A REALIZATION OF AN AUTOMATED DATA FLOW FOR DATA COLLECTING, PROCESSING, STORING AND RETRIEVING: GEONET
Explore the source record for details and available documents.
Friedsam, H.↗
SEARCH · Engineering Papers
Results for “data flow”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Processing Particle Data Flows with SmartNICs
Many distributed applications implement complex data flows and need a flexible mechanism for routing data between producers and consumers. Recent advances in programmable network interface cards, or SmartNICs, represent an opportunity to offload data-flow tasks into the network fabric, thereby freeing the hosts to perform other work. System architects in this space face multiple questions about the best way to leverage SmartNICs as processing elements in data flows. In this paper, we advocate the use of Apache Arrow as a foundation for implementing data-flow tasks on SmartNICs. We report on our experiences adapting a partitioning algorithm for particle data to Apache Arrow and measure the on-card processing performance for the BlueField-2 SmartNIC. Our experiments confirm that the BlueField-2’s (de)compression hardware can have a significant impact on in-transit workflows where data must be unpacked, processed, and repacked.
97 MATHEMATICS AND COMPUTING↗
Blood flow imaging by optimal matching of computational fluid dynamics to 4D-flow data
Three-dimensional, time-resolved blood flow measurement (4D-flow) is a powerful research and clinical tool, but improved resolution and scan times are needed. Therefore, this study aims to (1) present a postprocessing framework for optimization-driven simulation-based flow imaging, called 4D-flow High-resolution Imaging with a priori Knowledge Incorporating the Navier-Stokes equations and the discontinuous Galerkin method (4D-flow HIKING), (2) investigate the framework in synthetic tests, (3) perform phantom validation using laser particle imaging velocimetry, and (4) demonstrate the use of the framework in vivo. An optimizing computational fluid dynamics solver including adjoint-based optimization was developed to fit computational fluid dynamics solutions to 4D-flow data. Synthetic tests were performed in 2D, and phantom validation was performed with pulsatile flow. Reference velocity data were acquired using particle imaging velocimetry, and 4D-flow data were acquired at 1.5 T. In vivo testing was performed on intracranial arteries in a healthy volunteer at 7 T, with 2D flow as the reference. Results Synthetic tests showed low error (0.4%-0.7%). Phantom validation showed improved agreement with laser particle imaging velocimetry compared with input 4D-flow in the horizontal (mean -0.05 vs -1.11 cm/s, P < .001; SD 1.86 vs 4.26 cm/s, P < .001) and vertical directions (mean 0.05 vs -0.04 cm/s, P = .29; SD 1.36 vs 3.95 cm/s, P < .001). In vivo data show a reduction in flow rate error from 14% to 3.5%. Phantom and in vivo results from 4D-flow HIKING show promise for future applications with higher resolution, shorter scan times, and accurate quantification of physiological parameters.
4D-flow MRI↗
Creating an Interprocedural Analyst-Oriented Data Flow Representation for Binary Analysts (CIAO)
National security missions require understanding third-party software binaries, a key element of which is reasoning about how data flows through a program. However, vulnerability analysts protecting software lack adequate tools for understanding data flow in binaries. To reduce the human time burden for these analysts, we used human factors methods in a rolling discovery process to derive user-centric visual representation requirements. We encountered three main challenges: analysis projects span weeks, analysis goals significantly affect approaches and required knowledge, and analyst tools, techniques, conventions, and prioritization are based on personal preference. To address these challenges, we initially focused our human factors methods on an attack surface characterization task. We generalized our results using a two-stage modified sorting task, creating requirements for a data flow visualization. We implemented these requirements partially in manual static visualizations, which we informally evaluated, and partially in automatically generated interactive visualizations, which have yet to be integrated into workflows for evaluation. Our observations and results indicate that 1) this data flow visualization has the potential to enable novel code navigation, information presentation, and information sharing, and 2) it is an excellent time to pursue research applying human factors methods to binary analysis workflows.
42 ENGINEERING↗
Sap flow data from San Lorenzo, Panama (PA-SLZ), 2019
This data package contains sap flow data from the NGEE Tropics site in San Lorenzo, Panama (PA-SLZ) from 22 January 2019 to 31 December 2019. Sap flow measurements quantify plant-level transpiration, which varies across spatial and temporal scales. It is a key trait in understanding plant hydraulic functioning. Here sap flow was measured using ‘Granier-type’ thermal dissipation probes. Raw data downloaded from each data logger are in the 'Raw data' folder as CSV files, along with the Campbell data logger programs which can be read when opened with a text editor. The data was processed using the AquaFlux package in R. R code for preparing the data for use in AquaFlux is in the 'R code for pre-processing data folder'; it is saved as a text file that can be copy and pasted into R. The processed data for each data logger is in the 'Processed data' folder as CSV files. Documentation for the Granier sensors and AquaFlux package is in the ‘Manuals’ folder as PDF files.
54 ENVIRONMENTAL SCIENCES↗
Detector architecture and data flow of the CMS HGCAL Phase-2 Endcap Calorimeter Upgrade
The CMS High Granularity Calorimeter (HGCAL) is the Phase-2 endcap calorimeter upgrade for the High-Luminosity Large Hadron Collider, designed to provide precise spatial, energy, and timing measurements in a high-radiation, high-pileup environment. This work presents an overview of the HGCAL assembly process, detector architecture, and end-to-end data flow. The data flow goes from signal generation in silicon sensors and scintillator tiles through front-end electronics, optical transmission, and back-end data acquisition. The study highlights how the integrated detector, electronics, and readout systems enable efficient event reconstruction and support the physics objectives of the HL-LHC.
Osei Boakye, Prince [Grambling State U.]↗
A Network Management System for Handling Scientific Data Flows
Large scientific data transfers often occur at high rates causing increased burstiness in Internet traffic. To limit the adverse effects of these high-rate large-sized flows, which are referred to as $\alpha $ flows, on delay-sensitive audio/video flows, a network management system called Alpha Flow Traffic Engineering System (AFTES) is proposed for intra-domain traffic engineering. An offline approach is used in which AFTES analyzes NetFlow records collected by routers, extracts source–destination address prefixes of $\alpha $ flows, and uses these prefixes to configure firewall filters at ingress routers of a provider’s network to redirect future $\alpha $ flows to traffic-engineered paths and isolated queues. The effectiveness of this scheme was evaluated through an analysis of 7 months of NetFlow data obtained from an ESnet router. For this data set, 91 % of bytes generated by $\alpha $ flows during high-rate intervals would have been directed had AFTES been deployed. The negative aspect of using address prefixes in firewall filters, i.e., the redirection of $\beta $ flows to $\alpha $-flow paths/queues, was also quantified.
97 MATHEMATICS AND COMPUTING↗
A mathematical model of asynchronous data flow in parallel computers *
Abstract We present a simplified model of data flow on processors in a high-performance computing framework involving computations necessitating inter-processor communications. From this ordinary differential model, we take its asymptotic limit, resulting in a model which treats the computer as a continuum of processors and data flow as an Eulerian fluid governed by a conservation law. We derive a Hamilton–Jacobi equation associated with this conservation law for which the existence and uniqueness of solutions can be proven. We then present the results of numerical experiments for both discrete and continuum models; these show a qualitative agreement between the two and the effect of variations in the computing environment’s processing capabilities on the progress of the modelled computation.
Barnard, Richard C.↗
File-Based Data Flow in the CMS Filter Farm
During the LHC Long Shutdown 1, the CMS Data Acquisition system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and prepare the ground for future upgrades of the detector front-ends. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. This approach provides additional decoupling between the HLT algorithms and the input and output data flow. All the metadata needed for bookkeeping of the data flow and the HLT process lifetimes are also generated in the form of small “documents” using the JSON encoding, by either services in the flow of the HLT execution (for rates etc.) or watchdog processes. These “files” can remain memory-resident or be written to disk if they are to be used in another part of the system (e.g. for aggregation of output data). We discuss how this redesign improves the robustness and flexibility of the CMS DAQ and the performance of the system currently being commissioned for the LHC Run 2.
Andre, J. M.↗
Pass-efficient methods for compression of high-dimensional turbulent flow data
The future of high-performance computing, specifically on future Exascale computers, will presumably see memory capacity and bandwidth fail to keep pace with data generated, for instance, from massively parallel partial differential equation (PDE) systems. Current strategies proposed to address this bottleneck entail the omission of large fractions of data, as well as the incorporation of in situ compression algorithms to avoid overuse of memory. To ensure that post-processing operations are successful, this must be done in a way that a sufficiently accurate representation of the solution is stored. Moreover, in situations where the input/output system becomes a bottleneck in analysis, visualization, etc., or the execution of the PDE solver is expensive, the number of passes made over the data must be minimized. In the interest of addressing this problem, this work focuses on the utility of pass-efficient, parallelizable, low-rank, matrix decomposition methods in compressing high-dimensional simulation data from turbulent flows. Additionally, a particular emphasis is placed on using coarse representation of the data – compatible with the PDE discretization grid – to accelerate the construction of the low-rank factorization. This includes the presentation of a novel single-pass matrix decomposition algorithm for computing the so-called interpolative decomposition. The methods are described extensively and numerical experiments on two turbulent channel flow data are performed. In the first (unladen) channel flow case, compression factors exceeding 400 are achieved while maintaining accuracy with respect to first- and second-order flow statistics. In the particle-laden case, compression factors of 100 are achieved and the compressed data is used to recover particle velocities. These results show that these compression methods can enable efficient computation of various quantities of interest in both the carrier and disperse phases.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Discrete Regularization for Calibration of Geologic Facies Against Dynamic Flow Data
Abstract Subsurface flow model calibration involves many more unknowns than measurements, leading to ill‐posed problems with nonunique solutions. To alleviate nonuniqueness, the problem is regularized by constraining the solution space using prior knowledge. In certain sedimentary environments, such as fluvial systems, the contrast in hydraulic properties of different facies types tends to dominate the flow and transport behavior, making the effect of within facies heterogeneity less significant. Hence, flow model calibration in those formations reduces to delineating the spatial structure and connectivity of different lithofacies types and their boundaries. A major difficulty in calibrating such models is honoring the discrete, or piecewise constant, nature of facies distribution. The problem becomes more challenging when complex spatial connectivity patterns with higher‐order statistics are involved. This paper introduces a novel formulation for calibration of complex geologic facies by imposing appropriate constraints to recover plausible solutions that honor the spatial connectivity and discreteness of facies models. To incorporate prior connectivity patterns, plausible geologic features are learned from available training models. This is achieved by learning spatial patterns from training data, e.g., ‐SVD sparse learning or the traditional Principal Component Analysis. Discrete regularization is introduced as a penalty functions to impose solution discreteness while minimizing the mismatch between observed and predicted data. An efficient gradient‐based alternating directions algorithm is combined with variable splitting to minimize the resulting regularized nonlinear least squares objective function. Numerical results show that imposing learned facies connectivity and discreteness as regularization functions leads to geologically consistent solutions that improve facies calibration quality.
Khaninezhad, Mohammad‐Reza↗
GPU-Based Static Data-Flow Analysis for Fast and Scalable Android App Vetting
Many popular vetting tools for Android applications use static code analysis techniques. In particular, Interprocedural Data-Flow Graph (IDFG) construction is the computation at the core of Android static data-flow analysis and consumes most of the analysis time. Many analysis tools use a worklist algorithm, an iterative fixed-point approach, to construct the IDFG. In this paper, we observe that a straightforward GPU parallelization of the worklist algorithm leads to significant underutilization of the GPU resources. We identify four performance bottlenecks, namely, frequent dynamic memory allocations, high branch divergence, workload imbalance, and irregular memory access patterns. Accordingly, we propose GDroid, a GPU-based worklist algorithm implementation with multiple fine-grained optimizations tailored to common characteristics of Android applications. The optimizations considered are: matrix-based data structure, memory access-based node grouping, and worklist merging. Our experimental evaluation, performed on 1000 Android applications, shows that the proposed optimizations are beneficial to performance, and GDroid can achieve up to 128X speedups against a plain GPU implementation.
Android security↗
ARM: Aerosol Observing System (AOS): Air Flow Data 3
Aerosol Observing System (AOS): Air Flow Data 3
54 ENVIRONMENTAL SCIENCES↗
Data-flow parallelism for high-energy and nuclear physics computing frameworks
The processing tasks of a scientific workflow in high-energy and nuclear physics (HENP) can typically be represented as a directed acyclic graph formed according to the data flow—i.e. the data dependencies among algorithms executed as part of the workflow. With this representation, an HENP computing framework can optimally execute a workflow, exploiting the parallelism inherent among independent tasks. Despite such a natural description of a workflow, most HENP frameworks do not make use of technologies that provide concurrent execution of graph-based tasking structures. In this session, we describe Fermilab efforts to adopt a graph-based technology (specifically Intel’s oneTBB flow graph) for meeting the framework needs of its experiments, notably DUNE. After introducing the physics DUNE intends to explore, we will show that all common processing idioms supported by current HENP frameworks can naturally be supported by oneTBB’s data-flow technology, optimally leveraging the concurrent capabilities of the machine. In addition, we discuss collaborative efforts between Fermilab and the Intel oneTBB development team, who is considering improvements to the flow-graph technology to better support HENP use cases.
43 PARTICLE ACCELERATORS↗
Improving I/O-aware Workflow Scheduling via Data Flow Characterization and trade-off Analysis
The scientific computing paradigm has transitioned from compute-intensive to I/O-intensive and memory-intensive in the past decade, especially when data-driven science has become common practice. Numerous empirical I/O-aware scheduling optimizations have been developed by incorporating I/O capacity and bandwidth as constraints into scheduling. Unfortunately, there is a lack of data flow (I/O) characterization tool and an understanding of trade-offs between concurrency, locality, and I/O bandwidth. To bridge the gap, this work 1) presents a set of descriptors to characterize, organize, and visualize I/O profiles, including flow size, I/O bandwidth, and operation count, which group data flows by I/O types, tasks, and files; 2) proposes an I/O Roofline model-based trade-off analysis to find the optimal trade-off between flow operational intensity, concurrency, and flow performance. The I/O descriptors generate useful insights into complicated I/O behaviors, suggesting distinct concurrency, storage, and scheduling to be used by types, tasks, and files. The proposed trade-off analysis guides scheduling decisions that generate resource assignment with the best flow parallelism. We evaluate our I/O-aware scheduling methodology on a highly I/O-intensive workflow–1000 Genomes. The experimental results demonstrate speedups of up to 2.4× compared to the state-of-the- art methods.
Guo, Luanzheng [BATTELLE (PACIFIC NW LAB)]↗
Data-flow parallelism for high-energy and nuclear physics frameworks
The processing tasks of an event-processing workflow in high-energy and nuclear physics (HENP) can typically be represented as a directed acyclic graph formed according to the data flow—i.e. the data dependencies among algorithms executed as part of the workflow. With this representation, an HENP framework can optimally execute a workflow, exploiting the parallelism inherent among independent tasks. Despite such a natural description of a workflow, most HENP frameworks do not make use of technologies that provide concurrent execution of graph-based tasking structures. In this talk, we describe Fermilab efforts to adopt a graph-based technology (specifically Intel’s oneTBB flow graph) for meeting the framework needs of its experiments, notably DUNE. Building on the Meld project as presented at CHEP2023, we demonstrate that all common processing idioms supported by current frameworks can naturally be supported by oneTBB’s data-flow technology, optimally leveraging the concurrent capabilities of the machine. In addition, we discuss collaborative efforts between Fermilab and the Intel oneTBB development team, who is considering improvements to the flow-graph technology to better support HENP use cases.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Qualitative properties of mathematical model for data flow
In this paper, properties of a recently proposed mathematical model for data flow in large-scale asynchronous computer systems are analyzed. In particular, the existence of special weak solutions based on propagating fronts is established. Qualitative properties of these solutions are investigated, both theoretically and numerically.
97 MATHEMATICS AND COMPUTING↗