SEARCH · Engineering Papers

Results for “Data Movement”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

All sources NASA NTRS DOE OSTI

DOE OSTI2019

GROWTH on S190426c: Real-time Search for a Counterpart to the Probable Neutron Star–Black Hole Merger using an Automated Difference Imaging Pipeline for DECam

The discovery of a transient kilonova following the gravitational-wave (GW) event GW170817 highlighted the critical need for coordinated rapid and wide-field observations, inference, and follow-up across the electromagnetic spectrum. In the southern hemisphere, the Dark Energy Camera (DECam) on the Blanco 4 m telescope is well suited to this task, as it is able to cover wide fields quickly while still achieving the depths required to find kilonovae like the one accompanying GW170817 to ~500 Mpc, the binary neutron star (NS) horizon distance for current generation of LIGO/Virgo collaboration (LVC) interferometers. Here, as part of the multi-facility follow-up by the Global Relay of Observatories Watching Transients Happen collaboration, we describe the observations and automated data movement, data reduction, candidate discovery, and vetting pipeline of our target-of-opportunity DECam observations of S190426c, the first possible NS-black hole merger detected in GWs. Starting 7.5 hr after S190426c, over 11.28 hr of observations, we imaged an area of 525 deg 2 ( r band) and 437 deg 2 ( z band); this was 16.3% of the total original localization probability, and nearly all of the probability visible from the southern hemisphere. The machine-learning-based pipeline was optimized for fast turnaround, delivering transients for human vetting within 17 minutes, on average, of shutter closure. We reported nine promising counterpart candidates 2.5 hr before the end of our observations. One hour after our data-taking ended (roughly 20 hr after the announcement of S190426c), LVC released a refined skymap that reduced the probability coverage of our observations to 8.0%, demonstrating a critical need for localization updates on shorter (~hour) timescales. Our observations yielded no detection of a bona fide counterpart to m z = 21.7 and m r = 22.2 at the 5σ level of significance, consistent with the refined LVC positioning. Here, we view these observations and rapid inferencing as an important real-world test for this novel end-to-end wide-field pipeline.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2018

Complex Network Analysis and Intelligent Monitoring Platform

Data Products has particular interest in developing an efficient network analysis and monitoring system that utilizes advanced machine learning methods to mine complex network data of the type generated within large data centers, with an objective to provide tangible business-usable intelligence. The Laboratory is currently the lead institution for a U.S. Department of Energy Office of Science (DOE/SC) network research project called BigData Express (BDE). The project seeks to provide an orchestration service for high-performance data movement of data sets for large-scale science. This collaboration is believed to potentially be a highly symbiotic one, with Data Products platform development benefiting from the rich sets of network data available in the BDE development environment, and BDE benefiting from the Data Products machine learning analysis of BDE-generated data movement.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

Parallel performance optimizations on unstructured mesh-based simulations

This paper addresses two key parallelization challenges the unstructured mesh-based ocean modeling code, MPAS-Ocean, which uses a mesh based on Voronoi tessellations: (1) load imbalance across processes, and (2) unstructured data access patterns, that inhibit intra- and inter-node performance. Our work analyzes the load imbalance due to naive partitioning of the mesh, and develops methods to generate mesh partitioning with better load balance and reduced communication. Furthermore, we present methods that minimize both inter- and intranode data movement and maximize data reuse. Our techniques include predictive ordering of data elements for higher cache efficiency, as well as communication reduction approaches. We present detailed performance data when running on thousands of cores using the Cray XC30 supercomputer and show that our optimization strategies can exceed the original performance by over 2×. Additionally, many of these solutions can be broadly applied to a wide variety of unstructured grid-based computations.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

Compendium of Tour-Based Freight Modeling Literature

Transportation officials, metropolitan planning organizations (MPOs), and companies of varying size can all benefit from freight movement data. Obtaining freight movement data and route choice information is crucial to planning, improving current and future transportation infrastructure, locating businesses, improving efforts to streamline goods movement, especially urban goods movement, and reducing energy usage within freight transportation. Obtaining data on freight movements can also help in better planning transportation infrastructure to improve safety. Also, with increasing demands and changing consumption habits, mobility being offered as a service, and with new technologies such as 3-D printing, the landscape of freight planning is changing, as businesses compete to meet the needs of a growing customer base, while transportation planners aim at meeting these needs by maintaining and updating the transportation infrastructure.For decades, transportation research has focused on passenger travel behavior. Much of this research has examined route decision choice and, more recently, trip chaining of personal vehicles, both individually and collectively. Discrete choice modeling has commonly been used to better understand passenger movement and predict route decisions based on geographic and demographic data from Census data and personal surveys, and micro-simulation has been used to model estimated movements through the transportation network. However, there is a growing interest in examining freight movements, and although previous research has been done to look at freight route choice regarding individual trips, there is increasing interest in freight tours. This paper is an overview of the current tour-based freight modeling methodology found in the literature. Background information on tour-based freight modeling is provided, as well as common modeling methods, data sources, and findings. Consideration is given to new technologies and safety considerations excluded from the current literature. Suggestions are provided for future tour-based freight modeling studies.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2024

Network performance analysis for HPC datacenters (net_perf) v1.0

The software has two main features: (1) identify data movement trends in HPC data centers that use network flow monitoring (2) analyze the performance of individual data flows under the existing data movement management strategy and identify performance bottlenecks that impede timely data availability for science workflows. Its main advantage is that it is tailored for HPC network traffic by considering HPC data movement management intricacies.

Giannakou, Anna↗

DOE OSTI2021

RISE: Reducing I/O Contention in Staging-based Extreme-Scale In-situ Workflows

While in-situ workflow formulations have addressed some of the data-related challenges associated with extreme-scale scientific workflows, these workflows involve complex interactions and different modes of data exchange. In the context of increasing system complexity, such workflows present significant resource management challenges, requiring complex cost-performance tradeoffs. This paper presents RISE, an intelligent staging-based data management middleware, which builds on the DataSpaces framework and performs intelligent scheduling of data management operations to reduce I/O contention. In RISE, data are always written immediately to local buffers to reduce the effect of the transfer impact upon application performance. RISE identifies applications’ data access patterns and moves data towards data consumers only when the network is expected to be idle, reducing the impact of asynchronous background data movement upon critical data read/write requests. Here, we experimentally demonstrate that RISE can take advantage of staging nodes to offload data during writes without degrading application data movement performance.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

AnalyzeThis: An Analysis Workflow-Aware Storage System

The need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices enable processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrained in each and every layer of the storage, elevating data analyses as first-class citizens, and transforming AnalyzeThis into a potent analytics-aware appliance. We implement the AnalyzeThis storage system atop an emulation platform of the Active Flash array. Our results indicate that AnalyzeThis is viable, expediting workflow execution and minimizing data movement.

Data Analytics↗

DOE OSTI2018

An Analysis Workflow-Aware Storage System for Multi-Core Active Flash Arrays

Here, the need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices enable processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrained in each and every layer of the storage, elevating data analyses as first-class citizens, and transforming AnalyzeThis into a potent analytics-aware appliance. To evaluate the AnalyzeThis system, we have adopted both emulation and simulation approaches. In particular, we have evaluated AnalyzeThis by implementing the AnalyzeThis storage system on top of the Active Flash Array's emulation platform. We have also implemented an event-driven AnalyzeThis simulator, called AnalyzeThisSim, which allows us to address the limitations of the emulation platform, e.g., performance impact of using multi-core SSDs. The results from our emulation and simulation platforms indicate that AnalyzeThis is a viable approach for expediting workflow execution and minimizing data movement.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2015

AnalyzeThis: an analysis workflow-aware storage system

The need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices en-able processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrained in each and every layer of the storage, elevating data analyses as first-class citizens, and transforming AnalyzeThis into a potent analytics-aware appliance. We implement the AnalyzeThis storage system atop an emulation platform of the Active Flash array. Our results indicate that AnalyzeThis is viable, expediting workflow execution and minimizing data movement.

Computer Science↗

DOE OSTI2011

Resilient data staging through MxN distributed transactions.

Scientific computing-driven discoveries are frequently driven from workflows that use persistent storage as a staging area for data between operations. With the bad and progressively worse bandwidth vs. data size issues as we continue towards exascale, eliminating persistent storage through techniques like data staging will both enable these workflows to continue online, but also enable more interactive workflows reducing the time to scientific discoveries. Data staging has shown to be an effective way for applications running on high-end computing platforms to offload expensive I/O operations and to manage the tremendous amounts of data they produce. This data staging approach, however, lacks the ACID style guarantees traditional straight-to-disk methods provide. Distributed transactions are a proven way to add ACID properties to data movements, however distributed transactions follow 1xN data movement semantics, where our highly parallel HPC environments employ MxN data movement semantics. In this paper we present a novel protocol that extends distributed transaction terminology to include MxN semantics which allows our data staging areas to benefit from ACID properties. We show that with our protocol we can provide resilient data staging with a limited performance penalty over current data staging implementations.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2016

Characterization of bandwidth requirements of algorithms for extreme scale science

Technology trends have resulted in widely different rates of improvement in computational throughput as compared to data movement rates in computer systems. With future systems, the cost of data movement through the memory hierarchy is expected to become even more dominant relative to the cost of performing arithmetic operations, both in terms of time and energy. Therefore, optimizing data access costs will become ever more critical in the coming years. However, unlike the well understood model of computational complexity in terms of the number of elementary operations to execute an algorithm, the inherent data-movement complexity of computations is unknown for most algorithms. Given the crucial importance of optimizing data access costs in scalable parallel systems, it is of great interest to develop techniques for understanding the inherent data movement complexity of an algorithm. In this project, we focus on this fundamental and significant problem.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2014

The 'last mile' of data handling: Fermilab's IFDH tools

IFDH (Intensity Frontier Data Handling), is a suite of tools for data movement tasks for Fermilab experiments and is an important part of the FIFE[2] (Fabric for Intensity Frontier [1] Experiments) initiative described at this conference. IFDH encompasses moving input data from caches or storage elements to compute nodes (the 'last mile' of data movement) and moving output data potentially to those caches as part of the journey back to the user. IFDH also involves throttling and locking to ensure that large numbers of jobs do not cause data movement bottlenecks. IFDH is realized as an easy to use layer that users call in their job scripts (e.g. 'ifdh cp'), hiding the low level data movement tools. One advantage of this layer is that the underlying low level tools can be selected or changed without the need for the user to alter their scripts. Logging and performance monitoring can also be added easily. This system will be presented in detail as well as its impact on the ease of data handling at Fermilab experiments.

Lyon, Adam L.↗

DOE OSTI2019

Data Jockey: Automatic Data Management for HPC Multi-Tiered Storage Systems

We present the design and implementation of Data Jockey, a data management system for HPC multi-tiered storage systems. As a centralized data management control plane, Data Jockey automates bulk data movement and placement for scientific workflows and integrates into existing HPC storage infrastructures. Data Jockey simplifies data management by eliminating human effort in programming complex data movements, laying datasets across multiple storage tiers when supporting complex workflows, which in turn increases the usability of multitiered storage systems emerging in modern HPC data centers.Specifically, Data Jockey presents a new data management scheme called “goal driven data management” that can automatically infer low-level bulk data movement plans from declarative high-level goal statements that come from the lifetime of iterative runs of scientific workflows. While doing so, Data Jockey aims to minimize data wait times by taking responsibility for datasets that are unused or to be used, and aggressively utilizing the capacity of the upper, higher performant storage tiers.We evaluated a prototype implementation of Data Jockey under a synthetic workload based on a year’s worth of Oak Ridge Leadership Computing Facility’s (OLCF) operational logs. Our evaluations suggest that Data Jockey leads to higher utilization of the upper storage tiers while minimizing the programming effort of data movement compared to human involved, per-domain adhoc data management scripts.

Shin, Woong↗

DOE OSTI2021

When to use rsync

We have endeavored to show, using a series of data transfer results obtained from two testbeds, when to use the popular data copying tool rsync and related tools. Tests have been conducted in local area network (LAN) and wide area network (WAN) environments. We conclude that for files in a certain size range and network latency &lE; 10 ms round trip time (RTT), rsync is still useful for data moving tasks in the category 4 of the U.S. DOE Technical Report “Data Movement Categories”. For more demanding data movement requirements, tools of different classes are suggested. Sample histograms from two DOE user facilities are provided to further support our conclusions.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

Benchmark Generation and Simulation at Extreme Scale

The path to extreme scale high-performance computing (HPC) poses several challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Investigating the performance of parallel applications at scale on future architectures and the performance impact of different architectural choices is an important component of HPC hardware/software co-design. Simulations using models of future HPC systems and communication traces from applications running on existing HPC systems can offer an insight into the performance of future architectures. This work targets technology developed for scalable application tracing of communication events. It focuses on extreme-scale simulation of HPC applications and their communication behavior via lightweight parallel discrete event simulation for performance estimation and evaluation. Instead of simply replaying a trace within a simulator, this work promotes the generation of a benchmark from traces. This benchmark is subsequently exposed to simulation using models to reflect the performance characteristics of future-generation HPC systems. This technique provides a number of benefits, such as eliminating the data intensive trace replay and enabling simulations at different scales. The presented work features novel software co-design aspects, combining the ScalaTrace tool to generate scalable trace files, the ScalaBenchGen tool to generate the benchmark, and the xSim tool to assess the benchmark characteristics within a simulator.

Lagadapati, Mahesh↗

DOE OSTI2019

From Data to Models and Analytics: An Integrated Scientific Computing Platform to Accelerate Environmental Research SBIR Phase I (Final Technical Report)

Phase I research was conducted to investigate the technical feasibility of an integrated open source platform GeoLib to provide turnkey solutions for model-data integration for environmental systems science (ESS), using modern web technologies, advances in HPC and cloud computing and high-performance visualization. A key feature of our platform is that it minimizes and automates data movement between data storage and computing systems. Another critical feature of the design is that it enables users to perform high-fidelity post-processing by performing high-performance visualization server-side and in-situ on the compute cluster close to the data. The investigation, which was performed over a 12-month period ending April 2019, centered on developing prototype software to demonstrate the key enabling technologies required for this system. This included: (i) rapid access to data from multiple remote sources, via a unified data interface; (ii) software for scalable geospatial visualization of raster, vector, and point cloud data on the web; (iii) development of an easy-to-use API for creating simulation and modeling workflows; (iv) remote data visualization and processing on middle-tier servers; and (v) remote data processing on high-performance computing systems. A critical part of this work is that all of these capabilities be accessible to end users through standard desktop tools such as Project Jupyter, which is widely adopted for ESS simulation and modeling.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2013

Tools for Simulation and Benchmark Generation at Exascale

The path to exascale high-performance computing (HPC) poses several challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Investigating the performance of parallel applications at scale on future architectures and the performance impact of different architecture choices is an important component of HPC hardware/software co-design. Simulations using models of future HPC systems and communication traces from applications running on existing HPC systems can offer an insight into the performance of future architectures. This work targets technology developed for scalable application tracing of communication events and memory profiles, but can be extended to other areas, such as I/O, control flow, and data flow. It further focuses on extreme-scale simulation of millions of Message Passing Interface (MPI) ranks using a lightweight parallel discrete event simulation (PDES) toolkit for performance evaluation. Instead of simply replaying a trace within a simulation, the approach is to generate a benchmark from it and to run this benchmark within a simulation using models to reflect the performance characteristics of future-generation HPC systems. This provides a number of benefits, such as eliminating the data intensive trace replay and enabling simulations at different scales. The presented work utilizes the ScalaTrace tool to generate scalable trace files, the ScalaBenchGen tool to generate the benchmark, and the xSim tool to run the benchmark within a simulation.

Lagadapati, Mahesh↗

DOE OSTI2025

IRIS-MEMFLOW: Data Flow-Enabled Portable Memory Orchestration in IRIS Runtime for Diverse Heterogeneity

Task-based programming models and execution paradigms provide a means to decompose a computation by expressing it as a graph in which each node represents a specific computation operating on memory objects and the edges define the dependencies in the execution flow. In this execution model, independent nodes in the graph can be executed concurrently in different computing devices, making it suitable for heterogeneous systems in which computing devices with different architectures coexist. However, careful memory orchestration across heterogeneous devices is needed because copies of the same memory object may reside in multiple devices during execution. Manually ensuring such an orchestration is quite challenging. Not only must an application developer guard against race conditions, but they must also optimize data movement between the host and devices because unnecessary data movement significantly impacts performance. To mitigate these challenges, we enhance the IRIS heterogeneous runtime and introduce IRIS-MEMFLOW–a data flow–enabled portable memory abstraction for seamlessly orchestrating memory in diverse heterogeneous computing environments. By using data-flow analysis, IRIS-MEMFLOW guards against race conditions while multiple heterogeneous devices access memory objects. IRIS-MEMFLOW also optimizes data movement between the host and devices without manual intervention. As a result, IRIS provides improved programming productivity, performance, and portability for multidevice heterogeneous executions in high-performance computing and cloud systems that run diverse architectures from different vendors. The efficacy of IRIS-MEMFLOW is evaluated through experiments that show its capability in terms of programming productivity, multidevice heterogeneity, portability, and low overhead versus the state of the art.

Monil, M. A. H. [ORNL] (ORCID:0000000334194037)↗