Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data sharing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52

Status of DUNE Offline Computing

We summarize the status of Deep Underground Neutrino Experiment (DUNE) Offline Software and Computing program. We describe plans for the computing infrastructure needed to acquire, catalog, reconstruct, simulate and analyze the data from the DUNE experiment and its prototypes in pursuit of the experiment's physics goals of precision measurements of neutrino oscillation parameters, detection of astrophysical neutrinos, measurement of neutrino interaction properties and searches for physics beyond the Standard Model. In contrast to traditional HEP computational problems, DUNE's Liquid Argon Time Projection Chamber data consist of simple but very large (many GB) data objects which share many characteristics with astrophysical images. We have successfully reconstructed and simulated data from 4% prototype detector runs at CERN. The data volume from the full DUNE detector, when it starts commissioning late in this decade will present memory management challenges in conventional processing but significant opportunities to use advances in machine learning and pattern recognition as a frontier user of High Performance Computing facilities capable of massively parallel processing. Our goal is to develop infrastructure resources that are flexible and accessible enough to support creative software solutions as HEP computing evolves.

43 PARTICLE ACCELERATORS↗

A Workflow-based Intelligent Network Data Movement Advisor with End-to-end Performance Optimization

Next-generation eScience applications often generate large amounts of simulation, experimental, or observational data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate such massive data transfer. However, these technologies and services have not been fully utilized mainly because their use typically requires significant domain knowledge and in many cases application users are even not aware of their existence. By leveraging the functionalities of an existing Network-Aware Data Movement Advisor (NADMA) utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization for this DOE funded project. This WINDMA system integrates three major components: resource discovery, data movement, and status monitoring, and supports the sharing of common data movement workflows through account and database management. This system provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. We demonstrate the efficacy of the proposed transport-support workflow system in several use cases based on its implementation and deployment in DOE wide-area networks.

97 MATHEMATICS AND COMPUTING↗

AiiDA: automated interactive infrastructure and database for computational science

Computational science has seen in the last decades a spectacular rise in the scope, breadth, and depth of its efforts. Notwithstanding this prevalence and impact, it is often still performed using the renaissance model of individual artisans gathered in a workshop, under the guidance of an established practitioner. Great benefits could follow instead from adopting concepts and tools coming from computer science to manage, preserve, and share these computational efforts. We illustrate here our paradigm sustaining such vision, based around the four pillars of Automation, Data, Environment, and Sharing. We then discuss its implementation in the open-source AiiDA platform (http://www.aiida.net), that has been tuned first to the demands of computational materials science. AiiDA’s design is based on directed acyclic graphs to track the provenance of data and calculations, and ensure preservation and searchability. Remote computational resources are managed transparently, and automation is coupled with data storage to ensure reproducibility. Last, complex sequences of calculations can be encoded into scientific workflows. We believe that AiiDA’s design and its sharing capabilities will encourage the creation of social ecosystems to disseminate codes, data, and scientific workflows.

Materials Science↗

A Practical Framework for Sharing and Rendering Real-World Bidirectional Scattering Distribution Functions

The utilization of real-world materials has been hindered by a lack of standards for sharing and interpreting measured data. This paper presents an XML representation and an Open Source C library to support bidirectional scattering distribution functions (BSDFs) in data-driven lighting simulation and rendering applications.The library provides for the efficient representation, query, and Monte Carlo sampling of arbitrary BSDFs in amodel-free framework. Currently, we support two BSDF data representations: one using a fixed subdivision of thehemisphere, and one with adaptive density. The fixed type has advantages for certain matrix operations, while theadaptive type can more accurately represent highly peaked data. We discuss advanced methods for data-drivenBSDF rendering for both types, including the proxy of detailed geometry to enhance appearance and accuracy.We also present an advanced interpolation method to reduce measured data into these standard representations.We end with our plan for future extensions and sharing of BSDF data.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

A Task Based Approach for Co-Scheduling Ensemble Workloads on Heterogeneous Nodes

Scientific workflows consist of multiple, connected applications, with data and results flowing from one to another in a pipeline. Traditionally, such workflows are executed in sequential order, storing intermediate data in storage disks. Co-scheduling application workflows concurrently on the same compute nodes would greatly reduce the cost of moving data to/from storage and allow real-time analysis of intermediate results. Nevertheless, most parallel programming runtimes do not allow seamless integration of various applications in a scientific workflow, in part due to the complexity of managing data and resources. The situation is even more complicated for heterogeneous systems. In this work we extend the Minos Computing Library (MCL) runtime to accelerate pipe-lined and parallel workloads where multiple applications are running in the same system. MCL’s asynchronous task library and runtime dynamically manages resources to allow co-scheduling of multiple processes sharing heterogeneous resources. In addition, we design a custom ex- tension of the Open Compute Language (OpenCL) to enable multiple processes to share device memory. We enable MCL to coordinate these shared buffers to allow for easy, fast data sharing between applications. Using malleable micro-benchmarks and two application workflows that combine scientific simulation and AI-based analysis, we show that our method outperforms traditional approaches.

Index Terms—Parallel systems, Scheduling and Task ↗

A Web-based Collaborative Tool for Mars Analog Data Exploration

Solving today's complex research and modeling challenges are dependent on our ability to discover, access, integrate, and share information from multiple sources. The planetary sciences community is no exception'; over the last few years, the need for data mining and exploration tools that can expedite comparative studies between Martian and terrestrial analogs sites and aid the interpretation of Mars data sets has become evident. Data sharing maximizes scientific return from studies and data sets.

Necsoiu, M.↗

Interactive Sectoring and Animation of Global Change Data

In order to analyze and share results of global change climate data sets, scientists require a venue in which to exchange their results. The perfect medium for these collaborative efforts is the world wide web. Intuitive and efficient user interfaces, and background processes were developed at the Global Hydrology and Climate Center to interactively view weather satellite, radar, global temperature anomaly, and model output data using the world wide web. These tools combine scripts, Java, and C code, which allows the end user to easily interact with data, to create high resolution sector images, and sectored animation sequences. This paper examines the architecture and interfaces which were designed at the Global Hydrology and Climate Center and how they are used for collaborative research.

Meyer, Paul J.↗

Shared prefetching to reduce execution skew in multi-threaded systems

Mechanisms are provided for optimizing code to perform prefetching of data into a shared memory of a computing device that is shared by a plurality of threads that execute on the computing device. A memory stream of a portion of code that is shared by the plurality of threads is identified. A set of prefetch instructions is distributed across the plurality of threads. Prefetch instructions are inserted into the instruction sequences of the plurality of threads such that each instruction sequence has a separate sub-portion of the set of prefetch instructions, thereby generating optimized code. Executable code is generated based on the optimized code and stored in a storage device. The executable code, when executed, performs the prefetches associated with the distributed set of prefetch instructions in a shared manner across the plurality of threads.

97 MATHEMATICS AND COMPUTING↗

Biomass Harmonization and SAR Analysis with the Multi-mission Algorithm and Analysis Platform (MAAP)

The Multi‐mission Algorithm and Analysis Platform (MAAP) is a collaborative effort between NASA and the European Space Agency (ESA) to support above ground biomass (AGB) research in an open science framework. MAAP brings together relevant data, algorithms, and computing capabilities in a common cloud environment to address the challenges of sharing and processing data from field, airborne and satellite measurements. MAAP was publicly released in October 2021, providing computing capabilities co-located with the data, a collaborative coding and analysis environment, and a set of interoperable tools and algorithms developed to support the estimation and visualization of data. MAAP has allowed scientists from both North America and Europe to collaborate on the generation and analysis/visualization of data derived from multiple, discipline-adjacent missions in an open, collaborative environment that has reached beyond traditional scientific investigation. MAAP has been used to support multiple scientific activities. To date, existing LiDAR data from multiple platforms has been calibrated with field measurements and combined for more comprehensive and accurate estimates of above ground biomass AGB; these LiDAR platforms include airborne (e.g. LVIS), the International Space Station (NASA’s Global Ecosystem Dynamics Investigation (GEDI), and satellites (e.g. ICESat-2). The current challenge is to effectively and seamlessly combine the aforementioned LiDAR-based data with new data sources such as P-band RADAR from ESA’s upcoming BIOMASS mission, existing ESA Sentinel-1 C-band SAR, and the 30 PB/yr of high cadence global coverage L-band SAR data from the upcoming NASA-ISRO SAR (NISAR) mission. Recent analysis using MAAP merged ICESat-2 and optical data (Harmonized Landsat Sentinel) produced the most comprehensively precise estimate of boreal-wide AGB to date. Another effort using MAAP is the production and open distribution of global comparisons of AGB map estimates, including from ICESat-2 and GEDI, to bolster stakeholder uptake for policy applications. These map estimates will feed into the Intergovernmental Panel on Climate Change (IPCC) database, likely aiding the next Global Carbon Stocktake of the UNFCCC. Furthermore, the biomass retrieval intercomparison exercise BRIX-2 could benefit from the MAAP providing standardized test cases (based on airborne campaign and spaceborne data) allowing the community to develop and apply retrieval algorithms based on these test cases, while forthcoming SAR data training curricula could also use the MAAP as a teaching and learning platform. The MAAP is meeting the challenges inherent in international, open science collaboration and large scale computing with a platform that is entirely open source and cloud native, using open standards for data access, manipulation, protocols, and formats. The MAAP data system consists of a dedicated data store whose data is indexed in an online catalog conforming to established metadata, application programmatic interfaces (APIs), and service interface standards, using an implementation of the open sourced NASA Common Metadata Repository. Federation of user identities allows users from either NASA or ESA to access and consume services from the other using a unified metadata catalog for the data utilized across the ESA and NASA MAAP platforms. Similarly, we are exploring how to increase interoperability to achieve a common approach to packaging, orchestrating and executing algorithms, with interoperable access to data for subsetting, fast browse, and cloud-optimized access, all using interoperable standards such as those from the Open Geospatial Consortium (OGC). Designed for interoperability, ESA and NASA utilize a common architecture for the software platform. It provides a cloud-based algorithm development environment (ADE) that enables scientists to develop algorithms collaboratively with access to the MAAP data catalog as well as other data archives. MAAP provides an Eclipse Che-based ADE supporting both Python and R languages, popular in this biomass community. Algorithms developed and containerized within the ADE can be deployed to run to thousands of computational nodes in the MAAP’s data processing system (DPS), dramatically speeding up processing and giving scientists a rapid, iterative turnaround of results. NASA’s implementation of the DPS is based on the Hybrid Science Data System (HySDS) framework, used by NASA flight projects to produce Earth science standard products.

cloud computing↗

Layout-aware I/O Scheduling for terabits data movement

Many science facilities, such as the Department of Energy's Leadership Computing Facilities and experimental facilities including the Spallation Neutron Source, Stanford Linear Accelerator Center, and Advanced Photon Source, produce massive amounts of experimental and simulation data. These data are often shared among the facilities and with collaborating institutions. Moving large datasets over the wide-area network (WAN) is a major problem inhibiting collaboration. Next-generation, terabit-networks will help alleviate the problem, however, the parallel storage systems on the endsystem hosts at these institutions can become a bottleneck for terabit data movement. The parallel storage system (PFS) is shared by simulation systems, experimental systems, analysis and visualization clusters, in addition to wide-area data movers. These competing uses often induce temporary, but significant, I/O load imbalances on the storage system, which impact the performance of all the users. The problem is a serious concern because some resources are more expensive (e.g. super computers) or have time-critical deadlines (e.g. experimental data from a light source), but parallel file systems handle all requests fairly even if some storage servers are under heavy load. This paper investigates the problem of competing workloads accessing the parallel file system and how the performance of wide-area data movement can be improved in these environments. First, we study the I/O load imbalance problems using actual I/O performance data collected from the Spider storage system at the Oak Ridge Leadership Computing Facility. Second, we present I/O optimization solutions with layout-awareness on end-system hosts for bulk data movement. With our evaluation, we show that our I/O optimization techniques can avoid the I/O congested disk groups, improving storage I/O times on parallel storage systems for terabit data movement.

Kim, Youngjae↗

Layout-Aware I/O Scheduling for Terabits Data Movement

Many science facilities, such as the Department of Energy s Leadership Computing Facilities and experimental facilities including the Spallation Neutron Source, Stanford Linear Accelerator Center, and Advanced Photon Source, produce massive amounts of experimental and simulation data. These data are often shared among the facilities and with collaborating institutions. Moving large datasets over the wide- area network (WAN) is a major problem inhibiting collaboration. Next- generation, terabit-networks will help alleviate the problem, however, the parallel storage systems on the end-system hosts at these institutions can become a bottleneck for terabit data movement. The parallel storage system (PFS) is shared by simulation systems, experimental systems, analysis and visualization clusters, in addition to wide-area data movers. These competing uses often induce temporary, but significant, I/O load imbalances on the storage system, which impact the performance of all the users. The problem is a serious concern because some resources are more expensive (e.g. super computers) or have time-critical deadlines (e.g. experimental data from a light source), but parallel file systems handle all requests fairly even if some storage servers are under heavy load. This paper investigates the problem of competing workloads accessing the parallel file system and how the performance of wide-area data movement can be improved in these environments. First, we study the I/O load imbalance problems using actual I/O performance data collected from the Spider storage system at the Oak Ridge Leadership Computing Facility. Second, we present I/O optimization solutions with layout-awareness on end-system hosts for bulk data movement. With our evaluation, we show that our I/O optimization techniques can avoid the I/O congested disk groups, improving storage I/O times on parallel storage systems for terabit data movement.

I/O and Storage↗

Measuring and Modeling Shared Visual Attention

Multi-person teams are sometimes responsible for critical tasks, such as flying an airliner. Here we present a method using gaze tracking data to assess shared visual attention, a term we use to describe the situation where team members are attending to a common set of elements in the environment. Gaze data are quantized with respect to a set of N areas of interest (AOIs); these are then used to construct a time series of N dimensional vectors, with each vector component representing one of the AOIs, all set to 0 except for the component corresponding to the currently fixated AOI, which is set to 1. The resulting sequence of vectors can be averaged in time, with the result that each vector component represents the proportion of time that the corresponding AOI was fixated within the given time interval. We present two methods for comparing sequences of this sort, one based on computing the time-varying correlation of the averaged vectors, and another based on a chi-square test testing the hypothesis that the observed gaze proportions are drawn from identical probability distributions.We have evaluated the method using synthetic data sets, in which the behavior was modeled as a series of activities, each of which was modeled as a first-order Markov process. By tabulating distributions for pairs of identical and disparate activities, we are able to perform a receiver operating characteristic (ROC) analysis, allowing us to choose appropriate criteria and estimate error rates.We have applied the methods to data from airline crews, collected in a high-fidelity flight simulator (Haslbeck, Gontar Schubert, 2014). We conclude by considering the problem of automatic (blind) discovery of activities, using methods developed for text analysis.

attention↗

Measuring and Modeling Shared Visual Attention

Multi-person teams are sometimes responsible for critical tasks, such as flying an airliner. Here we present a method using gaze tracking data to assess shared visual attention, a term we use to describe the situation where team members are attending to a common set of elements in the environment. Gaze data are quantized with respect to a set of N areas of interest (AOIs); these are then used to construct a time series of N dimensional vectors, with each vector component representing one of the AOIs, all set to 0 except for the component corresponding to the currently fixated AOI, which is set to 1. The resulting sequence of vectors can be averaged in time, with the result that each vector component represents the proportion of time that the corresponding AOI was fixated within the given time interval. We present two methods for comparing sequences of this sort, one based on computing the time-varying correlation of the averaged vectors, and another based on a chi-square test testing the hypothesis that the observed gaze proportions are drawn from identical probability distributions. We have evaluated the method using synthetic data sets, in which the behavior was modeled as a series of "activities," each of which was modeled as a first-order Markov process. By tabulating distributions for pairs of identical and disparate activities, we are able to perform a receiver operating characteristic (ROC) analysis, allowing us to choose appropriate criteria and estimate error rates. We have applied the methods to data from airline crews, collected in a high-fidelity flight simulator (Haslbeck, Gontar & Schubert, 2014). We conclude by considering the problem of automatic (blind) discovery of activities, using methods developed for text analysis.

attention↗

Measuring and Modeling Shared Visual Attention

Multi-person teams are sometimes responsible for critical tasks, such as flying an airliner. Here we present a method using gaze tracking data to assess shared visual attention, a term we use to describe the situation where team members are attending to a common set of elements in the environment. Gaze data are quantized with respect to a set of N areas of interest (AOIs); these are then used to construct a time series of N dimensional vectors, with each vector component representing one of the AOIs, all set to 0 except for the component corresponding to the currently fixated AOI, which is set to 1. The resulting sequence of vectors can be averaged in time, with the result that each vector component represents the proportion of time that the corresponding AOI was fixated within the given time interval.We present two methods for comparing sequences of this sort, one based on computing the time varying correlation of the averaged vectors, and another based on a chi-square test testing the hypothesis that the observed gaze proportions are drawn from identical probability distributions.We have evaluated the method using synthetic data sets, in which the behavior was modeled as a series of activities, each of which was modeled as a first-order Markov process. By tabulating distributions for pairs of identical and disparate activities, we are able to perform a receiver operating characteristic (ROC) analysis, allowing us to choose appropriate criteria and estimate error rates. Using these criteria, we have applied the methods to data from airline crews, collected in a high-fidelity flight simulator (Gontar Hoermann, 2014). We conclude by considering the problem of automatic (blind) discovery of activities, using methods developed for text analysis.

Mulligan, Jeffrey B.↗

THE INFLUENCE OF CHANGES IN NUCLEAR COVARIANCE DATA ON THE CALCULATION OF Ck FOR HIGHLY ENRICHED URANIUM SOLUTION SYSTEMS

Since their initial development nearly 20 years ago, sensitivity/uncertainty (S/U) techniques have been increasingly applied to criticality safety validation. These techniques can generally be applied quickly and for a range of purposes after the initial investment of calculating sensitivities for a safety analysis model and a range of potential benchmark experiments. This paper discusses these applications and how they have been affected by changes in covariance data libraries.Critical experiment selection is the most prominent area in which S/U methods impact validation efforts. These methods allow for rigorous calculation of similarity between safety analysis models and potential benchmarks. The most widely used of these methods is the integral parameter ck, although other integral parameters such as E and some non-integral parameters have also been explored. The ck parameter calculates the fraction of nuclear data–induced uncertainty shared between an application and an experiment; this metric is predicated on the assessment that nuclear data errors are the most likely source of computational bias. Therefore, values of ck are inherently dependent on the nuclear covariance library used to calculate them and inevitably change when these data change.Past studies have shown how much the ck values can vary, especially in difficult validation cases like burnup credit. These significant changes are the result of differences in the relative uncertainties of nuclides with large sensitivities, and they imply differences in benchmark applicability to the safety analysis model. This paper demonstrates that more common criticality safety models involving single fissile species have significantly less variability of ck between covariance libraries, thus enhancing confidence in the application of S/U methods for experiment selection in these scenarios. This paper presents case studies examining the impact of these changes on some hypothetical safety analysis systems.

Clarity, Justin↗

WiP Abstract: Edge-Based Privacy of Naturalistic Driving Data Collection

Collecting large driving datasets is important for data-driven transportation research and studied in naturalistic driving [3]. Due to the standard implementation of a Controller Area Network (CAN) bus for a vehicle’s inter-module communication, many off-the-shelf devices can easily transform a vehicle into a rich data collection utility [2]. Vehicles with Adaptive Cruise Control (ACC) are an example of a feature resulting in emergent traffic behavior when scaled [4]. While these utilities were designed with particular data use cases, data may be publicly shared to benefit other researchers through online tools like CyVerse [1]. However, such data should only be shared when any private information is removed. This private information may exist as a set of GPS coordinates, since the start and end points of a trip may designate a driver’s place of residence or work. When considering larger continuously-collected data sets with a focus on naturalistic driving, many drivers are needed to make data collection feasible. Removal of private information becomes much more of a challenge since every driver may uniquely define their geographic privacy. This project aims to build upon the foundation of libpanda [2] by adding features of edge-based data privacy enforcement. In it’s current form, libpanda uses a GPS in conjunction with a CAN interface to record data. Libpanda feature s a set of startup and shutdown scripts to perform automatic data collection and upload. With additional support hardware on a Raspberry Pi, the Pi can maintain power on vehicle shutdown to automatically upload data before shutdown.

Bunting, Matt↗

Nonparallel transcriptional divergence during parallel adaptation

Abstract How underlying mechanisms bias evolution toward predictable outcomes remains an area of active debate. In this study, we leveraged phenotypic plasticity and parallel adaptation across independent lineages of Trinidadian guppies ( Poecilia reticulata ) to assess the predictability of gene expression evolution during parallel adaptation. Trinidadian guppies have repeatedly and independently adapted to high‐ and low‐predation environments in the wild. We combined this natural experiment with a laboratory breeding design to attribute transcriptional variation to the genetic influences of population of origin and developmental plasticity in response to rearing with or without predators. We observed substantial gene expression plasticity, as well as the evolution of expression plasticity itself, across populations. Genes exhibiting expression plasticity within populations were more likely to also differ in expression between populations, with the direction of population differences more likely to be opposite those of plasticity. While we found more overlap than expected by chance in genes differentially expressed between high‐ and low‐predation populations from distinct evolutionary lineages, the majority of differentially expressed genes were not shared between lineages. Our data suggest alternative transcriptional configurations associated with shared phenotypes, highlighting a role for transcriptional flexibility in the parallel phenotypic evolution of a species known for rapid adaptation.

Fischer, Eva K.↗