Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

ShopperWorkerRatio

These data characterize spatial and temporal variation in the ratio between shoppers and workers in public places - points of interest - in the United States between 2019 and 2020. The underlying data on foot traffic to public places is collected by SafeGraph, a commercial data aggregator. We estimate the ratio of shoppers to workers based on recorded visit duration.These data may be useful for understanding how use of public spaces have changed during the Coronavirus pandemic, and more generally for understanding activity patterns in public.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2014

Collective input/output under memory constraints

Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.

collective input/output↗
DOE OSTI2022

The ABCflux database: Arctic–boreal CO 2 flux observations and ancillary information aggregated to monthly time steps across terrestrial ecosystems

Past efforts to synthesize and quantify the magnitude and change in carbon dioxide (CO 2 ) fluxes in terrestrial ecosystems across the rapidly warming Arctic–boreal zone (ABZ) have provided valuable information but were limited in their geographical and temporal coverage. Furthermore, these efforts have been based on data aggregated over varying time periods, often with only minimal site ancillary data, thus limiting their potential to be used in large-scale carbon budget assessments. To bridge these gaps, we developed a standardized monthly database of Arctic–boreal CO 2 fluxes (ABCflux) that aggregates in situ measurements of terrestrial net ecosystem CO 2 exchange and its derived partitioned component fluxes: gross primary productivity and ecosystem respiration. The data span from 1989 to 2020 with over 70 supporting variables that describe key site conditions (e.g., vegetation and disturbance type), micrometeorological and environmental measurements (e.g., air and soil temperatures), and flux measurement techniques. Here, we describe these variables, the spatial and temporal distribution of observations, the main strengths and limitations of the database, and the potential research opportunities it enables. In total, ABCflux includes 244 sites and 6309 monthly observations; 136 sites and 2217 monthly observations represent tundra, and 108 sites and 4092 observations represent the boreal biome. The database includes fluxes estimated with chamber (19 % of the monthly observations), snow diffusion (3 %) and eddy covariance (78 %) techniques. The largest number of observations were collected during the climatological summer (June–August; 32 %), and fewer observations were available for autumn (September–October; 25 %), winter (December–February; 18 %), and spring (March–May; 25 %). ABCflux can be used in a wide array of empirical, remote sensing and modeling studies to improve understanding of the regional and temporal variability in CO 2 fluxes and to better estimate the terrestrial ABZ CO 2 budget.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2015

SU-E-T-255: Development of a Michigan Quality Assurance (MQA) Database for Clinical Machine Operations

Purpose: A unified database system was developed to allow accumulation, review and analysis of quality assurance (QA) data for measurement, treatment, imaging and simulation equipment in our department. Recording these data in a database allows a unified and structured approach to review and analysis of data gathered using commercial database tools. Methods: A clinical database was developed to track records of quality assurance operations on linear accelerators, a computed tomography (CT) scanner, high dose rate (HDR) afterloader and imaging systems such as on-board imaging (OBI) and Calypso in our department. The database was developed using Microsoft Access database and visual basic for applications (VBA) programming interface. Separate modules were written for accumulation, review and analysis of daily, monthly and annual QA data. All modules were designed to use structured query language (SQL) as the basis of data accumulation and review. The SQL strings are dynamically re-written at run time. The database also features embedded documentation, storage of documents produced during QA activities and the ability to annotate all data within the database. Tests are defined in a set of tables that define test type, specific value, and schedule. Results: Daily, Monthly and Annual QA data has been taken in parallel with established procedures to test MQA. The database has been used to aggregate data across machines to examine the consistency of machine parameters and operations within the clinic for several months. Conclusion: The MQA application has been developed as an interface to a commercially available SQL engine (JET 5.0) and a standard database back-end. The MQA system has been used for several months for routine data collection.. The system is robust, relatively simple to extend and can be migrated to a commercial SQL server.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2021

A Multi-criteria CCUS Screening Evaluation of the Gulf of Mexico, USA - Supplementary Data

The Wendt et al. study aims to incorporate multiple and disparate carbon capture, utilization, and storage (CCUS) decision-making criteria into a systematic, quantitative analytical approach to help identify areas with potentially high suitability to serve as offshore CO2 storage or EOR regions. Spatially-distributed data from publicly-available sources within the Gulf of Mexico (GOM) study area (limited to federal waters; state waters were not evaluated) was compiled using the U.S. Department of Energy's (DOE) National Energy Technology Laboratory (NETL) Cumulative Spatial Impact Layers™ (CSIL) tool to easily aggregate data based on evenly-distributed grids across the study region set at a resolution of approximately 25 square miles (65 square kilometers). The data included in this Microsoft Excel™ workbook provide the aggregated scores for each grid point across the study domain as well the weighting for each criterion under the four scenarios evaluated.

CCUS↗
DOE OSTI2019

A Comparative Study of the Efficacy of Local/Global and Parametric/Nonparametric Machine Learning Methods for Establishing Structure–Property Linkages in High-Contrast 3D Elastic Composites

Reduced-order structure–property (S-P) linkages play a pivotal role in the tailored design of materials for advanced engineering components. There is a critical need to distill these from the simulation datasets aggregated using sophisticated, computationally expensive, physics-based numerical tools (e.g., finite element methods). The recent emergence of materials data science approaches has opened new avenues for addressing this challenge. In this paper, we critically compare the relative merits of the application of four distinct machine learning approaches for their efficacy in extracting microstructure-property linkages from the finite element simulation data aggregated on high-contrast elastic composites with different microstructures. The machine learning approaches selected for the study have included different combinations of local/global and parametric/nonparametric approaches. Furthermore, the nonparametric approaches selected for this study are based on Gaussian Process (GP) models that allow for a formal treatment of uncertainty quantification in the predicted values. The predictive performances of these different approaches have been compared against each other using rigorous cross-validation error metrics. Furthermore, their sensitivity to both the dataset size and dimensionality has been investigated.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 250 TB/s aggregated data refactoring throughput—83% of theoretical peak—on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.

Data refactoring↗
DOE OSTI2021

Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 250 TB/s aggregated data refactoring throughput—83% of theoretical peak—on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.

Chen, Jieyang↗
DOE OSTI2024

Federated Machine Learning-Based Anomaly Detection System for Synchrophasor Network Using Heterogeneous Data Sets: Preprint

Synchrophasor technology is widely deployed in the energy management system to monitor the grid health at micro level and perform necessary corrective actions in real time; however, integrated phasor devices and data aggregators are exposed to several cybersecurity threats. This paper proposes a federated ML(FML)-based ADS to detect several data integrity attacks in the synchrophasor network. The proposed approach integrates the horizontal FML technique and consists of substation-based local models and a control center-based global model. The proposed methodology includes training local models using heterogeneous data sets that include network and grid information and updating the global model through multiple iterations by sharing model gradients. Finally, the trained global model is applied to identify cyberattacks, normal operation, and physical events. To validate the proof of concept, we used synthetic data sets generated by Mississippi State University and Oak Ridge National Laboratory for training and testing the classification models using the National Renewable Energy Laboratory's high performance computing resources. Our experimental results, computed through several performance measures, reveal that the proposed approach shows consistent performance during the binary, three-class, and multiclass classifications while ensuring privacy of synchrophasor data.

anomaly detection system↗
DOE OSTI2019

Scalable Topological Data Analysis and Visualization for Evaluating Data-Driven Models in Scientific Applications

With the rapid adoption of machine learning techniques for large-scale applications in science and engineering comes the convergence of two grand challenges in visualization. First, the utilization of black box models (e.g., deep neural networks) calls for advanced techniques in exploring and interpreting model behaviors. Second, the rapid growth in computing has produced enormous datasets that require techniques that can handle millions or more samples. Although some solutions to these interpretability challenges have been proposed, they typically do not scale beyond thousands of samples, nor do they provide the high-level intuition scientists are looking for. Here, we present the first scalable solution to explore and analyze high-dimensional functions often encountered in the scientific data analysis pipeline. By combining a new streaming neighborhood graph construction, the corresponding topology computation, and a novel data aggregation scheme, namely topology aware datacubes, we enable interactive exploration of both the topological and the geometric aspect of high-dimensional data. Following two use cases from high-energy-density (HED) physics and computational biology, we demonstrate how these capabilities have led to crucial new insights in both applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Exploring Climate-Disease Connections in Geopolitical Versus Ecological Regions: The Case of West Nile Virus in the United States

Many infectious disease forecasting models in the United States (US) are built with data partitioned into geopolitical regions centered on human activity as opposed to regions defined by natural ecosystems; although useful for data collection and intervention, this has the potential to mask biological relationships between the environment and disease. We explored this concept by analyzing the correlations between climate and West Nile virus (WNV) case data aggregated to geopolitical and ecological regions. We compared correlations between minimum, maximum, and mean annual temperature; precipitation; and annual WNV neuroinvasive disease (WNND) case data from 2005 to 2019 when partitioned into (a) climate regions defined by the National Oceanic and Atmospheric Administration (NOAA) and (b) Level I ecoregions defined by the Environmental Protection Agency (EPA). We found that correlations between climate and WNND in NOAA climate regions and EPA ecoregions were often contradictory in both direction and magnitude, with EPA ecoregions more often supporting previously established biological hypotheses and environmental dynamics underlying vector-borne disease transmission. Using ecological regions to examine the relationships between climate and disease cases can enhance the predictive power of forecasts at various scales, motivating a conceptual shift in large-scale analyses from geopolitical frameworks to more ecologically meaningful regions.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

SU-E-T-138: Automated Chart Review Module Including Cross-Vendor Data Transfer Verification Developed for IHE-RO Initiative

Purpose: To follow the Integrating Healthcare Enterprise - Radiation Oncology (IHE-RO) initiative of proper cross-vendor technology integration, an automated chart checker (ACC) was developed. ACC compares extracted data from an approved patient plan in the Eclipse treatment planning system (TPS) against data existing in the Mosaiq treatment management system (TMS). ACC automatically analyzes these parameters using built-in quality checklists to provide further aid in chart review. Methods: Eclipse TPS data are obtained using Eclipse scripting API (ESAPI) while Mosaiq TMS data are obtained from a radiotherapy-treatment-planning (RTP) file. Using this information, ACC identifies TPS-TMS discrepancies in 18 primary beam parameters including MU, energy, jaw positions, gantry angle, table angle, accessories, and bolus for up to 31 beams. Next, approximately 40 items from traditional quality checklists are evaluated such as prescription consistency, DRR graticule placement, plan approval status, global max dose, and dose tracking coefficients. Parameters were artificially modified to determine if ACC would detect an error in data transfer and to test each component of quality checklists. Results: Using ESAPI scripting and RTP file-processing, ACC was able to properly aggregate data from TPS and TMS for up to 31 beams. Errors were artificially introduced into each plan parameter, and ACC was able to successfully detect all of them within seconds. Next, ACC was able to successfully detect mistakes in the chart by identifying deviations with its quality checklists, within seconds. Conclusion: ACC effectively addresses the potential issue of faulty cross-vendor data transfer, as described by IHE-RO. In addition, ACC was also able to detect deviations from its built-in quality checklists. ACC is already an invaluable tool for efficient and standardized chart review and will continue to improve as its incorporated checklists become more comprehensive.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

National Energy Water Treatment & Speciation (NEWTS): A Water & Critical Mineral Database and Dashboard

The scarcity of water resources, the need for beneficial water reuse, and the challenges of wastewater treatment are becoming increasingly pressing in economic, social, and environmental domains. Addressing these concerns requires effective treatment strategies to manage wastewater streams and tackle environmental and economic issues. Furthermore, the recovery of critical minerals from the waste streams associated with energy production holds the promise of offsetting treatment costs and securing local sources of valuable minerals. However, relevant data on these waste streams are dispersed and challenging to locate. The process of ingesting such data into modeling software often involves multiple steps, requiring data restructuring to meet software-input requirements. The non-standardized reporting of water data makes data aggregation and reformatting a time-consuming process. Additionally, essential attributes necessary for modeling water treatment and mineral scale formation are frequently missing. Moreover, data gaps vary depending on the region of interest. Consequently, there is a pressing need for high-quality energy-water composition data that can be easily imported into water chemistry modeling software. To address this need, the National Energy Technology Laboratory has created the National Energy Water Treatment and Speciation (NEWTS) Database and Dashboard—a free online tool catering to community leaders and water researchers. NEWTS facilitates a comprehensive understanding of the composition of energy-related wastewater streams in the United States. The datasets provide detailed concentrations and speciation of major and minor aqueous compounds in energy-related wastewater streams, including power plant leachate, acid mine drainage, brackish water, and oil and gas produced water across the United States. Many of the aqueous species are critical minerals (Li, REEs) in high demand to modernize the world’s energy infrastructure. Many of the datasets also contain volumetric flow-rates needed to model the treatment and reuse scenarios in advanced aqueous chemistry software programs. The NEWTS Database and Dashboard offer public access to hitherto challenging-to-access datasets, presented in a standardized format that is tailored for easy input into aqueous chemistry modeling software. By performing the work needed to transform dispersed, disparate data sources into unified, model-ready datasets, NEWTS serves as an essential resource in advancing water treatment research and sustainable water resource management.

produced water management↗
DOE OSTI2018

A browser-based tool for visualization and analysis of diffusion MRI data

Human neuroscience research faces several challenges with regards to reproducibility. While scientists are generally aware that data sharing is important, it is not always clear how to share data in a manner that allows other labs to understand and reproduce published findings. Here we report a new open source tool, AFQ-Browser, that builds an interactive website as a companion to a diffusion MRI study. Because AFQ-Browser is portable—it runs in any web-browser—it can facilitate transparency and data sharing. Moreover, by leveraging new web-visualization technologies to create linked views between different dimensions of the dataset (anatomy, diffusion metrics, subject metadata), AFQ-Browser facilitates exploratory data analysis, fueling new discoveries based on previously published datasets. In an era where Big Data is playing an increasingly prominent role in scientific discovery, so will browser-based tools for exploring high-dimensional datasets, communicating scientific discoveries, aggregating data across labs, and publishing data alongside manuscripts.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

Cryogenic C-band wavelength division multiplexing system using an AIM Photonics Foundry process design kit

Cryogenic environments make superconducting computing possible by reducing thermal noise, electrical resistance and heat dissipation. Heat generated by the electronics and thermal conductivity of electrical transmission lines to the outside world constitute two main sources of thermal load in such systems. As a result, higher data rates require additional transmission lines which come at an increasingly higher cooling power cost. Hybrid or monolithic integration of silicon photonics with the electronics can be the key to higher data rates and lower power costs in these systems. We present a 4-channel wavelength division multiplexing photonic integrated circuit (PIC) built from modulators in the AIM Photonics process development kit (PDK) that operate at 25 Gbps at room temperature and 10 Gbps at 40 K. We further demonstrate 2-channel operation for 20 Gbps aggregate data rate at 40 K using two different modulators/wavelengths, with the potential for higher aggregate bit rates by utilizing additional channels.

42 ENGINEERING↗
DOE OSTI2024

Implementation of a realistic artificial data generator for crash data generation

In this paper, a framework is outlined to generate realistic artificial data (RAD) as a tool for comparing different models developed for safety analysis. The primary focus of transportation safety analysis is on identifying and quantifying the influence of factors contributing to traffic crash occurrence and its consequences. The current framework of comparing model structures using only observed data has limitations. With observed data, it is not possible to know how well the models mimic the true relationship between the dependent and independent variables. Further, real datasets do not allow researchers to evaluate the model performance for different levels of complexity of the dataset. RAD offers an innovative framework to address these limitations. Hence, we propose a RAD generation framework embedded with heterogeneous causal structures that generates crash data by considering crash occurrence as a trip level event impacted by trip level factors, demographics, roadway and vehicle attributes. Within our RAD generator we employ three specific modules: (a) disaggregate trip information generation, (b) crash data generation and (c) crash data aggregation. For disaggregate trip information generation, we employ a daily activity-travel realization for an urban region generated from an established activity-based model for the Chicago region. We use this data of more than 2 million daily trips to generate a subset of trips with crash data. For trips with crashes crash location, crash type, driver/vehicle characteristics, and crash severity. The daily RAD generation process is repeated for generating crash records at yearly or multi-year resolution. In conclusion, the crash databases generated can be employed to compare frequency models, severity models, crash type and various other dimensions by facility type - possibly establishing a universal benchmarking system for alternative model frameworks in safety literature.

42 ENGINEERING↗
DOE OSTI2023

A Privacy-Aware Federated Learning Framework for Distributed Energy Resource Analytics in Constrained Environments

To be resilient against extreme weather events, the rural communities in Puerto Rico are leveraging distributed energy resources (DER). However, computing frameworks sup-porting the grid in critical decision-making are still largely centralized. Sensitive consumer data are transmitted over the Internet or cellular networks to a secondary or tertiary node. It guarantees better situational awareness at the cost of a wider attack surface, jeopardizing user privacy, as more DER come online. Cloud, Edge, and Fog computing all require data aggregation at some level. This paper introduces a privacy-aware federated learning framework that leverages the Fog model by pushing analytics all the way to the DER and load assets. These local models train on individual asset data and transmit only learned parameters (such as weights) over secure communications to a global decision-maker. By abstracting personally identifiable consumer data without impacting decision optimality, this framework better aligns with distributed power generation paradigm.

Sundararajan, Aditya↗
DOE OSTI2019

Design and development of the DAQ and Timing Hub for CMS Phase-2

The CMS detector will undergo a major upgrade for Phase-2 of the LHC program, starting around 2026. The upgraded Level-1 hardware trigger will select events at a rate of 750 kHz. At an expected event size of 7.4 MB this corresponds to a data rate of up to 50 Tbit/s.Optical links will carry the signals from on-detector front-end electronics to back-end electronics in ATCA crates in the service cavern. A DAQ and Timing Hub board aggregates data streams from back-end boards over point-to-point links, provides buffering and transmits the data to the commercial data-to-surface network for processing and storage. This hub board is also responsible for the distribution of timing, control and trigger signals to the back-ends.This paper presents the current development towards the DAQ and Timing Hub and the design of the first prototype, to be used as for validation and integration with the first back-end prototypes in 2019-2020.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗