Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Multi-Source Data Aggregation and Real-Time Anomaly Classification and Localization in Power Distribution Systems

This paper proposes a real-time anomaly location and classification framework for power distribution systems to simultaneously determine the type of anomaly (i.e., short-circuit fault, cyber attack, DER switching) and its location. The proposed framework employs the data aggregation module to collect the measurement data from multiple field devices operating at different sampling rates, such as protection relays and D-PMUs. The output of the data aggregation is then fed into a multi-task learning-based long-based short-term memory (MTL-LSTM) to classify the type of anomaly and the location in two separate tasks. The proposed MTL-LSTM approach can be utilized in real-time operation in order to distinguish between normal and several anomalous operations and locate the anomaly. The proposed framework is tested on a modified IEEE 33-bus test feeder benchmark that integrates solar generation and energy storage. Furthermore, the results show that the proposed framework can locate and classify anomalies for several operation conditions with more than 96% accuracy. Further experiments highlight the impact of aggregating multiple sources of data on the performance of the proposed model.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2016

SU-G-TeP4-09: Development of a Plan Data Aggregator for Time Efficient Physics Second-Checks of Machine Parameters for External Beam Treatment Plans

Purpose: Physics second-checks for external beam radiation therapy are performed, in-part, to verify that the machine parameters in the Record-and-Verify (R&V) system that will ultimately be sent to the LINAC exactly match the values initially calculated by the Treatment Planning System (TPS). While performing the second-check, a large portion of the physicists’ time is spent navigating and arranging display windows to locate and compare the relevant numerical values (MLC position, collimator rotation, field size, MU, etc.). Here, we describe the development of a software tool that guides the physicist by aggregating and succinctly displaying machine parameter data relevant to the physics second-check process. Methods: A data retrieval software tool was developed using Python to aggregate data and generate a list of machine parameters that are commonly verified during the physics second-check process. This software tool imported values from (i) the TPS RT Plan DICOM file and (ii) the MOSAIQ (R&V) Structured Query Language (SQL) database. The machine parameters aggregated for this study included: MLC positions, X&Y jaw positions, collimator rotation, gantry rotation, MU, dose rate, wedges and accessories, cumulative dose, energy, machine name, couch angle, and more. Results: A GUI interface was developed to generate a side-by-side display of the aggregated machine parameter values for each field, and presented to the physicist for direct visual comparison. This software tool was tested for 3D conformal, static IMRT, sliding window IMRT, and VMAT treatment plans. Conclusion: This software tool facilitated the data collection process needed in order for the physicist to conduct a second-check, thus yielding an optimized second-check workflow that was both more user friendly and time-efficient. Utilizing this software tool, the physicist was able to spend less time searching through the TPS PDF plan document and the R&V system and focus the second-check efforts on assessing the patient-specific plan-quality.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2017

Reproducibility and Variability of I/O Performance on BG/Q: Lessons Learned from a Data Aggregation Algorithm

Reading and writing data efficiently from different tiers of storage is necessary for most scientific simulations to achieve good performance at scale. Many software solutions have been developed to decrease the I/O bottleneck. One wellknown strategy, in the context of collective I/O operations, is the two-phase I/O scheme. This strategy consists of selecting a subset of processes to aggregate contiguous pieces of data before performing reads/writes. In our previous work, we implemented the two-phase I/O scheme with a MPI-based topology-aware algorithm. Our algorithm showed very good performance at scale compared to the standard I/O libraries such as POSIX I/O and MPI I/O. However, the algorithm had several limitations hindering a satisfying reproducibility of our experiments. In this paper, we extend our work by 1) identifying the obstacles we face to reproduce our experiments and 2) discovering solutions that reduce the unpredictability of our results.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Scaling Irregular Applications through Data Aggregation and Software Multithreading

Bioinformatics, data analytics, semantic databases, knowledge discovery are emerging high performance application areas that exploit dynamic, linked data structures such as graphs, unbalanced trees or unstructured grids. These data structures usually are very large, requiring significantly more memory than available on single shared memory systems. Additionally, these data structures are difficult to partition on distributed memory systems. They also present poor spatial and temporal locality, thus generating unpredictable memory and network accesses. The Partitioned Global Address Space (PGAS) programming model seems suitable for these applications, because it allows using a shared memory abstraction across distributed-memory clusters. However, current PGAS languages and libraries are built to target regular remote data accesses and block transfers. Furthermore, they usually rely on the Single Program Multiple Data (SPMD) parallel control model, which is not well suited to the fine grained, dynamic and unbalanced parallelism of irregular applications. In this paper we present {\bf GMT} (Global Memory and Threading library), a custom runtime library that enables efficient execution of irregular applications on commodity clusters. GMT integrates a PGAS data substrate with simple fork/join parallelism and provides automatic load balancing on a per node basis. It implements multi-level aggregation and lightweight multithreading to maximize memory and network bandwidth with fine-grained data accesses and tolerate long data access latencies. A key innovation in the GMT runtime is its thread specialization (workers, helpers and communication threads) that realize the overall functionality. We compare our approach with other PGAS models, such as UPC running using GASNet, and hand-optimized MPI code on a set of typical large-scale irregular applications, demonstrating speedups of an order of magnitude.

Aggregation↗
DOE OSTI2014

Recovery Act: Geothermal Data Aggregation: Submission of Information into the National Geothermal Data System, (Final Report, June 24, 2014)

The National Geothermal Data System (NGDS) is a Department of Energy funded effort to create a single cataloged source for a variety of geothermal information through a distributed network of databases made available via web services. The NGDS will help identify regions suitable for potential development and further scientific data collection and analysis of geothermal resources as a source for clean, renewable energy.

15 GEOTHERMAL ENERGY↗
DOE OSTI2024

Divide and conquer: using RhizoVision Explorer to aggregate data from multiple root scans using image concatenation and statistical methods

Roots are important in agricultural and natural systems for determining plant productivity and soil carbon inputs. Sometimes, the amount of roots in a sample is too much to fit into a single scanned image, so the sample is divided among several scans, and there is no standard method to aggregate the data. Here, we describe and validate two methods for standardizing measurements across multiple scans: image concatenation and statistical aggregation. We developed a Python script that identifies which images belong to the same sample and returns a single, larger concatenated image. These concatenated images and the original images were processed with RhizoVision Explorer, a free and open-source software. An R script was developed, which identifies rows of data belonging to the same sample and applies correct statistical methods to return a single data row for each sample. These two methods were compared using example images from switchgrass, poplar, and various tree and ericaceous shrub species from a northern peatland and the Arctic. Most root measurements were nearly identical between the two methods except median diameter, which cannot be accurately computed by statistical aggregation. We believe the availability of these methods will be useful to the root biology community.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Putting Our Industry's Data to Work: A Case Study of Large-Scale Data Aggregation: Preprint

With increasing deployment of Advanced Metering Infrastructure (AMI), Building Automation System (BAS) controls, Internet of Things (IoT) network devices, and data-driven evaluation, measurement, and verification studies, the building sector is currently generating a staggering amount of energy-related data. In the right hands, these data sets can contribute to increased comfort and energy savings for building occupants and a more reliable electrical grid; however, due to a combination of factors, including significant privacy concerns, much of the data that are presently generated and stored are not used outside of basic operational applications. In the past year, our team has dedicated over 2,000 person-hours to accessing building energy data for a project funded by the U.S. Department of Energy (DOE) Building Technologies Office. We sought whole-building or end-use (e.g., lighting) timeseries data at the individual-building or equipment level where possible or aggregated information, such as timeseries averages and quartiles by building type (e.g., office, retail, hospital), where sharing individual building information was not an option. We are additionally working with IoT and BAS data sets to derive information important to the project. We have assembled an extensive data set that will enable the development of publicly available end-use load profiles to benefit the U.S. building and electricity industries. Here we present an overview of the data set that we have assembled to date, the motivators and approaches that got us here, and the lessons we learned through our efforts. We also discuss work underway that presents additional options for future data access.

building energy data↗
DOE OSTI2024

Aggregate data‐driven dynamic modeling of active distribution networks with DERs for voltage stability studies

Abstract Electric distribution networks increasingly host distributed energy resources based on power electronic converter (PEC) toward active distribution networks (ADN). Despite advances in computational capabilities, electromagnetic transient models are limited in scalability because of their reliance on exact data about the distribution system and each of its components. Similarly, the use of the DER_A model, which is intended to examine the combined dynamic behavior of many DERs, is limited by the difficulty in parameterization. There is a need for improved dynamic models of DERs for use in large power system simulations for stability analysis. This paper proposes an aggregate model‐free, data‐driven approach for deriving a dynamic partitioned model (DPM) of ADNs. Detailed residential distribution feeders were first developed, including PEC‐based DERs and composite load models (CMLDs), from which the aggregated DPM was derived. The performance was evaluated through various case studies and validated against the detailed ADN model and state‐of‐the‐art DER_A model with CMLD. The data‐driven DPM achieved a of over 90%, accurately representing the aggregated dynamic behavior of ADNs. Furthermore, the DPM significantly accelerated the simulation process with a computational speedup of 68 times compared to the detailed ADN and a 3.5 times speedup compared to the DER_A CMLD model.

42 ENGINEERING↗
DOE OSTI2025

HexWeather: Hexagonal Spatial Data Aggregation for Weather-Driven Grid Resilience Analysis

Extreme weather accounts for over 8 0 % of major U.S. power outages since 2000, highlighting the need for spatial tools that align weather data with the irregular boundaries of electric infrastructure. This paper introduces HexWeather, a modular, resolution-aware framework for aggregating historical and forecasted weather data using Uber's H3 hexagonal spatial indexing system. Unlike traditional methods that rely on state or county-level grids, HexWeather enables weather analysis across custom geographies such as utility service areas where public datasets are often unavailable or misaligned. Using Open-Meteo data, we evaluate how H3 resolution affects anomaly detection, spatial variability, and forecast uncertainty across three scales: state, county, and utility. Results show that while coarse resolutions suffice for broad trend tracking, finer resolutions are essential for identifying localized variability and operational risks. By applying metrics like Z-score standard deviation and interquartile range, HexWeather quantifies the spatial spread of both historical anomalies and forecasted conditions, allowing users to assess resolution adequacy for each analysis. This framework supports rapid weather data reuse, reproducible anomaly detection, and predictive modeling for infrastructure resilience. By bridging spatial misalignment in traditional datasets and enabling retrospective and forward-looking analysis within the same pipeline, HexWeather lays the groundwork for better post event analysis, outage prediction, and resilience planning.

Morris, Jacob [ORNL]↗
DOE OSTI2025

Toward Drilling the Perfect Geothermal Well: An International Research Coordination Network for Geothermal Drilling Optimization Supported by Deep Machine Learning and Cloud Based Data Aggregation

The EDGE project, supported by the U.S. Department of Energy Geothermal Technologies Office under award DE-EE0008793, established a data-driven framework for improving the efficiency, cost-effectiveness, and reliability of geothermal well drilling. The project focused on developing scalable data infrastructure, advanced machine learning and probabilistic models, and integrated analytics tools to support continuous drilling optimization. A central objective was to reduce geothermal drilling costs by up to seventy percent while minimizing the risk of well failure through predictive diagnostics and adaptive planning. Over the project period, a comprehensive data repository was designed and deployed, incorporating records from over one hundred geothermal wells across varied geological settings. This repository supported both structured and unstructured data and adhered to FAIR data principles, enabling provenance tracking, quality control, and standardized metadata. The project introduced automated ingestion pipelines and a cloud-hosted platform that facilitated access to raw, processed, and derived datasets. This infrastructure served as the foundation for model development and analysis. Machine learning workflows were developed to predict key drilling metrics including rate of penetration, non-productive time, and total drilling costs. Self-organizing maps and dimensionality reduction methods were used to uncover operational patterns and outliers, while supervised learning algorithms such as random forests and deep neural networks were applied to forecast performance outcomes. The models were validated on heterogeneous datasets from both U.S. and Icelandic fields, demonstrating variable but significant predictive accuracy. The results indicated that finer temporal resolution, inclusion of lithological data, and consistency in operational annotations could substantially improve model performance. The project also implemented process mining techniques to reconstruct state-transition models from drilling event logs. These models enabled the identification of deviations from optimal workflows and provided insights into recurring failure modes. Analysis of non-productive time highlighted the impact of equipment failures, geological challenges, and human factors, offering opportunities for targeted mitigation strategies. The EDGE Dashboard was developed as a web-based expert system integrating data visualization, model outputs, and user-driven queries. It provided an accessible interface for operators to explore historical data, evaluate predicted outcomes, and compare drilling scenarios. Initial feedback from project partners suggested that the dashboard could serve as a foundation for more advanced advisory and optimization tools. Overall, the EDGE project demonstrated the feasibility and value of applying modern data science techniques to geothermal drilling. It delivered a set of interoperable tools and models that can support more efficient, lower-risk well development. The findings point toward a viable path for transitioning from advisory analytics to semi-autonomous drilling systems, contingent on continued collaboration, expanded datasets, and field validation. The project results have immediate relevance for drilling operations, data management practices, and future geothermal R&D efforts aimed at achieving reliable, cost-competitive geothermal energy at scale.

15 GEOTHERMAL ENERGY↗
DOE OSTI2018

Lead exposure in young children over a 5-year period from urban environments using alternative exposure measures with the US EPA IEUBK model – A trial

Highlights: • IEUBK simulations of soil with petri dish dust (PDD), soil with handwipes, sweepings with PDD, sweepings with wipes. • Predicted blood lead levels (PbB) generally give good agreement with observed PbB in spite of differences in intake units for PDD and handwipes. • Analyses with dust loadings converted to concentration give higher predicted PbB. • Simulations on aggregated data for each child give similar results to age-stratified data. • Estimating house dust Pb from measured soil Pb concentrations (in the absence of measured house dust concentrations) results in over-prediction of PbB. The US Environmental Protection Agency (EPA) Integrated Exposure Uptake Biokinetic (IEUBK) model has been widely used to predict blood lead (PbB) levels in children especially around industrial sites. Exposure variables have strongly focussed on the major contribution of lead (Pb) in soil and interior dust to total intake and, in many studies, site-specific data for air, water, diet and measured PbB were not available. We have applied the IEUBK model to a comprehensive data set, including measured PbB, for 108 children monitored over a 5-year period in Sydney, New South Wales, Australia. To use this data set, we have substituted available data (with or without modification) for standard inputs as needed. For example, as an alternative measure for soil Pb concentration (μg/g), we have substituted exterior dust sweepings Pb concentration (μg/g). As alternative measures for interior dust Pb concentration (μg/g) we have used 1) 30-day cumulative petri dish deposition data (PDD) (as µg Pb/m{sup 2}/30days), or 2) hand wipe data (as μg Pb/hand). For comparison, simulations were also undertaken with estimates of dust Pb concentration derived from a prior regression of dust Pb concentration (μg/g) on dust Pb loading (μg/ft{sup 2}) as concentration is the unit specified for the Model. Simulations for each subject using observed data aggregated over the 5-year interval of the study, the most usual application of the IEUBK model, showed using Wilcoxon tests that there was a significant difference between the observed values and the values predicted by the Model containing soil with hand wipes (p < 0.001), and soil and PDD (p = 0.026) but not those for the other two sets of predictors, based on sweepings and PDD or sweepings and wipes. Overall, simulations of the Model using alternative exposure measures of petri dish dust (and possibly hand wipes) instead of vacuum cleaner dust and dust sweepings instead of soil provide predicted PbB which are generally consistent with each other and observed values. The predicted geometric mean PbBs were 2.17 ( ± 1.24) μg/dL for soil with PDD, 1.95 ( ± 1.17) μg/dL for soil with hand wipes, 2.36 ( ± 1.75) μg/dL for sweepings with PDD, and 2.15 ( ± 1.69) for sweepings with hand wipes. These results are in good agreement with the observed geometric mean PbB of 2.46 ( ± 0.99) μg/dL. In contrast to all other IEUBK model studies to our knowledge, we have stratified the data over the age ranges from 1 to 5 years. The median of the predicted values was lower than that for the observed values for every combination of age and set of measures; in some cases, the difference was statistically significant. The differences between observed and predicted PbB tended to be greatest for the soil plus wipes measure and for the oldest age group. Use of ‘default dust’ values calculated from the site-specific soil values, a common application of the IEUBK model, results in predicted PbB about 22% (range 0 to 52%) higher than those from soil with PDD data sets. Geometric mean contributions estimated from the Model to total Pb intake for a child aged 1–2 years was 0.09% for air, 42% for diet, 5.3% for water and 42% for soil and dust. Our results indicate that it is feasible to use alternative measures of soil and dust exposure to provide reliable predictions of PbB in urban environments.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2018

Aggregation and data analysis of corrosion studies in molten chloride and fluoride salts

Corrosion studies in molten fluoride and chloride salts were surveyed, and key data were aggregated into a single dataset. Studies were graphed by salt purity, temperature, sample material, container material, and experimental method (loop, capsule, crucible). To elucidate and quantify the factors that affect corrosion in molten salts, a data analysis using a newly-defined corrosion resistance score and a correlation analysis using techniques borrowed from machine learning were performed. It is shown that salt purity had the strongest correlation with corrosion rates in molten chlorides and fluorides. As a result, data analysis across varied works was inconsistent due to the lack of standardization among molten salt corrosion studies.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Data Integration Aggregated Model For Nuclear Deployment

The data of a nuclear utility or plant are stored in various isolated forms in different systems. These data have different structures and tools and are, therefore, used independently. The integration of data sources is often needed and is performed manually and on an as-needed basis. The integration of the data would enable creating a data warehouse for a single nuclear power plant (NPP) or multiple plants. A data model contains objects that make up a given domain along with their attributes and relationships and could be used as a standard for exchange between applications for simple integration and data aggregation. After a survey of existing models, it was determined that a new model was required to meet the scope of this problem. DIAMOND has been developed in the form of an ontology to meet these needs. An ontology is simply a data model that also acts as a taxonomy for the domain, organizing the objects in a hierarchy with the most general objects at the top.

Browning, JerenM↗
DOE OSTI2016

Authenticated sensor interface device

A system and method for the secure storage and transmission of data is provided. A data aggregate device can be configured to receive secure data from a data source, such as a sensor, and encrypt the secure data using a suitable encryption technique, such as a shared private key technique, a public key encryption technique, a Diffie-Hellman key exchange technique, or other suitable encryption technique. The encrypted secure data can be provided from the data aggregate device to different remote devices over a plurality of segregated or isolated data paths. Each of the isolated data paths can include an optoisolator that is configured to provide one-way transmission of the encrypted secure data from the data aggregate device over the isolated data path. External data can be received through a secure data filter which, by validating the external data, allows for key exchange and other various adjustments from an external source.

97 MATHEMATICS AND COMPUTING↗