Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

DOE EV Data Collection - Vehicle Data

Vehicle data consist of electric vehicle performance data collected directly from the vehicle during standard operations. Data were collected using onboard data loggers that were either installed by the project team or preinstalled by the original equipment manufacturer. Data recorded by the data loggers were made accessible via an online web portal or an application programming interface. Different data loggers were used (HEM, ViriCiti, and Geotab), and the method for each vehicle is defined in the vehicle attributes file. Some systems collected data on a “trip-level” basis, in which each row of a table represents a single trip (the period between a key-on and key-off event), whereas other data were collected on a per-day basis, in which each row represents a single day of operation. Data were collected over a range of data collection periods, depending on the project. Data have been anonymized by removing information or decreasing information resolution as necessary so that fleets are not identifiable. Due to the wide range of vehicle types represented and variation in data collection, data parameters and frequencies differ between vehicles and fleets The **Performance Data Daily/Trip Data Dictionaries** contain definitions for each available parameter associated with a vehicle’s operations, aggregated at either a daily or trip level. The parameters available will vary from vehicle to vehicle, but every possible parameter will be defined. The **Vehicle Attributes Data Dictionary** contains definitions for each available parameter associated with a vehicle’s physical and functional attributes and fleet context. The **Vehicle Attributes** table contains specific vehicle characteristics, coded to an anonymous Vehicle ID. This Vehicle ID can be used as a key between vehicle data and vehicle attribute tables. The **Vehicle Data** tables contain the data from each vehicle’s operations, aggregated at either a daily or trip level, coded to an anonymous Vehicle ID. This Vehicle ID can be used as a key between vehicle data and vehicle attribute tables. Data is being uploaded quarterly through 2023 and subject to change until the conclusion of the project.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

PV Fleet Performance Data Initiative Program and Methodology

The US Department of Energy’s PV Fleet Performance Data Initiative has been launched in order to collect and evaluate production data across multiple PV fleet partners. Performance statistics are anonymized, aggregated and shared to represent a snapshot of the US commercial and utility-scale fleet. Production data have been collected from over 1500 systems representing more than 1.3 GWdc capacity. Preliminary analysis indicates median performance loss rates are in line with previous publications of system degradation, on the order of –0.6%/yr to –0.9%/yr (preliminary numbers subject to change). These values are higher than module-only degradation rates which are often used in pro-forma estimates of project performance and economics, potentially exposing owner/operators to increased risk if systems under-perform over time.

41 EE - Solar Energy Technologies Office (EE-4S)↗
DOE OSTI2023

A review of privacy in energy applications

As the distribution system continues to experience an increase in distributed energy resource (DER) and electric vehicle (EV) penetration, so does the need for new solutions that can help grid operators manage and leverage their capabilities. This will undoubtedly lead to new operational schemes and business opportunities that will transform the traditional consumer into a prosumer who will be more actively engaged in grid operations. Although the field is still under active development, many of the potential use cases presented in literature or industry are built upon edge computing, two-way communications, and other innovative computational constructs to attain their goals. However, at their core, many use cases assume a great level of data access to aid with the decision-making process, an assumption that may need to be revised to ensure fair and equitable operational processes are maintained. This may be particularly true as edge resources are predicted to participate in retail-side, many-to-many, or peer-to-peer markets and thus may lead to financial impacts if data access considerations are ignored. The need to revise data access mechanisms can be further justified by the introduction of new participants into the operational process, who do not have the same level of trust, nor the incentives to focus on energy delivery as their primary objective. At the same time, more consumers are becoming aware of their own data, and the potential impacts of its abuse. To help solution developers better understand these risks, this report has been developed to offer an initial introduction to the topic of privacy. This is achieved by 1) Highlighting the need for privacy-aware solutions; 2) Encouraging system designers to be inquisitive about the status quo; 3) Documenting the existing threat space; 4) Presenting and evaluating tools that may be helpful towards enabling better privacy postures; and 5) Making recommendations to encourage the adoption of better practices. From a technical perspective, the report focuses on evaluating two potential techniques by applying them to the Transactive Energy Space. Based on the obtained results, it can be established that differential privacy (DP) methods may have limited applicability when highly correlated, time-series data records need to be protected. However, DP may be a powerful tool when it is used to aggregate and analyze mid-size and large-size data sets in a more traditional statistical environment. The second tool under evaluation is threshold cryptography, which can guarantee complete secrecy (and thus privacy) but requires the establishment of key management procedures and dedicated communication channels for key coordination. Therefore, due to its increased computational overhead, the use of threshold cryptography must be weighted using a cost/benefit analysis on a per-application basis.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Data for: Miscanthus × giganteus changes soil structure and increases maximum water holding capacity

The data provided include results from a comparative study evaluating the impact of Miscanthus × giganteus (miscanthus) versus maize on soil structural properties and maximum water holding capacity (MWHC) across two Iowa sites. The dataset includes MWHC values determined using the Funnel Filter Paper Drainage (FFPD) method, as well as additional measurements of MWHC following structural disruption of the soil to isolate the effect of aggregation. It also contains three-dimensional micro-computed tomography (microCT) data used to quantify total porosity and pore size distribution (PSD) of soil aggregates at a 5 µm resolution. All data are provided as raw replicate-level measurements, organized by site, crop, and depth, along with processed summary files in table form in CSV (.csv) format to support reproducibility and downstream analysis.

Misanthus x giganteus↗
DOE OSTI2024

Responses of Drying-rewetting (Transient Soil Moisture) and Steady State Soil Moisture Incubation on Soil Organic Carbon Dynamics in Three US Soils, 2017

This data set contains measurements of soil characteristics (aggregate size distribution and mean size, total aggregate associated carbon, extractable organic C, and microbial biomass C), microbial respiration, and soil metabolite concentrations from a transient and steady soil moisture incubation experiment using soils of different textures (sandy, loamy, and clayey). The study investigated mechanisms driving the Birch effect (increased carbon mineralization pulses with wetting following a drying period) in differing soil textures. Three different soils of distinctly different textures were collected from 0-15cm depth in Georgia (sandy, 2017-05-01), Missouri (loamy, 2017-06-14), and Texas (clayey, December 2017). Soils were incubated for 140 days with destructive harvests done on days 1, 29, 33, 56, 112, 116, and 140 in transient soil moisture incubation and on days 1, 33, 116, and 140 in steady state soil moisture incubation. This dataset contains six data files in comma separate (.csv) format. Additional metadata are provided: six data dictionaries and a file-level metadata file in comma separate (*.csv) format and a user guide in PDF (*.pdf) format.

1-Methyladenosine concentration↗
DOE OSTI2022

A round-robin approach provides a detailed assessment of biomolecular small-angle scattering data reproducibility and yields consensus curves for benchmarking

Through an expansive international effort that involved data collection on 12 small-angle X-ray scattering (SAXS) and four small-angle neutron scattering (SANS) instruments, 171 SAXS and 76 SANS measurements for five proteins (ribonuclease A, lysozyme, xylanase, urate oxidase and xylose isomerase) were acquired. From these data, the solvent-subtracted protein scattering profiles were shown to be reproducible, with the caveat that an additive constant adjustment was required to account for small errors in solvent subtraction. Further, the major features of the obtained consensus SAXS data over the q measurement range 0–1 Å −1 are consistent with theoretical prediction. The inherently lower statistical precision for SANS limited the reliably measured q -range to <0.5 Å −1 , but within the limits of experimental uncertainties the major features of the consensus SANS data were also consistent with prediction for all five proteins measured in H 2 O and in D 2 O. Thus, a foundation set of consensus SAS profiles has been obtained for benchmarking scattering-profile prediction from atomic coordinates. Additionally, two sets of SAXS data measured at different facilities to q > 2.2 Å −1 showed good mutual agreement, affirming that this region has interpretable features for structural modelling. SAS measurements with inline size-exclusion chromatography (SEC) proved to be generally superior for eliminating sample heterogeneity, but with unavoidable sample dilution during column elution, while batch SAS data collected at higher concentrations and for longer times provided superior statistical precision. Careful merging of data measured using inline SEC and batch modes, or low- and high-concentration data from batch measurements, was successful in eliminating small amounts of aggregate or interparticle interference from the scattering while providing improved statistical precision overall for the benchmarking data set.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Distributed Cross-Channel Hierarchical Aggregation for Foundation Models

Vision-based scientific foundation models hold significant promise for advancing scientific discovery and innovation. This potential stems from their ability to aggregate images from diverse sources—such as varying physical groundings or data acquisition systems—and to learn spatio-temporal correlations using transformer architectures. However, tokenizing and aggregating images can be compute-intensive, a challenge not fully addressed by current distributed methods. In this work, we introduce the Distributed Cross-Channel Hierarchical Aggregation (D-CHAG) approach designed for datasets with a large number of channels across image modalities. Our method is compatible with any model-parallel strategy and any type of vision transformer architecture, significantly improving computational efficiency. We evaluated D-CHAG on hyperspectral imaging and weather forecasting tasks. When integrated with tensor parallelism and model sharding, our approach achieved up to a 75% reduction in memory usage and more than doubled sustained throughput on up to 1,024 AMD GPUs on the Frontier Supercomputer.

Tsaris, Aristeidis (aris) [ORNL] (ORCID:0000000277↗
DOE OSTI2020

Automated Integration of Continental-Scale Observations in Near-Real Time for Simulation and Analysis of Biosphere–Atmosphere Interactions

The National Ecological Observatory Network (NEON) is a continental-scale observatory with sites across the US collecting standardized ecological observations that will operate for multiple decades. To maximize the utility of NEON data, we envision edge computing systems that gather, calibrate, aggregate, and ingest measurements in an integrated fashion. Edge systems will employ machine learning methods to cross-calibrate, gap-fill and provision data in near-real time to the NEON Data Portal and to High Performance Computing (HPC) systems, running ensembles of Earth system models (ESMs) that assimilate the data. For the first time gridded EC data products and response functions promise to offset pervasive observational biases through evaluating, benchmarking, optimizing parameters, and training new machine learning parameterizations within ESMs all at the same model-grid scale. Leveraging open-source software for EC data analysis, we are already building software infrastructure for integration of near-real time data streams into the International Land Model Benchmarking (ILAMB) package for use by the wider research community. We will present a perspective on the design and integration of end-to-end infrastructure for data acquisition, edge computing, HPC simulation, analysis, and validation, where Artificial Intelligence (AI) approaches are used throughout the distributed workflow to improve accuracy and computational performance.

Durden, David J.↗
DOE OSTI2022

Skopi: a simulation package for diffractive imaging of noncrystalline biomolecules

X-ray free-electron lasers (XFELs) have the ability to produce ultra-bright femtosecond X-ray pulses for coherent diffraction imaging of biomolecules. While the development of methods and algorithms for macromolecular crystallography is now mature, XFEL experiments involving aerosolized or solvated biomolecular samples offer new challenges in terms of both experimental design and data processing. Skopi is a simulation package that can generate single-hit diffraction images for reconstruction algorithms, multi-hit diffraction images of aggregated particles for training machine learning classifiers using labeled data, diffraction images of randomly distributed particles for fluctuation X-ray scattering algorithms, and diffraction images of reference and target particles for holographic reconstruction algorithms. Skopi is a resource to aid feasibility studies and advance the development of algorithms for noncrystalline experiments at XFEL facilities.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Prospects of federated machine learning in fluid dynamics

Physics-based models have been mainstream in fluid dynamics for developing predictive models. In recent years, machine learning has offered a renaissance to the fluid community due to the rapid developments in data science, processing units, neural network based technologies, and sensor adaptations. So far in many applications in fluid dynamics, machine learning approaches have been mostly focused on a standard process that requires centralizing the training data on a designated machine or in a data center. In this article, we present a federated machine learning approach that enables localized clients to collaboratively learn an aggregated and shared predictive model while keeping all the training data on each edge device. We demonstrate the feasibility and prospects of such a decentralized learning approach with an effort to forge a deep learning surrogate model for reconstructing spatiotemporal fields. Our results indicate that federated machine learning might be a viable tool for designing highly accurate predictive decentralized digital twins relevant to fluid dynamics.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Pricing Strategy of Electric Vehicle Aggregators Based on Locational Marginal Price to Minimize Photovoltaic (PV) Curtailment

The global climate crisis demands urgent action to mitigate global warming. Using renewable energy sources, such as solar and wind power, for electricity generation is crucial. This shift from centralized to distributed power systems, however, brings challenges, including voltage fluctuations and renewable energy curtailment. The rapid growth of the electric vehicle (EV) industry adds complexity, increasing overall electricity demand and straining the power supply during peak charging times. This paper proposes a scheduling strategy for EV aggregators to reduce renewable energy curtailment and stabilize grid operation by strategically scheduling EV charging. Using Multi -Agent Transport Simulation (MATSim), a traffic simulation tool, EV driving data in Denver, Colorado, USA, were modeled. The EV aggregator adjusts charging fees based on locational marginal prices, encouraging EVs to charge at different stations according to pricing. Simulations on an IEEE 33-bus system with distributed energy resources and EV charging stations validate the proposed algorithm, demonstrating its effectiveness in reducing curtailment by 12.55% and stabilizing grid operation.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2023

Field-scale dynamics of planting dates in the US Corn Belt from 2000 to 2020

Crop planting dates are a dynamic feature of agricultural systems that respond to short- and long-term climate signals, crop and cultivar selection, and technology changes. Planting date records are essential for yield gap analyses, accurate crop modeling, and tracking farmer adaptations to weather and climate change. Although planting dates have high variation at local scales due to heterogeneity in farm resources and decision-making, available long-term data on planting dates is largely restricted to aggregated regional statistics or, at best, satellite-derived datasets with limited spatiotemporal extent and at resolutions unable to distinguish individual fields (> 250 m). Here, we generated retrospective annual field-scale (30 m) planting date maps for both maize and soybeans spanning 2000-2020 across a 12 state region in the United States Corn Belt based on Landsat satellite data and a large ground sample of over 28,000 maize and soybean fields. Using training data from 2015-2020 for model selection, we found that planting date predictions improved with harmonic regression of Landsat data and additional annual weather covariates. The preferred random forests model approximately doubled performance compared to a null model based on state median planting dates, capturing 47% of field-level variation for maize (mean absolute error, MAE = 7.4 days) and 44% for soybeans (MAE = 7.5 days) against held-out ground truth test data for 2008-2014. We also evaluated the full 2000-2020 dataset with state agricultural statistics, finding strong agreement with median planting dates for maize (R 2 = 0.76, MAE = 4.4 days) and slightly lower agreement for soybeans (R 2 = 0.65, MAE = 5.4 days) when aggregated to the state level. We then used this new dataset to analyze environmental determinants of planting dates at a finer-scale than previously possible, controlling for unobserved variation at the sub-state district level. We found that during 2000-2020, each standard deviation increase in rainfall delayed planting by ~ 2.5 days, and fields with higher soil productivity ratings tended to be planted earlier. We did not find meaningful trends over the last two decades in planting dates for maize or soybeans, in contrast to trends towards earlier planting dates late last century and predicted for this period in climate adaptation studies. We hypothesize increases in early season rainfall may have inhibited these shifts towards earlier planting. Remotely sensed planting dates will be a useful tool for yield gap analyses, crop simulation modeling, and ongoing assessment of climate adaptation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

When to use rsync

We have endeavored to show, using a series of data transfer results obtained from two testbeds, when to use the popular data copying tool rsync and related tools. Tests have been conducted in local area network (LAN) and wide area network (WAN) environments. We conclude that for files in a certain size range and network latency &lE; 10 ms round trip time (RTT), rsync is still useful for data moving tasks in the category 4 of the U.S. DOE Technical Report “Data Movement Categories”. For more demanding data movement requirements, tools of different classes are suggested. Sample histograms from two DOE user facilities are provided to further support our conclusions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

International collaboration framework for the calculation of performance loss rates: Data quality, benchmarks, and trends (towards a uniform methodology)

Abstract The IEA PVPS Task 13 group, experts who focus on photovoltaic performance, operation, and reliability from several leading R&D centers, universities, and industrial companies, is developing a framework for the calculation of performance loss rates of a large number of commercial and research photovoltaic (PV) power plants and their related weather data coming across various climatic zones. The general steps to calculate the performance loss rate are (i) input data cleaning and grading; (ii) data filtering; (iii) performance metric selection, corrections, and aggregation; and finally, (iv) application of a statistical modeling method to determine the performance loss rate value. In this study, several high‐quality power and irradiance datasets have been shared, and the participants of the study were asked to calculate the performance loss rate of each individual system using their preferred methodologies. The data are used for benchmarking activities and to define capabilities and uncertainties of all the various methods. The combination of data filtering, metrics (performance ratio or power based), and statistical modeling methods are benchmarked in terms of (i) their deviation from the average value and (ii) their uncertainty, standard error, and confidence intervals. It was observed that careful data filtering is an essential foundation for reliable performance loss rate calculations. Furthermore, the selection of the calculation steps filter/metric/statistical method is highly dependent on one another, and the steps should not be assessed individually.

14 SOLAR ENERGY↗
DOE OSTI2022

Assessing the large-scale drivers of precipitation in the northeastern United States via linear orthogonal decomposition

This study examines the linear orthogonal modes associated with monthly precipitation in the northeastern United States, from CESM1 LENS (35 ensemble members, 1979–2005) and two reanalysis datasets (ERA5, 1950–2018 and NOAA-CIRES-DOE 20CRv3, 1950–2015). Calendar months are aggregated together, and any linear trends in data are removed. Using region-averaged precipitation anomaly time series and monthly anomalies for several global 2D atmospheric fields, a linear orthogonal decomposition method is implemented to iteratively extract time series (based on field and geographic location) of absolute maximum correlation. Linear modes associated with this method are then projected onto the full set of 2D fields to provide physical insight into the mechanisms involved in generating precipitation. In this region, the first mode is associated with vapor transport from the Atlantic seaboard, the second mode is characterized by westward vapor transport associated with extratropical cyclones, and the third mode captures vapor transport from the Gulf of Mexico during the fall and winter. However, the third mode is less robust in the spring and summer. Results are generally consistent across the datasets, and applying multiple linear regression with the linear modes to predict the precipitation anomalies produces R-squared values of around 0.54–0.65 for CESM1 LENS, and around 0.58–0.88 for reanalysis, with the lowest values generally in the spring and late summer. The influence of low-frequency climate variability on the modes is considered for CESM1 LENS, and the modes in late winter can be predicted with some success via a combination of several, prominent large-scale teleconnection patterns.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Parametric dynamic mode decomposition for reduced order modeling

Dynamic Mode Decomposition (DMD) is a model-order reduction approach, whereby spatial modes of fixed temporal frequencies are extracted from numerical or experimental data sets. The DMD low-rank or reduced operator is typically obtained by singular value decomposition of the temporal data sets. For parameter-dependent models, as found in many multi-query applications such as uncertainty quantification or design optimization, the only parametric DMD technique developed was a stacked approach, with data sets at multiple parameter values were aggregated together, increasing the computational work needed to devise low-rank dynamical reduced-order models. Here in this paper, we present two novel approach to carry out parametric DMD: one based on the interpolation of the reduced-order DMD eigen-pair and the other based on the interpolation of the reduced DMD (Koopman) operator. Numerical results are presented for diffusion-dominated nonlinear dynamical problems, including a multiphysics radiative transfer example. All three parametric DMD approaches are compared.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Linking structural and rheological memory in disordered soft materials

Linking the macroscopic flow properties and nanoscopic structure is a fundamental challenge to understanding, predicting, and designing disordered soft materials. Under small stresses, these materials are soft solids, while larger loads can lead to yielding and the acquisition of plastic strain, which adds complexity to the task. In this work, we connect the transient structure and rheological memory of a colloidal gel under cyclic shearing across a range of amplitudes via a generalized memory function using rheo-X-ray photon correlation spectroscopy (rheo-XPCS). Our rheo-XPCS data show that the nanometer scale aggregate-level structure recorrelates whenever the change in recoverable strain over some interval is zero. The macroscopic recoverable strain is therefore a measure of the nano-scale structural memory. We further show that yielding in disordered colloidal materials is strongly heterogeneous and that memories of prior deformation can exist even after the material has been subjected to flow.

Kamani, Krutarth M. [Univ. of Illinois at Urbana-C↗
DOE OSTI2026

ComStock Measure Documentation: High-Efficiency Rooftop Unit

Building on the 3-year End-Use Load Profiles project to calibrate and validate the U.S. Department of Energy's ResStock and ComStock models, this work produces national data sets that enable cities, states, utilities, and other stakeholders to answer a broad range of questions regarding their commercial building stock. ComStock is a highly granular, bottom-up model that uses various data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the commercial building stock across the United States. The "baseline" model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology of the baseline model is discussed in the ComStock Reference Documentation. The goal of this work is to develop energy efficiency and demand flexibility measures that cover market-ready technologies and study their mass adoption impact on the baseline building stock. "Measures" refers to various "what-if" scenarios that can be applied to buildings. The results for the baseline and measure scenario simulations are published in public data sets that provide insights into building stock characteristics, operational behaviors, utility bill impacts, and annual and sub-hourly energy usage by fuel type and end use. This report describes the modeling methodology for a single ComStock measure scenario - high-efficiency rooftop unit (RTU) - and briefly introduces key results. The full public data set can be accessed on the Comstock data lake or via the Data Viewer at comstock.nlr.gov. The public data set enables users to create custom aggregations of results for their use case (e.g., filter to a specific county or building type).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗