Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

The Value of Transparency in Distributed Solar PV Markets

Distributed solar photovoltaic (PV) markets are relatively non-transparent: PV price and product information is not readily available, searching for this information is costly (in terms of time and effort), and customers are mostly unfamiliar with the new technology. Quote aggregation, where third-party companies collect PV quotes on behalf of customers, may be one way to increase PV market transparency. In this paper, quote aggregation data are analyzed to study the value of transparency for distributed solar PV markets. The results suggest that easier access to more quotes results in lower prices. We find that installers tend to offer lower prices in more competitive market environments. We supplement the empirical analysis with key findings from interviews of residential PV installers.

14 SOLAR ENERGY↗

Utilizing Commercial Real Estate Owner and Investor Data to Analyze the Financial Performance of Energy Efficient, High-Performance Office Buildings

Evidence has shown that owning and operating energy-efficient, high-performance, “green” properties results in multiple benefits including lower utility bills, higher rents, improved occupancy, and greater net operating income. However, it is difficult to isolate and control moderating factors to identify the specific drivers behind improved financial performance and value to investors that results from sustainability in real estate. DOE is interested in facilitating deeper investigation of the correlation between energy efficiency and financial performance, reducing data acquisition and matching challenges, and developing a stronger understanding of how sustainable design and energy efficiency impact value. DOE commissioned this pilot study to test the logistical and empirical procedures required to establish a Commercial Real Estate Data Aggregation & Trends Analysis lab, determine the potential benefits available through the lab, and contribute to the existing body of evidence in this field.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

A Parallel Graph Environment for Real-World Data Analytics Workflows

Economic competitiveness and national security depend increasingly on the insightful analysis of large data sets. The diversity of real-world data sources and analytic workflows impose challenging hardware and software requirements for parallel graph platforms. The irregular nature of graph methods is not supported well by the deep memory hierarchies of conventional distributed systems, requiring new processor and runtime system designs to tolerate memory and synchronization latencies. Moreover, the efficiency of relational table operations and matrix computations are not attainable when data is stored in common graph data structures. In this paper, we present HAGGLE, a high-performance, scalable data analytics platform. The platform’s hybrid data model supports a variety of distributed, thread-safe data structures, parallel programming constructs, and persistent and streaming data. An abstract runtime layer enables us to map the stack to conventional, distributed computer systems with accelerators. The runtime uses multithreading, active messages, and data aggregation to hide memory and synchronization latencies on large-scale systems.

Graph Analytics↗

Model run output for WM-ABM 2024 HESS Publication

Model output for computational experiments introducing WM-ABM. See associated HESS publication here: https://egusphere.copernicus.org/preprints/2023/egusphere-2023-1604/ all_nldas_cdl_data_v3.txt - Cropland Data Layer data aggregated to 1/8 degree resolution for input into the PMP calibration process. Variables include NLDAS ID, crop type, year, the count of pixels in the grid cell of the crop type, CDL crop ID, and GCAM crop ID. 20230115 ABM runs - WM-ABM model outputs for various scenarios described in HESS publication. If multiple versions exist (e.g., v2, v3, etc.) latest version indicates dataset generated for manuscript. The "baseline" run is the primary run, where as the "memXX" runs indicate sensitivity runs testing different parameters of agent memory. Output files starting with "abm_results_" include groundwater irrigated acres and surface water irrigated acres for each farm/agent combination. Other CSV outputs provide summary statistics of various agent and water shortage outcomes, used for generation of figures in the HESS manuscript.

ABM↗

Integrated GW Farm ABM

This Data Repository includes data used for the integrated groundwater- farm ABM model, raw model output from scenario ensemble, and processed outputs that isolate the groundwater storage depletion outcomes for the 35,000 farm cells. Model Inputs: Farm ABM Inputs: This folder contains the input data used by the integrated groundwater - farm ABM modelling script (Python file) used for the high performance computing (HPC) experiments. The sub-folder "data inputs" contains all of the farm attribute data, while the three files in the folder have the hydrogeological data lookup table (NLDAS Cost Curve Attributes.csv), a lookup table (Theis well function table.csv) for the groundwater cost curve function, and the farm indexes and corresponding NLDAS ids for all of the cells run in this experiment (nldas farms subset final.csv). NLDAS Cost curve hydrogeological data: Hydrogeological data aggregated to 1/8 degree resolution and aligned with the NLDAS grid. Parameters include: water depth below ground surface [meters], subsurface porosity [unitless], aquifer depth from ground surface to aquifer bottom [meters], annual average recharge (USGS: mm, Doll: meters), and three different hydraulic conductivity (K) values (meters/day). The three K values represent the mean value from Gleeson et al. (2018), one standard deviation above the mean from Gleeson et al. (2018), and the de Graaf et al. 2020 modifications to certain lithologies. Additional information about these datasets and their processing are documented in the supplement to Yoon et al. 2025 (in review). Output: Raw outputs: This folder contains a .zip file that has model outputs for the entire scenario ensemble. There is one csv for each farm id, using the format "farm farmid cases.csv". The relationship between the farm id and NLDAS id is defined by the "nldas farms subset final.csv" located in the Farm ABM Inputs folder. Each csv has 625 rows, corresponding to 625 combinations of different scenario parameter values. Each row (scenario) represents the outcome of a 100 year simulation. Columns define scenario settings and summary statistics for each scenario. The first four columns define the scenario settings: "hydro ratio," "econ ratio," "K scenario," and "gamma scenario." The hydro and econ ratios are values passed to the modeling script that influence multipliers for other model parameters, as documented in the supplement to Yoon et al. 2025 (in review). The gamma multiplier is a coefficient multiplier applied to the baseline gamma values (values below 1 represent lower unobserved costs compared to baseline, values above 1 represent higher costs). The K scenario names represent K values of: "low": 0.5 m/d, "int 1": 2.5 m/d, "int 2": 10 m/d, "high": 50 m/d, and "gleeson": mean Gleeson K value. "Perc vol depleted" is the fraction of groundwater depleted at the end of the 100 simulation. Processed Output: Derived depletion outcomes from raw outputs: All of the individual csv files from the Raw outputs were aggregated into a single file that has the scenario settings and fraction depletion "Perc vol depleted" for every farm cell, for every scenario. The other two files define relationships between the farm id, NLDAS id, and local and major aquifer units, used for aquifer-level depletion analysis.

Agent based modeling↗

Conflux – An Asynchronous 2-to-1 Multiplexor for Time-Division Multiplexing and Clock-less, Token-less Readout

This paper describes an independent, self-arbitrating asynchronous two-to-one multiplexor called Conflux. Conflux can be used to implement clock-less and token-less Time-Division Multiplexing between two sources. This capability can then be used to create a clock-less and token-less aggregate data bus which can be used as a complete asynchronous readout architecture for a chip, a part of a chip or an entire multi-chip system. Conflux utilizes a classic, four-phased asynchronous handshake on both of its input ports as well as on its output port. Asynchronous Request is bundled with Data to ensure consistent propagation delay. The Request is also differential to account for propagation delay differences between Logical Ones and Zeros. Arbitration between the two Conflux input streams is accomplished by three Set-Reset latches. Finally, Conflux cells were used successfully to implement the readout architecture of the FCP130 prototype chip. Tests indicate single Conflux stage delays of 1.8ns and a bandwidth of approximately 20 Gbps in 130nm CMOS.

43 PARTICLE ACCELERATORS↗

NLIT 2022 Axonius Presentation

Axonius, a cybersecurity asset management tool used at INL, provides visibility into users, devices, software, and hardware in use at the lab. This visibility, provided by aggregating data on lab entities from many tools' perspectives, makes incident response, configuration management, and IT operations more capable.

42 ENGINEERING↗

Irradiation-induced damage in concrete-forming aggregates: revisiting literature data through micromechanics

The radiation-induced volumetric expansion (RIVE) of aggregate-forming minerals causes damage in concrete exposed to high levels of fast neutrons fluence ( \(>\,\sim \,10^{19}\, \hbox {n}\,\hbox {cm}^{-2}\) at kinetics energy above 0.1 MeV). Historical post-irradiation RIVE and Young’s modulus data obtained in test reactors were revisited using a polycrystalline homogenization model (self-consistent scheme) accounting for the aggregates’ minerals content and the formation of voids/cracks during irradiation. It was found that the formation of extra voids/cracks can contribute to the aggregate expansion more substantially than the cumulated expansions of aggregate-forming minerals. Furthermore, the rate of void creation appears to be inversely correlated to the silica content, although aggregates of higher silicate contents exhibit higher RIVEs. The loss of Young’s modulus decreases exponentially with the aggregates’ expansion. At a comparable RIVE level, the relative loss of modulus is more significant in aggregates of higher silica content.

36 MATERIALS SCIENCE↗

Consolidated Hydropower Data Repository: Value and Opportunities

Hydropower is one of several types of generating assets that provides energy, capacity, and services to electric power systems. It does so under rubrics and objectives—market driven and regulated, internal and external to asset and fleet owners—that address reliability, cost, price, and, increasingly, flexibility of output. The aggregation of data from multiple hydropower units can provide insights into asset operations and maintenance practices and needs and assist in meeting hydropower objectives. This paper examines the concept and potential benefits of aggregating hydropower asset data—primarily supervisory control and data acquisition (SCADA) information—with examples of insights developed from data aggregated by the Hydropower Research Institute (HRI). Data aggregation as discussed herein, and as implemented by the HRI, extends beyond multiple units in a powerhouse and beyond multiple hydropower facilities in an electric utility fleet or river system. Examples of research and analytics from such aggregated datasets range from unit load dependency analyses to modeling sensor measurements to detect and diagnose anomalies in assets. These examples showed the benefits of utilizing the entire dataset for insights into how the sensor layout of a single unit or set of units compares to the hydropower industry overall. Such insights include whether additional sensors are needed to complete analyses or to make decisions. In addition, utilizing multiple sensors of the same kind within a unit can provide an indication of possible current or upcoming problems with equipment. Although other analyses are possible, their use requires the development of complex models and, potentially, access to types of data that are currently not included with the example dataset used in this study. However, the examples studied herein confirmed the value of data aggregation in the fleet and unit contexts, and the value extends beyond multiple units in a powerhouse and beyond multiple hydropower facilities in an electric utility fleet or river system. The assessments also provided insights into potential extensions to the data aggregation concept that could further add to their value to the hydropower community, and these are included in this document as a set of recommendations.

13 HYDRO ENERGY↗

Proactive Data Containers for Scientific Storage (Final Report)

Emerging HPC systems are expected to be deployed with an unprecedented level of complexity, due to a deep system memory/storage hierarchy and heterogeneity of the storage hardware. This hierarchy is expected to range from CPU cache through several levels of volatile memory to nonvolatile memory, traditional hard disks, and tape. Simple and efficient methods of data management and movement through this hierarchy is critical for scientific applications using exascale systems. Existing storage system and I/O (SSIO) technologies face severe challenges in dealing with these requirements. POSIX and MPI I/O standards that are the basis for existing I/O libraries and parallel file systems present fundamental challenges in the areas of scalable metadata operations, semantics-based data movement performance tuning, asynchronous operation, and support for scalable consistency of distributed operations. Moving toward new paradigms for SSIO in the extreme-scale era, we have proposed to investigate novel object-based data abstractions and storage mechanisms that take advantage of the deep storage hierarchy and enable proactive automated performance tuning. In order to achieve these overarching goals, we initiated an effort to develop a fundamental new data abstraction, called Proactive Data Containers (PDC). A PDC is a container within a locus of storage (memory, NVRAM, disk, etc.) that stores science data in an object-centric manner. Managing data as objects enables powerful optimization opportunities for data movement and transformations. The R&D focus of this project are: 1) formulation of object-oriented PDCs and their mapping in different levels of the exascale storage hierarchy; 2) efficient strategies for moving data in deep storage hierarchies using PDCs; 3) techniques for transforming and reorganizing data based on application requirements; and 4) novel analysis paradigms for enabling data transformations and user-defined analysis on data in PDCs. Toward achieving these overarching goals, we designed an object-centric application programing interface (API) for HPC, scalable metadata management for object-centric storage systems, and data movement optimizations such as Data Elevator for moving data between two levels of storage devices and TAPIOCA for efficient aggregation of data on compute nodes. We then implemented several components of the PDC system. They include metadata management, data placement services, remote procedure calls, data aggregation, etc. We have put them together into the overall PDC framework.

97 MATHEMATICS AND COMPUTING↗

BAMCensus (The Behavior and Advanced Mobility Census Dataset Aggregator) [SWR-25-120]

This software is a high-performance tool developed in Rust for downloading and processing large-scale geospatial datasets, specifically focusing on US Census data. It is designed to address scaling limitations found in existing tools, such as R's [tidycensus](https://walker-data.com/tidycensus/), by providing performant streaming dataset JOIN operations between various US Census datasets (like ACS and LEHD) and their corresponding geometries stored on the TIGER/Lines web server. The tool automates the process of joining these data sources, returning aggregated data to the user based on a specified census GEOID type. The tool automates the process of joining these data sources, returning aggregated data to the user based on a specified census GEOID type. Its primary motivation stems from the need for a high-performance solution to combine spatial datasets with graph traversals within the context of mobility analysis tooling being developed at NREL's Behavior and Advanced Mobility (BAM) group.

Fitzgerald, Robert [National Renewable Energy Labo↗

Atmospheric Moisture Budget and Spatial Resolution Dependence of Precipitation Extremes in Aquaplanet Simulations

This study investigates the resolution dependency of precipitation extremes in an aqua-planet framework. Strong resolution dependency of precipitation extremes is seen over both tropics and extra-tropics, and the magnitude of this dependency also varies with dynamical cores. Moisture budget analyses based on aqua-planet simulations with the Community Atmosphere Model (CAM) using the Model for Prediction Across Scales (MPAS) and High Order Method Modeling Environment (HOMME) dynamical cores but the same physics parameterizations suggest that during precipitation extremes moisture supply for surface precipitation is mainly derived from advective moisture convergence. The resolution dependency of precipitation extremes mainly originates from advective moisture transport in the vertical direction. At most vertical levels over the tropics and in the lower atmosphere over the subtropics, the vertical eddy transport of mean moisture field dominates the contribution to precipitation extremes and its resolution dependency. Over the subtropics, the source of moisture, its associated energy, and the resolution dependency during extremes are dominated by eddy transport of eddies moisture at the mid- and upper-troposphere. With both MPAS and HOMME dynamical cores, the resolution dependency of the vertical advective moisture convergence is mainly explained by dynamical changes (related to vertical velocity or omega), although the vertical gradients of moisture act like averaging kernels to determine the sensitivity of the overall resolution dependency to the changes in omega at different vertical levels. The natural reduction of variability with coarser resolution, represented by areal data averaging (aggregation) effect, largely explains the resolution dependency in omega. The thermodynamic changes, which likely result from non-linear feedback in response to the large dynamical changes, are small compared to the overall changes in dynamics (omega). However, after excluding the data aggregation effect in omega, thermodynamic changes become relatively significant in offsetting the effect of dynamics leading to reduce differences between the simulated and aggregated results. Compared to MPAS, the simulated stronger vertical motion with HOMME also results in larger resolution dependency. Compared to the simulation at fine resolution, the vertical motion during extremes is insufficiently resolved/parameterized at the coarser resolution even after accounting for the natural reduction in variability with coarser resolution, and this is more distinct in the simulation with HOMME. To reduce uncertainties in simulated precipitation extremes, future development in cloud parameterizations must address their sensitivity to spatial resolution as well as dynamical cores.

aqua-planet↗

Aggregated carbon dioxide flux and hydrometeorology data from an Amazonian palm swamp peatland in Peru: 2018, 2019, and 2022

This dataset contains eddy covariance carbon dioxide flux and hydrometeorological measurements made in an Amazonian palm swamp peatland near Iquitos, Peru. These data have been aggregated from half-hourly observations that are available from AmeriFlux (https://ameriflux.lbl.gov/; site PE-QFR). These data files are CSV (comma separated values) format and can be imported using Matlab, R, or Excel. Three full years of data are reported (2018, 2019 and 2022) during which time there were large differences in annual net ecosystem carbon dioxide exchange. The gap was caused by instrument malfunction and extended delays in repairs because of the Covid-19 pandemic. This research was conducted to better understand the carbon cycle of tropical peatlands, and was supported by the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research, Terrestrial Ecosystem Science Program, under Award Number DE-SC0020167.

54 ENVIRONMENTAL SCIENCES↗

Real-Time Lifetime Prediction of Semiconductor Devices Using Hardware-in-the-Loop

This paper presents a unique approach to enable real-time lifespan prediction of semiconductor power modules using a Hardware-in-the-Loop (HIL) system. By integrating the module's overall loss characteristics-specifically switching and conduction losses-with a thermoelectric model of the thermal management system, this research demonstrates that the model can dynamically estimates the junction temperature profile of the semiconductor devices in response to a changing torque demand profile for the motor drive system. This capability enables continuous monitoring of the module's operational time and cumulative stress induced on the devices to compute accumulated remaining lifetime or time-to-failure (TTF). This study provides an architectural framework for the HIL system with high-fidelity component models of multiple physical domains, allowing simulation of dynamic behaviors of a closely-coupled motor drive system. The advanced real-time computation and measurement functionalities of the HIL system allow for both dynamic lifetime calculations based on simulated data and aggregate lifetime predictions utilizing historical data. Moreover, this paper details an algorithm that not only computes cumulative damage but also synthesizes these data into a comprehensive aggregated lifetime metric. This methodology can enhance the maintenance scheduling strategies and operational reliability of semiconductor devices in critical applications, ultimately extending their service life while optimizing performance.

hardware-in-the-loop (HIL)↗

INGENIOUS Thermal Conductivity Measurement Source Categorization

Thermal conductivity (TC) data taken for different wells at a specified drill depth. This is an abridged version of the complete SMU heat flow database, downloaded from the SMU node of the NGDS at the beginning of INGENIOUS (approximately April 2021), and filtered to the INGENIOUS study area. This National Geothermal Data System (NGDS) project aggregates geothermal data collected and curated by the SMU Geothermal Laboratory and its partner organizations. All columns in this database are the same as the SMU database, except for 2 additions associated with this project. Repeated columns are for data correlation purposes. Column descriptions and data types are the same as previous iterations of the SMU database. The new values that are the addition are two new columns developed as part of the INGENIOUS project: INGENIOUS TC Value | INGENIOUS notes INGENIOUS notes are individual notes that were written for specific data points during the analysis process. There are not always notes associated with each input value. INGENIOUS TC Value includes 4 values: 1. Assumed Measured These are values that are assumed to be measured thermal conductivity values, either within a specific well or within the same study region. Many of these have either a published reference, a reported standard deviation, or a unique thermal conductivity value. 2. Data release - assumed measured These are values in the SMU database that are from proprietary data that were added to the SMU database and are labeled as data release for their reference. These values were searched for in person at the SMU Geothermal Laboratory as well as virtual examination of data available on the NGDS. For many of these, there are reported thermal conductivity values associated with the heat flow data in the database, but no specific table or reference to measurements in the original data release files. 3. Known measured These are values that have a reported measurement, either as an original file in the SMU data files on the NGDS or a reported table in a publication. In the rare circumstances, Maria Richards or David Blackwell confirmed measurement. Confirmation of measurement would be written in the INGENIOUS notes column. 4. Unmeasured Unmeasured values are those that are known to be unmeasured, either estimated from another report or no information given. In the SMU database, there are wells that have a heat flow but no thermal conductivity. These are categorized as unmeasured. There are also heat flow values that are stated to have estimated or generalized average thermal conductivity values for the region and rock type. Because these are known to be unmeasured, they are categorized as such. 5. Blank Blank values are either A quality or X quality. These quality values are stated in the INGENIOUS notes. These values were not going to change associated with the heat flow analysis, so these were not examined.

15 GEOTHERMAL ENERGY↗

Monthly Quality-filtered Aggregation of NOAA Climate Data Record (CDR) of AVHRR Leaf Area Index (LAI) and Fraction of Absorbed Photosynthetically Active Radiation (FAPAR), Version 5

This dataset contains gridded monthly Leaf Area Index (LAI) derived from the daily NOAA Climate Data Record (CDR) of AVHRR Leaf Area Index (LAI) and Fraction of Absorbed Photosynthetically Active Radiation (FAPAR), Version 5. This data record spans from 1981 to 2018 using data from eight NOAA polar orbiting satellites: NOAA-7, -9, -11, -14, -16, -17, -18 and -19. The data are projected on a 0.05 degree x 0.05 degree global grid, as in the original CDR. The original CDR is one of the Land Surface CDR Version 5 products produced by the NASA Goddard Space Flight Center (GSFC) and the University of Maryland (UMD), which is accompanied by algorithm documentation, data flow diagram and source code for the NOAA CDR Program. This dataset is in the netCDF-4 file format following ACDD and CF Conventions. This dataset has applied quality assurance information to only include "OK" data from the original CDR in the monthly aggregation.

Vermote, Eric [NASA Goddard Space Flight Center (G↗

Monthly Quality-filtered Aggregation of NOAA Climate Data Record (CDR) of AVHRR (Version 5) and VIIRS (Version 1) Leaf Area Index (LAI) and Fraction of Absorbed Photosynthetically Active Radiation (FAPAR)

This dataset contains gridded monthly Leaf Area Index (LAI) derived from the daily NOAA Climate Data Record (CDR) of AVHRR (Version 5) and VIIRS (Version 1) Leaf Area Index (LAI) and Fraction of Absorbed Photosynthetically Active Radiation (FAPAR). This data record spans from 1981 to 2024 using data from NOAA polar orbiting satellites: NOAA-7, -9, -11, -14, -16, -17, -18, -19 and S-NPP. The data are projected on a 0.05 degree x 0.05 degree global grid, as in the original CDR. The original CDR is one of the Land Surface CDR products produced by the NASA Goddard Space Flight Center (GSFC) and the University of Maryland (UMD), which is accompanied by algorithm documentation, data flow diagram and source code for the NOAA CDR Program. This dataset is in the netCDF-4 file format following ACDD and CF Conventions. This dataset has applied quality assurance information to only include "OK" data from the original CDR in the monthly aggregation.

Vermote, Eric [NASA Goddard Space Flight Center (G↗