SEARCH · Engineering Papers

Results for “data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27

All sources NASA NTRS DOE OSTI

DOE OSTI2022

Parametric dynamic mode decomposition for reduced order modeling

Dynamic Mode Decomposition (DMD) is a model-order reduction approach, whereby spatial modes of fixed temporal frequencies are extracted from numerical or experimental data sets. The DMD low-rank or reduced operator is typically obtained by singular value decomposition of the temporal data sets. For parameter-dependent models, as found in many multi-query applications such as uncertainty quantification or design optimization, the only parametric DMD technique developed was a stacked approach, with data sets at multiple parameter values were aggregated together, increasing the computational work needed to devise low-rank dynamical reduced-order models. Here in this paper, we present two novel approach to carry out parametric DMD: one based on the interpolation of the reduced-order DMD eigen-pair and the other based on the interpolation of the reduced DMD (Koopman) operator. Numerical results are presented for diffusion-dominated nonlinear dynamical problems, including a multiphysics radiative transfer example. All three parametric DMD approaches are compared.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

Toward Molecule-Specific Geochemistry of Heavy Ends: Application to the Upstream Oil Industry

Detailed molecular analysis of the composition of petroleum is typically performed on only the light (volatile) fraction, using analyses involving gas chromatography (GC). Heavier (nonvolatile) components such as kerogen, bitumen, and asphaltenes, which are relevant to conventional reservoirs and particularly to unconventional gas shale and tight oil formations, are not amenable to analysis by GC and instead are typically studied by more phenomenological techniques. This report describes a suite of analytical techniques that can provide sufficient average molecular information on the heavy ends (defined as the nonvolatile fraction) to be useful for a variety of industrial applications, expanding the discipline of molecule-specific geochemistry into the non-GC amenable fractions of petroleum materials. Heavy ends are composed of the elements C, H, N, S, and O, and the bonding environments of each can be measured by various solid state spectroscopies: nuclear magnetic resonance (NMR) and infrared (IR) spectroscopies for C, H, and O; X-ray absorption near-edge structure (XANES) spectroscopy for S and N. Additionally, advances in mass spectrometry, including techniques such as laser desorption laser ionization mass spectrometry (L2MS), can be used to measure the molecular weight and dominant molecular architecture of asphaltenes. Moreover, higher-order structures can be measured: samples of kerogen with representative pore geometry can be prepared, and those pore systems are analyzed with techniques such as X-ray diffraction and neutron scattering. Colloidal structures of asphaltenes can be measured with a variety of techniques, which reveal a hierarchical aggregation structure. These analyses provide sufficient structural data to enable structure–property and structure–function relationships to be determined, which can be applied to the characterization of conventional and unconventional resources. In gas shale and tight oil formations, which contain abundant inorganic minerals as well as kerogen, diffuse reflectance infrared Fourier transform spectroscopy (DRIFTS) is used for simultaneous analysis of mineralogy, kerogen content, and maturity, typically performed at the wellsite. In conventional reservoirs, spatial variations in the asphaltene concentration measured by downhole fluid analysis logging tools are being interpreted with a recently developed equation of state to address upstream issues such as reservoir connectivity, tar mat formation, and fault block migration. Furthermore, as more heavy end analyses are developed, further scientific knowledge and industrial applications of heavy end geochemistry are anticipated.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2025

Linking structural and rheological memory in disordered soft materials

Linking the macroscopic flow properties and nanoscopic structure is a fundamental challenge to understanding, predicting, and designing disordered soft materials. Under small stresses, these materials are soft solids, while larger loads can lead to yielding and the acquisition of plastic strain, which adds complexity to the task. In this work, we connect the transient structure and rheological memory of a colloidal gel under cyclic shearing across a range of amplitudes via a generalized memory function using rheo-X-ray photon correlation spectroscopy (rheo-XPCS). Our rheo-XPCS data show that the nanometer scale aggregate-level structure recorrelates whenever the change in recoverable strain over some interval is zero. The macroscopic recoverable strain is therefore a measure of the nano-scale structural memory. We further show that yielding in disordered colloidal materials is strongly heterogeneous and that memories of prior deformation can exist even after the material has been subjected to flow.

Kamani, Krutarth M. [Univ. of Illinois at Urbana-C↗

DOE OSTI2017

Pattern Discovery in Internet Background Radiation

Internet Background Radiation (IBR) is observed in empty network address spaces. No traffic should arrive there, but it does in overwhelming quantities, gathering evidences of attacks, malwares and misconfigurations. The study of IBR helps to detect spreading network problems, common vulnerabilities and attack trends. However, network traffic data evolves quickly and is of high volume and diversity, i.e., an outstanding big data challenge. When used to assist network security, it also requires the online classification of dynamic streaming data. In this paper, we introduce an AGgregation & Mode (AGM) vector to represent network traffic. The AGM format characterizes IP hosts by extracting aggregated and mode values of IP header fields, and without inspecting payloads. We performed clustering and statistical analysis to explore six months of IBR from 2012 with the AGM mapping. The discovered patterns allow building a classification of IBR, which identifies phenomena that have been actively polluting the Internet for years. The AGM representation is light and tailored for monitoring and pattern discovery. We show that AGM vectors are suitable to analyze large volumes of network traffic: they capture permanent operations, such as long term scanning, as well as bursty events from targeted attacks and short term incidents.

Iglesias, Felix↗

DOE OSTI2014

MO-G-18C-05: Real-Time Prediction in Free-Breathing Perfusion MRI

Purpose: The aim is to minimize frame-wise difference errors caused by respiratory motion and eliminate the need for breath-holds in magnetic resonance imaging (MRI) sequences with long acquisitions and repeat times (TRs). The technique is being applied to perfusion MRI using arterial spin labeling (ASL). Methods: Respiratory motion prediction (RMP) using navigator echoes was implemented in ASL. A least-square method was used to extract the respiratory motion information from the 1D navigator. A generalized artificial neutral network (ANN) with three layers was developed to simultaneously predict 10 time points forward in time and correct for respiratory motion during MRI acquisition. During the training phase, the parameters of the ANN were optimized to minimize the aggregated prediction error based on acquired navigator data. During realtime prediction, the trained ANN was applied to the most recent estimated displacement trajectory to determine in real-time the amount of spatial Results: The respiratory motion information extracted from the least-square method can accurately represent the navigator profiles, with a normalized chi-square value of 0.037±0.015 across the training phase. During the 60-second training phase, the ANN successfully learned the respiratory motion pattern from the navigator training data. During real-time prediction, the ANN received displacement estimates and predicted the motion in the continuum of a 1.0 s prediction window. The ANN prediction was able to provide corrections for different respiratory states (i.e., inhalation/exhalation) during real-time scanning with a mean absolute error of < 1.8 mm. Conclusion: A new technique enabling free-breathing acquisition during MRI is being developed. A generalized ANN development has demonstrated its efficacy in predicting a continuum of motion profile for volumetric imaging based on navigator inputs. Future work will enhance the robustness of ANN and verify its effectiveness with human subjects. Research supported by National Institutes of Health National Cancer Institute Grant R01 CA159471-01.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2019

LBNL-Load

This software package is a set of Python-based tools for forecasting future time series of electrical power demand for a granular set of customer types in California, based on historical data on individual customers' demand, combined with aggregate forecasts of demand growth.

Gerke, Brian↗

DOE OSTI2026

ComStock Measure Documentation: High-Efficiency Rooftop Unit

Building on the 3-year End-Use Load Profiles project to calibrate and validate the U.S. Department of Energy's ResStock and ComStock models, this work produces national data sets that enable cities, states, utilities, and other stakeholders to answer a broad range of questions regarding their commercial building stock. ComStock is a highly granular, bottom-up model that uses various data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the commercial building stock across the United States. The "baseline" model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology of the baseline model is discussed in the ComStock Reference Documentation. The goal of this work is to develop energy efficiency and demand flexibility measures that cover market-ready technologies and study their mass adoption impact on the baseline building stock. "Measures" refers to various "what-if" scenarios that can be applied to buildings. The results for the baseline and measure scenario simulations are published in public data sets that provide insights into building stock characteristics, operational behaviors, utility bill impacts, and annual and sub-hourly energy usage by fuel type and end use. This report describes the modeling methodology for a single ComStock measure scenario - high-efficiency rooftop unit (RTU) - and briefly introduces key results. The full public data set can be accessed on the Comstock data lake or via the Data Viewer at comstock.nlr.gov. The public data set enables users to create custom aggregations of results for their use case (e.g., filter to a specific county or building type).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2026

ComStock Measure Documentation: Variable-Speed Pumps

Building on the 3-year End-Use Load Profiles project to calibrate and validate the U.S. Department of Energy's ResStock and ComStock models, this work produces national data sets that enable cities, states, utilities, and other stakeholders to answer a broad range of questions regarding their commercial building stock. ComStock is a highly granular, bottom-up model that uses various data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the commercial building stock across the United States. The "baseline" model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology of the baseline model is discussed in the ComStock Reference Documentation. The goal of this work is to develop energy efficiency and demand flexibility measures that cover market-ready technologies and study their mass adoption impact on the baseline building stock. "Measures" refers to various "what-if" scenarios that can be applied to buildings. The results for the baseline and measure scenario simulations are published in public data sets that provide insights into building stock characteristics, operational behaviors, utility bill impacts, and annual and sub-hourly energy usage by fuel type and end use. This report describes the modeling methodology for a single ComStock measure scenario - variable speed pumps - and briefly introduces key results. The full public data set can be accessed on the ComStock data lake or via the Data Viewer at comstock.nlr.gov. The public data set enables users to create custom aggregations of results for their use case (e.g., filter to a specific county or building type).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2018

Novel capabilities for examining and summarizing hierarchical and geographic resiliency profiles

The geospatial research and development team in the National and Homeland Security Division at Idaho National Laboratory was tasked with providing tools to derive insight from the substantial amount of data currently available - and continuously being produced - associated with the critical infrastructure of the US. This effort is in support of the Department of Homeland Security, whose mission includes the protection of this infrastructure and the enhancement of its resilience to hazards, both natural and human. We present geovisual-analytics-based approaches for analysis of vulnerabilities and resilience of critical infrastructure, designed so that decision makers, analysts, and infrastructure owners and managers can manage risk, prepare for hazards, and direct resources before and after an incident that might result in an interruption in service. Our designs are based on iterative discussions with DHS leadership and analysts, who in turn will use these tools to explore and communicate data in partnership with utility providers, law enforcement, and emergency response and recovery organizations, among others. In most cases these partners desire summaries of large amounts of data, but increasingly, our users seek the additional capability of focusing on, for example, a specific infrastructure sector, a particular geographic region, or time period, or of examining data in a variety of generalization or aggregation levels. These needs align well with tenets of in-formation-visualization design; in this paper, selected applications among those that we have designed are described and positioned within geovisualization, geovisual analytical, and information visualization frameworks.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

IntraShuffler: A Privacy Preserving Framework for Heterogeneous DP Federated Learning

Heterogeneous Differential Privacy (HDP) in Federated Learning (FL) allows clients to select individual privacy budgets () according to institutional policies and data sensitivity. In practice, many HDP-FL systems employ -aware server aggregation to improve model utility by re-weighting client updates according to their declared privacy budgets. However, gradient updates in FL retain structural patterns induced by non-independent and identically-distributed (non-IID) data, and these additional signals exposed by -aware aggregation create new opportunities for inference by an honest-but-curious server. In this work, we first show that a server equipped with gradient denoising and surrogate modeling can mount a Privacy Inference Attack that infers distributional attributes of clients and links updates from the same client across training rounds, measured via surrogate inference accuracy and linkage success, under realistic knowledge constraints. The Shuffle-Model has been widely studied as a defense against such inference risks by anonymizing update sources, but it is fundamentally incompatible with HDP-FL -aware aggregation. To address this challenge, we propose IntraShuffler, a middleware defense framework designed for HDP-FL systems. IntraShuffler introduces a privacy-aware shuffling mechanism that groups clients into privacy-compatible buckets and performs parameter-level shuffling within each bucket to disrupt persistent gradient structure while preserving -aware aggregation. Experiments across four different datasets show that IntraShuffler reduces gradient recoverability by over 60% and decreases surrogate inference accuracy from 0.78 to 0.33 while maintaining comparable model utility across multiple FL aggregation rules.

Riya, Farhin Farhad [ORNL]↗

DOE OSTI2025

Front-of-Meter Model Results

These files contains aggregations of key variables from the NREL Distributed Wind Futures Study using full parcel level data. These variables describe total technical and economic potential for distributed wind turbine deployment. Aggregations are available at the (1) county, (2) zipcode (zip code tabulation area or zcta), and (3) US Census block group level. Each scenario is coded with the scenario name (e.g., baseline) and year (e.g., 2022). Those files postfixed with 'econpot' contain results for only those parcels that are economically viable while the files postfixed with 'techpot' include results for all parcels that are technically feasible. Hence these correspond to technoeconomic and technical potential respectively. The data are available as CSV or Geopackage. Columns in the files are as follows: * geoid: geographic identifier (FIPS code or similar) * min_techpot_sum_kw: technical potential for all parcels in kW using turbines downsized to demand when appropriate * max_techpot_sum_kw: technical potential for all parcels in kW without downsizing turbines * aep_sum_kwh: annual energy production estimate in kWh * cf_mean_ratio: mean capacity factor * lcoe_mean_cents_per_kwh: mean levelized cost of energy for parcels in geography in cents per kWh * lcoe_std_cents_per_kwh: standard deviation of the above * parcel_area_sum_acres: total area of viable parcels in acres * n_turbines: number of cited turbines (one per viable parcel currently) Note: These are preliminary results from the full-parcel 2024 update of the Distributed Wind Energy Futures study. Please take care when making use of the data, and feel free to contact the team with any questions. Full documentation in support of these data is in progress and will follow.

17 WIND ENERGY↗

DOE OSTI2025

Behind-the-Meter Model Results

17 WIND ENERGY↗

DOE OSTI2024

Individual Data Sparsity in Smart Thermostat Big Data: Impacts on Modeling Thermostat Use Behavior Dynamics

This study explores the impacts of the sparsity of individual thermostat interaction data on modeling thermostat use behavior dynamics using a dataset of over 100,000 smart thermostats. In developing a data-driven model of Thermal Frustration Theory (TFT), we investigate the challenges and trade-offs in clustering occupant data to enhance predictive accuracy. Our findings reveal that a single, aggregated model fails to capture the diversity of occupant behaviors, resulting in extremely poor prediction performance. Conversely, excessive clustering exacerbates data sparsity, undermining model reliability. By identifying an optimal clustering strategy, we achieve a balance that significantly improves the prediction of manual setpoint changes during demand response (DR) events, enhancing energy management and occupant comfort

Fannon, David↗

DOE OSTI2016

Analysis of Application Power and Schedule Composition in a High Performance Computing Environment

As the capacity of high performance computing (HPC) systems continues to grow, small changes in energy management have the potential to produce significant energy savings. In this paper, we employ an extensive informatics system for aggregating and analyzing real-time performance and power use data to evaluate energy footprints of jobs running in an HPC data center. We look at the effects of algorithmic choices for a given job on the resulting energy footprints, and analyze application-specific power consumption, and summarize average power use in the aggregate. All of these views reveal meaningful power variance between classes of applications as well as chosen methods for a given job. Using these data, we discuss energy-aware cost-saving strategies based on reordering the HPC job schedule. Using historical job and power data, we present a hypothetical job schedule reordering that: (1) reduces the facility's peak power draw and (2) manages power in conjunction with a large-scale photovoltaic array. Lastly, we leverage this data to understand the practical limits on predicting key power use metrics at the time of submission.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

A novel stochastic modeling method to simulate cooling loads in residential districts

District cooling systems are widely used in urban residential communities in China. Most of such systems are oversized, which leads to wasted investment, low operational efficiency and, thus, waste of energy. The accurate prediction of district cooling loads that can support the rightsizing of cooling plant equipment remains a challenge. This study develops a novel stochastic modeling method that consists of (1) six prototype house models representing most apartments in a district, (2) occupant behavior models of residential buildings reflecting their spatial and temporal diversity as well as their complexity based on a large-scale residential survey in China, and (3) a stochastic sampling process to represent all apartments and occupants in the district. The stochastic method was applied to a case study using the Designer's Simulation Toolkit (DeST) to simulate the cooling loads of a residential district in Wuhan, China. The simulation results agreed well with the measured data based on five performance metrics representing the aggregated cooling consumption, the peak cooling loads, the spatial load distribution, the temporal load distribution and the load profiles. Two prevalent simulation methods were also employed to simulate the district cooling loads. Here, the results showed that oversimplified assumptions about occupant behavior could lead to significant overestimation of the peak cooling load and the total cooling loads in the district. Future work will aim to simplify the workflow and data requirements of the stochastic method for its application, and to explore its use in predicting district heating loads and in commercial or mixed-use districts.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2016

PRIMA-X Final Report

The PRIMA-X (Performance Retargeting of Instrumentation, Measurement, and Analysis Technologies for Exascale Computing) project is the successor of the DOE PRIMA (Performance Refactoring of Instrumentation, Measurement, and Analysis Technologies for Petascale Computing) project, which addressed the challenge of creating a core measurement infrastructure that would serve as a common platform for both integrating leading parallel performance systems (notably TAU and Scalasca) and developing next-generation scalable performance tools. The PRIMA-X project shifts the focus away from refactorization of robust performance tools towards a re-targeting of the parallel performance measurement and analysis architecture for extreme scales. The massive concurrency, asynchronous execution dynamics, hardware heterogeneity, and multi-objective prerequisites (performance, power, resilience) that identify exascale systems introduce fundamental constraints on the ability to carry forward existing performance methodologies. In particular, there must be a deemphasis of per-thread observation techniques to significantly reduce the otherwise unsustainable flood of redundant performance data. Instead, it will be necessary to assimilate multi-level resource observations into macroscopic performance views, from which resilient performance metrics can be attributed to the computational features of the application. This requires a scalable framework for node-level and system-wide monitoring and runtime analyses of dynamic performance information. Also, the interest in optimizing parallelism parameters with respect to performance and energy drives the integration of tool capabilities in the exascale environment further. Initially, PRIMA-X was a collaborative project between the University of Oregon (lead institution) and the German Research School for Simulation Sciences (GRS). Because Prof. Wolf, the PI at GRS, accepted a position as full professor at Technische Universität Darmstadt (TU Darmstadt) starting February 1st, 2015, the project ended at GRS on January 31st, 2015. This report reflects the work accomplished at GRS until then. The work of GRS is expected to be continued at TU Darmstadt. The first main accomplishment of GRS is the design of different thread-level aggregation techniques. We created a prototype capable of aggregating the thread-level information in performance profiles using these techniques. The next step will be the integration of the most promising techniques into the Score-P measurement system and their evaluation. The second main accomplishment is a substantial increase of Score-P’s scalability, achieved by improving the design of the system-tree representation in Score-P’s profile format. We developed a new representation and a distributed algorithm to create the scalable system tree representation. Finally, we developed a lightweight approach to MPI wait-state profiling. Former algorithms either needed piggy-backing, which can cause significant runtime overhead, or tracing, which comes with its own set of scaling challenges. Our approach works with local data only and, thus, is scalable and has very little overhead.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Development of a Reference Database for Beta-Delayed Neutron Emission

Beta-delayed neutron emission is important for nuclear structure and astrophysics as well as for reactor applications. Significant advances in nuclear experimental techniques in the past two decades have led to a wealth of new measurements that remain to be incorporated in the databases. In this work, we report on a coordinated effort to compile and evaluate all the available β-delayed neutron emission data. The different measurement techniques have been assessed and the data have been compared with semi-microscopic and microscopic-macroscopic models. The new microscopic database has been tested against aggregate total delayed neutron yields, time-dependent group parameters in 6-and 8-group re-presentation, and aggregate delayed neutron spectra. New recommendations of macroscopic delayed-neutron data for fissile materials of interest to applications are also presented. The new Reference Database for Beta-Delayed Neutron Emission Data is available online.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2016

Flexible Data-Aware Scheduling for Workflows over an In-Memory Object Store

This paper explores novel techniques for improving the performance of many-task workflows based on the Swift scripting language. We propose novel programmer options for automated distributed data placement and task scheduling. These options trigger a data placement mechanism used for distributing intermediate workflow data over the servers of Hercules, a distributed key-value store that can be used to cache file system data. We demonstrate that these new mechanisms can significantly improve the aggregated throughput of many-task workflows with up to 86x, reduce the contention on the shared file system, exploit the data locality, and trade off locality and load balance.

Duro, Francisco Rodrigo↗