Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 559 records · Page 31
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

A National Infrastructure for Artificial Intelligence on the Grid (NI4AI) (Final Scientific/Technical Report)

Electric utilities have traditionally taken a very pragmatic yet myopic approach with grid sensors and the resulting collected data. Sensors are purchased and deployed to solve a specific, known problem that has risen to sufficient awareness as to justify the effort of deploying sensors and the needed capital investment. This sensor data flows into proprietary software packages with limited functionality intended only to address the initial problem. This approach aligns with the financial incentives of the utility to deploy capital into fixed hardware assets for which the corporations earn a rate of return. This mentality stands in stark contrast to the big data revolution that started nearly 25 years ago with the rise of Google. In this worldview, data is a fundamental business asset; successful organizations collect, store, explore, merge, and exploit as much data as possible to not only solve problems well understood today but also to tackle new problems that will inevitably rise tomorrow. The ARPA-E Open Innovation 2018 project entitled A National Infrastructure for Artificial Intelligence on the Grid or NI4AI for short was designed to demonstrate this alternative paradigm for using data. To do this, the project was composed of three key thrust areas. The first major component deployed a variety of high-frequency grid sensors and captured terabytes of both wide-scale and localized grid measurements, generating high-value datasets for grid research and algorithm development. The second aspect made available PingThings’ PredictiveGridTM, a horizontally scalable, cloud-based data management and AI platform built for time series data to explore and exploit the collected data. Finally, the project fostered a diverse and open research community composed of experts from numerous fields through focused educational content, code sharing, and data science competitions. Shifting away from “single use” sensors and closed data silos within electric utilities is a major benefit to the public at large. This legacy approach to data is incredibly (1) capital intensive (new sensors must be deployed for each new problem and problems tend to arise continuously) and (2) painfully slow (new problems must be identified first and then new sensors must be deployed to collect data to begin to address the issue). The transition to a carbon neutral grid requires a massive transformation of the existing grid infrastructure and will continue to challenge the legacy grid in unforeseen ways. The only way to make the energy transition cost effective is for utilities to abandon this dated data paradigm and adopt more contemporary approaches. NI4AI has shown that it is technically possible and economically feasible to ingest, explore, and exploit grid data collected from even very high frequency sensing, such as continuous point on wave sensors collecting measurements 10,000 times a second. In fact, the PredictiveGrid platform used is commercially available and deployed at several utilities in the United States. Project accomplishments were numerous and included (1) making available a state of the art time series platform to the community, (2) collecting over 520 streams of time series data from grid sensors totaling over 1 trillion grid measurements, and (3) developing and nurturing a community within the industry focused on the use of data to create value for utilities and, ultimately, end consumers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Contrasting stream nitrate and sulfate response to recovery from experimental watershed acidification

Improvements in air quality have led to ecosystem recovery from acidic deposition, but the mechanisms and trajectories of this recovery are not fully understood. Here, we present long-term stream response and recovery data for paired watersheds at the Bear Brook Watershed in Maine (BBWM) during declining ambient SO 4 and NO 3 in precipitation. East Bear (EB) received ambient deposition from 1989 to 2018; West Bear (WB) received artificially elevated N+S from 1989 to 2016. The WB treatment was discontinued after 2016, the beginning of the recovery from both the experimental N+S and ambient decline. Stream SO 4 in WB gradually declined after the treatment ended, from ~147 µeq L -1 in 2010-16 to ~126 µeq L -1 in 2017-18. The declining S inputs induced desorption of SO 4 from soil phase surfaces, with stream loss far exceeding precipitation input. At the current rate of recovery, it will be many decades before the WB stream returns to pre-treatment SO 4 concentrations. In contrast, NO 3 is only weakly adsorbed in soil, and WB stream NO 3 concentrations rapidly declined from ~39 µeq L -1 in 2010-16 to ~5 µeq L -1 in 2017-18, comparable to the N-limited EB stream. The acid anions are strongly coupled to base cation chemistry in streams, and there was a distinct hysteretic response of Ca and Mg to the chronic acidification, as (Ca+Mg) increased rapidly during the initial years, followed by declining values due to depletion of the soil exchange complex. This 30-year monitoring study (1989-2019) provides insights into recovery mechanisms from acidic deposition and highlights the role of abiotic processes in soil that mediate nutrient cycling and retention. Finally, documenting the rapid response of N alongside the slower recovery for S identifies the temporal resolution necessary for other whole-watershed recovery studies.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)

This data release provides all data and code used in the paper " "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)" to model stream temperature, evaluate, and assess results. The associated manuscript explores current open questions in prediction in ungauged and unmonitored basins concerning top-down versus bottom-up approaches, tradeoffs between data available and input requirements, and the appropriate representation of catchment attributes as inputs to deep learning models. Modeling was done primarily with long short-term memory (LSTM) models, and stream site coverage spans 1362 locations across the conterminous United States. The data is organized into these items items:Code repository and data for the paper " "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)".Code: stream_temp_ml_regionalization.zip contains the code repositoryData to run the code: - data_dir.zip -- contains all files that should be moved to the "DATA_DIR" variable defined in the "set_env_vars.sh" script in the code repository- metadata_dir.zip -- contains all files that should be moved to the "METADATA_DIR" variable defined in the "set_env_vars.sh" script in the code repository- error_analysis_attribute_and_groundwater_dir.zip - workflows for the extended error analysis by stream attribute and groundwater influenceData produced by the code and used in the paper:- outputs_dir.zip - contains model output and results (outputs_dir/results), model weights (outputs_dir/models), and all other outputs used for the paper including feature importances.To cite this code, please use the following BibTeX or MLA entries:bibtex:@misc{willard2024streamdata, author = {Jared Willard and Fabio Ciulla and Helen Weierbach and Vipin Kumar and Charuleka Varadharajan}, title = {Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models"}, year = {2024}, doi = {10.15485/2448016}, publisher = {ESS-DIVE Repository}, url = {https://doi.org/10.15485/2448016}}MLA: Willard, Jared, et al. Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models". 2024. ESS-DIVE Repository, doi:10.15485/2448016.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Probability of Forming Gaps in the GD-1 Stream by Close Encounters of Globular Clusters

One of the most intriguing properties of the GD-1 stellar stream is the existence of three gaps. If these gaps were formed by close encounters with dark matter subhalos, the GD-1 stream opens an exciting window through which we can see the size, mass, and velocity distributions of the dark matter subhalos in the Milky Way. However, in order to use the GD-1 stream as a probe of the dark matter substructure, we need to disprove that these gaps are not due to the perturbations from baryonic components of the Milky Way. Here we ran a large number of test-particle simulations to investigate the probability that each of the known globular clusters (GCs) can form a GD-1-like gap, by using the kinematical data of the GD-1 stream and GCs from Gaia early data release 3 and by fully taking the observational uncertainty into account. We found that the probability that all of the three gaps were formed by GCs is as low as 1.7 × 10 -5 , and the expected number of gaps formed by GCs is only 0.057 in our fiducial model. Our result highly disfavors a scenario in which GCs form the gaps. Given that other baryonic perturbers (e.g., giant molecular clouds) are even less likely to form a gap in the retrograde-moving GD-1 stream, we conclude that at least one of the gaps in the GD-1 stream was formed by dark matter subhalos if the gaps were formed by flyby perturbations.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Continental Scale Hydrostratigraphy: Comparing Geologically Informed Data Products to Analytical Solutions

Abstract This study synthesizes two different methods for estimating hydraulic conductivity (K) at large scales. We derive analytical approaches that estimate K and apply them to the contiguous United States. We then compare these analytical approaches to three‐dimensional, national gridded K data products and three transmissivity (T) data products developed from publicly available sources. We evaluate these data products using multiple approaches: comparing their statistics qualitatively and quantitatively and with hydrologic model simulations. Some of these datasets were used as inputs for an integrated hydrologic model of the Upper Colorado River Basin and the comparison of the results with observations was used to further evaluate the K data products. Simulated average daily streamflow was compared to daily flow data from 10 USGS stream gages in the domain, and annually averaged simulated groundwater depths are compared to observations from nearly 2000 monitoring wells. We find streamflow predictions from analytically informed simulations to be similar in relative bias and Spearman's rho to the geologically informed simulations. R ‐squared values for groundwater depth predictions are close between the best performing analytically and geologically informed simulations at 0.68 and 0.70 respectively, with RMSE values under 10 m. We also show that the analytical approach derived by this study produces estimates of K that are similar in spatial distribution, standard deviation, mean value, and modeling performance to geologically‐informed estimates. The results of this work are used to inform a follow‐on study that tests additional data‐driven approaches in multiple basins within the contiguous United States.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

ESnet-JLab FPGA Accelerated Transport (data plane) [EJFAT (udplb)] v1.0

The ESnet-JLab FPGA Accelerated Transport system is a solution for streaming high-speed scientific measurement data from Data Acquisition Systems (DAQs) to high-performance computing facilties. It is generally compatible with many science workflows, and makes no assumptions about the specifics of any particular experiment. This program (udplb) implements the data plane portion of the EJFAT system. It is an FPGA design that rewrites and forwards data packets from a UDP-based scientific workflow to high-performance compute nodes. It depends on another program (udplbd, disclosed separately) to implement the control system.

Bengough, Peter [Malleable Networks, Inc.]↗
DOE OSTI2026

Real-time plasma monitoring framework for advanced plasma control and ML-research in DIII-D

Real-time and adaptive plasma control is crucial for robust tokamak operation, requiring sensitivity and tolerance measurements of the plasma state. This paper presents the implementation of an integrated real-time plasma monitoring framework on the DIII-D tokamak to support advanced control approaches, including machine-learning (ML) methods. The system is built on the SHIELD framework, a high-performance modular architecture that provides a unified pipeline for integrating diverse diagnostics. The framework leverages high-bandwidth digitizers, fast numerical processing, and deterministic, low-latency interconnects to stream high-fidelity data from diagnostics such as electron cyclotron emission (ECE), beam emission spectroscopy (BES), CO interferometers, and a visible tangential divertor camera (TangTV). The system’s validity is demonstrated through direct comparisons of real-time and offline data. Furthermore, we present two key applications of the developed plasma monitoring system with ML-based plasma control strategies, including real-time divertor detachment and active Alfvén Eigenmode control. As a result, this work presents a robust and scalable approach for integrating high-frequency, multidimensional diagnostics into advanced control algorithms for future fusion devices.

AI/ML↗
DOE OSTI2021

Accurate and confident prediction of electron beam longitudinal properties using spectral virtual diagnostics

Abstract Longitudinal phase space (LPS) provides a critical information about electron beam dynamics for various scientific applications. For example, it can give insight into the high-brightness X-ray radiation from a free electron laser. Existing diagnostics are invasive, and often times cannot operate at the required resolution. In this work we present a machine learning-based Virtual Diagnostic (VD) tool to accurately predict the LPS for every shot using spectral information collected non-destructively from the radiation of relativistic electron beam. We demonstrate the tool’s accuracy for three different case studies with experimental or simulated data. For each case, we introduce a method to increase the confidence in the VD tool. We anticipate that spectral VD would improve the setup and understanding of experimental configurations at DOE’s user facilities as well as data sorting and analysis. The spectral VD can provide confident knowledge of the longitudinal bunch properties at the next generation of high-repetition rate linear accelerators while reducing the load on data storage, readout and streaming requirements.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

Multiscale drivers of extreme southern California flooding: ENSO, MJO, North Pacific jet, and atmospheric rivers

Extreme rainfall and flooding, driven by a powerful atmospheric river (AR) and a persistent Madden-Julian Oscillation (MJO), hit Southern California in February 2024 during the 2023–2024 El Niño, affecting over 10 million people. ARs are key contributors to extreme rainfall and flooding along the U.S. West Coast. Although the AR-MJO link has been documented, its spatio-temporal variability remains a major forecasting and risk-management challenge. Combining precipitation, stream gauge and demographic data, we quantify the physical drivers and population exposure to this extreme event. Leveraging a Lagrangian MJO precipitation tracking algorithm, we unravel the multiscale interactions responsible for the AR’s development. El Niño favored a large, long-lived MJO that interacted with the North Pacific Jet (NPJ) over more than three weeks. The MJO convective outflow modulated the NPJ by inducing negative potential vorticity advection along the tropopause. The ensuing NPJ extension and acceleration induced explosive cyclogenesis, whose AR-driven moisture transport resulted in extreme rainfall.

Atmospheric dynamics↗
DOE OSTI2024

DUNE Database Development

The DUNE experiment will produce vast amounts of metadata, which describe the data coming from the read-out of the primary DUNE detectors. Various databases will make up the overall DB architecture for this metadata. ProtoDUNE at CERN is the largest existing prototype for DUNE and serves as a testing ground for - among other things - possible database solutions for DUNE. The subset of all metadata that is accessed during offline data reconstruction and analysis is referred to as ‘conditions data’ and it is stored in a dedicated database. As offline data reconstruction and analysis will be deployed on HTC and HPC resources, conditions data is expected to be accessed at very high rates. It is therefore crucial to store it in a granularity that matches the expected access patterns allowing for extensive caching. This requires a good understanding of the sources and use cases of conditions data. This contribution will briefly summarize the database architecture deployed at ProtoDUNE and explain the various sources of conditions data. We will present how the conditions data is retrieved and streamed from the databases and how it is handled to match expected access patterns.

Vizcaya Hernandez, Ana Paula↗
DOE OSTI2026

Cosmological neutrino mass: a frequentist overview in light of DESI

We derive constraints on the neutrino mass using a variety of recent cosmological datasets, including DESI BAO, the full-shape analysis of the DESI matter power spectrum and the one-dimensional power spectrum of the Lyman-α forest (P1D) from eBOSS quasars as well as the cosmic microwave background (CMB). The constraints are obtained in the frequentist formalism by constructing profile likelihoods and applying the Feldman-Cousins prescription to compute confidence intervals. This method avoids potential prior and volume effects that may arise in a comparable Bayesian analysis. Parabolic fits to the profiles allow one to distinguish changes in the upper limits from variations in the constraining power σ of the different data combinations. We find that all profiles in the ΛCDM model are cut off by the ∑m ν ≥ 0 bound, meaning that the corresponding parabolas reach their minimum in the unphysical sector. The most stringent 95% C.L. upper limit is obtained by the combination of DESI DR2 BAO, Planck PR4 and CMB lensing at 53 meV, below the minimum of 59 meV set by the normal ordering. The corresponding constraining power σ is 43 meV, which highlights the importance of the cut-off by negative values in the determination of the upper limit. Extending ΛCDM to non-zero curvature and w 0 w a CDM relaxes the constraints past 59 meV again, but only w 0 w a CDM exhibits profiles with a minimum at a positive value. Additionally, we extend the formalism to constrain the lightest neutrino mass. For DESI DR2 BAO, Planck PR4 and CMB lensing, we find confidence limits at 20 and 19 meV for normal and inverted ordering, respectively. Using a combination of DESI DR1 full-shape, BBN and eBOSS Lyman-α P1D, we successfully constrain the neutrino mass independently of the CMB. This combination yields m l ≤ 97 and 98 meV in the normal and inverted orderings, and total neutrino mass ∑m ν ≤ 285 meV (95% C.L.). The addition of DESI full-shape or Lyman-α P1D to CMB and DESI BAO results in small but noticeable improvement of the constraining power of the data. Lyman-α free-streaming measurements especially improve the constraint. Since they are based on eBOSS data, this sets a promising precedent for upcoming DESI data.

Frequentist statistics↗
DOE OSTI2025

Enhancing Smart Home Privacy: A Tutorial on Local Differential Privacy Techniques for Frequency and Mean Estimation

The ubiquity of Internet of Things (IoT) systems has seamlessly integrated into our daily lives, particularly in smart homes where devices continuously monitor and optimize our living environments. These systems significantly contribute to home automation, energy efficiency, and overall comfort. However, this widespread connectivity poses inherent risks linked to the streaming of sensitive household data, necessitating robust privacy preservation mechanisms. This tutorial systematically examines privacy preservation through local differential privacy (LDP), with a particular focus on frequency and mean estimation techniques for smart home applications. Here, we present a comprehensive taxonomy of smart home data formats and provide detailed implementation guidance for event-based and w-event LDP mechanisms. Through practical examples using smart thermostats and HVAC systems, we demonstrate how these techniques can be effectively deployed in real-world scenarios. The tutorial concludes by examining emerging research directions, including adaptive privacy budgets and federated learning approaches, establishing a foundation for privacy-preserving smart home deployments.

Kotevska, Olivera [Oak Ridge National Laboratory (↗
DOE OSTI2022

HIDES (Hybrid Intrusion Detection for Energy Systems) [SWR-19-65]

Hybrid Intrusion Detection for Energy Systems (HIDES) is a conglomerate detector that incorporates three major components: (1) Signature-based detection that involves information technology (IT) related anomaly detection rules, (2) Behavior-based detection that utilizes specially crafted rulesets for detecting abnormal behavior regarding the power system's supervisory control and data acquisition (SCADA) communication streams, and (3) Learning-based detection that combines SCADA communication with phasor measurements to identify otherwise non-detectible attack vectors. The software itself interconnects intrusion detection engine (IDE) enabling communications between the generated OT and IT alerts with phasor measurements for the learning-based detection system. The resulting detection system can detect a wide array of IT, OT, and physical attacks in a real-time setting.

Rivera, Joshua↗
DOE OSTI2021

Land Surface Modeling 2.0 for agricultural climate change impact assessments

Focus Area: This white paper addresses DOE AI4ESP focus area #2 by providing a sketch blueprint for a next generation, hybrid AI/process-based global Land Surface Modeling (LSM) framework to improve projections of climate change impacts on the land surface system including agriculture and the hydrological cycle. Science Challenge: Climate change impacts on agriculture are highly uncertain: -50% to +150% global production changes for major grains under high-end climate change, for example. The core challenge in land surface system predictability is the large gap between the scale at which the relevant biological processes act and the scale at which the risks need to be assessed. Conventional empirical and process-based modeling approaches are insufficient. A new multi-scale modeling paradigm which employs AI/ML to learn from new streams of remote sensing data and targeted ’gene-to-global’ simulations is needed.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Colorado State University Extension Industrial Assessment Center

Since its inception in 1984, the Colorado State University Industrial Assessment Center has performed industrial assessments at more than 720 manufacturing facilities in Colorado, Montana, Nebraska, Nevada, New Mexico, North Dakota, South Dakota, Utah, and Wyoming. From 2017 to 2019 there was a funding gap and the IAC shutdown. In 2020 through DOE extension funding the University relaunched the IAC as an extension center in order to provide assessments to underserved areas. Under this award, the CSU Industrial Assessment Center (IAC) was rebuilt with the help of student employees and the director. The CSU team experienced difficulty as the program was in the process of being restarted right as the 2020 pandemic hit. Nonetheless, the CSU IAC was instrumental in providing energy assessments to manufacturers in Colorado and Wyoming during the period of performance of 09/2019 – 12/2022. The Department of Energy's Industrial Assessment Centers (IACs) provide a valuable service to small and medium-sized manufacturers seeking to optimize their operations. These university-based centers offer no-cost, on-site assessments conducted by engineering faculty and students, analyzing energy consumption, production processes, and waste streams. They utilize advanced data acquisition systems to record operational data, and then use the data to create assessment recommendations. These recommendations form the foundation of the comprehensive energy report. The comprehensive report delivers actionable recommendations for enhancing energy efficiency, reducing waste, and reducing greenhouse gas emissions and improving productivity, often identifying significant cost savings. Furthermore, IACs facilitate access to implementation grants, enabling the businesses to readily adopt these improvements. This program not only strengthens individual businesses but also contributes to national goals of training the next generation of energy experts, as well as increasing industrial competitiveness and reduced environmental impact.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

An Efficient Storage-Driven Machine Learning Model for Performance in the Era of Multimodal Scientific Data

Scientific workflows are increasingly relying on machine learning (ML), simulation, and hybrid techniques to predict, understand, and optimize the behavior of complex experiments. High-performance computing has greatly improved researchers’ ability to acquire diverse data modalities in these workflows. Recent studies suggest that the performance of machine learning models can be improved by integrating data from various sources. Unfortunately, these workloads pose unprecedent pressure on the network storage to meet the demands associated with accessing these multimodal data. To mitigate the impact of intensive IO, we propose a solution that utilizes a multi-tier High-Performance Computing (HPC) distributed storage and data processing framework, placing computation where the data resides for better performance. By adopting this project, the scientific community will gain new opportunities to explore multimodal storage-driven possibilities, integrating multiple scientific data sources with advanced streaming frameworks. Additionally, our framework effectively utilizes computing resources and bridges the gaps identified by HPC experts. Our proposed approach tackles scalability and persistence challenges by leveraging native persistency, which has posed difficulties in traditional approaches. Furthermore, we seek to enhance fault-tolerance and load-balance of computations by leveraging real-time streaming in diverse scientific computing environments, thereby propelling advanced scientific computing research into the next generation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Waveform Simulation Framework: User Manual with Tutorials

This manuscript describes the Waveform Simulation Framework (WSF), a Python-based framework that provides a unified, programmable interface for generating synthetic seismograms for applications such as seismic array design, method development, and special event analysis. WSF standardizes how users define sources, receivers, and velocity models while abstracting simulator-specific configuration details, enabling workflows that are largely independent of the underlying numerical engine. The document provides installation guidance and tutorial-driven examples for three WSF simulator wrappers—WSF PyFK, WSF SW4, and WSF SPECFEM2D—illustrating end-to-end workflows from forward waveform simulation to common post-processing tasks (e.g., visualization and backprojection) using consistent data products (e.g., ObsPy Stream objects and SAC files).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

DUNE Database Development

The DUNE experiment will produce vast amounts of metadata, which describe the data coming from the read-out of the primary DUNE detectors. Various databases will make up the overall DB architecture for this metadata. ProtoDUNE at CERN is the largest existing prototype for DUNE and serves as a testing ground for - among other things - possible database solutions for DUNE. The subset of all metadata that is accessed during offline data reconstruction and analysis is referred to as ‘conditions data’ and it is stored in a dedicated database. As offline data reconstruction and analysis will be deployed on HTC and HPC resources, conditions data is expected to be accessed at very high rates. It is therefore crucial to store it in a granularity that matches the expected access patterns allowing for extensive caching. This requires a good understanding of the sources and use cases of conditions data. This contribution will briefly summarize the database architecture deployed at ProtoDUNE and explain the various sources of conditions data. We will present how the conditions data is retrieved and streamed from the databases and how it is handled to match expected access patterns.

43 PARTICLE ACCELERATORS↗