SEARCH · Engineering Papers

Results for “Data Distribution”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Standardized Data to Improve Understanding and Modeling of Soil Nitrogen at Continental Scale

Abstract Nitrogen (N) is a key limiting nutrient in terrestrial ecosystems, but there remain critical gaps in our ability to predict and model controls on soil N cycling. This may be in part due to lack of standardized sampling across broad spatial–temporal scales. Here, we introduce a continentally distributed, publicly available data set collected by the National Ecological Observatory Network (NEON) that can help fill these gaps. First, we detail the sampling design and methods used to collect and analyze soil inorganic N pool and net flux rate data from 47 terrestrial sites. We address methodological challenges in generating a standardized data set, even for a network using uniform protocols. Then, we evaluate sources of variation within the sampling design and compare measured net N mineralization to simulated fluxes from the Community Earth System Model 2 (CESM2). We observed wide spatiotemporal variation in inorganic N pool sizes and net transformation rates. Site explained the most variation in NEON’s stratified sampling design, followed by plots within sites. Organic horizons had larger pools and net N transformation rates than mineral horizons on a sample weight basis. The majority of sites showed some degree of seasonality in N dynamics, but overall these temporal patterns were not matched by CESM2, leading to poor correspondence between observed and modeled data. Looking forward, these data can reveal new insights into controls on soil N cycling, especially in the context of other environmental data sets provided by NEON, and should be leveraged to improve predictive modeling of the soil N cycle.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

DISTRIBUTION TRANSFORMER ASSET MONITORING ON THE GRID EDGE USING SMART SENSOR DATA

As new loads such as rooftop photovoltaics, electric vehicles and other distributed energy resources become commonplace on the distribution grid, the stress on already aging assets begins to escalate. This increased loading and changing dynamics can exacerbate failure rates. While traditional monitoring efforts focus on transmission and generation assets, utilities are now beginning to pay close attention to distribution assets in order to increase reliability indices and reduce cost from unexpected outages. This research develops low-cost and scalable methods to monitoring the health of a critical distribution grid asset: the service transformer. Existing methods in literature are either invasive and thus difficult to implement or require the device to be tested offline in an expensive lab setting. Data from the ubiquitous smart meter as well as a novel Bluetooth based transformer monitor are leveraged to automatically notify the utility of deteriorating or damaged transformers. Voltage, temperature, and vibration are some of the signals measured and analyzed by the proposed algorithms to predict transformer failures. Furthermore, these algorithms are designed to keep the implementation and processing costs low by taking advantage of edge computing where possible.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2024

Counter Data Paucity through Adversarial Invariance Encoding: A Case Study on Modeling Battery Thermal Runaway

Lithium-ion batteries, widely used for their durability and high energy storage, face the risk of internal short circuits leading to catastrophic thermal runaway events. These events, triggered by external stimuli like mechanical loads, pose safety concerns in applications such as electric vehicles. Detecting and understanding thermal runaway events is crucial, but physics-driven models struggle to explain the non-linear evolution of battery temperature during these events, considering factors like material composition and state-of-charge. Due to the rarity of these events and the cost of data collection, we propose a deep learning (DL) model to predict battery temperature responses during thermal runaway. The challenge lies in the scarcity of data, making traditional DL models prone to overfitting and learning low-quality representations of the complex process.Our approach introduces a novel few-shot architecture that incorporates an adversarially governed invariant encoding process. This architecture aims to distill "invariant" relationships by addressing distributional shifts in data across various battery properties, facilitating the detection of thermal runaway events. Specifically, our results demonstrate that deep learning models conditioned on these "invariant" representations outperform state-of-the-art baselines, achieving a remarkable 96.8% performance improvement in terms of the popular metric MAPE. This framework presents a promising direction for enhancing battery safety modeling, particularly in the context of rare and complex events like thermal runaway. Our code and code and dataset used for the paper are public1.

Tabassum, Anika [ORNL] (ORCID:0000000254600955)↗

DOE OSTI2024

Geometric Measures of Trustworthiness for Machine Learning Predictions

his report details the findings from the research and investigation of Geometric Measures of Trustworthiness for Machine Learning Predictions. We explored the trustworthiness of machine learning (ML) models’ predictions using geometric measures to quantify the similarity of a query point with the training data. Predictive uncertainty in ML can originate from at least three sources: (1) Model uncertainty, which represents the uncertainty in model form (e.g. decision tree, vs neural network) and estimating the model parameters from the training data, (2) Data uncertainty, which represents the natural complexities of the data such as class overlap and inherent noise, and (3) Distributional uncertainty, which represents the mismatch between the training and operational distributions. The proposed measures focus on measuring and explaining the data and distributional uncertainties by measuring the relationships of operational data with the training data.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Detecting Unclassified Electromagnetic Signals for Secure Wireless Communication Using Open Set Recognition

We developed multiple machine learning methods for the detection and classification of new wireless communication waveforms, which is critical for targeted attacks in wireless networks and electronic warfare. Our machine learning models are capable of dynamically detecting security threats in near real time through our advanced open set recognition (OSR) approach. This model has demonstrated significant improvements in the detection of unknown waveforms, thereby enhancing the security and reliability of mission critical communications. Our approach to detecting uncertain security threats is novel; we advanced OSR techniques by incorporating domain knowledge of wireless signals. Specifically, we combined time and frequency domain model features to enhance the model’s performance. Utilizing an OSR approach eliminates the need for training data to be distributed similarly to the deployment environment and removes the requirement for the training set to contains all possible threat classes. This is crucial because it is often infeasible to determine and characterize all potential security threats in advance. Our model were trained on simulated data, generated in partnership with the University at Albany, State of New York. The data set contained a diverse array of wireless signals, including those with additive white Gaussian noise and multipath signals, with and without line of sight. This comprehensive training set allowed us to optimize our models to detect unknown waveforms under various challenging scenarios, such as low signal-to-noise ratios. By training on various waveforms, varying signal-to-noise ratio, and different sample sizes under normal conditions, our models were fine tuned to perform effectively in challenging environments.

99 - GENERAL AND MISCELLANEOUS↗

DOE OSTI2022

Learning the temporal evolution of multivariate densities via normalizing flows

In this work, we propose a method to learn multivariate probability distributions using sample path data from stochastic differential equations. Specifically, we consider temporally evolving probability distributions (e.g., those produced by integrating local or nonlocal Fokker–Planck equations). Here, we analyze this evolution through machine learning assisted construction of a time-dependent mapping that takes a reference distribution (say, a Gaussian) to each and every instance of our evolving distribution. If the reference distribution is the initial condition of a Fokker–Planck equation, what we learn is the time-T map of the corresponding solution. Specifically, the learned map is a multivariate normalizing flow that deforms the support of the reference density to the support of each and every density snapshot in time. We demonstrate that this approach can approximate probability density function evolutions in time from observed sampled data for systems driven by both Brownian and Lévy noise. We present examples with two- and three-dimensional, uni- and multimodal distributions to validate the method.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Propagating Uncertainties in the SALT3 Model-training Process to Cosmological Constraints

Type Ia supernovae (SNe Ia) are standardizable candles that must be modeled empirically to yield cosmological constraints. To understand the robustness of this modeling to variations in the model-training procedure, we build an end-to-end pipeline to test the recently developed SALT3 model. We explore the consequences of removing pre-2000s low-z or poorly calibrated U-band data, adjusting the amount and fidelity of SN Ia spectra, and using a model-independent framework to simulate the training data. We find that the SALT3 model surfaces are improved by having additional spectra and U-band data, and can be shifted by ~5% if host-galaxy contamination is not sufficiently removed from SN spectra. We find that resulting measurements of w are consistent to within 2.5% for all of the training variants explored in this work, with the largest shifts coming from variants that add color-dependent calibration offsets or host-galaxy contamination to the training spectra and those that remove pre-2000s low-z data. These results demonstrate that the SALT3 model-training procedure is largely robust to reasonable variations in the training data, but that additional attention must be paid to the treatment of spectroscopic data in the training process. We also find that the training procedure is sensitive to the color distributions of the input data—the resulting w measurement can be biased by ~2% if the color distribution is not sufficiently wide. Future low-z data, particularly u-band observations and high signal-to-noise ratio SN Ia spectra, will help to significantly improve SN Ia modeling in the coming years.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Propagation of non-Gaussian voltage angle fluctuations in high-voltage power grids

Recent measurements have reported non-Gaussian tails in the distribution of frequency data in electric power grids. Large frequency deviations may induce grid instabilities and it is therefore crucial to understand how noise disturbances with long, non-Gaussian tails propagate. Here, we investigate how fluctuations in power feed-in, characterized by non-zero cumulants of their distribution, propagate through high-voltage power grids. Unlike previous investigations which focused on the white-noise limit, we consider the limit of long noise correlation time, where power feed-in fluctuates over times longer than the inherent dynamical time scales of the grid - the relevant regime for large-scale, high-voltage distribution grids. In this work, we show that in this limit, the skewness and kurtosis of the power feed-in distribution propagate similarly as its variance, independently of the distribution of inertia. Non-Gaussianities from individual sources of noise therefore persist throughout the entire network. This finding is corroborated by numerical results on a realistic model of the synchronous grid of continental Europe.

42 ENGINEERING↗

DOE OSTI2021

Deep Cellular Recurrent Network for Efficient Analysis of Time-Series Data With Spatial Information

Efficient processing of large-scale time series data is an intricate problem in machine learning. Conventional sensor signal processing pipelines with hand engineered feature extraction often involve huge computational cost with high dimensional data. Deep recurrent neural networks have shown promise in automated feature learning for improved time-series processing. However, generic deep recurrent models grow in scale and depth with increased complexity of the data. This is particularly challenging in presence of high dimensional data with temporal and spatial characteristics. Consequently, this work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to efficiently process complex multi-dimensional time series data with spatial information. Here, the cellular recurrent architecture in the proposed model allows for location-aware synchronous processing of time series data from spatially distributed sensor signal sources. Extensive trainable parameter sharing due to cellularity in the proposed architecture ensures efficiency in the use of recurrent processing units with high-dimensional inputs. This study also investigates the versatility of the proposed DCRNN model for classification of multi-class time series data from different application domains. Consequently, the proposed DCRNN architecture is evaluated using two time-series datasets: a multichannel scalp EEG dataset for seizure detection, and a machine fault detection dataset obtained in-house. The results suggest that the proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2021

Anomaly detection in Hyper Suprime-Cam galaxy images with generative adversarial networks

ABSTRACT The problem of anomaly detection in astronomical surveys is becoming increasingly important as data sets grow in size. We present the results of an unsupervised anomaly detection method using a Wasserstein generative adversarial network (WGAN) on nearly one million optical galaxy images in the Hyper Suprime-Cam (HSC) survey. The WGAN learns to generate realistic HSC-like galaxies that follow the distribution of the data set; anomalous images are defined based on a poor reconstruction by the generator and outlying features learned by the discriminator. We find that the discriminator is more attuned to potentially interesting anomalies compared to the generator, and compared to a simpler autoencoder-based anomaly detection approach, so we use the discriminator-selected images to construct a high-anomaly sample of ∼13 000 objects. We propose a new approach to further characterize these anomalous images: we use a convolutional autoencoder to reduce the dimensionality of the residual differences between the real and WGAN-reconstructed images and perform UMAP clustering on these. We report detected anomalies of interest including galaxy mergers, tidal features, and extreme star-forming galaxies. A follow-up spectroscopic analysis of one of these anomalies is detailed in the Appendix; we find that it is an unusual system most likely to be a metal-poor dwarf galaxy with an extremely blue, higher-metallicity H ii region. We have released a catalogue with the WGAN anomaly scores; the code and catalogue are available at https://github.com/kstoreyf/anomalies-GAN-HSC; and our interactive visualization tool for exploring the clustered data is at https://weirdgalaxi.es.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2026

An Autonomous MCP Bridge to Rucio: Enhancing Data Management Accessibility for High Energy Physics

The Rucio Data Management System [1] is an important tool used by High Energy Physics experiments, including those at Fermi National Accelerator Laboratory, to store and manage exabyte-scale scientific datasets. Despite its central role in coordinating data across globally distributed storage sites, Rucio's command line interface (CLI) presents a steep learning curve, and makes it difficult for scientists to navigate through. To solve this issue, a containerized Model Context Protocol (MCP) [2] server was built that connects Large Language Models directly to Rucio, allowing AI agents to handle data tasks by using simple, natural language rather than memorized terminal commands. The core engineering focus of this project was moving the server away from slow terminal commands that require text parsing and replacing them with a native Python Client API toolset and a planned REST API framework. Moving to the Python API handles data operations directly in memory, which helps clear up formatting errors, provides the AI with clean, structured JSON data and speeds up tool execution. To prove that the system actually works, a benchmarking pipeline was also built with various questions to test the AI across four different model configurations. The questions included finding data scopes, tracking down specific datasets, and checking replication rules. Through benchmarking, early runs showed that with raw terminal text, the model would get confused and stuck, whereas switching to the Python API to feed the AI clean, structured data yielded massive improvement. By creating an intelligent and autonomous bridge to a storage network, this project shows how AI can be implemented in scientific data management, which ultimately helps scientists at Fermilab spend less time sorting through data and more time focusing on their experiments and analysis.

Akella, Kashyap [William Rainey Harper Coll.]↗

DOE OSTI2021

Quantitative assessment of Distributed Acoustic Sensing at the Source Physics Experiment, Phase II

In this report, we assess the data recorded by a Distributed Acoustic Sensing (DAS) cable deployed during the Source Physics Experiment, Phase II (DAG) in comparison with the data recorded by nearby 4.5-Hz geophones. DAS is a novel recording method with unprecedented spatial resolution, but there are significant concerns around the data fidelity as the technology is ramped up to more common usage. Here we run a series of tests to quantify the similarity between DAS data and more conventional data and investigate cases where the higher spatial resolution of the DAS can provide new insights into the wavefield. These tests include 1D modeling with seismic refraction and bootstrap uncertainties, assessing the amplitude spectra with distance from the source, measuring the frequency dependent inter-station coherency, estimating time-dependent phase velocity with beamforming and semblance, and measuring the cross-correlation between the geophone and the particle velocity inferred from the DAS. In most cases, we find high similarity between the two datasets, but the higher spatial resolution of the DAS provides increased details and methods of estimating uncertainty.

47 OTHER INSTRUMENTATION↗

DOE OSTI2022

Quantitative assessment of Distributed Acoustic Sensing at the Source Physics Experiment (Phase II)

58 GEOSCIENCES↗

DOE OSTI2021

Geostatistical interpolation of streambed hydrologic attributes with addition of left censored data and anisotropy

Spatial geostatistical interpolation of point measurements of streambed attributes in the hyporheic zone may be constrained by the streambed anisotropy, and data density and spatial distribution may significantly impact the results. Spatial clustering and low spatial data density can be caused by bedrock outcropping at the streambed limiting installation of in-stream piezometers. This study examines parameter error variability of the geostatistical interpolation using anisotropic interpolation methods and increasing the data density by adding left censored values (i.e., data below measurement limit) to locations where measurements were limited by exposed bedrock lining the streambed. The reduction in relative standard error of the interpolation was determined for the spatial distributions of streambed attributes including hydraulic conductivity, seepage flux, and mercury solute flux measured in two different years along a study reach in East Fork Poplar Creek, Tennessee, USA. Here, two methods to impute the left censored values were compared including the conventional half the detection limit substitution method, and the Stochastic Approximation of Expectation-Maximization (SAEM) algorithm, which both had comparable results. Imputing left censored data increased the data density to recommended ranges, reduced data clustering, increased the spatial dependence for some attributes, and reduced the standard error for each of the three attributes. For the reach considered herein, addition of the left censored values resulted in a larger error reduction than the consideration of anisotropy within the interpolation, which confirms the benefit of data addition to increase data density within data-limited river corridors.

58 GEOSCIENCES↗

DOE OSTI2024

PHEV Distribution Grid Integration and Smart Systems Testing: Cooperative Research and Development Final Report

To continue NREL's mission, which includes preparing technologies and markets with "speed and scale," NREL researchers will collaborate with Toyota on plug-in electric vehicle technology. Toyota will provide a collection of plug-in hybrid electric vehicles ("PHEVs" [and ultimately including PEVs], collectively, the "Vehicles") to conduct infrastructure interaction testing at NREL facilities. NREL will use available resources in the Vehicle Testing and Integration Facility (VTIF), its staff parking garage with numerous commercial Electric Vehicle Supply Equipment ("EVSEs"), and its Energy Systems Integration Facility (ESIF) to safely execute research experiments that highlight potential distribution system power quality challenges related to plug-in electric vehicles. The expected outcome of this project is a better understanding of the correlation of individual vehicle power quality attributes to a system of vehicles on a distribution network. The data collected and the simulations conducted will be used to guide future experiments and project development that aid in the deployment of plug-in electric vehicles ("PEV").

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2023

Deep learning uncertainty quantification for clinical text classification

Machine learning algorithms are expected to work side-by-side with humans in decision-making pipelines. Thus, the ability of classifiers to make reliable decisions is of paramount importance. Deep neural networks (DNNs) represent the state-of-the-art models to address real-world classification. Although the strength of activation in DNNs is often correlated with the network’s confidence, in-depth analyses are needed to establish whether they are well calibrated. In this paper, we demonstrate the use of DNN-based classification tools to benefit cancer registries by automating information extraction of disease at diagnosis and at surgery from electronic text pathology reports from the US National Cancer Institute (NCI) Surveillance, Epidemiology, and End Results (SEER) population-based cancer registries. In particular, we introduce multiple methods for selective classification to achieve a target level of accuracy on multiple classification tasks while minimizing the rejection amount—that is, the number of electronic pathology reports for which the model’s predictions are unreliable. We evaluate the proposed methods by comparing our approach with the current in-house deep learning-based abstaining classifier. Overall, all the proposed selective classification methods effectively allow for achieving the targeted level of accuracy or higher in a trade-off analysis aimed to minimize the rejection rate. On in-distribution validation and holdout test data, with all the proposed methods, we achieve on all tasks the required target level of accuracy with a lower rejection rate than the deep abstaining classifier (DAC). Interpreting the results for the out-of-distribution test data is more complex; nevertheless, in this case as well, the rejection rate from the best among the proposed methods achieving 97% accuracy or higher is lower than the rejection rate based on the DAC. We show that although both approaches can flag those samples that should be manually reviewed and labeled by human annotators, the newly proposed methods retain a larger fraction and do so without retraining—thus offering a reduced computational cost compared with the in-house deep learning-based abstaining classifier.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

Distributed Grid-Sensing Service Layer: A System Architecture Blueprint

In this report, a scalable, decentralized, sensor-oriented communication blueprint is introduced. The proposed architecture has been specifically designed to support the mass deployment of sensors and also enable data consumers to access the underlying data streams using a distributed systems approach. With the proposed approach, sensors and data clients can be decoupled from the physical communication infrastructure and migrated into a modern, software-defined infrastructure ecosystem that can be configured to suit the end application’s requirements. In its current iteration, this blueprint defines the interactions, processes, and expected outcomes that each individual component within the architecture must support the overall ecosystem’s needs. Within this work, such an assembly of services is called the data bus; it represents the heart of the distributed grid sensing services architecture, a multiyear project that aims to design, implement, and test a reference sensor-data dissemination network that can support the needs of the extended grid state. We expect this blueprint to guide our future implementation efforts

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2023

New approaches to an old problem: addressing spatial gaps in the World Stress Map

Abstract A well-recognized characteristic of the World Stress Map (WSM) database is the continued presence of large spatial gaps in the distribution of the data records despite the more than 40-year development history of the database. The current release has more than 30 000 high-quality (A–C) data records (often referred to as ‘stress indicators’), but while some continental areas (such as Australia) have seen a significant increase in spatial converge with the latest release, other continental regions (Africa, central Asia, most of South America) remain markedly sparse. In this contribution we (1) review the current state of the spatial distribution of stress indicators in the continental regions (above sea-level); (2) quantify the clustering of the stress indicators in the latest WSM release using the Hopkins statistic as a way to explore the current spatial distribution of the indicators and assess future WSM releases; and (3) present three approaches (joint inversion, seismic anisotropy and InSAR) that provide a way to fill in the gaps (both in the S Hmax orientation and principal stress magnitudes) in regions that lack active seismicity and where borehole drilling is cost prohibitive. These three approaches have the potential to guide procedures for improving a priori estimates of the ambient stress field in the Earth's crust and reduce the uncertainty in predicting both the magnitude and orientation of the principal tectonic stresses.

58 GEOSCIENCES↗