SEARCH · Engineering Papers

Results for “noisy data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16

All sources NASA NTRS DOE OSTI

DOE OSTI2021

RapidEELS: machine learning for denoising and classification in rapid acquisition electron energy loss spectroscopy

Recent advances in detectors for imaging and spectroscopy have afforded in situ, rapid acquisition of hyperspectral data. While electron energy loss spectroscopy (EELS) data acquisition speeds with electron counting are regularly reaching 400 frames per second with near-zero read noise, signal to noise ratio (SNR) remains a challenge owing to fundamental counting statistics. In order to advance understanding of transient materials phenomena during rapid acquisition EELS, trustworthy analysis of noisy spectra must be demonstrated. In this study, we applied machine learning techniques to denoise high frame rate spectra, benchmarking with slower frame rate “ground truths”. The results provide a foundation for reliable use of low SNR data acquired in rapid, in-situ spectroscopy experiments. Such a tool-set is a first step toward both automation in microscopy as well as use of these methods to interrogate otherwise poorly understood transformations.

36 MATERIALS SCIENCE↗

DOE OSTI2025

Mathematical Morphological Filtering with a Self-Adaptive Reconstruction Technique and Application to Local Seismic Data

Recorded seismic data are generally contaminated by noise from different sources, which masks the signals of interest. In the seismology community, frequency filtering (FF) is the standard method for noise suppression. However, when the signal of interest and noise share the same frequency band, the latter cannot be filtered out without infringing on the former. We implemented a noise suppression approach based on the mathematical morphology theorem. The method involves compound operations of dilation and erosion using structuring elements of varying lengths and decomposes an input noisy waveform into several time functions with differing characteristics. Further, the filtered waveform is constructed from the time functions using a self-adaptive reconstruction technique. Application to a data set of >4700 local waveforms suggests that the implemented mathematical morphological filtering (MMF) approach is efficient for data with low signal-to-noise ratio (SNR) and significantly outperforms FF in that SNR range. For most of the dataset, FF, machine learning (ML) denoising, and continuous wavelet transform (CWT) thresholding result in higher SNR values compared with the MMF method. However, for ~42% of the waveforms, MMF outperforms FF, and the SNR gain achieved with MMF is as large as ~23 dB. Compared to ML denoising and CWT thresholding, this proportion drops to only ~10%–14%. Our results suggests that in an operational setting, MMF cannot replace the other noise suppression methods; however, signal detection can be improved if MMF is used to supplement them in some scenarios. MMF could help detect signals in problematic low-SNR data, which are currently being missed particularly when using FF alone.

58 GEOSCIENCES↗

DOE OSTI2022

A dictionary learning algorithm for compression and reconstruction of streaming data in preset order

There has been an emerging interest in developing and applying dictionary learning (DL) to process massive datasets in the last decade. Many of these efforts, however, focus on employing DL to compress and extract a set of important features from data, while considering restoring the original data from this set a secondary goal. On the other hand, although several methods are able to process streaming data by updating the dictionary incrementally as new snapshots pass by, most of those algorithms are designed for the setting where the snapshots are randomly drawn from a probability distribution. In this paper, we present a new DL approach to compress and denoise massive dataset in real time, in which the data are streamed through in a preset order (instances are videos and temporal experimental data), so at any time, we can only observe a biased sample set of the whole data. Here, our approach incrementally builds up the dictionary in a relatively simple manner: if the new snapshot is adequately explained by the current dictionary, we perform a sparse coding to find its sparse representation; otherwise, we add the new snapshot to the dictionary, with a Gram-Schmidt process to maintain the orthogonality. To compress and denoise noisy datasets, we apply the denoising to the snapshot directly before sparse coding, which deviates from traditional dictionary learning approach that achieves denoising via sparse coding. Compared to full-batch matrix decomposition methods, where the whole data is kept in memory, and other mini-batch approaches, where unbiased sampling is often assumed, our approach has minimal requirement in data sampling and storage: i) each snapshot is only seen once then discarded, and ii) the snapshots are drawn in a preset order, so can be highly biased. Through experiments on climate simulations and scanning transmission electron microscopy (STEM) data, we demonstrate that the proposed approach performs competitively to those methods in data reconstruction and denoising.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Constrained or unconstrained? Neural-network-based equation discovery from data

Throughout many fields, practitioners often rely on differential equations to model systems. Yet, for many applications, the theoretical derivation of such equations and/or the accurate resolution of their solutions may be intractable. Instead, recently developed methods, including those based on parameter estimation, operator subset selection, and neural networks, allow for the data-driven discovery of both ordinary and partial differential equations (PDEs), on a spectrum of interpretability. The success of these strategies is often contingent upon the correct identification of representative equations from noisy observations of state variables and, as importantly and intertwined with that, the mathematical strategies utilized to enforce those equations. Specifically, the latter has been commonly addressed via unconstrained optimization strategies. Representing the PDE as a neural network, we propose to discover the PDE (or the associated operator) by solving a constrained optimization problem and using an intermediate state representation similar to a physics-informed neural network (PINN). The objective function of this constrained optimization problem promotes matching the data, while the constraints require that the discovered PDE is satisfied at a number of spatial collocation points. We present a penalty method and a widely used trust-region barrier method to solve this constrained optimization problem, and we compare these methods on numerical examples. Our results on several example problems demonstrate that the latter constrained method outperforms the penalty method, particularly for higher noise levels or fewer collocation points. This work motivates further exploration into using sophisticated constrained optimization methods in scientific machine learning, as opposed to their commonly used, penalty-method or unconstrained counterparts. For both of these methods, we solve these discovered neural network PDEs with classical methods, such as finite difference methods, as opposed to PINNs-type methods relying on automatic differentiation. Here, we briefly highlight how simultaneously fitting the data while discovering the PDE improves the robustness to noise and other small, yet crucial, implementation details.

Data-driven discovery↗

DOE OSTI2020

SQUARE: Strategic Quantum Ancilla Reuse for Modular Quantum Programs via Cost-Effective Uncomputation

Compiling high-level quantum programs to machines that are size constrained (i.e. limited number of quantum bits) and time constrained (i.e. limited number of quantum operations) is challenging. In this paper, we present SQUARE (Strategic QUantum Ancilla REuse), a compilation infrastructure that tackles allocation and reclamation of scratch qubits (called ancilla) in modular quantum programs. At its core, SQUARE strategically performs uncomputation to create opportunities for qubit reuse. Current Noisy Intermediate-Scale Quantum (NISQ) computers and forward-looking Fault-Tolerant (FT) quantum computers have fundamentally different constraints such as data locality, instruction parallelism, and communication overhead. Our heuristic-based ancilla-reuse algorithm balances these considerations and fits computations into resource-constrained NISQ or FT quantum machines, throttling parallelism when necessary. To precisely capture the workload of a program, we propose an improved metric, the "active quantum volume," and use this metric to evaluate the effectiveness of our algorithm. Furthermore, our results show that SQUARE improves the average success rate of NISQ applications by 1.47X. Surprisingly, the additional gates for uncomputation create ancilla with better locality, and result in substantially fewer swap gates and less gate noise overall. SQUARE also achieves an average reduction of 1.5X (and up to 9.6X) in active quantum volume for FT machines.

compiler optimization↗

DOE OSTI2023

A Model Calibration Method for Grid-Forming Inverters Using Iterative Bayesian Optimization

As inverter-based resources (IBRs) are rapidly deployed, especially at the distribution and microgrid levels, the need to include their accurate representations in power systems models increases. With well-calibrated models, IBR-interactions can be examined, preventing any stability and operational issues, especially in islanded systems. This paper proposes a method to calibrate generic inverter models using Bayesian optimization, but with a parameter-grouping approach to improve speed and accuracy. This approach is illustrated for a grid-forming inverter using synthetic data and field measurements from a simple IBRbased microgrid. Tests show that the calibration algorithm has modest computation cost and is robust to noisy measurements..

Biswas, Shuchismita↗

DOE OSTI2026

Ensemble Federated Machine Learning‐Based Cybersecurity Situational Awareness in Microgrid Network

Cyber-physical microgrids are vulnerable to stealthy cybersecurity threats that disguise their actions through the exploitation of system knowledge. Such actions can severely impacts microgrids deployed in defense bases, slowing the response time of military forces during national emergencies. Several machine-learning algorithms have been proposed to detect intrusions in the grid networks; however, these traditional machine-learning algorithms lack data privacy and are subject to several adversarial machine-learning threats. This paper proposes a novel federated machine learning (FML)-based three-model framework to detect and identify stealthy data-integrity attacks while ensuring data privacy in microgrid networks. The proposed architecture uses a variational mode decomposition technique to extract derived features from incoming measurement and control datasets. The extraction of these derived features allows FML models to learn minute variations in data patterns that allow them to perform significantly better than the models trained with generic datasets consisting of raw features. Our experimental results show the efficient performance of the proposed methodology against different types of data integrity attacks while considering primary and secondary controllers in microgrids. Further, the applied FML-integrated random forest ensemble algorithm outperforms the existing generic FML algorithms during noisy and noise-free datasets with prediction latencies of only 91–134 µs per sample within the 0.1 s sampling interval and requires communication bandwidth of around ∼8.25 KB/s at the control center and ∼2.7 KB/s per edge client for communication.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2023

Data-driven quantum approximate optimization algorithm for power systems

Abstract Quantum technology provides a ground-breaking methodology to tackle challenging computational issues in power systems. It is especially promising for Distributed Energy Resources (DERs) dominant systems that have been widely developed to promote energy sustainability. In those systems, knowing the maximum sections of power and data delivery is essential for monitoring, operation, and control. However, high computational effort is required. By leveraging quantum resources, Quantum Approximate Optimization Algorithm (QAOA) provides a means to search for these sections efficiently. However, QAOA performance relies heavily on critical parameters, especially for weighted graphs. Here we present a data-driven QAOA, which transfers quasi-optimal parameters between weighted graphs based on the normalized graph density. We verify the strategy with 39,774 expectation value calculations. Without parameter optimization, our data-driven QAOA is comparable with the Goemans-Williamson algorithm. This work advances QAOA and pilots its practical application to power systems in noisy intermediate-scale quantum devices.

Jing, Hang (ORCID:0000000256343489)↗

DOE OSTI2023

Towards Redefining the Reproducibility in Quantum Computing: A Data Analysis Approach on NISQ Devices

Although the building of quantum computers has kept making rapid progress in recent years, noise is still the main challenge for any application to leverage the power of quantum computing. Existing works addressing noise in quantum devices proposed noise reduction when deploying a quantum algorithm to a specified quantum computer. The reproducibility issue of quantum algorithms has been raised since the noise levels vary on different quantum computers. Importantly, existing works largely ignore the fact that the noise of quantum devices varies as time goes by. Therefore, reproducing the results on the same hardware will even become a problem. We analyze the reproducibility of quantum machine learning (QML) algorithms based on daily model training and execution data collection. Our analysis shows a correlation between our QML models’ test accuracy and quantum computer hardware’s calibration features. We also demonstrate that noisy simulators for quantum computers are not a reliable tool for quantum machine learning applications.

Senapati, Priyabrata↗

DOE OSTI2023

Detection of False Data Injection Attacks in Battery Stacks Using Input Noise-Aware Nonlinear State Estimation and Cumulative Sum Algorithms

Grid-scale battery energy storage systems (BESSs) are vulnerable to false data injection attacks (FDIAs), which could be used to disrupt state of charge (SoC) estimation. Inaccurate SoC estimation has negative impacts on system availability, reliability, safety, and the cost of operation. In this article a combination of a Cumulative Sum (CUSUM) algorithm and an improved input noise-aware extended Kalman filter (INAEKF) is proposed for the detection and identification of FDIAs in the voltage and current sensors of a battery stack. The series-connected stack is represented by equivalent circuit models, the SoC is modeled with a charge reservoir model and the states are estimated using the INAEKF. Further, the root mean squared error of the states’ estimation by the modified INAEKF was found to be superior to the traditional EKF. By employing the INAEKF, this article addresses the research gap that many state estimators make asymmetrical assumptions about the noise corrupting the system. Additionally, the INAEKF estimates the input allowing for the identification of FDIA, which many alternative methods are unable to achieve. The proposed algorithm was able to detect attacks in the voltage and current sensors in 99.16% of test cases, with no false positives. Utilizing the INAEKF compared to the standard EKF allowed for the identification of FDIA in the input of the system in 98.43% of test cases.

25 ENERGY STORAGE↗

DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

DOE OSTI2022

Equation‐Free Surrogate Modeling of Geophysical Flows at the Intersection of Machine Learning and Data Assimilation

Abstract There is a growing interest in developing data‐driven reduced‐order models for atmospheric and oceanic flows that are trained on data obtained either from high‐resolution simulations or satellite observations. The data‐driven models are non‐intrusive in nature and offer significant computational savings compared to large‐scale numerical models. These low‐dimensional models can be utilized to reduce the computational burden of generating forecasts and estimating model uncertainty without losing the key information needed for data assimilation (DA) to produce accurate state estimates. This paper aims at exploring an equation‐free surrogate modeling approach at the intersection of machine learning and DA in Earth system modeling. With this objective, we introduce an end‐to‐end non‐intrusive reduced‐order modeling (NIROM) framework equipped with contributions in modal decomposition, time series prediction, optimal sensor placement, and sequential DA. Specifically, we use proper orthogonal decomposition (POD) to identify the dominant structures of the flow, and a long short‐term memory network to model the dynamics of the POD modes. The NIROM is integrated within the deterministic ensemble Kalman filter (DEnKF) to incorporate sparse and noisy observations at optimal sensor locations obtained through QR pivoting. The feasibility and the benefit of the proposed framework are demonstrated for the NOAA Optimum Interpolation Sea Surface Temperature (SST) V2 data set. Our results indicate that the NIROM is stable for long‐term forecasting and can model dynamics of SST with a reasonable level of accuracy. Furthermore, the prediction accuracy of the NIROM gets improved by almost one order of magnitude by the DEnKF algorithm.

Pawar, Suraj↗

DOE OSTI2022

Model-Form Epistemic Uncertainty Quantification for Modeling with Differential Equations: Application to Epidemiology

Modeling real-world phenomena to any degree of accuracy is a challenge that the scientific research community has navigated since its foundation. Lack of information and limited computational and observational resources necessitate modeling assumptions which, when invalid, lead to model-form error (MFE). The work reported herein explored a novel method to represent model-form uncertainty (MFU) that combines Bayesian statistics with the emerging field of universal differential equations (UDEs). The fundamental principle behind UDEs is simple: use known equational forms that govern a dynamical system when you have them; then incorporate data-driven approaches – in this case neural networks (NNs) – embedded within the governing equations to learn the interacting terms that were underrepresented. Utilizing epidemiology as our motivating exemplar, this report will highlight the challenges of modeling novel infectious diseases while introducing ways to incorporate NN approximations to MFE. Prior to embarking on a Bayesian calibration, we first explored methods to augment the standard (non-Bayesian) UDE training procedure to account for uncertainty and increase robustness of training. In addition, it is often the case that uncertainty in observations is significant; this may be due to randomness or lack of precision in the measurement process. This uncertainty typically manifests as “noisy” observations which deviate from a true underlying signal. To account for such variability, the NN approximation to MFE is endowed with a probabilistic representation and is updated using available observational data in a Bayesian framework. By representing the MFU explicitly and deploying an embedded, data-driven model, this approach enables an agile, expressive, and interpretable method for representing MFU. In this report we will provide evidence that Bayesian UDEs show promise as a novel framework for any science-based, data-driven MFU representation; while emphasizing that significant advances must be made in the calibration of Bayesian NNs to ensure a robust calibration procedure.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Spatiotemporal forecasting of the edge localized modes in tokamak plasmas using neural networks

Artificial intelligence techniques have been increasingly adopted by the plasma and fusion science to address problems like plasma reconstruction, surrogate modeling, and tokamak/stellarator optimization. A key focus in sustained fusion research is the prediction and mitigation of edge-localized-modes (ELMs), instabilities that occur in short, periodic bursts and can cause erosion to the tokamak vessel wall. Recent research has demonstrated the power of neural networks in approximating continuous functions. In this work, we build spatiotemporal forecasting models that can predict the onset of ELMs and their evolution at early stages. We leverage recent advances in generative modeling, sequence-to-sequence modeling, and Fourier neural operators to propose architectures and training strategies that can learn to forecast short to long term dynamics of the noisy signals due to ELMs. We benchmark the developed model against a state-of-the-art foundation model using the beam emission spectroscopy (BES) data that captures the plasma fluctuations due to ELMs over a 8 x 8 spatial grid. Our models demonstrate high accuracy, outperforming the baselines, in predicting the evolution of BES signals during ELM events. Furthermore, the developed models exhibit high accuracy in predicting the rapid rise and relaxation of the signals due to ELMs within 30–80 µs.

edge localized modes↗

DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific ap- plications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Net- works (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate band- widths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer perfor- mance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application- level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher pre- diction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

DOE OSTI2025

Estimating soybean yields from high-temporal-resolution multi-source data using deep learning

Accurate and timely crop yield prediction is crucial for ensuring food security and maintaining stable agricultural markets. In recent years, there has been a surge in interest in leveraging high-temporal-resolution, multi-source data for effective crop growth monitoring and yield estimation. A notable challenge arises from the difficulty in capturing the intricate interactions between variables across different time steps within these high-temporal-resolution time series datasets. This complexity hinders the reliable extraction of yield information from voluminous and often noisy datasets, especially during periods of extreme weather events. Here, in this study, we propose an Attention and Graph Isomorphism Network-enhanced Bi-directional Long Short-Term Memory network (AGB-LSTM) for estimating county-level soybean yield in the United States. This model integrates a diverse set of remote sensing data, including Near-Infrared Reflectance of Vegetation (NIRv), Sun-Induced chlorophyll Fluorescence (SIF), and Gross Primary Productivity (GPP), along with environmental covariates. The AGB-LSTM effectively leverages information related to crop yield from high-temporal-resolution time series data (5-days), achieving an accuracy of R²= 0.67 and rRMSE = 14.46%. This approach significantly outperforms traditional machine learning methods such as Random Forest (RF) (R²= 0.52, rRMSE = 17.36%) and Bi-LSTM (R²= 0.58, rRMSE = 16.17%). Sensitivity experiments with different time steps and ranges demonstrated that our model could accurately and stably predict yields 1 to 2 months before harvest. Moreover, data with a finer temporal resolution consistently improved prediction performance, resulting in an approximately 20% increase in and an approximately 20% decrease in rRMSE compared to using monthly composites. We also evaluated the robustness of the model under extreme climate events and observed strong performance (R²= 0.50, rRMSE = 21.32%). Finally, yield mapping for major soybean-producing regions in North America in 2023 revealed spatial patterns that closely matched USDA yield reports. Our findings suggest that the AGB-LSTM model is a promising and effective method for estimating yield and has notable potential for global crop yield forecasting.

Deep learning↗

DOE OSTI2022

Adaptive stabilization of quantum circuits executed on unstable devices

Conventional computers have evolved to device components that demonstrate failure rates of 10 −17 or less, while current quantum computing devices typically exhibit error rates of 10 −2 or greater. This raises concerns about the reliability and reproducibility of the results obtained from quantum computers. The problem is highlighted by experimental observation that today’s NISQ devices are inherently unstable. Remote quantum cloud servers typically do not provide users with an ability to calibrate the device themselves. Using inaccurate characterization data for error mitigation can have devastating impact on reproducibility. In this study, we investigate if one can infer the critical channel parameters dynamically from the noisy binary output of the executed quantum circuit and use it to improve program stability. An open question however is how well does this methodology scale. We discuss the efficacy and efficiency of our adaptive algorithm using canonical quantum circuits such as the uniform superposition circuit. Our metric of performance is the Hellinger distance between the post-stabilization observations and the reference (ideal) distribution.

Dasgupta, Samudra↗

DOE OSTI2025

SANE: strategic autonomous non-smooth exploration for multiple optima discovery in multi-modal and non-differentiable black-box functions

Both computational and experimental material discovery bring forth the challenge of exploring multidimensional and multimodal parameter spaces, such as phase diagrams of Hamiltonians with multiple interactions, composition spaces of combinatorial libraries, material structure image spaces, and molecular embedding spaces. Often these systems are black-boxes and time-consuming to evaluate, which resulted in strong interest towards active learning methods such as Bayesian optimization (BO). However, these systems are often noisy which make the black box function severely multi-modal and non-differentiable, where a vanilla BO can get overly focused near a single or faux optimum, deviating from the broader goal of scientific discovery. To address these limitations, here we developed Strategic Autonomous Non-Smooth Exploration (SANE) to facilitate an intelligent Bayesian optimized navigation with a proposed cost-driven probabilistic acquisition function to find multiple global and local optimal regions, avoiding the tendency to becoming trapped in a single optimum. To distinguish between a true and false optimal region due to noisy experimental measurements, a human (domain) knowledge driven dynamic surrogate gate is integrated with SANE. We implemented the gate-SANE into pre-acquired piezoresponse spectroscopy data of a ferroelectric combinatorial library with high noise levels in specific regions, and piezoresponse force microscopy (PFM) hyperspectral data. SANE demonstrated better performance than classical BO to facilitate the exploration of multiple optimal regions and thereby prioritized learning with higher coverage of scientific values in autonomous experiments. Our work showcases the potential application of this method to real-world experiments, where such combined strategic and human intervening approaches can be critical to unlocking new discoveries in autonomous research.

Biswas, Arpan [University of Tennessee, Knoxville,↗