Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Error Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Increasing the Measured Effective Quantum Volume with Zero Noise Extrapolation

Quantum volume is a full-stack benchmark for near-term quantum computers. It quantifies the largest size of a square circuit which can be executed on the target device with reasonable fidelity. Error mitigation is a set of techniques intended to remove the effects of noise present in the computation of noisy quantum computers when computing an expectation value of interest. Effective quantum volume is a proposed metric that applies error mitigation to the quantum volume protocol to evaluate the effectiveness not only of the target device but also of the error mitigation algorithm. Digital zero-noise extrapolation is an error mitigation technique that estimates the noiseless expectation value using circuit folding to amplify errors by known scale factors and then extrapolating computed expectation values to the zero-noise limit. Here we demonstrate that zero-noise extrapolation, with global and local unitary folding with fractional scale factors, in conjunction with dynamical decoupling, can increase the effective quantum volume over the vendor-measured quantum volume. Specifically, we measure the effective quantum volume of four IBM Quantum superconducting processor units, obtaining values that are larger than the vendor-measured quantum volume on each device. This is the first such increase reported.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Modeling land use and land cover change: using a hindcast to estimate economic parameters in gcamland v2.0

Abstract. Future changes in land use and cover have important implications for agriculture, energy, water use, and climate. Estimates of future land use and land cover differ significantly across economic models as a result of differences in drivers, model structure, and model parameters; however, these models often rely on heuristics to determine model parameters. In this study, we demonstrate a more systematic and empirically based approach to estimating a few key parameters for an economic model of land use and land cover change, gcamland. Specifically, we generate a large set of model parameter perturbations for the selected parameters and run gcamland simulations with these parameter sets over the historical period in the United States to quantify land use and land cover, determine how well the model reproduces observations, and identify parameter combinations that best replicate observations, assuming other model parameters are fixed. We also test alternate methods for forming expectations about uncertain crop yields and prices, including adaptive, perfect, linear, and hybrid approaches. In particular, we estimate parameters for six parameters used in the formation of expectations and three of seven logit exponents for the USA only. We find that an adaptive expectation approach minimizes the error between simulated outputs and observations, with parameters that suggest that for most crops, landowners put a significant weight on previous information. Interestingly, for corn, where ethanol policies have led to a rapid growth in demand, the resulting parameters show that a larger weight is placed on more recent information. We examine the change in model parameters as the metric of model error changes, finding that the measure of model fitness affects the choice of parameter sets. Finally, we discuss how the methodology and results used in this study could be used for other regions or economic models to improve projections of future land use and land cover change.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Review of Onsite Temperature and Solar Forecasting Models to Enable Better Building Design and Operations

Advanced building controls and energy optimization for new constructions and retrofits rely on accurate weather data. Traditionally, most studies utilize airport weather information as the decision inputs. However, most buildings are in environments that are quite different than those at the airport miles away. Tree cover, adjacent buildings, and micro-climate effects caused by the larger surrounding area can all yield deviations in air temperature, humidity, solar irradiance, and wind that are large enough to influence design and operation decisions. In order to overcome this challenge, there are many prior studies on developing weather forecasting algorithms from micro-to meso-scales. Additionally, this paper reviews and complies knowledge on common weather data resources, data processing methodologies and forecasting techniques of weather information. Commonly used statistical, machine learning and physical-based models are discussed and presented as two major categories: deterministic forecasting and probabilistic forecasting. Finally, evaluation metrics for forecasting errors are listed and discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Tuning the Interpolation Basis in a Multigrid Decomposition for Local Error Control

In the compression of scientific data, error-controlled compressors enable to considerably decrease the size of the dataset while maintaining adequate levels of accuracy. In this paper, we note that multi-level refactoring scheme such as MGARD i) rely on an approximation of the data based on the interpolation of coefficients, ii) estimate the resulting error with global metrics on the dataset. To improve on these two aspects, we propose a method that aims to divide the original dataset into blocks based on their smoothness and refactors each block separately with the most relevant interpolation order. We show the relevance of such a method on tailored datasets and the benefits and challenges when applying it to large scientific data.

Vidal, Nicolas [ORNL]↗
DOE OSTI2021

The Effect of Spectral Resolution of Light Sources on Photopic and a-opic Quantities

There is a growing interest in including light source spectrum in advanced lighting software tools and simulations. Given that traditional lighting software tools have been used primarily for calculating photometric quantities, their simplifying assumptions may not be suitable for calculating other a-opic quantities. Commonly used simulation tools use three values to represent the three primary colors (red, green, and blue), but others have expanded the number of bands (i.e., the spectral resolution) to 9, 27, or 81 in an attempt to more accurately represent variations in light source spectrum and downstream spectrally-derived quantities. It remains unclear, however, to what extent spectral resolution affects calculated quantities. To address this gap, a numerical analysis was completed using a large spectral power distribution database (n = 1,302 light sources). Calculated illuminance, a-opic irradiance, luminous efficacy of radiation (LER), melanopic efficacy of radiation (MER), and melanopic to photopic (M/P) ratio were compared for spectral resolutions of 3 and 9 bands compared to a baseline of 81 bands. Across all examined lighting quantities, considerable errors—a mean absolute percent error of 18.4%—were found when using 3 band calculations. These were reduced to 4% for 9 band calculations. The errors varied by metric and light source type. The results suggest that the use of 9 bands can more accurately characterize performance of different light source types across a range of metrics compared to using 3 bands.

Abboushi, Belal K.↗
DOE OSTI2025

Optimising the processing and storage of visibilities using lossy compression

The next-generation radio astronomy instruments are providing a massive increase in sensitivity and coverage, largely through increasing the number of stations in the array and the frequency span sampled. The two primary problems encountered when processing the resultant avalanche of data are the need for abundant storage and the constraints imposed by I/O, as I/O bandwidths drop significantly on cold storage. An example of this is the data deluge expected from the SKA Telescopes of more than 60 PB per day, all to be stored on the buffer filesystem. While compressing the data is an obvious solution, the impacts on the final data products are hard to predict. In this paper, we chose an error-controlled compressor – MGARD – and applied it to simulated SKA-Mid and real pathfinder visibility data, in noise-free and noise-dominated regimes. As the data have an implicit error level in the system temperature, using an error bound in compression provides a natural metric for compression. MGARD ensures the compression incurred errors adhere to the user-prescribed tolerance. To measure the degradation of images reconstructed using the lossy compressed data, we proposed a list of diagnostic measures, exploring the trade-off between these error bounds and the corresponding compression ratios, as well as the impact on science quality derived from the lossy compressed data products through a series of experiments. We studied the global and local impacts on the output images for continuum and spectral line examples. We found relative error bounds of as much as 10%, which provide compression ratios of about 20, have a limited impact on the continuum imaging as the increased noise is less than the image RMS, whereas a 1% error bound (compression ratio of 8) introduces an increase in noise of about an order of magnitude less than the image RMS. For extremely sensitive observations and for very precious data, we would recommend a 0.1% error bound with compression ratios of about 4. These have noise impacts two orders of magnitude less than the image RMS levels. At these levels, the limits are due to instabilities in the deconvolution methods. We compared the results to the alternative compression tool DYSCO, in both the impacts on the images and in the relative flexibility. MGARD provides better compression for similar error bounds and has a host of potentially powerful additional features.

Techniques: interferometric↗
DOE OSTI2020

Cross-Market Price Difference Forecast Using Deep Learning for Electricity Markets

Price forecasting is in the center of decision making in electricity markets. Many researches have been done in forecasting energy prices while little research has been reported on forecasting price difference between day-ahead and realtime markets due to its high volatility, which however plays a critical role in virtual trading. To this end, this paper takes the first attempt to employ novel deep learning architecture with Bidirectional Long-Short Term Memory (LSTM) units to forecast the price difference between day-ahead and real-time markets for the same node. The raw data is collected from PJM market, processed and fed into the proposed network. The Root Mean Squared Error (RMSE) and customized performance metric are used to evaluate the performance of the proposed method. Case studies show that it outperforms the traditional statistical models like ARIMA, and machine learning models like XGBoost and SVR methods in both RMSE and the capability of forecasting the sign of price difference. Additionally, to cross-market price difference forecast, the proposed approach has the potential to be applied to solve other forecasting problems such as price spread forecast in DA market for Financial Transmission Right (FTR) trading purpose.

DA/RT price difference↗
DOE OSTI2022

Energetic particle optimization of quasi-axisymmetric stellarator equilibria

Abstract An important goal of stellarator optimization is to achieve good confinement of energetic particles such as, in the case of a reactor, alphas created by deuterium–tritium fusion. In this work, a fixed-boundary stellarator equilibrium was re-optimized for energetic particle confinement via a two-step process: first, by minimizing deviations from quasi-axisymmetry (QA) on a single flux surface near the mid-radius, and secondly by maintaining this improved QA while minimizing the analytical quantity Γ C , which represents the angle between magnetic flux surfaces and contours of J | | , the second adiabatic invariant. This was performed multiple times, resulting in a group of equilibria with significantly reduced energetic particle losses, as evaluated by Monte Carlo simulations of alpha particles in scaled-up versions of the equilibria. This is the first time that energetic particle losses in a QA stellarator have successfully been reduced by optimizing Γ C . The relationship between energetic particle losses and metrics such as QA error ( E q a ) and Γ C in this set of equilibria were examined via statistical methods and a nearly linear relationship between volume-averaged Γ C and prompt particle losses was found.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2020

Radar Super Resolution using a Deep Convolutional Neural Network

Super-resolution involves synthetically increasing the resolution of gridded data beyond its native resolution. Typically, this is done using interpolation schemes, which estimate sub-grid scale values from neighboring data, and perform the same operation everywhere regardless of the large-scale context, or by requiring a network of radars with overlapping fields of view. Recently, significant progress has been made in single image super resolution using convolutional neural networks. Conceptually, a neural network may be able to learn relations between large scale precipitation features and the associated sub-pixel scale variability and outperform interpolation schemes. Here, we use a deep convolutional neural network to artificially enhance the resolution of NEXRAD PPI scans. The model is trained on 6-months of reflectivity observations from the Langley Hill WA (KLGX) radar, and we find that it substantially outperforms common interpolation schemes for x4 and x8 resolution increases based on several objective error and perceptual quality metrics.

radar, machine learning, super resolution, Remote ↗
DOE OSTI2022

Development of advanced machine learning models for analysis of plutonium surrogate optical emission spectra

This work investigates and applies machine learning paradigms seldom seen in analytical spectroscopy for quantification of gallium in cerium matrices via processing of laser-plasma spectra. Ensemble regressions, support vector machine regressions, Gaussian kernel regressions, and artificial neural network techniques are trained and tested on cerium-gallium pellet spectra. A thorough hyperparameter optimization experiment is conducted initially to determine the best design features for each model. The optimized models are evaluated for sensitivity and precision using the limit of detection (LoD) and root mean-squared error of prediction (RMSEP) metrics, respectively. Gaussian kernel regression yields the superlative predictive model with an RMSEP of 0.33% and an LoD of 0.015% for quantification of Ga in a Ce matrix. This study concludes that these machine learning methods could yield robust prediction models for rapid quality control analysis of plutonium alloys.

Rao, Ashwin P. (ORCID:0000000319312568)↗
DOE OSTI2023

Evaluating E. coli genome‐scale metabolic model accuracy with high‐throughput mutant fitness data

Abstract The Escherichia coli genome‐scale metabolic model (GEM) is an exemplar systems biology model for the simulation of cellular metabolism. Experimental validation of model predictions is essential to pinpoint uncertainty and ensure continued development of accurate models. Here, we quantified the accuracy of four subsequent E. coli GEMs using published mutant fitness data across thousands of genes and 25 different carbon sources. This evaluation demonstrated the utility of the area under a precision–recall curve relative to alternative accuracy metrics. An analysis of errors in the latest (iML1515) model identified several vitamins/cofactors that are likely available to mutants despite being absent from the experimental growth medium and highlighted isoenzyme gene‐protein‐reaction mapping as a key source of inaccurate predictions. A machine learning approach further identified metabolic fluxes through hydrogen ion exchange and specific central metabolism branch points as important determinants of model accuracy. This work outlines improved practices for the assessment of GEM accuracy with high‐throughput mutant fitness data and highlights promising areas for future model refinement in E. coli and beyond.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Comparison of Machine Learning-Based Predictive Models of the Nutrient Loads Delivered from the Mississippi/Atchafalaya River Basin to the Gulf of Mexico

Predicting nutrient loads is essential to understanding and managing one of the environmental issues faced by the northern Gulf of Mexico hypoxic zone, which poses a severe threat to the Gulf’s healthy ecosystem and economy. The development of hypoxia in the Gulf of Mexico is strongly associated with the eutrophication process initiated by excessive nutrient loads. Due to the complexities in the excessive nutrient loads to the Gulf of Mexico, it is challenging to understand and predict the underlying temporal variation of nutrient loads. The study was aimed at identifying an optimal predictive machine learning model to capture and predict nonlinear behavior of the nutrient loads delivered from the Mississippi/Atchafalaya River Basin (MARB) to the Gulf of Mexico. For this purpose, monthly nutrient loads (N and P) in tons were collected from US Geological Survey (USGS) monitoring station 07373420 from 1980 to 2020. Machine learning models—including autoregressive integrated moving average (ARIMA), gaussian process regression (GPR), single-layer multilayer perceptron (MLP), and a long short-term memory (LSTM) with the single hidden layer—were developed to predict the monthly nutrient loads, and model performances were evaluated by standard assessment metrics—Root Mean Square Error (RMSE) and Correlation Coefficient (R). The residuals of predictive models were examined by the Durbin–Watson statistic. The results showed that MLP and LSTM persistently achieved better accuracy in predicting monthly TN and TP loads compared to GPR and ARIMA. In addition, GPR models achieved slightly better test RMSE score than ARIMA models while their correlation coefficients are much lower than ARIMA models. Moreover, MLP performed slightly better than LSTM in predicting monthly TP loads while LSTM slightly outperformed for TN loads. Furthermore, it was found that the optimizer and number of inputs didn’t show effects on the LSTM performance while they exhibited impacts on MLP outcomes. This study explores the capability of machine learning models to accurately predict nonlinearly fluctuating nutrient loads delivered to the Gulf of Mexico. Further efforts focus on improving the accuracy of forecasting using hybrid models which combine several machine learning models with superior predictive performance for nutrient fluxes throughout the MARB.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Calibration of reactive burn and Jones-Wilkins-Lee parameters for simulations of a detonation-driven flow experiment with uncertainty quantification

Here, uncertainties in the explosive-specific parameters of the Jones-Wilkins-Lee (JWL) equation of state (EOS) are carefully considered in hydrodynamic simulations of an explosive experiment to minimize the error in the flow prediction. Experimental data of the leading shock position in the transverse direction over time serves as the prediction metric for quantifying simulation prediction error. The uncertainty quantification technique, global sensitivity analysis, is utilized to determine the JWL parameters to which the transverse shock propagation is most sensitive. A polynomial response surface (PRS) is constructed in the space of the most influential JWL parameters, and the point of minimum error between the experimental data and the PRS yields calibrated JWL parameters for the experimental flow. The simulation results following the parameter calibration show good agreement with the experimental data. It was found that two significant parameters, the heat release per unit mass of reactant Q and JWL model exponent R 1 are strongly related, which makes it difficult to identify accurate values.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Probabilistic Power Consumption Modeling for Commercial Buildings Using Logistic Regression Markov Chain

The total energy consumed by buildings takes up to 40% of U.S. energy use, in which a large portion is contributed by commercial buildings. Building performance optimization is desirable but requires accurate building models with uncertainties taken into account. This paper proposes a novel probabilistic modeling method using Logistic Regression Markov Chain (LRMC). The LRMC model enhances the performance of traditional Markov Chain (MC) models by adopting time-variant transition matrices calibrated using logistic regression with exogenous inputs. Compared with existing building models, the proposed model produces accurate multi-step modeling results with full probability distribution. The proposed probabilistic building model is tested using actual commercial building measurements and modeling performance is evaluated with two probabilisitc metrics. The results show that the LRMC model has higher accuracy than traditional MC model and Logistic Regression (LR) model in that it yields lower error scores under both evaluation metrics.

Building modeling↗
DOE OSTI2024

Benchmarking machine learning strategies for phase-field problems

Abstract We present a comprehensive benchmarking framework for evaluating machine-learning approaches applied to phase-field problems. This framework focuses on four key analysis areas crucial for assessing the performance of such approaches in a systematic and structured way. Firstly, interpolation tasks are examined to identify trends in prediction accuracy and accumulation of error over simulation time. Secondly, extrapolation tasks are also evaluated according to the same metrics. Thirdly, the relationship between model performance and data requirements is investigated to understand the impact on predictions and robustness of these approaches. Finally, systematic errors are analyzed to identify specific events or inadvertent rare events triggering high errors. Quantitative metrics evaluating the local and global description of the microstructure evolution, along with other scalar metrics representative of phase-field problems, are used across these four analysis areas. This benchmarking framework provides a path to evaluate the effectiveness and limitations of machine-learning strategies applied to phase-field problems, ultimately facilitating their practical application.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Highly accurate and constrained density functional obtained with differentiable programming

Using an end-to-end differentiable implementation of the Kohn-Sham self-consistent field equations, we obtain a highly accurate neural network–based exchange and correlation (XC) functional of the electronic density. The functional is optimized using information on both energy and density while exact constraints are enforced through an appropriate neural network architecture. Here we evaluate our model against different families of XC approximations and show that at the meta-GGA level our functional exhibits unprecedented accuracy for both energy and density predictions. For nonempirical functionals, there is a strong linear correlation between energy and density errors. We use this correlation to define an XC functional quality metric that includes both energy and density errors, leading to an improved way to rank different approximations.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Towards Scaling Law Analysis For Spatiotemporal Weather Data

Compute-optimal scaling laws are relatively well studied for NLP and CV, where objectives are typically single-step and targets are comparatively homogeneous. Weather forecasting is harder to characterize in the same framework: autoregressive rollouts compound errors over long horizons, outputs couple many physical channels with disparate scales and predictability, and globally pooled test metrics can disagree sharply with per-channel, late-lead behavior implied by short-horizon training. We extend neural scaling analysis for autoregressive weather forecasting from single-step training loss to long rollouts and per-channel metrics. We quantify (1) how prediction error is distributed across channels and how its growth rate evolves with forecast horizon, (2) if power law scaling holds for test error, relative to rollout length when error is pooled globally, and (3) how that fit varies jointly with horizon and channel for parameter, data, and compute-based scaling axes. We find strong cross-channel and cross-horizon heterogeneity: pooled scaling can look favorable while many channels degrade at late leads. We discuss implications for weighted objectives, horizon-aware curricula, and resource allocation across outputs.

Kiefer Jr, Alexander [ORNL] (ORCID:000000025398874↗
DOE OSTI2021

Probabilistic Day-Ahead Forecasting Using an Analog Ensemble Approach for Wind Farm Grid Services

Wind resource assessment and wind power forecasting are used in research and industry to anticipate future power output at scales ranging from individual wind turbines to entire wind farms. Probabilistic day-ahead wind forecasting is useful for anticipating how a wind farm could potentially participate in the day-ahead market by providing upper and lower bounds for expected power generation, thus informing grid operators of its uncertainty. Understanding this uncertainty is part of a larger project focused on building a platform that combines efforts in weather forecasting, aerodynamic and economic modeling to create maximum value of a wind plant to better provide services to the grid. This effort is also known as the Atmosphere to Electrons to Grid (A2E2G) project. One method for producing a probabilistic forecast is through the analog ensemble approach (Delle Monache et al., 2011). This method leverages historical forecasts and their corresponding observations as a training data set from which future forecasts can be made. For some future forecast, the most similar historical forecasts (analogs) are identified on a regular time basis such as once per a 3-hour window. The most similar analogs, based on a metric such as root mean square error (RMSE), are recorded and their corresponding verifying observations are used as an ensemble member for this future forecast. Prior work in this area demonstrates improvements over raw Numerical Weather Prediction (NWP) forecasts and shows skill similar to techniques such as logistic regression and machine learning (Delle Monache et al., 2013; Alessandrini et al., 2015). Here, we take the High-Resolution Rapid Refresh model (HRRR) day-ahead forecast (0-36 hours) to create a probabilistic day-ahead forecast using an analog ensemble approach. The HRRR has an hourly temporal resolution, with a spatial resolution of 3 km. The 12 UTC HRRR model run is downloaded every day for one year from August 2019 - July 2020, with the first 11 months serving as a bank of analogs from which the forecasting algorithm can create a probabilistic forecast. Once downloaded, the original HRRR forecast is temporally interpolated to 5-minutes, aligning with both the temporal resolution of the observations as well as the timescale relevant for day-ahead power forecasts. The forecast is validated at the M2 tower at the Flatirons Campus of the National Renewable Energy Laboratory (NREL) at a typical wind turbine height of 80 m. Variables such as wind speed, wind direction, and turbulence intensity are incorporated into the probabilistic forecast model and weighted according to their relative importance to the forecast. Based on metrics such as mean bias error (MBE), mean absolute error (MAE), and root mean square error, the analog ensemble forecast outperforms the raw HRRR forecast during the testing period of July 2020. Figure 1 illustrates an example day-ahead forecast compared against the verifying observations. The general variability and ramps are captured throughout the day, with potential to further improve the analog ensemble model through machine learning techniques.

numerical weather prediction↗