Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Error Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

A Nonlinear Least Squares Phasor Estimation Algorithm with a Trust Metric

The paper presents a separable nonlinear least squares (NLLS)-based approach for estimation of fundamental-frequency phasors from sampled point-on-wave measurements. An analytical connection is established between the NLLS cost function and the discrete Fourier transform (DFT)-based periodogram of the input signal. This periodogram-based interpretation of the cost function offers an intuitive and easy-to-implement solution for the frequency estimate. Using the residual error of the NLLS-fit, the paper also presents an insightful measure for ascertaining the quality of the phasor estimates and their validity, especially for data windows containing signal transients.

Chatterjee, Kaustav↗
DOE OSTI2021

Distributionally Safe Path Planning: Wasserstein Safe RRT

In this paper, we propose a Wasserstein metric-based random path planning algorithm. Wasserstein Safe RRT (W-Safe RRT) provides finite-sample probabilistic guarantees on the safety of a returned path in an uncertain obstacle environment. Vehicle and obstacle states are modeled as distributions based upon state and model observations. Additionally, we define limits on distributional sampling error so the Wasserstein distance between a vehicle state distribution and obstacle distributions can be bounded. This enables the algorithm to return safe paths with a confidence bound through combining finite sampling error bounds with calculations of the Wasserstein distance between discrete distributions. W-Safe RRT is compared against a baseline minimum encompassing ball algorithm, which ensures balls that minimally encompass discrete state and obstacle distributions do not overlap. The improved performance is verified in a 3D environment using single, multi, and rotating non-convex obstacle cases, with and without forced obstacle error in adversarial directions, showing that W-Safe RRT can handle poorly modeled complex environments.

42 ENGINEERING↗
DOE OSTI2021

Efficacy of the radial pair potential approximation for molecular dynamics simulations of dense plasmas

Macroscopic simulations of dense plasmas rely on detailed microscopic information that can be computationally expensive and is difficult to verify experimentally. In this work, we delineate the accuracy boundary between microscale simulation methods by comparing Kohn–Sham density functional theory molecular dynamics (KS-MD) and radial pair potential molecular dynamics (RPP-MD) for a range of elements, temperature, and density. By extracting the optimal RPP from KS-MD data using force matching, we constrain its functional form and dismiss classes of potentials that assume a constant power law for small interparticle distances. Our results show excellent agreement between RPP-MD and KS-MD for multiple metrics of accuracy at temperatures of only a few electron volts. The use of RPPs offers orders of magnitude decrease in computational cost and indicates that three-body potentials are not required beyond temperatures of a few eV. Due to its efficiency, the validated RPP-MD provides an avenue for reducing errors due to finite-size effects that can be on the order of ∼20%.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2023

Atomic protein structure refinement using all-atom graph representations and SE(3)-equivariant graph transformer

Abstract Motivation The state-of-art protein structure prediction methods such as AlphaFold are being widely used to predict structures of uncharacterized proteins in biomedical research. There is a significant need to further improve the quality and nativeness of the predicted structures to enhance their usability. In this work, we develop ATOMRefine, a deep learning-based, end-to-end, all-atom protein structural model refinement method. It uses a SE(3)-equivariant graph transformer network to directly refine protein atomic coordinates in a predicted tertiary structure represented as a molecular graph. Results The method is first trained and tested on the structural models in AlphaFoldDB whose experimental structures are known, and then blindly tested on 69 CASP14 regular targets and 7 CASP14 refinement targets. ATOMRefine improves the quality of both backbone atoms and all-atom conformation of the initial structural models generated by AlphaFold. It also performs better than two state-of-the-art refinement methods in multiple evaluation metrics including an all-atom model quality score—the MolProbity score based on the analysis of all-atom contacts, bond length, atom clashes, torsion angles, and side-chain rotamers. As ATOMRefine can refine a protein structure quickly, it provides a viable, fast solution for improving protein geometry and fixing structural errors of predicted structures through direct coordinate refinement. Availability and implementation The source code of ATOMRefine is available in the GitHub repository (https://github.com/BioinfoMachineLearning/ATOMRefine). All the required data for training and testing are available at https://doi.org/10.5281/zenodo.6944368.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Mbin v1.0

The Mbin software, is a software toolkit that implements the IMG metagenome binning pipeline. The software allows the user to process input metagenome contigs, and produces metagenome assembled genomes (metagenome bins) and valuation metrics per bin including completion and contamination estimates, quality assignment, predicted lineage and eukaryotic potential. It is currently packed as a portable docker container and provides the advantage of running the process of binning and analysis of the bins generated, using a suite of tools run sequentially with controls in place to capture errors and optional arguments to run a modified version depending on individual needs and capabilities.

Varghese, Neha↗
DOE OSTI2023

A Decision-Relevant Factor-Fixing Framework: Application to Uncertainty Analysis of a High-Dimensional Water Quality Model

Factor Fixing (FF) is a common method for reducing the number of model parameters to lower computational cost. FF typically starts with distinguishing the insensitive parameters from the sensitive and pursues uncertainty quantification (UQ) on the resulting reduced-order model, fixing each insensitive parameter at a fixed value. There is a need, however, to expand such a common approach to consider the effects of decision choices in the FF-UQ procedure on metrics of interest. Therefore, to guide the use of FF and increase confidence in the resulting dimension-reduced model, we propose a new adaptive framework consisting of four principles: (a) re-parameterize the model first to reduce obvious non-identifiable parameter combinations, (b) focus on decision relevance especially with respect to errors in quantities of interest (QoI), (c) conduct adaptive evaluation and robustness assessment of errors in the QoI across FF choices as sample size increases, and (d) reconsider whether fixing is warranted. The framework is demonstrated on a spatially-distributed water quality model. The error in estimates of QoI caused by FF can be estimated using a Polynomial Chaos Expansion (PCE) surrogate model. Built with 70 model runs, the surrogate is computationally inexpensive to evaluate and can provide global sensitivity indices for free. For the selected catchment, just two factors may provide an acceptably accurate estimate of model uncertainty in the average annual load of Total Suspended Solids (TSS), suggesting that reducing the uncertainty in these two parameters is a priority for future work before undertaking further formal uncertainty quantification.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Understanding and reducing mid-spatial frequency ripples during hemispherical sub-aperture tool glass polishing

We report during sub-aperture tool polishing of glass optics, mid-spatial surface ripples are generated because of material removal non-uniformities during tool linear translation (resulting in feed ripples) and tool pathway step overlaps (resulting in pitch ripples). A variety of tool influence function (TIF) spots, trenches, and patches were created to understand and minimize such ripples on fused silica workpieces after polishing with cerium oxide slurry using a rotating hemispherical pad-foam tool. The feed ripple amplitude can be decreased by reducing the non-uniformities in the pad texture and/or by minimizing a derived feed ripple metric ($r_f = V^{0.5}_{max}V_f/R_t$) via adjustments in processing parameters. Pitch ripples can be minimized by reducing relative step distance to spot radius ratio ($x_s/a_t$) and by achieving a flat bottom trench shape cross section or by reducing the material removal per pass. Using the combined methods, an overall ripple error of ~1.2nm rms has been achieved.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

Benchmarking performance changes in the simulation of extratropical modes of variability across CMIP generations

In this work, we evaluate extratropical modes of variability in the three most recent phases of the Coupled Model Intercomparison Project (CMIP3, 5, and 6) to gauge improvement of climate models over time. A suite of high-level metrics is employed to objectively evaluate how well climate models simulate the qobserved Northern Annular Mode (NAM), North Atlantic Oscillation (NAO), Pacific North America pattern (PNA), Southern Annular Mode (SAM), Pacific Decadal Oscillation (PDO), North Pacific Oscillation (NPO), and North Pacific Gyre Oscillation (NPGO). We apply a Common Basis Function (CBF) approach that projects model anomalies onto observed empirical orthogonal functions (EOFs), together with the traditional EOF approach, to CMIP Historical and AMIP models. We find simulated spatial patterns of those modes have been significantly improved in the newer models, although the skill improvement is sensitive to the mode and season considered. We identify some potential contributions to the pattern improvement of certain modes (e.g., the Southern Hemisphere jet and High-top vertical coordinate), however the performance changes are likely attributed to gradual improvement of the base climate and multiple relevant processes. Less performance improvement is evident in the mode amplitude of these modes and systematic overestimation of the mode amplitude in spring remains in the newer climate models. We find that the post-dominant season amplitude errors in atmospheric modes are not limited to coupled runs but are often already evident in AMIP simulations. This suggests that rectifying the egregious post-dominant season amplitude errors found in many models can be addressed in an atmospheric-only framework making it more tractable to address in the model development process.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

The DESC stellarator code suite. Part 1. Quick and accurate equilibria computations

Three-dimensional equilibrium codes are vital for stellarator design and operation, and high-accuracy equilibria are also necessary for stability studies. This paper details comparisons of two three-dimensional equilibrium codes: VMEC, which uses a steepest-descent algorithm to reach a minimum-energy plasma state, and DESC, which minimizes the magnetohydrodynamic (MHD) force error in real space directly. Accuracy as measured by satisfaction of MHD force balance is presented for each code, along with the computation time. It is shown that DESC is able to achieve more accurate solutions, especially near axis. The importance of higher-accuracy equilibria is shown in DESC's better agreement of stability metrics with asymptotic formulae. DESC's global Fourier–Zernike basis also yields solutions with analytic derivatives explicitly everywhere in the plasma volume, provides improved accuracy in the radial direction versus conventional finite differences and allows for exponential convergence. Further, DESC can compute a solution with the same accuracy as VMEC in order-of-magnitude less time.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2023

Earth Mover’s Distance as a Metric to Evaluate the Extent of Charge Transfer in Excitations Using Discretized Real-Space Densities

This paper presents a novel theoretical measure, μ EMD , based on the earth mover's distance (EMD), for quantifying the density shift caused by electronic excitations in molecules. As input, the EMD metric uses only the discretized ground- and excited-state electron densities in real space, rendering it compatible with almost all electronic structure methods used to calculate excited states. The EMD metric is compared against other popular theoretical metrics for describing the extent of electron-hole separation in a wide range of excited states (valence, Rydberg, charge transfer, etc.). Further, the results showcase the EMD metric's effectiveness across all excitation types and suggest that it is useful as an additional tool to characterize electronic excitations. The study also reveals that μ EMD can function as a promising diagnostic tool for predicting the failure of pure exchange-correlation functionals. Specifically, we show statistical relationships among the functional-driven errors, the exact exchange content within the functional, and the magnitude of μ EMD values.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

A deep learning approach for semantic segmentation of unbalanced data in electron tomography of catalytic materials

In computed TEM tomography, image segmentation represents one of the most basic tasks with implications not only for 3D volume visualization, but more importantly for quantitative 3D analysis. In case of large and complex 3D data sets, segmentation can be an extremely difficult and laborious task, and thus has been one of the biggest hurdles for comprehensive 3D analysis. Heterogeneous catalysts have complex surface and bulk structures, and often sparse distribution of catalytic particles with relatively poor intrinsic contrast, which possess a unique challenge for image segmentation, including the current state-of-the-art deep learning methods. To tackle this problem, we apply a deep learning-based approach for the multi-class semantic segmentation of a γ-Alumina/Pt catalytic material in a class imbalance situation. Specifically, we used the weighted focal loss as a loss function and attached it to the U-Net’s fully convolutional network architecture. We assessed the accuracy of our results using Dice similarity coefficient (DSC), recall, precision, and Hausdorff distance (HD) metrics on the overlap between the ground-truth and predicted segmentations. Our adopted U-Net model with the weighted focal loss function achieved an average DSC score of 0.96 ± 0.003 in the γ-Alumina support material and 0.84 ± 0.03 in the Pt NPs segmentation tasks. We report an average boundary-overlap error of less than 2 nm at the 90th percentile of HD for γ-Alumina and Pt NPs segmentations. The complex surface morphology of γ-Alumina and its relation to the Pt NPs were visualized in 3D by the deep learning-assisted automatic segmentation of a large data set of high-angle annular dark-field (HAADF) scanning transmission electron microscopy (STEM) tomography reconstructions.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Validation Metrics for Fixed Effects and Mixed-Effects Calibration

The modern scientific process often involves the development of a predictive computational model. To improve its accuracy, a computational model can be calibrated to a set of experimental data. A variety of validation metrics can be used to quantify this process. Some of these metrics have direct physical interpretations and a history of use, while others, especially those for probabilistic data, are more difficult to interpret. In this work, a variety of validation metrics are used to quantify the accuracy of different calibration methods. Frequentist and Bayesian perspectives are used with both fixed effects and mixed-effects statistical models. Through a quantitative comparison of the resulting distributions, the most accurate calibration method can be selected. Two examples are included which compare the results of various validation metrics for different calibration methods. It is quantitatively shown that, in the presence of significant laboratory biases, a fixed effects calibration is significantly less accurate than a mixed-effects calibration. This is because the mixed-effects statistical model better characterizes the underlying parameter distributions than the fixed effects model. The results suggest that validation metrics can be used to select the most accurate calibration model for a particular empirical model with corresponding experimental data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Exploring Hydrologic Model Process Connectivity at the Continental Scale Through an Information Theory Approach

Exploring water fluxes between hydrological model (HM) components is essential to assess and improve model realism. Many classical metrics for HM diagnosis rely solely on streamflow and hence provide limited insights into model performance across processes. This study applies an information theory measure known as “transfer entropy” (TE) to systematically quantify the transfer of information among major HM components. To test and demonstrate the benefits of TE, we use the Framework for Understanding Structural Errors (FUSE) model to mimic and compare four commonly used HM structures, VIC, PRMS, SACRAMENTO, and TOPMODEL, across 671 catchments spanning a variety of hydrologic regimes in the conterminous United States. We explore connections between HM components and catchment landscape characteristics (e.g., climate, topography, soil, and vegetation) and characterize their nonlinear associations using distance correlation and Spearman correlation coefficients. Our results indicate that while the information transferred from precipitation to runoff is similar across model structures (likely as a result of calibration), the information transferred among other components can vary significantly from a FUSE structure to another. We find that aridity, precipitation duration and frequency, snow fraction, mean elevation, forest area, and leaf area index are often significantly associated with TE between the main HM components. Here, we propose that the presence of meaningful nonlinear associations can be used to diagnose process representation in HMs. Our results highlight the necessity to enhance the conventional streamflow-only calibration approach for a more realistic representation of water dynamics in the models.

large-sample hydrology↗
DOE OSTI2024

Enriching OpenStreetMap network data for transportation applications: Insights into the impact of urban congestion on accessibility

OpenStreetMap (OSM) data is a valuable open-source resource for various transportation, traffic, and planning applications. However, OSM network data lack operating traffic speed information, which is critical for transport planning and operations. Addressing this shortcoming, this study leverages commercial vendor data (to serve as ground truth) with exogenous, open-source variables characterizing local transport infrastructure, land use, and demographic information to predict average congested traffic speeds on OSM networks. Three machine-learning models were tested and estimated for OSM links with and without speed limit information in the Denver metropolitan region. Among these, XGBoost performed best, with mean absolute errors of 3.27 and 3.62 mph for links with and without speed limits, respectively. The developed models accurately predicted traffic speeds for different hours and days of the week compared to ground truth data. Using these predicted speeds, drive accessibility scores were computed for the Denver region for different time periods using the Mobility Energy Productivity (MEP) metric to understand the impact of congestion on energy-efficient accessibility. Results show that congestion-adjusted drive accessibility can be significantly lower compared to accessibility calculated using free flow speeds. Specifically, weekday evening hours saw a 42 % drop in accessibility due to reduced speeds, particularly around downtown Denver. Across the Denver metro region, approximately half as many opportunities and jobs are accessible in under 20 min by car during the evening peak period relative to free flow conditions. These findings underscore the importance of using congestion-adjusted operating speeds rather than speed limits in accessibility calculations, as reliance on speed limits can substantially overestimate energy-efficient drive accessibility in large, car-centric cities susceptible to significant congestion. In conclusion, the methodology presented here could further enrich OSM network data, making them useful for an even broader range of transportation applications.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2025

Rapid Adaptation of Chemical Named Entity Recognition Using Few-Shot Learning and LLM Distillation

Named entity recognition (NER) has been widely used in chemical text mining for the automatic identification and extraction of chemical entities. However, existing chemical NER systems primarily focus on scenarios with abundant training data, requiring significant human effort on annotations. This poses challenges for applications in the chemical field, such as catalysis, where many advancements have traditionally relied on trial-and-error investigations and incremental adjustment of variables. This hinders catalysis science and technology progress in addressing emerging energy and environmental crises. In this work, we propose a few-shot NER model that can quickly adapt to extract new types of chemical entities by using only a limited number of annotated examples. Our model employs a metric-learning approach to transfer entity similarity knowledge from high-resource chemical domains (with abundant annotations) to enable effective entity recognition in low-resource specialized domains (limited annotation). We validate the effectiveness of our model on a few-shot chemical NER benchmark built based on six existing chemical NER data sets. Experiments show that the proposed few-shot NER model can achieve reasonable performance with only 5 examples per entity type and shows consistent improvement as the number of examples increases. Furthermore, we demonstrate how the proposed model can be trained with large language model (LLM) annotated data, opening a new pathway for rapid adaptation of NER systems. Furthermore, our approach leverages the knowledge broadness of large language models for chemistry while distilling this knowledge into a lightweight model suitable for efficient and in-house use.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Simulating organic aerosol in Delhi with WRF-Chem using the volatility-basis-set approach: exploring model uncertainty with a Gaussian process emulator

The nature and origin of organic aerosol in the atmosphere remain unclear. The gas–particle partitioning of semi-volatile organic compounds (SVOCs) that constitute primary organic aerosols (POAs) and the multigenerational chemical aging of SVOCs are particularly poorly understood. The volatility basis set (VBS) approach, implemented in air quality models such as WRF-Chem (Weather Research and Forecasting model with Chemistry), can be a useful tool to describe emissions of POA and its chemical evolution. However, the evaluation of model uncertainty and the optimal model parameterization may be expensive to probe using only WRF-Chem simulations. Gaussian process emulators, trained on simulations from relatively few WRF-Chem simulations, are capable of reproducing model results and estimating the sources of model uncertainty within a defined range of model parameters. In this study, a WRF-Chem VBS parameterization is proposed; we then generate a perturbed parameter ensemble of 111 model runs, perturbing 10 parameters of the WRF-Chem model relating to organic aerosol emissions and the VBS oxidation reactions. This allowed us to cover the model's uncertainty space and to compare outputs from each run to aerosol mass spectrometer observations of organic aerosol concentrations and O:C ratios measured in New Delhi, India. The simulations spanned the organic aerosol concentrations measured with the aerosol mass spectrometer (AMS). However, they also highlighted potential structural errors in the model that may be related to unsuitable diurnal cycles in the emissions and/or failure to adequately represent the dynamics of the planetary boundary layer. While the structural errors prevented us from clearly identifying an optimized VBS approach in WRF-Chem, we were able to apply the emulator in the following two periods: the full period (1–29 May) and a subperiod period of 14:00–16:00 h LT (local time) on 1–29 May. The combination of emulator analysis and model evaluation metrics allowed us to identify plausible parameter combinations for the analyzed periods. We demonstrate that the methodology presented in this study can be used to determine the model uncertainty and to identify the appropriate parameter combination for the VBS approach and hence to provide valuable information to improve our understanding of OA production.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Minimization of Measurement Uncertainty in Optical Frequency Domain Reflectometry

Optical frequency domain reflectometry (OFDR) is a technique for interrogating optical fiber sensors to generate relative, quasi-distributed measurements. Although Optical frequency domain reflectometry (OFDR) is increasingly being adopted for aerospace, energy production, and structural monitoring applications, the quantification of uncertainty for OFDR measurements has not been developed beyond sparse empirical relationships. To address this knowledge gap, an uncertainty metric for OFDR measurements was developed. This uncertainty metric was applied to weight the edges between OFDR measurements on directed correlation graphs and analyzed to minimize the cumulative uncertainty. In conclusion, this work is the first to propose an uncertainty metric for OFDR and provides a generalized mathematical framework for optimizing OFDR hardware selection, optical fiber sensor selection, and postprocessing strategy.

42 ENGINEERING↗
DOE OSTI2022

Challenges for unsupervised anomaly detection in particle physics

Anomaly detection relies on designing a score to determine whether a particular event is uncharacteristic of a given background distribution. One way to define a score is to use autoencoders, which rely on the ability to reconstruct certain types of data (background) but not others (signals). In this paper, we study some challenges associated with variational autoencoders, such as the dependence on hyperparameters and the metric used, in the context of anomalous signal (top and W) jets in a QCD background. We find that the hyperparameter choices strongly affect the network performance and that the optimal parameters for one signal are non-optimal for another. In exploring the networks, we uncover a connection between the latent space of a variational autoencoder trained using mean-squared-error and the optimal transport distances within the dataset. We then show that optimal transport distances to representative events in the background dataset can be used directly for anomaly detection, with performance comparable to the autoencoders. Whether using autoencoders or optimal transport distances for anomaly detection, we find that the choices that best represent the background are not necessarily best for signal identification. These challenges with unsupervised anomaly detection bolster the case for additional exploration of semi-supervised or alternative approaches.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗