SEARCH · Engineering Papers

Results for “error metric”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Model validation and error attribution for a drifting qubit

Qubit performance is often reported in terms of a variety of single-value metrics, each providing a facet of the underlying noise mechanism limiting performance. However, the value of these metrics may drift over long timescales, and reporting a single number for qubit performance fails to account for the low-frequency noise processes that give rise to this drift. Here, in this work, we demonstrate how we can use the distribution of these values to validate or invalidate candidate noise models. We focus on the case of randomized benchmarking (RB), where typically a single error rate is reported but this error rate can drift over time when multiple passes of RB are performed. We show that using a statistical test as simple as the Kolmogorov-Smirnov statistic on the distribution of RB error rates can be used to rule out noise models, assuming the experiment is performed over a long enough time interval to capture relevant low frequency noise. With confidence in a noise model, we show how care must be exercised when performing error attribution using the distribution of drifting RB error rate.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

NASA NTRS1990

Validity of the two-level model for Viterbi decoder gap-cycle performance

A two-level model has previously been proposed for approximating the performance of a Viterbi decoder which encounters data received with periodically varying signal-to-noise ratio. Such cyclically gapped data is obtained from the Very Large Array (VLA), either operating as a stand-alone system or arrayed with Goldstone. This approximate model predicts that the decoder error rate will vary periodically between two discrete levels with the same period as the gap cycle. It further predicts that the length of the gapped portion of the decoder error cycle for a constraint length K decoder will be about K-1 bits shorter than the actual duration of the gap. The two-level model for Viterbi decoder performance with gapped data is subjected to detailed validation tests. Curves showing the cyclical behavior of the decoder error burst statistics are compared with the simple square-wave cycles predicted by the model. The validity of the model depends on a parameter often considered irrelevant in the analysis of Viterbi decoder performance, the overall scaling of the received signal or the decoder's branch-metrics. Three scaling alternatives are examined: optimum branch-metric scaling and constant branch-metric scaling combined with either constant noise-level scaling or constant signal-level scaling. The simulated decoder error cycle curves roughly verify the accuracy of the two-level model for both the case of optimum branch-metric scaling and the case of constant branch-metric scaling combined with constant noise-level scaling. However, the model is not accurate for the case of constant branch-metric scaling combined with constant signal-level scaling.

Dolinar, S.↗

NASA NTRS1993

An empirical comparison of a dynamic software testability metric to static cyclomatic complexity

This paper compares the dynamic testability prediction technique termed 'sensitivity analysis' to the static testability technique termed cyclomatic complexity. The application that we chose in this empirical study is a CASE generated version of a B-737 autoland system. For the B-737 system we analyzed, we isolated those functions that we predict are more prone to hide errors during system/reliability testing. We also analyzed the code with several other well-known static metrics. This paper compares and contrasts the results of sensitivity analysis to the results of the static metrics.

Voas, Jeffrey M.↗

DOE OSTI2026

Crowdsourcing the Frontier: Advancing Hybrid Physics‐ML Climate Simulation via a $\$$50,000 Kaggle Competition

Subgrid machine-learning (machine learning [ML]) parameterizations have the potential to introduce a new generation of climate models that incorporate the effects of higher-resolution physics without incurring the prohibitive computational cost associated with more explicit physics-based simulations. However, important issues, ranging from online instability to inconsistent online performance, have limited their operational use for long-term climate projections. To more rapidly drive progress in solving these issues, domain scientists and ML researchers opened up the offline aspect of this problem to the broader ML and data science community with the release of ClimSim, a NeurIPS Data sets and Benchmarks publication, and an associated Kaggle competition. This paper reports on the downstream results of the Kaggle competition by coupling emulators inspired by the winning teams' architectures to an interactive climate model (including full cloud microphysics, a regime historically prone to online instability) and systematically evaluating their online performance. Our results demonstrate that online stability in the low-resolution real-geography setting is reproducible across multiple diverse architectures, which we consider a key milestone. All tested architectures exhibit strikingly similar offline and online biases, though their responses to architecture-agnostic design choices (e.g., expanding the list of input variables) can differ significantly. Multiple Kaggle-inspired architectures achieve state-of-the-art results on certain metrics such as zonal mean bias patterns and global Root Mean Squared Error, indicating that crowdsourcing the essence of the offline problem is one path to improving online performance in hybrid physics-AI climate simulation.

Environmental sciences↗

DOE OSTI2026

Fast baryonic field painting for Sunyaev-Zel’dovich analyses: Transfer function vs hybrid effective field theory

Here, we present two approaches for “painting” baryonic properties relevant to the Sunyaev-Zel’dovich (SZ) effect—optical depth and Compton-y—onto three-dimensional N-body simulations, using the MillenniumTNG suite as a benchmark. The goal of these methods is to produce fast and accurate reconstruction methods to aid future analyses of baryonic feedback using the SZ effect. The first approach employs a Gaussian process emulator to model the SZ quantities via a transfer function, while the second utilizes hybrid effective field theory (HEFT) to reproduce these quantities within the simulation. Our analysis involves comparing both methods to the true MillenniumTNG optical depth and Compton-y fields using several metrics, including the cross-correlation coefficient, power spectrum, and power spectrum error. Additionally, we assess how well the reconstructed fields correlate with dark matter haloes across various mass thresholds. The results indicate that the transfer function method yields more accurate reconstructions for fields with initially high correlations (r ≈ 1), such as between the optical depth and dark matter fields. Conversely, the HEFT-based approach proves more effective in enhancing correlations for fields with weaker initial correlations (r ∼ 0.5), such as between the Compton-y and dark matter fields. Lastly, we discuss extensions of our methods to improve the reconstruction performance at the field level.

Liu, R. Henry [University of California, Berkeley,↗

DOE OSTI2024

Towards Full Field-of-View Fourier Ptychography for Extreme Ultraviolet Microscope

We evaluate various Fourier ptychographic microscopy (FPM) reconstruction algorithms using both simulated and experimental data acquired from an Extreme Ultraviolet (EUV, 13.5 nm wavelength) microscope. We specifically focus on the algorithms' ability to robustly address field-dependent aberrations, which enables increased spatial resolution and quantitative phase imaging across an expanded field of view. We systematically compare the algorithms' performance under aberrations for a single zoneplate imaging system, utilizing Fourier Ring Correlation (FRC) as a systematic metric for assessing reconstruction quality. Furthermore, we explore the impact of systematic errors on the reconstruction of experimental data, aiming to increase the effective field of view by 25-fold, from the nominal 5x5 um2 diffraction-limited area. Additionally, our evaluation incorporates innovative FPM-adjacent methodologies, including the Angular Ptychographic Imaging with Closed-form method (APIC), for reconstructing EUV images.

Gu, Chaoying↗

DOE OSTI2023

Trust Model System for the Energy Grid of Things Network Communications

Network communication is crucial in the Energy Grid of Things (EGoT). Without a network connection, the energy grid becomes just a power grid where the energy resources are available to the customer uni-directionally. A mechanism to analyze and optimize the energy usage of the grid can only happen through a medium, a communications network, that enables information exchange between the grid participants and the service provider. Security implementers of EGoT network communication take extraordinary measures to ensure the safety of the energy grid, a critical infrastructure, as well as the safety and privacy of the grid participants. With the dynamic nature of network communication of the EGoT, the information provided by the customer or the service provider can be falsified by a malicious attacker. Therefore, a trust model is necessary to monitor any abnormal activities. This paper describes a distributed trust model system that meets the need of the EGoT. This paper describes methods for evaluating and improving the distributed trust model using standard hypothesis testing metrics such as true positive, false positive, true negative, false negative, equal error rate, and F1 score. Example calculations are shown based on generated sample data.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2022

The Exploitation of Data Reduction for Visualization

The disparity between the computational speed and storage bandwidth, as demonstrated in Figure 1, is a well known problem that grows with each successive generation. The visualization community is principally responding to this issue by using in situ to reduce which data must be written to storage. However, other communities are taking different, possibly complementary approaches. In particular, data compression is a common general approach to reduce storage demands. Data compression technologies are typically not designed with post processing in mind. The principal metrics measured are compression ratio, the improved bandwidth to storage, and the error introduced. It is assumed that data is inflated to its full size before any post processing can happen. Although when talking about bandwidth disparities, HPC’s dirty little secret is that no part of the memory nor interconnect hardware is increasing at the rate of computation. For example, the Summit supercomputer has a peak computation rate almost 10 times its predecessor, Titan, but only about 4 times the memory, less than twice the aggregate memory bandwidth, and almost no improvement in the interconnect bisection bandwidth. Naively inflating data for post processing does not help with limitations in the memory and interconnect systems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Development and evaluation of a new 4DEnVar-based weakly coupled ocean data assimilation system in E3SMv2

The development, implementation, and evaluation of a new weakly coupled ocean data assimilation (WCODA) system for the fully coupled Energy Exascale Earth System Model version 2 (E3SMv2) utilizing the four-dimensional ensemble variational (4DEnVar) method are presented in this study. The 4DEnVar method, based on the dimension-reduced projection four-dimensional variational (DRP-4DVar) approach, replaces the adjoint model with the ensemble technique, thereby reducing computational demands. Monthly mean ocean temperature and salinity data from the EN4.2.1 reanalysis are integrated into the ocean component of E3SMv2 from 1950 to 2021 with the goal of providing realistic initial conditions for decadal predictions and predictability studies. The performance of the WCODA system is assessed using various metrics, including the reduction rate of the cost function, root mean square error (RMSE) differences, correlation differences, and model biases. Results indicate that the WCODA system effectively assimilates the reanalysis data into the climate model, consistently achieving negative reduction rates of the cost function and notable improvements in RMSE and correlation across various ocean layers and regions. Significant enhancements are observed in the upper ocean layers across the majority of global ocean regions, particularly in the north Atlantic, north Pacific, and Indian Ocean. Model biases in sea surface temperature and salinity are also substantially reduced. For sea surface temperature, cold biases in the north Pacific and north Atlantic are diminished by about 1–2 °C, and warm biases in the Southern Ocean are corrected by approximately 1.5–2.5 °C. In terms of salinity, improvements are observed with bias reductions of about 0.5–1 psu in the north Atlantic and north Pacific and up to 1.5 psu in parts of the Southern Ocean. The ultimate goal of the WCODA system is to advance the predictive capabilities of E3SM for subseasonal to decadal climate predictions, thereby supporting research on strategic energy-sector policies and planning.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Count Every Trip: Finding the Uncertainty in Energy Estimates Made from Inferred Travel Modes

To properly inform transport policy and infrastructure changes, transportation related metrics need both measured values and uncertainties of those values. Travel monitoring smartphone apps can record people's travel behavior, but trip data quality is limited by sensor errors, user labeling rates and the accuracy of inference algorithms used for travel diary creation. We discuss the use of phone app recorded travel diary data to estimate energy consumption, and propose the use of propagation of variance to find error bars for such estimates. We define energy consumption for one trip as trip length times the energy intensity per distance unit of the travel mode used. We characterize trip length errors with relative error and inferred trip mode errors with confusion matrix columns. The resulting variances of each measurement are then propagated to the final calculated energy consumption. We tested our uncertainty methods on a dataset that used phone app data combined with prompted recall, consisting of 92,234 labeled trips for over 500,000 miles. Accounting for uncertainty using expected energy intensities and variance propagation gives a dataset-wide aggregate energy consumption percent error of about 9%, within one standard deviation from the truth. Future work could involve applying similar methods to other travel diary based metrics.

ADVANCED PROPULSION SYSTEMS,MATHEMATICS AND COMPUT↗

DOE OSTI2023

Count Every Trip: Finding the Uncertainty in Energy Estimates Made from Inferred Travel Modes

ADVANCED PROPULSION SYSTEMS↗

DOE OSTI2023

Estimating Travel Energy Consumption Uncertainty Based on Inferred Travel Mode and Sensed Travel Length

To properly inform transport policy and infrastructure changes, transportation related metrics need both measured values and uncertainties of those values. Travel monitoring smartphone apps can record people's travel behavior, but trip data quality is limited by sensor errors, user labeling rates and the accuracy of inference algorithms used for travel diary creation. We discuss the use of phone app recorded travel diary data to estimate energy consumption, and propose the use of propagation of variance to find error bars for such estimates. We define energy consumption for one trip as trip length times the energy intensity per distance unit of the travel mode used. We characterize trip length errors with relative error and inferred trip mode errors with confusion matrix columns. The resulting variances of each measurement are then propagated to the final calculated energy consumption. We tested our uncertainty methods on a dataset that used phone app data combined with prompted recall, consisting of 92,234 labeled trips for over 500,000 miles. Accounting for uncertainty using expected energy intensities and variance propagation gives a dataset-wide aggregate energy consumption percent error of about 8%, within one standard deviation from the truth. Future work could involve applying similar methods to other travel diary based metrics.

ADVANCED PROPULSION SYSTEMS,ENERGY PLANNING, POLIC↗

NASA NTRS1993

Specification-based software sizing: An empirical investigation of function metrics

For some time the software industry has espoused the need for improved specification-based software size metrics. This paper reports on a study of nineteen recently developed systems in a variety of application domains. The systems were developed by a single software services corporation using a variety of languages. The study investigated several metric characteristics. It shows that: earlier research into inter-item correlation within the overall function count is partially supported; a priori function counts, in themself, do not explain the majority of the effort variation in software development in the organization studied; documentation quality is critical to accurate function identification; and rater error is substantial in manual function counting. The implication of these findings for organizations using function based metrics are explored.

Jeffery, Ross↗

DOE OSTI2026

Scaling open-weight large language models for hydropower regulatory information extraction: A systematic analysis

Information extraction from regulatory and technical documents using large language models (LLMs) involves practical trade-offs between extraction quality and computational cost. We evaluate eight open-weight LLMs spanning 0.6B–70B parameters on hydropower licensing documents and report deployment-oriented evidence under a unified extraction schema and evaluation protocol. Across the model set, we observe clear scale-dependent trends in both baseline extraction quality and the effectiveness of reflective reasoning (self-checking) under our fixed-prompt, no-augmentation setting. Mid-scale models often provide a favorable balance of accuracy and efficiency, whereas the smallest models show limited or inconsistent gains from the reasoning variants tested. Larger models achieve the highest overall F1 scores but incur substantially greater compute and infrastructure requirements. We further find that reliability failure modes can distort conventional metrics in this domain: in particular, high recall can coincide with systematic extraction errors when models fabricate values for fields that are absent from the source text, underscoring the importance of conservative null handling and evidence-grounded evaluation. Overall, our study provides a reproducible resource–performance comparison for open-weight LLM-based extraction in hydropower regulatory documentation and offers practical guidance for model selection under different deployment constraints.

Evaluation protocol↗

DOE OSTI2024

The 4DEnVar-based weakly coupled land data assimilation system for E3SM version 2

Abstract. A new weakly coupled land data assimilation (WCLDA) system based on the four-dimensional ensemble variational (4DEnVar) method is developed and applied to the fully coupled Energy Exascale Earth System Model version 2 (E3SMv2). The dimension-reduced projection four-dimensional variational (DRP-4DVar) method is employed to implement 4DVar using the ensemble technique instead of the adjoint technique. With an interest in providing initial conditions for decadal climate predictions, monthly mean anomalies of soil moisture and temperature from the Global Land Data Assimilation System (GLDAS) reanalysis from 1980 to 2016 are assimilated into the land component of E3SMv2 within the coupled modeling framework with a 1-month assimilation window. The coupled assimilation experiment is evaluated using multiple metrics, including the cost function, assimilation efficiency index, correlation, root-mean-square error (RMSE), and bias, and compared with a control simulation without land data assimilation. The WCLDA system yields improved simulation of soil moisture and temperature compared with the control simulation, with improvements found throughout the soil layers and in many regions of the global land. In terms of both soil moisture and temperature, the assimilation experiment outperforms the control simulation with reduced RMSE and higher temporal correlation in many regions, especially in South America, central Africa, Australia, and large parts of Eurasia. Furthermore, significant improvements are also found in reproducing the time evolution of the 2012 US Midwest drought, highlighting the crucial role of land surface in drought lifecycle. The WCLDA system is intended to be a foundational resource for research to investigate land-derived climate predictability.

58 GEOSCIENCES↗

NASA NTRS2012

A Systems Modeling Approach for Risk Management of Command File Errors

The main cause of commanding errors is often (but not always) due to procedures. Either lack of maturity in the processes, incompleteness of requirements or lack of compliance to these procedures. Other causes of commanding errors include lack of understanding of system states, inadequate communication, and making hasty changes in standard procedures in response to an unexpected event. In general, it's important to look at the big picture prior to making corrective actions. In the case of errors traced back to procedures, considering the reliability of the process as a metric during its' design may help to reduce risk. This metric is obtained by using data from Nuclear Industry regarding human reliability. A structured method for the collection of anomaly data will help the operator think systematically about the anomaly and facilitate risk management. Formal models can be used for risk based design and risk management. A generic set of models can be customized for a broad range of missions.

probabilistic risk↗

NASA NTRS2017

Information Theoretic Evaluation of Satellite Soil Moisture Retrievals

Microwave radiometry has a long legacy of providing estimates of remotely sensed near surfacesoil moisture measurements over continental and global scales. A consistent assessment of theerrors and uncertainties associated with these retrievals is important for their effective utilization in modeling, data assimilation and end-use application environments. This article presents an evaluationof soil moisture retrieval products from AMSR-E, ASCAT, SMOS, AMSR2 and SMAPinstruments using information theory-based metrics. These metrics rely on time series analysis ofsoil moisture retrievals for estimating the measurement error, level of randomness (entropy) andregularity (complexity) of the data. The results of the study indicate that the measurement errors inthe remote sensing retrievals are significantly larger than that of the ground soil moisture measurements.The SMAP retrievals, on the other hand, were found to have reduced errors (comparable to Preprint submitted to Remote Sensing of Environment October 1, 2017those of in-situ datasets), particularly over areas with moderate vegetation. The SMAP retrievals also demonstrate high information content relative to other retrieval products, with higher levelsof complexity and reduced entropy. Finally, a joint evaluation of the entropy and complexity ofremotely sensed soil moisture products indicates that the information content of the AMSR-E, ASCAT,SMOS and AMSR2 retrievals is low, whereas SMAP retrievals show better performance. The use of information theoretic assessments is effective in quantifying the required levels of improvements needed in the remote sensing soil moisture retrievals to enhance their utility and information content.

Information theory↗

NASA NTRS2000

Performance and Optimization Metrics for Interferometric Radar

This paper provides a set of analytic equations that can be used to estimate system performance or to assist in the design of a system in that individual sources of height error can be balanced to minimize cost and maximize efficient use of resources.

optimization metrics interferometric radar interfe↗