Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “error metric”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Reactivity Coefficient Measurements and Sensitivity Studies [Abstract]

Nuclear data validation is often performed today using criticality measurements. The gold standard for criticality measurements is the International Criticality Safety Benchmark Experiment Project (ICSBEP). The validation specifically focuses on the effective multiplication factor (k eff ). K eff is a relatively easy parameter to infer and has reduced uncertainty due to being at or above critical. However, while k eff is a well-documented parameter with detailed sensitivity and uncertainty analysis, it cannot be used as a standalone metric to determine inaccuracies in nuclear data (e.g., cross section data, PFNS, nu), which is based on theory, physics, and differential measurements. The Experiments Underpinned by Computational Learning for Improvements in nuclear Data (EUCLID) project aims to identify compensating errors in specific nuclear data by optimally designing experiments that are sensitive to a suite of measurement parameters beyond k eff . By identifying how each parameter's nuclear data sensitivity differs from others, experiments can be designed to constrain questionable nuclear data. One sensitivity that is of particular interest to this project includes reactivity coefficient sensitivities.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

Quantum error mitigation by hidden inverses protocol in superconducting quantum devices

We present a method to improve the convergence of variational algorithms based on hidden inverses (HIs) to mitigate coherent errors. In the context of error mitigation, this means replacing the hardware implementation of certain Hermitian gates with their inverses. Doing so results in noise cancellation and a more resilient quantum circuit. This approach improves performance in a variety of two-qubit error models where the noise operator also inverts with the gate inversion. We apply the mitigation scheme on superconducting quantum processors running the variational quantum eigensolver (VQE) algorithm to find the H 2 ground-state energy. When implemented on superconducting hardware we find that the mitigation scheme effectively reduces the energy fluctuations in the parameter learning path in VQE, reducing the number of iterations for a converged value. We also provide a detailed numerical simulation of VQE performance under different noise models and explore how HIs and randomized compiling affect the underlying loss landscape of the learning problem. These simulations help explain our experimental hardware outcomes, helping to connect lower-level gate performance to application-specific behavior in contrast to metrics like fidelity which often do not provide an intuitive insight into observed high level performance.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2023

Bootstrapping outperforms community‐weighted approaches for estimating the shapes of phenotypic distributions

Abstract Estimating phenotypic distributions of populations and communities is central to many questions in ecology and evolution. These distributions can be characterized by their moments (mean, variance, skewness and kurtosis) or diversity metrics (e.g. functional richness). Typically, such moments and metrics are calculated using community‐weighted approaches (e.g. abundance‐weighted mean). We propose an alternative bootstrapping approach that allows flexibility in trait sampling and explicit incorporation of intraspecific variation, and show that this approach significantly improves estimation while allowing us to quantify uncertainty. We assess the performance of different approaches for estimating the moments of trait distributions across various sampling scenarios, taxa and datasets by comparing estimates derived from simulated samples with the true values calculated from full datasets. Simulations differ in sampling intensity (individuals per species), sampling biases (abundance, size), trait data source (local vs. global) and estimation method (two types of community‐weighting, two types of bootstrapping). We introduce the traitstrap R package, which contains a modular and extensible set of bootstrapping and weighted‐averaging functions that use community composition and trait data to estimate the moments of community trait distributions with their uncertainty. Importantly, the first function in the workflow, trait_fill , allows the user to specify hierarchical structures (e.g. plot within site, experiment vs. control, species within genus) to assign trait values to each taxon in each community sample. Across all taxa, simulations and metrics, bootstrapping approaches were more accurate and less biased than community‐weighted approaches. With bootstrapping, a sample size of 9 or more measurements per species per trait generally included the true mean within the 95% CI. It reduced average percent errors by 26%–74% relative to community‐weighting. Random sampling across all species outperformed both size‐ and abundance‐biased sampling. Our results suggest randomly sampling ~9 individuals per sampling unit and species, covering all species in the community and analysing the data using nonparametric bootstrapping generally enable reliable inference on trait distributions, including the central moments, of communities. By providing better estimates of community trait distributions, bootstrapping approaches can improve our ability to link traits to both the processes that generate them and their effects on ecosystems.

Maitner, Brian S.↗
DOE OSTI2021

CrossCheck: Rapid, Reproducible, and Interpretable Model Evaluation

Evaluation beyond aggregate performance metrics, e.g. F1-score, is crucial to both establish an appropriate level of trust in machine learning models and identify future model improvements. In this paper we demonstrate CrossCheck, an interactive visualization tool for rapid crossmodel comparison and reproducible error analysis. We describe the tool and discuss design and implementation details. We then present three use cases (named entity recognition, reading comprehension, and clickbait detection) that show the benefits of using the tool for model evaluation. CrossCheck allows data scientists to make informed decisions to choose between multiple models, identify when the models are correct and for which examples, investigate whether the models are making the same mistakes as humans, evaluate models’ generalizability and highlight models’ limitations, strengths and weaknesses. Furthermore, CrossCheck is implemented as a Jupyter widget, which allows rapid and convenient integration into data scientists’ model development workflows.

Arendt, Dustin L.↗
NASA NTRS1994

A Planar Approximation for the Least Reliable Bit Log-likelihood Ratio of 8-PSK Modulation

The optimum decoding of component codes in block coded modulation (BCM) schemes requires the use of the log-likelihood ratio (LLR) as the signal metric. An approximation to the LLR for the least reliable bit (LRB) in an 8-PSK modulation based on planar equations with fixed point arithmetic is developed that is both accurate and easily realizable for practical BCM schemes. Through an error power analysis and an example simulation it is shown that the approximation results in 0.06 dB in degradation over the exact expression at an E(sub s)/N(sub o) of 10 dB. It is also shown that the approximation can be realized in combinatorial logic using roughly 7300 transistors. This compares favorably to a look up table approach in typical systems.

Thesling, William H.↗
NASA NTRS2005

Preparing for the Huygens Probe Mission, Cassini orbit determination results for the first and second targeted Titan encounters

Dynamic modeling of the spacecraft and Saturn system, tracking data, including radio-metric and optical navigation data, and measurement modeling associated with the final trajectory analysis are described. Navigation predictions produced during the operational phase are compared with the final trajectory in order to gain insight into navigation performance and maneuver execution errors. Special attention is given to refinement of the dynamical environment of Saturn, particularly Titan, during the first two orbits.

navigation↗
NASA NTRS2013

Advanced Multimission Operations System (ATMO)

The HiiHat toolbox developed for CAT/ENVI provides principal investigators direct, immediate, flexible, and seamless interaction with their instruments and data from any location. Offering segmentation and neutral region division, it facilitates the discovery of key endmembers and regions of interest larger than a single pixel. Crucial to the analysis of hyperspectral data from Mars or Earth is the removal of unwanted atmospheric signatures. For Mars and the Compact Reconnaissance Imaging Spectrometer for Mars (CRISM), residual atmospheric CO2 absorption is both directly problematic and indicative of processing errors with implications to the scientific utility of any particular image region. Estimating this residual error becomes key both in selecting regions of low distortion, and also to select mitigating methods, such as neutral region division. This innovation, the ATMO estimator, provides a simple, 0-1 normalized scalar that estimates this distortion (see figure). The metric is defined as the coefficient of determination of a quadratic fit in the region of distorting atmospheric absorption (approx 2 micron). This mimics the behavior of existing CRISM team mineralogical indices to estimate the presence of known, interesting mineral signatures. This facilitates the ATMO metric's assimilation into existing planetary geology workflows.

Mandrake, Lucas↗
NASA NTRS2019

Seasonal Dependence of SMAP Radiometer-Based Soil Moisture Performance as Observed over Core Validation Sites

The NASA SMAP (Soil Moisture Active Passive) mission provides a global coverage of soil moisture measurements based on its L-band microwave radiometer every 2-3 days at about 40 km resolution. The soil moisture retrieval algorithms model the brightness temperature as a function of soil moisture, surface conditions and vegetation. External data sources inform the algorithms about the surface conditions and vegetation, which enable the retrieval of soil moisture. The inversion process contains uncertainties related to radiometer measurements, forward model assumptions and ancillary data sources. This study focuses on the uncertainties that depend on the seasonal evolution of the surface conditions and vegetation. This study compares the SMAP and core validation site (CVS) soil moisture values over a period of three years to extract the evolution of performance metrics over time. The analysis showed that most CVS that include managed agriculture exhibit significant time-dependent seasonal bias. This bias was linked to seasonal temperature cycle, which is a proxy to several features that can cause seasonally dependent errors in the SMAP product.

core validation site↗
NASA NTRSDate not supplied

Preliminary Steps in Developing Rapid Aero Modeling Technology

The Rapid Aero Modeling (RAM) approach is a method to efficiently and automatically obtain aerodynamic models during testing, significantly saving time and resources. Motivation for this technology results from demand for experimental efficiency and model fidelity that has increased with growing aircraft complexity and aerodynamic nonlinearities. These issues are typical in the responses presented by a class of vehicles categorized as Urban Air Mobility aircraft where many features from both airplane and rotorcraft are present. For UAM configurations, with typically many more factors than conventional aircraft, traditional test methods can lead to increased costs and missed interactions. RAM guides the test to obtain high-fidelity, statistically rigorous aircraft models, and the approach is applicable to computational, ground, or flight-test experiments. It combines concepts from Design of Experiment theory and Aircraft System Identification theory that allow the user the freedom to choose, in advance of the test, a specific level of fidelity typically, in terms of prediction error. RAM only collects data required to meet the user-specified fidelity and fidelity is only limited by the facility and test article capabilities. An initial wind tunnel test to support development of RAM was conducted to assess potential metrics, algorithms, and procedures. This paper presents results from initial tests for the development of RAM technology and highlights some of the unique features of RAM applied toa conventional configuration during a ground-based, static, wind-tunnel test.

Patrick C Murphy↗
NASA NTRS2023

Comment on “Advanced Testing of Low, Medium, and High ECS CMIP6 GCM Simulations Versus ERA5-T2m” by N. Scafetta (2022)

Scafetta (2022, https://doi.org/10.1029/2022gl097716) purports to test Coupled Model Intercomparison Project Phase 6 (CMIP6) climate models through a comparison of temperature changes over three decades. Unfortunately, the paper contains numerous conceptual and statistical errors that undermine all of the conclusions. First, no uncertainty is given for the observational temperature difference, making it impossible to assess compatibility with any model result. Second, the CMIP6 data are the ensemble means for each model, but the metric being tested is sensitive to the internal variability and so the full ensemble for each model must be used. When this is corrected, the conclusion that “all models with ECS > 3.0°C overestimate the observed global surface warming” is not sustained. Third, the statistical test in Section 2 would reject all models even in a perfect model setup given sufficient ensemble members, thus the second conclusion “that spatial t-statistics rejects the data-model agreement” is also not sustainable.

CMIP6↗
NASA NTRSDate not supplied

Evaluating Delta-V Dispersions Using Linear Covariance Techniques with Applications to Rendezvous and Docking

One of the attractive advantages of using linear covariance analysis (LinCov) is that it can accurately produce both navigation errors and trajectory dispersions in a fraction of the time comparable to the standard Monte Carlo analysis approach that is traditionally adopted. It also has the capability of generating delta-v dispersions which is perhaps the most influential performance metric since many aspects of a mission design revolve around the anticipated propellant usage. Accurately capturing this key performance criteria using linear covariance techniques has several subtle caveats raising reasonable doubts and a cautious sense of skepticism. This paper outlines multiple approaches of modeling delta-v usage in LinCov, highlights their limitations and advantages, and ultimately compares their corresponding results to the actual Monte Carlo performance. The theory is also extended to reliably account for an arbitrary thruster configuration and the resultant propellant consumptions due to both translational burns and attitude maneuvers. This provides near instant insight into the impacts of thruster layout design and different vehicle mass properties. These derived techniques are applied and verified using an NRHO rendezvous and docking scenario consistent with upcoming NASA Artemis missions for Orion.

Linear Covariance Analysis↗
DOE OSTI2023

Stochastic Framework for Optimal Control of Planetary Reentry Trajectories Under Multilevel Uncertainties

We present a novel stochastic optimal control framework that accounts for various types of uncertainties, with application to reentry trajectory planning. The formulation of the optimal trajectory control problem is presented in the context of an indirect method where a functional objective associated with the terminal vehicle speed is to be minimized. Uncertain input parameters in the optimal trajectory control model, including aerodynamic parameters and initial and terminal conditions, are modeled as aleatory random variables, while the statistical parameters of these aleatory distributions are themselves random variables. The parametric and model uncertainties are simultaneously propagated through an extended polynomial chaos expansion (EPCE) formalism. Several metrics are described to evaluate response statistics and presented as insightful tools for robust decision making. Specifically, the response probability density function (PDF) reflecting influence of both epistemic and aleatory uncertainties is obtained. By sampling over the random variables representing model error, an ensemble of response PDFs is generated and the associated failure probability is estimated as a random variable with its own polynomial chaos expansion. Besides, the sensitivity index functions of response PDF with respect to the statistical parameters are evaluated. Coupling parametric and model uncertainties within the EPCE framework leads to a robust and efficient paradigm for multilevel uncertainty propagation and PDF characterization in general optimal control problems.

Engineering↗
NASA NTRS2017

Validation of OCO-2 and ACOS-GOSAT using HIPPO and TCCON

Consistent validation of satellite CO2 estimates is a prerequisite for using multiple satellite CO2measurements for joint flux inversion and establishing a long-term atmospheric CO2 data record. Wevalidate recent satellite observation of OCO-2 v7 and ACOS-GOSAT v7.3 using similar analysis as previouswork (Kulawik et al. (2016) and Frankenberg et al. (2106)) through comparisons to the HIAPER Pole-to-Pole Observations (HIPPO) and the Total Carbon Column Observing Network (TCCON) to estimate biasesand errors affecting the understanding of carbon cycle science. CarbonTracker RT is also compared tothe validation data, and additionally used to evaluate the mismatch between the HIPPO observationtimeframe and the OCO-2 record, which are offset by 3-7 years. Some key metrics that are validatedinclude the seasonal cycle phase and amplitude, latitudinal gradient by season, regional biases, anderrors with respect to averaging.

Kulawik, Susan S.↗
NASA NTRSDate not supplied

The Impact of Controlled Rest on Neurobehavioral Outcomes at Top-of-Descent

INTRODUCTION: Long and irregular working hours can lead to fatigue in aviation operations. In some regions, a short nap taken on the flight deck (known as controlled rest) can be used as a countermeasure to unexpected in-flight sleepiness. We aimed to investigate the impact of taking controlled rest on neurobehavioral measures at top-of-descent. METHODS: Data from 120 long-haul (> 6 h flight duration), unaugmented flights were analyzed (n = 31 pilots). Pilots wore actigraphs and completed sleep logs before and during trips. At pre-flight and top-of-descent, pilots completed a 5-minute psychomotor vigilance task (PVT) and Karolinska Sleepiness Scale (KSS). A series of mixed-effects models were conducted to assess the impact of controlled rest on outcome measures at top-of-descent. Sleep duration in the prior 48 hours, timing of the flight, and pre-flight scores for each measure were included as covariates. RESULTS: Due to missing data, complete data from 76 flights (n = 28 participants) were available in the models examining the PVT metrics, and data from 83 flights (n = 29 participants) were available in the analyses of the KSS. Pilots who took controlled rest had faster response speeds at top-of-descent (p = .03, η2p = 0.07; estimated marginal mean [EMM] = 4.19, standard error [SE] = 0.07, 95% CI [4.08, 4.29]) than those who did not take controlled rest (EMM = 4.00, SE = 0.05, 95% CI [3.86, 4.14]). There were no differences by controlled rest status for KSS scores and PVT lapses (p values > .05, η2p values ≤ 0.01). DISCUSSION: Our results suggest that taking controlled rest may improve vigilant attention at critical phases of flight and, thus, may be a useful fatigue management tool during unaugmented flights. Further research is necessary to determine the impact of factors on the decision to take controlled rest (e.g., airline culture, personal preference) and how the controlled rest policy is applied in practice.

aviation↗
DOE OSTI2023

Stochastic multiscale modeling for quantifying statistical and model errors with application to composite materials

This paper provides a coherent and efficient computational framework for stochastic multiscale analysis of material systems in the presence of parametric uncertainties and modeling errors. Uncertainty in those model parameters that are not deduced as upscaled quantities is attributed to an uncertainty “germ”. While such parameters can appear at any scale, they are predominant at the finest analysis scale. Additional uncertainties stemming from statistical estimation, attributed to lack of data and model error, are associated with each submodel contributing to the multiscale system. Here, a robust and efficient framework based on a generalized extended polynomial chaos expansion (gEPCE) is proposed to simultaneously propagate all these uncertainties in order to provide a probabilistic representation of specific quantities of interest (QoI). We characterize the full probability distribution of the QoI and the uncertainty in the failure probability pertaining to its tails. By combining gEPCE with kernel density estimation (KDE) and directional derivatives, we construct sensitivity measures that connect these statistical metrics of QoI to the various sources of uncertainty to assess their individual and combined impacts. An illustrative problem featuring three-point bending of a composite beam is investigated to demonstrate the presented approach.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Three dimensional cluster analysis for atom probe tomography using Ripley’s K-function and machine learning

The size and structure of spatial molecular and atomic clustering can significantly impact material properties and is therefore important to accurately quantify. Ripley’s K-function (K(r)), a measure of spatial correlation, can be used to perform such quantification when the material system of interest can be represented as a marked point pattern. This work demonstrates how machine learning models based on K (r)-derived metrics can accurately estimate cluster size and intra-cluster density in simulated three dimensional (3D) point patterns containing spherical clusters of varying size; over 90% of model estimates for cluster size and intra-cluster density fall within 11% and 18% error of the true values, respectively. These K (r)-based size and density estimates are then applied to an experimental APT reconstruction to characterize MgZn clusters in a 7000 series aluminum alloy. Here we find that the estimates are more accurate, consistent, and robust to user interaction than estimates from the popular maximum separation algorithm. Using K (r) and machine learning to measure clustering is an accurate and repeatable way to quantify this important material attribute.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Intracardiac Electrical Imaging using the 12-lead ECG: A Machine Learning Approach using Synthetic Data

Current state-of-the-art techniques for non-invasive imaging of cardiac electrical phenomena require voltage recordings from dozens of different torso locations and anatomical models built from expensive medical diagnostic imaging procedures. Here this study aimed to assess if recent machine learning advances could alternatively reconstruct electroanatomical maps at clinically relevant resolutions using only the standard 12-lead electrocardiogram (ECG) as input. To that end, a computational study was conducted to generate a dataset of over 16000 detailed cardiac simulations, which was then used to train neural network (NN) architectures designed to exploit both spatial and temporal correlations in the ECG signal. Analysis over a validation set showed average errors in activation map reconstruction below 1.7 msec over 75 intracardiac locations. Furthermore, phenotypical patterns of activation and the morphology of the activation potential were correctly reconstructed. The approach offers opportunities to stratify patients non-invasively, both retrospectively and prospectively, using metrics otherwise only available through invasive clinical procedures.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1996

Advanced Air Traffic Management Research (Human Factors and Automation): NASA Research Initiatives in Human-Centered Automation Design in Airspace Management

NASA has initiated a significant thrust of research and development focused on providing the flight crew and air traffic managers automation aids to increase capacity in en route and terminal area operations through the use of flexible, more fuel-efficient routing, while improving the level of safety in commercial carrier operations. In that system development, definition of cognitive requirements for integrated multi-operator dynamic aiding systems is fundamental. The core processes of control and the distribution of decision making in that control are undergoing extensive analysis. From our perspective, the human operators and the procedures by which they interact are the fundamental determinants of the safe, efficient, and flexible operation of the system. In that perspective, we have begun to explore what our experience has taught will be the most challenging aspects of designing and integrating human-centered automation in the advanced system. We have performed a full mission simulation looking at the role shift to self-separation on board the aircraft with the rules of the air guiding behavior and the provision of a cockpit display of traffic information and an on-board traffic alert system that seamlessly integrates into the TCAS operations. We have performed and initial investigation of the operational impact of "Dynamic Density" metrics on controller relinquishing and reestablishing full separation authority. (We follow the assumption that responsibility at all times resides with the controller.) This presentation will describe those efforts as well as describe the process by which we will guide the development of error tolerant systems that are sensitive to shifts in operator work load levels and dynamic shifts in the operating point of air traffic management.

Corker, Kevin M.↗