Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “error metric”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20

The Impact of Truth Surrogate Variance on Quality Assessment/Assurance in Wind Tunnel Testing

Minimum data volume requirements for wind tunnel testing are reviewed and shown to depend on error tolerance, response model complexity, random error variance in the measurement environment, and maximum acceptable levels of inference error risk. Distinctions are made between such related concepts as quality assurance and quality assessment in response surface modeling, as well as between precision and accuracy. Earlier research on the scaling of wind tunnel tests is extended to account for variance in the truth surrogates used at confirmation sites in the design space to validate proposed response models. A model adequacy metric is presented that represents the fraction of the design space within which model predictions can be expected to satisfy prescribed quality specifications. The impact of inference error on the assessment of response model residuals is reviewed. The number of sites where reasonably well-fitted response models actually predict inadequately is shown to be considerably less than the number of sites where residuals are out of tolerance. The significance of such inference error effects on common response model assessment strategies is examined.

DeLoach, Richard↗

Downscaling atmospheric chemistry simulations with physically consistent deep learning

Abstract. Recent advances in deep convolutional neural network (CNN)-based super resolution can be used to downscale atmospheric chemistry simulations with substantially higher accuracy than conventional downscaling methods. This work both demonstrates the downscaling capabilities of modern CNN-based single image super resolution and video super-resolution schemes and develops modifications to these schemes to ensure they are appropriate for use with physical science data. The CNN-based video super-resolution schemes in particular incur only 39 % to 54 % of the grid-cell-level error of interpolation schemes and generate outputs with extremely realistic small-scale variability based on multiple perceptual quality metrics while performing a large (8×10) increase in resolution in the spatial dimensions. Methods are introduced to strictly enforce physical conservation laws within CNNs, perform large and asymmetric resolution changes between common model grid resolutions, account for non-uniform grid-cell areas, super-resolve lognormally distributed datasets, and leverage additional inputs such as high-resolution climatologies and model state variables. High-resolution chemistry simulations are critical for modeling regional air quality and for understanding future climate, and CNN-based downscaling has the potential to generate these high-resolution simulations and ensembles at a fraction of the computational cost.

54 ENVIRONMENTAL SCIENCES↗

The impact of mismatch on the performance of coded narrow-band FM with limiter/discriminator detection

An examination of the impact of mismatch on the performance of convolutionally encoded/Viterbi decoded narrow-band FM with limiter/discriminator detection is presented. Attention was given to the potential gain available by the combination of this type of system in terms of hard and soft decision decoding. Soft decision decoding was demonstrated to offer only approximately 0.3 dB better performance than hard decision coding. It was also shown, through a technique involving the number of clicks occurring in each detection interval, that both soft and hard decision decoding bit error probability performance could be improved. It is concluded that the mismatch between the coding channel and the decoding metric of the Viterbi algorithm is responsible for reducing the difference between hard and soft decoding metrics.

Simon, M. K.↗

A Near-Term Concept for Trajectory Based Operations with Air/Ground Data Link Communication

An operating concept and required system components for trajectory-based operations with air/ground data link for today's en route and transition airspace is proposed. Controllers are fully responsible for separation as they are today, and no new aircraft equipage is required. Trajectory automation computes integrated solutions to problems like metering, weather avoidance, traffic conflicts and the desire to find and fly more time/fuel efficient flight trajectories. A common ground-based system supports all levels of aircraft equipage and performance including those equipped and not equipped for data link. User interface functions for the radar controller's display make trajectory-based clearance advisories easy to visualize, modify if necessary, and implement. Laboratory simulations (without human operators) were conducted to test integrated operation of selected system components with uncertainty modeling. Results are based on 102 hours of Fort Worth Center traffic recordings involving over 37,000 individual flights. The presence of uncertainty had a marginal effect (5%) on minimum-delay conflict resolution performance, and windfavorable routes had no effect on detection and resolution metrics. Flight plan amendments and clearances were substantially reduced compared to today s operations. Top-of-descent prediction errors are the largest cause of failure indicating that better descent predictions are needed to reliably achieve fuel-efficient descent profiles in medium to heavy traffic. Improved conflict detections for climbing flights could enable substantially more continuous climbs to cruise altitude. Unlike today s Conflict Alert, tactical automation must alert when an altitude amendment is entered, but before the aircraft starts the maneuver. In every other failure case tactical automation prevented losses of separation. A real-time prototype trajectory trajectory-automation system is running now and could be made ready for operational testing at an en route Center in 1-2 years.

McNally, David↗

Long Term Performance Metrics of the GD SDR on the SCaN Testbed: The First Year on the ISS

The General Dynamics (GD) S-Band software defined radio (SDR) in the Space Communications and Navigation (SCaN) Testbed on the International Space Station (ISS) provides experimenters an opportunity to develop and demonstrate experimental waveforms in space. The SCaN Testbed was installed on the ISS in August of 2012. After installation, the initial checkout and commissioning phases were completed and experimental operations commenced. One goal of the SCaN Testbed is to collect long term performance metrics for SDRs operating in space in order to demonstrate long term reliability. These metrics include the time the SDR powered on, the time the power amplifier (PA) is powered on, temperature trends, error detection and correction (EDAC) behavior, and waveform operational usage time. This paper describes the performance of the GD SDR over the first year of operations on the ISS.

radio telemetry↗

On the Assessment of Acoustic Scattering and Shielding by Time Domain Boundary Integral Equation Solutions

Based on the time domain boundary integral equation formulation of the linear convective wave equation, a computational tool dubbed Time Domain Fast Acoustic Scattering Toolkit (TD-FAST) has recently been under development. The time domain approach has a distinct advantage that the solutions at all frequencies are obtained in a single computation. In this paper, the formulation of the integral equation, as well as its stabilization by the Burton-Miller type reformulation, is extended to cases of a constant mean flow in an arbitrary direction. In addition, a "Source Surface" is also introduced in the formulation that can be employed to encapsulate regions of noise sources and to facilitate coupling with CFD simulations. This is particularly useful for applications where the noise sources are not easily described by analytical source terms. Numerical examples are presented to assess the accuracy of the formulation, including a computation of noise shielding by a thin barrier motivated by recent Historical Baseline F31A31 open rotor noise shielding experiments. Furthermore, spatial resolution requirements of the time domain boundary element method are also assessed using point per wavelength metrics. It is found that, using only constant basis functions and high-order quadrature for surface integration, relative errors of less than 2% may be obtained when the surface spatial resolution is 5 points-per-wavelength (PPW) or 25 points-per-wavelength squared (PPW2).

Hu, Fang Q.↗

Uncertainty Estimates for Sonic-Boom Pressure Signatures and Loudness Carpets

A non-intrusive uncertainty quantification method is applied to computational analysis of supersonic, low-boom aircraft. The mean and standard deviation statistics of the pressure waveforms and loudness metrics are evaluated through use of numerical quadrature. The probability density function (p.d.f.) of these outputs is evaluated via kernel density estimation. The simulations use an inviscid, embedded-boundary Cartesian-mesh flow solver in the nearfield combined with an augmented Burgers’ equation solver for propagation in the farfield. The results show that the p.d.f. of the waveform is bimodal at shocks, which makes the mean and standard deviation statistics inappropriate. Despite this limitation, we show that the moment statistics can provide effective assessment of discrepancies when comparing with experimental data. This is demonstrated by presenting uncertainty analysis of a wind-tunnel test and showing that we significantly improve the predictions when we include the test uncertainties in the simulation. Normal distributions are obtained for the ground signature and loudness metrics, which is primarily due to the careful shaping of the low-boom waveform. Separation of variables and error control are used to reduce computational cost. We demonstrate that this is an efficient approach in the sense of balancing numerical errors in the statistics quadrature with discretization errors in the solvers.

ARMD↗

Global-scale Evaluation of SMAP, SMOS and ASCAT Soil Moisture Products Using Triple Collocation

Global-scale surface soil moisture products are currently available from multiple remote sensing platforms. Footprint-scale assessments of these products are generally restricted to limited number of densely-instrumented validation sites. However, by taking active and passive soil moisture products together with a third independent soil moisture estimates via land surface modeling, triple collocation (TC) can be applied to estimate the correlation metric of satellite soil moisture products (versus an unknown ground truth) over a quasi-global domain. Here, an assessment of Soil Moisture Active Passive (SMAP), Soil Moisture Ocean Salinity (SMOS) and Advanced SCATterometer (ASCAT) surface soil moisture retrievals via TC is presented. Considering the potential violation of TC error assumptions, the impact of active-passive and satellite-model error cross correlations on the TC-derived inter-comparison results is examined at in situ sites using quadruple collocation analysis. In addition, confidence intervals for the TC-estimated correlation metric are constructed from moving-block bootstrap sampling designed to preserve the temporal persistence of the original (unevenly-sampled) soil moisture time-series. This study is the first to apply TC to obtain a robust global-scale cross-assessment of SMAP, SMOS and ASCAT soil moisture retrieval accuracy in terms of anomaly temporal correlation. Our results confirm the overall advantage of SMAP (with a global average anomaly correlation of 0.76) over SMOS (0.66) and ASCAT (0.63) that has been established in several recent regional, ground-based studies. SMAP is also the best-performing product over the majority of applicable land pixels (52%), although SMOS and ASCAT each shows advantage in distinct geographic regions.

global-scale↗

A Quantum Volume Metric for Qudit Quantum Computers

Qudit-based quantum computers, such as the high-Q superconducting radio frequency resonator cavities being pursued by the SQMS Center hosted at Fermilab, may provide computational advantages relative to qubit-based machines for some problems. However, the novel system’s hardware and software will present different errors that must be understood and mitigated, along with other unforeseen inefficiencies across the entire stack. It is essential to benchmark these qudit platforms against existing systems. We propose such a benchmark by generalizing the qubit-based quantum volume metric to qudit devices. In particular, we show that the performance depends on the available gateset used to compile unitary operations, as well as realistic noise sources.

Quantum Computing↗

A Statistical Theory of Bidirectionality

Original concepts related to the quantification and assessment of bidirectionality in strain-gage balances were introduced by Ulbrich in 2012. These concepts are extended here in three ways: 1) the metric originally proposed by Ulbrich is normalized, 2) a categorical variable is introduced in the regression analysis to account for load polarity, and 3) the uncertainty in both normalized and non-normalized bidirectionality metrics is quantified. These extensions are applied to four representative balances to assess the bidirectionality characteristics of each. The paper is tutorial in nature, featuring reviews of certain elements of regression and formal inference. Principal findings are that bidirectionality appears to be a common characteristic of most balance outputs and that unless it is taken into account, it is likely to consume the entire error budget of a typical balance calibration experiment. Data volume and correlation among calibration loads are shown to have a significant impact on the precision with which bidirectionality metrics can be assessed.

DeLoach, Richard↗

A Comparison of Two Balance Calibration Model Building Methods

Simulated strain-gage balance calibration data is used to compare the accuracy of two balance calibration model building methods for different noise environments and calibration experiment designs. The first building method obtains a math model for the analysis of balance calibration data after applying a candidate math model search algorithm to the calibration data set. The second building method uses stepwise regression analysis in order to construct a model for the analysis. Four balance calibration data sets were simulated in order to compare the accuracy of the two math model building methods. The simulated data sets were prepared using the traditional One Factor At a Time (OFAT) technique and the Modern Design of Experiments (MDOE) approach. Random and systematic errors were introduced in the simulated calibration data sets in order to study their influence on the math model building methods. Residuals of the fitted calibration responses and other statistical metrics were compared in order to evaluate the calibration models developed with different combinations of noise environment, experiment design, and model building method. Overall, predicted math models and residuals of both math model building methods show very good agreement. Significant differences in model quality were attributable to noise environment, experiment design, and their interaction. Generally, the addition of systematic error significantly degraded the quality of calibration models developed from OFAT data by either method, but MDOE experiment designs were more robust with respect to the introduction of a systematic component of the unexplained variance.

DeLoach, Richard↗

Performance and power modeling and prediction using MuMMI and 10 machine learning methods

Energy-efficient scientific applications require insight into how high performance computing system features impact the applications' power and performance. This insight can result from the development of performance and power models. Here, in this article, we use the modeling and prediction tool MuMMI (Multiple Metrics Modeling Infrastructure) and 10 machine learning methods to model and predict performance and power consumption and compare their prediction error rates. We use an algorithm-based fault-tolerant linear algebra code and a multilevel checkpointing fault-tolerant heat distribution code to conduct our modeling and prediction study on the Cray XC40 Theta and IBM BG/Q Mira at Argonne National Laboratory and the Intel Haswell cluster Shepard at Sandia National Laboratories. Our experimental results show that the prediction error rates in performance and power using MuMMI are less than 10% for most cases. By utilizing the models for runtime, node power, CPU power, and memory power, we identify the most significant performance counters for potential application optimizations, and we predict theoretical outcomes of the optimizations. Based on two collected datasets, we analyze and compare the prediction accuracy in performance and power consumption using MuMMI and 10 machine learning methods.

97 MATHEMATICS AND COMPUTING↗

Perspective: Reproducible Coherence Characterization of Superconducting Quantum Devices

As the field of superconducting quantum computing approaches maturity, optimization of single-device performance is proving to be a promising avenue towards large-scale quantum computers. However, this optimization is possible only if performance metrics can be accurately compared among measurements, devices, and laboratories. Currently such comparisons are inaccurate or impossible due to understudied errors from a plethora of sources. In this Perspective, we outline the current state of error analysis for qubits and resonators in superconducting quantum circuits, and discuss what future investigations are required before superconducting quantum device optimization can be realized.

McRae, Corey H.↗

Evaluating Probabilistic Deep Learning Methods for Uncertainty Quantification of Precipitation Bias Correction

Climate models often exhibit biases in their precipitation predictions, particularly underestimating high-intensity events and overestimating low precipitation. Deep learning approaches offer promising solutions, but their epistemic uncertainty associated with a deep learning–based bias correction method has not previously been quantified for reliable downstream climate impact studies. While methods for capturing the epistemic uncertainty in deep learning frameworks exist, there is currently no consensus on the best method. In this work, we compare three uncertainty quantification (UQ) methods—Deep Ensembles (DEns), Monte Carlo Dropout (MCD), and Flipout—by assessing the reliability of their uncertainty estimates using standard measures such as sharpness and calibration. These UQ methods are applied to an existing deep learning precipitation bias correction model known as UFNet: a coupled U-Net and fully connected neural network. The methods utilized to assess the models’ uncertainties are 1) calibration, which ensures that the expected probabilities of the model align with reality and 2) sharpness, which is a measure of the precision of the model’s probabilistic predictions. Of the three UQ methods evaluated, the DEns and MCD methods demonstrated the best-calibrated performance (expected calibration error of 0.36 and 0.35, respectively), compared to Flipout (0.58). In contrast, Flipout had the sharpest predictions and the highest metric performance in bias correcting precipitation—especially for higher-order moments such as kurtosis with a spatial correlation of 72% compared to 32% and 55% spatial correlation for DEns and MCD, respectively. Of the three UQ methods, MCD was found to be the most suitable method for UQ purposes based on its calibration, sharpness, and computational requirements.

Bayesian methods↗

Sequential Syndrome Decoding of Convolutional Codes

The algebraic structure of convolutional codes are reviewed and sequential syndrome decoding is applied to those codes. These concepts are then used to realize by example actual sequential decoding, using the stack algorithm. The Fano metric for use in sequential decoding is modified so that it can be utilized to sequentially find the minimum weight error sequence.

Reed, I. S.↗

Advanced Navigation Strategies For Asteroid Sample Return Missions

Flyby and rendezvous missions to asteroids have been accomplished using navigation techniques derived from experience gained in planetary exploration. This paper presents analysis of advanced navigation techniques required to meet unique challenges for precision navigation to acquire a sample from an asteroid and return it to Earth. These techniques rely on tracking data types such as spacecraft-based laser ranging and optical landmark tracking in addition to the traditional Earth-based Deep Space Network radio metric tracking. A systematic study of navigation strategy, including the navigation event timeline and reduction in spacecraft-asteroid relative errors, has been performed using simulation and covariance analysis on a representative mission.

Getzandanner, K.↗

Improving Computational Efficiency of Prognostics Algorithms in Resource-Constrained Settings

In engineering and aerospace applications, it is vital to operational success to have insight into the expected performance and health of physical systems. The field of prognostics and health management provides quantitative methods for monitoring, predicting, and managing system health. Prognostics algorithms can be employed to assess the current state of a system, propagate the system throughout time, and predict potential anomalies or failures that may occur. While they can provide accurate prediction results, effective prognostics algorithms can be challenging to use in resource-constrained settings due to computational limitations and high computational latency, leading to obsolete predictions. Thus, computationally efficient and accurate algorithms are necessary for future remaining useful life predictions. In this work, we implement new algorithmic approaches for prediction, quantitatively compare them via a battery degradation use-case, and provide recommendations of potential improvements to a prognostics framework. One approach to prediction is through sampling, whereby the current state of a physical system is sampled many times and each sample is propagated forward until failure is reached, resulting in a distribution of failure values. To improve the efficiency of this process, we implemented five new algorithmic approaches to prediction, including three distinct sampling methods (standard Monte Carlo, Quasi-Monte Carlo, and Latin Hypercube Sampling), a variable time step algorithm, and a variable sample size algorithm. To compare the algorithms, we employ a variety of metrics designed specifically to analyze both computational efficiency and model accuracy. Our metrics include accuracy to compare the average predicted value to ground truth, mean absolute deviation to illustrate dispersion, specific percentile error to describe accuracy within a user-defined risk tolerance, and code run-time. To quantitatively analyze our results, we employ a use-case of degradation of a Lithium-ion battery. We use an electrochemistry-based model to describe the current health state of the battery, and implement our prediction algorithms to propagate forward in time until end-of-discharge (EOD) is reached. Notably, through this work it was found that none of our sampling approaches had a significant impact on computational efficiency or model accuracy in predicting EOD of the battery. We find that while the sampling methods are unique, the distributions they generate are similar, ultimately producing final predictions that are nearly identical. In exploring the effect of the time step within the prediction algorithm, we found that prediction accuracy was highly dependent on the time step used, and that implementing a variable time step within a particular prediction may provide an increase in computational efficiency while also maintaining prediction accuracy. Finally, implementing a variable sample size also affected prediction, and our results show that tuning both the magnitude and timing of the sample size adjustment can result in improved computation speed and maintained prediction accuracy. Taken together, our findings highlight the challenge of performing prognostics in resource-constrained settings, and illustrate the potential of developing new prediction algorithms to improve computational efficiency.

prognostics↗

QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation

The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, and so on, trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, comprising 25K circuit evaluations. In addition we also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system.

97 MATHEMATICS AND COMPUTING↗