Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Error Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

Metric analysis of minitrack optical and interferometer data

The network analysis program (NAP-2), which has the capability of simultaneously solving for orbits and tracking station error model terms, was, after several modifications, used in the calibration of minitrack stations using minitrack satellite measurements (self-calibration). Several support programs were written to aid in this task. A simultaneous four-arc solution was obtained. A comparison with optically determined arcs for the same time spans showed rms position differences of 67m, 86m, 124m and 168m for the 4 arcs considered. An apparatus incorporating a diffraction grating was designed and successfully used to measure the drift rate of the Fort Myers MOTS camera drive.

Brown, D. C.↗

Normalized Point Source Sensitivity for Off-Axis Optical Performance Evaluation of the Thirty Meter Telescope

The Normalized Point Source Sensitivity (PSSN) has previously been defined and analyzed as an On-Axis seeing-limited telescope performance metric. In this paper, we expand the scope of the PSSN definition to include Off-Axis field of view (FoV) points and apply this generalized metric for performance evaluation of the Thirty Meter Telescope (TMT). We first propose various possible choices for the PSSN definition and select one as our baseline. We show that our baseline metric has useful properties including the multiplicative feature even when considering Off-Axis FoV points, which has proven to be useful for optimizing the telescope error budget. Various TMT optical errors are considered for the performance evaluation including segment alignment and phasing, segment surface figures, temperature, and gravity, whose On-Axis PSSN values have previously been published by our group.

optical performance metric↗

The Impact of Truth Surrogate Variance on Quality Assessment/Assurance in Wind Tunnel Testing

Minimum data volume requirements for wind tunnel testing are reviewed and shown to depend on error tolerance, response model complexity, random error variance in the measurement environment, and maximum acceptable levels of inference error risk. Distinctions are made between such related concepts as quality assurance and quality assessment in response surface modeling, as well as between precision and accuracy. Earlier research on the scaling of wind tunnel tests is extended to account for variance in the truth surrogates used at confirmation sites in the design space to validate proposed response models. A model adequacy metric is presented that represents the fraction of the design space within which model predictions can be expected to satisfy prescribed quality specifications. The impact of inference error on the assessment of response model residuals is reviewed. The number of sites where reasonably well-fitted response models actually predict inadequately is shown to be considerably less than the number of sites where residuals are out of tolerance. The significance of such inference error effects on common response model assessment strategies is examined.

DeLoach, Richard↗

The impact of mismatch on the performance of coded narrow-band FM with limiter/discriminator detection

An examination of the impact of mismatch on the performance of convolutionally encoded/Viterbi decoded narrow-band FM with limiter/discriminator detection is presented. Attention was given to the potential gain available by the combination of this type of system in terms of hard and soft decision decoding. Soft decision decoding was demonstrated to offer only approximately 0.3 dB better performance than hard decision coding. It was also shown, through a technique involving the number of clicks occurring in each detection interval, that both soft and hard decision decoding bit error probability performance could be improved. It is concluded that the mismatch between the coding channel and the decoding metric of the Viterbi algorithm is responsible for reducing the difference between hard and soft decoding metrics.

Simon, M. K.↗

A Near-Term Concept for Trajectory Based Operations with Air/Ground Data Link Communication

An operating concept and required system components for trajectory-based operations with air/ground data link for today's en route and transition airspace is proposed. Controllers are fully responsible for separation as they are today, and no new aircraft equipage is required. Trajectory automation computes integrated solutions to problems like metering, weather avoidance, traffic conflicts and the desire to find and fly more time/fuel efficient flight trajectories. A common ground-based system supports all levels of aircraft equipage and performance including those equipped and not equipped for data link. User interface functions for the radar controller's display make trajectory-based clearance advisories easy to visualize, modify if necessary, and implement. Laboratory simulations (without human operators) were conducted to test integrated operation of selected system components with uncertainty modeling. Results are based on 102 hours of Fort Worth Center traffic recordings involving over 37,000 individual flights. The presence of uncertainty had a marginal effect (5%) on minimum-delay conflict resolution performance, and windfavorable routes had no effect on detection and resolution metrics. Flight plan amendments and clearances were substantially reduced compared to today s operations. Top-of-descent prediction errors are the largest cause of failure indicating that better descent predictions are needed to reliably achieve fuel-efficient descent profiles in medium to heavy traffic. Improved conflict detections for climbing flights could enable substantially more continuous climbs to cruise altitude. Unlike today s Conflict Alert, tactical automation must alert when an altitude amendment is entered, but before the aircraft starts the maneuver. In every other failure case tactical automation prevented losses of separation. A real-time prototype trajectory trajectory-automation system is running now and could be made ready for operational testing at an en route Center in 1-2 years.

McNally, David↗

Long Term Performance Metrics of the GD SDR on the SCaN Testbed: The First Year on the ISS

The General Dynamics (GD) S-Band software defined radio (SDR) in the Space Communications and Navigation (SCaN) Testbed on the International Space Station (ISS) provides experimenters an opportunity to develop and demonstrate experimental waveforms in space. The SCaN Testbed was installed on the ISS in August of 2012. After installation, the initial checkout and commissioning phases were completed and experimental operations commenced. One goal of the SCaN Testbed is to collect long term performance metrics for SDRs operating in space in order to demonstrate long term reliability. These metrics include the time the SDR powered on, the time the power amplifier (PA) is powered on, temperature trends, error detection and correction (EDAC) behavior, and waveform operational usage time. This paper describes the performance of the GD SDR over the first year of operations on the ISS.

radio telemetry↗

On the Assessment of Acoustic Scattering and Shielding by Time Domain Boundary Integral Equation Solutions

Based on the time domain boundary integral equation formulation of the linear convective wave equation, a computational tool dubbed Time Domain Fast Acoustic Scattering Toolkit (TD-FAST) has recently been under development. The time domain approach has a distinct advantage that the solutions at all frequencies are obtained in a single computation. In this paper, the formulation of the integral equation, as well as its stabilization by the Burton-Miller type reformulation, is extended to cases of a constant mean flow in an arbitrary direction. In addition, a "Source Surface" is also introduced in the formulation that can be employed to encapsulate regions of noise sources and to facilitate coupling with CFD simulations. This is particularly useful for applications where the noise sources are not easily described by analytical source terms. Numerical examples are presented to assess the accuracy of the formulation, including a computation of noise shielding by a thin barrier motivated by recent Historical Baseline F31A31 open rotor noise shielding experiments. Furthermore, spatial resolution requirements of the time domain boundary element method are also assessed using point per wavelength metrics. It is found that, using only constant basis functions and high-order quadrature for surface integration, relative errors of less than 2% may be obtained when the surface spatial resolution is 5 points-per-wavelength (PPW) or 25 points-per-wavelength squared (PPW2).

Hu, Fang Q.↗

Uncertainty Estimates for Sonic-Boom Pressure Signatures and Loudness Carpets

A non-intrusive uncertainty quantification method is applied to computational analysis of supersonic, low-boom aircraft. The mean and standard deviation statistics of the pressure waveforms and loudness metrics are evaluated through use of numerical quadrature. The probability density function (p.d.f.) of these outputs is evaluated via kernel density estimation. The simulations use an inviscid, embedded-boundary Cartesian-mesh flow solver in the nearfield combined with an augmented Burgers’ equation solver for propagation in the farfield. The results show that the p.d.f. of the waveform is bimodal at shocks, which makes the mean and standard deviation statistics inappropriate. Despite this limitation, we show that the moment statistics can provide effective assessment of discrepancies when comparing with experimental data. This is demonstrated by presenting uncertainty analysis of a wind-tunnel test and showing that we significantly improve the predictions when we include the test uncertainties in the simulation. Normal distributions are obtained for the ground signature and loudness metrics, which is primarily due to the careful shaping of the low-boom waveform. Separation of variables and error control are used to reduce computational cost. We demonstrate that this is an efficient approach in the sense of balancing numerical errors in the statistics quadrature with discretization errors in the solvers.

ARMD↗

Global-scale Evaluation of SMAP, SMOS and ASCAT Soil Moisture Products Using Triple Collocation

Global-scale surface soil moisture products are currently available from multiple remote sensing platforms. Footprint-scale assessments of these products are generally restricted to limited number of densely-instrumented validation sites. However, by taking active and passive soil moisture products together with a third independent soil moisture estimates via land surface modeling, triple collocation (TC) can be applied to estimate the correlation metric of satellite soil moisture products (versus an unknown ground truth) over a quasi-global domain. Here, an assessment of Soil Moisture Active Passive (SMAP), Soil Moisture Ocean Salinity (SMOS) and Advanced SCATterometer (ASCAT) surface soil moisture retrievals via TC is presented. Considering the potential violation of TC error assumptions, the impact of active-passive and satellite-model error cross correlations on the TC-derived inter-comparison results is examined at in situ sites using quadruple collocation analysis. In addition, confidence intervals for the TC-estimated correlation metric are constructed from moving-block bootstrap sampling designed to preserve the temporal persistence of the original (unevenly-sampled) soil moisture time-series. This study is the first to apply TC to obtain a robust global-scale cross-assessment of SMAP, SMOS and ASCAT soil moisture retrieval accuracy in terms of anomaly temporal correlation. Our results confirm the overall advantage of SMAP (with a global average anomaly correlation of 0.76) over SMOS (0.66) and ASCAT (0.63) that has been established in several recent regional, ground-based studies. SMAP is also the best-performing product over the majority of applicable land pixels (52%), although SMOS and ASCAT each shows advantage in distinct geographic regions.

global-scale↗

A Quantum Volume Metric for Qudit Quantum Computers

Qudit-based quantum computers, such as the high-Q superconducting radio frequency resonator cavities being pursued by the SQMS Center hosted at Fermilab, may provide computational advantages relative to qubit-based machines for some problems. However, the novel system’s hardware and software will present different errors that must be understood and mitigated, along with other unforeseen inefficiencies across the entire stack. It is essential to benchmark these qudit platforms against existing systems. We propose such a benchmark by generalizing the qubit-based quantum volume metric to qudit devices. In particular, we show that the performance depends on the available gateset used to compile unitary operations, as well as realistic noise sources.

Quantum Computing↗

A Statistical Theory of Bidirectionality

Original concepts related to the quantification and assessment of bidirectionality in strain-gage balances were introduced by Ulbrich in 2012. These concepts are extended here in three ways: 1) the metric originally proposed by Ulbrich is normalized, 2) a categorical variable is introduced in the regression analysis to account for load polarity, and 3) the uncertainty in both normalized and non-normalized bidirectionality metrics is quantified. These extensions are applied to four representative balances to assess the bidirectionality characteristics of each. The paper is tutorial in nature, featuring reviews of certain elements of regression and formal inference. Principal findings are that bidirectionality appears to be a common characteristic of most balance outputs and that unless it is taken into account, it is likely to consume the entire error budget of a typical balance calibration experiment. Data volume and correlation among calibration loads are shown to have a significant impact on the precision with which bidirectionality metrics can be assessed.

DeLoach, Richard↗

A Comparison of Two Balance Calibration Model Building Methods

Simulated strain-gage balance calibration data is used to compare the accuracy of two balance calibration model building methods for different noise environments and calibration experiment designs. The first building method obtains a math model for the analysis of balance calibration data after applying a candidate math model search algorithm to the calibration data set. The second building method uses stepwise regression analysis in order to construct a model for the analysis. Four balance calibration data sets were simulated in order to compare the accuracy of the two math model building methods. The simulated data sets were prepared using the traditional One Factor At a Time (OFAT) technique and the Modern Design of Experiments (MDOE) approach. Random and systematic errors were introduced in the simulated calibration data sets in order to study their influence on the math model building methods. Residuals of the fitted calibration responses and other statistical metrics were compared in order to evaluate the calibration models developed with different combinations of noise environment, experiment design, and model building method. Overall, predicted math models and residuals of both math model building methods show very good agreement. Significant differences in model quality were attributable to noise environment, experiment design, and their interaction. Generally, the addition of systematic error significantly degraded the quality of calibration models developed from OFAT data by either method, but MDOE experiment designs were more robust with respect to the introduction of a systematic component of the unexplained variance.

DeLoach, Richard↗

Sequential Syndrome Decoding of Convolutional Codes

The algebraic structure of convolutional codes are reviewed and sequential syndrome decoding is applied to those codes. These concepts are then used to realize by example actual sequential decoding, using the stack algorithm. The Fano metric for use in sequential decoding is modified so that it can be utilized to sequentially find the minimum weight error sequence.

Reed, I. S.↗

Advanced Navigation Strategies For Asteroid Sample Return Missions

Flyby and rendezvous missions to asteroids have been accomplished using navigation techniques derived from experience gained in planetary exploration. This paper presents analysis of advanced navigation techniques required to meet unique challenges for precision navigation to acquire a sample from an asteroid and return it to Earth. These techniques rely on tracking data types such as spacecraft-based laser ranging and optical landmark tracking in addition to the traditional Earth-based Deep Space Network radio metric tracking. A systematic study of navigation strategy, including the navigation event timeline and reduction in spacecraft-asteroid relative errors, has been performed using simulation and covariance analysis on a representative mission.

Getzandanner, K.↗

Improving Computational Efficiency of Prognostics Algorithms in Resource-Constrained Settings

In engineering and aerospace applications, it is vital to operational success to have insight into the expected performance and health of physical systems. The field of prognostics and health management provides quantitative methods for monitoring, predicting, and managing system health. Prognostics algorithms can be employed to assess the current state of a system, propagate the system throughout time, and predict potential anomalies or failures that may occur. While they can provide accurate prediction results, effective prognostics algorithms can be challenging to use in resource-constrained settings due to computational limitations and high computational latency, leading to obsolete predictions. Thus, computationally efficient and accurate algorithms are necessary for future remaining useful life predictions. In this work, we implement new algorithmic approaches for prediction, quantitatively compare them via a battery degradation use-case, and provide recommendations of potential improvements to a prognostics framework. One approach to prediction is through sampling, whereby the current state of a physical system is sampled many times and each sample is propagated forward until failure is reached, resulting in a distribution of failure values. To improve the efficiency of this process, we implemented five new algorithmic approaches to prediction, including three distinct sampling methods (standard Monte Carlo, Quasi-Monte Carlo, and Latin Hypercube Sampling), a variable time step algorithm, and a variable sample size algorithm. To compare the algorithms, we employ a variety of metrics designed specifically to analyze both computational efficiency and model accuracy. Our metrics include accuracy to compare the average predicted value to ground truth, mean absolute deviation to illustrate dispersion, specific percentile error to describe accuracy within a user-defined risk tolerance, and code run-time. To quantitatively analyze our results, we employ a use-case of degradation of a Lithium-ion battery. We use an electrochemistry-based model to describe the current health state of the battery, and implement our prediction algorithms to propagate forward in time until end-of-discharge (EOD) is reached. Notably, through this work it was found that none of our sampling approaches had a significant impact on computational efficiency or model accuracy in predicting EOD of the battery. We find that while the sampling methods are unique, the distributions they generate are similar, ultimately producing final predictions that are nearly identical. In exploring the effect of the time step within the prediction algorithm, we found that prediction accuracy was highly dependent on the time step used, and that implementing a variable time step within a particular prediction may provide an increase in computational efficiency while also maintaining prediction accuracy. Finally, implementing a variable sample size also affected prediction, and our results show that tuning both the magnitude and timing of the sample size adjustment can result in improved computation speed and maintained prediction accuracy. Taken together, our findings highlight the challenge of performing prognostics in resource-constrained settings, and illustrate the potential of developing new prediction algorithms to improve computational efficiency.

prognostics↗

Airborne Radar for sUAS Sense and Avoid

A primary challenge for the safe integration of small UAS operations into the National Airspace System (NAS) is traffic deconfliction, both from manned and unmanned aircraft. The UAS Traffic Management (UTM) project being conducted at the National Aeronautics and Space Administration (NASA) considers a layered approach to separation provision, ranging from segregation of operations through airspace volumes (geofences) to autonomous sense and avoid (SAA) technologies for higher risk, densely occupied airspace. Cooperative SAA systems, such as Automatic Dependent Surveillance-Broadcast (ADS-B) and/or vehicle-to-vehicle communication systems provide significant additional risk mitigation but they fail to adequately mitigate collision risks for non-cooperative (non-transponder equipped) airborne aircraft. The RAAVIN (Radar on Autonomous Aircraft to Verify ICAROUS Navigation) flight test being conducted by NASA and the Mid-Atlantic Aviation Partnership (MAAP) was designed to investigate the applicability and performance of a prototype, commercially available sUAS radar to detect and track non-cooperative airborne traffic, both manned and unmanned. The radar selected for this research was a Frequency Modulated Continuous Wave (FMCW) radar with 120 degree azimuth and 80 degree elevation field of view operating at 24.55GHz center frequency with a 200 MHz bandwidth. The radar transmits 2 watts of power thru a Metamaterial Electronically Scanning Array antenna in horizontal polarization. When the radar is transmitting, personnel must be at least 1 meter away from the active array to limit nonionizing radiation exposure. The radar physical dimensions are 18.7cm by 12.1cm by 4.1cm and it weighs less than 820 grams making it well suited for installation on small UASs. The onboard, SAA capability, known as ICAROUS, (Independent Configurable Architecture for Reliable Operations of Unmanned Systems), developed by NASA to support sUAS operations, will provide autonomous guidance using the traffic radar tracks from the onboard radar. The RAAVIN set of studies will be conducted in three phases. The first phase included outdoor, ground-based radar evaluations performed at the Virginia Tech’s Kentland Farm testing range in Blacksburg, VA. The test was designed to measure how well the radar could detect and track a small UAS flying in the radar’s field of view. The radar was used to monitor 5 test flights consisting of outbound, inbound and crossing routes at different ranges and altitudes. The UAS flown during the ground test was the Inspire 2, a quad copter weighing less than 4250 grams (10 pounds) at maximum payload. The radar was set up to scan and track targets over its full azimuthal field of view from 0 to 40 degrees in elevation. The radar was configured to eliminate tracks generated from any targets located beyond 2000 meters from the radar and moving at velocities under 1.45 meters per second. For subsequent phases of the study the radar will be integrated with a sUAS platform to evaluate its performance in flight for SAA applications ranging from sUAS to manned GA aircraft detections and tracking. Preliminary data analysis from the first outdoor ground tests showed the radar performed well at tracking the vehicle as it flew outbound and repeatedly maintained a track out to 1000 meters (maximum 1387 meters) until the vehicle slowed to a stop to reverse direction to fly inbound. As the Inspire flew inbound tracks from beyond 800 meters, a reacquisition time delay was consistently observed between when the Inspire exceeds a speed of 1.45 meters per second and when the radar indicated an inbound target was present and maintained its track. The time delay varied between 6 seconds to over 37 seconds for the inbound flights examined, and typically resulted in about a 200 meter closure distance before the Inspire track was maintained. The radar performed well at both acquiring and tracking the vehicle as it flew crossing routes out past 400 meters across the azimuthal field of view. The radar and ICAROUS software will be integrated and flown on a BFD-1400-SE8-E UAS during the next phase of the RAAVIN project. The main goal at the conclusion of this effort is to determine if this radar technology can reliably support minimum requirements for SAA applications of sUAS. In particular, the study will measure the range of vehicle detections, lateral and vertical angular errors, false and missed/late detections, and estimated distance at closest point of approach after an avoidance maneuver is executed. This last metric is directly impacted by sensor performance and indicates its suitability for the task.

Szatkowski, George N.↗

Verification of Viscous Goal-Based Anisotropic Mesh Adaptation

Adaptive unstructured mesh techniques have a limited, but growing impact on production analysis workflows where the control of discretization error is critical to obtaining reliable simulation results. Recent progress has matured a number of independent implementations of flow solvers, anisotropic metric construction methods, and anisotropic mesh adaptation mechanics. A key ingredient for the broader acceptance of unstructured mesh adaptation is the verification of these implementations. Anisotropic metric construction methods are evaluated with analytically defined primal fields and the corresponding entropy variables as adjoint fields. This allows the comparison of different metric formulations and different implementations of the same formulation without the complications of a flow and adjoint solution method. The convergence of the output associated with the entropy variable adjoint is studied for mesh adaptation to these fields and a manufactured solution. Mesh adapted drag output is studied for two simple wings in compressible laminar flow to show fine-mesh convergence of multiple metric construction methods to less than a single drag count. The documentation of these verification exercises helps to prepare these goal-based methods for routine use in more complex simulations for production workflows.

mesh adaptation↗

Validation of Satellite-Based Precipitation Products from TRMM to GPM

The global precipitation measurement mission (GPM) has been in operation for sevenyears and continues to provide a vast quantity of global precipitation data at finer temporospatialresolutions with improved accuracy and coverage. GPM’s signature algorithm, the integratedmultisatellite retrievals for GPM (IMERG) is a next-generation of precipitation product expectedfor wide variety of research and operational applications. This study evaluates the latest version(V06B) of IMERG and its predecessor, the tropical rainfall measuring mission (TRMM) multisatelliteprecipitation (TMPA) 3B42 (V7) using ground-based and gauge-corrected multiradar multisensorsystem (MRMS) precipitation products over the conterminous United States (CONUS). The spatialdistributions of all products are analyzed. The error characteristics are further examined for 3B42 andIMERG in winter and summer by an error decomposition approach, which partitions total bias intohit bias, biases due to missed precipitation and false precipitation. The volumetric and categoricalstatistical metrics are used to quantitatively evaluate the performance of the two satellite-basedproducts. All products show a similar precipitation climatology with some regional differences.The two satellite-based products perform better in the eastern CONUS than in the mountainousWestern CONUS. The evaluation demonstrates the clear improvement in IMERG precipitationproduct in comparison with its predecessor 3B42, especially in reducing missed precipitation inwinter and summer, and hit bias in winter, resulting in better performance in capturing lighter andheavier precipitation.

Jianxin Wang↗