Results for “Error Metrics” — Page 8

NASA NTRS2012

A Systems Modeling Approach for Risk Management of Command File Errors

The main cause of commanding errors is often (but not always) due to procedures. Either lack of maturity in the processes, incompleteness of requirements or lack of compliance to these procedures. Other causes of commanding errors include lack of understanding of system states, inadequate communication, and making hasty changes in standard procedures in response to an unexpected event. In general, it's important to look at the big picture prior to making corrective actions. In the case of errors traced back to procedures, considering the reliability of the process as a metric during its' design may help to reduce risk. This metric is obtained by using data from Nuclear Industry regarding human reliability. A structured method for the collection of anomaly data will help the operator think systematically about the anomaly and facilitate risk management. Formal models can be used for risk based design and risk management. A generic set of models can be customized for a broad range of missions.

probabilistic risk↗

NASA NTRS2017

Information Theoretic Evaluation of Satellite Soil Moisture Retrievals

Microwave radiometry has a long legacy of providing estimates of remotely sensed near surfacesoil moisture measurements over continental and global scales. A consistent assessment of theerrors and uncertainties associated with these retrievals is important for their effective utilization in modeling, data assimilation and end-use application environments. This article presents an evaluationof soil moisture retrieval products from AMSR-E, ASCAT, SMOS, AMSR2 and SMAPinstruments using information theory-based metrics. These metrics rely on time series analysis ofsoil moisture retrievals for estimating the measurement error, level of randomness (entropy) andregularity (complexity) of the data. The results of the study indicate that the measurement errors inthe remote sensing retrievals are significantly larger than that of the ground soil moisture measurements.The SMAP retrievals, on the other hand, were found to have reduced errors (comparable to Preprint submitted to Remote Sensing of Environment October 1, 2017those of in-situ datasets), particularly over areas with moderate vegetation. The SMAP retrievals also demonstrate high information content relative to other retrieval products, with higher levelsof complexity and reduced entropy. Finally, a joint evaluation of the entropy and complexity ofremotely sensed soil moisture products indicates that the information content of the AMSR-E, ASCAT,SMOS and AMSR2 retrievals is low, whereas SMAP retrievals show better performance. The use of information theoretic assessments is effective in quantifying the required levels of improvements needed in the remote sensing soil moisture retrievals to enhance their utility and information content.

Information theory↗

NASA NTRS2000

Performance and Optimization Metrics for Interferometric Radar

This paper provides a set of analytic equations that can be used to estimate system performance or to assist in the design of a system in that individual sources of height error can be balanced to minimize cost and maximize efficient use of resources.

optimization metrics interferometric radar interfe↗

NASA NTRS2007

Safety Performance of Airborne Separation: Preliminary Baseline Testing

The Safety Performance of Airborne Separation (SPAS) study is a suite of Monte Carlo simulation experiments designed to analyze and quantify safety behavior of airborne separation. This paper presents results of preliminary baseline testing. The preliminary baseline scenario is designed to be very challenging, consisting of randomized routes in generic high-density airspace in which all aircraft are constrained to the same flight level. Sustained traffic density is varied from approximately 3 to 15 aircraft per 10,000 square miles, approximating up to about 5 times today s traffic density in a typical sector. Research at high traffic densities and at multiple flight levels are planned within the next two years. Basic safety metrics for aircraft separation are collected and analyzed. During the progression of experiments, various errors, uncertainties, delays, and other variables potentially impacting system safety will be incrementally introduced to analyze the effect on safety of the individual factors as well as their interaction and collective effect. In this paper we report the results of the first experiment that addresses the preliminary baseline condition tested over a range of traffic densities. Early results at five times the typical traffic density in today s NAS indicate that, under the assumptions of this study, airborne separation can be safely performed. In addition, we report on initial observations from an exploration of four additional factors tested at a single traffic density: broadcast surveillance signal interference, extent of intent sharing, pilot delay, and wind prediction error.

Consiglio, Maria C.↗

NASA NTRS2014

The Uncertainty of Biomass Estimates from Modeled ICESat-2 Returns Across a Boreal Forest Gradient

The Forest Light (FLIGHT) radiative transfer model was used to examine the uncertainty of vegetation structure measurements from NASA's planned ICESat-2 photon counting light detection and ranging (LiDAR) instrument across a synthetic Larix forest gradient in the taiga-tundra ecotone. The simulations demonstrate how measurements from the planned spaceborne mission, which differ from those of previous LiDAR systems, may perform across a boreal forest to non-forest structure gradient in globally important ecological region of northern Siberia. We used a modified version of FLIGHT to simulate the acquisition parameters of ICESat-2. Modeled returns were analyzed from collections of sequential footprints along LiDAR tracks (link-scales) of lengths ranging from 20 m-90 m. These link-scales traversed synthetic forest stands that were initialized with parameters drawn from field surveys in Siberian Larix forests. LiDAR returns from vegetation were compiled for 100 simulated LiDAR collections for each 10 Mg · ha(exp -1) interval in the 0-100 Mg · ha(exp -1) above-ground biomass density (AGB) forest gradient. Canopy height metrics were computed and AGB was inferred from empirical models. The root mean square error (RMSE) and RMSE uncertainty associated with the distribution of inferred AGB within each AGB interval across the gradient was examined. Simulation results of the bright daylight and low vegetation reflectivity conditions for collecting photon counting LiDAR with no topographic relief show that 1-2 photons are returned for 79%-88% of LiDAR shots. Signal photons account for approximately 67% of all LiDAR returns, while approximately 50% of shots result in 1 signal photon returned. The proportion of these signal photon returns do not differ significantly (p greater than 0.05) for AGB intervals greater than 20 Mg · ha(exp -1). The 50m link-scale approximates the finest horizontal resolution (length) at which photon counting LiDAR collection provides strong model fits and minimizes forest structure uncertainty in the synthetic Larix stands. At this link-scale AGB greater than 20 Mg · ha(exp -1) has AGB error from 20-50% at the 95% confidence level. These results suggest that the theoretical sensitivity of ICESat-2 photon counting LiDAR measurements alone lack the ability to consistently discern differences in inferred AGB at 10 Mg · ha(exp -1) intervals in sparse forests characteristic of the taiga-tundra ecotone.

LiDAR↗

NASA NTRS2015

Effects of Resolution and Spectral Nudging in Simulation the Effects of Wintertime Atmospheric River Landfalls in the Western US

Landfalling atmospheric rivers (ARs) play a crucial role in the climate of the US Pacific coast region as they are frequently related with heavy precipitation and flash flooding events. Thus, the capability of climate models to accurately simulate AR landfalls and their key hydrologic effects is an important practical concern for WUS, from flood forecasting to future water resources projections. In order to examine the effects of model configuration, including the resolution and spectral nudging, in simulating the climatology of key weather events in the conterminous US, a NASA team has performed a hindcast experiment using the GEOS5 global and the NU-WRF regional models for Nov 1999 - Oct 2010. This study examines the skill of these hindcasts, with different models and their configurations, in simulating key footprints of landfalling ARs in the WUS region. Using an AR-landfall chronology based on the vertically-integrated water vapor flux calculated from the MERRA2 reanalysis, we have analyzed the observed and simulated precipitation and temperature anomalies associated with wintertime AR landfalls along the US Pacific coast. Model skill is measured using metrics including regional means, a skill score based on correlations and mean-square errors, and Taylor diagrams in four WUS Bukovsky regions. Results show that the AR-related anomalies of precipitation is more reliable than of surface temperatures. Model skill also varies according to regions. The AR temperature anomalies are well simulated in most of the WUS region except PNW. For precipitation, simulations with finer spatial resolution tend to generate larger spatial variability and agree better with the PRISM data in most regions. Such a resolution dependence of spatial variability is not found for temperatures; e.g., the MERRA2 reanalysis often outperforms, with similar spatial variability and higher pattern correlations with the PRISM data, finer-resolution NU-WRF runs in simulating temperature variations within subregions. Results from this study will be summarized to assist future (regional) climate experiments for climate change impact assessments and developing adaptationmitigation strategies, the key elements of the National Climate Assessment.

Simulation↗

NASA NTRS2015

Enabling Performance-Based Navigation Arrivals: Development and Simulation Testing of the Terminal Sequencing and Spacing System

NASA has developed an advanced arrival management capability for terminal controllers, known as Terminal Sequencing and Spacing (TSS). TSS increases use of performance-based navigation (PBN) arrival procedures during periods of high traffic demand. It extends two Federal Aviation Administration's operational systems with terminal metering and controller spacing tools. Sixteen high-fidelity human-in-the-loop simulations, involving more than five hundred hours of evaluation time, were conducted to mature TSS from proof-of- concept design to fully functional prototype. These simulations modeled arrival procedures at several U.S. airports, incorporated a broad range of traffic demand profiles and wind conditions, and used controllers with extensive operational experience. Two fundamental metrics are evaluated for these simulations: PBN Success Rate and Inter-Arrival Spacing Error. The PBN Success Rate shows a definitive trend when TSS is used. It increases from 42 percent for today's operations to 68 percent for terminal metering only and 92 percent for terminal metering with controller-managed spacing tools. Meanwhile, the Inter-Arrival Spacing Error improves 25 to 35 percent when TSS is used compared to not used. The TSS technology was transferred to the FAA and, and it is targeted for deployment to several busy airports in the U.S. starting in 2018.

terminal metering↗

NASA NTRSDate not supplied

Evaluating the Impact of Geostationary Sounders in the Context of International Coordination

The proposed NOAA/NASA Geostationary eXtended Observations (GeoXO) program plans to include a hyperspectral infrared (IR) sounder (GXS) on its central satellite, joining international counterparts. Ahead of launch, the NASA Global Modeling and Assimilation Office (GMAO) assessed the potential effectiveness of GXS both as a lone sounder in a GEO orbit and as part of a global ring of such instruments. Using an observing system simulation experiment (OSSE) framework from a global numerical weather prediction (NWP) perspective, the ability of GXS and the global ring to improve weather prediction of thermodynamic variables was assessed both globally and regionally. GXS dominated regional analysis and forecast improvements and contributed significantly to global increases in forecast skill. However, more sustained global improvements on the order of 4 days rely on international partnerships. Over CONUS, the FSOI metric showed the GXS observations provide the strongest radiance impact on the moist energy error norm reduction. Additionally, GXS shows the capability to improve hurricane forecast track errors, resulting in improved forecast warnings. Overall, the persistent atmospheric profile information from GXS over much of the western hemisphere provide an opportunity to improve the representation of weather systems and their forecasts.

Erica L. McGrath-Spangler↗

NASA NTRSDate not supplied

Evaluating the Impact of Geostationary Sounders in the Context of International Coordination

The proposed NOAA/NASA Geostationary eXtended Observations (GeoXO) program plans to include a hyperspectral infrared (IR) sounder (GXS) on its central satellite, joining international counterparts. Ahead of launch, the NASA Global Modeling and Assimilation Office (GMAO) assessed the potential effectiveness of GXS both as a lone sounder in a GEO orbit and as part of a global ring of such instruments. Using an observing system simulation experiment (OSSE) framework from a global numerical weather prediction (NWP) perspective, the ability of GXS and the global ring to improve weather prediction of thermodynamic variables was assessed both globally and regionally. GXS dominated regional analysis and forecast improvements and contributed significantly to global increases in forecast skill. However, more sustained global improvements on the order of 4 days rely on international partnerships. Over CONUS, the FSOI metric showed the GXS observations provide the strongest radiance impact on the moist energy error norm reduction. Additionally, GXS shows the capability to improve hurricane forecast track errors, resulting in improved forecast warnings. Overall, the persistent atmospheric profile information from GXS over much of the western hemisphere provide an opportunity to improve the representation of weather systems and their forecasts.

Erica McGrath-Spangler↗

NASA NTRS2015

Sensor Selection for Aircraft Engine Performance Estimation and Gas Path Fault Diagnostics

This paper presents analytical techniques for aiding system designers in making aircraft engine health management sensor selection decisions. The presented techniques, which are based on linear estimation and probability theory, are tailored for gas turbine engine performance estimation and gas path fault diagnostics applications. They enable quantification of the performance estimation and diagnostic accuracy offered by different candidate sensor suites. For performance estimation, sensor selection metrics are presented for two types of estimators including a Kalman filter and a maximum a posteriori estimator. For each type of performance estimator, sensor selection is based on minimizing the theoretical sum of squared estimation errors in health parameters representing performance deterioration in the major rotating modules of the engine. For gas path fault diagnostics, the sensor selection metric is set up to maximize correct classification rate for a diagnostic strategy that performs fault classification by identifying the fault type that most closely matches the observed measurement signature in a weighted least squares sense. Results from the application of the sensor selection metrics to a linear engine model are presented and discussed. Given a baseline sensor suite and a candidate list of optional sensors, an exhaustive search is performed to determine the optimal sensor suites for performance estimation and fault diagnostics. For any given sensor suite, Monte Carlo simulation results are found to exhibit good agreement with theoretical predictions of estimation and diagnostic accuracies.

Gas Turbine Engines↗

NASA NTRS2015

Sensor Selection for Aircraft Engine Performance Estimation and Gas Path Fault Diagnostics

This paper presents analytical techniques for aiding system designers in making aircraft engine health management sensor selection decisions. The presented techniques, which are based on linear estimation and probability theory, are tailored for gas turbine engine performance estimation and gas path fault diagnostics applications. They enable quantification of the performance estimation and diagnostic accuracy offered by different candidate sensor suites. For performance estimation, sensor selection metrics are presented for two types of estimators including a Kalman filter and a maximum a posteriori estimator. For each type of performance estimator, sensor selection is based on minimizing the theoretical sum of squared estimation errors in health parameters representing performance deterioration in the major rotating modules of the engine. For gas path fault diagnostics, the sensor selection metric is set up to maximize correct classification rate for a diagnostic strategy that performs fault classification by identifying the fault type that most closely matches the observed measurement signature in a weighted least squares sense. Results from the application of the sensor selection metrics to a linear engine model are presented and discussed. Given a baseline sensor suite and a candidate list of optional sensors, an exhaustive search is performed to determine the optimal sensor suites for performance estimation and fault diagnostics. For any given sensor suite, Monte Carlo simulation results are found to exhibit good agreement with theoretical predictions of estimation and diagnostic accuracies.

Gas Turbine Engines↗

NASA NTRS2016

Sensor Selection for Aircraft Engine Performance Estimation and Gas Path Fault Diagnostics

This paper presents analytical techniques for aiding system designers in making aircraft engine health management sensor selection decisions. The presented techniques, which are based on linear estimation and probability theory, are tailored for gas turbine engine performance estimation and gas path fault diagnostics applications. They enable quantification of the performance estimation and diagnostic accuracy offered by different candidate sensor suites. For performance estimation, sensor selection metrics are presented for two types of estimators including a Kalman filter and a maximum a posteriori estimator. For each type of performance estimator, sensor selection is based on minimizing the theoretical sum of squared estimation errors in health parameters representing performance deterioration in the major rotating modules of the engine. For gas path fault diagnostics, the sensor selection metric is set up to maximize correct classification rate for a diagnostic strategy that performs fault classification by identifying the fault type that most closely matches the observed measurement signature in a weighted least squares sense. Results from the application of the sensor selection metrics to a linear engine model are presented and discussed. Given a baseline sensor suite and a candidate list of optional sensors, an exhaustive search is performed to determine the optimal sensor suites for performance estimation and fault diagnostics. For any given sensor suite, Monte Carlo simulation results are found to exhibit good agreement with theoretical predictions of estimation and diagnostic accuracies.

Diagnosis↗

NASA NTRS2012

Uncertainty Quantification for Polynomial Systems via Bernstein Expansions

This paper presents a unifying framework to uncertainty quantification for systems having polynomial response metrics that depend on both aleatory and epistemic uncertainties. The approach proposed, which is based on the Bernstein expansions of polynomials, enables bounding the range of moments and failure probabilities of response metrics as well as finding supersets of the extreme epistemic realizations where the limits of such ranges occur. These bounds and supersets, whose analytical structure renders them free of approximation error, can be made arbitrarily tight with additional computational effort. Furthermore, this framework enables determining the importance of particular uncertain parameters according to the extent to which they affect the first two moments of response metrics and failure probabilities. This analysis enables determining the parameters that should be considered uncertain as well as those that can be assumed to be constants without incurring significant error. The analytical nature of the approach eliminates the numerical error that characterizes the sampling-based techniques commonly used to propagate aleatory uncertainties as well as the possibility of under predicting the range of the statistic of interest that may result from searching for the best- and worstcase epistemic values via nonlinear optimization or sampling.

Crespo, Luis G.↗

NASA NTRS1989

High rate concatenated coding systems using bandwidth efficient trellis inner codes

High-rate concatenated coding systems with bandwidth-efficient trellis inner codes and Reed-Solomon (RS) outer codes are investigated for application in high-speed satellite communication systems. Two concatenated coding schemes are proposed. In one the inner code is decoded with soft-decision Viterbi decoding, and the outer RS code performs error-correction-only decoding (decoding without side information). In the other, the inner code is decoded with a modified Viterbi algorithm, which produces reliability information along with the decoded output. In this algorithm, path metrics are used to estimate the entire information sequence, whereas branch metrics are used to provide reliability information on the decoded sequence. This information is used to erase unreliable bits in the decoded output. An errors-and-erasures RS decoder is then used for the outer code. The two schemes have been proposed for high-speed data communication on NASA satellite channels. The rates considered are at least double those used in current NASA systems, and the results indicate that high system reliability can still be achieved.

Deng, Robert H.↗

NASA NTRS1986

Error control techniques for satellite and space communications

High rate concatenated coding systems with trellis inner codes and Reed-Solomon (RS) outer codes for application in satellite communication systems are considered. Two types of inner codes are studied: high rate punctured binary convolutional codes which result in overall effective information rates between 1/2 and 1 bit per channel use; and bandwidth efficient signal space trellis codes which can achieve overall effective information rates greater than 1 bit per channel use. Channel capacity calculations with and without side information performed for the concatenated coding system. Concatenated coding schemes are investigated. In Scheme 1, the inner code is decoded with the Viterbi algorithm and the outer RS code performs error-correction only (decoding without side information). In scheme 2, the inner code is decoded with a modified Viterbi algorithm which produces reliability information along with the decoded output. In this algorithm, path metrics are used to estimate the entire information sequence, while branch metrics are used to provide the reliability information on the decoded sequence. This information is used to erase unreliable bits in the decoded output. An errors-and-erasures RS decoder is then used for the outer code. These two schemes are proposed for use on NASA satellite channels. Results indicate that high system reliability can be achieved with little or no bandwidth expansion.

Costello, D. J., Jr.↗

NASA NTRS1983

Metric analysis and data validation across FORTRAN projects

The desire to predict the effort in developing or explaining the quality of software has led to the proposal of several metrics. As a step toward validating these metrics, the Software Engineering Laboratory (SEL) has analyzed the software science metrics, cyclomatic complexity, and various standard program measures for their relation to effort (including design through acceptance testing), development errors (both discrete and weighted according to the amount of time to locate and fix), and one another. The data investigated are collected from a project FORTRAN environment and examined across several projects at once, within individual projects and by reporting accuracy checks demonstrating the need to validate a database. When the data comes from individual programmers or certain validated projects, the metrics' correlations with actual effort seem to be strongest. For modules developed entirely by individual programmers, the validity ratios induce a statistically significant ordering of several of the metrics' correlations. When comparing the strongest correlations, neither software science's E metric cyclomatic complexity not source lines of code appears to relate convincingly better with effort than the others.

Basili, Victor R.↗

NASA NTRS2019

Quantifying the Storm Time Thermospheric Neutral Density Variations Using Model and Observations

Accurate determination of thermospheric neutral density holds crucial importance for satellited rag calculations. The problem is twofold and involves the correct estimation of the quiet time climatology and storm time variations. In this work, neutral density estimations from two empirical and threephysics based models of the ionosphere thermosphere are compared with the neutral densities along the Challenging MicroSatellite Payload satellite track for six geomagnetic storms. Storm time variations are extracted from neutral density by (1) subtracting the mean difference between model and observation (bias),(2) setting climatological variations to zero, and (3) multiplying model data with the quiet time ratio between the model and observation. Several metrics are employed to evaluate the model performances. We find that the removal of bias or climatology reveals actual performance of the model in simulating the storm time variations. When bias is removed, depending on event and model, storm time errors in neutral density can decrease by an amount of 113% or can increase by an amount of 12% with respect to error in models with quiet time bias. It is shown that using only average and maximum values of neutral density to determine the model performances can be misleading since a model can estimate the averages fairly well but may not capture the maximum value or vice versa. Since each of the metrics used for determining model performances provides different aspects of the error, among these, we suggest employing mean absolute error, prediction efficiency, and normalized root mean square error together as a standard set ofmetrics for the neutral density.

Eyiguler, E. Ceren Kalafatoglu↗

NASA NTRS1985

Performance of a normalized energy metric without jammer state information for an FH/MFSK system in worst case partial band jamming

For a frequency-hopped noncoherent MFSK communication system without jammer state information (JSI) in a worst case partial band jamming environment, it is well known that the use of a conventional unquantized metric results in very poor performance. In this paper, a 'normalized' unquantized energy metric is suggested for such a system. It is shown that with this metric, one can save 2-3 dB in required signal energy over the system with hard decision metric without JSI for the same desired performance. When this very robust metric is compared to the conventional unquantized energy metric with JSI, the loss in required signal energy is shown to be small. Thus, the use of this normalized metric provides performance comparable to systems for which JSI is known. Cutoff rate and bit error rate with dual-k coding are used for the performance measures.

Lee, P. J.↗