Results for “Error Metrics” — Page 4

NASA NTRSDate not supplied

Effects of Communication Modality on Pilot-Controller Coordination during a Simulated m:N Operation

The last decade or so has seen growing interest in new control paradigms and concepts of operation for uncrewed aircraft systems (UAS) in which multiple aircraft are piloted remotely by a single or relatively small number of people. Referred to as “one-to-many” and “many-to- many” (alternatively, “multi-operator, multi-vehicle”)—and frequently expressed as the corresponding ratios, 1:N and m:N—such novel configurations of aircraft and the people who manage them are seen as critical to the path to future operations involving UAS. Examples of industry domains interested in these control paradigms are small package delivery services utilizing small UAS and passenger-carrying, short-range “Urban Air Mobility” (UAM) operations. Stakeholders in such operations have identified communication and coordination of flight activity with air traffic controllers (ATC) as a barrier to operations. In contrast to present-day flight operations, in which a pilot communicates with one ATC on one radio frequency for one aircraft, multi-vehicle operations potentially entail a significant increase in pilot task load for management of comms. New concepts, such as UAS Service Suppliers (USSs) and Providers of Services to UAM (PSUs), have been proposed to address the known bottleneck for Air Traffic Management (ATM) presented by multi-vehicle operations. While progress has been steadily made over years developing USSs and PSUs, it is generally expected that initial UAM operations will rely on traditional voice-over-radio communication with ATC for purposes of ATM. The current study was a human-in-the-loop simulation that had participants, each possessing a Private Pilot License, act as the ground-based pilot-in- command for multiple vehicles in a hypothetical UAM service in the San Francisco Bay Area. The experiment utilized a 2-by-3, within-subjects design in which the pilot’s Vehicle Load (4 vs. 12) and Comm System (Voice, Datalink, and a Hybrid) were manipulated. The task given to pilots was to use the Comm System to coordinate flight activity for all aircraft with appropriate controllers, having to obtain departure and arrival clearances at “vertiport” facilities and transition clearances for any intermediate airspaces along the route. Pilots were additionally responsible for compliance with vectoring instructions issued by ATC. Subjective workload questionnaires (NASA-TLX) were administered following each experimental trial. Screen recordings of the pilot’s Ground Control Station (GCS) and audio recordings of trials were subsequently coded to obtain performance metrics: response times and error rates. Presented in this paper are results related to pilot responses to vectoring instructions issued by ATC. Workload was found to be significantly higher in the 12-Vehicle condition compared to the 4-Vehicle condition, nearly maxing out the NASA-TLX overall workload scale. There was no significant difference made by the Comm System on workload ratings. Pilots’ response times to communications were fastest in the Voice condition, although overall “service time” for compliance was shorter in Datalink and Hybrid conditions in most cases. Errors by pilots were frequent in both Vehicle Load conditions, most perniciously when using the Voice system. The results of this study suggest tradeoffs in advantages and disadvantages of the three comm systems. Recommendations for communication system design are provided taking the tradeoffs into account.

Garrett G Sadler↗

NASA NTRSDate not supplied

Effects of Communication Modality on Pilot-Controller Coordination during a Simulated m:N Operation

The last decade or so has seen growing interest in new control paradigms and concepts of operation for uncrewed aircraft systems (UAS) in which multiple aircraft are piloted remotely by a single or relatively small number of people. Referred to as “one-to-many” and “many-to- many” (alternatively, “multi-operator, multi-vehicle”)—and frequently expressed as the corresponding ratios, 1:N and m:N—such novel configurations of aircraft and the people who manage them are seen as critical to the path to future operations involving UAS. Examples of industry domains interested in these control paradigms are small package delivery services utilizing small UAS and passenger-carrying, short-range “Urban Air Mobility” (UAM) operations. Stakeholders in such operations have identified communication and coordination of flight activity with air traffic controllers (ATC) as a barrier to operations. In contrast to present-day flight operations, in which a pilot communicates with one ATC on one radio frequency for one aircraft, multi-vehicle operations potentially entail a significant increase in pilot task load for management of comms. New concepts, such as UAS Service Suppliers (USSs) and Providers of Services to UAM (PSUs), have been proposed to address the known bottleneck for Air Traffic Management (ATM) presented by multi-vehicle operations. While progress has been steadily made over years developing USSs and PSUs, it is generally expected that initial UAM operations will rely on traditional voice-over-radio communication with ATC for purposes of ATM. The current study was a human-in-the-loop simulation that had participants, each possessing a Private Pilot License, act as the ground-based pilot-in- command for multiple vehicles in a hypothetical UAM service in the San Francisco Bay Area. The experiment utilized a 2-by-3, within-subjects design in which the pilot’s Vehicle Load (4 vs. 12) and Comm System (Voice, Datalink, and a Hybrid) were manipulated. The task given to pilots was to use the Comm System to coordinate flight activity for all aircraft with appropriate controllers, having to obtain departure and arrival clearances at “vertiport” facilities and transition clearances for any intermediate airspaces along the route. Pilots were additionally responsible for compliance with vectoring instructions issued by ATC. Subjective workload questionnaires (NASA-TLX) were administered following each experimental trial. Screen recordings of the pilot’s Ground Control Station (GCS) and audio recordings of trials were subsequently coded to obtain performance metrics: response times and error rates. Presented in this paper are results related to pilot responses to vectoring instructions issued by ATC. Workload was found to be significantly higher in the 12-Vehicle condition compared to the 4-Vehicle condition, nearly maxing out the NASA-TLX overall workload scale. There was no significant difference made by the Comm System on workload ratings. Pilots’ response times to communications were fastest in the Voice condition, although overall “service time” for compliance was shorter in Datalink and Hybrid conditions in most cases. Errors by pilots were frequent in both Vehicle Load conditions, most perniciously when using the Voice system. The results of this study suggest tradeoffs in advantages and disadvantages of the three comm systems. Recommendations for communication system design are provided taking the tradeoffs into account.

urban air mobility↗

NASA NTRS2024

Evaluation of Machine Learning and Deep Learning Algorithms for Fire Prediction in Southeast Asia

Vegetation fires are prevalent in South/Southeast Asian countries, making fire prediction crucial due to their potential environmental, economic, and social impacts. Accurate predictions of fires facilitate timely interventions, helping to mitigate uncontrolled fires that can lead to biodiversity loss and air quality issues. In this study, we utilize VIIRS satellite-derived fire data alongside six machine learning and deep learning models—Simple Persistence, Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), CNN-LSTM, and ConvLSTM—to determine the most effective fire prediction model, using Root Mean Square Error (RMSE) as the metric. Our results indicate that the CNN model is the most reliable in regions with spatial dependencies, such as Brunei, Indonesia, Malaysia, the Philippines, Timor-Leste, and Thailand. Conversely, the ConvLSTM model excels in countries with complex spatiotemporal dynamics like Laos, Myanmar, and Vietnam. The CNN-LSTM hybrid model also performed well in Cambodia, suggesting a need for a balanced approach in areas requiring both spatial and temporal feature extraction. Furthermore, simpler models like Persistence and MLP showed limitations in capturing dynamic patterns and temporal dependencies. Our findings highlight the importance of evaluating models before implementing any decision support systems (DSS) in fire management. By tailoring models to specific regional fire data, we can enhance prediction accuracy and responsiveness, ultimately improving fire risk management in Southeast Asia and beyond.

Deep learning↗

NASA NTRS2008

Similarity Metrics for Closed Loop Dynamic Systems

To what extent and in what ways can two closed-loop dynamic systems be said to be "similar?" This question arises in a wide range of dynamic systems modeling and control system design applications. For example, bounds on error models are fundamental to the controller optimization with modern control design methods. Metrics such as the structured singular value are direct measures of the degree to which properties such as stability or performance are maintained in the presence of specified uncertainties or variations in the plant model. Similarly, controls-related areas such as system identification, model reduction, and experimental model validation employ measures of similarity between multiple realizations of a dynamic system. Each area has its tools and approaches, with each tool more or less suited for one application or the other. Similarity in the context of closed-loop model validation via flight test is subtly different from error measures in the typical controls oriented application. Whereas similarity in a robust control context relates to plant variation and the attendant affect on stability and performance, in this context similarity metrics are sought that assess the relevance of a dynamic system test for the purpose of validating the stability and performance of a "similar" dynamic system. Similarity in the context of system identification is much more relevant than are robust control analogies in that errors between one dynamic system (the test article) and another (the nominal "design" model) are sought for the purpose of bounding the validity of a model for control design and analysis. Yet system identification typically involves open-loop plant models which are independent of the control system (with the exception of limited developments in closed-loop system identification which is nonetheless focused on obtaining open-loop plant models from closed-loop data). Moreover the objectives of system identification are not the same as a flight test and hence system identification error metrics are not directly relevant. In applications such as launch vehicles where the open loop plant is unstable it is similarity of the closed-loop system dynamics of a flight test that are relevant.

Whorton, Mark S.↗

NASA NTRS1992

A new neural net approach to robot 3D perception and visuo-motor coordination

A novel neural network approach to robot hand-eye coordination is presented. The approach provides a true sense of visual error servoing, redundant arm configuration control for collision avoidance, and invariant visuo-motor learning under gazing control. A 3-D perception network is introduced to represent the robot internal 3-D metric space in which visual error servoing and arm configuration control are performed. The arm kinematic network performs the bidirectional association between 3-D space arm configurations and joint angles, and enforces the legitimate arm configurations. The arm kinematic net is structured by a radial-based competitive and cooperative network with hierarchical self-organizing learning. The main goal of the present work is to demonstrate that the neural net representation of the robot 3-D perception net serves as an important intermediate functional block connecting robot eyes and arms.

Lee, Sukhan↗

NASA NTRS1985

An experimental evaluation of error seeding as a program validation technique

A previously reported experiment in error seeding as a program validation technique is summarized. The experiment was designed to test the validity of three assumptions on which the alleged effectiveness of error seeding is based. Errors were seeded into 17 functionally identical but independently programmed Pascal programs in such a way as to produce 408 programs, each with one seeded error. Using mean time to failure as a metric, results indicated that it is possible to generate seeded errors that are arbitrarily but not equally difficult to locate. Examination of indigenous errors demonstrated that these are also arbitrarily difficult to locate. These two results support the assumption that seeded and indigenous errors are approximately equally difficult to locate. However, the assumption that, for each type of error, all errors are equally difficult to locate was not borne out. Finally, since a seeded error occasionally corrected an indigenous error, the assumption that errors do not interfere with each other was proven wrong. Error seeding can be made useful by taking these results into account in modifying the underlying model.

Knight, J. C.↗

NASA NTRS2025

Uncertainty Quantification for Empirical X-59 Sonic Boom Loudness Levels

Estimates of the total uncertainty for empirically determined loudness levels are documented when GRS (Ground Recording System) noise monitors are used to record X 59 sonic boom waveforms. The total uncertainty is characterized by combining nine different sources of uncertainty that may affect the apparent gain of the measurement chain. These uncertainty estimates are presented as expected measurement error relative to the true loudness level, and separate error estimates are provided for eight different noise metrics in which NASA has interest. The behavior of the Perceived Level (PL) metric is studied within the report body, while the total uncertainties for the seven other noise metrics are summarized in appendices for brevity. The effects of four sources of uncertainty are estimated simply from information found on hardware specification sheets provided by the manufacturer. However, mock acoustic recordings are created to estimate the effects of other sources because those effects are expected to induce spectral coloration, so they may vary with noise metric type and sound level. These sources are not well modeled by simple gain adjustments. Importantly, measurement error is computable when processing mock recordings since the true levels are knowable, which is not the case when processing data recorded in the field. Specifically, the true levels are knowable because the components of the mock recordings are separable – e.g., loudness levels of booms can be computed with or without superimposed background noise. Mock acoustic recordings also have the benefit of allowing analysis of sonic booms from vehicles that are not yet flying, like the X-59, since the mock recordings are created by combining vehicle-specific predicted waveforms with other audio sources. The estimates of total measurement error are documented as a function of the signal-to-noise ratio (SNR) of the loudness level, where the corrected SNR is computed while accounting for the effects of the method that is used to correct for background noise contamination when computing the noise metric values. The corrected SNR calculations used here can be applied to both mock recordings and in-field measurements, so the uncertainty of in-field recordings can be found using pre-computed lookup tables that identify the relationship between metric type, corrected SNR, and the expected measurement error.

Sonic Boom↗

NASA NTRS1990

High-accuracy Mars approach navigation with radio metric and optical data

The aerocapture of a space vehicle on hyperbolic approach to Mars results in tight navigation requirements at atmospheric entry. The purpose of this paper is to examine several different methods for approach navigation and to determine what accuracies are possible. The methods are broken into four groups as follows (1) navigation with only Deep Space Network (DSN) tracking of the approach vehicle, (2) navigation with the DSN plus ranging between the approach vehicle and spacecraft in orbit about Mars, (3) navigation with the DSN plus optical data involving the Martian moons, and (4) navigation with DSN range data and differenced range data involving the approach spacecraft and orbiters at Mars. If the current modeling errors that affect earth-based radio metric data, such as errors in tracking station locations, the Martian ephemeris, and differences in the quasar and planetary coordinate frames, are improved, then perhaps earth-based tracking could meet the entry error requirements imposed by aerocapture. If not, then the other three options of intervehicular range, optical data, or differenced range provide highly accurate entry knowledge at least twelve hours before entry.

Konopliv, Alex↗

NASA NTRS2021

Development and Validation of an Empirical Ocean Color Algorithm with Uncertainties: A Case Study with the Particulate Backscattering Coefficient

We explored how algorithm (model) and in situ measurement (observation) uncertainties can effectively be incorporated into empirical ocean color model development and assessment. In this study we focused on methods for deriving the particulate backscattering coefficient at 555 nm, b(bp)(555)/(m). We developed a simple empirical algorithm for deriving b(bp)(555) as a function of a remote sensing reflectance line height (LH) metric. Model training was performed using a high-quality bio-optical dataset that contains coincident in situ measurements of the spectral remote sensing reflectances, R(rs)(λ)/(sr), and the spectral particulate backscattering coefficients, b(bp)(λ). The LH metric used is defined as the magnitude of Rrs(555) relative to a linear baseline drawn between R(rs)(490) and R(rs)(670). Using an independent validation dataset, we compared the skill of the LH-based model with two other models. We used contemporary validation metrics, including bias and mean absolute error (MAE), that were corrected for model and observation uncertainties. The results demonstrated that measurement uncertainties do indeed impact contemporary validation metrics such as mean bias and MAE. Zeta-scores and z-tests for overlapping confidence intervals were also explored as potential methods for assessing model skill.

ocean color↗

NASA NTRS1982

Software errors and complexity: An empirical investigation

The distributions and relationships derived from the change data collected during the development of a medium scale satellite software project show that meaningful results can be obtained which allow an insight into software traits and the environment in which it is developed. Modified and new modules were shown to behave similarly. An abstract classification scheme for errors which allows a better understanding of the overall traits of a software project is also shown. Finally, various size and complexity metrics are examined with respect to errors detected within the software yielding some interesting results.

Basili, V. R.↗

NASA NTRS1983

Software errors and complexity: An empirical investigation

The distributions and relationships derived from the change data collected during the development of a medium scale satellite software project show that meaningful results can be obtained which allow an insight into software traits and the environment in which it is developed. Modified and new modules were shown to behave similarly. An abstract classification scheme for errors which allows a better understanding of the overall traits of a software project is also shown. Finally, various size and complexity metrics are examined with respect to errors detected within the software yielding some interesting results.

Basili, Victor R.↗

NASA NTRS2008

A Study of the Effects of Atmospheric Phenomena on Mars Science Laboratory Entry Performance

At Earth during entry the shuttle has experienced what has come to be known as potholes in the sky or regions of the atmosphere where the density changes suddenly. Because of the small data set of atmospheric information where the Mars Science Laboratory (MSL) parachute deploys, the purpose of this study is to examine the effect similar atmospheric pothole characteristics, should they exist at Mars, would have on MSL entry performance. The study considers the sensitivity of entry design metrics, including altitude and range error at parachute deploy and propellant use, to pothole like density and wind phenomena.

Cianciolo, Alicia D.↗

NASA NTRS2016

Study of the Effect of Temporal Sampling Frequency on DSCOVR Observations Using the GEOS-5 Nature Run Results: Cloud Coverage - Part II

This is the second part of a study on how temporal sampling frequency affects satellite retrievals in support of the Deep Space Climate Observatory (DSCOVR) mission. Continuing from Part 1, which looked at Earth's radiation budget, this paper presents the effect of sampling frequency on DSCOVR-derived cloud fraction. The output from NASA's Goddard Earth Observing System version 5 (GEOS-5) Nature Run is used as the "truth". The effect of temporal resolution on potential DSCOVR observations is assessed by subsampling the full Nature Run data. A set of metrics, including uncertainty and absolute error in the subsampled time series, correlation between the original and the subsamples, and Fourier analysis have been used for this study. Results show that, for a given sampling frequency, the uncertainties in the annual mean cloud fraction of the sunlit half of the Earth are larger over land than over ocean. Analysis of correlation coefficients between the subsamples and the original time series demonstrates that even though sampling at certain longer time intervals may not increase the uncertainty in the mean, the subsampled time series is further and further away from the "truth" as the sampling interval becomes larger and larger. Fourier analysis shows that the simulated DSCOVR cloud fraction has underlying periodical features at certain time intervals, such as 8, 12, and 24 h. If the data is subsampled at these frequencies, the uncertainties in the mean cloud fraction are higher. These results provide helpful insights for the DSCOVR temporal sampling strategy.

GEOS-5↗

NASA NTRS2022

Landslide Likelihood Prediction using Machine Learning Algorithms

The supply of electricity via power plants is criticalto the operation of many critical infrastructure systems in mod-ern society. Natural hazards can disrupt the power supply, causepower outages that can halt economic growth, and impede emer-gency response until power is restored. The proposed work aimsto predict the landslides likelihood in these critical infrastructurelocations in the Northeastern USA using integrated databases ofexplanatory variables and machine learning algorithms. First,data related to landslides are obtained and merged, includingtopographic, soil moisture, and precipitation-related data. Fiveregression algorithms, namely: Random Forest, Extreme Gradi-ent Boosting (XGBoost), K-Nearest Neighbor regression (KNN),Linear Support Vector Regressor (SVR), and Linear regression,are utilized to predict the landslide probability and evaluatedon the dataset. The accuracy of the models is assessed by usingstatistical metrics such as mean absolute error (MAE), meansquared error (MSE), and root mean squared error (RMSE).The study results show that Random Forest outperformed othermodels with the mutual information feature selection method.It achieved an MSE of 0.0011 with mutual information-basedfeature selection and an MSE of 0.00157 without feature selection.KNN regressor outperformed the other models with an MSEof 0.00139 with correlation-based information selection. Theproposed landslide identification model with Random Forestalgorithm shows outstanding robustness and great potential intackling the landslide likelihood prediction by employing MLalgorithms.

Vasundhara Acharya↗

NASA NTRS2019

Estimation of coarse dead wood stocks in intact and degraded forests in the Brazilian Amazon using airborne lidar

Coarse dead wood is an important component of forest carbon stocks, but it is rarely measured in Amazon forests and is typically excluded from regional forest carbon budgets. Our study is based on line intercept sampling for fallen coarse dead wood conducted along 103 transects with a total length of 48 km matched with forest inventory plots where standing coarse dead wood was measured in the footprints of larger areas of airborne lidar acquisitions. We developed models to relate lidar metrics and Landsat time series variables to coarse dead wood stocks for intact, logged, burned, or logged and burned forests. Canopy characteristics such as gap area produced significant individual relations for logged forests. For total fallen plus standing coarse dead wood (hereafter defined as total coarse dead wood), the relative root mean square error for models with only lidar metrics ranged from 33 % in logged forest to up to 36 % in burned forests. The addition of historical information improved model performance slightly for intact forests (31 % against 35 % relative root mean square error), not justifying the use of a number of disturbance events from historical satellite images (Landsat) with airborne lidar data. Lidar-derived estimates of total coarse dead wood compared favorably with independent ground-based sampling for areas up to several hundred hectares. The relations found between total coarse dead wood and variables quantifying forest structure derived from airborne lidar highlight the opportunity to quantify this important but rarely measured component of forest carbon over large areas in tropical forests.

Brazilian Amazon↗

NASA NTRS1999

Advanced Communications Technology Satellite (ACTS) Fade Compensation Protocol Impact on Very Small-Aperture Terminal Bit Error Rate Performance

The Advanced Communications Technology Satellite (ACTS) communications system operates at Ka band. ACTS uses an adaptive rain fade compensation protocol to reduce the impact of signal attenuation resulting from propagation effects. The purpose of this paper is to present the results of an analysis characterizing the improvement in VSAT performance provided by this protocol. The metric for performance is VSAT bit error rate (BER) availability. The acceptable availability defined by communication system design specifications is 99.5% for a BER of 5E-7 or better. VSAT BER availabilities with and without rain fade compensation are presented. A comparison shows the improvement in BER availability realized with rain fade compensation. Results are presented for an eight-month period and for 24 months spread over a three-year period. The two time periods represent two different configurations of the fade compensation protocol. Index Terms-Adaptive coding, attenuation, propagation, rain, satellite communication, satellites.

Cox, Christina B.↗

NASA NTRS1992

Multiple symbol partially coherent detection of MPSK

It is shown that by using the known (or estimated) value of carrier tracking loop signal to noise ratio (SNR) in the decision metric, it is possible to improve the error probability performance of a partially coherent multiple phase-shift-keying (MPSK) system relative to that corresponding to the commonly used ideal coherent decision rule. Using a maximum-likeihood approach, an optimum decision metric is derived and shown to take the form of a weighted sum of the ideal coherent decision metric (i.e., correlation) and the noncoherent decision metric which is optimum for differential detection of MPSK. The performance of a receiver based on this optimum decision rule is derived and shown to provide continued improvement with increasing length of observation interval (data symbol sequence length). Unfortunately, increasing the observation length does not eliminate the error floor associated with the finite loop SNR. Nevertheless, in the limit of infinite observation length, the average error probability performance approaches the algebraic sum of the error floor and the performance of ideal coherent detection, i.e., at any error probability above the error floor, there is no degradation due to the partial coherence. It is shown that this limiting behavior is virtually achievable with practical size observation lengths. Furthermore, the performance is quite insensitive to mismatch between the estimate of loop SNR (e.g., obtained from measurement) fed to the decision metric and its true value. These results may be of use in low-cost Earth-orbiting or deep-space missions employing coded modulations.

Simon, M. K.↗

NASA NTRSDate not supplied

Urban Air Mobility Conflict Resolution: Centralized or Decentralized?

This work begins to address one of the critical questions in the urban air mobility and small unmanned aircraft communities: Should the en-route conflict resolution function in an urban air mobility traffic system be centralized or decentralized? Three conflict resolution architectures are modeled and analyzed: centralized, decentralized with uniform rules, and decentralized with mixed rules. This study compares these architectures and investigates their robustness to communication and state information errors in terms of safety and efficiency metrics. Experiments are conducted using a high-fidelity Monte Carlo traffic simulator and a generic set of traffic scenarios with increasing traffic density. When no errors were modeled, the centralized architecture marginally outperformed the decentralized architecture. However, performance of the centralized architecture was found to be adversely affected by the modeled input errors to a greater degree than was the decentralized architecture. Performance of the centralized architecture also was degraded significantly by the modeled transmission errors of the centralized resolution maneuvers. In the decentralized architecture, uniform rules outperformed mixed rules because, in the mixed rules case, system safety performance was undermined and dominated by the poor performers.

Urban Air Mobility (UAM) traffic system↗