Results for “evaluations” — Page 12

NASA NTRS2021

A New Approach to Evaluate and Reduce Uncertainty of Model-Based Biodiversity Projections for Conservation Policy Formulation

Biodiversity projections with uncertainty estimates under different climate, land-use, and policy scenarios are essential to setting and achieving international targets to mitigate biodiversity loss. Evaluating and improving biodiversity predictions to better inform policy decisions remains a central conservation goal and challenge. A comprehensive strategy to evaluate and reduce uncertainty of model outputs against observed measurements and multiple models would help to produce more robust biodiversity predictions. We propose an approach that integrates biodiversity models and emerging remote sensing and in-situ data streams to evaluate and reduce uncertainty with the goal of improving policy-relevant biodiversity predictions. In this article, we describe a multivariate approach to directly and indirectly evaluate and constrain model uncertainty, demonstrate a proof of concept of this approach, embed the concept within the broader context of model evaluation and scenario analysis for conservation policy, and highlight lessons from other modeling communities.

Bonnie J E Myers↗

NASA NTRSDate not supplied

Human-in-the-Loop Evaluations: Process and Mockup Fidelity

Human-in-the-loop (HITL) evaluations are iterative events used during design and development to identify issues with the implementation of human systems integration. HITL evaluations involve human test subjects who are user-representative participants performing activities with representative hardware, software, and procedures. The paper will describe the process by which HITL evaluations can be implemented in a program or project. This will include definitions for developmental and verification HITL evaluations, levels of mockup fidelity, and certification of HITL articles. The proposed process can be tailored depending on the type of HITL evaluation being conducted. It is highly recommended that findings are timely reported and incorporated in the hardware and/or software design.

Jackelynne Silva-Martinez↗

NASA NTRSDate not supplied

ISRU Potential Water Mine Sites; Preliminary Evaluation for NASA Artemis Campaign

The NASA Artemis Campaign aims to return to the Moon to maintain a sustainable presence [1], and In-Situ Resource Utilization (ISRU)is a key part of sustainability. The regions of interest for the Artemis campaign, as outlined in [1] and shown in Fig 1, are at Lunar the South Pole where water ice has been identified. The potential use of this water, and oxygen/hydrogen, for NASA and commercial applications such as refueling vehicles and power systems, and supplying life support consumables is one of the considerations the NASA Artemis team is using to evaluate these regions. As such, analyses are underway to evaluate the ISRU ice mining potential of these regions of interest. To do so, a set of ground rules for ISRU sites have been developed to align with current assumptions for customer needs, hardware capabilities, initially limited infrastructure, and lunar environments/terrain. The customer could be a lander, habitat, or other asset that makes use of ISRU product within the Artemis architecture. At this time, four of the regions of influence (the ‘western’ cluster in Fig. 1) have undergone preliminary ISRU evaluation. It should be noted that variety of other efforts have done similar evaluations of this nature with different assumptions or viewpoints, such as the most recent[2]. However, most evaluations focus on large permanently shadowed regions (PSRs) and craters due to orbital data resolution limitations, whereas early ice mining operations will likely occur in much smaller PSRs

In situ resource utilization↗

NASA NTRSDate not supplied

Evaluation of Response Surface Experiment Designs for Distributed Propulsion Aircraft Aero-Propulsive Modeling

Modern distributed hybrid and electric propulsion aircraft, including vertical, short, and conventional takeoff and landing configurations, exhibit significant aero-propulsive complexity and a large number of interacting test factors. This paper presents the development and evaluation of experiment designs for aero-propulsive characterization of distributed propulsion aircraft. Five different foundational response surface designs are evaluated to inform the development of two sequential design approaches tailored to complex aircraft aerodynamic characterization experiments. The first approach, which builds on sequential face-centered central composite designs, has been used previously to develop aero-propulsive models for complex aircraft using wind tunnel testing. The second approach is a new design strategy leveraging a regular I-optimal and nested I-optimal design that was developed for this study. The two sequential design strategies are compared for experiments with a large number of test factors using pre-experiment design evaluation metrics, as well as modeling results obtained from simulated wind tunnel data for the NASA LA-8 aircraft. The design evaluation metrics show that the sequential I-optimal base design has higher statistical power, lower correlation among candidate regressors, lower prediction variance, and more precise parameter estimates. The simulated wind tunnel experiments conducted using each design reveal that the sequential I-optimal base design has better predictive capability with fewer test points. The experiment design and evaluation procedures are described in detail to inform future aerodynamic characterization experiments for complex aircraft.

design of experiments↗

NASA NTRS2024

Evaluating Liftoff Debris for NASA’s Space Launch System (SLS) Prior to the Artemis I Launch

The SLS Artemis I launch vehicle is the first of several planned Artemis launch vehicles, with a number of design differences from earlier NASA missions that incur liftoff debris risk to the mission. As a test vehicle, the Artemis I hardware also endured environments and tests not planned for future missions, which led to several additional factors contributing to an evolving liftoff debris risk to the SLS vehicle. This paper will summarize these risk factors and address the processes used to evaluate and communicate the risks to support a successful Artemis I launch. It will discuss how the evolving risks that were quantified and evaluated by a Cross-Program team of debris Subject Matter Experts to mitigate liftoff debris hazards and communicate updated risk to the SLS vehicle. This process was performed through the inaugural use of an SLS debris day-of-launch (DOL) standard operating procedure that will be used for subsequent Artemis missions. This paper addresses the risk of liftoff debris, debris released by the vehicle or from the launch pad during liftoff through vehicle tower clear. Expected liftoff debris is well understood from previous NASA programs’ experience and from tests of materials, processes and functions that are known to release liftoff debris. These expected sources were assessed and cleared well ahead of launch day. However, given the ever-changing schedules and environments, processes were in place to evaluate any additional potential liftoff debris risks identified during launch countdown. Although many of the Artemis vehicle hardware components are similar to those on the NASA Shuttle Program, there are important differences in the architecture of the Artemis I vehicle which require new assessments of liftoff debris risk for the Artemis missions. The more favorable Artemis crew module location and surfaces are far less vulnerable to debris impacts; however, the longer vehicle can result in higher liftoff debris impact energies to those components on the aft end of the vehicle. Additionally, the positional change of the RS-25 liquid engines to nearer the Booster nozzle exit plane along with the change in Booster throat plug design is a disadvantage to the overall liftoff debris risk which resulted in additional test and analysis efforts for evaluating the integrated vehicle debris risk. In spite of the comprehensive tests and analyses of Artemis I expected liftoff debris, a number of additional tests/processes were completed prior to the Artemis I mission that were required to support a complete understanding of a new launch vehicle, but increased the risk of releasing liftoff debris. The hardware endured several additional cryogenic loading cycles, including the Green Run tests at Stennis Space Center, Wet Dress Rehearsals at Kennedy Space Center, and multiple launch attempts. Each of these cycles induced stresses in the thermal protection system (TPS) materials, increasing the risk of damage to and release of the TPS. Additionally, induced and weather environmental factors that could increase the likelihood of debris release were significant. Vibrations and stresses in the TPS were induced by a required roll-back to the Vehicle Assembly Building before Hurricane Ian to protect the vehicle from damage by high winds. Wind damage and potential internal stresses to several outer mold line materials on the integrated SLS vehicle and mobile launcher were caused by weathering Hurricane Nicole at Pad 39B the week before launch. A thorough imagery scan of the vehicle was performed after each event and the damage observed was repaired, removed, or assessed and the risk to the mission evaluated. Mitigation of debris risk can occur by tests and analyses to show debris impacted components as damage tolerant, by new/improved processes for prevention of debris availability, or redesign. Risk mitigation processes for Artemis I-specific liftoff debris events and the development and use of the SLS debris day of launch (DOL) procedures that will be used for subsequent Artemis missions will be described.

Space Launch System↗

NASA NTRSDate not supplied

Evaluating Liftoff Debris for NASA’s Space Launch System (SLS) Prior to the Artemis I Launch

The SLS Artemis I launch vehicle is the first of several planned Artemis launch vehicles, with a number of design differences from earlier NASA missions that incur liftoff debris risk to the mission. As a test vehicle, the Artemis I hardware also endured environments and tests not planned for future missions, which led to several additional factors contributing to an evolving liftoff debris risk to the SLS vehicle. This paper will summarize these risk factors and address the processes used to evaluate and communicate the risks to support a successful Artemis I launch. It will discuss how the evolving risks that were quantified and evaluated by a Cross-Program team of debris Subject Matter Experts to mitigate liftoff debris hazards and communicate updated risk to the SLS vehicle. This process was performed through the inaugural use of an SLS debris day-of-launch (DOL) standard operating procedure that will be used for subsequent Artemis missions. This paper addresses the risk of liftoff debris, debris released by the vehicle or from the launch pad during liftoff through vehicle tower clear. Expected liftoff debris is well understood from previous NASA programs’ experience and from tests of materials, processes and functions that are known to release liftoff debris. These expected sources were assessed and cleared well ahead of launch day. However, given the ever-changing schedules and environments, processes were in place to evaluate any additional potential liftoff debris risks identified during launch countdown. Although many of the Artemis vehicle hardware components are similar to those on the NASA Shuttle Program, there are important differences in the architecture of the Artemis I vehicle which require new assessments of liftoff debris risk for the Artemis missions. The more favorable Artemis crew module location and surfaces are far less vulnerable to debris impacts; however, the longer vehicle can result in higher liftoff debris impact energies to those components on the aft end of the vehicle. Additionally, the positional change of the RS-25 liquid engines to nearer the Booster nozzle exit plane along with the change in Booster throat plug design is a disadvantage to the overall liftoff debris risk which resulted in additional test and analysis efforts for evaluating the integrated vehicle debris risk. In spite of the comprehensive tests and analyses of Artemis I expected liftoff debris, a number of additional tests/processes were completed prior to the Artemis I mission that were required to support a complete understanding of a new launch vehicle, but increased the risk of releasing liftoff debris. The hardware endured several additional cryogenic loading cycles, including the Green Run tests at Stennis Space Center, Wet Dress Rehearsals at Kennedy Space Center, and multiple launch attempts. Each of these cycles induced stresses in the thermal protection system (TPS) materials, increasing the risk of damage to and release of the TPS. Additionally, induced and weather environmental factors that could increase the likelihood of debris release were significant. Vibrations and stresses in the TPS were induced by a required roll-back to the Vehicle Assembly Building before Hurricane Ian to protect the vehicle from damage by high winds. Wind damage and potential internal stresses to several outer mold line materials on the integrated SLS vehicle and mobile launcher were caused by weathering Hurricane Nicole at Pad 39B the week before launch. A thorough imagery scan of the vehicle was performed after each event and the damage observed was repaired, removed, or assessed and the risk to the mission evaluated. Mitigation of debris risk can occur by tests and analyses to show debris impacted components as damage tolerant, by new/improved processes for prevention of debris availability, or redesign. Risk mitigation processes for Artemis I-specific liftoff debris events and the development and use of the SLS debris day of launch (DOL) procedures that will be used for subsequent Artemis missions will be described.

Space Launch System↗

NASA NTRSDate not supplied

Advancing Electric Propulsion Aircraft Evaluation for Urban Air Mobility: Insights from NASA-Ames

This presentation delves into a recent evaluation conducted at NASA-Ames on the Vertical Motion Simulator, focusing on the handling qualities of Distributed Electric Propulsion VTOL (eVTOL) aircraft, specifically tailored for Urban Air Mobility (UAM) applications. The presentation will focus on the recent effort to adapt and refine use of the Aeronautical Design Standard -33 (ADS-33) rotorcraft handling qualities developed by the U.S. Army and NASA to meet the diverse needs of civilian (eVTOL) concept evaluation. A brief discussion of the author’s personal test pilot insights in the early development of military Fly-By-Wire evaluation methods will also be provided. The emergence of innovative eVTOL designs with unique lift capabilities and flight control systems, present both opportunities and challenges, particularly in ensuring safety amidst technological complexity. To navigate these challenges, our investigation examined evaluation criteria designed to accommodate the varied configurations and advanced automation systems inherent in modern eVTOL aircraft. By establishing a standardized approach to evaluation, our research not only fosters innovation but also upholds safety standards in the dynamic landscape of urban air mobility. Through this endeavor, we help to facilitate the seamless integration of novel aircraft capabilities into new operations, contributing to a new era of safe and efficient aerial transportation.

Loran Allen Haworth↗

NASA NTRSDate not supplied

Advancing Electric Propulsion Aircraft Evaluation for Urban Air Mobility: Insights from NASA Ames

This presentation delves into a recent evaluation conducted at NASA-Ames on the Vertical Motion Simulator, focusing on the handling qualities of Distributed Electric Propulsion VTOL (eVTOL) aircraft, specifically tailored for Urban Air Mobility (UAM) applications. The presentation will focus on the recent effort to adapt and refine use of the Aeronautical Design Standard -33 (ADS-33) rotorcraft handling qualities developed by the U.S. Army and NASA to meet the diverse needs of civilian (eVTOL) concept evaluation. A brief discussion of the author’s personal test pilot insights in the early development of military Fly-By-Wire evaluation methods will also be provided. The emergence of innovative eVTOL designs with unique lift capabilities and flight control systems, present both opportunities and challenges, particularly in ensuring safety amidst technological complexity. To navigate these challenges, our investigation examined evaluation criteria designed to accommodate the varied configurations and advanced automation systems inherent in modern eVTOL aircraft. By establishing a standardized approach to evaluation, our research not only fosters innovation but also upholds safety standards in the dynamic landscape of urban air mobility. Through this endeavor, we help to facilitate the seamless integration of novel aircraft capabilities into new operations, contributing to a new era of safe and efficient aerial transportation.

eVTOL↗

NASA NTRSDate not supplied

Evaluating the Trustworthiness of Deep Neural Networks in Deployment – A Comparative Study (Replicability Study)

As deep neural networks (DNNs) are increasingly used in safety critical applications, there is a growing concern for their trustworthiness. Even highly trained, high-performant networks are not 100% accurate. However, it is very difficult to predict their behaviour during deployment without ground truth. In this paper, we provide a comparative and replicability study on recent approaches that have been proposed to evaluate the trustworthiness of DNNs. We find that it is very difficult to run and reproduce the results for these approaches on their replication packages, and it is even more difficult to run the tools on artifacts other than their own. Further, it is difficult to compare the effectiveness of the tools, due to lack of clearly defined evaluation metrics. Our results indicate that more effort is needed in our research community to obtain sound techniques for evaluating the trustworthiness of neural networks in safety-critical domains. To this end, we contribute an evaluation framework that incorporates the considered approaches and enables evaluation on common benchmarks, using common metrics. Using this framework, we run a comparative study of the three approaches.

Trustworthy AI↗

NASA NTRS1975

CARETS: A prototype regional environmental information system. Volume 12: User evaluation of experimental land use maps and related products from the central Atlantic test site

The user interaction and evaluation phase of the USGS/NASA Central Atlantic Regional Ecological Test Site was designed to obtain the input of local, regional, State, and Federal agency users of land-resource information into the development, of a regional information system; to provide users with assistance and data resulting from CARETS research; and to have user. organizations evaluate to what extent the CARETS products meet their needs. The evaluation of CARETS land use and related products revealed that most user agencies interviewed, at all governmental levels, require more detailed data than that provided by the CARETS project. Few agencies found utility in the generalized ERTS Level I land--use maps. Level II data, though reported valuable by several users, was generally considered of secondary utility by most users. The products considered most useful by users at all levels were the high--altitude color-infrared photographs and the USGS orthophotoquads. Recommendations resulting from the evaluation reflect the need to establish a flexible and reliable system for providing more detailed raw and processed land-resource information as well as the need to improve the methods of making information available to users.

Land-use planning↗

NASA NTRS2003

An evaluation of the use of new Doppler methods for detecting longitudinal function abnormalities in a pacing-induced heart failure model

BACKGROUND: Doppler tissue echocardiography and color M-mode Doppler flow propagation velocity have proven useful in evaluating cross-sections of patients with left ventricular (LV) dysfunction, but experience with serial changes is limited. Purpose and methods: We tested their use by evaluating the temporal changes of LV function in a pacing-induced congestive heart failure model. Rapid ventricular pacing was initiated and maintained in 20 dogs for 4 weeks. Echocardiography was performed at baseline and weekly during brief pacing cessation. RESULTS: With rapid pacing, LV volume significantly increased and ejection fraction (57%-28%), stroke volume (37-18 mL), and mitral annulus systolic velocity (16.1-6.6 cm/s) by Doppler tissue echocardiography significantly decreased, with ejection fraction and mitral annulus systolic velocity closely correlated (r = 0.706, P <.0001). In contrast to the mitral inflow velocities, mitral annulus early diastolic velocity decreased steadily (12.3-7.3 cm/s) resulting in a dramatic decrease in mitral annulus early/late (1.22-0.57) diastolic velocity with no tendency toward pseudonormalization. The color M-mode Doppler flow propagation velocity also showed significant steady decrease (57-24 cm/s) throughout the pacing period. Multiple regression analysis chose mitral annulus systolic velocity (r = 0.895, P <.0001) and propagation velocity (r = 0.782, P <.0001) for the most important factor predicting LV systolic and diastolic function, respectively. CONCLUSIONS: Doppler tissue echocardiography and color M-mode Doppler flow could evaluate the serial deterioration in LV dysfunction throughout the pacing period. These were more useful in quantifying progressive LV dysfunction than conventional ehocardiographic techniques, and were probably relatively independent of preload. These techniques could be suitable for longitudinal evaluation in addition to the cross-sectional study.

Evaluation Studies↗

NASA NTRS2020

Identification of Scenarios for System Interface Design Evaluation: CAST SE-210 Output 2 Report 5 of 6

This report is one in a series of reports describing research to develop enhanced approaches to design and evaluation of flight deck interaction. The research was conducted to support the Commercial Aviation Safety Team (CAST) response to incidents and accidents caused by failure of the flight crew to maintain aircraft attitude or energy state awareness. This report partially responds to Safety Enhancement (SE) 210.2 (CAST, 2014) by describing development of operational scenarios for evaluation of flight crew interaction. Evaluations of human performance with an interface rely on asking humans to perform operational tasks in an appropriate operational context. The report includes the major sources of material for developing operational scenarios—which include system state, operational tasks, operational context, and safety events—and a method for developing an appropriate set of operational scenarios for interface evaluation.

commercial aviation↗

NASA NTRS2024

Evaluating Differences Among Crop Models in Simulating Soybean in-Season Growth

Crop models are useful tools for simulating agricultural systems that require continued model development and testing to increase their robustness and improve how they describe our current understanding of processes. Coordinated and “blind” evaluation of multiple models using same protocols and experimental datasets provides unique opportunities to further improve models and enhance their reliability. For soybean [Glycine max (L.) Merr.], there has been limited coordinated multi-model evaluations for the simulation of in-season plant growth dynamics. We evaluated ten dynamic soybean crop models for their simulation of in-season plant growth using data from five experiments conducted in Argentina, Brazil, France, and USA. We evaluated models after a Blind (using only phenology data) and a Full calibration (with in-season and end-of-season variables). Calibration reduced model uncertainty by reducing standard bias for the simulation of in-season variables (biomass, leaf, pod, and stem weights, and leaf area index, LAI). However, we found that most models had difficulty in reproducing leaf growth dynamics, with normalized root mean squared error (nRMSE) of 56% for leaf weight and 43% for LAI (across locations and models after Full calibration). Models with different levels of complexity and experience were capable of simulating final seed yield at maturity with reasonable accuracy (nRMSE of 8–31% after Full calibration). However, the nRMSE for pod weight (of 17–64% after Full calibration) was two-fold larger than that of seed yield. Moreover, the models differed in how they simulated timing from sowing to beginning seed growth (47–93 days) and effective seed filling period (18–54 days), owing to model structural differences in defining the reproductive developmental stages. Overall, we identified the following processes that can benefit from further model improvement: leaf expansion and senescence, reproductive phenology, and partitioning to reproductive growth. Simulation of pod wall tissue and individual seed cohorts is another aspect that many models currently lack. Model improvement can benefit from high-temporal resolution experimental datasets that concurrently account for phenology, plant growth, and partitioning. Further, we recommend collecting reproductive phenology in the field consistent with actual dry matter allocation to organs in the models and collecting multiple observations of seed and pod weight to aid model improvement for simulation of seed growth and yield formation.

Agricultural Model Intercomparison and Improvement↗

NASA NTRS1972

Flight program language requirements. Volume 2: Requirements and evaluations

The efforts and results are summarized for a study to establish requirements for a flight programming language for future onboard computer applications. Several different languages were available as potential candidates for future NASA flight programming efforts. The study centered around an evaluation of the four most pertinent existing aerospace languages. Evaluation criteria were established, and selected kernels from the current Saturn 5 and Skylab flight programs were used as benchmark problems for sample coding. An independent review of the language specifications incorporated anticipated future programming requirements into the evaluation. A set of detailed language requirements was synthesized from these activities. The details of program language requirements and of the language evaluations are described.

Source record↗

NASA NTRS1972

Evaluation of thermal network correction program using test temperature data

An evaluation process to determine the accuracy of a computer program for thermal network correction is discussed. The evaluation is required since factors such as inaccuracies of temperatures, insufficient number of temperature points over a specified time period, lack of one-to-one correlation between temperature sensor and nodal locations, and incomplete temperature measurements are not present in the computer-generated information. The mathematical models used in the evaluation are those that describe a physical system composed of both a conventional and a heat pipe platform. A description of the models used, the results of the evaluation of the thermal network correction, and input instructions for the thermal network correction program are presented.

Ishimoto, T.↗

NASA NTRS1972

Evaluation of spacecraft boom deployment dynamics by combined analysis and testing.

Discussion of the deployment of multiple hinged booms from the IMP-I satellite, which is a good example of the use of the combined analysis and testing approach for preflight system reliability evaluation. The procedures used for test and evaluation of the booms are described, and problems encountered and results achieved are considered. Analytical methods, using a digital computer program, and functional test operation with the best possible environmental simulation, were both applied to the problem and comparisons between analytical and test results were made to evaluate their validity and to reveal the effects of errors and imperfections. There is a synergistic advantage in the combined approach in that mutual comparison gives better evaluation of both analysis and test results than independent study of either. An appendix presents some results of a later and as yet incomplete test program involving the IMP-H satellite.

Lang, W. E.↗

NASA NTRS1973

Evaluation of the electro-optic direction sensor

Evaluation of a no-moving-parts single-axis star tracker called an electro-optic direction sensor (EODS) concept is described and the results are given in detail. The work involved experimental evaluation of a breadboard sensor yielding results which would permit design of a prototype sensor for a specific application. The laboratory work included evaluation of the noise equivalent input angle of the sensor, demonstration of a technique for producing an acquisition signal, constraints on the useful field-of-view, and a qualitative evaluation of the effects of stray light. In addition, the potential of the silicon avalanche-type photodiode for this application was investigated. No benefit in noise figure was found, but the easily adjustable gain of the avalanche device was useful. The use of mechanical tuning of the modulating element to reduce voltage requirements was also explored. The predicted performance of EODS in both photomultiplier and solid state detector configurations was compared to an existing state-of-the-art star tracker.

Johnson, A. R.↗

NASA NTRS1973

Sensitivity and comparison evaluation of Saturn 5 liquid penetrants

Results of a sensitivity and comparison evaluation performed on six liquid penetrants that were used on the Saturn 5 vehicle and other space hardware to detect surface discontinuities are described. The relationship between penetrant materials and crack definition capabilities, the optimum penetrant materials evaluation method, and the optimum measurement methods for crack dimensions were investigated. A unique method of precise developer thickness control was envolved, utilizing clear radiographic film and a densitometer. The method of evaluation included five aluminum alloy, 2219-T87, specimens that were heated and then quenched in cold water to produce cracks. The six penetrants were then applied, one at a time, and the crack indications were counted and recorded for each penetrant for comparison purposes. Measurements were made by determining the visual crack indications per linear inch and then sectioning the specimens for a metallographic count of the cracks present. This method provided a numerical approach for assigning a sensitivity index number to the penetrants. Of the six penetrants evaluated, two were not satisfactory (one was not sufficiently sensitive and the other was to sensitive, giving false indications). The other four were satisfactory with approximately the same sensitivity in the range of 78 to 80.5 percent of total cracks detected.

Jones, G. H.↗