SEARCH · Engineering Papers

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

Evaluating the Trustworthiness of Deep Neural Networks in Deployment – A Comparative Study (Replicability Study)

As deep neural networks (DNNs) are increasingly used in safety critical applications, there is a growing concern for their trustworthiness. Even highly trained, high-performant networks are not 100% accurate. However, it is very difficult to predict their behaviour during deployment without ground truth. In this paper, we provide a comparative and replicability study on recent approaches that have been proposed to evaluate the trustworthiness of DNNs. We find that it is very difficult to run and reproduce the results for these approaches on their replication packages, and it is even more difficult to run the tools on artifacts other than their own. Further, it is difficult to compare the effectiveness of the tools, due to lack of clearly defined evaluation metrics. Our results indicate that more effort is needed in our research community to obtain sound techniques for evaluating the trustworthiness of neural networks in safety-critical domains. To this end, we contribute an evaluation framework that incorporates the considered approaches and enables evaluation on common benchmarks, using common metrics. Using this framework, we run a comparative study of the three approaches.

Trustworthy AI↗

NASA NTRS1999

Framework Requirements for MDO Application Development

Frameworks or problem solving environments that support application development form an active area of research. The Multidisciplinary Optimization Branch at NASA Langley Research Center is investigating frameworks for supporting multidisciplinary analysis and optimization research. The Branch has generated a list of framework requirements, based on the experience gained from the Framework for Interdisciplinary Design Optimization project and the information acquired during a framework evaluation process. In this study, four existing frameworks are examined against these requirements. The results of this examination suggest several topics for further framework research.

Salas, A. O.↗

NASA NTRS2021

Passive Microwave Brightness Temperature Assimilation to Improve Snow Mass Estimation across Complex Terrain in Pakistan, Afghanistan, and Tajikistan

An ensemble Kalman filter is used to assimilate Advanced Microwave Scanning Radiometer-2 (AMSR2) observations of passive microwave (PMW) brightness temperatures (spectral differences, ΔT b ) into land surface model estimates of snow mass over northwestern high mountain Asia (HMA). Trained support vector machines serve as the observation operator and map the geophysical modeled variables into ΔT b space within the assimilation framework. Evaluation of the assimilation routine is carried out through comparison of assimilated snow mass estimates with an in situ dataset. The assimilation framework helps improve the land surface model estimates through PMW ΔT b assimilation, particularly in terms of decreasing the domain-wide bias. The assimilation framework proved more effective during the (dry) snow accumulation season and decreased the bias and root-mean-square error (RMSE) in snow mass estimates at 76% and 58% of the comparative pixels, respectively. During the snow ablation season, the PMW brightness temperature signal contained less information related to snow mass due to the presence of other concurrent geophysical features that effectively serve as noise during the snow mass update. The utilization of PMW ΔT b for accurate snow mass estimation in complex terrain such as HMA is dependent on a multitude of factors for optimal results; however, it does add utility to the land surface model if the relevant pitfalls are taken into consideration prior to the state variable update.

Jawairia Ahmad↗

NASA NTRS2017

Modelling the Risks Remotely Piloted Aircraft Pose to People on the Ground

Worldwide there is much e ort being directed towards the development of a framework of air- worthiness regulations for remotely piloted aircraft systems (RPAS). It is now broadly accepted that regulations should have a strong foundation in, and traceability to, the management of the safety risks. Existing risk models for RPAS operations do not provide a simple means for incorporating the wide range of technical and operational controls into the risk analysis and evaluation processes. This paper describes a new approach for modelling and evaluating the risks associated with RPAS operations near populous areas based on the barrier bow tie (BBT) model. A BBT model is used to structure the underlying risk management problem. The model focuses risk analysis, evaluation, and decision making activities on the devices, people, and processes that can be employed to reduce risk. The BBT model and a comprehensive set of example risk controls are presented. The general model can be applied to any RPAS operation. The foundations for quantitative and qualitative assessments using a BBT model are also presented. The modelling and evaluation framework is illustrated through its application to a case-study rotary wing RPAS for two operational scenarios. The model can be used as a basis for determining airworthiness certification requirements for RPAS.

Barrier bow tie model↗

NASA NTRS1999

Multidisciplinary Design Technology Development: A Comparative Investigation of Integrated Aerospace Vehicle Design Tools

This research effort is a joint program between the Departments of Aerospace and Mechanical Engineering and the Computer Science and Engineering Department at the University of Notre Dame. The purpose of the project was to develop a framework and systematic methodology to facilitate the application of Multidisciplinary Design Optimization (MDO) to a diverse class of system design problems. For all practical aerospace systems, the design of a systems is a complex sequence of events which integrates the activities of a variety of discipline "experts" and their associated "tools". The development, archiving and exchange of information between these individual experts is central to the design task and it is this information which provides the basis for these experts to make coordinated design decisions (i.e., compromises and trade-offs) - resulting in the final product design. Grant efforts focused on developing and evaluating frameworks for effective design coordination within a MDO environment. Central to these research efforts was the concept that the individual discipline "expert", using the most appropriate "tools" available and the most complete description of the system should be empowered to have the greatest impact on the design decisions and final design. This means that the overall process must be highly interactive and efficiently conducted if the resulting design is to be developed in a manner consistent with cost and time requirements. The methods developed as part of this research effort include; extensions to a sensitivity based Concurrent Subspace Optimization (CSSO) NMO algorithm; the development of a neural network response surface based CSSO-MDO algorithm; and the integration of distributed computing and process scheduling into the MDO environment. This report overviews research efforts in each of these focus. A complete bibliography of research produced with support of this grant is attached.

Renaud, John E.↗

NASA NTRS1998

Multidisciplinary Design Technology Development: A Comparative Investigation of Integrated Aerospace Vehicle Design Tools

This research effort is a joint program between the Departments of Aerospace and Mechanical Engineering and the Computer Science and Engineering Department at the University of Notre Dame. Three Principal Investigators; Drs. Renaud, Brockman and Batill directed this effort. During the four and a half year grant period, six Aerospace and Mechanical Engineering Ph.D. students and one Masters student received full or partial support, while four Computer Science and Engineering Ph.D. students and one Masters student were supported. During each of the summers up to four undergraduate students were involved in related research activities. The purpose of the project was to develop a framework and systematic methodology to facilitate the application of Multidisciplinary Design Optimization (N4DO) to a diverse class of system design problems. For all practical aerospace systems, the design of a systems is a complex sequence of events which integrates the activities of a variety of discipline "experts" and their associated "tools". The development, archiving and exchange of information between these individual experts is central to the design task and it is this information which provides the basis for these experts to make coordinated design decisions (i.e., compromises and trade-offs) - resulting in the final product design. Grant efforts focused on developing and evaluating frameworks for effective design coordination within a MDO environment. Central to these research efforts was the concept that the individual discipline "expert", using the most appropriate "tools" available and the most complete description of the system should be empowered to have the greatest impact on the design decisions and final design. This means that the overall process must be highly interactive and efficiently conducted if the resulting design is to be developed in a manner consistent with cost and time requirements. The methods developed as part of this research effort include; extensions to a sensitivity based Concurrent Subspace Optimization (CSSO) MDO algorithm; the development of a neural network response surface based CSSO-MDO algorithm; and the integration of distributed computing and process scheduling into the MDO environment. This report overviews research efforts in each of these focus. A complete bibliography of research produced with support of this grant is attached.

Renaud, John E.↗

NASA NTRS1974

Electron beam, laser beam and plasma arc welding studies

This program was undertaken as an initial step in establishing an evaluation framework which would permit a priori selection of advanced welding processes for specific applications. To this end, a direct comparison of laser beam, electron beam and arc welding of Ti-6Al-4V alloy was undertaken. Ti-6Al-4V was selected for use in view of its established welding characteristics and its importance in aerospace applications.

Banas, C. M.↗

NASA NTRS1978

Forest resource information system

The author has identified the following significant results. A benchmark classification evaluation framework was implemented. The FRIS preprocessing activities were refined. Potential geo-based referencing systems were identified as components of FRIS.

Mroczynski, R. P.↗

NASA NTRS1997

Why are Formal Methods Not Used More Widely?

Despite extensive development over many years and significant demonstrated benefits, formal methods remain poorly accepted by industrial practitioners. Many reasons have been suggested for this situation such as a claim that they extent the development cycle, that they require difficult mathematics, that inadequate tools exist, and that they are incompatible with other software packages. There is little empirical evidence that any of these reasons is valid. The research presented here addresses the question of why formal methods are not used more widely. The approach used was to develop a formal specification for a safety-critical application using several specification notations and assess the results in a comprehensive evaluation framework. The results of the experiment suggests that there remain many impediments to the routine use of formal methods.

Knight, John C.↗

NASA NTRS2020

COSMIC: Content-based Onboard Summarization to Monitor Infrequent Change

Interplanetary exploration occurs at vast distancesthat severely limit communication bandwidth to spacecraft exploringother planets. It is possible to collect much morescientific data than can ever be downlinked given current communicationcapabilities. Therefore, we are developing a systemcalled COSMIC (Content-based Onboard Summarization toMonitor Infrequent Change) that will opportunistically analyzedata onboard a Mars orbiter to alert scientists when meaningfulchanges have occurred. COSMIC will allow future spacecraftto continuously collect data to search for rare, transient phenomenasuch as fresh impacts or seasonally changing polarlandforms under a constrained downlink budget. In this paper,we describe the overall goals and architecture of COSMIC,plans to enable specific scientific studies, label acquisition toenable supervised approaches to surface landform classification,a new machine learning evaluation framework for analyzingthe trade-offs between classifier accuracy and computationalrequirements, and lessons learned about constraints that COSMICwill face operating onboard a spacecraft. In particular, wediscuss design considerations surrounding computational andstorage constraints, change detection strategies, and localizingdetected landforms of interest within a global coordinate frame.Finally, we describe challenges and open research questions thatmust be addressed prior to deploying COSMIC.

Trockman, Asher↗

NASA NTRSDate not supplied

Better Constraining Supercooled Clouds Could Reduce Projected Warming Spread

The increase of climate sensitivity to rising greenhouse gas concentrations in the coupled model intercomparison project phase 6 (CMIP6) Earth system models (ESMs) compared to CMIP5 ESMs is primarily attributed to a larger extratropical cloud response to climate change, referred to as cloud feedback. The ratio of supercooled liquid cloud water relative to all cloud water, termed liquid phase ratio (LPR), which has also notably increased in many recent ESMs, is thought to be a primary driver of the extratropical cloud feedback increase. Unlike the preponderance of previous studies that compare native model LPR directly with observations, here we evaluate LPR over three ESM generations against Cloud–Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) using an instrument simulator approach, which mimics instrument limitations and uses consistent cloud definitions and resolutions. We find that current coupled model intercomparison project (CMIP) ESMs collectively simulate greater LPR than previous CMIP generations and overestimate LPR compared to observations, contrary to previous findings, likely driven by past inconsistent comparisons of ESM outputs with CALIPSO observations. We further show that greater LPR in ESM present-day climate is unexpectedly correlated with a smaller extratropical cloud feedback, attributable to a decrease of cloud amount exceeding the increase of cloud optical depth. Finally, our results suggest that improving constraints on model LPR using our evaluation framework would likely reduce the spread in CMIP6 climate sensitivities, owing to its effect on extratropical cloud feedback from supercooled clouds.

greenhouse gas concentrations↗

NASA NTRSDate not supplied

Critical Progressive Damage Analysis Model Features for Low-Velocity Impact on Thermoset and Thermoplastic Panels

A previous verification and validation framework evaluated the ability of various progressive damage analysis (PDA) tools to predict the mechanical response and damage state in post-buckled stiffened aerospace structures composed of fiber-reinforced thermoset composite materials. Herein, the ability of PDA tools to predict damage in thermoplastic composite materials subject to low-velocity impact (LVI) loading has been assessed. LVI tests and analyses have been conducted for similar panels, comprising thermoset or thermoplastic composite materials. Preliminary test/analysis correlations revealed that using the same combination of PDA model features and finite element model-building guidelines for the thermoset and thermoplastic LVI specimens resulted in underprediction of matrix crack lengths and delaminated area in the thermoplastic analyses despite yielding good correlations for the thermoset specimens. In this paper, the quality of the test/analysis correlations for analyses of thermoplastic LVI specimens performed with different combinations of PDA model features using the NASA CompDam material model are presented. Model features that were assessed in terms of effect on the quality of the test/analysis correlation include the representation of damageable interfaces, residual thermal stresses, and fiber damage.

Frank Leone↗

NASA NTRSDate not supplied

Critical Progressive Damage Analysis Model Features for Low-Velocity Impact on Thermoset and Thermoplastic Panels

Frank Leone↗

NASA NTRSDate not supplied

Simulated Multipath Using Software Generated GPS Signals

Depending on the environment, multipath can be one of the largest error sources contributing to degradation in Global Navigation Satellite System (GNSS) (e.g., GPS) performance. Multipath is a phenomenon that occurs as radio signals reflect off of surfaces, such as buildings, producing multiple copies of the original signal. When this occurs with GPS signals, it results in one or more delayed signals arriving at the receiver with or without the on-time/direct GPS signal. The receiver measures the composite of these signals which, depending on the severity of the multipath, can substantially degrade the accuracy of the receiver's calculated position. Multipath is commonly experienced in cities due to tall buildings and its mitigation is an ongoing area of study. This research demonstrates a novel approach for simulating GPS multipath through the modification of an open-source tool, GPS-SDR-SIM. The resulting additional testing capability could allow for improved development of multipath mitigating technologies. Currently, open-source tools for simulating GPS signals are available and can be used in the testing and evaluation of GPS receiver equipment. These tools can generate GPS signals that, when used by a GPS receiver, result in computation of a position solution that was pre-determined at the time of signal generation. That is, the signals produced are properly formed for the pre-determined location and result in the receiver reporting that position. This allows for a GPS receiver under test to be exposed to various simulated locations and conditions without having to be physically subjected to them. Additionally, while these signals are generated by a software simulation, they can be processed by real or software defined GPS receivers. This work utilizes the GPS-SDR-SIM software tool for GPS signal generation and while this tool does implement some sources of error that are inherent to GPS, it cannot inject multipath. GPS-SDR-SIM was modified in this effort to produce additional copies of signals with pre-determined delays. These additional delayed signals mimic multipath and represent what happens to GPS signals in the real world as they reflect off of surfaces and arrive at a receiver in place of or alongside the direct GPS signal. A successful proof of concept was prototyped and demonstrated using this modified version of GPS-SDR-SIM to produce simulated GPS signals as well as additional simulated multipath signals. The generated data was processed using a software defined GPS receiver and it was found that the introduction of simulated multipath signals successfully produced the expected characteristics of a composite multipath signal. Further maturation of this work could allow for the development of a GPS receiver testing and evaluation framework and aid in the development of multipath mitigating technologies.

GPS↗

NASA NTRS1998

A Catchment-Based Land Surface Model for GCMs and the Framework for its Evaluation

A new GCM-scale land surface modeling strategy that explicitly accounts for subgrid soil moisture variability and its effects on evaporation and runoff is now being explored. In a break from traditional modeling strategies, the continental surface is disaggregated into a mosaic of hydrological catchments, with boundaries that are not dictated by a regular grid but by topography. Within each catchment, the variability of soil moisture is deduced from TOP-MODEL equations with a special treatment of the unsaturated zone. This paper gives an overview of this new approach and presents the general framework for its off-line evaluation over North-America.

Ducharen, A.↗

NASA NTRS2002

Software Engineering Support of the Third Round of Scientific Grand Challenge Investigations: Earth System Modeling Software Framework Survey

One of the most significant challenges in large-scale climate modeling, as well as in high-performance computing in other scientific fields, is that of effectively integrating many software models from multiple contributors. A software framework facilitates the integration task, both in the development and runtime stages of the simulation. Effective software frameworks reduce the programming burden for the investigators, freeing them to focus more on the science and less on the parallel communication implementation. while maintaining high performance across numerous supercomputer and workstation architectures. This document surveys numerous software frameworks for potential use in Earth science modeling. Several frameworks are evaluated in depth, including Parallel Object-Oriented Methods and Applications (POOMA), Cactus (from (he relativistic physics community), Overture, Goddard Earth Modeling System (GEMS), the National Center for Atmospheric Research Flux Coupler, and UCLA/UCB Distributed Data Broker (DDB). Frameworks evaluated in less detail include ROOT, Parallel Application Workspace (PAWS), and Advanced Large-Scale Integrated Computational Environment (ALICE). A host of other frameworks and related tools are referenced in this context. The frameworks are evaluated individually and also compared with each other.

Talbot, Bryan↗

NASA NTRS2012

A Framework for Probabilistic Evaluation of Interval Management Tolerance in the Terminal Radar Control Area

Projections of future traffic in the national airspace show that most of the hub airports and their attendant airspace will need to undergo significant redevelopment and redesign in order to accommodate any significant increase in traffic volume. Even though closely spaced parallel approaches increase throughput into a given airport, controller workload in oversubscribed metroplexes is further taxed by these approaches that require stringent monitoring in a saturated environment. The interval management (IM) concept in the TRACON area is designed to shift some of the operational burden from the control tower to the flight deck, placing the flight crew in charge of implementing the required speed changes to maintain a relative spacing interval. The interval management tolerance is a measure of the allowable deviation from the desired spacing interval for the IM aircraft (and its target aircraft). For this complex task, Formal Methods can help to ensure better design and system implementation. In this paper, we propose a probabilistic framework to quantify the uncertainty and performance associated with the major components of the IM tolerance. The analytical basis for this framework may be used to formalize both correctness and probabilistic system safety claims in a modular fashion at the algorithmic level in a way compatible with several Formal Methods tools.

Hercencia-Zapana, Heber↗

NASA NTRS2023

A Multi-Satellite Framework to Rapidly Evaluate Extreme Biosphere Cascades: The Western US 2021 Drought and Heatwave

The increasing frequency and intensity of climate extremes and complex ecosystem responses motivate the need for integrated observational studies at low-latency to determine biosphere responses and carbon-climate feedbacks. Here, we develop a satellite-based rapid attribution workflow and demonstrate its use at a 1–2-month latency to attribute drivers of the carbon cycle feedbacks during the 2020-2021 Western US drought and heatwave. In the first half of 2021, concurrent negative photosynthesis anomalies and large positive column CO 2 anomalies were detected with satellites. Using a simple atmospheric mass balance approach, we estimate a surface carbon efflux anomaly of 132 TgC in June 2021, a magnitude corroborated 28 independently with a dynamic global vegetation model. Integrated satellite observations of hydrologic processes, representing the soil-plant-atmosphere continuum (SPAC), show that these surface carbon flux anomalies are largely due to substantial reductions in photosynthesis because of a spatially widespread moisture-deficit propagation through the SPAC between 2020 and 2021. A causal model indicates deep soil moisture stores partially drove photosynthesis, maintaining its values in 2020 and driving its declines throughout 2021. The causal model also suggests legacy effects may have amplified photosynthesis deficits in 2021 beyond the direct effects of environmental forcing. The integrated, observation framework presented here provides a valuable first assessment of a biosphere extreme response and an independent testbed for improving drought propagation and mechanisms in models. The rapid identification of extreme carbon anomalies and hotspots can also aid mitigation and adaptation decisions.

Causal model↗