Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS2018

(GO)2-SIM: a GCM-Oriented Ground-Observation Forward-Simulator Framework for Objective Evaluation of Cloud and Precipitation Phase

General circulation model (GCM) evaluation using ground-based observations is complicated by inconsistencies in hydrometeor and phase definitions. Here we describe (GO)2-SIM, a forward simulator designed for objective hydrometeor-phase evaluation, and assess its performance over the North Slope of Alaska using a 1-year GCM simulation. For uncertainty assessment, 18 empirical relationships are used to convert model grid-average hydrometeor (liquid and ice, cloud, and precipitation) water contents to zenith polarimetric micropulse lidar and Ka-band Doppler radar measurements, producing an ensemble of 576 forward-simulation realizations. Sensor limitations are represented in forward space to objectively remove from consideration model grid cells with undetectable hydrometeor mixing ratios, some of which may correspond to numerical noise.Phase classification in forward space is complicated by the inability of sensors to measure ice and liquid signals distinctly. However, signatures exist in lidar–radar space such that thresholds on observables can be objectively estimated and related to hydrometeor phase. The proposed phase-classification technique leads to misclassification in fewer than 8% of hydrometeor-containing grid cells. Such misclassifications arise because, while the radar is capable of detecting mixed-phase conditions, it can mistake water- for ice-dominated layers. However, applying the same classification algorithm to forward-simulated and observed fields should generate hydrometeor-phase statistics with similar uncertainty. Alternatively, choosing to disregard how sensors define hydrometeor phase leads to frequency of occurrence discrepancies of up to 40%. So, while hydrometeor-phase maps determined in forward space are very different from model "reality" they capture the information sensors can provide and thereby enable objective model evaluation.

Lamer, K.↗
NASA NTRSDate not supplied

Traffic Prediction for Uncommunicative Aircraft in Terminal Airspace: Development Framework and Performance Evaluations

This paper presents an air traffic prediction algorithm that takes observations of an aircraft and classifies aircraft type, estimates the aircraft's intent to and method of joining an airport traffic pattern, and predicts the aircaft's future trajectory. To develop algorithms that enable autonomous aircraft to safely insert into un-towered traffic patterns, several challenges need to be addressed. These challenges range from traffic detection to sensor fusion to own-ship trajectory replanning. Critical to a trajectory replanning algorithm is information regarding the future behavior of all traffic aircraft in the operational environment. The presented traffic prediction algorithm generates this information using regular measurements of traffic aircraft position and velocity to classify the aircraft by speed-class, estimate how the aircraft will approach the runway, and construct a predicted trajectory to the runway including future positions and velocities at specific times. The predictions of the presented algorithm are the necessary inputs for any downstream traffic pattern sequencing and own-ship trajectory planning routines. The presented algorithm is benchmarked using approximately 300 randomized traffic trajectories, spanning four vehicle weight classes and eight traffic entry types. While the algorithm can process multiple traffic vehicles in the terminal area, there is no prediction of traffic-on-traffic interaction. Each traffic vehicle is processed separately.

John D McMinn↗
NASA NTRSDate not supplied

The Inspectability Metric: A Formalized System Of Measurement Enabling The Design For Inspection Framework

Nondestructive evaluation (NDE) engineers are often confronted with structural design choices that present challenges to meeting inspection requirements. These challenges, at best, increase the resources needed to design an inspection solution and, at worst, require resource intensive redesign of the structure. If the inspectability of the structure can be determined early in the design cycle, these challenging inspection scenarios can be avoided. The emergence of additive manufacturing has further compounded this problem by enabling the creation of highly optimized structures with no regard to inspection constraints. Design for inspection (DFI) offers a framework to integrate nondestructive evaluation (NDE) into the design process to alleviate the mechanisms that produce uninspectable designs. DFI is the concept of including inspectability in a multi-objective optimization framework so that it can be considered in parallel to other metrics such as mass and manufacturability. This allows rapid evaluation of the trade-off between design metrics to find solutions that meet the inspection needs of a particular material system, structural concept, or vehicle program. To enable DFI, there must be a system by which the inspectability of a structure can be measured. This system must be agile to produce results quickly, it must be versatile to work with the type of incomplete information one would encounter early in the design process (such as lack of inspection requirements), and it must be delivered in a form that is easily understood by designers. To meet this need, this presentation introduces the novel inspectability metric as a system to measure inspectability. The inspectability metric is a standardized, automation friendly procedure that uses simulations to determine inspectability. Along with guidelines to properly process designs and integrate with existing workflows, the inspectability metric provides a suite of simulation tests to interrogate the ability to find defects and the sensitivity to variability. The testing rubric is designed to maximize the coverage of the parameter space while minimizing the number of simulations needed. The inspectability metric has been in development in collaboration with industry partners to ensure compatibility with modern simulation tools and aerospace design workflows. In this study, we will demonstrate how the inspectability metric is able to determine the inspectability of multiple types of structures, including aerospace composites and additively manufactured parts. We will then show how the inspectability score can be plugged into existing design optimization tasks, such as structural sizing algorithms or design for manufacturing (DFM) frameworks.

Design for inspection↗
NASA NTRS1977

New model framework and structure and the commonality evaluation model

The development of a framework and structure for shuttle era unmanned spacecraft projects and the development of a commonality evaluation model is documented. The methodology developed for model utilization in performing cost trades and comparative evaluations for commonality studies is discussed. The model framework consists of categories of activities associated with the spacecraft system's development process. The model structure describes the physical elements to be treated as separate identifiable entities. Cost estimating relationships for subsystem and program-level components were calculated.

Source record↗
NASA NTRS2011

Advancing Aeronautics: A Decision Framework for Selecting Research Agendas

Publicly funded research has long played a role in the development of aeronautics, ranging from foundational research on airfoils to development of the air-traffic control system. Yet more than a century after the research and development of successful controlled, sustained, heavier-than-air flight vehicles, there are questions over the future of aeronautics research. The field of aeronautics is relatively mature, technological developments within it have become more evolutionary, and funding decisions are sometimes motivated by the continued pursuit of these evolutionary research tracks rather than by larger factors. These developments raise questions over whether public funding of aeronautics research continues to be appropriate or necessary and at what levels. Tightened federal budgets and increasing calls to address other public demands make these questions sharper still. To help it address the questions of appropriate directions for publicly funded aeronautics research, the National Aeronautics and Space Administration's (NASA's) Aeronautics Research Mission Directorate (ARMD) asked the RAND Corporation to assess the elements required to develop a strategic view of aeronautics research opportunities; identify candidate aeronautic grand challenges, paradigms, and concepts; outline a framework for evaluating them; and exercise the framework as an example of how to use it. Accordingly, this research seeks to address these questions: What aeronautics research should be supported by the U.S. government? What compelling and desirable benefits drive government-supported research? How should the government--especially NASA--make decisions about which research to support? Advancing aeronautics involves broad policy and decisionmaking challenges. Decisions involve tradeoffs among competing perspectives, uncertainties, and informed judgment.

AERONAUTICS↗
NASA NTRS2011

Theoretically-Driven Infrastructure for Supporting Healthcare Teams Training at a Military Treatment Facility

The Team Resource Center (TRC) at Naval Medical Center Portsmouth (NMCP) currently hosts a tri-service healthcare teams training course three times annually . The course consists of didactic learning coupled with simulation exercises to provide an interactive educational experience for healthcare professionals. The course is also the foundation of a research program designed to explore the use of simulation technologies for enhancing team training and evaluation. The TRC has adopted theoretical frameworks for evaluating training readiness and efficacy, and is using these frameworks to guide a systematic reconfiguration of the infrastructure supporting healthcare teams training and research initiatives at NMCP.

Turner, Robert T.↗
NASA NTRS2015

A Framework for Orbital Performance Evaluation in Distributed Space Missions for Earth Observation

Distributed Space Missions (DSMs) are gaining momentum in their application to earth science missions owing to their unique ability to increase observation sampling in spatial, spectral and temporal dimensions simultaneously. DSM architectures have a large number of design variables and since they are expected to increase mission flexibility, scalability, evolvability and robustness, their design is a complex problem with many variables and objectives affecting performance. There are very few open-access tools available to explore the tradespace of variables which allow performance assessment and are easy to plug into science goals, and therefore select the most optimal design. This paper presents a software tool developed on the MATLAB engine interfacing with STK, for DSM orbit design and selection. It is capable of generating thousands of homogeneous constellation or formation flight architectures based on pre-defined design variable ranges and sizing those architectures in terms of predefined performance metrics. The metrics can be input into observing system simulation experiments, as available from the science teams, allowing dynamic coupling of science and engineering designs. Design variables include but are not restricted to constellation type, formation flight type, FOV of instrument, altitude and inclination of chief orbits, differential orbital elements, leader satellites, latitudes or regions of interest, planes and satellite numbers. Intermediate performance metrics include angular coverage, number of accesses, revisit coverage, access deterioration over time at every point of the Earth's grid. The orbit design process can be streamlined and variables more bounded along the way, owing to the availability of low fidelity and low complexity models such as corrected HCW equations up to high precision STK models with J2 and drag. The tool can thus help any scientist or program manager select pre-Phase A, Pareto optimal DSM designs for a variety of science goals without having to delve into the details of the engineering design process.

Earth Science↗
NASA NTRS2022

Resiliency in Future Cislunar Space Architectures

This work introduces and explores the concept of resiliency as it relates to future cislunar space architectures by 1) citing examples of its growing demand across government; 2) describing potential characteristics of resilient systems; 3) introducing a framework for evaluating the linkages between resilient capabilities and visions for future cislunar architectures; and 4) exercising the framework to identify and evaluate resiliency-enabling technical capabilities for cislunar space architectures. We assert that resiliency can emerge from a layered approach of deliberately chosen capabilities with overlap and flexibility that, in aggerate, result in a resilient system. The challenge is to identify capabilities that contribute to resiliency and to accurately characterize their value. Resiliency is discussed through the lens of future architecture planning, outlining how the National Aeronautics and Space Administration (NASA) can benefit from a shift in approach when transitioning focus to the cislunar environment.

Jason Hay↗
NASA NTRSDate not supplied

Complementary Roles of LEO and GEO IR Sounders in NWP Evaluated using an OSSE Framework

The United States is currently preparing for the next-generation weather satellite program. Building off the multi-decadal successes of the Low-Earth Orbit (LEO) infrared (IR) sounder program, NOAA and NASA are planning for the Geostationary eXtended Observations (GeoXO) Sounder (GXS) that will join international counterparts to form a global ring from geosynchronous orbit (GEO). To address questions about the complementarities and roles of these two platform types in the context of numerical weather prediction (NWP) and forecast accuracy, NASA’s Global Modeling and Assimilation Office (GMAO) Observing System Simulation Experiment (OSSE) framework was used to test their impact individually and in concert. Analyses, forecasts, and forecast sensitivity-based observations impact (FSOI) are examined. Overall, each has a role in forecast error reduction with the most beneficial impacts occurring when both LEO and GEO sounders are used together to address the needs of global skill and targeted regional weather phenomena.

Erica Mcgrath-Spangler↗
NASA NTRS2019

Evaluating Biosignatures for Life Detection

The goal here is to develop a conceptual framework for evaluating the ability of different biosignatures to provide evidence for the presence of life in planned missions or observational studies. The focus is on intrinsic characteristics of biosignatures in a given space environment rather than on their detection, which depends on the current state of technology. Suitable evaluation procedures are based on an extensive body of previous work on related problems in which different options are evaluated to make decisions in business, engineering, medical fields and social or political arena. In these cases, three approaches have proven to be particularly useful. Two of them, signal detection theory (SDT) and Bayesian hypothesis testing are based on probabilities. The third approach, multi-attribute utility (MAU) is based on utility theory.

Pohorille, Andrew↗
NASA NTRS2011

Land Surface Verification Toolkit (LVT) - A Generalized Framework for Land Surface Model Evaluation

Model evaluation and verification are key in improving the usage and applicability of simulation models for real-world applications. In this article, the development and capabilities of a formal system for land surface model evaluation called the Land surface Verification Toolkit (LVT) is described. LVT is designed to provide an integrated environment for systematic land model evaluation and facilitates a range of verification approaches and analysis capabilities. LVT operates across multiple temporal and spatial scales and employs a large suite of in-situ, remotely sensed and other model and reanalysis datasets in their native formats. In addition to the traditional accuracy-based measures, LVT also includes uncertainty and ensemble diagnostics, information theory measures, spatial similarity metrics and scale decomposition techniques that provide novel ways for performing diagnostic model evaluations. Though LVT was originally designed to support the land surface modeling and data assimilation framework known as the Land Information System (LIS), it also supports hydrological data products from other, non-LIS environments. In addition, the analysis of diagnostics from various computational subsystems of LIS including data assimilation, optimization and uncertainty estimation are supported within LVT. Together, LIS and LVT provide a robust end-to-end environment for enabling the concepts of model data fusion for hydrological applications. The evolving capabilities of LVT framework are expected to facilitate rapid model evaluation efforts and aid the definition and refinement of formal evaluation procedures for the land surface modeling community.

Kumar, Sujay V.↗
NASA NTRS1983

Investigation of the applicability of using the triple redundant hydrogen sensor for methane sensing

Application specifications for the methane sensor were assembled and design guidelines, development goals and evaluation criteria were formulated. This was done to provide a framework to evaluate sensor performance and any design adjustments to the preprototype sensor that could be required to provide methane sensitivity. Good response to hydrogen was experimentally established for four hydrogen sensor elements to be later evaluated for methane response. Prior results were assembled and analyzed for other prototype hydrogen sensor performance parameters to form a comparison base. The four sensor elements previously shown to have good hydrogen response were experimentally evaluated for methane response in 2.5% methane-in-air. No response was obtained for any of the elements, despite the high methane concentration used (50% of the Lower Flammability Limit). It was concluded that the preprototype sensing elements were insensitive to methane and were hydrogen specific. Alternative sensor operating conditions and hardware design changes were considered to provide methane sensitivity to the preprototype sensor, including a variety of different methane sensing techniques. Minor changes to the existing sensor elements, sensor geometry and operating conditions will not make the preprototype hydrogen sensor respond to methane. New sensor elements that will provide methane and hydrogen sensitivity require replacement of the existing thermistor type elements. Some hydrogen sensing characteristics of the modified sensor will be compromised (larger in situ calibration gas volume and H2 nonspecificity). The preprototype hydrogen sensor should be retained for hydrogen monitoring and a separate methane sensor should be developed.

Lantz, J. B.↗
NASA NTRS2016

Regime-Based Evaluation of Cloudiness in CMIP5 Models

The concept of Cloud Regimes (CRs) is used to develop a framework for evaluating the cloudiness of 12 fifth Coupled Model Intercomparison Project (CMIP5) models. Reference CRs come from existing global International Satellite Cloud Climatology Project (ISCCP) weather states. The evaluation is made possible by the implementation in several CMIP5 models of the ISCCP simulator generating for each gridcell daily joint histograms of cloud optical thickness and cloud top pressure. Model performance is assessed with several metrics such as CR global cloud fraction (CF), CR relative frequency of occurrence (RFO), their product (long-term average total cloud amount [TCA]), cross-correlations of CR RFO maps, and a metric of resemblance between model and ISCCP CRs. In terms of CR global RFO, arguably the most fundamental metric, the models perform unsatisfactorily overall, except for CRs representing thick storm clouds. Because model CR CF is internally constrained by our method, RFO discrepancies yield also substantial TCA errors. Our findings support previous studies showing that CMIP5 models underestimate cloudiness. The multi-model mean performs well in matching observed RFO maps for many CRs, but is not the best for this or other metrics. When overall performance across all CRs is assessed, some models, despite their shortcomings, apparently outperform Moderate Resolution Imaging Spectroradiometer (MODIS) cloud observations evaluated against ISCCP as if they were another model output. Lastly, cloud simulation performance is contrasted with each model's equilibrium climate sensitivity (ECS) in order to gain insight on whether good cloud simulation pairs with particular values of this parameter.

cloud climatology↗
NASA NTRSDate not supplied

Combinatorial Auction-Based Strategic Deconfliction of Federated UTM Airspace

Unmanned Aerial Vehicles (UAVs) have become commonly used to perform a wide range of commercial activities such as cinematography and medical supply delivery. Consequently, regulators have become interested in designing UAV Traffic Management systems (UTMs) to coordinate UAV traffic among a collection of UAV operators. One framework which has been recently proposed for a UTM system is a combinatorial auction. In this framework, airspace is modelled as a 4D grid of space-time cells. UAV operators bid on cells which collectively form paths for their UAVs. Ideally, an airspace auction should reveal information about the current price of flight paths to bidders, allowing bidders to identify and bid on a select number of paths instead of placing as many bids as possible in the hopes of stumbling on a cheap path. Revealing too much information, however, can allow bad actors to place bids which are intended not to win but to raise the price that a rival bidder must pay. We address these twin challenges with a new information revelation framework which provides bidders with wide-ranging pricing information while suppressing bad actors. We evaluate our framework on scenarios based on a Japan Aerospace Exploration Agency (JAXA) case study and find that it can scale to thousands of bids.

Christopher J C Leet↗
NASA NTRS2024

Harmonizing Food Systems Emissions Accounting for More Effective Climate Action

Food systems—encompassing activities in food production, land-use change, supply chains and waste management—contribute significantly to climate change. Recent estimates indicate that food systems produce over 30% of annual anthropogenic greenhouse gas (GHG) emissions (about 20% of CO 2 , 50% of CH 4 , and 75% of N 2 O), with the Intergovernmental Panel on Climate Change (IPCC) estimating a notably broad range of 23%–42% of global GHG emissions. This paper synthesizes current research on the contributions of food systems to climate change, highlights challenges in quantifying their impact and proposes a harmonized accounting framework for more effective climate action. We recommend that an expert committee aligned with the IPCC develop guidance for food systems emissions accounting in four key areas, including: (1) defining system boundaries and nomenclature; (2) developing protocols to allocate broader sectoral emissions to food systems; (3) prioritizing critical areas for research into activity data and emissions factors; and (4) developing a balanced framework for evaluating the impact of mitigation interventions in light of other food systems imperatives. The committee should be integrated into two key international policy processes—the United Nations Framework Convention on Climate Change and the United Nations Food Systems Summit—to support coordinated action towards global net-zero goals. Guidance from the committee could significantly improve the ability of governments, companies, and researchers to estimate, report, monitor and ultimately reduce the climate impacts of food systems.

GHG accounting↗