Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

An Observation-Driven Framework for Modeling Post-Fire Hydrologic Response: Evaluation for Two Central California Case Studies

In a warming climate, wildfires are becoming increasingly common, especially in semi-arid environments. Wildfires can disrupt forest ecosystems and induce changes to the land surface. Collectively, these impacts can alter the hydrologic response of a catchment following a fire, resulting in increased potential for surface runoff, reduced evapotranspiration, and, ultimately, a higher risk for flash flooding and mass wasting. The timescale of post-fire recovery of hydrological processes to return to pre-fire conditions is not well established due to the lack of ground measurements. Accurate characterization of the impacts of fire on hydrologic response is also challenging to simulate, given the complex interplay of various processes. Here, we present a generalized framework to quantify the impacts of wildfire on runoff generation. We consider the disturbances in the vegetation and soil as the two main factors contributing to post-fire floods. Using an ensemble modeling structure to account for parameter uncertainty, remotely sensed leaf area index (LAI) is assimilated into a land surface model (LSM) to simulate vegetation disturbance, and the maximum land surface saturation LSM parameter is decreased to parameterize the soil disturbance following observed fires. We consider the impacts of fire-induced changes to LAI and soil saturation on hydrologic states like runoff and evapotranspiration for two case studies. These case studies demonstrate the general applicability of hydrophobicity formulation to serve as a guideline for exploring the range of hydrologic responses post-fire.

Fire Hydrology↗

An Integrated Approach to Life Cycle Analysis

Life Cycle Analysis (LCA) is the evaluation of the impacts that design decisions have on a system and provides a framework for identifying and evaluating design benefits and burdens associated with the life cycles of space transportation systems from a "cradle-to-grave" approach. Sometimes called life cycle assessment, life cycle approach, or "cradle to grave analysis", it represents a rapidly emerging family of tools and techniques designed to be a decision support methodology and aid in the development of sustainable systems. The implementation of a Life Cycle Analysis can vary and may take many forms; from global system-level uncertainty-centered analysis to the assessment of individualized discriminatory metrics. This paper will focus on a proven LCA methodology developed by the Systems Analysis and Concepts Directorate (SACD) at NASA Langley Research Center to quantify and assess key LCA discriminatory metrics, in particular affordability, reliability, maintainability, and operability. This paper will address issues inherent in Life Cycle Analysis including direct impacts, such as system development cost and crew safety, as well as indirect impacts, which often take the form of coupled metrics (i.e., the cost of system unreliability). Since LCA deals with the analysis of space vehicle system conceptual designs, it is imperative to stress that the goal of LCA is not to arrive at the answer but, rather, to provide important inputs to a broader strategic planning process, allowing the managers to make risk-informed decisions, and increase the likelihood of meeting mission success criteria.

Chytka, T. M.↗

Standalone Hazard Evaluation and Refinement From Instrument Findings (S.H.E.R.I.F.)

The Standalone Hazard Evaluation From Instrument Findings (SHERIF) system is a set of novel algorithms and associated framework designed to support the generation of Digital Elevation Maps (DEMs) from multiple LiDAR scans and perform Hazard Detection (HD) and Safe Site Identification (SSI) with no dependencies on other onboard systems. SHERIF can employ several techniques to perform robust 3D keypoint extraction and Point Cloud registration (PCR) on disparate LiDAR scans of a planetary surface to generate a DEM which evolves over the course of a trajectory. The framework also supports a variety of Hazard Detection and Safe Site Identification algorithms which can be applied to the evolving DEM being produced. SHERIF features a robust and modular construction, allowing the user a high degree of flexibility in selecting and implementing whichever keypoint identificaiton, PCR and HD/SSI algorithms they may prefer, while maintaining the data products and sensor independence of the core SHERIF framework. SHERIF was recently evaluated via simulation and hardware-in-the-loop experimental testing at NASA Johnson Space Center.

hazard detection↗

Standalone Hazard Evaluation and Refinement From Instrument Findings (S.H.E.R.I.F.)

The Standalone Hazard Evaluation From Instrument Findings (SHERIF) system is a set of novel algorithms and associated framework designed to support the generation of Digital Elevation Maps (DEMs) from multiple LiDAR scans and perform Hazard Detection (HD) and Safe Site Identification (SSI) with no dependencies on other onboard systems. SHERIF can employ several techniques to perform robust 3D keypoint extraction and Point Cloud registration (PCR) on disparate LiDAR scans of a planetary surface to generate a DEM which evolves over the course of a trajectory. The framework also supports a variety of Hazard Detection and Safe Site Identification algorithms which can be applied to the evolving DEM being produced. SHERIF features a robust and modular construction, allowing the user a high degree of flexibility in selecting and implementing whichever keypoint identificaiton, PCR and HD/SSI algorithms they may prefer, while maintaining the data products and sensor independence of the core SHERIF framework. SHERIF was recently evaluated via simulation and hardware-in-the-loop experimental testing at NASA Johnson Space Center.

hazard detection↗

First International Diagnosis Competition - DXC'09

A framework to compare and evaluate diagnosis algorithms (DAs) has been created jointly by NASA Ames Research Center and PARC. In this paper, we present the first concrete implementation of this framework as a competition called DXC 09. The goal of this competition was to evaluate and compare DAs in a common platform and to determine a winner based on diagnosis results. 12 DAs (model-based and otherwise) competed in this first year of the competition in 3 tracks that included industrial and synthetic systems. Specifically, the participants provided algorithms that communicated with the run-time architecture to receive scenario data and return diagnostic results. These algorithms were run on extended scenario data sets (different from sample set) to compute a set of pre-defined metrics. A ranking scheme based on weighted metrics was used to declare winners. This paper presents the systems used in DXC 09, description of faults and data sets, a listing of participating DAs, the metrics and results computed from running the DAs, and a superficial analysis of the results.

Kurtoglu, tolga↗

Advances in Landslide Hazard Forecasting: Evaluation of Global and Regional Modeling Approach

A prototype global satellite-based landslide hazard algorithm has been developed to identify areas that exhibit a high potential for landslide activity by combining a calculation of landslide susceptibility with satellite-derived rainfall estimates. A recent evaluation of this algorithm framework found that while this tool represents an important first step in larger-scale landslide forecasting efforts, it requires several modifications before it can be fully realized as an operational tool. The evaluation finds that the landslide forecasting may be more feasible at a regional scale. This study draws upon a prior work's recommendations to develop a new approach for considering landslide susceptibility and forecasting at the regional scale. This case study uses a database of landslides triggered by Hurricane Mitch in 1998 over four countries in Central America: Guatemala, Honduras, EI Salvador and Nicaragua. A regional susceptibility map is calculated from satellite and surface datasets using a statistical methodology. The susceptibility map is tested with a regional rainfall intensity-duration triggering relationship and results are compared to global algorithm framework for the Hurricane Mitch event. The statistical results suggest that this regional investigation provides one plausible way to approach some of the data and resolution issues identified in the global assessment, providing more realistic landslide forecasts for this case study. Evaluation of landslide hazards for this extreme event helps to identify several potential improvements of the algorithm framework, but also highlights several remaining challenges for the algorithm assessment, transferability and performance accuracy. Evaluation challenges include representation errors from comparing susceptibility maps of different spatial resolutions, biases in event-based landslide inventory data, and limited nonlandslide event data for more comprehensive evaluation. Additional factors that may improve algorithm performance accuracy include incorporating additional triggering factors such as tectonic activity, anthropogenic impacts and soil moisture into the algorithm calculation. Despite these limitations, the methodology presented in this regional evaluation is both straightforward to calculate and easy to interpret, making results transferable between regions and allowing findings to be placed within an inter-comparison framework. The regional algorithm scenario represents an important step in advancing regional and global-scale landslide hazard assessment and forecasting.

Kirschbaum, Dalia B.↗

Towards a comprehensive framework for reuse: A reuse-enabling software evolution environment

Reuse of products, processes and knowledge will be the key to enable the software industry to achieve the dramatic improvement in productivity and quality required to satisfy the anticipated growing demand. Although experience shows that certain kinds of reuse can be successful, general success has been elusive. A software life-cycle technology which allows broad and extensive reuse could provide the means to achieving the desired order-of-magnitude improvements. The scope of a comprehensive framework for understanding, planning, evaluating and motivating reuse practices and the necessary research activities is outlined. As a first step towards such a framework, a reuse-enabling software evolution environment model is introduced which provides a basis for the effective recording of experience, the generalization and tailoring of experience, the formalization of experience, and the (re-)use of experience.

Basili, V. R.↗

An Evaluation of the High Level Architecture (HLA) as a Framework for NASA Modeling and Simulation

The High Level Architecture (HLA) is a current US Department of Defense and an industry (IEEE-1516) standard architecture for modeling and simulations. It provides a framework and set of functional rules and common interfaces for integrating separate and disparate simulators into a larger simulation. The goal of the HLA is to reduce software costs by facilitating the reuse of simulation components and by providing a runtime infrastructure to manage the simulations. In order to evaluate the applicability of the HLA as a technology for NASA space mission simulations, a Simulations Group at Goddard Space Flight Center (GSFC) conducted a study of the HLA and developed a simple prototype HLA-compliant space mission simulator. This paper summarizes the prototyping effort and discusses the potential usefulness of the HLA in the design and planning of future NASA space missions with a focus on risk mitigation and cost reduction.

Reid, Michael R.↗

A Decision Support Framework for Feasibility Analysis of International Space Station (ISS) Research Capability Enhancing Options

The assembly and operation of the ISS has generated significant challenges that have ultimately impacted resources available to the program's primary mission: research. To address this, program personnel routinely perform trade-off studies on alternative options to enhance research. The approach, content level of analysis and resulting outputs of these studies vary due to many factors, however, complicating the Program Manager's job of selecting the best option. To address this, the program requested a framework be developed to evaluate multiple research-enhancing options in a thorough, disciplined and repeatable manner, and to identify the best option on the basis of cost, benefit and risk. The resulting framework consisted of a systematic methodology and a decision-support toolset. The framework provides quantifiable and repeatable means for ranking research-enhancing options for the complex and multiple-constraint domain of the space research laboratory. This paper describes the development, verification and validation of this framework and provides observations on its operational use.

Ortiz, James N.↗

Exploration Architecture Options - ECLSS, TCS, EVA Implications

Many options for exploration of space have been identified and evaluated since the Vision for Space Exploration (VSE) was announced in 2004. The Augustine Commission evaluated human space flight for the Obama administration then the Human Exploration Framework Teams (HEFT and HEFT2) evaluated potential exploration missions and the infrastructure and technology needs for those missions. Lunar architectures have been identified and addressed by the Lunar Surface Systems team to establish options for how to get to, and then inhabit and explore, the moon. This paper will evaluate the options for exploration of space for the implications of architectures on the Environmental Control and Life Support (ECLSS), Thermal Control (TCS), and Extravehicular Activity (EVA) Systems.

Chambliss, Joe↗

Modified Universal Design Survey: Enhancing Operability of Launch Vehicle Ground Crew Worksites

Operability is a driving requirement for next generation space launch vehicles. Launch site ground operations include numerous operator tasks to prepare the vehicle for launch or to perform preflight maintenance. Ensuring that components requiring operator interaction at the launch site are designed for optimal human use is a high priority for operability. To promote operability, a Design Quality Evaluation Survey based on Universal Design framework was developed to support Human Factors Engineering (HFE) evaluation for NASA s launch vehicles. Universal Design per se is not a priority for launch vehicle processing however; applying principles of Universal Design will increase the probability of an error free and efficient design which promotes operability. The Design Quality Evaluation Survey incorporates and tailors the seven Universal Design Principles and adds new measures for Safety and Efficiency. Adapting an approach proven to measure Universal Design Performance in Product, each principle is associated with multiple performance measures which are rated with the degree to which the statement is true. The Design Quality Evaluation Survey was employed for several launch vehicle ground processing worksite analyses. The tool was found to be most useful for comparative judgments as opposed to an assessment of a single design option. It provided a useful piece of additional data when assessing possible operator interfaces or worksites for operability.

Blume, Jennifer L.↗

Framework for Architecture Trade Study Using MBSE and Performance Simulation

Increasing complexity in modern systems as well as cost and schedule constraints require a new paradigm of system engineering to fulfill stakeholder needs. Challenges facing efficient trade studies include poor tool interoperability, lack of simulation coordination (design parameters) and requirements flowdown. A recent trend toward Model Based System Engineering (MBSE) includes flexible architecture definition, program documentation, requirements traceability and system engineering reuse. As a new domain MBSE still lacks governing standards and commonly accepted frameworks. This paper proposes a framework for efficient architecture definition using MBSE in conjunction with Domain Specific simulation to evaluate trade studies. A general framework is provided followed with a specific example including a method for designing a trade study, defining candidate architectures, planning simulations to fulfill requirements and finally a weighted decision analysis to optimize system objectives.

Ryan, Jessica↗

Identifying and Analyzing Uncertainty Structures in the TRMM Microwave Imager Precipitation Product over Tropical Ocean Basins

Despite continuous improvements in microwave sensors and retrieval algorithms, our understanding of precipitation uncertainty is quite limited, due primarily to inconsistent findings in studies that compare satellite estimates to in situ observations over different parts of the world. This study seeks to characterize the temporal and spatial properties of uncertainty in the Tropical Rainfall Measuring Mission Microwave Imager surface rainfall product over tropical ocean basins. Two uncertainty analysis frameworks are introduced to qualitatively evaluate the properties of uncertainty under a hierarchy of spatiotemporal data resolutions. The first framework (i.e. 'climate method') demonstrates that, apart from random errors and regionally dependent biases, a large component of the overall precipitation uncertainty is manifested in cyclical patterns that are closely related to large-scale atmospheric modes of variability. By estimating the magnitudes of major uncertainty sources independently, the climate method is able to explain 45-88% of the monthly uncertainty variability. The percentage is largely resolution dependent (with the lowest percentage explained associated with a 1 deg x 1 deg spatial/1 month temporal resolution, and highest associated with a 3 deg x 3 deg spatial/3 month temporal resolution). The second framework (i.e. 'weather method') explains regional mean precipitation uncertainty as a summation of uncertainties associated with individual precipitation systems. By further assuming that self-similar recurring precipitation systems yield qualitatively comparable precipitation uncertainties, the weather method can consistently resolve about 50 % of the daily uncertainty variability, with only limited dependence on the regions of interest.

Climate↗

Comparison of Human Exploration Architecture and Campaign Approaches

As part of an overall focus on space exploration, National Aeronautics and Space Administration (NASA) continues to evaluate potential approaches for sending humans beyond low Earth orbit (LEO). In addition, various external organizations are studying options for beyond LEO exploration. Recent studies include NASA's Evolvable Mars Campaign and Design Reference Architecture (DRA) 5.0, JPL's Minimal Mars Architecture; the Inspiration Mars mission; the Mars One campaign; and the Global Exploration Roadmap (GER). Each of these potential exploration constructs applies unique methods, architectures, and philosophies for human exploration. It is beneficial to compare potential approaches in order to better understand the range of options available for exploration. Since most of these studies were conducted independently, the approaches, ground rules, and assumptions used to conduct the analysis differ. In addition, the outputs and metrics presented for each construct differ substantially. This paper will describe the results of an effort to compare and contrast the results of these different studies under a common set of metrics. The paper will first present a summary of each of the proposed constructs, including a description of the overall approach and philosophy for exploration. Utilizing a common set of metrics for comparison, the paper will present the results of an evaluation of the potential benefits, critical challenges, and uncertainties associated with each construct. The analysis framework will include a detailed evaluation of key characteristics of each construct. These will include but are not limited to: a description of the technology and capability developments required to enable the construct and the uncertainties associated with these developments; an analysis of significant operational and programmatic risks associated with that construct; and an evaluation of the extent to which exploration is enabled by the construct, including the destinations visited and the exploration capabilities provided at those destinations. Based upon the comparison of constructs, the paper will identify trends and lessons learned across all of the candidate studies.

Goodliff, Kandyce↗

Investigation of Bipropellant Plume-Induced Contamination Effects on Coverglass Materials

Contamination and degradation of external spacecraft materials by unburned and partially combusted species from bipropellant thruster plumes has long been observed as a key component of the induced space environment. Space shuttle flight experiments and returned flight hardware from the International Space Station (ISS) have both experienced microscopic impact features induced by high-velocity thruster plume droplets. Analytical results have shown that droplet impingement angle relative to a receiving surface plays a key role in the surface damage. Although impacts with normal impingement angles contribute more severely to surface degradation than highly oblique angles, surface effects at higher impingement angles should not be dismissed. Thruster plume-induced materials degradation is a complex phenomenon that depends on a variety of parameters, including but not limited to material type, system temperature and pressure, plume composition, and thruster firing specifications such as number of pulses, pulse duration, and sample distance from the thruster. For space applications, attaining the vacuum pressure and temperature conditions necessary for flight-like plume expansion and exposure conditions is not a trivial task. The German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt, DLR) is a facility uniquely capable of simulating such conditions. Test coupons were exposed to bipropellant thruster firings under high vacuum at the DLR facility. Percent area coverage (PAC) and droplet size distributions were evaluated for the uncoated and coated solar array coverglass materials over a range of impingement angles (0̊ to 75̊). A post-test imaging workflow was developed that aimed to quantify changes in sample surface morphology obtained from scanning electron microscopy (SEM) images using the Image Processing and Analysis in Java (ImageJ) tool; an opensource image processing software. The goal was to create a framework through which to evaluate the effect of bipropellant-induced PAC and droplet size distribution on solar array coverglass optical transmission losses. Understanding this relationship is important because optical transmission losses are known to lead to current reduction in solar power generation systems. In addition to the development of surface characterization workflows, valuable lessons learned as they pertain to future investigations and experiments will be discussed. The authors hope that sharing these lessons will facilitate more utilization of DLR’s unique capabilities as well as open the conversation for how best to address experimental characterization of flight-like plume expansion and its impacts on materials surface degradation effects.

Gateway↗

A comparison of microwave techniques for measuring rainfall

The purpose of this paper is to evaluate within a common framework a large number of techniques for nearly instantaneous microwave measurements of rainfall and to determine the range of rainfall rates best suited to the various techniques and estimators. The physical basis of a technique as transformed by measurement imperfections of real instruments determines the ultimate performance capability of any microwave rain estimator. While many of the defects of the measurement process apply to all techniques, the physics behind each estimator differs. A method is presented for objectively evaluating the physical bases of the techniques and for quantifying estimator performance for perfect instruments. These results are then tempered by standard measurement errors to yield more realistic results. Analysis demonstrates that, in general, the minimization of rainfall estimate errors over a wide range of rainfall rates requires the simultaneous application of more than one microwave rainfall measurement technique.

Jameson, A. R.↗

Open Data Integration (ODIN): A Concurrent, Distributed Message-Based Architecture and Framework for Disaster Response

The Runtime for Airspace Concept Evaluation (RACE) is an open-source software architecture and framework to build configurable, highly concurrent and distributed message-based systems that offer scalable, low-latency performance on commodity hardware. RACE was used in commercial aviation applications to rapidly build systems that span several machines (including synchronized displays), interface existing hardware simulators and other live data feeds, and incorporate sophisticated visualization components such as NASA WorldWind. These RACE applications validated elements of the FAA’s System Wide Information Management (SWIM) Program, handling up to 1000 messages/sec from diverse sources (SFDPS, TFM-DATA, TAIS, ASDE-X, ITWS and local ADS) for 4,500 simultaneous flights tracked in the next-generation air transportation system’s digital backbone. We have since generalized RACE to support Open Data Integration (ODIN) applications outside aviation. Systems built with RACE/ODIN can be deployed in the field, on commodity hardware, and operate with limited or intermittent connectivity to the outside world. Our primary use case is a web-server with local/persistent data storage that runs within and only serves the stakeholder network (e.g. an incident command post). We are tailoring the RACE/ODIN system to support wildland fire management for the upcoming NASA Wildland Fire Safety Demonstration Series. RACE-ODIN is under consideration for application in the Scalable Traffic Management for Emergency Response Operations project, or STEReO, which aims to create a system that can be deployed during emergencies, to coordinate multiple elements of disaster response. Such data sources predominantly come from existing services on the internet (e.g. weather and satellite data, imported from so called "edge servers") but can also include dynamic (real-time) data from computer simulations and within the stakeholder network (such as aircraft and personnel tracking information). We will present the architecture and ODIN system demonstration incorporating local data from instrumented power-line towers, interpolated weather data and geospatial data from space-based platforms.

Joseph C Coughlan↗

Open Data Integration (ODIN): A Concurrent, Distributed Message-Based Architecture and Framework for Disaster Response

The Runtime for Airspace Concept Evaluation (RACE) is an open-source software architecture and framework to build configurable, highly concurrent and distributed message-based systems that offer scalable, low-latency performance on commodity hardware. RACE was used in commercial aviation applications to rapidly build systems that span several machines (including synchronized displays), interface existing hardware simulators and other live data feeds, and incorporate sophisticated visualization components such as NASA WorldWind. These RACE applications validated elements of the FAA’s System Wide Information Management (SWIM) Program, handling up to 1000 messages/sec from diverse sources (SFDPS, TFM-DATA, TAIS, ASDE-X, ITWS and local ADS) for 4,500 simultaneous flights tracked in the next-generation air transportation system’s digital backbone. We have since generalized RACE to support Open Data Integration (ODIN) applications outside aviation. Systems built with RACE/ODIN can be deployed in the field, on commodity hardware, and operate with limited or intermittent connectivity to the outside world. Our primary use case is a web-server with local/persistent data storage that runs within and only serves the stakeholder network (e.g. an incident command post). We are tailoring the RACE/ODIN system to support wildland fire management for the upcoming NASA Wildland Fire Safety Demonstration Series. RACE-ODIN is under consideration for application in the Scalable Traffic Management for Emergency Response Operations project, or STEReO, which aims to create a system that can be deployed during emergencies, to coordinate multiple elements of disaster response. Such data sources predominantly come from existing services on the internet (e.g. weather and satellite data, imported from so called "edge servers") but can also include dynamic (real-time) data from computer simulations and within the stakeholder network (such as aircraft and personnel tracking information). We will present the architecture and ODIN system demonstration incorporating local data from instrumented power-line towers, interpolated weather data and geospatial data from space-based platforms.

Guillaume P Brat↗