Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24

The Potential of GEO Sounders for Numerical Weather Prediction

The future of Earth observations from space is currently being decided upon in the United States. To better understand the impacts of various configurations, the Global Modeling and Assimilation Office (GMAO) observing system simulation experiment (OSSE) framework has been utilized to examine the impact of various hyperspectral infrared sounder constellations from the perspective of global numerical weather prediction (NWP). With an emphasis on the proposed NOAA/NASA Geostationary eXtended Observations (GeoXO) Sounder (GXS), planned to launch in the mid-2030s, the NWP benefit of a global ring of geostationary sounders, consistent with the WMO’s 2040 vision is evaluated. In addition, considerations extend to the future of the Low Earth Orbit (LEO) sounder program with the loss of several existing instruments and plans for future missions ongoing. The GMAO OSSE framework has examined the potential complementarities of GEO and LEO sounders in weather prediction improvement. Overall, the inclusion of both GEO and LEO sounders produces the most beneficial impact on NWP for regional and global scales and the most accurate global weather forecasts as evaluated using several metrics, including hurricanes and the forecast sensitivity observation impact (FSOI).

Erica Mcgrath-Spangler↗

A framework for assessing the adequacy and effectiveness of software development methodologies

Tools, techniques, environments, and methodologies dominate the software engineering literature, but relatively little research in the evaluation of methodologies is evident. This work reports an initial attempt to develop a procedural approach to evaluating software development methodologies. Prominent in this approach are: (1) an explication of the role of a methodology in the software development process; (2) the development of a procedure based on linkages among objectives, principles, and attributes; and (3) the establishment of a basis for reduction of the subjective nature of the evaluation through the introduction of properties. An application of the evaluation procedure to two Navy methodologies has provided consistent results that demonstrate the utility and versatility of the evaluation procedure. Current research efforts focus on the continued refinement of the evaluation procedure through the identification and integration of product quality indicators reflective of attribute presence, and the validation of metrics supporting the measure of those indicators. The consequent refinement of the evaluation procedure offers promise of a flexible approach that admits to change as the field of knowledge matures. In conclusion, the procedural approach presented in this paper represents a promising path toward the end goal of objectively evaluating software engineering methodologies.

Arthur, James D.↗

Theory underlying CRM training: Psychological issues in flight crew performance and crew coordination

What psychological theory and research can reveal about training in Cockpit Resource Management (CRM) is summarized. A framework is provided for the critical analysis of current approaches to CRM training. Background factors and definitions critical to evaluating CRM are reviewed, followed by a discussion of issues directly related to CRM training effectiveness. Some of the things not known about the optimization of crew performance and the research needed to make these efforts as effective as possible are described.

Helmreich, Robert L.↗

Functional categories for future flight deck designs

With the addition of each new system on the flight deck, the danger of increasing overall operator workload while reducing crew understanding of critical mission information exists. The introduction of more powerful onboard computers, larger databases, and the increased use of electronic display media may lead to a situation of flight deck 'sophistication' at the expense of losses in flight crew capabilities and situational awareness. To counter this potentially negative impact of new technology, research activities are underway to reassess the flight deck design process. The fundamental premise of these activities is that a human-centered, systems-oriented approach to the development of advanced civil aircraft flight decks will be required for future designs to remain ergonomically sound and economically competitive. One of the initial steps in an integrated flight deck process is to define the primary flight deck functions needed to support the mission goals of the vehicle. This would allow the design team to evaluate candidate concepts in relation to their effectiveness in meeting the functional requirements. In addition, this would provide a framework to aid in categorizing and bookkeeping all of the activities that are required to be performed on the flight deck, not just activities of the crew or of a specific system. This could then allow for a better understanding and allocation of activities in the design, an understanding of the impact of a specific system on overall system performance, and an awareness of the total crew performance requirements for the design. One candidate set of functional categories that could be used to guide an advanced flight deck design are described.

Abbott, Terence S.↗

Research in Satellite-Fiber Network Interoperability

This four part report evaluated the performance of high data rate transmission links using the ACTS satellite, and to provide a preparatory test framework for two of the space science applications that have been approved for tests and demonstrations as part of the overall ACTS program. The test plan will provide guidance and information necessary to find the optimal values of the transmission parameters and then apply these parameters to specific applications. The first part will focus on the satellite-to-earth link. The second part is a set of tests to study the performance of ATM on the ACTS channel. The third and fourth parts of the test plan will cover the space science applications, Global Climate Modeling and Keck Telescope Acquisition Modeling and Control.

Edelson, Burt↗

Comparisons of global aerosol simulations from GOCART and GMI-UMich

Global aerosol mass and optical thickness (AOT) simulated by two aerosol models (GOCART and GMI-UMich) are evaluated and the strengths and shortcomings of the model approaches are investigated. These two offline CTM models are driven by the same meteorological fields (NASA GEOS-4), use the same dynamic transport algorithm, and have the same spatial resolution. However, they differ partially in emission, dry/wet deposition, and chemistry. The simulated aerosol masses are evaluated by the surface long-term measurements and short-term field campaigns (ACE-Asia and TRACE-P). AOT are calculated using modeled aerosol mass but externally mixed for GOCART and partially internally mixed for GMI-Mich. These AOT are evaluated using satellite observations from MODIS and MISR and ground measurements from AERONET. Our work demonstrates that there are significant differences in simulated aerosol mass and AOT by the two approaches. Further investigation will be performed to evaluate each mechanism which induces the differences quantitatively by implementing GOCART-like emission, deposition, and chemistry into GMI framework.

Bian, Huisheng↗

Modeling Landslide Susceptibility Over Large Regions with Fuzzy Overlay

Landslide susceptibility mapping is most effective if detailed surface and subsurface information can be combined with authoritative landslide catalogs or a deep understanding of local conditions. However, these types of homogeneous input data and catalogs are frequently not available over large areas. In this study, we model landslide susceptibility in Central America and the Caribbean islands by combining three globally available datasets and one regional dataset with fuzzy overlay. This primarily heuristic model provides the flexibility to test a range of different contributing variables and the capability to compare landslide inventories within the model framework that vary greatly in their size, spatiotemporal scope, and collection methods. We create a regional susceptibility map and evaluate its performance using receiver operating characteristics for both continuous and binned susceptibility values. This susceptibility map forms the basis for a near-real-time landslide hazard assessment system that couples susceptibility with rainfall and soil moisture triggers to estimate potential landslide activity at a regional scale. The application of this susceptibility model at the regional scale provides a foundation for transferring the methodology to other geographic areas.

Landslide↗

Bond Line Thickness Estimation in Composite Structures Using Multiple Inspection Techniques

Imaging and other nondestructive evaluation techniques are commonly used for material characterization and defect recognition in safety critical aerospace applications, with data fusion providing the framework for uncertainty quantification in these contexts. Most commonly, forward physics-based modeling predicts the response conditioned on material properties and defect assumptions, and probabilistic methods are used to infer the hidden state of the subject of the inspection from a combination of prior information, likelihoods, and inspection data. In this paper Bayesian methods are used to estimate bond thickness in lap joints comprised of aluminum adherends using a combination of infrared thermography and ultrasound. The concept of the conflation of probability distributions is applied to combine the posterior distributions derived from thermography and ultrasound and the quality of the fused estimates are compared against the individual estimates against synthetic data that was created to mimic the inspection of a lap joint comprised of aluminum adherends.

thermal nondestructive evaluation↗

Handling Qualities Optimization for Rotorcraft Conceptual Design

Over the past decade, NASA, under a succession of rotary-wing programs has been moving towards coupling multiple discipline analyses to evaluate rotorcraft conceptual designs. Handling qualities is one of the component analyses to be included in such a future Multidisciplinary Analysis and Optimization framework for conceptual design of VTOL aircraft. Similarly, the future vision for the capability of the Concept Design and Assessment Technology Area of the U.S Army Aviation Development Directorate also includes a handling qualities component. SIMPLI-FLYD is a tool jointly developed by NASA and the U.S. Army to perform modeling and analysis for the assessment of the handling qualities of rotorcraft conceptual designs. Illustrative scenarios of a tiltrotor in forward flight and singlemain rotor helicopter at hover are analyzed using SIMPLI-FLYD and the conceptual design sizing tool, NDARC, integrated in a single process. The effects of variations of design parameters such as horizontal tail and tail rotor size were evaluated in the form of margins to fixed- and rotary-wing handling qualities metrics and the computed vehicle empty weight. The handling qualities design margins are shown to vary across the flight envelope due to both changing flight dynamic and control characteristics and changing handling qualities specification requirements. The current SIMPLI-FLYD capability and future developments are discussed in the context of an overall rotorcraft conceptual design process.

Rotorcraft Conceptual Design↗

Synergism of MODIS Aerosol Remote Sensing from Terra and Aqua

The MODerate-resolution Imaging Spectro-radiometer (MODIS) sensors, aboard the Earth Observing System (EOS) Terra and Aqua satellites, are showing excellent competence at measuring the global distribution and properties of aerosols. Terra and Aqua were launched on December 18, 1999 and May 4, 2002 respectively, with daytime equator crossing times of approximately 10:30 am and 1:30 pm respectively. Several aerosol parameters are retrieved at 10-km spatial resolution from MODIS daytime data over land and ocean surfaces. The parameters retrieved include: aerosol optical thickness (AOT) at 0.47, 0.55 and 0.66 micron wavelengths over land, and at 0.47, 0.55, 0.66, 0.87, 1.2, 1.6, and 2.1 microns over ocean; Angstrom exponent over land and ocean; and effective radii, and the proportion of AOT contributed by the small mode aerosols over ocean. Since the beginning of its operation, the quality of Terra-MODIS aerosol products (especially AOT) have been evaluated periodically by cross-correlation with equivalent data sets acquired by ground-based (and occasionally also airborne) sunphotometers, particularly those coordinated within the framework of the AErosol Robotic NETwork (AERONET). Terra-MODIS AOT data have been found to meet or exceed pre-launch accuracy expectations, and have been applied to various studies dealing with local, regional, and global aerosol monitoring. The results of these Terra-MODIS aerosol data validation efforts and studies have been reported in several scientific papers and conferences. Although Aqua-MODIS is still young, it is already yielding formidable aerosol data products, which are also subjected to careful periodic evaluation similar to that implemented for the Terra-MODIS products. This paper presents results of validation of Aqua-MODIS aerosol products with AERONET, as well as comparative evaluation against corresponding Terra-MODIS data. In addition, we show interesting independent and synergistic applications of MODIS aerosol data from both Terra and Aqua. In certain situations, this combined analysis of Terra- and Aqua-MODIS data offers an insight into the diurnal cycle of aerosol loading.

Ichoku, Charles↗

Data-driven landslide nowcasting at the global scale

Landslides affect nearly every country in the world each year. To better understand this global hazard, the Landslide Hazard Assessment for Situational Awareness (LHASA) model was developed previously. LHASA version 1 combines satellite precipitation estimates with a global landslide susceptibility map to produce a gridded map of potentially hazardous areas from 60° North-South every 3 h. LHASA version 1 categorizes the world’s land surface into three ratings: high, moderate, and low hazard with a single decision tree that first determines if the last seven days of rainfall were intense, then evaluates landslide susceptibility. LHASA version 2 has been developed with a data-driven approach. The global susceptibility map was replaced with a collection of explanatory variables, and two new dynamically varying quantities were added: snow and soil moisture. Along with antecedent rainfall, these variables modulated the response to current daily rainfall. In addition, the Global Landslide Catalog (GLC) was supplemented with several inventories of rainfall-triggered landslide events. These factors were incorporated into the machine-learning framework XGBoost, which was trained to predict the presence or absence of landslides over the period 2015–2018, with the years 2019–2020 reserved for model evaluation. As a result of these improvements, the new global landslide nowcast was twice as likely to predict the occurrence of historical landslides as LHASA version 1, given the same global false positive rate. Furthermore, the shift to probabilistic outputs allows users to directly manage the trade-off between false negatives and false positives, which should make the nowcast useful for a greater variety of geographic settings and applications. In a retrospective analysis, the trained model ran over a global domain for 5 years, and results for LHASA version 1 and version 2 were compared. Due to the importance of rainfall and faults in LHASA version 2, nowcasts would be issued more frequently in some tropical countries, such as Colombia and Papua New Guinea; at the same time, the new version placed less emphasis on arid regions and areas far from the Pacific Rim. LHASA version 2 provides a nearly real-time view of global landslide hazard for a variety of stakeholders.

XGBoos↗

Handling Qualities Optimization for Rotorcraft Conceptual Design

Over the past decade, NASA, under a succession of rotary-wing programs has been moving towards coupling multiple discipline analyses in a rigorous consistent manner to evaluate rotorcraft conceptual designs. Handling qualities is one of the component analyses to be included in a future NASA Multidisciplinary Analysis and Optimization framework for conceptual design of VTOL aircraft. Similarly, the future vision for the capability of the Concept Design and Assessment Technology Area (CD&A-TA) of the U.S Army Aviation Development Directorate also includes a handling qualities component. SIMPLI-FLYD is a tool jointly developed by NASA and the U.S. Army to perform modeling and analysis for the assessment of flight dynamics and control aspects of the handling qualities of rotorcraft conceptual designs. An exploration of handling qualities analysis has been carried out using SIMPLI-FLYD in illustrative scenarios of a tiltrotor in forward flight and single-main rotor helicopter at hover. Using SIMPLI-FLYD and the conceptual design tool NDARC integrated into a single process, the effects of variations of design parameters such as tail or rotor size were evaluated in the form of margins to fixed- and rotary-wing handling qualities metrics as well as the vehicle empty weight. The handling qualities design margins are shown to vary across the flight envelope due to both changing flight dynamic and control characteristics and changing handling qualities specification requirements. The current SIMPLI-FLYD capability and future developments are discussed in the context of an overall rotorcraft conceptual design process.

Handling Qualities↗

Performance Risk Model (PRisM) Proof-of-Concept: An Operational Decision Support Tool to Predict Crew Performance in Space from Available Performance Tests

The Crew Health and Performance-Probabilistic Risk Assessment (CHP-PRA) team at NASA Glenn Research Center has developed a range of tools to evaluate astronaut health during spaceflight and to optimize the medical set required for missions. Among these, the Performance Risk Model (PRisM) represents a novel advancement, extending CHP-PRA’s focus beyond medical systems into the domain of human performance. Such tool could be pivotal in optimizing astronaut capabilities during space travel, thereby enhancing overall mission success. PRisM leverages data from well-established performance assessments conducted during previous crewed space missions and Earth analogs to strategically predict outcomes for planned tasks, even when direct performance testing has not been conducted for those specific tasks. To evaluate performance, PRisM references the various metrics outlined in NASA-STD-3001 and employs a methodology to integrate different performance scales. This framework analyzes the contribution of various human system task categories (HSTCs) to task execution and compares these contributions to the HSTCs’ involvement in other known tests. The model further employs a Monte Carlo simulation to sample performance scores from their distribution in operationally relevant tests such as those in Mulavara et al. (2018) and, by leveraging similarities in HSTC involvement, transfers this knowledge to actual mission tasks, such as those outlined in the "Human Exploration of Mars: Preliminary List of Crew Tasks”. The current PRisM proof-of-concept includes analyses of the impact of exercise and specific medications on astronauts’ performance, with provisions to incorporate additional countermeasures as data becomes available. Furthermore, the tool is customizable to include any system necessary to fully encompass the domain of human systems and can be adapted to evaluate performance for any spaceflight activities as requested by operational stakeholders. PRisM has the potential to assist the Human Research Program in exploring the capabilities trade space for optimized crew performance.

performance modeling↗

A ROS-based Simulator for Testing the Enhanced Autonomous Navigation of the Mars 2020 Rover

In order to achieve the ambitious objectives of the Mars 2020 (M2020) mission, in particular the ability to autonomously traverse more challenging terrains more efficiently, new surface mobility software was developed for Enhanced Navigation (ENav). That decision was made early in the project, before most of the new surface flight software (FSW) existed, which created a need for a separate framework where the new navigation algorithms could be quickly prototyped and tested, before more realistic FSW-based testbeds became available. The JPL robotics team chose the Robot Operating System [1] (ROS) as the environment in which to test the new ENav algorithms. This made it possible to write the algorithms in the C language required by the FSW, so they could be directly ported over to the flight module later on, while leveraging all the C++ libraries and tools provided by ROS for simulation and testing. The ENav algorithms were developed as a separate C library, and stubs were used to replace any FSW-specific code, such as Event Reporting (EVRs) and data products (DPs). A ROS simulator was developed to generate a rich set of varied 3D terrains representative of the candidate Mars landing sites and simulate the physics of the rover motion, the point cloud perceived by the rover’s stereo vision system, and the new thinking-while-driving (TWD) navigation logic which directs the rover to drive autonomously to user-specified waypoints. To simulate the rover motion and perception, a ROS node was developed that uses a software library called HyperDrive Sim (HDSim), which is a wrapper for the Rover Sequencing and Visualization Program [2] (RSVP). That library provides roverterrain settling, realistic slip modelling, and camera rendering capability based on the rover’s NavCam machine vision models. To simulate the navigation logic, a ROS node was created that initializes and runs the ENav algorithms in a way that mimics the FSW execution, while also providing the capability to load and replay data products, including re-running the recorded inputs through the ENav algorithms for testing. An engineering Graphical User Interface (GUI) was also developed to visualize various elements, such as the rover pose during the drive, the simulated and perceived terrain, the selected local and global paths to the goal, the evaluated candidate paths and the reasons why they were rejected, the keep-in and keep-out zones (KIOZs), etc. Finally, an advanced Monte Carlo (MC) framework that can run many simulations in parallel on the Cloud and automatically generate reports that capture the key ENav performance metrics was developed to evaluate the system in a statisticallymeaningful way. This paper provides an overview of the ROSbased simulator used for testing the M2020 ENav algorithms.

Toupet, Olivier↗

Methodolgy For Evaluation Of Technology Impacts In Space Electric Power Systems

The Analysis and Management branch of the Power and Propulsion Office at NASA Glenn Research Center is responsible for performing complex analyses of the space power and In-Space propulsion products developed by GRC. This work quantifies the benefits of the advanced technologies to support on-going advocacy efforts. The Power and Propulsion Office is committed to understanding how the advancement in space technologies could benefit future NASA missions. They support many diverse projects and missions throughout NASA as well as industry and academia. The area of work that we are concentrating on is space technology investment strategies. Our goal is to develop a Monte-Carlo based tool to investigate technology impacts in space electric power systems. The framework is being developed at this stage, which will be used to set up a computer simulation of a space electric power system (EPS). The outcome is expected to be a probabilistic assessment of critical technologies and potential development issues. We are developing methods for integrating existing spreadsheet-based tools into the simulation tool. Also, work is being done on defining interface protocols to enable rapid integration of future tools. Monte Carlo-based simulation programs for statistical modeling of the EPS Model. I decided to learn and evaluate Palisade's @Risk and Risk Optimizer software, and utilize it's capabilities for the Electric Power System (EPS) model. I also looked at similar software packages (JMP, SPSS, Crystal Ball, VenSim, Analytica) available from other suppliers and evaluated them. The second task was to develop the framework for the tool, in which we had to define technology characteristics using weighing factors and probability distributions. Also we had to define the simulation space and add hard and soft constraints to the model. The third task is to incorporate (preliminary) cost factors into the model. A final task is developing a cross-platform solution of this framework.

Holda, Julie↗

Fighter agility metrics, research, and test

Proposed new metrics to assess fighter aircraft agility are collected and analyzed. A framework for classification of these new agility metrics is developed and applied. A completed set of transient agility metrics is evaluated with a high fidelity, nonlinear F-18 simulation provided by the NASA Dryden Flight Research Center. Test techniques and data reduction methods are proposed. A method of providing cuing information to the pilot during flight test is discussed. The sensitivity of longitudinal and lateral agility metrics to deviations from the pilot cues is studied in detail. The metrics are shown to be largely insensitive to reasonable deviations from the nominal test pilot commands. Instrumentation required to quantify agility via flight test is also considered. With one exception, each of the proposed new metrics may be measured with instrumentation currently available. Simulation documentation and user instructions are provided in an appendix.

Liefer, Randall K.↗

Fighter agility metrics

Fighter flying qualities and combat capabilities are currently measured and compared in terms relating to vehicle energy, angular rates and sustained acceleration. Criteria based on these measurable quantities have evolved over the past several decades and are routinely used to design aircraft structures, aerodynamics, propulsion and control systems. While these criteria, or metrics, have the advantage of being well understood, easily verified and repeatable during test, they tend to measure the steady state capability of the aircraft and not its ability to transition quickly from one state to another. Proposed new metrics to assess fighter aircraft agility are collected and analyzed. A framework for classification of these new agility metrics is developed and applied. A complete set of transient agility metrics is evaluated with a high fidelity, nonlinear F-18 simulation. Test techniques and data reduction methods are proposed. A method of providing cuing information to the pilot during flight test is discussed. The sensitivity of longitudinal and lateral agility metrics to deviations from the pilot cues is studied in detail. The metrics are shown to be largely insensitive to reasonable deviations from the nominal test pilot commands. Instrumentation required to quantify agility via flight test is also considered. With one exception, each of the proposed new metrics may be measured with instrumentation currently available.

Liefer, Randall K.↗

Multi-Disciplinary Design Optimization Using WAVE

The current preliminary design tools lack the product performance, quality and cost prediction fidelity required to design Six Sigma products. They are also frequently incompatible with the tools used in detailed design, leading to a great deal of rework and lost or discarded data in the transition from preliminary to detailed design. Thus, enhanced preliminary design tools are needed in order to produce adequate financial returns to the business. To achieve this goal, GEAE has focused on building the preliminary design system around the same geometric 3D solid model that will be used in detailed design. With this approach, the preliminary designer will no longer convert a flowpath sketch into an engine cross section but rather, automatically create 3D solid geometry for structural integrity, life, weight, cost, complexity, producibility, and maintainability assessments. Likewise, both the preliminary design and the detailed design can benefit from the use of the same preliminary part sizing routines. The design analysis tools will also be integrated with the 3D solid model to eliminate manual transfer of data between programs. GEAE has aggressively pursued the computerized control of engineering knowledge for many years. Through its study and validation of 3D CAD programs and processes, GEAE concluded that total system control was not feasible at that time. Prior CAD tools focused exclusively on detail part geometry and Knowledge Based Engineering systems concentrated on rules input and data output. A system was needed to bridge the gap between the two to capture the total system. With the introduction of WAVE Engineering from UGS, the possibilities of an engineering system control device began to formulate. GEAE decided to investigate the new WAVE functionality to accomplish this task. NASA joined GEAE in funding this validation project through Task Order No. 1. With the validation project complete, the second phase under Task Order No. 2 was established to develop an associative control structure (framework) in the UG WAVE environment enabling multi-disciplinary design of turbine propulsion systems. The capabilities of WAVE were evaluated to assess its use as a rapid optimization and productivity tool. This project also identified future WAVE product enhancements that will make the tool still more beneficial for product development.

Irwin, Keith↗