Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36

Modeling and Optimization of a High-Tc Hot-Electron Superconducting Mixer for Terahertz Applicaitons

The development of a YBa(sub 2)Cu(sub 3)O(sub 7-(kronecker delta))(YBCO) hot-electron bolometer (HEB) quasioptical mixer for a 2.5 heterodyne receiver is discussed. The modeled device is a submicron bridge made from a 10 nm thick film on a high thermal conductance substrate. The mixer performance expected for this device is analyzed in the framework of a two-temperature model which includes heating both of the electrons and the lattice. Also, the contribution of heat diffusion from the film through the substrate and from the film to the normal metal contacts is evaluated....a single sideband temperature of less than 2000k is predicted.

tetrahertz hot-electron bolometer quasioptical mix↗

Improving Rain/No-Rain Detection Skill by Merging Precipitation Estimates from Different Sources

Rain/no-rain detection error is a key source of uncertainty in regional and global precipitation products that propagates into offline hydrological and land surface modeling simulations. Such detection error is difficult to evaluate and/or filter without access to high-quality reference precipitation datasets. For cases where such access is not available, this study proposes a novel approach for improved rain/no-rain detection. Based on categorical triple collocation (CTC) and a probabilistic framework, a weighted merging algorithm (CTC-M) is developed to combine noisy, but independent, precipitation products into an optimal binary rain/no-rain time series. Compared with commonly used approaches that directly apply the best parent product for rain/no-rain detection, the superiority of CTC-M is demonstrated analytically and numerically using spatially dense precipitation measurements over Europe. Our analysis also suggests that CTC-M is tolerant to a range of cross-correlated rain/no-rain detection errors and detection biases of the parent products. As a result, CTC-M will benefit global precipitation estimation by improving the representation of precipitation occurrence in gauge-based and multisource merged precipitation products.

Jianzhi Dong↗

Smart Congestion Control for Delay- and Disruption Tolerant Networks

In this paper, we propose a novel congestion control framework for delay- and disruption tolerant networks (DTNs). The proposed framework, called Smart-DTN-CC, adjusts its operation automatically as a function of the dynamics of the underlying network. It employs reinforcement learning, a machine learning technique known to be well suited to problems in which the environment, in this case the network, plays a crucial role; yet, no prior knowledge about the target environment can be assumed, i.e., the only way to acquire information about the environment is to interact with it through continuous online learning. Smart-DTN-CC nodes get input from the environment (e.g., its buffer occupancy, set of neighbors, etc), and, based on that information, choose an action to take from a set of possible actions. Depending on an action’s effectiveness in controlling congestion, it will be given a reward. Smart-DTN-CC’s goal is to maximize the overall reward which translates to minimizing congestion. To our knowledge, Smart-DTN-CC is the first DTN congestion control framework that has the ability to automatically and continuously adapt to the dynamics of the target environment which allows Smart-DTNCC to deliver adequate performance in a variety of DTN applications and scenarios. As demonstrated by our experimental evaluation, Smart-DTN-CC is able to consistently outperform existing DTN congestion control mechanisms under a wide range of network conditions and characteristics.

Hirata, Celso M.↗

Incorporating Biological Knowledge into Evaluation of Casual Regulatory Hypothesis

Biological data can be scarce and costly to obtain. The small number of samples available typically limits statistical power and makes reliable inference of causal relations extremely difficult. However, we argue that statistical power can be increased substantially by incorporating prior knowledge and data from diverse sources. We present a Bayesian framework that combines information from different sources and we show empirically that this lets one make correct causal inferences with small sample sizes that otherwise would be impossible.

Chrisman, Lonnie↗

A Formal Approach for Designing and Evaluating Procedures

Operator interaction with modern control systems is a topic of great concern in high-risk industries such as nuclear power and commercial aviation. The issues associated with such systems focus on the ability of the operators (e.g., pilots) to achieve mission goals safely while containing failures. Operators must be able to interact safely and reliably with highly automatic and complex systems across the full spectrum of possible operating conditions, including normal, abnormal, and emergency situations. In environments such as commercial aviation, operator interaction with the machine is specified through a set of standard operating procedures (SOP). A procedure represents a collective agreement on the 'best' way to perform a given task. The intent of this paper is to suggest a formal methodology, for designing and evaluating procedures, that is both reliable and systematic. Our approach involves two major elements: a model of the machine and a list of the operator's task specifications (goals). We use formal modeling paradigms for describing the system and super-imposing on it the operator's tasks. Such paradigms, based on recent frameworks such as Statecharts and Hierarchical Hybrid Machines appear to be adequate methods for analyzing operator interaction with modern control systems. To illustrate this methodology, we model and analyze the sequence of actions for an emergency procedure. The procedure, Irregular Engine Start, for a medium-range aircraft, specifies the sequence of immediate actions that must be performed by the crew to avoid an uncontrolled rise in engine temperature during start-up. A model of engine behavior during a hot start is constructed. It also describes the various actions that can be taken by the crew and the resulting outcomes. The model is then opened up as a tree of all possible action sequences. This action tree allows us to trace the correct sequences necessary to achieve the desired end-goal (secure and shut down of the engine). In conclusion, we argue that the current process of designing and evaluating procedures can be improved. We discuss the implications of this approach for designing and evaluating this and other types of procedures. We conclude with insights about the benefits and limitation of this methodology, and offer suggestions for future research.

Degani, Asaf↗

Multiobjective optimization of hybrid regenerative life support technologies. Topic D: Technology Assessment

We developed simple mathematical models for many of the technologies constituting the water reclamation system in a space station. These models were employed for subsystem optimization and for the evaluation of the performance of individual water reclamation technologies, by quantifying their operational 'cost' as a linear function of weight, volume, and power consumption. Then we performed preliminary investigations on the performance improvements attainable by simple hybrid systems involving parallel combinations of technologies. We are developing a software tool for synthesizing a hybrid water recovery system (WRS) for long term space missions. As conceptual framework, we are employing the state space approach. Given a number of available technologies and the mission specifications, the state space approach would help design flowsheets featuring optimal process configurations, including those that feature stream connections in parallel, series, or recycles. We visualize this software tool to function as follows: given the mission duration, the crew size, water quality specifications, and the cost coefficients, the software will synthesize a water recovery system for the space station. It should require minimal user intervention. The following tasks need to be solved for achieving this goal: (1) formulate a problem statement that will be used to evaluate the advantages of a hybrid WRS over a single technology WBS; (2) model several WRS technologies that can be employed in the space station; (3) propose a recycling network design methodology (since the WRS synthesis task is a recycling network design problem, it is essential to employ a systematic method in synthesizing this network); (4) develop a software implementation for this design methodology, design a hybrid system using this software, and compare the resulting WRS with a base-case WRS; and (5) create a user-friendly interface for this software tool.

Manousiouthakis, Vasilios↗

Assessing the impact of modeling limits on intelligent systems

The knowledge bases underlying intelligent systems are validated. A general conceptual framework is provided for considering the roles in intelligent systems of models of physical, behavioral, and operational phenomena. A methodology is described for identifying limits in particular intelligent systems, and the use of the methodology is illustrated via an experimental evaluation of the pilot-vehicle interface within the Pilot's Associate. The requirements and functionality are outlined for a computer based knowledge engineering environment which would embody the approach advocated and illustrated in earlier discussions. Issues considered include the specific benefits of this functionality, the potential breadth of applicability, and technical feasibility.

Rouse, William B.↗

Assessing the impact of modeling limits on intelligent systems

The knowledge bases underlying intelligent systems are validated. A general conceptual framework is provided for considering the roles in intelligent systems of models of physical, behavioral, and operational phenomena. A methodology is described for identifing limits in particular intelligent systems, and the use of the methodology is illustrated via an experimental evaluation of the pilot-vehicle interface within the Pilot's Associate. The requirements and functionality are outlined for a computer based knowledge engineering environment which would embody the approach advocated and illustrated in earlier discussions. Issues considered include the specific benefits of this functionality, the potential breadth of applicability, and technical feasibility.

Rouse, William B.↗

Aerosol Modeling for the Global Model Initiative

The goal of this project is to develop an aerosol module to be used within the framework of the Global Modeling Initiative (GMI). The model development work will be preformed jointly by the University of Michigan and AER, using existing aerosol models at the two institutions as starting points. The GMI aerosol model will be tested, evaluated against observations, and then applied to assessment of the effects of aircraft sulfur emissions as needed by the NASA Subsonic Assessment in 2001. The work includes the following tasks: 1. Implementation of the sulfur cycle within GMI, including sources, sinks, and aqueous conversion of sulfur. Aerosol modules will be added as they are developed and the GMI schedule permits. 2. Addition of aerosol types other than sulfate particles, including dust, soot, organic carbon, and black carbon. 3. Development of new and more efficient parameterizations for treating sulfate aerosol nucleation, condensation, and coagulation among different particle sizes and types.

Weisenstein, Debra K.↗

The Super Tuesday Outbreak: Forecast Sensitivities to Single-Moment Microphysics Schemes

Forecast precipitation and radar characteristics are used by operational centers to guide the issuance of advisory products. As operational numerical weather prediction is performed at increasingly finer spatial resolution, convective precipitation traditionally represented by sub-grid scale parameterization schemes is now being determined explicitly through single- or multi-moment bulk water microphysics routines. Gains in forecasting skill are expected through improved simulation of clouds and their microphysical processes. High resolution model grids and advanced parameterizations are now available through steady increases in computer resources. As with any parameterization, their reliability must be measured through performance metrics, with errors noted and targeted for improvement. Furthermore, the use of these schemes within an operational framework requires an understanding of limitations and an estimate of biases so that forecasters and model development teams can be aware of potential errors. The National Severe Storms Laboratory (NSSL) Spring Experiments have produced daily, high resolution forecasts used to evaluate forecast skill among an ensemble with varied physical parameterizations and data assimilation techniques. In this research, high resolution forecasts of the 5-6 February 2008 Super Tuesday Outbreak are replicated using the NSSL configuration in order to evaluate two components of simulated convection on a large domain: sensitivities of quantitative precipitation forecasts to assumptions within a single-moment bulk water microphysics scheme, and to determine if these schemes accurately depict the reflectivity characteristics of well-simulated, organized, cold frontal convection. As radar returns are sensitive to the amount of hydrometeor mass and the distribution of mass among variably sized targets, radar comparisons may guide potential improvements to a single-moment scheme. In addition, object-based verification metrics are evaluated for their utility in gauging model performance and QPF variability.

Molthan, Andrew L.↗

Analysis of Control Strategies for Aircraft Flight Upset Recovery

This paper proposes a framework for studying the ability of a control strategy, consisting of a control law and a command law, to recover an aircraft from ight conditions that may extend beyond the normal ight envelope. This study was carried out (i) by evaluating time responses of particular ight upsets, (ii) by evaluating local stability over an equilibrium manifold that included stall, and (iii) by bounding the set in the state space from where the vehicle can be safely own to wings-level ight. These states comprise what will be called the safely recoverable ight envelope (SRFE), which is a set containing the aircraft states from where a control strategy can safely stabilize the aircraft. By safe recovery it is implied that the tran- sient response stays between prescribed limits before converging to a steady horizontal ight. The calculation of the SRFE bounds yields the worst-case initial state corresponding to each control strategy. This information is used to compare alternative recovery strategies, determine their strengths and limitations, and identify the most e ective strategy. In regard to the control law, the authors developed feedback feedforward laws based on the gain scheduling of multivariable controllers. In regard to the command law, which is the mechanism governing the exogenous signals driving the feed- forward component of the controller, we developed laws with a feedback structure that combines local stability and transient response considera- tions. The upset recovery of the Generic Transport Model, a sub-scale twin-engine jet vehicle developed by NASA Langley Research Center, is used as a case study.

Crespo, Luis G.↗

Sensorimotor Application of Proposed Methods to Combine the Effects of Multiple Countermeasures for PRisM

Risk associated with human systems is challenging to quantify but is critical for the mission planning and decision making required to enable future Lunar and Martian missions. To address this gap, the Crew Health and Performance-Probabilistic Risk Assessment (CHP-PRA) project is developing an integrated computational model for CHP mission risk. Much like how MEDPRAT is designed to allow medical resource trades informed by medical risk, CHP-PRA will enable analogous trades in human system risks across multiple CHP functions and capabilities. Human performance is one component of the risk intended to be captured by CHP-PRA through the Performance Risk Model (PRisM). The sensorimotor risk area provides a good frame of reference for investigating the structure of a performance model because most tasks that astronauts are expected to perform require input from the sensory system and/or movement/motor control. Additionally, sensorimotor countermeasures are an area of particular concern for NASA’s Human Research Program because of the increased sensorimotor risk associated with surface operations in Lunar and Martian missions. Thus, a tool that can quickly compare risk reductions of potential countermeasures would be beneficial in guiding research and development of effective countermeasures. In this proof of concept, we present a systematic way to combine multiple performance data sets for humans subjected to different countermeasures such that we can predict the countermeasure(s) that optimize astronaut performance on relevant tasks. PRisM assumes that both the tests that are used to measure countermeasure effectiveness (input data) and the tasks we use to represent astronaut performance, can be broken down and represented as a function/vector of the human systems required to perform that test/task. Through mathematical combination, test data are used to predict performance on astronaut tasks that use similar systems. We propose that when combining countermeasures evaluated using the same test, that only one value should be used to represent their combined effectiveness. We start our analysis with the assumption that two countermeasures together will perform better than each countermeasure individually. Our initial implementation of this framework compares various space motion sickness countermeasures and the most up to date analysis will be demonstrated at the IWS.

Caroline R Austin↗

Interface Consistency: Phase I Results & Phase II Status

Future exploration missions will rely on designing and developing vehicles and complex systems from within NASA and through multiple external commercial partners to meet mission goals. Despite existing consistency-related agency requirements, NASA’s approach to commercial spaceflight development encourages providers’ flexibility and innovation. This strategy is resulting in significant design diversity across Artemis vehicles. Design best practices and guidelines champion interface consistency to promote mental model development and knowledge transfer. However, research investigating the benefits of consistency is mixed, and little is known about its role in complex systems. Determining the level of risk that system diversity presents is difficult, as there is no established method for quantifying the degree of consistency within and across interfaces, nor is there information about the differential impacts of different types of inconsistency. Phase I of this project (Characterization and Measurement) served as a starting point to better understand the construct of consistency, its application, and the range of studies and methods for measuring it. The project team created a taxonomy of consistency to apply to interfaces as a framework to guide the development of tools to assess intersystem consistency. Checklist and cognitive walkthrough methods were developed for use by human factors (HF) and human-computer interaction (HCI) experts. The Intersystem Consistency Scale (ICS) was developed for interface evaluations with crew. A pilot study evaluated the methods’ ability to distinguish differences between Artemis-like prototype pairs exhibiting either high or low design consistency. In addition, click errors and time on task were collected within the ICS (crew-like) group. Results from our exploratory analysis and lessons learned from the pilot study will be discussed. The project team will also present the status of Phase II (Risk Assessment, Standards and Guidelines). This includes incorporating feedback to redesign the assessment tools, and inputs from displays and training Subject Matter Experts to update tasks and prototype designs. The team will present the risk assessment study design to identify the types and levels of inconsistency that pose the greatest risk to performance. Plans to apply these results toward agency standards and guideline recommendations will also be discussed.

Human-Computer Interaction↗

Beware of agents when flying aircraft: Basic principles behind a generic methodology for the evaluation and certification of advanced aviation systems

There is currently a growing interest in the aeronautical community to assess the effects of the increasing levels of automation on pilots' performance and overall safety. The first effect of automation is the change in the nature of the pilot's role on the flight deck. Pilots have become supervisors who monitor aircraft systems in usual situations and intervene only when unanticipated events occur. Instead of 'hand flying' the airplane, pilots contribute to the control of aircraft by acting as mediators, instructions given to the automation. By eliminating the need for manually controlling normal situations, such a role division has reduced the opportunities for the pilot to acquire experience and skills necessary to safely cope with abnormal events. Difficulties in assessing the state and behavior of automation arise mainly from four factors: (1) the complexity of current systems and consequence mode-related problems; (2) the intrinsic autonomy of automation which is able to fire mode transitions without explicit commands from the pilots; (3) the bad quality of feed-back from the control systems displays and interfaces to the pilots; and (4) the fact that the automation currently has no explicit representation of the current pilots' intentions and strategy. Assuming certification has among its major goals to guarantee the passengers' and pilots' safety and the airplane integrity under normal and abnormal operational conditions, the authors suggest it would be particularly fruitful to come up with a conceptual reference system providing the certification authorities both with a theoretical framework and a list of principles usable for assessing the quality of the equipment and designs under examination. This is precisely the scope of this paper. However, the authors recognize that the conceptual presented is still under development and would thus be best considered as a source of reflection for the design, evaluation and certification processes of advanced aviation technologies.

Javaux, Denis↗

The Earth Model Column Collaboratory (EMC2) v1.1: An Open-Source Ground-Based Lidar and Radar Instrument Simulator and Subcolumn Generator for Large-Scale Models

Climate models are essential for our comprehensive understanding of Earth's atmosphere and can provide critical insights on future changes decades ahead. Because of these critical roles, today's climate models are continuously being developed and evaluated using constraining observations and measurements obtained by satellites, airborne, and ground-based instruments. Instrument simulators can provide a bridge between the measured or retrieved quantities and their sampling in models and field observations while considering instrument sensitivity limitations. Here we present the Earth Model Column Collaboratory (EMC2), an open-source ground-based lidar and radar instrument simulator and subcolumn generator, specifically designed for large-scale models, in particular climate models, but also applicable to high-resolution model output. EMC2 provides a flexible framework enabling direct comparison of model output with ground-based observations, including generation of subcolumns that may statistically represent finer model spatial resolutions. In addition, EMC2 emulates ground-based (and air- or space-borne) measurements while remaining faithful to large-scale models' physical assumptions implemented in their cloud or radiation schemes. The simulator uses either single particle or bulk particle size distribution lookup tables, depending on the selected scheme approach, to perform the forward calculations. To facilitate model evaluation, EMC2 also includes three hydrometeor classification methods, namely, radar- and sounding-based cloud and precipitation detection and classification, lidar-based phase classification, and a Cloud Feedback Model Intercomparison Project Observational Simulator Package (COSP) lidar simulator emulator. The software is written in Python, is easy to use, and can be straightforwardly customized for different models, radars, and lidars. Following the description of the logic, functionality, features, and software structure of EMC2, we present a case study of highly supercooled mixed-phase cloud based on measurements from the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) West Antarctic Radiation Experiment (AWARE). We compare observations with the application of EMC2 to outputs from four configurations of the NASA Goddard Institute for Space Studies (GISS) climate model (ModelE3) in single-column model (SCM) mode and from a large-eddy simulation (LES) model. We show that two of the four ModelE3 configurations can form and maintain highly supercooled precipitating cloud for several hours, consistent with observations and LES. While our focus is on one of these ModelE3 configurations, which performed slightly better in this case study, both of these configurations and the LES results post-processed with EMC2 generally provide reasonable agreement with observed lidar and radar variables. As briefly demonstrated here, EMC2 can provide a lightweight and flexible framework for comparing the results of both large-scale and high-resolution models directly with observations, with relatively little overhead and multiple options for achieving consistency with model microphysical or radiation scheme physics.

Earth Model Column Collaboratory↗

Development of a Ground Test and Analysis Protocol for NASA's NextSTEP Phase 2 Habitation Concepts

The NASA Next Space Technologies for Exploration Partnerships (NextSTEP) program is a public-private partnership model that seeks commercial development of deep space exploration capabilities to support human spaceflight missions around and beyond cislunar space. NASA first issued the Phase 1 NextSTEP Broad Agency Announcement to U.S. industries in 2014, which called for innovative cislunar habitation concepts that leveraged commercialization plans for low-Earth orbit. These habitats will be part of the Deep Space Gateway (DSG), the cislunar space station planned by NASA for construction in the 2020s. In 2016, Phase 2 of the NextSTEP program selected five commercial partners to develop ground prototypes. A team of NASA research engineers and subject matter experts (SMEs) have been tasked with developing the ground-test protocol that will serve as the primary means by which these Phase 2 prototypes will be evaluated. Since 2008, this core test team has successfully conducted multiple spaceflight analog mission evaluations utilizing a consistent set of operational tools, methods, and metrics to enable the iterative development, testing, analysis, and validation of evolving exploration architectures, operations concepts, and vehicle designs. The purpose of implementing a similar evaluation process for the Phase 2 Habitation Concepts is to consistently evaluate different commercial partner ground prototypes to provide data-driven, actionable recommendations for Phase 3. This paper describes the process by which the ground test protocol was developed and the objectives, methods, and metrics by which the NextSTEP Phase 2 Habitation Concepts will be rigorously and systematically evaluated. The protocol has been developed using both a top-down and bottom-up approach. Top-down development began with the Human Exploration and Operations Mission Directorate (HEOMD) exploration objectives and ISS Exploration Capability Study Team (IECST) candidate flight objectives. Strategic questions and associated rationales, derived from these candidate architectural objectives, provide the framework by which the ground-test protocol will address the DSG stack elements and configurations, systems and subsystems, and habitation, science, and EVA functions. From these strategic questions, high-level functional requirements for the DSG were drafted and associated ground-test objectives and analysis protocols were established. Bottom-up development incorporated objectives from NASA SMEs in autonomy, avionics and software, communication, environmental control and life support systems, exercise, extravehicular activity, exploration medical operations, guidance navigation and control, human factors and behavioral performance, human factors and habitability, logistics, Mission Control Center operations, power, radiation, robotics, safety and mission assurance, science, simulation, structures, thermal, trash management, and vehicle health. Top-down and bottom-up objectives were integrated to form overall functional requirements - ground-test objectives and analysis mapping. From this mapping, ground-test objectives were organized into those that will be evaluated through inspection, demonstration, analysis, subsystem standalone testing, and human-in-the-loop (HITL) testing. For the HITL tests, mission-like timelines, procedures, and flight rules have been developed to directly meet ground test objectives and evaluate specific functional requirements. Data collected from these assessments will be analyzed to determine the acceptability of habitation element configurations and the combinations of capabilities that will result in the best habitation platform to be recommended by the test team for Phase 3.

Gernhardt, Michael L.↗

A review and preliminary evaluation of methodological factors in performance assessments of time-varying aircraft noise effects

The effects of aircraft noise on human performance is considered. Progress is reported in the following areas: (1) review of the literature to identify the methodological and stimulus parameters involved in the study of noise effects on human performance; (2) development of a theoretical framework to provide working hypotheses as to the effects of noise on complex human performance; and (3) data collection on the first of several experimental investigations designed to provide tests of the hypotheses.

Coates, G. D.↗

Influence of Plasma Environment on K-Line Emission in Highly Ionized Iron Atoms Evaluated Using a Debye-Huckel Model

The influence of plasma environment on the atomic parameters associated with the K-vacancy states has been investigated theoretically for several iron ions. To do this, a time-averaged Debye-Huckel potential for both the electron-nucleus and electron-electron interactions has been considered in the framework of relativistic multiconfiguration Dirac-Fock computations. More particularly, the plasma screening effects on ionization potentials, K-thresholds, transition energies, and radiative rates have been estimated in the astrophysical context of accretion disks around black holes. In the present paper, we describe the behavior of those atomic parameters for Ne-, Na-, Ar-, and K-like iron ions.

Deprince, J.↗