SEARCH · Engineering Papers

Results for “evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

All sources NASA NTRS DOE OSTI

NASA NTRS2012

Towards Reliable Evaluation of Anomaly-Based Intrusion Detection Performance

This report describes the results of research into the effects of environment-induced noise on the evaluation process for anomaly detectors in the cyber security domain. This research was conducted during a 10-week summer internship program from the 19th of August, 2012 to the 23rd of August, 2012 at the Jet Propulsion Laboratory in Pasadena, California. The research performed lies within the larger context of the Los Angeles Department of Water and Power (LADWP) Smart Grid cyber security project, a Department of Energy (DoE) funded effort involving the Jet Propulsion Laboratory, California Institute of Technology and the University of Southern California/ Information Sciences Institute. The results of the present effort constitute an important contribution towards building more rigorous evaluation paradigms for anomaly-based intrusion detectors in complex cyber physical systems such as the Smart Grid. Anomaly detection is a key strategy for cyber intrusion detection and operates by identifying deviations from profiles of nominal behavior and are thus conceptually appealing for detecting "novel" attacks. Evaluating the performance of such a detector requires assessing: (a) how well it captures the model of nominal behavior, and (b) how well it detects attacks (deviations from normality). Current evaluation methods produce results that give insufficient insight into the operation of a detector, inevitably resulting in a significantly poor characterization of a detectors performance. In this work, we first describe a preliminary taxonomy of key evaluation constructs that are necessary for establishing rigor in the evaluation regime of an anomaly detector. We then focus on clarifying the impact of the operational environment on the manifestation of attacks in monitored data. We show how dynamic and evolving environments can introduce high variability into the data stream perturbing detector performance. Prior research has focused on understanding the impact of this variability in training data for anomaly detectors, but has ignored variability in the attack signal that will necessarily affect the evaluation results for such detectors. We posit that current evaluation strategies implicitly assume that attacks always manifest in a stable manner; we show that this assumption is wrong. We describe a simple experiment to demonstrate the effects of environmental noise on the manifestation of attacks in data and introduce the notion of attack manifestation stability. Finally, we argue that conclusions about detector performance will be unreliable and incomplete if the stability of attack manifestation is not accounted for in the evaluation strategy.

cyber defense↗

NASA NTRS1993

Postflight hardware evaluation 360T026 (RSRM-26, STS-47)

The final report for the Clearfield disassembly evaluation and a continuation of the KSC postflight assessment for the 360T026 (STS-47) Redesigned Solid Rocket Motor (RSRM) flight set is provided. All observed hardware conditions were documented on PFOR's and are included in Appendices A, B, and C. Appendices D and E contain the measurements and safety factor data for the nozzle and insulation components. This report, along with the KSC Ten-Day Postflight Hardware Evaluation Report (TWR-64203), represents a summary of the 360T026 hardware evaluation. The as-flown hardware configuration is documented in TWR-60472. Disassembly evaluation photograph numbers are logged in TWA-1987. The 360T026 flight set disassembly evaluations described were performed at the RSRM Refurbishment Facility in Clearfield, Utah. The final factory joint demate occurred on 12 April 1993. Detailed evaluations were performed in accordance with the Clearfield Postflight Engineering Evaluation Plan (PEEP), TWR-50051, Revision A. All observations were compared against limits that are also defined in the PEEP. These limits outline the criteria for categorizing the observations as acceptable, reportable, or critical. Hardware conditions that were unexpected and/or determined to be reportable or critical were evaluated by the applicable CPT and tracked through the PFAR system.

Nielson, Greg↗

NASA NTRS1993

Final postflight hardware evaluation report RSRM-28 (STS-53)

The final report for the Clearfield disassembly evaluation and a continuation of the KSC postflight assessment for the RSRM-28 (STS-53) RSRM flight set is presented. All observed hardware conditions were documented on PFOR's and are included in Appendices A through C. Appendices D and E contain the measurements and safety factor data for the nozzle and insulation components. This report, along with the KSC Ten-Day Postflight Hardware Evaluation Report (TWR-64215), represents a summary of the RSRM-28 hardware evaluation. The as-flown hardware configuration is documented in TWR-63638. Disassembly evaluation photograph numbers are logged in TWA-1989. The RSRM-28 flight set disassembly evaluations described were performed at the RSRM Refurbishment Facility in Clearfield, Utah. The final factory joint demate occurred on July 15, 1993. Additional time was required to perform the evaluation of the stiffener rings per special issue 4.1.5.2 because of the washout schedule. The release of this report was after completion of all special issues per program management direction. Detailed evaluations were performed in accordance with the Clearfield PEEP, TWR-50051, Revision A. All observations were compared against limits that are also defined in the PEEP. These limits outline the criteria for categorizing the observations as acceptable, reportable, or critical. Hardware conditions that were unexpected and/or determined to be reportable or critical were evaluated by the applicable team and tracked through the PFAR system.

Starrett, William David, Jr.↗

NASA NTRS1994

Rule based design of conceptual models for formative evaluation

A Human-Computer Interface (HCI) Prototyping Environment with embedded evaluation capability has been investigated. This environment will be valuable in developing and refining HCI standards and evaluating program/project interface development, especially Space Station Freedom on-board displays for payload operations. This environment, which allows for rapid prototyping and evaluation of graphical interfaces, includes the following four components: (1) a HCI development tool, (2) a low fidelity simulator development tool, (3) a dynamic, interactive interface between the HCI and the simulator, and (4) an embedded evaluator that evaluates the adequacy of a HCI based on a user's performance. The embedded evaluation tool collects data while the user is interacting with the system and evaluates the adequacy of an interface based on a user's performance. This paper describes the design of conceptual models for the embedded evaluation system using a rule-based approach.

Moore, Loretta A.↗

NASA NTRS1994

Rule based design of conceptual models for formative evaluation

A Human-Computer Interface (HCI) Prototyping Environment with embedded evaluation capability has been investigated. This environment will be valuable in developing and refining HCI standards and evaluating program/project interface development, especially Space Station Freedom on-board displays for payload operations. This environment, which allows for rapid prototyping and evaluation of graphical interfaces, includes the following four components: (1) a HCI development tool; (2) a low fidelity simulator development tool; (3) a dynamic, interactive interface between the HCI and the simulator; and (4) an embedded evaluator that evaluates the adequacy of a HCI based on a user's performance. The embedded evaluation tool collects data while the user is interacting with the system and evaluates the adequacy of an interface based on a user's performance. This paper describes the design of conceptual models for the embedded evaluation system using a rule-based approach.

Moore, Loretta A.↗

NASA NTRS1999

Microgravity Workstation and Restraint Evaluations

Confined workstations, where the operator has limited visibility and physical access to the work area, may cause prolonged periods of unnatural posture. Impacts on performance, in terms of fatigue and posture, may occur especially if the task is tedious and repetitive or requires static muscle loading. The glovebox design is a good example of the confined workstation concept. Within the scope of the 'Microgravity Workstation and Restraint Evaluation' project, funded by the NASA Headquarters Life Sciences Division, it was proposed to conduct a series of evaluations in ground, KC-135 and Shuttle environments to investigate the human factors issues concerning confined/unique workstations, such as gloveboxes, and also including crew restraint requirements. As part of the proposed integrated evaluations, two Shuttle Detailed Supplementary Objectives (DSOs) were manifested; one on Space Transportation System (STS)-90 and one on STS-88. The DSO on STS-90 evaluated use of the General Purpose Workstation (GPWS). The STS-88 mission was planned to evaluate a restraint system at the Remote Manipulator System (RMS). In addition, KC- 1 35 flights were conducted to investigate user/workstation/restraint integration for long-duration microgravity use. The scope of these evaluations included workstations and restraints to be utilized in the ISS environment, but also incorporated other workstations/ restraints in an attempt to provide findings/requirements with broader applications across multiple programs (e.g., Shuttle, ISS, and future Lunar-Mars programs). In addition, a comprehensive electronic questionnaire has been prepared and is under review by the Astronaut Office which will compile crewmembers' lessons learned information concerning glovebox and restraint use following their missions. These evaluations were intended to be complementary and were coordinated with hardware developers, users (crewmembers), and researchers. This report is intended to provide a summary of the findings from each of the evaluations.

Chmielewski, C.↗

NASA NTRS1998

Objective Situation Awareness Measurement Based on Performance Self-Evaluation

The research was conducted in support of the NASA Safe All-Weather Flight Operations for Rotorcraft (SAFOR) program. The purpose of the work was to investigate the utility of two measurement tools developed by the British Defense Evaluation Research Agency. These tools were a subjective workload assessment scale, the DRA Workload Scale and a situation awareness measurement tool. The situation awareness tool uses a comparison of the crew's self-evaluation of performance against actual performance in order to determine what information the crew attended to during the performance. These two measurement tools were evaluated in the context of a test of innovative approach to alerting the crew by way of a helmet mounted display. The situation assessment data are reported here. The performance self-evaluation metric of situation awareness was found to be highly effective. It was used to evaluate situation awareness on a tank reconnaissance task, a tactical navigation task, and a stylized task used to evaluated handling qualities. Using the self-evaluation metric, it was possible to evaluate situation awareness, without exact knowledge the relevant information in some cases and to identify information to which the crew attended or failed to attend in others.

DeMaio, Joe↗

NASA NTRS2004

Efficient Evaluation Functions for Multi-Rover Systems

Evolutionary computation can be a powerful tool in cresting a control policy for a single agent receiving local continuous input. This paper extends single-agent evolutionary computation to multi-agent systems, where a collection of agents strives to maximize a global fitness evaluation function that rates the performance of the entire system. This problem is solved in a distributed manner, where each agent evolves its own population of neural networks that are used as the control policies for the agent. Each agent evolves its population using its own agent-specific fitness evaluation function. We propose to create these agent-specific evaluation functions using the theory of collectives to avoid the coordination problem where each agent evolves a population that maximizes its own fitness function, yet the system has a whole achieves low values of the global fitness function. Instead we will ensure that each fitness evaluation function is both "aligned" with the global evaluation function and is "learnable," i.e., the agents can readily see how their behavior affects their evaluation function. We then show how these agent-specific evaluation functions outperform global evaluation methods by up to 600% in a domain where a set of rovers attempt to maximize the amount of information observed while navigating through a simulated environment.

Agogino, Adrian↗

NASA NTRS2010

Risk Evaluation in the Pre-Phase A Conceptual Design of Spacecraft

Typically, the most important decisions in the design of a spacecraft are made in the earliest stages of its conceptual design the Pre-Phase A stages. It is in these stages that the greatest number of design alternatives is considered, and the greatest number of alternatives is rejected. The focus of Pre-Phase A conceptual development is on the evaluation and comparison of whole concepts and the larger-scale systems comprising those concepts. This comparison typically uses general Figures of Merit (FOMs) to quantify the comparative benefits of designs and alternative design features. Along with mass, performance, and cost, risk should be one of the major FOMs in evaluating design decisions during the conceptual design phases. However, risk is often given inadequate consideration in conceptual design practice. The reasons frequently given for this lack of attention to risk include: inadequate mission definition, lack of rigorous design requirements in early concept phases, lack of fidelity in risk assessment methods, and under-evaluation of risk as a viable FOM for design evaluation. In this paper, the role of risk evaluation in early conceptual design is discussed. The various requirements of a viable risk evaluation tool at the Pre-Phase A level are considered in light of the needs of a typical spacecraft design study. A technique for risk identification and evaluation is presented. The application of the risk identification and evaluation approach to the conceptual design process is discussed. Finally, a computational tool for risk profiling is presented and applied to assess the risk for an existing Pre-Phase A proposal. The resulting profile is compared to the risks identified for the proposal by other means.

Fabisinski, Leo L., III↗

NASA NTRS2022

Evaluation of remote sensing-based evapotranspiration products at low-latitude eddy covariance sites

Remote sensing-based evapotranspiration (ET) products have been evaluated primarily using data from northern middle latitudes; therefore, little is known about their performance at low latitudes. To address this bias, an evaluation dataset was compiled using eddy covariance data from 40 sites between latitudes 30° S and 30° N. The flux data were obtained from the emerging network in Mexico (MexFlux) and from openly available databases of FLUXNET, AsiaFlux, and OzFlux. This unique reference dataset was then used to evaluate remote sensing-based ET products in environments that have been underrepresented in earlier studies. The evaluated products were: MODIS ET (MOD16, both the discontinued collection 5 (C5) and the latest collection (C6)), Global Land Evaporation Amsterdam Model (GLEAM) ET, and Atmosphere-Land Exchange Inverse (ALEXI) ET. Products were compared with unadjusted fluxes (ETorig) and with fluxes corrected for the lack of energy balance closure (ETebc). Three common statistical metrics were used: coefficient of determination (R2), root mean square error (RMSE), and percent bias (PBIAS). The effect of a vegetation mismatch between pixel and site on product evaluation results was investigated by examining the relationship between the statistical metrics and product-specific vegetation match indexes. Evaluation results of this study and those published in the literature were used to examine the performance of the products across latitudes. Differences between the MOD16 collection 5 and 6 datasets were generally smaller than differences with the other products. Performance and ranking of the evaluated products depended on whether ETorig or ETebc was used. When using ETorig, GLEAM generally had the highest R2, smallest PBIAS, and best RMSE values across the studied land cover types and climate zones. Neither MOD16 nor ALEXI performed consistently better than the other. When using ETebc, none of the products stood out in terms of both low bias and strong correlations. The use of ETebc instead of ETorig affected the biases more than the correlations. The product evaluation results showed no significant relationship with the degree of match between the vegetation at the pixel and site scale. The latitudinal comparison showed tendencies of lower R2 (all products) but better PBIAS and normalized RMSE values (MOD16 and GLEAM) for forests at low latitudes than for forests at northern middle latitudes. For non-forest vegetation, the products showed no clear latitudinal differences in performance.

Diego Salazar-Martínez↗

NASA NTRSDate not supplied

Evaluation of the NASA Artemis Regions of Interest for ISRU Water Mine Potential

The NASA Artemis Campaign has a stated goal to return to the Moon to maintain a sustainable presence; In-Situ Resource Utilization (ISRU) is a key part of sustainability. The regions of interest identified for the Artemis campaign are at Lunar the South Pole where water ice, a valuable resource for ISRU, has been identified. As such, a preliminary evaluation of the ISRU ice mining potential has been performed for of these regions of interest. A set of ground rules for this evaluation were developed to align with current assumptions for customer needs, hardware capabilities, an initially limited infrastructure, and lunar environments/terrain. These ground rules, and the evaluation of six regions of interest, are presented here. The site selections (ISRU and customer assets) and their associated traverses are notional and were intended only to provide a broad preliminary evaluation of the water ISRU potential of the regions. Evaluation of these regions are subject to change as decisions regarding utilization are made. Water ISRU is possible at all regions, though the degree to which each criterion are met is variable. The two regions near Shackleton ranked highest in this evaluation, while the de Gerlache region presented the most difficulties meeting the current criteria. The regions were not explicitly ranked due to the nuances associated with the high number of variables but evaluation summaries of each are presented. It should also be noted that all ISRU ‘mine’ sites in this analysis focused on smaller (few kilometer) size permanently shadowed regions (PSRs). This was necessary to meet proximity requirements between these PSRs and the highly illuminated regions needed for customers and ISRU processing. The areas identified in this study are meant to focus exploration and reconnaissance efforts needed to better evaluate the ISRU potential.

In-situ resource utilization↗

NASA NTRSDate not supplied

Evaluation of the NASA Artemis Regions of Interest for ISRU Water Mine Potential

In-situ resource utilization↗

NASA NTRS1987

Evaluation of high-voltage, high-power, solid-state remote power controllers for amps

The Electrical Power Branch at Marshall Space Flight Center has a Power System Development Facility where various power circuit breadboards are tested and evaluated. This project relates to the evaluation of a particular remote power controller (RPC) energizing high power loads. The Facility equipment permits the thorough testing and evaluation of high-voltage, high-power solid-state remote power controllers. The purpose is to evaluate a Type E, 30 Ampere, 200 V dc remote power controller. Three phases of the RPC evaluation are presented. The RPC is evaluated within a low-voltage, low-power circuit to check its operational capability. The RPC is then evaluated while performing switch/circuit breaker functions within a 200 V dc, 30 Ampere power circuit. The final effort of the project relates to the recommended procedures for installing these RPC's into the existing Autonomously Managed Power System (AMPS) breadboard/test facility at MSFC.

Callis, Charles P.↗

NASA NTRS1990

Evaluation of NASA space grant consortia programs

The meaningful evaluation of the NASA Space Grant Consortium and Fellowship Programs must overcome unusual difficulties: (1) the program, in its infancy, is undergoing dynamic change; (2) the several state consortia and universities have widely divergent parochial goals that defy a uniform evaluative process; and (3) the pilot-sized consortium programs require that the evaluative process be economical in human costs less the process of evaluation comprise the effectiveness of the programs they are meant to assess. This paper represents an attempt to assess the context in which evaluation is to be conducted, the goals and limitations inherent to the evaluation, and to recommend appropriate guidelines for evaluation.

Eisenberg, Martin A.↗

NASA NTRS1993

Postflight hardware evaluation 360T025 (RSRM-25, STS-46)

The final report for the Clearfield disassembly evaluation and a continuation of the KSC postflight assessment for the 360T025 (STS-46) Redesign Solid Rocket Motor (RSRM) flight set is presented. All observed hardware conditions were documented on PFOR's and are included in Appendices A through C. Appendices D and E contain the measurements and safety factor data for the nozzle and insulation components. Along with the KSC Ten-Day Postflight Hardware Evaluation Report (TWR-60687), a summary of the 360T025 hardware evaluation is provided. The as-flown hardware configuration is documented in TWR-60470. Disassembly evaluation photograph numbers are logged in TWA-1986. The 360T025 flight set disassembly evaluations described were performed at the RSRM Refurbishment Facility in Clearfield, Utah. The final factory joint demate occurred on 16 Mar. 1993. Detailed evaluations were performed in accordance with the Clearfield PEEP, TWR-50051, Revision A. All observations were compared against limits that are also defined in the PEEP. These limits outline the criteria for categorizing the observations as acceptable, reportable, or critical. Hardware conditions that were unexpected and/or determined to be reportable or critical were evaluated by the applicable CPT and tracked through the PFAR system.

Morgan, Ferral↗

NASA NTRS1993

Postflight hardware evaluation (RSRM-29, STS-54)

This document is the final report for the Clearfield disassembly evaluation and a continuation of the KSC postflight assessment for the RSRM-29 flight set. All observed hardware conditions were documented on PFOR's and are included in Appendices A, B, and C. Appendices D and E contain the measurements and safety factor data for the nozzle and insulation components. This report, along with the KSC Ten-Day Postflight Hardware Evaluation Report (TWR-64221), represents a summary of the RSRM-29 hardware evaluation. Disassembly evaluation photograph numbers are logged in TWA-1990. The RSRM-29 flight set disassembly evaluations described in this document were performed at the RSRM Refurbishment Facility in Clearfield, Utah. The final factory joint demate occurred on September 9, 1993. Detailed evaluations were performed in accordance with the Clearfield PEEP, TWR-50051, Revision A. All observations were compared against limits that are also defined in the PEEP. These limits outline the criteria for categorizing the observations as acceptable, reportable, or critical. Hardware conditions that were unexpected and/or determined to be reportable or critical were evaluated by the applicable CPT and tracked through the PFAR system.

Source record↗

NASA NTRS1994

Extra-Vehicular Activity (EVA) glove evaluation test protocol

One of the most critical components of a space suit is the gloves, yet gloves have traditionally presented significant design challenges. With continued efforts at glove development, a method for evaluating glove performance is needed. This paper presents a pressure-glove evaluation protocol. A description of this evaluation protocol, and its development is provided. The protocol allows comparison of one glove design to another, or any one design to bare-handed performance. Gloves for higher pressure suits may be evaluated at current and future design pressures to drive out differences in performance due to pressure effects. Using this protocol, gloves may be evaluated during design to drive out design problems and determine areas for improvement, or fully mature designs may be evaluated with respect to mission requirements. Several different test configurations are presented to handle these cases. This protocol was run on a prototype glove. The prototype was evaluated at two operating pressures and in the unpressurized state, with results compared to bare-handed performance. Results and analysis from this test series are provided, as is a description of the configuration used for this test.

Hinman-Sweeney, E. M.↗

NASA NTRS1994

Subjective evaluations of integer cosine transform compressed Galileo solid state imagery

This paper describes a study conducted for the Jet Propulsion Laboratory, Pasadena, California, using 15 evaluators from 12 institutions involved in the Galileo Solid State Imaging (SSI) experiment. The objective of the study was to determine the impact of integer cosine transform (ICT) compression using specially formulated quantization (q) tables and compression ratios on acceptability of the 800 x 800 x 8 monochromatic astronomical images as evaluated visually by Galileo SSI mission scientists. Fourteen different images in seven image groups were evaluated. Each evaluator viewed two versions of the same image side by side on a high-resolution monitor; each was compressed using a different q level. First the evaluators selected the image with the highest overall quality to support them in their visual evaluations of image content. Next they rated each image using a scale from one to five indicating its judged degree of usefulness. Up to four preselected types of images with and without noise were presented to each evaluator.

Haines, Richard F.↗