Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Performance benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,693 records · Page 94
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Algorithmic and GPU enhancements for molecular dynamics in Cabana and LAMMPS

The focus of this milestone was to implement and test new algorithms and optimizations for improved GPU performance by molecular dynamics (MD) and related particle codes. The work was done in the CoPA Cabana library, the CabanaMD mini-app which uses Cabana to implement and benchmark prototypical MD models, and the LAMMPS MD code which is used by the ECP EXAALT project as well as for materials modeling generally. Efforts included (a) implementation of new or optimized algorithms for fundamental MD kernels like neighbor-list building and FFTs which are also widely used in other particle methods, (b) performance testing of data structure layouts for both simple pairwise and computationally intense neural network potentials, (c) optimizations aimed at improving GPU performance for models with small atom counts, and (d) optimizations specific to the SNAP interatomic potential due to its importance for the EXAALT project.

36 MATERIALS SCIENCE↗
DOE OSTI2024

CACTUS: Chemistry Agent Connecting Tool Usage to Science

Large language models (LLMs) have shown remarkable potential in various domains but often lack the ability to access and reason over domain-specific knowledge and tools. In this article, we introduce Chemistry Agent Connecting Tool-Usage to Science (CACTUS), an LLM-based agent that integrates existing cheminformatics tools to enable accurate and advanced reasoning and problem-solving in chemistry and molecular discovery. We evaluate the performance of CACTUS using a diverse set of open-source LLMs, including Gemma-7b, Falcon-7b, MPT-7b, Llama3-8b, and Mistral-7b, on a benchmark of thousands of chemistry questions. Our results demonstrate that CACTUS significantly outperforms baseline LLMs, with the Gemma-7b, Mistral-7b, and Llama3-8b models achieving the highest accuracy regardless of the prompting strategy used. Moreover, we explore the impact of domain-specific prompting and hardware configurations on model performance, highlighting the importance of prompt engineering and the potential for deploying smaller models on consumer-grade hardware without a significant loss in accuracy. By combining the cognitive capabilities of open-source LLMs with widely used domain-specific tools provided by RDKit, CACTUS can assist researchers in tasks such as molecular property prediction, similarity searching, and drug-likeness assessment.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRS2020

Durability Characterization of Low Ice Adhesion and Impact Resistance Coatings

Glaze, rime and mixed icing conditions can cause significant variation in aerodynamic performance, fuel efficiency, and flight safety. For commercial aircraft, active icing mitigation strategies are utilized to enable safe flight within icing conditions according to the FAR Part 25/29 Appendix C icing envelope. For general aviation and unmanned aerial vehicles that cannot support active anti-icing technologies, avoidance is the only recourse. Passive approaches have been investigated to reduce weight and energy consumption for active systems and to expand the operational envelope for smaller aircraft. Coatings are one passive method to reduce or mitigate ice accretion on frontal surfaces of commercial aircraft with the greatest region of interest being wing leading edges. Aircraft wing leading edges represent an extreme environment. Durability must be considered for any material to be applied as a coating in this area. Currently, there are no specifications regarding coating durability of a low ice adhesion material applied on the wing leading edge. Therefore, a reasonable starting point is to use the durability specifications put forth for an aircraft external coating. Besides the tests called out in these specifications, others were included such as Taber abrasion to simulate wear and erosion. Beyond durability, performance metrics with regards to adhesion strength of accreted in-flight (i.e., impact) icing are central to determine coating usefulness. Determination of this property though is particularly challenging and only a few facilities have the capability to quantify this. An instrument at NASA Langley Research Center [Adverse Environment Rotary Test Stand Jr., (AERTS Jr.)], based on the AERTS system at The Pennsylvania State University, allows for screening numerous coatings subjected to the icing environment. To establish a benchmark for comparison with research coatings, a state-of-the-art (SOA) commercial aircraft coating was subjected to an array of salient durability and performance experiments. The same tests were conducted using a baseline epoxy (BE) resin formulation that demonstrated some initial promising results for comparison. Results from this initial screening will be discussed herein.

Miranda L Beaudry↗
DOE OSTI2021

On the use of neural networks for energy reconstruction in high-granularity calorimeters

We contrasted the performance of deep neural networks — Convolutional Neural Network (CNN) and Graph Neural Network (GNN) — to current state of the art energy regression methods in a finely 3D-segmented calorimeter simulated by GEANT4. This comparative benchmark gives us some insight to assess the particular latent signals neural network methods exploit to achieve superior resolution. A CNN trained solely on a pure sample of pions achieved substantial improvement in the energy resolution for both single pions and jets over the conventional approaches. It maintained good performance for electron and photon reconstruction. We also used the Graph Neural Network (GNN) with edge convolution to assess the importance of timing information in the shower development for improved energy reconstruction. We implement a simple simulation based correction to the energy sum derived from the fraction of energy deposited in the electromagnetic shower component. This serves as an approximate dual-readout analogue for our benchmark comparison. Although this study does not include the simulation of detector effects, such as electronic noise, the margin of improvement seems robust enough to suggest these benefits will endure in real-world application. We also find reason to infer that the CNN/GNN methods leverage latent features that concur with our current understanding of the physics of calorimeter measurement.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Implementation of a Binary Neural Network on a Passive Array of Magnetic Tunnel Junctions

The increasing scale of neural networks and their growing application space have produced demand for more energy- and memory-efficient artificial-intelligence-specific hardware. Avenues to mitigate the main issue, the von Neumann bottleneck, include in-memory and near-memory architectures, as well as algorithmic approaches. In this report we leverage the low-power and the inherently binary operation of magnetic tunnel junctions (MTJs) to demonstrate neural network hardware inference based on passive arrays of MTJs. In general, transferring a trained network model to hardware for inference is confronted by degradation in performance due to device-to-device variations, write errors, parasitic resistance, and nonidealities in the substrate. To quantify the effect of these hardware realities, we benchmark 300 unique weight matrix solutions of a two-layer perceptron to classify the Wine dataset for both classification accuracy and write fidelity. Despite device imperfections, we achieve software-equivalent accuracy of up to 95.3% with proper tuning of network parameters in 15 x 15 MTJ arrays having a range of device sizes. The success of this tuning process shows that new metrics are needed to characterize the performance and quality of networks reproduced in mixed signal hardware.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Overview of Liquid-Metal PFC R&D at the University of Illinois Urbana-Champaign

The design and implementation of future flowing liquid-lithium plasma-facing components (LLPFCs) will be dependent on several factors. Of course, one of the most important is the need to be able to deal with high heat fluxes incident on the surface of the LLPFCs, but there are also several other important liquid-metal behaviors that have been identified for their critical impact on the feasibility of a LLPFC. One of these is the ability to constantly wet 100% of the plasma-facing component area and the best way to achieve that. Another key point is knowing and understanding the erosion and corrosion of the surfaces subject to a flowing liquid-lithium system and the ability for hydrogen and helium uptake by the system. The Center for Plasma Material Interactions (CPMI) has been tasked with looking at these various issues. The Mock-up Entry module for EAST device was used to investigate wetting and erosion effects and to design a suitable distribution and collection system with a liquid-lithium loop. The vapor shielding effects of lithium on the surface were also modeled and studied. A model coupling CRANE, an open-source global reaction network solver, and Zapdos, a plasma transport solver, is being developed to better understand the dynamics of the vapor cloud. Experiments on the Magnum-PSI at the Dutch Institute for Fundamental Energy Research have been carried out to study the vapor shielding effect and obtain experimental benchmarks to verify the model. Also, initial experiments using the Hybrid Illinois Device for Research and Applications have been performed to understand the pumping effects of lithium on helium. Experiments with a drop of liquid lithium (~100 mg) into a helium plasma have shown the ability of lithium to take out the cold recycling helium gas as well as hydrogen and oxygen impurity gases. The improvement in plasma performance was significant, and further understanding of this effect will have impacts on how future LLPFCs will be designed. Further investigation into the exact mechanism for helium pumping by lithium needs to be performed in the future. Finally, this paper presents a summary of the results obtained at the CPMI.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2020

Validation of neutron emission and neutron energy spectrum calculations on a Mega Ampere Spherical Tokamak with directional relativistic spectrum simulator

The recently developed directional relativistic spectrum simulator (DRESS) code has been validated for the first time against numerical calculations and experimental measurements performed on the Mega Ampere Spherical Tokamak . In this validation, the neutron emissivities and rates computed by DRESS are benchmarked against TRANSP/NUBEAM predictions while the neutron energy spectra provided by DRESS taking as input TRANSP/NUBEAM and ASCOT/BBNBI in Gyro-Orbit mode fast ion distributions are validated against proton pulse height spectra measured by the neutron flux monitor. Finally, excellent agreement was found between DRESS and TRANSP/NUBEAM predictions of local and total neutron emission.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

Novel technique for the study of pileup events in cryogenic bolometers

Precise characterization of detector time resolution is of crucial importance for next-generation cryogenic-bolometer experiments searching for neutrinoless double-beta decay, such as CUPID, in order to reject background due to pile-up of two-neutrino double-beta decay events. In this paper, we describe a technique developed to study the pile-up rejection capability of cryogenic bolometers. Our approach, which consists of producing controlled pile-up events with a programmable waveform generator, has the benefit that we can reliably and reproducibly control the time separation and relative energy of the individual components of the generated pile-up events. The resulting data allow us to optimize and benchmark analysis strategies to discriminate between individual and pile-up pulses. We describe a test of this technique performed with a small array of detectors at the Laboratori Nazionali del Gran Sasso, in Italy; we obtain a 90% rejection efficiency against pulser-generated pile-up events with rise time of ~15 ms down to time separation between the individual events of 2 ms.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

TPCPF: Three-Phase Continuation Power Flow Tool for Voltage Stability Assessment of Distribution Networks With Distributed Energy Resources

This article presents a three-phase unbalanced continuation power flow algorithm for voltage stability assessment of distribution systems with high penetration of distributed energy resources (DERs). Analyzing distribution system voltage stability with DER will allow high penetration of renewable energy necessary for the sustainability goals. The developed algorithm can analyze voltage stability for both the meshed and radial systems and the balanced and unbalanced three-phase distribution systems. The developed tool allows the voltage stability analysis to facilitate the planning, operation, control, and distribution system management. The impact of DER on the voltage stability of several test cases has been analyzed considering constant power (P Q) and regulated-voltage (P V ) modes of operation for DER units. Moreover, different voltage stability case studies are presented to demonstrate the impact of unbalance, load increment, and network topology on the maximum loading capacity. Finally, results using the IEEE 13-node feeder, the 18-bus balanced shipboard system, the 13-node CIGRE benchmark system, and the 136-bus redial distribution feeder demonstrate that the developed continuation power flow tool can efficiently perform voltage stability analysis for active distribution systems.

42 ENGINEERING↗
DOE OSTI2021

Critical and Subcritical Californium Source Driven Noise Analysis Experiments with Fresh PWR Fuels Pins

These experiments were performed in 1983 between August 1 and September 16 and used 35 operational days of critical facility time at the Babcock & Wilcox critical facility in Lynchburg, Va. These measurements were to assess the capability of the Cf source driven noise analysis method to measure the subcriticality of light water reactor fresh fuel configurations. This report documents the experiments of 1983 that were not reported at that time. These measurement with 2.459 wt. % 235 U enriched uranium oxide fresh fuel pins started at the delayed critical configuration of the fully assembled pressurized water reactor (PWR) core configuration with 1511 parts per million (ppm) natural boron (19.8 wt. % 10 B) in the water. This was a cylindrical configuration of 4961 fuel pins. After the initial measurements at delayed criticality, subcritical measurements by the Cf source driven noise analysis method were performed in which the number of detectors and Cf source locations were varied. Additional boron was added to the water moderator and reflector to reduce the neutron multiplication factor, with resulting boron concentrations of 1561, 1613, 1765, 1880, 2104, 2384, 2975, 3606, and 4303 ppm. After the completion of these measurements the increased, the boron concentration was returned to that at delayed critical (1511 ppm) and the reactivity was decreased by reducing the number of fuel pins in steps to 3713, 2553, 1281, 749, and 333 fuel pins and then to that (289) of a s ingle square 17 x 17 pin array, corresponding to a single PW fuel element. For all measurement that started at delayed criticality and then reduced the neutron multiplication factor to various subcritical states, inverse kinetics rod drop measurements at the higher reactivities were also performed but not reported here. At all subcritical states the break frequency noise analysis data was fitted to obtain the prompt neutron decay constant. Some data presented in this report are from notes not in the logbook. The purpose of this report is to document the experimental information for the measurements performed so that a later date someone could perform the required uncertainty and calculational analyses and documentation to use these data for an International Nuclear Criticality Safety Benchmark Program (INCSBEP) or Nuclear Energy Agency (NEA) benchmark. The data from these measurements is available from the Laboratory Records Department of Oak Ridge National Laboratory and the logbook is available from INCSBEP at Idaho National Laboratory.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2022

Small Hydropower Interconnections: Analysis of Interconnection Processes

Small hydropower projects have faced the challenge of navigating the process to interconnect their generation source to electricity distribution and transmission grids. Small hydropower developers have found interconnection procedures to be opaque and ultimately result in unexpected cost surprises and long timelines. Noting these challenges, the U.S. Department of Energy Water Power Technologies Office enlisted Pacific Northwest National Laboratory (PNNL) and Oak Ridge National Laboratory (ORNL) to investigate the small hydropower interconnection landscape across the United States. After reviewing the status of small hydropower (“Small Hydropower Interconnections: Small Hydropower in the United States”) and the interconnection procedures across the United States (“Small Hydropower Interconnections: State Interconnection Processes”) in the first two white papers of this series, this paper uses recent data from small hydropower interconnection applications to benchmark the efficacy of the process. Using data from interconnection queues hosted by utilities, balancing authorities, independent system operators (ISOs), and regional transmission organizations (RTOs), this paper provides context for the costs, timelines, and types of upgrades required for small hydropower projects. Interconnection applications and study reports for small hydropower projects were analyzed to collect key pieces of information about the interconnection process, timeline, costs, and type of upgrades required for interconnection. Information sourced from the reports was entered into an Interconnection Benchmarking database (IBdb), which may be found in Appendix A.1. Information from this database was used to evaluate the performance and challenges associated with interconnecting small hydropower projects. This white paper presents a description of the sources contained in the interconnection database (Section 2.0), an analysis of the interconnection timeline (Section 3.0), an evaluation the cost of interconnection upgrades (Section 4.0), and a description of the types of infrastructure upgrades (Section 5.0). The final paper in this series (“Small Hydropower Interconnections: Best Practices”) will use the analysis described here to outline best practices for interconnection processes that will help overcome barriers to future small hydropower development.

13 HYDRO ENERGY↗
DOE OSTI2021

Uranium–Molybdenum Alloy Critical Experiments for the Design of the Health Physics Research Reactor

Clean critical experiments with a uranium-molybdenum alloy (average of 10.1616 wt. % Mo with a density of 17.08 g/cm 3 ) were performed at the Oak Ridge Critical Experiments Facility in 1961 to support the design of the Health Physics Research Reactor (HPRR). The HPRR was similar to the Godiva burst reactor at Los Alamos National Laboratory and was designed to produce 50 microseconds burst of 10 17 fission pulses of radiation for dosimetry measurements, initially in support of the determination of the doses from the nuclear detonations in Japan during World War II. These experiments reported here were used to verify the calculational methods used to design the HPRR. These delayed critical measurements were:1) a solid unreflected and unmoderated 8-in.-dimeter U-Mo cylinder, 2) an unmoderated and unreflected annulus with 8-in.-outside diameter, 2-in.-inside diameter cylinder with a central void, 3) an unmoderated and unreflected annulus with 8-in.-outside diameter, 2-in.-inside diameter cylinder with a central void filled with stainless steel, 4) Same as 3) but with 3-in-thick Plexiglas reflector on top with and without cadmium between the reflector and the U-Mo alloy assembly with steel in the center, and 5) an unmoderated and unreflected annulus which was a modification of the second but with the lower 5 inches of the central hole enlarged to 3.5 in. with various reflector conditions. The reflector conditions were: 1-in.-thick Plexiglas on all outer surfaces-void in the center; 1-in.-thick Plexiglas on all outer surfaces-Plexiglas in the center; 2-in.- thick Plexiglas on radial surface-void in the center; 6-in.-thick Plexiglas on the bottom only-Plexiglas in the center; and 6-in.-thick Plexiglas on bottom, 1-in.-thick on top and on the lower 8.25-cm.-section of the radial surface-void in the center. For some of these reflector conditions 0.025-cm.thick cadmium was located between the reflector and the U-Mo alloy. The uranium contained 93.17 wt. % 235 U. Reflection was a safety concern for this unmoderated and unreflected reactor and reduction of reflection effects was also investigated by insertion of neutron absorber around the U-Mo alloy. The stainless steel 304 contained 18% nickel and 8% chromium and the rest iron. The reflector material was a methacrylate plastic (Plexiglas) containing 5.8 x 10 22 atoms/cm 3 of hydrogen and 3.6 x 10 22 atoms/cm 3 of carbon with a density of 1.20 g/cm 3 . The purpose of this report is to document the experimental information for the measurements performed so that at a later date researchers could perform the required uncertainty and calculational analyses and documentation to use these data for an International Nuclear Criticality Safety Benchmark Program (ICSBEP) or a EURATON Nuclear Energy Agency (NEA) benchmark. The data from the experiments described should be acceptable for use as criticality safety benchmark experiments for the ICSBEP and the NEA nuclear criticality safety benchmark program, once the uncertainty analysis is completed. Based on previous ICSBEP benchmarks with this enriched uranium metal at ORCEF, the uncertainties in k eff could be as low as ±0.0002 for some configurations.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2024

Godiva IV Simulated Radiation Field Characterization and Variance Reduction

Godiva IV is a system comprised of highly enriched uranium alloyed with molybdenum in the form of fuel plate rings. The reactor, along with its predecessors, was designed with the unique ability to satisfy interests in the super-prompt-critical reactor operation space. Originally, the reactor was part of the Los Alamos Critical Experiments Facility (LACEF) at Technical Area-18 (TA-18). The radiation field around Godiva at this facility was well characterized and understood. As a fast neutron system, the neutron spectrum in and around Godiva was close to a Watt Fission spectrum. The Kiva where Godiva IV was located at LACEF was made of thin, sheet metal walls which did not contribute significantly to the neutron spectrum. Following the transition of LACEF to the National Critical Experiments and Research Center (NCERC) in Nevada, Godiva-IV was moved from TA-18 to the Device Assembly Facility (DAF) at the Nevada National Security Site (NNSS). Part of this move brought renewed interest in radiation field characterization. The new facility introduced significant changes to the environment surrounding Godiva, and preliminary foil irradiation results suggested that the room contribution to the neutron spectrum was significant. Unlike at TA-18, a large thermal neutron signature was added to the fast spectrum from Godiva due to significant room return. A primary goal due to the additional complexity that the room return adds to the Godiva IV radiation emission spectrum was the development of an efficient Monte Carlo N-Particle (MCNP) calculation capable of characterizing the neutron spectrum anywhere in the room around Godiva. A campaign of activation foil irradiations and analysis were completed to support the validation of the MCNP model. The modeling of these foils in MCNP can be easily done with a standard volumetric neutron flux tally. However, given the multitude of locations and reaction rates to be modeled, further steps must be taken to increase the efficiency of these calculations in MCNP. During this study, a benchmark model currently under development for Godiva IV was used. A qualitative assessment of the thermal neutron contributors was performed using spatial neutron distribution plots. Additional detail was added to the model based on the qualitative results showing the thermal spectrum’s large sensitivity to hydrogenous material. Neutron energy spectra was evaluated at discrete locations in the room around Godiva to quantify the relative contribution of various components. It was discovered that the concrete walls are the largest contributor to the thermal signature, with minor contributions from plastic components surrounding Godiva. Following these results, two different variance reduction techniques were implemented to improve the problem efficiency in these calculations. In the first approach, an F5 point detector tally was implemented in the standard Godiva IV criticality problem. The second approach involved a weight-window generator implementation with an F5 point detector tally in a fixed source problem. The weight window implementation reduced the runtime from 42739.55 minutes to 1803.34 minutes (computer time), compared to the F5 KCODE implementation.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Lower Cost Renewable Nanoparticle Platform to Increases Production of Cathode Material (CRADA Final Report)

Employees of Sylvatex synthesized candidate cathode materials for lithium-ion batteries in PI’s labs and at the Molecular Foundry using sustainable methods. They then benchmarked the synthesized materials against available commercial cathode materials in lithium half-cells (coin cell configurations) to compare performance and ensure that their materials met or exceeded metrics of the commercial materials.

36 MATERIALS SCIENCE↗
DOE OSTI2020

DSCOVR/EPIC-derived global hourly and daily downward shortwave and photosynthetically active radiation data at 0.1° × 0.1° resolution

Downward shortwave radiation (SW) and photosynthetically active radiation (PAR) play crucial roles in Earth system dynamics. Spaceborne remote sensing techniques provide a unique means for mapping accurate spatiotemporally continuous SW–PAR, globally. However, any individual polar-orbiting or geostationary satellite cannot satisfy the desired high temporal resolution (sub-daily) and global coverage simultaneously, while integrating and fusing multisource data from complementary satellites/sensors is challenging because of co-registration, intercalibration, near real-time data delivery and the effects of discrepancies in orbital geometry. The Earth Polychromatic Imaging Camera (EPIC) on board the Deep Space Climate Observatory (DSCOVR), launched in February 2015, offers an unprecedented possibility to bridge the gap between high temporal resolution and global coverage and characterize the diurnal cycles of SW–PAR globally. In this study, we adopted a suite of well-validated data-driven machine-learning models to generate the first global land products of SW–PAR, from June 2015 to June 2019, based on DSCOVR/EPIC data. The derived products have high temporal resolution (hourly) and medium spatial resolution (0.1°×0.1°), and they include estimates of the direct and diffuse components of SW–PAR. We used independently widely distributed ground station data from the Baseline Surface Radiation Network (BSRN), the Surface Radiation Budget Network (SURFRAD), NOAA's Global Monitoring Division and the U.S. Department of Energy's Atmospheric System Research (ASR) program to evaluate the performance of our products, and we further analyzed and compared the spatiotemporal characteristics of the derived products with the benchmarking Clouds and the Earth's Radiant Energy System Synoptic (CERES) data. We found both the hourly and daily products to be consistent with ground-based observations (e.g., hourly and daily total SWs have low biases of -3.96 and -0.71 W m -2 and root-mean-square errors (RMSEs) of 103.50 and 35.40 W m -2 , respectively). The developed products capture the complex spatiotemporal patterns well and accurately track substantial diurnal, monthly, and seasonal variations in SW–PAR when compared to CERES data. They provide a reliable and valuable alternative for solar photovoltaic applications worldwide and can be used to improve our understanding of the diurnal and seasonal variabilities of the terrestrial water, carbon and energy fluxes at various spatial scales.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1983

Tilt rotor technology thrusts

This paper provides an overview of the results of on-going research investigations associated with the XV-15 program. The XV-15 data and an understanding of the technological elements that go into tilt rotor design can provide a series of benchmark points from which to proceed with new design. The work to refine the estimates of vertical-lift performance, the influence of airframe aerodynamics, and the work planned to establish rotor-limiting flight conditions are discussed. An initial look at short takeoff and landing (STOL) performance is provided and the lessons learned and on-going work in aeroelastic testing and data handling is described. Structural loads and stability and control systems work which have applicability to new design efforts are also shown. The paper concludes with some ideas for longer-range generic investigations to advance the state of the art.

Magee, J. P.↗
NASA NTRS1992

Exploratory study on performance measures as indicators of IS effectiveness

The Information Systems Directorate at JSC/NASA has undertaken the reevaluation of its performance measures process and measures. Under the direction of a quality approach it is essential to identify an external perspective of how well an organization is performing. This study was conducted with two major objectives: (1) survey and summarize the academic literature on performance measures as indicators of information systems (IS) effectiveness; and (2) survey organizations for their experience in measuring for IS effectiveness. Four approaches to measuring the effectiveness of IS performance were identified: (1) listen to the customer for the things they need; (2) align with corporate goals; (3) benchmark against well-respected organizations; and (4) ask yourself what critical factors lead to success. The list of known methods for soliciting customer feedback are as follows: (1) executive visit; (2) survey, interview, and focus group; (3) complaints and compliments; and (4) service level agreements. A common set of characteristics that satisfy customers was identified from the literature. The list includes elements such as the following: accuracy, timeliness, relevance, understandability, reliability, and completeness. Future research in this topic area should prove beneficial to determine the metrics for external validity.

Bishop, Peter C.↗
NASA NTRS2004

The Processing of Airspace Concept Evaluations Using FASTE-CNS as a Pre- or Post-Simulation CNS Analysis Tool

As NASA speculates on and explores the future of aviation, the technological and physical aspects of our environment increasing become hurdles that must be overcome for success. Research into methods for overcoming some of these selected hurdles have been purposed by several NASA research partners as concepts. The task of establishing a common evaluation environment was placed on NASA's Virtual Airspace Simulation Technologies (VAST) project (sub-project of VAMS), and they responded with the development of the Airspace Concept Evaluation System (ACES). As one examines the ACES environment from a communication, navigation or surveillance (CNS) perspective, the simulation parameters are built with assumed perfection in the transactions associated with CNS. To truly evaluate these concepts in a realistic sense, the contributions/effects of CNS must be part of the ACES. NASA Glenn Research Center (GRC) has supported the Virtual Airspace Modeling and Simulation (VAMS) project through the continued development of CNS models and analysis capabilities which supports the ACES environment. NASA GRC initiated the development a communications traffic loading analysis tool, called the Future Aeronautical Sub-network Traffic Emulator for Communications, Navigation and Surveillance (FASTE-CNS), as part of this support. This tool allows for forecasting of communications load with the understanding that, there is no single, common source for loading models used to evaluate the existing and planned communications channels; and that, consensus and accuracy in the traffic load models is a very important input to the decisions being made on the acceptability of communication techniques used to fulfill the aeronautical requirements. Leveraging off the existing capabilities of the FASTE-CNS tool, GRC has called for FASTE-CNS to have the functionality to pre- and post-process the simulation runs of ACES to report on instances when traffic density, frequency congestion or aircraft spacing/distance violations have occurred. The integration of these functions require that the CNS models used to characterize these avionic system be of higher fidelity and better consistency then is present in FASTE-CNS system. This presentation will explore the capabilities of FASTE-CNS with renewed emphasis on the enhancements being added to perform these processing functions; the fidelity and reliability of CNS models necessary to make the enhancements work; and the benchmarking of FASTE-CNS results to improve confidence for the results of the new processing capabilities.

Mainger, Steve↗