Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

An Operational Resilience Metric for Modern Power Distribution Systems

The electrical power system is the backbone of our nations critical infrastructure. It has been designed to withstand single component failures based on a set of reliability metrics which have proven acceptable during normal operating conditions. However, in recent years there has been an increasing frequency of extreme weather events. Many have resulted in widespread long-term power outages, proving reliability metrics alone do not provide adequate energy security. As a result, researchers have focused their efforts on a new set of metrics based on the concept of resilience to ensure efficient operation of power systems during extreme events. A resilient system has the ability to resist, adapt, and recover from disruptions. Therefore, resilience has demonstrated itself as a promising concept for currently faced challenges in power distribution systems. In this work, we propose a real-time resilience metric for modern power distribution systems. The metric is an aggregation of the controllable assets adaptive capacity, or their temporal flexibility in real and reactive power. This metric gives information to the magnitude and duration of a disturbance which the system can respond to without having to drop loads or have stability issues in voltage or frequency. We demonstrate the impact to resilience in a case study under normal operation and during a power contingency on a microgrid. In the future, this information can be used by operators to make more informed decisions based on system resilience in an effort to prevent power outages.

42 ENGINEERING↗

A Statistical Evaluation of Combining Human Productivity Metrics in the Indoor Environment

The potential of improving human productivity by providing healthy indoor environments has been a consistent interest in the building field for decades. This research field's long-standing challenge is to measure human productivity given the complex nature of office work. Previous studies have diversified productivity metrics, allowing greater flexibility in collecting human data; however, this diversity complicates the ability to combine productivity metrics from disparate studies within a meta-analysis. This study aims to categorize existing productivity metrics and statistically assess which categories show similar behavior when used to measure the impacts of indoor environmental quality. The 106 productivity metrics compiled were grouped into six productivity metric categories: neurobehavioral speed, accuracy, neurobehavioral response time, call handling time, self-reported productivity, and performance score. Then, this study set neurobehavioral speed as the baseline category given its fitness to the efficiency-based definition of productivity (i.e., output versus input) and conducted three statistical analyses with the other categories to evaluate their similarity. As results, the categories of neurobehavioral response time, self-reported productivity, and call handling time were found to have statistical similarity with neurobehavioral speed. This study contributes to creating a constructive research environment for future meta-analyses to understand which human productivity metrics can be combined with each other.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

SU-E-J-155: Automatic Quantitative Decision Making Metric for 4DCT Image Quality

Purpose: To develop a quantitative decision making metric for automatically detecting irregular breathing using a large patient population that received phase-sorted 4DCT. Methods: This study employed two patient cohorts. Cohort#1 contained 256 patients who received a phasesorted 4DCT. Cohort#2 contained 86 patients who received three weekly phase-sorted 4DCT scans. A previously published technique used a single abdominal surrogate to calculate the ratio of extreme inhalation tidal volume to normal inhalation tidal volume, referred to as the κ metric. Since a single surrogate is standard for phase-sorted 4DCT in radiation oncology clinical practice, tidal volume was not quantified. Without tidal volume, the absolute κ metric could not be determined, so a relative κ (κrel) metric was defined based on the measured surrogate amplitude instead of tidal volume. Receiver operator characteristic (ROC) curves were used to quantitatively determine the optimal cutoff value (jk) and efficiency cutoff value (τk) of κrel to automatically identify irregular breathing that would reduce the image quality of phase-sorted 4DCT. Discriminatory accuracy (area under the ROC curve) of κrel was calculated by a trapezoidal numeric integration technique. Results: The discriminatory accuracy of ?rel was found to be 0.746. The key values of jk and tk were calculated to be 1.45 and 1.72 respectively. For values of ?rel such that jk≤κrel≤τk, the decision to reacquire the 4DCT would be at the discretion of the physician. This accounted for only 11.9% of the patients in this study. The magnitude of κrel held consistent over 3 weeks for 73% of the patients in cohort#3. Conclusion: The decision making metric, ?rel, was shown to be an accurate classifier of irregular breathing patients in a large patient population. This work provided an automatic quantitative decision making metric to quickly and accurately assess the extent to which irregular breathing is occurring during phase-sorted 4DCT.

60 APPLIED LIFE SCIENCES↗

Resilience Metrics for the Electric Power System: A Performance-Based Approach.

Grid resilience is a concept related to a power system's ability to continue operating and delivering power even in the event that low probability, high-consequence disruptions such as hurricanes, earthquakes, and cyber-attacks occur. Grid resilience objectives focus on managing and, ideally, minimizing potential consequences that occur as a result of these disruptions. Currently, no formal grid resilience definitions, metrics, or analysis methods have been universally accepted. This document describes an effort to develop and describe grid resilience metrics and analysis methods. The metrics and methods described herein extend upon the Resilience Analysis Process (RAP) developed by Watson et al. for the 2015 Quadrennial Energy Review. The extension allows for both outputs from system models and for historical data to serve as the basis for creating grid resilience metrics and informing grid resilience planning and response decision-making. This document describes the grid resilience metrics and analysis methods. Demonstration of the metrics and methods is shown through a set of illustrative use cases.

24 POWER TRANSMISSION AND DISTRIBUTION↗

The Photometric LSST Astronomical Time-series Classification Challenge PLAsTiCC: Selection of a Performance Metric for Classification Probabilities Balancing Diverse Science Goals

Classification of transient and variable light curves is an essential step in using astronomical observations to develop an understanding of the underlying physical processes from which they arise. However, upcoming deep photometric surveys, including the Large Synoptic Survey Telescope (LSST), will produce a deluge of low signal-to-noise data for which traditional type estimation procedures are inappropriate. Probabilistic classification is more appropriate for such data but is incompatible with the traditional metrics used on deterministic classifications. Furthermore, large survey collaborations like LSST intend to use the resulting classification probabilities for diverse science objectives, indicating a need for a metric that balances a variety of goals. We describe the process used to develop an optimal performance metric for an open classification challenge that seeks to identify probabilistic classifiers that can serve many scientific interests. The Photometric LSST Astronomical Time-series Classification Challenge (PLASTICC) aims to identify promising techniques for obtaining classification probabilities of transient and variable objects by engaging a broader community beyond astronomy. Using mock classification probability submissions emulating realistically complex archetypes of those anticipated of PLASTICC, we compare the sensitivity of two metrics of classification probabilities under various weighting schemes, finding that both yield results that are qualitatively consistent with intuitive notions of classification performance. We thus choose as a metric for PLASTICC a weighted modification of the cross-entropy because it can be meaningfully interpreted in terms of information content. Finally, we propose extensions of our methodology to ever more complex challenge goals and suggest some guiding principles for approaching the choice of a metric of probabilistic data products.

79 ASTRONOMY AND ASTROPHYSICS↗

Handbook of aircraft noise metrics

Information is presented on 22 noise metrics that are associated with the measurement and prediction of the effects of aircraft noise. Some of the instantaneous frequency weighted sound level measures, such as A-weighted sound level, are used to provide multiple assessment of the aircraft noise level. Other multiple event metrics, such as day-night average sound level, were designed to relate sound levels measured over a period of time to subjective responses in an effort to determine compatible land uses and aid in community planning. The various measures are divided into: (1) instantaneous sound level metrics; (2) duration corrected single event metrics; (3) multiple event metrics; and (4) speech communication metrics. The scope of each measure is examined in terms of its: definition, purpose, background, relationship to other measures, calculation method, example, equipment, references, and standards.

Bennett, R. L.↗

The independence of software metrics taken at different life-cycle stages

Over the past few years a large number of software metrics have been proposed and, in varying degrees, a number of these metrics have been subjected to empirical validation which demonstrated the utility of the metrics in the software development process. Attempts to classify these metrics and to determine if the metrics in these different classes appear to be measuring distinct attributes of the software product are studied. Statistical analysis is used to determine the degree of relationship among the metrics.

Kafura, D.↗

Metric analysis and data validation across FORTRAN projects

The desire to predict the effort in developing or explaining the quality of software has led to the proposal of several metrics. As a step toward validating these metrics, the Software Engineering Laboratory (SEL) has analyzed the software science metrics, cyclomatic complexity, and various standard program measures for their relation to effort (including design through acceptance testing), development errors (both discrete and weighted according to the amount of time to locate and fix), and one another. The data investigated are collected from a project FORTRAN environment and examined across several projects at once, within individual projects and by reporting accuracy checks demonstrating the need to validate a database. When the data comes from individual programmers or certain validated projects, the metrics' correlations with actual effort seem to be strongest. For modules developed entirely by individual programmers, the validity ratios induce a statistically significant ordering of several of the metrics' correlations. When comparing the strongest correlations, neither software science's E metric cyclomatic complexity not source lines of code appears to relate convincingly better with effort than the others.

Basili, Victor R.↗

Fighter agility metrics, research, and test

Proposed new metrics to assess fighter aircraft agility are collected and analyzed. A framework for classification of these new agility metrics is developed and applied. A completed set of transient agility metrics is evaluated with a high fidelity, nonlinear F-18 simulation provided by the NASA Dryden Flight Research Center. Test techniques and data reduction methods are proposed. A method of providing cuing information to the pilot during flight test is discussed. The sensitivity of longitudinal and lateral agility metrics to deviations from the pilot cues is studied in detail. The metrics are shown to be largely insensitive to reasonable deviations from the nominal test pilot commands. Instrumentation required to quantify agility via flight test is also considered. With one exception, each of the proposed new metrics may be measured with instrumentation currently available. Simulation documentation and user instructions are provided in an appendix.

Liefer, Randall K.↗

A Validation of Object-Oriented Design Metrics

This paper presents the results of a study conducted at the University of Maryland in which we experimentally investigated the suite of Object-Oriented (00) design metrics introduced by [Chidamber and Kemerer, 1994]. In order to do this, we assessed these metrics as predictors of fault-prone classes. This study is complementary to [Lieand Henry, 1993] where the same suite of metrics had been used to assess frequencies of maintenance changes to classes. To perform our validation accurately, we collected data on the development of eight medium-sized information management systems based on identical requirements. All eight projects were developed using a sequential life cycle model, a well-known 00 analysis/design method and the C++ programming language. Based on experimental results, the advantages and drawbacks of these 00 metrics are discussed and suggestions for improvement are provided. Several of Chidamber and Kemerer's 00 metrics appear to be adequate to predict class fault-proneness during the early phases of the life-cycle. We also showed that they are, on our data set, better predictors than "traditional" code metrics, which can only be collected at a later phase of the software development processes.

Basili, Victor R.↗

A Validation of Object-Oriented Design Metrics as Quality Indicators

This paper presents the results of a study in which we empirically investigated the suits of object-oriented (00) design metrics introduced in another work. More specifically, our goal is to assess these metrics as predictors of fault-prone classes and, therefore, determine whether they can be used as early quality indicators. This study is complementary to the work described where the same suite of metrics had been used to assess frequencies of maintenance changes to classes. To perform our validation accurately, we collected data on the development of eight medium-sized information management systems based on identical requirements. All eight projects were developed using a sequential life cycle model, a well-known 00 analysis/design method and the C++ programming language. Based on empirical and quantitative analysis, the advantages and drawbacks of these 00 metrics are discussed. Several of Chidamber and Kamerer's 00 metrics appear to be useful to predict class fault-proneness during the early phases of the life-cycle. Also, on our data set, they are better predictors than 'traditional' code metrics, which can only be collected at a later phase of the software development processes.

Basili, Victor R.↗

Testing, Requirements, and Metrics

The criticality of correct, complete, testable requirements is a fundamental tenet of software engineering. Also critical is complete requirements based testing of the final product. Modern tools for managing requirements allow new metrics to be used in support of both of these critical processes. Using these tools, potential problems with the quality of the requirements and the test plan can be identified early in the life cycle. Some of these quality factors include: ambiguous or incomplete requirements, poorly designed requirements databases, excessive or insufficient test cases, and incomplete linkage of tests to requirements. This paper discusses how metrics can be used to evaluate the quality of the requirements and test to avoid problems later. Requirements management and requirements based testing have always been critical in the implementation of high quality software systems. Recently, automated tools have become available to support requirements management. At NASA's Goddard Space Flight Center (GSFC), automated requirements management tools are being used on several large projects. The use of these tools opens the door to innovative uses of metrics in characterizing test plan quality and assessing overall testing risks. In support of these projects, the Software Assurance Technology Center (SATC) is working to develop and apply a metrics program that utilizes the information now available through the application of requirements management tools. Metrics based on this information provides real-time insight into the testing of requirements and these metrics assist the Project Quality Office in its testing oversight role. This paper discusses three facets of the SATC's efforts to evaluate the quality of the requirements and test plan early in the life cycle, thus preventing costly errors and time delays later.

Rosenberg, Linda↗

A Sensor-Independent Gust Hazard Metric

A procedure for calculating an intuitive hazard metric for gust effects on airplanes is described. The hazard metric is for use by pilots and is intended to replace subjective pilot reports (PIREPs) of the turbulence level. The hazard metric is composed of three numbers: the first describes the average airplane response to the turbulence, the second describes the positive peak airplane response to the gusts, and the third describes the negative peak airplane response to the gusts. The hazard metric is derived from any time history of vertical gust measurements and is thus independent of the sensor making the gust measurements. The metric is demonstrated for one simulated airplane encountering different types of gusts including those derived from flight data recorder measurements of actual accidents. The simulated airplane responses to the gusts compare favorably with the hazard metric.

Stewart, Eric C.↗

Evaluation of Standard Gear Metrics in Helicopter Flight Operation

Each false alarm made by a machine monitoring system carries a high price tag. The machine must be taken out of service, thoroughly inspected with possible disassembly, and then made ready for service. Loss of use of the machine and the efforts to inspect it are costly. In addition, if a monitoring system is prone to false alarms, the system will soon be turned off or ignored. For aircraft applications, one growing concern is that the dynamic flight environment differs from the laboratory environment where fault detection methods are developed and tested. Vibration measurements made in flight are less stationary than those made in a laboratory, or test facility, and thus a given fault detection method may produce more false alarms in flight than might be anticipated. In 1977. Stewart introduced several metrics, including FM0 and FM4, for evaluating the health of a gear. These metrics are single valued functions of the vibration signal that indicate if the signal deviates from an ideal model of the signal. FM0 is a measure of the ratio of the peak-to-peak level to the harmonic energy in the signal. FM4 is the kurtosis of the signal with the gear mesh harmonics and first order side bands removed. The underlying theory is that a vibration signal from a gear in good condition is expected to be dominated by a periodic signal at the gear mesh frequency. If one or a small number of gear teeth contain damage or faults, the signal will change, possibly showing increased amplitude, local phase changes or both near the damaged region of the gear. FM0 increases if a signal contains a local increase in amplitude. FM4 increases if a signal contains a local increase in amplitude or local phase change in a periodic signal. Over the years, other single value metrics were also introduced to detect the onset and growth of damage in gears. These various metrics have detected faults in several gear tests in experimental test rigs. Conditions in these tests have been steady state in the sense that the rpm, torque and forces on the gear have been held steady. For gears used in a dynamic environment such as that occurring in aircraft, the rpm, torque and forces on the gear are constantly changing. The authors have measured significant variation in rpm and torque in the transmissions of helicopters in controlled steady flight conditions flown by highly proficient test pilots. Statistical analyses of the data taken in flight show significant nonstationarity in the vibration measurements. These deviations from stationarity may increase false alarms in gear monitoring during aircraft flight. In the proposed paper, the authors will study vibration measurements made in flight on an AH- 1 Cobra and an OH-58C Kiowa helicopters. The primary focus will be the development of a methodology to assess the impact of nonstationarity on false alarms. Issues to be addressed include how time synchronous averages are constructed from raw data as well as how lack of stationarity effects the behavior of single value metrics. Emphasis will be placed on the occurrence of false alarms with the use of standard metrics. In order to maintain an acceptable level of false alarms in the flight environment, this study will also address the determination of appropriate threshold levels, which may need to be higher than for test rigs.

Mosher, M.↗

NASA Aviation Safety Program Systems Analysis/Program Assessment Metrics Review

The goal of this project is to evaluate the metrics and processes used by NASA's Aviation Safety Program in assessing technologies that contribute to NASA's aviation safety goals. There were three objectives for reaching this goal. First, NASA's main objectives for aviation safety were documented and their consistency was checked against the main objectives of the Aviation Safety Program. Next, the metrics used for technology investment by the Program Assessment function of AvSP were evaluated. Finally, other metrics that could be used by the Program Assessment Team (PAT) were identified and evaluated. This investigation revealed that the objectives are in fact consistent across organizational levels at NASA and with the FAA. Some of the major issues discussed in this study which should be further investigated, are the removal of the Cost and Return-on-Investment metrics, the lack of the metrics to measure the balance of investment and technology, the interdependencies between some of the metric risk driver categories, and the conflict between 'fatal accident rate' and 'accident rate' in the language of the Aviation Safety goal as stated in different sources.

Louis, Garrick E.↗

Towards a Visual Quality Metric for Digital Video

The advent of widespread distribution of digital video creates a need for automated methods for evaluating visual quality of digital video. This is particularly so since most digital video is compressed using lossy methods, which involve the controlled introduction of potentially visible artifacts. Compounding the problem is the bursty nature of digital video, which requires adaptive bit allocation based on visual quality metrics. In previous work, we have developed visual quality metrics for evaluating, controlling, and optimizing the quality of compressed still images. These metrics incorporate simplified models of human visual sensitivity to spatial and chromatic visual signals. The challenge of video quality metrics is to extend these simplified models to temporal signals as well. In this presentation I will discuss a number of the issues that must be resolved in the design of effective video quality metrics. Among these are spatial, temporal, and chromatic sensitivity and their interactions, visual masking, and implementation complexity. I will also touch on the question of how to evaluate the performance of these metrics.

Watson, Andrew B.↗

Metrics for Linear Kinematic Features in Sea Ice

The treatment of leads as cracks or discontinuities (see Coon et al. presentation) requires some shift in the procedure of evaluation and comparison of lead-resolving models and their validation against observations. Common metrics used to evaluate ice model skills are by and large an adaptation of a least square "metric" adopted from operational numerical weather prediction data assimilation systems and are most appropriate for continuous fields and Eilerian systems where the observations and predictions are commensurate. However, this class of metrics suffers from some flaws in areas of sharp gradients and discontinuities (e.g., leads) and when Lagrangian treatments are more natural. After a brief review of these metrics and their performance in areas of sharp gradients, we present two new metrics specifically designed to measure model accuracy in representing linear features (e.g., leads). The indices developed circumvent the requirement that both the observations and model variables be commensurate (i.e., measured with the same units) by considering the frequencies of the features of interest/importance. We illustrate the metrics by scoring several hypothetical "simulated" discontinuity fields against the lead interpreted from RGPS observations.

Levy, G.↗

Evaluating Algorithm Performance Metrics Tailored for Prognostics

Prognostics has taken a center stage in Condition Based Maintenance (CBM) where it is desired to estimate Remaining Useful Life (RUL) of the system so that remedial measures may be taken in advance to avoid catastrophic events or unwanted downtimes. Validation of such predictions is an important but difficult proposition and a lack of appropriate evaluation methods renders prognostics meaningless. Evaluation methods currently used in the research community are not standardized and in many cases do not sufficiently assess key performance aspects expected out of a prognostics algorithm. In this paper we introduce several new evaluation metrics tailored for prognostics and show that they can effectively evaluate various algorithms as compared to other conventional metrics. Specifically four algorithms namely; Relevance Vector Machine (RVM), Gaussian Process Regression (GPR), Artificial Neural Network (ANN), and Polynomial Regression (PR) are compared. These algorithms vary in complexity and their ability to manage uncertainty around predicted estimates. Results show that the new metrics rank these algorithms in different manner and depending on the requirements and constraints suitable metrics may be chosen. Beyond these results, these metrics offer ideas about how metrics suitable to prognostics may be designed so that the evaluation procedure can be standardized. 1

Saxena, Abhinav↗