SEARCH · Engineering Papers

Results for “Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

All sources NASA NTRS DOE OSTI

NASA NTRS1991

Candidate control design metrics for an agile fighter

Success in the fighter combat environment of the future will certainly demand increasing capability from aircraft technology. These advanced capabilities in the form of superagility and supermaneuverability will require special design techniques which translate advanced air combat maneuvering requirements into design criteria. Control design metrics can provide some of these techniques for the control designer. Thus study presents an overview of control design metrics and investigates metrics for advanced fighter agility. The objectives of various metric users, such as airframe designers and pilots, are differentiated from the objectives of the control designer. Using an advanced fighter model, metric values are documented over a portion of the flight envelope through piloted simulation. These metric values provide a baseline against which future control system improvements can be compared and against which a control design methodology can be developed. Agility is measured for axial, pitch, and roll axes. Axial metrics highlight acceleration and deceleration capabilities under different flight loads and include specific excess power measurements to characterize energy meneuverability. Pitch metrics cover both body-axis and wind-axis pitch rates and accelerations. Included in pitch metrics are nose pointing metrics which highlight displacement capability between the nose and the velocity vector. Roll metrics (or torsion metrics) focus on rotational capability about the wind axis.

Murphy, Patrick C.↗

NASA NTRSDate not supplied

A Framework for Evaluating Climate Model Performance Metrics

The CMIP5 archive contains future climate projections from over 50 models provided by dozens of modeling centers from around the world. Individual model projections, however, are subject to biases created by structural model uncertainties. As a result, ensemble averaging of multiple models is often used to add value to model projections: consensus projections have been shown to consistently outperform individual models. Previous reports for the IPCC establish climate change projections based on an equal-weighted average of all model projections. However, certain models reproduce climate processes better than other models. Should models be weighted based on performance? Unequal ensemble averages have previously been constructed using a variety of mean state metrics. What metrics are most relevant for constraining future climate projections? This project develops a framework for systematically testing metrics in models to identify optimal metrics for unequal weighting multi-model ensembles. A unique aspect of this project is the construction and testing of climate process-based model evaluation metrics. A climate process-based metric is defined as a metric based on the relationship between two physically related climate variables?e.g., outgoing longwave radiation and surface temperature. Metrics are constructed using high-quality Earth radiation budget data from NASA's Clouds and Earth's Radiant Energy System (CERES) instrument and surface temperature data sets. It is found that regional values of tested quantities can vary significantly when comparing weighted and unweighted model ensembles. For example, one tested metric weights the ensemble by how well models reproduce the time-series probability distribution of the cloud forcing component of reflected shortwave radiation. The weighted ensemble for this metric indicates lower simulated precipitation (up to .7 mm/day) in tropical regions than the unweighted ensemble: since CMIP5 models have been shown to overproduce precipitation, this result could indicate that the metric is effective in identifying models which simulate more realistic precipitation. Ultimately, the goal of the framework is to identify performance metrics for advising better methods for ensemble averaging models and create better climate predictions.

Noel C Baker↗

DOE OSTI2015

Attenuation-based size metric for estimating organ dose to patients undergoing tube current modulated CT exams

Purpose: Task Group 204 introduced effective diameter (ED) as the patient size metric used to correlate size-specific-dose-estimates. However, this size metric fails to account for patient attenuation properties and has been suggested to be replaced by an attenuation-based size metric, water equivalent diameter (D{sub W}). The purpose of this study is to investigate different size metrics, effective diameter, and water equivalent diameter, in combination with regional descriptions of scanner output to establish the most appropriate size metric to be used as a predictor for organ dose in tube current modulated CT exams. Methods: 101 thoracic and 82 abdomen/pelvis scans from clinically indicated CT exams were collected retrospectively from a multidetector row CT (Sensation 64, Siemens Healthcare) with Institutional Review Board approval to generate voxelized patient models. Fully irradiated organs (lung and breasts in thoracic scans and liver, kidneys, and spleen in abdominal scans) were segmented and used as tally regions in Monte Carlo simulations for reporting organ dose. Along with image data, raw projection data were collected to obtain tube current information for simulating tube current modulation scans using Monte Carlo methods. Additionally, previously described patient size metrics [ED, D{sub W}, and approximated water equivalent diameter (D{sub Wa})] were calculated for each patient and reported in three different ways: a single value averaged over the entire scan, a single value averaged over the region of interest, and a single value from a location in the middle of the scan volume. Organ doses were normalized by an appropriate mAs weighted CTDI{sub vol} to reflect regional variation of tube current. Linear regression analysis was used to evaluate the correlations between normalized organ doses and each size metric. Results: For the abdominal organs, the correlations between normalized organ dose and size metric were overall slightly higher for all three differently (global, regional, and middle slice) reported D{sub W} and D{sub Wa} than they were for ED, but the differences were not statistically significant. However, for lung dose, computed correlations using water equivalent diameter calculated in the middle of the image data (D{sub W,middle}) and averaged over the low attenuating region of lung (D{sub W,regional}) were statistically significantly higher than correlations of normalized lung dose with ED. Conclusions: To conclude, effective diameter and water equivalent diameter are very similar in abdominal regions; however, their difference becomes noticeable in lungs. Water equivalent diameter, specifically reported as a regional average and middle of scan volume, was shown to be better predictors of lung dose. Therefore, an attenuation-based size metric (water equivalent diameter) is recommended because it is more robust across different anatomic regions. Additionally, it was observed that the regional size metric reported as a single value averaged over a region of interest and the size metric calculated from a single slice/image chosen from the middle of the scan volume are highly correlated for these specific patient models and scan types.

60 APPLIED LIFE SCIENCES↗

NASA NTRS1973

Metrication study for large space telescope

Various approaches which could be taken in developing a metric-system design for the Large Space Telescope, considering potential penalties on development cost and time, commonality with other satellite programs, and contribution to national goals for conversion to the metric system of units were investigated. Information on the problems, potential approaches, and impacts of metrication was collected from published reports on previous aerospace-industry metrication-impact studies and through numerous telephone interviews. The recommended approach to LST metrication formulated in this study cells for new components and subsystems to be designed in metric-module dimensions, but U.S. customary practice is allowed where U.S. metric standards and metric components are not available or would be unsuitable. Electrical/electronic-system design, which is presently largely metric, is considered exempt from futher metrication. An important guideline is that metric design and fabrication should in no way compromise the effectiveness of the LST equipment.

Creswick, F. A.↗

NASA NTRSDate not supplied

An Impedance-Based Complexity Metric for Unmanned Aircraft System Traffic Scenario Classification

This paper introduces an impedance-based metric to capture the complexity of a given unmanned aircraft system traffic scenario. The metric accounts for both the number of aircraft and the traffic flow pattern. The work presented here extends an earlier approach that introduced another scenario complexity metric based on the number of potential conflicts weighted by the conflict resolution cost associated. Complexity measurements for randomly-generated scenarios were produced through high-fidelity fast-time simulations and treated as baseline. Then the impedance based metric was evaluated, for the same scenarios, without the need for an actual flight simulation and a conflict resolution method. The results show that the impedance-based metric has a strong correlation to the baseline data and performs marginally better than the weighted conflict-based complexity metric introduced in the earlier work. The metric computation generates impedance maps which are useful for identifying high complexity regions in a scenario, where flight plan changes might be necessitated. This metric can therefore be used, in conjunction with other complexity metrics, to inform adequate traffic management strategies and classify a traffic scenario as acceptable, unacceptable or acceptable with changes made to flight plans that pass through the high complexity regions. The metric can also be used as a guidance metric for strategic conflict management methods.

Complexity↗

DOE OSTI2014

En route to Background Independence: Broken split-symmetry, and how to restore it with bi-metric average actions

The most momentous requirement a quantum theory of gravity must satisfy is Background Independence, necessitating in particular an ab initio derivation of the arena all non-gravitational physics takes place in, namely spacetime. Using the background field technique, this requirement translates into the condition of an unbroken split-symmetry connecting the (quantized) metric fluctuations to the (classical) background metric. If the regularization scheme used violates split-symmetry during the quantization process it is mandatory to restore it in the end at the level of observable physics. In this paper we present a detailed investigation of split-symmetry breaking and restoration within the Effective Average Action (EAA) approach to Quantum Einstein Gravity (QEG) with a special emphasis on the Asymptotic Safety conjecture. In particular we demonstrate for the first time in a non-trivial setting that the two key requirements of Background Independence and Asymptotic Safety can be satisfied simultaneously. Carefully disentangling fluctuation and background fields, we employ a ‘bi-metric’ ansatz for the EAA and project the flow generated by its functional renormalization group equation on a truncated theory space spanned by two separate Einstein–Hilbert actions for the dynamical and the background metric, respectively. A new powerful method is used to derive the corresponding renormalization group (RG) equations for the Newton- and cosmological constant, both in the dynamical and the background sector. We classify and analyze their solutions in detail, determine their fixed point structure, and identify an attractor mechanism which turns out instrumental in the split-symmetry restoration. We show that there exists a subset of RG trajectories which are both asymptotically safe and split-symmetry restoring: In the ultraviolet they emanate from a non-Gaussian fixed point, and in the infrared they loose all symmetry violating contributions inflicted on them by the non-invariant functional RG equation. As an application, we compute the scale dependent spectral dimension which governs the fractal properties of the effective QEG spacetimes at the bi-metric level. Earlier tests of the Asymptotic Safety conjecture almost exclusively employed ‘single-metric truncations’ which are blind towards the difference between quantum and background fields. We explore in detail under which conditions they can be reliable, and we discuss how the single-metric based picture of Asymptotic Safety needs to be revised in the light of the new results. We shall conclude that the next generation of truncations for quantitatively precise predictions (of critical exponents, for instance) is bound to be of the bi-metric type. - Highlights: • The Asymptotic Safety scenario in quantum gravity is explored. • A bi-metric generalization of the Einstein–Hilbert truncation is investigated. • We find that Background Independence can coexist with Asymptotic Safety. • RG trajectories restoring (background-quantum) split-symmetry are constructed. • The degree of validity of single-metric truncations is critically assessed.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2018

Catalog of Environmental Metrics for Hydropower

Evaluating the sustainability of new U.S. hydropower development will necessitate an approach that is comprehensive, transparent and consistent. The ORNL Environmental Metrics for Hydropower (EMH) project team addressed this need by conducting a literature review to extract and analyze a swath of environmental metrics related to hydropower projects. The assessed documents included peer-reviewed journal articles, U.S. Federal Energy Regulatory Commission licensing documents, and two sets of certification documents from the Low Impact Hydropower Institute and the International Hydropower Association. Review of 117 documents led to creation of a database of > 3000 current environmental metrics related to hydropower projects in 285 locations worldwide. Information recorded for each environmental metric was used to map the comprehensiveness of environmental measurement types gathered across primary categories, dam life cycle stages, spatial scales and locations. A majority of metrics were related to water quantity and water quality, a large number of metrics were related to biota and biodiversity, and relatively few metrics were related to the categories of geomorphology, land cover, connectivity and fragmentation, and infrastructure and design. Forty-five subcategories of environmental metrics emerged from our analysis, thereby suggesting an envelope of measurements that are important for understanding the environmental effects of new hydropower projects. The categories and subcategories of environmental metrics represented in each type of literature varied, however. The EMH database summarized in this Catalog will be used to inform stakeholder selection of appropriate environmental metrics for evaluating progress toward collective goals for new U.S. hydropower development.

13 HYDRO ENERGY↗

NASA NTRS1995

Goal-Driven Definition of Product Metrics Based on Properties

Defining product metrics requires a rigorous and disciplined approach, because useful metrics depend, to a very large extent, on one's goals and assumptions about the studied software process. Unlike in more mature scientific fields, it appears difficult to devise a "universal" set of metrics in software engineering, that can be used across application environments. We propose an approach for the definition of product metrics which is driven by the experimental goals of measurement, expressed via the Goal/Question/Metric (GQM) paradigm, and is based on the mathematical properties of the metrics. This approach integrates several research contributions from the literature into a consistent, practical and rigorous approach. The approach we outline should not be considered as a complete and definitive solution, but as a starting point for discussion about a product metric definition approach widely accepted in the software engineering community. At this point, we intend to provide an intellectual process that we think is necessary to define sound software product metrics. A precise and complete documentation of such an approach will provide the information needed to make the assessment and reuse of a new metric possible. Thus, product metrics are supported by a solid theory which facilitates their review and refinement. Moreover, their definition is made less exploratory and, as a consequence, one is less likely to identify spurious correlations between process and product metrics.

Briand, Lionel↗

NASA NTRS2007

A Contrast in Use of Metrics in Earth Science Data Systems

In recent years there has been a surge in the number of systems for processing, archiving and distributing remotely sensed data. Such systems, working independently as well as in collaboration, have been contributing greatly to the advances in the scientific understanding of the Earth system, as well as utilization of the data for nationally and internationally important applications. Among such systems, we consider those that are developed by or under the sponsorship of NASA to fulfill one of its strategic objectives: "Study Earth from space to advance scientific understanding and meet societal needs." NASA's Earth science data systems are of varying size and complexity depending on the requirements they are intended to meet. Some data systems are regarded as NASA's "Core Capabilities" that provide the basic infrastructure for processing, archiving and distributing a set of data products to a large and diverse user community in a robust and reliable manner. Other data systems constitute "Community Capabilities". These provide specialized and innovative services to data users and/or research products offering new scientific insight. Such data systems are generally supported by NASA through peer reviewed competition. Examples of Core Capabilities are 1. Earth Observing Data and Information System (EOSDIS) with its Distributed Active Archive Centers (DAACs), Science Investigator-led Processing Systems (SIPSs), and the EOS Clearing House (ECHO); 2. Tropical Rainfall Measurement Mission (TRMM) Science Data and Information System (TSDIS); 3. Ocean Data Processing System (ODPS); and 4. CloudSat Data Processing Center. Examples of Community Capabilities are projects under the Research, Education and Applications Solutions Network (REASON), and Advancing Collaborative Connections for Earth System Science (ACCESS) Programs. In managing these data system capabilities, it is necessary to have well-established goals and to measure progress relative to them. Progress is measured through "metrics", which can be a combination of quantitative as well as qualitative assessments. The specific metrics of interest depend on the user of the metrics as well as the type of data system. The users of metrics can be data system managers, program managers, funding agency or the public. Data system managers need metrics for assessing and improving the performance of the system and for future planning. Program managers need metrics to assess progress and the value of the data systems sponsored by them. Also, there is a difference in the metrics needed for core capabilities that tend to be more complex, larger and longer-term compared to community capabilities and the community capabilities that tend to be simpler, smaller and shorter-term. Even among community capabilities there are differences; hence the same set of metrics does not apply to all. Some provide data products to users, some provide services that enable better utilization of data or interoperability among other systems, and some are a part of a larger project where provision of data or services is only a minor activity. There is also a contrast between metrics used for internal and external purposes. Examples of internal purposes are: ensuring that the system meets its requirements, and planning for evolution and growth. Examples of external purposes are: providing to sponsors indicators of success of the systems, demonstrating the contributions of the system to overall program success, etc. This paper will consider EOSDIS, REASON and ACCESS programs to show the various types of metrics needed and how they need to be tailored to the types of data systems while maintaining the overall management goals of measuring progress and contributions made by the data systems.

Ramapriyan, Hampapuram↗

NASA NTRS2010

Coverage Metrics for Requirements-Based Testing: Evaluation of Effectiveness

In black-box testing, the tester creates a set of tests to exercise a system under test without regard to the internal structure of the system. Generally, no objective metric is used to measure the adequacy of black-box tests. In recent work, we have proposed three requirements coverage metrics, allowing testers to objectively measure the adequacy of a black-box test suite with respect to a set of requirements formalized as Linear Temporal Logic (LTL) properties. In this report, we evaluate the effectiveness of these coverage metrics with respect to fault finding. Specifically, we conduct an empirical study to investigate two questions: (1) do test suites satisfying a requirements coverage metric provide better fault finding than randomly generated test suites of approximately the same size?, and (2) do test suites satisfying a more rigorous requirements coverage metric provide better fault finding than test suites satisfying a less rigorous requirements coverage metric? Our results indicate (1) only one coverage metric proposed -- Unique First Cause (UFC) coverage -- is sufficiently rigorous to ensure test suites satisfying the metric outperform randomly generated test suites of similar size and (2) that test suites satisfying more rigorous coverage metrics provide better fault finding than test suites satisfying less rigorous coverage metrics.

Staats, Matt↗

NASA NTRSDate not supplied

Mineralogical, Elemental, and Tomographic Reconnaissance Investigation for CLPS (METRIC): A Payload Designed for Exploration of Terrestrial Planetary Bodies

Geological materials (indeed, all solid objects) are characterized by their crystal structure, elemental composition, and morphology. The Mineralogical, Elemental, and Tomographic Reconnaissance Investigation for CLPS (METRIC) instrument suite quantifies all three. These measurements address fundamental science questions (e.g., the origin and evolution of planetary bodies) and support the human exploration of space (e.g., the characterization of regolith for ISRU and the constraint of its geotechnical properties). METRIC comprises an X-ray Diffraction/X-ray Fluorescence instrument (XRD: mineral structure and XRF: elemental composition), an X-ray micro-Computed Tomography instrument (XCT: 3D internal micromorphology), and a hyperspectral imaging infrared spectrometer (IRS) to provide local/regional mineralogic context for these measurements. METRIC XRD/F draws heritage from the highly successful Mars Science Laboratory CheMin instrument. The METRIC XRD/F employs two separate sample cells, one optimized for XRD and one for XRF, resulting in more rapid XRD analysis (tens of minutes vs. tens of hours for CheMin) and an orders-of-magnitude improvement in XRF detection. XCT has not been deployed in space, so the METRIC XCT represents a new capability for solar system exploration. The XCT uses the same basic high-TRL components as METRIC XRD/F, decreasing its development cost for flight. The METRIC IRS is a derivative of the NASA Earth Science Technology Office funded Hyperspectral Thermal Imager instrument and utilizes the NASA Technology Transfer Program to incorporate a commercial-of-the-shelf infrared camera ruggedized for space by NASA Marshall Space Flight Center. The IRS spectral range (8–14 µm) and resolution (10.8 cm -1 ) are tailored to quantify mineralogy in rocks using their characteristic Reststrahlen bands and to characterize mineralogy of soils using the position of the Christensen Feature. The METRIC payload is currently designed for deployment to the Moon on a Commercial Lunar Payload Services (CLPS) mission, where the XRD/F and XCT would be located on a lander and the IRS would be on deployed on a companion rover to evaluate the mineralogical diversity of the landing site. A pneumatic drill designed by Honeybee Robotics would excavate regolith up to 50 cm below the lander and deliver multiple aliquots of regolith to the XRD/F and XCT. The METRIC payload could also be deployed on a rover. In this case, a sample handling system on a robotic arm could scoop regolith and/or drill rocks and deliver powder to the XRD/F and XCT located in the rover’s interior. Alternatively, METRIC instruments could be used singly or in combination on human space missions. The XRD/F and XCT could be used to characterize samples in a rover or in a science laboratory within a habitat. These data could help astronauts identify resource-enriched rocks and regolith and triage geologic samples to return samples of high interest for analysis in terrestrial laboratories. The IRS could be attached to a human-navigated rover to collect mineralogical data along a traverse and identify high-priority science samples.

E. B. Rampe↗

DOE OSTI2019

Review of environmental metrics used across multiple sectors and geographies to evaluate the effects of hydropower development

In this paper, as U.S. hydropower development continues to expand, a thorough inventory of potentially relevant environmental metrics will provide a comprehensive and transparent starting point for stakeholders seeking to evaluate the sustainability and environmental effects of this anticipated expansion. To address this need, we conducted a literature review to extract and analyze a swath of environmental metrics related to previous hydropower projects. Our examination of several U.S. Federal Energy Regulatory Commission licensing documents, several certification documents prepared by the Low Impact Hydropower Institute and the International Hydropower Association, and a set of peer-reviewed journal articles led to creation of a database of > 3000 current environmental metrics related to hydropower projects in 231 locations worldwide. Information recorded for each environmental metric was used to map the comprehensiveness and consistency of environmental measurements gathered across primary categories, dam life cycle stages, spatial scales and locations. We found that a majority of environmental metrics were related to water quantity and water quality, a large number of metrics were related to biota and biodiversity, and relatively few metrics were related to the categories of geomorphology, land cover, connectivity and fragmentation, and infrastructure and design. However, the categories of environmental metrics represented in each type of literature varied significantly. Forty-five subcategories of environmental metrics emerged from our analysis, thereby suggesting a potential envelope of environmental measurements that might be used to improve efficiency in evaluating the potential environmental effects of new U.S. hydropower projects.

13 HYDRO ENERGY↗

DOE OSTI2016

WE-H-BRC-05: Catastrophic Error Metrics for Radiation Therapy

Purpose: Intuitive evaluation of complex radiotherapy treatments is impractical, while data transfer anomalies create the potential for catastrophic treatment delivery errors. Contrary to prevailing wisdom, logical scrutiny can be applied to patient-specific machine settings. Such tests can be automated, applied at the point of treatment delivery and can be dissociated from prior states of the treatment plan, potentially revealing errors introduced early in the process. Methods: Analytical metrics were formulated for conventional and intensity modulated RT (IMRT) treatments. These were designed to assess consistency between monitor unit settings, wedge values, prescription dose and leaf positioning (IMRT). Institutional metric averages for 218 clinical plans were stratified over multiple anatomical sites. Treatment delivery errors were simulated using a commercial treatment planning system and metric behavior assessed via receiver-operator-characteristic (ROC) analysis. A positive result was returned if the erred plan metric value exceeded a given number of standard deviations, e.g. 2. The finding was declared true positive if the dosimetric impact exceeded 25%. ROC curves were generated over a range of metric standard deviations. Results: Data for the conventional treatment metric indicated standard deviations of 3%, 12%, 11%, 8%, and 5 % for brain, pelvis, abdomen, lung and breast sites, respectively. Optimum error declaration thresholds yielded true positive rates (TPR) between 0.7 and 1, and false positive rates (FPR) between 0 and 0.2. Two proposed IMRT metrics possessed standard deviations of 23% and 37%. The superior metric returned TPR and FPR of 0.7 and 0.2, respectively, when both leaf position and MUs were modelled. Isolation to only leaf position errors yielded TPR and FPR values of 0.9 and 0.1. Conclusion: Logical tests can reveal treatment delivery errors and prevent large, catastrophic errors. Analytical metrics are able to identify errors in monitor units, wedging and leaf positions with favorable sensitivity and specificity. In part by Varian.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2022

Toward Human-Centric Transportation and Energy Metrics: Influence of Mode, Vehicle Occupancy, Trip Distance, and Fuel Economy

Traditional metrics measuring transportation and energy outcomes can be augmented to better represent impacts on people's lives and systems-level performance. In this context, this study introduces two novel metrics: road capacity (as number of people traveling and accessing services) and energy intensity (as energy use for people traveling and accessing services). Current national-level distributions of available data in the United States for factors contributing to the two new integrated metrics are used as context to evaluate potential outcomes. These factors include vehicle occupancy, mode share, fuel economy, and trip distance. Variations in input values provide insights on how these factors shape efficiencies in road capacity and energy intensity. Parametric sensitivity analysis indicates that the impact of each input depends upon the metric being evaluated. For the human-centered road capacity mobility metric, increasing vehicle occupancy has the largest effect – twice that of increasing mode share for bike, walk, and transit. For the energy intensity mobility metric, the effect of improving fuel economy is the largest. However, when the focus is on accessibility (instead of mobility), for both metrics the effect of lowering average trip distance is the largest. Additionally, a novel interactive tool to visualize the results for various parameter combinations makes the metrics practitioner ready. The findings suggest that the diffusion of new human-centric metrics that benchmark outcomes associated with road capacity and energy may be significant in motivating new sustainable transportation investments and efficient utilization of infrastructure, mobility assets, and services.

ADVANCED PROPULSION SYSTEMS↗

NASA NTRSDate not supplied

Review of Noise Metric Sensitivities for Analysis of Quiet Supersonic Overflight

Six different noise metrics (PL, ASEL, BSEL, DSEL, ESEL, and ISBAP) are currently used to quantify sonic boom levels from overflight of supersonic aircraft. A previous meta-analysis using laboratory subjective data identified these metrics, each of which correlated well with human perception of low-boom sounds both indoors and outdoors. Because the analysis did not identify a single metric that was significantly superior, no internationally agreed-upon metric has been chosen for the quiet supersonic aircraft noise certification procedures currently under development. Other analyses of metric sensitivities using existing empirical and simulation datasets, however, have shown significant differences between metrics. Variability of metrics due to macro-atmospheric effects and atmospheric turbulence perturbations have been explored, as well as variability in measurements due to microphone setup configurations, array layouts, and ambient noise effects. This paper reviews these prior studies, summarizes the current understanding of metric sensitivities, and discusses how they may impact future downselection of metrics for certification procedures.

sonic boom↗

DOE OSTI2014

Software Quality Metrics for Geant4: An Initial Assessment - Paper 122

In the context of critical applications, such as shielding and radiation protection, ensuring the quality of simulation software they depend on is of utmost importance. The assessment of simulation software quality is crucial both to determine its adoption in experimental applications and to guarantee reproducibility of outcome over time. Geant4 is a simulation system used in a wide range of experimental applications, some of which, such as radiation protection and biomedical studies, may involve critical use cases and would especially profit from an objective assessment of the software quality. In this study, we present the first results from an ongoing analysis of Geant4 code based on established software metrics from the perspective of software engineering. The analysis evaluates the current status of the code to quantify its characteristics with respect to documented quality standards; further appraisals concern evolutions over a series of release distributions. The quantitative assessment of the software is initially focused on a subset of Geant4 packages, which play a key role in scientific applications and are representative of different software development processes. We describe the chosen software metrics that map the entities of a software system to numeric values through mathematical definitions, quantifying software attributes ranging from functionality to maintainability. To measure such metrics, we exploited a set of software metrics tools, i.e. programs that implement a set of software metrics definitions. They allow software engineers to assess a software system by extracting the required attributes from the software and providing the corresponding metric values. To correlate software metrics values in a well-defined way, we used software quality standard models, such as the ISO/IEC 9126 standard, which aggregate numerical values to aid quality analysis and assessment. With this work we aim to create a baseline for evaluating correlations between software quality embedded in the Geant4 development process and simulation observables produced by Geant4-based applications. Furthermore, we intend to highlight what metrics are most effective at evaluating radiation transport software quality. We are confident that the result will provide constructive guidance both to improve key software tools, such as Geant4, and to estimate their contribution in risk analysis concerning sensitive applications. The remainder of this paper is structured as follows. We first discuss the approach adopted to fulfil our assessment. Then, we present the setup of our experiments and the way we collected data. We provide an initial interpretation of a subset of data resulting from measurements made on the selected Geant4 packages. As the analysis is still ongoing at the time of submitting this abstract, some results will be documented in the final paper and in the conference presentation. With this work we aim to create a proper Geant4 data set for evaluating its software characteristics and identifying both strengths and weaknesses. In this initial study, we considered the geometry and processes packages, which play a major role in most simulation applications. The data analysis is in progress at the time of writing; its complexity is related to the differences among software metrics tools, to the large amount of data to be evaluated and to the pioneering character of this project. We are confident that the results will provide valuable indications for the Geant4 user community and constructive guidance to improve key Geant4 packages. Other Monte Carlo codes could profit from the methodology developed in this pilot project for similar assessments of their software quality. At the end of this study, we intend to create a baseline for evaluating correlations between software quality embedded in the Geant4 development process and simulation observables produced by Geant4-based applications. (authors)

61 RADIATION PROTECTION AND DOSIMETRY↗

DOE OSTI2017

MJO simulation in CMIP5 climate models: MJO skill metrics and process-oriented diagnosis

The Madden-Julian Oscillation (MJO) simulation diagnostics developed by MJO Working Group and the process-oriented MJO simulation diagnostics developed by MJO Task Force are applied to 37 Coupled Model Intercomparison Project phase 5 (CMIP5) models in order to assess model skill in representing amplitude, period, and coherent eastward propagation of the MJO, and to establish a link between MJO simulation skill and parameterized physical processes. Process-oriented diagnostics include the Relative Humidity Composite based on Precipitation (RHCP), Normalized Gross Moist Stability (NGMS), and the Greenhouse Enhancement Factor (GEF). Numerous scalar metrics are developed to quantify the results. Most CMIP5 models underestimate MJO amplitude, especially when outgoing longwave radiation (OLR) is used in the evaluation, and exhibit too fast phase speed while lacking coherence between eastward propagation of precipitation/convection and the wind field. The RHCP-metric, indicative of the sensitivity of simulated convection to low-level environmental moisture, and the NGMS-metric, indicative of the efficiency of a convective atmosphere for exporting moist static energy out of the column, show robust correlations with a large number of MJO skill metrics. The GEF-metric, indicative of the strength of the column-integrated longwave radiative heating due to cloud-radiation interaction, is also correlated with the MJO skill metrics, but shows relatively lower correlations compared to the RHCP- and NGMS-metrics. Our results suggest that modifications to processes associated with moisture-convection coupling and the gross moist stability might be the most fruitful for improving simulations of the MJO. Though the GEF-metric exhibits lower correlations with the MJO skill metrics, the longwave radiation feedback is highly relevant for simulating the weak precipitation anomaly regime that may be important for the establishment of shallow convection and the transition to deep convection.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2012

Optimization of the hop-byte metric for effective topology aware mapping

Suitable mapping of processes to the nodes of a massively parallel machine can substantially improve communication performance by reducing network congestion. The hop-byte metric has been used as a measure of the quality of such a mapping by several recent works. Optimizing this metric is NP hard, and thus heuristics are applied. However, the heuristics proposed so far do not directly try to optimize this metric. Rather, they use some intuitive methods for reducing congestion and use the metric just to evaluate the quality of the mapping. In fact, heuristics intending to optimize other metrics too don't directly optimize for them, but, rather, use the metric to evaluate the results of the heuristic. In contrast, we pose the mapping problem with the hop-byte metric as a quadratic assignment problem and use a heuristic to directly optimize for this metric. We evaluate our approach on realistic node allocations obtained on the Kraken system at NICS. Our approach yields values for the metric that are up to 75% lower than the default mapping and 66 % lower than existing heuristics. However, the time taken to produce the mapping can be substantially more, which makes this suitable for somewhat static, though possibly irregular, communication patterns. We introduce new heuristics that reduce the time taken to be comparable to that of existing fast heuristics, while still producing mappings of higher quality than existing ones. We also use theoretical lower bounds to suggest that our mapping may be close to optimal, at least for medium sized problems. Consequently, our work can also provide insight into the tradeoff between mapping quality and time taken by other mapping heuristics.

Sudheer, C. D.↗