Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28

Multiphysics FISPACT-II and TENDL-2019 simulation: neutron-induced damage metrics

Nuclear interactions can be the source of atomic displacement, irradiation-induced defects and transmutation in structural materials. Such quantities are derived from, or can be correlated to, nuclear kinematic simulations of primary atomic energy distributions spectra and the quantification of the numbers of secondary defects produced per primary as a function of the available recoils, residual and emitted, energies. Recoil kinematics of neutral, residual, gas, proton, alpha particle emissions are now more rigorously treated based on recent, complete and enhanced nuclear data parsed in state-of-the-art processing tools. Defect production metrics are the starting point in the complex problem of correlating and simulating the behaviour of materials under irradiation, as experimental information is rare or scattered. Detailed, segregated primary knock-on-atom metrics are now becoming available as the starting point of further simulation processes of isolated and clustered defects in material lattices. This allows more materials, lattices, neutron incident energy ranges, and irradiation conditions to be explored with sufficient data to adequately cover both standard and novel applications and materials: the broader reactor applications landscape. The damage metrics of of materials are systematically explored under typical but different reactor's type environment. The inventory code FISPACT-II combined with the enhanced nuclear data forms of the TENDL-2019 libraries allow one to not only calculate dpa from mostly scattering events but to also properly predict gas production, nuclear heating and transmutation under the same conditions.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Enhancing reproducibility in scientific computing: Metrics and registry for Singularity containers

Here we present Singularity Hub, a framework to build and deploy Singularity containers for mobility of compute, and the singularity-python software with novel metrics for assessing reproducibility of such containers. Singularity containers make it possible for scientists and developers to package reproducible software, and Singularity Hub adds automation to this workflow by building, capturing metadata for, visualizing, and serving containers programmatically. Our novel metrics, based on custom filters of content hashes of container contents, allow for comparison of an entire container, including operating system, custom software, and metadata. First we will review Singularity Hub’s primary use cases and how the infrastructure has been designed to support modern, common workflows. Next, we conduct three analyses to demonstrate build consistency, reproducibility metric and performance and interpretability, and potential for discovery. This is the first effort to demonstrate a rigorous assessment of measurable similarity between containers and operating systems. We provide these capabilities within Singularity Hub, as well as the source software singularity-python that provides the underlying functionality. Singularity Hub is available at https://singularity-hub.org, and we are excited to provide it as an openly available platform for building, and deploying scientific containers.

97 MATHEMATICS AND COMPUTING↗

Performance Assessment of the Overall Building Envelope Thermal Performance—Building Envelope Performance (BEP) Metric

Today, to describe the thermal performance of the building envelope and its components we use a variation of metrics; such as, R-value, ACH (air exchange rate per hour), SHGC (solar heat gain coefficient) of windows, U-factor etc. None of these performance indicators is meant to represent the overall thermal performance. In this paper, such a metric is introduced, the BEP (building envelope performance) value. Unlike the thermal resistance, typically expressed as an R-value, the BEP-value considers additional elements of heat transfer that affect the energy demand of the building because of exterior and interior (solar) thermal loads: conductive and radiant heat transfer, and air infiltration. To demonstrate BEP’s utility, validation studies were carried out by comparing the BEP-value to theoretical results using whole building energy simulation tools such as EnergyPlus and WUFI Plus. Results show that BEP calculations are comparable to calculations made using these simulation tools and that unlike other similar metrics, the BEP-value accounts for all heat transfer mechanisms that are relevant for the overall energy performance of the building envelope. The BEP-value thus allows comparing envelopes of buildings with different use types in a fair and realistic manner.

42 ENGINEERING↗

Review of Metrics Calculations Proposed by Lawrence Livermore National Laboratory

A multilaboratory effort to create metrics to describe the state of Surveillance programs was started several years ago. Two versions of metrics have been developed to date-one supported by Los Alamos National Laboratory (LANL) and Sandia National Laboratories (SNL) and the other developed and supported at Lawrence Livermore National Laboratory (LLNL). This paper reviews the LLNL metric calculation methodology and compares it with the LANLlSNL methodology.

42 ENGINEERING↗

Safety, codes and standards for hydrogen installations. Metrics development and benchmarking

Automakers and fuel providers have made public commitments to commercialize light duty fuel cell electric vehicles and fueling infrastructure in select US regions beginning in 2014. The development, implementation, and advancement of meaningful codes and standards is critical to enable the effective deployment of clean and efficient fuel cell and hydrogen solutions in the energy technology marketplace. Metrics pertaining to the development and implementation of safety knowledge, codes, and standards are important to communicate progress and inform future R&D investments. This document describes the development and benchmarking of metrics specific to the development of hydrogen specific codes relevant for hydrogen refueling stations. These metrics will be most useful as the hydrogen fuel market transitions from pre-commercial to early-commercial phases. The target regions in California will serve as benchmarking case studies to quantify the success of past investments in research and development supporting safety codes and standards R&D.

08 HYDROGEN↗

Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications

This report reviews existing literature describing forecast accuracy metrics, concentrating on those based on relative errors and percentage errors. We then review how the most common of these metrics, the mean absolute percentage error (MAPE), has been applied in recent radiation belt modeling literature. Finally, we describe metrics based on the ratios of predicted to observed values (the accuracy ratio) that address the drawbacks inherent in using MAPE. Specifically, we define and recommend the median log accuracy ratio as a measure of bias and the median symmetric accuracy as a measure of accuracy.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Validation Metrics for Deterministic and Probabilistic Data

The purpose of this document is to compare and contrast metrics that may be considered for use in validating computational models. Metrics suitable for use in one application, scenario, and/or quantity of interest may not be acceptable in another; these notes merely provide information that may be used as guidance in selecting a validation metric.

97 MATHEMATICS AND COMPUTING↗

Los Alamos National Laboratory Nuclear Material Management and Storage Program: Storage Health Metrics

The Strategic Materials Sustainment Program (SMSP) supports activities needed to manage the National Nuclear Security Administration’s (NNSA) inventory of strategic materials. As part of the SMSP, the Nuclear Material Management and Storage (NMM&S) program at the Los Alamos National Laboratory (LANL) is responsible for: (1) safe and secure storage of the SMSP strategic materials and others, (2) analysis, forecasting, and management of accountable nuclear materials; and (3) disposition of nuclear material through the Office of Nuclear Materials Integration (ONMI). The nuclear material management (NMM) mission is to ensure efficient nuclear material management lifecycle activities at LANL. The storage program function includes the management of storage capabilities across the site, supply chain management of nuclear materials, storage requirements integration, and nuclear container management (i.e., procurement, compliance, and safety). This report identifies key performance indicators (KPI) and describes the criteria and metrics used to assess the health of the storage component of the NMM&S program. “Health” is the ability of a process, system, facility, or program to safely and reliably accomplish its required mission. A health assessment provides timely identification of significant risks to enable mission performance. Significant risks identified are most effectively mitigated, avoided, or eliminated by evaluating and coordinating with stakeholders to specifically target, document, and often proceduralize strategies directed towards risk reduction. An effective health assessment involves status measurements of specific metrics that are tied directly to representative KPIs for a program. This document will baseline the KPI, criteria, and specific metrics that will be used to measure the “health” of the LANL Storage Program (i.e., NMM is not included).

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Metrics for the Complexity of Material Models

Quantitative measures are proposed for characterizing the complexity of material models used in computational mechanics. The algorithms for evaluating these metrics operate on the mathematical equations in the model rather than a code implementation and are different from software complexity measures. The metrics do not rely on a physical understanding of the model, using instead only a formal statement of the equations. A new algorithm detects the dependencies, whether explicit or implicit, between all the variables. The resulting pattern of dependencies is expressed in a set of pathways, each of which represents a chain of dependence between the variables. These pathways provide the raw data used in the metrics, which correlate with the expected ease of understanding, coding, and applying the model. Usage of the ComplexityMetrics code is described, with examples.

97 MATHEMATICS AND COMPUTING↗

Relationship between Metrics Used to Represent Displacement Damage in Materials

This report provides a set of consistent definitions for metrics relevant to the modeling of displacement damage in materials. The limitations/approximations built into the various metrics are discussed as is the intended application that gave rise to community use of the metric. Recommended sources for numerical tabulations of the neutron displacement kerma and the charged particle non-ionizing energy loss in some important semiconductor materials are also provided.

36 MATERIALS SCIENCE↗

FY2020 Fourth Quarter Performance Metric: Evaluate Improvement in Simulations of Mesoscale Convective Systems from New Parameterization Developments in E3SM

Mesoscale convective systems (MCSs) consist of an assembly of cumulonimbus clouds on scales of 100 km or more and produce mesoscale circulations (Houze, 2004, 2018). As the largest form of deep convective storms, MCSs contribute to 30% – 70% of annual and warm season rainfall in the U.S. and in the global tropics (Houze 2018; Stevenson & Schumacher, 2014; Feng et al., 2019; Haberlie & Ashley, 2019). Since MCSs contribute importantly to mean and extreme precipitation in the U.S. and many other regions around the world, understanding how well they are simulated by E3SM may guide future development towards more skillful modeling of convective storms and associated hydrologic impacts. The FY2020 Second Quarter Performance Metric Report documented comparisons of MCSs in the central and eastern U.S. in a high-resolution simulation produced by E3SM v1 at 25 km resolution (Caldwell et al. 2019) with observations. MCSs in the simulation occur less frequently and produce less intense precipitation, resulting in large underestimation of MCS volumetric rain-rate compared to observations. The first and third quarter performance metric report indicated that these model biases in simulating MCSs can be attributed to model limitations in parameterizing convection, clouds, and other related processes, as well as model biases in simulating the MCS large-scale environment. In the current FY2020 Fourth Quarter Performance Metric Report, we evaluate MCSs simulated in E3SM with several new developments in convection parameterizations that are being developed for its next generation. The goal is to summarize what have been improved with the new developments and highlight what need more work in the future.

54 ENVIRONMENTAL SCIENCES↗

Biases associated with a particular image volume metric

Image analysis area and volume metrics A 80 and V 80 , used by others, are studied as a function of image morphology, a 3D feature, and Monte Carlo noise in simulated images. The A 80 area metric is found to be stable, with consistent results (and meaning) obtainable without a detailed understanding of 3D features and/or simulation noise levels. However, the V 80 volume metric varies significantly with image morphology, 3D features, and simulation noise. If an experimental image set gives A 80 area inferences that are consistent with 2D image simulations, but V 80 volume inferences that are significantly larger than those corresponding to simulated images, then issues related to 3D features and simulation noise should be explored before conclusions are drawn related to a possible mismatch in the real physical size of experimental and simulated source volumes.

47 OTHER INSTRUMENTATION↗

MCNP® Code V.6.3.0 Unstructured-mesh Quality Metrics & Assessment

This report describes a variety of metrics calculated by the MCNP code for a user to assess the quality of an input unstructured mesh (UM). These UM quality metrics are calculated automatically, but the user has the opportunity to opt-out of doing so. At present, only linear tetrahedral elements and linear hexahedral elements have extensive quality metrics calculated and reported.

97 MATHEMATICS AND COMPUTING↗

Metrics and Methods for Radiation Detection Algorithm Characterization for Nuclear/Radiological Source Search

This report presents a series of recommendations for data to train and evaluate radiation detection algorithms and performance metrics to evaluate these algorithms. These recommendations were formed through a community consensus approach through the Detection Radiation Algorithms Group (DRAG), a multi-institution collaboration spanning eight Department of Energy laboratories and John Hopkins Applied Physics Laboratory. This report includes recommendations on background data variability, and metrics to quantify variability, sources and shielding configurations to include in data collection campaigns and detector response variability. In addition, this report describes several anomaly detection and identification algorithms and recommends metrics to report their performance. Finally, this report ends with a discussion on machine learning algorithms.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Integrating the Mobility Energy Productivity Metric into the CDOT Statewide Model

The Mobility Energy Productivity (MEP) metric measures the quality of mobility at a specific location and can be used to evaluate how changes to transportation systems impact the mobility of that location over time, such as through infrastructure investments. The objective of this study is to demonstrate integration of the MEP metric into CDOT's transportation planning process by leveraging data from their statewide travel demand model. We evaluate the MEP metric in 2015 as well as 2030 baselines and projected impacts in 2030 for two different regions in Colorado under multiple scenarios across multiple modes (driving, walking, biking, and transit). It was found that increasing development (increasing population density, jobs, and opportunities) had a significant impact on MEP, independent of any specific alternatives. For drive mode, there was a trade- off of increasing congestion on the road network and increasing job and opportunity access. Impacts to bike, walk, and transit MEP were also demonstrated in both regions. This report shows how MEP can be used as a tool to support evaluating the impacts of various transportation projects across the state. With projections of significant growth across the state of Colorado, access to the increasing opportunities and jobs will be important to understand through the context of energy efficiency. MEP could support future project evaluation and decision-making by enabling the unique and important dimension of energy- efficient accessibility of a transportation system.

33 ADVANCED PROPULSION SYSTEMS↗

Dense Image Matching Uncertainty Estimation and Confidence Metrics

Dense stereo matching takes overlapping image pairs as input and outputs a disparity map which encodes pixel-by-pixel matches between the images. Recently, there has been an interest in ranking the quality, or even quantifying the accuracy, of disparity estimates. The proposed methods can be described as either uncertainty estimators or confidence metrics. Uncertainty estimators are a small minority of the research. However, they have the potential to be the most useful because they estimate disparity accuracy (in pixel units) that can be used to threshold matches or carried forward using error propagation. The majority of the research deals with confidence metrics which give an ordinal (or binary) ranking of a match’s quality relative to other matches. Confidence metrics do not have units and thus are useful primarily for thresholding matches from mismatches. The methods could also be described as handcrafted or deep-learning based. The majority of the research focused on outdoor driving scenes. Hence, our interest–application to a satellite semi-global matching pipeline–is a domain shift that may challenge deep-learning based methods. We conclude by recommending five handcrafted and two deep-learning based methods for evaluation in our pipeline.

97 MATHEMATICS AND COMPUTING↗

Review of Grey Box/Black Box Data Contamination Metrics on Open and Commercial Models

Dataset contamination is a problem where benchmarks and tasks used to evaluate the capabilities of Large Language Models (LLMs) have been incorporated into the training dataset of the models. This gives a false sense of performance that can overestimate how these models will function on truly unseen data. This problem becomes worse with commercial LLMs with larger and non-accessible training data, so techniques have been developed to try to measure the degree to which a model is contaminated with a benchmark’s data. To understand the effectiveness of these techniques, particularly when evaluating contamination on coding tasks, we review trends and categorize techniques by the degree of access to the model that is required. The research literature on this topic has reported mixed effectiveness of these techniques, so we select a set of black box (text access only) and grey box (access to model loss/probabilities required) techniques and apply them to both commercial and non-commercial models. We implement these metrics as part of a framework to test the contamination of Python code in LLMs to see to what extent we can replicate the effectiveness (or ineffectiveness) of these contamination detection techniques. Though we find mixed results in the capabilities of these metrics to identify contamination, we do observe evidence that they can identify contamination (broadly) in fine-tuned models when both a baseline and fine-tuned model is present. Additionally, similarity metrics were able to identify between contaminated and uncontaminated data even in situations where the data is distributionally similar (e.g., drawn from the same set of code projects).

97 MATHEMATICS AND COMPUTING↗

Demonstration of Optimal Benchmark Selection Website and Validation of the q c Coverage Metric Using HEU-SOL-THERM-013-003 Experiment

In the work documented in this interim report, the experiment selection toolkit web site was demonstrated and q C coverage metric methodology was validated for IEU-MET-FAST-002-001, MIX-COMP-THERM 004-004, and HEU-SOL-THERM-013-003 experiments. 𝑞 𝐶 is an information-theoretic measure based on mutual information that quantifies the ability of candidate benchmark experiments to reduce the bias and uncertainty of a target criticality safety application. The metric and an accompanying open-source Python toolkit with a web-based interface were tested against a benchmark set of 425 experiments drawn from the International Criticality Safety Benchmark Evaluation Project Handbook. The interface is hosted at https://edim.covdef.com. It accepts sensitivity data files produced by the TSUNAMI-IP module of the SCALE code system and supports both (i) deterministic analysis using the ENDF/B-VII.0 covariance library and (ii) stochastic analysis based on user-supplied keff samples. Demonstrations on representative applications across a range of material composition, spectrum, and form show that q C -guided benchmark selection achieves greater uncertainty reduction with fewer experiments and yields more stable posterior bias and uncertainty estimates than traditional similarity coefficient ( c k )–based selection, while also capturing valuable low-ck experiments that one-to-one metrics overlook.

Abdel-khalik, Hany S. [Indiana Univ.-Purdue Univ. ↗