SEARCH · Engineering Papers

Results for “METRICS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Assessing United States County-Level Exposure for Research on Tropical Cyclones and Human Health

Tropical cyclone epidemiology can be advanced through exposure assessment methods that are comprehensive and consistent across space and time, as these facilitate multiyear, multistorm studies. Further, an understanding of patterns in and between exposure metrics that are based on specific hazards of the storm can help in designing tropical cyclone epidemiological research. a) Provide an open-source data set for tropical cyclone exposure assessment for epidemiological research; and b) investigate patterns and agreement between county-level assessments of tropical cyclone exposure based on different storm hazards. We created an open-source data set with data at the county level on exposure to four tropical cyclone hazards: peak sustained wind, rainfall, flooding, and tornadoes. The data cover all eastern U.S. counties for all land-falling or near-land Atlantic basin storms, covering 1996–2011 for all metrics and up to 1988–2018 for specific metrics. We validated measurements against other data sources and investigated patterns and agreement among binary exposure classifications based on these metrics, as well as compared them to use of distance from the storm’s track, which has been used as a proxy for exposure in some epidemiological studies. Our open-source data set was typically consistent with data from other sources, and we present and discuss areas of disagreement and other caveats. Over the study period and area, tropical cyclones typically brought different hazards to different counties. Therefore, when comparing exposure assessment between different hazard-specific metrics, agreement was usually low, as it also was when comparing exposure assessment based on a distance-based proxy measurement and any of the hazard-specific metrics. Our results provide a multihazard data set that can be leveraged for epidemiological research on tropical cyclones, as well as insights that can inform the design and analysis for tropical cyclone epidemiological research.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2024

FORESTR: Finding, Organizing, Representing, Explaining, Summarizing, and Thinning Random forests

Random forests have become popular models used for data driven predictions. As a result, random forests are currently used or being considered for high-consequence mission applications in national security, such as the prediction of yield from optical signals and malware detection. While random forests may provide accurate predictions, the complexity of the algorithm causes a lack of interpretability. Random forests are an ensemble of regression or decision trees. Individual regression and decision trees are interpretable, but ensembles are inherently difficult to interpret due to the compilation of many models. We aim to increase the interpretability of random forests by finding patterns in the ensemble of trees that can be used to “thin” (or remove) trees. As a starting point, in this report, we develop a new distance metric for quantifying the similarity between trees based on their topologies (i.e., shapes). We base the metric on a novel distance metric for graphs that is a proper mathematical distance, is invariant to transformations, has registration between graphs, and computes topological evolutions between graphs. We use the tree distance metric to compute tree statistics such as a “mean tree” and to identify clusters of trees. We apply the developed methodology to a toy dataset and a mission relevant product inspection dataset to demonstrate how the metric can provide insight into random forests. Furthermore, we discuss the limitations of the approach and ideas for future research into how the metric could be used as a thinning tool to develop less complex models.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Quantifying Technical Diversity Benefits of Wind as a Distributed Energy Resource

Distributed energy resources are increasingly used in power distribution systems and microgrids to supply cost competitive power, improve resilience, and provide a host of grid services. Diversifying variable renewable resources (e.g., by combining wind with solar) can increase energy usage efficiency and improve system resilience. However, when grid optimization and resilience studies consider multiple renewable resources, diversity benefits are usually captured only implicitly in the results of location-specific economic optimization. In this paper, metrics are introduced to express the technical value of resource diversity independent of jurisdiction-specific market structures. Specifically, marginal energy usage efficiency metrics are developed to quantify the ability of new distributed generation to produce useful energy and an incremental sustainable ride through metric is developed to express improvement in grid outage ride through capability. While there are economic implications for each of these metrics, the metrics themselves are technically-driven and could be used as components of a technical figure of merit that could be used to inform policy actions, such as development of resource-specific incentives and time-of-use tariff designs. Each of these metrics is demonstrated using balance-of-energy simulations that highlight the benefits of improving resource diversity.

Reiman, Andrew P.↗

DOE OSTI2021

Multifaceted radiation detection and classification system

A system identifying a source of radiation is provided. The system includes a radiation source detector and a radiation source identifier. The radiation source detector receives measurements of radiation; for one or more sources, generates a detection metric indicating whether that source is present in the measurements; and evaluates the detection metrics to detect whether a source is present in the measurements. When the presence of a source in the measurements is detected, the radiation source identifier for one or more sources, generates an identification metric indicating whether that source is present in the measurements; generates a null-hypothesis metric indicating whether no source is present in the measurements; evaluates the one or more identification metrics and the null-hypothesis metric to identify the source, if any, that is present in the measurements.

Labov, Simon E.↗

DOE OSTI2022

Multifaceted radiation detection and classification system

Labov, Simon E.↗

DOE OSTI2022

Capsule network-based semantic segmentation model for thermal anomaly identification on building envelopes

Thermography technology is widely used to inspect thermal anomalies in building façade systems. Computer vision-based techniques provide opportunities to autonomously detect such heat anomalies to significantly improve the efficiency of decision-making for building envelope retrofitting and maintenance. Here, in this work, we propose a novel Capsule Network-based deep learning model – CapsLab – that detects and identifies thermal anomalies by semantic segmentation. CapsLab is built based on our proposed prediction-tuning capsule (PT-Capsule) layer. Different from a traditional capsule layer, which consists of part-whole transformation and capsule-routing process, the proposed layer is composed of a prediction and tuning process, which helps decreasing the number of model parameters significantly. While the applicability of traditional Capsule Networks (CapsNets) has been limited to simpler tasks and smaller datasets due to their scalability issue, we can leverage the lightweight of the proposed PT-Capsule layer, and apply it to the semantic segmentation task. In this work, we also employ our previously presented performance metric, referred to as the Anomaly Identification Metric (AIM) (Kakillioglua et al. 2021), to evaluate the segmentation outputs. Traditional performance metrics do not accurately reflect the true performance of the segmentation models in thermal anomaly identification due to the high subjectivity in the annotation process and higher overlap ratio sensitivity of the standard metrics. AIM, on the other hand, is robust to these drawbacks. Experimental results show, both qualitatively and quantitatively, that our proposed segmentation method can effectively segment the thermal anomalies. Specifically, our model provides 9.38% and 13.53% improvements over the baseline model – DeepLabV3+ – based on traditional mIoU score and the AIM score, respectively, while requiring less model parameters and less computation at the same time. In addition, the scores that the AIM metric generates better align with the scores provided by building performance experts.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2021

Feedback density and causal complexity of simulation model structure

Measures of simulation model complexity generally focus on outputs; we propose measuring the complexity of a model’s causal structure to gain insight into its fundamental character. This article introduces tools for measuring causal complexity. First, we introduce a method for developing a model’s causal structure diagram, which characterises the causal interactions present in the code. Causal structure diagrams facilitate comparison of simulation models, including those from different paradigms. Next, we develop metrics for evaluating a model’s causal complexity using its causal structure diagram. We discuss cyclomatic complexity as a measure of the intricacy of causal structure and introduce two new metrics that incorporate the concept of feedback, a fundamental component of causal structure. The first new metric introduced here is feedback density, a measure of the cycle-based interconnectedness of causal structure. The second metric combines cyclomatic complexity and feedback density into a comprehensive causal complexity measure. Finally, we demonstrate these complexity metrics on simulation models from multiple paradigms and discuss potential uses and interpretations. These tools enable direct comparison of models across paradigms and provide a mechanism for measuring and discussing complexity based on a model’s fundamental assumptions and design.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Emulating climate extreme indices

We use simple pattern scaling and time-shift to emulate changes in a set of climate extreme indices under future scenarios, and evaluate the emulators' accuracy. We propose a metric for the error in emulation in the context of initial condition ensembles, to specifically characterize the role of internal variability in the emulation performance. Our metric separates systematic emulation errors from unavoidable discrepancies between emulated and target values due to internal variability. We compute the metricis at grid-point scale, and we show geographically resolved results, or aggregate them at global scale. We demonstrate the use of our error metric in the emulation of a suite of temperature and precipitation extreme indices. We test and compare simple pattern scaling and time-shift using a range of trajectories spanning targets inspired by the Paris agreement -- warming to 1.5C and 2.0C from the pre-industrial baseline -- and two of the longer-established trajectories, RCP4.5 and RCP8.5. With this suite of scenarios we can test the effects on the performance of the size of the temperature gap between emulation origin and target. We find that for most indices emulation the dominant source of discrepancy is internal variability. For at least one index, however, counting exceedances of a high temperature threshold, significant portions of the globally aggregated discrepancy and its regional pattern originate from the systematic emulation error. This error exceeds internal variability of both the target and the emulated quantities in large coherent regions at low latitudes, and the explanation can be found in the differential behavior of temperature distributions across latitudes. The metric also highlights a fundamental difference in the two methods related to the simulation of internal variability, which is dampened significantly by simple pattern scaling. This aspect is of consequence when using these methods for specific applications, where preserving variability for uncertainty quantification is deemed important. With this study we offer our metric as a diagnostic tool, facilitating the formulation of scientific hypotheses on the reasons for the error. In the meantime, we show that for many impact relevant indices by now traditional emulation techniques can be accurate within the variations unavoidably introduced by internal variability, establishing the fundamental condition for using their emulation in impact modeling.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

Nuclear Science User Facilities FY 2020 Program Performance Summary

The Nuclear Science User Facilities (NSUF) first gathered and reported metric data for Fiscal Year (FY) 2014. The goal of the FY 2014 NSUF Metric Report was to establish metric measures and to provide baseline data to evaluate future performance of the NSUF. Each metric goal and objective was established through NSUF discussions with Department of Energy (DOE) Idaho Operations Office and Office of Nuclear Energy staff. This report provides the data, analysis, and conclusions for the FY 2020 metrics.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2024

Justice Underpinning Science and Technology Research (JUST-R) Offline Tool: A Tool for Guiding Researchers in Addressing Energy Justice Considerations in Early-Stage Research

The Justice Underpinning Science and Technology Research (JUST-R) Metrics Framework is a set of metrics for assessing the energy justice implications of technologies that are currently under research and development (R&D). The JUST-R Metrics Framework guides researchers through an analysis of the many facets of their research processes, from material inputs to knowledge sources, that may contribute to energy injustice both during the research period and when the technology is scaled. This JUST-R Offline Tool consists of an Excel file and PDF guide to aid researchers, engineers, and project managers in their application of the JUST-R Metrics Framework. This tool enables researchers to 1) evaluate the baseline energy justice implications of their research; 2) develop justice-oriented changes to the research process; and 3) track the implementation of proposed changes. The metrics included in the framework are sorted into five aspects of research: Team Dynamics, Sources & Inputs, Processes & Protocols, Waste & Hazards, and Results & Dissemination. The JUST-R Tool can be found here: https://www.nrel.gov/analysis/just-r.html.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2020

Nuclear Science User Facilities FY 2019 Program Performance Summary

The Nuclear Science User Facilities (NSUF) first gathered and reported metric data for Fiscal Year (FY) 2014. The goal of the FY 2014 NSUF Metric Report was to establish metric measures and to provide baseline data to evaluate future performance of the NSUF. Each metric goal and objective were established through NSUF discussions with Department of Energy (DOE) Idaho Operations Office and Office of Nuclear Energy staff. This report provides the data, analysis, and conclusions for the FY 2019 metrics.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2025

AAPM Truth‐based CT (TrueCT) reconstruction grand challenge

Background: This Special Report summarizes the 2022, AAPM grand challenge on Truth-based CT image reconstruction. Purpose: To provide an objective framework for evaluating CT reconstruction methods using virtual imaging resources consisting of a library of simulated CT projection images of a population of human models with various diseases. Methods: Two hundred unique anthropomorphic, computational models were created with varied diseases consisting of 67 emphysema, 67 lung lesions, and 66 liver lesions. The organs were modeled based on clinical CT images of real patients. The emphysematous regions were modeled using segmentations from patient CT cases in the COPDGene Phase I dataset. For the lung and liver lesion cases, 1–6 malignant lesions were created and inserted into the human models, with lesion diameters ranging from 5.6 to 21.9 mm for lung lesions and 3.9 to 14.9 mm for liver lesions. The contrast defined between the liver lesions and liver parenchyma was 82 ± 12 HU, ranging from 50 to 110 HU. Similarly, the contrast between the lung lesions and the lung parenchyma was defined as 781 ± 11 HU, ranging from 725 to 805 HU. For the emphysematous regions, the defined HU values were −950 ± 17 HU ranging from −918 to −979 HU. The developed human models were imaged with a validated CT simulator. The resulting CT sinograms were shared with the participants. The participants reconstructed CT images from the sinograms and sent back their reconstructed images. Further, the reconstructed images were then scored by comparing the results against the corresponding ground truth values. The scores included both task-generic (root mean square error [RMSE] and structural similarity matrix [SSIM]), and task-specific (detectability index [d’] and lesion volume accuracy) metrics. For the cases with multiple lesions, the measured metric was averaged across all the lesions. To combine the metrics with each other, each metric was normalized to a range of 0 to 1 per disease type, with “0” and “1” being the worst and best measured values across all cases of the disease type for all received reconstructions. Results: The True-CT challenge attracted 52 participants, out of which 5 successfully completed the challenge and submitted the requested 200 reconstructions. Across all participants and disease types, SSIM absolute values ranged from 0.22 to 0.90, RMSE from 77.6 to 490.5 HU, d’ from 0.1 to 64.6, and volume accuracy ranged from 1.2 to 753.1 mm3. The overall scores demonstrated that participant “A” had the best performance in all categories, except for the metrics of d’ for lung lesions and RMSE for liver lesions. Participant “A” had an average normalized score of 0.41 ± 0.22, 0.48 ± 0.32, and 0.42 ± 0.33 for the emphysema, lung lesion, and liver lesion cases, respectively. Conclusions: The True-CT challenge successfully enabled objective assessment of CT reconstructions with the unique advantage of access to a diverse population of diseased human models with known ground truth. This study highlights the significant potential of virtual imaging trials in objective assessment of medical imaging technologies.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2026

Short-term electricity load forecasting: Application-driven evaluation of machine learning models across spatial and temporal scales

As we transition towards a decarbonized economy, the integration of variable renewable energy resources and new demands (e.g., electric vehicles, heat pumps) into the electricity grid places unprecedented pressure on grid operators to effectively anticipate and manage peak load. In this context, machine learning algorithms are proving to be indispensable for accurate short-term load forecasting, a crucial task to address these challenges. This study benchmarks 6 machine learning algorithms, including three neural networks and three tree-based algorithms, across various levels of spatial aggregation and time horizons (1, 4, 8, 24, and 48 h). The central contribution of this work is the comparison and analysis of load forecasting models not only based on statistical metrics, but also based on a novel error metric, which evaluates the cost implications of forecast errors for power system stakeholders. Results show that tree-based models outperform neural networks, based on statistical metrics, and yield less skewed error distributions for most spatial scales. However, through the lens of the novel error metric, neural networks are the more competitive choice, especially for forecast horizons that exceed 8 h. The study concludes with actionable recommendations to grid operators and highlights the need for the development of error metrics that link forecasting accuracy to operational costs. To promote transparency and open science, the datasets and Python code are open-sourced via a supplementary repository.

Houben, Nikolaus↗

DOE OSTI2022

Particle emissions from mobile sources: Discussion of ultrafine particle emissions and definition

There is no universally agreed upon definition for ultrafine particles (UFP). Commonly used definitions for UFP are either particle number below 100 nm or total particle number, but without an agreed upon lower cut point. For example, a lower cut point of 3 nm compared to 10 nm could result in a substantially higher count. Another definition for UFP is total particle mass but without a commonly agreed upon aerodynamic diameter upper cut point, e.g., below 100 nm, 200 nm, 300 nm, etc. Yet another definition is lung deposited surface area weighted by lung deposition fraction, found mainly in the particle mobility diameter range from 20 to 400 nm. It is clear from these definitions that there are inconsistencies in the way UFP is used and defined in the literature. Sometimes these metrics are well correlated, sometimes not. In this paper we suggest three exposure metrics: UFP-N, UFP-M, and UFP-S, that we believe will add clarity. These metrics represent total number, mass, and surface area below 500 nm, respectively. For surface area and mass, the 500 nm cut point can be either aerodynamic or mobility diameter depending upon measurement methodology. For all metrics, this cut point captures nearly all of the primary particle emissions from mobile sources. Furthermore, UFP-N would include a lower cut point of 3–6 nm and would not require an upper size cut point because there is very little particle number above 500 nm or even above 100 nm. Thus, our definition of UFP-N is consistent with the current definition of ultrafine number except for, importantly, the specification of a lower cut point. These exposure metrics can help facilitate consistency in the characterization of both short- and long-term UFP ambient exposures and associated health effects in epidemiological studies.

42 ENGINEERING↗

DOE OSTI2022

Measures of quasisymmetry for stellarators

Quasisymmetric stellarators are an attractive class of optimised magnetic confinement configurations. The property of quasisymmetry (QS) is in practice limited to be approximate, and thus the construction requires measures that quantify the deviation from the exact property. In this paper we study three measure candidates used in the literature, placing the focus on their origin and a comparison of their forms. The analysis shows clearly the lack of universality in these measures. As these metrics do not directly correspond to any physical property (except when exactly quasisymmetric), optimisation should employ additional physical metrics for guidance. It is suggested that close to QS minima, one should treat QS metrics through inequality constraints so that additional physics metrics dominate optimisation. The impact of different quasisymmetric measures on optimisation is presented through an example, for which the standard metric that weights the asymmetric Fourier modes of the field magnitude appears to perform best.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2025

A framework to evaluate machine learning crystal stability predictions

The rapid adoption of machine learning in various scientific domains calls for the development of best practices and community agreed-upon benchmarking tasks and metrics. We present Matbench Discovery as an example evaluation framework for machine learning energy models, here applied as pre-filters to first-principles computed data in a high-throughput search for stable inorganic crystals. We address the disconnect between (1) thermodynamic stability and formation energy and (2) retrospective and prospective benchmarking for materials discovery. Alongside this paper, we publish a Python package to aid with future model submissions and a growing online leaderboard with adaptive user-defined weighting of various performance metrics allowing researchers to prioritize the metrics they value most. To answer the question of which machine learning methodology performs best at materials discovery, our initial release includes random forests, graph neural networks, one-shot predictors, iterative Bayesian optimizers and universal interatomic potentials. We highlight a misalignment between commonly used regression metrics and more task-relevant classification metrics for materials discovery. Accurate regressors are susceptible to unexpectedly high false-positive rates if those accurate predictions lie close to the decision boundary at 0 eV per atom above the convex hull. The benchmark results demonstrate that universal interatomic potentials have advanced sufficiently to effectively and cheaply pre-screen thermodynamic stable hypothetical materials in future expansions of high-throughput materials databases.

Riebesell, Janosh↗

DOE OSTI2025

A building performance-based approach to determining energy resilience for grocery stores in the event of a power outage

Evaluating progress toward a built environment that is best equipped to serve communities during a regional power outage will require metrics that capture the energy resilience of the unique buildings and businesses most crucial to the well-being of those nearby. We focused on grocery stores as key buildings for which access, and thus energy resilience, is critical during a disaster when power is unavailable. We evaluated the energy resilience of these buildings by offering and testing building-scale metrics that assess business continuity potential during a power outage. Metrics proposed in this study are calculated based on the unique power loads characteristic to grocery stores, primarily refrigeration and maintaining safe indoor environmental conditions. Building simulations based on varying levels of backup power were carried out against occupant safety and comfort parameters to apply these metrics, with additional criteria imposed on grocery stores to capture the inventory and sales loss from food spoilage resulting from a lack of refrigeration power. Findings from this study demonstrate the feasibility of our proposed metrics and methodology to serve as a low-data burden means for stakeholders to evaluate the energy resilience of grocery stores, with greater implications in helping to understand the impact on community-scale energy resilience.

Siegel, Lino Sanchez↗

DOE OSTI2021

Quantitative Evaluation of Autonomous Driving in CARLA

There has been a great deal of recent advancements in end-to-end imitation and reinforcement learning for self-driving vehicles. Despite this, there is a severe lack of standardized metrics for evaluating the performance of autonomous self-driving agents. Existing metrics are generally lacking in their ability to capture a wide range of driving behaviors and compare the severity of different failure cases. In this work, we introduce the Quantitative Evaluation for Driving metric, or QED, which assigns a quantitative score from 0-100 that captures the quality of driving for any driving agent. Our QED metric assesses different aspects of driving behavior including the ability to stay in the center of the lane, avoid weaving and erratic behavior, follow the speed limit, and avoid collisions, and it can be used under a wide range of driving scenarios. To show the effectiveness of our QED metric, we compare the scores generated by QED against scores assigned by human evaluators on a total of 30 different drivers and 6 different towns in the CARLA driving simulator. In ``easy'' evaluation scenarios, where it is relatively straightforward to distinguish better drivers from worse drivers, QED attains an average Pearson correlation of 0.96 and average Spearman correlation of 0.97 when compared against human evaluators. In ``hard'' evaluation scenarios, where it is far more ambiguous how to rank/score different types of bad driving behavior, QED attains an average Pearson correlation of 0.82 and average Spearman correlation of 0.75 when compared against human evaluators, which are both slighter higher than when we compare human evaluators against each other. While QED may not capture every characteristic that defines good driving, we consider it an important foundation for reproducibility and standardization in the community.

Gao, Shang↗