Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “reliability metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Valuing Resilience for Microgrids: Challenges, Innovative Approaches, and State Needs

The United States depends on the delivery of reliable, affordable, clean, and safe electricity. Electric utilities invest billions of dollars each year in generation, transmission, and distribution assets to meet this need. However, experiences with recent natural disasters of increasing frequency and duration demonstrate the shortcomings of this approach in the face of modern threats. Further, as customers rely on electricity for a broader range of important needs, such as transportation, as well as critical life-saving services and mission critical facilities such as water treatment, medical care, shelters, telecommunications, and more, the need to minimize the likelihood and impacts of outages grows. Against this backdrop, resilience has emerged as a key consideration to guide electricity spending, whether from utilities, customers, or taxpayers. Although reliability has been defined and measured for decades with broadly accepted metrics that measure how many customers lose power and at what frequency and duration, resilience considers the electricity system’s response to a disruption and its subsequent impacts on customers. Developing tools and methods to accurately assess the costs and benefits of resilience investments is a critical step toward the goal of mitigating the impacts of outages on customers and society. Today, electric system resilience is largely treated as an externality due to challenges estimating the costs of long-duration outages, impacts of outages on society, and increasing reliance on electricity for a growing set of interdependent services. These interdependencies include the water, wastewater, telecommunications, natural gas, and health sectors. Without knowing how much a given resilience investment will benefit customers or society more broadly, investors, policymakers, and regulators are less likely to make or approve such investments, and less able to prioritize those investments. State Energy Offices and public utility commissions (PUCs) lead the development of state-level energy policy and utility regulation, respectively, and each has interests in encouraging appropriate public and private investments in resilience. To this end, the National Association of State Energy Officials (NASEO) and National Association of Regulatory Utility Commissioners (NARUC), with the support of the U.S. Department of Energy (DOE) Office of Electricity (OE), formed a joint Microgrids State Working Group to explore the costs and benefits of microgrids, barriers to broader deployment of microgrids to meet resilience and other objectives, and policy and regulatory strategies to optimize investments in resilience, including but not limited to microgrids. Although no universally accepted valuation tool for resilience exists, National Laboratories, utilities, researchers, and state and federal agencies have collaborated to develop, apply, and improve a number of approaches to quantify resilience, several of which are still in progress at the time this report is published. This report seeks to share these important advances by discussing current definitions of resilience (Section 1), how microgrids are defined and used to meet resilience objectives (Section 2), new approaches to valuing resilience (Section 3), steps State Energy Offices and PUCs have taken to further resilience valuation efforts (Section 4), and finally, considerations and suggested next steps for State Energy Offices and PUCs (Section 5). Relevant examples of specific microgrid projects and resilience valuation efforts are included throughout the report. While this report is written specifically for NASEO and NARUC members, it may be useful for utilities, local governments, and individual customers interested in improving the way public and private dollars are spent to achieve resilience outcomes.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Comparison of ML-Based Proxy Modeling Strategies: Lessons Learned from the SMART Initiative

Teams of researchers on Task 5 of the SMART project have developed a variety of modeling architectures to predict subsurface behavior during carbon injection and post-injection periods. One important part of this task was to compare the candidate approaches in terms of accuracy, reliability, speed, and memory use, all using a common set of metrics and visualizations for an “apples to apples” comparison. Dr. Jared Schuetter will share the details of this task, the results that were obtained, and the lessons learned.

Schuetter, Jared↗
DOE OSTI2026

SimLBR: Learning to Detect Fake Images by Learning to Detect Real Images

The rapid advancement of generative models has made the detection of AI-generated images a critical challenge for both research and society. Recent works have shown that most state-of-the-art fake image detection methods overfit to their training data and catastrophically fail when evaluated on curated hard test sets with strong distribution shifts. In this work, we argue that it is more principled to learn a tight decision boundary around the real image distribution and treat the fake category as a sink class. To this end, we propose SimLBR, a simple and efficient framework for fake image detection with Latent Blending Regularization (LBR). Our method significantly improves cross-generator generalization, achieving up to +24.85% accuracy and +69.62% recall on the challenging Chameleon benchmark. SimLBR is also highly efficient, training orders of magnitude faster than existing approaches. Furthermore, we emphasize the need for reliability-oriented evaluation in fake image detection, introducing risk-adjusted metrics and worst-case estimates to better assess model robustness. All the code and models are availabe at: https://github.com/mvrl/SimLBR

Dhakal, Aayush [Washington University, St. Louis]↗
DOE OSTI2020

Assessing the sensitivity and repeatability of permanganate oxidizable carbon as a soil health metric: An interlab comparison across soils

Soil organic matter is central to the soil health framework. Therefore, reliable indicators of Soil organic matter is central to the soil health framework. Therefore, reliable indicators of changes in soil organic matter are essential to inform land management decisions. Permanganate oxidizable carbon (PDXC), an emerging soil health indicator, has shown promise for being sensitive to soil management. However, strict standardization is required for widespread implementation in research and commercial contexts. Here, we used 36 soils-three from each of the 12 USDA soil orders-to determine the effects of sieve size and soil mass of analysis on PDXC results. Using replicated measurements across 12 labs in the US and the EU (n = 7951 samples), we quantified the relative importance of 1) variation between labs, 2) variation within labs, 3) effect soil mass, and 4) effect of soil sieve size on the repeatability of PDXC. We found a wide range of overall variability in PDXC values across labs (0.03 to 171.8%; mean = 13.4%), and much of this variability was attributable to within-lab variation (median = 6.5%) independently of soil mass or sieve size. Greater soil mass (2.5 g) decreased absolute PDXC values by a mean of 177 mg kg -1 soil and decreased analytical variability by 6.5%. For soils with organic carbon (SOC) >10%, greater soil mass (2.5 g) resulted in more frequent PDXC values above the limit of detection whereas the lower soil mass (0.75 g) resulted in PDXC values below the limit of detection for SOC contents <5%. A finer sieve size increased absolute values of PDXC by 124 mg kg -1 while decreasing the analytical variability by 1.8%. In general, soils with greater SOC contents had lower analytical variability. These results point to potential standardizations of the PDXC protocol that can decrease the variability of the metric. We recommend that the PDXC protocol be standardized to use 2.5 g for soils <10% SOC. Sieve size was a relatively small contributor to analytical variability and therefore we recommend that this decision be tailored to the study purpose. Tradeoffs associated with these standardizations can be mitigated, ultimately providing guidance on how to standardize PDXC for routine analysis.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

A call to standardize metrics for monitoring baleen whales near marine construction activities

Effective monitoring is necessary to protect marine mammal species during the construction of offshore infrastructure. The tools for detecting or monitoring marine mammals span traditional (e.g., visual observers, optical cameras), to newer (e.g., passive acoustic monitoring, infrared cameras, tags), and emerging (e.g., satellite imagery, environmental DNA, dimethyl sulfide concentration) technologies. Some are better suited for use during offshore development; however, peer-reviewed literature does not typically evaluate and report on the performance of these various technologies. We define a minimum set of metrics related to efficacy (i.e., confusion matrix, precision and recall, probability of missed mitigation), detection range (i.e., maximum and reliable detection range, spatial resolution), and data delivery (i.e., detection latency, system reliability, temporal resolution) that we recommend are needed to assess the utility of monitoring technologies for this purpose. Following a literature review of relevant studies, we highlight which publications reported these metrics and used multiple technologies to compare relative performance. We also emphasize the benefits of multi-modal approaches and recommend performance assessments through modeling or large-scale collaborative field testing. These metrics will standardize data collection, reporting, and analysis; promote consistent and comparable results; and foster collaboration among developers, regulatory agencies, and scientists. This may lead to the co-development of technology that achieves multiple goals, has greater application, and can answer research questions while collecting data to fulfill permitting requirements. These metrics may also inform decisions on what systems regulatory agencies might consider using and reduce monitoring costs, which is critical to support the marine sector's rapid growth alongside marine mammal conservation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

Trends in surface equivalent potential temperature: A more comprehensive metric for global warming and weather extremes

Significance The Earth has warmed by 1.2 ± 0.1 °C since the preindustrial era. The most common metric to measure the ongoing global warming is surface air temperature since it has long and reliable observational records. However, surface air temperature alone does not fully describe the nature of global warming and its impact on climate and weather extremes. Here we show that surface equivalent potential temperature, which combines the surface air temperature and humidity, is a more comprehensive metric not only for the global warming but also for its impact on climate and weather extremes including tropical deep convection and extreme heat waves. We recommend that it should be used more widely in future climate change studies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Avoidance of disruptions on KSTAR due to vertical displacement events via novel real-time stability assessment

Disruption avoidance via the DECAF approach has been achieved on KSTAR using a novel real-time vertical stability assessment and a multiactuator feedback control strategy. The development of disruption avoidance strategies with reactor-relevant reliability is an urgent activity, enabling future fusion power plants. The stability metric employed is based on a new formulation of a vertical force gradient balance metric evaluated across the poloidal cross section of the plasma, with parameters tuned using historical data. Evaluation of this metric on a validation set of 400 recent KSTAR shots indicates >82% of Vertical displacement events can be avoided via feedback control. Essential to its calculation is the two-dimensional toroidal current density distribution in the plasma. Measurement of this profile faster than fully-converged equilibrium reconstructions can deliver is found to improve forecaster performance and is achieved with a surrogate model that takes as input magnetic diagnostic measurements and outputs the current profile on a basis comprising the top principal components of historical current profiles (from past equilibrium reconstructions). This method solves the non-uniqueness problem typically faced when reconstructing current profiles directly from diagnostics, while improving computational time and accuracy. On average, profiles produced by this model reach coefficients of determination of >0.99 with respect to those from equilibrium reconstructions. The avoidance actuators employed include poloidal field coils and an electron cyclotron current drive system. The multiactuator approach, as shown in this first demonstration, allows disruption avoidance while minimizing impact to operational performance. This ability, along with its flexibility and speed, makes this new approach an attractive option for avoiding these types of disruptions in reactors.

Tobin, Matthew [Columbia Univ., New York, NY (Unit↗
DOE OSTI2025

Integrated Metrics for County-Level Resilience Ranking Using Entropy and TOPSIS

In the face of atypical weather events, power infrastructure failures, and limited resources for resilience investment, energy decision-makers need data-driven metrics to allocate resilience investments and maximize the reduction of power outage impacts. For state-level planning, for instance, ranking the resilience of each county is key to ensuring effective distribution of resources. In such cases, resilience for each spatial unit is multifaceted and is captured by a set of indicators (i.e., metrics) that can be combined into an overall score that reduces the complexity of power outage dynamics to a single decision metric. However, weighting of these indicators is often addressed by simplifying assumptions (i.e., equal weights) or semi-subjective methods that rely on user-defined weights that can introduce biases (e.g., weighted average score). Within the disaster risk reduction and resilience engineering community, a recurring challenge in multicriteria decision-making is the objective weighting of indicators for composite indices. To address this issue, we have leveraged a Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) combined with an entropy-based weighting approach to calculated the integrated scores. This method objectively determines the importance of each metric, better discerns between spatial units (i.e., counties), and offers a more reliable ranking of counties according to their relative resilience attributes. By improving methods for integrating resilience indicators, our approach helps planners and decision-makers prioritize resources more effectively for more efficient resilience investments.

Bhusal, Narayan [Oak Ridge National Laboratory (OR↗
DOE OSTI2025

Developments to the Distributed Holdup Monitoring System in Fiscal Year 2025

Permanently installed holdup monitoring would provide a myriad of benefits to nuclear facilities, ranging among decreased facility burden, increased safety bases, and increased accuracy of relevant material controls and accountancy metrics. However, the cost per detection system must be affordable to provide reliable coverage. For this project, the target unit price per system has been $\$$1,000. Recent work has focused on pushing costs lower through developing a plastic scintillator and silicon photomultiplier (SiPM)–based front end, which are components that are not commercially available in conjunction with each other. Other work has included further system development and use of a low-cost SiPM. Finally, a limited deployment at a processing facility was achieved. Results from this deployment and the development process are discussed in detail.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

A Large-Scale Analysis to Optimize the Control and V2V Communication Protocols for CDA Agreement-Seeking Cooperation

Cooperative driving automation (CDA) Class C, agreement-seeking cooperation, is an innovative and practical solution that can promote cooperation among general passenger vehicles on the road. However, more comprehensive studies are needed before establishing the standard protocols of agreement-seeking cooperation, such as communication frequency and the duration of cooperation. Here, this article presents an initiative study on the impacts of communication capabilities on agreement-seeking cooperation. Through a large-scale analysis by regulating vehicle-to-vehicle (V2V) communication metrics, this work suggests desirable system parameters that can maximize the benefits of cooperation and ensure reliable operability while avoiding exhaustive communication loads. As the first step, an example agreement-seeking cooperation system is created for a car-following scenario, including decision-making and control algorithms for autonomous vehicles. Then, software-in-the-loop tests explore the performance of the developed system as it encounters various communication risks, such as latency and message packet drops. The system performance metrics are evaluated from various angles, including the time consumed for the agreement-seeking process, cooperation ratio, and the ratio of faulty cooperation. Energy saving from the cooperation is assessed by using simulation software that can run multiple high-fidelity vehicle models simultaneously. Based on the analyses, this article suggests the V2V communication requirements for the reliable operation of CDA agreement-seeking, which can be referred to when developing the standard protocols of agreement-seeking cooperation.

42 ENGINEERING↗
DOE OSTI2021

Advanced Performance Metrics and their Application to the Sensitivity Analysis for Model Validation and Calibration

High-quality generator dynamic models are critical to reliable and accurate power systems studies and planning. With the availability of PMUs, measurement-based approach for model validation has gained significant prominence. In this approach, the quality of a model is analyzed by visually comparing measured generator response with the model-based simulated response for large system disturbances. This paper proposes a new set of performance metrics to assess the model validation results to facilitate automation of the model validation process. In the proposed methodology, first, the slow governor response and comparatively faster oscillatory response are separated, and then a separate set of performance metrics is calculated for each of these two components. These proposed metrics quantify the mismatch between the actual and model-based response in a comprehensive manner without missing any information enabling automation of the process. Furthermore, in this paper, we are also proposing that the sensitivity analysis for model calibration be performed with respect to the proposed metrics for the systematic identification of key parameters. In this work, results obtained using both simulated and real-world case-studies validate the effectiveness of the proposed performance metrics for model validation and their application to the sensitivity analysis for model calibration.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Translating dosimetry of Dibenzo[ def,p ]chrysene (DBC) and metabolites across dose and species using physiologically based pharmacokinetic (PBPK) modeling

We report that Dibenzo[def,p]chrysene (DBC) is an environmental polycyclic aromatic hydrocarbon (PAH) that causes tumors in mice and has been classified as a probable human carcinogen by the International Agency for Research on Cancer. Animal toxicity studies often utilize higher doses than are found in relevant human exposures. Additionally, like many PAHs, DBC requires metabolic bioactivation to form the ultimate toxicant, and species differences in DBC and DBC metabolite metabolism have been observed. To understand the implications of dose and species differences, a physiologically based pharmacokinetic model (PBPK) for DBC and major metabolites was developed in mice and humans. Metabolism parameters used in the model were obtained from experimental in vitro metabolism assays using mice and human hepatic microsomes. PBPK model simulations were evaluated against mice dosed with 15 mg/kg DBC by oral gavage and human volunteers orally microdosed with 29 ng of DBC. DBC and its primary metabolite DBC-11,12-diol were measured in blood of mice and humans, while in urine, the majority of DBC metabolites were obeserved as conjugated DBC-11,12-diol, conjugated DBC tetrols, and unconjugated DBC tetrols. The PBPK model was able to predict the time course concentrations of DBC, DBC-11,12-diol, and other DBC metabolites in blood and urine of human volunteers and mice with reasonable accuracy. Agreement between model simulations and measured pharmacokinetic data in mice and human studies demonstrate the success and versatility of our model for interspecies extrapolation and applicability for different doses. Furthermore, our simulations show that internal dose metrics used for risk assessment do not necessarily scale allometrically, and that PBPK modeling provides a reliable approach to appropriately account for interspecies differences in metabolism and physiology.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. Further, to demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed. Further, to help facilitate the deployment of UQ in Scientific Machine Learning research and practice, we present and develop in [1] an open-source Python library (github.com/Crunch-UQ4MI/neuraluq), termed NeuralUQ, that is accompanied by an educational tutorial and additional computational experiments.

11 physics-informed neural networks↗
DOE OSTI2022

Optimal Portfolio Design of Distributed Energy Resources on Puerto Rico Distribution Feeders with Long Outages after Hurricane Maria

This work details a project to design reliable, resilient, and cost-effective networked microgrids considering grid constraints and resilience metrics focused on Puerto Rico distribution feeder locations with long outages after Hurricane Maria. The project consisted primarily of modeling and simulation tasks that accomplished the following objectives: 1. Selected 10 distribution feeder models in vulnerable areas. The sample feeders are geographically distributed across Puerto Rico and vary in length to capture the wide variety of feeders on the island. 2. Determined the optimal location and sizing of distributed energy resources (DERs) on the identified distribution feeders. The systems considered as part of the microgrid solutions were solar photovoltaic (PV), battery energy storage systems (BESS) and distributed fossil fuel generation (DFFG). 3. Estimated the cost-benefit of the proposed DER portfolios. 4. Provided a set of final recommendations that inform decision making on how to do targeted planning analysis for microgrids that can supply energy to critical infrastructures.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

PV_ICE: PV in the Circular Economy, Dynamic Energy and Materials Tool (PV ICE)

This open-source tool implements Circular Economy metrics for photovoltaic (PV) materials. It can be used to quantify and assign a value framework to efforts on re-design, reduction, replacement, reuse, recycling, and lifetime and reliability increases in the PV value chain. The PV_ICE is leveraging published data from different sources on PV manufacturing and predicted technological changes. Input data is being compiled here This tool will help implement circularity metrics, quantify and assign a value framework to efforts on re-design, reduction, replacement, reusage, recycling, and lifetime and reliability increases on PV.

Ovaitt, nee Ayala Pelaez, Silvana↗
DOE OSTI2022

Assessment methods for determining small changes in hearing performance over time

Although the behavioral pure-tone threshold audiogram is considered the gold standard for quantifying hearing loss, assessment of speech understanding, especially in noise, is more relevant to quality of life but is only partly related to the audiogram. Metrics of speech understanding in noise are therefore an attractive target for assessing hearing over time. However, speech-in-noise assessments have more potential sources of variability than pure-tone threshold measures, making it a challenge to obtain results reliable enough to detect small changes in performance. Here, this review examines the benefits and limitations of speech-understanding metrics and their application to longitudinal hearing assessment, and identifies potential sources of variability, including learning effects, differences in item difficulty, and between- and within-individual variations in effort and motivation. We conclude by recommending the integration of non-speech auditory tests, which provide information about aspects of auditory health that have reduced variability and fewer central influences than speech tests, in parallel with the traditional audiogram and speech-based assessments.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

Power System Resilience Evaluation Framework and Metric Review: Preprint

Power system resilience is an emerging hot topic in recent years to study the increasing threats of extreme events such as natural disasters, severe weather, and cyberattacks. Although many research works have been done to define, model, and quantify resilience from different aspects, the lack of universally accepted resilience metrics and evaluation methods makes it difficult to assess and compare resilience across different power systems like what is typically done in power system reliability studies. In this paper, we first review the definitions of resilience and summarized two core concepts shared by most literature. On top of that, we conduct a thorough review of resilience metrics and develop a new framework to assess power system resilience from two perspectives, i.e., pre-event estimation and post-event evaluation, to capture system resilience performance in both general and specific fashions. Existing resilience metrics are summarized and categorized using the proposed framework, where recommendations are also proposed to capture core concepts of resilience.

power system resilience↗
DOE OSTI2024

Prioritizing circular economy strategies for sustainable PV deployment at the TW scale

Global decarbonization requires an unprecedented scale-up of photovoltaic (PV) manufacturing and deployment. The material demand and eventual end of life management associated with multi-TW scale deployment poses many challenges. Circular Economy (CE) and it's associated R-Actions (Reduce, Reuse, Recycle) have been proposed to mitigate end of life management and material sourcing concerns. However, CE metrics typically focus on a single product and only consider mass, excluding energy flows. This work leverages the PV in Circular Economy (PV ICE) tool to quantify the deployment, mass, and energy impacts of R-Actions and proposed sustainable PV designs in the context of achieving energy transition deployment goals (75 TW in 2050). 13 module scenarios are established and evaluated across 6 capacity, mass and energy metrics to identify tradeoffs and priorities. We find that increasing module efficiency can reduce near-term material demands up to 30% and improve energy metrics by up to 9%. Material circularity (recycling) can minimize lifecycle wastes and reduce material demands at the cost of higher energy demands. Increasing module lifetime, including reliability improvements and reuse strategies, is effective at reducing both material (>10%) and energy demands (24%). Uniquely, lifetime improvements maximize benefits and minimize the harms across all six metrics while achieving multi-TW scale deployment.

Photovoltaics↗