Results for “Error Metrics” — Page 19

DOE OSTI2021

New Capability at ORNL: High Precision Uranium Titration

Destructive analytical measurements establish a nuclear facility’s nuclear material inventory and inventory differences for Nuclear Material Accountancy and Control. A nuclear laboratory’s ability to perform accurate high-precision analytical measurements is key for tracking large inventories within a facility’s material balance areas and during production to track material movement through dynamic processes. For uranium, these measurements are made by using several established high-precision measurement protocols. These include isotope dilution mass spectrometry, gravimetry, and potentiometric titrations. Whatever measurement technique a nuclear lab chooses to use, reference materials (RMs) with certified attributes and accompanying uncertainties are used to calibrate measurement systems, and are the cornerstone for accurate results. In addition to calibration, RMs provide for metrological traceability, are used for method development and validation, and thus provide critical evaluations of the appropriateness and performance of analytical processes used. Evaluations may include validation that a method is fit-for-purpose, quantification of systematic and random biases, and the evaluation of long-term and short-term performance metrics. High-precision measurement techniques require that measured attributes be certified to a high degree of precision in the RMs used—ultimately, to a higher degree than that of the measurement technique itself. The US authority on the production of special nuclear material Certified Reference Materials is the NBL Program Office (NBLPO), formally known as New Brunswick Laboratory (NBL). The NBLPO is responsible for the sales and distribution of existing NBL certified reference materials (CRMs) and for the production of the next generation of nuclear RMs. To accomplish its mission, NBLPO is establishing key base capabilities within the DOE laboratory complex that formerly existed at the NBL laboratory. The Nuclear Analytical Chemistry (NAC) section within the Chemical Sciences Division (CSD) at Oak Ridge National Laboratory (ORNL) is currently working with NBLPO to set up laboratory and measurement capabilities to provide measurements and capabilities for production and/or recertification of existing and future CRMs for uranium assay. The NBL-developed high-precision titration (HPT) method is a critically-evaluated, extremely precise and accurate primary method utilized for the determination of uranium content in a variety of uranium materials. The HPT method, combined with detailed balance weighing protocols, provides for an analytical methodology that is unsurpassed in precision and one in which all sources of error have been evaluated, a requirement of CRM certification. HPT capability within the United States was lost with the closure of the labs at NBL. The NAC has been collaborating with NBLPO to stand up and demonstrate the capability to perform uranium assay via HPT at ORNL. HPT can produce results with an expanded uncertainty of approximately 0.01% for pure uranium compounds, with typical precisions of <0.006%. The major tasks required to stand up the method at ORNL were the refurbishment of a dedicated lab and equipment setup, procedure development, analyst training, establishing method-specific quality assurance, and qualification of an analyst to perform the method. This report summarizes these tasks, outlines the documents drafted, and gives the outcome of the qualification titrations.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗

DOE OSTI2021

A second order yield-temperature relation for accurate inference of burn-averaged quantities in multi-species plasmas

Measured yields and ion temperatures inferred from the fusion product energy spectra can be used as metrics for the performance of an ICF implosion. This can be used to infer species separation, thermal decoupling, flows, or other effects that can cause the inferred ion temperatures to deviate from the true underlying thermal temperature and the yield ratio to deviate from the expected value. Direct inference of the impact of these effects on observed temperatures and yields can be difficult to uncover due to the underlying dependence on the shape and time evolution of the temperature and density profiles of the fusing plasma. Due to differences in the temperature dependence of the reactivities, different fusion products are emitted from different regions and times within the implosion. In order to properly account for this, a second-order analytical expression relating the apparent temperatures and yield ratios is developed. This expression can be coupled to models of yield and/or temperature altering effects to infer their burn-averaged impact on an implosion. The second-order expression shows significant improvement over lower-order expressions in synthetic data studies. Demonstrations of its applications to synthetic data coupled with models of ion thermal decoupling and radial flows are presented. In the case of thermal decoupling, both first and second-order expressions show reasonable levels of accuracy. To consistently infer the amplitude of radial flow with a <10% error, the second-order equation is required.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2020

Characterization and Analysis of the Energy-Reporting Accuracy of Connected Devices

Emerging energy-efficient building systems increasingly exhibit greater functionality, often requiring multiple operating modes (e.g. white-tunability for lighting products and data traffic for devices with networked, integrated sensors). This increased functionality makes energy consumption estimates more complex. Given that these functions consume energy, the energy performance of such building systems is dependent on what operating modes they use and how much time they spend in each mode. Devices and systems that can report their own energy consumption mitigate this energy-performance uncertainty. This study explores the energy-reporting accuracy of market-available connected electrical outlets. The study considers two residential-market products (five units each, one outlet per unit) and three commercial-market products (two units each, 18 to 24 outlets per unit) with the ability to report power drawn and/or energy consumed by devices connected to their receptacles. The products were purchased through typical market channels. Pacific Northwest National Laboratory (PNNL) conducted testing in December 2018 at its Connected Lighting Test Bed (CLTB), using a custom-developed test setup and method adapted from industry standards. The setup collected energy-consumption data reported by the outlet devices under test (DUTs) at one-minute intervals and compared that data with measurements taken by a reference meter over a range of test conditions. The residential products reported power draw but not interval or cumulative energy consumption. The commercial products reported both power draw and cumulative energy consumption. Relative reporting error (RRE) was calculated for all measurements, and analysis of the results revealed variations across devices and test conditions. The total number of measurements (50 for each residential product, 60 for each commercial product) offers an appreciable comparison of performance at the make/model level. The average RRE of the residential products derived from reported power draw was -0.02% and -1.20%. The average RRE for two of the three the commercial products derived from reported power draw was worse than those of the residential products (-2.40%, -2.72%, -0.36%). The internal integration of power over time, used to calculate cumulative energy consumption, typically occurs at current and voltage sampling rates much higher than once per minute. This suggests that the average commercial-product RRE derived from reported energy consumption should be very consistent and better than performance based on reported power draw. However, the RRE derived from reported energy consumption varied significantly across the three makes of commercial-market products and was uniformly less accurate than performance based on reported power draw. Subsequent analysis identified a number of root causes for this decrease in performance, most of which were related to reporting resolution. The goals of this study are to generate awareness of building systems capable of reporting their own energy consumption, further interest in the value of energy data for a variety of uses, draw attention to how the accuracy of reported metrics can be characterized, and quantify the performance variation found in marketavailable products. The results of this study and subsequent related work may be relevant to stakeholders in industry-specification and standards-development organizations. The methods this study employs could inform test and measurement procedures and performance classifications for connected outlets, lighting products, and other building systems capable of reporting their own energy consumption. The study concludes with stakeholder recommendations, including the following: • Energy-reporting device and system manufacturers developing products that report energy consumption should characterize the accuracy of reported metrics using a reference meter calibrated by an independent laboratory that was accredited by an ILAC MRA signatory (and whose scope of accreditation explicitly covers energy measurement), and should include this information on product data sheets. • Standards and specification development organizations should develop application-specific performance classifications that end users can understand and relate to their energy-data use needs (e.g., 2% accuracy class for utility streetlight energy billing needs, or 10% accuracy class for ESCO performance verification needs). • Current or potential owners, operators, and specifiers of energy-reporting building systems should rigorously analyze the dependency of current and planned energy-data use cases on accuracy, noting in particular the dependence (or lack thereof) on relative vs. absolute accuracy, and on trueness vs. precision (i.e., repeatability), and should communicate use-case needs to industry standards and specification organizations.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2020

Characterization and Analysis of the Energy-Reporting Accuracy of Connected Devices

Emerging energy-efficient building systems increasingly exhibit greater functionality, often requiring multiple operating modes (e.g. white-tunability for lighting products and data traffic for devices with networked, integrated sensors). This increased functionality makes energy consumption estimates more complex. Given that these functions consume energy, the energy performance of such building systems is dependent on what operating modes they use and how much time they spend in each mode. Devices and systems that can report their own energy consumption mitigate this energy-performance uncertainty. This study explores the energy-reporting accuracy of market-available connected electrical outlets. The study considers two residential-market products (five units each, one outlet per unit) and three commercial-market products (two units each, 18 to 24 outlets per unit) with the ability to report power drawn and/or energy consumed by devices connected to their receptacles. The products were purchased through typical market channels. Pacific Northwest National Laboratory (PNNL) conducted testing in December 2018 at its Connected Lighting Test Bed (CLTB), using a custom-developed test setup and method adapted from industry standards. The setup collected energy-consumption data reported by the outlet devices under test (DUTs) at one-minute intervals and compared that data with measurements taken by a reference meter over a range of test conditions. The residential products reported power draw but not interval or cumulative energy consumption. The commercial products reported both power draw and cumulative energy consumption. Relative reporting error (RRE) was calculated for all measurements, and analysis of the results revealed variations across devices and test conditions. The total number of measurements (50 for each residential product, 60 for each commercial product) offers an appreciable comparison of performance at the make/model level. The average RRE of the residential products derived from reported power draw was -0.02% and -1.20%. The average RRE for two of the three the commercial products derived from reported power draw was worse than those of the residential products (-2.40%, -2.72%, -0.36%). The internal integration of power over time, used to calculate cumulative energy consumption, typically occurs at current and voltage sampling rates much higher than once per minute. This suggests that the average commercial-product RRE derived from reported energy consumption should be very consistent and better than performance based on reported power draw. However, the RRE derived from reported energy consumption varied significantly across the three makes of commercial-market products and was uniformly less accurate than performance based on reported power draw. Subsequent analysis identified a number of root causes for this decrease in performance, most of which were related to reporting resolution. The goals of this study are to generate awareness of building systems capable of reporting their own energy consumption, further interest in the value of energy data for a variety of uses, draw attention to how the accuracy of reported metrics can be characterized, and quantify the performance variation found in marketavailable products. The results of this study and subsequent related work may be relevant to stakeholders in industry-specification and standards-development organizations. The methods this study employs could inform test and measurement procedures and performance classifications for connected outlets, lighting products, and other building systems capable of reporting their own energy consumption. The study concludes with stakeholder recommendations, including the following: • Energy-reporting device and system manufacturers developing products that report energy consumption should characterize the accuracy of reported metrics using a reference meter calibrated by an independent laboratory that was accredited by an ILAC MRA signatory (and whose scope of accreditation explicitly covers energy measurement), and should include this information on product data sheets. • Standards and specification development organizations should develop application-specific performance classifications that end users can understand and relate to their energy-data use needs (e.g., 2% accuracy class for utility streetlight energy billing needs, or 10% accuracy class for ESCO performance verification needs). • Current or potential owners, operators, and specifiers of energy-reporting building systems should rigorously analyze the dependency of current and planned energy-data use cases on accuracy, noting in particular the dependence (or lack thereof) on relative vs. absolute accuracy, and on trueness vs. precision (i.e., repeatability), and should communicate use-case needs to industry standards and specification organizations.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2021

Calibration of the Diffusivity Predictions of Centipede Using Approximate Bayesian Computation and Applications in Nyx (Engineering Scale) and Xolotl-MARMOT (Meso-Scale) Simulations

Fission gas evolution and release in UO 2 nuclear fuel are important fuel performance metrics and occur in several distinct stages: 1) nucleation, growth and resolution of intra-granular bubbles, 2) diffusion to grain boundaries and 3) nucleation and growth of bubbles at grain boundaries, which eventually form a connected network (percolation) enabling release of gas from grain boundaries through connections to triple junctions, grain edges or free surfaces. The NE-SciDAC project is developing several computational tools to model this problem, which are connected in a hierarchical multi-scale framework. The information transfer in the multi-scale framework is a critical step that, in addition to best-estimates, should include uncertainty quantification. Despite taking a first-principles multi-scale approach, there is a need to perform parameter calibration to ensure consistency with available experimental data. In the present study, uncertainty quantification (UQ) and parameter calibration is demonstrated for one of the lower length scale codes in the multi-scale framework (Centipede) and then the results, including instances of the propagated uncertainties, are used in other codes within the framework, specifically Nyx and Xolotl-MARMOT. We calibrated the model parameters in Centipede, a computer code used to predict diffusivities of uranium (U) and xenon (Xe) in the context of the simulation of fission gas in uranium oxide (UO 2 ) nuclear fuel. The Centipede code depends on 183 parameters, all of which are subject to uncertainty. The three data sets used in our calibration effort are taken from the literature. This data is available as a set of measurements, including measurement errors. Our goal is to calibrate a statistical model that predicts both the value of the measurement and the uncertainty associated with the measurement. We perform a Bayesian calibration of the model parameters using a dedicated approximate Bayesian computation (ABC) likelihood function. To avoid excessive computational costs, we replace the expensive Centipede simulation code by a higher-order surrogate model, constructed using only the 9 most important parameters. These important parameters are identified by a preliminary global sensitivity analysis (GSA) study. Among the important parameters are T0 (the temperature at which UO 2 is perfectly stoichiometric) and Hf_pO2 (the temperature dependence of the oxygen (O) partial pressure) that should be considered as operating conditions to be estimated along with the other parameters. We consider two different cases: one where we define one set of these operating conditions for all data sets, and one where we define distinct operating condition parameters for each data set. The Xe diffusivities predicted by the latter case show distinct features that could not be observed in the former. Next, we use the diffusivity predictions by Centipede as input to Nyx, a reduced order fuel performance code focused on gas behavior alone, in order to estimate quantities associated with inter-granular bubble formation at conditions specified by the experiments. Finally, the diffusivities obtained from the calibrated Centipede runs were used in coupled Xolotl-MARMOT simulations of intra- and inter-granular gas evolution. The results are compared to simulations using the baseline diffusivities from Turnbull et al.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2026

Object-Based Evaluation of Dynamical and Statistical Downscaled Precipitation Products over CONUS

High-resolution precipitation data, generated through dynamical downscaling (DD) or statistical downscaling (SD) of global climate model output, provide critical information for regional climate assessment and adaptation planning. Most downscaling development and validation have focused on accurate gridscale precipitation construction and ignored the spatial structure of precipitation across model grids and at the event scale. However, many applications, e.g., hydrologic modeling and the analysis using the downscaled precipitation, require a reasonable representation of the spatial structure of precipitation within watersheds. Therefore, a set of standard metrics to evaluate the representation of the spatial structure of individual storms across diverse downscaled precipitation products is desired. To address this need, we conducted an object-based evaluation of precipitation in decades-long DD and SD products over the contiguous United States (CONUS). Specifically, we evaluate their ability to reproduce various features of precipitation objects in the observations: total volume, precipitation area, peak intensity, and spatial structure. Multiple metrics (bias, Perkins score, and nonparametric statistical tests) are used to quantify model performance. Our evaluation reveals notable variations in performance among individual products across different climate zones and seasons, as well as between extreme and nonextreme events. In general, most DD products exhibit balanced performance across the four precipitation object features, while SD products vary more significantly in their performance across products. Based on this comprehensive evaluation, we provide guidance on choosing downscaled products for specific regions, seasons, and precipitation object features. These findings and recommendations can inform precipitation-relevant modeling and analysis over CONUS, guide future downscaling technique developments, and provide actionable information for climate impact assessment and adaptation.

Downscaling↗

DOE OSTI2024

Integrated Spatial, Spectral, & Temporal Optical Reflectance System for Precision Occupancy & Location Sensing to Improve Building Energy Efficiency

Buildings consume approximately 35% of the electricity used in the U.S. and building owners can significantly reduce this energy use by providing services like heating, electrical power and lighting only when people are present. The ARPAe funded program titled “INTEGRATED SPATIAL, SPECTRAL, & TEMPORAL OPTICAL REFLECTANCE SYSTEM FOR PRECISION OCCUPANCY & LOCATION SENSING TO IMPROVE BUILDING ENERGY EFFICIENCY” demonstrates how a low cost sensor technology developed for measuring distances can be used to count and locate occupants with a high degree of precision with a very low error rates. This platform tells a building control system where occupants are located (but not who they are) so that energy consuming services can be provided only when the services are needed by building occupants. The original proof of concept involved using low cost, commercially available time-of-flight (TOF) sensors that measure distance, but the performance of these existing sensors was lacking, as they could not operate properly in the presence of sunlight, which blinded the simple TOF sensors and limited their utility in buildings. This project proposed a powerful new class of TOF sensors that used state-of-the-art integrated circuit (IC) fabrication processes that combined advanced photonics with conventional silicon chip circuitry for improved sensor performance. An equally important part of this project was to find ways to maximize occupant count and location accuracy while using the fewest number of sensors possible, in order to keep costs low. By using building blueprints to create digital twins of commercial building spaces, the team developed new algorithms to maximize occupant count and tracking accuracy by properly locating the minimum number of sensors at just the right spots in the building. This capability not only minimizes system costs but also simplified sensor installation and system commissioning. Our simulations of our sensor networks for a range of commercial floorplan designs demonstrated that our installed cost target of $0.08/sqft was attainable, though not fully demonstrated during the project. Finally, we noted that the TOF sensor concept could provide a valuable role in health and eldercare by tracking patients without the need for worn sensors and would be useful for fall detection and other patient safety metrics, including tracking healthcare/patient interactions. We feel that, when fully developed, this new class of sophisticated TOF sensors and support software will be a powerful new approach to improving building energy efficiency based on occupant centric control platforms and will also open new levels of patient safety in healthcare operations. To realize this potential, the team formed the Troy Sensor Company LLC to oversee licensing of the programs patents and continue to seek commercialization of this program’s activity sensing technologies.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2024

Methods for Incorporating Model Uncertainty into Exoplanet Atmospheric Analysis

A key goal of exoplanet spectroscopy is to measure atmospheric properties, such as abundances of chemical species, in order to connect them to our understanding of atmospheric physics and planet formation. In this new era of high-quality JWST data, it is paramount that these measurement methods are robust. When comparing atmospheric models to observations, multiple candidate models may produce reasonable fits to the data. Typically, conclusions are reached by selecting the best-performing model according to some metric. This ignores model uncertainty in favor of specific model assumptions, potentially leading to measured atmospheric properties that are overconfident and/or incorrect. In this paper, we compare three ensemble methods for addressing model uncertainty by combining posterior distributions from multiple analyses: Bayesian model averaging, a variant of Bayesian model averaging using leave-one-out predictive densities, and stacking of predictive distributions. We demonstrate these methods by fitting the Hubble Space Telescope (HST) + Spitzer transmission spectrum of the hot Jupiter HD 209458b using models with different cloud and haze prescriptions. All of our ensemble methods lead to uncertainties on retrieved parameters that are larger but more realistic and consistent with physical and chemical expectations. Since they have not typically accounted for model uncertainty, uncertainties of retrieved parameters from HST spectra have likely been underreported. We recommend stacking as the most robust model combination method. Our methods can be used to combine results from independent retrieval codes and from different models within one code. They are also widely applicable to other exoplanet analysis processes, such as combining results from different data reductions.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

Develop a new integrated macro→micro←nano (MMN) multiscale modeling framework to optimize high strength aluminum alloys and processes for vehicle light-weighting

Bending tests provide a means to study plane strain fracture performance of 6000 series aluminum alloys. Metrics from bending tests have been correlated with self-pierce riveting (SPR) performance of a high strength AA6111 automotive aluminum alloy in previous works. Using the ORNL HPC resources, this project developed an innovative macro→micro←nano (MMN) multiscale microstructure-based finite element (FE) code to further understanding of the relationship between microstructure and fracture properties of high-strength 6000-series alloys. This work started with microstructural characterization in both mesoscale and nanoscale and bending performance characterization of AA6111 HS2-T6 alloy at Ford, the MMN framework was applied to this alloy to simulate 3-point VDA bending. From the results of the macro-modeling of 3-point VDA bending, the critical region of fracture was identified, and the region geometry was used to construct the micro-model. The fracture criterion of micron-scale precipitates and aluminum matrix which contains submicron and nano particles (AL-SMP-NP) in the micro-model was calibrated and validated by comparing simulated and measured bending results. With the AL-SMP-NP fracture strain obtained, the fracture strain of Al-matrix containing nano particles (ALNP) will similarly be determined by a submicron scale-model using an edge-constrained FE modeling approach developed by Hu et al. With the ALNP fracture strain obtained, the fracture strain of Al-matrix containing no particles will similarly be determined by a nano scale-model using an edge-constrained FE modeling approach. After the MMN framework is built and fracture criterion calibrated, nano-model FE simulations with virtual microstructures was performed to obtain a reduced order model (ROM) of the fracture criterion of the ALNP as a function of volume fraction, size, and distribution of the nanoparticle. This nano→submicron→micro modeling part allows exploration of the influence of different material nanostructures from different process conditions on the bending properties within the multiscale bending simulation framework and the ROM of material bendability as a function of nanoparticle size and shape was established. This obtained reduced order model (ROM) could help guide the design and selection of materials to improve existing SPR process models that could replace trial-and-error rivet/die selection and help to design new rivet/die combinations capable of robustly joining new higher strength 6000 5 series alloys in automotive body structures. This would enable lightweighting of Ford vehicles leading to greater fuel efficiency and reduce manufacturing time and energy.

36 MATERIALS SCIENCE↗

DOE OSTI2023

Lithium-Ion Battery Diagnostics Using Electrochemical Impedance via Machine-Learning

Diagnosing battery states such as health, state-of-charge, or temperature is crucial for ensuring the safety and reliability of electrochemical energy storage systems. While some states, such as temperature, may be measured using cheap sensors, accurate diagnosis of battery health metrics usually requires time-consuming performance measurements, making them infeasible for use in real-world operation. These health metrics can be measured during lab-testing and then estimated on-line using predictive life models or via state observer algorithms such as Kalman filters, but these predictive methods should be supplemented by actual measurement of battery health whenever possible to ensure reliability. Rapid measurement of battery health may be done by various types of fast diagnostic techniques such as electrochemical impedance spectroscopy (EIS), which can be performed in only a few minutes and require only a fraction of the energy and power needed for a full charge and discharge measurement. But there is a substantial challenge for estimating battery health using EIS data, as EIS is sensitive to cell temperature, state-of-charge, current, and resting time in addition to health. Thus, utilizing EIS data to predict battery capacity requires correcting for all these additional variables, a task that is extremely difficult to handle analytically. This talk utilizes machine-learning methods to estimate the effectiveness of battery capacity prediction from EIS data, leveraging a data set of hundreds of EIS measurements recorded at varying temperature and state-of-charge throughout a 500-day aging study of 32 commercial, large-format NMC-Graphite lithium-ion batteries. Using EIS as input to machine-learning models is complicated by the nonlinear response of impedance to battery health, temperature, and state-of-charge, as well as the collinearity between the impedance response at neighboring frequencies, which can easily lead to overfit models. To train robust models, features from EIS data need to be extracted from the data or some subset of critical frequencies selected. Many approaches for extracting and selecting features from EIS data from electrochemical analysis and machine-learning fields were identified for analysis: using the entire raw spectra; selection of one, two, or many frequencies from the entire spectra; selecting interesting points from the EIS measurement using domain knowledge; fitting EIS with an equivalent-circuit model; calculating statistics on the raw impedance values; and reducing the dimensionality of the data using unsupervised linear (principal component analysis) and non-linear (uniform manifold approximation and projection) methods. These approaches were rigorously compared using a machine-learning pipeline approach, training linear, Gaussian process, and random forest regression models and quantifying performance using cross-validation as well as a held-out test set. An artificial neural network model trained on the raw spectra was also tested. Promising pipelines were fine-tuned via Bayesian hyperparameter optimization using cross-validation loss and training with class-specific weights to counter data set imbalance. The most reliable method for utilizing impedance in this work was the selection of two optimal frequencies through an exhaustive search, resulting in about 2% mean absolute error on test data for both Gaussian process and random forest model architectures. Interrogation of a variety of models reveals critical frequencies of 100 Hz and 103 Hz for this data set, though the optimal set of frequencies is not necessarily intuitive, i.e., the best performing models are not simply those that use impedance at frequencies that have the highest correlation to the relative discharge capacity. The best performing model is an ensemble model, which is able to predict battery capacity with 1.9% mean absolute error for unseen cells using impedance recorded at a variety of temperatures and states-of-charge.

battery↗