Results for “data quality metrics” — Page 35

NASA NTRS2019

The ILRS: Approaching 20 Years and Planning for the Future

The International Laser Ranging Service (ILRS) was established by the International Association of Geodesy (IAG) in 1998 to support programs in geodesy, geophysics, fundamental constants and lunar research, and to provide the International Earth Rotation Service with data products that are essential to the maintenance and improvement in the International Terrestrial Reference Frame (ITRF), the basis for metric measurements of changes in the Earth and Earth–Moon system. Other scientific products derived from laser ranging include precise geocentric positions and motions of ground stations, satellite orbits, components of Earth’s gravity field and their temporal variations, Earth Orientation Parameters, precise lunar ephemerides and information about the internal structure of the Moon. Laser ranging systems are already measuring the one-way distance to remote optical receivers in space and are performing very accurate time transfer between remote sites in the Earth and in Space. The ILRS works closely with the IAG’s Global Geodetic Observing System. The ILRS develops (1) the standards and specifications necessary for product consistency, and (2) the priorities and tracking strategies required to maximize network efficiency. The service collects, merges, analyzes, archives and distributes satellite and lunar laser ranging data to satisfy a variety of scientific, engineering, and operational needs and encourages the application of new technologies to enhance the quality, quantity, and cost effectiveness of its data products. The ILRS works with (1) new satellite missions in the design and building of retroreflector targets to maximize data quality and quantity, and (2) science programs to optimize scientific data yield. Since its inception, the ILRS has grown to include forty laser ranging stations distributed around the world. The ILRS stations track more than ninety satellites from low Earth orbit (LEO) to the geosynchronous orbit altitude as well as retroreflector arrays on the surface of the Moon. Applications have been expanded to include time transfer, asynchronous ranging for targets at extended ranges, free space quantum telecommunications, and the tracking of space debris. Laser ranging technology is moving to lower energy, higher repetition rates (kHz), single-photon-sensitive detectors, shorter pulse widths, shorter normal point intervals for faster data acquisition, and increased pass interleaving, automated to autonomous operation with remote access, and embedded software for real-time updates and decision making. An example of pass interleaving is presented for the Yarragadee station (see Fig. 4); tracking of LEO satellites is often accommodated during break in LEO and GNSS passes. New satellites arrays provide more compact targets and work continues on the development of lighter less expensive arrays for satellites and the moon. The service now provides operational ITRF products including daily/ weekly station positions and daily resolution Earth orientation products; the flow of weekly combination of satellite orbit files for LAGEOS/Etalon-1 and -2 has recently been established. New products are under testing through a pilot project on systematic error monitoring currently underway. The article will give an overview of activities underway within the service, paths forward presently envisioned, and current issues and challenges.

Laser retroreflectors↗

DOE OSTI2018

(Machine) learning to do more with less

Determining the best method for training a machine learning algorithm is critical to maximizing its ability to classify data. In this paper, we compare the standard “fully supervised” approach (which relies on knowledge of event-by-event truth-level labels) with a recent proposal that instead utilizes class ratios as the only discriminating information provided during training. This so-called “weakly supervised” technique has access to less information than the fully supervised method and yet is still able to yield impressive discriminating power. In addition, weak supervision seems particularly well suited to particle physics since quantum mechanics is incompatible with the notion of mapping an individual event onto any single Feynman diagram. We examine the technique in detail — both analytically and numerically — with a focus on the robustness to issues of mischaracterizing the training samples. Weakly supervised networks turn out to be remarkably insensitive to a class of systematic mismodeling. Furthermore, we demonstrate that the event level outputs for weakly versus fully supervised networks are probing different kinematics, even though the numerical quality metrics are essentially identical. This implies that it should be possible to improve the overall classification ability by combining the output from the two types of networks. For concreteness, we apply this technology to a signature of beyond the Standard Model physics to demonstrate that all these impressive features continue to hold in a scenario of relevance to the LHC. Example code is provided on GitHub.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2013

Assessing the Quality of Bioforensic Signatures

We present a mathematical framework for assessing the quality of signature systems in terms of fidelity, cost, risk, and utility—a method we refer to as Signature Quality Metrics (SQM). We demonstrate the SQM approach by assessing the quality of a signature system designed to predict the culture medium used to grow a microorganism. The system consists of four chemical assays designed to identify various ingredients that could be used to produce the culture medium. The analytical measurements resulting from any combination of these four assays can be used in a Bayesian network to predict the probabilities that the microorganism was grown using one of eleven culture media. We evaluated fifteen combinations of the signature system by removing one or more of the assays from the Bayes network. We demonstrated that SQM can be used to distinguish between the various combinations in terms of attributes of interest. The approach assisted in clearly identifying assays that were least informative, largely in part because they only could discriminate between very few culture media, and in particular, culture media that are rarely used. There are limitations associated with the data that were used to train and test the signature system. Consequently, our intent is not to draw formal conclusions regarding this particular bioforensic system, but rather to illustrate an analytical approach that could be useful in comparing one signature system to another.

Bayesian networks↗

NASA NTRS2010

HOPE: An On-Line Piloted Handling Qualities Experiment Data Book

A novel on-line database for capturing most of the information obtained during piloted handling qualities experiments (either flight or simulated) is described. The Hyperlinked Overview of Piloted Evaluations (HOPE) web application is based on an open-source object-oriented Web-based front end (Ruby-on-Rails) that can be used with a variety of back-end relational database engines. The hyperlinked, on-line data book approach allows an easily-traversed way of looking at a variety of collected data, including pilot ratings, pilot information, vehicle and configuration characteristics, test maneuvers, and individual flight test cards and repeat runs. It allows for on-line retrieval of pilot comments, both audio and transcribed, as well as time history data retrieval and video playback. Pilot questionnaires are recorded as are pilot biographies. Simple statistics are calculated for each selected group of pilot ratings, allowing multiple ways to aggregate the data set (by pilot, by task, or by vehicle configuration, for example). Any number of per-run or per-task metrics can be captured in the database. The entire run metrics dataset can be downloaded in comma-separated text for further analysis off-line. It is expected that this tool will be made available upon request

Jackson, E. B.↗

DOE OSTI2016

Further Investigation of Employing Binomial Small-Sample Estimator in M-LROM

Reduced Order Modeling (ROM) is essential to enable the identification of the so-called active parameter subspace for high-fidelity models that are employed in computationally intensive analyses such as uncertainty characterization and data assimilation. Given the complexity of models for real-world application, brute force methods to identify the active subspace may not be computationally efficient to realize the value of reduction, since in realistic problems, the full scale model must be executed few hundred times, which may be computationally taxing. Earlier work has introduced a new approach, which represents a further development of our previous work on ROM, referred to as multi-level ROM (M-LROM). In M-LROM, approximate models are employed to identify the active subspace instead of the full scale model. In this summary, we present our recent results on M-LROM, specifically in regard to the construction of reliable error metrics that can bound the errors resulting from the reduction. The idea is to employ a lower fidelity model, i.e., an approximation of the full scale or high fidelity model, to build the active subspace, and via the proposed error bound, the quality of the active subspace can be determined by calculating an upper bound on the errors resulting from the reduction. For this approach to be practical, the error bound must be realistic, implying that it should be close to the actual errors expected, and not much higher. Therefore, an important goal of this summary is to compare different error bounds in terms of the relation between their estimated bounds to the actual errors. In particular, we compare the proposed error bound which numerically construct an estimator using binomial distribution to the widely used small-sample norm estimators. Unlike most probabilistic error estimation methods, the proposed bound employs a binomial distribution to construct the estimator whereas in other conventional methods, the distribution employed is typically the standard Gaussian, normalized over a unit hypersphere as initially proposed by the seminal paper of J. Dixon on random matrix theory. Previous publications provided evidence that the binomial distribution calculates more realistic bounds as compared to the Gaussian distribution, which tends to be conservative, implying that the bound is much bigger than the actual errors, which diminishes the value of the reduction, as now must increase the size of the active subspace to reach the desired upper bound on the reduction errors. This work has provided a rigorous approach by which error bounds on the reduction errors can be established without having to execute the high fidelity model during the construction of the active subspace. The high fidelity model is only need to test the error bound using small number of executions. This methodology paves the way for a more efficient application of ROM to complex engineering models, where standard ROM techniques may be too taxing to optimally realize their value in practical situations. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2018

Mechanical Behavior of UO 2 at Sub-grain Length Scales: Quantification of Elastic, Plastic and Creep Properties via Microscale Testing

Techniques were developed to measure properties at sub-grain scales using depleted Uranium Oxide (d-UO2) samples heat-treated to obtain different grain sizes and oxygen stoichiometries, through three main tasks: 1) sample processing and characterization, 2) microscale and conventional testing and 3) modeling. Grain size and crystallography were characterized using Scanning Electron Microscopy (SEM), in conjunction with Electron Backscattering Diffraction (EBSD) and Electron Channeling Contrast Imaging (ECCI). Grains were then carefully selected based on their crystallographic orientations to perform ex-situ micromechanical tests with samples machined via Focused Ion Beam (FIB), with emphasis on micro-cantilever bending. These experiments were performed under controlled atmospheres, to insure stoichiometry control, at temperatures up to 700 °C and allowed measurements involving elastic (effective Young’s modulus), plastic (critical resolved shear stresses) and creep (creep strain rates) behavior. Conventional compression experiments were performed simultaneously to compare with the ex-situ measurements and study potential size effects. Modeling was implemented using anisotropic elasticity and inelastic constitutive relations for plasticity and creep based on kinematics and kinetics of dislocation glide that account for the effects of crystal orientation, and stress. The models will be calibrated and validated using the experimental data. This project provided insight on correlations among stoichiometry, crystallography and mechanical behavior in advanced oxide fuels, provided valuable experimental data to validate and calibrate mesoscale fuel performance codes and also a framework to measure sub-grain scale mechanical properties that should be suitable for use with irradiated samples due to small volumes required. The goals and metrics of the ongoing study of thermo-mechanical behavior in depleted uranium dioxide (d-UO 2 ) outlined in this project have been concluded successfully, resulting in: 1) the successful fabrication, processing, and characterization of large-grained samples with various orientations (up to and including single crystals) having stoichiometric and hyper-stoichiometric O/U ratios; 2) formulation, calibration, and validation of a crystal plasticity constitutive model to describe the creep deformation of UO 2 at the sub-grain length scale (single crystal level) at intermediate temperatures; 3) the successful calibration of a crystal plasticity constitutive model to describe the elasto-plastic deformation of microcantilever beams, also at moderate temperatures. Samples were prepared from natural uranium oxide powder of production-quality provided by Areva. The powder was pressed in a die to a pressure of 100 MPa to produce green pellets with no sintering aids, lubricants, or any other additives. The green pellets were then heated up to 1700 °C under ultra-high purity argon atmosphere (~1 ppm O2). The atmosphere was then changed to 79% Argon, 21% O 2 and the temperature was held at 1700 °C for 2 hours to sinter the pellets under oxidative conditions [1] that are known to increase grain growth kinetics in UO 2 [2]. Samples were then cooled down under Ar-4%H2 atmosphere to reduce the samples back to stoichiometric UO 2 . For macro-scale procedures, testing of UO 2 samples with large grains was performed at 1200 °C using a modified load frame capable of applying dead-weight loads to ensure constant stress conditions, while displacement of the sample produced by the applied load was measured with high precision micrometers to obtain strains. Stress steps were used during testing and the strains were monitored to measured creep strain rates under steady state for each level of stress used, so that stress exponents could be obtained. The results of the mechanical testing, along with sample geometry and crystal orientation of the grains in the samples, as well as post-test sample characterization were used to formulate a viscoplastic model to account for steady state (stage II) creep behavior, along with basic assumptions from crystal plasticity and kinematic constraints due to testing fixtures. In the micro-scale, testing of microcantilever beams at temperatures ranging from 25 to 570 °C was performed in-situ with a scanning electron microscope with a special attachment to apply load and measure displacement while the samples were at temperature. The load-displacement curves obtained showed linear behavior before fracture for all temperatures attempted except 570 °C, where clear deviations from non-linearity were observed before fracture. These deviations were consistently observed for all samples tested for a given orientation. A viscoplastic model was used to account for the presence of inelastic strain, along with basic assumptions from crystal plasticity and beam theory. These models were kept as simple as possible, and results from tests performed in a set of samples with a given crystal orientation were used to calibrate the material constants for the model, while results from a different sample set were then used for validation, thus satisfying the conditions of all main tasks within the parameters of this project. Details of these efforts are outlined in this report.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2024

Algal Biomass Production via Open Pond Algae Farm Cultivation: 2023 State of Technology and Future Research

The annual State of Technology (SOT) assessment is an essential activity for platform research conducted under the Bioenergy Technologies Office (BETO). It allows for the impact of research progress (both directly achieved in-house at the National Renewable Energy Laboratory [NREL] and furnished by partner organizations) to be quantified in terms of economic improvements in the overall biofuel production process for a particular biomass processing pathway, whether based on terrestrial or algal biomass feedstocks. As such, initial benchmarks can be established for currently demonstrated performance, and progress can be tracked toward out-year goals to ultimately demonstrate economically viable biofuel technologies. NREL's algae SOT benchmarking efforts historically focused both on front-end algal biomass production and separately on back-end conversion to fuels through NREL's "combined algae processing" (CAP) pathway. The production model is based on outdoor long-term cultivation data, enabled by comprehensive algal biomass production trials conducted under the Development of Integrated Screening, Cultivar Optimization, and Verification Research (DISCOVR) consortium efforts, driven by data furnished by Arizona State University (ASU) at the Arizona Center for Algae Technology and Innovation (AzCATI) testbed site. The CAP model is based on experimental efforts conducted primarily under NREL research and development projects. This report focuses on front-end algal biomass production, documenting the pertinent algal biomass cultivation parameters that were input to the NREL open pond algae farm model. Through partnerships under DISCOVR, collaborators at ASU furnished details on cultivation performance metrics including biomass productivity and harvest densities for recent growth trials done at the AzCATI site. The resulting biomass productivity was calculated at 16.7 g/m 2 /day (ash-free dry weight [AFDW], annual average) for seasonal cultivation of Picochlorum celeri TG2 and Monoraphidium minutum 26B-AM biomass strains at the ASU site. Picochlorum celeri achieved the best productivity from April to September, with Monoraphidium minutum 26B-AM being used between October and March. Tetraselmis striata LANL1001, usually part of the strain rotation in previous cultivation SOTs, was supplanted by Monoraphidium minutum 26B-AM in this year's outdoor cultivation trials. Finally, building from an industry case study presented in the 2022 SOT report, in the Appendix of this report we provide an update on further improved data furnished by an industry collaborator and resultant impacts on economics reflecting several seasonal scenarios. This case study provides a supplementary datapoint on work being performed elsewhere with a more dedicated focus on improved compositional quality, producing biomass enriched in lipids as may be more optimal for conversion upgrading to fuels and products.

09 BIOMASS FUELS↗

DOE OSTI2021

JGI QC impact on assembly, binning, phylogenomics, and functional analysis

Background Investigators using metagenomic sequencing to study their microbiomes are often provided data that has been trimmed and decontaminated or do it themselves without knowing the effect these procedures can have on their downstream analyses. Here we evaluated the impact that JGI trimming and decontamination procedures had on assembly and binning metrics, placement of metagenome assembled genomes into species trees, and functional profiles of metagenome-assembled genomes (MAGs) extracted from twenty three complex rhizosphere metagenomes. We also investigated how more aggressive trimming impacts these binning metrics. Results We found that JGI trimmed and decontamination of input reads had some significant impacts in assembly and binning metrics compared to raw reads, and that differences in placement of MAGs in species trees increased with decreasing completeness and contamination thresholds. More aggressive trimming beyond those used by JGI were found to reduce MAG counts. Conclusions Mild trimming and decontamination of metagenomics reads prior to assembly can change an investigator’s answer to the questions, “Who is there and what are they doing? However, mild trimming and decontamination of metagenomic reads with high quality scores is recommended for those who elect to do so.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

Impact of BBDuk metagenomic read trimming and decontamination

Background Investigators using metagenomic sequencing to study their microbiomes are often provided data that has been trimmed and decontaminated or do it themselves without knowing the effect these procedures can have on their downstream analyses. Here we evaluated the impact that JGI trimming and decontamination procedures had on assembly and binning metrics, placement of metagenome assembled genomes into species trees, and functional profiles of metagenome-assembled genomes (MAGs) extracted from twenty three complex rhizosphere metagenomes. We also investigated how more aggressive trimming impacts these binning metrics. Results We found that JGI trimmed and decontamination of input reads had some significant impacts in assembly and binning metrics compared to raw reads, and that differences in placement of MAGs in species trees increased with decreasing completeness and contamination thresholds. More aggressive trimming beyond those used by JGI were found to reduce MAG counts. Conclusions Mild trimming and decontamination of metagenomics reads prior to assembly can change an investigator’s answer to the questions, “Who is there and what are they doing? However, mild trimming and decontamination of metagenomic reads with high quality scores is recommended for those who elect to do so.

59 BASIC BIOLOGICAL SCIENCES↗

NASA NTRS2015

Improving Climate Projections Using "Intelligent" Ensembles

Recent changes in the climate system have led to growing concern, especially in communities which are highly vulnerable to resource shortages and weather extremes. There is an urgent need for better climate information to develop solutions and strategies for adapting to a changing climate. Climate models provide excellent tools for studying the current state of climate and making future projections. However, these models are subject to biases created by structural uncertainties. Performance metrics-or the systematic determination of model biases-succinctly quantify aspects of climate model behavior. Efforts to standardize climate model experiments and collect simulation data-such as the Coupled Model Intercomparison Project (CMIP)-provide the means to directly compare and assess model performance. Performance metrics have been used to show that some models reproduce present-day climate better than others. Simulation data from multiple models are often used to add value to projections by creating a consensus projection from the model ensemble, in which each model is given an equal weight. It has been shown that the ensemble mean generally outperforms any single model. It is possible to use unequal weights to produce ensemble means, in which models are weighted based on performance (called "intelligent" ensembles). Can performance metrics be used to improve climate projections? Previous work introduced a framework for comparing the utility of model performance metrics, showing that the best metrics are related to the variance of top-of-atmosphere outgoing longwave radiation. These metrics improve present-day climate simulations of Earth's energy budget using the "intelligent" ensemble method. The current project identifies several approaches for testing whether performance metrics can be applied to future simulations to create "intelligent" ensemble-mean climate projections. It is shown that certain performance metrics test key climate processes in the models, and that these metrics can be used to evaluate model quality in both current and future climate states. This information will be used to produce new consensus projections and provide communities with improved climate projections for urgent decision-making.

Baker, Noel C.↗

NASA NTRS2015

Improving Climate Projections Using "Intelligent" Ensembles

Recent changes in the climate system have led to growing concern, especially in communities which are highly vulnerable to resource shortages and weather extremes. There is an urgent need for better climate information to develop solutions and strategies for adapting to a changing climate. Climate models provide excellent tools for studying the current state of climate and making future projections. However, these models are subject to biases created by structural uncertainties. Performance metrics-or the systematic determination of model biases-succinctly quantify aspects of climate model behavior. Efforts to standardize climate model experiments and collect simulation data-such as the Coupled Model Intercomparison Project (CMIP)-provide the means to directly compare and assess model performance. Performance metrics have been used to show that some models reproduce present-day climate better than others. Simulation data from multiple models are often used to add value to projections by creating a consensus projection from the model ensemble, in which each model is given an equal weight. It has been shown that the ensemble mean generally outperforms any single model. It is possible to use unequal weights to produce ensemble means, in which models are weighted based on performance (called "intelligent" ensembles). Can performance metrics be used to improve climate projections? Previous work introduced a framework for comparing the utility of model performance metrics, showing that the best metrics are related to the variance of top-of-atmosphere outgoing longwave radiation. These metrics improve present-day climate simulations of Earth's energy budget using the "intelligent" ensemble method. The current project identifies several approaches for testing whether performance metrics can be applied to future simulations to create "intelligent" ensemble-mean climate projections. It is shown that certain performance metrics test key climate processes in the models, and that these metrics can be used to evaluate model quality in both current and future climate states. This information will be used to produce new consensus projections and provide communities with improved climate projections for urgent decision-making.

Baker, Noel C.↗

NASA NTRS2016

Demonstration of Mars Crosslink Occultation Measurements for Future Small Spacecraft Constellations

Future planetary atmospheric profiling via radio occultations (RO) benefits significantly from increased received signal-to-noise ratio as well as geometrical coverage of links between two or more spacecraft in orbit around a target planet. These can be small spacecraft, possibly dedicated to quality radio-metrics with precision clock reference. Motivated by long-term research investigating the optimum SNR for the science performance, the optimum reference clock stability, the number and combination of wavelengths driving the number of transmitters and antennas, the optimum number and orbital spacecraft configuration, and design modifications to existing radio communication systems that would allow these new science objectives on future missions, three crosslink occultation experiments have been acquired between the Mars Odyssey and Mars Reconnaissance Orbiter spacecraft to probe the Martian atmosphere. While crosslink occultations between Earth orbiting satellites have long been used to profile the Earth’s atmosphere, this represents the first demonstration of crosslink occultation measurements at another planet. These measurements leverage the proximity link telecommunication payloads on each orbiter, which were designed to provide relay communication and navigation services to Mars landers and rovers. Analysis of the observed Doppler shift on each crosslink measurement reveals a clear signature of the Martian atmosphere, primarily the ionosphere. Inversion of the observed Doppler data yields vertical profiles of the Martian refractivity and electron density. The electron density profiles show the presence of two layers with peak densities and peak heights that are consistent with empirical models.

Asmar, Sami↗

DOE OSTI2018

An Efficient Pipeline for Abdomen Segmentation in CT Images

Computed tomography (CT) scans usually include some disadvantages due to the nature of the imaging procedure, and these handicaps prevent accurate abdomen segmentation. Discontinuous abdomen edges, bed section of CT, patient information, closeness between the edges of the abdomen and CT, poor contrast, and a narrow histogram can be regarded as the most important handicaps that occur in abdominal CT scans. Currently, one or more handicaps can arise and prevent technicians obtaining abdomen images through simple segmentation techniques. In other words, CT scans can include the bed section of CT, a patient’s diagnostic information, low-quality abdomen edges, low-level contrast, and narrow histogram, all in one scan. These phenomena constitute a challenge, and an efficient pipeline that is unaffected by handicaps is required. In addition, analysis such as segmentation, feature selection, and classification has meaning for a real-time diagnosis system in cases where the abdomen section is directly used with a specific size. A statistical pipeline is designed in this study that is unaffected by the handicaps mentioned above. Intensity-based approaches, morphological processes, and histogram-based procedures are utilized to design an efficient structure. Performance evaluation is realized in experiments on 58 CT images (16 training, 16 test, and 26 validation) that include the abdomen and one or more disadvantage(s). The first part of the data (16 training images) is used to detect the pipeline’s optimum parameters, while the second and third parts are utilized to evaluate and to confirm the segmentation performance. The segmentation results are presented as the means of six performance metrics. Thus, the proposed method achieves remarkable average rates for training/test/validation of 98.95/99.36/99.57% (jaccard), 99.47/99.67/99.79% (dice), 100/99.91/99.91% (sensitivity), 98.47/99.23/99.85% (specificity), 99.38/99.63/99.87% (classification accuracy), and 98.98/99.45/99.66% (precision). In summary, a statistical pipeline performing the task of abdomen segmentation is achieved that is not affected by the disadvantages, and the most detailed abdomen segmentation study is performed for the use before organ and tumor segmentation, feature extraction, and classification.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

NASA NTRS1994

Method for optimizing resource allocation in a government organization

The managers in Federal agencies are challenged to control the extensive activities in government and still provide high-quality products and services to the American taxpayers. Considering today's complex social and economic environment and the $3.8 billion daily cost of operating the Federal Government, it is evident that there is a need to develop decision-making tools for accurate resource allocation and total quality management. The goal of this thesis is to provide a methodical process that will aid managers in Federal Government to make budgetary decisions based on the cost of services, the agency's objectives, and the customers' perception of the agency's product. A general resource allocation procedure was developed in this study that can be applied to any government organization. A government organization, hereafter the 'organization,' is assumed to be a multidivision enterprise. This procedure was applied to a small organization for the proof of the concept. This organization is the Technical Services Directorate (TSD) at the NASA Lewis Research Center in Cleveland, Ohio. As part of the procedure, a nonlinear programming model was developed to account for the resources of the organization, the outputs produced by the organization, the decision-maker's views, and the customers' satisfaction with the organization. The information on the resources of the organization was acquired from current budget levels of the organization and the human resources assigned to the divisions. The outputs of the organization were defined and measured by identifying metrics that assess the outputs, the most challenging task in this study. The decision-maker's views are represented in the model as weights assigned to the various outputs and were quantified by using the analytic hierarchy process. The customer's opinions regarding the outputs of the organization were collected through questionnaires that were designed for each division individually. Following the philosophy of total quality management, information on customers' satisfaction is presented in the model as the quality of output. The model is a nonlinear one whose objective is to maximize customers' satisfaction such that the total cost of operation does not exceed the organization's budget. This model represents a structured approach or policy mechanism, at the agency level, to make capital investment decisions based on the priorities of the agency and the quality of outputs. This procedure applied to TSD resulted in a resources allocation scheme that was reasonable and acceptable to the decision-makers and, as expected, dependent on the assumptions and accuracy of the data used in the model.

Afarin, James↗

DOE OSTI2016

Investigation of similarity metrics for simulation based scaling analysis

Scaling analysis is a technique for characterizing degree of similarity between processes to occur in a real prototypic system and those to occur in its reduced scale (experimental) analog. In the context of nuclear reactor thermal-hydraulics and safety, scaling analysis is instrumental in assessing the applicability of data obtained from a scaled test facility for calibration of models and validation of simulation codes. Quantification of scale-related distortions is the key element in scaling analysis. In this work, we examine an approach for scaling analysis using computer simulation codes. Similarity analysis using information-theoretic approach is performed by comparing performance of experiment and application to variation of system conditions. Three different information theoretic measures namely mutual information, Kullback-Leibler divergence and overlap coefficient are selected for their potential use as metric for similarity analysis. We test the metrics by comparing them against physics-based scaling ratios, which are well known in fundamental thermal-hydraulic processes. In this exploratory analysis, a case of pressure drop of a steady-state flow through a channel is employed to test the proposed approach. It was found that in the absence of model uncertainties, mutual information serve as an effective measure of the scale effects present in the experiment. However, model uncertainty can significantly impact the quality of similarity analysis. Our analysis indicates that model uncertainties can have significant impact on the information theoretic metrics used for scaling analysis. Furthermore, if the model uncertainties exceeds beyond a critical value, the scale effect present in the experiment cannot be adequately evaluated. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2019

PNNL Report: New Study Validates Color-Rendition Specification Criteria Based on TM-30

Pacific Northwest National Laboratory (PNNL) has released the results of a study – published in Lighting Research & Technology – that validates color-preference specification criteria based on ANSI/IES TM-30-18. The result provides additional data to support TM-30 Annex E (in press), which will include tiers of recommended specification criteria for different goals (color preference, color vividness, and color fidelity), among other information related to color rendition specification. Standardized methods for evaluating the color rendition of light sources have progressed rapidly in the past five years, beginning with the publication of IES TM-30-15, followed by the adoption of CIE 224:2017, and culminating with the issuance of ANSI/IES TM-30-18 (which revised TM-30-15, made it a national standard, and unified the calculation framework of CIE 224 and IES TM-30). Importantly, none of these documents provided guidance on how to specify color rendition using the included measures; that is, no value or combination of values was identified that leads to any desired quality of the light. Dozens of experiments over the past 60 years have examined subjective evaluations of color quality—such as preference, naturalness, or vividness—although the focus has traditionally been on developing a single-number color quality metric rather than specification criteria. Further, many of these pre-LED and pre-TM-30 studies were limited in their ability to vary the lighting characteristics, preventing them from generating a complete and accurate picture of what influences color rendering quality in architectural spaces. A renewed emphasis in this area of research over the past four years has yielded a growing consensus about how measures of color fidelity (Rf), gamut area (Rg), and red chroma (Rcs,h1) can be used in combination to effectively describe several important perceptual attributes. In the new study, which was the third in a series conducted at PNNL, 90 lighting conditions with systematically varied color rendition and chromaticity were evaluated by 25 participants, who provided ratings for normalness, saturation, preference, and acceptability. The lighting conditions were specifically designed to test the bounds of previously proposed color preference criteria, while simultaneously verifying the applicability of the criteria across different values of CCT and Duv. In line with the preceding work, there were substantial differences in ratings based on color rendition, but minimal differences based on chromaticity. The tested criteria performed well, although after the experiment, some small adjustments in lower limits provided more effective divisions between tiers of performance. The adjusted criteria are: • Level 1: Rf = 78, Rg = 95, -1% = Rcs,h1 = 15% • Level 2: Rf = 74, Rg = 92, -7% = Rcs,h1 = 19% • Level 3: Rf = 70, Rg = 89, -12% = Rcs,h1 = 23%. These criteria were concurrently checked against the results from other, independent experiments, and will be included in ANSI/IES TM-30 Annex E as the specifications for color preference. The new criteria will be a valuable tool for improving lighting quality and balancing the tradeoff between color rendition and energy efficiency.

Royer, Michael P.↗

DOE OSTI2023

Explainable AI (XAI)-driven vibration sensing scheme for surface quality monitoring in a smart surface grinding process

Local Interpretable and Model-agnostic Explanation (LIME), an explainable artificial intelligence (XAI) approach is adapted to identify the globally important time-frequency bands for predicting average surface roughness (Ra) in a smart grinding process. The smart grinding setup consisted of a Supertech CNC precision surface grinding machine, instrumented with a Dytran piezoelectric accelerometer attached to the tailstock quill along the tangential direction (Y-axis). For every grinding pass, vibration signatures were captured, and the ground truth surface roughness values were recorded using a Mahr Marsurf M300C portable surface roughness profilometer. The roughness values ranged from 0.06 to 0.14 microns over the complete set of experiments. Time-frequency domain spectrogram frames were extracted for each of the vibration signals collected during the grinding process. Convolutional Neural Networks (CNNs) were modeled to predict the surface roughness based on these spectrogram frames and their image augmentations. The best CNN model was able to predict the roughness values with an overall R2-score of 0.95, training R2-score of 0.99, and testing R2-score of 0.81 with only 80 sets of vibration signals corresponding to 4 experiments with 20 trials each. Although the data size is not large enough to guarantee such performance metrics in real-world scenarios, one can extract statistically consistent explanations underlying the relationships these complex deep learning models capture. Further, the LIME methodology was implemented on the developed surface roughness CNN model to identify the important time-frequency bands (i.e., the superpixels of a spectrogram) influencing the predictions. Based on the identified important regions on the spectrogram frames, the corresponding frequency characteristics were determined that influence the surface roughness predictions. The important frequency range based on LIME results was approximately 11.7 to 19.1 kHz. The power of XAI was demonstrated by cutting down the sampling rate from 160 kHz to 30, 20, 10, and 5 kHz based on the important frequency range and considering Nyquist criteria. Separate CNN models were developed for these ranges by only extracting time-frequency contents below their corresponding Nyquist cut-offs. A proper data acquisition strategy is proposed by comparing the model performances to argue the selection of a sufficient sampling rate to capture the grinding process successfully and robustly.

42 ENGINEERING↗

DOE OSTI2023

Reprint of: Explainable AI (XAI)-driven vibration sensing scheme for surface quality monitoring in a smart surface grinding process

Local Interpretable and Model-agnostic Explanation (LIME), an explainable artificial intelligence (XAI) approach is adapted to identify the globally important time-frequency bands for predicting average surface roughness (Ra) in a smart grinding process. The smart grinding setup consisted of a Supertech CNC precision surface grinding machine, instrumented with a Dytran piezoelectric accelerometer attached to the tailstock quill along the tangential direction (Y-axis). For every grinding pass, vibration signatures were captured, and the ground truth surface roughness values were recorded using a Mahr Marsurf M300C portable surface roughness profilometer. The roughness values ranged from 0.06 to 0.14 microns over the complete set of experiments. Time-frequency domain spectrogram frames were extracted for each of the vibration signals collected during the grinding process. Convolutional Neural Networks (CNNs) were modeled to predict the surface roughness based on these spectrogram frames and their image augmentations. The best CNN model was able to predict the roughness values with an overall R2-score of 0.95, training R2-score of 0.99, and testing R2-score of 0.81 with only 80 sets of vibration signals corresponding to 4 experiments with 20 trials each. Although the data size is not large enough to guarantee such performance metrics in real-world scenarios, one can extract statistically consistent explanations underlying the relationships these complex deep learning models capture. Further, the LIME methodology was implemented on the developed surface roughness CNN model to identify the important time-frequency bands (i.e., the superpixels of a spectrogram) influencing the predictions. Based on the identified important regions on the spectrogram frames, the corresponding frequency characteristics were determined that influence the surface roughness predictions. The important frequency range based on LIME results was approximately 11.7 to 19.1 kHz. The power of XAI was demonstrated by cutting down the sampling rate from 160 kHz to 30, 20, 10, and 5 kHz based on the important frequency range and considering Nyquist criteria. Separate CNN models were developed for these ranges by only extracting time-frequency contents below their corresponding Nyquist cut-offs. A proper data acquisition strategy is proposed by comparing the model performances to argue the selection of a sufficient sampling rate to capture the grinding process successfully and robustly.

47 OTHER INSTRUMENTATION↗