Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “error propagation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Accuracy and precision of ESI-Orbitrap-IRMS observations of hours to tens of hours via reservoir injection

Here, orbitrap isotope ratio mass spectrometry (Orbitrap-IRMS) has recently been applied to high-precision, natural-abundance isotope ratio measurements of a diverse range of compounds, including amino acids, oxyanions, fatty acids, and metals. These measurements can characterize many isotope ratios simultaneously at high (≈1.0‰) precision. In a successful experiment, observed precision will track the shot-noise limit and be limited by experimental time. Some isotope ratios, for example those involving 17 O in organic compounds or multiply-substituted (‘clumped’) isotopologues, require experimental times of hours to tens of hours to achieve desired precision, while current sample introduction techniques focus on observations on the order of seconds to tens of minutes. In this study, we characterize Orbitrap-IRMS performance for three long duration measurements (individual acquisitions ≥1 h and as long as 24 h) using an automated reservoir injection system coupled to a Q Exactive HF Orbitrap with an electrospray ionization (ESI) source. First, we characterize long-term intra-measurement stability through a 24-h long measurement of acetone. We report the following isotope ratios and precisions (as acquisition errors, errors on the observed ratio within this measurement (σ AE )): 13 C/ 12 C (σ AE = 0.07‰), 17 O/ 16 O (σ AE = 1.1‰), 18 O/ 16 O (σ AE = 0.3‰), and 13 C 13 C/ 12 C (σ AE = 0.65‰). The σ AE of each tracks the shot noise limit throughout and is limited by the challenging conditions (high resolution and low numbers of ions per scan) required for 17 O/ 16 O measurement in the presence of 13 C via Orbitrap. Second, we characterize inter-measurement stability via a sequence of seven 75-min analyses of perchlorate. We observe the following ratios and acquisition errors: 37 Cl/ 35 Cl (σ AE = 0.09‰); 17 O/ 16 O (σ AE = 1.6‰); 18 O/ 16 O (σ AE = 0.7‰), 37 Cl 17 O/ 35 Cl 16 O (σ AE = 2.7‰), and 37 Cl 18 O/ 35 Cl 16 O (σ AE = 1.2‰). However, we find that inter-measurement drift between acquisitions limits our accuracy and precision for standardized measurements (i.e., error on reported δ values) to ≈1‰ for the 37 Cl/ 35 Cl measurement. Hence, the benefits of low σ AE may not be fully realized. Third, we demonstrate accuracy via sample/standard comparisons of a methionine sample with 13 C enrichment of ≈20‰ relative to a known standard. Using a sequence of seven 60-min analyses, we recover the following isotope ratios and standardized precisions (i.e., error on reported δ values, denoted propagated acquisition errors, σ PAE ): 33 S/ 32 S (σ PAE = 1.0‰), 34 S/ 32 S (σ PAE = 0.7‰), 15 N/ 14 N (σ PAE = 2.1‰), 2 H/ 1 H (σ PAE = 3.2‰), 13 C/ 12 C (σ PAE = 0.4‰), 18 O/ 16 O (σ PAE = 1.6‰), & 13 C 13 C/ 12 C (σ PAE = 2.8‰) with confirmation of accurate results for the known 13 C/ 12 C and 13 C 13 C/ 12 C enrichments. Together, our results demonstrate the viability of Orbitrap-IRMS for long duration measurements of diverse sample types via an automated reservoir injection system. Inter-measurement stability remains a challenge; we expect our methods to be most applicable to extended measurements of hard-to-observe properties, such as 17 O in organics and clumped isotopologues.

58 GEOSCIENCES↗
DOE OSTI2023

Recipes for when physics fails: recovering robust learning of physics informed neural networks

Abstract Physics-informed neural networks (PINNs) have been shown to be effective in solving partial differential equations by capturing the physics induced constraints as a part of the training loss function. This paper shows that a PINN can be sensitive to errors in training data and overfit itself in dynamically propagating these errors over the domain of the solution of the PDE. It also shows how physical regularizations based on continuity criteria and conservation laws fail to address this issue and rather introduce problems of their own causing the deep network to converge to a physics-obeying local minimum instead of the global minimum. We introduce Gaussian process (GP) based smoothing that recovers the performance of a PINN and promises a robust architecture against noise/errors in measurements. Additionally, we illustrate an inexpensive method of quantifying the evolution of uncertainty based on the variance estimation of GPs on boundary data. Robust PINN performance is also shown to be achievable by choice of sparse sets of inducing points based on sparsely induced GPs. We demonstrate the performance of our proposed methods and compare the results from existing benchmark models in literature for time-dependent Schrödinger and Burgers’ equations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Evaluating Diesel/Biofuel Blends Using Artificial Neural Networks and Linear/Nonlinear Equations

Abstract The use of biomass-derived additives in diesel fuel mixtures has the potential to increase the fuel’s efficiency, decrease the formation of particulate matter during its combustion, and retain the fuel’s behavior in cold weather. To this end, identifying compounds that enable these behaviors is paramount. The present work utilizes a series of linear and non-linear equations in series with artificial neural networks to predict the cetane number, yield sooting index, kinematic viscosity, cloud point, and lower heating value of multi-component blends. Property values of pure components are predicted using artificial neural networks trained with existing experimental data, and these predictions and their expected errors are propagated through linear and non-linear equations to obtain property predictions for multi-component blends. Individual component property prediction errors, defined by blind prediction median absolute error, are 4.91 units, 7.84 units, 0.06 cSt, 4.00 °C, and 0.55 MJ/kg for cetane number, yield sooting index, kinematic viscosity, cloud point, and lower heating value respectively. On average, property predictions for blends are shown to be accurate to within 6% of the blends’ experimental values. Further, a multitude of compounds expected to be produced from catalytically upgrading products of fast pyrolysis are evaluated with respect to their behavior in diesel fuel blends.

09 BIOMASS FUELS↗
DOE OSTI2020

SpotSDC: Revealing the Silent Data Corruption Propagation in High-Performance Computing Systems

We report the trend of rapid technology scaling is expected to make the hardware of high-performance computing (HPC) systems more susceptible to computational errors due to random bit flips. Some bit flips may cause a program to crash or have a minimal effect on the output, but others may lead to silent data corruption (SDC), i.e., undetected yet significant output errors. Classical fault injection analysis methods employ uniform sampling of random bit flips during program execution to derive a statistical resiliency profile. However, summarizing such fault injection result with sufficient detail is difficult, and understanding the behavior of the fault-corrupted program is still a challenge. In this article, we introduce SpotSDC, a visualization system to facilitate the analysis of a program's resilience to SDC. SpotSDC provides multiple perspectives at various levels of detail of the impact on the output relative to where in the source code the flipped bit occurs, which bit is flipped, and when during the execution it happens. SpotSDC also enables users to study the code protection and provide new insights to understand the behavior of a fault-injected program. Based on lessons learned, we demonstrate how what we found can improve the fault injection campaign method.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Comparing Model Representations of Physiological Limits on Transpiration at a Semi-arid Ponderosa Pine Site

Mechanistic representations of biogeochemical processes in ecosystem models are rapidly advancing, requiring advancements in model evaluation approaches. Here we quantify multiple aspects of model functional performance to evaluate improved process representations in ecosystem models. We compare semi-empirical stomatal models with hydraulic constraints against more mechanistic representations of stomatal and hydraulic functioning at a semi-arid pine site using a suite of metrics and analytical tools. We find that models generally perform similarly under unstressed conditions, but performance diverges under atmospheric and soil drought. The more empirical models better capture synergistic information flows between soil water potential and vapor pressure deficit to transpiration, while the more mechanistic models are overly deterministic. Although models can be parameterized to yield similar functional performance, alternate parameterizations could not overcome structural model constraints that underestimate the unique information contained in soil water potential about transpiration. Additionally, both multilayer canopy and big-leaf models were unable to capture the magnitude of canopy temperature divergence from air temperature, and we demonstrate that errors in leaf temperature can propagate to considerable error in simulated transpiration. This study demonstrates the value of merging underutilized observational data streams with emerging analytical tools to characterize ecosystem function and discriminate among model process representations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Atomic masses with machine learning for the astrophysical r process

The astrophysical r process plays a vital role in the production of heavy elements. Modeling of the r process is sensitive to masses and further requires knowledge of masses beyond current experimental reach. Therefore, simulations of the r process offer a unique test bed for predicting mass extrapolations. We take a Machine-Learning (ML) approach to model the masses across the entire chart of nuclides. For the first time, we simulate r-process nucleosynthesis with a physics-based ML mass model. We compare simulated abundances to solar data in order to evaluate the model's performance far from stability. The resulting r-process abundances up to thorium and uranium qualitatively match those of the observed solar system abundance pattern, with the characteristic peaks well positioned. We propagate the mass uncertainties obtained from the ML model to r-process abundance yields to estimate an uncertainty band associated with our approach. The size of the uncertainty band is approximately one order of magnitude which aligns with the uncertainty reported using alternative techniques.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Autonomous Inversion of In Situ Deformation Measurement Data for Injection-Induced Stress Change

Geologic carbon storage (GCS) is likely to play a key part of the global effort to dramatically reduce CO2 emissions and perhaps even reduce atmospheric CO2 concentrations through carbon negative operations. A critical part of effort to commercialize and widely deploy this technology is developing the capability to rapidly assimilate real-time monitoring data into a form that will enable site operators to make decisions to manage the safe and efficient operations. Two of the risks associate with GCS are the risk of inducing fractures in the sealing formations that can create leakage pathways and the risk of inducing earthquakes of sufficient magnitude to cause public concern, property damage, or safety risks. To properly manage these risks the site operator needs to know the initial state of stress, the change in stress induced by injection, and the relationship between operational parameters such as injection rate and pressure and the change in stress. Current methods of estimating the change in stress require choosing the type of constitutive model and the model parameters based on core, log, and geophysical data during the characterization phase, with little feedback from operational observations to validate or refine these choices. These characterization methods interrogate the geologic formations using length scales, loading rates or magnitudes that are quite different from those encountered by the actual storage system. It is shown that errors in the assumed constitutive response, even when informed by laboratory tests on core samples, are likely to be common, large, and underestimate the magnitude of stress change caused by injection. Recent advances in borehole-based strain instruments and borehole and surface-based tilt and displacement instruments have now enabled monitoring of the deformation of the storage system throughout its operational lifespan. This data can enable validation and refinement of the knowledge of the geomechanical properties and state of the system, but brings with it a challenge to transform the raw data into actionable knowledge. We demonstrate a method that uses automatic differentiation and a finite-element based geomechanical model perform a gradient-based deterministic inversion of geomechanical monitoring data. This approach allows autonomous integration of the instrument data without the need for time consuming manual interpretation and selection of updated model parameters. Furthermore, only isotropic linear elasticity is considered in this paper, the approach presented is very flexible as to what type of geomechanical constitutive response can be used. The approach is easily adaptable to nonlinear physics-based constitutive models to account for common rock behaviors such as creep and plasticity. The approach also enables training of machine learning-based constitutive models by allowing back propagation of errors through the finite element calculations. This enables strongly enforcing known physics, such as conservation of momentum and continuity, while allowing data-driven models to learn the truly unknown physics such as the constitutive or petrophysical responses.

Burghardt, Jeffrey A.↗
DOE OSTI2022

Position-specific carbon isotope analysis of serine by gas chromatography/Orbitrap mass spectrometry, and an application to plant metabolism

Position-specific 13 C/ 12 C ratios within amino acids remain largely unexplored in environmental samples due to methodological limitations. We hypothesized that natural-abundance isotope patterns in serine may serve as a proxy for plant metabolic fluxes including photorespiration. Here we describe an Orbitrap method optimized for the position-specific carbon isotope analysis of serine to test our hypothesis and discuss the generalizability of this method to other amino acids. Position-specific carbon isotope ratios of serine were measured using a Thermo Scientific™ Q Exactive™ GC Orbitrap™. Amino acids were hydrolyzed from Arabidopsis biomass, purified from potential matrix interferences, and derivatized alongside standards. Derivatized serine (N,O-bis(trifluoroacetyl)methyl ester) was isolated using gas chromatography, trapped in a reservoir, and purged into the electron ionization source over tens of minutes, producing fragment ions containing different combinations of atoms from the serine-derivative molecule. The 13 C/ 12 C ratios of fragments with monoisotopic masses of 110.0217, 138.0166, and 165.0037 Da were monitored in the mass analyzer and used to calculate position-specific δ 13 C values relative to a working standard. This methodology constrains position-specific δ 13 C values for nanomole amounts of serine isolated from chemically complex mixtures. The δ 13 C values of fragment ions of serine were characterized with ≤1‰ precisions, leading to propagated standard errors of 0.7–5‰ for each carbon position. Position-specific δ 13 C values differed by up to ca 28 ± 5‰ between serine molecules hydrolyzed from plants grown under contrasting pCO 2 , selected to promote different fluxes through photosynthesis and photorespiration. The method was validated using pure serine standards characterized offline. Here this study presents the first Orbitrap-based measurements of natural-abundance, position-specific carbon isotope variation in an amino acid isolated from a biological matrix. We present a method for the precise characterization of isotope ratios in serine and propose applications probing metabolism in plants. We discuss the potential for extending these approaches to other amino acids, paving the way for novel applications.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Direct sensitivity analysis on the parameterization of crystal plasticity models

Various methods for calibrating crystal plasticity finite element (CPFE) models lead to non-unique input parameter values, which subsequently introduce uncertainty in the predicted mechanical response. Sensitivity analysis (SA) conducted on crystal plasticity models is used to identify how variability in these parameters contribute to output uncertainty. Traditional SA on CPFE parameters uses simplified surrogate models to save computational time. However, the accuracy of the surrogate models depends on the quantity of training data used, and any modeling error can propagate into the SA results, potentially affecting their reliability. In this work, the elementary effects test (EET) method, a global SA technique using direct CPFE simulations was employed, and the results obtained were compared with the First Order Second Moment (FOSM) method. ExaConstit, an open-source GPU-enabled CPFE code, was used to perform the simulations and direct SA. The EET method was accurately able to capture the non-linear effects of all the input parameters on the output and is a valuable approach for reliably attributing parameter sensitivities in CPFE models. Based on the results, efficient strategies to perform future parameter calibration and SA are discussed. Additionally, the SA trends observed in different single crystal orientations closely mirrored the activity of the slip systems.

Elementary Effects Test↗
DOE OSTI2020

Towards real-time monitoring: data assimilated time-lapse full waveform inversion for seismic velocity and uncertainty estimation

SUMMARY Rapid development of time-lapse seismic monitoring instrumentations has made it possible to collect dense time-lapse data for tomographically retrieving time-lapse (even continuous) images of subsurface changes. While traditional time-lapse full waveform inversion (TLFWI) algorithms are designed for sparse time-lapse surveys, they lack of effective temporal constraint on time-lapse data, and, more importantly, lack of the uncertainty estimation of the TLFWI results that is critical for further interpretation. Here, we propose a new data assimilation TLFWI method, using hierarchical matrix powered extended Kalman filter (HiEKF) to quantify the image uncertainty. Compared to existing Kalman filter algorithms, HiEKF allows to store and update a data-sparse representation of the cross-covariance matrices and propagate model errors without expensive operations involving covariance matrices. Hence, HiEKF is computationally efficient and applicable to 3-D TLFWI problems. Then, we reformulate TLFWI in the framework of HiEKF (termed hereafter as TLFWI-HiEKF) to predict time-lapse images of subsurface spatiotemporal velocity changes and simultaneously quantify the uncertainty of the inverted velocity changes over time. We demonstrate the validity and applicability of TLFWI–HiEKF with two realistic CO2 monitoring models derived from Frio-II and Cranfield CO2 injection sites, respectively. In both 2-D and 3-D examples, the inverted high-resolution time-lapse velocity results clearly reveal a continuous velocity reduction due to the injection of CO2. Moreover, the accuracy of the model is increasing over time by assimilating more time-lapse data while the standard deviation is decreasing over lapsed time. We expect TLFWI-HiEKF to be equipped with real-time seismic monitoring systems for continuously imaging the distribution of subsurface gas and fluids in the future large-scale CO2 sequestration experiments and reservoir management.

58 GEOSCIENCES↗
DOE OSTI2026

Efficient Simulation of Logical Magic State Preparation Protocols

Developing space- and time-efficient logical magic state preparation (MSP) protocols will likely be an essential step toward building a large-scale fault-tolerant quantum computer. Motivated by this need, we introduce a scalable method for simulating logical MSP protocols under the standard circuit-level noise model. When applied to protocols based on code-switching, magic state cultivation, and magic state distillation, our method yields a complexity polynomial in (i) the number of qubits and (ii) the nonstabilizerness, e.g., stabilizer rank or Pauli rank, of the target encoded magic state. The efficiency of our simulation method is rooted in a curious fact: every circuit-level Pauli error in these protocols propagates to a Clifford error at the end. This property is satisfied by a large family of protocols, including those that repeatedly measure a transversal Clifford that squares to a Pauli. We provide a proof-of-principle numerical simulation that prepares a magic state using such logical Clifford measurements. Our work enables practical simulation of logical MSP protocols without resorting to approximations or resource-intensive state-vector simulations.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2024

Minimization of Measurement Uncertainty in Optical Frequency Domain Reflectometry

Optical frequency domain reflectometry (OFDR) is a technique for interrogating optical fiber sensors to generate relative, quasi-distributed measurements. Although Optical frequency domain reflectometry (OFDR) is increasingly being adopted for aerospace, energy production, and structural monitoring applications, the quantification of uncertainty for OFDR measurements has not been developed beyond sparse empirical relationships. To address this knowledge gap, an uncertainty metric for OFDR measurements was developed. This uncertainty metric was applied to weight the edges between OFDR measurements on directed correlation graphs and analyzed to minimize the cumulative uncertainty. In conclusion, this work is the first to propose an uncertainty metric for OFDR and provides a generalized mathematical framework for optimizing OFDR hardware selection, optical fiber sensor selection, and postprocessing strategy.

42 ENGINEERING↗
DOE OSTI2022

Comparison of the Baseline USL Calculation Methods for Loosely-Coupled and Novel Neutronic Systems [Slides]

Current work includes reconstructing 187-group ENDF B/VII.1 covariance matrix for comparison study to 44-group ENDF B/VII.1 matrix to determine how covariance matrix structures affect the USL calculations and investigating the exact domination mechanism of region-wise sensitivities in a loosely-coupled system. Future work will involve seeking to understand how bias distributions change with reactivity and how to calculate propagate distribution error into USL calculations, as well as how cross section perturbation studies can be extended to other types of calculations, such as shielding calculations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

SSTDR and FDR Detection of Un-Energized and Energized Cable Anomalies Including Thermal Degradation Using Machine Learning

Historically, cables are initially qualified for nuclear power plant use for 40 years. As plants extend their operating license to 60 and 80 years, continued use of these cables must shift to a performance-based approach since it is cost prohibitive to completely replace cables that are likely still capable of performing their design function. A variety of cable tests are available and are commonly applied during outages when the cables can be taken out of service. Frequency domain reflectometry (FDR) is one of these test methods that is being more broadly accepted and used because it not only detects anomalies along the cable with a low-voltage signal that does not stress the cable insulation, but the technique also locates the anomalies. This supports follow-up local inspection and local repair or partial replacement of a damaged cable segment. Currently, FDR testing is only applied to cables that are taken out of service since the test instrument would be damaged by operational voltages. A related technology that has found some acceptance in the aircraft and rail industry is spread spectrum time domain reflectometry (SSTDR). This technology has been implemented with a custom commercial instrument by LiveWire Innovation that is designed to operate on live cables up to 1000 volts and with a bandwidth of 48 MHz. Initial evaluation by the Pacific Northwest National Laboratory (PNNL) of the Live Wire system indicated that a broader bandwidth (BW) SSTDR may be better for many kinds of flaws. This led PNNL to develop an SSTDR laboratory instrument suitable for tests up to 500 MHz bandwidth. Testing on energized cables is also desirable for online monitoring systems so an inductive clamshell coupler was developed that allows energized cables to be tested up to at least 5 kV and likely higher voltage levels. Dielectric spectroscopy and tan delta testing plus various laboratory destructive tests were included in this data acquisition campaign directed to feed a machine learning (ML) study. With these kinds of developments, online energized cable tests may be possible with industrial adoption of such hardware advances but it will be completely impractical to have highly skilled data analysts continually examine these complex signals for indications of damage or compromised conditions. If online testing is to be implemented in new test hardware, it must be accompanied by software that can interpret the signals and alert plant operators of changing or degraded conditions. The thermally aged, shielded cable investigated here was separately treated for ML analysis. Visual analysis of electrical data showed generally increasing peaks where the cable entered and exited the oven. These peaks were not exactly aligned with expected locations, but these differences were attributed to velocity of propagation calibration errors. Only supervised ML was applied to the thermally aged data as this data was only available shortly before the committed publication date of this report. The supervised ML was structured to divide the 0 to 70-day responses as ‘normal’ from 0 to 35 days or ‘anomalous’ from 36 to 70 days, based on cable tensile elongation at break (EAB) insulation characterization. Using 80% of the data for training and 20% for testing, the supervised ML predicted normal versus anomalous was 70% accurate. Important conclusions include: • Accuracy to predict the presence of cable damage is improved from the 2023 effort by more training data. Weighted accuracies for comparisons among the instruments ranged from 67 to 89 % for unsupervised ML and 71 to 99% for supervised ML. • Based on the synthetic data tests, the unsupervised models are more generalizable to unseen anomalies. The Multi-Layer Perceptron classifier (MLP) model reported as high as 99.7% accuracy on the test data, but this dropped to 58.3% when tested on the synthetic data. In contrast, the unsupervised Pointwise model only achieved 89.7% accuracy on the experimental data but reported 78.3% accuracy on the synthetic data. • The best anomaly indicators are higher frequency (400 MHz BW) FDR data. Other tests may be interesting but for this study, this was the best predicter.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

On The Jacobian of the ECEF J2 Gravitation Model

An Earth-centered, Earth-fixed (ECEF) inertial navigation system must compute the Jacobian of its employed gravitation model with respect to position while time-propagating the error covariance of the system. One commonly used gravitation model is the ‘J2 model’ which is a second-order truncation of the Earth’s spherical harmonic gravitation model. The J2 model is popular because it can quickly and efficiently be evaluated, and the truncation error is small: The ‘J3 term’ --- the third term in the spherical harmonic expansion --- is approximately 1000 times smaller than the J2 term.

42 ENGINEERING↗
DOE OSTI2022

Understanding Uncertainty in Microstructure Evolution and Constitutive Properties in Additive Process Modeling

Coupled process–microstructure–property modeling, and understanding the sources of uncertainty and their propagation toward error in part property prediction, are key steps toward full utilization of additive manufacturing (AM) for predictable quality part development. The OpenFOAM model for process conditions, the ExaCA model for as-solidified grain structure, and the ExaConstit model for constitutive mechanical properties are used as part of the ExaAM modeling framework to examine a few of the various sources of uncertainty in the modeling workflow. In addition to “random” uncertainty (due to random number generation in the orientations and locations of grains present), the heterogeneous nucleation density N 0 and the mean substrate grain spacing S 0 are varied to examine their impact of grain area development as a function of build height in the simulated microstructure. While mean grain area after 1 mm of build is found to be sensitive to N 0 and S 0 , particularly at small N 0 and large S 0 (despite some convergence toward similar values), the resulting grain shapes and overall textures develop in a reasonably similar manner. As a result of these similar textures, ExaConstit simulation using ExaCA representative volume elements (RVEs) from various permutations of N 0 , S 0 , and location within the build resulted in similar yield stress, stress–strain curve shape, and stress triaxiality distributions. It is concluded that for this particular material and scan pattern, 15 layers is sufficient for ExaCA texture and ExaConstit predicted properties to become relatively independent of additional layer simulation, provided that reasonable estimates for N 0 and S 0 are used. However, additional layers of ExaCA will need to be run to obtain mean grain areas independent of build height and baseplate structure.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.

58 GEOSCIENCES↗