Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model validation and calibration”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Cooperative Transmission Expansion Planning Experiment Data and Results

GO WEST is an open-source power grid modeling framework for U.S. Western Interconnection, which allows users to tailor the model depending on their research study and science questions. It covers 28 balancing authorities (BA) and 12 states in U.S. Western Interconnection. GO WEST allows users to select different number of nodes and come up with a simplified network by utilizing 10,000 nodal topology of U.S. Western Interconnection created by Texas A&M University. Users can try and select different number of nodes, mathematical formulations (linear programming vs. mixed-integer linear programming), transmission line limit scaling factors, and hurdle rate scaling factors. GO WEST offers a unit commitment and economic dispatch (UC/ED) module to simulate grid operations on an hourly scale. In this sense, users can calibrate and validate their model versions by comparing model outputs to historical datasets. TEP is an open-source transmission capacity expansion model, built on GO WEST framework. It utilizes linear programming to optimize transmission capacity addition investment on existing lines within GO WEST framework. In this sense, TEP model only increases the thermal capacity of existing transmission lines and does not add new lines to the system, which leaves the topology preserved. TEP minimizes the total cost of the system which comprises the operational cost of satisfying electricity demand (i.e., generation cost), cost of loss of load (i.e., unserved energy), cost of power flow, and cost of new transmission capacity additions (i.e., investment cost). In order to use TEP model, users need to create scenarios with GO WEST framework. In this analysis, outputs from several models are used to create future inputs to GO WEST and TEP models, including GCAM-USA, TELL, CERF and reV. This dataset includes experiment inputs and outputs from three different transmission expansion scenarios (cooperative, intermediate, and individual) for 2019 and 2059. For 2019, a base scenario to illustrate the default (i.e., historical) power grid operations is also included. This study utilizes rcp45hotter_ssp3 scenario from a previous version of GCAM-USA simulations. Sources of the shapefiles in supplementary data are HIFLD Open and U.S. Energy Atlas. Please see the README file for a detailed description of the main and supplementary data.

Capacity Expansion Model↗
DOE OSTI2023

IM3 GO WEST Parameter Search Dataset

GO WEST is an open-source power grid modeling framework for U.S. Western Interconnection, which allows users to tailor the model depending on their research study and science questions. It is developed to address weather and water dynamics, and associated vulnerabilities in this bulk power system. It covers 28 balancing authorities (BA) and 12 states in U.S. Western Interconnection. GO WEST allows users to select different number of nodes and come up with a simplified network by utilizing 10,000 nodal topology of U.S. Western Interconnection created by Texas A&M University. Users can try and select different number of nodes, mathematical formulations (linear programming vs. mixed-integer linear programming), transmission line limit scaling factors, and hurdle rate scaling factors. GO WEST offers a unit commitment and economic dispatch (UC/ED) module to simulate grid operations on an hourly scale. In this sense, users can calibrate and validate their model versions by comparing model outputs to historical datasets. Therefore, GO WEST can help researchers to strike a balance between model fidelity (i.e. accuracy) and computational complexity (i.e. runtime). This dataset includes model inputs and outputs from 600 model versions for each 2019, 2020, and 2021. The folder naming convention is as follows: Exp{Number of Nodes}_{Mathematical Formulation}_{Transmission Line Limit Scaling Factor in MW}_{Hurdle Rate Scaling Factor in %}_{Year}. Linear programming is designated with "simple" label whereas mixed-integer linear programming is designated with "coal" label. For example, "Exp100_simple_1000_50_2019" folder contains inputs and outputs from 2019 model version with 100 nodes, linear programming, +1000 MW transmission line limit scaling factor, and +50% hurdle rate scaling factor. GO WEST GitHub repository hosts all raw datasets, processing scripts, and model scripts. Please refer to the README file for a detailed description of the included files.

Economics↗
DOE OSTI2021

Optical calibration of the SNO+ detector in the water phase with deployed sources

SNO+ is a large-scale liquid scintillator experiment with the primary goal of searching for neutrinoless double beta decay, and is located approximately 2 km underground in SNOLAB, Sudbury, Canada. The detector acquired data for two years as a pure water Cherenkov detector, starting in May 2017. During this period, the optical properties of the detector were measured in situ using a deployed light diffusing sphere, with the goal of improving the detector model and the energy response systematic uncertainties. The measured parameters included the water attenuation coefficients, effective attenuation coefficients for the acrylic vessel, and the angular response of the photomultiplier tubes and their surrounding light concentrators, all across different wavelengths. Finally, the calibrated detector model was validated using a deployed tagged gamma source, which showed a 0.6% variation in energy scale across the primary target volume.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

Validation Metrics for Fixed Effects and Mixed-Effects Calibration

The modern scientific process often involves the development of a predictive computational model. To improve its accuracy, a computational model can be calibrated to a set of experimental data. A variety of validation metrics can be used to quantify this process. Some of these metrics have direct physical interpretations and a history of use, while others, especially those for probabilistic data, are more difficult to interpret. In this work, a variety of validation metrics are used to quantify the accuracy of different calibration methods. Frequentist and Bayesian perspectives are used with both fixed effects and mixed-effects statistical models. Through a quantitative comparison of the resulting distributions, the most accurate calibration method can be selected. Two examples are included which compare the results of various validation metrics for different calibration methods. It is quantitatively shown that, in the presence of significant laboratory biases, a fixed effects calibration is significantly less accurate than a mixed-effects calibration. This is because the mixed-effects statistical model better characterizes the underlying parameter distributions than the fixed effects model. The results suggest that validation metrics can be used to select the most accurate calibration model for a particular empirical model with corresponding experimental data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Development, calibration, and validation of a novel gray-box energy model for residential split air conditioners

Energy models for vapor compression refrigeration in residential air conditioners have been developed through white-box, gray-box, and black-box methods in decades. However, existing white-box and gray-box models require complicated equations with detailed geometries while black-box models require substantial experimental data. Further, this paper aims to develop and validate a simple gray-box steady-state energy model without the need for detailed geometries, which can accurately predict the cooling capacity and electrical power input based on outdoor and indoor air conditions, and supply airflow rates. First three state variables, including the evaporation and condensation temperatures, and refrigerant mass flowrate, are applied to develop the energy model and are solved by three physical equations, including the energy conservations at the evaporator and condenser, and the refrigerant volume-mass flow correlation of the compressor. Secondly, seven performance property functions related to three state variables and three physical equations are identified and calibrated by simple temperature and power measurements. Finally, field experiments are conducted on a residential air conditioner to calibrate these performance property functions and validate the developed model. The validated results reveal the model can accurately predict the cooling capacity and electrical power input, with the normalized root mean square errors of 2.3% and 0.87% respectively.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Alkali-silica reaction expansion model for confined concrete with stress-dependency and casting direction anisotropy

Alkali-silica reaction (ASR) is a deleterious chemical reaction between alkali hydroxyl ions and types of silica found in some aggregates of concrete. Owners and regulators of nuclear power plants aim to ensure the safety of the concrete structures with optimal maintenance strategies. A new model was developed for predicting the expansion of concrete structures affected by alkali-silica reaction. The model includes a novel combination of existing models as an alkali-silica reaction advancement model, a casting direction anisotropic expansion model, a stress-dependent anisotropic expansion model, and a material property evolution model dependent on the degree of ASR expansion. The model parameters were calibrated based on existing literature data and data generated by previous efforts of this study. The calibrated model was then validated with the experiments carried out in previous efforts of this study. The model was shown to accurately predict the ASR-expansion of large-scale reinforced concrete specimens with confinement.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Defects and anomalies in powder bed fusion metal additive manufacturing

Metal additive manufacturing is a disruptive technology that is revolutionizing the manufacturing industry. Despite its unrivaled capability for directly fabricating metal parts with complex geometries, the wide realization of the technology is currently limited by microstructural defects and anomalies, which could significantly degrade the structural integrity and service performance of the product. Accurate detection, characterization, and prediction of these defects and anomalies have an important and immediate impact in manufacturing fully-dense and defect-free builds. As such, this review seeks to elucidate common defects/anomalies and their formation mechanisms in powder bed fusion additive manufacturing processes. They could arise from raw materials, processing conditions, and post-processing. While defects/anomalies in laser welding have been studied extensively, their formation and evolution remain unclear. Additionally, the existence of powder in powder bed fusion techniques may generate new types of defects, e.g., porosity transferring from powder to builds. Practical strategies to mitigate defects are also addressed through fundamental understanding of their formation. Such explorations enable the validation and calibration of models and ease the process qualification without costly trial-and-error experimentation.

42 ENGINEERING↗
DOE OSTI2021

Plasticity and fracture behavior of Inconel 625 manufactured by laser powder bed fusion: Comparison between as-built and stress relieved conditions

In this work, the influence of stress relief on the plasticity and fracture behavior of Inconel 625 fabricated through laser powder bed fusion additive manufacturing (AM) was investigated. The as-built versus stress relieved microstructures were compared, showing similar grain structures but the presence of ~10 vol % δ phase in the stress relieved condition, and no δ phase in the as-built condition. Mechanical tests under plane strain tension were performed on the stress relieved samples, and an anisotropic plasticity model was calibrated and validated using finite element simulations. Uniaxial and notched tension tests were performed on both as-built and stress relieved samples to probe the effect of stress relief on stress state- and direction-dependent fracture behavior. It was found that on average, the fracture strain of the stress relieved samples along the build direction was 30% higher than that along the perpendicular build direction in the stress state range studied, and the stress relief heat treatment resulted in a 45% decrease in fracture strain. The fracture strain in stress relieved samples was more strongly dependent on stress state than in as-built samples.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Development of new baseline models for U.S. medium office buildings based on commercial buildings energy consumption survey data

Building energy estimation for the building sector under various scenarios are needed for building energy regulation and policy making. This often starts with representative baselines (either empirical baseline or modeled baseline). Commercial Buildings Energy Consumption Survey (CBECS) data is a widely used empirical baseline for U.S. commercial buildings, but none of the existing baseline model are developed to represent the CBECS data. This paper aims to develop new baseline models for the U.S. medium office buildings, which can produce modeled baselines consistent with the CBECS data. Here, we introduced the methodology to create baseline models and the criteria to evaluate the performance of baseline models. The methodology consists of three phases: (1) identification of model inputs, (2) model calibration, and (3) model validation with uncertainty analysis. The evaluation index is the coefficient of variation of the root-mean-square deviation (CV(RMSD)) of site energy use intensities (EUIs) between the modeled baseline and empirical baseline. Then 30 new baseline models for two vintages (pre- and post-1980) and 15 climate zones were created. The evaluation shows that the CV(RMSD) is lower than 0.05 for the modeled baselines produced by the new baseline models. As a comparison, the CV(RMSD) is higher than 0.1 for the existing modeled baselines generated by DOE Commercial Reference Building Models. Further analysis shows that the new baseline models are able to capture the uncertainties of the representative features of existing buildings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Investigating the Effects of Individual Neutron-Induced Defects in Bipolar Junction Transistors

Here, this study investigates neutron-induced displacement damage in Bipolar Junction Transistors (BJTs) using TCAD models informed by Deep-Level-Transient-Spectroscopy (DLTS) data. These models are calibrated and validated against experimental measurements performed at various neutron fluences. Both npn and pnp transistor configurations are studied to analyze the effects of individual traps on carrier recombination and base leakage currents. In npn transistors, deep traps (0.42 eV from the conduction band) dominate at low voltages, while shallow traps (0.17 eV from the conduction band) become prominent at higher voltages. Conversely, pnp transistors have base leakage current predominantly due to deep-level traps. The study observes a notable trend in trap density versus fluence, characterized by a linear relationship on a log-log scale. These insights into defect evolution under radiation conditions are crucial for optimizing semiconductor device reliability and performance in radiation-prone environments.

42 ENGINEERING↗
DOE OSTI2025

ASME Code change proposal to implement new universal high temperature constitutive models for Section III, Division 5

This report completes work on a universal high temperature constitutive model suitable for use with the ASME Boiler & Pressure Vessel Code Section III, Division 5 rules for the design by inelastic analysis of Class A nuclear reactor components. The goals of this work are to provide a simple model form that adequately captures the high temperature response of materials and can be applied to any future Code material. Additionally, the report describes an automated process for calibrating a model against test data. The idea is to simplify the effort required to generate a constitutive model for an arbitrary material, provided test data is available. This will accelerate the process of qualifying new Code materials in the future. In addition, the report provides calibrated models and detailed validation comparisons to test data for five currently-qualified or soon-to-be qualified materials: 316H, Grade 91, Alloy 800H, Alloy 617, and Alloy 709. The report surveys the available data for the remaining two ASME Class Materials --- 2.25Cr-1Mo and 304H --- concluding that there is enough data data to generate a model for 2.25Cr-1Mo steel provided some additional sources of non-public data can be included in the test database, but that a dedicated cyclic test campaign would be needed for 304H. Supplemental material includes the full text of an ASME Code change proposal to incorporate the models for the four currently-qualified Class A material, detailed validation comparisons to test data for the five material models, and input files for reference implementations of the constitutive models in the NEML and NEML2 modeling frameworks.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2022

Scalable deep learning for watershed model calibration

Watershed models such as the Soil and Water Assessment Tool (SWAT) consist of high-dimensional physical and empirical parameters. These parameters often need to be estimated/calibrated through inverse modeling to produce reliable predictions on hydrological fluxes and states. Existing parameter estimation methods can be time consuming, inefficient, and computationally expensive for high-dimensional problems. In this paper, we present an accurate and robust method to calibrate the SWAT model (i.e., 20 parameters) using scalable deep learning (DL). We developed inverse models based on convolutional neural networks (CNN) to assimilate observed streamflow data and estimate the SWAT model parameters. Scalable hyperparameter tuning is performed using high-performance computing resources to identify the top 50 optimal neural network architectures. We used ensemble SWAT simulations to train, validate, and test the CNN models. We estimated the parameters of the SWAT model using observed streamflow data and assessed the impact of measurement errors on SWAT model calibration. We tested and validated the proposed scalable DL methodology on the American River Watershed, located in the Pacific Northwest-based Yakima River basin. Our results show that the CNN-based calibration is better than two popular parameter estimation methods (i.e., the generalized likelihood uncertainty estimation [GLUE] and the dynamically dimensioned search [DDS], which is a global optimization algorithm). For the set of parameters that are sensitive to the observations, our proposed method yields narrower ranges than the GLUE method but broader ranges than values produced using the DDS method within the sampling range even under high relative observational errors. The SWAT model calibration performance using the CNNs, GLUE, and DDS methods are compared using R 2 and a set of efficiency metrics, including Nash-Sutcliffe, logarithmic Nash-Sutcliffe, Kling-Gupta, modified Kling-Gupta, and non-parametric Kling-Gupta scores, computed on the observed and simulated watershed responses. The best CNN-based calibrated set has scores of 0.71, 0.75, 0.85, 0.85, 0.86, and 0.91. The best DDS-based calibrated set has scores of 0.62, 0.69, 0.8, 0.77, 0.79, and 0.82. The best GLUE-based calibrated set has scores of 0.56, 0.58, 0.71, 0.7, 0.71, and 0.8. The scores above show that the CNN-based calibration leads to more accurate low and high streamflow predictions than the GLUE and DDS sets. Our research demonstrates that the proposed method has high potential to improve our current practice in calibrating large-scale integrated hydrologic models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

ResStock Measure Documentation: Residential Two-Stage Geothermal Heat Pump (4.0 COP, 20.5 EER) With Envelope Improvements and Advanced Air Sealing

The goal of this work is to develop energy efficiency, demand flexibility, and other retrofit end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) measures. "Measures" refers to retrofits that can be applied to buildings during modeling. An "end-use savings shape" is the difference in energy consumption between a baseline building and a building with an energy efficiency, demand flexibility, or other retrofit measure applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each time step. ResStock is a highly granular, physics-based, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the residential building stock across the United States. The baseline model intends to represent the U.S. residential building stock as it existed in 2018. Technical documentation for the inputs and assumptions in the baseline building stock model is available in Reyna et al. (2025). Calibration and validation of the baseline model results are available in the final technical report of the End-Use Load Profiles project (Wilson, et al. 2022). This document focuses on a single end-use savings shape measure: Residential Two-Stage Geothermal Heat Pump (GHP) (4.0 COP, 20.5 EER) With Envelope Improvements. This measure combines a two-stage GHP with envelope improvements as a single package. As this package is a combination of two other measures, this document focused on documenting the results associated with this combination of technologies, with individual measure documents for two-stage GHPs and envelope improvements providing the information on the details of these measures. When the two technologies are combined, envelope improvements can modestly reduce energy consumption by a further 10%-15%, but also reduce the required size of the ground heat exchanger and heat pump by approximately 33% on average across all sites. The cost of installing envelope improvements in these homes is likely to be more than paid for by the reduction in equipment and drilling costs in these buildings for the majority of the stock.

15 GEOTHERMAL ENERGY↗
DOE OSTI2026

ResStock Measure Documentation: Residential Single-Stage Geothermal Heat Pump (3.8 COP, 18.6 EER)

The goal of this work is to develop energy efficiency, demand flexibility, and other retrofit end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) measures. "Measures" refers to retrofits that can be applied to buildings during modeling. An "end-use savings shape" is the difference in energy consumption between a baseline building and a building with an energy efficiency, demand flexibility, or other retrofit measure applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each time step. ResStock is a highly granular, physics-based, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the residential building stock across the United States. The baseline model intends to represent the U.S. residential building stock as it existed in 2018. Technical documentation for the inputs and assumptions in the baseline building stock model is available in Reyna et al. (2025). Calibration and validation of the baseline model results are available in the final technical report of the End-Use Load Profiles project (Wilson et al. 2022). This documentation focuses on a single end-use savings shape measure: Residential Single-Stage Geothermal Heat Pump (GHP).?Single-stage GHPs are able to reduce energy consumption by 31% for the entire stock. Additional results provided below detail how savings changes for sections of the housing stock with different base heating fuel and in different climate zones, as well as the savings potential by state for both heating and cooling. Utility bills and electric panel impacts are also shown and discussed.

15 GEOTHERMAL ENERGY↗
DOE OSTI2026

ResStock Measure Documentation: Residential Variable-Speed Geothermal Heat Pump (4.4 COP, 30.9 EER)

The goal of this work is to develop energy efficiency, demand flexibility, and other retrofit end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) measures. "Measures" refers to retrofits that can be applied to buildings during modeling. An "end-use savings shape" is the difference in energy consumption between a baseline building and a building with an energy efficiency, demand flexibility, or other retrofit measure applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each time step. ResStock (TM) is a highly granular, physics-based, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the residential building stock across the United States. The baseline model intends to represent the U.S. residential building stock as it existed in 2018. Technical documentation for the inputs and assumptions in the baseline building stock model is available in Reyna et al. (2025). Calibration and validation of the baseline model results are available in the final technical report of the End-Use Load Profiles project (Wilson et al. 2022). This documentation focuses on a single end-use savings shape measure: Residential Variable-Speed Geothermal Heat Pump (GHP). This document provides the relevant new modeling information for variable-speed systems not previously covered in either the single-stage or two-stage documents. Variable-speed GHPs represent the most efficient option available for this technology: They provide the most savings, with up to 46% for the applicable portion of the housing stock, compared to 31% for less efficient single-stage GHPs. Additional results shown here detail how the savings change for sections of the housing stock with different base heating fuels and in different climate zones, and they show the savings potential by state for both heating and cooling. Utility bills and electric panel impacts are also shown and discussed.

15 GEOTHERMAL ENERGY↗
DOE OSTI2026

ResStock Measure Documentation: Residential Two-Stage Geothermal Heat Pump (4.0 COP, 20.5 EER)

The goal of this work is to develop energy efficiency, demand flexibility, and other retrofit end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) measures. "Measures" refers to retrofits that can be applied to buildings during modeling. An "end-use savings shape" is the difference in energy consumption between a baseline building and a building with an energy efficiency, demand flexibility, or other retrofit measure applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each time step. ResStock is a highly granular, physics-based, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the residential building stock across the United States. The baseline model intends to represent the U.S. residential building stock as it existed in 2018. Technical documentation for the inputs and assumptions in the baseline building stock model is available in Reyna et al. (2025). Calibration and validation of the baseline model results are available in the final technical report of the End-Use Load Profiles project (Wilson et al. 2022). This document focuses on a single end-use savings shape measure: Residential Two-Stage Geothermal Heat Pump (4.0 COP, 20.5 EER). This document builds on details established in the single-stage document (Maguire et al. 2025) to detail differences in the approach to modeling this higher efficiency, but more commonly deployed, type of geothermal heat pump. Specific EnergyPlus objects and product specific curves used are highlighted along with showing the results of this measure compared to the baseline and single-speed geothermal heat pumps. Two-speed geothermal heat pumps are able to save even more energy and on utility bills than single-speed products, albeit at the expense of a higher first cost.

15 GEOTHERMAL ENERGY↗
DOE OSTI2023

Toward Improved Regional Hydrological Model Performance Using State-Of-The-Science Data-Informed Soil Parameters

Accurate soil moisture and streamflow data are an aspirational need of many hydrologically relevant fields. Model simulated soil moisture and streamflow hold promise but models require validation prior to application. Calibration methods are commonly used to improve model fidelity but misrepresentation of the true dynamics remains a challenge. In this study, we leverage soil parameter estimates from the Soil Survey Geographic (SSURGO) database and the probability mapping of SSURGO (POLARIS) to improve the representation of hydrologic processes in the Weather Research and Forecasting Hydrological modeling system (WRF-Hydro) over a central California domain. Our results show WRF-Hydro soil moisture exhibits increased correlation coefficients ( r ), reduced biases, and increased Kling-Gupta Efficiencies (KGEs) across seven in situ soil moisture observing stations after updating the model's soil parameters according to POLARIS. Compared to four well-established soil moisture data sets including Soil Moisture Active Passive data and three Phase 2 North American Land Data Assimilation System land surface models, our POLARIS-adjusted WRF-Hydro simulations produce the highest mean KGE (0.69) across the seven stations. More importantly, WRF-Hydro streamflow fidelity also increases, especially in the case where the model domain is set up with SSURGO-informed total soil thickness. The magnitude and timing of peak flow events are better captured, r increases across nine United States Geological Survey stream gages, and the mean KGE across seven of the nine gages increases from 0.12 to 0.66. Our pre-calibration parameter estimate approach, which is transferable to other spatially distributed hydrological models, can substantially improve a model's performance, helping reduce calibration efforts and computational costs.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Too Good to Be True? Evaluation of Colonoscopy Sensitivity Assumptions Used in Policy Models

Background: Models can help guide colorectal cancer screening policy. Although models are carefully calibrated and validated, there is less scrutiny of assumptions about test performance. Methods: We examined the validity of the CRC-SPIN model and colonoscopy sensitivity assumptions. Standard sensitivity assumptions, consistent with published decision analyses, assume sensitivity equal to 0.75 for diminutive adenomas (<6 mm), 0.85 for small adenomas (6–10 mm), 0.95 for large adenomas (≥10 mm), and 0.95 for preclinical cancer. We also selected adenoma sensitivity that resulted in more accurate predictions. Targets were drawn from the Wheat Bran Fiber study. In the study, we examined how well the model predicted outcomes measured over a three-year follow-up period, including the number of adenomas detected, the size of the largest adenoma detected, and incident colorectal cancer. Results: Using standard sensitivity assumptions, the model predicted adenoma prevalence that was too low (42.5% versus 48.9% observed, with 95% confidence interval 45.3%–50.7%) and detection of too few large adenomas (5.1% versus 14.% observed, with 95% confidence interval 11.8%–17.4%). Predictions were close to targets when we set sensitivities to 0.20 for diminutive adenomas, 0.60 for small adenomas, 0.80 for 10- to 20-mm adenomas, and 0.98 for adenomas 20 mm and larger. Conclusions: Colonoscopy may be less accurate than currently assumed, especially for diminutive adenomas. Alternatively, the CRC-SPIN model may not accurately simulate onset and progression of adenomas in higher-risk populations. Impact: Misspecification of either colonoscopy sensitivity or disease progression in high-risk populations may affect the predicted effectiveness of colorectal cancer screening. When possible, decision analyses used to inform policy should address these uncertainties.

60 APPLIED LIFE SCIENCES↗