Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “conformal prediction”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Extending Conformal Prediction to Hidden Markov Models with Exact Validity via de Finetti’s Theorem for Markov Chains

Conformal prediction is a widely used method to quantify uncertainty in settings where the data is independent and identically distributed (IID), or more generally, exchangeable. Conformal prediction takes in a pre-trained classifier and a calibration dataset as inputs, and returns a function which maps feature vectors to subsets of classes. The output of the returned function for a new feature vector is guaranteed to contain the true class with a pre-specified confidence. Despite its success and usefulness in IID settings, extending conformal prediction to non-exchangeable (e.g., Markovian) data in a manner that provably preserves all desirable theoretical properties has largely remained an open problem. As a solution, we extend conformal prediction to the setting of a Hidden Markov Model (HMM) with unknown parameters. The key idea behind the proposed method is to partition the non-exchangeable Markovian data from the HMM into exchangeable blocks by exploiting the de Finetti’s Theorem for Markov Chains discovered by Diaconis and Freedman (1980). The permutations of the exchangeable blocks are then viewed as randomizations of the observed Markovian data from the HMM. The proposed method provably retains all desirable theoretical guarantees offered by the classical conformal prediction framework and is general enough to be useful in many sequential prediction problems.

Nettasinghe, Don Buddhika Wijayantha↗
DOE OSTI2022

Robust and scalable uncertainty estimation with conformal prediction for machine-learned interatomic potentials

Abstract Uncertainty quantification (UQ) is important to machine learning (ML) force fields to assess the level of confidence during prediction, as ML models are not inherently physical and can therefore yield catastrophically incorrect predictions. Established a-posteriori UQ methods, including ensemble methods, the dropout method, the delta method, and various heuristic distance metrics, have limitations such as being computationally challenging for large models due to model re-training. In addition, the uncertainty estimates are often not rigorously calibrated. In this work, we propose combining the distribution-free UQ method, known as conformal prediction (CP), with the distances in the neural network’s latent space to estimate the uncertainty of energies predicted by neural network force fields. We evaluate this method (CP+latent) along with other UQ methods on two essential aspects, calibration, and sharpness, and find this method to be both calibrated and sharp under the assumption of independent and identically-distributed (i.i.d.) data. We show that the method is relatively insensitive to hyperparameters selected, and test the limitations of the method when the i.i.d. assumption is violated. Finally, we demonstrate that this method can be readily applied to trained neural network force fields with traditional and graph neural network architectures to obtain estimates of uncertainty with low computational costs on a training dataset of 1 million images to showcase its scalability and portability. Incorporating the CP method with latent distances offers a calibrated, sharp and efficient strategy to estimate the uncertainty of neural network force fields. In addition, the CP approach can also function as a promising strategy for calibrating uncertainty estimated by other approaches.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2024

Non-conformity Scores for High-Quality Uncertainty Quantification from Conformal Prediction

High-quality uncertainty quantification (UQ) is a critical component of enabling trust in deep learning (DL) models and is especially important if DL models are to be deployed in high-consequence applications. Conformal prediction (CP) methods represent an emerging nonparametric approach for producing UQ that is easily interpretable and, under weak assumptions, provides a guarantee regarding UQ quality. This report describes the research outputs of an Exploratory Express Laboratory Directed Research and Development (LDRD) project at Sandia National Laboratories. This project focused on how best to implement CP methods for DL models. This report introduces new methodology for obtaining high-quality UQ from DL models using CP methods, describes a novel system of assessing UQ quality, and provides experimental results that demonstrate the quality of the new methodology and utility of the UQ quality assessment system. Avenues for future research and discussion of potential impacts at Sandia and in the wider research community are also given.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Uncertainty quantification of the virial black hole mass with conformal prediction

Precise measurements of the black hole mass are essential to gain insight on the black hole and host galaxy co-evolution. A direct measure of the black hole mass is often restricted to nearest galaxies and instead, an indirect method using the single-epoch virial black hole mass estimation is used for objects at high redshifts. However, this method is subjected to biases and uncertainties as it is reliant on the scaling relation from a small sample of local active galactic nuclei. In this study, we propose the application of conformalized quantile regression (CQR) to quantify the uncertainties of the black hole predictions in a machine learning setting. We compare CQR with various prediction interval techniques and demonstrated that CQR can provide a more useful prediction interval indicator. In contrast to baseline approaches for prediction interval estimation, we show that the CQR method provides prediction intervals that adjust to the black hole mass and its related properties. That is it yields a tighter constraint on the prediction interval (hence more certain) for a larger black hole mass, and accordingly, bright and broad spectral line width source. Using a combination of neural network model and CQR framework, the recovered virial black hole mass predictions and uncertainties are comparable to those measured from the Sloan Digital Sky Survey. The code is publicly available.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

D–MOPH–25: diverse MOF–molecule pairs for Henry’s constants prediction

Computational methods like grand-canonical Monte Carlo simulations and machine learning (ML) have accelerated metal–organic frameworks (MOF) exploration but are typically limited to a narrow range of adsorbates due to data availability and force field constraints. In this study, we introduce a dataset of diverse MOF–molecule pairs for Henry’s constant prediction, D–MOPH–25, which systematically explores a diverse chemical space by combining 113 molecular adsorbates with over 5000 MOF structures through an active learning process. D–MOPH–25 constitutes the most diverse adsorbate dataset used in any ML study of molecular adsorption in MOFs to date. Our workflow builds a benchmark for predicting Henry’s constants at 300 K, leveraging conformal prediction for uncertainty quantification. Assessment through Shannon entropy and uniform manifold approximation and projection confirms the comprehensiveness of D–MOPH–25 while highlighting the importance of robust classification to filter out unphysical data points in regression tasks. Although future enhancements in model architecture and sampling criteria could improve predictive performance, our dataset already spans the target space using only 2.31% of total possibilities. This comprehensive dataset facilitates assessment of model generalizability across adsorbate species and can establish a foundation for high-throughput MOF screening and ML-driven separation processes.

active learning↗
DOE OSTI2025

An investigation on machine learning predictive accuracy improvement and uncertainty reduction using VAE-based data augmentation

The confluence of ultrafast computers with large memory, rapid progress in Machine Learning (ML) algorithms, and the availability of large datasets place multiple engineering fields at the threshold of dramatic progress. However, a unique challenge in nuclear engineering is data scarcity because experimentation on nuclear systems is usually more expensive and time-consuming than most other disciplines. One potential way to resolve the data scarcity issue is deep generative learning, which uses certain ML models to learn the underlying distribution of existing data and generate synthetic samples that resemble the real data. In this way, one can significantly expand the dataset to train more accurate predictive ML models. In this study, our objective is to evaluate the effectiveness of data augmentation using variational autoencoder (VAE)-based deep generative models. We investigated whether the data augmentation leads to improved accuracy in the predictions of a deep neural network (DNN) model trained using the augmented data. Additionally, the DNN prediction uncertainties are quantified using Bayesian Neural Networks (BNN) and conformal prediction (CP) to assess the impact on predictive uncertainty reduction. To test the proposed methodology, we used TRACE simulations of steady-state void fraction data based on the NUPEC Boiling Water Reactor Full-size Fine-mesh Bundle Test (BFBT) benchmark. Here, we found that augmenting the training dataset using VAEs has improved the DNN model’s predictive accuracy, improved the prediction confidence intervals, and reduced the prediction uncertainties.

Bayesian neural network↗
DOE OSTI2026

Conformalized-KANs: Uncertainty Quantification with Coverage Guarantees for Kolmogorov-Arnold Networks (KANs) in Scientific Machine Learning

This paper explores uncertainty quantification (UQ) methods in the context of Kolmogorov–Arnold Networks (KANs). We apply an ensemble approach to KANs to obtain a heuristic measure of UQ, enhancing interpretability and robustness in modeling complex functions. Building on this, we introduce Conformalized-KANs, which integrate conformal prediction, a distribution-free UQ technique, with KAN ensembles to generate calibrated prediction intervals with guaranteed coverage.} Extensive numerical experiments are conducted to evaluate the effectiveness of these methods, focusing particularly on the robustness and accuracy of the prediction intervals under various hyperparameter settings. We show that the conformal KAN predictions can be applied to recent extensions of KANs, including Finite Basis KANs (FBKANs) and multifideilty KANs (MFKANs). The results demonstrate the potential of our approaches to significantly improve the reliability and applicability of KANs in scientific machine learning.

• Artificial intelligence (AI) / machine learning ↗
DOE OSTI2025

Uncertainty Quantification for Data-Driven Machine Learning Models in Nuclear Engineering Applications: Where We Are and What Do We Need?

Machine learning (ML) has been leveraged to tackle a diverse range of tasks in almost all branches of nuclear engineering. Many of the successes in ML applications can be attributed to the recent performance breakthroughs in deep learning, the growing availability of computational power, data, and easy-to-use ML libraries. However, these empirical successes have often outpaced our formal understanding of the ML algorithms. An important but under-rated area is uncertainty quantification (UQ) of ML. ML-based models are subject to approximation uncertainty when they are used to make predictions, due to sources including but not limited to, data noise, data coverage, extrapolation, imperfect model architecture and the stochastic training process. The goal of this paper is to clearly explain and illustrate the importance of UQ of ML. We will elucidate the differences in the basic concepts of UQ of physics-based models and data-driven ML models. Various sources of uncertainties in physical modeling and data-driven modeling will be discussed, demonstrated, and compared. We will also present and demonstrate a few techniques to quantify the ML prediction uncertainties, including Monte Carlo dropout, deep ensemble, Bayesian neural networks, Gaussian Processes and conformal prediction. Lastly, we will discuss the need for building a verification, validation and UQ framework to establish ML credibility.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

A high-throughput workflow to analyze sequence-conformation relationships and explore hydrophobic patterning in disordered peptoids

Understanding how a macromolecule’s primary sequence governs its conformational landscape is crucial for elucidating its function, yet these design principles are still emerging for macromolecules with intrinsic disorder. Herein, we introduce a high-throughput workflow that implements a practical colorimetric conformational assay, introduces a semi-automated sequencing protocol using matrix-assisted laser desorption/ionization and tandem mass spectrometry (MALDI-MS/MS), and develops a generalizable sequence-structure algorithm. Using a model system of 20mer peptidomimetics containing polar glycine and hydrophobic N-butylglycine residues, we identified nine classifications of conformational disorder and isolated 122 unique sequences across varied compositions and conformations. Conformational distributions of three compositionally identical library sequences were corroborated through atomistic simulations and ion mobility spectrometry coupled with liquid chromatography. A data-driven strategy was developed using existing sequence variables and data-derived “motifs” to inform a machine-learning algorithm toward conformation prediction. Here, this multifaceted approach enhances our understanding of sequence-conformation relationships and offers a powerful tool for accelerating the discovery of materials with conformational control.

data-driven analysis↗
DOE OSTI2025

mphys-surrogate-model

This repository contains python scripts for building and studying reduced-order-modeling representations of droplet coalescence for eventual use in atmospheric models. The included data are generated from high-fidelity superdroplet methods and are utilized by machine learning pipelines to build data-driven models of droplet size distributions that evolve under coalescence. This repository further includes scripts to determine prediction (uncertainty) intervals on the data-driven model products based on conformal prediction.

Katona, JonasE [Lawrence Livermore National Labora↗
DOE OSTI2022

Machine Learning with Gradient-Based Optimization of Nuclear Waste Vitrification with Uncertainties and Constraints

Gekko is an optimization suite in Python that solves optimization problems involving mixed-integer, nonlinear, and differential equations. The purpose of this study is to integrate common Machine Learning (ML) algorithms such as Gaussian Process Regression (GPR), support vector regression (SVR), and artificial neural network (ANN) models into Gekko to solve data based optimization problems. Uncertainty quantification (UQ) is used alongside ML for better decision making. These methods include ensemble methods, model-specific methods, conformal predictions, and the delta method. An optimization problem involving nuclear waste vitrification is presented to demonstrate the benefit of ML in this field. ML models are compared against the current partial quadratic mixture (PQM) model in an optimization problem in Gekko. GPR with conformal uncertainty was chosen as the best substitute model as it had a lower mean squared error of 0.0025 compared to 0.018 and more confidently predicted a higher waste loading of 37.5 wt% compared to 34 wt%. The example problem shows that these tools can be used in similar industry settings where easier use and better performance is needed over classical approaches. Future works with these tools include expanding them with other regression models and UQ methods, and exploration into other optimization problems or dynamic control.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2021

Shotgun Immunoproteomic Approach for the Discovery of Linear B-Cell Epitopes in Biothreat Agents Francisella tularensis and Burkholderia pseudomallei

Peptide-based subunit vaccines are coming to the forefront of current vaccine approaches, with safety and cost-effective production among their top advantages. Peptide vaccine formulations consist of multiple synthetic linear epitopes that together trigger desired immune responses that can result in robust immune memory. The advantages of linear compared to conformational epitopes are their simple structure, ease of synthesis, and ability to stimulate immune responses by means that do not require complex 3D conformation. Prediction of linear epitopes through use of computational tools is fast and cost-effective, but typically of low accuracy, necessitating extensive experimentation to verify results. On the other hand, identification of linear epitopes through experimental screening has been an inefficient process that requires thorough characterization of previously identified full-length protein antigens, or laborious techniques involving genetic manipulation of organisms. In this study, we apply a newly developed generalizable screening method that enables efficient identification of B-cell epitopes in the proteomes of pathogenic bacteria. As a test case, we used this method to identify epitopes in the proteome of Francisella tularensis (Ft), a Select Agent with a well-characterized immunoproteome. Our screen identified many peptides that map to known antigens, including verified and predicted outer membrane proteins and extracellular proteins, validating the utility of this approach. We then used the method to identify seroreactive peptides in the less characterized immunoproteome of Select Agent Burkholderia pseudomallei (Bp). This screen revealed known Bp antigens as well as proteins that have not been previously identified as antigens. Although B-cell epitope prediction tools Bepipred 2.0 and iBCE-EL classified many of our seroreactive peptides as epitopes, they did not score them significantly higher than the non-reactive tryptic peptides in our study, nor did they assign higher scores to seroreactive peptides from known Ft or Bp antigens, highlighting the need for experimental data instead of relying on computational epitope predictions alone. The present workflow is easily adaptable to detecting peptide targets relevant to the immune systems of other mammalian species, including humans (depending upon the availability of convalescent sera from patients), and could aid in accelerating the discovery of B-cell epitopes and development of vaccines to counter emerging biological threats.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

Repetitive proteins that undergo large conformational changes evade structural prediction algorithms

Protein structure prediction algorithms, such as AlphaFold, have accelerated protein design and advanced the understanding of the relationship between amino acid sequence and protein structure. However, these algorithms are limited in their ability to predict the structures of conformationally dynamic, intrinsically disordered, and stimuli-responsive proteins. To evaluate sequence-to-structure predictions of such challenging proteins, we explored a class of conformationally dynamic, repeats-in-toxin (RTX) proteins. RTX proteins adopt intrinsically disordered conformations in the absence of calcium and undergo reversible folding into β-roll structures upon binding to calcium. RTX proteins are characterized by tandem repeats of the sequence GGXGXDXUX, in which X can be any amino acid and U is an aliphatic amino acid. We designed RTX sequence variants with global substitutions of nonconserved amino acids, tandem repeats of consensus sequences GGAGXDTLY, and tandem repeats of scrambled sequences GGAGXDTYL. AlphaFold2 and AlphaFold3 predicted that all of these RTX variants adopt β-roll structures, characteristic of wild-type RTX bound to calcium. However, modeling the predicted structures with molecular dynamics simulations and characterizing the protein variants with circular dichroism spectroscopy, small-angle x-ray scattering, and x-ray crystallography revealed that variants adopt diverse, sequence-dependent structures in the absence and presence of calcium. To better design proteins for applications in biotechnology and sustainability, it is critical to build predictive tools that consider intrinsically disordered protein states and validate these tools with multi-mode, multi-scale experimental data.

Chang, Marina P. [Stanford Univ., CA (United State↗
DOE OSTI2022

Adaptive Conformer Sampling for Property Prediction Using the Conductor-like Screening Model for Real Solvents

The valorization of lignocellulose-derived bioproducts requires effective separation from excessive water. Liquid–liquid extraction is a promising low-energy separation technology, but effective extraction requires solvent selection based on the thermodynamic properties of the bioproduct and solvent components. We propose a computational framework for predicting such properties by developing an adaptive conformer selection approach for use with COSMO-RS (conductor-like screening model for real solvents) calculations. In this framework, molecular dynamics simulations are used to generate many molecular structures (conformers) at representative temperatures in varying solvent environments. Conformers are then clustered based on structural metrics in a low-dimensional space and selected using a mixed-integer quadratic programming problem to iteratively insert a sampled conformer. At each iteration, we determine bioproduct properties using COSMO-RS. Here, we demonstrate the capability of the proposed framework on representative bioproducts to show convergence of the adaptive sampling toward experimentally measured properties with fewer calculations than required by random conformer sampling, enabling the improved screening of solvent systems for liquid-phase separation.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Functional protein mining with conformal guarantees

Molecular structure prediction and homology detection offer promising paths to discovering protein function and evolutionary relationships. However, current approaches lack statistical reliability assurances, limiting their practical utility for selecting proteins for further experimental and in-silico characterization. To address this challenge, we introduce a statistically principled approach to protein search leveraging principles from conformal prediction, offering a framework that ensures statistical guarantees with user-specified risk and provides calibrated probabilities (rather than raw ML scores) for any protein search model. Our method (1) lets users select many biologically-relevant loss metrics (i.e. false discovery rate) and assigns reliable functional probabilities for annotating genes of unknown function; (2) achieves state-of-the-art performance in enzyme classification without training new models; and (3) robustly and rapidly pre-filters proteins for computationally intensive structural alignment algorithms. Our framework enhances the reliability of protein homology detection and enables the discovery of uncharacterized proteins with likely desirable functional properties.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Molecular Cage Reports on Its Contents: Spectroscopic Signatures of Cryo-Cooled K + - and Ba 2+ -Benzocryptand Complexes

UV photofragment spectroscopy and IR–UV double resonance methods are used to determine the structure and spectroscopic responses of a three-dimensional [2.2.2]-benzocryptand cage to the incorporation of a single K + or Ba 2+ imbedded inside it (labeled as K + -BzCrypt, Ba 2+ -BzCrypt). We studied the isolated ion-cryptand complex under cryo-cooled conditions, brought into the gas phase by nano-electrospray ionization. Incorporation of a phenyl ring in place of the central ethyl group in one of the three N-CH 2 -CH 2 -O-CH 2 -CH 2 -O-CH 2 -CH 2 -N chains provides a UV chromophore whose S 0 –S 1 transition we probe. K + -BzCrypt and Ba 2+ -BzCrypt have their S 0 –S 1 origin transitions at 35,925 and 36,446 cm –1 , respectively, blue-shifted by 174 and 695 cm –1 from that of 1,2-dimethoxybenzene. These origins are used to excite a single conformation of each complex selectively and record their IR spectra using IR–UV dip spectroscopy. The alkyl CH stretch region (2800–3000 cm –1 ) is surprisingly sensitive to the presence and nature of the encapsulated ion. We carried out an exhaustive conformational search of cage conformations for K + -BzCrypt and Ba 2+ -BzCrypt, identifying two conformations (A and B) that lie below all others in energy. Here, we extend our local mode anharmonic model of the CH stretch region to these strongly bound ion-cage complexes to predict conformation-specific alkyl CH stretch spectra, obtaining quantitative agreement with experiment for conformer A, the gas-phase global minimum. The large electrostatic effect of the charge on the O- and N-lone pairs affects the local mode frequencies of the CH 2 groups adjacent to these atoms. The localized CH 2 scissors modes are pushed up in frequency by the adjacent O/N-atoms so that their overtones have little effect on the alkyl CH stretch region. However, the localized CH 2 wags are nearly degenerate and strongly coupled to one another, producing an array of delocalized wag normal modes, whose highest frequency members reach up above 1400 cm –1 . As such, their overtones mix significantly with the CH stretch modes, most notably involving the CH 2 symmetric stretch fundamentals of the central ethyl groups in the all-alkyl chains and the CH stretches adjacent to the N-atoms and antiperiplanar to the nitrogen lone pair.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗