SEARCH · Engineering Papers

Results for “deterministic method”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52

All sources NASA NTRS DOE OSTI

DOE OSTI2021

The dynamic evolution of swelling in nickel concentrated solid solution alloys through in situ property monitoring

Defects and microstructural features spanning the atomic level to the microscale play deterministic roles in the expressed properties of materials. Yet studies of material evolution in response to environmental stimuli most often correlate resulting performance with one dominant microstructural feature only. In this work, the dynamic evolution of swelling in a series of Ni-based concentrated solid solution alloys under high-temperature irradiation exposure is observed using continuous, in situ measurements of thermoelastic properties in bulk specimens. Unlike traditional evaluation techniques which account only for volumetric porosity identified using electron microscopy, direct property evaluation provides an integrated response across all defect length scales. In particular, the evolution in elastic properties during swelling is found to depend significantly on the entire size spectrum of defects, from the nano- to meso-scales, some of which are not resolvable in imaging. Observed changes in thermal transport properties depend sensitively on the partitioning of electronic and lattice thermal conductivity. This emerging class of in situ experiments, which directly measure integrated performance in relevant conditions, provides unique insight into material dynamics otherwise unavailable using traditional methods.

36 MATERIALS SCIENCE↗

DOE OSTI2023

ClimSim: A large multi-scale dataset for hybrid physics-ML climate emulation

Modern climate projections lack adequate spatial and temporal resolution due to computational constraints. A consequence is inaccurate and imprecise predictions of critical processes such as storms. Hybrid methods that combine physics with machine learning (ML) have introduced a new generation of higher fidelity climate simulators that can sidestep Moore’s Law by outsourcing compute-hungry, short, high-resolution simulations to ML emulators. However, this hybrid ML-physics simulation approach requires domain-specific treatment and has been inaccessible to MLexperts because of lack of training data and relevant, easy-to-use workflows. Wepresent ClimSim, the largest-ever dataset designed for hybrid ML-physics research. It comprises multi-scale climate simulations, developed by a consortium of climate scientists and ML researchers. It consists of 5.7 billion pairs of multivariate input and output vectors that isolate the influence of locally-nested, high-resolution, high-fidelity physics on a host climate simulator’s macro-scale physical state. The dataset is global in coverage, spans multiple years at high sampling frequency, and is designed such that resulting emulators are compatible with downstream coupling into operational climate simulators. We implement a range of deterministic and stochastic regression baselines to highlight the ML challenges and their scoring. The data (https://huggingface.co/datasets/LEAP/ClimSim_high-res2) and code(https://leap-stc.github.io/ClimSim)arereleasedopenlytosupport the development of hybrid ML-physics and high-fidelity climate simulations for the benefit of science and society.

artificial intelligence, machine learning↗

NASA NTRS2013

Improving Photometry and Stellar Signal Preservation with Pixel-Level Systematic Error Correction

The Kepler Mission has demonstrated that excellent stellar photometric performance can be achieved using apertures constructed from optimally selected CCD pixels. The clever methods used to correct for systematic errors, while very successful, still have some limitations in their ability to extract long-term trends in stellar flux. They also leave poorly correlated bias sources, such as drifting moiré pattern, uncorrected. We will illustrate several approaches where applying systematic error correction algorithms to the pixel time series, rather than the co-added raw flux time series, provide significant advantages. Examples include, spatially localized determination of time varying moiré pattern biases, greater sensitivity to radiation-induced pixel sensitivity drops (SPSDs), improved precision of co-trending basis vectors (CBV), and a means of distinguishing the stellar variability from co-trending terms even when they are correlated. For the last item, the approach enables physical interpretation of appropriately scaled coefficients derived in the fit of pixel time series to the CBV as linear combinations of various spatial derivatives of the pixel response function (PRF). We demonstrate that the residuals of a fit of soderived pixel coefficients to various PRF-related components can be deterministically interpreted in terms of physically meaningful quantities, such as the component of the stellar flux time series which is correlated with the CBV, as well as, relative pixel gain, proper motion and parallax. The approach also enables us to parameterize and assess the limiting factors in the uncertainties in these quantities.

Kolodzijczak, Jeffrey J.↗

DOE OSTI2025

Virtual to Physical: Reinforcement Learning to Optimize SNS Particle Accelerator Controls

Complex accelerators must have control systems that can handle dynamic nonlinear environments. This makes traditional control methods unsuitable as they can struggle to adapt to these uncertainties. This provides an ideal environment for reinforcement learning algorithms as they are adaptable and generalizable. We present a reinforcement learning pipeline that can effectively handle the dynamics of a complex accelerator. We test and prove our pipelines capabilities on multiple environments including the Spallation Neutron Source (SNS) and the Beam Test Facility (BTF) at Oakridge National Lab (ORNL). Due to the limited time available to train an online algorithm like reinforcement learning on a real accelerator, we utilize a virtual twin accelerator (VIRAC) developed by ORNL to pretrain the policy and show its ability to converge in the virtual environment. We then test the adaptability of the pretrained RL model by applying it on the real accelerator and comparing the results. Utilizing our Scientific Optimization and Controls Toolkit (SOCT) and open-source standards such as Gymnasium we create and solve for a MEBT orbit correction problem in the SNS and an emittance maximization problem in the BTF. We show how Twin Delayed Deep Deterministic Policy Gradient (TD3) can solve this optimization environment in the virtual accelerator and transfer this policy onto the real accelerator for inference and model retraining. We show how reinforcement learning can be utilized as a control system for complex accelerators and provide a model pipeline for how an implementation performs and can be adapted to new accelerator control problems.

Kasparian, Armen [Thomas Jefferson National Accele↗

DOE OSTI2024

Reduced-Order Modeling of Multigroup Neutron Cross Sections for High-Temperature Gas-cooled Reactors

Abstract – Deterministic neutronics calculations rely on multigroup neutron cross section libraries, which consist of databases of tabulated values, used to calculate the neutron cross sections through multivariate linear interpolation. However, interpolation of the multidimensional cross section data becomes memory inefficient and time consuming as the number of tabulations increases, significantly slowing down the neutronics calculation, especially in the case of microscopic cross section libraries where every isotope (on the order of hundreds) has its own set of specific reactions and cross sections. In order to address this challenge, this work constructs efficient and robust reduced-order models (ROMs) of the multi-group cross sections to support the Griffin simulation of high-temperature gas-cooled reactors (HTGRs). The first part of the study investigates the linearity of the multigroup cross section data across isotopes, reaction types, and energy groups on pre-generated datasets for the purpose of dimensionality reduction. Secondly, a down-selection of ROM techniques is presented on representative classical machine learning (ML) techniques, including variants of linear regression, kernel-based methods, tree-based algorithms, and artificial neural networks. The selection criteria jointly consider the memory efficiency, predictive accuracy, prediction speed, and scalability in comparison to the multidimensional interpolation. Among all the ML techniques, deep neural networks (DNNs) have proven to be the best selection with sufficient accuracy, high robustness, good memory efficiency, great scalability, and superior flexibility. DNNs have been trained for all isotopes in this work and systematic Griffin testing is ongoing to ensure the feasibility of this ROM technique for predicting cross section and reducing memory requirements without a significant sacrifice in computational performance.

42 - ENGINEERING↗

DOE OSTI2024

Advanced Cross Section Library Generation using Reduced Order Models

Deterministic neutronics calculations rely on multigroup neutron cross section libraries, which consist of databases of tabulated values, used to calculate the neutron cross sections through multivariate linear interpolation. However, interpolation of the multidimensional cross section data becomes memory inefficient and time consuming as the number of tabulations increases, significantly slowing down the neutronics calculation, especially in the case of microscopic cross section libraries where every isotope (on the order of hundreds) has its own set of specific reactions and cross sections. In order to address this challenge, this work constructs efficient and robust reduced-order models (ROMs) of the multi-group cross sections to support the Griffin simulation of high-temperature gas-cooled reactors (HTGRs). The first part of the study investigates the linearity of the multigroup cross section data across isotopes, reaction types, and energy groups on pre-generated datasets for the purpose of dimensionality reduction. Secondly, a down-selection of ROM techniques is presented on representative classical machine learning (ML) techniques, including variants of linear regression, kernel-based methods, tree-based algorithms, and artificial neural networks. The selection criteria jointly consider the memory efficiency, predictive accuracy, prediction speed, and scalability in comparison to the multidimensional interpolation. Among all the ML techniques, deep neural networks (DNNs) have proven to be the best selection with sufficient accuracy, high robustness, good memory efficiency, great scalability, and superior flexibility. DNNs have been trained for all isotopes in this work and systematic Griffin testing is ongoing to ensure the feasibility of this ROM technique for predicting cross section and reducing memory requirements without a significant sacrifice in computational performance.

42 - ENGINEERING↗

DOE OSTI2023

Preservation of kinetics parameters generated by Monte Carlo calculations in two-step deterministic calculations

The generation of accurate kinetic parameters such as mean generation time Λ and effective delayed neutron fraction β eff via Monte Carlo codes is established. Employing these in downstream deterministic codes warrants another step to ensure no additional error is introduced by the low-order transport operator when computing forward and adjoint fluxes for bilinear weighting of these parameters. Another complexity stems from applying superhomogenization (SPH) equivalence in non-fundamental mode approximations, where reference and low-order calculations rely on a 3D full core model. In these cases, SPH factors can optionally be computed for only part of the geometry while preserving reaction rates and K-effective, but the impact of such approximations on kinetics parameters has not been thoroughly studied. This paper aims at studying the preservation of bilinearly-weighted quantities in the Serpent–Griffin calculation procedure. Diffusion and transport evaluations of IPEN/MB-01, Godiva, and Flattop were carried out with the Griffin reactor physics code, testing available modeling options using Serpent-generated multigroup cross sections and equivalence data. Verifying Griffin against Serpent indicates sensitivities to multigroup energy grid selection and regional application of SPH equivalence, introducing significant errors; these were demonstrated to be reduced through the use of a transport method together with a finer energy grid.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2024

Reduce-Order Modeling of Multigroup Neutron Cross Sections for High-Temperature Gas-cooled Reactors

Deterministic neutronics calculations rely on multigroup neutron cross section libraries, which usually consists of a database of tabulated values, used to calculate the cross sections through multivariate linear interpolation. However, interpolation of the multidimensional cross section data becomes memory inefficient and time consuming as the number of tabulations increases, significantly slowing down the neutronics calculation, especially in the case of micro cross section libraries where every isotope (on the order of hundreds) has its own set of specific reactions and cross sections. To address this challenge, this work constructs efficient and robust reduced-order models (ROMs) of the multi-group cross sections to support the Griffin simulation of high-temperature gas-cooled reactors (HTGRs). The first part of the study investigates the linearity of the multi-group cross section data across isotopes, reaction types and energy groups on pre-generated datasets for the purpose of dimensionality reduction. Secondly, a down-selection of ROM techniques is presented on representative classical machine learning (ML) techniques, including variants of linear regression, kernel-based methods, tree-based algorithms, and artificial neural networks. The selection criteria jointly consider the memory efficiency, predictive accuracy, prediction speed, and scalability in comparison to the multidimensional interpolation. Among all the ML techniques, deep neural networks (DNNs) have proven to be the best selection with sufficient accuracy, high robustness, good memory efficiency, great scalability, and superior flexibility. DNNs for have been trained for all isotopes in this work and systematic Griffin testing is ongoing at this moment to ensure the feasibility of this ROM technique for cross section predictions.

42 - ENGINEERING↗

DOE OSTI2019

A Bi-Level Formulation And Solution Method For The Integration Of Process Design And Scheduling

Design and operation decisions of manufacturing plants can often be expressed within a hierarchical structure, where optimal decisions at lower operating level provide constraints for the decision raking at a higher decision level. In this work, we are addressing the case of a hierarchical design and scheduling optimization problem. The integration of design and scheduling decisions can play a big role in designing economically profitable plants and improving their operational performance. This problem can be expressed as a bi-level problem, where design related decisions occur at the upper level and operational scheduling decisions at the lower level. Since discrete decisions are involved in both optimization levels, the resulting formulation typically corresponds to bi-level mixed-integer programming problems (B-MIP). The solution of B-MIP problems is very challenging, and typically requires the use of global optimization techniques, with many algorithms not able to guarantee feasibility. To overcome the challenges in solving this class of problems we propose the use of a multi-parametric based algorithm for the solution of bi-level mixed-integer linear and quadratic programming problems, capable of providing the exact and global solution. The main idea of this approach is to treat the lower scheduling problem as a multi-parametric programming problem in which the design decisions are considered as parameters. Here, the resulting parametric solutions can then be substituted into the upper level design problem, which can be solved as a set of single-level deterministic programming problems. Through the developed formulation and algorithm we are able to supply the decision makers with the exact solution of the integrated design and scheduling problem.

42 ENGINEERING↗

DOE OSTI2024

A Data-Driven Methodology for Contextual Unit Commitment Using Regression Residuals

Day after day, system operators are faced with the challenge of taking unit commitment (UC) decisions under uncertain net load conditions. The standard operating procedure for taking UC decisions begins by leveraging auxiliary data on covariates (such as the day of the week or latest weather information) to generate a point prediction for net load, which is used in solving a deterministic UC problem. Such an approach, however, is known to deliver a notoriously poor out-of-sample (OOS) performance, as it completely disregards the stochastic nature of net load. While stochastic programming models explicitly represent uncertainty, they mostly do so using a generic set of scenarios that neglect covariate observations, squandering useful auxiliary data that could be harnessed to glean insights into uncertainty. In this article, we discuss a contextual stochastic optimization approach to UC, which effectively exploits covariate observations while explicitly assessing uncertainty so as to boost the OOS performance of UC decisions. The key thrust of our approach is to leverage regression models, along with their empirical residuals, to set up and solve sample average approximation problems. Not only do we prove that our approach satisfies the requisite conditions for asymptotic optimality and consistency laid out in (Kannan et al., 2022), but we also assess its performance on several case studies conducted using real-world data collected in California ISO and New York ISO grids. In conclusion, results show that the proposed approach can significantly improve OOS performance compared to alternative methods proposed in the literature under varying dataset sizes.

Yurdakul, Ogun↗

NASA NTRS2009

Reliability-Based Design Optimization of a Composite Airframe Component

A stochastic design optimization methodology (SDO) has been developed to design components of an airframe structure that can be made of metallic and composite materials. The design is obtained as a function of the risk level, or reliability, p. The design method treats uncertainties in load, strength, and material properties as distribution functions, which are defined with mean values and standard deviations. A design constraint or a failure mode is specified as a function of reliability p. Solution to stochastic optimization yields the weight of a structure as a function of reliability p. Optimum weight versus reliability p traced out an inverted-S-shaped graph. The center of the inverted-S graph corresponded to 50 percent (p = 0.5) probability of success. A heavy design with weight approaching infinity could be produced for a near-zero rate of failure that corresponds to unity for reliability p (or p = 1). Weight can be reduced to a small value for the most failure-prone design with a reliability that approaches zero (p = 0). Reliability can be changed for different components of an airframe structure. For example, the landing gear can be designed for a very high reliability, whereas it can be reduced to a small extent for a raked wingtip. The SDO capability is obtained by combining three codes: (1) The MSC/Nastran code was the deterministic analysis tool, (2) The fast probabilistic integrator, or the FPI module of the NESSUS software, was the probabilistic calculator, and (3) NASA Glenn Research Center s optimization testbed CometBoards became the optimizer. The SDO capability requires a finite element structural model, a material model, a load model, and a design model. The stochastic optimization concept is illustrated considering an academic example and a real-life raked wingtip structure of the Boeing 767-400 extended range airliner made of metallic and composite materials.

Patnaik, Surya N.↗

DOE OSTI2022

Hydrogen Plus Other Alternative Fuels Risk Assessment Models (HyRAM+) Version 4.1 Technical Reference Manual

The HyRAM+ software toolkit provides a basis for conducting quantitative risk assessment and consequence modeling for hydrogen, methane, and propane systems. HyRAM+ is designed to facilitate the use of state-of-the-art models to conduct robust, repeatable assessments of safety, hazards, and risk. HyRAM+ integrates deterministic and probabilistic models for quantifying accident scenarios, predicting physical effects, characterizing hazards (thermal effects from jet fires, overpressure effects from delayed ignition), and assessing impacts on people. HyRAM+ is developed at Sandia National Laboratories to support the development and revision of national and international codes and standards, and to provide developed models in a publicly-accessible toolkit usable by all stakeholders. This document provides a description of the methodology and models contained in HyRAM+ version 4.1. The two most significant changes for HyRAM+ version 4.1 from HyRAM+ version 4.0 are direct incorporation of unconfined overpressure into the QRA calculations and modification of the models for cryogenic liquid flow through an orifice. In QRA mode, the user no longer needs to input peak overpressure and impulse values that were calculated separately; rather, the unconfined overpressure is estimated for the given system inputs, leak size, and occupant location. The orifice flow model now solves for the maximum mass flux through the orifice at constant entropy while conserving energy, which does not require a direct speed of sound calculation. This does not affect the mass flow for all-gaseous releases; the method results in the same speed of sound for choked flow. However, this method does result in a higher (and more realistic) mass flow rate for a given leak size for liquid releases than was previously calculated.

08 HYDROGEN↗

DOE OSTI2022

Two-step neutronics calculations with Shift and Griffin for advanced reactor systems

This research develops the initial coupling of the Shift Monte Carlo (MC) code and the Griffin reactor physics code for reactor analysis of non–light-water reactor systems. The novelty of this work is twofold. It is the first application of Shift to produce the multigroup cross sections needed for Griffin as applied to a non–light-water reactor system; and, the first investigation and analysis of characteristics of the Empire microreactor benchmark that should be considered for steady state and transient reactor physics calculations. This application uses the previously developed two-step neutronics analysis workflow to demonstrate this initial coupling. Here, we outline the two-step neutronics analysis workflow in which the Shift MC code is used to generate the multigroup cross sections and fluxes needed by the Griffin deterministic solver. Details on how these multigroup cross sections are generated using MC tallies are given, as well as the practicalities and limitations of the two-step neutronics workflow. The Empire microreactor benchmark was used to investigate and validate this coupling. Results using this benchmark show good agreement between Griffin calculations using Serpent-generated cross sections and Shift-generated cross sections. Analysis of the characteristics of this Empire benchmark show larger eigenvalue differences between heterogeneous and pin–homogenized solutions compared to those of traditional light-water reactor (LWR) designs, thus requiring super homogenization factor corrections for accurate eigenvalue and power distribution predictions.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2018

Watts Bar I ex-core analyses using VERA - 25430

The integration and direct coupling of the Oak Ridge National Laboratory (ORNL) Shift Monte Carlo code within the Consortium for Advanced Simulation of Light Water Reactors (CASL) Virtual Environment for Reactor Applications (VERA) offers unique capabilities that combine high-fidelity in-core and ex-core radiation transport. In this recent development activity, the deterministic neutronics code MPACT performs the in-core radiation transport with temperature feedback using COBRA-TF (CTF), and provides the fission source to Shift for a follow-on fixed source radiation transport calculation to determine ex-core quantities of interest. The coupling of MPACT to Shift allows the neutron source for the ex-core transport to be calculated from the time-dependent, fully coupled MPACT solution. This paper presents the first use of VERA to perform coupon irradiation for code-to-code comparison of its ex-core capabilities. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2017

Some Notes on Neutron Up-Scattering and the Doppler-Broadening of High-Z Scattering Resonances

When neutrons are scattered by target nuclei at elevated temperatures, it is entirely possible that the neutron will actually gain energy (i.e., up-scatter) from the interaction. This phenomenon is in addition to the more usual case of the neutron losing energy (i.e., down-scatter). Furthermore, the motion of the target nuclei can also cause extended neutron down-scattering, i.e., the neutrons can and do scatter to energies lower than predicted by the simple asymptotic models. In recent years, more attention has been given to temperature-dependent scattering cross sections for materials in neutron multiplying systems. This has led to the inclusion of neutron up-scatter in deterministic codes like Partisn and to free gas scattering models for material temperature effects in Monte Carlo codes like MCNP and cross section processing codes like NJOY. The free gas scattering models have the effect of Doppler Broadening the scattering cross section output spectra in energy and angle. The current state of Doppler-Broadening numerical techniques used at Los Alamos for scattering resonances will be reviewed, and suggestions will be made for further developments. The focus will be on the free gas scattering models currently in use and the development of new models to include high-Z resonance scattering effects. These models change the neutron up-scattering behavior.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2020

Super-resolution surface slope metrology of x-ray mirrors

We present experimental, analytical, and numerical methods developed for reconstruction (deconvolution) of one-dimensional (1D) surface slope profiles over the spatial frequency range where the raw data are significantly perturbed due to the limited resolution of the measurement instrument. We characterize the spatial resolution properties of a profiler with the instrument's transfer function (ITF). To precisely measure the ITF, we apply a recently developed method utilizing test surfaces with 1D linear chirped height profiles of constant slope amplitude. Based on the results of the ITF calibration, we determine parameters of an analytical model for the ITF that is used in the original reconstruction software. Here, we treat surface slope metrology data obtained with the Optical Surface Measuring System (OSMS), using as a sensor an electronic autocollimator (AC) ELCOMAT-3000. The spatial resolution of the OSMS is limited by the size of the AC light-beam-collimating aperture. For the purposes of this investigation, the OSMS is equipped with a circular aperture with a diameter of 2.5 mm. This is a typical arrangement of most AC-based slope profilers developed for surface slope metrology of state-of-the-art x-ray mirrors. Using the example of surface slope metrology of two state-of-the-art elliptically shaped x-ray focusing mirrors, we demonstrate that the developed data reconstruction procedure allows us to significantly improve the accuracy of surface slope metrology with the OSMS over the spatial wavelength range from ~1.6 mm to 7 mm. Thus, the amplitude of the quasi-periodic error characteristic of the deterministic polishing process used appears to be higher by a factor of~2 than is apparent from the rough metrology data. Underestimation of the surface slope errors in this spatial wavelength range can lead to serious errors in the expected performance of x-ray mirrors in synchrotron beamlines, especially at modern light sources utilizing coherent x rays, where the perturbations can lead to increased speckle-like intensity variation.

47 OTHER INSTRUMENTATION↗

DOE OSTI2022

Status of the OECD/NEA Watts Bar unit 1 benchmark and calculations of local reactor core power data of the Zero Power physics tests

The paper aims to provide an update on the status of the Organization for Economic Cooperation and Development (OECD) / the Nuclear Energy Agency (NEA) Tennessee Valley Authority (TVA) Watts Bar 1 (WB1) Multi-Physics Multi-Cycle depletion benchmark and the results associated with Exercise 1. The benchmark relies on the set of benchmark progression problems developed by the Department of Energy (DOE) Consortium for the Advanced Simulation of Light Water Reactors (CASL) for the Virtual Environment for Reactor (VERA), which are based on real plant design and operational data. The OECD/NEA TVA WB1 benchmark is designed for validation of both traditional and novel high-fidelity multi-physics codes to analyze Pressurized Water Reactors (PWR) depletion cycles. The activities are conducted under the Expert Group on Reactor Systems Multi-Physics (EGMUP) at NEA/OECD. In this work, we analyze Exercise 1 of the benchmark: stand-alone Three-Dimensional (3D) neutronics at Start-up Zero Power Physics Test (ZPPT) at Hot Zero Power Conditions (HZP) and the resulting power maps. The code used to model the exercise is the continuous energy Monte Carlo code Serpent 2.1.31 along with the nuclear data library ENDF/B-VII.1. Serpent results are compared with the high-fidelity deterministic code MPACT, which is part of VERA. The paper presents three selected results from power calculations required output. The results compared are the normalized axially integrated radial core power maps, the normalized axial averaged core power shapes, and the normalized core hottest and coldest assemblies radial power maps. The Root Mean Square Deviation (RMSD) between Serpent and MPACT is 0.51 % for the axially integrated radial core power map, 1.41 % for the axial averaged core power shape, 2.42 % for the hottest assembly axial power shape, 0.88 % for the coldest assembly axial integrated hottest assembly radial pin power map, and 0.17 % for the axially integrated coldest assembly radial pin power map. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2024

An evaluation of GPT models for phenotype concept recognition

Clinical deep phenotyping and phenotype annotation play a critical role in both the diagnosis of patients with rare disorders as well as in building computationally-tractable knowledge in the rare disorders field. These processes rely on using ontology concepts, often from the Human Phenotype Ontology, in conjunction with a phenotype concept recognition task (supported usually by machine learning methods) to curate patient profiles or existing scientific literature. With the significant shift in the use of large language models (LLMs) for most NLP tasks, we examine the performance of the latest Generative Pre-trained Transformer (GPT) models underpinning ChatGPT as a foundation for the tasks of clinical phenotyping and phenotype annotation. The experimental setup of the study included seven prompts of various levels of specificity, two GPT models (gpt-3.5-turbo and gpt-4.0) and two established gold standard corpora for phenotype recognition, one consisting of publication abstracts and the other clinical observations. The best run, using in-context learning, achieved 0.58 document-level F1 score on publication abstracts and 0.75 document-level F1 score on clinical observations, as well as a mention-level F1 score of 0.7, which surpasses the current best in class tool. Without in-context learning, however, performance is significantly below the existing approaches. Our experiments show that gpt-4.0 surpasses the state of the art performance if the task is constrained to a subset of the target ontology where there is prior knowledge of the terms that are expected to be matched. While the results are promising, the non-deterministic nature of the outcomes, the high cost and the lack of concordance between different runs using the same prompt and input make the use of these LLMs challenging for this particular task.

59 BASIC BIOLOGICAL SCIENCES↗