Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “EVALUATION”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,603 records · Page 89
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Evaluation of Model Summertime Boundary Layer Cloud Development over Complex Terrain in New York State

Weather forecasting over complex terrain with diverse land cover is challenging. Utilizing the high-resolution observations from New York State Mesonet (NYSM), we are able to evaluate the surface processes of the Weather Research Forecast (WRF) Model in a detailed, scale-dependent manner. Here, in the study, possible impacts of land–atmosphere interaction on surface meteorology and boundary layer cloud development are investigated with different model resolutions, land surface models (LSMs), and planetary boundary layer (PBL) physical parameterizations. The High-Resolution Rapid Refresh, version 3 (HRRR), forecasting model is used as a reference for the sensitivity evaluation. Results show that over complex terrain, the high-resolution simulations (1 km × 60 vertical levels) generally perform better compared to low-resolution (3 km × 50 levels) in both surface meteorology and cloud fields. LSMs play a more important role in surface meteorology compared to PBL schemes. The NoahMP land surface model exhibits daytime warmer and drier biases compared to the Rapid Update Cycle (RUC) due to better prediction of the Bowen ratio in RUC. The PBL schemes would affect the convective strength in the boundary layer. The Shin–Hong (SH) scale-aware scheme tends to produce the strongest convective strength in the PBL, while the ACM2 PBL scheme rarely resolved convection even at 1-km resolution. By considering the radiation effect of subgrid-scale (SGS) clouds, the Mellor–Yamada–Nakanishi–Niino eddy diffusivity mass flux (MYNN-EDMF) predicted the highest cloud coverage and lowest surface solar radiation bias. The configuration of SGS clouds in MYNN-EDMF would not only significantly reduce shortwave radiation bias, but also affect the convection behaviors through land surface–cloud–radiation interaction.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Lidar-Based Evaluation of HRRR Performance in California’s Diablo Range

The performance of the NOAA High-Resolution Rapid Refresh (HRRR) model for capturing low-level winds near a wind energy production site during summer 2019 is evaluated. This study catalogs the ability of HRRR to predict boundary layer dynamics relevant to wind energy interests over complex terrain, which has presented challenges for weather and energy forecasting. Performance is evaluated by comparing HRRR output to wind-profiling Doppler lidars at Lawrence Livermore National Laboratory Site 300. HRRR captured the diurnal profile of horizontal winds in the observed 150-m layer, despite strong underpredictions (∼4 m s −1 ) during evening and nighttime hours. These underpredictions may be a result of local speedup flows observed by the lidars, which were unresolved in HRRR due to their small spatial extent. HRRR bias magnitude relative to observations was found to be minimal during days with synoptic-scale troughs and strong 850-hPa geopotential gradients, while bias magnitude was maximal during days with synoptic ridging and weak 850-hPa geopotential gradients. To translate wind speed predictions to energy forecasting, generic turbine models were used to estimate power generation for turbines characteristic of the nearby Altamont Pass Wind Resource Area. Results show that HRRR-based energy estimates predicted daytime power generation adequately relative to lidar-based estimates with an 18-h lead time (bias magnitude < 0.4 MW from 0900 to 1400 LT) but overpredicted power during the rest of the diurnal cycle (bias > 1 MW). These results demonstrate conditions under which HRRR performs well for wind energy applications in complex terrain, while highlighting biases that require further investigation to support usage of a high-resolution model for wind energy forecasts.

Boundary layer↗
DOE OSTI2024

Development and Evaluation of Ensemble Learning-based Environmental Methane Detection and Intensity Prediction Models

The environmental impacts of global warming driven by methane (CH 4 ) emissions have catalyzed significant research initiatives in developing novel technologies that enable proactive and rapid detection of CH 4 . Several data-driven machine learning (ML) models were tested to determine how well they identified fugitive CH 4 and its related intensity in the affected areas. Various meteorological characteristics, including wind speed, temperature, pressure, relative humidity, water vapor, and heat flux, were included in the simulation. We used the ensemble learning method to determine the best-performing weighted ensemble ML models built upon several weaker lower-layer ML models to (i) detect the presence of CH 4 as a classification problem and (ii) predict the intensity of CH 4 as a regression problem. The classification model performance for CH 4 detection was evaluated using accuracy, F1 score, Matthew’s Correlation Coefficient (MCC), and the area under the receiver operating characteristic curve (AUC ROC), with the top-performing model being 97.2%, 0.972, 0.945 and 0.995, respectively. The R 2 score was used to evaluate the regression model performance for CH 4 intensity prediction, with the R 2 score of the best-performing model being 0.858. The ML models developed in this study for fugitive CH 4 detection and intensity prediction can be used with fixed environmental sensors deployed on the ground or with sensors mounted on unmanned aerial vehicles (UAVs) for mobile detection.

Majumder, Reek↗
DOE OSTI2020

Computationally efficient evaluation of optimum homogeneous charge compression ignition operating range with accelerated multizone engine cycle simulation

The very intensive calculations necessary to define a performance map requiring evaluation of over a hundred individual operating points can be efficiently conducted with accelerated multizone for engine cycle simulation, leading to a definition of regions of acceptable and optimum homogeneous charge compression ignition operation. Accelerated multizone for engine cycle simulation has the virtue of enabling accurate evaluation of many operating conditions based on thermal stratification data from a single fluid mechanics run at motored conditions. This is possible because thermal stratification is more sensitive to engine geometry than to operating conditions. In this article, accuracy of accelerated multizone for engine cycle simulation is demonstrated by comparison with experimental data for iso-octane homogeneous charge compression ignition operation over a broad range of lean equivalence ratios (0.14–0.28). The validated accelerated multizone for engine cycle simulation model is then applied to generating a performance map for an engine controlled by appropriately adjusting equivalence ratio and internal exhaust gas recirculation. Regions of acceptable and optimum combustion are identified. It is finally demonstrated that while indicated mean effective pressure remains low for optimum homogeneous charge compression ignition operation (1–4 bar), this is sufficient for a large fraction of typical driving in light-duty vehicles. Much driving including idle can therefore be done in homogeneous charge compression ignition mode at high efficiency and low (essentially zero) NOx and particulate matter emissions.

30 DIRECT ENERGY CONVERSION↗
DOE OSTI2021

Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program

We applied a three-step process, abstracting and analyzing program budgets to examine how Colorectal Cancer Control Program (CRCCP) awardees are structuring their programs and to assess the fidelity of program design to the CRCCP public health model. We reviewed 23 state, one tribal organization, and six university awardee budgets. We assessed resource allocations, staffing structures, and contracted partners and their activities. Awardees allocated 83% of all funds to contracts and personnel. Program managers were the most budgeted personnel type across three measures: number of people, full-time equivalency, and personnel costs. Awardees not only contracted with health care systems and clinics (39% of all contracts) but also contracted other partner types. Contractors were mainly funded to implement evidence-based interventions (25%) and conduct evaluation (24%). Program design varied among awardees in the number of staff (0–22), number of full-time equivalencies (0–5.4), and the number of contracts (1–11) budgeted. State awardees budgeted more resources to contracts, compared with university awardees (57% vs. 31%), while universities budgeted more for total personnel costs (41% vs. 30%). We learned that awardees designed their programs with fidelity to the CRCCP model. Although implementation approaches varied, overall results suggest implementation requires a combination of internal capacities and contracted partners. Budgets provide opportunities to use already existing program data to evaluate program design, partnerships, and planned activities.

Uhd, Justin↗
DOE OSTI2026

PRIME: An evaluation framework for protein representation inference and generalization in viral mutation space

Background Protein language models (PLMs) have revolutionized protein fitness prediction, yet their application to rapidly evolving viral pathogens is often confounded by extreme sequence homology. This homology leads to “data leakage” in standard random validation splits, yielding inflated performance metrics that fail to translate into real-world biosurveillance utility. Results We present Protein Representation Inference for Mutation Evaluation (PRIME), a framework that integrates domain-specific fine-tuning with a rigorous position-stratified validation protocol to evaluate viral threats. Using a dataset of 347,432 SARS-CoV-2 receptor binding domain (RBD) sequences, we demonstrate that while random training data split yields deceptive R 2 values (> 0.90), they fail to generalize to novel mutational sites. By benchmarking models up to 650 M parameters, we show that domain-specific fine-tuning of the ESM-C 600 M model with correctly stratified data provides an initial demonstration of predictive signal for binding affinity and expression at unseen mutational sites of binding affinity and expression on unseen sites (R 2 ~0.23), a significant advancement over base foundation models which exhibit no predictive power (R 2 <0). PRIME’s embedding-based clustering identified 3.03% of bat coronavirus sequences as candidates for further experimental prioritization based on their functional similarity to human-infective strains in embedding space, offering a perspective complementary to traditional phylogenetic methods. Conclusion PRIME establishes a new benchmark for the application of PLMs in pathogen surveillance. Our findings demonstrate that state-of-the-art models and fine-tuning, when paired with stratified validation, provide biologically meaningful insights into pathogen evolution and zoonotic risk.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Generation and evaluation of synthetic patient data

Background: Machine learning (ML) has made a significant impact in medicine and cancer research; however, its impact in these areas has been undeniably slower and more limited than in other application domains. A major reason for this has been the lack of availability of patient data to the broader ML research community, in large part due to patient privacy protection concerns. High-quality, realistic, synthetic datasets can be leveraged to accelerate methodological developments in medicine. By and large, medical data is high dimensional and often categorical. These characteristics pose multiple modeling challenges. Methods: In this paper, we evaluate three classes of synthetic data generation approaches; probabilistic models, classification-based imputation models, and generative adversarial neural networks. Metrics for evaluating the quality of the generated synthetic datasets are presented and discussed. Results: While the results and discussions are broadly applicable to medical data, for demonstration purposes we generate synthetic datasets for cancer based on the publicly available cancer registry data from the Surveillance Epidemiology and End Results (SEER) program. Specifically, our cohort consists of breast, respiratory, and non-solid cancer cases diagnosed between 2010 and 2015, which includes over 360,000 individual cases. Conclusions: We discuss the trade-offs of the different methods and metrics, providing guidance on considerations for the generation and usage of medical synthetic data.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Evaluating crown scorch predictions from a computational fluid dynamics wildland fire simulator

Abstract Background Crown scorch—the heating of live leaves, needles, and buds in the vegetative canopy to lethal temperatures without widespread combustion—is one of the most common fire effects shaping post-fire canopies. Despite the ability of computational fluid dynamic models to finely resolve fire activity and buoyant plume dynamics including heterogenous 3D distributions of forest canopy heating, these models have had only limited use in simulating fire effects and have not been used to evaluate crown scorch. Here, we demonstrate a method of evaluating crown scorch using a computational fluid dynamics model, FIRETEC, and validate this approach by simulating the experiments that were used to develop Van Wagner’s 1973 crown scorch model. Results The average scorch height prediction from FIRETEC compares well with the empirical model derived by Van Wagner, which is the most widely used empirical model for crown scorch. We further find that the 3D buoyant plume dynamics from a steady and homogeneous idealized heat source on the ground results in a spatially heterogenous crown scorch pattern reflecting complex heating dynamics that are best represented by percent scorch rather than height of scorch. Conclusions The ability of the computational fluid dynamics model to capture variation in crown scorch due to 3D buoyant plume dynamics provides direct links between forest structure, fire behavior, and fire effects that can be used by forest managers and researchers to better understand how fires result in crown damage under various environmental and management scenarios.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

EXPERIMENTAL EVALUATION OF ULTRASONIC VELOCITIES AND ANISOTROPY IN THE TUSCALOOSA MARINE SHALE FORMATION

Tuscaloosa Marine Shale (TMS) formation is a clay- and organic-rich emerging shale play with a considerable amount of hydrocarbon resources. Despite the substantial potential, there have been only a few wells drilled and produced in the formation over the recent years. The analyzed TMS samples contain an average of 50 wt% total clay, 27 wt% quartz and 14 wt% calcite and the mineralogy varies considerably over the small intervals. The high amount of clay leads to pronounced anisotropy and the frequent changes in mineralogy result in the heterogeneity of the formation. We studied the compressional (V P ) and shear-wave (V S ) velocities to evaluate the degree of anisotropy and heterogeneity, which impact hydraulic fracture growth, borehole instabilities, and subsurface imaging. The ultrasonic measurements of P- and S-wave velocities from five TMS wells are the best fit to the linear relationship with R 2 = 0.84 in the least-squares criteria. We observed that TMS S-wave velocities are relatively lower when compared to the established velocity relationships. Most of the velocity data in bedding-normal direction lie outside constant V P /V S lines of 1.6–1.8, a region typical of most organic-rich shale plays. For all of the studied TMS samples, the S-wave velocity anisotropy exhibits higher values than P-wave velocity anisotropy. In the samples in which the composition is dominated by either calcite or quartz minerals, mineralogy controls the velocities and V P /V S ratios to a great extent. Additionally, the organic content and maturity account for the velocity behavior in the samples in which the mineralogical composition fails to do so. The results provide further insights into TMS Formation evaluation and contribute to a better understanding of the heterogeneity and anisotropy of the play.

Geochemistry & Geophysics↗
DOE OSTI2022

A Generalizable Evaluated Approach, Applying Advanced Geospatial Statistical Methods, to Identify High Lead Exposure Locations at Census Tract Scale: Michigan Case Study

BACKGROUND: Despite great progress in reducing environmental lead (Pb) levels, many children in the United States are still being exposed. OBJECTIVE: Our aim was to develop a generalizable approach for systematically identifying, verifying, and analyzing locations with high prevalence of children’s elevated blood Pb levels (EBLLs) and to assess available Pb models/indices as surrogates, using a Michigan case study. METHODS: We obtained ~1:9 million BLL test results of children <6 years of age in Michigan from 2006–2016; we then evaluated them for data representativeness by comparing two percentage EBLL (%EBLL) rates (number of children tested with EBLL divided by both number of children tested and total population). We analyzed %EBLLs across census tracts over three time periods and between two EBLL reference values (≥5 vs. ≥10 μg/dL) to evaluate consistency. Locations with high %EBLLs were identified by a top 20 percentile method and a Getis-Ord Gi* geospatial cluster “hotspot” analysis. For the locations identified, we analyzed convergences with three available Pb exposure models/indices based on old housing and sociodemographics. RESULTS: Analyses of 2014–2016 %EBLL data identified 11 Michigan locations via cluster analysis and 80 additional locations via the top 20 percentile method and their associated census tracts. Data representativeness and consistency were supported by a 0.93 correlation coefficient between the two EBLL rates over 11 y, and a Kappa score of ~0:8 of %EBLL hotspots across the time periods (2014–2016) and reference values. Many EBLL hotspot locations converge with current Pb exposure models/indices; others diverge, suggesting additional Pb sources for targeted interventions. DISCUSSION: This analysis confirmed known Pb hotspot locations and revealed new ones at a finer geographic resolution than previously available, using advanced geospatial statistical methods and mapping/visualization. It also assessed the utility of surrogates in the absence of blood Pb data. This approach could be applied to other states to inform Pb mitigation and prevention efforts. https://doi.org/10.1289/EHP9705

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Proposal and application of ROM-Lasso method for sensitivity coefficient evaluation

We propose a novel method for evaluating sensitivity coefficients of neutronics parameters to cross sections, so-called the reduced-order modeling technique ROM-Lasso. In this method, cross sections of interest are randomly sampled, and corresponding perturbed core analyses are performed. Then, the sensitivity coefficient vector of the higher-level model is expanded via the active subspace bases obtained with the lower-level model whose dimensional complexity is smaller than that of the higher-level model, and the expansion coefficients are estimated by the Lasso regression. A unique feature of the ROM-Lasso method allows the use of different bases optimized for each neutronics parameter. We conducted a verification calculation for an accelerator-driven system and demonstrated that the ROM-Lasso method can reproduce the sensitivity coefficients with a much smaller number of forward calculations than the direct method. The proposed method can be used to practically evaluate sensitivity coefficients. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2025

Remote sensing images, DEM, and point clouds associated with “Accuracy evaluation of cost-effective 3D reconstruction approaches for hydrobiogeochemical processes in non-perennial stream riverbeds”

This data package is associated with the publication “Accuracy evaluation of cost-effective 3D reconstruction approaches for hydrobiogeochemical processes in non-perennial stream riverbeds” published in Frontiers in Environmental Science, Environmental Informatics and Remote Sensing (Bao et al., 2026; doi: 10.3389/fenvs.2026.1725258). This data package includes the drone photos for a section of Umtanum Creek in Washington, Unted States. The photos were used to reconstruct the 3-dimensional (3D) digital elevation model (DEM) of the riverbed for the investigated stream section. The reconstruction results from four approaches are provided: (1) unoccupied aerial vehicle (UAV, colloquially known as drone) imagery-based Structure-from-Motion (SfM), (2) a machine learning-based 3D reconstruction model, Visual Geometry Grounded Deep Structure from Motion (VGGSfM), (3) Visual Geometry Grounded Transformer for long sequence of images (VGGT-Long), and (4) handheld smartphone LiDAR scanning. The ground truth measurements by tripod-mounted optical level kit and ground control points GPS locations for evaluating the accuracy of the four reconstruction approaches are also provided in this data package. A preliminary version of this data package was published in October 2025 at the time of manuscript submission. It was updated in March 2026, at the time of manuscript acceptance, to include additional metadata (this readme, data dictionary, and file level metadata). The data did not change. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to a readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. This dataset is comprised of (1) 8 folders; (2) the detailed flight configuration html files; (3) field metadata; (4) a readme; (5) a data dictionary; and (6) file-level metadata. The folders “2024_10_18_d01” and “2024_10_18_d02” contain the original drone photos for the two drone flights (d01 and d02) on October 18, 2024. The reconstruction results from each of the approaches are in the folders called “ODM_SfM”, “VGGSfM”, “VGGTLong”, and “LiDAR”. The ground truth measurements are in the folder called “optical_level_kit”. Lastly, results comparing the different approaches are in the folder called “comparisons”. All files are .csv, .html, .jpg, .obj, .txt, and .npy. For information on using the .obj and .npy files, see the readme files within the same folder as the files.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages

The rapid evolution of large language models (LLMs) has opened new possibilities for automating various tasks in software development. This paper evaluates the capabilities of the LLaMA 2-70B model in automating these tasks for scientific applications written in commonly used programming languages. Using representative test problems, we assess the model's capacity to generate code, documentation, and unit tests, as well as its ability to translate existing code between commonly used programming languages. Our comprehensive analysis evaluates the compilation, runtime behavior, and correctness of the generated and translated code. Additionally, we assess the quality of automatically generated code, documentation, and unit tests. Here, our results indicate that while LLaMA 2-70B frequently generates syntactically correct and functional code for simpler numerical tasks, it encounters substantial difficulties with more complex, parallelized, or distributed computations, requiring considerable manual corrections. We identify key limitations and suggest areas for future improvements to better leverage AI-driven automation in scientific computing workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

A Million Person Study Innovation: Evaluating Cognitive Impairment and other Morbidity Outcomes from Chronic Radiation Exposure Through Linkages with the Centers for Medicaid and Medicare Services Assessment and Claims Data

Here, the study of One Million U.S. Radiation Workers and Veterans, the Million Person Study (MPS), examines the health consequences, both cancer and non-cancer, of exposure to ionizing radiation received gradually over time. Recently the MPS has focused on mortality patterns from neurological and behavioral conditions, e.g., Parkinson's disease, Alzheimer's disease, dementia, and motor neuron disease such as amyotrophic lateral sclerosis. A fuller picture of radiation-related late effects comes from studying both mortality and the occurrence (incidence) of conditions not leading to death. Accordingly, the MPS is identifying neurocognitive diagnoses from fee-for-service insurance claims from the Centers for Medicare and Medicaid Services (CMS), among Medicare beneficiaries beginning in 1999 (the earliest date claims data are available). Linkages to date have identified ∼540,000 workers with available health information. Such linkages provide individual information on important co-factor and confounding variables such as smoking, alcohol consumption, blood pressure, obesity, diabetes and many other health and demographic characteristics. The total person-level set of time-dependent variables, outcomes, organ-specific dose measures, co-factors, and demographics will be massive and much too large to be evaluated with standard software. Thus, development of specialized open-source software designed for large datasets (Colossus) is nearly complete. The wealth of information available from CMS claims data, coupled with individual dose reconstructions, will thus greatly enhance the quality and precision of health evaluations for this new field of low-dose radiation and neurocognitive effects.

Dauer, Lawrence T.↗
DOE OSTI2022

Supplementary Data for "Evaluation of the Economic Implications of Varied Pressure Drawdown Strategies Generated Using a Real-time, Rapid Predictive, Multi-fidelity Model for Unconventional Oil and Gas Wells" by Bello, K., Vikara, D., Sheriff, A., Viswanathan, H., Carr, T., Sweeney, M., O'Malley, D., Marquis, M., Vactor, R.T., and Cunha, L.

The Bello et al. study evaluates the impact of contrasting pressure drawdown on gas productivity and the resulting economics of a well in the Marcellus Shale of the Appalachian Basin. This research applies a techno-economic analysis approach to help identify potential ways pressure management strategies can be used to improve cumulative recovery of hydraulically fractured horizontal wells while maintaining project profitability. Gas production forecast outlook scenarios of the Marcellus Shale Energy and Environment Laboratory Laboratory's MIP-3H well were generated under varying pressure drawdowns using two approaches: 1) a novel physics-informed machine learning (PIML) workflow and 2) via traditional reservoir simulation in Computer Modeling Group’s (CMG) GEM Compositional & Unconventional Simulator. Cash flow and other economic metrics of interest were compiled on the production outlook using the U.S. Department of Energy's (DOE) National Energy Technology Laboratory (NETL) Unconventional Shale Well Economic Model (UShWEM).The sheets within this Microsoft ExcelTM workbook provide the economic metric outputs for the baseline condition and the one-at-a-time (OAT) sensitivity analysis of UShWEM's input parameters for each of the production scenarios evaluated.

Fracture Network Model↗
DOE OSTI2025

CO2 Enhanced Oil Recovery Evaluation System (CO2_E_EvSystem), Version 2025

The United States Department of Energy’s (DOE’s) Office of Fossil Energy and the National Energy Technology Laboratory (NETL) developed a suite of screening-level, techno-economic models/tools, known as the CO2_E_EvSystem, to evaluate technical aspects and costs of using carbon dioxide (CO2) enhanced oil recovery (EOR) to store CO2 and produce oil. CO2_E_EvSystem has three software components and several input and output files. The software components are CO2_E_EvTool, CO2_Prophet, and CO2_E_COM. Almost all computational work is performed by CO2_Prophet and CO2_E_COM, which are both Fortran programs. The primary role of CO2_E_EvTool is to manage input and output files for the two programs, run the two programs, allow multiple oilfields to be evaluated in a single run, and generate files that summarize the results for all the oilfields run. This version of the CO2_E_EvSystem includes a residual oil zone dataset from the San Andres formation, Permian Basin, to demonstrate the system and guide users on attributes needed for dataset inputs.

AS↗
DOE OSTI2023

Evaluation of the first version of the new RFPI system dedicated to PIP-II project accelerator

The Low-Level Radio Frequency (LLRF) control system is one of the most critical superconducting linac infrastructures responsible for the parameters of the beam acceleration. The LLRF system mainly focuses on the electromagnetic field parameters inside the cavity. While it incorporates fast feedback algorithms to optimize energy transfer to the passing particle beam it does not follow other cryomodule or cavity parameter changes. The Radio Frequency Protection Interlock (RFPI) system closely monitors various factors (like cryomodule vacuum, beamline vacuum, field emission probe current level, temperature, RF signal leakage, etc). Its simple but reliable logic has to provide an instant decision about the LLRF system or high-power amplifier output signal blocking in case of safety region excitation. This contribution presents a new version of the RFPI system which logic is implemented in the FPGA chip. The initial work on the prototype of the new system design resulted in the PoC (Proof of Concept) device. The PoC offers the possibility of various protection logic configurations, input signals parameters evaluation, and modularity aspects verification. The structure and test results from the device evaluation are summarized and discussed in this contribution.

43 PARTICLE ACCELERATORS↗
DOE OSTI2024

Evaluating the Effectiveness of Retrieval-Augmented Large Language Models in Scientific Document Reasoning

Despite the dramatic progress in Large Language Model (LLM) development, LLMs often provide seemingly plausible but not factual information, often referred as hallucinations. Retrieval-augmented LLMs provide a non-parametric approach to solve these issues by retrieving relevant information from external data sources and augment the training process. These models helps to trace evidence from an externally provided knowledge base allowing the model predictions to be better interpreted and verified. In this work, we critically evaluate these models in their ability to perform in scientific document reasoning tasks. To this end, we tuned multiple such model variants with science-focused instructions and evaluated them on a scientific document reasoning benchmark for the usefulness of the retrieved document passages. Our findings suggest that models justify predictions in science tasks with fabricated evidence and leveraging scientific corpus as pretraining data does not alleviate the risk of evidence fabrication.

• Artificial intelligence (AI) / machine learning ↗