Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Evaluating Model Robustness for Defect Identification and Classification in a Composite Aerostructure Material

Aircraft structures are required to have a high level of quality to satisfy their need for light weight, efficient flight, and withstanding high loads over their lifespan. These aerostructures are typically made from a composite material due to their good tensile strength and resistance to compression. To ensure their structural integrity, the composite material requires inspection for common flaws such as porosity, delaminations, voids, foreign object debris, and other defects. Ultrasonic testing (UT) is a popular non-destructive inspection (NDI) technique used for effectively evaluating the composite material. Current inspection methods rely heavily on human experience and are extremely time consuming. Therefore, there is a need for the development of techniques to reduce the manual inspection time. This work compares the performance of different deep learning-based methods in the identification and classification of defects. Deep learning has shown great promise in numerous fields, and we show its effectiveness in the evaluation of the composite aerostructure material. The methods developed here are both highly reliable with a top recall value of 98.64% as well as extremely efficient requiring an average of 4 s during the inferencing stage to evaluate new composites. Lastly, we investigate model robustness to concept drift by measuring its performance over time.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Data Files for Runoff Evaluation in an Earth System Land Model for Permafrost Regions

Modeling of hydrological runoff is essential for accurately capturing spatiotemporal feedbacks within the land–atmosphere system, particularly in sensitive regions such as permafrost landscapes. However, substantial uncertainties persist in the terrestrial runoff parameterization schemes used in Earth system and land surface models. This is particularly true in permafrost regions, where landscape heterogeneity is high and reliable observational data are scarce.This data set includes all files that were produced and applied in the paper Runoff Evaluation in an Earth System Land Model for Permafrost Regions [Xiang et al. in review]. The paper is in review as of July 1 2025 in Geoscientific Model Development (GMD). In this study, we evaluate the performance of runoff parameterization schemes in the Energy Exascale Earth System Model (E3SM) land model (ELM). Our proposed framework leverages simulation results from the Advanced Terrestrial Simulator (ATS), which is a physics-rich integrated surface/subsurface hydrologic model that has been successfully evaluated previously in Arctic tundra regions. We used ATS to simulate runoff from 22 representative hillslopes in the Sagavanirktok River basin, located on the North Slope of Alaska, then compared the output with ELM’s parameterized representation of total runoff. This dataset contains 2 figure image files (*.png, *jpg) that describe the study site and methods, as well as folders (Figure*.zip) that contain the associated data files (*.csv, *.dat) and python code notebooks (*.ipynb) for figures 3-7 in the paper. Jupyter notebook (*.ipynb) files that produce the figure files using the associated data files will run within a python environment configured with Jupyter Lab or Notebook packages.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

ARM Data-Oriented Metrics and Diagnostics Package for Climate Model Evaluation

A Python-based metrics and diagnostics package is currently being developed by the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Infrastructure Team at Lawrence Livermore National Laboratory (LLNL) to facilitate the use of long-term, high-frequency measurements from the ARM Facility in evaluating the regional climate simulation of clouds, radiation, and precipitation. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data. The Coupled Model Intercomparison Project (CMIP) model data sets are also included in the package to enable model intercomparison as demonstrated in Zhang et al. (2017). The mean of the CMIP model can serve as a reference for individual models. Basic performance metrics are computed to measure the accuracy of mean state and variability of climate models. The evaluated physical quantities include cloud fraction, temperature, relative humidity, cloud liquid water path, total column water vapor, precipitation, sensible and latent heat fluxes, and radiative fluxes, with plan to extend to more fields, such as aerosol and microphysics properties. Process-oriented diagnostics focusing on individual cloud- and precipitation-related phenomena are also being developed for the evaluation and development of specific model physical parameterizations. The version 1.0 package is designed based on data collected at ARM’s Southern Great Plains (SGP) Research Facility, with the plan to extend to other ARM sites. The metrics and diagnostics package is currently built upon standard Python libraries and additional Python packages developed by DOE (such as CDMS and CDAT). The ARM metrics and diagnostic package is available publicly with the hope that it can serve as an easy entry point for climate modelers to compare their models with ARM data. In this report, we first present the input data, which constitutes the core content of the metrics and diagnostics package in section 2, and a user's guide documenting the workflow/structure of the version 1.0 codes, and including step-by-step instruction for running the package in section 3.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Evaluation of the Turbine Integrated Mortality Reduction (TIMR SM ) Technology as a Smart Curtailment Approach (Final Summary Report)

Wind energy is a crucial technology for achieving net-zero emissions by 2050. However, the growth and deployment of wind energy in North America have led to the deaths of many bat species due to operating wind turbines. Hundreds of thousands of bats are estimated to die at wind turbines annually in North America. Operational minimization, which includes feathering turbine blades and curtailment, has been documented to reduce bat fatality effectively. Curtailment refers to altering turbine operation based on wind speed, time of year, temperature, sensors, and activity models. However, when turbines are curtailed, they do not generate power, resulting in energy loss and revenue for wind energy facilities. The Electric Power Research Institute (EPRI) funded the development of Turbine Integrated Mortality Reduction (TIM SM ) Technology, which curtails turbine operation when bats are detected. The initial TIMR system research showed promising results, with an 85% reduction in overall bat fatalities and a 91% reduction for the little brown bat. However, these results were based on a single site during one fall season, and it was unclear if similar results could be replicated at other wind energy facilities. This research aimed to validate the TIMR system results from the prior field study at a second site in the U.S., estimate the power production and reduction in bat mortality at turbines with installed TIMR systems relative to blanket curtailment and fully operational turbines, test the TIMR system in two calendar years and during the summer and fall periods, and evaluate the operational and commercial characteristics of the TIMR system for potential wind industry adoption. The study was conducted at a 500.9-MW wind energy facility in southeast Adair County, Iowa. Three experimental treatments were involved in this randomized block design study: TIMR, Curtailment at 5.0 m/s, and Normal Operation. In 2021, three treatments were used at 18 turbines, expanding to four treatments across 36 turbines in 2022. The TIMR system worked as designed throughout the entire study; however, because of unexpected wind turbine operational challenges in 2021, there was not sufficient sample size to evaluate the treatment differences. In 2022, there were significant differences in fatality levels between treatment types and normal operating turbines. Curtailment at 5.0 m/s reduced fatalities by 30.8% compared to normal operations, and TIMR decreased fatalities by 48.6% compared to normal operations. Two different methods were used to evaluate the differences in energy loss for each treatment. The TIMR system resulted in 1.3% to 1.6 % annual energy loss in 2021 and 1.0% to 1.2 % in 2022. The Curtailment at 5.0 m/s resulted in 0.6% to 0.8 % annual energy loss in 2021 and 0.5% to 0.6 % in 2022. The project achieved all the stated objectives and demonstrated that TIMR is an effective technology that balances bat fatality reduction with energy generation. The results will support the deployment of TIMR and other acoustic sensor-based technologies. The research provides valuable insights into the impact of different treatments on fatality rates and energy outputs, contributing to the ongoing efforts to mitigate the environmental impact of wind energy.

17 WIND ENERGY↗
DOE OSTI2024

​​Integrity Enhancing Protocols (IEP) Evaluation Framework​

​Protocols play an essential role in Advance Reactor systems. A diverse set of protocols are available to these reactors. Advanced Reactors benefit from technologies that can minimize their resource utilization and costs. Evaluation frameworks are often used when assessing protocols and processes related to cryptographic security systems. The following report discusses the various characteristics associated with these protocol evaluation frameworks, and derives a novel evaluative framework.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Chlorine Nuclear Data Evaluation Aided Through New LANSCE Measurements

The collaboration between the Los Alamos National Laboratory Neutron Science Center (LANSCE) and TerraPower LLC enables the enhanced understanding of fast spectrum critical systems consisting of chlorine. Specifically, TerraPower is interested in updating the nuclear data for the stable isotopes of chlorine, 35 Cl and 37 Cl, because these nuclides are the primary constituents of the chloride fuel salt in the Molten Chloride Reactor Experiment (MCRE), for which TerraPower is leading the design. The Cooperative Research and Development Agreement (CRADA) between the parties is funded by DOE’s Office of Nuclear Energy’s Gateway for Accelerated Innovation in Nuclear (GAIN) initiative to provide the nuclear community with access to the technical, regulatory, and financial support necessary to motivate innovative nuclear reactor technologies toward commercialization. New measurements of 35 Cl(n,p total ) and 35 Cl(n,α total ) were completed at LANSCE to constrain the reaction theory models that are used to generate the updated evaluations. The updated evaluations were then tested across the sensitivities of the MCRE by TerraPower to provide direct feedback to the evaluation for application specific sensitivities.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗
DOE OSTI2024

2023 Reftek Wrangler Digitizer Type Approval Evaluation

Sandia National Laboratories has tested and evaluated the Reftek Wrangler digitizer. The Wrangler digitizers are intended to record sensor output for seismic and infrasound monitoring applications. The purpose of this digitizer evaluation is to measure the performance characteristics in such areas as power consumption, input impedance, sensitivity, full scale, self-noise, dynamic range, system noise, response, passband, and timing. The Wrangler digitizers are being evaluated for potential use in the International Monitoring System (IMS) and On-site Inspection (OSI) components of Comprehensive Nuclear-Test-Ban Treaty (CTBT).

58 GEOSCIENCES↗
DOE OSTI2024

Optimization and Evaluation of Energy Savings for Connected and Autonomous Off-Road Vehicles

Off-road vehicles, such as wheel loaders, excavators, and harvesters, are extensively utilized across a wide range of industries, including construction, agriculture, and mining. These machines have become indispensable in supporting the day-to-day operational needs of a nation, playing a critical role in various sectors' infrastructure and productivity. However, despite their utility, off-road vehicles are significant consumers of fossil fuels, resulting in substantial emissions that contribute to environmental degradation. This highlights the pressing need for research and technological advancements aimed at improving their energy efficiency and reducing their carbon footprint. There are, however, two primary challenges that must be addressed to achieve these goals. First, off-road vehicles typically perform both driving and working tasks simultaneously, which introduces a high level of complexity into their overall dynamic systems. Analysis the interactions between these functions is challenging. Second, research into off-road vehicles is inherently interdisciplinary, demanding expertise across several domains such as fluid power systems, vehicle dynamics, control theory, optimization techniques, and real-world implementation. Recognizing these challenges, we proposed the project titled "Optimization and Evaluation of Energy Savings for Connected and Autonomous Off-Road Vehicles" as a comprehensive solution to enhance fuel efficiency while simultaneously improving productivity. This project specifically focuses on autonomous off-road vehicles, with particular attention to wheel loaders, and seeks to develop novel methods to optimize energy consumption without sacrificing operational performance. The project integrates real-time control algorithms, vehicle dynamics modeling, and co-optimization of powertrain system and vehicle system to achieve these goals. Our optimization strategy dynamically co-optimizes critical parameters at both the powertrain and vehicle levels, including vehicle speed, working tool movements, powertrain dynamics, and engine operations in real-time. To streamline this optimization process, we developed a vehicle model that captures the key dynamics while significantly enhancing computational efficiency. This allows the system to intelligently minimize fuel consumption, all while maintaining or even improving productivity through real-time calculations during various off-road operations. To validate the effectiveness of this energy optimization method, we introduced a state-of-the-art Hardware-in-the-Loop (HIL) testbed. This reconfigurable testbed seamlessly integrates the actual engine with virtual models of the wheel loader's subsystems, allowing for accurate emulation of real-world operational loads and environments. By simulating these conditions, the HIL testbed enables us to evaluate the wheel loader’s performance under diverse working scenarios, ensuring the developed solution is applicable in real-world operations. This testbed proved to be instrumental in validating the optimization algorithms and demonstrating the system's practical effectiveness. During the evaluation and testing phase, we employed the HIL testbed to rigorously assess the energy savings and productivity improvements generated by the optimized system. The results were highly encouraging, revealing that the automated wheel loader achieved over 30% fuel savings compared to traditional, human-operated cycles, with comparable or even enhanced levels of productivity. The insights gained from this HIL-based testing provided critical validation of our approach and highlighted the potential for deploying these optimized autonomous technologies in real-world off-road vehicles.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

2023 Reftek Colt Seismometer Exploratory Evaluation

Sandia National Laboratories has tested and evaluated three Colt broadband seismometers designed and manufactured by Reftek. The purpose of this seismometer evaluation is to measure performance characteristics in areas such as power consumption, sensitivity, frequency response, full scale, self-noise, dynamic range, calibration system response, and passband. The Colt model of sensors are being evaluated to explore the potential for a future seismometer Type Approval process in the International Monitoring System (IMS) of the Comprehensive Nuclear-Test-Ban Treaty (CTBT).

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Tables of Neutron Thermal Cross Sections, Westcott Factors, Resonance Integrals, Maxwellian Averaged Cross Sections, Astrophysical Reaction Rates, and r-process Abundances Calculated from the ENDF/B-VIII.1, JEFF-3.3, JENDL-5.0, BROND-3.1, and CENDL-3.2 Evaluated Data Libraries

We present calculations of neutron thermal cross sections, Westcott factors, resonance integrals, Maxwellian-averaged cross sections, astrophysical reaction rates, and solar system r-process abundances using the latest data from the major evaluated nuclear libraries for 849 ENDF target materials. The recent release of ENDF/B-VIII.1 library, progress in 252 Cf(SF) evaluation, extensive analysis of newly-evaluated neutron reaction cross sections, neutron covariances, and improvements in data processing techniques motivated us to calculate the nuclear industry and neutron physics parameters, produce s-process Maxwellian-averaged cross sections and astrophysical reaction rates, extract r-process abundances, systematically calculate uncertainties, and provide additional insights on currently available neutron-induced reaction data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Enabling Evaluation of a Southern Company Distribution Feeder on NREL ADMS Test Bed: Cooperative Research and Development (Final Report)

The objective of this project is to enable evaluation of a Southern Company distribution feeder on the Advanced Distribution Management System (ADMS) test bed. The long-term goal is to evaluate a federated distributed energy resource (DER) management solution that aggregates DERs through either direct control, transactive control or an aggregator to provide bulk services while observing distribution system voltage and power constraints. The DER aggregation needs to be coordinated with an ADMS that is responsible for reliable power delivery across the distribution systems. This project takes the first step towards enabling such evaluation by deploying an ADMS from Oracle (Southern Company's ADMS supplier) with a Southern Company feeder at NREL.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Evaluation of Pozzolan-substituted Cast Stone for Pretreated Hanford Tank Waste

Portland cement. The future availability of quality Class F fly ash has come into question as coal-fired power plants are being converted to gas or taken offline. Consequently, alternative pozzolanic reagents are being evaluated for fly ash replacement in cementitious matrices for treating radioactive waste and debris. In 2023 and 2024, Savannah River National Laboratory (SRNL) was funded to evaluate natural pozzolans for the replacement of Class F fly in the Savannah River Site (SRS) Saltstone formulation. Based on the promising results and uncertain availability of Class F fly ash in the northwestern US, Hanford Tank Waste Operations & Closure, LLC (H2C) requested SRNL to evaluate the potential for substituting natural pozzolans in the Hanford West Area Pretreated Low Activity Tank Waste (PTW/LAW) Cast Stone formulation1 . Task 1 in this request was to determine the change in Cast Stone volume resulting from stabilization/solidification of 1 L of PTW/LAW tank waste. The densities of three (Hess Pumice, tephra pumice, and clinoptilolite zeolite) pozzolan-substituted Baseline Cast Stone mixes were determined, and volume changes relative to the fly ash-Baseline Mix with a water-to-dry mix = 0.40 were calculated.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2024

ORNL FY2024 Nuclear Data Evaluation Contributions: Cu, La, N, and Ta [Slides]

63,65 Cu angular distributions resolved for ENDF/B-VIII.1, consistent with both differential data and integral benchmark performance. 139 La RRR and URR evaluations will be merged with the LANL high energy evaluation, and to be submitted to the future ENDF-B release (post-VIII.1). 14 N RRR evaluation is planned to produce n+ 14 N, p+ 14 C, and a+ 11 B contributions to be submitted to the future ENDF-B release (post-VIII.1). 181 Ta covariances repaired and reported as intended in an errata to ENDF/B-VIII.1.

139-La↗
DOE OSTI2025

Evaluation of LLM-Generated Kokkos Code Using Compile-Time and Run-Time Testing

Due to the growing use of large language models (LLMs) by developers and researchers, it has become essential to reliably evaluate their ability to generate code that uses specialized libraries. We explore the use of compile-time and run-time evaluation of LLM-generated Kokkos code through extending the methods used by OpenAI with the HumanEval dataset. Our evaluation framework is based on the first 40 prompts from the Kokkos138 dataset. We start by discussing two different forms of LLM prompting, using entirely plain English or providing pseudocode for added context. These two methods are used to generate Kokkos code with the Llama-3.1-8B-Instruct and CodeQwen1.5-7B-Chat models. We found that both forms of prompting led to high failure rates and difficulties with reliably parsing LLM-generated code, while prompts with pseudocode for context generally led to improved results on more complicated tests.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Long-Term Evaluation of Remedial Technology Performance in the Laboratory: Multi-year Experimental Test Plan

An understanding of the long-term effectiveness of remediation technologies is central to sustainable environmental cleanup. Long-term experiments are valuable for reducing uncertainty and predicting remediation outcomes at scales required for regulatory compliance. However, these types of tests can be costly and challenging to interpret. Therefore, contaminated sites often rely on short-term laboratory experiments that may not account for the potentially significant effects of gradual, time- dependent processes governing contaminant retention, release, and species transformation. For example, short-term lab experiments (from months to a year) conducted with sediments from the unsaturated and saturated zones at the Hanford Site play an important role in initial evaluations of remediation technologies but cannot capture the full extent of time-dependent reactions, leaving uncertainties in field-scale deployment. Here, long-term testing will be conducted on select technologies based on their performance in short-term testing. The overall objective is to directly address the challenges described above by generating and analyzing data on long-term (2–10 years) efficacy of selected remediation technologies, integrating this understanding into models, and providing critical input for remediation planning, monitoring, and 5-year review cycles for field-implemented remedies. Specific objectives include: 1. Evaluating long-term efficiency of promising remedies under site-specific conditions. 2. Generating robust parameters for modeling, reducing uncertainty in predictive simulations. 3. Advancing integrated monitoring by combining geochemical and geophysical observations. 4. Informing field-scale implementation by incrementally advancing technologies, identifying failure mechanisms early, and prioritizing robust, cost-effective remedies. Through systematic evaluation of technologies in laboratory-scale column experiments, integrated monitoring, and modeling support, this project is designed to bolster confidence in the long-term robustness of selected technologies. The ultimate outcome is the identification and deployment of more reliable, cost-effective remedies that safeguard human health and the environment while reducing the uncertainties that have historically hindered cleanup progress at Hanford Site. An experimental approach was developed and initiated for long-term testing potentially up to 10 years. The table below summarizes the experimental approach developed for testing select technologies and presented in this multi-year experimental test plan.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Optimizing Alabama’s CO 2 Storage in Shelby County (Project OASIS) Milestone 6.0: Evaluation of Class VI Readiness

Introduction. Project OASIS is approximately 30 miles southeast of Birmingham, Alabama, and approximately 5 miles north-northwest of Alabama Power Company's Plant Gaston. Geologically, the Project area is in the Alabama fold and thrust belt province. This work builds on the initiatives of the Southeast Regional Carbon Utilization and Storage Acceleration Partnership (SECARB-USA, DE-FE0031830) that identified nearly 500 million metric tonnes of CO 2 emitted on an annual basis that is not collocated with prospective storage geology (the Coastal Plain of the Southeastern US in this context). This observation suggests costly investments in connective infrastructure (e.g., pipelines) or exploratory well drilling campaigns to identify CO 2 storage opportunities in under explored areas. While not traditionally thought of for saline storage, these studies suggest that storage prospects in the Valley and Ridge Province occur in relatively flat lying structural panels between thrust faults. For the Project OASIS region, available geologic studies related to hydrocarbon exploration suggest that Cambro-Ordovician carbonates and Cambrian clastic units offer multiple potential storage intervals, and that regional confining systems are present, such as the tectonically thickened Floyd-Parkwood Shale. The Project OASIS surface property is owned by a timber and land stewardship company, The Westervelt Company, Inc., who worked with the Project Team to select and prepare adequate sites for geologic assessment. The purpose of drilling the Westover Stratigraphic Test Well #2 was to collect geologic data to model the feasibility of commercial scale CO 2 injection and storage in an under explored region. This initiative benefits the regions emitters as the data generated from this study can inform their own internal decision making. The field program included geological and geophysical evaluations, reservoir engineering analyses, and risk assessments. This report evaluates existing data, as well as a variety of modeling scenarios to evaluate project readiness. Importantly, the impact of this study is not limited to Alabama as there are numerous large emitters throughout Appalachia, in similar geologic settings, contemplating their decarbonization options.

20 FOSSIL-FUELED POWER PLANTS↗
DOE OSTI2025

Scalable workflow for evaluating and optimizing large language models

This work describes the improved workflow for evaluating open-source large language models (LLMs) for trustworthiness. The workflow facilitates the acquisition of LLMs, the generation of LLM responses, and the evaluation of the responses for their trustworthiness. As a use case, the workflow is employed to evaluate dense, quantized, and pruned Meta Llama3.1 LLMs for their truthfulness. The outcome of the project could set the stage for understanding and developing trustworthy models in the future projects.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

NCSP supports 240 Pu prompt fission neutron spectrum (PFNS) evaluation

A new 240 Pu PFNS evaluation was recently undertaken at LANL as a strategic priority. It truly is an NCSP end-to-end product. It factors in a new differential experiment funded by NCSP, builds on theoretical work coming out of a previous NCSP nuclear data evaluation milestone and was validated with an NCERC experiment that was recently evaluated as an integral benchmark with NCSP funds.

240Pu↗