Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2014

Benchmarking the performance of density functional theory and point charge force fields in their description of sI methane hydrate against diffusion Monte Carlo

High quality reference data from diffusion Monte Carlo calculations are presented for bulk sI methane hydrate, a complex crystal exhibiting both hydrogen-bond and dispersion dominated interactions. The performance of some commonly used exchange-correlation functionals and all-atom point charge force fields is evaluated. Our results show that none of the exchange-correlation functionals tested are sufficient to describe both the energetics and the structure of methane hydrate accurately, while the point charge force fields perform badly in their description of the cohesive energy but fair well for the dissociation energetics. By comparing to ice I{sub h}, we show that a good prediction of the volume and cohesive energies for the hydrate relies primarily on an accurate description of the hydrogen bonded water framework, but that to correctly predict stability of the hydrate with respect to dissociation to ice I{sub h} and methane gas, accuracy in the water-methane interaction is also required. Our results highlight the difficulty that density functional theory faces in describing both the hydrogen bonded water framework and the dispersion bound methane.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2014

Benchmarking all-atom simulations using hydrogen exchange

Molecular dynamics simulations have recently become capable of observing multiple protein unfolding and refolding events in a single-millisecond–long trajectory. This major advance produces atomic-level information with nanosecond resolution, a feat unmatched by experimental methods. Such simulations are being extensively analyzed to assess their description of protein folding, yet the results remain difficult to validate experimentally. We apply a combination of hydrogen exchange, NMR, and other techniques to test the simulations with a resolution of single H-bonds. Several significant discrepancies between the simulations and experimental data were uncovered for regions of the energy surface outside of the native basin. This comparison yields suggestions for improving the force fields and provides a general method for experimentally validating folding simulations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Initial Efforts Organizing WPNCS SG-8: Preservation of Expert Knowledge and Judgement Applied to Criticality Benchmarks

The Working Party on Nuclear Criticality Safety (WPNCS) under the guidance of the Organization for Economic Co-operation and Development (OECD) Nuclear Energy Agency (NEA) has over 20 years of experience addressing concerns related to static and transient configurations encountered within the nuclear fuel cycle: fuel fabrication, transportation, reprocessing, storage, and geological disposal. One of the cornerstone activities of the WPNCS is the International Criticality Safety Benchmark Evaluation Project (ICSBEP), which was established to identify a comprehensive set of criticality benchmark data, evaluate the data, including quantification of overall uncertainties; compile the data into a standardized format, perform sample calculations utilizing modern nuclear data sets and codes utilized in nuclear criticality safety, and formally document the work into a single source of verified benchmark data. Annually, members of the ICSBEP Technical Review Group (TRG) contribute evaluated benchmark data that undergoes comprehensive technical review prior to publication in the ICSBEP Handbook. In the years since the ICSBEP was established, there has been much work to prepare benchmark data to support validation activities in nuclear criticality safety. The 2020 edition of the ICSBEP Handbook contains acceptable benchmark specifications for 5,053 critical, subcritical, or near-critical configurations in 582 benchmark evaluations. Modern benchmark development benefits from decades of experienced international participants, a well-established handbook format, supplementary guides to deal with uncertainty quantification, and a comprehensive review process based upon independent reviews from international experts. The ICSBEP Handbook also contains 838 configurations deemed unacceptable to support criticality safety efforts. They are recorded, with the reasoning for their rejection, to preserve the experimental data, prevent reevaluation of data that are incomplete or contain known errors, and/or to potentially allow future reevaluation of the experiment pending the identification of sufficient data to resolve identified inconsistencies and errors. Users of the ICSBEP Handbook today might notice that the rigor and quality of modern criticality safety benchmarks is much greater than those prepared within the initial decade of the project. Benchmarks with 1s uncertainties in k eff greater than 1% were traditionally rejected unless they were identified as unique experiment types that encompassed materials, fuels, or designs not available from other benchmark experiments. However, benchmarks developed using modern experimental techniques and practices typically have uncertainties on the order of a few tenths of a percent. There have been ongoing efforts to improve the overall quality of previously published benchmark evaluations. Seventy-eight evaluations, containing approximately 600 configurations, have been revised just within the past decade. An additional eleven benchmarks are under revision for updated release in the 2020 edition of the ICSBEP Handbook. If some of the historic benchmarks were resubmitted in their current form to the TRG today, they would be rejected due to lack of data, missing components in the uncertainty analysis, or incomplete benchmark model development. The use of historic criticality safety benchmarks that underestimate the total uncertainty, lack properly quantified biases, or provide inadequate benchmark specifications do not sufficiently support modern criticality safety and nuclear data efforts. Although the ICSBEP Handbook is recognized by regulating bodies to support criticality safety, users are required to justify their reasons to ignore historic benchmark data and include additional safety margins within their designs. Discussions were held at the WPNCS 23rd Annual Meeting in September 2019 regarding the aforementioned issues. The resultant decision was to establish Subgroup 8 (SG-8): Preservation of Expert Knowledge and Judgement Applied to Criticality Benchmarks. The current activities of SG-8 are discussed herein.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Uniformly Ordered Binary Decision Algorithm for Benchmark Experiment Correlations in Whisper Validation

When performing a validation exercise for determining the upper subcritical limit of a nuclear criticality safety application, an analyst should select and perform a statistical analysis on a population of benchmark experiments that are neutronically similar to the application. The size of this population should be sufficiently large such that the statistical analysis has a high degree of confidence that the bias plus bias uncertainty (calculational margin) has been accurately quantified. A complication arises because many benchmark experiments share common components, leading to correlations in their measured effective multiplication factors. Correlations between benchmark experiments within the population reduces its predictive power. This motivates the need for methods that consider benchmark experiment correlations and ensure adequate statistical significance of results. The Whisper code is a statistical analysis pack- age that incorporates nuclear data sensitivity coefficients from MCNP to assess benchmark experiment similarity and then performs an extreme-value analysis to estimate the bias plus bias uncertainty. The original methodology in Whisper does not consider the effect of benchmark experiment correlations when making this estimation, and this summary proposes the uniformly ordered binary decision algorithm to address this shortcoming. The original methodology in Whisper computes similarity coefficients ck for an application compared to all benchmark experiments in its library and develops weighting factors for a selected population proportional to the ck values. The methodology can be interpreted as statistically emulating a validation exercise for a particular application where the weighting factors may be viewed as the likelihood that an analyst would include a particular benchmark experiment within the population. The effective sample size of the population is the expected or mean number of benchmark experiments in the population. The uniformly ordered binary decision algorithm identifies clusters of correlated benchmark experiments within the population and then computes adjusted weighting factors based on the magnitude of the correlation coefficients within the cluster to compute a reduced effective sample size accounting for the lower information content because of correlations. Benchmark experiments within the cluster are ordered randomly with equal probability and probabilistic decisions are made as to whether a benchmark. experiment within the cluster should treated as redundant with a previous one; if two redundant benchmark experiments are included, then the conservative worst case bias plus bias uncertainty is used and the pair is counted as a single benchmark experiment in the population. Results are provided for HEU solutions in a research version of the Whisper software using benchmark experiment correlations provided by DICE, the Database for the International Criticality Safety Benchmark Evaluation Project (ICSBEP). These show that there can be a significant increase in the bias plus bias uncertainty because the effective sample size is reduced, and therefore the algorithm, needing to meet sample size requirements, expands the benchmark experiment population by accepting less similar benchmark experiments that would have otherwise not been included.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2023

U.S. Solar Photovoltaic System and Energy Storage Cost Benchmarks, With Minimum Sustainable Price Analysis: Q1 2023

The U.S. Department of Energy's (DOE's) Solar Energy Technologies Office (SETO) aims to accelerate the advancement and deployment of solar technology in support of an equitable transition to a decarbonized economy no later than 2050, starting with a decarbonized power sector by 2035. Its approach to achieving this goal includes driving innovations in technology, hardware, and soft cost reductions to make solar affordable and accessible for all. As part of this effort, SETO must track solar cost trends so it can focus its research and development (R&D) on the highest-impact activities. The benchmarks in this report are bottom-up cost estimates of all major inputs to PV and energy storage system installations. Bottom-up costs are based on national averages and do not necessarily represent typical costs in all local markets. Like last year's report, this year's report includes two distinct sets of benchmarks: minimum sustainable price (MSP) benchmarks and modeled market price (MMP) benchmarks. MSP benchmarks can be interpreted as the minimum price a company needs to charge to remain financially solvent in the long term based on the minimum sustainable prices of all inputs including minimum sustainable profit margins. MMP benchmarks can be interpreted as the actual cash sales price a company charges in the given benchmark period. These simplified estimates are useful for tracking technological progress, but they do not reflect all experiences. In fact, no individual estimate under any approach can reflect the diversity of the PV and storage manufacturing and installation industries. Our residential MMP benchmark ($2.90 per watt direct current [Wdc]) is 24% higher than the MSP benchmark ($2.34/Wdc) and 9% lower than our MMP benchmark ($3.18/Wdc) from Q1 2022 in 2022 U.S. dollars (USD). For community solar, our MMP benchmark ($1.75/Wdc) is 18% higher than our MSP benchmark ($1.49/Wdc). Our Q1 2022 benchmark report has no community solar system for comparison. For utility-scale systems with one-axis tracking, our MMP benchmark ($1.17/Wdc) is 22% higher than our MSP benchmark ($0.96/Wdc) and 10% higher than its counterpart ($1.07/Wdc) in Q1 2022 in 2022 USD.

14 SOLAR ENERGY↗
DOE OSTI2012

Benchmarking ENDF/B-VII.1, JENDL-4.0 and JEFF-3.1.1 with MCNP6

Recent releases of three major world nuclear reaction data libraries, ENDF/B-VII.1, JENDL-4.0, and JEFF-3.1.1, have been tested extensively using benchmark calculations. The calculations were performed with the latest release of the continuous energy Monte Carlo neutronics code MCNP, i.e. MCNP6. Three types of benchmarks were used, viz. criticality safety benchmarks, (fusion) shielding benchmarks, and reference systems for which the effective delayed neutron fraction is reported. For criticality safety, more than 2000 benchmarks from the International Handbook of Criticality Safety Benchmark Experiments were used. Benchmarks from all categories were used, ranging from low-enriched uranium, compound fuel, thermal spectrum ones (LEU-COMP-THERM), to mixed uranium-plutonium, metallic fuel, fast spectrum ones (MIX-MET-FAST). For fusion shielding many benchmarks were based on IAEA specifications for the Oktavian experiments (for Al, Co, Cr, Cu, LiF, Mn, Mo, Si, Ti, W, Zr), Fusion Neutronics Source in Japan (for Be, C, N, O, Fe, Pb), and Pulsed Sphere experiments at Lawrence Livermore National Laboratory (for {sup 6}Li, {sup 7}Li, Be, C, N, O, Mg, Al, Ti, Fe, Pb, D2O, H2O, concrete, polyethylene and teflon). The new functionality in MCNP6 to calculate the effective delayed neutron fraction was tested by comparison with more than thirty measurements in widely varying systems. Among these were measurements in the Tank Critical Assembly (TCA in Japan) and IPEN/MB-01 (Brazil), both with a thermal spectrum, two cores in Masurca (France) and three cores in the Fast Critical Assembly (FCA, Japan), all with fast spectra. The performance of the three libraries, in combination with MCNP6, is shown to be good. The results for the LEU-COMP-THERM category are on average very close to the benchmark value. Also for most other categories the results are satisfactory. Deviations from the benchmark values do occur in certain benchmark series, or in isolated cases within benchmark series. Such instances can often be related to nuclear data for specific non-fissile elements, such as C, Fe, or Gd. Indications are that the intermediate and mixed spectrum cases are less well described. The results for the shielding benchmarks are generally good, with very similar results for the three libraries in the majority of cases. Nevertheless there are, in certain cases, strong deviations between calculated and benchmark values, such as for Co and Mg. Also, the results show discrepancies at certain energies or angles for e.g. C, N, O, Mo, and W. The functionality of MCNP6 to calculate the effective delayed neutron fraction yields very good results for all three libraries.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

AI Benchmark Democratization and Carpentry

Benchmarks are a cornerstone of modern machine learning, enabling reproducibility, comparison, and scientific progress. However, AI benchmarks are increasingly complex, requiring dynamic, AI-focused workflows. Rapid evolution in model architectures, scale, datasets, and deployment contexts makes evaluation a moving target. Large language models often memorize static benchmarks, causing a gap between benchmark results and real-world performance. Beyond traditional static benchmarks, continuous adaptive benchmarking frameworks are needed to align scientific assessment with deployment risks. This calls for skills and education in AI Benchmark Carpentry. From our experience with MLCommons, educational initiatives, and programs like the DOE's Trillion Parameter Consortium, key barriers include high resource demands, limited access to specialized hardware, lack of benchmark design expertise, and uncertainty in relating results to application domains. Current benchmarks often emphasize peak performance on top-tier hardware, offering limited guidance for diverse, real-world scenarios. Benchmarking must become dynamic, incorporating evolving models, updated data, and heterogeneous platforms while maintaining transparency, reproducibility, and interpretability. Democratization requires both technical innovation and systematic education across levels, building sustained expertise in benchmark design and use. Benchmarks should support application-relevant comparisons, enabling informed, context-sensitive decisions. Dynamic, inclusive benchmarking will ensure evaluation keeps pace with AI evolution and supports responsible, reproducible, and accessible AI deployment. Community efforts can provide a foundation for AI Benchmark Carpentry.

von Laszewski, Gregor [Virginia U.]↗
DOE OSTI2018

Policy, Permit, Perform: Using City Benchmarking Data and Building Construction Permit History to Identify Energy Performance Improvements

The City of Portland (City) and Multnomah County 2015 Climate Action Plan targets a 40 percent reduction in carbon emissions below 1990 levels by 2030. To reach this goal, the plan includes a key objective to reduce the total energy use of existing buildings by 25 percent. Buildings are responsible for one-half of carbon emissions in Portland, Oregon, U.S., and improving their performance is critical to achieving the City’s climate goals. As part of its climate action planning, the City passed an ordinance on April 22, 2015, requiring commercial buildings 20,000 ft2 and larger to benchmark and disclose annual energy performance metrics through ENERGY STAR® Portfolio Manager® (ESPM), including greenhouse gas emissions, building energy use intensity (EUI) and ENERGY STAR scores. Twenty-five cities in the U.S. have adopted similar policies requiring energy benchmarking. These policies are generating a treasure trove of data about the drivers of building performance, but practitioners are only beginning to link benchmarking data to other datasets to produce actionable insight into improving building performance. In 2016, the City received a financial assistance award from the U.S. Department of Energy (DOE) through the Cities Leading through Energy Analysis and Planning project (Award Number DE-0007737). Portland’s project evaluated the application of the DOE Building Energy Asset Score Tool and rating system developed by Pacific Northwest National Laboratory (PNNL) to commercial buildings that report to the City. The City used building construction permit history to develop Asset Scores for a set of office buildings that reported energy performance for 2015 – the first year the City mandated energy benchmarking and reporting for commercial buildings 50,000 ft2 and larger. The primary project objective was to link energy benchmarking data and commercial building permit data to analyze commercial buildings systems that present the best, specific opportunities to improve energy performance. Integrating these datasets from two City departments, the Bureau of Development Services and the Bureau of Planning and Sustainability, could expand Portland’s use of analytics to inform climate policy decisions and strategic targeting of financial incentives for building owners and managers who are best positioned to improve the performance of their buildings. The project began with the commercial benchmarking data received for 340 commercial buildings that submitted calendar year 2015 energy reports. The City retained Research Into Action and Energy 350 (RIA team) to evaluate this self-reported benchmarking data. Based on a survey of 53 building managers, the RIA team found that most respondents used the City’s compliance tools, including a How-to Guide and Help Desk, and found access to utility data to be straightforward. However, almost one-third of the respondents encountered challenges when estimating the gross floor area (GFA), and some experienced confusion regarding the input of property use types and details into ESPM. Based on the RIA team’s recommendations, the City refined its How-to Guide, created online answers to frequently asked questions, and hosted office hour sessions to help building managers accurately characterize their building’s physical characteristics and operations in ESPM. Concurrently with the RIA team’s evaluation, the City worked with PNNL to estimate DOE Asset Score Preview (Preview) scores for 181 of the 340 commercial buildings that submitted ESPM reports to the City. Asset Score Tool is a web-based tool developed by PNNL for DOE to evaluate and rate the as-built energy efficiency of a building. Asset Score Tool’s full analysis capability runs a detailed energy model based on building geometry, age, envelope, lighting and mechanical systems using standard operating assumptions to give the building a score, similar to a miles per gallon rating for a car. Preview, a simplified deployment of the full Asset Score Tool, provides a preliminary score range using only seven data points, six of which are readily available in the ESPM benchmarking data. Analysis of the combined Preview scores and the operational ESPM scores identified buildings with a high potential for energy improvements through either retro-commissioning and retuning of existing operations or capital investments into building systems. The City contracted Whole Building Solutions (WBS) to mine building permit history and develop full Asset Scores for a subset of 26 office buildings that were eligible for both ENERGY STAR scores and Asset Scores. WBS researched permits and building plans to identify mechanical, lighting and envelope systems and other building characteristics necessary to complete Asset Scores. This historical data was recorded using the web-based Asset Score Tool, which evaluates the as designed energy efficiency of existing buildings. Much of this City information was available online after 2010; however, WBS had to search through microfiche files and the City’s database to obtain permit and plan history prior to 2010. Although building geometry, mechanical system type and basic envelope information was consistently identifiable, WBS found permit data gaps that prevented complete and accurate Asset Scores for most of the office buildings. If the City decides to develop Asset Scores based on building permit information in the future, WBS recommended requiring lighting plans, control diagrams and as-built drawings that reflect how the buildings are actually constructed. In addition, WBS recommended uploading plans and permits from microfiche storage into a more comprehensive, searchable database consistent with the City’s current online permit information. To identify the actual lighting types, WBS and City staff conducted site visits of all 26 office buildings. As part of managing the flow of ESPM and Asset Score data for the project, Earth Advantage, a sub-recipient of the Cities-LEAP award, created a DOE Standard Energy Efficiency Data (SEED) (USDOE 2018) instance for the City and used Asset Score’s Application Programming Interface (API) (Asset Score 2018) capabilities to develop two new features in the SEED Platform™: (1) automated Asset Score Preview ranges calculated based on benchmarking data and the City’s estimate of number of building floors; and (2) storage of full Asset Score information. Leveraging the construction permit history and site visit information for the 26 office buildings, The Cadmus Group (Cadmus) analyzed benchmarking data and Asset Score information to identify opportunities and drivers to improve performance. Correlation and regression analysis of the combined dataset was completed to determine whether building operations or specific systems – mechanical, lighting or envelope – present the best opportunities to improve energy performance. The central outcome of this project was Cadmus’ detailed analysis of the richer dataset that was created by combining the City of Portland’s commercial building permit activity and building-specific energy performance information based on metrics from ESPM and Asset Score Tool. Some of their findings collaborate with well-known facts, such as buildings with less efficient lighting systems are good targets for efficiency improvements through lighting upgrades. This analysis also helped identify that individual heating, ventilation, and air conditioning (HVAC) equipment efficiency does not necessarily correlate with building performance. However, overall HVAC system performance, as measured with the Total System Performance Ratio (TSPR) metric, is a better indicator of building performance (Goel et al. 2014). As a metric, TSPR considers whole system performance that includes equipment efficiency and system controls, such as resets for static pressure and chilled water setpoints, which can be more effective and financially feasible options for improving building performance. Given the high cost associated with envelope retrofits, opportunities to upgrade envelope insulation are likely to be limited. However, buildings with single-pane and non-thermal break windows are good candidates for window replacements, such as upgrading to double-pane windows with thermal breaks. This research project determined that developing Asset Scores based on site visits is preferable to mining City building permit data. In addition, the use of benchmarking data to develop Asset Score Preview Scores is recommended as a first step in engaging building managers to develop full Asset Scores. Asset Score Tool outputs can help building managers quickly identify opportunities for improvement and then work with the utility incentive administrator and energy service providers to develop their retrofit plan. Where full Asset Scores are available, analysis using a quadrant matrix with ENERGY STAR scores is recommended to identify buildings that present the best opportunities for operational-behavioral improvements versus ones with opportunities for upgrades to physical building systems. This workflow and the tools will help cities, utility incentive administrators, building portfolio managers and energy service providers develop energy programs and strategies for screening and analysis of buildings with the greatest energy savings potential. This report describes the project approach and research results to inform future coordination of ESPM and Asset Score data, including the use of benchmarking data to generate Preview scores. Given better awareness of energy performance, building managers can make more informed decisions to reduce energy consumption and carbon emissions. With cities, states and jurisdictions adopting benchmarking and auditing ordinances, there are substantial amounts of data submitted to cities that could be used to inform investments in energy efficiency.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2022

Health Physics Research Reactor Criticality Accident Alarm System Benchmark Overview

From the countless critical experiments performed in the world during the past century, high-quality integral benchmarks experiments have been collected and gathered into the International Handbook of Evaluated Criticality Safety Benchmark Experiments (ICSBEP Handbook), managed by the International Criticality Safety Benchmark Evaluation Project (ICSBEP) Working Group. This information preservation and dissemination effort is crucial for reactor licensing as well as criticality and radiation transport modeling validation. This summary reports on the status of a tentative benchmark addition to the ICSBEP Handbook. The proposed benchmark arises from legacy operation data of the Oak Ridge National Laboratory (ORNL) Health Physics Research Reactor (HPRR). The HPRR was a small, unmoderated, unshielded fast burst reactor that was used for research in health physics and radiobiology as well as teaching and training. As part of a comprehensive investigation of the available HPRR operation data and characteristics, different possibilities for use of the valuable results were studied. A critical experiment benchmark evaluation was performed, analyzing data coming from sub-critical and critical operation of the HPRR during operator training, steady-state irradiation of samples and before critical bursts. The results of the evaluation do not satisfy for the ICSBEP standards as the benchmark relative standard uncertainty is of about 4% for k eff , and the relative difference between sample calculations and expected k eff results is of about 1.5%. Due to those unsatisfactory results, it was decided not to pursue critical experiments evaluation of the HPRR presently and to focus instead on shielding type data for the creation of a criticality accident alarm system (CAAS) and shielding category benchmark, which is currently very scarce in the ICSBEP handbook—especially concerning critical, pulsed assembly, or reactor operation data. Several dosimetry and shielding experiments from HPRR burst operation were evaluated, with different benchmark metrics as sulfur fluence, Element 57 dose, or neutron fluence at different distances and under different shield materials conditions. An evaluation focusing on the Element 57 neutron dose as a benchmark metric was submitted to the ICSBEP Technical Review Group (TRG) meeting in October 2021, and the inclusion of the evaluation in the ICSBEP Handbook was deferred. The main change proposed by the international experiment evaluation experts is to use the neutron fluence measured by Bonner spheres as a benchmark metric. This represents a quantity closer to that actually measured by the experimentalists of the HPRR compared to the Element 57 dose, which adds another step of data transformation, thus potentially adding uncertainty to the benchmark. The evaluation has been updated and will be submitted to the 2022 ICSBEP TRG meeting for inclusion in the 2023 edition of the ICSBEP Handbook. The evaluation is performed using the KENO and MAVRIC combination from the SCALE 6.2.4 code suite which was previously used in similar CAAS benchmarks to allow for the use of variance reduction techniques.

61 RADIATION PROTECTION AND DOSIMETRY↗