Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

Evaluation of the Pool Critical Assembly Benchmark with Explicitly-Modeled Geometry using MCNP6

Despite being one of the most widely used benchmarks for qualifying light water reactor (LWR) radiation transport methods and data, no benchmark calculation of the Oak Ridge National Laboratory (ORNL) Pool Critical Assembly (PCA) pressure vessel wall benchmark facility (PVWBF) using MCNP6 with explicitly modeled core geometry exists. As such, this paper provides results for such an analysis. First, a criticality calculation is used to construct the fixed source term. Next, ADVANTG-generated variance reduction parameters are used within the final MCNP6 fixed source calculations. These calculations provide unadjusted dosimetry results using three sets of dosimetry reaction cross sections of varying ages (those packaged with MCNP6, from the IRDF-2002 multi-group library, and from the ACE-formatted IRDFF v1.05 library). These results are then compared to two different sets of measured reaction rates. The comparison agrees in an overall sense within 2% and on a specific reaction- and dosimetry location-basis within 5%. Except for the neptunium dosimetry, the individual foil raw calculation-to-experiment comparisons usually agree within 10% but is typically greater than unity. Finally, in the course of developing these calculations, geometry that has previously not been completely specified is provided herein for the convenience of future analysts.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

An Analytic Benchmark for Neutron Boltzmann Transport with Downscattering—Part I: Flux and Eigenvalue Solutions

Computing in the energy dimension is one of the greatest challenges confronting present-day deterministic neutron transport solvers. Accurately resolving the neutron flux as neutrons downscatter across resonances in the nuclear cross sections currently requires considerable computing power and suffers from approximation errors. Flux uncertainty resulting from the uncertainty of the resonance structure is the single-largest cause of reactivity uncertainty. Any additional reference solution for the critical neutron downscattering problem with resonance phenomena would be a boon to verification and validation of neutronics codes. This paper establishes a benchmark to verify the accuracy of neutron transport criticality solvers along the energy dimension. For the first time, the analytic solution of the flux amplitude is derived in the particular case of an infinite homogeneous medium with isotropic scattering in the center of mass and an arbitrary number of no-threshold, neutral particle reaction resonances (e.g., radiative capture, fission, and resonance scattering). Furthermore, original analytic expressions are established to quantify the discrepancy between the ψ k (E) and ψ α (E) flux amplitudes, respective solutions of the multiplication factor k, or the exponential time-evolution frequency α eigenproblems. The physical study of these relations led to analysis of their first-order relative difference near the criticality condition α=0. Finally, numerical solutions are provided to a benchmark problem constituted of the first resonance of 239 Pu, the 6.67-eV resonance of 238 U, and a scattering isotope with a flat cross section, allowing for the computational verification of the energy resolution of current neutron transport criticality codes. Through these novel results, this analytic benchmark can serve as a reference to verify the energy resolution and sensitivity analysis of neutron transport criticality calculations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Benchmark of Neutron Thermalization in Graphite Using a Pulsed Slowing-Down-Time Experiment

A benchmark has been developed using a pulsed slowing-down-time experiment to isolate the thermalization process in graphite. The experiment was conducted at the Oak Ridge Electron Linear Accelerator facility at Oak Ridge National Laboratory, and it measured the time spectrum of neutrons leaking from a graphite pile during slowing down and thermalization within graphite. Simulations of the benchmark experiment were performed using the MCNP6.1 Monte Carlo code and the ENDF/B-VII.1 and ENDF/B-VIII.0 cross-section databases. The benchmark provides a time spectrum (i.e., time-dependent counts in a detector) that allows for validation of the graphite thermal scattering libraries (TSLs). The impact on the simulations using a suite of graphite TSLs was compared with the experimental results. Given the density of nuclear graphite, the TSL corresponding to graphite with 30% porosity, as implemented in ENDF/B-VIII.0, was found to most accurately represent the measured time spectrum corresponding to the thermal energy range with an average deviation of ±1.7%.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Benchmark of the Chlorine Worth Study Experiments in Support of Chlorine Nuclear Data Validation for Nuclear Criticality Safety

The Chlorine Worth Study (CWS) was a critical experiment to address an urgent need for thermal chlorine nuclear data validation in plutonium systems. This urgent need is tied directly to plutonium recycle and recovery operations in the plutonium facility at Los Alamos National Laboratory, where exceptionally conservative criticality safety limits are used because no credit is taken for the neutron capture by chlorine. The experiment used weapons-grade plutonium metal plates clad in stainless steel, known as the PANN (plutonium aluminum no nickel) ZPPR (zero power physics reactor) plates. The plutonium was reflected and moderated by high-density polyethylene and included combinations of polyvinyl chloride (PVC) and chlorinated polyvinyl chloride (CPVC) as absorbers. The experiment and benchmark included three configurations mimicking 30 g 239 Pu/L plutonium, 300 g 239 Pu/L plutonium, and 600 g 239 Pu/L plutonium in an aqueous chloride solution. Uncertainties in the benchmark included five broad categories: (1) criticality measurement, (2) mass and density, (3) dimensions, (4) material compositions, and (5) positioning. The largest contribution to the overall uncertainties for all three cases came from the material compositions, in particular the PVC and CPVC absorber compositions. A detailed model was created to be a near match (that is within expectations of transport code users) and a simplified model was created to minimize offset dimensions and expedite modeling for code validation. Sample calculations were completed in MCNP6.3 with ENDF/B-VIII.0 and ENDF/B-VII.1 nuclear data. For the detailed and simplified models, the average difference between the computed and experimental k eff was 951 pcm. CWS will serve as the key validation experiment for nuclear criticality safety in support of aqueous chloride operations. The sensitivity to the chlorine capture cross section is orders of magnitude greater than other existing benchmarks. The current limits, as defined by nuclear criticality safety, are 520 g Pu per batch, i.e. the minimum critical mass of the Pu solution infinitely reflected by water [Criticality Handbook: Volume II, (1969)]. This extremely conservative critical mass limit does not credit any neutron capture by chlorine (in particular neutron capture by 35 Cl) and greatly impedes the throughput required for current and future operations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

2D axial-azimuthal particle-in-cell benchmark for low-temperature partially magnetized plasmas

The increasing need to demonstrate the correctness of computer simulations has highlighted the importance of benchmarks. We define in this paper a representative simulation case to study low-temperature partially-magnetized plasmas. Seven independently developed particle-in-cell codes have simulated this benchmark case, with the same specified conditions. The characteristics of the codes used, such as implementation details or computing times and resources, are given. First, we compare at steady-state the time-averaged axial profiles of three main discharge parameters (axial electric field, ion density and electron temperature). We show that the results obtained exhibit a very good agreement within 5% between all the codes. As E x B discharges are known to cause instabilities propagating in the direction of electron drift, an analysis of these instabilities is then performed and a similar behaviour is retrieved between all the codes. A particular attention has been paid to the numerical convergence by varying the number of macroparticles per cell and we show that the chosen benchmark case displays a good convergence. Detailed outputs are given in the supplementary data, to be used by other similar codes in the perspective of code verification.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Benchmark for two-dimensional large scale coherent structures in partially magnetized E × B plasmas—community collaboration & lessons learned

Low-temperature plasmas (LTPs) are essential to both fundamental scientific research and critical industrial applications. As in many areas of science, numerical simulations have become a vital tool for uncovering new physical phenomena and guiding technological development. Code benchmarking remains crucial for verifying implementations and evaluating performance. This work continues the Landmark benchmark initiative, a series specifically designed to support the verification of LTP codes. In this study, seventeen simulation codes from a collaborative community of nineteen international institutions modeled a partially magnetized E × B Penning discharge. The emergence of large scale coherent structures, or rotating plasma spokes, endows this configuration with an enormous range of time scales, making it particularly challenging to simulate. The codes showed excellent agreement on the rotation frequency of the spoke as well as key plasma properties, including time-averaged ion density, plasma potential, and electron temperature profiles. Achieving this level of agreement came with challenges, and we share lessons learned on how to conduct future benchmarking campaigns. Comparing code implementations, computational hardware, and simulation runtimes also revealed interesting trends, which are summarized with the aim of guiding future plasma simulation software development.

benchmarking↗

Results of the GABLS3 diurnal-cycle benchmark for wind energy applications

We present results of the GABLS3 model intercomparison benchmark revisited for wind energy applications. The case consists of a diurnal cycle, measured at the 200-m tall Cabauw tower in the Netherlands, including a nocturnal low-level jet. The benchmark includes a sensitivity analysis of WRF simulations using two input meteorological databases and five planetary boundary-layer schemes. A reference set of mesoscale tendencies is used to drive microscale simulations using RANS k-ϵ and LES turbulence models. The validation is based on rotor-based quantities of interest. Cycle-integrated mean absolute errors are used to quantify model performance. The results of the benchmark are used to discuss input uncertainties from mesoscale modelling, different meso-micro coupling strategies (online vs offline) and consistency between RANS and LES codes when dealing with boundary-layer mean flow quantities. Altogether, all the microscale simulations produce a consistent coupling with mesoscale forcings.

17 WIND ENERGY↗

Phase-covariant quantum benchmarks

We give a quantum benchmark for teleportation and quantum storage experiments suited for pure and mixed test states. The benchmark is based on the average fidelity over a family of phase-covariant states and certifies that an experiment cannot be emulated by a classical setup, i.e., by a measure-and-prepare scheme. We give an analytical solution for qubits, which shows important differences with standard state estimation approach, and compute the value of the benchmark for coherent and squeezed states, both pure and mixed.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Quantum benchmarks for the storage or transmission of quantum light from minimal resources

We investigate several recently published benchmark criteria for storage or transmission of continuous-variable quantum information. A comparison reveals that criteria based on a Gaussian distribution of coherent states are most resilient to noise. We then address the issue of experimental resources and derive an equally strong benchmark, solely based on three coherent states and homodyne detection. This benchmark is further simplified in the presence of naturally occurring random phases, which remove the need for active input-state modulation.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Quantum-teleportation benchmarks for independent and identically distributed spin states and displaced thermal states

A successful state-transfer (or teleportation) experiment must perform better than the benchmark set by the 'best' measure and prepare procedure. We consider the benchmark problem for the following families of states: (i) displaced thermal equilibrium states of a given temperature; (ii) independent identically prepared qubits with a completely unknown state. For the first family we show that the optimal procedure is heterodyne measurement followed by the preparation of a coherent state. This procedure was known to be optimal for coherent states and for squeezed states with the 'overlap fidelity' as the figure of merit. Here, we prove its optimality with respect to the trace norm distance and supremum risk. For the second problem we consider n independent and identically distributed (i.i.d.) spin-(1/2) systems in an arbitrary unknown state {rho} and look for the measurement-preparation pair (M{sub n},P{sub n}) for which the reconstructed state {omega}{sub n}:=P{sub n} circle M{sub n}({rho}{sup xn}) is as close as possible to the input state (i.e., parallel {omega}{sub n}-{rho}{sup xn} parallel {sub 1} is small). The figure of merit is based on the trace norm distance between the input and output states. We show that asymptotically with n this problem is equivalent to the first one. The proof and construction of (M{sub n},P{sub n}) uses the theory of local asymptotic normality developed for state estimation which shows that i.i.d. quantum models can be approximated in a strong sense by quantum Gaussian models. The measurement part is identical to 'optimal estimation', showing that 'benchmarking' and estimation are closely related problems in the asymptotic set up.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Strong quantitative benchmarking of quantum optical devices

Quantum communication devices, such as quantum repeaters, quantum memories, or quantum channels, are unavoidably exposed to imperfections. However, the presence of imperfections can be tolerated, as long as we can verify that such devices retain their quantum advantages. Benchmarks based on witnessing entanglement have proven useful for verifying the true quantum nature of these devices. The next challenge is to characterize how strongly a device is within the quantum domain. We present a method, based on entanglement measures and rigorous state truncation, which allows us to characterize the degree of quantumness of optical devices. This method serves as a quantitative extension to a large class of previously known quantum benchmarks, requiring no additional information beyond what is already used for the nonquantitative benchmarks.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Qutrit Randomized Benchmarking

Ternary quantum processors offer significant potential computational advantages over conventional qubit technologies, leveraging the encoding and processing of quantum information in qutrits (three-level systems). Therefore, to evaluate and compare the performance of such emerging quantum hardware it is essential to have robust benchmarking methods suitable for a higher-dimensional Hilbert space. We demonstrate extensions of industry standard randomized benchmarking (RB) protocols, developed and used extensively for qubits, suitable for ternary quantum logic. Using a superconducting five-qutrit processor, we find an average single-qutrit process infidelity of 3.8×10 -3 . Through interleaved RB, we characterize a few relevant gates, and employ simultaneous RB to fully characterize crosstalk errors. Finally, we apply cycle benchmarking to a two-qutrit CSUM gate and obtain a two-qutrit process fidelity of 0.85. Our results present and demonstrate RB-based tools to characterize the performance of a qutrit processor, and a general approach to diagnose control errors in future qudit hardware.

97 MATHEMATICS AND COMPUTING↗

Neural Architecture Search Benchmarks: Insights and Survey

Neural Architecture Search (NAS), a promising and fast-moving research field, aims to automate the architectural design of Deep Neural Networks (DNNs) to achieve better performance on the given task and dataset. NAS methods have been very successful in discovering efficient models for various Computer Vision, Natural Language Processing, etc. The major obstacles to the advancement of NAS techniques are the demand for large computation resources and fair evaluation of various search methods. The differences in training pipeline and setting make it challenging to compare the efficiency of two NAS algorithms. A large number of NAS Benchmarks to simulate the architecture evaluation in seconds have been released over the last few years to ease the computation burden of training neural networks and can aid in the unbiased assessment of different search methods. This paper provides an extensive review of several publicly available NAS Benchmarks in the literature. We provide technical details and a deeper understanding of each benchmark and point out future directions.

97 MATHEMATICS AND COMPUTING↗

Pianola: A script-based I/O benchmark

Script-based I/O benchmarks record the I/O behavior of applications by using an instrumentation library to trace I/O events and their timing. A replay engine can then reproduce these events from the script in the absence of the original application. This type of benchmark reproduces real-world I/O workloads without the need to distribute, build, or run complex applications. However, faithfully recreating the I/O behavior of the original application requires careful design in both the instrumentation library and the replay engine. This paper presents the Pianola script-based benchmarking system, which includes an accurate and unobtrusive instrumentation system and a simple-to-use replay engine, along with some additional utility programs to manage the creation and replay of scripts. We show that for some sample applications, Pianola reproduces the qualitative features of the I/O behavior. Moreover, the overall replay time and the cumulative read and write times are usually within 10% of the values measured for the original applications.

99 GENERAL AND MISCELLANEOUS↗

SU-E-J-30: Benchmark Image-Based TCP Calculation for Evaluation of PTV Margins for Lung SBRT Patients

Purpose: Tumor control probability (TCP) calculated with accumulated radiation doses may help design appropriate treatment margins. Image registration errors, however, may compromise the calculated TCP. The purpose of this study is to develop benchmark CT images to quantify registration-induced errors in the accumulated doses and their corresponding TCP. Methods: 4DCT images were registered from end-inhale (EI) to end-exhale (EE) using a “demons” algorithm. The demons DVFs were corrected by an FEM model to get realistic deformation fields. The FEM DVFs were used to warp the EI images to create the FEM-simulated images. The two images combined with the FEM DVF formed a benchmark model. Maximum intensity projection (MIP) images, created from the EI and simulated images, were used to develop IMRT plans. Two plans with 3 and 5 mm margins were developed for each patient. With these plans, radiation doses were recalculated on the simulated images and warped back to the EI images using the FEM DVFs to get the accumulated doses. The Elastix software was used to register the FEM-simulated images to the EI images. TCPs calculated with the Elastix-accumulated doses were compared with those generated by the FEM to get the TCP error of the Elastix registrations. Results: For six lung patients, the mean Elastix registration error ranged from 0.93 to 1.98 mm. Their relative dose errors in PTV were between 0.28% and 6.8% for 3mm margin plans, and between 0.29% and 6.3% for 5mm-margin plans. As the PTV margin reduced from 5 to 3 mm, the mean TCP error of the Elastix-reconstructed doses increased from 2.0% to 2.9%, and the mean NTCP errors decreased from 1.2% to 1.1%. Conclusion: Patient-specific benchmark images can be used to evaluate the impact of registration errors on the computed TCPs, and may help select appropriate PTV margins for lung SBRT patients.

60 APPLIED LIFE SCIENCES↗

Experimental depth dose curves of a 67.5 MeV proton beam for benchmarking and validation of Monte Carlo simulation

Purpose: To measure depth dose curves for a 67.5 ± 0.1 MeV proton beam for benchmarking and validation of Monte Carlo simulation. Methods: Depth dose curves were measured in 2 beam lines. Protons in the raw beam line traversed a Ta scattering foil, 0.1016 or 0.381 mm thick, a secondary emission monitor comprised of thin Al foils, and a thin Kapton exit window. The beam energy and peak width and the composition and density of material traversed by the beam were known with sufficient accuracy to permit benchmark quality measurements. Diodes for charged particle dosimetry from two different manufacturers were used to scan the depth dose curves with 0.003 mm depth reproducibility in a water tank placed 300 mm from the exit window. Depth in water was determined with an uncertainty of 0.15 mm, including the uncertainty in the water equivalent depth of the sensitive volume of the detector. Parallel-plate chambers were used to verify the accuracy of the shape of the Bragg peak and the peak-to-plateau ratio measured with the diodes. The uncertainty in the measured peak-to-plateau ratio was 4%. Depth dose curves were also measured with a diode for a Bragg curve and treatment beam spread out Bragg peak (SOBP) on the beam line used for eye treatment. The measurements were compared to Monte Carlo simulation done with GEANT4 using TOPAS. Results: The 80% dose at the distal side of the Bragg peak for the thinner foil was at 37.47 ± 0.11 mm (average of measurement with diodes from two different manufacturers), compared to the simulated value of 37.20 mm. The 80% dose for the thicker foil was at 35.08 ± 0.15 mm, compared to the simulated value of 34.90 mm. The measured peak-to-plateau ratio was within one standard deviation experimental uncertainty of the simulated result for the thinnest foil and two standard deviations for the thickest foil. It was necessary to include the collimation in the simulation, which had a more pronounced effect on the peak-to-plateau ratio for the thicker foil. The treatment beam, being unfocussed, had a broader Bragg peak than the raw beam. A 1.3 ± 0.1 MeV FWHM peak width in the energy distribution was used in the simulation to match the Bragg peak width. An additional 1.3–2.24 mm of water in the water column was required over the nominal values to match the measured depth penetration. Conclusions: The proton Bragg curve measured for the 0.1016 mm thick Ta foil provided the most accurate benchmark, having a low contribution of proton scatter from upstream of the water tank. The accuracy was 0.15% in measured beam energy and 0.3% in measured depth penetration at the Bragg peak. The depth of the distal edge of the Bragg peak in the simulation fell short of measurement, suggesting that the mean ionization potential of water is 2–5 eV higher than the 78 eV used in the stopping power calculation for the simulation. The eye treatment beam line depth dose curves provide validation of Monte Carlo simulation of a Bragg curve and SOBP with 4%/2 mm accuracy.

60 APPLIED LIFE SCIENCES↗

SU-F-T-152: Experimental Validation and Calculation Benchmark for a Commercial Monte Carlo Pencil BeamScanning Proton Therapy Treatment Planning System in Heterogeneous Media

Purpose: Eclipse AcurosPT 13.7, the first commercial Monte Carlo pencil beam scanning (PBS) proton therapy treatment planning system (TPS), was experimentally validated for an IBA dedicated PBS nozzle in the CIRS 002LFC thoracic phantom. Methods: A two-stage procedure involving the use of TOPAS 1.3 simulations was performed. First, Geant4-based TOPAS simulations in this phantom were experimentally validated for single and multi-spot profiles at several depths for 100, 115, 150, 180, 210 and 225 MeV proton beams, using the combination of a Lynx scintillation detector and a MatriXXPT ionization chamber array. Second, benchmark calculations were performed with both AcurosPT and TOPAS in a phantom identical to the CIRS 002LFC, with the exception that the CIRS bone/mediastinum/lung tissues were replaced with similar tissues that are predefined in AcurosPT (a limitation of this system which necessitates the two stage procedure). Results: Spot sigmas measured in tissue were in agreement within 0.2 mm of TOPAS simulation for all six energies, while AcurosPT was consistently found to have larger spot sigma (<0.7 mm) than TOPAS. Using absolute dose calibration by MatriXXPT, the agreements between profiles measurements and TOPAS simulation, and calculation benchmarks are over 97% except near the end of range using 2 mm/2% gamma criteria. Overdosing and underdosing were observed at the low and high density side of tissue interfaces, respectively, and these increased with increasing depth and decreasing energy. Near the mediastinum/lung interface, the magnitude can exceed 5 mm/10%. Furthermore, we observed >5% quenching effect in the conversion of Lynx measurements to dose. Conclusion: We recommend the use of an ionization chamber array in combination with the scintillation detector to measure absolute dose and relative PBS spot characteristics. We also recommend the use of an independent Monte Carlo calculation benchmark for the commissioning of a commercial TPS. Partially supported by Varian Medical System under the master agreement between Varian and University of pennsylvania.

60 APPLIED LIFE SCIENCES↗

Parameterized Micro-benchmarking: An Auto-tuning Approach for Complex Applications

Auto-tuning has emerged as an important practical method for creating highly optimized implementations of key computational kernels and applications. However, the growing complexity of architectures and applications is creating new challenges for auto-tuning. Complex applications can involve a prohibitively large search space that precludes empirical auto-tuning. Similarly, architectures are becoming increasingly complicated, making it hard to model performance. In this paper, we focus on the challenge to auto-tuning presented by applications with a large number of kernels and kernel instantiations. While these kernels may share a somewhat similar pattern, they differ considerably in problem sizes and the exact computation performed. We propose and evaluate a new approach to auto-tuning which we refer to as parameterized micro-benchmarking. It is an alternative to the two existing classes of approaches to auto-tuning: analytical model-based and empirical search-based. Particularly, we argue that the former may not be able to capture all the architectural features that impact performance, whereas the latter might be too expensive for an application that has several different kernels. In our approach, different expressions in the application, different possible implementations of each expression, and the key architectural features, are used to derive a simple micro-benchmark and a small parameter space. This allows us to learn the most significant features of the architecture that can impact the choice of implementation for each kernel. We have evaluated our approach in the context of GPU implementations of tensor contraction expressions encountered in excited state calculations in quantum chemistry. We have focused on two aspects of GPUs that affect tensor contraction execution: memory access patterns and kernel consolidation. Using our parameterized micro-benchmarking approach, we obtain a speedup of up to 2 over the version that used default optimizations, but no auto-tuning. We demonstrate that observations made from microbenchmarks match the behavior seen from real expressions. In the process, we make important observations about the memory hierarchy of two of the most recent NVIDIA GPUs, which can be used in other optimization frameworks as well.

auto-tuning↗