Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,513 records · Page 84

Ensemble Canonical Correlation Prediction of Seasonal Precipitation Over the United States: Raising the Bar for Dynamical Model Forecasts

This paper presents preliminary results of an ensemble canonical correlation (ECC) prediction scheme developed at the Climate and Radiation Branch, NASA/Goddard Space Flight Center for determining the potential predictability of regional precipitation, and for climate downscaling studies. The scheme is tested on seasonal hindcasts of anomalous precipitation over the continental United States using global sea surface temperature (SST) for 1951-2000. To maximize the forecast skill derived from SST, the world ocean is divided into non-overlapping sectors. The canonical SST modes for each sector are used as the predictor for the ensemble hindcasts. Results show that the ECC yields a substantial (10-25%) increase in prediction skills for all the regions of the US in every season compared to traditional CCA prediction schemes. For the boreal winter, the tropical Pacific contributes the largest potential predictability to precipitation in the southwestern and southeastern regions, while the North Pacific and the North Atlantic are responsible to the enhanced forecast skills in the Pacific Northwest, the northern Great Plains and Ohio Valley. Most importantly, the ECC increases skill for summertime precipitation prediction and substantially reduces the spring predictability barrier over all the regions of the US continent. Besides SST, the ECC is designed with the flexibility to include any number of predictor fields, such as soil moisture, snow cover and additional local observations. The enhanced ECC forecast skill provides a new benchmark for evaluating dynamical model forecasts.

Lau, William K. M.↗

Advancing a Model-Validated Statistical Method for Decomposing the Key Oceanic Drivers of Regional Climate: Focus on Northern and Tropical African Climate Variability in the Community Earth System Model (CESM)

This study advances the practicality and stability of the traditional multivariate statistical method, generalized equilibrium feedback assessment (GEFA), for decomposing the key oceanic drivers of regional atmospheric variability, especially when available data records are short. An advanced stepwise GEFA methodology is introduced, in which unimportant forcings within the forcing matrix are eliminated through stepwise selection. Method validation of stepwise GEFA is performed using the CESM, with a focused application to northern and tropical Africa (NTA). First, a statistical assessment of the atmospheric response to each primary oceanic forcing is carried out by applying stepwise GEFA to a fully coupled control run. Then, a dynamical assessment of the atmospheric response to individual oceanic forcings is performed through ensemble experiments by imposing sea surface temperature anomalies over focal ocean basins. Finally, to quantify the reliability of stepwise GEFA, the statistical assessment is evaluated against the dynamical assessment in terms of four metrics: the percentage of grid cells with consistent response sign, the spatial correlation of atmospheric response patterns, the area-averaged seasonal cycle of response magnitude, and consistency in associated mechanisms between assessments. In CESM, tropical modes, namely El Niño–Southern Oscillation and the tropical Indian Ocean Basin, tropical Indian Ocean dipole, and tropical Atlantic Niño modes, are the dominant oceanic controls of NTA climate. In complementary studies, stepwise GEFA is validated in terms of isolating terrestrial forcings on the atmosphere, and observed oceanic and terrestrial drivers of NTA climate are extracted to establish an observational benchmark for subsequent coupled model evaluation and development of process-based weights for regional climate projections.

54 ENVIRONMENTAL SCIENCES↗

Few multi-year precipitation-reduction experiments find a shift in the productivity-precipitation relationship

Well-defined productivity–precipitation relationships of ecosystems are needed as benchmarks for the validation of land models used for future projections. The productivity–precipitation relationship may be studied in two ways: the spatial approach relates differences in productivity to those in precipitation among sites along a precipitation gradient (the spatial fit, with a steeper slope); the temporal approach relates interannual productivity changes to variation in precipitation within sites (the temporal fits, with flatter slopes). Precipitation–reduction experiments in natural ecosystems represent a complement to the fits, because they can reduce precipitation below the natural range and are thus well suited to study potential effects of climate drying. Here, we analyse the effects of dry treatments in eleven multiyear precipitation–manipulation experiments, focusing on changes in the temporal fit. We expected that structural changes in the dry treatments would occur in some experiments, thereby reducing the intercept of the temporal fit and displacing the productivity–precipitation relationship downward the spatial fit. Seventy two percent of expiriments showed that dry treatments did not alter the temporal fit. This implies that current temporal fits are to be preferred over the spatial fit to benchmark land-model projections of productivity under future climate within the precipitation ranges covered by the experiments. Moreover, in two experiments, the intercept of the temporal fit unexpectedly increased due to mechanisms that reduced either water loss or nutrient loss. The expected decrease of the intercept was observed in only one experiment, and only when distinguishing between the late and the early phases of the experiment. This implies that we currently do not know at which precipitation–reduction level or at which experimental duration structural changes will start to alter ecosystem productivity. Our study highlights the need for experiments with multiple, including more extreme, dry treatments, to identify the precipitation boundaries within which the current temporal fits remain valid.

54 ENVIRONMENTAL SCIENCES↗

OptiBench: An Optimization Benchmark Tool for Renewable Energy Problems

We propose a benchmark framework and visualization tool, OptiBench, for analyzing the performance of state-of-the-art optimization solvers across a variety of optimization problems in renewable energy research. Our framework is designed from the ground up in the Julia programming language and enables analysis at scale on high performance computing (HPC) systems. Our visualization tool allows effortless evaluation of optimization solver performance, robustness, and accuracy through intuitive plots, e.g., performance profiles, heat maps, and distribution plots. We have tested three benchmark suites relevant to the modeling of renewable energy systems, viz., CUTEst, PGLib-OPF, and WaterTAP water treatment optimization problems. We illustrate benchmarking of CUTEst using OptiBench on the National Renewable Energy Laboratory's (NREL) HPC Kestrel. Our findings indicate that MA57 HSL linear solver demonstrated the best overall performance for an experimental IPOPT implementation. Our work is ongoing and we intend to add support for more optimization solvers and benchmark test suites in the future.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

OptiBench: An Optimization Benchmark Tool for Renewable Energy Problems

We propose a benchmark framework and visualization tool, OptiBench, for analyzing the performance of state-of-the-art optimization solvers across a variety of optimization problems in renewable energy research. Our framework is designed from the ground up in the Julia programming language and enables analysis at scale on high performance computing (HPC) systems. Our visualization tool allows effortless evaluation of optimization solver performance, robustness, and accuracy through intuitive plots, e.g., performance profiles, heat maps, and distribution plots. We have tested three benchmark suites relevant to the modeling of renewable energy systems, viz., CUTEst, PGLib-OPF, and WaterTAP water treatment optimization problems. We illustrate benchmarking of CUTEst using OptiBench on the National Laboratory of the Rockies's (NLR) HPC Kestrel. Our findings indicate that MA57 HSL linear solver demonstrated the best overall performance for an experimental IPOPT implementation. Our work is ongoing and we intend to add support for more optimization solvers and benchmark test suites in the future.

97 MATHEMATICS AND COMPUTING↗

Surrogate Modeling of Subgrid Turbulent Transport Based on 3D Radiative Hydrodynamic Simulations of the Quiet Sun

Turbulent Transport: Plays a critical role in astrophysical plasmas, such as the solar interior, spanning multiple scales and challenging traditional modeling approaches. Objective: Develop machine learning (ML) models—MLP and CNN—to predict subgrid Reynolds stress tensors from StellarBox 3D simulations of the solar atmosphere. Benchmarking: Compare ML-driven models against physics-based Gradient and Smagorinsky approaches.

SMD↗

Impact-Seismic Investigations of the InSight Mission

Impact investigations will be an important aspect of the InSight mission. One of the scientific goals of the mission is a measurement of the current impact rate at Mars. Impacts will additionally inform the major goal of investigating the interior structure of Mars. In this paper, we review the current state of knowledge about seismic signals from impacts on the Earth, Moon, and laboratory experiments. We describe the generalized physical models that can be used to explain these signals. A discussion of the appropriate source time function for impacts is presented, along with spectral characteristics including the cutoff frequency and its dependence on impact momentum. Estimates of the seismic efficiency (ratio between seismic and impact energies) vary widely. Our preferred value for the seismic efficiency at Mars is 5 × 10−4, which we recommend using until we can measure it during the InSight mission, when seismic moments are not used directly. Effects of the material properties at the impact point and at the seismometer location are considered. We also discuss the processes by which airbursts and acoustic waves emanate from bolides, and the feasibility of detecting such signals. We then consider the case of impacts on Mars. A review is given of the current knowledge of present-day cratering on Mars: the current impact rate, characteristics of those impactors such as velocity and directions, and the morphologies of the craters those impactors create. Several methods of scaling crater size to impact energy are presented. The Martian atmosphere, although thin, will cause fragmentation of impactors, with implications for the resulting seismic signals. We also benchmark several different seismic modeling codes to be used in analysis of impact detections, and those codes are used to explore the seismic amplitude of impactinduced signals as a function of distance from the impact site. We predict a measurement of the current impact flux will be possible within the timeframe of the prime mission (one Mars year) with the detection of ∼ a few to several tens of impacts. However, the error bars on these predictions are large. Specific to the InSight mission, we list discriminators of seismic signals from impacts that will be used to distinguish them from marsquakes. We describe the role of the InSight Impacts Science Theme Group during mission operations, including a plan for possible night-time meteor imaging. The impacts detected by these methods during the InSight mission will be used to improve interior structure models, measure the seismic efficiency, and calculate the size frequency distribution of current impacts.

Ingrid Daubar↗

SU-G-BRC-10: Feasibility of a Web-Based Monte Carlo Simulation Tool for Dynamic Electron Arc Radiotherapy (DEAR)

Purpose: DEAR is a radiation therapy technique utilizing synchronized motion of gantry and couch during delivery to optimize dose distribution homogeneity and penumbra for treatment of superficial disease. Dose calculation for DEAR is not yet supported by commercial TPSs. The purpose of this study is to demonstrate the feasibility of using a web-based Monte Carlo (MC) simulation tool (VirtuaLinac) to calculate dose distributions for a DEAR delivery. Methods: MC simulations were run through VirtuaLinac, which is based on the GEANT4 platform. VirtuaLinac utilizes detailed linac head geometry and material models, validated phase space files, and a voxelized phantom. The input was expanded to include an XML file for simulation of varying mechanical axes as a function of MU. A DEAR XML plan was generated and used in the MC simulation and delivered on a TrueBeam in Developer Mode. Radiographic film wrapped on a cylindrical phantom (12.5 cm radius) measured dose at a depth of 1.5 cm and compared to the simulation results. Results: A DEAR plan was simulated using an energy of 6 MeV and a 3×10 cm{sup 2} cut-out in a 15×15 cm{sup 2} applicator for a delivery of a 90° arc. The resulting data were found to provide qualitative and quantitative evidence that the simulation platform could be used as the basis for DEAR dose calculations. The resulting unwrapped 2D dose distributions agreed well in the cross-plane direction along the arc, with field sizes of 18.4 and 18.2 cm and penumbrae of 1.9 and 2.0 cm for measurements and simulations, respectively. Conclusion: Preliminary feasibility of a DEAR delivery using a web-based MC simulation platform has been demonstrated. This tool will benefit treatment planning for DEAR as a benchmark for developing other model based algorithms, allowing efficient optimization of trajectories, and quality assurance of plans without the need for extensive measurements.

60 APPLIED LIFE SCIENCES↗

Depth Dose Distribution Study within a Phantom Torso after Irradiation with a Simulated Solar Particle Event at NSRL

The adequate knowledge of the radiation environment and the doses incurred during a space mission is essential for estimating an astronaut's health risk. The space radiation environment is complex and variable, and exposures inside the spacecraft and the astronaut's body are compounded by the interactions of the primary particles with the atoms of the structural materials and with the body itself Astronauts' radiation exposures are measured by means of personal dosimetry, but there remains substantial uncertainty associated with the computational extrapolation of skin dose to organ dose, which can lead to over- or underestimation of the health risk. Comparisons of models to data showed that the astronaut's Effective dose (E) can be predicted to within about a +10% accuracy using space radiation transport models for galactic cosmic rays (GCR) and trapped radiation behind shielding. However for solar particle event (SPE) with steep energy spectra and for extra-vehicular activities on the surface of the moon where only tissue shielding is present, transport models predict that there are large differences in model assumptions in projecting organ doses. Therefore experimental verification of SPE induced organ doses may be crucial for the design of lunar missions. In the research experiment "Depth dose distribution study within a phantom torso" at the NASA Space Radiation Laboratory (NSRL) at BNL, Brookhaven, USA the large 1972 SPE spectrum was simulated using seven different proton energies from 50 up to 450 MeV. A phantom torso constructed of natural bones and realistic distributions of human tissue equivalent materials, which is comparable to the torso of the MATROSHKA phantom currently on the ISS, was equipped with a comprehensive set of thermoluminescence detectors and human cells. The detectors are applied to assess the depth dose distribution and radiation transport codes (e.g. GEANT4) are used to assess the radiation field and interactions of the radiation field with the phantom torso. Lymphocyte cells are strategically embedded at selected locations at the skin and internal organs and are processed after irradiation to assess the effects of shielding on the yield of chromosome damage. The initial focus of the present experiment is to correlate biological results with physical dosimetry measurements in the phantom torso. Further on, the results of the passive dosimetry within the anthropomorphic phantoms represent the best tool to generate reliable data to benchmark computational radiation transport models in a radiation field of interest. The presentation will give first results of the physical dose distribution, the comparison with GEANT4 computer simulations based on a Voxel model of the phantom, and a comparison with the data from the chromosome aberration study.

Berger, Thomas↗

Image-to-Image Wildfire Detection via Quantum-Compatible Variational Segmentation from Remotely-sensed Data

Over the last decade, the incidence of wildfires has surged, causing widespread destruction globally. To better comprehend and manage these incidents, remote sensing and aerial missions have been implemented in recent efforts. However, this has resulted in an exponential rise in the amount of remote sensing data utilization, leading to a need for intelligent automation of data extraction in wildfire studies. Machine learning provides an accurate automated approach for detecting these natural anomalies and facilitates decision-makers to take prompt actions. To make insightful decisions in wildfire management, it is imperative to move beyond simple detection and explore the potential of probabilistic generative machine learning for creating "what-if" scenarios for various wildfire conditions. Such models offer improved representation of the stochastic nature of wildfire events. However, the optimization of these models can be computationally expensive, especially when using classical computers. Quantum computers have recently emerged as a promising solution to reduce the computational cost of training such models and improve their performance. In this study, we aim to utilize quantum-compatible machine learning techniques to implement our probabilistic generative approach. To that end, we propose a supervised probabilistic variational model consisting of a U-NET-based image-to-image component along with encoder and decoder networks which work as a variational autoencoder (VAE) component. Additionally, we explore the type of latent distribution type in the VAE component and implement different means for modeling the prior distribution. We further investigate the quantum-compatible versions of the model compared to the classical counterpart and benchmark potential benefits of quantum compatibility over the classical model.

quantum machine learning↗

Conflict Detection in Open RAN with Recurrent Neural Networks Using Geometric Manifolds

Allowing third-party applications on Radio Access Network (RAN) Intelligent Controllers (RICs) within the OpenRAN (O-RAN) framework introduces conflicting interactions that are often difficult to detect in advance. These conflicts, occurring between third-party applications in the Near RealTime RIC (Near-RT RIC), known as xApps, can lead to performance degradation and instability in O-RAN if not identified early. Existing conflict detection and mitigation solutions in the literature assume that the conflicts are known beforehand, which is not always accurate due to the complex and often hidden relationships between control parameters and Key Performance Indicators (KPIs). In this paper, we propose a novel Recurrent Neural Network (RNN) to detect both known and unknown conflicts in O-RAN xApps as specified in the O-RAN standards. We model the xApps, control parameters, and KPIs with nodes and edges to create graph structures and use the hidden nonEuclidean geometric properties of the Riemannian manifold to train the RNN model. The performance of this proposed model is validated using evaluation metrics and compared with benchmarks. Results demonstrate that the proposed RNN model, leveraging Riemannian geometric properties, can achieve 100% of the F1-score provided by an optimal solution in just 20 iterations.

5G↗

Conflict Detection in Open RAN with Recurrent Neural Networks Using Geometric Manifolds

Allowing third-party applications on Radio Access Network (RAN) Intelligent Controllers (RICs) within the OpenRAN (O-RAN) framework introduces conflicting interactions that are often difficult to detect in advance. These conflicts, occurring between third-party applications in the Near RealTime RIC (Near-RT RIC), known as xApps, can lead to performance degradation and instability in O-RAN if not identified early. Existing conflict detection and mitigation solutions in the literature assume that the conflicts are known beforehand, which is not always accurate due to the complex and often hidden relationships between control parameters and Key Performance Indicators (KPIs). In this paper, we propose a novel Recurrent Neural Network (RNN) to detect both known and unknown conflicts in O-RAN xApps as specified in the O-RAN standards. We model the xApps, control parameters, and KPIs with nodes and edges to create graph structures and use the hidden nonEuclidean geometric properties of the Riemannian manifold to train the RNN model. The performance of this proposed model is validated using evaluation metrics and compared with benchmarks. Results demonstrate that the proposed RNN model, leveraging Riemannian geometric properties, can achieve 100% of the F1-score provided by an optimal solution in just 20 iterations.

5G↗

Uncertainty quantification and sensitivity analysis for SPERT III E-core reactivity measurement benchmarking

The Special Power Excursion Reactor Test (SPERT) III E-core experiment is important because it provides critical data on reactor behavior under significant reactivity insertions, which is essential for validating computer simulations and ensuring the safety of modern light water reactors. Its design similarities to contemporary reactors make it a valuable resource for understanding and mitigating extreme hazards in nuclear operations. The current study details the application of formal parametric uncertainty quantification and sensitivity analysis to a model of the SPERT-III E-core model for zero power reactivity benchmarking. Additionally, the reactivity impact from various modeling assumptions is quantified. Overall, an conservative estimate for an uncertainty in k$_{\text{eff}}$ of $\pm$1257 was observed. A less conservative, more realistic, uncertainty estimate of $\pm$1096 pcm can be justified by the potential for various parametric uncertainties to become negligible when sampled independently across the ~1400 pins in the core. The experimental results fall within both of these uncertainty bounds. Standardized regression coefficient as well as Sobol indices are used to identify the guide tube thicknesses as the primary contributors to the uncertainty in k$_{\text{eff}}$. Overall, this study provides information on how the uncertainties in input parameters and modeling methods impact simulated k$_{\text{eff}}$ values and can be used to aid model building efforts for future code validation with the SPERT-III E-core experiment.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

Benchmarking of collision operators with momentum source corrections

Several linearized collision operator models with parallel momentum conservation enforced by source functions are benchmarked against the correct collision operator based on Rosenbluth potentials. The model operators without energy diffusion allow for an analytic solution of a generalized Spitzer problem. For the other operators under investigation, an integrodifferential equation is solved. The benchmarking is performed both for the parallel conductivity with the ions assumed at rest and for the bootstrap current, where ion and electron flows are collisionally coupled. The accuracy of the results obtained with the operators varies from rather poor in the case of the simplest monoenergetic model to quite satisfactory for the models employing an energy-weighted parallel momentum source function.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Landsat near-infrared (NIR) band and ELM-FATES sensitivity to forest disturbances and regrowth in the Central Amazon

Forest disturbance and regrowth are key processes in forest dynamics, but detailed information on these processes is difficult to obtain in remote forests such as the Amazon. We used chronosequences of Landsat satellite imagery (Landsat 5 Thematic Mapper and Landsat 7 Enhanced Thematic Mapper Plus) to determine the sensitivity of surface reflectance from all spectral bands to windthrow, clear-cut, and clear-cut and burned (cut + burn) and their successional pathways of forest regrowth in the Central Amazon. We also assessed whether the forest demography model Functionally Assembled Terrestrial Ecosystem Simulator (FATES) implemented in the Energy Exascale Earth System Model (E3SM) Land Model (ELM), ELM-FATES, accurately represents the changes for windthrow and clear-cut. The results show that all spectral bands from the Landsat satellites were sensitive to the disturbances but after 3 to 6 years only the near-infrared (NIR) band had significant changes associated with the successional pathways of forest regrowth for all the disturbances considered. In general, the NIR values decreased immediately after disturbance, increased to maximum values with the establishment of pioneers and early successional tree species, and then decreased slowly and almost linearly to pre-disturbance conditions with the dynamics of forest succession. Statistical methods predict that NIR values will return to pre-disturbance values in about 39, 36, and 56 years for windthrow, clear-cut, and cut + burn disturbances, respectively. The NIR band captured the observed, and different, successional pathways of forest regrowth after windthrow, clear-cut, and cut + burn. Consistent with inferences from the NIR observations, ELM-FATES predicted higher peaks of biomass and stem density after clear-cuts than after windthrows. ELM-FATES also predicted recovery of forest structure and canopy coverage back to pre-disturbance conditions in 38 years after windthrows and 41 years after clear-cut. The similarity of ELM-FATES predictions of regrowth patterns after windthrow and clear-cut to those of the NIR results suggests the NIR band can be used to benchmark forest regrowth in ecosystem models. Our results show the potential of Landsat imagery data for mapping forest regrowth from different types of disturbances, benchmarking, and the improvement of forest regrowth models.

54 ENVIRONMENTAL SCIENCES↗

Validation of OpenPronghorn for Periodic Hill Flow Separation

OpenPronghorn is an open-source, MOOSE-based thermal-hydraulics simulation tool used for advanced reactor analysis. As an open-source code, it offers a transparent framework for validating governing equations, assumptions, and numerical methods against established benchmarks. This study evaluates OpenPronghorn's RANS turbulence model against the ERCOFTAC Case 81 periodic hill benchmark, a standard test case for separated turbulent flow featuring curved-wall separation, recirculation, shear-layer development, and reattachment. Streamwise velocity profiles predicted by OpenPronghorn were compared to reference LES data at multiple x/h locations. Results show that OpenPronghorn captures the overall trend of the velocity profiles, but the largest discrepancies occur in the separated-flow region, where turbulence is highly anisotropic and strongly affected by adverse pressure gradients and wall curvature—conditions that are inherently difficult for standard RANS models to resolve. Future work will test alternative k-e model variants and correction terms to improve prediction accuracy in this region.

42 - ENGINEERING↗

Phase-field modeling of rate-dependent fluid-driven fracture initiation and propagation

The rate-dependent behavior associated with deformation and fracturing of materials, such as natural rocks, poses significant challenges for modeling. In addition to the complications of the viscoelastic response, the speed of fracture propagation reflects micromechanical mechanisms in the fracture process zone(FPZ). In order to represent these complicated behaviors, a thermodynamically consistent, rate-dependent fracture model is required. Based on rigorous thermodynamic principles, we derive a rate-dependent phase-field mechanical model coupled with single-phase fluid flow in both the matrix and the fracture. The model is guaranteed to satisfy energy conservation during fracture propagation. Here, the system of equations is solved using the introduced solution procedure and a novel preconditioner that accounts for the complex fluid-structure interaction. The proposed phase-field model is tested against several benchmark problems on solid-fluid coupling, fluid-driven fracture propagation and rate-dependent viscoelastic deformation. The model serves as a strong basis for investigating rate-dependent fracturing experiments and for making predictions of material behaviors under new conditions.

42 ENGINEERING↗

Analysis of CRRES PHA Data for Low-Energy-Deposition Events

This effort analyzed the low-energy deposition Pulse Height Analyzer (PHA) data from the Combined Release and Radiation Effects Satellite (CRRES). The high-energy deposition data had been previously analyzed and shown to be in agreement with spallation reactions predicted by the Clemson University Proton Interactions in Devices (CUPID) simulation model and existing environmental and orbit positioning models (AP-8 with USAF B-L coordinates). The scope of this project was to develop and improve the CUPID model by increasing its range to lower incident particle energies, and to expand the modeling to include contributions from elastic interactions. Before making changes, it was necessary to identify experimental data suitable for benchmarking the codes; then, the models to the CRRES PHA data could be applied. It was also planned to test the model against available low-energy proton or neutron SEU data obtained with mono-energetic beams.

McNulty, P. J.↗