Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,711 records · Page 95

Model data for numerical evaluation of photosensitive tracers as a strategy for separating surface and subsurface transient storage in streams

This model-data archive pertains to a study aimed at the numerical evaluation of the photosensitive tracers as a potential strategy for separating the effects of surface and hyporheic storage zones (SSZs and HSZs, respectively). Separating effects of SSZ and HSZ are important for accurately representing stream function as HSZs and SSZs expose solutes to significantly different biogeochemical conditions. We perform numerical experiments using a multiscale reactive transport model for stream corridors implemented in ATS code, which allows for representing multiple storage zones with their respective travel time distributions and biogeochemistry. For each of the numerical experiment, we provide in this model-data archive python wrapper to drive ATS (forward_model.py), synthetic observation (synthetic_btc.py, BTC_observed.csv), forward model files including ATS input (multiscale_transport.tpl), PFLOTRAN inputs (reactions_channel.tpl, reactions_hz.tpl, reactions_sz.tpl), MCMC (mcmc_run.py), predictive uncertainty (pred_uncert.py, BTCs_simulated_day.csv, BTCs_simulated_night.csv), MCMC outputs (tracer_test_GR.npy, tracer_test_logps.npy, tracer_test_parameters.npy) and Jupyter notebook for post-processing and visualization (post-processing.ipynb). For the denitrification application, ATS input file (denitrification_multisubgrid.tpl), PFLOTRAN input files (denitrification_channel.in, denitrification_hz.in), predictive uncertainty (pred_uncert_denitrification.py, BTCs_simulated_DO.csv, BTCs_simulated_DOC.csv, BTCs_simulated_Nitrate.csv).

54 ENVIRONMENTAL SCIENCES↗

Turbulent Processes that Influence Boundary-Layer Cloud Structure (Final Technical Report)

Many models of the atmosphere simulate vertical profiles of observable fields, such as temperature, moisture, and vertical velocity. To assess the realism of such model output, it is important to evaluate the modeled profiles against observations. However, many extant evaluations consist of comparing time averaged profiles from the model and the observations. Such time averages can smooth out unrealistic features that appear in instantaneous profiles. In this way, the time averages may obscure certain model errors from detection. For instance, many models misrepresent turbulent processes, leading to profiles that are too smoothed out or too noisy. E.g., if a model has coarse vertical resolution and overly strong diffusion, then the model may be unable to produce observed sharp peaks. On the other hand, a model may have artificial approximations that produce spurious jumps in the vertical. Much can be learned by browsing through instantaneous snapshots of model output. But the volume of output produced by a global model is far too great to rely purely on visual inspection. In addition, we need a way to statistically characterize the profiles, so that anomalies can be detected more conveniently.

54 ENVIRONMENTAL SCIENCES↗

Comparison of Entry Descent and Landing Aerodynamic Databases with Uncertainty Quantification Developed Using Machine Learning Techniques

When developing the aerodynamic databases for use in trajectory simulations, it is important to develop a system of metrics to qualify which aerodynamic models are best to use. Since aerodynamics are just one input into trajectory simulations, the results of these simulations do not reflect on the quality of the aerodynamic database used. This means that aerodynamic database comparisons must be done offline. While traditional metrics that focus on mean/nominal predictions are a good first step, more robust estimates of the prediction interval become important as more focused uncertainty models are developed. We explore the limitations of evaluating aerodynamic models based purely on nominal-centered response surfaces. Before elaborating and evaluating metrics based on distributed models, the value of evaluating prediction interval and confidence interval are discussed to conclude that prediction intervals are more relevant to the use of trajectory analysis. Several metrics to evaluate the prediction interval are introduced with a focus on the standard calibration metric. Finally, we compare candidate models using both mean and distributed metrics. A finalized candidate model developed using state of the art machine learning methods is compared to a baseline model developed using traditional aerodynamic database modeling techniques.

Aerodynamic Database↗

Enhanced climate reproducibility testing with false discovery rate correction

Simulating the Earth's climate is an important and complex problem, thus climate models are similarly complex, comprised of millions of lines of code. In order to appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures to improve their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non-bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we propose a short-simulation ensemble-based test for an atmosphere model to evaluate the null hypothesis that modified model results are statistically equivalent to that of the original model. We implement this test in version 2 of the US Department of Energy's Energy Exascale Earth System Model (E3SM). The test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the currently implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence.

Kelleher, Michael E. [Oak Ridge National Laborator↗

Evaluating Nextgen Closely Spaced Parallel Operations Concepts with Validated Human Performance Models: Flight Deck Guidelines

The objectives of the current research were to develop valid human performance models (HPMs) of approach and land operations; use these models to evaluate the impact of NextGen Closely Spaced Parallel Operations (CSPO) on pilot performance; and draw conclusions regarding flight deck display design and pilot-ATC roles and responsibilities for NextGen CSPO concepts. This document presents guidelines and implications for flight deck display designs and candidate roles and responsibilities. A companion document (Gore, Hooey, Mahlstedt, & Foyle, 2013) provides complete scenario descriptions and results including predictions of pilot workload, visual attention and time to detect off-nominal events.

guidelines↗

Aeroacoustic Measurements of a Wing/Slat Model

Aeroacoustic evaluations of high-lift devices have been carried out in the Quiet Flow Facility of the NASA Langley Research Center. The present paper deals with detailed flow and acoustic measurements that have been made to understand, and to possibly predict and reduce, the noise from a wing leading edge slat configuration. The acoustic database is obtained by a moveable Small Aperture Directional Array (SADA) of microphones designed to electronically steer to different portions of models under study. The slat is shown to be a uniform distributed noise source. The data was processed such that spectra and directivity were determined with respect to a one-foot span of slat. The spectra are normalized in various fashions to demonstrate slat noise character. In order to equate portions of the spectra to different slat noise components, trailing edge noise predictions using measured slat boundary layer parameters as inputs are compared to the measured slat noise spectra.

Mendoza, Jeff M.↗

Plant Phenology Evaluation of CRESCENDO Land Surface Models–Part 1: Start and End of the Growing Season

Plant phenology plays a fundamental role in land–atmosphere interactions, and its variability and variations are an indicator of climate and environmental changes. For this reason, current land surface models include phenology parameterizations and related biophysical and biogeochemical processes. In this work, the climatology of the beginning and end of the growing season, simulated by the land component of seven state-of-the-art European Earth system models participating in the CMIP6, is evaluated globally against satellite observations. The assessment is performed using the vegetation metric leaf area index and a recently developed approach, named four growing season types. On average, the land surface models show a 0.6-month delay in the growing season start, while they are about 0.5 months earlier in the growing season end. The difference with observation tends to be higher in the Southern Hemisphere compared to the Northern Hemisphere. High agreement between land surface models and observations is exhibited in areas dominated by broadleaf deciduous trees, while high variability is noted in regions dominated by broadleaf deciduous shrubs. Generally, the timing of the growing season end is accurately simulated in about 25 % of global land grid points versus 16 % in the timing of growing season start. The refinement of phenology parameterization can lead to better representation of vegetation-related energy, water, and carbon cycles in land surface models, but plant phenology is also affected by plant physiology and soil hydrology processes. Consequently, phenology representation and, in general, vegetation modelling is a complex task, which still needs further improvement, evaluation, and multi-model comparison.

Phenology↗

Dynamic metrics of natural ventilation cooling effectiveness for interactive modeling

The evaluation of natural ventilation potential for cooling indoor spaces during the early design phases is of great interest to researchers and practitioners. Among various definitions and usages for natural ventilation potential (NVP) in early design evaluation, this paper reviews and identifies the key performance indicators, and proposes two new dynamic metrics—natural ventilation cooling effectiveness (NVCE) and climate potential utilization ratio (CPUR). The metrics are dynamically responsive to various design options, in both steady and transient states, allowing consideration of thermal mass. Assisting in design development processes, the metrics quantify how well indoor spaces make use of natural ventilation’s cooling capacity. Case studies are presented to demonstrate how NVCE and CPUR enable designers to evaluate the predicted performance and how to apply the information to improve building design. Finally, the results of the design iterations showed that the relationship among various design parameters should be dynamically understood in order to evaluate the performance of natural ventilation, confirming that “the more the airflow, the greater the potential,” and “the heavier the thermal mass, the greater the energy saving” were not always true.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Seasonal versus Episodic Performance Evaluation for an Eulerian Photochemical Air Quality Model

This study presents detailed evaluation of the seasonal and episodic performance of the Community Multiscale Air Quality (CMAQ) modeling system applied to simulate air quality at a fine grid spacing (4 km horizontal resolution) in central California, where ozone air pollution problems are severe. A rich aerometric database collected during the summer 2000 Central California Ozone Study (CCOS) is used to prepare model inputs and to evaluate meteorological simulations and chemical outputs. We examine both temporal and spatial behaviors of ozone predictions. We highlight synoptically driven high-ozone events (exemplified by the four intensive operating periods (IOPs)) for evaluating both meteorological inputs and chemical outputs (ozone and its precursors) and compare them to the summer average. For most of the summer days, cross-domain normalized gross errors are less than 25% for modeled hourly ozone, and normalized biases are between {+-}15% for both hourly and peak (1 h and 8 h) ozone. The domain-wide aggregated metrics indicate similar performance between the IOPs and the whole summer with respect to predicted ozone and its precursors. Episode-to-episode differences in ozone predictions are more pronounced at a subregional level. The model performs consistently better in the San Joaquin Valley than other air basins, and episodic ozone predictions there are similar to the summer average. Poorer model performance (normalized peak ozone biases <-15% or >15%) is found in the Sacramento Valley and the Bay Area and is most noticeable in episodes that are subject to the largest uncertainties in meteorological fields (wind directions in the Sacramento Valley and timing and strength of onshore flow in the Bay Area) within the boundary layer.

54 ENVIRONMENTAL SCIENCES↗

Large-scale features of Last Interglacial climate: results from evaluating the lig127k simulations for the Coupled Model Intercomparison Project (CMIP6)–Paleoclimate Modeling Intercomparison Project (PMIP4)

The modeling of paleoclimate, using physically based tools, is increasingly seen as a strong out-of-sample test of the models that are used for the projection of future climate changes. New to the Coupled Model Intercomparison Project (CMIP6) is the Tier 1 Last Interglacial experiment for 127 000 years ago (lig127k), designed to address the climate responses to stronger orbital forcing than the midHolocene experiment, using the same state-of-the-art models as for the future and following a common experimental protocol. Here we present a first analysis of a multi-model ensemble of 17 climate models, all of which have completed the CMIP6 DECK (Diagnostic, Evaluation and Characterization of Klima) experiments. The equilibrium climate sensitivity (ECS) of these models varies from 1.8 to 5.6°C. The seasonal character of the insolation anomalies results in strong summer warming over the Northern Hemisphere continents in the lig127k ensemble as compared to the CMIP6 piControl and much-reduced minimum sea ice in the Arctic. The multi-model results indicate enhanced summer monsoonal precipitation in the Northern Hemisphere and reductions in the Southern Hemisphere. These responses are greater in the lig127k than the CMIP6 midHolocene simulations as expected from the larger insolation anomalies at 127 than 6 ka. New synthesis for surface temperature and precipitation, targeted for 127 ka, have been developed for comparison to the multi-model ensemble. The lig127k model ensemble and data reconstructions are in good agreement for summer temperature anomalies over Canada, Scandinavia, and the North Atlantic and for precipitation over the Northern Hemisphere continents. The model–data comparisons and mismatches point to further study of the sensitivity of the simulations to uncertainties in the boundary conditions and of the uncertainties and sparse coverage in current proxy reconstructions. The CMIP6–Paleoclimate Modeling Intercomparison Project (PMIP4) lig127k simulations, in combination with the proxy record, improve our confidence in future projections of monsoons, surface temperature, and Arctic sea ice, thus providing a key target for model evaluation and optimization.

Last Interglacial climate↗

Evaluation of nonrigid registration models for interfraction dose accumulation in radiotherapy

Purpose: Interfraction dose accumulation is necessary to evaluate the dose distribution of an entire course of treatment by adding up multiple dose distributions of different treatment fractions. This accumulation of dose distributions is not straightforward as changes in the patient anatomy may occur during treatment. For this purpose, the accuracy of nonrigid registration methods is assessed for dose accumulation based on the calculated deformations fields. Methods: A phantom study using a deformable cubic silicon phantom with implanted markers and a cylindrical silicon phantom with MOSFET detectors has been performed. The phantoms were deformed and images were acquired using a cone-beam CT imager. Dose calculations were performed on these CT scans using the treatment planning system. Nonrigid CT-based registration was performed using two different methods, the Morphons and Demons. The resulting deformation field was applied on the dose distribution. For both phantoms, accuracy of the registered dose distribution was assessed. For the cylindrical phantom, also measured dose values in the deformed conditions were compared with the dose values of the registered dose distributions. Finally, interfraction dose accumulation for two treatment fractions of a patient with primary rectal cancer has been performed and evaluated using isodose lines and the dose volume histograms of the target volume and normal tissue. Results: A significant decrease in the difference in marker or MOSFET position was observed after nonrigid registration methods (p<0.001) for both phantoms and with both methods, as well as a significant decrease in the dose estimation error (p<0.01 for the cubic phantom and p<0.001 for the cylindrical) with both methods. Considering the whole data set at once, the difference between estimated and measured doses was also significantly decreased using registration (p<0.001 for both methods). The patient case showed a slightly underdosed planning target volume and an overdosed bladder volume due to anatomical deformations. Conclusions: Dose accumulation using nonrigid registration methods is possible using repeated CT imaging. This opens possibilities for interfraction dose accumulation and adaptive radiotherapy to incorporate possible differences in dose delivered to the target volume and organs at risk due to anatomical deformations.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Challenges Beyond the Standard Model: An Examination and Evaluation of Alternatives with Respect to Unsolved Problems in Energetics and Signaling

The standard model of particle physics is regarded as one of the greatest achievements of modern physics. It has successfully predicted the existence of numerous hadrons; e.g. the Higgs Boson, the W and Z Bosons, B mesons, etc. Moreover it accurately models many phenomena such as Beta and Radioactive decay, pair production, hadron mass splitting, and others. In spite of this, the standard model has many shortfalls. For example it fails to account for or explain the presence of dark matter and dark energy. Many alternative models of the substructure of matter etc. have been proposed. Some of these are examined. In addition the strongly related standard model of cosmology, that is the big bang theory, accounts for all observed celestial phenomena. What it cannot do is satisfactorily justify all astrophysical events. As noted, dark matter and dark energy are not well understood yet they are core elements of the big bang model. Many elements of this standard model of cosmology are challenged regularly, yet somehow the community always fits the problems with the model into another mysterious element. This model and others are examined for consistency and reconciliation. Evidence of particles that move faster than the speed of light has long been sought, but remains elusive. Some of the known space-like observations of physics are examined. Potential consequences of such particles are likewise investigated. Some conclusions are drawn with respect to the framework of tachyons in a rational but non-rigorous sense. Finally, some of my ideas are presented, concerning all of the above material. The framework of theories of everything is examined, considering what such a theory requires for validity. Sociological notes on the structure of research in physics are given.

Coxe, Alexander M.↗

Beginner's Guide to Understanding Power System Model Results for Long-Term Resource Plans

This guide was developed to improve decision-making in the electricity planning process by strengthening dialogue between electricity system planners and stakeholders throughout the process. This document was created for anyone who is first getting exposed to scenario results from power system planning models, or for those who are beginning to interact with power system modelers. It does not assume any experience with power system modeling, or familiarity with electricity planning concepts. This guide can be helpful to a wide range of people, including but not limited to state utility commissioners and commission staff, state energy offices, and intervenors in the planning process. It is targeted to those who are evaluating or reviewing model results presented by a utility, which typically span utility-scale generation and storage (and associated transmission) assets. This guide is designed to provide enough background and guidance that stakeholders can formulate and ask questions to evaluate model results through a deeper understanding of the key assumptions, data, and methods that drive decision outcomes. The ultimate purpose of this work is to increase transparency in the electricity planning process by enabling stakeholders to better engage and evaluate model results. A follow-on guide will provide a deeper dive into emerging topics, which are expected to become more prominent in the electricity planning process, but whose methods and approaches are still being refined.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Evaluation of atmospheric density models and preliminary functional specifications for the Langley Atmospheric Information Retrieval System (LAIRS)

This document presents the results of an extensive survey and comparative evaluation of current atmosphere and wind models for inclusion in the Langley Atmospheric Information Retrieval System (LAIRS). It includes recommended models for use in LAIRS, estimated accuracies for the recommended models, and functional specifications for the development of LAIRS.

Lee, T.↗

SU-E-T-659: Quantitative Evaluation of Patient Setup Accuracy of Stereotactic Radiotherapy with the Frameless 6D-ExacTrac System Using Statistical Modeling

Purpose: To evaluate patient setup accuracy and quantify individual and cumulative positioning uncertainties associated with different hardware and software components of the stereotactic radiotherapy (SRS/SRT) with the frameless-6D-ExacTrac system. Methods: A statistical model was used to evaluate positioning uncertainties of the different components of SRS/SRT treatment with the BrainLAB 6D-ExacTrac system using the positioning shifts of 35 patients having cranial lesions (49 total lesions treated in 1, 3, 5 fractions). All these patients were immobilized with rigid head-and-neck masks, simulated with BrainLAB-localizer and planned with iPlan treatment planning system. Infrared imaging (IR) was used initially to setup patients. Then, stereoscopic x-ray images (XC) were acquired and registered to corresponding digitally-reconstructed-radiographs using bony-anatomy matching to calculate 6D-translational and rotational shifts. When the shifts were within tolerance (0.7mm and 1°), treatment was initiated. Otherwise corrections were applied and additional x-rays were acquired (XV) to verify that patient position was within tolerance. Results: The uncertainties from the mask, localizer, IR-frame, x-ray imaging, MV and kV isocentricity were quantified individually. Mask uncertainty (Translational: Lateral, Longitudinal, Vertical; Rotational: Pitch, Roll, Yaw) was the largest and varied with patients in the range (−1.05−1.50mm, −5.06–3.57mm, −5.51−3.49mm; −1.40−2.40°, −1.24−1.74°, and −2.43−1.90°) obtained from mean of XC shifts for each patient. Setup uncertainty in IR positioning (0.88,2.12,1.40mm, and 0.64,0.83,0.96°) was extracted from standard-deviation of XC. Systematic uncertainties of the localizer (−0.03,−0.01,0.03mm, and −0.03,0.00,−0.01°) and frame (0.18,0.25,−1.27mm,−0.32,0.18, and 0.47°) were extracted from means of all XV setups and mean of all XC distributions, respectively. Uncertainties in isocentricity of the MV radiotherapy machine were (0.27,0.24,0.34mm) and kV-imager (0.15,−0.4,0.21mm). Conclusion: A statistical model was developed to evaluate the individual and cumulative systematic and random uncertainties induced by the different hardware and software components of the 6D-ExacTrac-system. The immobilization mask was associated with the largest positioning uncertainty.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Performance Evaluation of a Two-Dimensional Flood Model on Heterogeneous High-Performance Computing Architectures

This paper describes the implementation of a two-dimensional hydrodynamic flood model with two different numerical schemes on heterogeneous high-performance computing architectures. Both schemes were able to solve the nonlinear hyperbolic shallow water equations using an explicit upwind first-order approach on finite differences and finite volumes, respectively, and were conducted using MPI and CUDA. Four different test cases were simulated on the Summit supercomputer at Oak Ridge National Laboratory. Both numerical schemes scaled up to 128 nodes (768 GPUs) with a maximum 98.2x speedup of over 1 GPU. The lowest run time for the 10 day Hurricane Harvey event simulation at 5 meter resolution (272 million grid cells) was 50 minutes. GPUDirect communication proved to be more convenient than the standard communication strategy. Both strong and weak scaling are shown.

Sharif, Md Bulbul↗