Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 757 records · Page 42

Real-time automated failure analysis for on-orbit operations

A system which is to provide real-time failure analysis support to controllers at the NASA Johnson Space Center Control Center Complex (CCC) for both Space Station and Space Shuttle on-orbit operations is described. The system employs monitored systems' models of failure behavior and model evaluation algorithms which are domain-independent. These failure models are viewed as a stepping stone to more robust algorithms operating over models of intended function. The described system is designed to meet two sets of requirements. It must provide a useful failure analysis capability enhancement to the mission controller. It must satisfy CCC operational environment constraints such as cost, computer resource requirements, verification, and validation. The underlying technology and how it may be used to support operations is also discussed.

Kirby, Sarah↗

Improving Climate Projections Using "Intelligent" Ensembles

Recent changes in the climate system have led to growing concern, especially in communities which are highly vulnerable to resource shortages and weather extremes. There is an urgent need for better climate information to develop solutions and strategies for adapting to a changing climate. Climate models provide excellent tools for studying the current state of climate and making future projections. However, these models are subject to biases created by structural uncertainties. Performance metrics-or the systematic determination of model biases-succinctly quantify aspects of climate model behavior. Efforts to standardize climate model experiments and collect simulation data-such as the Coupled Model Intercomparison Project (CMIP)-provide the means to directly compare and assess model performance. Performance metrics have been used to show that some models reproduce present-day climate better than others. Simulation data from multiple models are often used to add value to projections by creating a consensus projection from the model ensemble, in which each model is given an equal weight. It has been shown that the ensemble mean generally outperforms any single model. It is possible to use unequal weights to produce ensemble means, in which models are weighted based on performance (called "intelligent" ensembles). Can performance metrics be used to improve climate projections? Previous work introduced a framework for comparing the utility of model performance metrics, showing that the best metrics are related to the variance of top-of-atmosphere outgoing longwave radiation. These metrics improve present-day climate simulations of Earth's energy budget using the "intelligent" ensemble method. The current project identifies several approaches for testing whether performance metrics can be applied to future simulations to create "intelligent" ensemble-mean climate projections. It is shown that certain performance metrics test key climate processes in the models, and that these metrics can be used to evaluate model quality in both current and future climate states. This information will be used to produce new consensus projections and provide communities with improved climate projections for urgent decision-making.

Baker, Noel C.↗

Improving Climate Projections Using "Intelligent" Ensembles

Recent changes in the climate system have led to growing concern, especially in communities which are highly vulnerable to resource shortages and weather extremes. There is an urgent need for better climate information to develop solutions and strategies for adapting to a changing climate. Climate models provide excellent tools for studying the current state of climate and making future projections. However, these models are subject to biases created by structural uncertainties. Performance metrics-or the systematic determination of model biases-succinctly quantify aspects of climate model behavior. Efforts to standardize climate model experiments and collect simulation data-such as the Coupled Model Intercomparison Project (CMIP)-provide the means to directly compare and assess model performance. Performance metrics have been used to show that some models reproduce present-day climate better than others. Simulation data from multiple models are often used to add value to projections by creating a consensus projection from the model ensemble, in which each model is given an equal weight. It has been shown that the ensemble mean generally outperforms any single model. It is possible to use unequal weights to produce ensemble means, in which models are weighted based on performance (called "intelligent" ensembles). Can performance metrics be used to improve climate projections? Previous work introduced a framework for comparing the utility of model performance metrics, showing that the best metrics are related to the variance of top-of-atmosphere outgoing longwave radiation. These metrics improve present-day climate simulations of Earth's energy budget using the "intelligent" ensemble method. The current project identifies several approaches for testing whether performance metrics can be applied to future simulations to create "intelligent" ensemble-mean climate projections. It is shown that certain performance metrics test key climate processes in the models, and that these metrics can be used to evaluate model quality in both current and future climate states. This information will be used to produce new consensus projections and provide communities with improved climate projections for urgent decision-making.

Baker, Noel C.↗

A fusion protein of HCMV IE1 exon4 and IE2 exon5 stimulates potent cellular immunity in an MVA vaccine vector

A therapeutic CMV vaccine incorporating an antigenic repertoire capable of eliciting a cellular immune response has yet to be successfully implemented for patients who already have acquired an infection. To address this problem, we have developed a vaccine candidate derived from modified vaccinia Ankara (MVA) that expresses three immunodominant antigens (pp65, IE1, IE2) from CMV. The novelty of this vaccine is the fusion of two adjacent exons from the immediate-early region of CMV, their successful expression in MVA, and robust immunogenicity in both primary and memory response models. Evaluation of the immunogenicity of the viral vaccine in mouse models shows that it can stimulate primary immunity against all three antigens in both the CD4{sup +} and CD8{sup +} T cell subsets. Evaluation of human PBMC from healthy CMV-positive donors or patients within 6 months of receiving hematopoietic cell transplant shows robust stimulation of existing CMV-specific CD4{sup +} and CD8{sup +} T cell subsets.

60 APPLIED LIFE SCIENCES↗

SU-D-BRB-02: Patient-Specific Rectal Toxicity Predictor Based Plan Quality Control for Prostate Stereotactic Body Radiation Therapy (SBRT)

Purpose: To develop a patient-specific rectal toxicity predictor guided plan quality control tool for prostate SBRT plans. Methods: For prostate SBRT cases, four segments of rectal walls including peri-prostatic anterior rectal wall, peri-prostatic lateral rectal walls, peri-prostatic posterior rectal wall and rectum superior to prostate are identified as organs at risk and the circumference of rectal wall receiving more than 39 Gy (CRW39) and 24 Gy (CRW24) are rectal toxicity predictors. In this new geometry-dosimetry model, a patient geometry descriptor, differential circumference of rectal wall (dCRW) is used as model input geometry parameters and plan dosimetric endpoints CRW39 and CRW24 are output dosimetric parameters. Linear models are built to correlate dCRW to both CRW39 and CRW24 and established with both a linear regression method and a modified bagging ensemble machine learning method. 27 SBRT prostate cases are retrospectively studied from a dose-escalated clinical trial research. 20 prescribed 50 Gy SBRT cases are recruited to train the model and the other rescaled 7 cases are used to evaluated model feasibility and accuracy. Results: Each solved linear coefficient sequence related to CRW39 or CRW24 is a one-dimensional decreasing function of the distance from the PTV boundary, indicating that the different locations of each rectal circumference have different contributions to each particular dosimetric endpoint. The fitting errors for those trained 20 prostate SBRT cases are small with mean values of 2.39%, 2.45% relative to the endpoint values for SBRT rectal toxicity predictor CRW39 and CRW24 respectively. 1 out of 7 evaluation plans is identified as poor quality plan. After re-planning, the CRW39 and CRW24 can be reduced by 3.34% and 3%, without sacrificing PTV coverage. Conclusion: The proposed patient geometry-plan toxicity predictor model for SBRT plans can be successfully applied to plan quality control for prostate SBRT cases.

60 APPLIED LIFE SCIENCES↗

Incorporating the Acclimation of Photosynthesis and Leaf Respiration in the Noah‐MP Land Surface Model: Model Development and Evaluation

Abstract Realistic simulation of leaf photosynthetic and respiratory processes is needed for accurate prediction of the global carbon cycle. These two processes systematically acclimate to long‐term environmental changes by adjusting photosynthetic and respiratory traits (e.g., the maximum photosynthetic capacity at 25°C ( V cmax,25 ) and the leaf respiration rate at 25°C ( R 25 )) following increasingly well‐understood principles. While some land surface models (LSMs) now account for thermal acclimation, they do so by assigning empirical parameterizations for individual plant functional types (PFTs). Here, we have implemented an Eco‐Evolutionary Optimality (EEO)‐based scheme to represent the universal acclimation of photosynthesis and leaf respiration to multiple environmental effects, and that therefore requires no PFT‐specific parameterizations, in a standard version of the widely used LSM, Noah MP. We evaluated model performance with plant trait data from a 5‐year experiment and extensive global field measurements, and carbon flux measurements from FLUXNET2015. We show that observed R 25 and V cmax,25 vary substantially both temporally and spatially within the same PFT ( C.V. >20%). Our EEO‐based scheme captures 62% of the temporal and 70% of the spatial variations in V cmax,25 (73% and 54% of the variations in R 25 ). The standard scheme underestimates gross primary production by 10% versus 2% for the EEO‐based scheme and generates a larger spread in r (correlation coefficient) across flux sites (0.79 ± 0.16 vs. 0.84 ± 0.1, mean ± S.D.). The standard scheme greatly overestimates canopy respiration (bias: ∼200% vs. 8% for the EEO scheme), resulting in less CO 2 uptake by terrestrial ecosystems. Our approach thus simulates climate‐carbon coupling more realistically, with fewer parameters.

Ren, Yanghang [Department of Earth System Science ↗

Normal Tissue Complication Probability Modeling of Radiation-Induced Hypothyroidism After Head-and-Neck Radiation Therapy

Purpose: To determine the dose-response relationship of the thyroid for radiation-induced hypothyroidism in head-and-neck radiation therapy, according to 6 normal tissue complication probability models, and to find the best-fit parameters of the models. Methods and Materials: Sixty-five patients treated with primary or postoperative radiation therapy for various cancers in the head-and-neck region were prospectively evaluated. Patient serum samples (tri-iodothyronine, thyroxine, thyroid-stimulating hormone [TSH], free tri-iodothyronine, and free thyroxine) were measured before and at regular time intervals until 1 year after the completion of radiation therapy. Dose-volume histograms (DVHs) of the patients' thyroid gland were derived from their computed tomography (CT)-based treatment planning data. Hypothyroidism was defined as increased TSH (subclinical hypothyroidism) or increased TSH in combination with decreased free thyroxine and thyroxine (clinical hypothyroidism). Thyroid DVHs were converted to 2 Gy/fraction equivalent doses using the linear-quadratic formula with {alpha}/{beta} = 3 Gy. The evaluated models included the following: Lyman with the DVH reduced to the equivalent uniform dose (EUD), known as LEUD; Logit-EUD; mean dose; relative seriality; individual critical volume; and population critical volume models. The parameters of the models were obtained by fitting the patients' data using a maximum likelihood analysis method. The goodness of fit of the models was determined by the 2-sample Kolmogorov-Smirnov test. Ranking of the models was made according to Akaike's information criterion. Results: Twenty-nine patients (44.6%) experienced hypothyroidism. None of the models was rejected according to the evaluation of the goodness of fit. The mean dose model was ranked as the best model on the basis of its Akaike's information criterion value. The D{sub 50} estimated from the models was approximately 44 Gy. Conclusions: The implemented normal tissue complication probability models showed a parallel architecture for the thyroid. The mean dose model can be used as the best model to describe the dose-response relationship for hypothyroidism complication.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Structure of ducted particle-laden turbulent jets

New and existing measurements have been used to develop and evaluate models of particle-laden jets and sprays, yielding encouraging results for a stochastic separated flow (SSF) model which treats effects of interphase slip and turbulence on the basis of random walk computations of particle or drop motion. An evaluation of the SSF model using the measurements of Modarress et al. (1982, 1983) yielded results comparable to earlier work, suggesting that data from coflowing jets in ducts should be accompanied by static pressure measurements, since even small pressure gradients appreciably influence jet properties.

Zhang, Q.-F.↗

Model and Analytic Processes for Export License Assessments

This paper represents the Department of Energy Office of Nonproliferation Research and Development (NA-22) Simulations, Algorithms and Modeling (SAM) Program's first effort to identify and frame analytical methods and tools to aid export control professionals in effectively predicting proliferation intent; a complex, multi-step and multi-agency process. The report focuses on analytical modeling methodologies that alone, or combined, may improve the proliferation export control license approval process. It is a follow-up to an earlier paper describing information sources and environments related to international nuclear technology transfer. This report describes the decision criteria used to evaluate modeling techniques and tools to determine which approaches will be investigated during the final 2 years of the project. The report also details the motivation for why new modeling techniques and tools are needed. The analytical modeling methodologies will enable analysts to evaluate the information environment for relevance to detecting proliferation intent, with specific focus on assessing risks associated with transferring dual-use technologies. Dual-use technologies can be used in both weapons and commercial enterprises. A decision-framework was developed to evaluate which of the different analytical modeling methodologies would be most appropriate conditional on the uniqueness of the approach, data availability, laboratory capabilities, relevance to NA-22 and Office of Arms Control and Nonproliferation (NA-24) research needs and the impact if successful. Modeling methodologies were divided into whether they could help micro-level assessments (e.g., help improve individual license assessments) or macro-level assessment. Macro-level assessment focuses on suppliers, technology, consumers, economies, and proliferation context. Macro-level assessment technologies scored higher in the area of uniqueness because less work has been done at the macro level. An approach to developing testable hypotheses for the macro-level assessment methodologies is provided. The outcome of this works suggests that we should develop a Bayes Net for micro-level analysis and continue to focus on Bayes Net, System Dynamics and Economic Input/Output models for assessing macro-level problems. Simultaneously, we need to develop metrics for assessing intent in export control, including the risks and consequences associated with all aspects of export control.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

PDRD (SR13046) Tritium Production Final Report

Utilizing the results of Texas A&M University (TAMU) senior design projects on tritium production in four different small modular reactors (SMR), the Savannah River National Laboratory’s (SRNL) developed an optimization model evaluating tritium production versus uranium utilization under a FY2013 plant directed research development (PDRD) project. The model is a tool that can evaluate varying scenarios and various reactor designs to maximize the production of tritium per unit of unobligated United States (US) origin uranium that is in limited supply.

07 ISOTOPE AND RADIATION SOURCES↗

Software enhancements to the IVSEM model of the CTBTO IMS.

Sandia National Laboratories (SNL) developed the Integrated Verification System Evaluation Model (IVSEM) to estimate the performance of the International Monitoring System (IMS) operated by the Comprehensive Nuclear Test Ban Treaty Organization (CTBTO). IVSEM was developed in several phases between 1995 and 2000. The model was developed in FORTRAN with an IDL-based user interface and was compiled for Windows and UNIX operating systems. Continuing interest in this analysis capability, coupled with numerous advances in desktop computer hardware and software since IVSEM was written, enabled significant improvements to IVSEM run-time performance and data analysis capabilities. These improvements were implemented externally without modifying the FORTRAN executables, which had been previously verified. This paper describes the parallelization approach developed to significantly reduce IVSEM run-times and the new test setup and analysis tools developed to facilitate better IVSEM operation.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Moving beyond the total sea ice extent in gauging model biases

Here, reproducing characteristics of observed sea ice extent remains an important climate modeling challenge. This study describes several approaches to improve how model biases in total sea ice distribution are quantified, and applies them to historically forced simulations contributed to phase 5 of the Coupled Model Intercomparison Project (CMIP5). The quantity of hemispheric total sea ice area, or some measure of its equatorward extent, is often used to evaluate model performance. A new approach is introduced that investigates additional details about the structure of model errors, with an aim to reduce the potential impact of compensating errors when gauging differences between simulated and observed sea ice. Using multiple observational datasets, several new methods are applied to evaluate the climatological spatial distribution and the annual cycle of sea ice cover in 41 CMIP5 models. It is shown that in some models, error compensation can be substantial, for example resulting from too much sea ice in one region and too little in another. Error compensation tends to be larger in models that agree more closely with the observed total sea ice area, which may result from model tuning. The results herein suggest that consideration of only the total hemispheric sea ice area or extent can be misleading when quantitatively comparing how well models agree with observations. Further work is needed to fully develop robust methods to holistically evaluate the ability of models to capture the finescale structure of sea ice characteristics; however, the “sector scale” metric used here aids in reducing the impact of compensating errors in hemispheric integrals.

54 ENVIRONMENTAL SCIENCES↗

A critical evaluation of the upper ocean heat budget in the Climate Forecast System Reanalysis data for the south central equatorial Pacific

Coupled ocean-atmospheric models suffer from the common bias of a spurious rain belt south of the central equatorial Pacific throughout the year. Observational constraints on key processes responsible for this bias are scarce. The recently available reanalysis from a coupled model system for the National Centers for Environmental Prediction (NCEP) Climate Forecast System Reanalysis (CFSR) data is a potential benchmark for climate models in this region. Its suitability for model evaluation and validation, however, needs to be established. This paper examines the mixed layer heat budget and the ocean surface currents - key factors for the sea surface temperature control in the double Inter-Tropical Convergence Zone in the central Pacific - from 5{sup o}S to 10{sup o}S and 170{sup o}E to 150{sup o}W. Two independent approaches are used. The first approach is through comparison of CFSR data with collocated station observations from field experiments; the second is through the residual analysis of the heat budget of the mixed layer. We show that the CFSR overestimates the net surface flux in this region by 23 W m{sup -2}. The overestimated net surface flux is mainly due to an even larger overestimation of shortwave radiation by 44 W m{sup -2}, which is compensated by a surface latent heat flux overestimated by 14 W m{sup -2}. However, the quality of surface currents and the associated oceanic heat transport in CFSR are not compromised by the surface flux biases, and they agree with the best available estimates. The uncertainties of the observational data from field experiments are also briefly discussed in the present study.

54 ENVIRONMENTAL SCIENCES↗

Reliability estimation of an advanced nuclear fuel using coupled active learning, multifidelity modeling, and subset simulation

Tristructural isotropic (TRISO)-coated particle fuel is a robust nuclear fuel and determining its reliability is critical for the success of advanced nuclear technologies. However, TRISO failure probabilities are small and the associated computational models are expensive. Here we used coupled active learning, multifidelity modeling, and subset simulation to estimate the failure probabilities of TRISO fuels using several 1D and 2D models. With multifidelity modeling, we replaced expensive high-fidelity (HF) model evaluations with information fusion from two low-fidelity (LF) models. For the 1D TRISO models, we considered three multifidelity modeling strategies: only Kriging, Kriging LF prediction plus Kriging correction, and deep neural network (DNN) LF prediction plus Kriging correction. While the results across these multifidelity modeling strategies compared satisfactorily, strategies employing information fusion from two LF models called the HF model least often. Next, for the 2D TRISO model, we considered two multifidelity modeling strategies: DNN LF prediction plus Kriging correction (data-driven) and 1D TRISO LF prediction plus Kriging correction (physics-based). The physics-based strategy, as expected, consistently required the fewest calls to the HF model. However, the data-driven strategy had a lower overall simulation time since the DNN predictions are instantaneous, and the 1D TRISO model requires a non-negligible simulation time.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

The Monolithic Heat Pipe Microreactor Reference Plant Model Updates

This work presents the latest improvements to, and investigations performed with, the generic monolithic heat-pipe-cooled microreactor reference plant model for the United States Nuclear Regulatory Commission. This model serves as the foundation for the future detailed design evaluation models based on license applications. This model has been developed with the Comprehensive Reactor Analysis Bundle (BlueCRAB) and its specifications are based on open literature publications for the eVinci™ design . BlueCRAB is the U.S. Nuclear Regulatory Commission non-light-water reactor analysis system based on MOOSE, the Multiphysics Object-Oriented Simulation Environment framework, which can couple the Griffin, BISON, and Sockeye applications to resolve the various physics that are essential for the safety analysis of this type of reactor system. The core specifications include tristructural isotropic fuel, graphite monolith, graphite reflectors, and drums composed of graphite and B 4 C.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

Optimal control of a low wing-loading STOL aircraft

Linear optimal quadratic control theory is applied to a low wing-loading STOL aircraft for ride quality and flight path following. Design criteria include minimum rms response to wind turbulence and desired transient response characteristics. Design techniques include proper choosing of design versus evaluation models, choosing appropriate performance index responses, and use of classical evaluation techniques. Results are obtained through a combination of frequency response shaping and gust observation. Effects of control rate and authority saturation are examined with a new rapid calculation of random input describing functions. Parameter sensitivity is also evaluated using a Liapunov type matrix equation.

Cunningham, T. B.↗

Classification of Clouds and Deep Convection from GEOS-5 Using Satellite Observations

With the increased resolution of global atmospheric models and the push toward global cloud resolving models, the resemblance of model output to satellite observations has become strikingly similar. As we progress with our adaptation of the Goddard Earth Observing System Model, Version 5 (GEOS-5) as a high resolution cloud system resolving model, evaluation of cloud properties and deep convection require in-depth analysis beyond a visual comparison. Outgoing long-wave radiation (OLR) provides a sufficient comparison with infrared (IR) satellite imagery to isolate areas of deep convection. We have adopted a binning technique to generate a series of histograms for OLR which classify the presence and fraction of clear sky versus deep convection in the tropics that can be compared with a similar analyses of IR imagery from composite Geostationary Operational Environmental Satellite (GOES) observations. We will present initial results that have been used to evaluate the amount of deep convective parameterization required within the model as we move toward cloud system resolving resolutions of 10- to 1-km globally.

Putman, William↗

MUlti-SpEctral, MUlti-SpEcies, MUlti-SatEllite (MUSES) Retrieval Algorithm: Towards Extending Multi-Decadal NASA EOS Atmospheric Composition Data Records

Multi-Spectra, Multi-Species, Multi-Sensors (MUSES): Builds off of heritage from the Tropospheric Emission Spectrometer (TES) optimal estimation (OE) algorithm to combine a priori and satellite data, including rigorous error analysis diagnostics and observation operators needed for trend analysis, climate model evaluation, and data assimilation; has generic design to incorporate forward model radiances from hyperspectral measurements from multiple sensors into the joint retrieval algorithm.

Fu, Dejian↗