Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Improving parameter estimation for column experiments by multi-model evaluation and comparison

Reliability of column experiment data interpretation by fitting the data to a transport model using the nonlinear least squares method is of concern when the complexity of the experiment increases. Misfits often occur and are accepted with the excuse of system complexity. More often, overfits are published and accepted uncritically. The objective of this work was to improve the reliability of column experiment data interpretation to provide insights for future investigation and select estimates for application prediction. We proposed to achieve this goal by evaluating and comparing various estimates that consider the uncertainties in experimental conditions, sensitivities of model parameters, alternative parameterizations and models. Two examples from the literature were selected for the illustration. Sensitivity analysis showed that the experimentally-controlled parameters (water content, flow rate and pulse volume) are much more sensitive than the transport parameters (dispersivity and mass transfer coefficient) in breakthrough curve fitting. Fixing these sensitive parameters may lead to misfit or biased estimate. Estimating sensitive parameters helps diagnose and remediate the misfit. The dispersion coefficient was parameterized as a function of solute molecular diffusion coefficient, medium dispersivity and average pore water velocity to reduce the number of estimated parameters for multiple tracers. Nonlinear least squares statistics were used to evaluate the goodness-of-fit for the multiple estimates obtained using different fitting schemes (e.g., fixing or estimating parameters), parameterizations and models (e.g., equilibrium or nonequilibrium). The F-test and Akaike s Information Criteria were used for the discrimination of statistically reasonable estimates. Our results suggest that sensitivity/uncertainty analysis and model evaluation /discrimination are useful to improve the reliability of column experiment data interpretation.

58 GEOSCIENCES↗

Human Thermal Model Evaluation Using the JSC Human Thermal Database

The human thermal database developed at the Johnson Space Center (JSC) is used to evaluate a set of widely used human thermal models. This database will facilitate a more accurate evaluation of human thermoregulatory response using in a variety of situations, including those situations that might otherwise prove too dangerous for actual testing--such as extreme hot or cold splashdown conditions. This set includes the Wissler human thermal model, a model that has been widely used to predict the human thermoregulatory response to a variety of cold and hot environments. These models are statistically compared to the current database, which contains experiments of human subjects primarily in air from a literature survey ranging between 1953 and 2004 and from a suited experiment recently performed by the authors, for a quantitative study of relative strength and predictive quality of the models. Human thermal modeling has considerable long term utility to human space flight. Such models provide a tool to predict crew survivability in support of vehicle design and to evaluate crew response in untested environments. It is to the benefit of any such model not only to collect relevant experimental data to correlate it against, but also to maintain an experimental standard or benchmark for future development in a readily and rapidly searchable and software accessible format. The Human thermal database project is intended to do just so; to collect relevant data from literature and experimentation and to store the data in a database structure for immediate and future use as a benchmark to judge human thermal models against, in identifying model strengths and weakness, to support model development and improve correlation, and to statistically quantify a model s predictive quality.

Cognata, T.↗

An evaluative model of system performance in manned teleoperational systems

Manned teleoperational systems are used in aerospace operations in which humans must interact with machines remotely. Manual guidance of remotely piloted vehicles, controling a wind tunnel, carrying out a scientific procedure remotely are examples of teleoperations. A four input parameter throughput (Tp) model is presented which can be used to evaluate complex, manned, teleoperations-based systems and make critical comparisons among candidate control systems. The first two parameters of this model deal with nominal (A) and off-nominal (B) predicted events while the last two focus on measured events of two types, human performance (C) and system performance (D). Digital simulations showed that the expression A(1-B)/C+D) produced the greatest homogeneity of variance and distribution symmetry. Results from a recently completed manned life science telescience experiment will be used to further validate the model. Complex, interacting teleoperational systems may be systematically evaluated using this expression much like a computer benchmark is used.

Haines, Richard F.↗

Sequentially Executed Model Evaluation Framework

Provides a message passing framework between generic input, model and output drivers, and specifies an API for developing such drivers. Also provides batch and real-time controllers which step the model and 1/0 through the time domain (or other discrete domain), and sample 1/0 drivers. This is a Framework library framework, and does not, itself, solve any problems or execute any modelling. The SeMe framework aids in development of models which operate on sequential information, such as time-series, where evaluation is based on prior results combined with new data for this iteration. Ha) applications in quality monitoring, and was developed as part of the CANARY-EDS software, where real-time water quality data is being analyzed

Hart, David↗

User's appraisal of yield model evaluation criteria

The five major potential USDA users of AgRISTAR crop yield forecast models rated the Yield Model Development (YMD) project Test and Evaluation Criteria by the importance placed on them. These users were agreed that the "TIMELINES" and "RELIABILITY" of the forecast yields would be of major importance in determining if a proposed yield model was worthy of adoption. Although there was considerable difference of opinion as to the relative importance of the other criteria, "COST", "OBJECTIVITY", "ADEQUACY", AND "MEASURES OF ACCURACY" generally were felt to be more important that "SIMPLICITY" and "CONSISTENCY WITH SCIENTIFIC KNOWLEDGE". However, some of the comments which accompanied the ratings did indicate that several of the definitions and descriptions of the criteria were confusing.

Warren, F. B.↗

ARID Raceway Temperature Model Evaluation

The ARID (Algae Raceway Integrated Design) raceway temperature model, written in Visual Basic for Applications, calculates algae culture temperature in ARID and conventional raceways as a function of meteorological data from agricultural weather stations. The model is an atmospheric boundary layer model and uses energy balance terms from the Penman-Monteith equation. The original model used evapotranspiration, as calculated by the weather station, and assumed an average daily albedo. In this research, equations for albedo and for direct calculation of evaporation from atmospheric parameters were added to the ARID model. Albedo was calculated with the Fresnel equations. The Penman-Monteith model was adjusted for algae raceway evaporation rather than reference evapotranspiration. The new model components were calibrated with data from a 21-day algae growth experiment in the ARID raceway during winter 2011. The model was also adapted to model variable circulation times and heat transfer through a canal cover. In summary, this research improved and calibrated the ARID temperature model for future use.

09 BIOMASS FUELS↗

Pumped Storage Hydro Model Evaluation Report Task 1: Tool Assessment

Pumped storage hydro (PSH) projects support various aspects of power system operations. However, determining the value of PSH projects and the many services and contributions to the system they provide can be a challenge. The PSH techno-economic studies being carried out by a 5-lab consortium are defining extensive modeling approaches for evaluating the economic benefits of PSH projects. To enhance the impact of the valuation approach and follow-on techno-economic studies, Pacific Northwest National Laboratory (PNNL) will be developing a set of online tools that industry could adopt and use to advance PSH projects in the US. The problem is that the research team is not entirely clear on the features of the tools that would offer the most significant impact to industry. Thus, year one of this effort is designed to first define the needs of industry before developing the PSH valuation tool. The first step in defining industry needs is to evaluate existing tool packages, including those created by DOE and those by industry groups, to both measure usage levels and define tool attributes that maximize industry impact. The second step in this process will be to hold a workshop designed to elicit direct industry input. Given the breadth and nature of the task, which is to define ideal model attributes, the research team did not constrain the assessment to storage valuation tools. Several tools were assessed, and their characteristics compared against a list of “ideal” attributes that included ease of use, generalizability of results, and analytical flexibility. Information regarding usage and impact were then compared to specific model attributes to define which attributes appear to drive success. The following key lessons and implications can be drawn from the analysis: Models with high ease of use were often the most adopted by industry, though they did not necessarily have high publication reference records, meaning a published article made direct reference to the tool. For example, REScheck TM and COMcheck TM , two tools which share a site, have seen a high amount of web traffic and usage. Similarly, Storage Value Estimate Tool (StorageVET®), an energy storage valuation model, has a large number of organizations using it despite a low publication reference record. A common feature between these models is that they are web-hosted, theoretically making them more easily accessible with graphic user interfaces (GUIs) that are more navigable; Models with higher usage statistics were typically free and/or open source. While it may be unsurprising that this attribute could be considered a draw for users, there is the potential for overestimating the tool's value based on popularity due to the fact that some users may have used the model simply because it was free and not because it is the best model; Models that offer more in-depth analysis and a wide coverage of applications (e.g. VOLTTRON™, GridLAB-D™, and Integrated Valuation of Ecosystem Services and Tradeoffs (InVest)) have shown a high number of publication references as well as showing a high number of downloads in the case of GridLAB-D™. It should be noted that the figure below does not include the model with the highest number of publication references (EnergyPlus™) as it has been removed to show clearer distinction between the remaining tools; The most popular model within the evaluation list was EnergyPlus™. This tool has the highest number of publication references (2,034) of all models considered and contains both a high ease of use level as well as broad application flexibility. Its high usage may be correlated, however, with the presumed wider audience conducting energy consumption modeling compared to the audiences for some of the other tools in this list; Models that were low in both general usage and publication reference generally did not exhibit characteristics considered ideal by the research team. Some of these models offered a narrower range of analytical applications or lower generalizability of results. This would automatically restrict the pool of potential users to those who fall within a narrow range of specific technologies or applications. This report also explores two model structures for the PSH tool: decision tree-based, which would guide the user through the valuation methods defined in the PSH techno-economic studies currently being carried out by DOE, and an all-inclusive comprehensive model. The strengths of each were explored and a hybrid approach was defined that would embed a price taker model into the decision tree approach. Such an approach would provide a comprehensive approach for modeling small-scale PSH while recognizing the challenges associated with implementing a price-maker approach in an all-inclusive model. Insight gained in this analysis of other models can aid in narrowing down the list of effective and sought-after attributes, and in defining model structure. Next steps toward developing a model for storage valuation will be to gain direct industry input through workshops.

13 HYDRO ENERGY↗

Evaluating Model Parameterizations of Submicron Aerosol Scattering and Absorption with in situ Data from ARCTAS 2008

Accurate modeling of the scattering and absorption of ultraviolet and visible radiation by aerosols is essential for accurate simulations of atmospheric chemistry and climate. Closure studies using in situ measurements of aerosol scattering and absorption can be used to evaluate and improve models of aerosol optical properties without interference from model errors in aerosol emissions, transport, chemistry, or deposition rates. Here we evaluate the ability of four externally mixed, fixed size distribution parameterizations used in global models to simulate submicron aerosol scattering and absorption at three wavelengths using in situ data gathered during the 2008 Arctic Research of the Composition of the Troposphere from Aircraft and Satellites (ARCTAS) campaign. The four models are the NASA Global Modeling Initiative (GMI) Combo model, GEOS-Chem v9- 02, the baseline configuration of a version of GEOS-Chem with online radiative transfer calculations (called GC-RT), and the Optical Properties of Aerosol and Clouds (OPAC v3.1) package. We also use the ARCTAS data to perform the first evaluation of the ability of the Aerosol Simulation Program (ASP v2.1) to simulate submicron aerosol scattering and absorption when in situ data on the aerosol size distribution are used, and examine the impact of different mixing rules for black carbon (BC) on the results. We find that the GMI model tends to overestimate submicron scattering and absorption at shorter wavelengths by 10-23 percent, and that GMI has smaller absolute mean biases for submicron absorption than OPAC v3.1, GEOS-Chem v9-02, or GC-RT. However, the changes to the density and refractive index of BC in GCRT improve the simulation of submicron aerosol absorption at all wavelengths relative to GEOS-Chem v9-02. Adding a variable size distribution, as in ASP v2.1, improves model performance for scattering but not for absorption, likely due to the assumption in ASP v2.1 that BC is present at a constant mass fraction throughout the aerosol size distribution. Using a core-shell mixing rule in ASP overestimates aerosol absorption, especially for the fresh biomass burning aerosol measured in ARCTAS-B, suggesting the need for modeling the time-varying mixing states of aerosols in future versions of ASP.

Alvarado, Matthew J.↗

Exploring the use of dynamic linear panel data models for evaluating energy/economy/environment models — an application for the transportation sector

This paper evaluates the RoSE transportation sector scenarios of the GCAM and REMIND models for the U.S. region by comparing these models’ projected future final energy use levels (1) to historical trends and (2) across each other. To estimate and compare the intrinsic model elasticities, we explore the use of dynamic linear panel data models. On the basis of 26 scenarios (panels) from each of the two energy system models, our analysis suggests that most intrinsic model elasticities with respect to final energy use are roughly comparable to each other and to those observed historically. The key difference is the income elasticity of final energy use, which is significantly smaller for the two energy models than that derived from the historical development. This difference and other minor inconsistencies are interpreted through key assumptions underlying both energy system models.

54 ENVIRONMENTAL SCIENCES↗

An AeroCom–AeroSat study: intercomparison of satellite AOD datasets for aerosol model evaluation

To better understand and characterize current uncertainties in the important observational constraint of climate models of aerosol optical depth (AOD), we evaluate and intercompare 14 satellite products, representing nine different retrieval algorithm families using observations from five different sensors on six different platforms. The satellite products (super-observations consisting of 1°×1° daily aggregated retrievals drawn from the years 2006, 2008 and 2010) are evaluated with AErosol RObotic NETwork (AERONET) and Maritime Aerosol Network (MAN) data. Results show that different products exhibit different regionally varying biases (both under- and overestimates) that may reach ±50 %, although a typical bias would be 15 %–25 % (depending on the product). In addition to these biases, the products exhibit random errors that can be 1.6 to 3 times as large. Most products show similar performance, although there are a few exceptions with either larger biases or larger random errors. The intercomparison of satellite products extends this analysis and provides spatial context to it. In particular, we show that aggregated satellite AOD agrees much better than the spatial coverage (often driven by cloud masks) within the 1°×1° grid cells. Up to ∼50 % of the difference between satellite AOD is attributed to cloud contamination. The diversity in AOD products shows clear spatial patterns and varies from 10 % (parts of the ocean) to 100 % (central Asia and Australia). More importantly, we show that the diversity may be used as an indication of AOD uncertainty, at least for the better performing products. This provides modellers with a global map of expected AOD uncertainty in satellite products, allows assessment of products away from AERONET sites, can provide guidance for future AERONET locations and offers suggestions for product improvements. We account for statistical and sampling noise in our analyses. Sampling noise, variations due to the evaluation of different subsets of the data, causes important changes in error metrics. The consequences of this noise term for product evaluation are discussed.

Aerosol↗

Evaluation of the cytotoxic and cellular proteome impacts of food-grade TiO2 (E171) using simulated gastrointestinal digestions and a tri-culture small intestinal epithelial model

Engineered nanomaterials (ENMs) are widely used in the food industry; however, regulations for ENMs in food are still in the early stages of development due to insufficient health data. This study investigated the cytotoxicity and changes to the proteomic profile in an in vitro small intestinal epithelium model after exposure to digested food models containing the ubiquitous engineered particulate food additive, TiO2 (E171). TiO2 at 0.75% or 1.5% (w/w) concentrations in either a fasting food model (FFM) or a standardized food model (SFM) based on American diet were digested using an in vitro oral-gastric-small intestinal simulator, and the resulting digestas were applied to a small intestinal epithelium tri-culture cellular model. Effects on cell layer integrity, cytotoxicity, and oxidative stress were assessed. In order to explore impact on cellular processes beyond basic cytotoxicity, mass spectrometrybased quantitative proteomic analyses of control and exposed tri-culture cells was performed. TiO2 in FFM, but not in SFM, produced significant, dose-dependent cytotoxicity, and at the higher dose caused significant oxidative stress, indicative of a food matrix effect. No significant perturbations of the cellular proteome were observed with TiO2 in either FFM or SFM food models. However, proteins involved in the tricarboxylic acid (TCA) cycle, gluconeogenesis, and translation were up-regulated by digestas from SFM compared to those from FFM, also indicative of food matrix effects in the case of food models alone. Interestingly, the differences in profiles between the two food models was more pronounced in the presence of TiO2 indicative of food matrix effects. Together, these results indicate that TiO2 in a fasting diet may be slightly cytotoxic, and that ingested TiO2 does not significantly alter the epithelial proteome, whereas the food matrix alone can have a dramatic effect on the proteome.

Cao, Xiaoqiong↗

One-way coupling of an integrated assessment model and a water resources model: evaluation and implications of future changes over the US Midwest

An integrated model is being developed to advance our understanding of the interactions between human activities, terrestrial system and water cycle, and how system interactions will be affected by a changing climate at the regional scale. As a first step towards that goal, a global integrated assessment model including a waterdemand model is coupled offline with a land surface hydrology – routing – water resources management model. A spatial and temporal disaggregation approach is developed to project the annual regional water demand simulations into a daily time step and subbasin representation. The model demonstrated reasonable ability to represent the historical flow regulation and water supply over the Midwest (Missouri, Upper Mississippi and Ohio). Implications for the future flow regulation, water supply and supply deficit are investigated using a climate change projection with the B1 emission scenario which affects both natural flow and water demand. Over the Midwest, changes in flow regulation are mostly driven by the change in natural flow due to the limited storage capacity over the Ohio and Upper Mississippi river basins. The changes in flow and demand have a combined effect on the Missouri Summer regulated flow. The supply deficit tends to be driven by the change in flow over the region. Spatial analysis demonstrates the relationship between the supply deficit and the change in demand over urban areas not along a main river or with limited storage, and over areas upstream of groundwater dependent fields with therefore overestimated demand.

Environmental Molecular Sciences Laboratory↗

Simplified model evaluation of cooling rates for glass-containing lunar compositions

The simplified model of glass formation and the development of partial crystallinity in cooled bodies has been applied to lunar compositions 10060, 15028, 15086, 15101, 15286, 15301, 15498, 15499, 60255, 65016, 77017, Apollo 15 green glass and LUNA 24 highland basalt. The critical cooling rates for glass formation predicted by the simplified model are found to be in good agreement (to within an order of magnitude) with those predicted by the exact treatment of crystallization statistics. These predicted critical cooling rates are in even better agreement (a factor of 2) with measured values of the rates required to form glasses of the materials.

Uhlmann, D. R.↗

Assessment of the ParFlow–CLM CONUS 1.0 integrated hydrologic model: evaluation of hyper-resolution water balance components across the contiguous United States

Abstract. Recent advancements in computational efficiency and Earth system modeling have awarded hydrologists with increasingly high-resolution models of terrestrial hydrology, which are paramount to understanding and predicting complex fluxes of moisture and energy. Continental-scale hydrologic simulations are, in particular, of interest to the hydrologic community for numerous societal, scientific, and operational benefits. The coupled hydrology–land surface model ParFlow–CLM configured over the continental United States (PFCONUS) has been employed in previous literature to study scale-dependent connections between water table depth, topography, recharge, and evapotranspiration, as well as to explore impacts of anthropogenic aquifer depletion to the water and energy balance. These studies have allowed for an unprecedented process-based understanding of the continental water cycle at high resolution. Here, we provide the most comprehensive evaluation of PFCONUS version 1.0 (PFCONUSv1) performance to date by comparing numerous modeled water balance components with thousands of in situ observations and several remote sensing products and using a range of statistical performance metrics for evaluation. PFCONUSv1 comparisons with these datasets are a promising indicator of model fidelity and ability to reproduce the continental-scale water balance at high resolution. Areas for improvement are identified, such as a positive streamflow bias at gauges in the eastern Great Plains, a shallow water table bias over many areas of the model domain, and low bias in seasonal total water storage amplitude, especially for the Ohio, Missouri, and Arkansas River basins. We discuss several potential sources for model bias and suggest that minimizing error in topographic processing and meteorological forcing would considerably improve model performance. Results here provide a benchmark and guidance for further PFCONUS model development, and they highlight the importance of concurrently evaluating all hydrologic components and fluxes to provide a multivariate, holistic validation of the complete modeled water balance.

O'Neill, Mary M. F. (ORCID:0000000221455165)↗

A simple physiologically based pharmacokinetic model evaluating the effect of anti-nicotine antibodies on nicotine disposition in the brains of rats and humans

Physiologically based pharmacokinetic (PBPK) modeling was applied to investigate the effects of anti-nicotine antibodies on nicotine disposition in the brains of rats and humans. Successful construction of both rat and human models was achieved by fitting model outputs to published nicotine concentration time course data in the blood and in the brain. Key parameters presumed to have the most effect on the ability of these antibodies to prevent nicotine from entering the brain were selected for investigation using the human model. These parameters, which included antibody affinity for nicotine, antibody cross-reactivity with cotinine, and antibody concentration, were broken down into different, clinically-derived in silico treatment levels and fed into the human PBPK model. Model predictions suggested that all three parameters, in addition to smoking status, have a sizable impact on anti-nicotine antibodies' ability to prevent nicotine from entering the brain and that the antibodies elicited by current human vaccines do not have sufficient binding characteristics to reduce brain nicotine concentrations. If the antibody binding characteristics achieved in animal studies can similarly be achieved in human studies, however, nicotine vaccine efficacy in terms of brain nicotine concentration reduction is predicted to meet threshold values for alleviating nicotine dependence. - Highlights: • Modelling of nicotine disposition in the presence of anti-nicotine antibodies • Key vaccine efficacy factors are evaluated in silico in rats and in humans. • Model predicts insufficient antibody binding in past human nicotine vaccines. • Improving immunogenicity and antibody specificity may lead to vaccine success.

54 ENVIRONMENTAL SCIENCES↗