Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,783 records · Page 99

Comparison of Resource Requirements for a Wind Tunnel Test Designed with Conventional vs. Modern Design of Experiments Methods

The factors that determine data volume requirements in a typical wind tunnel test are identified. It is suggested that productivity in wind tunnel testing can be enhanced by managing the inference error risk associated with evaluating residuals in a response surface modeling experiment. The relationship between minimum data volume requirements and the factors upon which they depend is described and certain simplifications to this relationship are realized when specific model adequacy criteria are adopted. The question of response model residual evaluation is treated and certain practical aspects of response surface modeling are considered, including inference subspace truncation. A wind tunnel test plan developed by using the Modern Design of Experiments illustrates the advantages of an early estimate of data volume requirements. Comparisons are made with a representative One Factor At a Time (OFAT) wind tunnel test matrix developed to evaluate a surface to air missile.

DeLoach, Richard↗

A constitutive model with damage for high temperature superalloys

A unified constitutive model is searched for that is applicable for high temperature superalloys used in modern gas turbines. Two unified inelastic state variable constitutive models were evaluated for use with the damage parameter proposed by Kachanov. The first is a model (Bodner, Partom) in which hardening is modeled through the use of a single state variable that is similar to drag stress. The other (Ramaswamy) employs both a drag stress and back stress. The extension was successful for predicting the tensile, creep, fatigue, torsional and nonproportional response of Rene' 80 at several temperatures. In both formulations, a cumulative damage parameter is introduced to model the changes in material properties due to the formation of microcracks and microvoids that ultimately produce a macroscopic crack. A back stress/drag stress/damage model was evaluated for Rene' 95 at 1200 F and is shown to predict the tensile, creep, and cyclic loading responses reasonably well.

Sherwood, J. A.↗

Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledge

Understanding the interactions and regulatory relationships among biomolecules is essential for deciphering complex biological systems and elucidating the mechanisms behind diverse biological functions. Traditionally, the collection of such molecular interaction data has relied on expert curation, a process that is both time-consuming and labor-intensive. To address these limitations, this study explores the use of large language models (LLMs) to automate the genome-scale extraction of molecular interaction knowledge. Here, we evaluate the performance of various LLMs on key biological tasks, including the identification of protein-protein interactions, detection of genes associated with pathways influenced by low-dose radiation, and inference of gene regulatory relationships. Our findings demonstrate that larger LLMs tend to perform better, particularly in extracting intricate gene and protein interactions. Despite their strengths, these models face challenges in recognizing functionally diverse gene groups and highly correlated regulatory relationships. Through a comprehensive analysis using established molecular interaction and pathway databases, we show that LLMs possess the potential to identify relevant biomolecules and predict their interactions, offering valuable insights and marking a significant step toward AI-driven biological knowledge discovery.

63 RADIATION, THERMAL, AND OTHER ENVIRON. POLLUTAN↗

Evaluation of a Model-Based Groundwater Drought Indicator in the Conterminous U.S.

Monitoring groundwater drought using land surface models is a valuable alternative given the current lack of systematic in situ measurements at continental and global scales and the low resolution of current remote sensing based groundwater data. However, uncertainties inherent to land surface models may impede drought detection, and thus should be assessed using independent data sources. In this study, we evaluated a groundwater drought index (GWI) derived from monthly groundwater storage output from the Catchment Land Surface Model (CLSM) using a GWI similarly derived from in situ groundwater observations. Groundwater observations were obtained from unconfined or semi-confined aquifers in eight regions of the central and northeastern U.S. Regional average GWI derived from CLSM exhibited strong correlation with that from observation wells, with correlation coefficients between 0.43 and 0.92. GWI from both in situ data and CLSM was generally better correlated with the Standard Precipitation Index (SPI) at 12 and 24 month timescales than at shorter timescales, but it varied depending on climate conditions. The correlation between CLSM derived GWI and SPI generally decreases with increasing depth to the water table, which in turn depends on both bedrock depth (a CLSM parameter) and mean annual precipitation. The persistence of CLSM derived GWI is spatially varied and again shows a strong influence of depth to groundwater. CLSM derived GWI generally persists longer than GWI derived from in situ data, due at least in part to the inability of coarse model inputs to capture high frequency meteorological variability at local scales. The study also showed that groundwater can have a significant impact on soil moisture persistence where the water table is shallow. Soil moisture persistence was estimated to be longer in the eastern U.S. than in the west, in contrast to previous findings that were based on models that did not represent groundwater. Assimilation of terrestrial water storage data from the Gravity Recovery and Climate Experiment (GRACE) satellite mission improved the correlation between CLSM based regional average GWI and that based on in situ data in six of the eight regions. Practical issues regarding the application of GRACE assimilated groundwater storage for drought detection are discussed. An important conclusion of this study is that model parameters that control the depth to the water table, including bedrock depth, strongly influence the evolution and persistence of simulated groundwater and require careful configuration for drought monitoring.

hydrology↗

Evaluating the Sensitivity of Agricultural Model Performance to Different Climate Inputs: Supplemental Material

Projections of future food production necessarily rely on models, which must themselves be validated through historical assessments comparing modeled and observed yields. Reliable historical validation requires both accurate agricultural models and accurate climate inputs. Problems with either may compromise the validation exercise. Previous studies have compared the effects of different climate inputs on agricultural projections but either incompletely or without a ground truth of observed yields that would allow distinguishing errors due to climate inputs from those intrinsic to the crop model. This study is a systematic evaluation of the reliability of a widely used crop model for simulating U.S. maize yields when driven by multiple observational data products. The parallelized Decision Support System for Agrotechnology Transfer (pDSSAT) is driven with climate inputs from multiple sources reanalysis, reanalysis that is bias corrected with observed climate, and a control dataset and compared with observed historical yields. The simulations show that model output is more accurate when driven by any observation-based precipitation product than when driven by non-bias-corrected reanalysis. The simulations also suggest, in contrast to previous studies, that biased precipitation distribution is significant for yields only in arid regions. Some issues persist for all choices of climate inputs: crop yields appear to be oversensitive to precipitation fluctuations but under sensitive to floods and heat waves. These results suggest that the most important issue for agricultural projections may be not climate inputs but structural limitations in the crop models themselves.

simulation↗

THERMALLY DRIVEN ATMOSPHERIC ESCAPE

Accurately determining the escape rate from a planet's atmosphere is critical for determining its evolution. A large amount of Cassini data is now available for Titan's upper atmosphere and a wealth of data is expected within the next decade on escape from Pluto, Mars, and extra-solar planets. Escape can be driven by upward thermal conduction of energy deposited well below the exobase, as well as by nonthermal processes produced by energy deposited in the exobase region. Recent applications of a model for escape driven by upward thermal conduction, called the slow hydrodynamic escape model, have resulted in surprisingly large loss rates for the atmosphere of Titan, Saturn's largest moon. Based on a molecular kinetic simulation of the exobase region, these rates appear to be orders of magnitude too large. Therefore, the slow hydrodynamic model is evaluated here. It is shown that such a model cannot give a reliable description of the atmospheric temperature profile unless it is coupled to a molecular kinetic description of the exobase region. Therefore, the present escape rates for Titan and Pluto must be re-evaluated using the atmospheric model described here.

79 ASTRONOMY AND ASTROPHYSICS↗

A Perspective on Computational Human Performance Models as Design Tools

The design of interactive systems, including levels of automation, displays, and controls, is usually based on design guidelines and iterative empirical prototyping. A complementary approach is to use computational human performance models to evaluate designs. An integrated strategy of model-based and empirical test and evaluation activities is particularly attractive as a methodology for verification and validation of human-rated systems for commercial space. This talk will review several computational human performance modeling approaches and their applicability to design of display and control requirements.

Jones, Patricia M.↗

Benchmarking carbon fluxes of the ISIMIP2a biome models

The purpose of this study is to evaluate the eight ISIMIP2a biome models against independent estimates of long-term net carbon fluxes (i.e. Net Biome Productivity, NBP) over terrestrial ecosystems for the recent four decades (1971–2010). Here, we evaluate modeled global NBP against 1) the updated global residual land sink (RLS) plus land use emissions (E LUC ) from the Global Carbon Project (GCP), presented as R + L in this study by Le Quéré et al (2015), and 2) the land CO 2 fluxes from two atmospheric inversion systems: Jena CarboScope s81_v3.8 and CAMS v15r2, referred to as F Jena and F CAMS respectively. The model ensemble-mean NBP (that includes seven models with land-use change) is higher than but within the uncertainty of R + L, while the simulated positive NBP trend over the last 30 yr is lower than that from R + L and from the two inversion systems. ISIMIP2a biome models well capture the interannual variation of global net terrestrial ecosystem carbon fluxes. Tropical NBP represents 31 ± 17% of global total NBP during the past decades, and the year-to-year variation of tropical NBP contributes most of the interannual variation of global NBP. According to the models, increasing Net Primary Productivity (NPP) was the main cause for the generally increasing NBP. Significant global NBP anomalies from the long-term mean between the two phases of El Niño Southern Oscillation (ENSO) events are simulated by all models (p < 0.05), which is consistent with the R + L estimate (p = 0.06), also mainly attributed to NPP anomalies, rather than to changes in heterotrophic respiration (Rh). The global NPP and NBP anomalies during ENSO events are dominated by their anomalies in tropical regions impacted by tropical climate variability. Multiple regressions between R + L, F Jena and F CAMS interannual variations and tropical climate variations reveal a significant negative response of global net terrestrial ecosystem carbon fluxes to tropical mean annual temperature variation, and a non-significant response to tropical annual precipitation variation. According to the models, tropical precipitation is a more important driver, suggesting that some models do not capture the roles of precipitation and temperature changes adequately.

54 ENVIRONMENTAL SCIENCES↗

Experimental evaluation criteria for constitutive models of time dependent cyclic plasticity

Notched members were tested at temperatures far above those recorded till now. Simulation of the notch root stress response was accomplished to establish notch stress-strain behavior. Cyclic stress-strain profiles across the net-section were recorded and on-line direct notch strain control was accomplished. Data are compared to three analysis techniques with good results. The objective of the study is to generate experimental data that can be used to evaluate the accuracy of constitutive models of time dependent cyclic plasticity.

Martin, J. F.↗

On testing models for the pressure-strain correlation of turbulence using direct simulations

Direct simulations of homogeneous turbulence have, in recent years, come into widespread use for the evaluation of models for the pressure-strain correlation of turbulence. While work in this area has been beneficial, the increasingly common practice of testing the slow and rapid parts of these models separately in uniformly strained turbulent flows is shown in this paper to be unsound. For such flows, the decomposition of models for the total pressure-strain correlation into slow and rapid parts is ambiguous. Consequently, when tested in this manner, misleading conclusions can be drawn about the performance of pressure-strain models. This point is amplified by illustrative calculations of homogeneous shear flow where other pitfalls in the evaluation of models are also uncovered. More meaningful measures for testing the performance of pressure-strain models in uniformly strained turbulent flows are proposed and the implications for turbulence modeling are discussed.

Speziale, Charles G.↗

CAM-chem: description and evaluation of interactive atmospheric chemistry in the Community Earth System Model

We discuss and evaluate the representation of atmospheric chemistry in the global Community Atmosphere Model (CAM) version 4, the atmospheric component of the Community Earth System Model (CESM). We present a variety of configurations for the representation of tropospheric and stratospheric chemistry, wet removal, and online and offline meteorology. Results from simulations illustrating these configurations are compared with surface, aircraft and satellite observations. Major biases include a negative bias in the high-latitude CO distribution, a positive bias in upper-tropospheric/lower-stratospheric ozone, and a positive bias in summertime surface ozone (over the United States and Europe). The tropospheric net chemical ozone production varies significantly between configurations, partly related to variations in stratosphere-troposphere exchange. Aerosol optical depth tends to be underestimated over most regions, while comparison with aerosol surface measurements over the United States indicate reasonable results for sulfate, especially in the online simulation. Other aerosol species exhibit significant biases. Overall, the model-data comparison indicates that the offline simulation driven by GEOS5 meteorological analyses provides the best simulation, possibly due in part to the increased vertical resolution (52 levels instead of 26 for online dynamics). The CAM-chem code as described in this paper, along with all the necessary datasets needed to perform the simulations described here, are available for download at www.cesm.ucar.edu.

54 ENVIRONMENTAL SCIENCES↗

Thermally anisotropic building envelope for thermal management: finite element model calibration using field evaluation data

The thermally anisotropic building envelope (TABE) is an active building envelope that redistributes thermal loads in response to weather conditions and building energy demand. Conductive layers throughout the TABE distribute low-grade heat among hydronic loops, altering heat flow direction and intensity. Finite element models of TABE roof and wall panels were developed and calibrated using field evaluation data. The calibration results showed that heat flux differences between the experimental data and finite element models averaged –0.42% and 3.57%, with a maximum mean square error of 1.78 and 3.96 for roof and wall panels, respectively. A reduction in heat flux from the environment to the building living space over the entire testing period (weeks in July/August) was found to be 85% for roof panels and 335% (load reversed) for wall panels. Finally, these results indicate TABE can effectively harness low-grade thermal energy sources to achieve high energy efficiency and promote demand-side management.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Estimation of Interfacial Tension between Organic Liquid Mixtures and Water

Knowledge of IFT values for chemical mixtures helps guide the design and analysis of various processes, including NAPL remediation with surfactants or alcohol flushing, enhanced oil recovery, and chemical separation technologies, yet available literature values are sparse. A comprehensive comparison of thermodynamic and empirical models for estimating interfacial tension (IFT) of organic chemical mixtures with water is conducted, mainly focusing on chlorinated organic compounds for 14 ternary, three quaternary, and one quinary systems. Emphasis is placed on novel results for systems with three and four organic chemical compounds, and for systems with composite organic compounds like lard oil and mineral oil. Seven models are evaluated: the ideal and nonideal monolayer models (MLID and MLNID), the ideal and nonideal mutual solubility models (MSID and MSNID), an empirical model for ternary systems (EM), a linear mixing model based on mole fractions (LMMM), and a newly developed linear mixing model based on volume fractions of organic mixtures (LMMV) for higher order systems. The two ideal models (MLID and MSID) fit ternary systems of chlorinated organic compounds without surface active compounds relatively well. However, both ideal models did not perform well for the mixtures containing a surface active compound. However, for these systems, both the MLNID and MSNID models matched the IFT data well. It is shown that the MLNID model with a surface coverage value (0.00341 mmol/m2) obtained in this study can practically be used for chlorinated organic compounds. The LMMM results in poorer estimates of the IFT as the difference in IFT values of individual organic compounds in a mixture increases. The EM, with two fitting parameters, provided accurate results for all 14 ternary systems including composite organic compounds. The new LMMV method for quaternary and higher component systems was successfully tested. This study shows that the LMMV may be able to be used for higher component systems and it can be easily incorporated into compositional multiphase flow models using only parameters from ternary systems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Evaluating the Impact of Model Resolutions and Cumulus Parameterization on Precipitation in NU-WRF: A Case Study in the Central Great Plains

Regional climate models are expected to exhibit improved skill at finer spatial resolutions due to improved representation of land surface heterogeneity. However, at spatial scales between 1 to 10 km (grey scales), these improvements are often illusive due to the competing benefits from spatial resolution and cumulus parameterization. This study provides insights into the impact of model resolution and cumulus parameterization on precipitation prediction in the Central Great Plains by using an object-based evaluation method. Our results show limited improvement solely from finer resolution but larger improvement without using the cumulus scheme at a 4-kmresolution. Compared to traditional evaluation methods, the object-based analysis shows that without the cumulus scheme the spatial properties of precipitation are better represented. In contrast, all model configurations show a dry bias in precipitation days and a tendency to produce widespread precipitation but with fewer hours with precipitationwhich indicates other shortcomings in the model.

Yuqi Zhang↗

A comparison between backscattering models for rough surfaces

The ease of applicability of three scattering models is examined. This is accomplished by considering the time taken to numerically evaluate these models and by comparing their predictions as a function of surface roughness, frequency, incident angle and polarization with the moment-method solution in two dimensions. The complexity of the analytic models in three dimensions and their analytic reduction to high- and low-frequency regions are also compared. The selected models are an integral equation model (IEM), a full wave model (FWM), and the phase perturbation model (PPM). It is noted that in three dimensions, the full-wave model requires an evaluation of a tenfold integral and the phase perturbation model requires a fourfold and a twofold integral, whereas the integral equation model is an algebraic equation in like polarization under single scattering conditions. It is found that both the IEM and PPM agree with the moment-method solution from low to high frequencies numerically.

Chen, K. S.↗

Informing Robust Functional Relationship Benchmarks: An Evaluation of the Temperature Sensitivity of Ecosystem Respiration Across the Arctic-Boreal Region

During land model development, simulated carbon dynamics are often benchmarked against observational data sets to evaluate model performance. Functional relationship benchmarks are the relationship between a driving variable (e.g., temperature) and a response variable (e.g., ecosystem respiration) and are a promising tool for assessing model performance by evaluating modeled sensitivities to changing environmental conditions. However, observed functional relationships can be influenced by choices made during data collection and throughout the benchmarking process, impacting the inferred skill of land models. To avoid misrepresenting a model's true performance, it is necessary to systematically evaluate best practices when constructing functional relationship benchmarks. We developed a set of guidelines for constructing functional relationship benchmarks, considering the choice of data set, number of daily observations, temporal extent, and temporal resolution across Alaska and Canada over a 20-year period from 2001 to 2020. The temperature sensitivity of ecosystem respiration from observations, evaluated through an apparent Q 10 , is highly variable both spatially and as a result of the data processing approach applied in the benchmark formation. When benchmarking 13 models from the Warming Permafrost Model Intercomparison Project (WrPMIP), the range in inferred model skill is substantially impacted by the choices applied in constructing functional relationship benchmarks. The inferred performance of a given model is most sensitive to the number of daily observations and temporal extent, followed by choice of benchmark data set and temporal averaging. Results from this analysis can guide the development of consistent and robust functional relationships for future model evaluation studies.

Poe, Jeralyn [Northern Arizona University, Flagsta↗