Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21

Investigating Downscaling Methods and Evaluating Climate Models for Use in Estimating Regional Water Resources in Mountainous Regions under Changing Climatic Conditions

The purpose of this three-year study is to develop and evaluate techniques to estimate the range of potential hydrological impacts of climate change in mountainous areas. Three main objectives are set out in the proposal. (1) To develop and evaluate transfer functions to link tropospheric circulation to regional snowfall. (2) To evaluate a suite of General Circulation Models (GCMs) for use in estimating synoptic scale circulation and the resultant regional snowfall. And (3) to estimate the range of potential hydrological impacts of changing climate in the two case study areas: the Upper Colorado River basin, and the Catskill Mountains of southeastern New York State. Both regions provide water to large populations.

Frei, Allan↗

Albedo and Diffuse POA Measurements to Evaluate Transposition Model Uncertainty

Albedo and diffuse plane of array (DPOA) measurements are used in addition to standard global horizontal irradiance (GHI), direct normal irradiance (DNI), diffuse horizontal irradiance (DNI), and plane of array irradiance (POA) measurements to determine the impact of albedo on transposition model performance. Albedo measurements averaged 0.214. Daily albedo values ranged from 0.148 to 0.236 and were found to be correlated to daily relative humidity. DPOA measurements were compared to calculated DPOA values (from POA and DNI), and helped identify a suspected deviation from due south in the azimuth of the POA measurement. Since the measured albedo average was close to the typical fixed $^{albedo = 0.2}$ assumption, little difference was seen between using measured and fixed albedo (~0.15% differences in mean bias difference (MBD) and root mean squared difference (RMSD)). However, evaluation of transposition models at other fixed albedos showed an albedo change of 0.1 is found to increase total modeled insolation by approximately 1%. Thus, for locations with different ground surfaces (e.g., persistent snow cover of black surfaces), the impact of using measured albedo instead of the fixed $^{albedo = 0.2}$ assumption may be greater. Measurement deviations resulted in up to 2% changes in MBD and RIVISD when switching between interrelated measurements (e.g., GHI and DHI as inputs to transposition models versus DNI and DHI as inputs). Variation among transposition models was also up to 2% MBD and RMSD. Thus, for this data set, measurement deviation and transposition model selection are found to have more impact than using measured albedo instead fixed albedo.

42 ENGINEERING↗

Experimental design for three interrelated marine ice sheet and ocean model intercomparison projects: MISMIP v. 3 (MISMIP +), ISOMIP v. 2 (ISOMIP +) and MISOMIP v. 1 (MISOMIP1)

Coupled ice sheet-ocean models capable of simulating moving grounding lines are just becoming available. Such models have a broad range of potential applications in studying the dynamics of marine ice sheets and tidewater glaciers, from process studies to future projections of ice mass loss and sea level rise. The Marine Ice Sheet-Ocean Model Intercomparison Project (MISOMIP) is a community effort aimed at designing and coordinating a series of model intercomparison projects (MIPs) for model evaluation in idealized setups, model verification based on observations, and future projections for key regions of the West Antarctic Ice Sheet (WAIS). Here we describe computational experiments constituting three interrelated MIPs for marine ice sheet models and regional ocean circulation models incorporating ice shelf cavities. These consist of ice sheet experiments under the Marine Ice Sheet MIP third phase (MISMIP+), ocean experiments under the Ice Shelf-Ocean MIP second phase (ISOMIP+) and coupled ice sheet-ocean experiments under the MISOMIP first phase (MISOMIP1). All three MIPs use a shared domain with idealized bedrock topography and forcing, allowing the coupled simulations (MISOMIP1) to be compared directly to the individual component simulations (MISMIP+ and ISOMIP+). The experiments, which have qualitative similarities to Pine Island Glacier Ice Shelf and the adjacent region of the Amundsen Sea, are designed to explore the effects of changes in ocean conditions, specifically the temperature at depth, on basal melting and ice dynamics. In future work, differences between model results will form the basis for the evaluation of the participating models.

58 GEOSCIENCES↗

Experimental Studies and Energy Modeling for Evaluating Performance of Various Deep Wall Retrofits

The Pacific Northwest National Laboratory, Oak Ridge National Laboratory, and the University of Minnesota conducted a three-year, multipart study on residential retrofit wall assemblies. The project, which was funded by the U.S. Department of Energy’s Building Technology Office, aimed to compare a range of residential wall retrofit systems that prioritized affordability, durability, and energy savings potential. The research team identified, constructed, tested, simulated, and analyzed the feasibility and economics of 16 wall retrofit assemblies (14 test configurations and two baseline configurations) that can be applied to the exterior side of existing walls (either with or without the existing cladding). The 16 wall assemblies were installed in an in-situ laboratory environment, to evaluate the ease of construction and study the thermal and hygrothermal performance of the walls. This paper presents the methodology used to evaluate the thermal performance of the walls and discusses the energy modeling results of the study. The results from the experiments were used to calibrate a THERM model of each wall assembly, which was then applied to a whole building using the EnergyPlus 8.6 simulation engine. A residential prototype building was used to extrapolate whole-building energy savings in each U.S. climate zone. To capture the conditions of the largest number of homes in the United States, the most frequent building characteristics (e.g., attic insulation level, window specifications, foundation insulation, etc.) were extracted from ResStock data and applied to the prototype model. Results from the energy modeling showed that the climate zones with the highest potential for retrofit savings are those which are heating-dominated (i.e., Cold and Very Cold climate designations). In these climate zones, heating and cooling energy savings due to the wall retrofits alone ranged from 21.5% to 38.2%.

Nagda, Harshil↗

Energy Model to Evaluate Thermal Energy Storage Integrated with Air Source Heat Pumps: Preprint

Full decarbonization in buildings requires the replacement of combustion appliances with electric ones, and air source heat pumps (ASHP) are a candidate alternative. However, technical limitations, such as the efficiency decrease when operating in cold weather, limit their adoption in the global heating market. Among several options to improve ASHP efficiency operating in colder climates, thermal energy storage (TES) has been considered, as it may provide heating when it is cold and shift ASHP operation to times when the weather is warmer. It may also take advantage of time of use electricity rates and support defrosting when necessary. The evaluation of ASHP-TES systems, however, is still limited because traditional metrics do not capture their full economic and environmental benefits. In this work, a python framework is presented to model the ASHP with and without the presence of TES. Metrics are proposed to analyze the system performance in terms of costs, equivalent CO2 emission, and efficiency metrics to evaluate and compare alternative systems. Model validation against experimental data obtained for a commercial heat pump is provided, as well as an application example using Denver, Colorado, to highlight the model capabilities.

cold climate↗

Evaluation of Model Summertime Boundary Layer Cloud Development over Complex Terrain in New York State

Weather forecasting over complex terrain with diverse land cover is challenging. Utilizing the high-resolution observations from New York State Mesonet (NYSM), we are able to evaluate the surface processes of the Weather Research Forecast (WRF) Model in a detailed, scale-dependent manner. Here, in the study, possible impacts of land–atmosphere interaction on surface meteorology and boundary layer cloud development are investigated with different model resolutions, land surface models (LSMs), and planetary boundary layer (PBL) physical parameterizations. The High-Resolution Rapid Refresh, version 3 (HRRR), forecasting model is used as a reference for the sensitivity evaluation. Results show that over complex terrain, the high-resolution simulations (1 km × 60 vertical levels) generally perform better compared to low-resolution (3 km × 50 levels) in both surface meteorology and cloud fields. LSMs play a more important role in surface meteorology compared to PBL schemes. The NoahMP land surface model exhibits daytime warmer and drier biases compared to the Rapid Update Cycle (RUC) due to better prediction of the Bowen ratio in RUC. The PBL schemes would affect the convective strength in the boundary layer. The Shin–Hong (SH) scale-aware scheme tends to produce the strongest convective strength in the PBL, while the ACM2 PBL scheme rarely resolved convection even at 1-km resolution. By considering the radiation effect of subgrid-scale (SGS) clouds, the Mellor–Yamada–Nakanishi–Niino eddy diffusivity mass flux (MYNN-EDMF) predicted the highest cloud coverage and lowest surface solar radiation bias. The configuration of SGS clouds in MYNN-EDMF would not only significantly reduce shortwave radiation bias, but also affect the convection behaviors through land surface–cloud–radiation interaction.

54 ENVIRONMENTAL SCIENCES↗

Evaluation of the CEAS trend and monthly weather data models for soybean yields in Iowa, Illinois, and Indiana

The CEAS models evaluated use historic trend and meteorological and agroclimatic variables to forecast soybean yields in Iowa, Illinois, and Indiana. Indicators of yield reliability and current measures of modeled yield reliability were obtained from bootstrap tests on the end of season models. Indicators of yield reliability show that the state models are consistently better than the crop reporting district (CRD) models. One CRD model is especially poor. At the state level, the bias of each model is less than one half quintal/hectare. The standard deviation is between one and two quintals/hectare. The models are adequate in terms of coverage and are to a certain extent consistent with scientific knowledge. Timely yield estimates can be made during the growing season using truncated models. The models are easy to understand and use and are not costly to operate. Other than the specification of values used to determine evapotranspiration, the models are objective. Because the method of variable selection used in the model development is adequately documented, no evaluation can be made of the objectivity and cost of redevelopment of the model.

French, V.↗

An evaluation of several compressible turbulent boundary-layer models Effects of pressure gradient and Reynolds number

Computations, employing several turbulence models, are compared with a series of attached supersonic turbulent boundary-layer experiments over an extensive range of Reynolds numbers (11.7 x 10 to the 6th to 314 x 10 to the 6th). These experiments included measurements of surface pressure and skin friction for adverse pressure gradients ranging in magnitude from those of previous investigations to an order of magnitude greater. The turbulence models evaluated include algebraic and two-equation eddy-viscosity models and two full Reynolds stress models. In general all the models tested performed well independently of the magnitude of the pressure gradient or Reynolds number and could predict the measured skin friction for most cases with sufficient accuracy for engineering purposes.

Horstman, C. C.↗

Dilated causal convolutional neural networks for forecasting zone airflow to estimate short-term energy consumption

Here this paper investigates the use of dilated causal convolutional neural networks for fine- grained temporal forecasting of building zone states. Specifically, we build and evaluate models using a small set of exogenous features (e.g., external temperature) to autoregressively predict zone airflow setpoints every minute for a 24-hour prediction window. We carefully explore the trade-off between generality and specificity in these models, training and evaluating them based on zone, zone type, month, season, and combinations thereof. When evaluated for a commercial office building in Eastern Washington with 16 zones served by variable air volume air handling units, we find that the highest performance comes from a zone-specific, season-agnostic approach; with it, we obtain an R 2 of 0.704 (averaged over zones) and an average normalized root mean square error (nRMSE) of 0.111. In contrast, the most general model (trained across all zones and seasons) yields an R 2 of only 0.416 and a nRMSE of 0.168, while a baseline zone-specific reduced order model obtains 0.443 R 2 and 0.159 nRMSE. We also report on factors affecting airflow forecasting performance, on the ability of models trained on a specific zone to generalize to other zones, and on the capability of those models trained on a specific month to generalize to other months.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Evaluation of a Multi-Decadal Simulation of Stratospheric Ozone by Comparison with Total Ozone Mapping Spectrometer (TOMS) Observations

One key application of atmospheric chemistry and transport models is prediction of the response of ozone and other constituents to various natural and anthropogenic perturbations. These include changes in composition, such as the previous rise and recent decline in emission of man-made chlorofluorcarbons, changes in aerosol loading due to volcanic eruption, and changes in solar forcing. Comparisons of hindcast model results for the past few decades with observations are a key element of model evaluation and provide a sense of the reliability of model predictions. The 25 year data set from Total Ozone Mapping Spectrometers is a cornerstone of such model evaluation. Here we report evaluation of three-dimensional multi-decadal simulation of stratospheric composition. Meteorological fields for this off-line calculation are taken from a 50 year simulation of a general circulation model. Model fields are compared with observations from TOMS and also with observations from the Stratospheric Aerosol and Gas Experiment (SAGE), Microwave Limb Sounder (MLS), Cryogenic Limb Array Etalon Spectrometer (CLAES), and the Halogen Occultation Experiment (HALOE). This overall evaluation will emphasize the spatial, seasonal, and interannual variability of the simulation compared with observed atmospheric variability.

Douglass, Anne R.↗

Thermal sensing of cryogenic wind tunnel model surfaces Evaluation of silicon diodes

Different sensors and installation techniques for surface temperature measurement of cryogenic wind tunnel models were investigated. Silicon diodes were selected for further consideration because of their good inherent accuracy. Their average absolute temperature deviation in comparison tests with standard platinum resistance thermometers was found to be 0.2 K in the range from 125 to 273 K. Subsurface temperature measurement was selected as the installation technique in order to minimize aerodynamic interference. Temperature distortion caused by an embedded silicon diode was studied numerically.

Daryabeigi, K.↗

Thermal sensing of cryogenic wind tunnel model surfaces - Evaluation of silicon diodes

Different sensors and installation techniques for surface temperature measurement of cryogenic wind tunnel models were investigated. Silicon diodes were selected for further consideration because of their good inherent accuracy. Their average absolute temperature deviation in comparison tests with standard platinum resistance thermometers was found to be 0.2 K in the range from 125 to 273 K. Subsurface temperature measurement was selected as the installation technique in order to minimize aerodynamic interference. Temperature distortion caused by an embedded silicon diode was studied numerically.

Daryabeigi, Kamran↗

Using Remotely Sensed Data and Watershed and Hydrodynamic Models to Evaluate the Effects of Land Cover Land Use Change on Aquatic Ecosystems in Mobile Bay, AL

Alabama coastal systems have been subjected to increasing pressure from a variety of activities including urban and rural development, shoreline modifications, industrial activities, and dredging of shipping and navigation channels. The impacts on coastal ecosystems are often observed through the use of indicator species. One such indicator species for aquatic ecosystem health is submerged aquatic vegetation (SAV). Watershed and hydrodynamic modeling has been performed to evaluate the impact of land cover land use (LCLU) change in the two counties surrounding Mobile Bay (Mobile and Baldwin) on SAV stressors and controlling factors (temperature, salinity, and sediment) in the Mobile Bay estuary. Watershed modeling using the Loading Simulation Package in C++ (LSPC) was performed for all watersheds contiguous to Mobile Bay for LCLU scenarios in 1948, 1992, 2001, and 2030. Remotely sensed Landsat-derived National Land Cover Data (NLCD) were used in the 1992 and 2001 simulations after having been reclassified to a common classification scheme. The Prescott Spatial Growth Model was used to project the 2030 LCLU scenario based on current trends. The LSPC model simulations provided output on changes in flow, temperature, and sediment for 22 discharge points into the estuary. These results were inputted in the Environmental Fluid Dynamics Computer Code (EFDC) hydrodynamic model to generate data on changes in temperature, salinity, and sediment on a grid throughout Mobile Bay and adjacent estuaries. The changes in the aquatic ecosystem were used to perform an ecological analysis to evaluate the impact on SAV habitat suitability. This is the key product benefiting the Mobile Bay coastal environmental managers that integrates the influences of temperature, salinity, and sediment due to LCLU driven flow changes with the restoration potential of SAVs. Data products and results are being integrated into NOAA s EcoWatch and Gulf of Mexico Data Atlas online systems for dissemination to coastal resource managers and stakeholders. Objective 1: Develop and utilize Land Use scenarios for Mobile and Baldwin Counties, AL as input to models to predict the affects on water properties (temperature,salinity,)for Mobile Bay through 2030. Objective 2: Evaluate the impact of land use change on seagrasses and SAV in Mobile Bay. Hypothesis: Urbanization will significantly increase surface flows and impact salinity and temperature variables that effect seagrasses and SAVs.

Al-Hamdan, Mohammad↗

Documentation of a Linked-Population Viability Model to Evaluate Recovery Options for Snake River Fall Chinook Salmon

We developed an age- and life-history based population viability analysis model for linked populations of fall Chinook salmon (LPVA). LPVA was designed to quantify the costs and benefits of re-establishing access to an upstream spawning area using managed reconnection options such as hatchery stocking and translocation. The model incorporates stochasticity in the age at which salmon return to spawn, sex ratio, juvenile life history, and survival rates. Response variables include the proportion of replicate populations that persisted for 80 y (roughly 20 generations), the linked population’s growth rate, λ, time to extinction and summaries of population sizes including average female spawner abundance, final female spawner abundance, and total female abundance. This report attempts to provide a complete model description, including simulation of management options and options for simulating forward with a joint distribution of parameters based on their likelihoods with respect to historical data. Intermediate results illustrating the Bayesian approach to identifying poorly-known parameters are reported here. LPVA simulates reconnection of linked populations by representing management processes such as trapping of adults migrating upstream at two dams and juveniles migrating downstream, and hatchery operations for one or two hatcheries. The purpose of the model is to support efforts to recover the threatened Snake River fall Chinook salmon ESU by evaluating population viability using existing spawning areas with and without a hatchery and to assess the potential costs and benefits of programs to re-establish an upstream spawning area.

Jager, Henriette (Yetta) I.↗

Assessing Mountains as Natural Reservoirs With a Multimetric Framework

Anthropogenic climate change will continue to diminish the unique role that mountains perform as natural reservoirs and alter long‐held assumptions of water management. Climate models are important tools to help constrain uncertainty and understand processes that shape this decline. To ensure that climate model estimates provide stakeholder relevant information, the formulation of multimetric model evaluation frameworks informed by stakeholder interactions are critical. In this study, we present one such multimetric framework to evaluate snowpack data sets in the California Sierra Nevada: the snow water equivalent (SWE) triangle. SWE triangle metrics help to describe snowpack characteristics associated with total water volume buildup, peak water availability, and the rate of water release. This approach highlights compensating errors that would not be reflected in conventional large‐scale spatiotemporal analysis. To test our multimetric evaluation framework, we evaluate several publicly available snow products including the Sierra Nevada Snow Reanalysis, Livneh (L15), and the North American Land Data Assimilation System version 2 data sets. We then evaluate regional climate model skill within the North American Coordinated Regional Climate Downscaling Experiment. All data sets analyzed show variation across the various SWE triangle metrics, even within observationally constrained snow products. This spread was especially shown in spring season melt rates. Melt rate biases were prevalent throughout most regional climate model simulations, regardless of snow accumulation dynamics, and will need to be addressed to improve their utility for water stakeholders.

54 ENVIRONMENTAL SCIENCES↗

Multilevel and multifidelity uncertainty quantification for cardiovascular hemodynamics

Standard approaches for uncertainty quantification in cardiovascular modeling pose challenges due to the large number of uncertain inputs and the significant computational cost of realistic three-dimensional simulations. Here, we propose an efficient uncertainty quantification framework utilizing a multilevel multifidelity Monte Carlo (MLMF) estimator to improve the accuracy of hemodynamic quantities of interest while maintaining reasonable computational cost. This is achieved by leveraging three cardiovascular model fidelities, each with varying spatial resolution to rigorously quantify the variability in hemodynamic outputs. We employ two low-fidelity models (zero- and one-dimensional) to construct several different estimators. Our goal is to investigate and compare the efficiency of estimators built from combinations of these two low-fidelity model alternatives and our high-fidelity three-dimensional models. We demonstrate this framework on healthy and diseased models of aortic and coronary anatomy, including uncertainties in material property and boundary condition parameters. Our goal is to demonstrate that for this application it is possible to accelerate the convergence of the estimators by utilizing a MLMF paradigm. Therefore, we compare our approach to single fidelity Monte Carlo estimators and to a multilevel Monte Carlo approach based only on three-dimensional simulations, but leveraging multiple spatial resolutions. We demonstrate significant, on the order of 10 to 100 times, reduction in total computational cost with the MLMF estimators. We also examine the differing properties of the MLMF estimators in healthy versus diseased models, as well as global versus local quantities of interest. As expected, global quantities such as outlet pressure and flow show larger reductions than local quantities, such as those relating to wall shear stress, as the latter rely more heavily on the highest fidelity model evaluations. Similarly, healthy models show larger reductions than diseased models. In all cases, our workflow coupling Dakota’s MLMF estimators with the SimVascular cardiovascular modeling framework makes uncertainty quantification feasible for constrained computational budgets.

59 BASIC BIOLOGICAL SCIENCES↗

Toward equitable environmental exposure modeling through convergence of data, open, and citizen sciences: an example of air pollution exposure modeling amidst increasing wildfire smoke

Exposure modeling is critical in environmental epidemiology and human health but may face challenges (e.g., skewed data, unequal error, context-insensitive validation, and computational demands). Modeling decisions reflect the intended use of the models and the values that modelers prioritize. We aimed to provide a conceptual framework and machine learning (ML) modeling protocols that address these issues. With 500m-gridded hourly PM 2.5 and O 3 levels in Illinois before, during, and after the 2023 Canadian wildfire season as a motivating example, we conducted modeling experiments to evaluate modeling methods, guided by three domains we propose based on theories of science: 1) Data Diversity, leveraging open and citizen science data to enhance inclusivity, parsimony, and representativeness; 2) Equitable Accuracy, ensuring fairly distributed uncertainties across subpopulations; and 3) Sustainable Modeling, balancing accuracy with reducing computational demands to promote accessibility for under-resourced researchers. Here, we found that ML with publicly available data can achieve high accuracy. Depending on methods, performance may vary substantially, even with identical input data. Large but skewed data may reduce performance. Misuse of cross-validation protocols can underestimate prediction error; although we observed R 2 s of ∼98 %, the modeled estimates varied significantly, indicating the need for careful model validation. By using new modeling protocols including representativeness-considered training and validation data and a new loss function, we achieved high agreement between estimates and ground-based measurements (e.g., R 2 = ∼90 % for PM 2.5 ; ∼80 % for O 3 ), equally distributed errors across sociodemographic strata and urban–rural divides, and reduction in computation time—from several weeks or months to a few days.

Exposure assessment↗

A Hindcast Experiment Using the GCAM 3.0 Agriculture and Land-Use Module

We report results of a “hindcast” experiment focusing on the agricultural and land-use component of the Global Change Assessment Model (GCAM). We initialize GCAM to reproduce observed agriculture and land use in 1990 and forecast agriculture and land use patterns on 1-year time steps to 2010. We report overall model performance for 9 crops in 14 regions. We report areas where the hindcast is in relatively good agreement with observations and areas where the correspondence is poorer. We find that when given observed crop yields as input data, producers in GCAM implicitly have perfect foresight for yields leading to over compensation for year-to-year yield variation. We explore a simple model in which planting decisions are based on expectations but production depends on actual yields and find that this addresses the implicit perfect foresight problem. Second, while existing policies are implicitly calibrated into IAMs, changes in those policies over the period of analysis can have a dramatic effect on the fidelity model output. Third, we demonstrate that IAMs can employ techniques similar to those used by the climate modeling community to evaluate model skill. We find that hindcasting has the potential to yield substantial benefits to the IAM community.

hindcast, gc, agriculture and landuse modeling, va↗