Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,405 records · Page 78

Evaluating the SWAT Model for Hydrological Modeling in the Xixian Watershed and A Comparison with the XAJ Model

Already declining water availability in Huaihe River, the 6th largest river in China, is further stressed by climate change and intense human activities. There is a pressing need for a watershed model to better understand the interaction between land use activities and hydrologic processes and to support sustainable water use planning. In this study, we evaluated the performance of SWAT for hydrologic modeling in the Xixian River Basin, located at the headwaters of the Huaihe River, and compared its performance with the Xinanjiang (XAJ) model that has been widely used in China

54 ENVIRONMENTAL SCIENCES↗

A coupled hydrologic-agroeconomic modeling framework to evaluate adaptive irrigation strategies under groundwater withdrawal restrictions

Growing groundwater scarcity requires integrated tools to capture interactions among hydrology, agricultural production, markets, and land use. This study presents an iterative modeling framework that couples hydrologic, crop-yield, and economic models to capture two-way feedback among water availability, agricultural production, and market responses under groundwater constraints. The primary goal of this paper is to describe the methodological development of the coupled framework and demonstrate the significance of iterative model interaction. Applied to the western United States, we evaluated adaptive responses to restricting groundwater use beyond recharge levels, represented through changes in irrigation management and expansion or shrinkage of crop markets through land reallocation. Results demonstrate that the iterative coupling converges to stable equilibrium responses within 10 iterations. At equilibrium, deficit irrigation emerges as the dominant adaptation strategy in California, with irrigation levels stabilizing at approximately 70% of full irrigation demand, while Arizona and New Mexico experience stronger yield sensitivities. Early iterations produce commodity price increases of up to 10% for fruit and vegetable crops; however, these responses moderate as land allocation and production patterns adjust across regions. Deficit irrigation and spatial reallocation of irrigated land partially offset production losses, with variability observed across different states: California maintains yields primarily via deficit irrigation, whereas Arizona and New Mexico will rely mainly on reducing irrigated area to absorb the shock. By capturing feedback between biophysical and economic processes, this approach highlights how irrigation strategies and land-use decisions evolve under water stress and provides a transferable platform for evaluating water management policies.

54 ENVIRONMENTAL SCIENCES↗

A Validated Set of MIDAS V5 Task Network Model Scenarios to Evaluate Nextgen Closely Spaced Parallel Operations Concepts

The Closely Spaced Parallel Operations (CSPO) scenario is a complex, human performance model scenario that tested alternate operator roles and responsibilities to a series of off-nominal operations on approach and landing (see Gore, Hooey, Mahlstedt, Foyle, 2013). The model links together the procedures, equipment, crewstation, and external environment to produce predictions of operator performance in response to Next Generation system designs, like those expected in the National Airspaces NextGen concepts. The task analysis that is contained in the present report comes from the task analysis window in the MIDAS software. These tasks link definitions and states for equipment components, environmental features as well as operational contexts. The current task analysis culminated in 3300 tasks that included over 1000 Subject Matter Expert (SME)-vetted, re-usable procedural sets for three critical phases of flight; the Descent, Approach, and Land procedural sets (see Gore et al., 2011 for a description of the development of the tasks included in the model; Gore, Hooey, Mahlstedt, Foyle, 2013 for a description of the model, and its results; Hooey, Gore, Mahlstedt, Foyle, 2013 for a description of the guidelines that were generated from the models results; Gore, Hooey, Foyle, 2012 for a description of the models implementation and its settings). The rollout, after landing checks, taxi to gate and arrive at gate illustrated in Figure 1 were not used in the approach and divert scenarios exercised. The other networks in Figure 1 set up appropriate context settings for the flight deck.The current report presents the models task decomposition from the tophighest level and decomposes it to finer-grained levels. The first task that is completed by the model is to set all of the initial settings for the scenario runs included in the model (network 75 in Figure 1). This initialization process also resets the CAD graphic files contained with MIDAS, as well as the embedded operator models that comprise MIDAS. Following the initial settings, the model progresses to begin the first tasks required of the two flight deck operators, the Captain (CA) and the First Officer (FO). The task sets will initialize operator specific settings prior to loading all of the alerts, probes, and other events that occur in the scenario. As a note, the CA and FO were terms used in developing this model but the CA can also be thought of as the Pilot Flying (PF), while the FO can be considered the Pilot-Not-Flying (PNF)or Pilot Monitoring (PM). As such, the document refers to the operators as PFCA and PNFFO respectively.

task network model↗

Evaluating and Constraining Models’ Stratocumulus and Cumulus Cloud Feedbacks in the Tropics using Satellite Observations to Reduce Uncertainties in Future Climate Projections

Low-cloud feedbacks are thought to be the greatest source of uncertainty in climate projections on the centennial scale. Past research has mainly focused on the global low-cloud feedback as a whole whereas only a few studies have examined separated contributions from the two main low cloud categories: stratocumulus (Sc) and shallow cumulus (Cu). Here we aim to evaluate and constrain Sc and Cu cloud feedbacks in models using satellite observations to ultimately reduce their contribution to the spread in equilibrium climate sensitivity. We utilize the Cumulus and Stratocumulus CloudSat-CALIPSO Dataset (CASCCAD) recently developed by Cesana et al. (2019), which discriminates Sc from Cu clouds. This algorithm – based on cloud morphological characteristics at the orbital level – outperforms previous cloud-type classifications stemming from passive-sensor measurements. We investigate the observed relationship between the Sc and Cu cloud cover in tropical subsidence oceanic regions and their link to the primary low-cloud controlling factors (sea surface temperature, estimated inversion strength). To gain further insight into the Sc-Cu cloud partitioning, we explore the Sc-Cu relationship during the four meteorological seasons (DJF, MAM, JJA, SON) as well as on monthly timescale. We find a substantial negative correlation between the Sc and Cu cloud cover on annual, seasonal and monthly timescales. We finally evaluate how well climate models of the Coupled Model Intercomparison Project phase 5 (CMIP5) and phase 6 (CMIP6) agree with the observations. These results establish a framework to advance model parameterizations of the underlying physical processes driving the stratocumulus and shallow cumulus clouds (turbulence and convection).

Cloud feedbacks↗

Investigation of Wave Energy Converter Effects on Near-shore Wave Fields: Model Generation Validation and Evaluation - Kaneohe Bay HI.

The numerical model, SWAN (Simulating WAves Nearshore) , was used to simulate wave conditions in Kaneohe Bay, HI in order to determine the effects of wave energy converter ( WEC ) devices on the propagation of waves into shore. A nested SWAN model was validated then used to evaluate a range of initial wave conditions: significant wave heights (H s ) , peak periods (T p ) , and mean wave directions ( MWD) . Differences between wave height s in the presence and absence of WEC device s were assessed at locations in shore of the WEC array. The maximum decrease in wave height due to the WEC s was predicted to be approximately 6% at 5 m and 10 m water depths. Th is occurred for model initiation parameters of H s = 3 m (for 5 m water depth) or 4 m (10 m water depth) , T p = 10 s, and MWD = 330deg . Subsequently, bottom orbital velocities were found to decrease by about 6%.

Roberts, Jesse D.↗

A Benchmarking Framework for Evaluating Large Language Model Capabilities in Nuclear Reactor Safety Applications

Large language models (LLMs) are increasingly capable of answering technical questions, synthesizing domain knowledge, and supporting engineering workflows. For nuclear science and engineering, these capabilities require careful, domain-specific evaluation before they can be credibly incorporated into safety-related activities, regulatory review, or technical decision support. This paper presents preliminary results from benchmarking framework for evaluating LLM capabilities in nuclear contexts. The framework is organized into three evaluation categories: nuclear fundamentals, general dual-use knowledge, and plant specific knowledge. These categories are intended to distinguish general nuclear engineering competence from broader technical reasoning and more context-dependent nuclear knowledge. Initial evaluations focus on nuclear fundamentals using questions representative of the knowledge expected of a nuclear professional engineer. Results indicate that contemporary frontier models perform at a high level and substantially exceed the performance of older model generations, with some models approaching saturation of the current benchmark. These findings suggest both the rapid improvement of LLM capabilities in specialized technical domains and the need for more discriminating evaluation methods. The paper presents the benchmark structure, preliminary model-comparison results, and ongoing work. This work supports development of verifiable, responsible, and safety-conscious methods for assessing AI systems in nuclear engineering applications.

21 - SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLAN↗

Evaluation of Mathematical Turbulence Models

Simplified models for internal flow described, and their predictions compared with experimental results. Report presents account of various models used in computation of turbulent flows. Applications of these models to internal flows evaluated by analysis of predictions of various turbulence models in some important flow configurations.

Nallasamy, M.↗

Performance evaluation of CMIP6 models on the Arctic-Siberian Plain teleconnection affecting the East Asian heat waves

The frequency and intensity of summer heat waves in East Asia have increased sharply in recent decades, significantly impacting public health and the economy. The Arctic-Siberian Plain (ASP) teleconnection pattern has been identified as a key driver, with ASP warming amplifying atmospheric circulation patterns conducive to extreme temperatures. This study evaluates the ability of Coupled Model Inter-comparison Project phase 6 models to simulate the ASP pattern across interannual variability (IAV) and intra-seasonal variability (ISV) timescales using the Common Basis Function method. The multi-model mean shows statistically significant pattern correlations with ERA5 reanalysis, with correlation coefficients of 0.90 and 0.99 for IAV and ISV, respectively. While the ASP pattern is generally well captured, models exhibit substantial inter-model diversity in the intensity and position of anticyclonic anomalies over the ASP and East Asia. Models with ASP pattern variability similar to reanalysis better reproduce extreme East Asian temperatures, whereas those over- or underestimating ASP variability exhibit lower skill. These performance differences are related to differences in simulating key variables associated with the development of the ASP pattern. Our findings highlight the role of the ASP pattern in modulating extreme heat events, as models with improved ASP simulations align more closely with observed temperature extremes. Refining ASP representations in models could enhance seasonal heat wave predictions, improving climate adaptation strategies.

Arctic-Siberian Plain (ASP)↗

Evaluation of historical CMIP6 model simulations of extreme precipitation over contiguous US regions

Simulated historical precipitation is evaluated for Coupled Model Intercomparison Project Phase 6 (CMIP6) models using precipitation indices defined by the Expert Team on Climate Change Detection and Indices. The model indices are evaluated against corresponding indices from the CPC unified gauge-based analyses of precipitation over seven geographical regions across the contiguous US (CONUS). The regions assessed match those in recent US National Climate Assessment Reports. To estimate observational uncertainty, precipitation indices for three other observational datasets (HadEx2, Livneh and PRISM) are evaluated against the CPC analyses. Both the moderate and extreme mean precipitation intensities are overestimated over the western CONUS and underestimated in the areas of the Central Great Plains (CGP) in most CMIP6 models tested. Most CMIP6 models overestimate the mean and variability of wet spell durations and underestimate the mean and variability of dry spell durations across the CONUS. Biases in interannual variability of most of the indices have similar patterns to those in corresponding mean biases. The median and interquartile model spreads in CMIP6 model biases are clearly smaller than those in CMIP5 model biases for wet spell durations. Multimodel medians of CMIP6 (CMIP6-MMM) and CMIP5 (CMIP5-MMM) have similar biases in climatology and variability but biases tend to be smaller in CMIP6-MMM. Depending on the index, extreme precipitation is slightly better in parts of the eastern half of the CONUS in CMIP6-MMM, otherwise, the biases in climatology and variability are similar to CMIP5-MMM. CMIP6-MMM performs better than individual models and even observational datasets in some cases. Differences between observational datasets for most indices are comparable to the CMIP6 interquartile model spread. The better-performing observational and model datasets are different in different parts of the CONUS.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Evaluation of the Global Aerosol Microphysical ModelE2-TOMAS Model Against Satellite and Ground-Based Observations

The TwO-Moment Aerosol Sectional (TOMAS) microphysics model has been integrated into the state-of-the- art general circulation model, GISS ModelE2. This paper provides a detailed description of the ModelE2-TOMAS model and evaluates the model against various observations including aerosol precursor gas concentrations, aerosol mass and number concentrations, and aerosol optical depths. Additionally, global budgets in ModelE2-TOMAS are compared with those of other global aerosol models, and the ModelE2-TOMAS model is compared to the default aerosol model in ModelE2, which is a one-moment aerosol (OMA) model (i.e. no aerosol microphysics). Overall, the ModelE2- TOMAS predictions are within the range of other global aerosol model predictions, and the model has a reasonable agreement (mostly within a factor of 2) with observations of sulfur species and other aerosol components as well as aerosol optical depth. However, ModelE2-TOMAS (as well as ModelE2-OMA) cannot capture the observed vertical distribution of sulfur dioxide over the Pacific Ocean, possibly due to overly strong convective transport and over-predicted precipitation. The ModelE2-TOMAS model simulates observed aerosol number concentrations and cloud condensation nuclei concentrations roughly within a factor of 2. Anthropogenic aerosol burdens in ModelE2-OMA differ from ModelE2-TOMAS by a few percent to a factor of 2 regionally, mainly due to differences in aerosol processes including deposition, cloud processing, and emission parameterizations. We observed larger differences for naturally emitted aerosols such as sea salt and mineral dust, as those emission rates are quite different due to different upper size cutoff assumptions.

Lee, Y. H.↗

Existing and Required Modeling Capabilities for Evaluating ATM Systems and Concepts

ATM systems throughout the world are entering a period of major transition and change. The combination of important technological developments and of the globalization of the air transportation industry has necessitated a reexamination of some of the fundamental premises of existing Air Traffic Management (ATM) concepts. New ATM concepts have to be examined, concepts that may place more emphasis on: strategic traffic management; planning and control; partial decentralization of decision-making; and added reliance on the aircraft to carry out strategic ATM plans, with ground controllers confined primarily to a monitoring and supervisory role. 'Free Flight' is a case in point. In order to study, evaluate and validate such new concepts, the ATM community will have to rely heavily on models and computer-based tools/utilities, covering a wide range of issues and metrics related to safety, capacity and efficiency. The state of the art in such modeling support is adequate in some respects, but clearly deficient in others. It is the objective of this study to assist in: (1) assessing the strengths and weaknesses of existing fast-time models and tools for the study of ATM systems and concepts and (2) identifying and prioritizing the requirements for the development of additional modeling capabilities in the near future. A three-stage process has been followed to this purpose: 1. Through the analysis of two case studies involving future ATM system scenarios, as well as through expert assessment, modeling capabilities and supporting tools needed for testing and validating future ATM systems and concepts were identified and described. 2. Existing fast-time ATM models and support tools were reviewed and assessed with regard to the degree to which they offer the capabilities identified under Step 1. 3 . The findings of 1 and 2 were combined to draw conclusions about (1) the best capabilities currently existing, (2) the types of concept testing and validation that can be carried out reliably with such existing capabilities and (3) the currently unavailable modeling capabilities that should receive high priority for near-term research and development. It should be emphasized that the study is concerned only with the class of 'fast time' analytical and simulation models. 'Real time' models, that typically involve humans-in-the-loop, comprise another extensive class which is not addressed in this report. However, the relationship between some of the fast-time models reviewed and a few well-known real-time models is identified in several parts of this report and the potential benefits from the combined use of these two classes of models-a very important subject-are discussed in chapters 4 and 7.

Odoni, Amedeo R.↗

Dynamic Modelling and Performance Evaluation of a Chemisorption Heat Pump for Cold Climate

Space heating is a significant contributor to the energy consumption in residential buildings, especially in cold climates where electric-driven heat pumps lose the advantage of high efficiency and conventional gas-combustion furnaces have limited efficiency. Chemical adsorption (chemisorption) heat pumps (CSHP) have the potential for producing heating with high energy efficiency under cold operating conditions. A CSHP operates intermittently between desorption and adsorption modes, and their performance is affected by system design and transient system control. This study developed a dynamic model and used it to evaluate the daily system performance under different designs and controls, particularly the buffer tank size and the controlled minimum heating power. The simulation results revealed the effects of these parameters on the system performance and helped identify the optimal design and control for the highest coefficient of performance and heating capacity.

Yang, Zhiyao↗

Chemistry-Climate Interactions in the GISS GCM: Tropospheric Chemistry Model Description and Evaluation - Part 1

A tropospheric chemistry module has been developed for use within the Goddard Institute for Space Studies (GISS) general circulation model (GCM) to study interactions between chemistry and climate change. The model uses a simplified chemistry scheme based on CO-NOx-CH4 chemistry, and also includes a parameterization for emissions of isoprene, the most important non-methane hydrocarbon. The model reproduces present day annual cycles and mean distributions of key trace gases fairly well, based on extensive comparisons with available observations. Examining the simulated change between present day and pre-industrial conditions, we find that the model has a similar response to that seen in other simulations. It shows a 45% increase in the global tropospheric ozone burden, within the 25% - 57% range seen in other studies. Annual average zonal mean ozone increases by more than 125% at Northern Hemisphere middle latitudes near the surface. Comparison of model runs that allow the calculated ozone to interact with the GCM's radiation and meteorology with those that do not shows only minor differences for ozone. The common usage of ozone fields that are not calculated interactively seems to be adequate to simulate both the present day and the pre-industrial ozone distributions. However, use of coupled chemistry does alter the change in tropospheric oxidation capacity, enlarging the overall decrease in OH concentrations from the pre-industrial to the present by about 10% (-5.3% global annual average in uncoupled mode, -5.9% in coupled mode). This indicates that there may be systematic biases in the simulation of the pre-industrial to present day decrease in the oxidation capacity of the troposphere (though a 10% difference is well within the total uncertainty). Global annual average radiative forcing from pre-industrial to present day ozone change is 0.32 W/sq m. The forcing seems to be increased by about 10% when the chemistry is coupled to the GCM. Forcing values greater than 0.8 W/sq m are seen over large areas of the United States, Southern Europe, North Africa, the Middle East, Central Asia, and the Arctic. Radiative forcing is greater than 1.5 W/sq m over parts of these areas during Northern summer Though there are local differences, the radiative forcing is overall in good agreement with the results of other modeling studies in both its magnitude and spatial distribution, demonstrating that the simplified chemistry is adequate for climate studies.

Shindell, Drew T.↗

Evaluation of Mesoscale Model Phenomenological Verification Techniques

Forecasters at the Spaceflight Meteorology Group, 45th Weather Squadron, and National Weather Service in Melbourne, FL use mesoscale numerical weather prediction model output in creating their operational forecasts. These models aid in forecasting weather phenomena that could compromise the safety of launch, landing, and daily ground operations and must produce reasonable weather forecasts in order for their output to be useful in operations. Considering the importance of model forecasts to operations, their accuracy in forecasting critical weather phenomena must be verified to determine their usefulness. The currently-used traditional verification techniques involve an objective point-by-point comparison of model output and observations valid at the same time and location. The resulting statistics can unfairly penalize high-resolution models that make realistic forecasts of a certain phenomena, but are offset from the observations in small time and/or space increments. Manual subjective verification can provide a more valid representation of model performance, but is time-consuming and prone to personal biases. An objective technique that verifies specific meteorological phenomena, much in the way a human would in a subjective evaluation, would likely produce a more realistic assessment of model performance. Such techniques are being developed in the research community. The Applied Meteorology Unit (AMU) was tasked to conduct a literature search to identify phenomenological verification techniques being developed, determine if any are ready to use operationally, and outline the steps needed to implement any operationally-ready techniques into the Advanced Weather Information Processing System (AWIPS). The AMU conducted a search of all literature on the topic of phenomenological-based mesoscale model verification techniques and found 10 different techniques in various stages of development. Six of the techniques were developed to verify precipitation forecasts, one to verify sea breeze forecasts, and three were capable of verifying several phenomena. The AMU also determined the feasibility of transitioning each technique into operations and rated the operational capability of each technique on a subjective 1-10 scale: (1) 1 indicates that the technique is only in the initial stages of development, (2) 2-5 indicates that the technique is still undergoing modifications and is not ready for operations, (3) 6-8 indicates a higher probability of integrating the technique into AWIPS with code modifications, and (4) 9-10 indicates that the technique was created for AWIPS and is ready for implementation. Eight of the techniques were assigned a rating of 5 or below. The other two received ratings of 6 and 7, and none of the techniques a rating of 9-10. At the current time, there are no phenomenological model verification techniques ready for operational use. However, several of the techniques described in this report may become viable techniques in the future and should be monitored for updates in the literature. The desire to use a phenomenological verification technique is widespread in the modeling community, and it is likely that other techniques besides those described herein are being developed, but the work has not yet been published. Therefore, the AMIU recommends that the literature continue to be monitored for updates to the techniques described in this report and for new techniques being developed whose results have not yet been published. 111

Lambert, Winifred↗

Simulating boundary layer transition with low-Reynolds-number k-epsilon turbulence models. I - An evaluation of prediction characteristics. II - An approach to improving the predictions

The capability of two k-epsilon low-Reynolds number (LRN) turbulence models, those of Jones and Launder (1972) and Lam and Bremhorst (1981), to predict transition in external boundary-layer flows subject to free-stream turbulence is analyzed. Both models correctly predict the basic qualitative aspects of boundary-layer transition with free stream turbulence, but for calculations started at low values of certain defined Reynolds numbers, the transition is generally predicted at unrealistically early locations. Also, the methods predict transition lengths significantly shorter than those found experimentally. An approach to overcoming these deficiencies without abandoning the basic LRN k-epsilon framework is developed. This approach limits the production term in the turbulent kinetic energy equation and is based on a simple stability criterion. It is correlated to the free-stream turbulence value. The modification is shown to improve the qualitative and quantitative characteristics of the transition predictions.

Schmidt, R. C.↗

Wind Tunnel Evaluation of a Model Helicopter Main-Rotor Blade With Slotted Airfoils at the Tip

Data for rotors using unconventional airfoils are of interest to permit an evaluation of this technology's capability to meet the U.S. Army's need for increased helicopter mission effectiveness and improved safety and survivability. Thus, an experimental investigation was conducted in the Langley Transonic Dynamics Tunnel (TDT) to evaluate the effect of using slotted airfoils in the rotor blade tip region (85 to 100 percent radius) on rotor aerodynamic performance and loads. Four rotor configurations were tested in forward flight at advance ratios from 0.15 to 0.45 and in hover in-ground effect. The hover tip Mach number was 0.627, which is representative of a design point of 4000-ft geometric altitude and a temperature of 95 F. The baseline rotor configuration had a conventional single-element airfoil in the tip region. A second rotor configuration had a forward-slotted airfoil with a -6 deg slat, a third configuration had a forward-slotted airfoil with a -10 slat, and a fourth configuration had an aft-slotted airfoil with a 3 deg flap (trailing edge down). The results of this investigation indicate that the -6 deg slat configuration offers some performance and loads benefits over the other three configurations.

Noonan, Kevin W.↗