Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “performance model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
NASA NTRS2020

A Machine Learning-Based Cloud Detection and Thermodynamic Phase Classification Algorithm using Passive Spectral Observations

We trained two Random Forest (RF) machine-learning models for cloud mask and cloud thermodynamic phase detection using spectral observations from VIIRS on Suomi NPP (SNPP). Observations from CALIOP were carefully selected to provide reference labels. The two RF models were trained for all-day and daytime-only conditions using a 4-year collocated VIIRS/CALIOP dataset from 2013 to 2016. Due to the orbit difference, the collocated CALIOP and SNPP VIIRS training samples cover a broad viewing zenith angle range, which is a great benefit to overall model performance. The all-day model uses 3 VIIRS infrared (IR) bands (8.6,11, and 12 μm) and the daytime model uses 5 Near-IR (NIR) and Shortwave-IR (SWIR) bands (0.86, 1.24, 1.38, 1.64 and 2.25 μm) together with the 3 IR bands to detect clear, liquid water, and ice cloud pixels. Up to 7 surface types, namely, ocean/water, forest, cropland, grassland, snow/ice, barren/desert, and shrubland, were considered separately to enhance performance for both models. Detection of cloudy pixels and thermodynamic phase with the two RF models were compared against collocated CALIOP products from 2017. It is shown that, with a conservative screening process that excludes the most challenging cloudy pixels for passive remote sensing, the two RF models have high accuracy rates in comparison with the CALIOP reference for both cloud detection and thermodynamic phase. Other existing SNPP VIIRS and Aqua MODIS cloud mask and phase products are also evaluated, with results showing that the two RF models and the MODIS MYD06 optical property phase product are the top 3 algorithms with respect to lidar observations during the daytime. During the nighttime, the RF all-day model works best for both cloud detection and phase, in particular for pixels over snow/ice surfaces. The present RF models can be extended to other similar passive instruments if training samples can be collected from CALIOP or other lidars. However, the quality of reference labels and potential sampling issues that may impact model performance would need further attention.

cloud detection↗
DOE OSTI2024

Accurate and Data‐Efficient Micro X‐ray Diffraction Phase Identification Using Multitask Learning: Application to Hydrothermal Fluids

Traditional analysis of highly distorted micro X‐ray diffraction (μ‐XRD) patterns from hydrothermal fluid environments is a time‐consuming process, often requiring substantial data preprocessing and labeled experimental data. Herein, the potential of deep learning with a multitask learning (MTL) architecture to overcome these limitations is demonstrated. MTL models are trained to identify phase information in μ‐XRD patterns, minimizing the need for labeled experimental data and masking preprocessing steps. Notably, MTL models show superior accuracy compared to binary classification convolutional neural networks. Additionally, introducing a tailored cross‐entropy loss function improves MTL model performance. Most significantly, MTL models tuned to analyze raw and unmasked XRD patterns achieve close performance to models analyzing preprocessed data, with minimal accuracy differences. This work indicates that advanced deep learning architectures like MTL can automate arduous data handling tasks, streamline the analysis of distorted XRD patterns, and reduce the reliance on labor‐intensive experimental datasets.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2001

Future Modeling Needs in Pulse Detonation Rocket Engine Design

This paper presents a performance model rocket engine design that takes advantage of pulse detonation to generate thrust. The contents include: 1) Introduction to the Pulse Detonation Rocket Engine (PDRE); 2) PDRE modeling issues and options; 3) Discussion of the PDRE Performance Workshop held at Marshall Space Flight Center; and 4) Identify needs involving an open performance model for Pulse Detonation Rocket Engines. This paper is in viewgraph form.

Meade, Brian↗
NASA NTRS1995

A Performance Prediction Model for a Fault-Tolerant Computer During Recovery and Restoration

The modeling and design of a fault-tolerant multiprocessor system is addressed. Of interest is the behavior of the system during recovery and restoration after a fault has occurred. The multiprocessor systems are based on the Algorithm to Architecture Mapping Model (ATAMM) and the fault considered is the death of a processor. The developed model is useful in the determination of performance bounds of the system during recovery and restoration. The performance bounds include time to recover from the fault, time to restore the system, and determination of any permanent delay in the input to output latency after the system has regained steady state. Implementation of an ATAMM based computer was developed for a four-processor generic VHSIC spaceborne computer (GVSC) as the target system. A simulation of the GVSC was also written on the code used in the ATAMM Multicomputer Operating System (AMOS). The simulation is used to verify the new model for tracking the propagation of the delay through the system and predicting the behavior of the transient state of recovery and restoration. The model is shown to accurately predict the transient behavior of an ATAMM based multicomputer during recovery and restoration.

Obando, Rodrigo A.↗
NASA NTRS2018

Generation of Mars Helicopter Rotor Model for Comprehensive Analyses

The present research is aimed at providing a performance model for the Mars Helicopter (MH), to understand the complexity of the flow, and identify future regions of improvement. The low density of the Martian atmosphere and the relatively small MH rotor, result in very low chord-based Reynolds number flows. The low density and Reynolds numbers reduce the lifting force and lifting efficiency, respectively. The high drag coefficients in subcritical flow, especially for thicker sections, are attributed to laminar separation from the rear of the airfoil. In the absence of test data, efforts have been made to explore these effects using prior very low Reynolds number research efforts. The rotor chord-based Reynolds number range is observed to be subcritical, which makes boundary layer transition unlikely to occur. The state of the two-dimensional rotor boundary layer in hover is approximated by calculating the instability point, laminar separation point, and the transition location to provide understanding of the flow state in the high Mach-low Reynolds number regime. The results are used to investigate the need for turbulence modeling in Computational Fluid Dynamics (CFD) calculations afterwards. The goal is to generate a performance model for the MH rotor for a free wake analysis, because the computational budget for a complete Navier-Stokes solution for a rotating body-fitted rotor is substantial. In this study, a Reynolds-Averaged Navier-Stokes (RANS) based approach is used to generate the airfoil deck using C81Gen with stitched experimental data for very high angles of attack. A full Grid Resolution Study is performed and over 4,500 cases are completed to create the full airfoil deck. The laminar separation locations are predicted within the accuracy of the approximate method when compared with the CFD calculations. The model is presented through airfoil data tables (c81 files) that are used by comprehensive rotor analysis codes such as CAMRADII, or the mid-fidelity CFD solver RotCFD. Finally, the rotor performance is compared with experimental data from the 25ft Space Simulator at the NASA Jet Propulsion Laboratory (JPL) and shows good correlation for the rotor Figure of Merit over the available thrust range.

Model↗
DOE OSTI2023

Technical Report on Waveform Fit Metrics for Global Models

The new WAVEFORMS Initiative in the Ground-based Nuclear Detonation Detection (GNDD) program includes an increased emphasis on the development of Earth models and methods to predict entire seismic and acoustic waveforms more accurately. In general, this increased emphasis is predicated on the need to better characterize seismic events and provide improved model-based discrimination between event types including earthquakes and explosions. More specifically, while current moment tensor inversion methods tend to work well for larger events (M>~4) using tuned 1-D Earth models, the development of state-of-the-art 3-D models and methods is required for the prediction of shorter period waves over large areas for discrimination of smaller events. There is no standard metric for model-based waveform prediction accuracy used in the waveform modeling/inversion community. However, there are several popular waveform misfit definitions; and minimizing the corresponding objective functions is the goal of waveform inversion. Some example misfit definitions employed for adjoint waveform tomography include measures of simple travel time differences (e.g. Tape et al., 2010), cross-correlation travel time differences (e.g. Luo and Schuster, 1991), multi-taper frequency dependent methods (e.g. Lei et al., 2020), time-frequency phase misfit functions (e.g. Fichtner 2010; Rodgers et al., 2022), normalized cross-correlation methods (e.g. Tao et al., 2018), and others. In some cases, these misfit definitions also involve complicated weighting schemes and summations over multiple frequency bands making it difficult to duplicate the misfit measurement with alternative models and datasets. Although each of the misfit definitions mentioned above are useful for developing waveform models, the actual misfit values are not usually meaningful outside of a given project, model, and/or dataset. Therefore, it is difficult to understand and communicate model performance for predicting waveforms and comparing to other models and/or new model iterations with a different dataset. Therefore, there is a need for a generalized method for evaluating overall model performance that is independent from the specific misfit chosen to develop the waveform models that is also intuitive and meaningful. In this report, we describe a new metric we refer to as ‘Percent of Correlated Signal’. The following sections describe and demonstrate the metric with a case study event and a more rigorous test using a random selection of globally distributed events. While the focus here is on global tomography models, the metric is meant to applicable to regional ‘wiggle-for-wiggle’ waveform models/studies as well.

58 GEOSCIENCES↗
NASA NTRS2011

CEDAR Electrodynamics Thermosphere Ionosphere (ETI) Challenge for Systematic Assessment of Ionosphere/Thermosphere Models: NmF2, hmF2, and Vertical Drift Using Ground-Based Observations

Objective quantification of model performance based on metrics helps us evaluate the current state of space physics modeling capability, address differences among various modeling approaches, and track model improvements over time. The Coupling, Energetics, and Dynamics of Atmospheric Regions (CEDAR) Electrodynamics Thermosphere Ionosphere (ETI) Challenge was initiated in 2009 to assess accuracy of various ionosphere/thermosphere models in reproducing ionosphere and thermosphere parameters. A total of nine events and five physical parameters were selected to compare between model outputs and observations. The nine events included two strong and one moderate geomagnetic storm events from GEM Challenge events and three moderate storms and three quiet periods from the first half of the International Polar Year (IPY) campaign, which lasted for 2 years, from March 2007 to March 2009. The five physical parameters selected were NmF2 and hmF2 from ISRs and LEO satellites such as CHAMP and COSMIC, vertical drifts at Jicamarca, and electron and neutral densities along the track of the CHAMP satellite. For this study, four different metrics and up to 10 models were used. In this paper, we focus on preliminary results of the study using ground-based measurements, which include NmF2 and hmF2 from Incoherent Scatter Radars (ISRs), and vertical drifts at Jicamarca. The results show that the model performance strongly depends on the type of metrics used, and thus no model is ranked top for all used metrics. The analysis further indicates that performance of the model also varies with latitude and geomagnetic activity level.

Shim, J. S.↗
DOE OSTI2020

High-Performance Computational Modeling of Strong Ground Motion for Seismic Response Analysis with Implications for High-Hazard and/or Nuclear Facilities and Critical Infrastructure at the NNSSNLV-015-19, Year 2

Presentation to be given at the FY 2020 SDRD Annual Meeting, to be held Sept. 23-24, on WebEx. There is one video also attached. This presentation is a reduced slide deck of the recently STIPped version, DOE/NV/03624--0875, presented 9-15-20 at the work in progress seminar. High-Performance Computational Modeling of Strong Ground Motion for Seismic Response Analysis with Implications for High-Hazard and/or Nuclear Facilities and Critical Infrastructure at the NNSSNLV-015-19, Year 2

58 GEOSCIENCES↗
NASA NTRS2016

Regime-Based Evaluation of Cloudiness in CMIP5 Models

The concept of Cloud Regimes (CRs) is used to develop a framework for evaluating the cloudiness of 12 fifth Coupled Model Intercomparison Project (CMIP5) models. Reference CRs come from existing global International Satellite Cloud Climatology Project (ISCCP) weather states. The evaluation is made possible by the implementation in several CMIP5 models of the ISCCP simulator generating for each gridcell daily joint histograms of cloud optical thickness and cloud top pressure. Model performance is assessed with several metrics such as CR global cloud fraction (CF), CR relative frequency of occurrence (RFO), their product (long-term average total cloud amount [TCA]), cross-correlations of CR RFO maps, and a metric of resemblance between model and ISCCP CRs. In terms of CR global RFO, arguably the most fundamental metric, the models perform unsatisfactorily overall, except for CRs representing thick storm clouds. Because model CR CF is internally constrained by our method, RFO discrepancies yield also substantial TCA errors. Our findings support previous studies showing that CMIP5 models underestimate cloudiness. The multi-model mean performs well in matching observed RFO maps for many CRs, but is not the best for this or other metrics. When overall performance across all CRs is assessed, some models, despite their shortcomings, apparently outperform Moderate Resolution Imaging Spectroradiometer (MODIS) cloud observations evaluated against ISCCP as if they were another model output. Lastly, cloud simulation performance is contrasted with each model's equilibrium climate sensitivity (ECS) in order to gain insight on whether good cloud simulation pairs with particular values of this parameter.

cloud climatology↗
NASA NTRS1999

Data Reduction Methods Applied to the Fastrac Engine

The Fastrac rocket engine is currently being developed for the X-34 technology demonstrator vehicle. The engine performance model must be calibrated to support accurate performance prediction. Data reduction is the process of estimating hardware characteristics from available test data, and is essential for effective performance model calibration and prediction. A new data reduction procedure was developed, implemented, and tested using data from Fastrac engine tests. The procedure selects hardware and test measurements to use in the reduction process based on examination of the model influence matrix condition number. Predicted hardware characteristics are recovered from the solution of a quadratic programming problem. Computational tests indicate that the new procedure provides a significant improvement in test data reduction capability. Enhancements include improved test data utilization and time history data reduction capability. The new method is generically applicable to other systems.

Santi, L. Michael↗
NASA NTRS2013

Modeling and Evaluating Pilot Performance in NextGen: Review of and Recommendations Regarding Pilot Modeling Efforts, Architectures, and Validation Studies

NextGen operations are associated with a variety of changes to the national airspace system (NAS) including changes to the allocation of roles and responsibilities among operators and automation, the use of new technologies and automation, additional information presented on the flight deck, and the entire concept of operations (ConOps). In the transition to NextGen airspace, aviation and air operations designers need to consider the implications of design or system changes on human performance and the potential for error. To ensure continued safety of the NAS, it will be necessary for researchers to evaluate design concepts and potential NextGen scenarios well before implementation. One approach for such evaluations is through human performance modeling. Human performance models (HPMs) provide effective tools for predicting and evaluating operator performance in systems. HPMs offer significant advantages over empirical, human-in-the-loop testing in that (1) they allow detailed analyses of systems that have not yet been built, (2) they offer great flexibility for extensive data collection, (3) they do not require experimental participants, and thus can offer cost and time savings. HPMs differ in their ability to predict performance and safety with NextGen procedures, equipment and ConOps. Models also vary in terms of how they approach human performance (e.g., some focus on cognitive processing, others focus on discrete tasks performed by a human, while others consider perceptual processes), and in terms of their associated validation efforts. The objectives of this research effort were to support the Federal Aviation Administration (FAA) in identifying HPMs that are appropriate for predicting pilot performance in NextGen operations, to provide guidance on how to evaluate the quality of different models, and to identify gaps in pilot performance modeling research, that could guide future research opportunities. This research effort is intended to help the FAA evaluate pilot modeling efforts and select the appropriate tools for future modeling efforts to predict pilot performance in NextGen operations.

pilot human performance models↗
DOE OSTI2025

Adaptive Client Selection in Federated Learning: A Network Anomaly Detection Use Case

Federated Learning (FL) has become a ubiquitous approach for training machine learning models on decentralized data, addressing the myriad privacy concerns inherent in traditional centralized methods. However, the efficiency of FL depends on effective client selection and robust privacy preservation mechanisms. Inadequate client selection may lead to suboptimal model performance, while insufficient privacy measures risk exposing sensitive data. This paper proposes a client selection framework for FL that integrates differential privacy and fault tolerance. Our adaptive approach dynamically adjusts the number of selected clients based on model performance and system constraints, ensuring privacy through calibrated noise addition. We evaluate our method on a network anomaly detection use case using the UNSW-NB15 and ROAD datasets. Results show up to a 7% increase in accuracy and a 25% reduction in training time compared to FedL2P. Moreover, we highlight the trade-offs between privacy budgets and model performance, with higher privacy budgets reducing noise and improving accuracy. Our fault tolerance mechanism, while causing a slight performance drop, enhances robustness to client failures. Statistical validation using Mann-Whitney U tests confirms the significance of these improvements (p < 0.05).

Marfo, William [University of Texas at El Paso,Dep↗
DOE OSTI2026

An improved dataset for predicting mammal infecting viruses from genetic sequence information

There have been several attempts to develop machine learning (ML) models to identify human infecting viruses from their genomic sequences, with varying degrees of success. Direct comparison between models is problematic, because these models are typically trained and evaluated on different datasets with alternative data splitting schemes, features, and model performance metrics. In this paper we present a standardized dataset of mammal infecting and non-infecting viral pathogens, refined from the previous work of Mollentze et al. to include the latest literature evidence, roughly doubling the number of curated host-virus records available to the community, and new host target labels, primate and mammal. The new host labels were included for several reasons, including previous reports that classification performance is better at broader taxonomic ranks and the idea that there may be more data for primate infection that might serve as a suitable proxy for zoonotic potential and avoidance of false positives for human infection due to absence of evidence. On this dataset, we report the performance of eight machine learning models for predicting mammal-infecting viruses from their genomic sequences. We find that randomly assigning cases in our improved dataset to training/testing sets, when compared to the original assignments into training/testing in Mollentze et al., increases the overall average ROC AUC of prediction of human infection from 0.663 ± 0.070 to 0.784 ± 0.013, consistent with the reduction in phylogenetic distance between train and test sets (relative entropy change from 3.00 to 0.08). The broadest host category of mammal infection can be predicted most reliably at 0.850 ± 0.020. We share our improved dataset and code to enable standardized comparisons of machine learning methods to predict human host infections. Overall, we have presented preliminary evidence that classification of virus host infection is more tractable at higher taxonomic ranks, that unsurprisingly reducing the phylogenetic distance between training and test sets can improve predictive performance, that peptide kmer features appear to be harmful to out of sample model performance, and we are left with the question of whether models for virus host prediction can reasonably be expected to perform well in out of sample scenarios given the likelihood that viruses do not share a common ancestor. Consistent with this concern, when the data is resampled such that there is no overlap between viral families in training and test sets (relative entropy > 24), models perform no better than random chance at prediction of human infection regardless of whether kmers are included (ROC AUC 0.50 ± 0.08) or not (ROC AUC 0.50 ± 0.04).

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Implementing a neural network interatomic model with performance portability for emerging exascale architectures

The two main thrusts of computational science are increasingly accurate predictions and faster calculations; to this end, the zeitgeist in molecular dynamics (MD) simulations is pursuing machine learned and data driven interatomic models, e.g. neural network potentials, and novel hardware architectures, e.g. GPUs. Current implementations of neural network potentials are orders of magnitude slower than traditional interatomic models and while looming exascale computing offers the ability to run large, accurate simulations with these models, achieving portable performance for MD with new and varied exascale hardware requires rethinking traditional algorithms, using novel data structures, and library solutions. We re-implement a neural network interatomic model in CabanaMD, an MD proxy application, built on libraries developed for performance portability. Our implementation shows significantly improved thread scaling in this complex kernel as compared to a current LAMMPS implementation, across both strong and weak scaling. Our single-source solution enables simulations up to 20 million atoms on a single CPU node and 4 million atoms with improved performance on a single GPU. Furthermore, we also explore parallelism and data layout choices (using flexible data structures called AoSoAs) and their effect on performance, seeing up to ~50% and ~5% improvements in performance on a GPU by choosing the right level of parallelism and data layout respectively.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2016

Evaluation of Contamination Inspection and Analysis Methods through Modeling System Performance

Contamination is usually identified as a risk on the risk register for sensitive space systems hardware. Despite detailed, time-consuming, and costly contamination control efforts during assembly, integration, and test of space systems, contaminants are still found during visual inspections of hardware. Improved methods are needed to gather information during systems integration to catch potential contamination issues earlier and manage contamination risks better. This research explores evaluation of contamination inspection and analysis methods to determine optical system sensitivity to minimum detectable molecular contamination levels based on IEST-STD-CC1246E non-volatile residue (NVR) cleanliness levels. Potential future degradation of the system is modeled given chosen modules representative of optical elements in an optical system, minimum detectable molecular contamination levels for a chosen inspection and analysis method, and determining the effect of contamination on the system. By modeling system performance based on when molecular contamination is detected during systems integration and at what cleanliness level, the decision maker can perform trades amongst different inspection and analysis methods and determine if a planned method is adequate to meet system requirements and manage contamination risk.

Seasly, Elaine↗
DOE OSTI2020

Numerical modeling and performance analysis of a membrane-based air dehumidifier using ionic liquid desiccant

Membrane-based Liquid Desiccant Air Dehumidifier (MLDAD) has the potential to overcome the shortcomings of the conventional open-tower liquid desiccant air dehumidifier. The MLDAD uses membranes with high water vapor permeability to separate air stream from the liquid desiccant and thus eliminates carry-over and the resulting corrosion issues. Recently, the ionic liquid desiccant becomes a promising alternative to conventional liquid desiccant. The ionic liquid desiccant has a large potential of dehumidification, which is also non-corrosive to metals and non-crystallizable. A two-dimensional numerical heat and mass transfer model of the MLDAD using a recently identified ionic liquid desiccant, [EMIM]OAc, is presented in this paper. We report this model can simulate the performance of the MLDAD with various designs, including different selections of liquid desiccant and membrane materials, dimensions of the MLDAD, and flow patterns. For porous membranes, this model accounts for several micro-scale mass transfer mechanisms of vapor transportation across the membrane. It can also model the mass transfer performance of dehumidifiers using non-porous membranes based on the experimentally measured membrane permeance. The numerical model was validated against the performance data available from literature and experimental tests. The maximum discrepancy of the latent effectiveness between the measured and the model-predicted results was about 6% when porous membranes were used. A parametric study was conducted with the numerical model. The results indicated that the membrane permeability, air path height, and solution to air mass ratio were the most critical parameters, determining the dehumidification performance of the MLDAD.

42 ENGINEERING↗
DOE OSTI2020

Numerical modeling and performance analysis of a membrane-based air dehumidifier using ionic liquid desiccant

Membrane-based Liquid Desiccant Air Dehumidifier (MLDAD) has the potential to overcome the shortcomings of the conventional open-tower liquid desiccant air dehumidifier. The MLDAD uses membranes with high water vapor permeability to separate air stream from the liquid desiccant and thus eliminates carry-over and the resulting corrosion issues. Recently, the ionic liquid desiccant becomes a promising alternative to conventional liquid desiccant. The ionic liquid desiccant has a large potential of dehumidification, which is also non-corrosive to metals and non-crystallizable. A two-dimensional numerical heat and mass transfer model of the MLDAD using a recently identified ionic liquid desiccant, [EMIM]OAc, is presented here. This model can simulate the performance of the MLDAD with various designs, including different selections of liquid desiccant and membrane materials, dimensions of the MLDAD, and flow patterns. For porous membranes, this model accounts for several micro-scale mass transfer mechanisms of vapor transportation across the membrane. It can also model the mass transfer performance of dehumidifiers using non-porous membranes based on the experimentally measured membrane permeance. The numerical model was validated against the performance data available from literature and experimental tests. The maximum discrepancy of the latent effectiveness between the measured and the model-predicted results was about 6% when porous membranes were used. A parametric study was conducted with the numerical model. Finally, the results indicated that the membrane permeability, air path height, and solution to air mass ratio were the most critical parameters, determining the dehumidification performance of the MLDAD.

42 ENGINEERING↗
DOE OSTI2025

Impacts of benchmarking choices on inferred model skill of the Arctic–Boreal terrestrial carbon cycle

Abstract Land surface models require continuous validation against observations to improve and reduce simulation uncertainty. However, inferred model performance can be heavily influenced by subjective choices made in the selection and application of observational data products. A key area often misrepresented by models is the Arctic–Boreal region, which is a potential tipping point region in Earth’s climate system due to large permafrost carbon stocks that are vulnerable to release with climate warming. We use the International Land Model Benchmarking (ILAMB) framework to evaluate how the model skill of TRENDY-v9 models varies based on the choice of observational-based benchmark and how benchmarks are applied in model evaluation. This analysis uses global datasets integrated into ILAMB and new, regionally-specific observational products from the Arctic–Boreal Vulnerability Experiment. Our results cover the overall time period of 1979–2019 and show that model scores can vary substantially depending on the data product applied, with higher model scores indicating better model performance against observations. The lowest model scores occur when benchmarked against regional, compared to global, datasets. We also evaluate observed and modeled functional relationships between ecosystem respiration and air temperature and between gross primary production and precipitation. Here, we find that the magnitude and shape of the responses are strongly impacted by the choice of observational dataset and the approach used to construct the functional relationship benchmark. These results suggest that model evaluation studies could conclude a false sense of model skill if only using a single benchmark data product or if not applying regional data products when performing a regional model analysis. Collectively, our findings highlight the influence of benchmarking choices on model evaluation and point to the need for benchmarking guidelines when assessing model skill.

Poe, Jeralyn (ORCID:0000000318495278)↗