Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41

Proposed Critical Thinking Metrics for Power Grid Trainees

The TRS Intro to Critical Thinking for Operational Decision-Making course for power grid operators seeks to help trainees improve their use of critical thinking skills in operation environments. To determine the effectiveness of this training, we have identified 3 categories of evaluation measures: critical thinking metrics, situation awareness (SA) evaluations, and behavioral observations. Critical thinking metrics seek to directly measure trainee’s ability to use critical thinking skills through their verbalized responses to different questions or scenarios. SA will be evaluated during various points throughout simulated exercises completed during the training. SA evaluations aim to determine the level of a trainee’s ability to perceive, comprehend and anticipate events in the operational environment. This construct will be assessed through verbalized responses to questions informed by the Situation Awareness Global Assessment Technique (SAGAT). Behavioral observations capture how trainees use the system, including which simulator displays or windows they use or actions they perform and in what order. These observations help us understand how the critical thinking training may inform operator action. We hypothesize that while these measures are correlated with each other, they provide different perspectives on each trainee’s ability to gain SA and use critical thinking skills in operation environments. By comparing these measures in a pre-assessment and a post-assessment, we can determine how much improvement a trainee gains through the training. These measures may also be used during the training as appropriate. This document specifies these 3 categories of measures, which components of these measures currently seem to be the best candidates to use in the training course, and which parts of the training course is related to each component.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Robust Explanations using Diverse Adversarially Trained Ensembles, Multi-Modal Contrastive Learning, and Attribution-based Confidence Metrics

The primary objective of this project is to strengthen the trustworthiness of AI systems by designing algorithms that make their internal decision-making processes more understandable to human users. This involves creating clear, interpretable explanations for AI decisions and developing metrics to assess these explanations' validity and reliability. Significant progress has been achieved through (i) developing symbolic explanations, (ii) generating meaningful interpretive insights, (iii) establishing accuracy and confidence metrics, and (iv) devising methods to evaluate the knowledge boundaries of AI models. To date, the research findings have been shared in peer-reviewed publications, with accompanying scientific and technical information (STI) detailed below.

97 MATHEMATICS AND COMPUTING↗

Effect of realistic routing on the social burden metric

The distance people travel to reach critical services is a key input to the Social Burden metric used by Sandia’s Resilient Node Cluster Analysis Tool (ReNCAT) in the optimization’s objective function. By default, ReNCAT utilizes Euclidian distances between population blocks and critical facilities when calculating Social Burden. However, these straight-line distances do not reflect how most residents or goods would travel throughout the area. As distance is a vital input to the burden calculation, a more realistic distance calculation will yield more realistic burden values. This work uses real road networks and calculates the shortest distance path between population centers and critical facilities using a standard graph theory approach. These realistic route distances are then used to compute Social Burden for four areas of study. It was found that distances using real road routes are generally, but not always, longer than the Euclidean distance. The increased length increases the final Social Burden metric, however, the overall burden percent change ranged between 17% and 52%, which means the impact of realistic routes relies heavily upon the area’s road topology. It was found that rural locations within an area may have larger burden increases than urban areas as more dense road networks allow routes to more closely follow a straight-line path. Additionally, using the most straight forward routing algorithms requires high computational effort for areas with large road networks. While it is believed this process can be made more performant, that task is beyond this scope of work.

99 GENERAL AND MISCELLANEOUS↗

Thermal Tolerance Metrics for Freshwater Fish, CONUS, Version 1

This dataset is a compilation of 13 thermal response metrics for 834 freshwater fish species across the conterminous United States (CONUS). The data were extracted from six published sources, many of which are compilations of data from other sources. The data were harmonized for comparison, and additional variables were added to summarize the metrics. The dataset is presented as a spreadsheet containing 17 sheets. The first sheet (datasets) describes the data sources. Other sheets describe the source and compilation variables in detail.

13 HYDRO ENERGY↗

Dataset for: Electric Vulnerability Index: Targeted Energy Storage Implementation Metric

Uninterrupted access to electricity is critical to the safety and security of American households. More frequent and extreme emergency events increase outages across the country, disproportionately impacting vulnerable communities that experience the most frequent and longest outages, are most sensitive to the loss of electric power, and have the least capacity to adapt to these conditions. This study devises a metric, the Electric Vulnerability Index (EVI), and validates this metric against the 2021 Winter Storm Uri in Texas. Though not ubiquitous, similar trends were observed between adjacent areas with higher EVI and those with higher outage rates from this storm. EVI is offered as a viable approach to quantify a population’s vulnerability to electric outages and maps that index across the continental United States to aid policymakers, advocates, and energy system stakeholders in the targeted deployment of resilience solutions, such as energy storage, to communities most in need. This dataset includes the geopackage file containing all relevant attributes used to generate the maps used in the accompanying paper.

Kerby, Jessica [Pacific Northwest National Laborat↗

Simulative Prediction of Solar Illuminance and Application of the Du-Sharples Model in Estimating Adapted Daylighting Metrics for an Urban Environment

The practice of daylighting in indoor spaces can significantly reduce electricity consumption and carbon emissions, improve human productivity, and enhance mood and cognitive perception. This work discussed the recent developments in daylighting science and practice, computed the periodic variations in average diurnal daylight levels for each month, quantified in terms of global horizontal illuminance and diffuse horizontal illuminance, for Kolkata, India, a city with tropical wet and dry climate, with two empirical luminous efficacy models of estimating solar illuminance, and assessed daylighting metrics with the Du-Sharples model. A program was formulated that could compute and generate daylight data with monthly-hourly solar irradiation data and the Du-Sharples model was utilized to predict dirt-corrected daylighting metrics for three glazing transmittance values and five elemental carbon deposition levels on glazing material. The highest monthly average global horizontal illuminance is recorded in April (64.05 klx for Littlefair model and 66.82 klx for Muneer-Kinghorn model) and the highest monthly average diffuse horizontal illuminance is recorded in July (33.23 klx for Littlefair model and 30.63 klx for Muneer-Kinghorn model). Further, the computed yearly average global and diffuse horizontal illuminance levels agree well with a previous study that applied the Perez model. Yearly average horizontal work surface illuminance level remained >1.5 klx for window-towall area ratio >30 %. The approach adopted in this work and the temporal variation charts of computed exterior daylight level data may assist building service engineers, architects, and indoor lighting practitioners in making informed policy decisions at different stages of building planning.

Engineering↗

Entropy of the Quantum–Classical Interface: A Potential Metric for Security

Hybrid quantum–classical systems are emerging as key platforms in quantum computing, sensing, and communication technologies, but the quantum–classical interface (QCI)—the boundary enabling these systems—introduces unique and largely unexplored security vulnerabilities. This position paper proposes using entropy-based metrics to monitor and enhance security, specifically at the QCI. We present a theoretical security outline that leverages well-established information-theoretic entropy measures, such as Shannon entropy, von Neumann entropy, and quantum relative entropy, to detect anomalous behaviors and potential breaches at the QCI. By linking entropy fluctuations to scenarios of practical relevance—including quantum key distribution, quantum sensing, and hybrid control systems—we promote the potential value and applicability of entropy-based security monitoring. While explicitly acknowledging practical limitations and theoretical assumptions, we argue that entropy-based metrics provide a complementary approach to existing security methods, inviting further empirical studies and theoretical refinements that can strengthen future quantum technologies.

97 MATHEMATICS AND COMPUTING↗

Properties of F Stars with Stable Radial Velocity Timeseries: A Useful Metric for Selecting Low-jitter F Stars

In a companion paper, we have conducted an in-depth analysis of radial velocity jitter of over 600 stars, examining the astrophysical origins including stellar granulation, oscillation, and magnetic activity. In this paper, we highlight a subsample of those stars, specifically the main sequence and “retired” F stars—which we refer to as “MSRF” stars—that show low levels of RV jitter (<10 m s{sup −1}). We describe the observational signatures of these stars that allow them to be identified in radial velocity planet programs, for instance, those performing follow-up of transiting planets discovered by TESS. We introduce a “jitter metric” that combines the two competing effects of RV jitter with age: activity and convection. Using thresholds in the jitter metric, we can select both “complete” and “pure” samples of low jitter F stars. We also provide recipes for identifying these stars using only Gaia colors and magnitudes. Finally, we describe a region in the Gaia color–magnitude diagram where low jitter F stars are most highly concentrated. By fitting a ninth-order polynomial to the Gaia main sequence, we use the height above the main sequence as a proxy for evolution, allowing for a crude selection of low jitter MSRF stars when activity measurements are otherwise unavailable.

79 ASTRONOMY AND ASTROPHYSICS↗

The Cloud Feedback Model Intercomparison Project (CFMIP) Diagnostic Codes Catalogue – metrics, diagnostics and methodologies to evaluate, understand and improve the representation of clouds and cloud feedbacks in climate models

The CFMIP Diagnostic Codes Catalogue assembles cloud metrics, diagnostics and methodologies, together with programs to diagnose them from general circulation model (GCM) outputs written by various members of the CFMIP community. This aims to facilitate use of the diagnostics by the wider community studying climate and climate change. Here, this paper describes the diagnostics and metrics which are currently in the catalogue, together with examples of their application to model evaluation studies and a summary of some of the insights these diagnostics have provided into the main shortcomings in current GCMs. Analysis of outputs from CFMIP and CMIP6 experiments will also be facilitated by the sharing of diagnostic codes via this catalogue. Any code which implements diagnostics relevant to analysing clouds – including cloud–circulation interactions and the contribution of clouds to estimates of climate sensitivity in models – and which is documented in peer-reviewed studies, can be included in the catalogue. We very much welcome additional contributions to further support community analysis of CMIP6 outputs.

58 GEOSCIENCES↗

Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3

Systematic, routine, and comprehensive evaluation of Earth system models (ESMs) facilitates benchmarking improvement across model generations and identifying the strengths and weaknesses of different model configurations. By gauging the consistency between models and observations, this endeavor is becoming increasingly necessary to objectively synthesize the thousands of simulations contributed to the Coupled Model Intercomparison Project (CMIP) to date. The Program for Climate Model Diagnosis and Intercomparison (PCMDI) Metrics Package (PMP) is an open-source Python software package that provides quick-look objective comparisons of ESMs with one another and with observations. The comparisons include metrics of large- to global-scale climatologies, tropical inter-annual and intra-seasonal variability modes such as the El Niño–Southern Oscillation (ENSO) and Madden–Julian Oscillation (MJO), extratropical modes of variability, regional monsoons, cloud radiative feedbacks, and high-frequency characteristics of simulated precipitation, including its extremes. The PMP comparison results are produced using all model simulations contributed to CMIP6 and earlier CMIP phases. An important objective of the PMP is to document the performance of ESMs participating in the recent phases of CMIP, together with providing version-controlled information for all datasets, software packages, and analysis codes being used in the evaluation process. Among other purposes, this also enables modeling groups to assess performance changes during the ESM development cycle in the context of the error distribution of the multi-model ensemble. Quantitative model evaluation provided by the PMP can assist modelers in their development priorities. In this paper, we provide an overview of the PMP, including its latest capabilities, and discuss its future direction.

54 ENVIRONMENTAL SCIENCES↗

New framework for benchmarking decadal predictions leveraging the PCMDI Metric Package with interactive visualization

Reliable climate predictions across multiple timescales are increasingly critical as climate-related risks continue to rise. With the growing number and diversity of climate prediction systems, systematic intercomparison has become essential. Here, we present a comprehensive evaluation framework based on the PCMDI Metric Package to assess the performance of multiple decadal climate prediction systems. Unlike uninitialized simulations, initialized predictions exhibit bias and predictive skill that evolve with forecast lead time. To address this, we introduce (1) model-by-lead-time portrait plots, which efficiently summarize metrics of global temperature, precipitation, and Arctic/Antarctic sea-ice extent, and (2) an HTML-based interactive visualization platform that provides detailed regional and seasonal diagnostics of model bias, skill scores, and ensemble spread for each model and lead time. Comparisons with uninitialized simulations further quantify the relative impacts of initialization and external forcing on prediction skill. The proposed framework provides a scalable and transparent approach for multi-model climate prediction assessments and can be readily extended to a wide range of operational and research forecasting systems.

54 ENVIRONMENTAL SCIENCES↗

ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP

A community diagnostics and performance metrics tool for the evaluation of Earth system models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected essential climate variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases, and soil hydrology–climate interactions, as well as atmospheric CO 2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the Coupled Model Intercomparison Project (CMIP) ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within CMIP and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

58 GEOSCIENCES↗

Process-level model evaluation: a snow and heat transfer metric

Land models require evaluation in order to understand results and guide future development. Examining functional relationships between model variables can provide insight into the ability of models to capture fundamental processes and aid in minimizing uncertainties or deficiencies in model forcing. This study quantifies the proficiency of land models to appropriately transfer heat from the soil through a snowpack to the atmosphere during the cooling season (Northern Hemisphere: October–March). Using the basic physics of heat diffusion, we investigate the relationship between seasonal amplitudes of soil versus air temperatures due to insulation from seasonal snow. Observations demonstrate the anticipated exponential relationship of attenuated soil temperature amplitude with increasing snow depth and indicate that the marginal influence of snow insulation diminishes beyond an effective snow depth of about 50 cm. A snow and heat transfer metric (SHTM) is developed to quantify model skill compared to observations. Land models within the CMIP5 experiment vary widely in SHTM scores, and deficiencies can often be traced to model structural weaknesses. The SHTM value for individual models is stable over 150 years of climate, 1850–2005, indicating that the metric is insensitive to climate forcing and can be used to evaluate each model's representation of the insulation process.

58 GEOSCIENCES↗

Metrics tool for for evaluating atmospheric rivers in climate data

The metric tool code is designed for evaluating atmospheric rivers (AR) in climate models and reanalysis. It is python based, with built in metrics of AR frequency, AR precipitation, AR peak day, AR characteristics (width, length, area, latitude and longitude), and output of diagnostic plots. Is there

Dong, Bo↗

Nuclear Data Impact on Key Metrics for a Representative Molten Chloride Fast Reactor Model

Nuclear data are an essential component of the foundation on which all modeling and simulation methods and tools are relying upon, from the front end to the back end of the nuclear fuel cycle. In this study, the impact of uncertainties in nuclear data is investigated for a representative molten chloride fast reactor, for several important metrics, including eigenvalue, reactivity differences, and nuclide inventories in fuel at 5-yr irradiation. Uncertainty of keff for a full core model was found to be similar between the fresh fuel and the irradiated fuel states (1.7-1.8%), with its primary driver being the uncertainty in the 235U (n,γ) cross section. The results obtained for the reactivity differences show large uncertainties, of over 100%, in elastic scattering sensitivities of several nuclides, which led to large uncertainties of temperature reactivity differences for cladding and reflector. These results provide evidence that the currently applied methods may not be sufficiently adequate for ensuring the reliable determination of such metrics.

Procop, Germina [ORNL] (ORCID:0000000342226393)↗

DOE Cold Climate Heat Pump Challenge: Development, Metrics, and Early Field Observations

Space heating in residential buildings is a major contributor of Greenhouse Gas (GHG) emissions in the United States. New, advanced electric heat pumps are poised to provide a low carbon alternative to traditional fossil-based heating, especially in colder climates. Widespread deployment of cold climate heat pumps could help address the significant portion of building emissions and primary energy used in American households, but these gains will require broader acceptance from consumers and decision makers. As part of the “Energy, Emissions, and Equity” (E3) initiative, the U.S. Department of Energy launched the Cold Climate Heat Pump Challenge (CCHP) in 2021 to accelerate the deployment of the next generation of air source heat pumps. The Challenge is currently focused on residential, centrally ducted, electric heat pumps, with a nominal cooling capacity greater than or equal to 24,000 Btu/h (2 tons) and less than or equal to 65,000 Btu/h (5 tons). The Challenge specifications represent a best-in-class heat pump product that provides high-efficiency heating performance in cold climates, employs environmentally friendly low-Global Warming Potential (GWP) refrigerants, and is designed to be grid interactive. Spearheaded by DOE in partnership with the US Environmental Protection Agency (EPA) and Natural Resources Canada (NRCan), the CCHP Challenge brings together numerous major heat pump manufacturers and key stakeholders including utilities and state agencies across the country. In addition to performance testing in a laboratory environment, a key component of this research is evaluating the in-field performance of the prototype heat pumps developed as part of the Challenge. This paper will discuss the development of the CCHP Challenge, key performance specifications, and the energy and non-energy metrics that will be evaluated through the field study. The types of data that are being collected from the units installed in homes across North America as part of the field validation will be described, including details about the sensors and data acquisition system. Data cleaning and analysis methodologies will be discussed, along with early observations from the winter 2022-2023, spring and summer performance periods, and an assessment of non-energy metrics through pre- and post-installation homeowner surveys. Challenges uncovered and lessons learned throughout the process will also be discussed. Finally, the paper will discuss next steps for the second winter assessment period, areas needing additional research, and potential applications of the data collected and analyzed through this work.

Mendon, Vrushali V.↗

An exhaustive survey of regular peptide conformations using a new metric for backbone handedness ( h )

The Ramachandran plot is important to structural biology as it describes a peptide backbone in the context of its dominant degrees of freedom—the backbone dihedral angles φ and ψ (Ramachandran, Ramakrishnan & Sasisekharan, 1963). Since its introduction, the Ramachandran plot has been a crucial tool to characterize protein backbone features. However, the conformation or twist of a backbone as a function of φ and ψ has not been completely described for both cis and trans backbones. Additionally, little intuitive understanding is available about a peptide’s conformation simply from knowing the φ and ψ values of a peptide (e.g., is the regular peptide defined by φ = ψ = - 100° left-handed or right-handed?). This report provides a new metric for backbone handedness ( h ) based on interpreting a peptide backbone as a helix with axial displacement d and angular displacement θ , both of which are derived from a peptide backbone’s internal coordinates, especially dihedral angles φ , ψ and ω . In particular, h equals sin( θ ) d / d |, with range [-1, 1] and negative (or positive) values indicating left(or right)-handedness. The metric h is used to characterize the handedness of every region of the Ramachandran plot for both cis ( ω = 0°) and trans ( ω = 180°) backbones, which provides the first exhaustive survey of twist handedness in Ramachandran ( φ , ψ ) space. These maps fill in the ‘dead space’ within the Ramachandran plot, which are regions that are not commonly accessed by structured proteins, but which may be accessible to intrinsically disordered proteins, short peptide fragments, and protein mimics such as peptoids. Finally, building on the work of (Zacharias & Knapp, 2013), this report presents a new plot based on d and θ that serves as a universal and intuitive alternative to the Ramachandran plot. The universality arises from the fact that the co-inhabitants of such a plot include every possible peptide backbone including cis and trans backbones. The intuitiveness arises from the fact that d and θ provide, at a glance, numerous aspects of the backbone including compactness, handedness, and planarity.

59 BASIC BIOLOGICAL SCIENCES↗

NLR HPC Eagle Jobs Data and Additional Energy Metrics

Overview: Anonymized job-level records from the Eagle high-performance computing (HPC) system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, resource utilization, CPU/GPU energy consumption, and efficiency metrics. Sensitive fields (user, account, job name) are replaced with cryptographic hashes. System & Timeframe: Eagle was a 2,000-node, 8-petaflop system operated at NLR from 2019–2024. Data covers the full operational lifetime of the system. Slurm data was processed nightly; timestamps are in Mountain Time. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.eagle.job-anon.zip — Core anonymized job records (Hive-partitioned Parquet) esif.hpc.eagle.job-anon-energy-metrics.zip — Same records with additional iLO and Ganglia energy metrics datacard.md — Full dataset documentation ~13.8 million rows, 62 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct through a pipeline: Eagle Jobs API → Redpanda → StreamSets → HPCMON API → PostgreSQL. Node-level power from iLO (HP Integrated Lights-Out); GPU power from Ganglia monitoring, joined to jobs via node lists and time ranges. Preprocessing: Anonymization of name, user, and account fields via cryptographic hashing Derived columns: queue_wait, cpu_eff, max_mem_eff Simplified job state mapping (e.g., "CANCELLED BY 12345" → "CANCELLED") QoS accounting rules (buy-in, standby, or Slurm QoS value) CPU energy estimated from TDP (200W, Intel Xeon Gold 6154, 18 cores) Timezone-aware columns (_tz) sourced from LEX accounting database to correctly handle DST transitions Key Variables: Scheduling: job_id, partition, state_simple, submit_time_tz, start_time_tz, end_time_tz, queue_waitResources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, node_energy_total_watt_hours (iLO), gpu0/1_energy_total_watt_hours (Ganglia) Partitions: bigmem, bigmem-8600, bigscratch, csc, dav, ddn, debug, gpu, haswell, long, mono, short, standard Job States: CANCELLED, COMPLETED, FAILED, NODE_FAIL, OUT_OF_MEMORY, PENDING, RUNNING, TIMEOUT QoS Levels: Unknown, normal, buy-in, debug, penalty, high, standby Important Notes: Non-_tz timestamp columns may be off by one hour across DST boundaries; use _tz columns for time difference calculations Energy fields are null for jobs without monitoring coverage Job step records and raw Slurm JSONB fields are excluded from this extract Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗