Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

TRIM: AI Guided Random Number Generation for Resource-Constrained IoT Systems

Random numbers often serve as the backbone for many security solutions in diverse domains such as cryptography, side channel leakage prevention, and moving target defense. However, generating true random numbers requires a physical source of entropy (e.g. hardware, quantum, environmental phenomenon) making it difficult to realize at a large scale and at a low cost. On the flip side, pseudorandom number generators (easy to implement) following a specific distribution (e.g. Gaussian) can be easily compromised given a sufficient amount of traces. In this work, we have developed a machine learning-guided generative approach that can be used to create portable, resource-efficient, and cost-effective random number generators with high throughput and true randomness characteristics. We implement the proposed approach as a highly parameterized framework and perform extensive evaluation for different settings. The framework was able to learn from true random sources such as irrational numbers and environmental audio noise and imitate those sources towards generating new good quality random numbers on demand. We have generated more than 1 billion bits and observed robust performance in terms of true randomness metrics obtained from NIST SP 800-22 and FIPS 140-1 randomness test suites achieving a throughput of up to 142.85 Mbps. Compared to the state-of-the-art (SOTA) technique, the iso-cost setup of our framework can achieve more than 500 Mbps in a distributed setting. We have evaluated the efficacy of running the true randomness imitation AI models on target edge devices such as Raspberry Pi 4 (Model B), Nvidia Jetson Nano, Nvidia Jetson Orin Nano and Nvidia Jetson Xavier. We have also looked at the security of the TRIM framework itself against different adversarial threat models.

Cybersecurity↗

Updating the BNB Flux Prediction at SBND

Precise, accurate neutrino flux predictions for neutrino beam experiments are crucial for physics results. Flux uncertainties contribute significantly to the total systematic uncertainties seen in modern accelerator neutrino measurements such as cross sections, oscillations, and BSM studies.. For over a decade, experiments utilizing Fermilab’s Booster Neutrino Beam (BNB) have relied on the 2009 MiniBooNE flux prediction. However, the high-statistics era of the Short-Baseline Neutrino (SBN) program, with both SBND and ICARUS now operating, demands a modernized flux model and framework. While SBND is using the MiniBooNE flux model for its Generation 1 analyses, including many upcoming cross section measurements, future work will be based on a new flux model. In this talk, the ongoing work towards this new model will be outlined, including a new simulation framework (G4BNB), a new evaluation of model parameters from hadron scattering data, and a new framework for evaluating systematic uncertainties (BNBFP). Additionally, expansions to the flux model to include BSM contributions from neutral mesons will be discussed, as well as the PRISM capabilities of SBND to observe a wide range of off axis angles of the neutrino beam.

Paton, Josephine [Fermilab]↗

Updating the BNB Flux Prediction at SBND

Precise, accurate neutrino flux predictions for neutrino beam experiments are crucial for physics results. Flux uncertainties contribute significantly to the total systematic uncertainties seen in modern accelerator neutrino measurements such as cross sections, oscillations, and BSM studies.. For over a decade, experiments utilizing Fermilab’s Booster Neutrino Beam (BNB) have relied on the 2009 MiniBooNE flux prediction. However, the high-statistics era of the Short-Baseline Neutrino (SBN) program, with both SBND and ICARUS now operating, demands a modernized flux model and framework. While SBND is using the MiniBooNE flux model for its Generation 1 analyses, including many upcoming cross section measurements, future work will be based on a new flux model. In this talk, the ongoing work towards this new model will be outlined, including a new simulation framework (G4BNB), a new evaluation of model parameters from hadron scattering data, and a new framework for evaluating systematic uncertainties (BNBFP). Additionally, expansions to the flux model to include BSM contributions from neutral mesons will be discussed, as well as the PRISM capabilities of SBND to observe a wide range of off axis angles of the neutrino beam.

Paton, Josephine [Fermilab]↗

AutoBEM: A scalable framework for nationwide building energy simulation and retrofit evaluation in the United States

This paper presents AutoBEM, an integrated, automated framework for nationwide building energy modeling and retrofit evaluation in the United States. Unlike prior UBEM platforms that either rely primarily on representative stock sampling or operate at city scale, AutoBEM automates the generation of building-resolved, physics-based EnergyPlus/OpenStudio simulation models at national scale using GIS-derived geometry, prototype-based assumptions, and standardized scalable workflows. Leveraging the Model America dataset and high-performance computing, AutoBEM generates and simulates energy models for 122.9 million buildings, representing 97.8% of the U.S. building stock. These models are being made publicly and freely available as the Model America v1.0 (MAv1) dataset. AutoBEM supports detailed, building-level assessments of energy consumption, CO2 emissions, and post-processed anthropogenic heat emissions (AHE), and evaluates 151 energy conservation measures (ECMs) using localized utility pricing and building characteristics. In addition, AutoBEM incorporates both typical and future climate conditions through integration with Typical Meteorological Year (TMY) and Future TMY (fTMY) weather data derived from IPCC scenarios. In a case study of Phoenix, Arizona, AutoBEM identified several high-efficiency HVAC upgrades and selected envelope measures with short modeled payback periods (1.5 years) for certain building types and standards. Simulations under future climate scenarios (SSP5–RCP8.5) project an 11.3% increase in electricity use and a 32% reduction in natural gas demand by 2100, underscoring the need for climate-adaptive retrofit planning. By enabling reproducible, bottom-up, and location-specific analysis at scale, AutoBEM provides a step toward a national digital twin of the built environment and supports data-driven screening and planning for decarbonization, resilience, and energy equity.

Li, Hang [ORNL] (ORCID:0000000306001920)↗

Evaluation and Demonstration of Blockchain Applicability Framework

Blockchain technology has been gaining great interest from a variety of industry sectors, including financial, food processing, and power and energy markets. Realizing the strength of blockchain technology beyond the successful application in the cryptocurrency arena, researchers have been evaluating and using blockchain for applications such as supply chain management, transactive industry (both financial and energy), system integrity, device cybersecurity, identity management, and much more. One of the unique elements of the blockchain technology that made it such a captivating technology to researchers is its plethora of features. Some of the features include smart contracts, cryptocurrency and tokenizing, immutable distributed ledger, cryptographic hashing, and digital signature. In addition, there are multiple types of blockchains, such as permissioned/private and permissionless/public, and various consensus models, such as proof-of-work, proof-of-authority, proof-of-burn, and proof-of-stake. Therefore, it is often non-trivial to determine if an application requires a blockchain. If so, what kind of blockchain and consensus is most appropriate? This paper discusses the blockchain applicability framework (BAF), which was specifically designed with the purpose to answer those questions. BAF is divided into five domains, 18 subdomains, and about 100 controls. It is designed to ingest detailed user requirements to perform a weighted evaluation that is built on mathematical constructs to determine the ideal combination of blockchain that is appropriate for an application. Along with the core logical formulation of BAF, this paper depicts the efficacy of BAF through two use cases

Gourisetti, Sri Nikhil G.↗

Evaluation of the Self Retrieval Augmented Generation Technique on Common Security Advisory Framework Data

This small experimental report evaluates a variation of Retrieval Augmented Generation (RAG), called Self-RAG. This method uses a generative language model that incorporates retrieved facts into its generation and is explicitly trained to be able to determine whether retrieved information is enough to answer the input query, with a user-defined threshold for confidence. We performed an experiment using data from the publicly available CISA Common Security Advisory Framework (CSAF) repository (https://github.com/cisagov/CSAF) as the database of facts to be used in retrieval. Qualitative results from the experiment demonstrate that the Self-RAG method has some ability to provide reasonable answers to queries that are in the dataset and will often ignore irrelevant information when asked outside of domain questions (e.g., general facts). In settings with deliberately confusing questions (the question is within domain, but asks about a fabricated advisory), it was able to refuse 40% of the time without further adjustments to the original framework. While this performance is not sufficient for current practical use, further improvements to data formatting, disambiguating results, and leveraging threshold values could improve performance significantly. However, evaluating this will require more extensive evaluations on larger datasets and potentially better models.

97 MATHEMATICS AND COMPUTING↗

Evaluating Function-as-a-Service (FaaS) frameworks for the Accelerator Control System

As particle accelerator control systems evolve in complexity and scale, the need for responsive, scalable, and cost-effective computational infrastructure becomes increasingly critical. Function-as-a-Service (FaaS) offers an alternative to traditional monolithic architecture by enabling event-driven execution, automatic scaling, and fine-grained resource utilization. This paper explores the applicability and performance of FaaS frameworks in the context of a modern particle accelerator control system, with the objective of evaluating their suitability for short lived and triggered workloads. In this paper, we evaluate prominent open-source FaaS platforms in executing functional logic, triggers, and diagnostics routines. Evaluation metrics consist of cold-start latency, scalability, performance, integration with other open-source tools like Kafka. Experimental workloads were designed to simulate real-world control tasks when implemented as stateless FaaS functions. These workloads were benchmarked under various invocation loads and network conditions. Self-hosted FaaS platforms, when deployed within accelerator networks, offer greater control over execution environment, better integration with legacy systems, and support for real-time guarantees when paired with message queues. Based on lessons learned and evaluation metrics, this paper describes reliability of the FaaS framework for the Accelerator Control Systems (ACS).

Jaikar, A. [Fermilab] (ORCID:0000000332046217)↗

A proposed framework for the development and qualitative evaluation of West Nile virus models and their application to local public health decision-making

West Nile virus (WNV) is a globally distributed mosquito-borne virus of great public health concern. The number of WNV human cases and mosquito infection patterns vary in space and time. Many statistical models have been developed to understand and predict WNV geographic and temporal dynamics. However, these modeling efforts have been disjointed with little model comparison and inconsistent validation. In this paper, we describe a framework to unify and standardize WNV modeling efforts nationwide. WNV risk, detection, or warning models for this review were solicited from active research groups working in different regions of the United States. A total of 13 models were selected and described. The spatial and temporal scales of each model were compared to guide the timing and the locations for mosquito and virus surveillance, to support mosquito vector control decisions, and to assist in conducting public health outreach campaigns at multiple scales of decision-making. Our overarching goal is to bridge the existing gap between model development, which is usually conducted as an academic exercise, and practical model applications, which occur at state, tribal, local, or territorial public health and mosquito control agency levels. The proposed model assessment and comparison framework helps clarify the value of individual models for decision-making and identifies the appropriate temporal and spatial scope of each model. This qualitative evaluation clearly identifies gaps in linking models to applied decisions and sets the stage for a quantitative comparison of models. Specifically, whereas many coarse-grained models (county resolution or greater) have been developed, the greatest need is for fine-grained, short-term planning models (m–km, days–weeks) that remain scarce. We further recommend quantifying the value of information for each decision to identify decisions that would benefit most from model input.

60 APPLIED LIFE SCIENCES↗

A seamless approach for evaluating climate models across spatial scales

In regions of the world where topography varies significantly with distance, most global climate models (GCMs) have spatial resolutions that are too coarse to accurately simulate key meteorological variables that are influenced by topography, such as clouds, precipitation, and surface temperatures. One approach to tackle this challenge is to run climate models of sufficiently high resolution in those topographically complex regions such as the North American Regionally Refined Model (NARRM) subset of the Department of Energy’s (DOE) Energy Exascale Earth System Model version 2 (E3SM v2). Although high-resolution simulations are expected to provide unprecedented details of atmospheric processes, running models at such high resolutions remains computationally expensive compared to lower-resolution models such as the E3SM Low Resolution (LR). Moreover, because regionally refined and high-resolution GCMs are relatively new, there are a limited number of observational datasets and frameworks available for evaluating climate models with regionally varying spatial resolutions. As such, we developed a new framework to quantify the added value of high spatial resolution in simulating precipitation over the contiguous United States (CONUS). To determine its viability, we applied the framework to two model simulations and an observational dataset. We first remapped all the data into Hierarchical Equal-Area Iso-Latitude Pixelization (HEALPix) pixels. HEALPix offers several mathematical properties that enable seamless evaluation of climate models across different spatial resolutions including its equal-area and partitioning properties. The remapped HEALPix-based data are used to show how the spatial variability of both observed and simulated precipitation changes with resolution increases. This study provides valuable insights into the requirements for achieving accurate simulations of precipitation patterns over the CONUS. It highlights the importance of allocating sufficient computational resources to run climate models at higher temporal and spatial resolutions to capture spatial patterns effectively. Furthermore, the study demonstrates the effectiveness of the HEALPix framework in evaluating precipitation simulations across different spatial resolutions. This framework offers a viable approach for comparing observed and simulated data when dealing with datasets of varying spatial resolutions. By employing this framework, researchers can extend its usage to other climate variables, datasets, and disciplines that require comparing datasets with different spatial resolutions.

54 ENVIRONMENTAL SCIENCES↗

Performance Evaluation of Python Based Data Analytics Frameworks in Summit: Early Experiences

The explosion in the volumes of data generated from ever-larger simulation campaigns and experiments or observations necessitates competent tools for data wrangling and analysis). While the Oak Ridge Leadership Computing Facility (OLCF) provides a variety of tools to perform data wrangling and data analysis tasks, Python based tools often lack scalability, or the ability to fully exploit the computational capability of OLCF’s Summit supercomputer. NVIDIA RAPIDS and Dask offer a promising solution to accelerate and distribute data analytics workloads from personal computers to heterogeneous supercomputing systems. We discuss early performance evaluation results of RAPIDS and Dask on Summit to understand their capabilities, scalability, and limitations. Our evaluation includes a subset of RAPIDS libraries, i.e., cuDF, cuML, and cuGraph, and Chainer’s CuPy, and their multi-GPU variants when available.We also draw on the observed trends from the performance evaluation results to discuss best practices for maximizing performance.

Hernandez Arreguin, Benjamin↗

Adsorption of Natural Gas in Metal–Organic Frameworks: Selectivity, Cyclability, and Comparison to Methane Adsorption

Evaluation of metal–organic frameworks (MOFs) for adsorbed natural gas (ANG) technology employs pure methane as a surrogate for natural gas (NG). This approximation is problematic, as it ignores the impact of other heavier hydrocarbons present in NG, such as ethane and propane, which generally have more favorable adsorption interactions with MOFs compared to methane. Herein, using quantitative Raman spectroscopic analysis and Monte Carlo calculations, we demonstrate the adsorption selectivity of high-performing MOFs, such as MOF-5, MOF-177, and SNU-70, for a methane and ethane mixture (95:5) that mimics the composition of NG. The impact of selectivity on the storage and deliverable capacities of these adsorbents during successive cycles of adsorption and desorption, simulating the filling and emptying of an ANG tank, is also demonstrated. Finally, the study reveals a gradual reduction in the storage performance of MOFs, particularly with smaller pore volumes, due to ethane accumulation over long-term cycling, until a steady state is reached with substantially degraded storage performance.

03 NATURAL GAS↗

Evaluating precipitation distributions at regional scales: a benchmarking framework and application to CMIP5 and 6 models

Abstract. As the resolution of global Earth system models increases, regional-scale evaluations are becoming ever more important. This study presents a framework for quantifying precipitation distributions at regional scales and applies it to evaluate Coupled Model Intercomparison Project (CMIP) 5 and 6 models. We employ the Intergovernmental Panel on Climate Change (IPCC) sixth assessment report (AR6) climate reference regions over land and propose refinements to the oceanic regions based on the homogeneity of precipitation distribution characteristics. The homogeneous regions are identified as heavy-, moderate-, and light-precipitating areas by K-means clustering of Integrated Multi-satellitE Retrievals for Global Precipitation Measurement (GPM) version 6 final run product (IMERG) precipitation frequency and amount distributions. With the global domain partitioned into 62 regions, including 46 land and 16 ocean regions, we apply 10 established precipitation distribution metrics. The collection includes metrics focused on the maximum peak, lower 10th percentile, and upper 90th percentile in precipitation amount and frequency distributions; the similarity between observed and modeled frequency distributions; an unevenness measure based on cumulative amount; average total intensity on all days with precipitation; and number of precipitating days each year. We apply our framework to 25 CMIP5 and 41 CMIP6 models, as well as six observation-based products of daily precipitation. Our results indicate that many CMIP5 and 6 models substantially overestimate the observed light-precipitation amount and frequency, as well as the number of precipitating days, especially over midlatitude regions outside of some land regions in the Americas and Eurasia. Improvement from CMIP5 to 6 is shown in some regions, especially in midlatitude regions, but it is not evident globally, and over the tropics most metrics point toward degradation.

54 ENVIRONMENTAL SCIENCES↗

CONUS-wide Projected Flood Frequency and Uncertainty Estimates, Version 1.0

This dataset presents a large-ensemble of CONUS-wide projected flood frequency and uncertainty estimates across ~2.7 million NHDPlusV2 river reaches over the CONUS. The framework producing this dataset leverages a multi-model, uncertainty-aware modeling framework that allows evaluating shifts in flood frequences at the stream reach level across the CONUS. CONUS-wide ensemble streamflow projections generated from hydrologic simulations driven by downscaled and bias-corrected Coupled Model Intercomparison Project Phase 6 (CMIP6) outputs are used to derive these flood frequency and uncertainty estimates over the period 1980 - 2099. A spatially consistent regional L-moment algorithm is applied across clusters defined by the US Hydrologic Unit Code Subregions (HUC4s and HUC8s) and NHDPlusV2 stream orders to estimate flood frequencies. The dataset also includes at-site based flood estimates that allow for the comparison between local and regional approach-based estimates, assess projected changes, and characterize their uncertainties. For more reliable estimation of rare flood frequencies such as 500 and 1000-year return periods, super-ensemble based estimates are also included in the dataset. This dataset is derived to support the "Impact-Informed Dam Safety Risk Assessment for Securing Hydropower Assests" project for the US Department of Energy (DOE) Hydropower and Hydrokinetic Office (H2O). For further details, refer to Kao et al. (2022), Ghimire et al. (2023), Ghimire et al. (2025), and Hosking and Wallis (1997).

Ghimire, Ganesh [ORNL] (ORCID:0000000242843941)↗

Bayesian Monte-Carlo Framework: New Methods for Resonance Parameter Evaluation [Slides]

BMC evaluation is a tool to address imperfect data & models, non-linear models, and non-normal PDFs. ENDF-6 format does not allow non-normal parameter PDFs. Storing posterior sets allow for variance, covariance, skewness, etc. To better predict criticality, we could document non-normal parameter PDFs (i.e. asymmetric uncertainty), consider non-linear sensitivity of $\kappa$ eff to resonance parameters, and reduce uncertainty in key resonance parameters.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

A Flexible Quasi-Static Mooring Design Optimization Method for Floating Structures

This paper presents a flexible and efficient design method for optimizing the mooring systems of floating structures. Mooring system optimization is challenging because of the strong nonlinearity of mooring system behavior and the many technical constraints that must be satisfied. Furthermore, different mooring configurations can have very different design spaces. While some successful examples of mooring design optimization exist in the literature, developing an optimization approach that can work across various mooring design problems is a larger challenge. We present such a method based on a flexible parameterization that allows a wide variety of mooring designs to be described by a list of variables, a quasi-static mooring model that provides efficient evaluation of a mooring design without directly considering mooring system dynamics, and an optimization framework that generates, evaluates, and adjusts the mooring design while considering user-specified constraints such as offset limits, strength safety factors, and seabed contact limits. We demonstrate the design optimization framework on four mooring design problems, each for a different type of mooring system. We compare the use of different design modes to simplify the optimization problem, showing that they can reduce the computation time by up to 75%. We also compare different optimization algorithms and find that the resulting computational speed can vary by up to 51 times. We perform a sensitivity study on one design and find that the local sensitivity of anchoring radius to water depth has a positive correlation of 0.29, but the global sensitivity shows large nonlinearities. Lastly, we perform a coupled dynamic analysis on one of the optimized designs and find that the predicted mean platform motions and mooring line tensions are within 1% of dynamic results and the extreme motions and tensions are within 14%. Lastly, we show that a DEA-Chain-Polyester mooring configuration is cost-optimal for the given design problem of the demonstrations, which aligns with general industry practice.

16 TIDAL AND WAVE POWER↗

An Authentication Vulnerability Assessment of Connected Lighting Systems

Emerging connected lighting systems (CLS) that incorporate distributed intelligence, network interfaces, and sensors can become data-collection platforms that enable a wide range of valuable new capabilities as well as greater energy savings in buildings and cities. However, CLS technology is currently at an early stage of development, and its increased connectivity introduces cybersecurity risks that are new to the lighting industry and that must be addressed for successful integration with other systems. While a number of existing frameworks, guidelines, and tests for evaluating cybersecurity vulnerability may apply to CLS in whole or in part, there is currently no mandatory requirement for cybersecurity testing or certification. The lighting industry, including technology developers and specification organizations, is currently evaluating the suitability of existing frameworks and guidelines for CLS. To support these efforts, Pacific Northwest National Laboratory (PNNL) is conducting a series of studies intended to educate lighting industry stakeholders on specific cybersecurity practices and characterize their implementation in commercially available CLS with varying system architectures, network-communication technologies, and degrees of maturity. This study demonstrates that tests for authentication vulnerabilities can be developed with objective pass/fail criteria, therby facilitating comparisons between CLS. Based on the limited results of this study, it appears that the CLS that are being brought to market have varying levels of authentication vulnerability. It is hoped that these evaluations will support and perhaps accelerate industry discussions on the risks of specific security vulnerabilities, what vulnerabilities should be addressed by in-development of future lighting-specific best practices, and whether any such practices should be included in voluntary lighting standards.

42 ENGINEERING↗