SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
This is the completion documentation for the 2022 ASC L2 7905: TOSS4 Validation on CTS-2. At a high-level, this L2 is based on the porting and validation of the TOSS operating system onto the new CTS-2 Linux clusters.
This program developed an advanced model-based monitoring and model-predictive control algorithms for a coal fired power plant (CFPP), and deployed these algorithms in a real-time platform to demonstrate performance benefits for transient flexibility and plant operation efficiency. More specifically, the objectives were successfully achieved through a combination of (i) developing a high-fidelity transient plant model in Apros, which was used as a high-fidelity plant simulation between $100-50\% TMCR$ where TMCR denotes the turbine maximum continuous rating, i.e., baseload, (ii) developing a very fast physics-based reduced-order model (ROM) of the plant, which ran more than $100\times$ faster than real-time, enabling its use as real-time embedded model for model-based estimation (MBE) and model predictive control (MPC) (iii) implementing a real-time MBE based on ROM using a robust unscented Kalman filter (UKF) to continuously tune the ROM to match the measurements from high-fidelity Apros plant model despite significant plant-model mismatch, and thus, obtain a Digital Twin of the plant (iv) designing and implementing a real-time MPC with dual objectives of transient plant load tracking with high ramp rates and minimizing coal consumption, i.e., improving plant efficiency in the baseload-partload operation range of $100-50\% TMCR$. Each key element above was developed and tested individually, and has been reported in corresponding Topical Reports. Finally, all the individual elements were integrated in an overall closed-loop system, that was successfully tested in desktop Simulink test harness simulations with ROM or high-fidelity model as the plant. Thereafter, the Simulink implementation was used to auto-generate C-code and deploy as real-time Docker microservice containers in Linux, and validate that they can run in real-time in the hardware-in-the loop (HIL) setup and produce the same results as in Simulink. The results of the integrated simulation tests in Simulink as well as the real-time HIL deployment are documented in this final report, showing good load tracking for load ramps at $3-4\%/min$ ramp rates, and achieving up to $5.5\%$ reduction in coal relative to baseline operation at $50\% TMCR$ load. The desktop and HIL simulations show successful performance of the overall model based estimation and control solution and achieve the key objectives of the program for flexible, efficient and reliable operation of subcritical coal fired power plants.
There are versions of QUIC for 64-bit Windows, 64-bit Linux, and 64-bit Intel Mac OS X. As compared to earlier 32-bit versions of QUIC, the 64-bit versions allow the QUIC transport and dispersion codes to access greater than 2 GB of RAM and thus larger problems can be run.
A‐STEP is a first‐of‐its‐kind, integrated, open‐source software tool aimed at guiding freight rail decarbonization decision‐making. It has tools for studying energy use details for individual trains, networks of trains, battery and hydrogen charging stations, national energy sourcing and pricing, and overall decarbonization costs and environmental impacts. It gives analysts an ability to study the challenges of making such change happen. Completely amenable to analyst specified inputs and parameter values, it can be customized to provide outputs for a wide variety of assumptions about future energy conditions and technological advances. Written in Python, C++, and VB.Net, A‐STEP can be implemented on both Windows and Linux‐based platforms.
Fuel Analysis under Steady-state and Transients (FAST) is the US Nuclear Regulatory Commission (NRC)’s computer code that calculates the steady-state and transient response of nuclear reactor fuel rods during long-term in-reactor burnup, anticipated operational occurrences (AOOs), design basis accidents (DBAs), and dry storage conditions. The code calculates the temperature, pressure, and deformation of a fuel rod as functions of time-dependent fuel rod power and coolant boundary conditions. The phenomena modeled by the code include: heat conduction through the fuel and other materials, heat transfer from the cladding-to-coolant, cladding elastic and plastic deformation, including creep, fuel-cladding mechanical interaction, fission gas release from the fuel, rod internal pressure and void volume, and cladding oxidation. The code contains necessary material and coolant properties, as well as clad-to-coolant heattransfer correlations, for normal operation through postulated accidents for today’s US-based light water reactor (LWR) fuel designs. FAST-1.0 also contains preliminary materials and models for new LWR fuel concepts, such as accident tolerant fuel (ATF), and non-LWR fuel concepts such as metallic fuels for sodium fast reactors (SFRs). FAST has been developed for use on Windows and Linux operating systems. This document describes FAST-1.0, which is the first official version of this code. This document is one of a series of documents on FAST; the other documents detail the material properties used by FAST as well as its integral assessment to experiments and commercial data.
Improving the understanding of how computational fluid dynamics (CFD) direct numerical simulations (DNS) of flows in the High Flux Isotope Reactor (HFIR) perform when run in parallel using the high performance computing (HPC) platform Summit at the Oak Ridge Leadership Computing Facility (OLCF) is of particular importance to boost the computational tools used to support HFIR conversion to low enriched fuel (LEU). Evaluation of scaling performance was driven by the increasing importance of graphics processing unit (GPU) usage in HPC, which is becoming the standard for modern supercomputers such as Summit. The desired results are to obtain a strong positive correlation between the computational resources dedicated to a problem and the relative speed-up of the simulation in comparison to a benchmark. This capability will allow substantially improvement in HFIR flow analytical capabilities, specifically when predicting turbulence properties at high Reynolds numbers. The study leverages previous simulation results performed with code PHASTA (finite element) on HPC platforms Cori (NERSC) and Theta (ALCF) [1] with computing options provided in the computing platform OpenFOAM (finite volume) at OLCF. Transitioning from PHASTA to OpenFOAM will (1) eliminate dependence on third-party software for mesh generation and manipulation, (2) reduce resource needs by employing modern architectures, and (3) build expertise for future modeling of HFIR-specific problems like heat transfer in involute geometry, entrance effects, flow structure in channel corners, and so on—all important issues when defining the available thermal margins in the transition to LEU. CPUs and GPUs differ significantly in their architecture and utilization, as discussed in the literature [2]. The most important differences are in the approach to computations and their memory. A single GPU contains a large quantity of cores, enabling it to perform with a much higher throughput than a CPU, but execution requires a different approach. GPU codes execute instructions using the Single-Instruction Multiple-Thread (SIMT) approach in which a single instruction is used for groups of threads called warps. A warp typically consists of 32 threads which must execute the same set of instructions, although on separate threads. Alternately, a CPU has far fewer cores that are much more flexible in their operation, excelling at quickly performing more complex serial computations. This is why GPUs have greater throughput when properly utilized. The second important difference is seen when comparing their memory spaces. Limited memory allocations and CPU–GPU communications cause a significant bottleneck in GPU-accelerated programs. Further study was required to properly take advantage of GPU resources. A comprehensive analysis of code performance and the model-specific features of turbulence constitutes the core of this work. In this study, a DNS simulation of HFIR channel turbulence was performed with the finite volume CFD code OpenFOAM v2112 and CUDA v11.0 on Red Hat Enterprise Linux v8.2. The OpenFOAM installation had AMGx integrated to enable GPU acceleration and utilizes the PETSc4FOAM library. The computational resources and the problem size were scaled on CPU and CPU + GPU architectures to gain a better understanding of the performance of a DNS problem on modern computing hardware. The study aimed to analyze the scaling of the code exclusively on CPUs and then to examine the scaling of the codes with GPU acceleration enabled. Scaling studies included CPU and GPU acceleration on a mesh of varying resolution to analyze the impact of problem size relative to computational resources. In the course of preparing the GPU configuration on Summit, mainly using the AMGX solvers, difficulties were encountered stemming from constant changes resulting from extensive ongoing development activities and the changing environment. This resulted in the inability to complete the GPU portion of the work. The code was compiled and tested, but production runs to assess acceleration were not performed because the used discretional compute time allocation expired as year-end approached. The Summit HPC platform is scheduled for decommissioning in 2024, making it unattractive for future use with Nvidia-based GPUs. Therefore, the work will be moved onto NERSC machines in FY24. An application was prepared and submitted, and sufficient node-hours were awarded to continue the research in the next calendar year. This report summarizes work performed thus far, which mostly focused on CPU OpenFOAM computing.
Fuel Analysis under Steady-state and Transients (FAST) is the U.S. Nuclear Regulatory Commission (NRC)’s computer code that calculates the steady-state and transient response of nuclear reactor fuel rods during long-term in-reactor burnup, anticipated operational occurrences (AOOs), design basis accidents (DBAs), and dry storage conditions. The code calculates the temperature, pressure, and deformation of a fuel rod as functions of time-dependent fuel rod power and coolant boundary conditions. The phenomena modeled by the code include: * heat conduction through the fuel and other materials * heat transfer from the cladding-to-coolant * cladding elastic and plastic deformation, including creep * fuel-cladding mechanical interaction * fission gas release from the fuel * rod internal pressure and void volume * cladding oxidation The code contains necessary material and coolant properties, as well as clad-to-coolant heat transfer correlations, for normal operation through postulated accidents and AOOs for today’s U.S.-based light water reactor (LWR) fuel designs. FAST-1.1 also contains preliminary materials and models for new LWR fuel concepts, such as accident tolerant fuel (ATF), and non-LWR fuel concepts such as metallic fuels for sodium fast reactors (SFRs). FAST has been developed for use on Windows and Linux operating systems. This document describes FAST-1.1 and is one of a series of documents on the code; the other documents detail the material properties used by FAST as well as its integral assessment to experiments and commercial data.
SLATE is discontinuing the previously used FTS3 container images so this project focuses on migrating these images, updating them from Scientific Linux 7 (SL7) to AlmaLinux 9, building a continuous integration (CI) pipeline, and executing self-hosted CI runners. This allows FNAL to no longer rely on the SLATE images.
This report outlines the results of the DOE-funded SULI (Science Undergraduate Laboratory Internship) research completed over the summer 2024 on site at Fermilab. The project focused on the \textit{otsdaq} software responsible for data acquisition. The project included drafting updated documentation for new users including new procedures relevant for the Alma-Linux 9 operating system update, and the UPS package managing system replacement with spack (scientific software package manager). A new \textit{art} module (ReadTriggerCounts) to track trigger rates was created, with compatibility to be displayed via a Grafana server. The author also assisted in calorimeter cable installation in the detector hall.
Containers are a revolutionary technology in software development and deployment that provides a lightweight, portable environment for ensuring consistency across multiple computing environments. In anticipation of the MCNP 6.3.1 release, two Docker container images have been released on DockerHub for general use. The MCNP source code is not included in the images, and users are still required to obtain it through RSICC. The images produced by Docker are compliant with the OCI (Open Container Initiative) standards, ensuring compatibility with other container engines such as Podman or Kubernetes’ CRI-O. Initially, the images are stored under the author’s personal space on DockerHub (docker.io/azukaitis), but they will be relocated to a dedicated MCNP group space once approved. In the future, they will also be available through the registry feature of the https://github.com/lanl/mcnp-containers project. The use of Docker provides a pre-configured environment for building and running MCNP, ensuring reproducibility of results across various host architectures. This significantly improves consistency when running MCNP on different systems. Notably, executables and installers from the Docker images have successfully passed the MCNP development branch testing suite on x86-64 architectures, including Windows, macOS, and Linux operating systems. Furthermore, testing has demonstrated compatibility with macOS Docker in emulation mode on the latest Apple Mac M2 Ultra hardware, ensuring robust support even on the latest platforms. In this document, we will provide a step-by-step guide to using the Docker images across multiple platforms. Additionally, we will present performance numbers for building and running the MCNP test suite.
Monitoring the status of a high throughput computing cluster running computationally intensive production jobs is a crucial yet challenging system administration task due to the complexity of such systems. To this end, we train autoencoders using the Linux kernel CPU metrics of the cluster. Additionally, we explore assisting these models with graph neural networks to share information across threads within a compute node. The models are compared in terms of their ability to: 1) Produce a compressed latent representation that captures the salient features of the input, 2) Detect anomalous activity, and 3) Make distinction between different kinds of jobs run at Jefferson Lab. The goal is to have a robust encoder whose compressed embeddings are used for several downstream tasks. We extend this study further by deploying these models in a human-in-the-loop production-based setting for the anomaly detection task and discuss the associated implementation aspects such as continual learning and the criterion to generate alarms. This study represents a first step in the endeavor towards building self-supervised large-scale foundation models for computing centers.
Fuel Analysis under Steady-state and Transients (FAST) is the U.S. Nuclear Regulatory Commission (NRC)’s computer code that calculates the steady-state and transient response of nuclear reactor fuel rods during long-term in-reactor burnup, anticipated operational occurrences (AOOs), design basis accidents (DBAs), and dry storage conditions. The code calculates the temperature, pressure, and deformation of a fuel rod as functions of time-dependent fuel rod power and coolant boundary conditions. The phenomena modeled by the code include: • heat conduction through the fuel and other materials • heat transfer from the cladding-to-coolant • cladding elastic and plastic deformation, including creep • fuel-cladding mechanical interaction • fission gas release from the fuel • rod internal pressure and void volume • cladding oxidation The code contains necessary material and coolant properties, as well as clad-to-coolant heat-transfer correlations, for normal operation through postulated accidents and AOOs for today’s U.S.-based light water reactor (LWR) fuel designs. FAST-1.2 also contains preliminary materials and models for new LWR fuel concepts, such as accident tolerant fuel (ATF), and non-LWR fuel concepts such as metallic fuels for sodium fast reactors (SFRs). FAST has been developed for use on Windows and Linux operating systems. This document describes FAST-1.2 and is one of a series of documents on the code; the other documents detail the material properties used by FAST as well as its integral assessment to experiments and commercial data.
We present K-Foundry, a framework that enables the integration of Simple Linux Utility for Resource Management (SLURM) with Kubernetes (K8s) via a Kubernetes-like Control Plane (KCP). Our implementation seamlessly enables a unified communication and scheduling layer for a fleet of multiple diverse computing platforms. While SLURM and K8s traditionally support distinct scheduling models, they have recently started aligning their objectives by moving towards a more converged execution environment.
With the recent diversification of the hardware landscape in the high-performance computing (HPC) community, performance-portability solutions are becoming more and more important. One of the most popular choices is Kokkos, which recently became a Linux Foundation project. Most of its development is supported by the US Department of Energy and the French Alternative Energies and Atomic Energy Commission. Kokkos is implemented as a C++ library with multiple backends to support CPUs as well as various GPU architectures. These backends include OpenMP, CUDA, HIP, and also SCYL. This approach enables users to leverage the preferred vendor toolchain for the respective platform (e.g. CUDA, ROCm, OneAPI). The SYCL backend is used to target Intel GPUs, in particular to support the Aurora exascale supercomputer. However, SYCL itself also offers a large degree of portability, and in fact Kokkos’ CI for SYCL has been running on NVIDIA hardware due to a lack of access to Intel GPUs. In this report, we describe our experience with using Kokkos SYCL backend on AMD GPUs targeting the Frontier supercomputer at Oak Ridge National Laboratory. The two major SYCL implementations are DPC++ and AdaptiveCpp. While the Kokkos SYCL backend has been implemented using the former, the latter was the first implementation to target AMD GPUs. We will discuss the experience with both of these SYCL implementations in terms of functionality and performance. Using Kokkos to evaluate SYCL toolchains has a number of benefits. Kokkos’ use of SYCL is fairly complex, exercising features such as graphs, relocatable device functions, atomics – including for non-arithmetic types, as well as pinned and page migratable memory allocations. Kokkos also needs to implement capabilities such as Kokkos’ hierarchical parallelism that are not a straight-forward mapping to SYCL capabilities. Furthermore, a large number of libraries and applications that represent diverse use cases are implemented in Kokkos, providing readily available test cases for a toolchain evaluation. Preliminary results show that support for AMD GPUs in DPC++ is much less mature than for NVIDIA GPUs or Intel GPUs. While the situation has improved significantly over the last year, we still encounter many runtime failures, dispatching problems, and code generation issues. With AdaptiveCpp the challenges arise even earlier in the evaluation process. Since Kokkos’ SYCL implementation is largely focused on supporting Intel GPUs, we opted to leverage SYCL extensions which are available in DPC++ but not in AdaptiveCpp. Furthermore, AdaptiveCpp appears to be less conformant with the SYCL2020 standard which Kokkos relies on. In some cases, we are able to work around the lack of feature support, in other cases we have to disable certain Kokkos capabilities to evaluate the toolchain. Our evaluation will leverage Kokkos’ unit tests to establish basic functionality and feature completeness. We then use simple benchmarks for components of a CG implementation as a measure of usability and performance of the SYCL toolchains.
We developed an analysis pipeline for transposase-accessible chromatin sequencing (ATAC-Seq) data derived from bulk samples, which brings together publicly available R packages in addition to command-line tools designed for analysis of bulk ATAC-Seq data and can be run on any computer running a Linux-like operating system such as Ubuntu or Apple OSX.
The Industroyer2 and Wiper Malware Targeting Ukrainian Energy Provider 2022 Precursor Analysis Report leverages publicly available information about the Industroyer2 cyber attack and catalogs anomalous observables for each technique employed in the attack. This analysis is based upon the methodology of the Cybersecurity for the Operational Technology Environment (CyOTE) program. An adversary attempted to cause a blackout in Ukraine in April 2022 by using the Industroyer2 malware against a regional Ukrainian energy provider. The adversary targeted eight high-voltage electrical substations and utilized the malware in tandem with disk wipers for Windows, Linux, and Solaris operating systems in an attempt to make response and recovery efforts more difficult. The adversary reused a piece of the original Industroyer malware designed to open circuit breakers and de-energize target substations. The adversary gained initial access to the victim’s enterprise network through unknown means in February 2022 and was able to perform reconnaissance, pivot to the operations network, and reside in the system for at least 51 days. This gave the adversary a detailed understanding of the environment and allowed them to customize the Industroyer2 malware to the victim’s operations network. However, defenders detected and stopped the attack before the adversary could achieve their intended impact. Had the Industroyer2 attack been successful, it could have caused a blackout for more than two million people during the early stages of Russia’s invasion of Ukraine. Researchers and analysts identified 22 unique techniques (used in a sequence of 31 steps) utilized during the attack with a total of 297 observables using MITRE ATT&CK® for Industrial Control Systems. The CyOTE program assesses observables accompanying techniques used prior to the triggering event to identify opportunities to detect malicious activity. If observables accompanying the attack techniques are perceived and investigated prior to the triggering event, earlier comprehension of malicious activity can take place. Twenty-three of the identified techniques used during the Industroyer2 cyber attack were precursors to the triggering event. Analysis identified 224 observables associated with these precursor techniques, 122 of which were assessed to have an increased likelihood of being perceived in the 51 days preceding the triggering event. The response and comprehension time could have been reduced if the observables had been identified earlier. The information gathered in this report contributes to a library of observables tied to a repository of artifacts, data sources, and technique detection references for practitioners and developers to support the comprehension of indicators of attack. Asset owners and operators can use these products if they experience similar observables or to prepare for comparable scenarios.
Fuel Analysis under Steady-state and Transients (FAST) is the U.S. Nuclear Regulatory Commission (NRC)’s computer code that calculates the steady-state and transient response of nuclear reactor fuel rods during long-term in-reactor burnup, anticipated operational occurrences (AOOs), design basis accidents (DBAs), and dry storage conditions. The code calculates the temperature, pressure, and deformation of a fuel rod as functions of time-dependent fuel rod power and coolant boundary conditions. The phenomena modeled by the code include heat conduction through the fuel and other materials, heat transfer from the cladding-to-coolant, cladding elastic and plastic deformation (including creep), fuel-cladding mechanical interaction, fission gas release from the fuel, rod internal pressure, void volume, and cladding oxidation. The code contains necessary material and coolant properties, as well as clad-to-coolant heat transfer correlations, for normal operation through postulated accidents and AOOs for today’s U.S.-based light water reactor (LWR) fuel designs. FAST-1.2.2 also contains preliminary materials and models for new LWR fuel concepts, such as accident tolerant fuel (ATF), and non-LWR fuel concepts such as metallic fuels for sodium fast reactors (SFRs). FAST has been developed for use on Windows and Linux operating systems. This document describes FAST-1.2.2 and is one of a series of documents on the code; the other documents detail the material properties used by FAST as well as its integral assessment to experiments and commercial data.
The PSEC6 is an application-specific integrated circuit (ASIC) designed for a readout system for a large area picosecond photodetector (LAPPD). The PSEC6 is currently in fabrication and pending testing. The testing system for the PSEC5, the previous iteration of the ASIC, required expensive and non-portable equipment, because the ASIC needs twelve adjustable reference voltages. The new testing system consists of an low-cost, open-source, cross-platform graphical user interface (GUI), a digital system, and a biasing board. The digital system is the interface between the GUI and biasing board, and can be implemented on a microcontroller or field-programmable gate array (FPGA). The biasing board contains twelve digital-to-analog converters (DACs) that are configurable via the GUI, which gives users the ability to write voltage values to all or specific DACs. The GUI was developed in C on Linux using the widget library GTK4 and cross-compiled for Windows compatibility. I2C and SPI protocols were implemented on an Adafruit Feather ESP32-S3 microcontroller to write commands to the DACs and PSEC6. A hardware implementation of the I2C protocol is in development on an FPGA. Since LAPPDs will be used by the Accelerator Neutrino Neutron Interaction Experiment (ANNIE) at Fermilab, the PSEC6 testing system in this internship project can potentially benefit future neutrino research. The project is relevant to the Department of Energy’s microelectronics mission, because the PSEC6 is an ASIC that will handle fast time signals arriving from the detector for readout. It also provided experience with building a cross-platform user interface, practicing digital design and implementation in hardware description language (HDL), and using simulations to inform new design iterations.