STNS01-6 BEE - FY20 P6-2: Support for HPC resource managers
BEE provides a portable, modular, HPC-focused workflow engine capable of managing containerized applications at scale.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
BEE provides a portable, modular, HPC-focused workflow engine capable of managing containerized applications at scale.
This document is the final report for project OR16-ML-3DTomography-PD3Jb, “3D Tomography and Image Processing Using Fast Neutrons.” This project was motivated by the availability of three-dimensional (3D) tomographic neutron transmission and induced-reaction data following the development of associated-particle imaging (API) systems with two-dimensional (2D) or “area” neutron detectors such as the Advanced Portable Neutron Imaging System, which is capable of simultaneous measurement of transmission and induced-reaction neutrons.
Californium-252 (Cf-252) has been synthesized on a large scale (mg) since the 1970s for use as a portable neutron source. With a half-life of 2.645 years and a spontaneous fission fraction of 3.09%, Cf-252 emits 2.31 × 10 12 neutrons/gram second. Today most of the Cf-252 produced in the world is made at Oak Ridge National Laboratory (ORNL) using the High Flux Isotope Reactor (HFIR) and the Radiochemical Engineering Development Center (REDC). Because of its highly radioactive nature, all manipulations of Cf-252 need to be performed in hot cell facilities by highly trained personnel. The Cf-252 produced at ORNL is done at the REDC, which is conveniently located next to the HFIR. Cf-252 is sold as bulk wire, which has been described as palladium (Pd) metal incasing californium oxide (Cf 2 O 3 ). The process was developed by Mosley et al. at Savannah River Laboratory (SRL) where it was modeled after the thorium work done by Fuschillo et al. This process has been used since the 1970s to synthesize this unique wire, but its fundamental chemistry and materials characteristics were never openly reported. Here we aim to better understand the chemistry and the materials characteristics of Cf-252 wire production for process improvements and to better our fundamental understanding of the mid-late actinides. Though there are many challenges in working with radioactive elements such as Cf, the isotope Cf-252 is even more complicated due to its neutron emission. One challenge with neutron emission is the ability of neutrons to penetrate much deeper through materials, causing radiation damage. This makes using instrumentation to characterize the chemistry and materials properties of this process difficult. For this reason, this study will first focus on terbium (Tb) as a surrogate for Cf. Currently Tb is used as a carrier or filler in Cf production when small amounts of Cf are being processed. This, along with their similar size and charge, makes Tb a reasonable surrogate for this study.
TChem is an open source software library for solving complex computational chemistry problems and analyzing detailed chemical kinetic models. The software provides support for: complex kinetic models for gas-phase and surface chemistry; thermodynamic properties based on NASA polynomials; species production/consumption rates; stable time integrator for solving stiff time ordinary differential equations; and, reactor models such as homogenous gas-phase ignition (with analytical Jacobian matrices), continuously stirred tank reactor, plug-flow reactor. This toolkit builds upon earlier versions that were written in C and featured tools for gas-phase chemistry only. The current version of the software was completely refactored in C++, uses an object-oriented programming model, and adopts Kokkos as its portability layer to make it ready for the next generation computing architectures i.e., multi/many core computing platforms with GPU accelerators. We have expanded the range of kinetic models to include surface chemistry and have added examples pertaining to Continuously Stirred Tank Reactors (CSTR) and Plug Flow Reactor (PFR) models to complement the homogenous ignition examples present in the earlier versions. To exploit the massive parallelism available from modern computing platforms, the current software interface is designed to evaluate samples in parallel, which enables large scale parametric studies, e.g. for sensitivity analysis and model calibration.
Bladewerx TM LLC (Rio Rancho, NM) manufactures instrumentation, neutron shielding and activation foils for the radiation protection industry. Specializing in portable alpha/beta air monitors and sample counters, Bladewerx is the source of Speclon TM PTFE filter media that they recommend for high-resolution alpha spectroscopy. Los Alamos National Laboratory (LANL) utilizes Speclon TM filter material in CAM (Continuous Air Monitor) samplers for workplace air monitoring. The LANL Aerosol Engineering Facility received air filter material from Bladewerx, referred to as “New Speclon 5” in this document, in order to distinguish from filter material that was previously received (referred to as “Speclon 5” in this document). In this document, the aerosol collection efficiency and airflow resistance (pressure drop) were measured for the New Speclon 5 filter material.
In 2015, the Lawrence Livermore National Laboratory started development of next-generation multiphysics simulation capabilities for the National Nuclear Security Administration under the Advanced Technologies Development and Mitigation (ATDM) element of the Advanced Simulation and Computing program in collaboration with the Exascale Computing Project (ECP). A key driver for this effort across the NNSA tri-lab was the emergence of advanced high performance computing (HPC) architectures based on heterogeneous compute capabilities, including GPU based systems, as part of the national drive toward exascale computing platforms at multiple Department of Energy (DOE) facilities. Developing a multiphysics code capable of meeting the various simulation needs of the NNSA as defined by the current generation of integrated codes (or ICs), initially developed as part of the Accelerated Strategic Computing Initiative (ASCI) program beginning in 1996, and able to scale to the current 100 petaflop class pre-exascale systems, as well the forthcoming exaflop class computers, is a daunting challenge. To accomplish this ambitious goal, LLNL has embraced two key themes: use of high-order numerical methods and a modular approach to code development. The LLNL next generation effort is organized under the Multi-Physics on Advanced Platforms Project (MAPP). A foundational component of MAPP is the Axom computer science (CS) toolkit which provides infrastructure for the development of modular, performance portable, multi-physics application codes. MARBL is a next-generation application code built on the Axom base to address the modeling needs of the high energy density physics (HEDP) community for simulating high-explosive, magnetic or laser driven experiments such as inertial confinement fusion (ICF), pulsed-power magneto-hydrodynamics (MHD), equation of state (EOS) and material strength studies as part of the NNSA’s stockpile stewardship program (SSP).
The objective of this research is to design a portable neutron detector that is lightweight and responds to neutrons with energies of 20 MeV and greater. The instrument will be used as a ‘sniffer’ probe for identifying shielding weaknesses at high-energy particle accelerators and be paired with a plastic scintillator to quantify the high-energy neutron dose equivalent. The sniffer probes are two different sizes and comprised of either cesium iodide or sodium iodide, all doped with thallium, making a total of four detectors. The probes were calibrated to respond only to radiations with 8 MeV electron equivalent energy or higher, vastly reducing detector response from background and lower energy radiations. The sniffer probes were tested using californium- 252 and americium-beryllium sources, followed by high-energy (up to 800 MeV) neutron exposures at the Los Alamos Neutron Science Center Weapons Neutron Research (WNR) facility. The sniffer probes were then used to identify areas around the accelerator facility where elevated levels of high-energy radiation was present and the plastic scintillators were placed in these locations to determine the high-energy neutron dose equivalent. The dose equivalent is calculated by measuring the 12 C(n,2n) 11 C reaction, which has a 20.4 MeV neutron threshold. Two locations behind the WNR facility were identified with the probes and the dose equivalent rates from neutron energies above 20 MeV were calculated to be 1.24 mrem/h behind flightpath 30-left and 1.53 behind flightpath 30-right. Of the four sniffer probes, the 2-inch x 2-inch cesium iodide is the recommended probe for performing high-energy shielding surveys. The larger crystal size yields higher sensitivity at lower count rates and the more dense material increases the likelihood of interaction within the crystal.
This year, the Nuclear Energy Advanced Modeling Simulation program (NEAMS) thermal-hydraulics verification and validation (V&V) work has focused in three areas of Nek5000 V&V-driven development. First, in a close collaborative effort with the U. S. Nuclear Regulatory Commission (NRC) staff, we have continued V&V efforts for the HYMERES-2 project using the OECD/NEA sponsored testing in the PSI PANDA facility. This year’s focus of ANL-NRC collaboration involves Nek5000 setups and validation for a range of problems relevant to and including the HYMERES-2 benchmark from PSI. The primary outcome of this year efforts is a more efficient geometry and inlet modeling simplification after a careful sensitivity study of the inlet profiles and pipe geometries. The resulting modeling choice of a short recycling/fully-developed turbulent inlet is within the experimental uncertainty estimate. This finding simplifies the next step of the cross-V&V HYMERES-2 project. In addition, the ANL team continue to provide assistance to the NRC staff in the form of Nek5000 application support in general and on the use of the HPC platforms of ALCF and INL in particular. This supports the NRC’s assessment of Nek5000 for use with the NRC Blue CRAB code suite. Second, we have implemented and tested more robust model of URANS, namely the k – τ model, a variant of the k-ω model, along with other improvements to RANS Nek5000 modeling in general. Because of its demonstrated robustness and stability, the k – τ model is the only RANS model that has been implemented in the new GPU version of the Nek5000 code, nekRS. Lastly, we report the initial implementation of Jacobian-free Newton Krylov approach to the direct Newton method for steady fluid solvers aimed at acceleration of RANS modeling and at IC improvement for LES campaigns. Also leveraging the Exascale Computing Project (ECP) ANL/CEED & SMR team’s software development effort to support NEAMS problems at large scale of the advanced computing architectures, NekRS, a GPU variant of Nek5000, built on top of kernels from libParanumal using OCCA for portability, has been successfully run on the full system of Summit (4608 nodes, 27648 GPUs).
As we have seen the arithmetic power of hardware architectures growing much faster than the memory bandwidth, there is an urgent need to rethink the way we operate, communicate, and store data. A promising strategy for overcoming the “memory wall” is the decoupling of memory precision from arithmetic precision as proposed in Anzt et al. Such a decoupling of memory precision and arithmetic precision requires an “accessor” that handles the conversion between formats (resp. compression of the data) on the fly in registers. Such an accessor has to be general in the sense that it can handle a wide range of floating point formats and compression strategies, and still allows for further extension. At the same time, is important that the use of the accessor is not a burden for the developer. Therefore, it is important to abstract all implementation details from the user, hide the underlying storage format from the user, and make the accessor cross-platform portable. In this document we present the design of an accessor that aims at fulfilling all these requirements.
In order to support the machine learning co-design needs of ECP applications in current and future DOE HPC hardware, we have developed a generative adversarial network (GAN) proxy application, miniGAN, that has been released through the ECP proxy application suite. The proxy application is representative of the needs of ExaLearn's target applications, specifically the Cosmoflow and ExaGAN cosmology applications and the ExaWind energy application. The proxy application also demonstrates the first use of performance portable kernels within widely-used machine learning frameworks: PyTorch (Facebook) and Horovod (Uber). We provide performance scaling results for similar workloads to ExaGAN and a profile of individual GAN training components.
The 2018 Wood Stove Design Challenge was an international design competition which sought to identify top performing residential wood stoves based on automation. To prepare for the event, the Northeast States for Coordinated Air Use Management (NESCAUM) developed a testing protocol that challenged the stoves by testing them in more field-like conditions—capturing emissions from start-up, reloading a stove, and the use of larger piece sizes. Flue gas emissions were measured using a combination of in-stack and novel dilution sampling methods, as the event occurred on the National Mall in Washington D.C. in a non-laboratory setting. Particulate matter (PM), carbon monoxide (CO), carbon dioxide (CO2), and methane (CH4) were measured from three stoves in real-time. A combustion efficiency was also calculated for each stove in real-time. Measured PM emission rates ranged between 1.8 and 8.0 g/hr for all stoves and operating conditions, with the highest emissions being measured during cold start in all cases. The test average PM emission rates were 2.4, 4.0, and 2.4 g/hr for stoves A, B, and C, respectively. Results showed emissions measured during transient operations, which are often excluded in current certification methods, can be significantly higher than steady-state periods—echoing other studies and highlighting the importance of testing in various operational modes for more realistic emission estimates. Additionally, the stoves had overall CO emission rates of 48.8, 284.8, and 100.8 g/hr, for stoves A, B and C, respectively. Calculated PM and CO emission factors overall were low considering the testing protocol sought to follow more challenging yet realistic practices in terms of fuel loading and operating procedures. The overall estimated combustion efficiency for stoves A, B, and C was 85%, 78%, 76%, respectively and in all cases was lowest during the cold start period.This report details the successfully proposed and assembled instrumentation that was relatively portable for field-site testing and the results of the emissions measurements for the 2018 Wood Stove Design Challenge competition. Overall, the repeatability of each stove was favorable with coefficients of variation (COV) ranging from 7 to 15% for measured PM concentrations.
As part of the Cyclotron Road program, Takachar seeks to investigate the scaling laws and behaviors of a small-scale biomass torrefaction reactor design and its technoeconomic feasibility in various potential applications. Traditionally, biomass is logistically expensive and complicated to utilize in various energy and chemical industries, because it is often loose, wet, bulky, and present in small pockets in dispersed regions. Biomass torrefaction is an approach whereby raw biomass is subject to moderate heat, releasing low-energy molecules. The resultant torrefied biomass becomes energy and volume dense, which reduces the relative transportation cost, and in many cases can even support a local value chain that would otherwise not be possible. However, conventional biomass torrefaction systems are often large-scale, centralized in design, rending them difficult to deploy in many biomass applications in rural/decentralized regions. By building upon prior research at Massachusetts Institute of Technology (MIT), the concept of small-scale, low-cost, and portable biomass torrefaction units was explored further. This investigation has led to the advancement and scale-up of the MIT technology which can be deployed to a much wider spectrum of the biomass market than the conventional technologies.
BEE will have the ability to archive, clone, and re-run workflows that have been previously executed. This will enable reproducibility and portability of scientific workflows both within and between DOE facilities.
We propose the CodeFlow toolchain for Flash-X that realizes the “recipe-to-source” code transformation for Flash-X simulations and that is necessary to achive performance portability. We design a high-level language to express operations of simulations in so-called recipes, which are given as input to the toolchain. The tools of the CodeFlow pipeline include code transformation with tree-based source code representation techniques and code orchestration and generation based on control flow graphs. The generated source code utilizes a new runtime, developed for Flash-X, that orchestrates dynamic and asynchronous data movement and task execution. The functionality of CodeFlow is demonstrated using a hydrodynamic problem with a strong shock.
The United States has a large, domestic source of lithium in geothermal fluids, especially at the Salton Sea region of southern California, where estimates of lithium pass-through at geothermal plants exceed 24,000 metric tons per year, based on 2019 geothermal plant operations. Lithium extraction from geothermal brines offers the potential to provide the United States with a secure, domestic supply of lithium to meet the increasing demands of electric vehicles, grid energy storage, portable electronics, and other end-use applications. Additionally, the use of direct extraction technologies allows for a more sustainable lithium supply relative to current evaporative brine and hardrock mining operations in terms of land use, water use, time to market with lithium products, and carbon intensity of operations. This report is part of an effort to assess geothermal brines as a source of commercial lithium supply for the United States. In this study, the National Renewable Energy Laboratory (NREL) reviews and summarizes public technoeconomic analyses of lithium extraction technologies. The work was coordinated with the Critical Minerals Institute at the Colorado School of Mines who focused on supply chain analysis of lithium.
Characterization of low-level waste (LLW) at Los Alamos National Laboratory (LANL) is primarily performed by the Environmental Protection and Compliance - Environmental Stewardship (EPC-ES) Characterization Services team and/or the Nuclear Process Infrastructure (NPI--9) Nuclear Material Support Services team (hereafter referred to as the gamma spectroscopy teams). The gamma spectroscopy teams use portable high-purity germanium (HPGe) detector systems for the nondestructive assay (NDA) characterization of LLW. Controlled-approved procedures and processes for the use of such systems to assay LLW items are currently in place. Additionally, a number of performance studies have been conducted by the gamma spectroscopy teams to support the efficacy and quality of assay results generated by the established NDA process.
This manual describes the use of the Portable, Extensible Toolkit for Scientific Computation (PETSc) for the numerical solution of partial differential equations and related problems on high-performance computers. PETSc is a suite of data structures and routines that provide the building blocks for the implementation of large-scale application codes on parallel (and serial) computers. PETSc uses the MPI standard for all distributed memory communication.
Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia's Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia's ElectroMagnetic Plasma In Realistic Environments ( EMPIRE ) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia's HPC users.