SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
INL High Performance Computing Overview presentation by Eric Whiting for the National Science Users Facility 2020 Annual Program Review on November 10, 2020; data covering our Collaborative Computing Center (C3), the supercomputers and their data.
The Department of Energy (DOE) ensures America’s security and prosperity by addressing its energy, environmental, and nuclear challenges through transformative science and technology solutions. The DOE’s Office of Science (SC) delivers groundbreaking scientific discoveries and major scientific tools that transform our understanding of nature and advance the energy, economic, and national security of the United States. The SC’s programs advance DOE mission science across a wide range of disciplines and have developed the research infrastructure needed to remain at the forefront of scientific discovery. The DOE SC’s world-class research infrastructure — exemplified by the 28 SC scientific user facilities — provides the research community with premier observational, experimental, computational, and network capabilities. Each user facility is designed to provide unique capabilities to advance core DOE mission science for its sponsor SC program and to stimulate a rich discovery and innovation ecosystem. Research communities gather and flourish around each user facility, bringing together diverse perspectives. A hallmark of many facilities is the large population of students, postdoctoral researchers, and early-career scientists who contribute as full-fledged users. These facility staff and users collaborate over years to devise new approaches to utilizing the user facility’s core capabilities. The history of the SC user facilities has many examples of wildly inventive researchers challenging operational orthodoxy to pioneer new vistas of discovery; for example, the use of the synchrotron X-ray light sources for study of proteins and other large biological molecules. This continual reinvention of the practice of science — as users and staff forge novel approaches expressed in research workflows — unlocks new discoveries and propels scientific progress. Within this research ecosystem, the high-performance computing (HPC) and networking user facilities stewarded by SC’s Advanced Scientific Computing Research (ASCR) program play a dynamic cross-cutting role, enabling complex workflows demanding high performance data, networking, and computing solutions. The DOE SC’s three HPC user facilities and the Energy Sciences Network (ESnet) high-performance research network serve all of the SC’s programs as well as the global research community. Argonne Leadership Computing Facility (ALCF), the National Energy Research Scientific Computing Center (NERSC), and Oak Ridge Leadership Computing Facility (OLCF) conceive, build, and provide access to a range of supercomputing, advanced computing, and large-scale data-infrastructure platforms, while ESnet interconnects DOE SC research infrastructure and enables seamless exchange of scientific data. All four facilities operate testbeds to expand the frontiers of computing and networking research. Together, the ASCR facilities enterprise seeks to understand and meet the needs and requirements across SC and DOE domain science programs and priority efforts, highlighted by the formal requirements reviews (RRs) methodology. In recent years, the research communities around the SC user facilities have begun experimenting with and demanding solutions integrated with HPC and data infrastructure. This rise of integrated-science approaches is documented in many community and high-level government reports. At the dawn of the era of exascale science and the acceleration of artificial intelligence (AI) innovation, there is a broad need for integrated computational, data, and networking solutions. In response to these drivers, DOE has developed a vision for an Integrated Research Infrastructure (IRI): To empower researchers to meld DOE’s world-class research tools, infrastructure, and user facilities seamlessly and securely in novel ways to radically accelerate discovery and innovation.
The National Energy Research Scientific Computing Center (NERSC), as the high-performance computing (HPC) facility for the Department of Energy’s Office of Science, recognizes the essential role of quantum computing in its future mission. In this report, we analyze the NERSC workload and identify materials science, quantum chemistry, and high-energy physics as the science domains and application areas that stand to benefit most from quantum computers. These domains jointly make up over 50% of the current NERSC production workload, which is illustrative of the impact quantum computing could have on NERSC’s mission going forward. We perform an extensive literature review and determine the quantum resources required to solve classically intractable problems within these science domains. This review also shows that the quantum resources required have consistently decreased over time due to algorithmic improvements and a deeper understanding of the problems. At the same time, public technology roadmaps from a collection of ten quantum computing companies predict a dramatic increase in capabilities over the next five to ten years. Our analysis reveals a significant overlap emerging in this time frame between the technological capabilities and the algorithmic requirements in these three scientific domains. We anticipate that the execution time of large-scale quantum workflows will become a major performance parameter and propose a simple metric, the Sustained Quantum System Performance (SQSP), to compare system-level performance and throughput for a heterogeneous workload.
The small interaction cross-section of neutrinos makes experimental neutrino physics particularly responsive to technological advancements. A significant development leveraged by the NOvA experiment is large-scale parallel processing, enabling novel computational approaches to longstanding experimental challenges. Central to managing the resulting high-throughput data is NOvA’s implementation of the Freight Train model, designed for efficient data production and handling.This dissertation details the methodology and execution of the NOvA 2024 3-Flavor Oscillation Analysis, supported by a comprehensive dataset spanning ten years. It emphasizes frequentist results refined through the Feldman-Cousins (FC) technique, specifically addressing confidence interval corrections in parameter estimation. The computational intensity associated with Feldman-Cousins arises from extensive Monte Carlo simulations, which were substantially mitigated through parallel computing on the Perlmutter supercomputer at the National Energy Research Scientific Computing Center (NERSC), employing the MPI framework.To further enhance computational efficiency, an Importance Sampling method is introduced and evaluated, demonstrating significant potential to reduce complexity, particularly in exploring extreme parameter space regions. This thesis presents both the successful application of advanced computational resources and the development of sophisticated statistical techniques, aiming to enhance the precision and scope of neutrino oscillation analyses.
We present our work on developing and training scalable, trustworthy, and energy-efficient predictive graph foundation models (GFMs) using HydraGNN, a multi-headed graph convolutional neural network architecture. HydraGNN expands the boundaries of graph neural network (GNN) computations in both training scale and data diversity. It abstracts over message passing algorithms, allowing both reproduction of and comparison across algorithmic innovations that define nearest-neighbor convolution in GNNs. This work discusses a series of optimizations that have allowed scaling up the GFMs training to tens of thousands of GPUs on datasets consisting of hundreds of millions of graphs. Our GFMs use multitask learning (MTL) to simultaneously learn graph-level and node-level properties of atomistic structures, such as energy and atomic forces. Using over 154 million atomistic structures for training, we illustrate the performance of our approach along with the lessons learned on two state-of-the-art US Department of Energy (US-DOE) supercomputers, namely the Perlmutter petascale system at the National Energy Research Scientific Computing Center and the Frontier exascale system at Oak Ridge Leadership Computing Facility. The HydraGNN architecture enables the GFM to achieve near-linear strong scaling performance using more than 2000 GPUs on Perlmutter and 16,000 GPUs on Frontier.
Functional layered material (LM) architectures will dominate nanomaterials science in this century. We have developed theory, modeling, simulation, and software and data tools that enhance understanding and AI guide synthesis, enable characterization of complex structures, and improve capabilities in the predictive design and growth of LMs. Research at the Center has focused on: Computational synthesis and characterization: AI guided synthesis and experimental synthesis of stacked LMs with tailored properties via optimized chemical vapor deposition (CVD) growth and liquid-phase exfoliation; study defects, edges, grain boundaries, wrinkling of atomic layers and their effects on chemical, mechanical, electrical, and optical properties. Far-from-equilibrium processes: Joint experimental and simulation based probe of electronic processes with NAQMD and ultrafast X-ray free-electron laser (XFEL) and ultrafast electron diffraction (UED) facilities at Stanford. Experimentally validate NAQMD by ultrafast electron diffraction and X-ray spectroscopy studies of structural and excited state dynamics, shape fluctuations, and phonon dynamics. Scalable software: Simulation engines for desktop-to-exascale platforms using low-overhead, linear-scaling QMD algorithms; divide-conquer-recombine NAQMD with electronic excitations; extended-Lagrangian reactive molecular dynamics (RMD), machine learning (ML) based neural-network quantum molecular dynamics (NNQMD), and super-state accelerated molecular dynamics (AMD) and kinetic Monte Carlo codes; thermal and electrical transport software; and design 3D architectures of LMs with desired functionality using scalable software. Distribution of software and data, and training: Software and simulation-experimental data generated within the Center are distributed to the materials science community via Berkeley Materials Project (MP) framework. We have also organized three workshops for software distribution and training at USC (Nov. 2017, Mar. 2018) and Gaithersburg, MD (Nov. 2018) to train researchers, with the last one in focused on underrepresented groups, in collaboration with Howard University which is one of the largest HBCUs. The Center supported a total of 46 personnel and 6 undergraduate students. These include 14 faculty, 11 postdoctoral research associates, 20 graduate research assistants, and mentored 6 undergraduate students. This resulted in the publications of 63 research papers that include 46 publications on Reactive and Quantum Dynamics Simulations, 13 publications on Machine Learning for Quantum Materials, and 4 publications on Quantum Computing.
The mission of the DOE Energy Frontier Research Centers CCDM (2014-2018) and CCM (2018-2021) was to theoretically develop, computationally apply, and experimentally validate electronic structure methods for all materials, with a focus on the complex materials, especially layered and two-dimensional materials, strongly-correlated materials, and liquid water. This was achieved by over 200 published journal articles authored by about 17 senior investigators from physics and chemistry and from theory, computation, and experiment, plus their collaborators. In particular, the Centers confirmed the predictive power of the SCAN (strongly constrained and appropriately normed) density functional, which was constructed to satisfy 17 known exact constraints and several appropriate norms. Without being fitted to real bonded systems, and at a modest computational cost, SCAN correctly predicted covalent, ionic, metallic, hydrogen, and van der Waals bonds in many challenging materials. SCAN gave an improved description of defects in semiconductors, surface properties of metals, seven phases of ice, liquid water, liquid and supercooled silicon, subtle structural distortions in ferroelectrics, formation energies and structural predictions for solids, and critical pressures for structural phase transitions. Perhaps most remarkably, SCAN correctly described some strongly-correlated materials that were previously believed to be beyond the reach of density-functional approximations. SCAN is the only density functional that correctly predicts the band gap closing under chemical doping of the cuprate high-temperature superconducting materials. SCAN also predicts a landscape of competing stripe and magnetic phases in the cuprates. For some materials with some codes, SCAN has convergence problems that are greatly reduced by the CCM-developed r 2 SCAN, without loss of accuracy or rigor. SCAN and r 2 SCAN still make some self-interaction error, which is greatly reduced by the CCDM/ CCM-developed local orbital scaling correction (LOSC). These Centers further proved that the fundamental energy gaps of a solid from an orbital energy difference and from total energy differences are the same for a large class of generalized Kohn-Sham (GKS) functionals, including SCAN and standard hybrid functionals, and that symmetry breaking arises when a dynamic density fluctuation drops to zero frequency. The Centers identified new mechanisms for catalysis in layered materials with ions intercalated between the layers, investigated charge density waves both in model systems and in real layered materials, studied changes of band gap with the number of layers, and explored topological ultrathin films, bent nanoribbons, and defects.
The ATLAS experiment has 18+ years of experience using workload management systems to deploy and develop workflows to process and to simulate data on the distributed computing infrastructure. Simulation, processing and analysis of LHC experiment data require the coordinated work of heterogeneous computing resources. In particular, the ATLAS experiment utilizes the resources of 250 computing centers worldwide, the power of supercomputing centres, and national, academic and commercial cloud computing resources. In this contribution, we present new techniques for cost-effectively improving efficiency introduced in workflow management system software. The evolution from a mesh framework to new types of computing facilities such as cloud and HPCs is described, as well as new types of production and analysis workflows.
This code provides annotation and an interface to 10+ hours of video activities in a high performance datacenter with 20+ different types of anomalous activities. The purpose is to enable machine learning for video surveillance systems in high performance computing centers. This is the first code of this type addressing the space of high performance computing datacenters.
The Mu2e experiment at the Fermilab will search for a coherent neutrinoless conversion of a muon into an electron in the field of an aluminum nucleus with a sensitivity improvement by a factor of 10,000 over existing limits. In this work, the Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq framework as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates several framework components — an artdaq-based DAQ, an art-based event processing, and an EPICS-based detector control system (DCS), and provides a uniform multi-user interface to its components through a web browser. Data streams from the Mu2e tracker and calorimeter are handled by the artdaq-based DAQ and processed by a one-level software trigger implemented within the art framework. Events accepted by the trigger have their data combined, post-trigger, with the separately read out data from the Mu2e Cosmic Ray Veto system. Foundation of the Mu2e DCS, EPICS – an Experimental Physics and Industrial Control System – is an open-source platform for monitoring, controlling, alarming, and archiving. A prototype of the TDAQ and the DCS systems has been built and tested over the last three years at Fermilab’s Feynman Computing Center, and now the production system installation is underway. This work presents their status and focus on the installation plans and procedures for racks, workstations, network switches, gateway computers, DAQ hardware, slow controls implementation, and testing.
March 2022 is the 70th anniversary of the MANIAC, the first fully electronic computer at Los Alamos. The MANIAC was constructed when electronic computing and its associated industry were still in their infancy. Early computer centers like Los Alamos had to build their own systems and invent the methods for using them. This pioneering effort gave Los Alamos and the few places like it an outsized influence on the emerging electronic computing field, shaping the computing technologies and methods we experience today. This report highlights the history of the MANIAC and "recalls the hard work and ingenuity of the people at Los Alamos who helped to invent the modern field of scientific computing, using vacuum tubes and paper tape to build the Lab’s continuing computing legacy."
Monitoring services play a crucial role in the day-to-day operation of distributed computing systems. The ATLAS Experiment at LHC uses the Production and Distributed Analysis workload management system (PanDA WMS), which allows a million computational jobs to run daily at over 170 computing centers of the WLCG and opportunistic resources, utilizing 600k cores simultaneously on average. The BigPanDA monitor is an essential part of the monitoring infrastructure for the ATLAS Experiment that provides a wide range of views, from top-level summaries to a single computational job and its logs. Over the past few years of the PanDA WMS advancement in the ATLAS Experiment, several new components were developed, such as Harvester, iDDS, Data Carousel, and Global Shares. Due to its modular architecture, the BigPanDA monitor naturally grew into a platform where the relevant data from all PanDA WMS components and accompanying services are accumulated and displayed in the form of interactive charts and tables. Moreover the system has been adopted by other experiments beyond HEP. In this paper we describe the evolution of the BigPanDA monitor system, the development of new modules, and the integration process into other experiments.
The Los Alamos National Laboratory’s (LANL) Monte Carlo N-Particle (MCNP)1 transport code version 6.3 (also known as MCNP6.3) has the capability for tracking particles on unstructured mesh (UM) geometry models embedded into constructive solid geometry (CSG) cells. This feature has been developed for performing calculations of complex geometry models because manually creating CSG models is time-consuming and error-prone as the complexities of geometries increase. A UM geometry model is a collection of finite elements representing a solid geometry. The first step of the MCNP UM calculation is using other software packages to create a finite element mesh representation of a solid 3D geometry because the MCNP code cannot be used to generate a UM model. Computer-aided design (CAD) software is typically used to create a solid geometry model, which is later imported into mesh generation software to create a UM model. Some mesh generation software packages may also be used to create solid geometries and thus CAD files are not needed. The MCNP UM feature was originally designed for models generated by the Abaqus/CAE software suite. The MCNP code version 6.0 and later can process UM models formatted as Abaqus input files. Starting with a 6.3 version, the MCNP code can process HDF5 mesh input files. We only focus on the UM models formatted as Abaqus input files in this report since currently no external software can be used to generate HDF5 mesh input files for MCNP UM calculations. The MCNP code version 6.3 can be used to convert the Abaqus mesh input files into the HDF5 mesh input files, but this option is typically used by the MCNP code development team to test the HDF5 mesh input file feature. Several software packages (such as Abaqus, Attila4MC, or Cubit) can be used to create the Abaqus input files for MCNP UM calculations. An MCNP UM calculation using an Abaqus model requires two input file types: MCNP and Abaqus input files. The Abaqus input files needed for MCNP UM calcu lations must have the correct Abaqus syntax and meet the additional requirements by the MCNP code. The MCNP code can process only Abaqus input files that make use of part and assembly definitions, where elements in each part must be grouped into one or more element sets (i.e., elset) using *Elset keyword lines with specified naming formats. The MCNP and Abaqus input files required for MCNP UM simulations must be related; pseudo-cells in an MCNP input file must be constructed from mesh model data from an Abaqus input file. For large complex UM models, it is tedious to manually create MCNP UM input files. The um pre op (unstructured mesh pre operations) program with the -m option can be used to create a skeleton MCNP input file from an Abaqus input file [6]. Since the um pre op program was written in Fortran and was not written for optimized performance, this program is a deprecated feature in the MCNP code version 6.3 and may be removed in the next release of the code. To improve calculation flow of multiphysics calculations, a Python3 code called write mcnp um input has been developed to generate an MCNP input file instead of using the um_pre_op -m option. This Python code was initially released to the public in 2020. We have updated this Python code for MCNP6.3 and it was used to generate the MCNP input files used to verify the MCNP6.3 code. The write_mcnp_um_input code is included with the MCNP6.3 code package which will be released to the public through the Radiation Safety Information Computational Center (RSICC) at Oak Ridge National Laboratory. This report is a revision of LA-UR-20-27139 report.
Scientific computing centers increasingly face workloads with diverse urgency requirements, driven by applications that demand rapid or even immediate execution. Appropriately configured scheduling policies can significantly improve both user satisfaction and overall cluster utilization. In this work, we present a systematic analysis of scheduler configurations under scenarios where a fraction of jobs have urgent computing needs. We evaluate multiple job scheduling simulators, develop a lightweight job-submission emulation framework, and create tools to analyze and visualize the resulting scheduling data. Our study identifies key trade-offs between responsiveness, fairness, and efficiency, and offers a set of practical scheduling configurations (particularly for Slurm) that can be tailored to HPC environments supporting mixed-urgency workloads.
This errata is to correct the acknowledgment in the original published article. G.B. (partial support), V.-A.G., M.-T.N., and R.R. were supported by the U.S. Department of Energy, Office of Science, Basic Energy Sciences, Chemical Sciences, Geosciences, and Biosciences Division. Computer resources were provided by Research Computing at Pacific Northwest National Laboratory and the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility operated under Contract No. DE-AC02-05CH11231. PNNL is operated by Battelle for the U.S. Department of Energy under Contract DE-AC05-76RL01830. G.B. acknowledges the EPSRC for partial support in PhD funding. The authors acknowledge the use of the IRIDIS High Performance Computing Facility (IRIDIS 5) and associated support services at the University of Southampton in the completion of this work. We are grateful to the UK Materials and Molecular Modeling Hub (Thomas HPC) for computational resources, which is partially funded by EPSRC (EP/P020194/1).
We report a Kohn–Sham density functional theory calculation of a system with more than 200 000 atoms and 800 000 electrons using a real-space high-order finite-difference method to investigate the electronic structure of large spherical silicon nanoclusters. Our system of choice was a 20 nm large spherical nanocluster with 202 617 silicon atoms and 13 836 hydrogen atoms used to passivate the dangling surface bonds. To speed up the convergence of the eigenspace, we utilized Chebyshev-filtered subspace iteration, and for sparse matrix–vector multiplications, we used blockwise Hilbert space-filling curves, implemented in the PARSEC code. For this calculation, we also replaced our orthonormalization + Rayleigh–Ritz step with a generalized eigenvalue problem step. We utilized all of the 8192 nodes (458 752 processors) on the Frontera machine at the Texas Advanced Computing Center. We achieved two Chebyshev-filtered subspace iterations, yielding a good approximation of the electronic density of states. Our work pushes the limits on the capabilities of the current electronic structure solvers to nearly 106 electrons and demonstrates the potential of the real-space approach to efficiently parallelize large calculations on modern high-performance computing platforms.
Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.
Using a real-space high-order finite-difference approach, we investigate the electronic structure of large spherical silicon nanoclusters. Within Kohn-Sham density functional theory and using pseudopotentials, we report the self-consistent field convergence of a system with over 100000 atoms: a Si 107,641 H 9,084 nanocluster with a diameter of 16 nm. Our approach uses Chebyshev-filtered subspace iteration to speed up the convergence of the eigenspace, and blockwise Hilbert space-filling curves to speed up sparse matrix-vector multiplications, all of which are implemented in the parsec code. For the largest system, we utilized 2048 nodes (114 688 cores) on the Frontera machine in the Texas Advanced Computing Center. Our quantitative analysis of the electronic structure shows how it gradually approaches its bulk counterpart as a function of nanocluster size. The band gap is enlarged due to quantum confinement in nanoclusters, but decreases as the system size increases, as expected. In conclusion, our work serves as a proof of concept for the capacity of the real-space approach in efficiently parallelizing very large calculations using high-performance computer platforms, which can straightforwardly be replicated in other systems with more than 10 5 atoms.