Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “HPC Utilization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Adaptive elasticity policies for staging-based in situ visualization

In situ processing aims to alleviate the growing gap between computation and I/O capabilities by performing data processing close to the data source. In situ processing is widely used to process data generated by multiple data sources, including observation data from edge devices or scientific observational facilities and the simulation data generated by scientific computation on a high-performance computing (HPC) platform. For a scientific workflow that is run on an HPC platform and composed of a simulation program and an in situ data analytics or visualization (abbreviated as ana/vis) task, there is an implicit assumption that the computing resources assigned to the workflow keep static during the workflow execution. However, with the converging trend between the HPC and cloud computing platform, running the in situ ana/vis task in an elastic way is promising to decrease its overhead and improve its resource utilization rate. Resource elasticity represents the ability to change resource configurations such as the number of computing nodes/processes during workflow execution. An elastic job may dynamically adjust resource configurations; it may use a few resources at the beginning and more resources toward the end of the job when interesting data appear. However, it is hard to predict a priori how many computing nodes/processes need to be added/removed during the workflow execution to adapt to changing workflow needs. How to efficiently guide elasticity operations, such as growing or shrinking the number of processes used for in situ analysis during workflow execution, is an open-ended research question. In this article, we present adaptive elasticity policies that adopt workflow runtime information collected during workflow execution to predict how to trigger the addition/removal of processes in order to minimize in situ processing overhead. Taking in situ visualization tasks as an example, we integrate the presented elasticity policies into a staging-based elastic workflow and evaluate its efficiency in multiple elasticity scenarios. Compared with the situation without elasticity or with a static elasticity policy that uses a fixed number of processes for each rescaling operation, the adaptive elasticity policy can save overhead in finding a proper resource configuration and improve resource utilization efficiency. Furthermore, one experiment illustrates that the adaptive elasticity policy saves 41% of core-hours compared with the situation without the resource elasticity.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

DAAP (Data Analytics Application Profiling)

The Data Analytics Application Profiling (DAAP) utility is a software program comprised of a library and scripts to provide basic instrumenting of parallel scientific software applications while they are running on an HPC cluster, and transmit this data over a secure connection as individual records to aggregators. When used in conjunction with additional open source software components (Telegraf, FluentD, and RabbitMQ), the data collected by DAAP can be transmitted off-cluster to an analytics system of the user’s choice for further analysis.

Shereda, Charles↗
DOE OSTI2023

Computational Requirements in Clean Energy and Manufacturing: Summary report of the virtual workshop held on June 28-29, 2021

On June 28–29, 2021, the US Department of Energy’s (DOE’s) Advanced Scientific Computing Research (ASCR) program in the Office of Science convened a workshop with the Energy Efficiency and Renew able Energy (EERE) program offices to assess the future need for advanced computing resources in the areas of clean energy and advanced manufacturing. In part, this discussion served as an update to earlier workshops and townhalls. ASCR is guided by DOE mission needs as it develops research programs, computers, and networks at the leading edge of technologies. As the exascale computing era dawns, technology changes are creating new opportunities for those who must use high-performance computing (HPC) and data systems effectively. The ASCR computing facilities are augmenting their strategy to adapt to changing science needs and emerging technologies and to leverage the utility of exascale computing across the federal government.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

2020 Budget Request for the DOE Computational Science Graduate Fellowship (CSGF) Grant

The Department of Energy Computational Science Graduate Fellowship (DOE CSGF) is essential for addressing the increasingly complex national workforce demands stemming from the growth of computational science and engineering challenges. Computational science and engineering (CSE) takes a multidisciplinary approach that utilizes scientific computing to tackle practical problems and provide technical tools across the spectrum of scientific discovery. The DOE CSGF specifically highlights high-performance computing (HPC) as a critical enabling technology in CSE, driving advancements in science and engineering that are vital to both the DOE and the broader economy. Over the past half-century, HPC has been an essential tool for DOE’s success. During this period, important missions, such as nuclear stockpile stewardship, have turned to HPC as an essential technology. Entire science disciplines have been transformed through the augmentation of scientific observation via HPC. At government laboratories, academic institutions, and in industry, DOE CSGF alumni are helping push traditional HPC boundaries while contributing to discoveries in high-energy physics, quantum information systems, fusion-reactor design, machine learning, additive manufacturing, nano materials for next-generation batteries and transistors, and advanced nuclear reactor modeling. In addition, HPC is used to address national health needs that will eventually point to cures both by helping cancer researchers manage and analyze huge troves of data, by simulating biological mechanisms, and by accelerating drug development. A 2023 report from the ASCAC Subcommittee on American Competitiveness and Innovation to the ASCR office, “Can the United States Maintain Its Leadership in High-Performance Computing?” says of the Program, “The CSGF program provides a barometer for disciplines that will be of interest to future DOE computing. Computational biology, machine learning, and quantum computing are among the subjects that began to swell in the ranks of CSGF applicants before the labs were hiring as high a percentage of employees in these categories.” The explosion of scientific and technological data has heightened the demand for advanced high-performance computing (HPC) to transform these data into meaningful scientific insights. As access to vast amounts of data increases, the fields of Machine Learning and Artificial Intelligence are experiencing a resurgence, enhancing the established practices of computational modeling and simulation. In its September 2020 subcommittee report on "AI/ML, Data Intensive Science, and High-Performance Computing," the DOE Advanced Scientific Computing Advisory Committee (ASCAC) specifically called for a fellowship program to train computational and data scientists to address exascale and data-intensive computing challenges. This integration of empirical and theoretical modeling will increasingly guide federal policymakers in making decisions that impact American society and future generations. It demands a workforce of highly skilled and intellectually agile computational scientists capable of navigating the rapid advancements in scientific computing within the DOE National Laboratory research environment. The DOE CSGF program has consistently addressed this critical need.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Mastering HPC Runtime Prediction: From Observing Patterns to a Methodological Approach

The continual expansion of high-performance computing (HPC) brings with it an increasing need for efficiency. Heavy investment in energy, hardware, and software infrastructure to support peta- and exascale computing requires the optimization of existing systems and, wherever possible, the discernment and adoption of best-practices towards these goals. Such is the case for runtime prediction. When a job is submitted to an HPC system, an estimate of its runtime is provided by the user in the form of "requested wallclock". Error in this user-provided estimate can lead to jobs being prematurely killed by the scheduler, increased wait time on the queue, and decreased system utilization. More than fifteen years of research has been directed at mitigating these effects by using data-driven runtime predictions. Codified here is a set of commonalities and insights emerging from this body of work, which we present as recommendations and best practices. These practices are combined into a methodological approach described and evaluated on an 11-million-job dataset from the National Renewable Energy Laboratory's petascale HPC system, Eagle. This dataset and the accompanying codebase have been released to the public domain for the benefit of the wider HPC research community.

high performance computing↗
DOE OSTI2023

Mastering HPC Runtime Prediction: From Observing Patterns to a Methodological Approach: Preprint

The continual expansion of high-performance computing (HPC) brings with it an increasing need for efficiency. Heavy investment in energy, hardware, and software infrastructure to support peta- and exascale computing requires the optimization of existing systems and, wherever possible, the discernment and adoption of best-practices towards these goals. Such is the case for runtime prediction. When a job is submitted to an HPC system, an estimate of its runtime is provided by the user in the form of "requested wallclock''. Error in this user-provided estimate can lead to jobs being prematurely killed by the scheduler, increased wait time on the queue, and decreased system utilization. More than fifteen years of research has been directed at mitigating these effects by using data-driven runtime predictions. Codified here is a set of commonalities and insights emerging from this body of work, which we present as recommendations and best practices. These practices are combined into a methodological approach described and evaluated on an 11-million-job dataset from the National Renewable Energy Laboratory's petascale HPC system, Eagle. This dataset and the accompanying codebase have been released to the public domain for the benefit of the wider HPC research community.

high performance computing↗
DOE OSTI2022

Distributed deep learning training using silicon photonic switched architectures

The scaling trends of deep learning models and distributed training workloads are challenging network capacities in today’s datacenters and high-performance computing (HPC) systems. We propose a system architecture that leverages silicon photonic (SiP) switch-enabled server regrouping using bandwidth steering to tackle the challenges and accelerate distributed deep learning training. In addition, our proposed system architecture utilizes a highly integrated operating system-based SiP switch control scheme to reduce implementation complexity. To demonstrate the feasibility of our proposal, we built an experimental testbed with a SiP switch-enabled reconfigurable fat tree topology and evaluated the network performance of distributed ring all-reduce and parameter server workloads. The experimental results show up to 3.6× improvements over the static non-reconfigurable fat tree. Our large-scale simulation results show that server regrouping can deliver up to 2.3× flow throughput improvement for a 2× tapered fat tree and a further 11% improvement when higher-layer bandwidth steering is employed. The collective results show the potential of integrating SiP switches into datacenters and HPC systems to accelerate distributed deep learning training.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Batched Sparse Linear Algebra (Final Report for Subcontract B648960)

This report finalizes design specifications for developing batched kernels for small tensor operations for unassembled matrix-free iterative solvers, batched solvers for partially assembled operators, and batched solvers with support for various sparse formats. The outcome of the project milestones is a set of interfaces to Batched Sparse LA solvers running on hardware accelerators for use in ECP Libraries and Applications. It is part of the development of sparse batched kernels, solvers/preconditioners as well as creating interoperability in xSDK libraries with sparse and dense batched functions to benefit ECP applications. The participants included representatives from ECP libraries (not limited to the xSDK project), applications, and vendors (AMD, Intel, and NVIDIA). Batched sparse linear algebra solvers form the new frontier for algorithmic development and performance engineering. Many applications (ECP and non-ECP alike) require simultaneous solutions of small linear systems of equations that are structurally sparse. To move towards high hardware utilization, it is important to provide these applications with appropriate interfaces to efficient batched sparse solvers running on modern hardware accelerators. We present interface designs in use by HPC software libraries supporting batched sparse linear algebra and the development of sparse batched kernel codes for solvers and preconditioners. We also address the potential interoperability opportunities to keep the software portable between the major hardware accelerators from AMD, Intel, and NVIDIA. The presented interface specifications includes batched band, sparse iterative, and sparse direct solvers. This report summarizes progress in Kokkos Kernels and the xSDK libraries MAGMA, Ginkgo, hypre, SUNDIALS, and SuperLU_dist.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

The Virtual Blast Furnace - An Integrated High Performance Computing Modeling, Simulation, and Visualization Capability for Steel Manufacturing (Final Report)

Many manufacturing industries require substantial capital and utilize energy intensive processes that involve complex phenomena. One example of such an industry is the steel industry, which is the fourth largest energy consuming industry in the U.S. By harnessing the power of High-Performance Computing (HPC) to enhance current simulation and visualization methods in the steel industry, it should be possible to increase resolution and/or decrease time of these methods by a factor of 1000. In this way, information can be obtained in a time frame that is useful for making business and engineering decisions, optimizing manufacturing processes and, ultimately, improving the completeness of U.S. industries. For example, if coke usage in blast furnaces were optimized such that the average coke rate was reduced from 797 lb/net tonne of hot metal (NTHM) to 604 lb/NTHM, costs could be reduced by $894 million/year. Additionally, members of the steel industry need the flexibility to efficiently operate blast furnaces at a range of production rates in order to meet fluctuating market demands. Large scale parameter studies can be utilized to discover workable operating parameters at a range of production rates.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Integrating and Characterizing HPC Task Runtime Systems for hybrid AI-HPC workloads

Scientific workflows increasingly involve both HPC and machine-learning tasks, combining MPI-based simulations, training, and inference in a single execution. Launchers such as Slurm’s srun constrain concurrency and throughput, making them unsuitable for dynamic and heterogeneous workloads. We present a performance study of RADICAL-Pilot (RP) integrated with Flux and Dragon, two complementary runtime systems that enable hierarchical resource management and high-throughput function execution. Using synthetic and production-scale workloads on Frontier, we characterize the task execution properties of RP across runtime configurations. RP+Flux sustains up to 930 tasks/s, and RP+Flux+Dragon exceeds 1,500 tasks/s with over 99.6% utilization. In contrast, srun peaks at 152 tasks/s and degrades with scale, with utilization below 50%. For IMPECCABLE.v2 drug discovery campaign, RP+Flux reduces makespan by 30–60% relative to srun/Slurm and increases throughput more than four times on up to 1,024. These results demonstrate hybrid runtime integration in RP as a scalable approach for hybrid AI-HPC workloads.

HPC-AI↗
DOE OSTI2023

HQ-Sim: High-performance State Vector Simulation of Quantum Circuits on Heterogeneous HPC Systems

Quantum circuit simulations are applied in more and more circum- stances as the quantum computing community becomes broader. It helps researchers to evaluate quantum algorithms and relieve the burden of limited quantum computing resources. However, most of the state-of-the-art quantum simulators utilize either CPU or GPU to store and calculate the state vector, which results in resource starvation. Moreover, the maximum number of qubits supported by the simulator is bounded by the memory, since the memory utilization increases exponentially with the number of qubits. In this study, we leverage Heterogeneous computing to utilize both CPU and GPU to store and update state vectors. We also integrate lossy data compression to reduce memory requirements. Specifically, we develop a heterogeneous framework that has a dynamic scheduler to fully utilize the computing resources. We apply lossy compression to chunked state vector to make the maximum number of qubits higher than the regular simulators, the compression also benefits the data movement between CPU and GPU.

Zhang, Boyuan↗
DOE OSTI2024

Advanced Computing is at the Forefront of a New “Moonshot” Revolutionizing the North American Power Grid

In the 50+ years since the first humans landed on the moon, computing has grown at breakneck speed. We are faced with another challenge that is just as daunting, and just as important to overcome-modernizing the North American electric power grid-and high-performance computing (HPC) systems with specialized software will be an important element in rising to this challenge. We describe at a high level how software developed in the ExaSGD project addresses this "moonshot" goal by utilizing exascale computing and a novel high performance solver software stack to support the mission of decarbonizing power grid operations in an environment of uncertain weather and climate. To reach the exascale benchmark the team has made a number of first-of-their-kind innovations, including novel method for stochastic optimization, fine grained parallel methods for modeling power systems, and GPU resident sparse numerical linear solvers.

17 WIND ENERGY↗
DOE OSTI2022

Management and Storage of Scientific Data

Scientific discoveries rely heavily on efficient access, search, and management of massive data sets. Data management technologies have, for decades, provided foundational capabilities for scientific computing. Just as storage, input/output (I/O), and data management have been fundamental to simulation-based science for many years, so too are capable data-management technologies key to the success of today’s scientific workflows utilizing data intensive and machine learning (ML) techniques. The Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) program has invested broadly in data-management research focused on high-performance computing (HPC) systems, from parallel file systems that store data to application software that makes these systems more productive. Still, advances in technology combined with growing diversity of supported science strongly motivate continued investment in this area. In January 2022, ASCR convened a workshop to identify priority research directions in the area of data management for high-performance and scientific computing. Attendees were challenged to identify promising approaches that would support the breadth of the DOE mission, including the explosion of artificial intelligence (AI) uses and the growing needs of experimental and observational science. Technological and science drivers were identified and considered as they relate to key aspects of data management such as interfaces, architectural design, and FAIR principles (Findable, Accessible, Interoperable, and Reusable). The thoughts of the workshop participants were distilled into a set of four priority research directions with the potential for high impact on DOE science. These research directions are summarized in the following pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2022

PV Hosting Capacity Estimation: Experiences with Scalable Framework; Preprint

Hosting capacity is an indication of the amount of photovoltaics (PV) can be hosted in a distribution system. This paper presents a framework for estimating distributed PV hosting capacity at scale. We first analyze the key challenges of performing relevant large scale simulation including computational and modeling challenges. Then, we develop two python-based software tools in order to conduct repeatable distribution analyses: Distribution Integration Solution Cost Options (DISCO) for configuring and analyzing simulations, and JADE for parallelizing jobs on HPC clusters. A case study of hosting capacity estimation for SMART-DS SFO 2000+ synthetic feeders is used to demonstrate the capability of the developed framework and tools. The framework and tools can help utilities assess the overall hosting capacity of their service territory, through which the overall upgrade cost can be better planned in order to integrate more PV in the future.

distributed energy resources↗
DOE OSTI2025

Creating Apptainer Workflows with Docker-Compose-like Utilities

Creating Apptainer Workflows with Docker-Compose-like Utilities In this presentation, I will explore the utilization of a tool called process-compose, inspired by docker-compose, to create Apptainer-based services. This approach allows for easy deployment and management of fully containerized applications on High Performance Computing (HPC) systems without requiring elevated privileges. Benefits to the Ecosystem: By incorporating process-compose and Apptainer, I aim to address several key challenges in the HPC ecosystem: Simplified Workflow Management: Process-compose provides a user-friendly interface for defining and managing complex containerized application services, reducing the setup time and lowering the barrier to entry for new users. Enhanced Portability: Apptainer ensures that containerized applications can run consistently across different HPC environments, promoting greater portability and reducing compatibility issues. Process-compose is also a single binary that does not need to be installed by admin level users. Community Driven Solutions: This approach aligns with the goals of the High Performance Software Foundation (HPSF) to advance community-driven solutions. By sharing our experiences and insights, I hope to foster collaboration and innovation within the HPC community. Increased Productivity: The combination of process-compose and Apptainer streamlines the serve deployment process, allowing researchers and developers to focus more on their scientific work rather than the intricacies of system or service administration. Through this presentation, attendees will gain valuable insights into the practical implementation of containerized workflows on HPC systems, learn about the benefits of using process-compose and Apptainer, and understand how these tools can contribute to a more efficient HPC ecosystem.

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2024

High-Fidelity and High-Performance Computational Simulations for Rapid Design Optimization of Sulfur Thermal Energy Storage (CRADA Report)

NREL and Element 16 collaborated on sulfur thermal energy storage modeling using NREL’s high performance computing (HPC) resources to assist its application in industrial processes. Industrial process heat (IPH) accounts for ~70% of US manufacturing energy use and is primarily produced by fossil fuel combustion. Approximately, 1500 TWht (~60% Terawatt hour thermal) of IPH demand is in the temperature range of 100-300°C. Industrial applications in this temperature range include drying, hydrothermal processing, thermal enhanced oil recovery, food and beverage, bioethanol production, etc. Cost-effective thermal energy storage (TES) that increases the utilization of waste and renewable heat (solar, geothermal, etc.) could provide significant energy savings and reliable heat sources, decrease emissions, and increase US manufacturing competitiveness through reductions in fuel consumption. This HPC4EI project facilitated Element 16’s development of low-cost and high-impact molten sulfur TES for dispatchable IPH. The development of a high-fidelity model validated by experimental data and HPC simulations enabled the successful resolution of the complex interplay between fluid dynamics and heat transfer processes during transient operation of sulfur TES, overcoming the numerical challenges posed by the non-linear temperature-dependent physical properties of sulfur. The project helped accelerate Element 16’s molten sulfur TES product design and support its broad applications.

25 ENERGY STORAGE↗
DOE OSTI2024

Integrating Quantum Computing with High-Performance Computing: A Streamlined Approach

In recent years, quantum computing has demon-strated the potential to revolutionize specific algorithms and applications by solving problems exponentially faster than classical computers. However, its widespread adoption for general computing remains a future prospect. This paper discusses the integration of quantum computing within High-Performance Computing (HPC) environments, focusing on a resource management framework designed to streamline quantum simulators' use and enhance runtime performance and efficiency. The proposed framework facilitates hybrid applications' transition from simulation backends to real quantum hardware, optimizing resource utilization and providing a flexible infrastructure for developing and testing quantum algorithms.

Shehata, Amir↗
DOE OSTI2025

I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey

Growing interest in Artificial Intelligence (AI) has resulted in a surge in demand for faster methods of Machine Learning (ML) model training and inference. This demand for speed has prompted the use of high performance computing (HPC) systems that excel in managing distributed workloads. Because data is the main fuel for AI applications, the performance of the storage and I/O subsystem of HPC systems is critical. In the past, HPC applications accessed large portions of data written by simulations or experiments or ingested data for visualizations or analysis tasks. ML workloads perform small reads spread across a large number of random files. This shift of I/O access patterns poses several challenges to modern parallel storage systems. In this paper, we survey I/O in ML applications on HPC systems, and target literature within a 6-year time window from 2019 to 2024. We define the scope of the survey, provide an overview of the common phases of ML, review available profilers and benchmarks, examine the I/O patterns encountered during offline data preparation, training, and inference, and explore I/O optimizations utilized in modern ML frameworks and proposed in recent literature. Lastly, we seek to expose research gaps that could spawn further R&D.

97 MATHEMATICS AND COMPUTING↗