Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “High performance Computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

PandAna: A Python Analysis Framework for Scalable High Performance Computing in High Energy Physics

Modern experiments in high energy physics analyze millions of events recorded in particle detectors to select the events of interest and make measurements of physics parameters. These data can often be stored as tabular data in files with detector information and reconstructed quantities. Current techniques for event selection in these files lack the scalability needed for high performance computing environments. We describe our work to develop a high energy physics analysis framework suitable for high performance computing. This new framework utilizes modern tools for reading files and implicit data parallelism. Framework users analyze tabular data using standard, easy-to-use data analysis techniques in Python while the framework handles the file manipulations and parallelism without the user needing advanced experience in parallel programming. In future versions, we hope to provide a framework that can be utilized on a personal computer or a high performance computing cluster with little change to the user code.

Groh, Micah↗
DOE OSTI2022

Accelerating Simulation for High-Fidelity PV Inverter System Reliability Assessment with High-Performance Computing

The overall cost of photovoltaic (PV) systems has shown a downward trend during the last decade; however, PV inverter failures account for the highest cost of operation and maintenance. To address this, reliability tools with powerful computation and better accuracy are required for the lifetime prediction and degradation evaluation of PV inverters. This paper proposes an event-driven parallel computing-based simulator. The proposed simulator applies high-performance computing techniques and other accessory optimization techniques-including cluster merging, adaptive model updates, and steady-state identification-to make reliability assessments for PV inverters under given input mission profiles and operating conditions with high efficiency and high fidelity. The main idea of the simulator and its workflow are introduced. Then, a demo PV inverter system simulator is implemented, and the speedup of the total simulations of the switching model reaches 123.03 times.

high-performance computing↗
DOE OSTI2024

Frontiers in Scientific Workflows: Pervasive Integration With High-Performance Computing

Herein we address the increasing complexity of scientific workflows in the context of high-performance computing (HPC) and their associated need for robust, adaptable, and flexible computational support systems. We explore five key trends as well as future challenges and opportunities for scientific workflows and HPC technologies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

INL High Performance Computing Overview

INL High Performance Computing Overview presentation by Eric Whiting for the National Science Users Facility 2020 Annual Program Review on November 10, 2020; data covering our Collaborative Computing Center (C3), the supercomputers and their data.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2025

Toward a persistent event-streaming system for high-performance computing applications

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.

HPC↗
DOE OSTI2026

Queue wait time prediction in high performance computing (HPC) systems

High Performance Computing (HPC) systems are critical enablers for groundbreaking scientific research across various domains. Efficient resource allocation, facilitated by job scheduling, is paramount for maximizing the utilization of HPC systems. However, the variability in wait times for queued jobs poses challenges for users, necessitating accurate job wait time estimation. This paper explores the influence of job characteristics, including job size (the number of nodes requested and walltime), the queue to which the job is submitted and other resource requirements, on job wait times in leadership-class HPC systems. Focusing on the Theta Cray XC40 and Polaris machines at Argonne National Laboratory, the study evaluates the performance of different supervised learning algorithms in predicting job wait times. It also evaluates the impact of data preprocessing, including outlier detection, Principal Component Analysis (PCA), and feature selection, on the performance of wait time prediction models. The findings reveal insights into the relationship between job characteristics and wait times, offering a foundation for optimizing resource allocation and enhancing user experience. The methodologies and tools developed in this study are adaptable to other leadership-class HPC systems, providing a valuable contribution to the broader HPC community aiming to improve job scheduling efficiency and user satisfaction.

Okafor, Nwamaka↗
DOE OSTI2023

Auto-HPCnet: An Automatic Framework to Build Neural Network-based Surrogate for High-Performance Computing Applications

High-performance computing communities are increasingly adopt- ing Neural Networks (NN) as surrogate models in their applications to generate scientific insights. Replacing an execution phase in the application with NN models can bring significant performance im- provement. However, there is a lack of tools that can help domain scientists automatically apply NN-based surrogate models to HPC applications. We introduce a framework, named Auto-HPCnet, to democratize the usage of NN-based surrogates. Auto-HPCnet is the first end-to-end framework that makes past proposals for the NN-based surrogate model practical and disciplined. Auto-HPCnet introduces a workflow to address unique challenges when apply- ing the approximation, such as feature acquisition and meeting the application-specific constraint on the quality of final computation outcome. We show that Auto-HPCnet can leverage NN for a set of HPC applications and achieve 5.50× speedup on average (up to 16.8× speedup and with data preparation cost included) while meeting the application-specific constraint on the final computation quality.

Dong, Wenqian↗
DOE OSTI2022

Transforming Energy Through Computational Excellence: High-Performance Computing for Energy Innovation

The challenges associated with energy efficiency of manufacturing and advanced materials often cannot be addressed through experimentation alone, whether because of scale, complexity, or practicality. High-performance computing (HPC) enables fast tackling of these challenges in the manufacturing sector - vital to achieving net-zero carbon emissions by 2050. The National Renewable Energy Laboratory (NREL) and industry partners leverage HPC to apply advanced modeling, simulation, and data analysis to improve manufacturing efficiency, explore new materials for energy applications, and develop technologies to manage carbon across the life cycle. From improving additive manufacturing processes to increasing the energy efficiency of jet-engine components, advanced computing can help manage emissions produced by manufacturing in a wide variety of ways.

advanced materials↗
DOE OSTI2024

Integrating Quantum Computing with High-Performance Computing: A Streamlined Approach

In recent years, quantum computing has demon-strated the potential to revolutionize specific algorithms and applications by solving problems exponentially faster than classical computers. However, its widespread adoption for general computing remains a future prospect. This paper discusses the integration of quantum computing within High-Performance Computing (HPC) environments, focusing on a resource management framework designed to streamline quantum simulators' use and enhance runtime performance and efficiency. The proposed framework facilitates hybrid applications' transition from simulation backends to real quantum hardware, optimizing resource utilization and providing a flexible infrastructure for developing and testing quantum algorithms.

Shehata, Amir↗
DOE OSTI2020

PLEXUS: A Pattern-Oriented Runtime System Architecture for Resilient Extreme-Scale High-Performance Computing Systems

For high-performance computing (HPC) system designers and users, meeting the myriad challenges of next-generation exascale supercomputing systems requires rethinking their approach to application and system software design. Among these challenges, providing resiliency and stability to the scientific applications in the presence of high fault rates requires new approaches to software architecture and design. As HPC systems become increasingly complex, they require intricate solutions for detection and mitigation for various modes of faults and errors that occur in these large-scale systems, as well as solutions for failure recovery. These resiliency solutions often interact with and affect other system properties, including application scalability, power and energy efficiency. Therefore, resilience solutions for HPC systems must be thoughtfully engineered and deployed.In previous work, we developed the concept of resilience design patterns, which consist of templated solutions based on well-established techniques for detection, mitigation and recovery. In this paper, we use these patterns as the foundation to propose new approaches to designing runtime systems for HPC systems. The instantiation of these patterns within a runtime system enables flexible and adaptable end-to-end resiliency solutions for HPC environments. The paper describes the architecture of the runtime system, named Plexus, and the strategies for dynamically composing and adapting pattern instances under runtime control. This runtime-based approach enables actively balancing the cost-benefit trade-off between performance overhead and protection coverage of the resilience solutions. Based on a prototype implementation of PLEXUS, we demonstrate the resiliency and performance gains achieved by the pattern-based runtime system for a parallel linear solver application.

Hukerikar, Saurabh↗
DOE OSTI2024

Converged Computing: A Best of Both Worlds of High-Performance Computing and Cloud

Collaboration between Cloud and High Performance Computing (HPC) communities has accelerated in the last half decade. A common goal to run batch workloads combined with a desire for reproducibility, automation, and optimization has led to successful projects that range from container technologies to workload management and security. This span of current and future work defines a novel “Converged Computing” paradigm that aims to combine the best of both worlds, both from a technological and cultural standpoint. Furthermore, in this Special Issue, we review common themes in the space, showcasing current work and encouraging a continued effort toward innovative ideas that will enable the next generations of scientific discovery.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Data Center Immersion Cooling: A Case Study and Summary of High-Performance Computing Cooling Technologies

The future of High Performance Computing (HPC) is carved out for all high-performance facilities. As computer processing increases exponentially, especially with the demand for AI training, more speed will equate to more computing density and ultimately more power draw and resource demand. Sandia National Laboratories has not seen or known of anything that would lead us to think this might change in the next 5 to 10 years.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

DRAS: Deep Reinforcement Learning for Cluster Scheduling in High Performance Computing

Cluster schedulers are crucial in high-performance computing (HPC). They determine when and which user jobs should be allocated to available system resources. Existing cluster scheduling heuristics are developed by human experts based on their experience with specific HPC systems and workloads. However, the increasing complexity of computing systems and the highly dynamic nature of application workloads have placed tremendous burden on manually designed and tuned scheduling heuristics. More aggressive optimization and automation are needed for cluster scheduling in HPC. In this work, we present an automated HPC scheduling agent named DRAS (Deep Reinforcement Agent for Scheduling) by leveraging deep reinforcement learning. DRAS is built on a hierarchical neural network incorporating special HPC scheduling features such as resource reservation and backfilling. An efficient training strategy is presented to enable DRAS to rapidly learn the target environment. Once being provided a specific scheduling objective given by the system manager, DRAS automatically learns to improve its policy through interaction with the scheduling environment and dynamically adjusts its policy as workload changes. We implement DRAS into a HPC scheduling platform called CQGym. CQGym provides a common platform allowing users to flexibly evaluate DRAS and other scheduling methods such as heuristic and optimization methods. Furthermore, the experiments using CQGym with different production workloads demonstrate that DRAS outperforms the existing heuristic and optimization approaches by up to 50%.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Quantifying the Impact of Advanced Web Platforms on High Performance Computing Usage

The deployment of Science Gateways for High Performance Computing (HPC) systems can alter long-accepted usage patterns on supercomputing systems in positive ways as an ever-increasing number of users migrate their workflows to HPC systems. Idaho National Laboratory (INL) has deployed two separate advanced web platforms, Open OnDemand and NICE DCV, for integration with HPC resources to improve web accessibility for HPC users. Researchers conducted a multi-year study on how HPC usage pat- terns changed in the presence of these platforms. This work reports the results of that study and quantifies the observed impacts, including adoption by visualization and Jupyter Notebook/Lab users, decreased job submission friction, rapid uptake of HPC by Windows users, and increased overall system utilization. The most significant impacts were observed from the deployment of Open OnDemand, and this work also identifies some best practices for Open OnDemand deployment for HPC datacenters.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Integration of scanning probe microscope with high-performance computing: Fixed-policy and reward-driven workflows implementation

The rapid development of computation power and machine learning algorithms has paved the way for automating scientific discovery with a scanning probe microscope (SPM). The key elements toward operationalization of the automated SPM are the interface to enable SPM control from Python codes, availability of high computing power, and development of workflows for scientific discovery. Here, we build a Python interface library that enables controlling an SPM from either a local computer or a remote high-performance computer, which satisfies the high computation power need of machine learning algorithms in autonomous workflows. We further introduce a general platform to abstract the operations of SPM in scientific discovery into fixed-policy or reward-driven workflows. Furthermore, our work provides a full infrastructure to build automated SPM workflows for both routine operations and autonomous scientific discovery with machine learning.

47 OTHER INSTRUMENTATION↗
DOE OSTI2026

The role of quantum computing in advancing scientific high-performance computing: A perspective from the ADAC institute

Quantum computing (QC) has gained significant attention over the past two decades due to its potential for speeding up classically demanding tasks. This transition from an academic focus to a thriving commercial sector is reflected in substantial global investments. While advancements in qubit counts and functionalities continue at a rapid pace, current quantum systems still lack the scalability for practical applications, facing challenges such as too high error rates and limited coherence times. Here, this perspective paper examines the relationship between QC and high-performance computing (HPC), highlighting their complementary roles in enhancing computational efficiency. It is widely acknowledged that even fully error-corrected QC will not be suited for all computational tasks. Rather, future compute infrastructures are anticipated to employ quantum acceleration within hybrid systems that integrate HPC and QC. While QC can enhance classical computing, traditional HPC remains essential for maximizing quantum acceleration. This integration is a priority for supercomputing centers and companies, sparking innovation to address the challenges of merging these technologies. The novelty of this work lies in its unique perspective, reflecting the collective insights of the Accelerated Data Analytics and Computing (ADAC) Institute, a global consortium of over 20 leading HPC centers. Recognizing the growing importance of QC, ADAC established a Quantum Computing Working Group in 2023 to foster collaboration and knowledge-sharing among its members. This paper synthesizes insights from the group’s collaborative efforts and incorporates findings from a member survey that captures shared experiences, ongoing projects, and strategic directions. By outlining the current landscape and challenges of QC integration into HPC ecosystems, this work offers HPC specialists practical and forward-looking guidance on the opportunities and implications of QC in computationally intensive endeavors.

Accelerated Data Analytics and↗
DOE OSTI2022

aphBO-2GP-3B: a budgeted asynchronous parallel multi-acquisition functions for constrained Bayesian optimization on high-performing computing architecture

High-fidelity complex engineering simulations are often predictive, but also computationally expensive and often require substantial computational efforts. The mitigation of computational burden is usually enabled through parallelism in high-performance cluster (HPC) architecture. Optimization problems associated with these applications is a challenging problem due to the high computational cost of the high-fidelity simulations. In this paper, an asynchronous parallel constrained Bayesian optimization method is proposed to efficiently solve the computationally expensive simulation-based optimization problems on the HPC platform, with a budgeted computational resource, where the maximum number of simulations is a constant. The advantage of this method are three-fold. Firstly, the efficiency of the Bayesian optimization is improved, where multiple input locations are evaluated parallel in an asynchronous manner to accelerate the optimization convergence with respect to physical runtime. This efficiency feature is further improved so that when each of the inputs is finished, another input is queried without waiting for the whole batch to complete. Second, the proposed method can handle both known and unknown constraints. Third, the proposed method samples several acquisition functions based on their rewards using a modified GP-Hedge scheme. The proposed framework is termed aphBO-2GP-3B, which means asynchronous parallel hedge Bayesian optimization with two Gaussian processes and three batches. The numerical performance of the proposed framework aphBO-2GP-3B is comprehensively benchmarked using 16 numerical examples, compared against other 6 parallel Bayesian optimization variants and 1 parallel Monte Carlo as a baseline, and demonstrated using two real-world high-fidelity expensive industrial applications. The first engineering application is based on finite element analysis (FEA) and the second one is based on computational fluid dynamics (CFD) simulations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Accelerating Computational Materials Discovery with Machine Learning and Cloud High-Performance Computing: from Large-Scale Screening to Experimental Validation

High-throughput computational materials discovery has promised significant acceleration of the design and discovery of new materials for many years. Despite a surge in interest and activity, the constraints imposed by large-scale computational resources present a significant bottleneck. Furthermore, examples of large-scale computational discovery carried through experimental validation remain scarce, especially for materials with product applicability. In this paper, we demonstrate how this vision became reality by first combining state-of-the-art artificial intelligence (AI) models and traditional physics-based models on cloud high performance computing (HPC) resources to quickly navigate through more than 32 million candidates and predict around half a million potentially stable materials. Focusing on solid-state electrolytes for battery applications, our discovery pipeline further identified 18 promising candidates with new compositions and rediscovered a decade’s worth of collective knowledge in the field as a byproduct. By employing around one thousand virtual machines in the cloud, this process took less than 80 hours. We then synthesized and experimentally characterized the structures and conductivities of our top candidates, the Na x Li 3-x YCl 6 (0.5 ≤ x ≤ 2.5) series, demonstrating the potential of these compounds to serve as solid electrolytes. Additional candidate materials are currently under experimental investigation that could offer more examples of the computational discovery of new phases of Li- and Na-conducting solid electrolytes. We believe this unprecedented approach of synergistically integrating AI models and cloud HPC not only accelerates materials discovery but also showcases the potency of AI-guided experimentation in unlocking transformative scientific breakthroughs with real-world applications.

36 MATERIALS SCIENCE↗