Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “HPC applications”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Performance Characterization and Provenance of Distributed Task-based Workflows on HPC Platforms

Understanding performance and provenance of task-based workflows poses significant challenges, particularly in distributed configurations where resources are shared by multiple applications. Task-based workflow management systems further complicate performance predictability because of their dynamicity that subtly alters task execution order from run to run. In this paper we propose a layered characterization framework for performance and task provenance for Dask.distributed workflows running on high-performance computing (HPC) platforms. It collects data from jobs, the workflow management system, and the operating system to aid in understanding the performance of these workflows. Our approach encompasses three main contributions: first, an extension of Dask.distributed to capture high-fidelity task provenance using Mochi data services; second, the adaptation of the established HPC I/O characterization tool Darshan to gather high-fidelity I/O data, thereby enhancing the granularity of our analysis; and third, a framework to combine and process the collected data and provide helpful insights into performance characterization and reproducibility, alongside our lessons learned.

Dask↗
DOE OSTI2020

2020 Exascale Computing Project Annual Meeting (Executive Summary Report)

The Exascale Computing Project (ECP) delivers specific applications, software products, and outcomes on DOE computing facilities. Integration across these elements for specific hardware technologies for exascale system instantiations is fundamental to ECP success. The outcome of the ECP is the delivery of a capable exascale computing ecosystem to provide breakthrough solutions addressing our most critical challenges in scientific discovery, energy assurance, economic competitiveness, and national security. This outcome is not a matter of ensuring more powerful computing systems. The ECP is designed to create more valuable and rapid insights from a wide variety of applications (“capable”), which requires a much higher level of inherent efficacy in all methods, software tools, and ECP-enabled computing technologies to be acquired by DOE laboratories (“ecosystem”). The ECP annual meeting provides a unique opportunity for the core technical expertise in the United States focused on achieving this next plateau of computational science and computing performance to engage in direct discussions on project execution. Face-to-face gatherings in technical communities like this are common and needed for the exchange of scientific ideas and technical performance. The ECP annual meeting stands apart from other technical conferences and meetings in the computing community as it is uniquely and solely focused on the execution of the ECP and the integration of technical activities leading to the creation of the exascale computing ecosystem for the future. The direct interaction of key critical technical staff, who are leaders in their respective fields, and the resulting give-and-take between software, applications, and hardware and the technical co-design therein, is unique and essential to the effective execution of the ECP. The first annual meeting was held in Knoxville, Tennessee, January 31 – February 2, 2017 and brought together, for the first time, a diverse collection of researchers from 16 DOE national laboratories as well as university computer and computational science researchers to discuss shared problems and joint solutions for the development of a capable exascale computing ecosystem. These interactions resulted in focused technical plans and an energized community centered on advances for ECP. The second annual meeting was held in Knoxville, Tennessee, February 5–9, 2018. It included 643 individual thought leaders and performers in application development, software research and deployment, and hardware research and integrators, all of whom are part of the multifaceted, billion dollar HPC community. This meeting provided a platform to discuss and disseminate numerous examples where researchers with common goals and synergistic solutions came together for the first time to deliver tangible results. Additionally, at the 2018 meeting, ECP researchers had the opportunity to digest all US HPC vendor R&D product roadmaps pointing to exascale – not only to learn how their research can play a role, but, more importantly, to influence those roadmaps to ensure successful delivery on DOE applications that will contribute to (if not solve) problems of national interest in national security, science, energy, and health, as well as growing security threats. The third annual meeting was held in Houston, Texas, January 14–17, 2019. With a 19% increase in the number of registrations (768 people), and the change in location, the third annual meeting was considered the most impactful of the three at the time. The new website provided a better platform for the dissemination of the content, the new venue as a meeting hotel instead of a conference center facilitated interactions and discussions after event hours, and the addition of an award-winning mobile event conference app (Whova) transformed dramatically the attendee experience at the event. This fourth annual meeting was held in Houston, Texas, February 3-7, 2020. This meeting had an increase in the number of attendees for a total of 824 people registered (782 attendees) and included numerous enhancements based on feedback and lessons learned from previous meetings, some of which are listed here: Improved quality of the sessions, their material and the whole program.; Had more industry participation and addition of external collaborators from overseas.; Published the full agenda earlier to better accommodate attendance and travel plans based on schedule.; Centralized all sessions in one venue.; Provided additional hotels and room blocks for the attendees.; Improved communication with the audience (links, material, directions, notifications, etc.) to go paperless.; Enhanced side meeting scheduling, management and user experience.; Made available additional space and tables for impromptu meetings and side discussions.; Improved IT and A/V solutions for speakers. In addition, our final survey captured the following points as opportunities for improvement in future meetings: consider a different meeting location that is more pedestrian friendly, reduce talks during working meals to allow more collaboration and informal time, adapt the agenda to acknowledge attendees from different timezones, consider recording some of the tutorials and/or sessions to share broadly with the HPC community, have a larger poster room, provide additional power strips, and improve the WiFi.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Accelerating Computational Materials Discovery with Machine Learning and Cloud High-Performance Computing: from Large-Scale Screening to Experimental Validation

High-throughput computational materials discovery has promised significant acceleration of the design and discovery of new materials for many years. Despite a surge in interest and activity, the constraints imposed by large-scale computational resources present a significant bottleneck. Furthermore, examples of large-scale computational discovery carried through experimental validation remain scarce, especially for materials with product applicability. In this paper, we demonstrate how this vision became reality by first combining state-of-the-art artificial intelligence (AI) models and traditional physics-based models on cloud high performance computing (HPC) resources to quickly navigate through more than 32 million candidates and predict around half a million potentially stable materials. Focusing on solid-state electrolytes for battery applications, our discovery pipeline further identified 18 promising candidates with new compositions and rediscovered a decade’s worth of collective knowledge in the field as a byproduct. By employing around one thousand virtual machines in the cloud, this process took less than 80 hours. We then synthesized and experimentally characterized the structures and conductivities of our top candidates, the Na x Li 3-x YCl 6 (0.5 ≤ x ≤ 2.5) series, demonstrating the potential of these compounds to serve as solid electrolytes. Additional candidate materials are currently under experimental investigation that could offer more examples of the computational discovery of new phases of Li- and Na-conducting solid electrolytes. We believe this unprecedented approach of synergistically integrating AI models and cloud HPC not only accelerates materials discovery but also showcases the potency of AI-guided experimentation in unlocking transformative scientific breakthroughs with real-world applications.

36 MATERIALS SCIENCE↗
DOE OSTI2021

A Hierarchical Task Scheduler for Heterogeneous Computing

Heterogeneous computing is one of the future directions of HPC. Task scheduling in heterogeneous computing must balance the challenge of optimizing the application performance and the need for an intuitive interface with the programming run-time to maintain programming portability. The challenge is further compounded by the varying data communication time between tasks. This paper proposes RANGER, a hardware-assisted task-scheduling framework. By integrating RISC-V cores with accelerators, the RANGER scheduling framework divides scheduling into global and local levels. At the local level, RANGER further partitions each task into fine-grained subtasks to reduce the overall makespan. At the global level, RANGER maintains the coarse granularity of the task specification, thereby maintaining programming portability. The extensive experimental results demonstrate that RANGER achieves a 12.7× performance improvement on average, while only requires 2.7% of area overhead.

Miniskar, Narasinga Rao↗
DOE OSTI2025

Assessment of ESM Readiness Level for Exascale HPC

Advancement of Earth System Models (ESMs) is becoming increasingly challenging due to a confluence of factors including increasing model complexity – to more fully represent the earth system, increasing spatial resolution - to achieve higher accuracy by resolving fine-scale dynamical to physical, biological, and chemical processes and their interaction, increasing ensemble size - to more accurately represent predictive uncertainty, and increased computing requirements – to enable more accurate and timely weather predictions and climate projections for societal benefit. The belief by many that computing will take care of itself is no longer valid given the disruptive changes in HPC that are driving up the cost of computing, increasing the difficulty of using emerging HPC effectively, and exposing limits in parallelism, portability and scalability of the ESM applications themselves.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Orchestrating Fault Prediction with Live Migration and Checkpointing

Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ~20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ~29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.

Behera, Subhendu↗
DOE OSTI2022

Understanding and Leveraging the I/O Patterns of Emerging Machine Learning Analytics

The scientific community is currently experiencing unprecedented amounts of data generated by cutting-edge science facilities. Soon facilities will be producing up to 1 PB/s which will force scientist to use more autonomous techniques to learn from the data. The adoption of machine learning methods, like deep learning techniques, in large-scale workflows comes with a shift in the workflow’s computational and I/O patterns. These changes often include iterative processes and model architecture searches, in which datasets are analyzed multiple times in different formats with different model configurations in order to find accurate, reliable and efficient learning models. This shift in behavior brings changes in I/O patterns at the application level as well at the system level. These changes also bring new challenges for the HPC I/O teams, since these patterns contain more complex I/O workloads. In this paper we discuss the I/O patterns experienced by emerging analytical codes that rely on machine learning algorithms and highlight the challenges in designing efficient I/O transfers for such workflows. We comment on how to leverage the data access patterns in order to fetch in a more efficient way the required input data in the format and order given by the needs of the application and how to optimize the data path between collaborative processes. We will motivate our work and show performance gains with a study case of medical applications.

Gainaru, Ana↗
DOE OSTI2026

Shaping the FutureWorkforce: Challenges and Lessons Learned in HPC Education from National Labs and Computing Centers

Workforce training at national laboratories and computing centers is essential and typically falls into two categories: foundational training for newcomers and advanced training for experienced users. Foundational topics—such as version control, build systems, and basic HPC usage—are largely transferable across institutions, while cluster-specific training varies due to differences in hardware, job schedulers, and local workflows. Training on emerging technologies is split between hardware-specific content and broadly applicable programming paradigms. Here, to reduce redundancy and increase impact, national labs, computing centers, and vendors are collaborating through initiatives like the HPC Training Working Group to share best practices, co-develop materials, and broaden outreach. These coordinated efforts aim to make HPC training more accessible, scalable, and consistent across the community.

HPC↗
DOE OSTI2025

Regen: An object layout regenerator on large-scale production HPC systems

This article proposes an object layout regenerator called Regen which regenerates and removes the object layout dynamically to improve the read performance of applications. Regen first detects frequent access patterns from the I/O requests of the applications. Second, Regen reorganizes the objects and regenerates or preallocates new object layouts according to the identified access patterns. Finally, Regen removes or reuses the obsolete or regenerated object layouts as necessary. As a result, Regen accelerates access to objects by providing a flexible object layout. We implement Regen as a framework on top of Proactive Data Container (PDC) and evaluate it on Cori supercomputer, a production-scale HPC system, by using realistic HPC I/O benchmarks. The experimental results show that Regen improves the I/O performance by up to 16.92 × compared with an existing system.

Distributed file system↗
DOE OSTI2023

Code modernization strategies for short-range non-bonded molecular dynamics simulations

Modern HPC systems are increasingly relying on greater core counts and wider vector registers. Thus, applications need to be adapted to fully utilize these hardware capabilities. One class of applications that can benefit from this increase in parallelism are molecular dynamics simulations. In this paper, we describe our efforts at modernizing the ESPResSo++ simulation package for molecular dynamics by restructuring its particle data layout for efficient memory accesses and applying vectorization techniques to benefit the calculation of short-range non-bonded forces, which results in an overall three times speedup and serves as a baseline for further optimizations. We also implement fine-grained parallelism for multi-core CPUs through HPX, a C++ runtime system which uses lightweight threads and an asynchronous many-task approach to maximize concurrency. Our goal is to evaluate the performance of an HPX-based approach compared to the bulk-synchronous MPI-based implementation. This requires the introduction of an additional layer to the domain decomposition scheme that defines the task granularity. On spatially inhomogeneous systems, which impose a corresponding load-imbalance in traditional MPI-based approaches, we demonstrate that by choosing an optimal task size, the efficient work-stealing mechanisms of HPX can overcome the overhead of communication resulting in an overall 1.4 times speedup compared to the baseline MPI version.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

GPU Direct I/O with HDF5

Exascale HPC systems are being designed with accelerators, such as GPUs, to accelerate parts of applications. In machine learning workloads as well as large-scale simulations that use GPUs as accelerators, the CPU (or host) memory is currently used as a buffer for data transfers between GPU (or device) memory and the file system. If the CPU does not need to operate on the data, then this is sub-optimal because it wastes host memory by reserving space for duplicated data. Furthermore, this “bounce buffer” approach wastes CPU cycles spent on transferring data. A new technique, NVIDIA GPUDirect Storage (GDS), can eliminate the need to use the host memory as a bounce buffer. Thereby, it becomes possible to transfer data directly between the device memory and the file system. This direct data path shortens latency by omitting the extra copy and enables higher-bandwidth. To take full advantage of GDS in existing applications, it is necessary to provide support with existing I/O libraries, such as HDF5 and MPI-IO, which are heavily used in applications. In this paper, we describe our effort of integrating GDS with HDF5, the top I/O library at NERSC and at DOE leadership computing facilities. We design and implement this integration using a HDF5 Virtual File Driver (VFD). The GDS VFD provides a file system abstraction to the application that allows HDF5 applications to perform I/O without the need to move data between CPUs and GPUs explicitly. We compare performance of the HDF5 GDS VFD with explicit data movement approaches and demonstrate superior performance with the GDS method.

Ravi, J↗
DOE OSTI2023

5G Energy FRAME: The Design and Implementation of Data, Model, and Use Case (Year 2 Report)

This report summarizes the Year 2 work of Pacific Northwest National Laboratory’s (PNNL’s) 5G Fabricated Resource and Asset Management Encompassment for energy infrastructure (Energy FRAME) project funded by the Department of Energy Office of Science’s Advanced Scientific Computing Research (ASCR) Program. In this report, newest 5G equipment testing results are presented, along with two 5G-enabled AI/ML examples for grid applications; in addition, the work flow of grid edge, cloud, and High Performance Computing (HPC) platform is introduced, to support and interface the cross-domain simulation for power system transmission, distribution, and communication networks. Last but not least, the outlook for Year 3 work and the overarching impact of 5G Energy FRAME work to a multitude of stakeholders are provided. Additional 5G performance data now is shared through the publicly available weblink, https://www.pnnl.gov/projects/5g-energy-frame/publications

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-aligned engine which provides up to 2.3× improved performance than when using the existing DAOS-POSIX interface, without requiring any application modifications.

Venkatesh, Ranjan Sarpangala↗
NASA NTRS2017

pFlogger: The Parallel Fortran Logging Utility

In the context of high performance computing (HPC), software investments in support of text-based diagnostics, which monitor a running application, are typically limited compared to those for other types of IO. Examples of such diagnostics include reiteration of configuration parameters, progress indicators, simple metrics (e.g., mass conservation, convergence of solvers, etc.), and timers. To some degree, this difference in priority is justifiable as other forms of output are the primary products of a scientific model and, due to their large data volume, much more likely to be a significant performance concern. In contrast, text-based diagnostic content is generally not shared beyond the individual or group running an application and is most often used to troubleshoot when something goes wrong. We suggest that a more systematic approach enabled by a logging facility (or 'logger)' similar to those routinely used by many communities would provide significant value to complex scientific applications. In the context of high-performance computing, an appropriate logger would provide specialized support for distributed and shared-memory parallelism and have low performance overhead. In this paper, we present our prototype implementation of pFlogger - a parallel Fortran-based logging framework, and assess its suitability for use in a complex scientific application.

Clune, Tom↗
DOE OSTI2024

Integrating Quantum Computing with High-Performance Computing: A Streamlined Approach

In recent years, quantum computing has demon-strated the potential to revolutionize specific algorithms and applications by solving problems exponentially faster than classical computers. However, its widespread adoption for general computing remains a future prospect. This paper discusses the integration of quantum computing within High-Performance Computing (HPC) environments, focusing on a resource management framework designed to streamline quantum simulators' use and enhance runtime performance and efficiency. The proposed framework facilitates hybrid applications' transition from simulation backends to real quantum hardware, optimizing resource utilization and providing a flexible infrastructure for developing and testing quantum algorithms.

Shehata, Amir↗
DOE OSTI2020

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Scientific applications at exascale generate and analyze massive amounts of data. A critical requirement of these applications is the capability to access and manage this data efficiently on exascale systems. As such, parallel I/O, the key technology enables moving data between compute nodes and storage, faces monumental challenges from new applications, memory, and storage architectures considered in the designs of exascale systems. As the storage hierarchy is expanding to include node-local persistent memory, burst buffers, etc., as well as disk-based storage, data movement among these layers must be efficient. Parallel I/O libraries of the future should be capable of handling file sizes of many terabytes and beyond. In this paper, we describe new capabilities we have developed in Hierarchical Data Format version 5 (HDF5), the most popular parallel I/O library for scientific applications. HDF5 is one of the most used libraries at the leadership computing facilities for performing parallel I/O on existing HPC systems. The state-of-the-art features we describe include: Virtual Object Layer (VOL), Data Elevator, asynchronous I/O, full-featured single-writer and multiple-reader (Full SWMR), and parallel querying. In this paper, we introduce these features, their implementations, and the performance and feature benefits to applications and other libraries.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Transforming Energy Through Computational Excellence: High-Performance Computing for Energy Innovation

The challenges associated with energy efficiency of manufacturing and advanced materials often cannot be addressed through experimentation alone, whether because of scale, complexity, or practicality. High-performance computing (HPC) enables fast tackling of these challenges in the manufacturing sector - vital to achieving net-zero carbon emissions by 2050. The National Renewable Energy Laboratory (NREL) and industry partners leverage HPC to apply advanced modeling, simulation, and data analysis to improve manufacturing efficiency, explore new materials for energy applications, and develop technologies to manage carbon across the life cycle. From improving additive manufacturing processes to increasing the energy efficiency of jet-engine components, advanced computing can help manage emissions produced by manufacturing in a wide variety of ways.

advanced materials↗
DOE OSTI2023

Workflows Community Summit 2022: A Roadmap Revolution

Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from the execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing (often referred to as a computing continuum) and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large-scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC, enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enable the publication of workflows and their associated products according to the FAIR principles.

97 MATHEMATICS AND COMPUTING↗