Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “HPC Access”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

S&TR September 2025: Computing Grand Challenge Turns 20

Livermore’s Computing Grand Challenge Program enters its 20th year with more unclassified high-performance computing (HPC) power than ever before. This unique, peer-reviewed competition awards HPC allocations on top supercomputers to multidisciplinary teams with high-impact projects. The Grand Challenge encourages researchers to innovate, pushes scientific discovery to new heights, improves the Laboratory’s HPC capabilities, and extends HPC accessibility to collaborators. Awardees must adapt to successive generations of HPC hardware and learn to run simulations at scale. The feature article spotlights three Grand Challenge teams whose research broke new ground in key scientific pursuits—the essence of dark matter, explosion-generated seismic waves, and protein interactions linked to cancer—while underscoring the importance of academic partnerships and considering the program’s future.

07 ISOTOPE AND RADIATION SOURCES↗

A Hands-On Curriculum for Training in HPC Cluster Deployment and Management

This paper presents the design, methodology, and outcomes of the High-Performance Computing Technologies (HPCT) course, a hands-on training program focused on the system-side of HPC cluster deployment and administration. Delivered as part of the Master in High Performance Computing (MHPC) program, the course introduces students to key concepts in cluster configuration, including networking, software stack provisioning, job scheduling, and monitoring. Initially taught in person, the course was transitioned to an online format during the COVID-19 pandemic. This shift led to the development of openly available instructional material and a flipped-classroom approach that continues to support both in-person and hybrid delivery. All course materials are publicly available at www.hpc.temple.edu/mhpc/hpc-technology/index.html. By documenting the structure, infrastructure, and evolution of HPCT, this paper offers a model for accessible HPC system training that supports workforce development in computational science.

Posada Correa, Fernando [ORNL] (ORCID:000000022565↗

Inl Open Ondemand Applications

Open OnDemand is a software tool that is used to access HPC resources. It provides a framework for organizations to create apps and other additional functionality that may be useful to the organization. This code expands upon the pre-existing INL applications, including the NEAMS Workbench application, MOOSE Herd applications, and others. These changes significantly expand upon the functionality originally provided. Due to the extensive functionality that we added, these changes would not be added to the original application but would function as a standalone application that other organizations would be able to utilize on their own systems.

Biggs, BrandonS.↗

Inl Open Ondemand Dashboard Applications

Open OnDemand is a software tool that is used to access HPC resources. It provides a framework for organizations to create apps and other additional functionality that may be useful to the organization. This code creates new INL applications. These changes significantly expand upon the functionality originally provided. Due to the extensive functionality that we added, these changes would not be added to the original application but would function as additional applications that other organizations would be able to utilize on their own systems.

Biggs, Brandon [Idaho National Laboratory (INL), I↗

INL High Performance Computing Overview

INL High Performance Computing Overview presentation by Eric Whiting for the National Science Users Facility 2020 Annual Program Review on November 10, 2020; data covering our Collaborative Computing Center (C3), the supercomputers and their data.

99 GENERAL AND MISCELLANEOUS↗

I/O Access Patterns in HPC Applications: A 360-Degree Survey

The high-performance computing I/O stack has been complex due to multiple software layers, the inter-dependencies among these layers, and the different performance tuning options for each layer. In this complex stack, the definition of an “I/O access pattern” has been reappropriated to describe what an application is doing to write or read data from the perspective of different layers of the stack, often comprising a different set of features. It has become common to have to redefine what is meant when discussing a pattern in every new study, as no assumption can be made. This survey aims to propose a baseline taxonomy, harnessing the I/O community’s knowledge over the past 20 years. This definition can serve as a common ground for high-performance computing I/O researchers and developers to apply known I/O tuning strategies and design new strategies for improving I/O performance. We seek to summarize and bring a consensus to the multiple ways to describe a pattern based on common features already used by the community over the years.

97 MATHEMATICS AND COMPUTING↗

Quantifying the Impact of Advanced Web Platforms on High Performance Computing Usage

The deployment of Science Gateways for High Performance Computing (HPC) systems can alter long-accepted usage patterns on supercomputing systems in positive ways as an ever-increasing number of users migrate their workflows to HPC systems. Idaho National Laboratory (INL) has deployed two separate advanced web platforms, Open OnDemand and NICE DCV, for integration with HPC resources to improve web accessibility for HPC users. Researchers conducted a multi-year study on how HPC usage pat- terns changed in the presence of these platforms. This work reports the results of that study and quantifies the observed impacts, including adoption by visualization and Jupyter Notebook/Lab users, decreased job submission friction, rapid uptake of HPC by Windows users, and increased overall system utilization. The most significant impacts were observed from the deployment of Open OnDemand, and this work also identifies some best practices for Open OnDemand deployment for HPC datacenters.

97 MATHEMATICS AND COMPUTING↗

Analyzing File Access Patterns on Large-Scale HPC Systems: Opportunities for File Prefetching

This paper explores the potential opportunities for implementing file prefetching techniques on large-scale high-performance computing (HPC) systems. Specifically, we investigate the file access patterns of various applications across multiple scientific domains using two years' worth of Darshan I/O traces obtained from the Summit supercomputer. We identify recurring trends and patterns which indicate that prefetching can be effectively leveraged to improve data access performance on HPC systems. This study serves as a valuable reference for system architects and developers in the HPC community, providing insights into the opportunities and challenges associated with enabling file prefetching on large-scale HPC systems.

Karimi, Ahmad Maroof↗

I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey

Growing interest in Artificial Intelligence (AI) has resulted in a surge in demand for faster methods of Machine Learning (ML) model training and inference. This demand for speed has prompted the use of high performance computing (HPC) systems that excel in managing distributed workloads. Because data is the main fuel for AI applications, the performance of the storage and I/O subsystem of HPC systems is critical. In the past, HPC applications accessed large portions of data written by simulations or experiments or ingested data for visualizations or analysis tasks. ML workloads perform small reads spread across a large number of random files. This shift of I/O access patterns poses several challenges to modern parallel storage systems. In this paper, we survey I/O in ML applications on HPC systems, and target literature within a 6-year time window from 2019 to 2024. We define the scope of the survey, provide an overview of the common phases of ML, review available profilers and benchmarks, examine the I/O patterns encountered during offline data preparation, training, and inference, and explore I/O optimizations utilized in modern ML frameworks and proposed in recent literature. Lastly, we seek to expose research gaps that could spawn further R&D.

97 MATHEMATICS AND COMPUTING↗

HPC Campaign Management: Remote data access with user-defined error bound using ADIOS and ZFP

Remote access to large-scale scientific datasets, like those generated by combustion simulations or other high-performance computing (HPC) applications, presents a significant challenge. Downloading entire datasets is often impractical due to their size and the bandwidth limitations of typical networks. To address this challenge, we propose a novel approach that enables efficient remote access to large datasets distributed across multiple facilities. Our method enables technologies to download only the data values of a select variable, in a select region of interest, to a user-defined accuracy. For this purpose, we extended the ADIOS IO library to provide read functions with user-defined accuracy, a remote data server that understands multidimensional selections of specific variables, steps and accuracy from an ADIOS dataset, and which uses lossy compression on the remote site to reduce the data to be transferred back to the client. In addition, our extension of the ADIOS library collects metadata from multiple datasets in small files called Campaign Archives, which can be shared among project participants on any HPC, cloud or laptop, and which can easily facilitate the discovery of content and pointers to the data location as well as remote access to the data by local tools as if data was local. This feature called Campaign Management, enables a group of scientists to manage related datasets stored in multiple files, across multiple facilities as if it was in a single file/database. We demonstrate the effectiveness of our approach using a 1.5 TB dataset from the S3D combustion simulation on Frontier at the Oak Ridge Leadership Facility. Even a single variable from this dataset, at 64 GB, is too large to be processed on a standard laptop. We show two different reading patterns for 2D plots and 3D visualization, with careful settings that a scientist studying combustion data would do and show that running the same Python scripts on Frontier directly takes comparable time than running them on the local laptop with remote access to the data on Frontier.

Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗

dCache: The Storage System of Choice for Data-Intensive Applications

The ever-increasing volumes of data produced by modern scientific facilities like EuXFEL and LHC put significant stress on data management infrastructure operated by laboratories and research centers. The challenges to be addressed span the entire data life cycle, from ingest and efficient data analysis to long-term preservation, typically involving large tape libraries. dCache, a storage system developed in collaboration between the Deutsches Elektronen-Synchrotron (DESY), Fermi National Accelerator Laboratory, and Nordic e-Infrastructure Collaboration (NeIC), is designed to manage a large number of disk servers and to facilitate transparent data migration to and from archival storage. Its multifaceted approach offers a unified method to support a variety of scientific use cases with the same storage infrastructure, including high-throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and long-term data preservation on tertiary storage. Initially developed for high energy physics (HEP) experiments, dCache is now used by various scientific communities, including astrophysics, biomedical research, and life sciences, each having specific requirements. This paper presents architecture, deployment strategies, performance and scalability enhancements, and recent advancements in dCache addressing the needs of scientific communities. Finally, we touch on the development and release process, ensuring the software’s high quality.

DCache↗

dCache: Inter-disciplinary storage system

The dCache project provides open-source software deployed internationally to satisfy ever more demanding storage requirements. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters and long term data persistence on a tertiary storage. Though it was originally developed for the HEP experiments, today it is used by various scientific communities, including astrophysics, biomed, life science, which have their specific requirements. In this paper we describe some of the new requirements as well as demonstrate how dCache developers are addressing them.

Mkrtchyan, Tigran↗

dCache project status and update

The dCache project delivers an open-source, massively scalable, distributed storage system deployed internationally to satisfy today’s scientists’ ever-demanding storage requirements. Its multifaceted approach supports different use cases with the same storage, from high throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and longterm data persistence on tertiary storage. Even though dCache was initially developed for HEP experiments, today, it is used by various scientific communities, including astrophysics, biomed, and life science, each with their specific requirements. To match the needs of these new communities and keep up with the scaling demands of existing experiments, dCache is permanently evolving. With this contribution, we would like to highlight the recent developments in dCache regarding integration with CERN Tape Archive (CTA), advanced metadata handling, token-based authorization support, bulk API for QoS transitions, REST API to control interaction with the tape system, and future development directions.

Mkrtchyan, Tigran [DESY]↗

Final Reports of the 2020 Los Alamos National Laboratory Computational Physics Student Summer Workshop

For the past ten years, the workshop has been bringing a highly talented and diverse group of student every summer. Students work in teams of two, alongside typically two mentors, on research projects reflecting a broad range of topics within computational physics. In addition, students attend a series of lectures on topics within computational physics, facility tours, and networking events. The program lasts ten weeks, with this year’s workshop running from June 8 to August 14. At the end of the summer, students give a final presentation, along with a written report. Those reports are what make up the remaining sections of this document. Admission to the workshop is by a competitive process, with the mentors forming the selection committee. One of the important accomplishments of the workshop has been to create a student pipeline from diverse schools that sometimes are not normally tapped by LANL recruiting. Many workshop students maintain a continuing relationship with LANL, returning as students interns, post-doctoral researchers, and staff members. Additionally, workshop alumni act as ambassadors for LANL. The result is a wider awareness both of LANL as a potential employer, and of the technical work that happens at LANL. This year, the workshop format was changed in several ways, in order to accommodate the off-site, virtual format. Students worked on LANL virtual desktop systems remotely, also accessing LANL HPC resources. In order to facilitate communication, student were given accounts on both Webex, a video teleconferencing platform, and Mattermost, an online team collaboration and chat platform, similar to Slack. Daily communication between students and mentors was primarily on Mattermost, with Webex conferencing as needed. The lectures were all done on Webex. Given the difficulty of the virtual format, and a concern that students might have video teleconferencing burn-out after an academic semester largely moved to that format, all lectures were optional this year. In spite of this, the attendance was generally high. Lecturers were asked to try to move to a more high-level, ”What is it?,” format. Once again, the students did a tremendous job. Over the course of ten weeks, they did important research across a staggering array of disciplines. The following pages contain the final report for each team’s research efforts. We hope you will find reading them as exciting as it was for us to produce them.

36 MATERIALS SCIENCE↗

Final Reports of the 2021 Los Alamos National Laboratory Computational Physics Student Summer Workshop

Since 2011, the Los Alamos National Laboratory Computational Physics Student Summer Workshop has been bringing together a highly talented and diverse group of students every summer. Students work in teams of two, alongside typically two mentors, on research projects reflecting a broad range of topics within computational physics. In addition, students attend a series of lectures on topics within computational physics, facility tours, and networking events. The program lasts ten weeks, with this year’s workshop running from June 7 to August 13. At the end of the summer, students give a final presentation, along with a written report. Those reports are what make up the remaining sections of this document. Admission to the workshop is by a competitive process, with the mentors forming the selection committee. One of the important accomplishments of the workshop has been to create a student pipeline from diverse schools that sometimes are not normally tapped by LANL recruiting. Many workshop students maintain a continuing relationship with LANL, returning as student interns, post-doctoral researchers, and staff members. Additionally, workshop alumni act as ambassadors for LANL. The result is a wider awareness both of LANL as a potential employer, and of the technical work that happens at LANL. This year, the workshop was once again in an off-site, virtual format. Students worked on LANL virtual desktop systems remotely, also accessing LANL HPC resources. In order to facilitate communication, student were given accounts on both Webex, a video teleconferencing platform, and Mattermost, an online team collaboration and chat platform, similar to Slack. Daily communication between students and mentors was primarily on Mattermost, with Webex conferencing as needed. The lectures were all done on Webex. Given the difficulty of the virtual format, and a concern that students might have video teleconferencing burn-out, all lectures were optional this year. In spite of this, the attendance was generally high. Lecturers were asked to try to move to a more high-level, ”What is it?,” format. Once again, the students did a tremendous job. Over the course of ten weeks, they did important research across a staggering array of disciplines. The following pages contain the final report for each team’s research efforts. Enjoy!

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Intro to HPC Bootcamp: Engaging New Communities Through Energy Justice Projects

The U.S. Department of Energy (DOE) is a long-standing leader in research and development of high-performance computing (HPC) in the pursuit of science. However, we face daunting challenges in fostering a robust and diverse HPC workforce. Basic HPC is not typically taught at early stages of students' academic careers, and the capacity and knowledge of HPC at many institutions are limited. Even so, such topics are prerequisites for advanced training programs, internships, graduate school, and ultimately for careers in HPC. To help address this challenge, as part of the DOE Exascale Computing Project's Broadening Participation Initiative, we recently launched the Introduction to HPC Training and Workforce Pipeline Program to provide accessible introductory material on HPC, scalable AI, and analytics. We describe the Intro to HPC Bootcamp, an immersive program designed to engage students from underrepresented groups as they learn foundational HPC skills. Here, the program takes a novel approach to HPC training by turning the traditional curriculum upside down. Instead of focusing on technology and its applications, the bootcamp focuses on energy justice to motivate the training of HPC skills through project-based pedagogy and real-life science stories. Additionally, the bootcamp prepares students for internships and future careers at DOE labs. The first bootcamp, hosted by the advanced computing facilities at Argonne, Lawrence Berkeley, and Oak Ridge National Labs and organized by Sustainable Horizons Institute, took place in August 2023.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Shaping the FutureWorkforce: Challenges and Lessons Learned in HPC Education from National Labs and Computing Centers

Workforce training at national laboratories and computing centers is essential and typically falls into two categories: foundational training for newcomers and advanced training for experienced users. Foundational topics—such as version control, build systems, and basic HPC usage—are largely transferable across institutions, while cluster-specific training varies due to differences in hardware, job schedulers, and local workflows. Training on emerging technologies is split between hardware-specific content and broadly applicable programming paradigms. Here, to reduce redundancy and increase impact, national labs, computing centers, and vendors are collaborating through initiatives like the HPC Training Working Group to share best practices, co-develop materials, and broaden outreach. These coordinated efforts aim to make HPC training more accessible, scalable, and consistent across the community.

HPC↗

MemFriend: Understanding Memory Performance with Spatial-Temporal Affinity

In HPC applications, memory access behavior is one of the main factors affecting performance. Improving an application’s memory access behavior involves optimizing data layout and/or restructuring code, and requires studying spatial-temporal data locality. Existing data locality analyses focus on single-location metrics and are restricted to evaluating temporal locality. We introduce spatial-temporal affinity metrics that quantify temporal access proximity, forward access correlation, and nearby access correlation between pairs of memory locations. We describe methods for distinguishing between potential vs. realized affinity and for reasoning about affinity at multiple resolutions (3D, 2D, 1D). Finally, we construct spatial-temporal affinity signatures that classify memory behavior and that be used to reason about changes in software (data relayout, code refactoring) or hardware (caching, prefetching). We describe methods for signature visualization, interpretation, and quantitative comparison of signatures. We evaluate our methodology using applications with variants that contrast data structures, data layouts and algorithms. We show that spatial-temporal affinity analysis provides novel insights and enables predictive reasoning about application performance when contrasted with reuse distance analysis.

Suriyakumar, Yasodhadevi↗