Towards Access Pattern Aware Checkpointing For Kokkos Applications.
Abstract not provided.
Tan, Nigel↗
SEARCH · Engineering Papers
Results for “checkpointing”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Elastic Recovery from Self-Descriptive Checkpoint Files in HPC Applications
Explore the source record for details and available documents.
Computer science↗
Error containment for enabling local checkpoint and recovery
Various embodiments include a parallel processing computer system that detects memory errors as a memory client loads data from memory and disables the memory client from storing data to memory, thereby reducing the likelihood that the memory error propagates to other memory clients. The memory client initiates a stall sequence, while other memory clients continue to execute instructions and the memory continues to service memory load and store operations. When a memory error is detected, a specific bit pattern is stored in conjunction with the data associated with the memory error. When the data is copied from one memory to another memory, the specific bit pattern is also copied, in order to identify the data as having a memory error.
Cherukuri, Naveen↗
Aviation security screening optimizer for risk and throughput (ASSORT)
The increasing number of air travelers each year presents a challenge as many airports are near their capacity in terms of resources and space for passenger screening. Fortunately, advancements in technologies like next-generation millimeter wave scanning offer solutions to ease this strain. The focus remains on managing risk while enhancing the passenger experience for the traveling public. The risk model presented in this paper known as the Aviation Security Screening Optimizer for Risk and Throughput (ASSORT) is designed to assess risk-based approaches for passenger screening and checkpoint operations. Additionally, ASSORT is exploring various traveler categories — general, trusted, and trusted-plus — along with different checkpoint screening Concept of Operations tailored to each traveler type. For instance, travelers with a higher trust level may experience fewer screening technologies, resulting in quicker processing times at the checkpoint. The output of ASSORT provides a risk score for predefined threat scenarios, as well as the overall risk to the checkpoint, aircraft, and airport by traveler type. In conclusion, benefits of using this tool include assessing the trade-offs between the overall risk associated with checkpoints and the throughput rate of passengers screened. We show for example the impact that different passenger volumes at the checkpoint can have on risk.
99 GENERAL AND MISCELLANEOUS↗
The Calpain-7 protease functions together with the ESCRT-III protein IST1 within the midbody to regulate the timing and completion of abscission
The Endosomal Sorting Complexes Required for Transport (ESCRT) machinery mediates the membrane fission step that completes cytokinetic abscission and separates dividing cells. Filaments composed of ESCRT-III subunits constrict membranes of the intercellular bridge midbody to the abscission point. These filaments also bind and recruit cofactors whose activities help execute abscission and/or delay abscission timing in response to mitotic errors via the NoCut/Abscission checkpoint. We previously showed that the ESCRT-III subunit IST1 binds the cysteine protease Calpain-7 (CAPN7) and that CAPN7 is required for both efficient abscission and NoCut checkpoint maintenance. Here, we report biochemical and crystallographic studies showing that the tandem microtubule-interacting and trafficking (MIT) domains of CAPN7 bind simultaneously to two distinct IST1 MIT interaction motifs. Structure-guided point mutations in either CAPN7 MIT domain disrupted IST1 binding in vitro and in cells, and depletion/rescue experiments showed that the CAPN7-IST1 interaction is required for (1) CAPN7 recruitment to midbodies, (2) efficient abscission, and (3) NoCut checkpoint arrest. CAPN7 proteolytic activity is also required for abscission and checkpoint maintenance. Hence, IST1 recruits CAPN7 to midbodies, where its proteolytic activity is required to regulate and complete abscission.
59 BASIC BIOLOGICAL SCIENCES↗
An autoencoder compression approach for accelerating large-scale inverse problems
Abstract Partial differential equation (PDE)-constrained inverse problems are some of the most challenging and computationally demanding problems in computational science today. Fine meshes required to accurately compute the PDE solution introduce an enormous number of parameters and require large-scale computing resources such as more processors and more memory to solve such systems in a reasonable time. For inverse problems constrained by time-dependent PDEs, the adjoint method often employed to compute gradients and higher order derivatives efficiently requires solving a time-reversed, so-called adjoint PDE that depends on the forward PDE solution at each timestep. This necessitates the storage of a high-dimensional forward solution vector at every timestep. Such a procedure quickly exhausts the available memory resources. Several approaches that trade additional computation for reduced memory footprint have been proposed to mitigate the memory bottleneck, including checkpointing and compression strategies. In this work, we propose a close-to-ideal scalable compression approach using autoencoders to eliminate the need for checkpointing and substantial memory storage, thereby reducing the time-to-solution and memory requirements. We compare our approach with checkpointing and an off-the-shelf compression approach on an earth-scale ill-posed seismic inverse problem. The results verify the expected close-to-ideal speedup for the gradient and Hessian-vector product using the proposed autoencoder compression approach. To highlight the usefulness of the proposed approach, we combine the autoencoder compression with the data-informed active subspace (DIAS) prior showing how the DIAS method can be affordably extended to large-scale problems without the need for checkpointing and large memory.
Mathematics↗
Stability-preserving Lossy Compression for Large-scale Partial Differential Equations
Checkpoint/Restart (C/R) strategies are vital for fault tolerance in PDE-based scientific simulations, yet traditional checkpointing incurs significant I/O overhead. Lossy compression offers a scalable solution by reducing checkpoint data size, but conventional methods often lack control over physical invariants (e.g., energy), leading to instability such as oscillations or divergence in Partial Differential Equations (PDE) systems. This paper introduces a stability-preserving compression approach tailored for PDE simulations by explicitly controlling kinetic and potential energy perturbations to ensure stable restarts. Extensive experiments conducted across diverse PDE configurations demonstrate that our method maintains numerical stability with minimal error magnification—even across multiple checkpoint-restart cycles—outperforming state-of-the-art lossy compressors. Parallel evaluations on the Frontier supercomputer show up to 8.4× improvement in checkpoint write performance and 6.3× in read performance, while maintaining relative L2 errors ∼ 2e-6 throughout continued simulation. These results provide practical guidance for balancing compression accuracy, stability, and computational efficiency in large-scale PDE applications.
Gong, Qian [ORNL] (ORCID:0000000235704142)↗
Hindsight logging for model training
In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive conservative logging, or letting training run fast via minimalist optimistic logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint---a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch.
Computer Science↗
Implementing Software Resiliency in HPX for Extreme Scale Computing
The DOE Office of Science Exascale Computing Project (ECP) outlines the next milestones in the supercomputing domain. The target computing systems under the project will deliver 10x performance while keeping the power budget under 30 megawatts. With such large machines, the need to make applications resilient has become paramount. The benefits of adding resiliency to mission critical and scientific applications, includes the reduced cost of restarting the failed simulation both in terms of time and power. Most of the current implementation of resiliency at the software level makes use of a Coordinated Checkpoint and Restart (C/R). This technique of resiliency generates a consistent global snapshot, also called a checkpoint. Generating snapshots involves global communication and coordination and is achieved by synchronizing all running processes. The generated checkpoint is then stored in some form of persistent storage. On failure detection, the runtime initiates a global rollback to the most recent previously saved checkpoint. This involves aborting all running processes, rolling them back to the previous state and restarting them.
97 MATHEMATICS AND COMPUTING↗
Tactical Analysis for Calculating Contextual Risk at Boundaries: Summary of Laboratory Directed Research & Development Effort
The Tactical Analysis for Calculating Contextual Risk at Boundaries (TACCRAB) tool is an innovative digital twin (DT) platform and automated risk algorithm designed to transform operational decision-making in structured screening environments, with an initial focus on Southern Border Land Ports of Entry (POEs). The invention provides integration points for advanced artificial intelligence, predictive modeling, and real-time data analysis to produce a comprehensive risk management tool that enables proactive, data-informed security strategies. The core inventive features of TACCRAB center on its unique risk algorithm, which dynamically calculates contextual risk by synthesizing historical data, near real-time streaming data from the checkpoints themselves, and AI-generated predictions. Unlike traditional risk assessment methods, TACCRAB utilizes a DT to provide comprehensive operational insights, allowing stakeholders to visualize, simulate, and optimize checkpoint configurations with unprecedented speed and contextual awareness. TACCRAB's key innovation lies in its ability to combine multiple complex inputs - including technology detection probabilities, resource availability, screening pathway characteristics, and threat actor behavioral patterns - into a unified risk calculation and update these inputs based on changing operational and environmental conditions. By leveraging a DT that continuously updates and learns from linked data, TACCRAB can suggest adaptive mitigation strategies that minimize risk while maintaining operational efficiency. Particularly novel is the platform's approach to decision support, which goes beyond static risk assessment. The DT provides dynamic metrics such as wait times, resource allocation effectiveness, and potential emerging threat scenarios, enabling users to view sophisticated, relevant what-if simulations and optimize checkpoint operations in near real-time. The system's architecture allows for generalized application across different screening environments, such as secure facilities, ports of entry, and soft targets, making it a versatile tool for security and operational management. The invention distinguishes itself through its comprehensive integration of predictive modeling, AI-driven pattern discovery, and user-friendly interface design. By combining these elements, TACCRAB transforms complex risk data into actionable insights, supporting decision-makers at various organizational levels - from booth agents making split-second screening decisions to checkpoint managers optimizing the day's resource allocation to strategic planners managing long-term investments.
45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
Metadata compression
A computer-executable method, system, and computer program product for managing a Parallel-Logged File System (PLFS) on a data storage system, wherein the data storage system is in communication with an application that updates the data storage system with checkpoint data, the computer-executable method, system, and computer program product comprising receiving a first checkpoint including data from the application; analyzing the data in the first checkpoint to determine whether a pattern exists in the checkpoint data; and upon a positive determination, adding the pattern into PLFS.
97 MATHEMATICS AND COMPUTING↗
Shortwave‐Infrared‐Emitting Nanoprobes for CD8 Targeting and In Vivo Imaging of Cytotoxic T Cells in Breast Cancer
Checkpoint immunotherapy has made great strides in the treatment of solid tumors, but many patients do not respond to immune checkpoint inhibitors. Identification of tumor‐infiltrating cytotoxic T cells (CTLs) has the potential to stratify patients and monitor immunotherapy responses. In this study, the design of cluster of differentiation (CD8 + ) T cell‐targeted nanoprobes that emit shortwave infrared (SWIR) light in the second tissue‐transparent window for noninvasive, real‐time imaging of CTLs in murine models of breast cancer is presented. SWIR‐emitting rare‐earth nanoparticles encapsulated in human serum albumin are conjugated with anti‐CD8α to target CTLs with high specificity. CTL targeting is validated in vitro through binding of nanoprobes to primary mouse CTLs. The potential for the use of SWIR fluorescence intensity to determine CTL presence is validated in two syngeneic mammary fat pad tumor models, EMT6 and 4T1, which differ in immune infiltration. SWIR imaging using CD8‐targeted nanoprobes successfully identifies the presence of CTLs in the more immunogenic EMT6 model, while imaging confirms the lack of substantial immune infiltration in the nonimmunogenic 4T1 model. In this work, the opportunity for SWIR imaging using CD8‐targeted nanoprobes to assess CTL infiltration in tumors for the stratification and monitoring of responders to checkpoint immunotherapy is highlighted.
Shah, Jay V.↗
Current Status of Clinical Trials for Cervical and Uterine Cancer Using Immunotherapy Combined With Radiation
Novel therapies combined with radiation continue to be of significant interest in the developmental treatment paradigm of gynecologic cancers. Clinical implementation of immunotherapy in oncology has rapidly changed the treatment landscape, options, paradigm, and outcomes through clinical trials. Immunotherapy has emerged as a therapeutic pillar in the treatment of solid tumors with demonstrable synergistic activity when combined with radiation therapy and chemoradiotherapy by an alteration or enhancement of the immune system. In solid tumors, radiation therapy induces migration of dendritic cells, T cell activation, and proliferation, and increases in tumor-infiltrating lymphocytes. These immunomodulatory effects in conjunction with immune checkpoint blockade are currently under active investigation in the adjuvant, definitive, and metastatic settings. Results from early phase trials demonstrate promising efficacy and overall tolerable toxicity profiles of combined modality treatment. There is significant interest in optimizing the treatment for patients with locally advanced cervical cancer beyond the standard of care—chemoradiation—which has been in place for the last 30 years. The majority of cervical cancer emerges after persistent infection with a high-risk subtype of the human papillomavirus, where viral oncoproteins lead to cellular changes and immortalization. As a result, immune tolerance can develop, resulting in cancer. Knowledge of the mechanism of human papillomavirus–related oncogenesis suggests that immune therapy or checkpoint blockade can reinvigorate an antitumor immune response. Current clinical trials are exploring the therapeutic potential of these approaches. Uterine cancers have been grouped into 4 molecular subclasses by their driver mutations, mutational burden, and copy-number alterations. Of these subgroups, the polymerase epsilon–mutated and microsatellite-unstable may represent up to 40% of endometrial cancers, and they have been shown to be immunogenic. Because of the inherent immunogenicity of these MSI-high tumors, combined immune modulation strategies, including chemotherapy, radiation, and immunotherapy and immune checkpoint inhibitor therapy, are being explored to improve treatment outcomes. In this review, we explore current immunomodulatory and multimodality therapeutic approaches in the treatment of cervical and uterine cancer through ongoing clinical trials investigating the combination of immunotherapy and radiation therapy.
62 RADIOLOGY AND NUCLEAR MEDICINE↗
Graph-based Reversible Evaluation and Tangents Library
GRETL is a C++ library for evaluation, re-evaluation and algorithmic differentiation of functional operations on an arbitrary computational graph with limited memory usage. Similar to popular machine learning frameworks in Python, like PyTorch and JAX, it tracks and stores both operations and output data as functions are evaluated. Once this composition of functions is built up, the entire chain of operations can be back propagated to compute sensitivities of the final result with respect to any number of inputs. In contrast to most machine learning applications, memory usage becomes the bottleneck for back propagation in many physics applications, especially for time-dependent PDEs. Dynamic check pointing becomes essential. An important distinguishing feature of GRETL is its ability to limit the maximum memory usage by automatically dynamic checkpointing the data output for each graph operation (see Wang, Moin, Iaccarino, 2009). During backpropagation, parts of the graph that are no longer in memory are automatically re-evaluated from upstream checkpointed states as needed for derivative sensitivity calculations (or more precisely, for vector-Jacobian products). GRETL is particularly beneficial for applications, such as coupled multi-physics, where deriving adjoint-based sensitivities and managing checkpoint memory across modules becomes onerous. Cases which can be readily handled by the GRETL library include: different time-integration algorithms per physics (e.g., coupled predictor-corrector algorithms, IMEX, etc.), sub-cycling, asynchronous integrators, state dependent timestep sizes, iterative solvers and coupling algorithms, controller algorithms, and more.
Tupek, MichaelR [Lawrence Livermore National Labor↗
Resiliency in numerical algorithm design for extreme scale simulations
Here this work is based on the seminar titled ‘Resiliency in Numerical Algorithm Design for Extreme Scale Simulations’ held March 1–6, 2020, at Schloss Dagstuhl, that was attended by all the authors. Advanced supercomputing is characterized by very high computation speeds at the cost of involving an enormous amount of resources and costs. A typical large-scale computation running for 48 h on a system consuming 20 MW, as predicted for exascale systems, would consume a million kWh, corresponding to about 100k Euro in energy cost for executing 10 23 floating-point operations. It is clearly unacceptable to lose the whole computation if any of the several million parallel processes fails during the execution. Moreover, if a single operation suffers from a bit-flip error, should the whole computation be declared invalid? What about the notion of reproducibility itself: should this core paradigm of science be revised and refined for results that are obtained by large-scale simulation? Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to background storage at frequent intervals will create intolerable overheads in runtime and energy consumption. Forecasts show that the mean time between failures could be lower than the time to recover from such a checkpoint, so that large calculations at scale might not make any progress if robust alternatives are not investigated. More advanced resilience techniques must be devised. The key may lie in exploiting both advanced system features as well as specific application knowledge. Research will face two essential questions: (1) what are the reliability requirements for a particular computation and (2) how do we best design the algorithms and software to meet these requirements? While the analysis of use cases can help understand the particular reliability requirements, the construction of remedies is currently wide open. One avenue would be to refine and improve on system- or application-level checkpointing and rollback strategies in the case an error is detected. Developers might use fault notification interfaces and flexible runtime systems to respond to node failures in an application-dependent fashion. Novel numerical algorithms or more stochastic computational approaches may be required to meet accuracy requirements in the face of undetectable soft errors. These ideas constituted an essential topic of the seminar. The goal of this Dagstuhl Seminar was to bring together a diverse group of scientists with expertise in exascale computing to discuss novel ways to make applications resilient against detected and undetected faults. In particular, participants explored the role that algorithms and applications play in the holistic approach needed to tackle this challenge. This article gathers a broad range of perspectives on the role of algorithms, applications and systems in achieving resilience for extreme scale simulations. The ultimate goal is to spark novel ideas and encourage the development of concrete solutions for achieving such resilience holistically.
79 ASTRONOMY AND ASTROPHYSICS↗
Checkpoint/Restart Vision and Strategies for NERSC’s Production Workloads
As a primary approach to fault-tolerant computing, Checkpoint/Restart (C/R) improves scientific productivity for users, provides scheduling flexibility for computing centers, and protects against system failures. While both applicationspecific (or application-level) and transparent C/R are used in practice, we are interested in transparent checkpointing, which is vital for system-level checkpointing. Developing and maintaining transparent C/R tools for HPC applications, however, is labor intensive and highly complex due to ever-changing HPC systems and diverse production workloads. Existing C/R tools are often research-oriented, so there is a gap to close before they can be used reliably with production workloads, especially on cutting edge HPC systems. In this position paper, we present our journey to prepare a production-ready MPI-Agnostic Network-Agnostic (MANA) transparent checkpointing tool for NERSC, and share our vision and strategies to bring transparent C/R capabilities to NERSC’s production workloads on current and future systems.
42 ENGINEERING↗
MAPSTER: Automated Geospatial Data Sharing – Version 1.4.0
The US Department of Energy’s (DOE) Oak Ridge National Laboratory (ORNL) developed MAPSTER which is a geospatial data management tool that aggregates, organizes, and shares data from dispersed sources such as unmanned aerial systems (UAS). Built specifically for use in environments where communications may be limited, MAPSTER utilizes two key technologies to effectively manage data in the field and enable easy data sharing with authorized partners: Observer and Checkpoint. Observer is a lightweight software package on an edge device, such as a laptop, that automatically detects newly processed UAS data and sends to a central server called Checkpoint. Checkpoint is a centralized server at ORNL that receives and manages data from all Observer instances. Even in a very low bandwidth environment, Observer can still send information about the UAS data product almost instantly as it generates its own metadata package on the size of KB (kilobytes). MAPSTER is not only for UAS data but for any geospatial data collected at the austere edge and dispersed sources.
97 MATHEMATICS AND COMPUTING↗
DenKv: Addressing Design Trade-offs of Key-value Stores for Scientific Applications
High-performance computing (HPC) facilities have employed flash-based storage tier near to compute nodes to absorb high I/O demand by HPC applications during periodic system-level checkpoints. To accelerate these checkpoints, proxy-based distributed key-value stores (PD-KVS) gained particular attention for their flexibility to support multiple backends and different network configurations. PD-KVS rely internally on monolithic KVS, such as LevelDB or RocksDB, to exploit the KV interface and query support. However, PD-KVS are unaware of the high redundancy factor in checkpoint data, which can be up to GBs to TBs, and therefore, tend to generate high write and space amplification on these storage layers. In this paper, we propose DenKv which is deduplication-extended node-local LSM-tree-based KVS. DenKv employs asynchronous partially inline dedup (APID) and aims to maintain the performance characteristics of LSM-tree-based KVS while reducing the write and space amplification problems. We implemented DenKv atop BlobDB and showed that our proposed solution maintains performance while reducing write amplification up to 2× and space amplification by 4× on average.
Khan, Awais↗