Workload-Adaptive Scheduling for Efficient Use of Parallel File Systems in High-Performance Computing Clusters
Results from work on feedback control of a queuing system for HPC jobs with shared storage resources.
Allan, Benjamin Andrew↗
SEARCH · Engineering Papers
Results for “Adaptive Scheduling”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Workload-adaptive Scheduling for Efficient Use of Parallel File Systems in High-performance Computing Clusters
Explore the source record for details and available documents.
Goponenko, Alexander V.↗
Quantum/AI Topology-Aware Latency-Adaptive HPC Workflow Scheduling Optimization
The growing demand for more powerful high-performance computing (HPC) systems has led to a steady rise in energy consumption by supercomputing worldwide. This study is focused on comparing our Application-Topology Mapper (ATMapper) to the popular Simple Linux Utility for Resource Management (SLURM) for the purpose of exploring methods that can further optimize job-scheduling within HPC systems. ATMapper is an Artificial-Intelligence based approach to job-scheduling that is currently being enhanced with quantum annealing (QA) to generate optimal schedules faster. We are applying QA to speedup our ATMapper process to achieve higher computing efficiency, thereby reducing HPC energy consumption. Here, we examine how four job-scheduling approaches perform in processor node assignment when using an example network architecture of 4 interconnected nodes. Using a specialized script, we are assessing the schedule of a computation flow with 11 interdependent tasks. The data movements among nodes were tracked to count for the number of interactions (network hops) between nodes needed to complete the tasks. The total number of hops and the job completion time were then used to quantify the efficiency of the different mapping approaches. In addition to SLURM, we also compare our ATMapper to the QA-enabled LBNL TIGER and the D-Wave Distributed Computing processor assignment approaches. The preliminary results showed that our topology-aware, latency-adaptive ATMapper is significantly more efficient when compared to the other scheduling approaches due to its load-imbalance network allocation. The scheduler displayed a computing efficiency of 53% by performing significantly fewer network hops than its alternatives. By reducing the number of hops, ATMapper was able to perform all 11 tasks by using only 3 nodes out of given 4. This research indicates the potential to use QA/AI for HPC job-scheduling. Later, we will test a SLURM simulator program to draw further comparisons on the effectiveness of ATMapper's scheduling approach. The results of this comparison will serve as a baseline for later improving SLURM's performance using a QA-enhanced ATMapper approach.
Caraveo, Braulio [University of Huston - Clear Lak↗
Adaptive PID Gain Scheduling Control for Hydropower Turbine Using Neural CDE and Stochastic Distribution Shaping
This paper introduces a gain-scheduling PID controller design strategy for hydroturbine frequency control mode. This scheme first uses real data to learn the nonlinear dynamics of the hydroturbine using neural controlled differential equations and then perturbs the obtained nonlinear system at different equilibrium points, based on which a static output feedback adaptive dynamic programming algorithm is then used to optimize the PID gains for each equilibrium point. Moreover, a continuous-time version of stochastic distribution control is proposed to further fine-tune the optimized PID gains. Finally, the controller is obtained by implementing linear interpolation between the optimized PID control gains. The simulation results show that the proposed gain-scheduling PID controller can control a larger range of operation points compared with the given fixed PID controller and the baseline method. Compared with the given fixed PID controller, the proposed gain-scheduling PID controller can regulate hydroturbine frequency against disturbances induced by power-load variation with over 50% less overshoot for some operation points.
13 HYDRO ENERGY↗
Adaptive Framework for Maintenance Scheduling Based on Dynamic Preventive Intervals and Remaining Useful Life Estimation
Data-based prognostic methods exploit sensor data to forecast the remaining useful life (RUL) of industrial settings to optimize the scheduling of maintenance actions. However, implementing sensors may not be cost-effective or practical for all components. Traditional preventive approaches are not based on sensor data; however, they schedule maintenance at equally spaced intervals, which is not a cost-effective approach since the distribution of the time between failures changes with the degradation state of other parts or changes in working conditions. This study introduces a novel framework comprising two maintenance scheduling strategies. In the absence of sensor data, we propose a novel dynamic preventive policy that adjusts intervention intervals based on the most recent failure data. When sensor data are available, a method for RUL prediction, designated k-LSTM-GFT, is enhanced to dynamically account for RUL prediction uncertainty. The results demonstrate that dynamic preventive maintenance can yield cost reductions of up to 51.8% compared to conventional approaches. The predictive approach optimizes the exploitation of RUL, achieving costs that are only 3–5% higher than the minimum cost achievable while ensuring the safety of critical systems since all of the failures are avoided.
Nunes, Pedro (ORCID:0000000180012172)↗
Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows
The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.
Etz, Brian [ORNL] (ORCID:0000000208554863)↗
Decentralized Distributed Proximal Policy Optimization (DD-PPO) for High Performance Computing Scheduling on Multi-User Systems
Resource allocation in High Performance Computing (HPC) environments presents a complex and multifaceted challenge for job scheduling algorithms. Beyond the efficient allocation of system resources, schedulers must account for and optimize multiple performance metrics, including job wait time and system throughput. Traditional heuristic-based scheduling algorithms increasingly struggle and lack the efficiency needed to meet the demands and address the complexity and scale of modern HPC systems. Consequently, recent research efforts have focused on leveraging advancements in Artificial Intelligence (AI) and Deep Learning (DL), particularly Reinforcement Learning (RL), to develop more adaptable and intelligent scheduling strategies. Previous RL-based scheduling approaches have explored a range of algorithms, from Deep Q-Networks (DQN) to Proximal Policy Optimization (PPO), and more recently, hybrid methods that integrate Graph Neural Networks (GNNs) with RL techniques. However, a common limitation across these methods is their reliance on relatively small datasets, with few methods being evaluated using large-scale, multi-million-job trace datasets representative of real-world HPC workloads. Moreover, existing RL schedulers face scalability issues due to centralized policy updates, which hinder training efficiency and performance when applied to large datasets. This study introduces a novel RL-based scheduler utilizing Decentralized Distributed Proximal Policy Optimization (DD-PPO) algorithm, which supports large-scale distributed training across multiple workers without requiring parameter synchronization at every step. By eliminating reliance on centralized updates to a shared policy, the DD-PPO scheduler enhances scalability, training efficiency, and sample utilization. Experimental validation using a large real-world dataset containing over 11.5 million job traces collected from petascale HPC systems over six years assesses the influence of dataset scale on training effectiveness and compares DD-PPO performance to traditional and advanced scheduling approaches. The experimental results demonstrate improved scheduling performance in comparison to both heuristic-based schedulers and existing RL-based scheduling algorithms.
AI↗
iDDS: intelligent distributed dispatch and scheduling for workflow orchestration
The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.
97 MATHEMATICS AND COMPUTING↗
Predictive Model for Starlink Maritime Performance Using Multi-Horizon RandomForest
Low Earth orbit (LEO) satellite systems have become a crucial enabler of broadband access for maritime industries, where traditional networks are unavailable. However, the high mobility of LEO constellations and constantly changing weather conditions result in unpredictable link fluctuations, limiting the ability of maritime platforms to plan bandwidth usage proactively. To the best of our knowledge, no prior work has developed a short-term predictive model for maritime LEO connectivity using real experimental field measurements. This paper proposes a data-driven forecasting model that predicts future downlink throughput using multi-horizon RandomForest regression. The model is trained using real experimental coastal measurement data incorporating recent throughput history, network-layer indicators, and environmental variables. The proposed approach reduces mean absolute error by approximately 31% compared to a persistence baseline for 15-minute horizons. It maintains a measurable improvement at 30 minutes, despite increased stochasticity. These findings confirm that proactive bandwidth awareness is feasible on maritime platforms and can effectively support operational decisions such as adaptive streaming, routing, and resource scheduling. The performance gap between forecasting horizons also highlights the need for expanded offshore datasets to improve prediction robustness under harsher maritime environments.
97 MATHEMATICS AND COMPUTING↗
Energy-efficient multimodal mobility networks in transportation digital twins: Strategies and optimization
The study proposes a comprehensive Transportation Mobility (TransitMo) framework covering conceptual design, model formulation, optimization, simulation, and impact analysis of the transportation mobility system. TransitMo is composed of a transportation digital twin developed in Simulation of Urban MObility (SUMO) and an Intelligent Traffic Management and Control Center (ITMCC) that identifies the best ways to improve the movement of people within urban areas using various modes of transportation. This study encompasses advanced modeling techniques, algorithms, and strategic testing to optimize energy efficiency and mobility in a multimodal shared mobility network. TransitMo’s practical applications are exemplified through a city-scaled simulation network in Chattanooga, TN, employing demographic data to analyze historical traffic patterns and forecast future demands. Central to this methodology are three models: the User Preference Model (UP), the Energy Consumption Model (EC), and the System Optimization Model (SO). These models work in concert to iteratively devise the optimal travel incentives and minimize the total system cost in a real-time manner. In conclusion, test results verified that the proposed adaptive incentive program and optimized bus scheduling can improve network performance by increasing public transit ridership.
42 ENGINEERING↗
Inertia estimation for power grids: A review of methods, challenges, and future prospects
The electric power grid is undergoing a significant transformation, shifting from traditional synchronous generators to inverter-based resources (IBRs) such as solar photovoltaics, wind turbines, and energy storage systems. This evolution leads to a reduction in system inertia, a critical attribute for maintaining frequency stability in response to disturbances. Consequently, the ability to monitor and estimate system inertia has become increasingly essential. This paper provides a comprehensive review of existing inertia estimation methodologies, analyzing them from multiple perspectives, including the types of data utilized, underlying estimation principles, operational modes, and system-wide applicability. A comparative summary table is included to distill commonalities and key characteristics across various studies. In addition, the paper examines practical implementations of inertia estimation across several major power systems worldwide, including the U.S. interconnections, the Nordic power system, and the U.K. grid. Key challenges are identified, particularly in estimating contributions from virtual inertia sources and load-induced inertia in increasingly converter-dominated networks. To address these emerging challenges, the paper proposes an integrated framework for real-time inertia estimation and monitoring. This framework encompasses critical components such as data acquisition, inertia estimation from both synchronous and non-synchronous sources, load-induced effects, optimization techniques, forecasting, and virtual inertia scheduling. Collectively, these elements enable dynamic, system-wide monitoring and adaptive control of grid inertia.
Inertia estimation↗
Multi-Agent Control Planes for Quantum Networks: A Scalable Architecture for Autonomous Quantum Internet Management
Quantum networks are expected to enable distributed quantum computing, secure communication, and global entanglement distribution. However, operating such networks presents significant challenges, including stochastic quantum processes, fragile entanglement resources, dynamic topology, and cross-layer control requirements. Current quantum network control architectures largely rely on centralized or hierarchical controllers inspired by classical software-defined networking (SDN). While effective for small testbeds, these approaches face scalability, latency, and reliability limitations as quantum networks grow. This paper proposes a multi-agent control plane architecture for quantum networks. In this design, intelligent software agents operate at quantum nodes, repeaters, and orchestration layers, collectively managing entanglement generation, routing, purification, and scheduling. The distributed intelligence of the agent system allows the network to adapt dynamically to quantum hardware variability and environmental noise. We argue that multi-agent systems provide significant advantages over centralized control approaches, including scalability, resilience, local autonomy, and real-time adaptation. The paper discusses architectural design principles, agent coordination mechanisms, and research challenges in deploying multi-agent control planes for the emerging quantum Internet.
Alnajjar, Anees [ORNL] (ORCID:0000000237101601)↗
Comparison of Routes of Administration, Frequency, and Duration of Favipiravir Treatment in Mouse and Guinea Pig Models of Ebola Virus Disease
Favipiravir is a ribonucleoside analogue that has been explored as a therapeutic for the treatment of Ebola Virus Disease (EVD). Promising data from rodent models has informed nonhuman primate trials, as well as evaluation in patients during the 2013–2016 West African EVD outbreak of favipiravir treatment. However, mixed results from these studies hindered regulatory approval of favipiravir for the indication of EVD. This study examined the influence of route of administration, duration of treatment, and treatment schedule of favipiravir in immune competent mouse and guinea pig models using rodent-adapted Zaire ebolavirus (EBOV). A dose of 300 mg/kg/day of favipiravir with an 8-day treatment was found to be fully effective at preventing lethal EVD-like disease in BALB/c mice regardless of route of administration (oral, intraperitoneal, or subcutaneous) or whether it was provided as a once-daily dose or a twice-daily split dose. Preclinical data generated in guinea pigs demonstrates that an 8-day treatment of 300 mg/kg/day of favipiravir reduces mortality following EBOV challenge regardless of route of treatment or duration of treatments for 8, 11, or 15 days. This work supports the future translational development of favipiravir as an EVD therapeutic.
60 APPLIED LIFE SCIENCES↗
Zero-Emission Transit Bus Needs Assessment
The transition to zero-emissions vehicles (ZEVs) in public transit has gained traction due to significant federal investments from the Bipartisan Infrastructure Law (BIL) and the Inflation Reduction Act (IRA). This needs assessment, commissioned by the Joint Office of Energy and Transportation and conducted by researchers at the Idaho National Laboratory, explores the current state of electrification in transit agencies, identifying barriers to implementation, potential funding sources, and operational considerations necessary for a successful transition. The assessment involved qualitative interviews with representatives from 19 transit service providers across diverse geographic regions. Key findings highlight the challenges related to bus facilities and operations, which require careful planning for charging infrastructure and maintenance capabilities to accommodate battery electric buses (BEBs) and hydrogen fuel cell buses (HFCBs). Agencies reported operational hurdles due to the shorter range of BEBs compared to diesel buses, necessitating revised scheduling and routing strategies. Despite these challenges, many agencies expressed optimism about their capacity to adapt. Funding availability emerged as a critical factor influencing the transition to ZEVs. While agencies welcomed increased financial support, particularly from the Low or No Emission Grant Program (Lo-No), concerns about the sustainability of this funding and the ongoing operational costs were prevalent. The need for a comprehensive funding inventory was underscored to ensure transit agencies are aware of all available resources. Technological constraints were significant barriers to ZEV adoption. The limited range of BEBs was frequently cited as a concern, leading to operational challenges and reliability issues. Agencies reported difficulties in sourcing replacement parts, which exacerbated downtime and maintenance challenges. Workforce development and training were identified as pivotal for a successful transition. Many agencies rely heavily on manufacturers for technician training, highlighting the need for scalable training programs that equip staff with the necessary skills to maintain electric powertrains effectively. This assessment offers actionable recommendations for the Joint Office, including enhancing outreach to transit agencies, developing resources for effective utility partnerships, and facilitating comprehensive training programs. Establishing a zero-emission bus evaluation program to track performance metrics such as cost, range, and reliability could provide valuable insights for transit agencies. The needs assessment provides a detailed examination of the challenges and opportunities facing transit agencies in their transition to zero-emissions bus fleets. By addressing these issues through targeted support, stakeholders can collaboratively work towards a cleaner, more sustainable public transportation system that benefits all communities.
33 - ADVANCED PROPULSION SYSTEMS↗
CRiSPPy: An advanced hydropower scheduling tool for the Colorado River Storage Project
The Western Area Power Administration (WAPA) plays a vital role in delivering reliable and cost-effective hydroelectric power to millions of customers across the western United States. The Colorado River Storage Project (CRSP) carries out WAPA’s mission in Arizona, Utah, Colorado, New Mexico, Nevada, Wyoming and Texas. Achieving this mission requires effective management of the Colorado River system, and depends on the use of advanced analytical tools and modeling methodologies. For many years, CRSP has relied on the Generation and Transmission Maximization Superlite (GTMax SL) model for its mid-term and long-term hydroscheduling needs. However, the evolving energy market, power system operations, environmental rules, and hydrology conditions, coupled with advancements in computational capabilities, have necessitated the development of a more modern and robust solution. This report introduces the Colorado River Storage Project Python-based (CRiSPPy) model, a new, advanced hydropower scheduling tool developed to address CRSP ever-evolving challenges. CRiSPPy represents a significant leap forward in our ability to model and optimize the operation of the Colorado River system. It incorporates state-of-the-art optimization algorithms, enhanced data management capabilities, and an advanced graphical user interface, providing WAPA CRSP personnel with unprecedented insights and decision-making support. This document details the development, capabilities, and implementation of CRiSPPy. It is intended to serve as a comprehensive resource for WAPA staff, stakeholders, and anyone interested in the future of hydropower scheduling in the Colorado River Basin. We are confident that CRiSPPy will enhance WAPA's mission while adapting to the challenges of a dynamic and increasingly complex environment. The version of CRiSPPy described in this report is the version 2.3. New versions of CRiSPPy will be developed as the tool keeps evolving to address CRSP challenges.
13 HYDRO ENERGY↗
Queue wait time prediction in high performance computing (HPC) systems
High Performance Computing (HPC) systems are critical enablers for groundbreaking scientific research across various domains. Efficient resource allocation, facilitated by job scheduling, is paramount for maximizing the utilization of HPC systems. However, the variability in wait times for queued jobs poses challenges for users, necessitating accurate job wait time estimation. This paper explores the influence of job characteristics, including job size (the number of nodes requested and walltime), the queue to which the job is submitted and other resource requirements, on job wait times in leadership-class HPC systems. Focusing on the Theta Cray XC40 and Polaris machines at Argonne National Laboratory, the study evaluates the performance of different supervised learning algorithms in predicting job wait times. It also evaluates the impact of data preprocessing, including outlier detection, Principal Component Analysis (PCA), and feature selection, on the performance of wait time prediction models. The findings reveal insights into the relationship between job characteristics and wait times, offering a foundation for optimizing resource allocation and enhancing user experience. The methodologies and tools developed in this study are adaptable to other leadership-class HPC systems, providing a valuable contribution to the broader HPC community aiming to improve job scheduling efficiency and user satisfaction.
Okafor, Nwamaka↗
Future foundries: A convergent manufacturing platform
This article introduces the Future Foundries platform developed at Oak Ridge National Laboratory, a first-generation research system designed to demonstrate convergent manufacturing. Convergent manufacturing brings together additive, subtractive, and transformative processes in a digitally interconnected environment to enable end-to-end production workflows. By linking traditionally discrete steps, convergent platforms accelerate production, improve repeatability, and support high-mix, low-volume manufacturing. The Future Foundries platform exemplifies this vision in practice by combining four modular, vendor-agnostic process cells that include robotic WAAM, induction heating, optical metrology, and machining, coordinated through an automated pallet handler and a ROS 2-based digital thread. This architecture provides the flexibility and scalability needed for agile production in small and medium-sized manufacturing enterprises and for field deployable manufacturing. Two case studies illustrate the platform’s capabilities. The first presents an integrated workflow for fabricating, transforming, and repairing critical replacement components, showing how consolidated thermal, additive, inspection, and machining operations reduce manual part handling and streamline process flow. The second case study highlights coordinated multi-part production enabled by automated pallet logistics and multi-cell scheduling. Together, these examples showcase convergent manufacturing as a practical and scalable strategy for strengthening domestic casting and forging capacity, improving supply-chain resilience, and enabling rapid, adaptable production of mission-critical components.
Convergent manufacturing↗
EASY-SHIFT v Alpha
The software is a generic, price- and load-responsive control algorithm integrating heat pumps with thermal energy storage. The algorithm leverages simple models of the system and easily accessible data to schedule operation of heat pumps and thermal energy storage in ways that minimize the cost of operating the heating/cooling system. This tool is specifically designed to be easy to interact with, and something that industry partners are able to adopt. There are two current state of the art approaches. Industry tends to develop very simple algorithms, with predetermined schedules that are not capable of changing operation in response to changes in operating environment. For example, a control designed to avoid high-price electricity from 5-8 PM will not be able to adapt if the high-price period changes to 4-9 PM. Academia commonly develops algorithms called Model predictive control (MPC). MPC requires extensive data and highly trained staff to develop a specific type of simulation model of the building, connect the building to optimization algorithms, and leverage powerful computers. Industry, with limited time/finance budgets for any project, is resistant to adopting MPC due to the associated high complexity and cost.
Grant, Peter [Lawrence Berkeley National Laborator↗