Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “cluster scheduling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Exploring Continuous Seismic Data at an Industry Facility Using Unsupervised Machine Learning

Seismic data recorded at industrial sites contain valuable information on anthropogenic activities. With advances in machine learning and computing power, new opportunities have emerged to explore the seismic wavefield in these complex environments. We applied two unsupervised machine learning algorithms to analyze continuous seismic data collected from an industrial facility in Texas, United States. The Uniform Manifold Approximation and Projection for Dimension Reduction algorithm was used to reduce the dimensionality of the data and generate 2D embeddings. Then, the Hierarchical Density-Based Spatial Clustering of Applications with Noise method was employed to automatically group these embeddings into distinct signal clusters. Our analysis of over 1400 hr (around 59 days) of continuous seismic data revealed five and seven signal clusters at two separate stations. At both stations, we identified clusters associated with background noise and vehicle traffic, with the latter’s temporal patterns aligning closely with the facility’s work schedule. Furthermore, the algorithms detected signal clusters from unknown sources and underline the ability of unsupervised machine learning for uncovering previously unrecognized patterns. Our analysis demonstrates the effectiveness of unsupervised approaches in examining continuous seismic data without requiring prior knowledge or pre-existing labels.

58 GEOSCIENCES↗

Smart thermostat data-driven U.S. residential occupancy schedules and development of a U.S. residential occupancy schedule simulator

Occupancy schedule is one of the key inputs in Building Energy Modeling (BEM) to reflect the interaction between buildings and occupants. Over the past decades, standardized occupancy schedules, developed mainly by engineering rule-of-thumb, have been widely used in BEM due to its simplicity and lack of real measured occupancy data. However, the BEM community has recognized their association with uncertainty and reliability in simulation results from BEM. This study introduces representative occupancy schedules in the U.S. residential buildings, derived from a large smart thermostat dataset and time-series K-means clustering, and an open-source tool to generate a stochastic residential occupancy schedule. Over 90,000 residential occupancy schedules were estimated from the ecobee Donate Your Data dataset. Then, the representative occupancy schedules were identified through clustering. This study further investigated the impacts of three parameters (day, house type, and state) on residential occupancy schedules. Then, a tool, the Residential Occupancy Schedule Simulator (ROSS), is developed using the representative occupancy schedules derived in this study. Details of this tool are presented in this paper. In conclusion, the derived representative occupancy schedules and the ROSS tool can help improve the energy modeling of residential buildings.

42 ENGINEERING↗

Enabling Low-Overhead HT-HPC Workflows at Extreme Scale using GNU Parallel

GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-throughput high-performance computing (HT-HPC) workflows. Through real-world examples, we highlight GNU Parallel’s performance across various HPC workloads, including GPU computing, container-based workloads, and node-local NVMe storage. Our results on two leading supercomputers, OLCF’s Frontier and NERSC’s Perlmutter, showcase GNU Parallel’s rapid process dispatching ability and its capacity to maintain low overhead even at extreme scales. We explore GNU Parallel’s application in massive parallel file transfers using a scheduled Data Transfer Node (DTN) cluster, emphasizing its broad utility in diverse scientific workflows. Beyond its direct application as a viable workflow manager, GNU Parallel can be employed in conjunction with other workflow systems as a "last-mile" parallelizing driver and as a quick prototyping tool to design and extract parallel profiles from application executions. We then argue that the potential for GNU Parallel to transform workflow management at extreme scales is substantial, paving the way for more efficient and effective scientific discoveries.

Maheshwari, Ketan↗

Grid Optimization (GO) Competition Platform

A software for a multi-challenge power-flow grid optimization competition was developed. The platform brings together high performance computing clusters, webservers, databases, competition datasets, schedulers, evaluation codes, and a multitude of language compilers and optimization solvers to host the competition. The original video announcing the competition, from former Secretary Perry, at: https://www.youtube.com/watch?v=hZwX3P9vS8M

Veeramany, Arun↗

Operational Dynamic Configuration Analysis

Sectors may combine or split within areas of specialization in response to changing traffic patterns. This method of managing capacity and controller workload could be made more flexible by dynamically modifying sector boundaries. Much work has been done on methods for dynamically creating new sector boundaries [1-5]. Many assessments of dynamic configuration methods assume the current day baseline configuration remains fixed [6-7]. A challenging question is how to select a dynamic configuration baseline to assess potential benefits of proposed dynamic configuration concepts. Bloem used operational sector reconfigurations as a baseline [8]. The main difficulty is that operational reconfiguration data is noisy. Reconfigurations often occur frequently to accommodate staff training or breaks, or to complete a more complicated reconfiguration through a rapid sequence of simpler reconfigurations. Gupta quantified a few aspects of airspace boundary changes from this data [9]. Most of these metrics are unique to sector combining operations and not applicable to more flexible dynamic configuration concepts. To better understand what sort of reconfigurations are acceptable or beneficial, more configuration change metrics should be developed and their distribution in current practice should be computed. This paper proposes a method to select a simple sequence of configurations among operational configurations to serve as a dynamic configuration baseline for future dynamic configuration concept assessments. New configuration change metrics are applied to the operational data to establish current day thresholds for these metrics. These thresholds are then corroborated, refined, or dismissed based on airspace practitioner feedback. The dynamic configuration baseline selection method uses a k-means clustering algorithm to select the sequence of configurations and trigger times from a given day of operational sector combination data. The clustering algorithm selects a simplified schedule containing k configurations based on stability score of the sector combinations among the raw operational configurations. In addition, the number of the selected configurations is determined based on balance between accuracy and assessment complexity.

Lai, Chok Fung↗

NASA Tech Briefs, July 2013

Dielectrophoresis-Based Particle Sensor Using Nanoelectrode Arrays; Multi-Dimensional Damage Detection for Surfaces and Structures; ULTRA: Underwater Localization for Transit and Reconnaissance Autonomy; Autonomous Cryogenic Leak Detector for Improving Launch Site Operations; Submillimeter Planetary Atmospheric Chemistry Exploration Sounder; Method for Reduction of Silver Biocide Plating on Metal Surfaces; Silicon Micromachined Microlens Array for THz Antennas; Forward-Looking IED Detector Ground Penetrating Radar; Fully Printed, Flexible, Phased Array Antenna for Lunar Surface Communication, Battery Charge Equalizer with Transformer Array; An Efficient, Highly Flexible Multi-Channel Digital Downconverter Architecture; Dimmable Electronic Ballast for a Gas Discharge Lamp; Conductive Carbon Nanotube Inks for Use with Desktop Inkjet Printing Technology; Enhanced Schapery Theory Software Development for Modeling Failure of Fiber-Reinforced Laminates; High-Performance, Low-Temperature-Operating, Long-Lifetime Aerospace Lubricants; Carbon Nanotube Microarrays Grown on Nanoflake Substrates; Differential Muon Tomography to Continuously Monitor Changes in the Composition of Subsurface Fluids; Microgravity Drill and Anchor System; 20 Granular Media-Based Tunable Passive Vibration Suppressor; 21 Miga Aero Actuator and 2D Machined Mechanical Binary Latch; Micro-XRF for In Situ Geological Exploration of Other Planets; Hydrogen-Enhanced Lunar Oxygen Extraction and Storage Using Only Solar Power; Uplift of Ionospheric Oxygen Ions During Extreme Magnetic Storms; Miniaturized, High-Speed, Modulated X-Ray Source; Hollow-Fiber Spacesuit Water Membrane Evaporator 25 High-Power Single-Mode 2.65-micrometers InGaAsSb/AlInGaAsSb Diode Lasers; Optical Device for Converting a Laser Beam Into Two Co-aligned but Oppositely Directed Beams; A Hybrid Fiber/Solid-State Regenerative Amplifier with Tunable Pulse Widths for Satellite Laser Ranging; X-Ray Diffractive Optics; SynGenics Optimization System (SynOptSys); 29 CFD Script for Rapid TPS Damage Assessment; radEq Add-On Module for CFD Solver Loci-CHEM; Science Opportunity Analyzer (SOA) Version 8; 30 Autonomous Byte Stream Randomizer; Distributed Engine Control Empirical/Analytical Verification Tools; Dynamic Server-Based KML Code Generator Method for Level-of-Detail Traversal of Geospatial Data; Automated Planning of Science Products Based on Nadir Overflights and Alerts for Onboard and Ground Processing; Linked Autonomous Interplanetary Satellite Orbit Navigation; Risk-Constrained Dynamic Programming for Optimal Mars Entry, Descent, and Landing; Scheduling Operations for Massive Heterogeneous Clusters; Deepak Condenser Model (DeCoM); Flight Software Math Library; Recirculating 1-K-Pot for Pulse-Tube Cryostats; 35 Method for Processing Lunar Regolith Using Microwaves; Wells for In Situ Extraction of Volatiles from Regolith (WIEVR); and Estimating the Backup Reaction Wheel Orientation Using Reaction Wheel Spin Rates Flight Telemetry from a Spacecraft.

Source record↗

Evaluating HPC Scheduling Strategies for Urgent Workloads

Scientific computing centers increasingly face workloads with diverse urgency requirements, driven by applications that demand rapid or even immediate execution. Appropriately configured scheduling policies can significantly improve both user satisfaction and overall cluster utilization. In this work, we present a systematic analysis of scheduler configurations under scenarios where a fraction of jobs have urgent computing needs. We evaluate multiple job scheduling simulators, develop a lightweight job-submission emulation framework, and create tools to analyze and visualize the resulting scheduling data. Our study identifies key trade-offs between responsiveness, fairness, and efficiency, and offers a set of practical scheduling configurations (particularly for Slurm) that can be tailored to HPC environments supporting mixed-urgency workloads.

Maheshwari, Ketan [ORNL] (ORCID:000000033800662X)↗

Predicting runtime and resource utilization of jobs on integrated cloud and HPC systems

Recent advances in virtualization technologies used in cloud computing offer performance that closely approaches bare-metal levels. Combined with specialized instance types and high-speed networking services for cluster computing, cloud platforms have become a compelling option for high-performance computing (HPC). However, most current batch job schedulers in HPC systems are designed for homogeneous clusters and make decisions based on limited information about jobs and system status. Scientists typically submit computational jobs to these schedulers with a requested runtime that is often over- or under-estimated. More accurate runtime predictions can help schedulers make better decisions and reduce job turnaround times. Here, they can also support decisions about migrating jobs to the cloud to avoid long queue wait times in HPC systems.

97 MATHEMATICS AND COMPUTING↗

Jet-Intracluster Medium Interactions of the Head Tail Radio Galaxy 3C 129

The 50 ksec XMM observations of the galaxy cluster 3C 129 were taken as scheduled, and the data are of good quality. We analyzed the data in the following way. After standard cleaning, we flat-fielded the XMM surface brightness maps. Combining the data from the EPIC MOS and PN Camera CCDs, we performed a cross-correlation analysis of the X-ray surface brightness distribution with the 1.4 GHz VLA radio map. We found evidence for cavities in the X-ray emitting Intra-Cluster Medium (ICM) associated with the radio tail of the head-tail radio galaxy 3C 129. This discovery is very interesting as it excludes the presence of a large fraction of thermal plasma in the radio tail. Together with the observation of an apparent pressure mismatch between the radio plasma and the ICM, and an upper limit on the magnetic field inside the radio tail (from the radio spectral indices map) the observation implies that the tail pressure is dominated either by low-energy electrons/positrons, or, by relativistic protons. Furthermore, we studied the energy spectrum of an X-ray "hot-spot" associated with the head of the radio galaxy 3C 129. It seems likely that the X-ray hot-spot originates from shocked gas in front of the radio galaxy. , The analysis turned out to be much more difficult than anticipated. The main reason is the lack of a comprehensive, publicly available background model that is key for the analysis of extended sources. Small groups like our do not have the man-power to come up with a background model themselves. We used the model from Read & Ponman (A&A 409, 395, 2003). However, the background subtracted X-ray surface brightness maps show a bright ring in the outer 20% of the camera. We tried to get rid of this ring and contacted the XMM helpdesk and Read & Ponman, the authors of the background paper. However, up to this day, we did not entirely succeed to remove the brightness enhancement at the outer parts of the camera. Unfortunately, our results are somewhat sensitive to the uncertainty, as the radio galaxy 3C 129 is very large and occupies a rather large fraction of the XMM field of view. We are now working on a paper describing the results. The paper will include a detailed discussion of the uncertainties associated with the non- perfect background subtraction.

Krawczynski, Henric S. W.↗

Oak Ridge Computing Academy: An HPC cluster deployment and management pilot

The High Performance Computing Technologies (HPCT) course is a hands-on High Performance Computing (HPC) cluster deployment and management training program offered as part of the International School for Advanced Studies (SISSA) and the International Center for Theoretical Physics (ICTP) Master in High Performance Computing (MHPC) specialization. Here, this training program introduces students to key concepts in cluster configuration. which include networking, software stack provisioning, job scheduling, and monitoring. The publicly available course materials feature several examples and underlying methods that are broadly applicable to cluster deployment and management. This paper discusses the design of a new workforce development program at the Oak Ridge National Laboratory that is based on HPCT, the Oak Ridge Computing Academy (ORCA). The ORCA pilot program was hosted by the Oak Ridge Leadership Computing Facility (OLCF) in Summer 2025. As a part of this discussion, HPCT and ORCA course contents and infrastructure are outlined, ORCA participant experiences are detailed, and potential opportunities for improvement are discussed.

Education↗

Preventive Power Outage Estimation Based on A Novel Scenario Clustering Strategy: Preprint

The increasing occurrence of extreme weather events is challenging the power grid operation. In front of the extreme weather, the system operator is responsible for estimating the power outage and scheduling the restoration resources. This paper proposes an outage evaluation framework to identify the possible unserved load profiles, vulnerable areas, and mobile energy adequacy. The predicted vulnerable lines of an outage prediction model tool are utilized to generate numerous faulted line scenarios. Next, each scenario's nodal unserved load profile is obtained by solving a three-phase restoration model that considers the schedule of repair crews and mobile energy resources. Then, a novel scenario clustering strategy is developed to cluster the unserved load profiles into multiple representative ones for straightforward analysis. Finally, case studies on a distribution system evaluate the damage level brought by extreme weather and verify the effectiveness of the proposed scenario clustering strategy.

mobile energy resources↗

A Hands-On Curriculum for Training in HPC Cluster Deployment and Management

This paper presents the design, methodology, and outcomes of the High-Performance Computing Technologies (HPCT) course, a hands-on training program focused on the system-side of HPC cluster deployment and administration. Delivered as part of the Master in High Performance Computing (MHPC) program, the course introduces students to key concepts in cluster configuration, including networking, software stack provisioning, job scheduling, and monitoring. Initially taught in person, the course was transitioned to an online format during the COVID-19 pandemic. This shift led to the development of openly available instructional material and a flipped-classroom approach that continues to support both in-person and hybrid delivery. All course materials are publicly available at www.hpc.temple.edu/mhpc/hpc-technology/index.html. By documenting the structure, infrastructure, and evolution of HPCT, this paper offers a model for accessible HPC system training that supports workforce development in computational science.

Posada Correa, Fernando [ORNL] (ORCID:000000022565↗

NAS Requirements Checklist for Job Queuing/Scheduling Software

The increasing reliability of parallel systems and clusters of computers has resulted in these systems becoming more attractive for true production workloads. Today, the primary obstacle to production use of clusters of computers is the lack of a functional and robust Job Management System for parallel applications. This document provides a checklist of NAS requirements for job queuing and scheduling in order to make most efficient use of parallel systems and clusters for parallel applications. Future requirements are also identified to assist software vendors with design planning.

Jones, James Patton↗

Combining Quick-Turnaround and Batch Workloads at Scale

NAS uses PBS Professional to schedule and manage the workload on Pleiades, an 11,000+ node 1B cluster. At this scale the user experience for quick-turnaround jobs can degrade, which led NAS initially to set up two separate PBS servers, each dedicated to a particular workload. Recently we have employed PBS hooks and scheduler modifications to merge these workloads together under one PBS server, delivering sub-1-minute start times for the quick-turnaround workload, and enabling dynamic management of the resources set aside for that workload.

Matthews, Gregory A.↗

A Photo Album of Earth Scheduling Landsat 7 Mission Daily Activities

Landsat7 is a member of a new generation of Earth observation satellites. Landsat7 will carry on the mission of the aging Landsat 5 spacecraft by acquiring high resolution, multi-spectral images of the Earth surface for strategic, environmental, commercial, agricultural and civil analysis and research. One of the primary mission goals of Landsat7 is to accumulate and seasonally refresh an archive of global images with full coverage of Earth's landmass, less the central portion of Antarctica. This archive will enable further research into seasonal, annual and long-range trending analysis in such diverse research areas as crop yields, deforestation, population growth, and pollution control, to name just a few. A secondary goal of Landsat7 is to fulfill imaging requests from our international partners in the mission. Landsat7 will transmit raw image data from the spacecraft to 25 ground stations in 20 subscribing countries. Whereas earlier Landsat missions were scheduled manually (as are the majority of current low-orbit satellite missions), the task of manually planning and scheduling Landsat7 mission activities would be overwhelmingly complex when considering the large volume of image requests, the limited resources available, spacecraft instrument limitations, and the limited ground image processing capacity, not to mention avoidance of foul weather systems. The Landsat7 Mission Operation Center (MOC) includes an image scheduler subsystem that is designed to automate the majority of mission planning and scheduling, including selection of the images to be acquired, managing the recording and playback of the images by the spacecraft, scheduling ground station contacts for downlink of images, and generating the spacecraft commands for controlling the imager, recorder, transmitters and antennas. The image scheduler subsystem autonomously generates 90% of the spacecraft commanding with minimal manual intervention. The image scheduler produces a conflict-free schedule for acquiring images of the "best" 250 scenes daily for refreshing the global archive. It then equitably distributes the remaining resources for acquiring up to 430 scenes to satisfy requests by international subscribers. The image scheduler selects candidate scenes based on priority and age of the requests, and predicted cloud cover and sun angle at each scene. It also selects these scenes to avoid instrument constraint violations and maximizes efficiency of resource usage by encouraging acquisition of scenes in clusters. Of particular interest to the mission planners, it produces the resulting schedule in a reasonable time, typically within 15 minutes.

Potter, William↗

Shaping the FutureWorkforce: Challenges and Lessons Learned in HPC Education from National Labs and Computing Centers

Workforce training at national laboratories and computing centers is essential and typically falls into two categories: foundational training for newcomers and advanced training for experienced users. Foundational topics—such as version control, build systems, and basic HPC usage—are largely transferable across institutions, while cluster-specific training varies due to differences in hardware, job schedulers, and local workflows. Training on emerging technologies is split between hardware-specific content and broadly applicable programming paradigms. Here, to reduce redundancy and increase impact, national labs, computing centers, and vendors are collaborating through initiatives like the HPC Training Working Group to share best practices, co-develop materials, and broaden outreach. These coordinated efforts aim to make HPC training more accessible, scalable, and consistent across the community.

HPC↗