25 Years of Cluster Computing at Los Alamos National Laboratory
Explore the source record for details and available documents.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
We present a summary of a recent workshop held at Duke University on Partonic Transverse Momentum in Hadrons: Quark Spin-Orbit Correlations and Quark-Gluon Interactions. The transverse momentum dependent parton distribution functions (TMDs), parton-to-hadron fragmentation functions, and multi-parton correlation functions, were discussed extensively at the Duke workshop. In this paper, we summarize first the theoretical issues concerning the study of partonic structure of hadrons at a future electron-ion collider (EIC) with emphasis on the TMDs. We then present simulation results on experimental studies of TMDs through measurements of single spin asymmetries (SSA) from semi-inclusive deep-inelastic scattering (SIDIS) processes with an EIC, and discuss the requirement of the detector for SIDIS measurements. The dynamics of parton correlations in the nucleon is further explored via a study of SSA in D (`D) production at large transverse momenta with the aim of accessing the unexplored tri-gluon correlation functions. The workshop participants identified the SSA measurements in SIDIS as a golden program to study TMDs in both the sea and valence quark regions and to study the role of gluons, with the Sivers asymmetry measurements as examples. Such measurements will lead to major advancement in our understanding of TMDs in the valence quark region, and more importantly also allow for the investigation of TMDs in the sea quark region along with a study of their evolution.
Cluster schedulers are crucial in high-performance computing (HPC). They determine when and which user jobs should be allocated to available system resources. Existing cluster scheduling heuristics are developed by human experts based on their experience with specific HPC systems and workloads. However, the increasing complexity of computing systems and the highly dynamic nature of application workloads have placed tremendous burden on manually designed and tuned scheduling heuristics. More aggressive optimization and automation are needed for cluster scheduling in HPC. In this work, we present an automated HPC scheduling agent named DRAS (Deep Reinforcement Agent for Scheduling) by leveraging deep reinforcement learning. DRAS is built on a hierarchical neural network incorporating special HPC scheduling features such as resource reservation and backfilling. An efficient training strategy is presented to enable DRAS to rapidly learn the target environment. Once being provided a specific scheduling objective given by the system manager, DRAS automatically learns to improve its policy through interaction with the scheduling environment and dynamically adjusts its policy as workload changes. We implement DRAS into a HPC scheduling platform called CQGym. CQGym provides a common platform allowing users to flexibly evaluate DRAS and other scheduling methods such as heuristic and optimization methods. Furthermore, the experiments using CQGym with different production workloads demonstrate that DRAS outperforms the existing heuristic and optimization approaches by up to 50%.
The use of computer clusters for computational sciences including computational physics is vital as it provides computing power to crunch big numbers at a faster rate. In compute intensive applications that requires high resolution such as Monte Carlo simulation, the use of computer clusters in a grid form that supplies computational power to any nodes within the grid that needs computing power, has now become a necessity. In this paper, we described how the clusters running on a specific application could use resources within the grid, to run the applications to speed up the computing process.
Sustainable and effective computing infrastructure depends critically on the skills and expertise of domain scientists and of committed and well-trained advanced computing professionals. But, in its ongoing High Performance Computing (HPC) work, Los Alamos National Laboratory noted a persistent shortage of well-prepared applicants, particularly for entry-level cluster administration, file systems administration, and high speed networking positions. Further, based upon recruiting efforts and interactions with universities graduating students in related majors of interest (e.g., computer science (CS)), there has been a long standing skillset gap, as focused training in HPC topics is typically lacking or absent in undergraduate and in even many graduate programs. Given that the effective operation and use of HPC systems requires specialized and often advanced training, that there is a recognized HPC skillset gap, and that there is intense global competition for computing and computational science talent, there is a long-standing and critical need for innovative approaches to help bridge the gap and create a well-prepared, next generation HPC workforce. Our paper places this need in the context of the HPC work and workforce requirements at Los Alamos National Laboratory (LANL) and presents one such innovative program conceived to address the need, bridge the gap, and grow an HPC workforce pipeline at LANL. The Computer System, Cluster, and Networking Summer Institute (CSCNSI) completed its 10th year in 2016. The story of the CSCNSI and its evolution is detailed below with a description of the design of its Boot Camp, and a summary of its success and some key factors that have enabled that success.
Supported subnano-cluster catalysts are highly dynamic, developing true active sites only under the pressures and temperatures of reaction conditions. Operando X-ray absorption near-edge structure (XANES) spectroscopy can track changes in the oxidation state and the local environment of cluster atoms, providing insight into the development of these active sites. While bulk metal, oxide, and hydroxide standards are often used for fitting experimental XANES spectra to obtain average oxidation states, we recently showed that computed cluster standards of relevant compositions are a more suitable basis, producing more accurate fits. Here, we theoretically interpret the operando XANES of supported Cu 3 Pd and Cu 4 clusters during temperature-programmed reaction (TPRx) of oxidative dehydrogenation of propane. We use an expanded basis set including both globally optimized computed clusters and bulk standards. Not only can we track reversible composition/oxidation state change with temperature, but also the irreversible growth of the bulk fraction upon heating, which we attribute to cluster sintering. This has important implications for the mechanism of the catalyzed reaction and the nature of the available active sites. Here, we propose that operando XANES provides most significant insight into the nature of supported cluster catalysts in reaction conditions when interpreted using mixed computed cluster and bulk standards.
We re-examine the problem of load balancing in conservatively synchronized parallel, discrete-event simulations executed on high-performance computing clusters, focusing on simulations where computational and messaging load tend to be spatially clustered. Such domains are frequently characterized by the presence of geographic 'hot-spots' - regions that generate significantly more simulation events than others. Examples of such domains include simulation of urban regions, transportation networks and networks where interaction between entities is often constrained by physical proximity. Noting that in conservatively synchronized parallel simulations, the speed of execution of the simulation is determined by the slowest (i.e most heavily loaded) simulation process, we study different partitioning strategies in achieving equitable processor-load distribution in domains with spatially clustered load. In particular, we study the effectiveness of partitioning via spatial scattering to achieve optimal load balance. In this partitioning technique, nearby entities are explicitly assigned to different processors, thereby scattering the load across the cluster. This is motivated by two observations, namely, (i) since load is spatially clustered, spatial scattering should, intuitively, spread the load across the compute cluster, and (ii) in parallel simulations, equitable distribution of CPU load is a greater determinant of execution speed than message passing overhead. Through large-scale simulation experiments - both of abstracted and real simulation models - we observe that scatter partitioning, even with its greatly increased messaging overhead, significantly outperforms more conventional spatial partitioning techniques that seek to reduce messaging overhead. Further, even if hot-spots change over the course of the simulation, if the underlying feature of spatial clustering is retained, load continues to be balanced with spatial scattering leading us to the observation that spatial scattering can often obviate the need for dynamic load balancing.
Real-time computing has traditionally been considered largely in the context of single-processor and embedded systems, and indeed, the terms real-time computing, embedded systems, and control systems are often mentioned in closely related contexts. However, real-time computing in the context of multinode systems, specifically high-performance, cluster-computing systems, remains relatively unexplored. Imposing real-time constraints on a parallel (cluster) computing environment introduces a variety of challenges with respect to the formal verification of the system's timing properties. In this paper, we give a motivating example to demonstrate the need for such a system--- an application to estimate the electromechanical states of the power grid--- and we introduce a formal method for performing verification of certain temporal properties within a system of parallel processes. We describe our work towards a full real-time implementation of the target application--- namely, our progress towards extracting a key mathematical kernel from the application, the formal process by which we analyze the intricate timing behavior of the processes on the cluster, as well as timing measurements taken on our test cluster to demonstrate use of these concepts.
With a longer-term goal of addressing the comparative behavior of the aqueous halides F−, Cl−, Br−, and I− on the basis of quasi-chemical theory (QCT), here we study structures and free energies of hydration clusters for those anions. We confirm that energetically optimal (H2O)nX clusters, with X = Cl−, Br−, and I−, exhibit surface hydration structures. Computed free energies, based on optimized surface hydration structures utilizing a harmonic approximation, typically (but not always) disagree with experimental free energies. To remedy the harmonic approximation, we utilize single-point electronic structure calculations on cluster geometries sampled from an AIMD (ab initio molecular dynamics) simulation stream. This rough-landscape procedure is broadly satisfactory and suggests unfavorable ligand crowding as the physical effect addressed. Nevertheless, this procedure can break down when n≳4, with the characteristic discrepancy resulting from a relaxed definition of clustering in the identification of (H2O)nX clusters, including ramified structures natural in physical cluster theories. With ramified structures, the central equation for the present rough-landscape approach can acquire some inconsistency. Extension of these physical cluster theories in the direction of QCT should remedy that issue, and should be the next step in this research direction.
An earlier report described a procedure for optimal sensor set selection and its implementation on a computational cluster. This new and innovative capability was developed to facilitate a reduction in operations staffing levels to improve plant economics. By automating surveillance and maintenance tasks through early detection of degrading sensors and equipment, staff can be more efficiently deployed. The method uses automated reasoning and domain knowledge in the form of the conservation equations to infer from plant measurements the state of equipment health. Inclusion of domain knowledge addresses the problem that exists with pure data-driven methods that there are no rigorous guidelines for determining what constitutes an adequate sensor set. Formalizing the procedure for sensor set selection as we have done results in a more reliable and explainable diagnosis of plant equipment health. Importantly, from the standpoint of the plant owner, personnel are provided with an early and explicit diagnosis of an equipment problem. That in principle automates the process and eliminates having to send personnel into the plant to find the cause as typically occurs when a data-driven method detects an anomaly. In this report we describe first results obtained using a computational cluster to solve the sensor set selection problem as framed above. The case described addresses the problem of equipment health monitoring in the high-pressure (HP) feedwater system of a pressurized light water reactor as seen through the eyes of our collaborating utility partner. Maintenance of this system can amount to millions of dollars per year if equipment health issues go undiagnosed and lead to loss of function. On examining the potential that is inherent in the installed sensor set for diagnosing equipment health degradation, it was found that greater fault resolution capability can be achieved using a sensor set that is 20 percent fewer in number. The take-away is that compared to the installed sensor set there exists a more strategic assignment of sensors that will furnish better health monitoring capability and with fewer sensors. Where the problem defies solution by manual inspection, as is the case here, one can be found by an algorithm. The solution was obtained in four hours using 30 computational cores. The HP feedwater problem as posed above illustrates the added value of approaching the sensor selection problem as one amenable to algorithmic solution. This problem is of interest to advanced reactor designers and to utilities that are setting up remote monitoring and diagnostic centers.
We examine cluster states transformed by stochastic local operations and classical communication, as a resource for deterministic universal computation driven strictly by projective measurements. We identify circumstances under which such states in one dimension constitute resources for random-length single-qubit rotations, in one case quasideterministically (N-U-N states) and in another probabilistically (B-U-B states). In contrast to the cluster states, the N-U-N states exhibit spin correlation functions that decay exponentially with distance, while the B-U-B states can be arbitrarily locally pure. A two-dimensional square N-U-N lattice is a universal resource for quasideterministic measurement-based quantum computation. Measurements on cubic B-U-B states yield two-dimensional cluster states with bond defects, whose connectivity exceeds the percolation threshold for a critical value of the local purity.
With the advent of big data, the I/O subsystems of large-scale compute clusters are becoming a center of focus, with more applications putting greater demands on end-to-end I/O performance. These subsystems are often complex in design. They comprise of multiple hardware and software layers to cope with the increasing capacity, capability and scalability requirements of data intensive applications. The sharing nature of storage resources and the intrinsic interactions across these layers make it to realize user-level, end-to-end performance gains a great challenge. We propose a topology-aware resource load balancing strategy to improve per-application I/O performance. We demonstrate the effectiveness of our algorithm on an extreme-scale compute cluster, Titan, at the Oak Ridge Leadership Computing Facility (OLCF). Our experiments with both synthetic benchmarks and a real-world application show that, even under congestion, our proposed algorithm can improve large-scale application I/O performance significantly, resulting in both the reduction of application run times and higher resolution simulation runs.
With the advent of big data, the I/O subsystems of large-scale compute clusters are becoming a center of focus. More applications are putting greater demands on end-to-end I/O performance. These subsystems are often complex in design. They comprise of multiple hardware and software layers to cope with the increasing capacity, capability, and scalability requirements of data intensive applications. However, the sharing nature of storage resources and the intrinsic interactions across these layers make it a great challenge to realize end-to-end performance gains. This paper proposes a topology-aware strategy to balance the load across resources, to improve the per-application I/O performance. We demonstrate the effectiveness of our algorithm on an extreme-scale compute cluster, Titan, at the Oak Ridge Leadership Computing Facility (OLCF). Our experiments with both synthetic benchmarks and a real-world application show that, even under congestion, our proposed algorithm can improve large-scale application I/O performance significantly, resulting in both a reduction in application run time as well as a higher resolution of simulation run.
Coupled-cluster Green’s function (GFCC) calculation has drawn much attention in the recent years for targeting the molecular and material electronic structure problems from a many-body perspective in a systematically improvable way. However, GFCC calculations on scientific computing clusters usually suffer from expensive higher di- mensional tensor contractions in the complex space, expensive inter-process communi- cation, and severe load imbalance, which limits it’s routine use for tackling electronic structure problems. Here we present a numerical library prototype that is specifically designed for large-scale GFCC calculations. The design of the library is focused on a systematically optimal computing strategy to improve its scalability and efficiency. The performance of the library is demonstrated by the relevant profiling analysis of running GFCC calculations on remote giant computing clusters. The capability of the library is highlighted by computing a wide near valence band of a fullerene C60 molecule for the first time at the GFCCSD level that shows excellent agreement with the experimental spectrum.
Coupled-cluster theory is a powerful tool for first-principles calculations of atomic nuclei, enabling accurate predictions of nuclear observables across the Segrè chart. While coupled-cluster computations are especially efficient at shell closures, extensions have been developed to tackle open-shell nuclei, by exploiting the equation-of-motion method or by expanding the coupled-cluster wave function on top of a symmetry-breaking (either deformed or superfluid) reference state. In this study, we provide a comprehensive comparison of these different formulations applied to the calcium and nickel isotopes using nuclear two-and three-body interactions from chiral effective field theory. Here, based on ground-state energies, two-neutron separation energies, and two-neutron shell gaps, different coupled-cluster computations—based on symmetry-broken reference states and equationof-motion techniques— offer consistent descriptions of bulk properties across medium-mass isotopic chains.
With February 11 marking the International Day of Women and Girls in Science, female physicists, engineers and computer scientists from CERN and Fermilab share their experiences of building careers in science. Krista Majewski came to Fermilab nine years ago as a software developer and now works in an operations role. She is a computing services specialist and part of the grid and cloud operations group, which manages computing clusters for the CMS experiment and other intensity frontier experiments at the lab. "My typical day would be maintaining the services that our group is responsible for and the machines that they run on. So if there are problems with a computing cluster where a user can't run their jobs, or they're trying to run on resources on the grid and they're unable to, then our group would be responsible for helping debug some of those issues," Krista says. Her path to programming at Fermilab has been winding. After receiving a degree in biomedical engineering, she went to work doing web development and programming for a large consulting firm, then moved to the financial industry. But programming called her back, and she earned a master's degree in computer science to start her new career. For Krista, programming is similar to something she enjoyed as child: solving puzzles. "I think it's a skill that you use in a lot of different ways," she says. Credits: Video production - Fermilab Creative Services Music Copyright: Universal Production Music Astral Light Chevalier Idea originally conceived by Tanya Levshina, Fermilab and Maria Dimou, CERN.