Leveraging Cluster Computing for Large-Scale Data Processing and Automated Analysis for Radiation Portal Monitor Data
Explore the source record for details and available documents.
Smith, Jacob↗
SEARCH · Engineering Papers
Results for “cluster computing”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
DRAS: Deep Reinforcement Learning for Cluster Scheduling in High Performance Computing
Cluster schedulers are crucial in high-performance computing (HPC). They determine when and which user jobs should be allocated to available system resources. Existing cluster scheduling heuristics are developed by human experts based on their experience with specific HPC systems and workloads. However, the increasing complexity of computing systems and the highly dynamic nature of application workloads have placed tremendous burden on manually designed and tuned scheduling heuristics. More aggressive optimization and automation are needed for cluster scheduling in HPC. In this work, we present an automated HPC scheduling agent named DRAS (Deep Reinforcement Agent for Scheduling) by leveraging deep reinforcement learning. DRAS is built on a hierarchical neural network incorporating special HPC scheduling features such as resource reservation and backfilling. An efficient training strategy is presented to enable DRAS to rapidly learn the target environment. Once being provided a specific scheduling objective given by the system manager, DRAS automatically learns to improve its policy through interaction with the scheduling environment and dynamically adjusts its policy as workload changes. We implement DRAS into a HPC scheduling platform called CQGym. CQGym provides a common platform allowing users to flexibly evaluate DRAS and other scheduling methods such as heuristic and optimization methods. Furthermore, the experiments using CQGym with different production workloads demonstrate that DRAS outperforms the existing heuristic and optimization approaches by up to 50%.
97 MATHEMATICS AND COMPUTING↗
Cluster Node Computing for Target Generation Systems in Aircraft Simulations
Target generation systems provide the positions of aircraft in air traffic simulations. As the scope of the simulation domains expand, there is a need to develop systems that can provide position reports for thousands of aircraft simultaneously and at update rates that support out-the-window visualization. This paper discusses the motivation and reasoning behind investigating development of a next generation target generator through distributed computing using clustered node processing and how a target generation system benefit future research that utilizes human-in-the-loop simulations.
air traffic simulations↗
Cluster Node Computing for Target Generation Systems in Aircraft Simulations
Target generation systems provide virtual position and state reports of aircraft for air traffic simulations. As the scope and scale of the simulation domains expand, there is a need to develop systems that can generate position reports for thousands of simulated aircraft simultaneously at high update rates that support out-the-window visualization. This paper discusses the motivation and reasoning behind investigating development of a next generation target generator through distributed computing using clustered node processing and the benefits such a target generation system has on future research that utilizes human-in-the-loop simulations.
Monheim, Spencer↗
An Innovative Approach to Bridge a Skill Gap and Grow a Workforce Pipeline: The Computer System, Cluster, and Networking Summer Institute
Sustainable and effective computing infrastructure depends critically on the skills and expertise of domain scientists and of committed and well-trained advanced computing professionals. But, in its ongoing High Performance Computing (HPC) work, Los Alamos National Laboratory noted a persistent shortage of well-prepared applicants, particularly for entry-level cluster administration, file systems administration, and high speed networking positions. Further, based upon recruiting efforts and interactions with universities graduating students in related majors of interest (e.g., computer science (CS)), there has been a long standing skillset gap, as focused training in HPC topics is typically lacking or absent in undergraduate and in even many graduate programs. Given that the effective operation and use of HPC systems requires specialized and often advanced training, that there is a recognized HPC skillset gap, and that there is intense global competition for computing and computational science talent, there is a long-standing and critical need for innovative approaches to help bridge the gap and create a well-prepared, next generation HPC workforce. Our paper places this need in the context of the HPC work and workforce requirements at Los Alamos National Laboratory (LANL) and presents one such innovative program conceived to address the need, bridge the gap, and grow an HPC workforce pipeline at LANL. The Computer System, Cluster, and Networking Summer Institute (CSCNSI) completed its 10th year in 2016. The story of the CSCNSI and its evolution is detailed below with a description of the design of its Boot Camp, and a summary of its success and some key factors that have enabled that success.
97 MATHEMATICS AND COMPUTING↗
NASA Multidimensional Stirling Convertor Code Developed
A high-efficiency Stirling Radioisotope Generator (SRG) for use on potential NASA Space Science missions is being developed by the Department of Energy, Lockheed Martin, Stirling Technology Company, and the NASA Glenn Research Center. These missions may include providing spacecraft onboard electric power for deep space missions or power for unmanned Mars rovers. Glenn is also developing advanced technology for Stirling convertors, aimed at substantially improving the specific power and efficiency of the convertor and the overall power system. Performance and mass improvement goals have been established for second- and third-generation Stirling radioisotope power systems. Multiple efforts are underway to achieve these goals, both in house at Glenn and under various grants and contracts. These efforts include the development of a multidimensional Stirling computational fluid dynamics code, high-temperature materials, advanced controllers, an end-to-end system dynamics model, low-vibration techniques, advanced regenerators, and a lightweight convertor. Under a NASA grant, Cleveland State University (CSU) and its subcontractors, the University of Minnesota (UMN) and Gedeon Associates, have developed a twodimensional computer simulation of a CSUmod Stirling convertor. The CFD-ACE commercial software developed by CFD Research Corp. of Huntsville, Alabama, is being used. The CSUmod is a scaled version of the Stirling Technology Demonstrator Convertor (TDC), which was designed and fabricated by the Stirling Technology Company and is being tested by NASA. The schematic illustrates the structure of this model. Modeled are the fluid-flow and heat-transfer phenomena that occur in the expansion space, the heater, the regenerator, the cooler, the compression space, the surrounding walls, and the moving piston and displacer. In addition, the overall heat transfer, the indicated power, and the efficiency can be calculated. The CSUmod model is being converted to a two-dimensional model of the TDC at NASA Glenn. Validation of the multidimensional Stirling code is an important part of the grant effort. UMN has been generating data in an oscillating-flow test facility using two different test sections: a 90 turn and a cooler/regenerator/heater test section. CSU has created computational fluid dynamics models of both these test sections and has been making comparisons with the data, then improving their models to improve the agreement with the test data. CSU has also been using data available in the literature for code validation. UMN is now preparing to begin fabrication of a new 180 turn test section that will be more representative of certain portions of the Stirling engine geometry. Simulations to almost periodic steady state with the two-dimensional CSUmod model indicate that, to reach periodic steady state on a single 2-GHz desktop computer, 75 to 100 complete simulation cycles would be required and between 1 and 2 months of computer time. Therefore, Glenn has purchased the first 8 computers, of a 64-computer cluster, to be run in parallel to accelerate the simulation. On the basis of CFD Research Corp.'s experience with running the parallelized version of CFD-ACE on their clusters, we estimate that the complete 64-computer cluster will reduce simulation computing time by a factor of about 40. Plans are to continue development of these multidimensional Stirling codes and to use them to study the fluid-flow and heat-transfer phenomena that occur inside Stirling convertors. This is expected to lead to improved thermodynamic loss understanding, onedimensional design and performance codes, and engine performance.
Tew, Roy C.↗
Interpreting the Operando X-ray Absorption Near-Edge Structure of Supported Cu and CuPd Clusters in Conditions of Oxidative Dehydrogenation of Propane: Dynamic Changes in Composition and Size
Supported subnano-cluster catalysts are highly dynamic, developing true active sites only under the pressures and temperatures of reaction conditions. Operando X-ray absorption near-edge structure (XANES) spectroscopy can track changes in the oxidation state and the local environment of cluster atoms, providing insight into the development of these active sites. While bulk metal, oxide, and hydroxide standards are often used for fitting experimental XANES spectra to obtain average oxidation states, we recently showed that computed cluster standards of relevant compositions are a more suitable basis, producing more accurate fits. Here, we theoretically interpret the operando XANES of supported Cu 3 Pd and Cu 4 clusters during temperature-programmed reaction (TPRx) of oxidative dehydrogenation of propane. We use an expanded basis set including both globally optimized computed clusters and bulk standards. Not only can we track reversible composition/oxidation state change with temperature, but also the irreversible growth of the bulk fraction upon heating, which we attribute to cluster sintering. This has important implications for the mechanism of the catalyzed reaction and the nature of the available active sites. Here, we propose that operando XANES provides most significant insight into the nature of supported cluster catalysts in reaction conditions when interpreted using mixed computed cluster and bulk standards.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Automated Production of Movies on a Cluster of Computers
A method of accelerating and facilitating production of video and film motion-picture products, and software and generic designs of computer hardware to implement the method, are undergoing development. The method provides for automation of most of the tedious and repetitive tasks involved in editing and otherwise processing raw digitized imagery into final motion-picture products. The method was conceived to satisfy requirements, in industrial and scientific testing, for rapid processing of multiple streams of simultaneously captured raw video imagery into documentation in the form of edited video imagery and video derived data products for technical review and analysis. In the production of such video technical documentation, unlike in production of motion-picture products for entertainment, (1) it is often necessary to produce multiple video derived data products, (2) there are usually no second chances to repeat acquisition of raw imagery, (3) it is often desired to produce final products within minutes rather than hours, days, or months, and (4) consistency and quality, rather than aesthetics, are the primary criteria for judging the products. In the present method, the workflow has both serial and parallel aspects: processing can begin before all the raw imagery has been acquired, each video stream can be subjected to different stages of processing simultaneously on different computers that may be grouped into one or more cluster(s), and the final product may consist of multiple video streams. Results of processing on different computers are shared, so that workers can collaborate effectively.
Nail, Jasper↗
A Parallel Processing Algorithm for Remote Sensing Classification
A current thread in parallel computation is the use of cluster computers created by networking a few to thousands of commodity general-purpose workstation-level commuters using the Linux operating system. For example on the Medusa cluster at NASA/GSFC, this provides for super computing performance, 130 G(sub flops) (Linpack Benchmark) at moderate cost, $370K. However, to be useful for scientific computing in the area of Earth science, issues of ease of programming, access to existing scientific libraries, and portability of existing code need to be considered. In this paper, I address these issues in the context of tools for rendering earth science remote sensing data into useful products. In particular, I focus on a problem that can be decomposed into a set of independent tasks, which on a serial computer would be performed sequentially, but with a cluster computer can be performed in parallel, giving an obvious speedup. To make the ideas concrete, I consider the problem of classifying hyperspectral imagery where some ground truth is available to train the classifier. In particular I will use the Support Vector Machine (SVM) approach as applied to hyperspectral imagery. The approach will be to introduce notions about parallel computation and then to restrict the development to the SVM problem. Pseudocode (an outline of the computation) will be described and then details specific to the implementation will be given. Then timing results will be reported to show what speedups are possible using parallel computation. The paper will close with a discussion of the results.
Gualtieri, J. Anthony↗
A comparison of queueing, cluster and distributed computing systems
Using workstation clusters for distributed computing has become popular with the proliferation of inexpensive, powerful workstations. Workstation clusters offer both a cost effective alternative to batch processing and an easy entry into parallel computing. However, a number of workstations on a network does not constitute a cluster. Cluster management software is necessary to harness the collective computing power. A variety of cluster management and queuing systems are compared: Distributed Queueing Systems (DQS), Condor, Load Leveler, Load Balancer, Load Sharing Facility (LSF - formerly Utopia), Distributed Job Manager (DJM), Computing in Distributed Networked Environments (CODINE), and NQS/Exec. The systems differ in their design philosophy and implementation. Based on published reports on the different systems and conversations with the system's developers and vendors, a comparison of the systems are made on the integral issues of clustered computing.
Kaplan, Joseph A.↗
Free Energies of Hydrated Halide Anions: High Through-Put Computations on Clusters to Treat Rough Energy-Landscapes
With a longer-term goal of addressing the comparative behavior of the aqueous halides F−, Cl−, Br−, and I− on the basis of quasi-chemical theory (QCT), here we study structures and free energies of hydration clusters for those anions. We confirm that energetically optimal (H2O)nX clusters, with X = Cl−, Br−, and I−, exhibit surface hydration structures. Computed free energies, based on optimized surface hydration structures utilizing a harmonic approximation, typically (but not always) disagree with experimental free energies. To remedy the harmonic approximation, we utilize single-point electronic structure calculations on cluster geometries sampled from an AIMD (ab initio molecular dynamics) simulation stream. This rough-landscape procedure is broadly satisfactory and suggests unfavorable ligand crowding as the physical effect addressed. Nevertheless, this procedure can break down when n≳4, with the characteristic discrepancy resulting from a relaxed definition of clustering in the identification of (H2O)nX clusters, including ramified structures natural in physical cluster theories. With ramified structures, the central equation for the present rough-landscape approach can acquire some inconsistency. Extension of these physical cluster theories in the direction of QCT should remedy that issue, and should be the next step in this research direction.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Performance Results for Sensor Assignment Problem as Solved on a Multi-Node Cluster
An earlier report described a procedure for optimal sensor set selection and its implementation on a computational cluster. This new and innovative capability was developed to facilitate a reduction in operations staffing levels to improve plant economics. By automating surveillance and maintenance tasks through early detection of degrading sensors and equipment, staff can be more efficiently deployed. The method uses automated reasoning and domain knowledge in the form of the conservation equations to infer from plant measurements the state of equipment health. Inclusion of domain knowledge addresses the problem that exists with pure data-driven methods that there are no rigorous guidelines for determining what constitutes an adequate sensor set. Formalizing the procedure for sensor set selection as we have done results in a more reliable and explainable diagnosis of plant equipment health. Importantly, from the standpoint of the plant owner, personnel are provided with an early and explicit diagnosis of an equipment problem. That in principle automates the process and eliminates having to send personnel into the plant to find the cause as typically occurs when a data-driven method detects an anomaly. In this report we describe first results obtained using a computational cluster to solve the sensor set selection problem as framed above. The case described addresses the problem of equipment health monitoring in the high-pressure (HP) feedwater system of a pressurized light water reactor as seen through the eyes of our collaborating utility partner. Maintenance of this system can amount to millions of dollars per year if equipment health issues go undiagnosed and lead to loss of function. On examining the potential that is inherent in the installed sensor set for diagnosing equipment health degradation, it was found that greater fault resolution capability can be achieved using a sensor set that is 20 percent fewer in number. The take-away is that compared to the installed sensor set there exists a more strategic assignment of sensors that will furnish better health monitoring capability and with fewer sensors. Where the problem defies solution by manual inspection, as is the case here, one can be found by an algorithm. The solution was obtained in four hours using 30 computational cores. The HP feedwater problem as posed above illustrates the added value of approaching the sensor selection problem as one amenable to algorithmic solution. This problem is of interest to advanced reactor designers and to utilities that are setting up remote monitoring and diagnostic centers.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Graphics Processing Unit Assisted Thermographic Compositing
Objective: To develop a software application utilizing general purpose graphics processing units (GPUs) for the analysis of large sets of thermographic data. Background: Over the past few years, an increasing effort among scientists and engineers to utilize the GPU in a more general purpose fashion is allowing for supercomputer level results at individual workstations. As data sets grow, the methods to work them grow at an equal, and often great, pace. Certain common computations can take advantage of the massively parallel and optimized hardware constructs of the GPU to allow for throughput that was previously reserved for compute clusters. These common computations have high degrees of data parallelism, that is, they are the same computation applied to a large set of data where the result does not depend on other data elements. Signal (image) processing is one area were GPUs are being used to greatly increase the performance of certain algorithms and analysis techniques. Technical Methodology/Approach: Apply massively parallel algorithms and data structures to the specific analysis requirements presented when working with thermographic data sets.
Ragasa, Scott↗
Graphics Processing Unit Assisted Thermographic Compositing
Objective: To develop a software application utilizing general purpose graphics processing units (GPUs) for the analysis of large sets of thermographic data. Background: Over the past few years, an increasing effort among scientists and engineers to utilize the GPU in a more general purpose fashion is allowing for supercomputer level results at individual workstations. As data sets grow, the methods to work them grow at an equal, and often greater, pace. Certain common computations can take advantage of the massively parallel and optimized hardware constructs of the GPU to allow for throughput that was previously reserved for compute clusters. These common computations have high degrees of data parallelism, that is, they are the same computation applied to a large set of data where the result does not depend on other data elements. Signal (image) processing is one area were GPUs are being used to greatly increase the performance of certain algorithms and analysis techniques.
Ragasa, Scott↗
GFCCLib: Scalable and efficient coupled-cluster Green's function library for accurately tackling many-body electronic structure problems
Coupled-cluster Green’s function (GFCC) calculation has drawn much attention in the recent years for targeting the molecular and material electronic structure problems from a many-body perspective in a systematically improvable way. However, GFCC calculations on scientific computing clusters usually suffer from expensive higher di- mensional tensor contractions in the complex space, expensive inter-process communi- cation, and severe load imbalance, which limits it’s routine use for tackling electronic structure problems. Here we present a numerical library prototype that is specifically designed for large-scale GFCC calculations. The design of the library is focused on a systematically optimal computing strategy to improve its scalability and efficiency. The performance of the library is demonstrated by the relevant profiling analysis of running GFCC calculations on remote giant computing clusters. The capability of the library is highlighted by computing a wide near valence band of a fullerene C60 molecule for the first time at the GFCCSD level that shows excellent agreement with the experimental spectrum.
Peng, Bo↗
PyPele Rewritten To Use MPI
A computer program known as PyPele, originally written as a Pythonlanguage extension module of a C++ language program, has been rewritten in pure Python language. The original version of PyPele dispatches and coordinates parallel-processing tasks on cluster computers and provides a conceptual framework for spacecraft-mission- design and -analysis software tools to run in an embarrassingly parallel mode. The original version of PyPele uses SSH (Secure Shell a set of standards and an associated network protocol for establishing a secure channel between a local and a remote computer) to coordinate parallel processing. Instead of SSH, the present Python version of PyPele uses Message Passing Interface (MPI) [an unofficial de-facto standard language-independent application programming interface for message- passing on a parallel computer] while keeping the same user interface. The use of MPI instead of SSH and the preservation of the original PyPele user interface make it possible for parallel application programs written previously for the original version of PyPele to run on MPI-based cluster computers. As a result, engineers using the previously written application programs can take advantage of embarrassing parallelism without need to rewrite those programs.
Hockney, George↗
From closed shells to open shells: Coupled-cluster calculations of atomic nuclei
Coupled-cluster theory is a powerful tool for first-principles calculations of atomic nuclei, enabling accurate predictions of nuclear observables across the Segrè chart. While coupled-cluster computations are especially efficient at shell closures, extensions have been developed to tackle open-shell nuclei, by exploiting the equation-of-motion method or by expanding the coupled-cluster wave function on top of a symmetry-breaking (either deformed or superfluid) reference state. In this study, we provide a comprehensive comparison of these different formulations applied to the calcium and nickel isotopes using nuclear two-and three-body interactions from chiral effective field theory. Here, based on ground-state energies, two-neutron separation energies, and two-neutron shell gaps, different coupled-cluster computations—based on symmetry-broken reference states and equationof-motion techniques— offer consistent descriptions of bulk properties across medium-mass isotopic chains.
Marino, Francesco [Johannes Gutenberg-Universität ↗
Women in Science: Fermilab Scientific Computing Specialist Krista Majewski
With February 11 marking the International Day of Women and Girls in Science, female physicists, engineers and computer scientists from CERN and Fermilab share their experiences of building careers in science. Krista Majewski came to Fermilab nine years ago as a software developer and now works in an operations role. She is a computing services specialist and part of the grid and cloud operations group, which manages computing clusters for the CMS experiment and other intensity frontier experiments at the lab. "My typical day would be maintaining the services that our group is responsible for and the machines that they run on. So if there are problems with a computing cluster where a user can't run their jobs, or they're trying to run on resources on the grid and they're unable to, then our group would be responsible for helping debug some of those issues," Krista says. Her path to programming at Fermilab has been winding. After receiving a degree in biomedical engineering, she went to work doing web development and programming for a large consulting firm, then moved to the financial industry. But programming called her back, and she earned a master's degree in computer science to start her new career. For Krista, programming is similar to something she enjoyed as child: solving puzzles. "I think it's a skill that you use in a lot of different ways," she says. Credits: Video production - Fermilab Creative Services Music Copyright: Universal Production Music Astral Light Chevalier Idea originally conceived by Tanya Levshina, Fermilab and Maria Dimou, CERN.
99 GENERAL AND MISCELLANEOUS↗