SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
On July 11-12, mobility and transportation leaders from across the country are coming to Washington, D.C. for the inaugural Sustainable Transportation Summit hosted by the Office of Energy Efficiency and Renewable Energy (EERE).
This video provides a recap of the 2017 ARPA-E Energy Innovation Summit.
We observe and analyze usage of the login nodes of the leadership class Summit supercomputer from the perspective of an ordinary user—not a system administrator—by periodically sampling user activities (job queues, running processes, etc.) for two full years (2020–2021). Our findings unveil key usage patterns that evidence misuse of the system, including gaming the policies, impairing I/O performance, and using login nodes as a sole computing resource. Our analysis highlights observed patterns for the execution of complex computations (workflows), which are key for processing large-scale applications.
Recently, a full-scale data processing workflow (Fig. 1) of the Square Kilometre Array (SKA) Phase 1 was successfully executed on the world’s fastest supercomputer Summit, proving that scientists have the expertise, software tools and computing resources to process the SKA data.
The analysis of vast amounts of data and the processing of complex computational jobs have traditionally relied upon high performance computing (HPC) systems, which offer reliable and efficient management of large-scale computational and data resources. Understanding these analyses’ needs is paramount for designing solutions that can lead to better science, and similarly, understanding the characteristics of the user behavior on those systems is important for improving user experiences on HPC systems. A common approach to gathering data about user behavior is to extract workload characteristics from system log data available only to system administrators. Recently at Oak Ridge Leadership Computing Facility (OLCF), however, we unveiled user behavior about the Summit supercomputer by collecting data from a user’s point of view with ordinary Unix commands.In this paper, we discuss the process, challenges, and lessons learned while preparing this dataset for publication and submission to an open data challenge. The original dataset contains personal identifiable information (PII) about the users of OLCF which needed be masked prior to publication, and we determined that anonymization, which scrubs PII completely, destroyed too much of the structure of the data to be interesting for the data challenge. We instead chose to pseudonymize the dataset, which reduced the linkability of the dataset to the users’ identities. Pseudonymization is significantly more computationally expensive than anonymization, and the size of our dataset, which is approximately 175 million lines of raw text, necessitated the development of a parallelized workflow that could be reused on different HPC machines. We demonstrate the scaling behavior of the workflow on two leadership class HPC systems at OLCF, and we show that we were able to bring the overall makespan time from an impractical 20+ hours on a single node down to around 2 hours. As a result of this work, we release the entire pseudonymized dataset and make the workflows and source code publicly available.
The rapid growth and wide applicability of Deep Learning (DL) frameworks poses challenges to computing centers which need to deploy and support the software, and also to domain scientists who have to keep up with the system environment and scale up scientific exploration through DL. We offer recommendations for deploying and scaling DL frameworks on the Summit supercomputer, currently atop the Top500 list, at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). We discuss DL software deployment in the form of containers, and compare performance of native-built frameworks and containerized deployment. Software containers show no noticeable negative performance impact and exhibit faster Python loading times and promise easier maintenance. To explore strategies for scaling up DL model training campaigns, we assess DL compute kernel performance, discuss and recommend I/O data formats and staging, and identify communication needs for scalable message exchange for DL runs at scale. We recommend that users take a step-wise tuning approach beginning with algorithmic kernel choice, node I/O configuration, and communications tuning as best-practice. We present baseline examples of scaling efficiency 87% for a DL run of ResNet50 running on 1024 nodes (6144 V100 GPUs).
The rise of machine learning (ML) applications and their use of mixed precision to perform interesting science are driving forces behind AI for science on HPC. The convergence of ML and HPC with mixed precision offers the possibility of transformational changes in computational science. The HPL-AI benchmark is designed to measure the performance of mixed precision arithmetic as opposed to the HPL benchmark which measures double precision performance. Pushing the limits of systems at extreme scale is nontrivial -little public literature explores optimization of mixed precision computations at this scale. In this work, we demonstrate how to scale up the HPL-AI benchmark on the pre-exascale Summit and exascale Frontier systems at the Oak Ridge Leadership Computing Facility (OLCF) with a cross-platform design. We present the implementation, performance results, and a guideline of optimization strategies employed for delivering portable performance on both AMD and NVIDIA GPUs at extreme scale.
This work presents a workflow for simulating and processing the full-scale low-frequency telescope data of the Square Kilometre Array (SKA) Phase 1. The SKA project will enter the construction phase soon, and once completed, it will be the world’s largest radio telescope and one of the world’s largest data generators. The authors used Summit to mimic an endto-end SKA workflow, simulating a dataset of a typical 6 hour observation and then processing that dataset with an imaging pipeline. This workflow was deployed and run on 4,560 compute nodes, and used 27,360 GPUs to generate 2.6 PB of data. This was the first time that radio astronomical data were processed at this scale. Results show that the workflow has the capability to process one of the key SKA science cases, an Epoch of Reionization observation. This analysis also helps reveal critical design factors for the next-generation radio telescopes and the required dedicated processing facilities.
Protein-ligand docking is an in silico tool used to screen potential drug compounds for their ability to bind to a given protein receptor within a drug-discovery campaign. Experimental drug screening is expensive and time consuming, and it is desirable to carry out large scale docking calculations in a high-throughput manner to narrow the experimental search space. Few of the existing computational docking tools were designed with high performance computing in mind. Therefore, optimizations to maximize use of high-performance computational resources available at leadership-class computing facilities enables these facilities to be leveraged for drug discovery. Here we present the porting, optimization, and validation of the AutoDock-GPU program for the Summit supercomputer, and its application to initial compound screening efforts to target proteins of the SARS-CoV-2 virus responsible for the current COVID-19 pandemic.
GPU memory corruption and in particular double-bit errors (DBEs) remain one of the least understood aspects of HPC system reliability. Albeit rare, their occurrences always lead to job termination and can potentially cost thousands of node-hours, either from wasted computations or as the overhead from regular checkpointing needed to minimize the losses. As supercomputers and their components simultaneously grow in scale, density, failure rates, and environmental footprint, the efficiency of HPC operations becomes both an imperative and a challenge. We examine DBEs using system telemetry data and logs collected from the Summit supercomputer, equipped with 27,648 Tesla V100 GPUs with 2nd-generation high-bandwidth memory (HBM2). Using exploratory data analysis and statistical learning, we extract several insights about memory reliability in such GPUs. We find that GPUs with prior DBE occurrences are prone to experience them again due to otherwise harmless factors, correlate this phenomenon with GPU placement, and suggest manufacturing variability as a factor. On the general population of GPUs, we link DBEs to short- and long-term high power consumption modes while finding no significant correlation with higher temperatures. We also show that the workload type can be a factor in memory’s propensity to corruption.
As we move into the exascale era, the power and energy footprints of high-performance computing (HPC) systems have grown significantly larger. Due to the harsh power and thermal conditions the system, components are exposed to extreme operating conditions. Operation of such modern HPC systems requires deep insights into long term system behavior to maintain its efficiency as well as its longevity. To help the HPC community to gain such insights, we provide a dataset that records the long-term power and thermal behavior of the 200PF pre-exascale supercomputer at the Oak Ridge Leadership Computing Facility (OLCF), Summit. This system is an IBM AC922 based system that has 9,252 IBM Power9 CPUs and 27,756 Nvidia V100 GPUs and can consume up to 13MW power at peak. Heat removal is performed using medium temperature direct liquid cooling and rear-door heat exchanger based secondary cooling loop. Originally extracted from a high-resolution (1Hz) per-component (GPUs, CPUs) measurements from the system, we primarily provide a dataset that has 10-second and 1-minute mean power and thermal measurements selected from five month-long segments over the course of 2020 (January and August), 2021 (February and August), and 2022 (January). For convenience, we also provide various sub datasets randomly sampled from the time and space (hosts) of the cluster. Further details and example code for analysis can be found in the following GitHub repository: https://github.com/at-aaims/summit_power_and_thermal_data
The Suisun Summit 2022 was a gathering of 70 clinicians, educators, and researchers in quantitative electroencephalography and neurofeedback. During this 5-day event, several themes emerged in talks or discussion groups: EEG/qEEG Reading Skills; Medication Effects and Pharmaco-EEG; Technological Advancements; Emerging Concerns; and Growing Community Prestige through Research. Participants were asked to summarize what they believed to be the most important messages from the event to share with colleagues who were not in attendance, resulting in this review. A unifying concept for all the themes was a desire for higher quality, standardized EEG/qEEG education that provides depth as well as breadth. Models of clinical care that encourage open communication with prescribers and functional medicine specialists were strongly emphasized. Abstracts from all presentations are attached in Addendum B.
This report captures the initial conclusions of the DOE seven National Lab team collaborating on the “Solving the Information Technology Energy Challenge Beyond Moore’s Law” initiative from the DOE Big Idea Summit III held in April of 2016. The seven Labs held a workshop in Albuquerque, NM in late July 2016 and gathered 40 researchers into 5 working groups: 4 groups spanning the levels of the co-design framework shown below, and a 5th working group focused on extending and advancing manufacturing approaches and coupling their constraints to all of the framework levels. These working groups have identified unique capabilities within the Labs to support the key challenges of this Beyond Moore’s Law Computing (BMC) vision, as well as example first steps and potential roadmaps for technology development.
The purpose of this document is to provide an initial overview of (new) information regarding the gaps in DOE's work on High Performance Computing(HPC), Machine Learning and Algorithms (IVILA) and Human Systems(HS) emerging from the Sep. 2018 Summit, and to briefly forecast the attendee community's next steps.
The NATO allies must react to the nuclear policy and posture choices of the Trump administration. In some respects, there are good reasons why the Alliance should move closer to the Trump Administration. In other cases, Allies should seek agreement on slightly different policies. While it is unrealistic to expect that the NATO allies will be able to resolve all the questions posed by the 2018 Nuclear Posture Review at the Summit in Brussels, there are several things that they should be able to agree upon. This includes more specific language about the Russia nuclear challenges, general recognition of U.S. efforts to bolster NATO nuclear deterrence posture, a clear language on the value of nuclear sharing arrangements, and reaffirmation of some of NATO’s statements on arms control and disarmament. Yet, a signal in Brussels of NATO’s efforts to build a new nuclear consensus would be stronger if heads of state and government initiated a broader process aimed at developing a comprehensive deterrence and defense strategy that would treat nuclear deterrence as an integral part of a multi-domain approach.
Nearest-neighbor communication is at the heart of many high-performance parallel computations. We report on the performance of such communication on the Oak Ridge Leadership Computing Facility system Summit in the context of the PETSc communication module. The analysis in this report includes basic Ping-Pong point-to-point communication and regular and irregular nearest-neighbor communication.
We evaluated the orientation matching step in the M-TIP SPI workflow for potential offloading to accelerators. We ported the code to GPUs, benchmarked it, optimized and down-selected the best versions. The accelerated version of the orientation matching code that was developed at LANL (LANL GPU v3) is 34-55X faster than sequential, 2.4-4.9X faster than the fastest OpenMP open source version we found (FAISS OpenMP) and 1.5-4X faster than the fastest GPU open source version we found (FAISS GPU). Summit single-node GPU versions were somewhat faster than Cori GPU. Image size plays a role; mid-range image sizes take more time. The LANL CUDA multi-node, multi-GPU implementation shows mostly linear strong scaling. I/O also plays a large role; splitting data into parts improves read time and burst buffers dramatically improve read times. This work will be integrated into the M-TIP workflow as part of the next milestone ADSE13-193.
This report is a part of the deliverables for technical assistance provided to Cat Creek Energy for the Coral Summit Trybrid (Triple Hybrid-Pumped Storage Hydropower, Battery Energy Storage System, and photovoltaic solar energy) energy project. This report explores the operational benefits and challenges of hybridizing an open loop PSH (200MW) located at Mackay, Custer County, Idaho with solar PV (Ground mount 300MW and floating 40MW) and battery (720MWhr). This document reports two activities performed as a part of the hybridization assessment task 1) optimal resource allocation and energy management strategy, and 2) power quality and reliability assessment. From optimal resource allocation and energy management strategy (activity 1), the following key findings can be observed: • Conventional PSH (CPSH) with two reversible pump turbines and separate penstocks can provide required flexibility equivalent to that from two ternary PSH with separate penstock. With single unit CPSH, upper reservoir head cannot be maintained accurately, the variation of water level is rapid and pump mode flexibility is not available. These disadvantages can be overcome by single unit TPSH. However, using two CPSH units with separate penstocks also overcome these disadvantages with the formation of the hydraulic short circuit between two conventional units. • Flooding of the lower reservoir is a severe concern when considering continuous operation for black start. This limits the duration of continuous operation from PSH alone to around 50 hours. Due to the complementary PV and battery action, the duration of continuous operation and smooth power output can be extended. • An optimization problem is framed that maximizes the power output on an hourly basis while minimizing constraint violations and respecting seasonal variations of solar PV and load profiles . Two value streams, arbitrage and baseload generation are served by this profile. It was uncovered that for smooth power output during regular operation, PV curtailment will be required, or the battery capacity needs to be increased above 90MW to accommodate additional PV. From power quality and reliability assessment (activity 2) the following takeaway points can be observed: • The Trybrid, when integrated at the Lost River bus, and limited to 250MW in generation mode and -150MW in the pump mode, causes no violation of voltage or flow.