Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Exascale computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,243 records · Page 69

ForTrilinos Design Document

With the development of a Fortran Interface to Trilinos, ForTrilinos, modelers using modern Fortran will be able to provide their codes the capability to use solvers and other capabilities on exascale machines via a straightforward infrastructure that accesses Trilinos. This document outlines what ForTrilinos does and explains briefly how it works. We show it provides a general access to packages via an entry point and uses an xml file from fortran code. With the first release, ForTrilinos will enable Teuchos to take xml parameter lists from Fortran code and set up data structures. It will provide access to linear solvers and eigen solvers. Several examples are provided to illustrate the capabilities in practice. We explain what the user should have already with their code and what Trilinos provides and returns to the Fortran code. We provide information about the build process for ForTrilinos, with a practical example. In future releases, nonlinear solvers, time iteration, advanced preconditioning techniques, and inversion of control (IoC), to enable callbacks to Fortran routines, will be available.

97 MATHEMATICS AND COMPUTING↗

Enabling Scalable and Extensible Memory-mapped Datastores in Userspace

Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Furthermore, through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9 × . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.

97 MATHEMATICS AND COMPUTING↗

Towards Accurate Application Characterization for Exascale (APEX)

Sandia National Laboratories has been engaged in hardware and software codesign activities for a number of years, indeed, it might be argued that prototyping of clusters as far back as the CPLANT machines and many large capability resources including ASCI Red and RedStorm were examples of codesigned solutions. As the research supporting our codesign activities has moved closer to investigating on-node runtime behavior a nature hunger has grown for detailed analysis of both hardware and algorithm performance from the perspective of low-level operations. The Application Characterization for Exascale (APEX) LDRD was a project conceived of addressing some of these concerns. Primarily the research was to intended to focus on generating accurate and reproducible low-level performance metrics using tools that could scale to production-class code bases. Along side this research was an advocacy and analysis role associated with evaluating tools for production use, working with leading industry vendors to develop and refine solutions required by our code teams and to directly engage with production code developers to form a context for the application analysis and a bridge to the research community within Sandia. On each of these accounts significant progress has been made, particularly, as this report will cover, in the low-level analysis of operations for important classes of algorithms. This report summarizes the development of a collection of tools under the APEX research program and leaves to other SAND and L2 milestone reports the description of codesign progress with Sandia’s production users/developers.

97 MATHEMATICS AND COMPUTING↗

Domain Specific Language Support for Exascale Application Development

The University of Oregon has produced a Fortran parser based on an SDF (Syntax Definition Formalism) grammar. Primary partners in this effort were Rice University and Lawrence Livermore National Laboratory. The developed Fortran parser incorporates the full Fortran 2008 standard with extensions from the more recent document, TS 29113, “Further Interoperability of Fortran with C,” (to be included in the Fortran 2018 standard). In the final year of the project, the Fortran parser was updated to comply with the complete 2018 standard, as specified in the ISO/IEC document, JTC 1/SC 22/WG 5/N2146. The research explored in this project provided the expression of the Fortran grammar in SDF form for integration with the Rosebud DSL framework developed by Rice University. The Rosebud framework is a cohesive interface allowing modular DSL plugins to be developed independently and then distributed and combined in HPC applications. These plugins implement a form of generalized DSL which includes both concrete languages used to write application source code and abstract languages used within the compilation process. ROSE’s predefined DSL Building Blocks (for example the Fortran base language) are represented as abstract language plugins and DSL developers can use Rosebud to construct additional building blocks in the same way.

97 MATHEMATICS AND COMPUTING↗

Cross-Layer Self-Adaptive/Self-Aware System Software for Exascale Systems

The extreme level of parallelism coupled with the limited available power budget expected in the exascale era brings unprecedented challenges that demand optimization of performance, power and resiliency in unison. Scalability on such systems is of paramount importance, while power and reliability issues may change the execution environment in which a parallel application runs. To solve these challenges exascale systems will require an introspective system software that combines system and application observations across all system stack layers with online feedback and adaptation mechanisms. In this paper we propose the design of a novel self-aware, self-adaptive system software in which a kernel-level Monitor, which continuously inspects the evolution of the target system through observation of Sensors, is combined with a user-level Controller, which reacts to changes in the execution environment, explores opportunities to increase performance, save power and adapts applications to new execution scenarios. We show that the monitoring system accurately monitors the evolution of parallel applications with a runtime overhead below 1-2%. As a test case, we design and implement a user-runtime system that aims at optimizing application’s performance and system power consumption on complex hierarchical architectures. Our results show that our adaptive system reaches 98% of performance efficiency of manually-tuned applications.

High Performance computing, Self-aware/self-adapti↗

Continuous soil temperature and soil deformation measurements, Teller road Mile 47, Nome, Alaska.

The dataset comprises Soil Temperature and Deformation monitoring data gathered from a watershed situated along the Nome-Teller road at Mile 47 in Alaska. Its primary objective is to enhance comprehension of deformation mechanisms in permafrost environments by using 51 probes designed following Wielandt et al. (2022). The dataset comprises a description of Probe_ID, Location, start time, end time and probe length in the "ProbesLocations_StartEndDates.csv" file, the 51 data files encapsulated within "Data_files.zip" and the computed deformation for 2023 and 2022 in "Deformation_T47.zip".Each file in Data_files.zip is named as followed "ProbeId_StartMonth_EndMonth.csv". Within each file, the initial column denotes the timestamp in UTC, followed by the sensor’s battery voltage and temperature and acceleration values (X, Y, Z) in subsequent columns. The data collection frequency is set at 30-minute intervals. Each file in Deformation_T47.zip is named as followed "ProbeId_Temp_def_year.csv". Within each file, the initial column denotes the timestamp in UTC, followed by the temperature at each sensor and cumulative deformation from last to first sensor in subsequent columns. The data collection is daily averaged.This dataset forms an integral component of the Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic). The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

GASNet-EX Performance Improvements Due to Specialization for the Cray Aries Network

GASNet-EX is a portable, open-source, high-performance communication library designed to efficiently support the networking requirements of PGAS runtime systems and other alternative models on future exascale machines. This paper reports on the improvements in performance observed on Cray XC-series systems due to enhancements made to the GASNet-EX software. These enhancements, known as "specializations", primarily consist of replacing network-independent implementations of several recently added features with implementations tailored to the Cray Aries network. Performance gains from specialization include (1) Negotiated-Payload Active Messages improve bandwidth of a ping-pong test by up to 14%, (2) Immediate Operations reduce running time of a synthetic benchmark by up to 93%, (3) non-bulk RMA Put bandwidth is increased by up to 32%, (4) Remote Atomic performance is 70% faster than the reference on a point-to-point test and allows a hot-spot test to scale robustly, and (5) non-contiguous RMA interfaces see up to 8.6x speedups for an intra-node benchmark and 26% for inter-node. These improvements are all available in GASNet-EX version 2018.3.0 and later.

97 MATHEMATICS AND COMPUTING↗

Understanding and Exploiting Spatial Properties of System Failures on Extreme-Scale HPC Systems, In: 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks

As we approach exascale, the scientific simulations are expected to experience more interruptions due to increased system failures. Designing better HPC resilience techniques requires understanding the key characteristics of system failures on these systems. While temporal properties of system failures on HPC systems have been well-investigated, there is limited understanding about the spatial characteristics of system failures and its impact on the resilience mechanisms. Therefore, we examine the spatial characteristics and behavior of system failures. We investigate the interaction between spatial and temporal characteristics of failures and its implications for system operations and resilience mechanisms on large-scale HPC systems. We show that system failures have "spatial locality" at different granularity in the system, study impact of different failure-types, and investigate the correlation among different failure-types. Finally, we propose a novel scheme that exploits the spatial locality in failures to improve application and system performance. Our evaluation shows that the proposed scheme significantly improves the system performance in a dynamic and production-level HPC system.

Computer Science↗

Improving North American Wildfire Prediction by Integrating a Machine-Learning Fire Model in a Land Surface Model

Wildfires have shown increasing trends in both frequency and severity across the Contiguous United States (CONUS). However, process-based fire models have difficulties in accurately simulating the burned area over the CONUS due to a simplification of the physical process and cannot capture the interplay among fire, ignition, climate, and human activities. The deficiency of burned area simulation deteriorates the description of fire impact on energy balance, water budget, and carbon fluxes in the Earth System Models (ESMs). Alternatively, machine learning (ML) based fire models, which capture statistical relationships between the burned area and environmental factors, have shown promising burned area predictions and corresponding fire impact simulation. We develop a hybrid framework (ML4Fire-XGB) that integrates a pretrained eXtreme Gradient Boosting (XGBoost) wildfire model with the Energy Exascale Earth System Model (E3SM) land model (ELM) version 2.1. A Fortran-C-Python deep learning bridge is adapted to support online communication between ELM and the ML fire model. Specifically, the burned area predicted by the ML-based wildfire model is directly passed to ELM to adjust the carbon pool and vegetation dynamics after disturbance, which are then used as predictors in the ML-based fire model in the next time step. Evaluated against the historical burned area from Global Fire Emissions Database 5 from 2001-2020, the ML4Fire-XGB model outperforms process-based fire models in terms of spatial distribution and seasonal variations. Sensitivity analysis confirms that the ML4Fire-XGB well captures the responses of the burned area to rising temperatures. The ML4Fire-XGB model has proved to be a new tool for studying vegetation-fire interactions, and more importantly, enables seamless exploration of climate-fire feedback, working as an active component in E3SM.

54 ENVIRONMENTAL SCIENCES↗

UMap: An application-oriented user level memory mapping library

Exploiting the prominent role of complex memories in exascale node architecture, the UMap page fault handler offers new capabilities to access large memory-mapped data sets directly. UMap provides flexible configuration options to customize page handling to each application, including analysis of massive observational and simulation data sets. The high-performance design features I/O decoupling, dynamic load balancing, and application-level controls. Page faults triggered by application threads and processes accessing data mapped to a UMapp’ed region are handled via the Linux userfaultfd protocol, an asynchronous message-oriented kernel-user communication mechanism that avoids the context switch penalty of traditional signal fault handlers. UMap is fully open source. In this paper, we give an overview of the UMap library architecture, its extensible plugin architecture, and the use/performance of UMap in emerging heterogeneous memory hierarchies such as near-node Non-volatile Memory (NVM) and network attached memories. We highlight new capabilities in two pagefault management plugins, the NetworkStore and SparseStore. We demonstrate the integration between UMap and multiple ECP products including Caliper, Metall, ZFP, Mochi, and Ripples.

97 MATHEMATICS AND COMPUTING↗

Earth System Model Development - Energy Exascale Earth System Model (2020 Principal Investigator Meeting Report)

As part of the U.S. Department of Energy (DOE) Office of Biological and Environmental Research (BER), the Earth System Model Development (ESMD) program area supports significant research in Earth system model development, including major efforts in the Energy Exascale Earth System Model (E3SM) project. Convened virtually on October 26–29, the 2020 ESMD-E3SM Principal Investigator (PI) Meeting provided a forum to survey ESMD sponsored research, recent progress made by the ESMD science team and the broader Earth system modeling community within and outside of DOE, foster interactions across ESMD projects, and discuss research gaps and opportunities for future advancements.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Performance tuning of CEED software and 1st and 2nd wave apps

The goal of this milestone was the performance tuning of the CEED software, as well as the use and tuning of CEED to accelerate the first and second wave of targeted ECP applications. In this milestone, the CEED team developed optimization techniques and tuned for performance the CEED software to accelerate the first and second wave target ECP applications. Specifically, the focus was on the following: (1) Efficient use of the memory sub-system for managing data motion and interactions among different physics packages and libraries; this included integration of the new developments in a libCEED 0.5 software release; (2) Enhancing the CEED libraries with GPU and AMD GPU support in close interaction with vendors; (3) Optimal data locality and motion, and enhanced scalability and parallelism; this included vendors interactions for improvements in data motion and making strong scaling easier and more efficient; and (4) Continue boosting performance for the first-wave and second-wave of ECP target applications, including ExaSMR, ExaWind, Urban, MARBL, E3SM, and ExaAM. A main part of this milestone was the performance tuning of the CEED first-wave and second-wave ECP applications. This included the ExaSMR application – Coupled Monte Carlo Neutronics and Fluid Flow Simulation of Small Modular Reactors (ORNL), the MARBL application – Next-Gen Multi-physics Simulation Code (LLNL), and the ExaAM ExaConstit – a miniapp for the Transforming Additive Manufacturing through Exascale Simulation applications project (ExaAM), as well as ExaWind, Urban, and E3SM. The artifacts delivered include performance improvements in CEED’s 1st and 2nd wave of applications, and tuned CEED software for various architectures through a number of backends, freely available in the CEED’s repository on GitHub. See the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed for more details. In addition to details and results from the above R&D efforts, in this document we are also reporting on other project-wide activities performed in Q4 of FY19 including: CEED’s third annual meeting, MFEM GPU kernels scaling improvements, libCEED 0.5 and NekRS releases, an overview paper covering latest results on a set of bake-off high-order finite element problems (BPs), and other outreach efforts.

97 MATHEMATICS AND COMPUTING↗

In Situ Inference for Earth System Predictability

Focal Area: Focal Area 3: Insight gleaned from complex simulated data using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: An understanding of future evolution in precipitation extremes is critical to numerous DOE mission questions. Extreme events are by nature short time-scale events that are difficult to diagnose in available model data. Accurate modeling of extreme events necessarily requires high spatial resolution at the storm scale locally. However, the environment in which storms grow is dependent on global, remote, processes. These complex spatiotemporal relationships are impossible to diagnose at resolutions required to accurately model storms responsible for extreme precipitation. At exascale, climate simulations will produce results at fine enough resolution to investigate these relationships. However, the resulting data from these simulations will be far too large to save for post-simulation analysis. We advocate for fitting statistical models inside the simulations as they run, a context known as in situ, which will facilitate scientific investigations using the full fine-scale data stream. Figure 1 shows an example of the type of model we could consider, a Bayesian hierarchical spatial regression model. Precipitation extremes at each grid cell are modeled using extreme value distributions. Since extremes are rare, fitting models to individual grid cells can result in high variance and poor estimates. Instead, the model can be made more robust by smoothing the parameters of the extreme value model across space. Additionally, the parameters themselves can be functionally linked to other variables elsewhere in the simulation. Thus, we can use the fine-scale data to build more robust models for extremes that link extreme behavior to other climate patterns.

54 ENVIRONMENTAL SCIENCES↗

Unraveling Network-induced Memory Contention: Deeper Insights with Machine Learning

Remote Direct Memory Access (RDMA) is expected to be an integral communication mechanism for future exascale systems enabling asynchronous data transfers, so that applications may fully utilize CPU resources while simultaneously sharing data amongst remote nodes. We examine Network-induced Memory Contention (NiMC) on Infiniband networks. We expose the interactions between RDMA, main-memory and cache, when applications and out-of-band services compete for memory resources. We then explore NiMCs resulting impact on application-level performance. For a range of hardware technologies and HPC workloads, we quantify NiMC and show that NiMCs impact grows with scale resulting in up to 3X performance degradation at scales as small as 8K processes even in applications that previously have been shown to be performance resilient in the presence of noise. In addition, this work examines the problem of predicting NiMC's impact on applications by leveraging machine learning and easily accessible performance counters. This approach provides additional insights about the root cause of NiMC and facilitates dynamic selection of potential solutions. Finally, we evaluated three potential techniques to reduce NiMCs impact, namely hardware offloading, core reservation and network throttling.

97 MATHEMATICS AND COMPUTING↗

LIGHT-bgcArgo-1.0: using synthetic float capabilities in E3SMv2 to assess spatiotemporal variability in ocean physics and biogeochemistry

Since their advent over 2 decades ago, autonomous Argo floats have revolutionized the field of oceanography, and, more recently, the addition of biogeochemical and biological sensors to these floats has greatly improved our understanding of carbon, nutrient, and oxygen cycling in the ocean. While Argo floats offer unprecedented horizontal, vertical, and temporal coverage of the global ocean, uncertainties remain about whether Argo sampling frequency and density capture the true spatiotemporal variability in physical, biogeochemical, and biological properties. As the true distributions of, e.g., temperature or oxygen are unknown, these uncertainties remain difficult to address with Argo floats alone. Numerical models with synthetic observing systems offer one potential avenue to address these uncertainties. Here, we implement synthetic biogeochemical Argo floats into the Energy Exascale Earth System Model version 2 (E3SMv2), which build on the Lagrangian In Situ Global High-Performance Particle Tracking (LIGHT) module in E3SMv2 (E3SMv2-LIGHT-bgcArgo-1.0). Since the synthetic floats sample the model fields at model run time, the end user defines the sampling protocol ahead of any model simulation, including the number and distribution of synthetic floats to be deployed, their sampling frequency, and the prognostic or diagnostic model fields to be sampled. Using a 6-year proof-of-concept simulation, we illustrate the utility of the synthetic floats in different case studies. In particular, we quantify the impact of (i) sampling density on the float-derived detection of deep-ocean change in temperature or oxygen and on float-derived estimates of phytoplankton phenology, (ii) sampling frequency and sea-ice cover on float trajectory lengths and hence float-derived estimates of current velocities, and (iii) short-term variability in ecosystem stressors on estimates of their seasonal variability.

54 ENVIRONMENTAL SCIENCES↗

ISOBAR Preconditioner for Effective and High-throughput Lossless Data Compression

Efficient handling of large volumes of data is a necessity for exascale scientific applications and database systems. To address the growing imbalance between the amount of available storage and the amount of data being produced by high speed (FLOPS) processors on the system, data must be compressed to reduce the total amount of data placed on the file systems. General-purpose loss less compression frameworks, such as zlib and bzlib2, are commonly used on datasets requiring loss less compression. Quite often, however, many scientific data sets compress poorly, referred to as hard-to-compress datasets, due to the negative impact of highly entropic content represented within the data. An important problem in better loss less data compression is to identify the hard-to-compress information and subsequently optimize the compression techniques at the byte-level. To address this challenge, we introduce the In-Situ Orthogonal Byte Aggregate Reduction Compression (ISOBAR-compress) methodology as a preconditioner of loss less compression to identify and optimize the compression efficiency and throughput of hard-to-compress datasets.

Computer Science↗

Impact of Physics Parameterization Ordering in a Global Atmosphere Model

Abstract Because weather and climate models must capture a wide variety of spatial and temporal scales, they rely heavily on parameterizations of subgrid‐scale processes. The goal of this study is to demonstrate that the assumptions used to couple these parameterizations have an important effect on the climate of version 0 of the Energy Exascale Earth System Model (E3SM) General Circulation Model (GCM), a close relative of version 1 of the Community Earth System Model (CESM1). Like most GCMs, parameterizations in E3SM are sequentially split in the sense that parameterizations are called one after another with each subsequent process feeling the effect of the preceding processes. This coupling strategy is noncommutative in the sense that the order in which processes are called impacts the solution. By examining a suite of 24 simulations with deep convection, shallow convection, macrophysics/microphysics, and radiation parameterizations reordered, process order is shown to have a big impact on predicted climate. In particular, reordering of processes induces differences in net climate feedback that are as big as the intermodel spread in phase 5 of the Coupled Model Intercomparison Project. One reason why process ordering has such a large impact is that the effect of each process is influenced by the processes preceding it. Where output is written is therefore an important control on apparent model behavior. Application of k ‐means clustering demonstrates that the positioning of macro/microphysics and shallow convection plays a critical role on the model solution.

54 ENVIRONMENTAL SCIENCES↗

Hanford Site Composite Analysis: LLBG-200-W B Vadose Zone Model

The objectives of the vadose modeling for the updated Hanford Site Composite Analysis (CA) are to simulate the flow and transport of water and radionuclide releases from the surface to the water table and to provide radionuclide transfer rates to the CA saturated zone model (CP-57037, Model Package Report: Plateau to River Groundwater Model, Version 8.3). Water additions include natural recharge and water discharged to the ground as a result of industrial processes associated with Hanford Site operations. Contaminant sources include radionuclides in water discharged to the ground during operations and radionuclides disposed “dry” in solid waste burial grounds or other means. The following 16 radionuclides were selected for this modeling effort: carbon-14 (C-14), chlorine-36 (Cl-36), tritium (H-3), iodine-129 (I-129), neptunium-237 (Np-237), rhenium-187 (Re-187), strontium-90 (Sr-90), technetium-99 (Tc-99), uranium-232 (U-232), uranium-233 (U-233), uranium-234 (U-234), uranium-235 (U-235), uranium-236 (U-236), uranium-238 (U-238), radium-226 (Ra-226), and thorium-230 (Th-230). The simulation time starts in 1943 and ends at 12070, which is 10,000 years after assumed Hanford Site closure in 2070. The parallel version of the Subsurface Transport Over Multiple Phases (STOMP 1 ) simulator, officially named the exascale Subsurface Transport Over Multiple Phases (eSTOMP), is used to simulate flow and transport for the vadose models. The documentation for the STOMP code is comprehensive. The theoretical and numerical approaches applied in the STOMP code are documented in a published theory guide (PNNL-12030, STOMP Subsurface Transport Over Multiple Phases Version 2.0 Theory Guide). The code has undergone a rigorous verification procedure against analytical solutions, laboratory-scale experiments, and field-scale demonstrations. The application guide (PNNL-11216, STOMP Subsurface Transport Over Multiple Phases Application Guide) provides instructive examples in the application of the code to classical groundwater problems. The user’s guide (PNNL-15782, STOMP: Subsurface Transport Over Multiple Phases Version 4.0: User’s Guide) describes the general use, input file formatting, compilation, and execution of the code. The primary output of the vadose zone modeling is radionuclide transfer rates to the groundwater for input into the saturated zone model. The rates will be summed over the 100 by 100 m saturated zone model grid cells that fall within the vadose zone model source domain.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗