Engineering PapersSearch

SEARCH · Engineering Papers

Results for “MPI”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

General field evaluation in high-order meshes on GPUs

Robust and scalable function evaluation at any arbitrary point in the finite/spectral element mesh is required for querying the partial differential equation solution at points of interest, comparison of solution between different meshes, and Lagrangian particle tracking. This is a challenging problem, particularly for high-order unstructured meshes partitioned in parallel with MPI, as it requires identifying the element that overlaps a given point and computing the corresponding reference space coordinates. Here, we present a robust and efficient technique for general field evaluation in large-scale high-order meshes with quadrilaterals and hexahedra. In the proposed method, a combination of globally partitioned and processor-local maps are used to first determine a list of candidate MPI ranks, and then locally candidate elements that could contain a given point. Next, element-wise bounding boxes further reduce the list of candidate elements. Finally, Newton’s method with trust region is used to determine the overlapping element and corresponding reference space coordinates. Since GPU-based architectures have become popular for accelerating computational analyses using meshes with tensor-product elements, specialized kernels have been developed to utilize the proposed methodology on GPUs. The method is also extended to enable general field evaluation on surface meshes. The paper concludes by demonstrating the use of the proposed method in various applications ranging from mesh-to-mesh transfer during r-adaptivity to Lagrangian particle tracking.

97 MATHEMATICS AND COMPUTING

Changing windstorm characteristics over the US Northeast in a single model large ensemble

Abstract Extreme windstorms pose a significant hazard to infrastructure and public safety, particularly in the highly populated US Northeast (NE). However, the influence climate change and changing land use will have on these events remains unclear. A large ensemble generated using the Max-Planck Institute (MPI) Earth system model is used to generate projections of NE windstorms under different shared socioeconomic pathways (SSPs) and to attribute changes to projected land use land cover (LULC) change, externally forced changes and internal climate variability. To reduce the influence of coarse grid cell resolution and uncertainties in surface roughness lengths, windstorms are identified using simultaneous widespread exceedance of local 99th percentile 10 m wind speeds (U 99 ). Projected declines in forest cover in the NE and the resulting reductions in surface roughness length under SSP3-7.0 lead to projections of large increases in U 99 and derived windstorm intensity and scale. However, these projected changes in regional LULC under SSP3-7.0 are unprecedented in a historical context and may not be realistic. After corrections are applied to remove the influence of LULC on wind speeds, regionally averaged U 99 exhibit declines for most of the single model initial-condition large ensemble (SMILE) members which are broadly proportional to the radiative forcing and global air temperature increase in the SSPs, with a median value of −0.15 ms −1 °C −1 . While weak cyclones are projected to decline in frequency in the NE, intense cyclones and the resulting windstorms and indices of socioeconomic loss do not. Where present, significant trends in these loss indices are positive, and some MPI SMILE members generate future windstorms that are unprecedented in the historical period.

Coburn, Jacob (ORCID:0000000309538117)

Enabling Scientific Applications with Performance-Portability and High-Productivity for Multi-GPU Programming with JACC.Multi

This work bridges the gap between multi-GPU computing and high-productivity, performance-portable programming solutions. Our goal is to enhance scientific applications with a productive and portable solution—program once, deploy everywhere—for multi-GPU programming with no cost to programmability. To accomplish this, we implemented JACC.Multi, which is part of the Julia for ACCelerators (JACC) performance-portable framework. JACC. Multi is the only high-level, portable metaprogramming solution that targets multi-GPU environments and is integrated in a readily accessible programming language (e.g., Julia language). With transparent GPU-to-GPU communication, JACC. Multi is optimized for scientific application workloads and is portable for NVIDIA and AMD accelerators. For the evaluation, we use two modern multi-GPU systems: Hudson, which features two NVIDIA H100 Hopper GPUs per node, and Frontier, which features four AMD MI250X GPUs per node, each with two Graphics Compute Dies (GCDs) for a total of eight GCDs per node. Additionally, as part of the evaluation, we use JACC (one GPU), MPI+JACC, and JACC. Multi codes that implement well-known and widely used scientific algorithms/kernels such as the conjugate gradient algorithm and an explicit forward Euler solver that requires GPU-to-GPU communication. Overall, JACC. Multi codes achieve better performance than MPI+JACC codes and significant speedups over JACC (one GPU), with up to 1.9× on Hudson and 6× on Frontier.

Valero Lara, Pedro [ORNL] (ORCID:0000000214794310)

Large-Message All-to-All Communication at Frontier Scale

Near the full scale of exascale supercomputers, latency can dominate the cost of all-to-all communication even for very large message sizes. We describe GPU-aware all-to-all implementations designed to reduce latency for large message sizes at extreme scales, and we present their performance using 65536 tasks (8192 nodes) on the Frontier supercomputer at the Oak Ridge Leadership Computing Facility. Two implementations perform best for different ranges of message size, and all outperform the vendor-provided MPI_Alltoall. Our results show promising options for improving implementations of MPI_Alltoall_init.

White, Trey [ORNL] (ORCID:000900052186075X)

Scalable multilevel Monte Carlo methods exploiting parallel redistribution on coarse levels

Here, we study an element agglomeration coarsening strategy that requires data redistribution at coarse levels when the number of coarse elements becomes smaller than the number of MPI processes used on the finest level. The overall procedure generates coarse elements (general unstructured unions of fine grid elements) within the framework of element-based algebraic multigrid methods (or AMGe) studied previously. The AMGe-generated coarse spaces have the ability to exhibit approximation properties of the same order as the fine-level spaces since by construction they contain the piecewise polynomials of the same order as on the fine level. These approximation properties are key for the successful use of AMGe in multilevel solvers for nonlinear partial differential equations as well as for multilevel Monte Carlo (MLMC) simulations. The ability to coarsen without being constrained by the number of MPI processes, as described in the present paper, allows to improve the scalability of these solvers as well as the overall MLMC method. The paper illustrates this latter fact with detailed scalability study of MLMC simulations applied to model Darcy equations with a stochastic log-normal permeability field.

AMGe

Venado acceptance: results and tips [Slides]

Nvidia compiler support is not available through cray-mpich/compiler wrapper interface. Adjust CMAKE files to use the correct COMPILER_ID in conditionals and explicit variables to package flags. Use pinned host memory in cray-libsci_acc and cublasXt calls. Set a large blockDim for cublasXt calls. Try MPS and/or explicit numactl binding if performance is lackluster. Use CRAY_MALLOPT_OFF=1 if unexpected OOM errors appear using cce. Use MPICH_SMP_SINGLE_COPY_MODE=CMA for xpmem issues. Use MPICH_OPT_THREAD_SYNC=0 for MPI_THREAD issues. Poor CUDA-aware MPI performance remains an issue.

97 MATHEMATICS AND COMPUTING

Atmospheric H 2 observations from the NOAA Cooperative Global Air Sampling Network

Abstract. The NOAA Global Monitoring Laboratory (GML) measures atmospheric hydrogen (H2) in grab samples collected weekly as flask pairs at over 50 sites in the Cooperative Global Air Sampling Network. Measurements representative of background air sampling show higher H2 in recent years at all latitudes. The marine boundary layer (MBL) global mean H2 was 552.8 ppb in 2021, 20.2 ± 0.2 ppb higher compared to 2010. A 10 ppb or more increase over the 2010–2021 average annual cycle was detected in 2016 for MBL zonal means in the tropics and in the Southern Hemisphere. Carbon monoxide measurements in the same-air samples suggest large biomass burning events in different regions likely contributed to the observed interannual variability at different latitudes. The NOAA H2 measurements from 2009 to 2021 are now based on the World Meteorological Organization Global Atmospheric Watch (WMO GAW) H2 mole fraction calibration scale, developed and maintained by the Max Planck Institute for Biogeochemistry (MPI-BGC), Jena, Germany. GML maintains eight H2 primary calibration standards to propagate the WMO scale. These are gravimetric hydrogen-in-air mixtures in electropolished stainless steel cylinders (Essex Industries, St. Louis, MO), which are stable for H2. These mixtures were calibrated at the MPI-BGC, the WMO Central Calibration Laboratory (CCL) for H2, in late 2020 and span the range 250–700 ppb. We have used the CCL assignments to propagate the WMO H2 calibration scale to NOAA air measurements performed using gas chromatography and helium pulse discharge detector instruments since 2009. To propagate the scale, NOAA uses a hierarchy of secondary and tertiary standards, which consist of high-pressure whole-air mixtures in aluminum cylinders, calibrated against the primary and secondary standards, respectively. Hydrogen at the parts per billion level has a tendency to increase in aluminum cylinders over time. We fit the calibration histories of these standards with zero-, first-, or second-order polynomial functions of time and use the time-dependent mole fraction assignments on the WMO scale to reprocess all tank air and flask air H2 measurement records. The robustness of the scale propagation over multiple years is evaluated with the regular analysis of target air cylinders and with long-term same-air measurement comparison efforts with WMO GAW partner laboratories. Long-term calibrated, globally distributed, and freely accessible measurements of H2 and other gases and isotopes continue to be essential to track and interpret regional and global changes in the atmosphere composition. The adoption of the WMO H2 calibration scale and subsequent reprocessing of NOAA atmospheric data constitute a significant improvement in the NOAA H2 measurement records.

Pétron, Gabrielle

Spectral radiance calibrations between 165-300 nm - An interlaboratory comparison

The spectral radiance of deuterium lamps calibrated by the Max-Planck-Institut fuer Astronomie (MPI), by the U.S. National Bureau of Standards (NBS), and by the Physikalisch-Technische Bundesanstalt (PTB) are compared to check the agreement of UV radiometric scales. The NBS group used the optically thin continuum radiation from a wall-stabilized hydrogen arc as its fundamental radiometric standard, while the MPI and PTB groups used the synchrotron radiation facility in DESY. It is found that the spectral radiance scales based upon the DESY synchrotron and the NBS hydrogen arc are consistent, at least for one wavelength relative to another.

Bridges, J. M.

System identification of the JPL micro-precision interferometer truss - Test-analysis reconciliation

The JPL Micro-Precision Interferometer (MPI) is a testbed for studying the use of control-structure interaction technology in the design of space-based interferometers. A layered control architecture will be employed to regulate the interferometer optical system to tolerances in the nanometer range. An important aspect of designing and implementing the control schemes for such a system is the need for high fidelity, test-verified analytical structural models. This paper focuses on one aspect of the effort to produce such a model for the MPI structure, test-analysis model reconciliation. Pretest analysis, modal testing, and model refinement results are summarized for a series of tests at both the component and full system levels.

Red-Horse, J. R.

3D Multistage Simulation of Each Component of the GE90 Turbofan Engine

A 3D multistage simulation of each component of the GE90 Turbofan engine has been made. This includes 49 blade rows. A coupled simulation of all blade rows will be made very soon. The simulation is running using two levels of parallelism. The first level is on a blade row basis with information shared using files. The second level is using a grid domain decomposition with information shared using MPI. Timings will be shown for running on the SP2, an SGI Origin and a distributed system of HP workstations. On the HP workstations, the CHIMP version of MPI is used, with queuing supplied by LSF (Load Sharing Facility). A script-based control system is used to ensure reliability. An MPEG movie illustrating the flow simulation of the engine has been created using PV3, a parallel visualization library created by Bob Haimes of MIT. PVM is used to create a virtual machine from 10 HP workstations and display on an SGI workstation. A representative component simulation will be compared to rig data to demonstrate its usefulness in turbomachinery design and analysis.

Turner, Mark

Aerodynamic Shape Optimization Using A Combined Distributed/Shared Memory Paradigm

Current parallel computational approaches involve distributed and shared memory paradigms. In the distributed memory paradigm, each processor has its own independent memory. Message passing typically uses a function library such as MPI or PVM. In the shared memory paradigm, such as that used on the SGI Origin 2000 machine, compiler directives are used to instruct the compiler to schedule multiple threads to perform calculations. In this paradigm, it must be assured that processors (threads) do not simultaneously access regions of memory in such away that errors would occur. This paper utilizes the latest version of the SGI MPI function library to combine the two parallelization paradigms to perform aerodynamic shape optimization of a generic wing/body.

Cheung, Samson

Towards Flange-to-Flange Turbopump Simulations for Liquid Rocket Engines

The primary objective of this research is to support the design of liquid rocket systems for the Advanced Space Transportation System. Since the space launch systems in the near future are likely to rely on liquid rocket engines, increasing the efficiency and reliability of the engine components is an important task. One of the major problems in the liquid rocket engine is to understand fluid dynamics of fuel and oxidizer flows from the fuel tank to plume. Understanding the flow through the entire turbopump geometry through numerical simulation will be of significant value toward design. This will help to improve safety of future space missions. One of the milestones of this effort is to develop, apply and demonstrate the capability and accuracy of 3D CFD methods as efficient design analysis tools on high performance computer platforms. The development of the MPI and MLP versions of the INS3D code is currently underway. The serial version of INS3D code is a multidimensional incompressible Navier-Stokes solver based on overset grid technology. INS3D-MPI is based on the explicit massage-passing interface across processors and is primarily suited for distributed memory systems. INS3D-MLP is based on multi-level parallel method and is suitable for distributed-shared memory systems. For the entire turbopump simulations, moving boundary capability and an efficient time-accurate integration methods are build in the flow solver. To handle the geometric complexity and moving boundary problems, overset grid scheme is incorporated with the solver that new connectivity data will be obtained at each time step. The Chimera overlapped grid scheme allows subdomains move relative to each other, and provides a great flexibility when the boundary movement creates large displacements. The performance of the two time integration schemes for time-accurate computations is investigated. For an unsteady flow which requires small physical time step, the pressure projection method was found to be computationally efficient since it does not require any subiterations procedure. It was observed that the artificial compressibility method requires a fast convergence scheme at each physical time step in order to satisfy incompressibility condition. This was obtained by using a GMRES-ILU(0) solver in our computations. When a line-relaxation scheme was used, the time accuracy was degraded and time-accurate computations became very expensive. The current geometry for the LOX boost turbopump has various rotating and stationary components, such as inducer, stators, kicker, hydrolic turbine, where the flow is extremely unsteady. Figure 1 shows the geometry and computed surface pressure of the inducer. The inducer and the hydrolic turbine rotate in different rotational speed.

Kiris, Cetin

Supercomputing Aspects for Simulating Incompressible Flow

The primary objective of this research is to support the design of liquid rocket systems for the Advanced Space Transportation System. Since the space launch systems in the near future are likely to rely on liquid rocket engines, increasing the efficiency and reliability of the engine components is an important task. One of the major problems in the liquid rocket engine is to understand fluid dynamics of fuel and oxidizer flows from the fuel tank to plume. Understanding the flow through the entire turbo-pump geometry through numerical simulation will be of significant value toward design. One of the milestones of this effort is to develop, apply and demonstrate the capability and accuracy of 3D CFD methods as efficient design analysis tools on high performance computer platforms. The development of the Message Passage Interface (MPI) and Multi Level Parallel (MLP) versions of the INS3D code is currently underway. The serial version of INS3D code is a multidimensional incompressible Navier-Stokes solver based on overset grid technology, INS3D-MPI is based on the explicit massage-passing interface across processors and is primarily suited for distributed memory systems. INS3D-MLP is based on multi-level parallel method and is suitable for distributed-shared memory systems. For the entire turbo-pump simulations, moving boundary capability and efficient time-accurate integration methods are built in the flow solver, To handle the geometric complexity and moving boundary problems, an overset grid scheme is incorporated with the solver so that new connectivity data will be obtained at each time step. The Chimera overlapped grid scheme allows subdomains move relative to each other, and provides a great flexibility when the boundary movement creates large displacements. Two numerical procedures, one based on artificial compressibility method and the other pressure projection method, are outlined for obtaining time-accurate solutions of the incompressible Navier-Stokes equations. The performance of the two methods is compared by obtaining unsteady solutions for the evolution of twin vortices behind a flat plate. Calculated results are compared with experimental and other numerical results. For an unsteady flow, which requires small physical time step, the pressure projection method was found to be computationally efficient since it does not require any subiteration procedure. It was observed that the artificial compressibility method requires a fast convergence scheme at each physical time step in order to satisfy the incompressibility condition. This was obtained by using a GMRES-ILU(0) solver in present computations. When a line-relaxation scheme was used, the time accuracy was degraded and time-accurate computations became very expensive.

Kwak, Dochan

Reflective Insulation

NRC-2 Superinsulation, manufactured by Metallized Products, Inc. (MPI), is designed for superconducting magnets used in MRI systems and particle accelerators. It is a thin, polyester film characterized by a unique crinkled surface that provides surface stand-off between layers and minimizes heat transfer in multilayer applications. NRC-2/Two is a two-sided metallized film. The material, originally developed as a skin for balloon-like satellites, was later used by NASA as a thermal barrier. MPI worked with NASA on the development of the original material and now supplies it for both space and consumer applications.

Source record

NAS Parallel Benchmarks

We describe a new problem size, called Class D, for the NAS Parallel Benchmarks (NPB), whose MPI source code implementation is being released as NPB 2.4. A brief rationale is given for how the new class is derived. We also describe the modifications made to the MPI (Message Passing Interface) implementation to allow the new class to be run on systems with 32-bit integers, and with moderate amounts of memory. Finally, we give the verification values for the new problem size.

VanderWijngaart, Rob

Performance Analysis of a Hybrid Overset Multi-Block Application on Multiple Architectures

This paper presents a detailed performance analysis of a multi-block overset grid compu- tational fluid dynamics app!ication on multiple state-of-the-art computer architectures. The application is implemented using a hybrid MPI+OpenMP programming paradigm that exploits both coarse and fine-grain parallelism; the former via MPI message passing and the latter via OpenMP directives. The hybrid model also extends the applicability of multi-block programs to large clusters of SNIP nodes by overcoming the restriction that the number of processors be less than the number of grid blocks. A key kernel of the application, namely the LU-SGS linear solver, had to be modified to enhance the performance of the hybrid approach on the target machines. Investigations were conducted on cacheless Cray SX6 vector processors, cache-based IBM Power3 and Power4 architectures, and single system image SGI Origin3000 platforms. Overall results for complex vortex dynamics simulations demonstrate that the SX6 achieves the highest performance and outperforms the RISC-based architectures; however, the best scaling performance was achieved on the Power3.

Djomehri, M. Jahed

GSRP/David Marshall: Fully Automated Cartesian Grid CFD Application for MDO in High Speed Flows

With the renewed interest in Cartesian gridding methodologies for the ease and speed of gridding complex geometries in addition to the simplicity of the control volumes used in the computations, it has become important to investigate ways of extending the existing Cartesian grid solver functionalities. This includes developing methods of modeling the viscous effects in order to utilize Cartesian grids solvers for accurate drag predictions and addressing the issues related to the distributed memory parallelization of Cartesian solvers. This research presents advances in two areas of interest in Cartesian grid solvers, viscous effects modeling and MPI parallelization. The development of viscous effects modeling using solely Cartesian grids has been hampered by the widely varying control volume sizes associated with the mesh refinement and the cut cells associated with the solid surface. This problem is being addressed by using physically based modeling techniques to update the state vectors of the cut cells and removing them from the finite volume integration scheme. This work is performed on a new Cartesian grid solver, NASCART-GT, with modifications to its cut cell functionality. The development of MPI parallelization addresses issues associated with utilizing Cartesian solvers on distributed memory parallel environments. This work is performed on an existing Cartesian grid solver, CART3D, with modifications to its parallelization methodology.

Source record

Hot Chips and Hot Interconnects for High End Computing Systems

I will discuss several processors: 1. The Cray proprietary processor used in the Cray X1; 2. The IBM Power 3 and Power 4 used in an IBM SP 3 and IBM SP 4 systems; 3. The Intel Itanium and Xeon, used in the SGI Altix systems and clusters respectively; 4. IBM System-on-a-Chip used in IBM BlueGene/L; 5. HP Alpha EV68 processor used in DOE ASCI Q cluster; 6. SPARC64 V processor, which is used in the Fujitsu PRIMEPOWER HPC2500; 7. An NEC proprietary processor, which is used in NEC SX-6/7; 8. Power 4+ processor, which is used in Hitachi SR11000; 9. NEC proprietary processor, which is used in Earth Simulator. The IBM POWER5 and Red Storm Computing Systems will also be discussed. The architectures of these processors will first be presented, followed by interconnection networks and a description of high-end computer systems based on these processors and networks. The performance of various hardware/programming model combinations will then be compared, based on latest NAS Parallel Benchmark results (MPI, OpenMP/HPF and hybrid (MPI + OpenMP). The tutorial will conclude with a discussion of general trends in the field of high performance computing, (quantum computing, DNA computing, cellular engineering, and neural networks).

Saini, Subhash