Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “supercomputer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,081 records · Page 60

Using Docker Containers to Extend Reproducibility Architecture for the NASA Earth Exchange (NEX)

NASA Earth Exchange (NEX) is a data, supercomputing and knowledge collaboratory that houses NASA satellite, climate and ancillary data where a focused community can come together to address large-scale challenges in Earth sciences. As NEX has been growing into a petabyte-size platform for analysis, experiments and data production, it has been increasingly important to enable users to easily retrace their steps, identify what datasets were produced by which process chains, and give them ability to readily reproduce their results. This can be a tedious and difficult task even for a small project, but is almost impossible on large processing pipelines. We have developed an initial reproducibility and knowledge capture solution for the NEX, however, if users want to move the code to another system, whether it is their home institution cluster, laptop or the cloud, they have to find, build and install all the required dependencies that would run their code. This can be a very tedious and tricky process and is a big impediment to moving code to data and reproducibility outside the original system. The NEX team has tried to assist users who wanted to move their code into OpenNEX on Amazon cloud by creating custom virtual machines with all the software and dependencies installed, but this, while solving some of the issues, creates a new bottleneck that requires the NEX team to be involved with any new request, updates to virtual machines and general maintenance support. In this presentation, we will describe a solution that integrates NEX and Docker to bridge the gap in code-to-data migration. The core of the solution is saemi-automatic conversion of science codes, tools and services that are already tracked and described in the NEX provenance system, to Docker - an open-source Linux container software. Docker is available on most computer platforms, easy to install and capable of seamlessly creating and/or executing any application packaged in the appropriate format. We believe this is an important step towards seamless process deployment in heterogeneous environments that will enhance community access to NASA data and tools in a scalable way, promote software reuse, and improve reproducibility of scientific results.

earth exchange↗

A GeoNEX-Based High-Spatiotemporal-Resolution Product of Land Surface Downward Shortwave Radiation and Photosynthetically Active Radiation

Surface downward shortwave radiation (DSR) and photosynthetically active radiation (PAR) play critical roles in the Earth’s surface processes. As the main inputs of various ecological, hydrological, carbon, and solar photovoltaic models, increasing requirements for high-spatiotemporal-resolution DSR and PAR es- timation with high accuracy have been observed in recent years. However, few existing products satisfy all of these requirements. This study employed a well-established physical-based lookup table (LUT) approach to the GeoNEX gridded top-of-atmosphere bidirectional reflectance factor data acquired by the Advanced Hi- mawari Imager (AHI) and Advanced Baseline Imager (ABI) sensors. It produced a data product of DSR and PAR over both AHI and ABI coverage at an hourly temporal step with a 1 km spatial resolution. GeoNEX DSR data were validated over 63 stations, and GeoNEX PAR data were validated over 27 stations. The vali- dation showed that the new GeoNEX DSR and PAR products have accuracy higher than other existing prod- ucts, with root mean square error (RMSE) of hourly GeoNEX DSR achieving 74.3Wm−2 (18.0%), daily DSR estimation achieving 18.0 W m−2 (9.2 %), hourly GeoNEX PAR achieving 34.9 W m−2 (19.6 %), and daily PAR achieving 9.5 W m−2 (10.5 %). The study also demonstrated the application of the high-spatiotemporal- resolution GeoNEX DSR product in investigating the spatial heterogeneity and temporal variability of surface solar radiation. The data product can be freely accessed through the NASA Advanced Supercomputing Division GeoNEX data portal: https://data.nas.nasa.gov/geonex/geonexdata/GOES16/GEONEX-L2/DSR-PAR/ (last ac- cess: 12 March 2023) and https://data.nas.nasa.gov/geonex/geonexdata/HIMAWARI8/GEONEX-L2/DSR-PAR/ (last access: 12 March 2023) (https://doi.org/10.5281/zenodo.7023863; Wang and Li, 2022).

Surface downward shortwave radiation (DSR)↗

A Comparative Study of Contrail Frequency Indices and GOES-16 Contrail Data Set

Contrail formations have been shown to contribute to the greenhouse effect: they are practically transparent to incoming solar radiation and do little to reflect heat away from Earth but are highly effective at trapping heat within Earth’s atmosphere. To understand the impact contrails have on climate change, contrail frequency indices (CFIs) can be used as a method to quantify aircraft-induced persistent contrails. These indices are capable of tracking long-term contrail formation and identify regions of airspace with the highest contrail formation rates. In this aper, an algorithm is proposed which is capable of using NASA Sherlock and Global Forecast System (GFS) datasets and computing CFIs over large geographic regions and long temporal intervals using NASA Ames’ High-End Computing Capability (HECC) supercomputing system. CFIs are computed using nowcast weather data and previously flown flight tracks. This paper calculated the CFIs of all twenty Air Route Traffic Control Centers in the National Airspace System on October 28th, 2019 and compared the distribution of non-zero CFIs with observed contrail data collected from GOES-16 Satellite data in order to assess the accuracy of the CFI system as a contrail prediction model. It was ultimately determined that the computed CFIs were broadly distributed in the same way as the GOES-16 contrail data and that the individual CFIs computed at the latitude/longitude points at which GOES-16 contrail masks were available had high precision and recall (at 0.75 and 0.86 respectively). While these validation results bode well for the accuracy of the CFI method, the number of provided GOES-16 masks was quite small. Future work should aim to increase the size of the GOES-16 dataset in order to perform a more comprehensive comparison between these two datasets.

Contrails↗

A Comparative Study of Contrail Frequency Indices and GOES-16 Contrail Data Set

Contrail formations have been shown to contribute to the greenhouse effect: they are practically transparent to incoming solar radiation and do little to reflect heat away from Earth but are highly effective at trapping heat within Earth’s atmosphere. To understand the impact contrails have on climate change, contrail frequency indices (CFIs) can be used as a method to quantify aircraft-induced persistent contrails. These indices are capable of tracking long-term contrail formation and identify regions of airspace with the highest contrail formation rates. In this aper, an algorithm is proposed which is capable of using NASA Sherlock and Global Forecast System (GFS) datasets and computing CFIs over large geographic regions and long temporal intervals using NASA Ames’ High-End Computing Capability (HECC) supercomputing system. CFIs are computed using nowcast weather data and previously flown flight tracks. This paper calculated the CFIs of all twenty Air Route Traffic Control Centers in the National Airspace System on October 28th, 2019 and compared the distribution of non-zero CFIs with observed contrail data collected from GOES-16 Satellite data in order to assess the accuracy of the CFI system as a contrail prediction model. It was ultimately determined that the computed CFIs were broadly distributed in the same way as the GOES-16 contrail data and that the individual CFIs computed at the latitude/longitude points at which GOES-16 contrail masks were available had high precision and recall (at 0.75 and 0.86 respectively). While these validation results bode well for the accuracy of the CFI method, the number of provided GOES-16 masks was quite small. Future work should aim to increase the size of the GOES-16 dataset in order to perform a more comprehensive comparison between these two datasets.

Contrails↗

End-to-end experiment management in HPC

Experiment management in any domain is challenging. There is a perpetual feedback loop cycling through planning, execution, measurement, and analysis. The lifetime of a particular experiment can be limited to a single cycle although many require myriad more cycles before definite results can be obtained. Within each cycle, a large number of subexperiments may be executed in order to measure the effects of one or more independent variables. Experiment management in high performance computing (HPC) follows this general pattern but also has three unique characteristics. One, computational science applications running on large supercomputers must deal with frequent platform failures which can interrupt, perturb, or terminate running experiments. Two, these applications typically integrate in parallel using MPI as their communication medium. Three, there is typically a scheduling system (e.g. Condor, Moab, SGE, etc.) acting as a gate-keeper for the HPC resources. In this paper, we introduce LANL Experiment Management (LEM), an experimental management framework simplifying all four phases of experiment management. LEM simplifies experiment planning by allowing the user to describe their experimental goals without having to fully construct the individual parameters for each task. To simplify execution, LEM dispatches the subexperiments itself thereby freeing the user from remembering the often arcane methods for interacting with the various scheduling systems. LEM provides transducers for experiments that automatically measure and record important information about each subexperiment; these transducers can easily be extended to collect additional measurements specific to each experiment. Finally, experiment analysis is simplified by providing a general database visualization framework that allows users to quickly and easily interact with their measured data.

97 MATHEMATICS AND COMPUTING↗

Factors impacting performance of multithreaded sparse triangular solve.

As computational science applications grow more parallel with multi-core supercomputers having hundreds of thousands of computational cores, it will become increasingly difficult for solvers to scale. Our approach is to use hybrid MPI/threaded numerical algorithms to solve these systems in order to reduce the number of MPI tasks and increase the parallel efficiency of the algorithm. However, we need efficient threaded numerical kernels to run on the multi-core nodes in order to achieve good parallel efficiency. In this paper, we focus on improving the performance of a multithreaded triangular solver, an important kernel for preconditioning. We analyze three factors that affect the parallel performance of this threaded kernel and obtain good scalability on the multi-core nodes for a range of matrix sizes.

99 GENERAL AND MISCELLANEOUS↗

Parallel k-Means Clustering for Quantitative Ecoregion Delineation Using Large Data Sets

Identification of geographic ecoregions has long been of interest to environmental scientists and ecologists for identifying regions of similar ecological and environmental conditions. Such classifications are important for predicting suitable species ranges, for stratification of ecological samples, and to help prioritize habitat preservation and remediation efforts. Hargrove and Hoffman (1999, 2009) have developed geographical spatio-temporal clustering algorithms and codes and have successfully applied them to a variety of environmental science domains, including ecological regionalization; environmental monitoring network design; analysis of satellite-, airborne-, and ground-based remote sensing, and climate model-model and model-measurement intercomparison. With the advances in state-of-the-art satellite remote sensing and climate models, observations and model outputs are available at increasingly high spatial and temporal resolutions. Long time series of these high resolution datasets are extremely large in size and growing. Analysis and knowledge extraction from these large datasets are not just algorithmic and ecological problems, but also pose a complex computational problem. This paper focuses on the development of a massively parallel multivariate geographical spatio-temporal clustering code for analysis of very large datasets using tens of thousands processors on one of the fastest supercomputers in the world.

54 ENVIRONMENTAL SCIENCES↗

Self-Consistant Numerical Modeling of E-Cloud Driven Instability of a Bunch Train in the CERN SPS

The simulation package WARP-POSINST was recently upgraded for handling multiple bunches and modeling concurrently the electron cloud buildup and its effect on the beam, allowing for direct self-consistent simulation of bunch trains generating, and interacting with, electron clouds. We have used the WARP-POSINST package on massively parallel supercomputers to study the growth rate and frequency patterns in space-time of the electron cloud driven transverse instability for a proton bunch train in the CERN SPS accelerator. Results suggest that a positive feedback mechanism exists between the electron buildup and the e-cloud driven transverse instability, leading to a net increase in predicted electron density. Comparisons to selected experimental data are also given. Electron clouds have been shown to trigger fast growing instabilities on proton beams circulating in the SPS and other accelerators. So far, simulations of electron cloud buildup and their effects on beam dynamics have been performed separately. This is a consequence of the large computational cost of the combined calculation due to large space and time scale disparities between the two processes. We have presented the latest improvements of the simulation package WARP-POSINST for the simulation of self-consistent ecloud effects, including mesh refinement, and generation of electrons from gas ionization and impact at the pipe walls. We also presented simulations of two consecutive bunches interacting with electrons clouds in the SPS, which included generation of secondary electrons. The distribution of electrons in front of the first beam was initialized from a dump taken from a preceding buildup calculation using the POSINST code. In this paper, we present an extension of this work where one full batch of 72 bunches is simulated in the SPS, including the entire buildup calculation and the self-consistent interaction between the bunches and the electrons. Comparisons to experimental data are also given.

43 PARTICLE ACCELERATORS↗

Direct Numerical Modeling of E-Cloud Driven Instability of a Bunch Train in the CERN SPS

The simulation package WARP-POSINST was recently upgraded for handling multiple bunches and modeling concurrently the electron cloud buildup and its effect on the beam, allowing for direct self-consistent simulation of bunch trains generating, and interacting with, electron clouds. We have used the WARP-POSINST package on massively parallel supercomputers to study the buildup and interaction of electron clouds with a proton bunch train in the CERN SPS accelerator. Results suggest that a positive feedback mechanism exists between the electron buildup and the e-cloud driven transverse instability, leading to a net increase in predicted electron density. Electron clouds have been shown to trigger fast growing instabilities on proton beams circulating in the SPS and other accelerators. So far, simulations of electron cloud buildup and their effects on beam dynamics have been performed separately. This is a consequence of the large computational cost of the combined calculation due to large space and time scale disparities between the two processes. We have presented the latest improvements of the simulation package WARP-POSINST for the simulation of self-consistent ecloud effects, including mesh refinement, and generation of electrons from gas ionization and impact at the pipe walls. We also presented simulations of two consecutive bunches interacting with electrons clouds in the SPS, which included generation of secondary electrons. The distribution of electrons in front of the first beam was initialized from a dump taken from a preceding buildup calculation using the POSINST code. In this paper, we present an extension of this work where one full batch of 72 bunches is simulated in the SPS, including the entire buildup calculation and the self-consistent interaction between the bunches and the electrons.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

PyMercury: Interactive Python for the Mercury Monte Carlo Particle Transport Code

Monte Carlo particle transport applications are often written in low-level languages (C/C++) for optimal performance on clusters and supercomputers. However, this development approach often sacrifices straightforward usability and testing in the interest of fast application performance. To improve usability, some high-performance computing applications employ mixed-language programming with high-level and low-level languages. In this study, we consider the benefits of incorporating an interactive Python interface into a Monte Carlo application. With PyMercury, a new Python extension to the Mercury general-purpose Monte Carlo particle transport code, we improve application usability without diminishing performance. In two case studies, we illustrate how PyMercury improves usability and simplifies testing and validation in a Monte Carlo application. In short, PyMercury demonstrates the value of interactive Python for Monte Carlo particle transport applications. In the future, we expect interactive Python to play an increasingly significant role in Monte Carlo usage and testing.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Factors impacting performance of multithreaded triangular solve.

As computational science applications grow more parallel with multi-core supercomputers having hundreds of thousands of computational cores, it will become increasingly difficult for solvers to scale. Our approach is to use hybrid MPI/threaded numerical algorithms to solve these systems in order to reduce the number of MPI tasks and increase the parallel efficiency of the algorithm. However, we need efficient threaded numerical kernels to run on the multi-core nodes in order to achieve good parallel efficiency. In this paper, we focus on improving the performance of a multithreaded triangular solver, an important kernel for preconditioning. We analyze three factors that affect the parallel performance of this threaded kernel and obtain good scalability on the multi-core nodes for a range of matrix sizes.

99 GENERAL AND MISCELLANEOUS↗

Bridging the gaps : joining information sources with Splunk.

Supercomputers are composed of many diverse components, operated at a variety of scales, and function as a coherent whole. The resulting logs are thus diverse in format, interrelated at multiple scales, and provide evidence of faults across subsystems. When combined with system configuration information, insights on both the downstream effects and upstream causes of events can be determined. However, difficulties in joining the data and expressing complex queries slow the speed at which actionable insights can be obtained. Effectively connecting data experts and data miners faces similar hurdles. This paper describes our experience with applying the Splunk log analysis tool as a vehicle to combine both data, and people. Splunk's search language, lookups, macros, and subsearches reduce hours of tedium to seconds of simplicity, and its tags, saved searches, and dashboards offer both operational insights and collaborative vehicles.

99 GENERAL AND MISCELLANEOUS↗

Numerical Verification of the Power Transfer and Wakefield Coupling in the Clic Two-Beam Accelerator

The Compact Linear Collider (CLIC) provides a path to a multi-TeV accelerator to explore the energy frontier of High Energy Physics. Its two-beam accelerator (TBA) concept envisions complex 3D structures, which must be modeled to high accuracy so that simulation results can be directly used to prepare CAD drawings for machining. The required simulations include not only the fundamental mode properties of the accelerating structures but also the Power Extraction and Transfer Structure (PETS), as well as the coupling between the two systems. Time-domain simulations will be performed to understand pulse formation, wakefield damping, fundamental power transfer and wakefield coupling in these structures. Applying SLAC's parallel finite element code suite, these large-scale problems will be solved on some of the largest supercomputers available. The results will help to identify potential issues and provide new insights on the design, leading to further improvements on the novel two-beam accelerator scheme.

43 PARTICLE ACCELERATORS↗

Scientists and artists: ""Hey! You got art in my science! You got science on my art

The pairing of science and art has proven to be a powerful combination since the Renaissance. The combination of these two seemingly disparate disciplines ensured that even complex scientific theories could be explored and effectively communicated to both the subject matter expert and the layman. In modern times, science and art have frequently been considered disjoint, with objectives, philosophies, and perspectives often in direct opposition to each other. However, given the technological advances in computer science and high fidelity 3-D graphics development tools, this marriage of art and science is once again logically complimentary. Art, in the form of computer graphics and animation created on supercomputers, has already proven to be a powerful tool for improving scientific research and providing insight into nuclear phenomena. This paper discusses the power of pairing artists with scientists and engineers in order to pursue the possibilities of a widely accessible lightweight, interactive approach. We will use a discussion of photo-realism versus stylization to illuminate the expected beneficial outcome of such collaborations and the societal advantages gained by a non-traditional pa11nering of these two fields.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Python for Development of OpenMP and CUDA Kernels for Multidimensional Data

Design of data structures for high performance computing (HPC) is one of the principal challenges facing researchers looking to utilize heterogeneous computing machinery. Heterogeneous systems derive cost, power, and speed efficiency by being composed of the appropriate hardware for the task. Yet, each type of processor requires a specific organization of the application state in order to achieve peak performance. Discovering this and refactoring the code can be a challenging and time-consuming task for the researcher, as the data structures and the computational model must be co-designed. We present a methodology that uses Python as the environment for which to explore tradeoffs in both the data structure design as well as the code executing on the computation accelerator. Our method enables multi-dimensional arrays to be used effectively in any target environment. We have chosen to focus on OpenMP and CUDA environments, thus exploring the development of optimized kernels for the two most common classes of computing hardware available today: multi-core CPU and GPU. Python s large palette of file and network access routines, its associative indexing syntax and support for common HPC environments makes it relevant for diverse hardware ranging from laptops through computing clusters to the highest performance supercomputers. Our work enables researchers to accelerate the development of their codes on the computing hardware of their choice.

43 PARTICLE ACCELERATORS↗

Data Mining in Earth System Science (DMESS 2011)

From field-scale measurements to global climate simulations and remote sensing, the growing body of very large and long time series Earth science data are increasingly difficult to analyze, visualize, and interpret. Data mining, information theoretic, and machine learning techniques - such as cluster analysis, singular value decomposition, block entropy, Fourier and wavelet analysis, phase-space reconstruction, and artificial neural networks - are being applied to problems of segmentation, feature extraction, change detection, model-data comparison, and model validation. The size and complexity of Earth science data exceed the limits of most analysis tools and the capacities of desktop computers. New scalable analysis and visualization tools, running on parallel cluster computers and supercomputers, are required to analyze data of this magnitude. This workshop will demonstrate how data mining techniques are applied in the Earth sciences and describe innovative computer science methods that support analysis and discovery in the Earth sciences.

99 GENERAL AND MISCELLANEOUS↗

Dynamic Time-Variant Connection Management for PGAS Models on InfiniBand

InfiniBand (IB) has established itself as a promising network infrastructure for high-end cluster computing systems as evidenced by its usage in the Top500 supercomputers today. While the IB standard describes multiple communication models (including reliable-connection (RC), and unreliable datagram (UD)), most of its promising features such as remote direct memory access (RDMA), hardware atomics and network fault tolerance are only available for the RC model which requires connections between communicating process pairs. In the past, several researchers have proposed on-demand connection management techniques that establish connections when there is a need to communicate, and not before. While such techniques work well for algorithms and applications that only communicate with a small set of processes in their life-time, there exists a broad set of applications that do not follow this trend. For example, applications that perform dynamic load balancing and adaptive work stealing have a small set of communicating neighbors at any given time, but over time the total number of neighbors can be very high; in some cases, equal to the entire system size. In this paper, we present a dynamic time-variant connection management approach that establishes connections on-demand like previous approaches, but further intelligently tears down some of the unused connections as well. While connection tear-down itself is relevant for any programming model, different models have different complexities. In this paper, we study the Global Arrays (GA) PGAS model for two reasons: (1) the simple one-sided communication primitives provided by GA and other PGAS models ensure that connection requests are always initiated by the origin process without explicit synchronization with the target process---this makes connection tear-down simpler to handle; and (2) GA supports several applications that demonstrate this behavior making it an obvious first target for the proposed enhancements. We evaluate our proposed approach using several micro-benchmarks as well as the NWChem computational chemistry application on more than 6000 processes, and show that our approach can significantly reduce the memory requirements of the communication library while maintaining its performance. To the best of our knowledge, this is the first design, implementation and evaluation of connection tear-down protocols over InfiniBand.

99 GENERAL AND MISCELLANEOUS↗

Probabilistic Communication and I/O Tracing with Deterministic Replay at Scale

With today's petascale supercomputers, applications often exhibit low efficiency, such as poor communication and I/O performance, that can be diagnosed by analysis tools. However, these tools either produce extremely large trace files that complicate performance analysis, or sacrifice accuracy to collect high-level statistical information using crude averaging. This work contributes Scala-H-Trace, which features more aggressive trace compression than any previous approach, particularly for applications that do not show strict regularity in SPMD behavior. Scala-H-Trace uses histograms expressing the probabilistic distribution of arbitrary communication and I/O parameters to capture variations. Yet, where other tools fail to scale, Scala-H-Trace guarantees trace files of near constant size, even for variable communication and I/O patterns, producing trace files orders of magnitudes smaller than using prior approaches. We demonstrate the ability to collect traces of applications running on thousands of processors with the potential to scale well beyond this level. We further present the first approach to deterministically replay such probabilistic traces (a) without deadlocks and (b) in a manner closely resembling the original applications. Our results show either near constant sized traces or only sub-linear increases in trace file sizes irrespective of the number of nodes utilized. Even with the aggressively compressed histogram-based traces, our replay times are within 12% to 15% of the runtime of original codes. Such concise traces resembling the behavior of production-style codes closely and our approach of deterministic replay of probabilistic traces are without precedence.

99 GENERAL AND MISCELLANEOUS↗