Performance Analysis for Using Non-Volatile Memory DIMMs: Opportunities and Challenges.
Abstract not provided.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Abstract not provided.
DRAM technology is the main building block of main memory, however, DRAM scaling is becoming very challenging. The main issues for DRAM scaling are the increasing error rates with each new generation, the geometric and physical constraints of scaling the capacitor part of the DRAM cells, and the high power consumption caused by the continuous need for refreshing cell values. At the same time, emerging Non- Volatile Memory (NVM) technologies, such as Phase-Change Memory (PCM), are emerging as promising replacements for DRAM. NVMs, when compared to current technologies e.g., NAND-based ash, have latencies comparable to DRAM. Additionally, NVMs are non-volatile, which eliminates the need for refresh power and enables persistent memory applications. Finally, NVMs have promising densities and the potential for multi-level cell (MLC) storage.
As supercomputers continue to get faster and more powerful in the future, they will also have more nodes. If nothing is done, then the amount of memory in supercomputer clusters will soon grow large enough that memory failures will be unmanageable to deal with by manually replacing memory DIMMs. "Improving Memory Error Handling Using Linux" is a process oriented method to solve this problem by using the Linux kernel to disable (offline) faulty memory pages containing bad addresses, preventing them from being used again by a process. The process of offlining memory pages simplifies error handling and results in reducing both hardware and manpower costs required to run Los Alamos National Laboratory (LANL) clusters. This process will be necessary for the future of supercomputing to allow the development of exascale computers. It will not be feasible without memory error handling to manually replace the number of DIMMs that will fail daily on a machine consisting of 32-128 petabytes of memory. Testing reveals the process of offlining memory pages works and is relatively simple to use. As more and more testing is conducted, the entire process will be automated within the high-performance computing (HPC) monitoring software, Zenoss, at LANL.
The opsim4 operations simulation program for the LSST astronomical survey uses a database of seeing values covering the range of times to besimulated. Idescribethe creation of such a database using Dual Image Motion Monitor(DIMM)datacollected at Cerro Pachon from 2004-03-17 to 2019-10-07. In times during which the data overlap, I compare the distribution of DIMM seeing values to the seeing measured in DECamimages,takenatasite 10kmaway. Becauseinstrumentalproblemsinthe DIMMmay indicate unreliablemeasurements,cutsonimagequality(asindicatedby the measured Strehlratio)wereexplored. TheDIMMhassignificantgaps,soImodel thedata(withandwithoutcutsonStrehlratio)andgenerateartificialdatainthegaps according to the model. The model consists of a sinusoidal variation with a period of one year, an autoregressive (AR1) model for variations in mean seeing from one night to the next, and another AR1 model for variations on a 5 minute timescale. I create four databases according to thisprocedure, twobasedonDIMMdatastarting 2006-01-01 (with and without a Strehl ratio cut), and two starting 2009-01-01. I then run opsim simulations using each, and an otherwise identical simulation using the default seeing database, and explore the differences
As part of the ATLAS Phase-I Upgrade, the global Feature EXtractor (gFEX) is one of several hardware modules designed to help maintain the ATLAS Level-1 trigger acceptance rate with the increasing Large Hadron Collider (LHC) luminosity and the increasing Pile-Up conditions. The gFEX is used to identify patterns of energy associated with the hadronic decays of high momentum Higgs, W & Z bosons, top quarks, and exotic particles in real time at the 40 MHz LHC bunch crossing rate. The board is required to receive coarse-granularity (Δη×Δphiv = 0.2×0.2 gTower) information from the entire ATLAS calorimeters on 276 optical fibers. A prototype v1 with one Xilinx ZYNQ FPGA, and one Vertex-7 FPGA for technology validation has been designed and tested in 2015. With the lessons learned from the prototype v1, a prototype v2 with three Vertex UltraScale FPGAs and one ZYNQ FPGA has been implemented to verify full functionalities of gFEX in 2016. Based on the prototype v2 design, a prototype v3, the final gFEX prototype, is implemented, which is an ATCA module consisting of three Vertex UltraScale+ FPGAs, one ZYNQ UltraScale+ SoC, and 35 MiniPODs. This board receives up to 300 fiber optical links from calorimeters and transmits trigger data on 96 links to the to the ATLAS Level-1 Topological trigger (L1Topo [1]) at the speed up to 12.8 Gb/s. There are also 24 electrical links on board for communication between two FPGAs with the speed up to 25.6 Gb/s. The performance of three prototype boards have been tested and evaluated. For the prototype v3 board, the high-speed optical links are stable at 12.8 Gb/s with Bit Error Ratio (BER) < 1 × 10 -15 . The low-latency parallel GPIO (General Purpose I/O) buses between FPGAs are stable at 1.12 Gb/s. The peripheral components of ZYNQ UltraScale+ SoC, such as 16 GB DDR4 DIMM, UART, SPI flashes, and Ethernet, have also been verified. Here, the test results of the prototype v3 board validate the gFEX technologies, architecture and full functionalities. Now the final production board is being produced.
Memory size has long limited large-scale applications on high-performance computing (HPC) systems. Since compute nodes frequently do not have swap space, physical memory often limits problem sizes. Increasing core counts per chip and power density constraints, which limit the number of DIMMs per node, have exacerbated this problem. Further, DRAM constitutes a significant portion of overall HPC system cost. Therefore, instead of adding more DRAM to the nodes, mechanisms to manage memory usage more efficiently - preferably transparently - could increase effective DRAM capacity and thus the benefit of multicore nodes for HPC systems. MPI application processes often exhibit significant data similarity. These data regions occupy multiple physical locations across the individual rank processes within a multicore node and thus offer a potential savings in memory capacity. These regions, primarily residing in heap, are dynamic, which makes them difficult to manage statically. Our novel memory allocation library, SBLLmalloc, automatically identifies identical memory blocks and merges them into a single copy. SBLLmalloc does not require application or OS changes since we implement it as a user-level library. Overall, we demonstrate that SBLLmalloc reduces the memory footprint of a range of MPI applications by 32.03% on average and up to 60.87%. Further, SBLLmalloc supports problem sizes for IRS over 21.36% larger than using standard memory management techniques, thus significantly increasing effective system size. Similarly, SBLLmalloc requires 43.75% fewer nodes than standard memory management techniques to solve an AMG problem.
Memory size has long limited large-scale applications on high-performance computing (HPC) systems. Increasing core counts per chip and power density constraints, which limit the number of DIMMs per node, have exacerbated this problem. Mechanisms to manage memory usage - preferably transparently - more efficiently could increase effective DRAM capacity and, thus, the benefit of multicore similarity. These data regions occupy multiple physical locations within a multicore node and thus offer a potential savings in memory capacity. These data, primarily residing in heap, are quite dynamic, which makes them difficult to manage statically. SBLLmalloc is a memory allocation library that automatically identifies the replicated memory blocks and merges them into a single copy. SBLLmalloc does not require application or OS changes since we implement it as a user-level library that can be linked at runtime. SBLLmalloc provides the capability to run larger problems using the same hardware configuration in multicore nodes. SBLLmalloc has been implemented for Linux clusters.
Simsee is a program for generating artificial seeing data for simulations of astronomical surveys. It includes a jupyter notebook for deriving model parameters from Dual Image Motion Monitor (DIMM) data, and a program that reads these model parameters and creates a data file suitable for use by survey strategy simulation code (which is outside the scope of this package). It is specifically designed to provide input data for LSST’s opsim4 survey simulator, but may be adapted for use with different simulators as well. The code is verified to work on a Linux workstation with an Intel Xeon E5-6980 CPU and 256GB or memory, but should work fine on systems with far more limited resources.
Over the past three years in this project, what we have observed is that the primary reason for data movement in large-scale systems is that the per-node capacity is not large enough—i.e., one of the solutions to the data-movement problem (certainly not the only solution that is required, but a significant one nonetheless) is to increase per-node capacity so that inter-node traffic is reduced. This unfortunately is not as simple as it sounds. Today’s main memory systems for datacenters, enterprise computing systems, and supercomputers, fail to provide high per-socket capacity [Dirik & Jacob 2009; Cooper-Balis et al. 2012], except at extremely high price points (factors of 10–100x the cost/bit of consumer main-memory systems) [Stokes 2008]. The reason is that our choice of technology for today’s main memory systems—i.e., DRAM, which we have used as a main-memory technology since the 1970s [Jacob et al. 2007]—can no longer keep up with our needs for density and price per bit. Main memory systems have always been built from the cheapest, densest, lowest-power memory technology available, and DRAM is no longer the cheapest, the densest, nor the lowest-power storage technology out there. It is now time for DRAM to go the way that SRAM went: move out of the way for a cheaper, slower, denser storage technology, and become a cache instead. This inflection point has happened before, in the context of SRAM yielding to DRAM. There was once a time that SRAM was the storage technology of choice for all main memories [Tomasulo 1967; Thornton 1970; Kidder 1981]. However, once DRAM hit volume production in the 1970s and 80s, it supplanted SRAM as a main memory technology because it was cheaper, and it was denser. It also happened to be lower power, but that was not the primary consideration of the day. At the time, it was recognized that DRAM was much slower than SRAM, but it was only at the supercomputer level (For instance the Cray X-MP in the 1980s and its follow-on, the Cray Y-MP, in the 1990s) that could one afford to build ever- larger main memories out of SRAM—the reasoning for moving to DRAM was that an appropriately designed memory hierarchy, built of DRAM as main memory and SRAM as a cache, would approach the performance of SRAM, at the price-per-bit of DRAM [Mashey 1999]. Today it is quite clear that, were one to build an entire multi-gigabyte main memory out of SRAM instead of DRAM, one could improve the performance of almost any computer system by up to an order of magnitude—but this option is not even considered, because to build that system would be prohibitively expensive. It is now time to revisit the same design choice in the context of modern technologies and modern systems. For reasons both technical and economic, we can no longer afford to build ever-larger main memory systems out of DRAM. Flash memory, on the other hand, is significantly cheaper and denser than DRAM and therefore should take its place. While it is true that flash is significantly slower than DRAM, one can afford to build much larger main memories out of flash than out of DRAM, and we show that an appropriately designed memory hierarchy, built of flash as main memory and DRAM as a cache, will approach the performance of DRAM, at the price-per-bit of flash. In our studies as part of this project, we have investigated Non-Volatile Main Memory (NVMM), a new main-memory architecture for large-scale computing systems, one that is specifically designed to address the weaknesses described previously. In particular, it provides the following features: non-volatility: The bulk of the storage is comprised of NAND flash, and in this organization DRAM is used only as a cache, not as main memory. Furthermore, the flash is journaled, which means that operations such as checkpoint/restore are already built into the system. 1+ terabytes of storage per socket: SSDs and DRAM DIMMs have roughly the same form factor (several square inches of PCB surface area), and terabyte SSDs are now commonplace. performance approaching that of DRAM: DRAM is used as a cache to the flash system. price-per-bit approaching that of NAND: Flash is currently well under $0.50 per gigabyte; DDR3 SDRAM is currently just over $10 per gigabyte [Newegg 2014]. Even today, one can build an easily affordable main memory system with a terabyte or more of NAND storage per CPU socket (which would be extremely expensive were one to use DRAM), and our cycle- accurate, full-system experiments show that this can be done at a performance point that lies within a factor of two of DRAM.
Memory size has long limited large-scale applications on high-performance computing (HPC) systems. Increasing core counts per chip and power density constraints, which limit the number of DIMMs per node, have exacerbated this problem. Mechanisms to manage memory usage - preferably transparently - more efficiently could increase effective DRAM capacity and, thus, the benefit of multicore similarity. These data regions occupy multiple physical locations within a multicore node and thus offer a potential savings in memory capacity. These data, primarily residing in heap, are quite dynamic, which makes them difficult to manage statically. SBLLmalloc is a memory allocation library that automatically identifies the replicated memory blocks and merges them into a single copy. SBLLmalloc does not require application or OS changes since we implement it as a user-level library that can be linked at runtime. SBLLmalloc provides the capability to run larger problems using the same hardware configuration in multicore nodes. SBLLmalloc has been implemented for Linux clusters.
Power consumption in US data centers has been escalating at an alarming rate. In response to Public Law 109-431, EPA reported that electricity usage by US data centers accounted for 1.5% of the total electricity used in the US in 2006. They projected that if current data center operating practices continue, electricity usage will almost double to 2.9% of the total electricity used in the US in 2011 (EPA, 2007). Consequently EPA issued a call to action to both government and industry to collaborate and set aggressive goals to reduce power consumption in data centers. In the spirit of responding to the findings and recommendations of the EPA report, the Pacific Northwest National Laboratory (PNNL) has teamed with several key organizations including: The Green Grid (TGG), ASHRAE TC9.9, IBM, 3M, and SprayCool. As part of this effort, a highly instrumented liquid-cooled cluster has been installed at PNNL. The cluster is housed in an 800 ft2 data center which resides in a mixed-use data center with a significant amount of instrumentation as well. The eventual objective of the effort is to be able to report the real-time power consumption, energy efficiency, and productivity of the liquid-cooled data center. Preliminary results from the effort at PNNL are reported in this paper. Thermal results are reported for the hottest server components, including the microprocessors and memory DIMMs. Under all conditions tested, the components have not exceeded manufacturers’ specifications. More importantly, the data show that the liquid-cooled servers can be maintained within specifications while rejecting to non-chilled facility water at 78°F (25.6°C). Furthermore, a reasonable extrapolation suggests that the specifications can still be maintained at 86°F (30°C). In an effort to address global warming, work has started on the qualification of a new 3M Fluoroketone fluid that has a Global Warming Potential (GWP) of 1. This GWP is the lowest published value of all commercially available coolants. Details are provided in the body of this paper.