C3 Datacenter Hourly Power Usage
Explore the source record for details and available documents.
W. Anderson, Matthew↗
SEARCH · Engineering Papers
Results for “datacenter”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
C3 Datacenter Hourly Power Usage
Explore the source record for details and available documents.
W. Anderson, Matthew↗
LANL HPC Datacenter Monitoring
Explore the source record for details and available documents.
Computer Science↗
There and Back Again: Reimagining Cryogenic Cooling for Scalable Arrays of Dilution Refrigerators for future Quantum Datacenters
While pulse tube cryocoolers enabled the rapid expansion of dilution refrigerator technology over the past two decades, the transition to large-scale quantum systems is now driving a reassessment of the DR’s higher-temperature-stage cooling strategies and how these systems can be effectively scaled in a modular way. Quasi-wet architectures based on centralized cryoplants and forced-flow helium distribution offer compelling advantages in energy efficiency, operational cost, and scalability. With appropriate redundancy, standardized interfaces, and optimized distribution system designs, these architectures will provide a practical and robust path forward for the next generation of quantum computing infrastructure.
Hansen, B. [Fermilab]↗
Data Center Power Systems: Architectures, Impact on Grid Reliability, Modeling Considerations, and Megawatt-Scale Hardware Testing [Slides]
This slide deck describes typical power systems of large datacenters along with reliability problems to bulk power systems from large-scale integration of datacenters. The slide deck covers the architecture of datacenter power systems, different power electronic converters used inside datacenters, their operation modes, and R&Dopportunities in maintaining grid stability.
24 POWER TRANSMISSION AND DISTRIBUTION↗
High Performance Computing Peak Shaving for Microreactor Operation
There are multiple nuclear microreactors currently under development that are designed to provide autonomous power for as many as ten or more years without refueling and are designed to power high performance computing (HPC) datacenters. But the load-follow speeds for a nuclear microreactor will be much slower than grid power and slower than the power variance typical of a HPC system. HPC datacenters experience peak power load variance driven by several factors ranging from the operation of cooling systems to remove heat from the servers to supporting a wide range of user application workflows and architectures each with different power signatures. One mechanism to support the limited load-follow of a microreactor is peak shaving where an energy storage mechanism is used to shed peak load and reduce significant power variance. This work explores peak electrical load shaving using uninterruptible power supply (UPS) systems designed for HPC support in the context of peak shaving when operating using a nuclear microreactor with a load-follow limited to 10% of load per minute. Using a self contained HPC datacenter complete with stand-alone cooling system and provisioned with an x86 cluster, an ARM cluster, and a graphics processing unit (GPU) cluster, peak shaving for microreactor operation using the UPS battery backup is explored while running two classes of typical HPC user applications. HPC architecture suitability for microreactor operation under this type of peak shaving is examined.
97 MATHEMATICS AND COMPUTING↗
SU-E-T-152: Building An Ultrafast, Earthquake Proof Treatment Planning System
Purpose: Situated 20 miles from 5 major fault lines in California’s Bay Area, Stanford Universityhas a critical need for IT infrastructure planning to handle the high probability devastating earthquakes. Recently, a multi-million dollar project has been underway to overhaul Stanford’s radiation oncology information systems, maximizing planning system performance and providing true disaster recovery abilities. An overview of the project will be given with particular focus on lessons learned throughout the build. Methods: In this implementation, two isolated external datacenters provide geographical redundancy to Stanford’s main campus datacenter. Real-time mirroring is made of all data stored to our serial attached network (SAN) storage. In each datacenter, hardware/software virtualization was heavily implemented to maximize server efficiency and provide a robust mechanism to seamlessly migrate users in the event of an earthquake. System performance is routinely assessed through the use of virtualized data robots, able to log in to the system at scheduled times, perform routine planning tasks and report timing results to a performance dashboard. A substantial dose calculation framework (608 CPU cores) has been constructed as part of the implementation. Results: Migration to a virtualized server environment with a high performance SAN has resulted in up to a 45% speed up of common treatment planning tasks. Switching to a 608 core DCF has resulted in a 280% speed increase in dose calculations. Server tuning was found to further improved read/write performance by 20%. Disaster recovery tests are carried out quarterly and, although successful, remain time consuming to perform and verify functionality. Conclusion: Achieving true disaster recovery capabilities is possible through server virtualization, support from skilled IT staff and leadership. Substantial performance improvements are also achievable through careful tuning of server resources and disk read/write operations. Developing a streamlined method to comprehensively test failover is a key requirement to the system’s success.
60 APPLIED LIFE SCIENCES↗
Distributed deep learning training using silicon photonic switched architectures
The scaling trends of deep learning models and distributed training workloads are challenging network capacities in today’s datacenters and high-performance computing (HPC) systems. We propose a system architecture that leverages silicon photonic (SiP) switch-enabled server regrouping using bandwidth steering to tackle the challenges and accelerate distributed deep learning training. In addition, our proposed system architecture utilizes a highly integrated operating system-based SiP switch control scheme to reduce implementation complexity. To demonstrate the feasibility of our proposal, we built an experimental testbed with a SiP switch-enabled reconfigurable fat tree topology and evaluated the network performance of distributed ring all-reduce and parameter server workloads. The experimental results show up to 3.6× improvements over the static non-reconfigurable fat tree. Our large-scale simulation results show that server regrouping can deliver up to 2.3× flow throughput improvement for a 2× tapered fat tree and a further 11% improvement when higher-layer bandwidth steering is employed. The collective results show the potential of integrating SiP switches into datacenters and HPC systems to accelerate distributed deep learning training.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Compiler-Driven FPGA Virtualization with SYNERGY
FPGAs are increasingly common in modern applications, and cloud providers now support on-demand FPGA acceleration in datacenters. Applications in datacenters run on virtual infrastructure, where consolidation, multi-tenancy, and workload migration enable economies of scale that are fundamental to the provider's business. However, a general strategy for virtualizing FPGAs has yet to emerge. While manufacturers struggle with hardware-based approaches, we propose a compiler/runtime-based solution called Synergy. We show a compiler transformation for Verilog programs that produces code able to yield control to software atsub-clock-tickgranularity according to the semantics of the original program. Synergy uses this property to efficiently support core virtualization primitives: suspend and resume, program migration, and spatial/temporal multiplexing, on hardware which is availabletoday.We use Synergy to virtualize FPGA workloads across a cluster of Intel SoCs and Xilinx FPGAs on Amazon F1. The workloads require no modification, run within 3--4x of unvirtualized performance, and incur a modest increase in FPGA fabric usage.
Computer Science↗
MarFS, a Near-POSIX Interface to Cloud Objects
The engineering forces driving development of “cloud” storage have produced resilient, cost-effective storage systems that can scale to 100s of petabytes, with good parallel access and bandwidth. These features would make a good match for the vast storage needs of High-Performance Computing datacenters, but cloud storage gains some of its capability from its use of HTTP-style Representational State Transfer (REST) semantics, whereas most large datacenters have legacy applications that rely on POSIX file-system semantics. MarFS is an open-source project at Los Alamos National Laboratory that allows us to present cloud-style object-storage as a scalable near-POSIX file system. We have also developed a new storage architecture to improve bandwidth and scalability beyond what’s available in commodity object stores, while retaining their resilience and economy. Additionally, we present a scheme for scaling the POSIX interface to allow billions of files in a single directory and trillions of files in total.
97 MATHEMATICS AND COMPUTING↗
Enabling power measurement and control on Astra: The first petascale Arm supercomputer
Astra, deployed in 2018, was the first petascale supercomputer to utilize processors based on the ARM instruction set. The system was also the first under Sandia's Vanguard program which seeks to provide an evaluation vehicle for novel technologies that with refinement could be utilized in demanding, large-scale HPC environments. In addition to ARM, several other important first-of-a-kind developments were used in the machine, including new approaches to cooling the datacenter and machine. Here we document our experiences building a power measurement and control infrastructure for Astra. While this is often beyond the control of users today, the accurate measurement, cataloging, and evaluation of power, as our experiences show, is critical to the successful deployment of a large-scale platform. While such systems exist in part for other architectures, Astra required new development to support the novel Marvell ThunderX2 processor used in compute nodes. In addition to documenting the measurement of power during system bring up and for subsequent on-going routine use, we present results associated with controlling the power usage of the processor, an area which is becoming of progressively greater interest as data centers and supercomputing sites look to improve compute/energy efficiency and find additional sources for full system optimization.
97 MATHEMATICS AND COMPUTING↗
Nodal capacity expansion planning with flexible large-scale load siting
We propose explicitly incorporating large-scale load siting into a stochastic nodal power system capacity expansion planning model that concurrently co-optimizes generation, transmission, and storage expansion. The potential operational flexibility of some of these large loads is also taken into account by considering them as consisting of a set of tranches with different reliability requirements, which are modeled as a constraint on expected served energy across operational scenarios. We implement our model as a two-stage stochastic mixed-integer optimization problem with cross-scenario expectation constraints. To overcome the challenge of scalability, we build upon existing work to implement this model on a high performance computing platform and exploit scenario parallelization using an augmented Progressive Hedging Algorithm. The algorithm is implemented using the bounding features of mpisppy, which have shown to provide satisfactory provable optimality gaps despite the absence of theoretical guarantees of convergence. We test our approach and assess the value of this proactive planning framework on total system cost and reliability metrics using realistic testcases geographically assigned to San Diego and South Carolina, with datacenter and direct air capture facilities as large loads.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Demonstrating the data center as a flexible grid asset using a C-HIL setup
Increasing data center demand is outpacing grid infrastructure development. Artificial intelligence workloads and hyperscale cloud growth are creating unprecedented demand for power, while traditional grid expansion faces multiyear development timelines. Verrus is developing an innovative datacenter solution for this challenge, data centers that act as active grid-supportive assets rather than passive loads. Our approach integrates a novel grid-aware power flow management system with battery energy storage systems(BESS) into a microgrid-controlled, medium-voltage power distribution architecture that delivers critical capabilities, such as: * Fast response to grid disturbances such over/ under voltage or over/ under frequency * Demand flexibility that can service requests from the utility within 10 s * Uninterrupted transition to islanded operation during grid outages * Continuous uptime assurance for compute loads while maintaining all customer service level agreements. Through Verrus' strategic partnership with the National Renewable Energy Laboratory (NREL), these capabilities were validated using NREL's Advanced Research on Integrated Energy Systems (ARIES) virtual emulation environment to model a 70-MW grid-interactive data center. This paper outlines the design, methodology, and results of this emulated deployment, demonstrating that data centers can provide both critical load resilience and ancillary grid support without compromising uptime requirements. Specifically, we present a digital real time simulation of a 70 MW data center integrated with a physical microgrid controller, and demonstrate the data center response in the event of a grid voltage and frequency event, utility demand response request and utility outage.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Technoeconomic analysis of hydrogen storage using 1,4-butanediol (BDO)/γ-butyrolactone (GBL) as a stationary backup power system
Liquid organic hydrogen carriers (LOHCs) are compounds that store and release hydrogen in stable forms at high density. While one-way carriers such as methanol and ammonia have gained attention, their economic advantages are often realized from their use as an export product and direct use as a fuel. Liquid carrier materials that can instead be cycled for energy storage have promise for stationary power applications. In particular, the reversible LOHC system 1,4-butanediol (BDO, H 2 -rich) and gamma-butyrolactone (GBL, H 2 -lean) has a lower enthalpy of dehydrogenation compared to conventional cyclic hydrocarbons and can utilize non-precious metal copper-based catalysts. Here, in this study, BDO/GBL system capital and operating expenses are characterized for vapor phase versus liquid phase hydrogenation and dehydrogenation in a 10 MW backup power application corresponding to sizing of Tier 2 datacenters as well as other critical infrastructure such as hospitals. Costs are benchmarked against two incumbent technologies: a well-established methylcyclohexane/toluene carrier system and compressed gas storage. BDO/GBL storage costs are found to differ substantially between operating modes, with liquid phase hydrogenation coupled with liquid phase dehydrogenation leading to the lowest LCOS of $\$$4.58/kg H 2 in the absence of byproduct formation. In this bounding case, LCOS for the BDO/GBL system is lower than for MCH/TOL ($\$$6.97/kg H 2 ) and compressed gas ($\$$8.48/kg H 2 at 170 bara and $\$$12.05/kg H 2 at 350 bara). However, escalating costs of carrier replacement due to byproduct formation (ranging from an added $\$$6–11/kg H 2 ) illustrate the need for highly selective catalysts to ensure BDO/GBL carrier viability.
BDO/GBL↗
Experience with dynamic resource provisioning of the CMS online cluster using a cloud overlay
The primary goal of the online cluster of the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC) is to build event data from the detector and to select interesting collisions in the High Level Trigger (HLT) farm for offline storage. With more than 1500 nodes and a capacity of about 850 kHEPSpecInt06, the HLT machines represent similar computing capacity of all the CMS Tier1 Grid sites together. Moreover, it is currently connected to the CERN IT datacenter via a dedicated 160 Gbps network connection and hence can access the remote EOS based storage with a high bandwidth. In the last few years, a cloud overlay based on OpenStack has been commissioned to use these resources for the WLCG when they are not needed for data taking. This online cloud facility was designed for parasitic use of the HLT, which must never interfere with its primary function as part of the DAQ system. It also allows to abstract from the different types of machines and their underlying segmented networks. During the LHC technical stop periods, the HLT cloud is set to its static mode of operation where it acts like other grid facilities. The online cloud was also extended to make dynamic use of resources during periods between LHC fills. These periods are a-priori unscheduled and of undetermined length, typically of several hours, once or more a day. For that, it dynamically follows LHC beam states and hibernates Virtual Machines (VM) accordingly. Finally, this work presents the design and implementation of a mechanism to dynamically ramp up VMs when the DAQ load on the HLT reduces towards the end of the fill.
97 MATHEMATICS AND COMPUTING↗
Power consumption monitoring using additional monitoring device
Today, emphasis is placed on reducing power consumption. Computers are large consumers; therefore it is important to know the total consumption of computing systems. Since their optimal functioning requires quite strict environmental conditions, without much variation in temperature and humidity, reducing energy consumption cannot be made without monitoring environmental parameters. Thus, the present work uses a multifunctional electric meter UPT 210 for power consumption monitoring. Two applications were developed: software which carries meter readings provided by electronic and programming facilitates remote device and a device for temperature monitoring and control. Following temperature variations that occur both in the cooling system, as well as the ambient, can reduce energy consumption. For this purpose, some air conditioning units or some computers are stopped in different time slots. These intervals were set so that the economy is high, but the work's Datacenter is not disturbed.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Advancing microelectronics through nanoscale science: A perspective on needs and opportunities from the nanoscale science research centers
Microelectronics are the cornerstone of the modern world, enhancing our daily lives by providing services such as communications and datacenters. These resources are accessible thanks to the continual pursuit of a deeper understanding of the chemical and physical phenomena underlying the materials synthesis approaches and fabrication processes used to create microelectronic components and subsequently the components' responses to electrical, optical, and other stimuli that are utilized within microelectronic systems. Today, further development of microelectronics requires multidisciplinary expertise across scientific disciplines and fields of study—synthesis, materials characterization, nanoscale fabrication, and performance characterization—with focus placed on comprehending the nanoscale forms and features of microelectronic components. The Nanoscale Science Research Centers (NSRCs) are Department of Energy, Office of Science user facilities that support the international scientific community in advancing nanoscale science and technology. As a key component of the U.S. Government's National Nanotechnology Initiative, the NSRCs enable transformative discoveries by providing world-class facilities, expertise, and collaborative opportunities. Here, in this perspective, we showcase a non-exhaustive cross-section of the capabilities housed at and developed by the NSRCs and their user communities to address fundamental synthesis, metrology, fabrication, and performance considerations toward advancing the development of new microelectronics. Finally, we provide a timely outlook on the next major areas of necessary development in nanoscale sciences to continue the innovation of microelectronics into the next generation.
2D materials↗
Evolving Energy Efficient Convolutional Neural Networks
As deep neural networks have been deployed in more and more applications over the past half decade and are finding their way into an ever increasing number of operational systems, their energy consumption becomes a concern whether running in the datacenter or on edge devices. Hyperparameter optimization and automated network design for deep learning is a quickly growing field, but much of the focus has remained only on optimizing for the performance of the machine learning task. In this work, we demonstrate that the best performing networks created through this automated network design process have radically different computational characteristics (e.g. energy usage, model size, inference time), presenting the opportunity to utilize this optimization process to make deep learning networks more energy efficient and deployable to smaller devices. Optimizing for these computational characteristics is critical as the number of applications of deep learning continues to expand.
Young, Steven↗