Designing computing system architecture and models for the HL-LHC era
This work describes a programme to study the computing model in CMS after the next long shutdown near the end of the decade.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
SEARCH · Engineering Papers
Results for “SHUTDOWN”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Pooling the resources of the CMS Tier-1 sites
The CMS experiment at the LHC relies on 7 Tier-1 centres of the WLCG to perform the majority of its bulk processing activity, and to archive its data. During the first run of the LHC, these two functions were tightly coupled as each Tier-1 was constrained to process only the data archived on its hierarchical storage. This lack of flexibility in the assignment of processing workflows occasionally resulted in uneven resource utilisation and in an increased latency in the delivery of the results to the physics community.The long shutdown of the LHC in 2013-2014 was an opportunity to revisit this mode of operations, disentangling the processing and archive functionalities of the Tier-1 centres. The storage services at the Tier-1s were redeployed breaking the traditional hierarchical model: each site now provides a large disk storage to host input and output data for processing, and an independent tape storage used exclusively for archiving. Movement of data between the tape and disk endpoints is not automated, but triggered externally through the WLCG transfer management systems.With this new setup, CMS operations actively controls at any time which data is available on disk for processing and which data should be sent to archive. Thanks to the high-bandwidth connectivity guaranteed by the LHCOPN, input data can be freely transferred between disk endpoints as needed to take advantage of free CPU, turning the Tier-1s into a large pool of shared resources. The output data can be validated before archiving them permanently, and temporary data formats can be produced without wasting valuable tape resources. Lastly, the data hosted on disk at Tier-1s can now be made available also for user analysis since there is no risk any longer of triggering chaotic staging from tape.In this contribution, we describe the technical solutions adopted for the new disk and tape endpoints at the sites, and we report on the commissioning and scale testing of the service. We detail the procedures implemented by CMS computing operations to actively manage data on disk at Tier-1 sites, and we give examples of the benefits brought to CMS workflows by the additional flexibility of the new system.
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
Online Data Handling and Storage at the CMS Experiment
During the LHC Long Shutdown 1, the CMS Data Acquisition (DAQ) system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and support new detector back-end electronics. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. All the metadata needed for bookkeeping are stored in files as well, in the form of small documents using the JSON encoding. The Storage and Transfer System (STS) is responsible for aggregating these files produced by the HLT, storing them temporarily and transferring them to the T0 facility at CERN for subsequent offline processing. The STS merger service aggregates the output files from the HLT from ~62 sources produced with an aggregate rate of ~2GB/s. An estimated bandwidth of 7GB/s in concurrent read/write mode is needed. Furthermore, the STS has to be able to store several days of continuous running, so an estimated of 250TB of total usable disk space is required. In this article we present the various technological and implementation choices of the three components of the STS: the distributed file system, the merger service and the transfer system.
Andre, J. M.↗
File-Based Data Flow in the CMS Filter Farm
During the LHC Long Shutdown 1, the CMS Data Acquisition system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and prepare the ground for future upgrades of the detector front-ends. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. This approach provides additional decoupling between the HLT algorithms and the input and output data flow. All the metadata needed for bookkeeping of the data flow and the HLT process lifetimes are also generated in the form of small “documents” using the JSON encoding, by either services in the flow of the HLT execution (for rates etc.) or watchdog processes. These “files” can remain memory-resident or be written to disk if they are to be used in another part of the system (e.g. for aggregation of output data). We discuss how this redesign improves the robustness and flexibility of the CMS DAQ and the performance of the system currently being commissioned for the LHC Run 2.
Andre, J. M.↗
A new event builder for CMS Run II
The data acquisition system (DAQ) of the CMS experiment at the CERN Large Hadron Collider (LHC) assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GB/s to the high-level trigger (HLT) farm. The DAQ system has been redesigned during the LHC shutdown in 2013/14. The new DAQ architecture is based on state-of-the-art network technologies for the event building. For the data concentration, 10/40 Gbps Ethernet technologies are used together with a reduced TCP/IP protocol implemented in FPGA for a reliable transport between custom electronics and commercial computing hardware. A 56 Gbps Inniband FDR CLOS network has been chosen for the event builder. This paper discusses the software design, protocols, and optimizations for exploiting the hardware capabilities. In conclusion, ee present performance measurements from small-scale prototypes and from the full-scale production system.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
The CMS Data Acquisition - Architectures for the Phase-2 Upgrade
The upgraded High Luminosity LHC, after the third Long Shutdown (LS3), will provide an instantaneous luminosity of 7.5 × 10 34 cm -2 s -1 (levelled), at the price of extreme pileup of up to 200 interactions per crossing. In LS3, the CMS Detector will also undergo a major upgrade to prepare for the phase-2 of the LHC physics program, starting around 2025. The upgraded detector will be read out at an unprecedented data rate of up to 50 Tb/s and an event rate of 750 kHz. Complete events will be analysed by software algorithms running on standard processing nodes, and selected events will be stored permanently at a rate of up to 10 kHz for offline processing and analysis. Here in this paper we discuss the baseline design of the DAQ and HLT systems for the phase-2, taking into account the projected evolution of high speed network fabrics for event building and distribution, and the anticipated performance of general purpose CPU. Implications on hardware and infrastructure requirements for the DAQ “data center” are analysed. Emerging technologies for data reduction are considered. Novel possible approaches to event building and online processing, inspired by trending developments in other areas of computing dealing with large masses of data, are also examined. We conclude by discussing the opportunities offered by reading out and processing parts of the detector, wherever the front-end electronics allows, at the machine clock rate (40 MHz). This idea presents interesting challenges and its physics potential should be studied.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
A new ATLAS muon CSC readout system with system on chip technology on ATCA platform
The ATLAS muon Cathode Strip Chamber (CSC) backend readout system has been upgraded during the LHC 2013-2015 shutdown to be able to handle the higher Level-1 trigger rate of 100 kHz and the higher occupancy at Run-2 luminosity. The readout design is based on the Reconfigurable Cluster Element (RCE) concept for high bandwidth generic DAQ implemented on the Advanced Telecommunication Computing Architecture (ATCA) platform. The RCE design is based on the new System on Chip XILINX ZYNQ series with a processor-centric architecture with ARM processor embedded in FPGA fabric and high speed I/O resources. Together with auxiliary memories, all of these components form a versatile DAQ building block that can host applications tapping into both software and firmware resources. The Cluster on Board (COB) ATCA carrier hosts RCE mezzanines and an embedded Fulcrum network switch to form an online DAQ processing cluster. More compact firmware solutions on the ZYNQ for high speed input and output fiberoptic links and TTC allowed the full system of 320 input links from the 32 chambers to be processed by 6 COBs in one ATCA shelf. The full system was installed in September 2014. In conclusion, we will present the RCE/COB design concept, the firmware and software processing architecture, and the experience from the intense commissioning for LHC Run 2.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
The conversion of CESR to operate as the Test Accelerator, CesrTA. Part 3: Electron cloud diagnostics
Cornell's electron/positron storage ring (CESR) was modified over a series of accelerator shutdowns beginning in May 2008, which substantially improves its capability for research and development for particle accelerators. CESR's energy span from 1.8 to 5.6 GeV with both electrons and positrons makes it ideal for the study of a wide spectrum of accelerator physics issues and instrumentation related to present light sources and future lepton damping rings. Additionally a number of these are also relevant for the beam physics of proton accelerators. This paper is the third in a series of four describing the conversion of CESR to the test accelerator, CESRTA. The first two papers discuss the overall plan for the conversion of the storage ring to an instrument capable of studying advanced accelerator physics issues [1] and the details of the vacuum system upgrades [2]. This paper focuses on the necessary development of new instrumentation, situated in four dedicated experimental regions, capable of studying such phenomena as electron clouds (ECs) and methods to mitigate EC effects. The fourth paper in this series describes the vacuum system modifications of the superconducting wigglers to accommodate the diagnostic instrumentation for the study of EC behavior within wigglers. Lastly, while the initial studies of CESRTA focused on questions related to the International Linear Collider damping ring design, CESRTA is a very versatile storage ring, capable of studying a wide range of accelerator physics and instrumentation questions.
43 PARTICLE ACCELERATORS↗
The upgrade of the CMS hadron calorimeter with silicon photomultipliers
The upgrade of the hadron calorimeter of the CMS experiment at the CERN Large Hadron Collider is currently underway. The endcap sections will be upgraded in the winter of 2016–2017 and the barrel sections during the second LHC long shutdown in 2019. The existing photosensors will be replaced with about 16 000 new silicon photomultipliers (SiPMs), resulting in the first large installation of SiPMs in a radiation environment. All associated front-end electronics will also be upgraded. Here, this paper discusses the motivation for the upgrade and provides a description 17 of the new system, including the SiPMs with associated control electronics and the front-end readout cards.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
The construction of the phase-1 upgrade of the CMS pixel detector
The innermost layers of the original CMS tracker were built out of pixel detectors arranged in three barrel layers and two forward disks in each endcap. The original CMS detector was designed for the nominal instantaneous LHC luminosity of 1×10 34 cm -2 s -1 . Under the conditions expected in the coming years, which will see an increase of a factor two of the instantaneous luminosity, the CMS pixel detector would have seen a dynamic inefficiency caused by data losses due to buffer overflows. For this reason the CMS collaboration has installed a replacement pixel detector during the recent extended end of year shutdown. The phase-1 upgrade of the CMS pixel detector will operate at high efficiency at an instantaneous luminosity of 2×10 34 cm -2 s -1 with increased detector acceptance and additional redundancy for the tracking, while at the same time reducing the material budget. These goals are achieved using a new read-out chip and modified powering and read-out schemes, one additional tracking layer both in the barrel and in the disks, and new detector supports including a CO 2 based evaporative cooling system, that contribute to the reduction of the material in the tracking volume. This contribution will review the design and technological choices of the phase-1 detector.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Study of the material of the ATLAS inner detector for Run 2 of the LHC
The ATLAS inner detector comprises three different sub-detectors: the pixel detector, the silicon strip tracker, and the transition-radiation drift-tube tracker. The Insertable B-Layer, a new innermost pixel layer, was installed during the shutdown period in 2014, together with modifications to the layout of the cables and support structures of the existing pixel detector. The material in the inner detector is studied with several methods, using a low-luminosity √s=13 TeV pp collision sample corresponding to around 2.0 nb -1 collected in 2015 with the ATLAS experiment at the LHC. In this paper, the material within the innermost barrel region is studied using reconstructed hadronic interaction and photon conversion vertices. For the forward rapidity region, the material is probed by a measurement of the efficiency with which single tracks reconstructed from pixel detector hits alone can be extended with hits on the track in the strip layers. The results of these studies have been taken into account in an improved description of the material in the ATLAS inner detector simulation, resulting in a reduction in the uncertainties associated with the charged-particle reconstruction efficiency determined from simulation.
47 OTHER INSTRUMENTATION↗
Performance of the upgraded PreProcessor of the ATLAS Level-1 Calorimeter Trigger
The PreProcessor of the ATLAS Level-1 Calorimeter Trigger prepares the analogue trigger signals sent from the ATLAS calorimeters by digitising, synchronising, and calibrating them to reconstruct transverse energy deposits, which are then used in further processing to identify event features. During the first long shutdown of the LHC from 2013 to 2014, the central components of the PreProcessor, the Multichip Modules, were replaced by upgraded versions that feature modern ADC and FPGA technology to ensure optimal performance in the high pile-up environment of LHC Run 2. This paper describes the features of the new Multichip Modules along with the improvements to the signal processing achieved.
47 OTHER INSTRUMENTATION↗
The CMS Phase-1 Pixel Detector Upgrade
The CMS detector at the CERN LHC features a silicon pixel detector as its innermost subdetector. The original CMS pixel detector has been replaced with an upgraded pixel system (CMS Phase-1 pixel detector) in the extended year-end technical stop of the LHC in 2016/2017. The upgraded CMS pixel detector is designed to cope with the higher instantaneous luminosities that have been achieved by the LHC after the upgrades to the accelerator during the first long shutdown in 2013–2014. Compared to the original pixel detector, the upgraded detector has a better tracking performance and lower mass with four barrel layers and three endcap disks on each side to provide hit coverage up to an absolute value of pseudorapidity of 2.5. This paper describes the design and construction of the CMS Phase-1 pixel detector as well as its performance from commissioning to early operation in collision data-taking.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Hollow electron lenses for beam collimation at the High-Luminosity Large Hadron Collider (HL-LHC)
Electrons lenses produce a high-intensity electron beam and have a variety of applications to circular hadron accelerators. Electron beams of different transverse cross sections and distributions may be designed, depending on the desired application, and they are produced and steered along the orbit of the hadron beam, overlapping with it for typical distances of a few meters before being deflected away and disposed of. Hollow electron beams find applications to high-intensity beam collimation for machines like the CERN Large Hadron Collider (LHC). Such devices can be integrated in a collimation system to improve the halo-cleaning performance through an active control of the halo dynamics: the annular distribution of the electrons excites resonantly the beam tails surrounding the beam core, while the core itself remains unperturbed, as ideally it only “sees” the field-free “hole” in the electron distribution. Hollow electron lenses are part of the upgrade baseline of the High-Luminosity project of the LHC (HL-LHC) and will be installed in the machine during a long shutdown in 2025–2027 to mitigate effects from beam losses so to improve the collimation system performance. This paper describes the hollow electron lens project within the HL-LHC collimation upgrade.
43 PARTICLE ACCELERATORS↗
The ATLAS Fast TracKer system
The ATLAS Fast TracKer (FTK) was designed to provide full tracking for the ATLAS high-level trigger by using pattern recognition based on Associative Memory (AM) chips and fitting in high-speed field programmable gate arrays. The tracks found by the FTK are based on inputs from all modules of the pixel and silicon microstrip trackers. The as-built FTK system and components are described, as is the online software used to control them while running in the ATLAS data acquisition system. Also described is the simulation of the FTK hardware and the optimization of the AM pattern banks. An optimization for long-lived particles with large impact parameter values is included. A test of the FTK system with the data playback facility that allowed the FTK to be commissioned during the shutdown between Run 2 and Run 3 of the LHC is reported. The resulting tracks from part of the FTK system covering a limited $\eta$-$\phi$ region of the detector are compared with the output from the FTK simulation. It is shown that FTK performance is in good agreement with the simulation.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Analysis of humidity sensitivity of silicon strip sensors for ATLAS upgrade tracker, pre- and post-irradiation
During the prototyping phase of the new ATLAS Inner-Tracker (ITk) strip sensors, a degradation of the device breakdown voltage at high humidity was observed. Although the degradation was temporary, showing a fast recovery in dry conditions, the study of the influence of humidity on the sensor performance was critical to establish counter-measures and handling protocols during production testing in order to ensure the proper performance of the upgraded detector. The work presented here has the objective to study for the first time the breakdown voltage deterioration in presence of ambient humidity of ATLAS ITk production-layout strip sensors with different surface properties, before and after proton, neutron and gamma irradiations. A study of the humidity sensitivity of miniature ATLAS ITk strip sensors, before and after proton irradiations, is also presented to compare the sensitivity of devices with different sizes. The sensors were also exposed for several days to high humidity with the aim to recreate and evaluate the influence of the detector integration environment expected during the Large Hadron Collider (LHC) Long Shutdown 3 (LS3) in 2026, where the sensors will be exposed to ambient humidity for prolonged times.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Laser calibration of the ATLAS Tile Calorimeter during LHC Run 2
This article reports the laser calibration of the hadronic Tile Calorimeter of the ATLAS experiment in the LHC Run 2 data campaign. The upgraded Laser II calibration system is described. The system was commissioned during the first LHC Long Shutdown, exhibiting a stability better than 0.8% for the laser light monitoring. The methods employed to derive the detector calibration factors with data from the laser calibration runs are also detailed. These allowed to correct for the response fluctuations of the 9852 photomultiplier tubes of the Tile Calorimeter with a total uncertainty of 0.5% plus a luminosity-dependent sub-dominant term. Finally, we report the regular monitoring and performance studies using laser events in both standalone runs and during proton collisions. These studies include channel timing and quality inspection, and photomultiplier linearity and response dependence on anode current.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
The ATLAS trigger system for LHC Run 3 and trigger performance in 2022
The ATLAS trigger system is a crucial component of the ATLAS experiment at the LHC. It is responsible for selecting events in line with the ATLAS physics programme. This paper presents an overview of the changes to the trigger and data acquisition system during the second long shutdown of the LHC, and shows the performance of the trigger system and its components in the proton-proton collisions during the 2022 commissioning period as well as its expected performance in proton-proton and heavy-ion collisions for the remainder of the third LHC data-taking period (2022–2025).
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗