Engineering Papers⌕ Search

Engineering topics

Kowalkowski, James B.

Publications and source records attributed to Kowalkowski, James B..

Shape-shifting Elephants: Multi-modal Transport for Integrated Research Infrastructure

Data Acquisition (DAQ) workloads form an important class of scientific network traffic that by its nature (1) flows across different research infrastructure, including remote instruments and supercomputer clusters, (2) has ever-increasing throughput demands, and (3) has ever-increasing integration demands---for example, observations at one instrument could trigger a reconfiguration of another instrument. Today's DAQ transfers rely on UDP and (heavily tuned) TCP, but this is driven by convenience rather than suitability. The mismatch between Internet transport protocols and scientific workloads becomes more stark with the steady increase in link capacities, data generation, and integration across research infrastructure.This position paper argues the importance of developing specialized transport protocols for DAQ workloads. It proposes a new transport feature for this kind of elephant flow: multi-modality involves the network actively configuring the transport protocol to change how DAQ flows are processed across different underlying networks that connect scientific research infrastructure. Multi-modality is a layering violation that is proposed as a pragmatic technique for DAQ transport protocol design. It takes advantage of programmable network hardware that is increasingly being deployed in scientific research infrastructure. The paper presents an initial evaluation through a pilot study that includes a Tofino2 switch and Alveo FPGA cards, and using data from a particle detector.

97 MATHEMATICS AND COMPUTING↗

High-Throughput Custom Monitoring for the Mu2e TDAQ System

In this project we are studying the application of programmable network hardware to provide a custom monitoring capability for the Mu2e Trigger and Data Acquisition System (TDAQ) system. The goal of the Mu2e experiment is to search for a charged-lepton flavor violating processes where a negative muon converts into an electron in the field of an aluminum nucleus. This experiment is intended to improve by four orders of magnitude the search sensitivity reached so far. We have a working prototype of a system that provides high-throughput, custom monitoring for the Mu2e TDAQ system. The custom Mu2e network packet header format is parsed as it crosses the network switch. Parsing extracts bits that convey information about error states at read-out controllers (ROCs). This information is periodically relayed to the switch controller, which in turn alerts experiment operators.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

High-Throughput Custom Monitoring for the Mu2e TDAQ System

In this project we are studying the application of programmable network hardware to provide a custom monitoring capability for the Mu2e Trigger and Data Acquisition System (TDAQ) system. The goal of the Mu2e experiment is to search for a charged-lepton flavor violating processes where a negative muon converts into an electron in the field of an aluminum nucleus. This experiment is intended to improve by four orders of magnitude the search sensitivity reached so far. We have a working prototype of a system that provides high-throughput, custom monitoring for the Mu2e TDAQ system. The custom Mu2e network packet header format is parsed as it crosses the network switch. Parsing extracts bits that convey information about error states at read-out controllers (ROCs). This information is periodically relayed to the switch controller, which in turn alerts experiment operators.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NuGraph2: A Graph Neural Network for Neutrino Event Reconstruction

Neutrino experiments are set to probe some of the most important open questions in physics, from CP violation and the nature of dark matter. The technology of choice for many of these experiments is the liquid argon time projection chamber (LArTPC). In current LArTPC experiments, reconstruction performance often represents a limiting factor for the sensitivity. New developments are therefore needed to unlock the full potential of LArTPC experiments. NuGraph2 is a state of the art Graph Neural Network for reconstruction of data in LArTPC experiments. NuGraph2 utilizes a heterogeneous graph structure, with separate subgraphs of 2D nodes (hits in each plane) connected across planes via 3D nodes (space points). The model provides a consistent description of the neutrino interaction across all planes. NuGraph2 is a multi-purpose network, with a common message-passing attention engine connected to multiple decoders with different classification or regression tasks. These include the classification of detector hits according to the particle type that produced them (semantic segmentation) and the separation of hits from the neutrino interaction from hits due to noise or cosmic-ray background. Additional decoders are being developed, performing tasks such as the regression of the neutrino interaction vertex position. Performance results will be presented based on publicly available samples from MicroBooNE. These include both physics performance metrics, achieving 95% accuracy for semantic segmentation and 98% classification of neutrino hits, as well as computational metrics for training and for inference on CPU or GPU. The status of the NuGraph integration in the LArSoft software framework will be presented, as well as initial studies about model interpretability and injection of domain knowledge.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NuGraph2: A Graph Neural Network for Neutrino Event Reconstruction

Neutrino experiments are set to probe some of the most important open questions in physics, from CP violation and the nature of dark matter. The technology of choice for many of these experiments is the liquid argon time projection chamber (LArTPC). In current LArTPC experiments, reconstruction performance often represents a limiting factor for the sensitivity. New developments are therefore needed to unlock the full potential of LArTPC experiments. NuGraph2 is a state of the art Graph Neural Network for reconstruction of data in LArTPC experiments. NuGraph2 utilizes a heterogeneous graph structure, with separate subgraphs of 2D nodes (hits in each plane) connected across planes via 3D nodes (space points). The model provides a consistent description of the neutrino interaction across all planes. NuGraph2 is a multi-purpose network, with a common message-passing attention engine connected to multiple decoders with different classification or regression tasks. These include the classification of detector hits according to the particle type that produced them (semantic segmentation) and the separation of hits from the neutrino interaction from hits due to noise or cosmic-ray background. Additional decoders are being developed, performing tasks such as the regression of the neutrino interaction vertex position. Performance results will be presented based on publicly available samples from MicroBooNE. These include both physics performance metrics, achieving 95% accuracy for semantic segmentation and 98% classification of neutrino hits, as well as computational metrics for training and for inference on CPU or GPU. The status of the NuGraph integration in the LArSoft software framework will be presented, as well as initial studies about model interpretability and injection of domain knowledge.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

In-Network DAQ Functions

A revolution in networking is changing how we compute, but we lack the tools that can channel this new capability to benefit science. It is now possible to write programs that operate "in" the network—on network cards (NICs) and network switches themselves, rather than on servers. These programs can analyze and reduce huge volumes of data as they flow through the network—at higher throughput, lower latency, and lower power consumption than if servers (containing CPUs or GPUs) were used. That equipment offers appealing features for scientific experiments that involve huge quantities of data. This poster describes a prototype LArTPC raw-waveform hit finder based on DUNE’s Trigger Primitives generator. We built this as part of ongoing research to better understand how to put programmable network hardware to use in large scientific experiments.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Quantum computing hardware for HEP algorithms and sensing

Quantum information science harnesses the principles of quantum mechanics to realize computational algorithms with complexities vastly intractable by current computer platforms. Typical applications range from quantum chemistry to optimization problems and also include simulations for high energy physics. The recent maturing of quantum hardware has triggered preliminary explorations by several institutions (including Fermilab) of quantum hardware capable of demonstrating quantum advantage in multiple domains, from quantum computing to communications, to sensing. The Superconducting Quantum Materials and Systems (SQMS) Center, led by Fermilab, is dedicated to providing breakthroughs in quantum computing and sensing, mediating quantum engineering and HEP based material science. The main goal of the Center is to deploy quantum systems with superior performance tailored to the algorithms used in high energy physics. In this Snowmass paper, we discuss the two most promising superconducting quantum architectures for HEP algorithms, i.e. three-level systems (qutrits) supported by transmon devices coupled to planar devices and multi-level systems (qudits with arbitrary N energy levels) supported by superconducting 3D cavities. For each architecture, we demonstrate exemplary HEP algorithms and identify the current challenges, ongoing work and future opportunities. Furthermore, we discuss the prospects and complexities of interconnecting the different architectures and individual computational nodes. Finally, we review several different strategies of error protection and correction and discuss their potential to improve the performance of the two architectures. This whitepaper seeks to reach out to the HEP community and drive progress in both HEP research and QIS hardware.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

The Story of HDF5 in High Energy Physics

For several years now, Fermilab has been investigating the use of HDF5 for large-scale analysis of experimental high energy physics (HEP) data.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Impacts of noise and structure on quantum information encoded in a quantum memory

As larger, higher-quality quantum devices are built and demonstrated in quantum information applications, such as quantum computation and quantum communication, the need for high-quality quantum memories to store quantum states becomes ever more pressing. Future quantum devices likely will use a variety of physical hardware, some being used primarily for processing of quantum information and others for storage. Here we study the correlation of the structure of quantum information with physical noise models of various possible quantum memory implementations. Through numerical simulation of different noise models and approximate analytical formulas applied to a variety of interesting quantum states, we provide comparisons between quantum hardware with different structure, including both qubit- and qudit-based quantum memories. Our findings point to simple, experimentally relevant formulas for the relative lifetimes of quantum information in different quantum memories and have relevance to the design of hybrid quantum devices.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Apprentice for Event Generator Tuning

Apprentice is a tool developed for event generator tuning. It contains a range of conceptual improvements and extensions over the tuning tool Professor. Its core functionality remains the construction of a multivariate analytic surrogate model to computationally expensive Monte-Carlo event generator predictions. The surrogate model is used for numerical optimization in chi-square minimization and likelihood evaluation. Apprentice also introduces algorithms to automate the selection of observable weights to minimize the effect of mis-modeling in the event generators. We illustrate our improvements for the task of MC-generator tuning and limit setting.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

PandAna: A Python Analysis Framework for Scalable High Performance Computing in High Energy Physics

Modern experiments in high energy physics analyze millions of events recorded in particle detectors to select the events of interest and make measurements of physics parameters. These data can often be stored as tabular data in files with detector information and reconstructed quantities. Current techniques for event selection in these files lack the scalability needed for high performance computing environments. We describe our work to develop a high energy physics analysis framework suitable for high performance computing. This new framework utilizes modern tools for reading files and implicit data parallelism. Framework users analyze tabular data using standard, easy-to-use data analysis techniques in Python while the framework handles the file manipulations and parallelism without the user needing advanced experience in parallel programming. In future versions, we hope to provide a framework that can be utilized on a personal computer or a high performance computing cluster with little change to the user code.

Groh, Micah↗

Geant4 Computing Performance Benchmarking and Monitoring

Performance evaluation and analysis of large scale computing applications is essential for optimal use of resources. As detector simulation is one of the most compute intensive tasks and Geant4 is the simulation toolkit most widely used in contemporary high energy physics (HEP) experiments, it is important to monitor Geant4 through its development cycle for changes in computing performance and to identify problems and opportunities for code improvements. All Geant4 development and public releases are being profiled with a set of applications that utilize different input event samples, physics parameters, and detector configurations. Results from multiple benchmarking runs are compared to previous public and development reference releases to monitor CPU and memory usage. Observed changes are evaluated and correlated with code modifications. Besides the full summary of call stack and memory footprint, a detailed call graph analysis is available to Geant4 developers for further analysis. The set of software tools used in the performance evaluation procedure, both in sequential and multi-threaded modes, include FAST, IgProf and Open|Speedshop. In conclusion, the scalability of the CPU time and memory performance in multi-threaded application is evaluated by measuring event throughput and memory gain as a function of the number of threads for selected event samples.

97 MATHEMATICS AND COMPUTING↗

Message correlation analysis tool for NOvA

A complex running system, such as the NOvA online data acquisition, consists of a large number of distributed but closely interacting components. This paper describes a generic real-time correlation analysis and event identification engine, named Message Analyzer. Its purpose is to capture run time abnormalities and recognize system failures based on log messages from participating components. The initial design of analysis engine is driven by the data acquisition (DAQ) of the NOvA experiment. The Message Analyzer performs filtering and pattern recognition on the log messages and reacts to system failures identified by associated triggering rules. The tool helps the system maintain a healthy running state and to minimize data corruption. This paper also describes a domain specific language that allows the recognition patterns and correlation rules to be specified in a clear and flexible way. In addition, the engine provides a plugin mechanism for users to implement specialized patterns or rules in generic languages such as C++.

Lu, Qiming↗

LQCD workflow execution framework: Models, provenance and fault-tolerance

Large computing clusters used for scientific processing suffer from systemic failures when operated over long continuous periods for executing workflows. Diagnosing job problems and faults leading to eventual failures in this complex environment i s difficult, specifically when the success of an entire workflow might be affected by a single job failure. In this paper, we introduce a model-based, hierarchical, reliable execution framework that encompass workflow specification, data provenance, execu tion tracking and online monitoring of each workflow task, also referred to as participants. The sequence of participants is described in an abstract parameterized view, which is translated into a concrete data dependency based sequence of participants wi th defined arguments. As participants belonging to a workflow are mapped onto machines and executed, periodic and on-demand monitoring of vital health parameters on allocated nodes is enabled according to pre-specified rules. These rules specify condition s that must be true pre-execution, during execution and post-execution. Monitoring information for each participant is propagated upwards through the reflex and healing architecture, which consists of a hierarchical network of decentralized fault manageme nt entities, called reflex engines. They are instantiated as state machines or timed automatons that change state and initiate reflexive mitigation action(s) upon occurrence of certain faults. We describe how this cluster reliability framework is combined with the workflow execution framework using formal rules and actions specified within a structure of first order predicate logic that enables a dynamic management design that reduces manual administrative workload, and increases cluster-productivity.

Piccoli, Luciano↗