Engineering topics

Holzman, Burt [Fermilab]

Publications and source records attributed to Holzman, Burt [Fermilab].

3 records

DOE OSTI2025

Enabling Innovative Analysis on Heterogeneous Clusters through HTCdaskgateway

High energy particle (HEP) physics research is going through fundamental changes as we move to collect larger amounts of data from the Large Hadron Collider (LHC). Analysis facilities and distributed computing, through HTCs, have come together to create the next pythonic generation of analysis by utilizing HTCdaskgateway, a Dask gateway extension, allowing users to spawn workers compatible with both their analysis and heterogeneous clusters in line with authentication requirements. This is enabling physicists to engage with scientific python in ways they had not before because of domain specific C++ tools. An example of HTCdaskgateway’s use is Fermilab’s Elastic Analysis Facility.

Chavez, Elise [U. Wisconsin, Madison (main)]↗

DOE OSTI2025

The Elastic Analysis Facility's (EAF's) Contribution to the Future of Analysis at Multi-Experiment Institutions and Future Colliders

The Elastic Analysis Facility (EAF) hosted at Fermi National Accelerator Laboratory (Fermilab) is a platform being developed with the goal of providing a fast and efficient facility for physics analysis. As high-energy physics moves towards collecting larger datasets, such as those from the High-Luminosity LHC, the EAF strives to provide a powerful and adaptable framework for future colliders and multi-experiment institutions. Currently, the EAF supports several experiments including CMS, NOvA, and DUNE as well as serving accelerator physicists and beam line operations through integrated software and secure connections to Fermilab's computing resources. In addition, the EAF was designed with a user-friendly interface, intended to be more intuitive for emerging generations of physicists, that is still accessible for established styles of analysis. The EAF can also achieve better analysis efficiency due to the modernization of software and tools that can better utilize Fermilab's computing power. Furthermore, its design incorporates industry standards whenever possible, enhancing its sustainability and making it a possible template for other national or international laboratories and research facilities. Overall, the EAF is a forward-looking solution that will meet the evolving needs of particle physics, ensuring readiness for future colliders and multi-experiment research institutions.

Chavez, Elise [Wisconsin U., Madison]↗

DOE OSTI2025

On-Demand Column Joining for High Energy Physics

As the Large Hadron Collider (LHC) transitions into the High-Luminosity LHC (HL-LHC) era, the volume of data to be processed is expected to increase significantly. The CMS Experiment currently utilizes various data formats, including AOD, MiniAOD, and NanoAOD, each with different levels of detail and storage requirements. This paper addresses the challenges of data duplication and storage inefficiencies in high-energy physics (HEP) analyses by proposing an on-demand column-joining solution. This approach aims to reduce data duplication by enabling the dynamic combination of NanoAOD data with auxiliary information from larger data tiers, such as MiniAOD. The proposed solution leverages Trino, a high-performance distributed SQL query engine, to perform efficient and scalable data joins. Benchmarks using CMS OpenData demonstrate the feasibility of this approach, showing that it can handle large datasets with low latency. Integration with the scikit-hep ecosystem and the coffea analysis framework is also discussed, highlighting the potential for seamless end-to-end data processing and analysis. Ongoing and future work focuses on expanding benchmarks, integrating ServiceX for data transformation, and exploring the use of native object storage solutions.

Manganelli, Nicholas [Northeastern U.]↗