Hydra: Computer Vision for Data Quality Monitoring
Explore the source record for details and available documents.
Britton, Thomas↗
Engineering topics
Britton, Thomas
Publications and source records attributed to Britton, Thomas.
Experimental Physics Software and Computing Infrastructure
EPSCI Group activities including Hydra, AIEC, AIOP, AIOPP, EJFAT, ERSAP, and JANA.
Lawrence, David↗
AI Driven Experiment Calibration and Control
One critical step on the path from data taking to physics analysis is calibration. For many experiments this step is both time consuming and computationally expensive. The AI Experimental Calibration and Control project seeks to address these issues, starting first with the GlueX Central Drift Chamber (CDC). We demonstrate the ability of a Gaussian Process to estimate the gain correction factor (GCF) of the GlueX CDC accurately, and also the uncertainty of this estimate. Using the estimated GCF, the developed system infers a new high voltage (HV) setting that stabilizes the GCF in the face of changing environmental conditions. This happens in near real time during data taking and produces data which are already approximately gain-calibrated, eliminating the cost of performing those calibrations which vary ±15% with fixed HV. We also demonstrate an implementation of an uncertainty aware system which exploits a key feature of a Gaussian process.
Britton, Thomas↗
Hydra: Computer Vision for Online Data Quality Monitoring
Hydra is a system utilizing computer vision for near real-time data quality monitoring. Currently operational across all of Jefferson Lab’s experimental halls, it reduces the workload of shift takers by autonomously monitoring diagnostic plots during experiments. Hydra uses "off-the-shelf" supervised learning technologies and is supported by a comprehensive MySQL database. To simplify access, web apps have been developed to facilitate both labeling and monitoring of Hydra’s inferences. Hydra can connect with the alarm system and incorporates complete historical tracking, enabling it to identify issues that shift takers could miss. When issues are detected, a natural first question is: "Why does Hydra think there is a problem?" To answer, Hydra employs Gradient-weighted Class Activation Maps (GradCAM) to identify regions of the image that are important for the specific classification. This interpretive layer enhances transparency and trustworthiness, which is essential for integration with experiment workflows and operation. The Hydra system, results, and sociological considerations for deployment will be discussed.
Jeske, Torri↗
Hydra: Computer Vision for Data Quality Monitoring
Hydra: Computer Vision for Data Quality Monitoring Mar 13, 2024, 4:15 PM 30m Charles B. Wang Center, Stony Brook University Poster Track 1: Computing Technology for Physics Research Poster session with coffee break Speaker Ayan Roy Description Hydra is an extensible framework for training, managing, and deploying machine learning models for near real-time data quality monitoring. It is designed to take some of the burden off of shift crews by providing ‘round-the-clock’ monitoring of plots representing the data being collected. The Hydra system is backed by a database which is leveraged for near push button training and is primarily controlled and viewed through a set of web interfaces. This web interface contains a simple-to-use web based GUI for labeling of the datasets used in training; making it possible to label thousands of images quickly and efficiently. To aid in the analysis of Hydra inferences gradCAM, a method of interpretability, is performed and overlaid on the target image, highlighting regions of interest, making diagnosis of problems much faster. Development began in 2019 for the GlueX Experiment in Hall-D, the Hydra system has grown to encompass all of the experimental halls at Jefferson Laboratory. This talk will focus on the features of Hydra as well as provide details of the challenges present with deploying to disparate experimental halls. With a roadmap of development established, Hydra aims to continue to grow in richness of feature set and expand to encompass other monitoring tasks in differing communities.
Roy, Ayan↗
Using AI to predict calibration constants for the central drift chamber in GlueX at Jefferson Lab
The AI for Experimental Controls project team at Jefferson Lab has developed an AI system to control and calibrate a large drift chamber system in near-real time. The AI system will monitor environmental and experimental variables to recommend voltage settings that maintain consistent dE/dx gain and optimal resolution throughout the experiment. At present, calibrations are performed after data have been recorded and require a considerable amount of time and attention from experts. The calibrations currently require multiple iterations and depend on accurate tracking information. Our approach uses environmental data, such as atmospheric pressure and gas temperature, and beam conditions, such as the flux of incident particles, as inputs to a Gaussian Process Regression (GPR) model. For the data taken during the GlueX 2020 run period, the GPR is able to predict the existing gain correction factors to within 3.5%. This talk will briefly describe the development, testing, and future plans for this system at Jefferson Lab.
Jeske, Torri↗
Control and Calibration of GlueX Central Drift Chamber Using Gaussian Process Regression
The Gluonic Excitations (GlueX) experiment is designed to search for exotic hybrid mesons using photoproduction, and to study the hybrid meson spectrum predicted from Lattice Quantum Chromodynamics. For the first time, the GlueX Central Drift Chamber was controlled autonomously using machine learning (ML) to calibrate in real time while recording cosmic ray tracks. We demonstrate the ability of a Gaussian Process to predict the gain correction calibration factor used to determine a high voltage setting that will stabilize the CDC gain in response to changing environmental conditions; this is in contrast to the traditional, computationally expensive method of calibrating raw data after data collection is complete.
McSpadden, Helen↗
Gaussian process for calibration and control of GlueX Central Drift Chamber
We have developed and implemented a machine learning based system to calibrate and control the GlueX Central Drift Chamber at Jefferson Lab, VA, in near real-time. The system monitors environmental and experimental conditions during data taking and uses those as inputs to a Gaussian process (GP) with learned prior. The GP predicts calibration constants in order to recommend a high voltage (HV) setting for the detector that maintains consistent detector performance (gain and resolution) throughout data taking. This approach is in stark contrast to traditional detector operations in which the detector operates at fixed HV and its calibration parameters vary quite considerably with time. Additionally, the ML based system utilizes uncertainty quantification to correct the recommended control parameters when appropriate. We will present results from the ML system autonomously during the Charged Pion Polarizability (CPP) experiment conducted in Hall D at Jefferson Lab.
McSpadden, Helen↗
AI Data Quality Monitoring with Hydra
Hydra is an extensible framework for training and managing AI for near real time monitoring that aims to replace the tedious and repetitive data quality monitoring activities the shift crew and online monitoring coordinator typically perform. It continuously scans incoming data in the form of monitoring plots for signs of problems, flagging them for human review. A web app was developed such that experts can efficiently label images for training. Labels are stored in a database for use in training and model validation. Backed up by a comprehensive database, it utilizes an additional web based front-end for viewing the current monitoring status from anywhere in the world. The system has been in production use for the GlueX experiment at Jefferson Lab for more than 2 years with new features still under active development.
Britton, Thomas↗
Gaussian process for calibration and control of GlueX Central Drift Chamber
We have developed and implemented a machine learning based system to calibrate and control the GlueX Central Drift Chamber at Jefferson Lab, VA, in near real-time. The system monitors environmental and experimental conditions during data taking and uses those as inputs to a Gaussian process (GP) with learned prior. The GP predicts calibration constants in order to recommend a high voltage (HV) setting for the detector that maintains consistent detector performance (gain and resolution) throughout data taking. This approach is in stark contrast to traditional detector operations in which the detector operates at fixed HV and its calibration parameters vary quite considerably with time. Additionally, the ML based system utilizes uncertainty quantification to correct the recommended control parameters when appropriate. We will present results from the ML system autonomously during the Charged Pion Polarizability (CPP) experiment conducted in Hall D at Jefferson Lab.
McSpadden, Helen↗
Time series anomaly detection in power electronics signals with recurrent and ConvLSTM autoencoders
The anomalies in the high voltage converter modulator (HVCM) remain a major down time for the spallation neutron source facility, that delivers the most intense neutron beam in the world for scientific materials research. In this work, we propose neural network architectures based on Recurrent AutoEncoders (RAE) to detect anomalies ahead of time in the power signals coming from the HVCM. Bi-directional gated recurrent unit, bi-directional long-short term memory (LSTM), and convolutional LSTM (ConvLSTM) are developed, trained, and tested using real experimental signals from the HVCM module. The results show a good performance of the proposed RAE models, achieving precision up to 91%, recall up to 88%, false omission rate as low as 20% (i.e. 80% of the anomalies were detected), and area under the ROC curve up to 0.9. The three RAE models provide very comparable performance, with LSTM showing slightly better performance than GRU and ConvLSTM. The RAE models are benchmarked against other anomaly detection methods, including isolation forest, support vector machine, local outlier factor, feedforward and convolutional autoencoders, and others; showing a better performance. Here, the results of this study demonstrate the promising potential of RAE in anomaly detection for real-world power systems, and for increasing the reliability of the HVCM modules in the spallation neutron source.
42 ENGINEERING↗
Progress on Machine Learning for the SNS High Voltage Converter Modulators
The High-Voltage Converter Modulators (HVCM) used to power the klystrons in the Spallation Neutron Source (SNS) linac were selected as one area to explore machine learning due to reliability issues in the past and the availability of large sets of archived waveforms. Progress in the past two years has resulted in generating a significant amount of simulated and measured data for training neural network models such as recurrent neural networks, convolutional neural networks, and variational autoencoders. Applications in anomaly detection, fault classification, and prognostics of capacitor degradation were pursued in collaboration with the Jefferson Laboratory, and early promising results were achieved. This paper will discuss the progress to date and present results from these efforts.
Pappas, Chris↗
AI for Experimental Controls at Jefferson Lab
We report the AI for Experimental Controls project is developing an AI system to control and calibrate detector systems located at Jefferson Laboratory. Currently, calibrations are performed offline and require significant time and attention from experts. This work would reduce the amount of data and the amount of time spent calibrating in an offline setting. The first use case involves the Central Drift Chamber (CDC) located inside the GlueX spectrometer in Hall D. We use a combination of environmental and experimental data, such as atmospheric pressure, gas temperature, and the flux of incident particles as inputs to a sequential Neural Network (NN) to recommend a high voltage setting and the corresponding calibration constants in order to maintain consistent gain and optimal resolution throughout the experiment. Utilizing AI in this manner represents an initial shift from offline calibration towards near real time calibrations performed at Jefferson Laboratory.
47 OTHER INSTRUMENTATION↗
AI Enabled Data Quality Monitoring with Hydra
Data quality monitoring is critical to all experiments impacting the quality of any physics results. Traditionally, this is done through an alarm system, which detects low level faults, leaving higher level monitoring to human crews. Artificial Intelligence is beginning to find its way into scientific applications, but comes with difficulties, relying on the acquisition of new skill sets, either through education or acquisition, in data science. This paper will discuss the development and deployment of the Hydra monitoring system in production at Gluex. It will show how “off-the-shelf” technologies can be rapidly developed, as well as discuss what sociological hurdles must be overcome to successfully deploy such a system. Early results from production running of Hydra will also be shared as well as a future outlook for development of Hydra.
Britton, Thomas↗
Automated and Distributed Monte Carlo Generation for GlueX
MCwrapper is a set of systems that manages the entire Monte Carlo production workflow for GlueX and provides standards for how that Monte Carlo is produced. MCwrapper was designed to be able to utilize a variety of batch systems in a way that is relatively transparent to the user, thus enabling users to quickly and easily produce valid simulated data at home institutions worldwide. Additionally, MCwrapper supports an autonomous system that takes user’s project submissions via a custom web application. The system then atomizes the project into individual jobs, matches these jobs to resources, and monitors the jobs status. The entire system is managed by a database which tracks almost all facets of the systems from user submissions to the individual jobs themselves. Users can interact with their submitted projects online via a dashboard or, in the case of testing failure, can modify their project requests from a link contained in an automated email. Beginning in 2018 the GlueX Collaboration began to utilize the Open Science Grid (OSG) to handle a bulk of simulation tasks; these tasks are currently being performed on the OSG automatically via MCwrapper. This talk will outline the entire system of MCwrapper, its use cases, and the unique challenges facing the system.
Britton, Thomas↗
ML Track Fitting in Nuclear Physics
Charged particle tracking represents the largest consumer of CPU resources in high data volume Nuclear Physics (NP) experiments. An effort is underway to develop machine learning (ML) networks that will reduce the resources required for charged particle tracking. Tracking in NP experiments represent some unique challenges compared to high energy physics (HEP). In particular, track finding typically represents only a small fraction of the overall tracking problem in NP. This presentation will outline the differences and similarities between NP and HEP charged particle tracking and areas where ML learning may provide a benefit. The status of the specific effort taking place at Jefferson Lab will also be shown.
Britton, Thomas↗
Meson Spectroscopy: a Perspective from GlueX
In the last 15 years collaborations world-wide have contributed to the discovery of many exotic candidate states, such as those made from two quarks and two antiquarks. With these discoveries there has been renewed interest in spectroscopy and in particular, searches for other such particles. Thomas Jefferson National Accelerator Facility is home to the GlueX experiment which uses a 12 GeV electron beam incident on a diamond radiator to produce a linearly polarized, coherent Bremsstrahlung photon beam. The primary goal of GlueX is to search for so-called exotic hybrid mesons (comprised of a quark, an anti quark, and a gluon) by first mapping out the conventional spectrum of mesons starting with the often poorly understood excited vector mesons. Photoproduction at these energies is relatively unexplored and the use of a polarized beam allows GlueX to discriminate between various production mechanisms which may aid in the identification of exotic hybrid mesons. Successfully commissioned in 2015, GlueX has recently completed data taking for its first phase (GlueX-I) which represents orders of magnitude more statistics than previous photoproduction experiments. In this talk we will present an overview of the spectroscopy being performed at Jefferson Lab with a focus on GlueX.
Britton, Thomas↗
Latest results from GlueX
The GlueX experiment is housed in the newest experimental hall at the Thomas Jefferson National Accelerator Facility in Newport News, Virginia. It was successfully commissioned in 2015 and is in its third year of data taking. GlueX uses a 12 GeV electron beam incident on a diamond radiator, producing a linearly polarized, coherent Bremsstrahlung photon beam. The ultimate goal of GlueX is to search for exotic hybrid mesons (e.g. $q\bar{q}g$), with either exotic or conventional quantum numbers, whose existence, or lack thereof, would allow for the exploration of the gluon-gluon coupling present in QCD through the manifestation of hadrons with gluonic degrees of freedom. Photo-production at these energies is fairly unexplored and the linear beam polarization allows GlueX to discriminate between various production mechanisms which may be an effective way to identify such exotic hybrid mesons. In addition to exotic mesons, GlueX will also be poised to map out the conventional meson spectrum and to study the spectrum of excited vector mesons, which are often poorly understood. Here in these proceedings, we will present an overview of the GlueX experiment, its goals, current physics results, and future plans.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗