Advancing anomaly detection in computational workflows with active learning
Not Available
Raghavan, Krishnan↗
Engineering topics
Mandal, Anirban
Publications and source records attributed to Mandal, Anirban.
Shaping the Future of Self-Driving Autonomous Laboratories Workshop
The "Shaping the Future of Self-Driving Autonomous Laboratories" workshop, held in Denver on November 7-8, 2024, brought together leading experts from materials science and computing to address the growing need to revolutionize scientific research through AI-driven autonomous laboratories. The workshop identified critical challenges, including the integration of heterogeneous data, development of AI systems that understand fundamental physical principles, and comprehensive safety protocols. Key recommendations emerged around developing universal laboratory equipment interfaces, implementing automated metadata collection systems, and creating hybrid AI approaches that combine data-driven learning with scientific principles. The workshop emphasized maintaining human oversight while leveraging automation, transforming scientific education to prepare the next generation of researchers, and establishing a national consortium leveraging DOE facilities as anchors for broader collaboration with academia and industry. Participants stressed the urgency of addressing the growing disconnect between human decision-making timescales and modern instrumentation capabilities, highlighting the need for strategic automation while preserving essential human insight and oversight in the research process.
36 MATERIALS SCIENCE↗
FlowBench Raw Data Archive
The repo that provides data archive for DOE PoSeiDon project. It also contains scripts and instructions to parse the data.
George, Papadimitriou↗
Graph neural networks for detecting anomalies in scientific workflows
Identifying and addressing anomalies in complex, distributed systems can be challenging for reliable execution of scientific workflows. We model these workflows as directed acyclic graphs (DAGs), where the nodes and edges of the DAGs represent jobs and their dependencies, respectively. We develop graph neural networks (GNNs) to learn patterns in the DAGs and to detect anomalies at the node (job) and graph (workflow) levels. We investigate workflow-specific GNN models that are trained on a particular workflow and workflow-agnostic GNN models that are trained across the workflows. Our GNN models, which incorporate both individual job features and topological information from the workflow, show improved accuracy and efficiency compared to conventional learning methods for detecting anomalies. While joint trained with multiple scientific workflows, our GNN models reached an accuracy more than 80% for workflow level and 75% for job level anomalies. In addition, we illustrate the importance of hyperparameter tuning method in our study that can significantly improve the metric(s) measure of evaluating the GNN models. Finally, we integrate explainable GNN methods to provide insights on job features in the workflow that cause an anomaly.
97 MATHEMATICS AND COMPUTING↗
High-Pressure Dynamic Strength Measurements and Continuum Modeling of Granular Boron Carbide via the Tamped Richtmyer-Meshkov Instability Method.
Abstract not provided.
Voorhees, Travis↗
Novel Proposals for FAIR, Automated, Recommendable, and Robust Workflows
Lightning talks of the Workflows in Support of Large-Scale Science (WORKS) workshop are a venue where the workflow community (researchers, developers, and users) can discuss work in progress, emerging technologies and frameworks, and training and education materials. This paper summarizes the WORKS 2022 lightning talks, which cover five broad topics: data integrity of scientific workflows; a machine learning-based recommendation system; a Python toolkit for running dynamic ensembles of simulations; a cross-platform, high-performance computing utility for processing shell commands; and a meta(data) framework for reproducing hybrid workflows.
Abhinit, Ishan↗
WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows
This paper introduces WIRE that manages resources for the DAG-based workflows on IaaS clouds. WIRE predicts and plans resources over the MAPE (Monitor-Analyze-Plan-Execute) loops to: 1) Estimate task performance with online data, 2) Conduct simulations to predict the upcoming loads based on online estimates and workflow DAGs, 3) Apply a resource-steering policy to size cloud instance pools for the maximal parallelism that is consistent with low cost. We implement WIRE on Pegasus WMS/HTCondor and evaluate its performance on the ExoGENI network cloud. The results show that WIRE attains low resource cost with the performance that is typically within a factor of two of optimal.
Xie, Bing↗
End-to-end online performance data capture and analysis for scientific workflows
With the increased prevalence of employing workflows for scientific computing and a push towards exascale computing, it has become paramount that we are able to analyze characteristics of scientific applications to better understand their impact on the underlying infrastructure and vice-versa. Such analysis can help drive the design, development, and optimization of these next generation systems and solutions. Here, we present the architecture, integrated with existing well-established and newly developed tools, to collect online performance statistics of workflow executions from various, heterogeneous sources and publish them in a distributed database (Elasticsearch). Using this architecture, we are able to correlate online workflow performance data, with data from the underlying infrastructure, and present them in a useful and intuitive way via an online dashboard. We have validated our approach by executing two classes of real-world workflows, both under normal and anomalous conditions. The first is an I/O-intensive genome analysis workflow; the second, a CPU- and memory-intensive material science workflow. Based on the data collected in Elasticsearch, we are able to demonstrate that we can correctly identify anomalies that we injected. The resulting end-to-end data collection of workflow performance data is an important resource of training data for automated machine learning analysis.
97 MATHEMATICS AND COMPUTING↗
Strength of porous α-SiO 2 in a shock loaded environment: Calibration via Richtmyer–Meshkov instability and validation via Mach lens
The strength of brittle porous media is of concern in numerous applications, for example, earth penetration, crater formation, and blast loading. Thus, it is of importance to possess techniques that allow for constitutive model calibration within the laboratory setting. The goal of the current work is to demonstrate an experimental technique allowing for strength assessment of porous media subjected to shock loading, which can be implemented into pressure-dependent yield surfaces within numerical simulation schemes. As a case study, the deviatoric response of distended α-SiO 2 has been captured in a tamped Richtmyer–Meshkov instability (RMI) environment at a pressure regime of 4–10 GPa. Hydrocode simulations were used to interpret RMI experimental data, and a resulting pressure-dependent yield surface akin to the often employed modified Drucker–Prager model was calibrated. Simulations indicate that the resulting jet length generated by the RMI is sensitive to the porous media strength, thereby providing a feasible experimental platform capable of capturing the pressurized granular deviatoric response. Furthermore, in efforts to validate the RMI-calibrated strength model, a set of Mach-lens experiments was performed and simulated with the calibrated pressure-dependent yield surface. Excellent agreement between the resulting Mach-lens length in experiment and simulation provides additional confidence to the RMI yield-surface calibration scheme.
36 MATERIALS SCIENCE↗
Direct observations of shock-induced melting in a porous solid using time-resolved x-ray diffraction
Here in this work we provide direct evidence of shock-induced melting and associated kinetics in a porous solid (aluminum powder) using time-resolved x-ray diffraction. Unambiguous evidence of melting in 50% porous aluminum (Al) powder samples, shocked to peak pressures between ~13 - 19 GPa, was provided by the broadening of the Debye-Scherrer ring corresponding to the (111) peak. Shocked Al powder did not melt completely in any of our experiments within the durations of measurement. Incomplete (partial) melting of the powder, even after several hundreds of nanoseconds of shock loading, provides insights into thermal transport with Al powder particles under high-pressure dynamic loading. Such insights are quite valuable for developing well-constrained melting models and thermodynamic equations of state for porous Al and other porous solids relevant to planetary and materials science.
36 MATERIALS SCIENCE↗
Detecting anomalous packets in network transfers: investigations using PCA, autoencoder and isolation forest in TCP
Large-scale scientific workflows rely heavily on high-performance file transfers. These transfers require strict quality parameters such as guaranteed bandwidth, no packet loss or data duplication. To have successful file transfers, methods such as predetermined thresholds and statistical analysis need to be done to determine abnormal patterns. Network administrators routinely monitor and analyze network data for diagnosing and alleviating these, making decisions based on their experience. However, as networks grow and become complex, monitoring large data files and quickly processing them, makes it improbable to identify errors and rectify these. Abnormal file transfers have been classified by simply setting alert thresholds, via tools such as PerfSonar and TCP statistics (Tstat). This paper investigates the feasibility of unsupervised feature extraction methods for identifying network anomaly patterns with three unsupervised classification methods—principal component analysis, autoencoder and isolation forest. Here, we collect file transfer statistics from two experiment sets—synthetic iPerf generated traffic and 1000 Genome workflow runs, with synthetically introduced anomalies. Our results show that while PCA and a simple autoencoder finds it difficult to detect clusters, the tree-variant isolation forest is able to identify anomalous packets by breaking down TCP traces into tree classes early.
97 MATHEMATICS AND COMPUTING↗
Direct observation of the hcp-bcc phase transition and melting along the principal Hugoniot of Mg
Abstract not provided
Beason, M. T.↗