Malcolm slides for NSPA workshop
Summary of Malcolm, an INL open source project dealing with network traffic analysis. History and overview of the project. All information is open source and is already publicly available on the IdahoLab GitHub page.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Summary of Malcolm, an INL open source project dealing with network traffic analysis. History and overview of the project. All information is open source and is already publicly available on the IdahoLab GitHub page.
In this work, we derived S-parameter based expressions for the intrinsic quality factor of an arbitrary coupled multiport microwave cavity and a non-ideal test fixture. Practical accuracy limitations of the obtained expressions specifically for superconducting accelerator cavities were evaluated both analytically and with the simulation software. The resulting formulas can be used to extract the intrinsic quality factor of normal-conducting and superconducting cavities directly from the calibrated S-parameter measurements.
We investigate the advantages of using co-packaged optics in next-generation data center and AI supercomputer networks. The increased escape bandwidth offered by co-packaged optics provides multiple possibilities for building 50T switches and beyond, expanding the opportunities in both the data center and supercomputing domains. Furthermore, this provides network architects with the opportunity to expand their design space and develop simplified networks with enhanced network locality properties. Co-packaging at the switch and server points enables networks with double capacity while reducing the switch count by 64% compared to state-of-the-art systems. We evaluate these concepts through discrete-event simulations using all-to-all and all-reduce traffic patterns that simulate collective communications commonly found in network-bound applications. Initially, we investigate the all-to-all overhead involved in distributing the virtual machines of the applications across multiple leaf switches and compare it to the scenario in which all VMs are placed under a single switch. Subsequently, we evaluate the performance of an AI supercomputing cluster by simulating both patterns for different message sizes, while also varying the number of participating nodes. The results suggest that networks with improved locality properties become increasingly important as the network stack operates at higher speeds; for a stack latency of 1.25 µs, placing the applications under multiple switches can result in up to 68% higher completion times than placing them under a single switch. For AI supercomputers, significant improvements are observed in the mean server throughput, reaching more than 90% for configurations involving 256 nodes and message sizes of at least 128 KiB.
The feature interpretability code is a python module that interprets and analyzes neural networks trained on hydrodynamic simulation output in the form of numpy arrays. The code takes trained neural networks and extracts internal model states in the form of images. Additionally, tools for covariance analysis of network weights and predictions are provided. This code is built on the TensorFlow and PyTorch python libraries, and includes trained networks and example input data for demonstration purposes.
The code is written in Python and consists of the following pipeline that is implemented in Apache Airflow. This pipeline intends to understand the companies that are directly or indirectly involved with a type of critical infrastructure system at some point in that system's lifecycle. The pipeline takes a configuration file that specifies a list of initial companies to consider, a geographic region of interest, and a set of SEC form types as well as other data sources (e.g. CrunchBase) from which to extract entities and relations. There are four main components to this pipeline as currently implemented: Entity Extraction, Network Construction, Analysis, and Visualization. First, Entity Extraction, is implemented as the `topear-extract_organizations` Apache Airflow workflow. Given an initial query that specifies a geographic region of interest and a time interval, the software will extract CI facilities of interest and organizations that have a direct influence relationship to those facilities (e.g. ownership). During the course of the LDRD, we focused on Electric Vehicle charging stations and this information is available via the Department of Energy (DOE) database on fueling stations maintained by NREL. Within the context of the DOE CESER project, we have focused on Battery Energy Storage Systems (BESS). Second, the Network Extraction component will iteratively construct a social network graph given the set of organizations and people extracted in the previous step. Organizations (and eventually People if desired) are then fed as a query to the `topgear-construct_social_network` Apache Airflow workflow which given a set of initial companies and data sets (e.g. SEC EDGAR form types, OpenCorporates, Crunchbase). This Airflow workflow will iteratively query such data sources to discover relationships with new organizations and people. For example, this module can iteratively query SEC EDGAR for metadata that documents the number of each type of form for the given set of companies and their location. This forms metadata represents a catalog of data sources from SEC EDGAR for the extracted social network knowledge graph. The pipeline then downloads these forms from the website and saves them in a build directory for further processing. These documents are then parsed for entities and relations. Again, we note that in additional to SEC data sources, this step can also pull in information on organizations via API services such as CrunchBase and OpenCorporates or bulk data sources. At the end of this step, the resultant social network, the Critical Infrastructure network, and the edges that encode relationships between organizations and CI facilities, form the Adversarial Socio-Technical Network (ASTN) that informs the analysis. Third, the Analysis component processes these generated ASTN. Previously, that has included the ability to compare prevalence of different vendors for a given infrastructure component type across different regions as well as identify common public and private investors across those vendors. This was demonstrated for EV Charging Stations across several different metropolitan areas within an IEEE PES GridEdge publication. More recently, we have looked at ways to identify infrastructure owners and operators of BESS with the most nameplate capacity across different states as well as other indictors of risk resulting from changes in ownership over time. Finally, the Visualization component consists of an HTML/CSS/JS framework by which users can interact geospatial, operational, and organizational relationships across a given portfolio of Critical Infrastructure facilities. The objective is to provide a library of UI/UX modules that can be repurposed for stakeholder-specific dashboards. All of the modules are related via a common event model that enables UI actions in one view to percolate across the other views.
The evolving energy landscape requires novel tools to efficiently perform contingency analysis and reliability assessment of power grids, potentially in real-time. The high computational cost of traditional power flow solvers limits their applicability in practice. Machine learning (ML) surrogates such as deep neural networks (NNs) accelerate power flow solvers computations, enabling high-order contingency analysis and real-time decision-making by learning highly nonlinear functions and integrating grid topology via graph architectures. However, (graph) NNs lack predictive power away from training data and do not provide predictive confidence estimates. Here, we present a Bayesian residual graph NN that integrates knowledge from low-fidelity data via residual training and embeds granular quantification of uncertainties, improving trustworthiness critical for high-consequence decision-making. Applying Bayesian concepts to NNs is challenging due to the high-dimensionality of both the parameter space, complicating derivation of a meaningful prior, and the output space in large grid systems, requiring enhanced techniques to assess the predicted high-dimensional uncertainties. Our contributions include: (1) Deriving a prior for fully connected and graph NNs that leverages low-fidelity data to guide mean predictions and appropriately control prior predictive uncertainty. (2) Integrating this prior within an ensembling with anchoring scheme for efficient approximate posterior inference. (3) Deriving enhanced metrics to assess accuracy of both the mean and uncertainty predictions in high dimensions, appropriately accounting for correlations propagated through graph layers. The resulting Bayesian residual graph NN is tested on a contingency analysis task for 14-bus and 118-bus grids.
We evaluate uncertainty quantification (UQ) methods for deep learning applied to liquid argon time projection chamber (LArTPC) physics analysis tasks. As deep learning applications enter widespread usage among physics data analysis, neural networks with reliable estimates of prediction uncertainty and robust performance against overconfidence and out-of-distribution (OOD) samples are critical for their full deployment in analyzing experimental data. While numerous UQ methods have been tested on simple datasets, performance evaluations for more complex tasks and datasets are scarce. Here we assess the application of selected deep learning UQ methods on the task of particle classification using the PiLArNet monte carlo 3D LArTPC point cloud dataset. We observe that UQ methods not only allow for better rejection of prediction mistakes and OOD detection, but also generally achieve higher overall accuracy across different task settings. We assess the precision of uncertainty quantification using different evaluation metrics, such as distributional separation of prediction entropy across correctly and incorrectly identified samples, receiver operating characteristic curves (ROCs), and expected calibration error from observed empirical accuracy. We conclude that ensembling methods can obtain well calibrated classification probabilities and generally perform better than other existing methods in deep learning UQ literature.
Electronic health record (EHR) systems are critical to modern healthcare delivery, yet the dynamic workflows that govern electronic order processing remain underexplored. Inefficiencies in these digital pathways can cause delays in care, repetitive workloads, and even patient harm. This study presents a discrete-event simulation framework used to reconstruct and evaluate EHR-based order workflows in a large integrated healthcare system. Using real-world data extracted from the Veterans Health Administration’s Corporate Data Warehouse, the authors mapped order events to standardized state transitions and modeled their progression across different facilities of varying complexity levels. After being calibrated with empirical distributions of transition times and validated against observed time-in-system metrics, the simulation demonstrates close alignment with historical performance. Scenario analyses reveal that resource capacity constraints significantly amplify the impact of electronic order surges, which are reflected in the disproportionate growth in backlogs and processing delays. Adjustments in transition probabilities further increased recirculation and extended workflow paths. Network-based analysis identified Reserved, InProgress, and Completed as structurally critical states that function as hubs within the process network but the transitions in-between also act as major bottlenecks. These results showcased the effectiveness of simulation-based approaches in monitoring EHR order processing performance and evaluating consequences of workflow changes on healthcare network resources planning. The proposed simulation framework provides a scalable data-driven tool to support operational decision-making and improve the efficiency of electronic order management in complex healthcare environments.
We briefly review previous applications of machine learning (ML) in power grid analyses and introduce our ongoing effort toward developing a generative-adversarial (GA) model for fast and reliable grid contingency analyses. According to our review, the persisting limitation of traditional ML techniques in grid analyses is the need for an exhaustive amount of training data for model generalization and accurate predictions. GA models overcome this limitation by first learning true data distribution from a small training set, from which new samples assimilating true data are generated with some variations. Subsequently, GA models can transfer learn or super-generalize with increased accuracy, that is, accurately predict n - (k + 2) contingencies from a small n - k training set and generated n - (k + 1) data. The joint effort between Idaho National Lab and Florida State University strives to develop a zero-shot and deep learning-based contingency analysis tool, named Smart Contingency Analysis Neural Network (SCANN), by leveraging the aforementioned advantages of GA models. The basic architecture of SCANN stems from the Latent Encoding of Atypical Perturbations network combined with an adversarial network, and it is designed to generate imbalanced power flow data from learned true data distributions for prediction purposes. Here we also introduce the abstract concept of resilience-chaos plots, a new resilience characterization tool proposed to complement SCANN by aiding in the assessment of large amounts of high-order contingency predictions.
Topological (graph-theoretic) analysis of critical infrastructure networks provides insight on several aspects of resilience. Graph clustering or community detection, which identifies densely connected components in a graph, has been employed for analysis. In this paper, we propose employing consensus clustering, which is a technique to determine consensus from a collection of different clusters on an input, such that the resulting clustering is robust to disruptions, where a disruption is represented as loss of one or more vertices or edges in the graph. Using two critical infrastructure networks as case studies, we empirically demonstrate the need to compute consensus clustering in order to address the drastic changes in the topology due to disruptions in the network.
Power grids are critical infrastructures that require robust resilience analysis to ensure reliable and uninterrupted electricity supply. Traditional simulation-based methods for grid resilience analysis suffer from computational complexity and limited ability to capture the full spectrum of potential disruptions. This paper presents a novel approach to enhance grid resilience by leveraging transductive graph neural network (GNN) learning to identify critical nodes and links. By leveraging the graph structure and system features, GNNs effectively learn resilience metrics and accurately identify critical nodes based on actual grid operational behavior. The efficacy of the proposed approach is demonstrated through case studies on node criticality scoring and critical node/line identification in cascading outage scenarios. The results highlight the advantages of learning-based methods over traditional simulation-based approaches and their potential to revolutionize grid resilience analysis. The contributions of this paper include a graph-based scalable approach for fast cascading analysis, an inductive formulation for training GNN models, and a transfer learning-based approach to scale the model to largescale power systems.
Even though the discovery of lytic polysaccharide monooxygenases (LPMOs) has fundamentally shifted our understanding of biomass degradation, most of the current studies focused on their roles in carbohydrate oxidation. However, no study demonstrated if LPMO could directly participate to the process of lignin degradation in lignin-degrading microbes. This study showed that LPMO could synergize with lignin-degrading enzymes for efficient lignin degradation in white-rot fungi. The transcriptomics analysis of fungi Irpex lacteus and Dichomitus squalens during their lignocellulosic biomass degradation processes surprisingly highlighted that LPMOs co-regulated with lignin-degrading enzymes, indicating their more versatile roles in the redox network. Biochemical analysis further confirmed that the purified LPMO from I. lacteus CD2 could use diverse electron donors to produce H 2 O 2 , drive Fenton reaction, and synergize with manganese peroxidase for lignin oxidation. The results thus indicated that LPMO might uniquely leverage the redox network toward dynamic and efficient degradation of different cell wall components.
The evolution of the ω phase in titanium–vanadium (Ti–V) alloys is critical for their mechanical properties, particularly in aerospace and biomedical applications. Here, this study employs a Rapid Artificial Neural Network (RANN) potential to model the ω phase evolution at the atomistic level, demonstrating a high degree of consistency with experimental observations, unlike the Modified Embedded Atom Method (MEAM), which fails to capture this phase transformation accurately. RANN simulations replicate key phenomena such as the nucleation of α precipitates at ω/β interfaces and accurate lattice orientations, enhancing our understanding of phase stability and transformation kinetics. The findings affirm that RANN potentials can significantly improve the prediction accuracy of complex material behaviors, offering a powerful tool for designing advanced materials with tailored properties such as solute effect in various stacking fault energies. This approach not only bridges the gap between theoretical predictions and empirical data but also sets a new direction for future research in materials science, emphasizing the integration of machine learning techniques in the development and optimization of new alloys.
Severe malarial anemia (SMA, Hb < 6.0 g/dL) is a leading cause of childhood morbidity and mortality in holoendemic Plasmodium falciparum transmission zones. This study explored the entire expressed human transcriptome in whole blood from 66 Kenyan children with non-SMA (Hb ≥ 6.0 g/dL, n = 41) and SMA (n = 25), focusing on host immune response networks. RNA-seq analysis revealed 6862 differentially expressed genes, with equally distributed up-and down-regulated genes, indicating a complex host immune response. Deconvolution analyses uncovered leukocytic immune profiles indicative of a diminished antigenic response, reduced immune priming, and polarization toward cellular repair in SMA. Weighted gene co-expression network analysis revealed that immune-regulated processes are central molecular distinctions between non-SMA and SMA. A top dysregulated immune response signaling network in SMA was the HSP60-HSP70-TLR2/4 signaling pathway, indicating altered pathogen recognition, innate immune activation, stress responses, and antigen recognition. Validation with high-throughput gene expression from a separate cohort of Kenyan children (n = 50) with varying severities of malarial anemia (n = 38 non-SMA and n = 12 SMA) confirmed the RNA-seq findings. Proteomic analyses in 35 children with matched transcript and protein abundance (n = 19 non-SMA and n = 16 SMA) confirmed dysregulation in the HSP60-HSP70-TLR2/4 signaling pathway. Additionally, glutamine transporter and glutamine synthetase genes were differentially expressed, indicating altered glutamine metabolism in SMA. This comprehensive analysis underscores complex immune dysregulation and novel pathogenic features in SMA.
While several data-driven models have been developed for transient stability assessment, how to consider the uncertainties from load and renewable generations and provide interpretation of data-driven assessment results are still open. This paper proposes an explainable Bayesian Neural Network (BNN) for probabilistic transient stability assessment (TSA). By extracting the uncertainties from loads and wind farms, the BNN model can make a reliable prediction and quantify the prediction uncertainties. We also develop the Gradient Shap algorithm to make the global and local explanations for the probabilistic TSA model, a significant advantage over existing black-box data-driven methods. Numerical results on the modified IEEE 39-bus system show that the proposed method outperforms the existing methods in terms of prediction accuracy and uncertainty quantification capabilities. The explainability of the proposed method allows system operators to design preventive controls for enhancing system stability.
Most transcriptomic studies of SARS-CoV-2 infection have focused on differentially expressed genes, which do not necessarily reveal the genes mediating the transcriptomic changes. In contrast, exploiting curated biological network, our PathExt tool identifies central genes from the differentially active paths mediating global transcriptomic response. Here we apply PathExt to multiple cell line infection models of SARS-CoV-2 and other viruses, as well as to COVID-19 patient-derived PBMCs. The central genes mediating SARS-CoV-2 response in cell lines were uniquely enriched for ATP metabolic process, G1/S transition, leukocyte activation and migration. In contrast, PBMC response reveals dysregulated cell-cycle processes. In PBMC, the most frequently central genes are associated with COVID-19 severity. Importantly, relative to differential genes, PathExt-identified genes show greater concordance with several benchmark anti-COVID-19 target gene sets. We propose six novel anti-SARS-CoV-2 targets ADCY2, ADSL, OCRL, TIAM1, PBK, and BUB1, and potential drugs targeting these genes, such as Bemcentinib, Phthalocyanine, and Conivaptan.
Scalable data management is essential for processing large scientific dataset on HPC platforms for distributed deep learning. In-memory distributed storage is preferred for its speed, enabling rapid, random, and frequent data access required by stochastic optimizers. Processes use one-sided or collective communication to fetch remote data, with optimal performance depending on (i) dataset characteristics, (ii) training scale, and (iii) interconnection network. Empirical analysis shows collective communication excels with larger mini-batch sizes and/or fewer processes, whereas one-sided communication outperforms at larger scales. We propose MDLoader, a hybrid in-memory data loader for distributed graph neural network training. MDLoader features a model-driven performance estimator that dynamically selects between one-sided and collective communication at the beginning of training using Tree of Parzen Estimators (TPE). Evaluations on NERSC Perlmutter and OLCF Summit show MDLoader outperforms single-backend loaders by up to 2.83 × and predicts the suitable communication method with 96.3% (Perlmutter) and 94.3% (Summit) success rate.
We report high-latitude tundra ecosystems are increasingly affected by climate warming. As an important fraction of soil microorganisms, fungi play essential roles in carbon degradation, especially the old, chemically recalcitrant carbon. However, it remains obscure how fungi respond to climate warming and whether fungi, in turn, affect carbon stability of tundra. In a 2-year winter soil warming experiment of 2°C by snow fences, we investigated responses of fungal communities to warming in the active layer of an Alaskan tundra. Although fungal community composition, revealed by the 28S rRNA gene amplicon sequencing, remained unchanged (p > .05), fungal functional gene composition, revealed by a microarray named GeoChip, was altered (p < .05). Changes in functional gene composition were linked to winter soil temperature, thaw depth, soil moisture, and gross primary productivity (canonical correlation analysis, p < .05). Specifically, relative abundances of fungal genes encoding invertase, xylose reductase and vanillin dehydrogenase significantly increased (p < .05), indicating higher carbon degradation capacities of fungal communities under warming. Accordingly, we detected changes in fungal gene networks under warming, including higher average path distance, lower average clustering coefficient and lower percentage of negative links, indicating that warming potentially changed fungal interactions. Together, our study reveals higher carbon degradation capacities of fungal communities under short-term warming and highlights the potential impacts of fungal communities on tundra ecosystem respiration, and consequently future carbon stability of high-latitude tundra.