Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “core tools”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

The E3SM Diagnostics Package (E3SM Diags v2.7): a Python-based diagnostics package for Earth system model evaluation

Abstract. The E3SM Diagnostics Package (E3SM Diags) is a modern, Python-based Earth system model (ESM) evaluation tool (with Python module name e3sm_diags), developed to support the Department of Energy (DOE) Energy Exascale Earth System Model (E3SM). E3SM Diags provides a wide suite of tools for evaluating native E3SM output, as well as ESM data on regular latitude–longitude grids, including output from Coupled Model Intercomparison Project (CMIP) class models. E3SM Diags is modeled after the National Center for Atmospheric Research (NCAR) Atmosphere Model Working Group (AMWG, 2022) diagnostics package. In its version 1 release, E3SM Diags included a set of core essential diagnostics to evaluate the mean physical climate from model simulations. As of version 2.7, more process-oriented and phenomenon-based evaluation diagnostics have been implemented, such as analysis of the quasi-biennial oscillation (QBO), the El Niño–Southern Oscillation (ENSO), streamflow, the diurnal cycle of precipitation, tropical cyclones, ozone and aerosol properties. An in situ dataset from DOE's Atmospheric Radiation Measurement (ARM) program has been integrated into the package for evaluating the representation of simulated cloud and precipitation processes. This tool is designed with enough flexibility to allow for the addition of new observational datasets and new diagnostic algorithms. Additional features include customizable figures; streamlined installation, configuration and execution; and multiprocessing for fast computation. The package uses an up-to-date observational data repository maintained by its developers, where recent datasets are added to the repository as they become available. Finally, several applications for the E3SM Diags module were introduced to fit a diverse set of use cases from the scientific community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Biofoundries: Principles, Tools, and Applications

This chapter aims to provide a broad overview of biofoundries and introduces the principles, concepts, and case studies. We first outline the underlying principles of the Design-Build-Test-Learn (DBTL) framework and the role of automation, digital integration, and standardization. The chapter then explores core biofoundry technologies including robotic liquid handlers, high-throughput analytical instruments, and digital infrastructure for data management and workflow scheduling. Case studies spanning DNA assembly, protein engineering, metabolic engineering, and mammalian cell culture demonstrate the practical applications of the biofoundries. Economic and societal impacts are assessed alongside current limitations. We discuss the emerging trends including artificial intelligence integration and cloud-based distributed facilities to highlight its potential for biotechnology and the bioeconomy.

Singh, Nilmani↗
DOE OSTI2026

Mauka Energy FEVER Tool Dataset

Mauka Energy’s dataset, developed under the Forestry Electric Vehicle Energy Routing (FEVER) project and funded by the U.S. Department of Energy’s Small Business Innovation Research program, is a high-resolution geospatial resource designed to support energy modeling for electric log trucks in complex forestry environments. The dataset integrates detailed spatial and road network data to enable accurate simulation of vehicle performance across varied terrain. At its core, the dataset incorporates lidar-derived elevation models, road alignments, and surface classifications from Oregon State University’s McDonald-Dunn Research Forest. These data capture fine-scale variations in slope, curvature, and surface conditions across forest road systems, allowing for vehicle-level analysis of energy consumption and recovery. The dataset also includes data collected on the surrounding public and private road networks in Benton County, Oregon, used in real-world haul routes. These connecting segments provide critical context for modeling transitions between forest operations and regional transportation infrastructure, incorporating attributes such as grade profiles, elevation change, and speed constraints. This combined dataset underpins the development of Mauka Energy’s rolldown tool, which quantifies energy use and regenerative braking potential on downhill and variable-grade segments. By leveraging high-resolution terrain and road data, the FEVER project enables more accurate assessment of electric vehicle feasibility and performance in forestry applications.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Ard [SWR-25-18]

A wind farm optimization suite for wind energy that is built for modular, gradient-enabled multi-disciplinary and multi-fidelity optimizations. Dig into wind farm design. An ard is a type of simple and lightweight plow, used through the single-digit centuries to prepare a farm for planting. The intent of Ard is to be a modular, full-stack multi-disciplinary optimization tool for wind farms. The problem with wind farms is that they are complicated, multi-disciplinary objects. They are aerodynamic machines, with complicated control systems, power electronic devices, social and political objects, and the core value (and cost) of complicated financial instruments. Moreover, the design of one of these aspects affects all the rest! Ard seeks to make plant-level design choices that can incorporate these different aspects and their interactions to make wind energy projects more successful.

Frontin, Cory [National Renewable Energy Laborator↗
DOE OSTI2025

Community Requirements Meta-Analysis: Characterizing Needs and Opportunities for HPDF

This High Performance Data Facility (HPDF) Project is creating a new scientific user facility to provide advanced infrastructure for data-intensive science, supporting the DOE’s Office of Science (SC) community. HPDF’s mission is to enable and accelerate scientific discovery by delivering state-of-the-art data management infrastructure, capabilities, and tools. This meta-analysis examines the needs of the breadth of the SC community, captured in publicly available community reports or mission documents. The meta-analysis identifies and provides initial characterization of fifteen core requirements for the HPDF Project team to consider during the conceptual design phase. The fifteen requirements illustrate how scientific work among SC communities requires modern, seamless user experiences across the ASCR Ecosystem to advance the use of large volumes of heterogeneous data. The scientific community requires support for the missing middle of compute between local and HPC to interactively and collaboratively use growing datasets. Data producers and end users will benefit from enhanced data catalogs and portals that improve data access through advanced search of well curated data. The fifteen requirements are examined here organized across five themes for discussion. Examples in each theme illustrate the array of scientific needs that convey the important role that the fully realized and operational High Performance Data Facility will be able to play as an integral part of the evolving ASCR Ecosystem. Our amalgamated data tables from ESnet reports demonstrate ranges to the volumes of data HPDF must be concerned with, but limitations are inherent to this meta-analysis (see Key Challenges & Limitations). Feedback and validation of these requirements along with additional details and emergent community requirements will be gathered through user research and design activities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

MOOSE framework enhancements for meshing reactor geometries

MOOSE is an open-source, parallel finite element framework designed to permit rapid development of robust multi-physics modeling capabilities. Under the DOE-NEAMS program, numerous solvers have been developed utilizing the open-source MOOSE framework for multiphysics reactor analysis. These solvers require input finite element meshes representing the discretized geometry. Typically, reactor analysts turn to licensed external tools for creation of reactor geometry meshes. Recently, enhancements have been added to the MOOSE framework to mesh common reactor geometries and improve MOOSE-based application user workflows. Support for hexagonal pins, assemblies, and cores has been added, and Cartesian support has been extended. Options for modeling static and rotating control drums within a hexagonal assembly are now available. Pin, assembly, and plane regions can be identified through automatically applied tags on the mesh called 'reporting IDs' for easier post- processing of physics results. An external open-source triangle routine has been leveraged within MOOSE to mesh core periphery zones. A set of reactor geometry builder routines further streamline the construction of hexagonal and Cartesian cores and include the ability to assign materials to regions during mesh generation. The new meshing routines are available through the MOOSE framework in the open-source 'Reactor' module, and the resulting directly within MOOSE-based applications or exported as Exodus II files for use in other finite element solvers. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2025

Dynamics modeling of molten salt reactor with reduced and expanded representations of delayed neutron precursors

Molten salt reactors (MSRs) present unique challenges in dynamic behavior due to the mobility of their fuel. In these reactors, delayed neutron precursors (DNPs) drift with the fuel circulation through the primary loop. As a result, a fraction of DNPs decays outside the core, effectively reducing the available delayed neutron population for reactivity control. Consequently, precise modeling of the distribution and behavior of DNPs is critical for accurate reactor dynamics simulations. In this study, the System Dynamics Analysis Tool (SDAT) was used to simulate a thermal-spectrum MSR under steady-state conditions and following transients. The effects of using reduced and expanded representations of DNPs with fewer or more groups than the conventional 6-group model were investigated. Their impact on the simulated distribution of precursors in the primary loop, reactivity loss value, and reactor response to transients was analyzed. Simulation results showed that reduced models lead to the loss of the actual DNPs distribution data, resulting in less accurate estimates of reactivity loss. Reactor power predictions using these reduced models showed significant deviations compared to those using the conventional 6-group model in transient simulations. Expanded models offered a more accurate representation of the distribution of DNPs and reactivity loss estimates. Reactor power predictions using expanded models showed minimal deviation from the conventional 6-group model during the simulated transients.

analysis↗
DOE OSTI2022

CTF: A modernized, production-level, thermal hydraulic solver for the solution of industry-relevant challenge problems in pressurized water reactors

CTF is a thermal hydraulic (T/H) subchannel tool that has been extensively developed over the past ten years as part of the Consortium for Advanced Simulation of Light Water Reactors (CASL) program. The code was selected early in the CASL program for support of high-impact challenge problems that were found to be relevant to the nuclear industry and its currently operating fleet of pressurized water reactors (PWRs), including issues such as departure from nucleate boiling (DNB), crud-induced power shifts (CIPSs), and reactivity-insertion accidents (RIAs). By incorporating CTF into the multiphysics Virtual Environment for Reactor Application (VERA) core simulator software developed by CASL, CTF has become the primary means of providing fluid and fuel thermal feedback, as well as T/H figure-of-merits (FOMs) in large-scale reactor simulations. With the goal of solving industry challenge problems, CASL placed great emphasis on developing high-quality, high-performance, validated software tools that offer higher fidelity than what is currently possible with current industry methods. In support of this effort, CTF was developed from a research tool into an nuclear quality assurance (NQA-1)–compliant, production-level software tool that is capable of addressing the stated challenge problems and goals of CASL. This work presents a review of the major technological achievements that were realized in developing CTF over the past decade of the CASL program and presents an overview of the code solution approach and closure models.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2025

Summary Report Of The FY25 Computational Fluid Dynamics Verification And Validation Exercises In The Advanced Reactor Technologies - Gas-cooled Reactor Program

Verification and Validation (V&V) of numerical tools is critical for ensuring reasonable predictions during design, safety analysis, and licensing. Recent work in the Advanced Reactor Technologies - Gas-cooled Reactor (ART-GCR) program has focused on V&V of common Computational Fluid Dynamics (CFD) tools that are used within the Untied States. This report presents an update on these CFD V&V activities. These Generation IV Forum (GIF) Very High Temperature Reactor (VHTR) Computational Methods, Validation, and Benchmarking (CMVB) is an international organization dedicated to the verification and validation of High Temperature Gas-Cooled Reactor (HTGR) simulation tools. Participation in the CMVB provides additional value to the V&V activities, as it allows for access to a wider range of data, and provides valuable benchmarking exercises. Three HTGR phenomena are targeted: Reactor Cavity Cooling System (RCCS) performance, core bypass flow, and lower plenum mixing. Simulations of the University of Wisconsin-Madison (UW-Madison) RCCS facilities are performed with Reynolds Averaged Navier-Stokes (RANS) in StarCCM+. Results are compared for both forced and natural convection conditions, with both exhibiting good agreement with experimental measurements. The Idaho National Laboratory (INL) matched index of refraction (MIR) and Korean Atomic Energy Research Institute (KAERI) bypass flow expeirments are used to validation CFD predictions of bypass flow. Simulations are performed with RANS in StarCCM+ and with Large Eddy Simulation (LES) in NekRS. Finally, preliminary simulations of the Institute of Nuclear and New Energy Technology (INET) lower plenum mixing facilities are presented. Initial work has developed models with LES, RANS, and porous media models. These preliminary models are presented and compared to each other to gauge differences in predictions with each of the three methods.

and Benchmarking (CMVB)↗
DOE OSTI2020

CoreCruncher : Fast and Robust Construction of Core Genomes in Large Prokaryotic Data Sets

The core genome represents the set of genes shared by all, or nearly all, strains of a given population or species of prokaryotes. Inferring the core genome is integral to many genomic analyses, however, most methods rely on the comparison of all the pairs of genomes; a step that is becoming increasingly difficult given the massive accumulation of genomic data. Here, we present CoreCruncher; a program that robustly and rapidly constructs core genomes across hundreds or thousands of genomes. CoreCruncher does not compute all pairwise genome comparisons and uses a heuristic based on the distributions of identity scores to classify sequences as orthologs or paralogs/xenologs. Although it is much faster than current methods, our results indicate that our approach is more conservative than other tools and less sensitive to the presence of paralogs and xenologs. CoreCruncher is freely available from: https://github.com/lbobay/CoreCruncher. CoreCruncher is written in Python 3.7 and can also run on Python 2.7 without modification. It requires the python library Numpy and either Usearch or Blast. Certain options require the programs muscle or mafft.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

Fully generalized, turbulent trace impurity transport with Gkeyll and Flan in the DIII-D far-SOL

The Monte Carlo trace impurity turbulent transport code Flan is introduced for the first time. Flan follows impurities in a turbulent background plasma from Gkeyll using the Lorentz force to resolve the full particle gyro orbit. Collisions are handled using the Nanbu collision algorithm (Nanbu 1997 Phys. Rev. E 55 4642–52), and ionization/recombination is handled via ADAS coupling. The far-SOL of a generic DIII-D L-mode is simulated with and without the collision model to show how collisions affect radial tungsten transport. Anomalous diffusion coefficient (D r ) and pinch velocity (v p ) profiles are extracted from fits to the results. With collisions, D r and v p are between 0–1.0 m 2 s −1 and −100–100 m s −1 , respectively. Without collisions, D r and v p are between 0–0.3 m 2 s −1 and −50–50 m s −1 , respectively. Exponential fits to the radial W density profiles and experimental data from W deposition along a collector probe are in good agreement, demonstrating Flan as a useful interpretive modeling tool. Additional simulations show that impurity transport away from the wall increases with atomic number, though it is not clear why. Flan has the potential to better interpret existing data and improve reactor scale predictions of core contamination because the underlying physics model is very general and does not rely on arbitrary user-defined transport coefficients.

DIII-D↗
DOE OSTI2021

Femtosecond laser fabrication of nanograting-based distributed fiber sensors for extreme environmental applications

Abstract The femtosecond laser has emerged as a powerful tool for micro- and nanoscale device fabrication. Through nonlinear ionization processes, nanometer-sized material modifications can be inscribed in transparent materials for device fabrication. This paper describes femtosecond precision inscription of nanograting in silica fiber cores to form both distributed and point fiber sensors for sensing applications in extreme environmental conditions. Through the use of scanning electron microscope imaging and laser processing optimization, high-temperature stable, Type II femtosecond laser modifications were continuously inscribed, point by point, with only an insertion loss at 1 dB m −1 or 0.001 dB per point sensor device. High-temperature performance of fiber sensors was tested at 1000 °C, which showed a temperature fluctuation of ±5.5 °C over 5 days. The low laser-induced insertion loss in optical fibers enabled the fabrication of a 1.4 m, radiation-resilient distributed fiber sensor. The in-pile testing of the distributed fiber sensor further showed that fiber sensors can execute stable and distributed temperature measurements in extreme radiation environments. Overall, this paper demonstrates that femtosecond-laser-fabricated fiber sensors are suitable measurement devices for applications in extreme environments.

Wang, Mohan (ORCID:0000000166784993)↗
DOE OSTI2023

Engineering Computational Practices (Rev. 1)

This manual will attempt to motivate the use of an automated build system for the purposes of computational science and engineering. As part of this motivation, the surrounding computational practices of version control, documen tation, compute environment management, and regression testing will also be addressed as applied to the practice of computational engineering. Specifically, this manual intends to motivate the adoption of these traditional software engineering practices for use in research and production engineering simulation projects. This manual is not the first such effort in the greater scientific computing community. In fact, the authors relied heavily on the lesson plans of the Software Carpentry, established to teach computing skills to researchers in 1998. As the intention for this manual is to lay out fundamental practices of engineering computing, it will not attempt to fully teach the underlying concepts and will instead reference the well designed lesson plans of the Software Carpentry. Where possible, this manual will explain to general computing practices and concepts and limit discussion of specific software implementations to examples or vehicles for practice in concrete application. The specific software taught by the Software Carpentry curriculum is an excellent starting point to learn the core concepts of computational engi neering. However, the authors have found that applications to engineering simulation and analysis require translation of these software development concepts into the language and workflows of computational engineers. Adopting these computational tools may require engineers to re-imagine their workflows in some combination of traditional engineer ing and software concepts. It has also been necessary to extend existing software build systems for engineering practices beyond the simple wrap ping of engineering software execution. Where necessary, examples of specific software and their method of extension to engineering simulations will be given, with reference to the User Manual for recommended practical use. Where this manual relies on specific implementation examples, it should be understood that the practicing engineer may find that different software is more amenable to their specific work. It is always the overall collection of computational practices is more important than any specific software implementation. The ability to recognize which concepts are implemented by a software package will make a practicing engineer agile to changing project needs, computing resources, numeric solvers, programming languages, and even available funding.

42 ENGINEERING↗
DOE OSTI2024

Description of FY25 Theory and Simulation Performance Target: Development of an integrated modeling framework for fusion reactor design and assessment

The urgency to deliver fusion power is growing now more than ever, with increasing pressure for both public programs and private companies to meet milestones timelines and overcome significant remaining technical challenges to ensure growth of a nascent fusion industry in time to meet rapidly growing clean energy demands. With incredible advancements in computation and years of investment in fusion model development and validation, integrated modeling is poised to fill a key role in accelerating the timeline to a fusion pilot plant (FPP). Future fusion pilot plants will operate in regimes far beyond current experience, and device design will rely on physics-based prediction and extrapolation. Many concepts will also rely on simulation to assess safety (shielding, tritium management, materials activation and lifetimes), economics and scalability before the decision to build. Importantly, integrated simulation can be used to reveal and solve the complexities of system integration that may otherwise not be apparent in physical components or models developed in isolation. New experimental test facilities that produce relevant conditions to validate and resolve key technical challenges for various subsystems (materials, blankets, fuel cycle, etc.) have been repeatedly called for by the fusion community but are not yet realized. Integrated modeling has an important role in identifying realistic load conditions (thermal, electromagnetic, plasma, neutron and photon loads, etc.) and defining the components and experiments for these test facilities in order to ensure meaningful validation that sufficiently reduces modeling uncertainties and technical risk for the full integrated reactor. The Fusion REactor Design and Assessment (FREDA) SciDAC project is building a component-based integrated modeling framework & data structure to enable self-consistent, multi-fidelity, iterative optimization workflows for the fusion reactor design process. FREDA aims to shorten the time to viable designs by providing a set of flexible workflows to support the various stages of the design process using an integrated model hierarchy, ranging from the simple analytic descriptions to the highest fidelity, theory-based plasma and engineering modeling developed by the fusion and fission communities. These tools are expected to be needed for timely support of FPP design in the milestone program and in the FIRE collaboratives. The plasma simulation backbone of FREDA is IPS-FASTRAN with newly developed coupled Core-Edge Pedestal-SOL (CESOL) workflows, which is being extended to the far-SOL region up to the plasma facing components. FREDA incorporates the FERMI engineering modeling suite and will enable self-consistent evaluation of the thermal shields, limiters, blanket, magnets, and other surrounding structures with predictions of temperatures, erosion, dpa, activation, tritium generation and transport, creep, corrosion, material degradation, etc. Parametric generation of 3D CAD enables rapid iteration of component geometry in response to plasma and loading specifications.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

FunM2C: A Filter for Uncertainty Visualization of Multivariate Data on Multi-Core Devices

Uncertainty visualization is an emerging research topic in data visualization because neglecting uncertainty in visualization can lead to inaccurate assessments. In this paper, we study the propagation of multivariate data uncertainty in visualization. Although there have been a few advancements in probabilistic uncertainty visualization of multivariate data, three critical challenges remain to be addressed. First, the state-of-the-art probabilistic uncertainty visualization framework is limited to bivariate data (two variables). Second, existing uncertainty visualization algorithms use computationally intensive techniques and lack support for cross-platform portability. Third, as a consequence of the computational expense, integration into production visualization tools is impractical. In this work, we address all three issues and make a threefold contribution. First, we take a step to generalize the state-of-the-art probabilistic framework for bivariate data to multivariate data with an arbitrary number of variables. Second, through utilization of VTK-m’s shared-memory parallelism and cross-platform compatibility features, we demonstrate acceleration of multivariate uncertainty visualization on different many-core architectures, including OpenMP and AMD GPUs. Third, we demonstrate the integration of our algorithms with the ParaView software. We demonstrate the utility of our algorithms through experiments on multivariate simulation data with three and four variables.

Hari, Gautam↗
DOE OSTI2025

Web-Based Tools for Data-Informed Remedy Optimization: Software Theory and User Guide

This report documents the development and application of two web-based decision-support tools for pump-and-treat (P&T) groundwater remediation systems: PTOLEMY (Pump-and-Treat Optimized Location Evaluation to Maximize Yields) and OPTIMA (Optimization for Pump-and-Treat Implementation, Management, & Assessment). These tools enhance remedy design and management by leveraging advanced computational methods – specifically deep learning and multi-objective optimization – within a user-friendly platform. By integrating data-driven models with established hydrogeological knowledge, PTOLEMY and OPTIMA enable more efficient evaluation of well placement and operational strategies, helping site managers balance multiple remediation objectives under complex conditions. Both tools are implemented as modules within the SOCRATES (Suite Of Comprehensive Rapid Analysis Tools for Environmental Sites) web platform, which provides data access, visualization, and analytics to support remedy optimization across sites in the U.S. Department of Energy Office of Environmental Management complex. PTOLEMY is a rapid screening module designed to identify promising locations for new extraction wells. It employs a multi-channel three-dimensional convolutional neural network (MC3D-CNN) trained on high-fidelity simulation data to predict the relative performance (in terms of contaminant mass recovery) of potential well sites. Through an interactive web interface, PTOLEMY visualizes the probability of high performance across a site, highlighting areas where an extraction well is likely to yield above-threshold contaminant removal over a multi-year period. PTOLEMY’s map-based displays and exportable results support transparent communication of screening analyses. By focusing attention on the most favorable candidate locations, the tool augments traditional engineering judgment and physics-based modeling, providing a data informed basis for subsequent detailed evaluations. OPTIMA is a multi objective optimization module designed to find wellfield layouts and operating schedules that meet various cleanup goals. It quickly evaluates thousands of candidate setups – combinations of well locations, timing, and rates – and returns a small set of best trade-off options for comparison. At its core, OPTIMA uses a U-Net-based surrogate model – a deep-learning emulator of a groundwater flow and transport simulator – to dramatically accelerate scenario evaluations. Coupling this fast surrogate with the NSGA-II (Non-dominated Sorting Genetic Algorithm II) evolutionary algorithm, OPTIMA explores a wide decision space of well locations and schedules to identify Pareto-optimal solutions that trade off key objectives (e.g., minimizing cleanup time, maximizing contaminant mass removal, and minimizing plume extent). The tool outputs a family of optimal configurations and visualizes their trade-offs (Pareto frontiers of cleanup metrics and maps of optimized well placements). Site managers can use these results to understand the range of viable strategies and to select candidate designs for more detailed verification. OPTIMA is currently under active development and not yet fully released; this guide provides early documentation to support planning and gather user feedback.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Towards High-Performance AI4NP Applications on Modern GPU Platforms

The evolution of modern heterogeneous accelerators, such as GPUs, has significantly advanced the landscape of artificial intelligence (AI). There is a notable surge to adopt AI within the nuclear physics domain (AI4NP). While most AI4NP studies focus on feasibility analysis, our attention is directed towards evaluating their performance on contemporary GPUs that integrate tensor cores. We first benchmark the throughput of hyperparameterized multi-layer perceptron (MLP) models. We then examine the performance of an AI4NP application: Hydra. We assess the performance gain and accuracy loss caused by the tensor cores for low-precision floating-point operations. Our experiments encompass the PyTorch and TensorFlow Keras frameworks on NVIDIA’s T4 and A100 GPUs. We explore the behavior of different GPU hardware platforms and AI software tools. This study can be a valuable resource for guiding the performance optimization of larger-scale deployments of AI4NP applications.

Mei, Xinxin↗
DOE OSTI2021

Scalable Pattern Matching in Metadata Graphs via Constraint Checking

Pattern matching is a fundamental tool for answering complex graph queries. Unfortunately, existing solutions have limited capabilities: They do not scale to process large graphs and/or support only a restricted set of search templates or usage scenarios. Moreover, the algorithms at the core of the existing techniques are not suitable for today’s graph processing infrastructures relying on horizontal scalability and shared-nothing clusters, as most of these algorithms are inherently sequential and difficult to parallelize. In this article we present an algorithmic pipeline that bases pattern matching on constraint checking. The key intuition is that each vertex and edge participating in a match has to meet a set of constraints implicitly specified by the search template. These constraints can be verified independently and typically are less expensive to compute than searching the full template. The pipeline we propose generates these constraints and iterates over them to eliminate all the vertices and edges that do not participate in any match, thus reducing the background graph to a subgraph that is the union of all template matches—the complete set of all vertices and edges that participate in at least one match. Additional analysis can be performed on this annotated, reduced graph, such as full match enumeration, match counting, or computing vertex/edge centrality. Furthermore, a vertex-centric formulation for constraint checking algorithms exists, and this makes it possible to harness existing high-performance, vertex-centric graph processing frameworks. This technique (i) enables highly scalable pattern matching in metadata (labeled) graphs; (ii) supports arbitrary patterns with 100% precision; (iii) enables tradeoffs between precision and time-to-solution, while always selects all vertices and edges that participate in matches, thus offering 100% recall; and (iv) supports a set of popular data analytics scenarios. We implement our approach on top of HavoqGT, an open-source asynchronous graph processing framework, and demonstrate its advantages through strong and weak scaling experiments on massive scale real-world (up to 257 billion edges) and synthetic (up to 4.4 trillion edges) labeled graphs, respectively, and at scales (1,024 nodes / 36,864 cores), orders of magnitude larger than used in the past for similar problems. This article serves two purposes: First, it synthesises the knowledge accumulated during a long-term project. Second, it presents new system features, usage scenarios, optimizations, and comparisons with related work that strengthen the confidence that pattern matching based on iterative pruning via constraint checking is an effective and scalable approach in practice. The new contributions include the following: (i) We demonstrate the ability of the constraint checking approach to efficiently support two additional search scenarios that often emerge in practice, interactive incremental search and exploratory search. (ii) We empirically compare our solution with two additional state-of-the-art systems, Arabsque and TriAD. (iii) We show the ability of our solution to accommodate a more diverse range of datasets with varying properties, e.g., scale, skewness, label distribution, and match frequency. (iv) We introduce or extend a number of system features (e.g., work aggregation, load balancing, and the ability to cap the generated traffic) and design optimizations and demonstrate their advantages with respect to improving performance and scalability. (v) We present bottleneck analysis and insights into artifacts that influence performance. (vi) We present a theoretical complexity argument that motivates the performance gains we observe.

97 MATHEMATICS AND COMPUTING↗