Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “plugins”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Scalable Deep Learning via I/O Analysis and Optimization

Scalable deep neural network training has been gaining prominence because of the increasing importance of deep learning in a multitude of scientific and commercial domains. Consequently, a number of researchers have investigated techniques to optimize deep learning systems. Much of the prior work has focused on runtime and algorithmic enhancements to optimize the computation and communication. Despite these enhancements, however, deep learning systems still suffer from scalability limitations, particularly with respect to data I/O. This situation is especially true for training models where the computation can be effectively parallelized, leaving I/O as the major bottleneck. In fact, our analysis shows that I/O can take up to 90% of the total training time. Thus, in this article, we first analyze LMDB, the most widely used I/O subsystem of deep learning frameworks, to understand the causes of this I/O inefficiency. Based on our analysis, we propose LMDBIO—an optimized I/O plugin for scalable deep learning. LMDBIO includes six novel optimizations that together address the various shortcomings in existing I/O for deep learning. Lastly, our experimental results show that LMDBIO significantly outperforms LMDB in all cases and improves overall application performance by up to 65-fold on a 9,216-core system.

97 MATHEMATICS AND COMPUTING↗

Extending C++ for Heterogeneous Quantum-Classical Computing

In this report we present qcor - a language extension to C++ and compiler implementation that enables heterogeneous quantum-classical programming, compilation, and execution in a single-source context. Our work provides a first-of-its-kind C++ compiler enabling high-level quantum kernel (function) expression in a quantum-language agnostic manner, as well as a hardware-agnostic, retargetable compiler workflow targeting a number of physical and virtual quantum computing backends. qcor leverages novel Clang plugin interfaces and builds upon the XACC system-level quantum programming framework to provide a state-of-the-art integration mechanism for quantum-classical compilation that leverages the best from the community at-large. qcor translates quantum kernels ultimately to the XACC intermediate representation, and provides user-extensible hooks for quantum compilation routines like circuit optimization, analysis, and placement. This work details the overall architecture and compiler workflow for qcor, and provides a number of illuminating programming examples demonstrating its utility for near-term variational tasks, quantum algorithm expression, and feed-forward error correction schemes.

97 MATHEMATICS AND COMPUTING↗

OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPC

Federated Learning (FL) is critical for edge and High Performance Computing (HPC) where data is not centralized and privacy is crucial. We present OmniFed, a modular framework designed around decoupling and clear separation of concerns for configuration, orchestration, communication, and training logic. Its architecture supports configuration-driven prototyping and code-level override-what-you-need customization. We also support different topologies, mixed communication protocols within a single deployment, and popular training algorithms. It also offers optional privacy mechanisms including Differential Privacy (DP), Homomorphic Encryption (HE), and Secure Aggregation (SA), as well as compression strategies. These capabilities are exposed through well-defined extension points, allowing users to customize topology and orchestration, learning logic, and privacy/compression plugins, all while preserving the integrity of the core system. We evaluate multiple models and algorithms to measure various performance metrics. By unifying topology configuration, mixed-protocol communication, and pluggable modules in one stack, OmniFed streamlines FL deployment across heterogeneous environments. Github repository is available at https://github.com/at-aaims/OmniFed.

Tyagi, Sahil [ORNL] (ORCID:0009000783144745)↗

CGSim: A Simulation Framework for Large Scale Distributed Computing Environment

Large-scale distributed computing infrastructures such as the Worldwide LHC Computing Grid (WLCG) require comprehensive simulation tools for evaluating performance, testing new algorithms, and optimizing resource allocation strategies. However, existing simulators suffer from limited scalability, hardwired algorithms, lack of real-time monitoring, and inability to generate datasets suitable for modern machine learning approaches. We present CGSim, a simulation framework for large-scale distributed computing environments that addresses these limitations. Built upon the validated SimGrid simulation framework, CGSim provides high-level abstractions for modeling heterogeneous grid environments while maintaining accuracy and scalability. Key features include a modular plugin mechanism for testing custom workflow scheduling and data movement policies, interactive real-time visualization dashboards, and automatic generation of event-level datasets suitable for AI-assisted performance modeling. We demonstrate CGSim’s capabilities through a comprehensive evaluation using production ATLAS PanDA workloads, showing significant calibration accuracy improvements across WLCG computing sites. Scalability experiments show near-linear scaling for multi-site simulations, with distributed workloads achieving 6 × better performance compared to single-site execution. The framework enables researchers to simulate WLCG-scale infrastructures with hundreds of sites and thousands of concurrent jobs within practical time budget constraints on commodity hardware.

Vatsavai, Sairam Sri [Brookhaven National Laborato↗

Fine-Grained Application Energy and Power Measurements on the Frontier Exascale System

The increasing complexity and power/energy demands of heterogeneous exascale systems, such as the Frontier supercomputer, present significant challenges for measuring and optimizing power consumption in applications. Current tools either lack the resolution to capture fine-grained power and energy measurements, fail to validate in-band measurements against out-of-band power sensors, or cannot integrate this information with application performance events in a scalable manner. This paper introduces a novel open-source performance toolkit that integrates extended PAPI components with Score-P plugins to enable in-band, fine-grained power and energy measurements, while also supporting validation using power meter measurements for both CPUs and GPUs. One key contribution is the ability to perform millisecond-level power and energy measurements for AMD MI250X GPUs, mapping them to application performance events within a single trace and measurement system that scales. Our toolkit combines coarse-grained measurements from cray_pm counters with high-resolution metrics from rocm_smi and RAPL, converting GPU instantaneous accumulated energy into power to capture both transient and steady-state power behavior, a capability often missed by out-of-band and monitoring tools. By mapping these metrics to specific application regions, developers can identify energy hotspots, address inefficiencies in GPU kernel execution, and validate in-band measurements against external measurements. We demonstrate the effectiveness of this approach through case studies using benchmarks such as GPU rocblas_sgemm, BLIS c_blas_dgemm, and rocHPL, highlighting the variability of the measurements and the impact of transient power spikes on kernel-level efficiency.

Hernandez Mendoza, Oscar [ORNL] (ORCID:00000002538↗

Coopr Forum v 1.0

SNL Coopr Forum is a set of Python routines that provide extensions to the Coopr optimization software. This software augments the open-source Coopr project to provide a more comprehensive capability for formulating and solving optimization applications.Optimization is a common analysis tool used in engineering design and scientific discovery. The SNL Coopr Forum software augments the Coopr Forum open source project to provide extensions of the Coopr optimization framework. Specifically, Coopr Forum provides plugin tools that allow Coopr to apply different optimization solvers, manage optimization solvers in a distributed manner, and to process input and output files. Thus, Coopr Forum is comprised of a variety of distinct software components, each of which provides a different capability that can be used within Coopr.

Hart, William↗

F3D Image Processing and Analysis for Many - and Multi-core Platforms

F3D is written in OpenCL, so it achieve[sic] platform-portable parallelism on modern mutli-core CPUs and many-core GPUs. The interface and mechanims to access F3D core are written in Java as a plugin for Fiji/ImageJ to deliver several key image-processing algorithms necessary to remove artifacts from micro-tomography data. The algorithms consist of data parallel aware filters that can efficiently utilizes[sic] resources and can work on out of core datasets and scale efficiently across multiple accelerators. Optimizing for data parallel filters, streaming out of core datasets, and efficient resource and memory and data managements over complex execution sequence of filters greatly expedites any scientific workflow with image processing requirements. F3D performs several different types of 3D image processing operations, such as non-linear filtering using bilateral filtering and/or median filtering and/or morphological operators (MM). F3D gray-level MM operators are one-pass constant time methods that can perform morphological transformations with a line-structuring element oriented in discrete directions. Additionally, MM operators can be applied to gray-scale images, and consist of two parts: (a) a reference shape or structuring element, which is translated over the image, and (b) a mechanism, or operation, that defines the comparisons to be performed between the image and the structuring element. This tool provides a critical component within many complex pipelines such as those for performing automated segmentation of image stacks. F3D is also called a "descendent" of Quant-CT, another software we developed in the past. These two modules are to be integrated in a next version. Further details were reported in: D.M. Ushizima, T. Perciano, H. Krishnan, B. Loring, H. Bale, D. Parkinson, and J. Sethian. Structure recognition from high-resolution images of ceramic composites. IEEE International Conference on Big Data, October 2014.

Source record↗

DMS Analysis Tool Manager

The DMS Analysis Tool Manager is a part of PRISM, the Proteomics Research Information and Management System. The analysis manager performs automated processing of available job steps, tracked by the in the DMS_Pipeline database. Plugin DLLs implement the processing details for each analysis tool.

Gibbons, Bryson↗

DMS Capture Task Manager

The DMS Capture Task Manager is a part of PRISM, the Proteomics Research Information and Management System. The capture manager processes instrument data files, copying them from the instrument to storage servers, validating the data, creating QC graphics, and pushing the data into MyEMSL. Plugin DLLs implement the processing details for each tool.

Gibbons, Bryson↗

XFEMParaviewPlugin

The XFEMClipPartialElements tool is a plugin for the open-source Paraview visualization software to correctly visualize results with discontinuities represented using the extended finite element method (XFEM) in MOOSE-based codes.

Spencer, Ben↗

Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets v1.1.0

BASIN-3D (Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets) uses a data brokering approach to synthesize diverse earth science data from a variety of remote sources in real-time without the need for additional storage. BASIN-3D is an extendable Django web framework application using a generalized data synthesis model that makes the synthesized data available via REpresentational State Transfer (REST) Application Programming Interface (API). We have currently implemented our data synthesis model to represent sensor-based time series earth science observations across a hierarchy of spatial locations. Supporting our data synthesis model is a plugin framework that allows developers to map data sources of interest to the BASIN-3D synthesis model

Varadharajan, Charuleka↗

TEAL

TEAL is a financial performance calculator plugin for the RAVEN code, framework, resolving around the computation of Net Present Value and associated financial metrics. TEAL can make use of inflation rates, taxation, escalation factors, capital expenditure economy of scale scaling factors. The unique feature of TEAL is the capability to be linked with RAVEN external models and build corresponding cash flows using the variables computed by those external models. In addition to be able to use the capability to generate cash flows derived from complex physical models generated by RAVEN, another distinctive feature of TEAL is the capability to provide financial risk/probabilistic metrics that can empower RAVEN to perform optimization/analysis driven by financial risk augmentations. Optimization, robust optimization, parametric studies, large parallel simulations, sensitivity analysis, data mining, etc. are just some of the capabilities that can be leveraged.

Alfonsi, Andrea↗

Industrial Control Systems Network Protocol Parsers

Industrial Control Systems protocol parsers plugins for the Zeek network security monitoring framework. Currently we have four fully developed protocol parsers but we plan on adding more in the future. The protocol parsers we currently have developed are for BACnet, DNP3, Ethernet/IP, and Modbus.

Rasmussen, BrettD↗

Saloon

Saloon is a Vim plugin that simplifies Python code linter / fixer configuration and usage. Saloon's menu lets developers toggle which static analysis tools to use and delegates those changes to ALE's (Asynchronous Lint Engine) API. Since prospector (python linter aggregate) already handles multiple tools, and is integrated with ALE, most of the actual linting will initially be handled via prospector calls.

Bloss, DavidK.↗

Warewulf 3.0

Warewulf is a scalable systems management suite originally developed to manage large high-performance Linux clusters. Focused on general scalable systems management, it includes a framework for system configuration, management, provisioning/installation, monitoring, event notification, and more via a modular plugin architecture. Install the components and features you need or leverage the existing system configuration stored within Warewulf to create custom solutions to meet your performance needs. One of the goals of the project is to grow the project by allowing other organizations and developers to contribute back to the project under an open source BSD license. We have been successful with gaining support within the open source community already and have contributors and support in not only the government and academic space but also the commercial sector (e.g. Intel). Being both open source and vendor neutral puts Warewulf in a unique position to be supported and utilized as an open standard in High Performance Computing (HPC). Warewulf is an active member of various HPC initiatives and groups and is considered one of the most utilized, extensible and scalable clustering toolkits available.

Kurtzer, GregoryM.↗

Xi-CAM v1.2.3

Xi-CAM aims to provide a community driven platform for multimodal analysis in synchrotron science. The platform core provides a robust plugin infrastructure for extensibility, allowing continuing development to simply add further functionality. Current modules include tools for characterization with (GI)SAXS, Tomography, and XAS. This will continue to serve as a development base as algorithms for multimodal analysis develop. Seamless remote data access, visualization and analysis are key elements of Xi-CAM, and will become critical to synchrotron data infrastructure as expectations for future data volume and acquisition rates rise with continuously increasing throughputs. The highly interactive design elements of Xi-cam will similarly support a generation of users which depend on immediate data quality feedback during high-throughput or burst acquisition modes.

Pandolfi, Ronald↗

HQ-Tracer

HQ-Tracer is a software plugin related to the open-source project Ghidra. It is used for taking an emulation instruction trace from open source platforms and displaying inside of Ghidra. It allows for software debugging, reverse engineering, and enables interactive forward and backward stepping and searching through the execution traces. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2021-4552 O

Wright, ChristopherM.↗