Results for “software development framework” — Page 44

DOE OSTI2025

Final report- UFL - RAPIDS2: A SciDAC Institute for Computer Science, Data, and Artificial Intelligence

The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

How close are urban scale building simulations to measured data? Examining bias derived from building metadata in urban building energy modeling

Residential and commercial buildings in the United States accounted for 40% of total energy in 2020. Building energy modeling (BEM) is a useful tool that allows individuals, researchers, companies, or utilities to save energy by optimizing buildings through estimation of building technology savings and performance projection of building energy under various environmental conditions. Urban building energy modeling (UBEM) expands the scope beyond individual buildings to the buildings in a neighborhood, city, utility and more. Yet there is a knowledge gap in the literature as to how these models compare to measured data on an individual and aggregated basis. As UBEM data and methods continue to develop, it is important to consider the accuracy, bias, and limitations of the models. Here, nation-scale data and UBEM software suite named Automatic Building Energy Modeling (AutoBEM) was used to model 50,843 buildings in Chattanooga, Tennessee. The uncalibrated simulation results were compared to aggregated 15-minute electricity data for the year 2019 with visualizations highlighting sources of bias in building data and the AutoBEM framework while considering how they relate to other UBEM methods. Estimation of building type and year of constructions are found to be the major sources of bias. Accounting for the amount of conditioned area per building significantly improves the overall fit of the simulated energy use intensity. it was found that inherent variation in building energy use contributes to R 2 values between 0.008 and 0.095 across building types but slope values near 1 for the total number of buildings. This indicates the need for building aggregation for representative building energy modeling with data sources available at an urban scale while illustrating the need for additional individual building data and model improvement beyond the originally produced UBEM models for individual building analysis.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2022

Strym: A Python Package for Real-time CAN Data Logging, Analysis and Visualization to Work with USB-CAN Interface

In this report, we describe a data analysis tool developed for decoding and analyzing vehicle data obtained from a passenger vehicle’s onboard controller area network (CAN) bus. The tool developed in this paper provides a timeseries framework to perform domain-specific analysis at scale when interpreting data from a vehicle or a collection of vehicles in light of how to design intelligent vehicle applications. The tool, called Strym, exploits the CAN bus mechanism of modern vehicles to capture data using commercially available CAN-to-USB hardware Comma.ai Panda devices, managed through open-source software Libpanda. Strym permits the decoding of vendor-specific CAN messages in a vehicle-agnostic manner. Through this, a researcher can characterize data throughput, assess data quality, and perform analyses. Such analyses are useful in a number of research such as studying human driving behavior in mixed-autonomy, new driver models, rare-event detection, traffic flow estimation, and custom control of vehicles.

Performance evaluation, Smart cities, Intelligent ↗

DOE OSTI2018

Development of Application Function Blocks for Power-Hardware-in-the-Loop Testing of Grid-Connected Inverters: Preprint

Stability and accuracy of the PHIL platform has been a common problem and challenging issue for a long time, thus continuous efforts and research have been devoted to developing interface algorithms to improve upon this. Some of existing works exhibit promising solutions to compensate the errors in the PHIL platform. However, there is a need to harmonize various solutions into a standard framework that provides a general solution for various PHIL test beds and that can be reused with minimum modifications to avoid extra effort. A good way to do this is by using application function blocks (AFBs), which are model-based and can be developed as a standard library component, thus achieving satisfactory reusability and reconfigurability for various platforms. Since PHIL simulation has the flexibility of the software simulation, proper functions can be easily implemented in the digital real-time simulator (DRTS) to preprocess the reference signal before sending it to the power amplifier. Therefore, this paper aims to develop standard AFBs in DRTS for the PHIL interface to test grid-connected inverters. The PHIL interface algorithm is separated into five AFBs that will be presented in detail in this paper.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2014

Evaluation of Visual Analytics Environments: The Road to the Visual Analytics Science and Technology Challenge Evaluation Methodology

The evaluation of visual analytics environments was a topic in Illuminating the Path [Thomas 2005] as a critical aspect of moving research into practice. For a thorough understanding of the utility of the systems available, evaluation not only involves assessing the visualizations, interactions or data processing algorithms themselves, but also the complex processes that a tool is meant to support (such as exploratory data analysis and reasoning, communication through visualization, or collaborative data analysis [Lam 2012; Carpendale 2007]). Researchers and practitioners in the field have long identified many of the challenges faced when planning, conducting, and executing an evaluation of a visualization tool or system [Plaisant 2004]. Evaluation is needed to verify that algorithms and software systems work correctly and that they represent improvements over the current infrastructure. Additionally to effectively transfer new software into a working environment, it is necessary to ensure that the software has utility for the end-users and that the software can be incorporated into the end-user’s infrastructure and work practices. Evaluation test beds require datasets, tasks, metrics and evaluation methodologies. As noted in [Thomas 2005] it is difficult and expensive for any one researcher to setup an evaluation test bed so in many cases evaluation is setup for communities of researchers or for various research projects or programs. Examples of successful community evaluations can be found [Chinchor 1993; Voorhees 2007; FRGC 2012]. As visual analytics environments are intended to facilitate the work of human analysts, one aspect of evaluation needs to focus on the utility of the software to the end-user. This requires representative users, representative tasks, and metrics that measure the utility to the end-user. This is even more difficult as now one aspect of the test methodology is access to representative end-users to participate in the evaluation. In many cases the sensitive nature of data and tasks and difficult access to busy analysts puts even more of a burden on researchers to complete this type of evaluation. User-centered design goes beyond evaluation and starts with the user [Beyer 1997, Shneiderman 2009]. Having some knowledge of the type of data, tasks, and work practices helps researchers and developers know the correct paths to pursue in their work. When access to the end-users is problematic at best and impossible at worst, user-centered design becomes difficult. Researchers are unlikely to go to work on the type of problems faced by inaccessible users. Commercial vendors have difficulties evaluating and improving their products when they cannot observe real users working with their products. In well-established fields such as web site design or office software design, user-interface guidelines have been developed based on the results of empirical studies or the experience of experts. Guidelines can speed up the design process and replace some of the need for observation of actual users [heuristics review references]. In 2006 when the visual analytics community was initially getting organized, no such guidelines existed. Therefore, we were faced with the problem of developing an evaluation framework for the field of visual analytics that would provide representative situations and datasets, representative tasks and utility metrics, and finally a test methodology which would include a surrogate for representative users, increase interest in conducting research in the field, and provide sufficient feedback to the researchers so that they could improve their systems.

evaluation↗

DOE OSTI2013

Autonomous Intelligent Assembly Systems LDRD 105746 (Final Report)

This report documents a three-year to develop technology that enables mobile robots to perform autonomous assembly tasks in unstructured outdoor environments. This is a multi-tier problem that requires an integration of a large number of different software technologies including: command and control, estimation and localization, distributed communications, object recognition, pose estimation, real-time scanning, and scene interpretation. Although ultimately unsuccessful in achieving a target brick stacking task autonomously, numerous important component technologies were nevertheless developed. Such technologies include: a patent-pending polygon snake algorithm for robust feature tracking, a color grid algorithm for uniquely identification and calibration, a command and control framework for abstracting robot commands, a scanning capability that utilizes a compact robot portable scanner, and more. This report describes this project and these developed technologies.

Anderson, Robert J.↗

DOE OSTI2020

Development of a Physics-Based Combustion Model for Engine Knock Prediction

The objective of this project is to improve the prediction of engine knock by developing a new combustion modeling framework. Engine knock is a limiting factor to constrain the increase of fuel efficiency for spark ignition (SI) engines in most passenger cars. Efforts to increase fuel efficiency, increasing the compression ratio or downsizing, lead to the increase in the tendency of the knock occurrence. The knock is an undesired ignition of the end-gas, unburned fuel/air mixture ahead of the spark-ignited premixed flame, resulting in rapid in-cylinder pressure rises and engine damages. The combustion modeling framework developed in this project can consider turbulence-chemistry interactions during end-gas ignition, while using a reasonably detailed chemical mechanism developed for ignition and combustion reactions under engine relevant conditions, and the subtle characteristics of spark-ignited flame propagation. It is developed in the context of large eddy simulation (LES), which can capture stochastic in-cylinder processes. The developed model is incorporated into a commercial software for engine simulation, CONVERGE CFD, as a user defined function, and validated. Engine knock and knock-free experiments as well as direct numerical simulation (DNS) of end-gas ignition in homogeneous turbulence are performed to help model development and provide data sets for model validation. With further validation, the developed model is expected to advance the predictive capability for engine knock simulations and thus contribute to improving the fuel efficiency.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2023

DFTT report for Signal Analysis (2023)

The Detection Framework Testbed and Toolkit (DFTT) is a database and associated java programs intended to facilitate the development and testing of algorithms for operating suites of correlation and subspace detectors. This framework is a generalization of the system described in Harris and Dodge (2011) and Dodge and Harris (2016). DFTT allows retrospective processing of sequences of data using various system configurations. Results are saved in a database, so it is easy to compare the results obtained using different configurations of the system. The principal software components of DFTT are the ConfigCreator, the framework_runner and the Builder program. ConfigCreator assembles the files necessary to define a particular configuration used for processing. The framework_runner operates suites of detectors and saves the results in an Oracle™ database. Builder allows visual examination of templates and detected signals and allows editing and creation of detectors. In addition, there are tools for importing continuous data into the database. DFTT is licensed under the MIT license and has LLNL release number LLNL-CODE-801881. In the current fiscal year DFTT has been used in two different projects. The first (NTL-funded) project investigates the feasibility of screening nuisance detections using correlation detectors. The second (GNEM-funded) project seeks to extend the results of Harris and Dodge (2021) to 3- D sources and multiple stations. Each of these efforts required that additional functionality be added to DFTT. In this report I will summarize the enhancements to DFTT in the context of the relevant research effort.

58 GEOSCIENCES↗

DOE OSTI2024

NuGraph2: A Graph Neural Network for Neutrino Event Reconstruction

Neutrino experiments are set to probe some of the most important open questions in physics, from CP violation and the nature of dark matter. The technology of choice for many of these experiments is the liquid argon time projection chamber (LArTPC). In current LArTPC experiments, reconstruction performance often represents a limiting factor for the sensitivity. New developments are therefore needed to unlock the full potential of LArTPC experiments. NuGraph2 is a state of the art Graph Neural Network for reconstruction of data in LArTPC experiments. NuGraph2 utilizes a heterogeneous graph structure, with separate subgraphs of 2D nodes (hits in each plane) connected across planes via 3D nodes (space points). The model provides a consistent description of the neutrino interaction across all planes. NuGraph2 is a multi-purpose network, with a common message-passing attention engine connected to multiple decoders with different classification or regression tasks. These include the classification of detector hits according to the particle type that produced them (semantic segmentation) and the separation of hits from the neutrino interaction from hits due to noise or cosmic-ray background. Additional decoders are being developed, performing tasks such as the regression of the neutrino interaction vertex position. Performance results will be presented based on publicly available samples from MicroBooNE. These include both physics performance metrics, achieving 95% accuracy for semantic segmentation and 98% classification of neutrino hits, as well as computational metrics for training and for inference on CPU or GPU. The status of the NuGraph integration in the LArSoft software framework will be presented, as well as initial studies about model interpretability and injection of domain knowledge.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2013

Algae Biofuels Co-Location Assessment Tool

ABCLAT was built to help any model user with spatially explicit Nitrogen, Phosphorous, and Carbon Dioxide nutrient flux information, and solar resource information evaluate algal cultivation potential. Initial applications of this modeling framework include Algae Biofuels Co-Location Assessment Tool Canada and Australia. The Canadian application was copyrighted November 29th 2011 as the Algae Biofuels Co-Location Assessment Tool for Canada. This copyright assertion is for the general framework from which any country or region with the requisite data could create a regionally specific application. The ABCLAT model framework developed by SNL looks at the growth potential in a given region as a function of available nutrients from wastewater and other sources, carbon dioxide from power plants, available solar potential, and if available, land cover and use information. The model framework evaluates the biomass potential, fixed carbon dioxide, potential algal biocrude and required land area for nutrient sources. ABCLAT is built with an object-oriented software program that can provide an easy to use interface for exploring questions related to aigal biomass production.

Source record↗

DOE OSTI2020

Development of a Physics-Based Combustion Model for Engine Knock Prediction

The objective of this project is to improve the prediction of engine knock by developing a new combustion modeling framework. Engine knock is a limiting factor to constrain the increase of fuel efficiency for spark ignition (SI) engines in most passenger cars. Efforts to increase fuel efficiency, increasing the compression ratio or downsizing, lead to the increase in the tendency of the knock occurrence. The knock is an undesired ignition of the end-gas, unburned fuel/air mixture ahead of the spark-ignited premixed flame, resulting in rapid in-cylinder pressure rises and engine damages. The combustion modeling framework developed in this project can consider turbulence-chemistry interactions during end-gas ignition, while using a reasonably detailed chemical mechanism developed for ignition and combustion reactions under engine relevant conditions, and the subtle characteristics of spark-ignited flame propagation. It is developed in the context of large eddy simulation (LES), which can capture stochastic in-cylinder processes. The developed model is incorporated into a commercial software for engine simulation, CONVERGE CFD, as a user defined function, and validated. Engine knock and knock-free experiments as well as direct numerical simulation (DNS) of end-gas ignition in homogeneous turbulence are performed to help model development and provide data sets for model validation. With further validation, the developed model is expected to advance the predictive capability for engine knock simulations and thus contribute to improving the fuel efficiency.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2024

NuGraph2: A Graph Neural Network for Neutrino Event Reconstruction

Neutrino experiments are set to probe some of the most important open questions in physics, from CP violation and the nature of dark matter. The technology of choice for many of these experiments is the liquid argon time projection chamber (LArTPC). In current LArTPC experiments, reconstruction performance often represents a limiting factor for the sensitivity. New developments are therefore needed to unlock the full potential of LArTPC experiments. NuGraph2 is a state of the art Graph Neural Network for reconstruction of data in LArTPC experiments. NuGraph2 utilizes a heterogeneous graph structure, with separate subgraphs of 2D nodes (hits in each plane) connected across planes via 3D nodes (space points). The model provides a consistent description of the neutrino interaction across all planes. NuGraph2 is a multi-purpose network, with a common message-passing attention engine connected to multiple decoders with different classification or regression tasks. These include the classification of detector hits according to the particle type that produced them (semantic segmentation) and the separation of hits from the neutrino interaction from hits due to noise or cosmic-ray background. Additional decoders are being developed, performing tasks such as the regression of the neutrino interaction vertex position. Performance results will be presented based on publicly available samples from MicroBooNE. These include both physics performance metrics, achieving 95% accuracy for semantic segmentation and 98% classification of neutrino hits, as well as computational metrics for training and for inference on CPU or GPU. The status of the NuGraph integration in the LArSoft software framework will be presented, as well as initial studies about model interpretability and injection of domain knowledge.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

NuGraph2: A Graph Neural Network for Neutrino Event Reconstruction

Neutrino experiments are set to probe some of the most important open questions in physics, from CP violation and the nature of dark matter. The technology of choice for many of these experiments is the liquid argon time projection chamber (LArTPC). In current LArTPC experiments, reconstruction performance often represents a limiting factor for the sensitivity. New developments are therefore needed to unlock the full potential of LArTPC experiments. NuGraph2 is a state of the art Graph Neural Network for reconstruction of data in LArTPC experiments [https://arxiv.org/abs/2403.11872]. NuGraph2 utilizes a heterogeneous graph structure, with separate subgraphs of 2D nodes (hits in each plane) connected across planes via 3D nodes (space points). The model provides a consistent description of the neutrino interaction across all planes. NuGraph2 is a multi-purpose network, with a common message-passing attention engine connected to multiple decoders with different classification or regression tasks. These include the classification of detector hits according to the particle type that produced them (semantic segmentation) and the separation of hits from the neutrino interaction from hits due to noise or cosmic-ray background. Additional decoders are being developed, performing tasks such as the regression of the neutrino interaction vertex position. Performance results will be presented based on publicly available samples from MicroBooNE. These include both physics performance metrics, achieving 95% accuracy for semantic segmentation and 98% classification of neutrino hits, as well as computational metrics for training and for inference on CPU or GPU. The status of the NuGraph integration in the LArSoft software framework will be presented, as well as initial studies about model interpretability and injection of domain knowledge.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

AEcroscopy: A Software–Hardware Framework Empowering Microscopy Toward Automated and Autonomous Experimentation

Microscopy has been pivotal in improving the understanding of structure-function relationships at the nanoscale and is by now ubiquitous in most characterization labs. However, traditional microscopy operations are still limited largely by a human-centric click-and-go paradigm utilizing vendor-provided software, which limits the scope, utility, efficiency, effectiveness, and at times reproducibility of microscopy experiments. Here, in this work, a coupled software–hardware platform is developed that consists of a software package termed AEcroscopy (short for Automated Experiments in Microscopy), along with a field-programmable-gate-array device with LabView-built customized acquisition scripts, which overcome these limitations and provide the necessary abstractions toward full automation of microscopy platforms. The platform works across multiple vendor devices on scanning probe microscopes and electron microscopes. It enables customized scan trajectories, processing functions that can be triggered locally or remotely on processing servers, user-defined excitation waveforms, standardization of data models, and completely seamless operation through simple Python commands to enable a plethora of microscopy experiments to be performed in a reproducible, automated manner. This platform can be readily coupled with existing machine-learning libraries and simulations, to provide automated decision-making and active theory-experiment optimization to turn microscopes from characterization tools to instruments capable of autonomous model refinement and physics discovery.

47 OTHER INSTRUMENTATION↗

DOE OSTI2024

Creating Continuous Integration Infrastructure for Software Development on U.S. Department of Energy High-Performance Computing Systems

The Exascale Computing Project (ECP) software deployment effort developed and advanced DevOps capabilities. One goal was to enable robust continuous integration (CI) workflows that span the protected high performance computing (HPC) environments found within many of the Department of Energy’s (DOE) national laboratories. This article highlights several challenges encountered with enabling automation, such as charging models for CI jobs, and meeting individualized security requirements that revolve around strongly associating running code with a human identity. Here, it also describes how the Jacamar CI tool evolved to meet latter requirements and became a key aspect of the solutions currently offered. Derived from this experience, we offer a conceptual framework for understanding current and future CI challenges at DOE facilities and offer suggestions for long-term solutions.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

An end-to-end workflow for executing a classically bootstrapped variational quantum algorithm on an academic quantum computer

Academic quantum computing platforms often face unique challenges in executing quantum workloads due to fragmented software environments and limited engineering support. Unlike commercial ecosystems, academic devices typically evolve without full-stack integration in mind, making it difficult to run complex applications—such as variational quantum algorithms (VQA)—reliably and efficiently. Issues such as incompatible software layers and lack of automated job management significantly increase the overhead of theory-experiment collaboration. To address these challenges, we develop a modular, end-to-end workflow that decouples application-layer code from low-level hardware control, automates circuit submission and result collection, and supports fine-grained circuit-level job scheduling and recovery. The architecture employs a dual-end application programming interface (API) design, enabling robust operation across unstable or resource-constrained hardware backends. For practical use, the framework is lightweight and user-friendly, allowing rapid prototyping of full-stack workflows using basic Python tools. We validate this workflow on a high-fidelity trapped-ion quantum computer by demonstrating a variational quantum eigensolver (VQE) experiment with a classically bootstrapped ansatz initialization technique. The system successfully executed over 60,000 circuits across multiple molecular test cases with minimal human intervention, highlighting the framework’s effectiveness in enabling reproducible, resilient quantum experimentation in academic settings.

Clifford↗

DOE OSTI2020

CALPHAD Uncertainty Quantification and TDBX

CALPHAD uncertainty quantification (UQ) is the foundation of materials design with quantified confidence. We report a framework and software packages to enable CALPHAD UQ assessment and calculation using commercial CALPHAD software (Thermo-Calc). This Bayesian inference framework is coupled with a Markov chain Monte Carlo algorithm to establish uncertainty traces with a given thermodynamic database file (TDB) and corresponding experimental data points. This general framework is demonstrated with the Ni–Cr binary system. The algorithm is firstly validated on synthetic data with known ground truth. Then it is applied to real experimental data to generate posterior traces. We develop a file format named TDBX, which provides a single source of truth by combining the original TDB content and the traces for each assessed Gibbs energy parameter. CALPHAD UQ calculations are performed based on the TDBX file, from which uncertainties for phase boundaries, enthalpy curves, and solidification range are collected as examples of basic design parameters. This TDBX file with corresponding scripts are made open-source. Finally, the combination of CALPHAD UQ assessments and calculations connected by TDBX supports uncertainty-assisted modeling, enabling the integrated application of modern design with uncertainty methodologies to computational materials design.

36 MATERIALS SCIENCE↗

DOE OSTI2015

Breaking the silos: The art documentation suite

The art event-processing framework is used by almost all new experiments at Fermilab, and by several outside of Fermilab. All use art as an external product in the same sense that the compiler, ROOT, Geant4, CLHEP and boost are external products. The art team has embarked on a campaign to document art and develop training materials for new users. Many new users of art have little or no knowledge of C++, software engineering, build systems or the many external packages used by art or their experiments, such as ROOT, CLHEP, HEPPDT, and boost. To effectively teach art requires that the training materials include appropriate introductions to these topics as they are encountered. Experience has shown that simply referring readers to the existing native documentation does not work, too often a simple idea that they need to understand is described in a context that presumes prerequisites that are unimportant for a beginning user of art. There is the additional complication that the training materials must be presented in a way that does not presume knowledge of any of the experiments using art. Finally, new users of art arrive at random times throughout the year and the training materials must allow them to start to learn art at any time. This presentation will explain the strategies adopted by the art team to develop a documentation suite that complies with these boundary conditions. It will also show the present status of the documentation suite, including feedback the art team has received from pilot users.

97 MATHEMATICS AND COMPUTING↗