Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “multi-modal learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

43 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Unveiling the Hidden Evolution of Crystal Defects and Disorder in Energy Materials

Control of point defects and disorder in functional thin films and 2D materials is critical to realizing their full potential in applications ranging from energy storage to advanced electronics. However, these phenomena are often poorly understood, difficult to characterize, and challenging to direct with precision. This presentation explores emerging multi-modal computer vision to decipher and predict order in materials across multiple length scales in the electron microscope, from the atomic to the nanoscale. By fusing data from diverse sources, these powerful models provide unprecedented insights into materials' lifecycles, enabling the control of defects and their associated properties at a fundamental level. This capability promises to transform materials design and accelerate the development of next-generation technologies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Multi-modal dynamic radiography using short-pulse laser-generated probe beams

Radiography is an important tool for the interrogation of dynamic experiments in the fields of dynamic properties of materials, and in condensed matter, high explosive, and high-energy-density physics. Multi-modal radiography advances the hypothesis that combining the information delivered by multiple radiographic modalities can lead to more constrained (improved) “reconstruction” of the scene than can be obtained from a single probe. We identify four modalities: multi-probe, time sequence, multi-view, and multi-messenger. Multi-probe radiography is a promising candidate for a next-generation dynamic radiographic facility. High-energy X-rays are the most frequently used probe for dynamic radiography, although recent developments show the utility of proton (pRad), electron (eRad), and neutron probe beams. Because each probing species interacts with material in the radiographic scene through quantitatively different mechanisms, each returns independent information about the scene, which can add extra constraints to the reconstruction process. How to conduct detailed, quantitative “co-analysis” of multiple data streams remains an area of active research. Multi-beam, short-pulse, laser-generated probes offer sufficient dose, an appropriate spectrum, and appropriate spatio-temporal resolution to produce high-quality dynamic radiographs. This paper reports on technology development to advance the state of the art of multi-modal/multi-probe radiography and the pursuit of both deterministic and inferential (AI/ML assisted) co-analysis methodologies to produce more constrained reconstructions from multi-modal data.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

Object detection with deep learning for rare event search in the GADGET II TPC

In the pursuit of identifying rare two-particle events within the GADGET II Time Projection Chamber (TPC), this paper presents a comprehensive approach for leveraging Convolutional Neural Networks (CNNs) and various data processing methods. To address the inherent complexities of 3D TPC track reconstructions, the data is expressed in 2D projections and 1D quantities. This approach capitalizes on the diverse data modalities of the TPC, allowing for the efficient representation of the distinct features of the 3D events, with no loss in topology uniqueness. Additionally, it leverages the computational efficiency of 2D CNNs and benefits from the extensive availability of pre-trained models. Given the scarcity of real training data for the rare events of interest, simulated events are used to train the models to detect real events. To account for potential distribution shifts when predominantly depending on simulations, significant perturbations are embedded within the simulations. This produces a broad parameter space that works to account for potential physics parameter and detector response variations and uncertainties. These parameter-varied simulations are used to train sensitive 2D CNN object detectors. When combined with 1D histogram peak detection algorithms, this multi-modal detection framework is highly adept at identifying rare, two-particle events in data taken during experiment 21072 at the Facility for Rare Isotope Beams (FRIB), demonstrating a 100% recall for events of interest. Here, we present the methods and outcomes of our investigation and discuss the potential future applications of these techniques.

Convolutional neural network↗
DOE OSTI2026

Ripening of Rh Nanoparticle Catalysts in Reverse Water–Gas Shift via a Data-Driven Model Combining Physics, Theory, and Experiment

Degradation via sintering is an ongoing challenge that impedes the broad commercial success of supported metallic nanoparticle catalysts. To mitigate degradation via informed catalyst design and process operations, here we aim to disambiguate the underlying mechanisms of sintering by combining theory and experiment in a quantitative framework. While mechanistic sintering models exist, they only model a single sintering pathway, even though multiple sintering mechanisms can occur simultaneously or dominate at different stages of the process. Data-driven machine learning models have emerged as a means to represent complex processes through data regression. However, machine learning models have very large data needs and lack mechanistic insights due to their black-box encoding. To develop an interpretive model of catalyst degradation via sintering, we constructed a hybrid model combining mechanistic “physics-based” models and data-driven methods to obtain both reliable predictions and mechanistic insights regarding experimentally observed sintering phenomena. Focusing on nanoparticle sintering in the Rh–TiO 2 catalyst for the reverse water–gas shift (RWGS) reaction, the hybrid model couples a mechanistic term for Ostwald ripening with energy values calculated via density functional theory (DFT) with a parametric, data-driven discrepancy function term for unmodeled mechanisms. The hybrid model is trained using Bayesian inference with data collected from small-angle X-ray scattering (SAXS) in situ experiments wherein average nanoparticle diameter versus time was measured at three relevant operating temperatures. The calibrated hybrid model results show that an Ostwald ripening-only model parameterized with fixed DFT energies does not fully capture the time and temperature dependence of the SAXS-observed sintering kinetics, and that an additional functional contribution, or DFT energy calibration, is required to reconcile simulation and experiment. Analysis of the hybrid-model error confirms that the hybrid model outperforms both the purely mechanistic and purely data-driven alternatives in terms of expected predictive accuracy for time-evolving average particle sizes. Furthermore, the results support the hypothesis that the Ostwald ripening mechanism is less important for explaining the sintering phenomena as operating temperature increases under an assumed fixed DFT parameterization. This could be explained in one of two ways: either latent, unmodeled sintering mechanisms dominate at higher temperatures, or the DFT uncertainty increases with temperature. The proposed modeling approach directly links theory to experiments and simulations via a statistical hybrid modeling framework and can be extended to other catalytic systems to improve predictive models and mechanistic understanding.

Bayesian hybrid modeling↗
DOE OSTI2025

Enhancing Automotive Intrusion Detection Through Multi-Modal Fusion: A CAN FD-LiDAR Approach

As vehicles become smarter and more autonomous, they increasingly depend on advanced sensors and communication technologies to operate securely. However, such growing dependence on technology—whether it’s CAN (Controller Area Network) for internal communication or LiDAR (Light Detection and Ranging) for sensing the world around them—also expands the attack surface for the types of cyber attacks. Traditional intrusion detection systems (IDS) typically monitor these systems in isolation, limiting their ability to detect sophisticated, crosssystem attacks. To address this, we propose a multi-modal fusion approach that combines real-world CAN FD signals (from the HCRL dataset) with LiDAR features (from the nuScenes dataset) to enhance attack detection. Our method employs a twostage ensemble approach. Calibrated XGBoost and LightGBM models initially process CAN FD (Fuzzing Data) and LiDAR data independently, detecting timing anomalies and space abnormalities. They are subsequently logarithmically combined with a logistic regression meta-model along with 17 engineered features capturing cross-modal behavior, prediction conflicts, and nonlinear interactions. This approach achieves an AUC of 0.87 and an F1-score of 0.82, surpassing single-modality baselines and early fusion methods, at merely 2 ms inference latency. Compared with deep learning competitors, it is 3 times more efficient, providing a lightweight, interpretable, and real time solution to automotive cybersecurity.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Cell to Ecosystem: Understanding Methane and Associated Nutrient Cycling by Sediment Hosted Syntrophic Consortia and Their Viral Predators

The anaerobic oxidation of methane (AOM) is a significant worldwide microbial process in anoxic lake and ocean sediments, responsible for sequestering up to 80% of this greenhouse gas. Often considered a metabolism on the edge of thermodynamic probability, the impact of ANaerobic MEthane-oxidizing ‘ANME’ archaea on carbon and nutrient cycling in sediment ecosystems is far reaching. They not only serve as a sink for methane coupled to diverse electron acceptors, but also catalyze the transformation of many important nutrients including nitrogen, phosphate, and iron. While information about the potential mechanisms supporting metabolism in AOM is now available, remarkably little has been learned about their nutritional requirements, their dependencies on bacterial partners, and consequentially their ultimate impact on nutrient transformation and bioavailability within sedimentary ecosystems, and beyond. Further, the role of viruses within sediment ecosystems represents an essential but vastly understudied aspect of the transformation of carbon and nutrients by AOM. Viruses are now widely appreciated as central players in biogeochemical cycles across diverse ecosystems. These nanoscale predators have been shown to enhance the turnover of essential nutrients, thus stimulating microbial growth and environmental viruses themselves may constitute an important reservoir of nitrogen and phosphorous. Little is known about the role of viruses in methane-impacted sedimentary ecosystems, however prior genomic and microscopy evidence suggests that AOM consortia are susceptible to phage infection. The overarching scientific goal of this multi-disciplinary research proposal is to build on these recent discoveries and expand our understanding of interactions and fundamental activities involved in cycling of carbon and nutrients by syntrophic methanotrophic archaeal-bacterial consortia and associated viruses in anoxic sedimentary environments. Our three specific objectives are to 1) Quantify energy and nutrient exchange (e.g. N, P, Fe and vitamins) within AOM consortia and between ANME-bacterial partners; 2) Identify virus-host interactions associated with AOM and assess C and N transfer through viruses in methane-impacted sediment ecosystems; 3) Model energy and nutrient exchange in AOM consortia and viral-host interactions (i.e. viral activity), and their environmental distribution patterns. Our experimental emphasis cuts across scales that are important for understanding microbial and viral interactions and activities within their habitats, as well as community wide biogeochemical transformations. These research goals will be accomplished through the application of novel molecular techniques targeting DNA, RNA, proteins, and metabolites combined with a unique multi-modal analytical imaging pipeline. We will then model the ecophysiological capabilities of diverse sediment-hosted methanotrophic consortia to develop a more comprehensive understanding of the energetic and nutritional interactions between different AOM partner couplings that occur in sediments.

03 NATURAL GAS↗
DOE OSTI2024

Hiding-in-Plain-Sight (HiPS) Attack on CLIP for Targetted Object Removal from Images

Machine learning models are known to be vulnerable to adversarial attacks, but prior works have mostly focused on single-modalities. With the rise of large multi-modal models (LMMs) like CLIP, which combine vision and language capabilities, new vulnerabilities have emerged. However, these multimodal targeted attacks aim to completely change the model's output to what the adversary wants. In many realistic scenarios, an adversary might seek to make only subtle modifications to the output, so that the changes go unnoticed by downstream models or even by humans. We introduce Hiding-in-Plain-Sight (HiPS) attacks, a novel class of adversarial attacks that subtly modifies model predictions by selectively concealing target object(s), as if the target object was absent from the scene. We propose two HiPS attack variants, HiPS-cls and HiPS-cap, and demonstrate their effectiveness in transferring to downstream image captioning models, such as CLIP-Cap, for targeted object removal from image captions.

Daw, Arka [ORNL] (ORCID:0009000633191271)↗