Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “multimodal data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Multimodal Chemical Characterization of Brown Carbon in Atmosphere and Snowpack during SAIL Field Campaign Report

The main goal of this field project was to characterize the chemical and physical properties of light-absorbing particles (LAP) present in the atmosphere and snowpack. The specific field efforts included: 1) online measurement and sampling of light-absorbing particles using a Magee Scientific model AE33 aethalometer (AE33) and a time-resolved aerosol collector (TRAC); and 2) sampling of snowpack during observed events of LAP deposition on snow, confirmed by onsite U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility measurements and meteorological data. The ongoing research objectives of our follow-up sample and data analysis include: 1) Characterization of aerosol regimes and sources, identifying particle-type populations, mixing states, and atmospheric transformations based on correlative analysis of our detailed chemical imaging and chemical characterization measurements and real-time records from the second ARM Mobile Facility (AMF2) and other instruments available from the Surface Atmosphere Integrated Field Laboratory (SAIL) experiment; 2) Assessment of the optical and chemical properties of snowpack deposits to investigate how aerosol deposition influences snowpack lifetime.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

KS4A-Omics1.0_FspDS682

Soil fungi facilitate the translocation of inorganic nutrients from soil minerals to other microorganisms and plants. This ability is particularly advantageous in impoverished soils, because fungal mycelial networks can bridge otherwise spatially disconnected and inaccessible nutrient hotspots. However, the molecular mechanisms underlying fungal mineral weathering and transport through soil remains poorly understood. Here, we addressed this knowledge gap by directly visualizing nutrient acquisition and transport through fungal hyphae in a mineral doped soil micromodel using a multimodal imaging approach. Here, we observed how a representative of common saprotrophic soil fungi, Fusarium sp. DS 682, exhibited a mechanosensory response (thigmotropism) around obstacles and through pore spaces (~12 μm) in the presence of minerals.This study establishes the significance of fungal biology and nutrient translocation mechanisms in maintaining fungal growth under water and nutrient limitations in a soil-like microenvironment, using a high-throughput multi-omic analysis approach. Data package KS4A-Omics.1.0_FspDS682 (Publication: Fungal Mineral Weathering Mechanisms Revealed Through Direct Molecular Visualization) contents reported here are the first version (1.0) and contain pre- and post-processed data using high throughput data capture technologies for multi-omic analysis and integration, this data package contains raw and post-processed experimental data for X-Ray Absorption Near Edge Structure Spectroscopy (XANES/XRF), Optical Microscopy, Proteomics, Scanning Electron Microscope (SEM), Time-of-Flight Secondary Ion Mass Spectroscopy (ToF-SIMS), X-Ray Diffraction Spectroscopy (XRD) files, and X- Ray Photoelectron Spectroscopy (XPS) using EMSL capabilities. This data package DOI contains a comprehensive collection of high-throughput multi-omics data and process metadata catalog. Support files include additional data download contents “Read Me” with dataset descriptor information and data source method application ontologies (see data dictionary section). Reported data download content is structured for compliance with reported guidelines provided by community standard initiatives and publisher stakeholder policies supporting FAIR data principles.

47 OTHER INSTRUMENTATION↗
DOE OSTI2020

In situ multimodal imaging for nanoscale visualization of tribofilm formation

A third of the energy from fuel combustion in passenger car gasoline engines is lost due to friction. Carefully designed engine lubricants can recover some of these losses by reducing friction and wear by forming a nanometer-scale chemico-physico tribofilm between surfaces. Accordingly, attention has focused on developing oil formulations that form low-friction tribofilms. However, analyses of resultant tribofilms are typically conducted after tribo-tests with conventional characterization tools and do not offer insights into tribofilm formation and evolution, precluding information critical to tuning tribofilm properties. In this work, we developed a unique multimodal methodology based on Atomic Force Microscopy (AFM) with local probe heating for in situ tribological studies that activates friction modifiers and simultaneously captures the evolution of friction and surface roughness, with nanometer resolution. As a platform to demonstrate the ability of this methodology to visualize dynamics of tribofilm formation in situ, we apply it to molybdenum-based friction modifiers to distinguish key factors in their functionality and correlate nanoscale AFM and Friction Force Microscopy data to bench tribo-tests used in the industry. To decode the formation mechanisms observed in situ and underlying chemistry of tribofilms, we performed ab initio Molecular Dynamics (AIMD) simulations at comparable conditions. AIMD simulations confirmed both nanoscale and bench tribo-test results and showed deviations in molecular organization in tribofilms that are formed from different molybdenum-based friction modifiers that corroborates with surface functionality. With this innovative methodology, we demonstrate proof-of-principle in situ formation of molybdenum-based tribofilms directly on steel surfaces that could be applied generally to studying tribofilm formation.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Bistable random momentum transfer in a linear on-chip resonator

Optical switches and bifurcation rely on the nonlinear response of materials. Here, we demonstrate linear temporal bifurcation responses in a passive multimode microresonator, with strongly coupled chaotic and whispering gallery modes (WGMs). In microdisks, the chaotic modes exhibit broadband transfer within the deformed cavities, but their transient response is less explored and yields a random output of the analog signal distributed uniformly from “0” to “1.” Here, we build chaotic states by perturbing the multimode microring resonators with densely packed silicon nanocrystals on the waveguide surface. In vivo measurements reveal random and “digitized” output that ONLY populates around 0 and 1 intensity levels. The bus waveguide mode couples first to chaotic modes, then either dissipates or tunnels into stable WGMs. This binary pathway generates high-contrast, digitized outputs. In conclusion, the fully passive device enables real-time conversion of periodic clock signals into binary outputs with contrasts exceeding 12.3 dB, data rates of up to 10 7 · bits per second, and 20 dB dynamic range.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2024

A Deep Multimodal Representation Learning Framework for Accurate Molecular Properties Prediction

Drug discovery is a complex and challenging process, requiring the optimization of candidate compounds to identify those with the potential to become safe and effective drugs. Predicting molecular properties is an indispensable step in the drug discovery pipeline. Traditionally, this process is costly and time-intensive, involving multiple rounds of experiments and clinical trials, rendering it impractical for every candidate compound. Deep learning techniques have emerged as a promising approach to drug discovery to reduce the cost and time required to identify novel drugs. However, prevalent research in deep learning models focused on predicting molecular properties has primarily fixated on single-modal models, which utilize a single modality of data, neglecting the potential benefits of combining different data modalities. To overcome this limitation, we introduce MRL-Mol: a deep \textbf{M}ultimodal \textbf{R}epresentation \textbf{L}earning framework for accurate \textbf{Mol}ecular properties prediction. MRL-Mol harnesses three data modalities: sequence, graph, and image, augmenting the depth of comprehension. Leveraging a large-scale unlabeled dataset~($\sim$1M unique molecules), we pretrain MRL-Mol to extract inter- and intra-modal information. Our study demonstrates the superior performance of MRL-Mol in predicting molecular properties across six benchmark datasets, including both classification and regression tasks. Notably, MRL-Mol outperforms other state-of-the-art molecular properties prediction models. These findings suggest that by combining information from multiple data modalities, MRL-Mol can comprehend molecules better than single-modal deep learning models and identify molecular properties with better accuracy.

Yang, Yuxin↗
DOE OSTI2020

Multimodal imaging and machine learning to enhance microscope images of shale

A machine learning based image processing workflow is presented to enhance shale source rock microscopic images obtained using diverse imaging platforms. Images were acquired from a 30 μm diameter cylindrical Vaca Muerta shale sample using both nondestructive Transmission X-Ray Microscopy (TXM, alternately referred to as nano computed tomography) and destructive Focused Ion Beam-Scanning Electron Microscopy (FIB-SEM). Output cross-sectional images from each modality were aligned using a combination of manual and automated registration techniques to create a registered image dataset. We then apply this dataset for two image processing tasks: prediction of image cross sections with SEM-like resolution from nondestructive TXM data and repair of charged region artifacts (localized accumulation of electrons) within SEM images. The image processing algorithms for both tasks use deep learning models, specifically image-to-image Convolutional Neural Networks (CNNs) and conditional Generative Adversarial Networks (cGANs). In the image enhancement tasks, we are able to achieve significant qualitative and quantitative improvement in TXM images. Here, the best model reaches an average Peak Signal to Noise Ratio (PSNR) of 15.8 dB. Conditioning on TXM data is also shown to reduce artifacts from SEM charging, achieving an average PSNR of 25.8 dB. Furthermore, our results suggest that properly trained and validated networks are capable of significant enhancement of images obtained using nondestructive techniques, thereby improving interpretation of two- and three-dimensional images while preserving samples for future use.

58 GEOSCIENCES↗
DOE OSTI2026

Multimodal Approaches for Leveraging Domain Knowledge with State-of-the-Art Machine Learning to Engineer Biocatalysts

This grant aimed to accelerate the development of specialized enzymes—biological catalysts essential for sustainable manufacturing and medicine—by integrating traditional laboratory evolution with cutting-edge artificial intelligence. To achieve this, we developed a suite of high-throughput sequencing tools and a centralized database to bridge the gap between a protein’s genetic "code" and its physical function. By training machine learning models on large datasets, we also demonstrated the ability to move beyond slow, trial-and-error testing to a "generative" approach, where AI can independently design new, versatile enzymes like tryptophan synthases. Ultimately, these findings demonstrate that combining laboratory data with computer-guided design enables the engineering of highly efficient biological tools with unprecedented speed and precision.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Stationary Bragg reflection of laser light in inhomogeneous absorbing plasmas inside inertial confinement fusion Hohlraums

Laser-produced plasma in inertial confinement fusion (ICF) Hohlraums are marked with density non-uniformity whose length scale can go down to micrometers. This scale is of the order of the laser wavelength. The WKB approximation, which is classically used in radiation-hydrodynamic codes to compute the laser trajectory, cannot correctly take into account such small-scale inhomogeneity of the plasma. Going beyond this approximation, we predict a novel mechanism for the laser reflection. Here we show that an electromagnetic plane wave with wave number k resonates with the k B =2 k Fourier component of a multimode perturbation of the background density and generates a reflected wave. It is the first time that this reflection is considered for stationary inhomogeneous ICF plasmas, and the energy absorption is taken into account. This mechanism, which is a form of Bragg reflection, can occur away from the critical surface and generate a drift of the location of the laser absorption. Furthermore, this absorption will be periodically modulated with a k B wave number. The stationary Bragg reflection can explain ongoing discrepancies between experimental and numerical data about laser trajectory and absorption in ICF Hohlraums.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

Examination of probability distribution of mixture fraction in LES/FDF modelling of a turbulent partially premixed jet flame

An accurate prediction of the probability density function (PDF) of the mixture fraction is crucial to the prediction of combustion since mixing plays an important role in turbulent non-premixed and partially premixed flames. This work provides an assessment of the large-eddy simulation (LES)/filtered density function (FDF) method for the prediction of the PDF of the mixture fraction. The advantage of the LES/FDF method is that it provides the full predictions of the statistical distribution of scalars including the mixture fraction. The predictive accuracy of the method for the PDF is yet to be fully validated. Assessing the prediction of the PDF of the mixture fraction, a conserved scalar, is an important starting point. The Sydney/Sandia inhomogeneous inlet jet flame is used as a test case. A quick comparison shows that the LES/FDF predicted PDF shapes of the mixture fraction deviate significantly from the commonly presumed Beta-PDF as well as from the experimental data in the flame. Here, to examine the source of the discrepancy, we clarify the different PDF definitions used in the comparison among the predictions, measurements, and the presumed shape PDFs. The discrepancy observed from the comparison is largely reconciled by clarifying the difference between the PDFs that are examined. The PDF of the resolved mixture fraction is shown to be close to the Beta-PDF in both the measurements and predictions, while the PDF directly deduced from the LES/FDF particles deviates significantly from the Beta-PDF. A multimodal PDF analysis and a pseudo convergence analysis are conducted to provide plausible evidence to support the predicted multimodal PDF shapes. The sub-filter scale FDF is shown to be close to the Beta-PDF too through the construction of a synthesized PDF, which supports the common presumed Beta-PDF assumption used in the presumed PDF methods when combined with LES.

42 ENGINEERING↗
DOE OSTI2024

Multimodal 3D quantification of particle stimulated nucleation in industrially manufactured aluminium AA5182 sheet

Particle stimulated nucleation is a dominant recrystallisation mechanism observed in many industrially relevant aluminium alloys during thermomechanical processing. Here, in this work, we quantify particle stimulated nucleation in 3D in an aluminium AA5182 alloy sheet cold-rolled to 75% thickness reduction. Second phase particles and nuclei are mapped in the same sample volume by conventional laboratory absorption X-ray tomography and synchrotron X-ray Laue micro-diffraction. The large second phase particles are classified as Fe- and Mg-rich phases. It is found that 84% of the nuclei are particle stimulated and 40% of the particles stimulate nucleation. The critical particle diameter is found to be 4 μm. Deviatoric elastic strains are derived from micro-diffraction data and it is found that elastic strains are present in the recrystallised nuclei. The effects of the different particle types, particle clustering, particle size and aspect ratio as well as strain inheritance are discussed. This work provides a full 3D quantification of particle stimulated nucleation behaviour in AA5182 alloy sheet deformed to high strain.

Deviatoric strain↗
DOE OSTI2026

A unified large language model–based framework for heterogeneous PV image diagnosis

With advances in imaging technologies, modern photovoltaic (PV) systems generate large volumes of heterogeneous image data, including visible, electroluminescence (EL), and infrared (IR) images. Existing PV image analysis models, particularly deep learning approaches, are typically task-specific and lack cross-modality generalization. To address this limitation, this paper proposes an open-source large language model (LLM)–based unified framework for heterogeneous PV image diagnostics. Through task-aware diagnostic prompting, the framework enables analysis of visible, EL, and IR images within a single pipeline, supporting both zero-shot and few-shot inference and binary and multiclass classification. It is compatible with state-of-the-art multimodal LLMs, including ChatGPT, Gemini, Claude, Qwen, and CLIP. The framework is evaluated on PV module condition classification (clean, soiling, snow, hail, and bird droppings) using visible images, cell crack detection using EL images, and hotspot detection using IR images. GPT-5.1 in few-shot mode achieves the best performance, with classification accuracy exceeding 97.3%. Open-source models such as Qwen and CLIP also deliver competitive results on visible images (around 90% accuracy), though their performance is more limited on EL and IR modalities. On the full ELPV dataset, the framework achieves 83.5% zero-shot accuracy, within 2.8% of the supervised CNN baseline, confirming scalability to larger benchmarks. Practical aspects such as reproducibility, response latency, and confidence estimation are systematically analyzed. The framework operates across PV image modalities without modality- or task-specific training, making it well suited as a rapid pre-screening tool to support downstream detailed diagnostics. A benchmark dataset of diverse labeled PV images is also released.

Li, Baojie↗
DOE OSTI2023

Resolution-enhanced X-ray fluorescence microscopy via deep residual networks

Abstract Multimodal hard X-ray scanning probe microscopy has been extensively used to study functional materials providing multiple contrast mechanisms. For instance, combining ptychography with X-ray fluorescence (XRF) microscopy reveals structural and chemical properties simultaneously. While ptychography can achieve diffraction-limited spatial resolution, the resolution of XRF is limited by the X-ray probe size. Here, we develop a machine learning (ML) model to overcome this problem by decoupling the impact of the X-ray probe from the XRF signal. The enhanced spatial resolution was observed for both simulated and experimental XRF data, showing superior performance over the state-of-the-art scanning XRF method with different nano-sized X-ray probes. Enhanced spatial resolutions were also observed for the accompanying XRF tomography reconstructions. Using this probe profile deconvolution with the proposed ML solution to enhance the spatial resolution of XRF microscopy will be broadly applicable across both functional materials and biological imaging with XRF and other related application areas.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Dara: Automated Multiple-Hypothesis Phase Identification and Refinement from Powder X-ray Diffraction

Powder X-ray diffraction (XRD) is a foundational technique for characterizing crystalline materials. However, the reliable interpretation of XRD patterns, particularly in multiphase systems, remains a manual and expertise-demanding task. As a characterization method that only provides structural information, multiple reference phases can often be fit to a single pattern, leading to potential misinterpretation when alternative solutions are overlooked. To ease humans’ efforts and address the challenge, we introduce Dara (data-driven automated Rietveld analysis), a framework designed to automate the robust identification and refinement of multiple phases from powder XRD data. Dara performs an exhaustive tree search over all plausible phase combinations within a given chemical space and validates each hypothesis using the BGMN Rietveld refinement routine. Key features include structural database filtering, automatic clustering of isostructural phases during tree expansion, and peak-matching-based scoring to identify promising phases for refinement. When ambiguity exists, Dara generates multiple hypothesis which can then be decided between by human experts or with further characterization tools. By enhancing the reliability and accuracy of phase identification, Dara enables scalable analysis of realistic complex XRD patterns and provides a foundation for integration into multimodal characterization workflows, moving toward fully self-driving materials discovery.

Biological databases↗
DOE OSTI2025

Edge AI-Enhanced Traffic Monitoring and Anomaly Detection Using Multimodal Large Language Models

This paper addresses the challenge of traffic monitoring and incident detection in remote areas, utilizing multimodal large language models (LLMs) deployed on edge AI devices. The key novelty of the LLM is to convert real-time video streams into descriptive texts, enabling low-bandwidth transmissions and reliable detection of anomalies and incidents in environments of intermittent connectivity. The model is developed based on fine-tuning open-source LLMs and extending it with multi-modal capabilities to analyze video frames. Our work also involves deploying this model on edge devices such as Nvidia IGX Orin and is planned to be tested in realistic environments in future work. The methodology includes data set curation, iterative model fine-tuning and compression, and hardware-based optimization. This approach aims to enhance traffic safety and response speed in remote areas, marking a significant advancement in the application of AI for traffic monitoring and safety management.

Peruski, Ryan [University of Tennessee, Knoxville ↗
DOE OSTI2020

Characterization of groove density variation of VLS gratings with ALS XROL LTP-II in different operation modes

The long trace profiler, LTP-II, available at the Advanced Light Source (ALS) X-ray Optics Laboratory (XROL), was recently upgraded by replacing a multimode diode laser light source with a single-mode, wavelength-stabilized, fibercoupled diode laser system. The upgrade enables us to reliably characterize the lateral variation of groove density of variable-line-spacing (VLS) x-ray diffraction gratings. Here, we discuss the LTP-II performance with an example of measurements with a VLS grating with the groove density at the grating center of 300 lines/mm. For the measurements, we use the LTP-II in two different operation arrangements, the single Gaussian beam and the pencil beam interferometer arrangements. For each operation arrangement, we apply two data processing algorithms: with calculating the centroid position and with determining the position of a characteristic features of the detected beam intensity distributions. Additionally, we discuss the observed strong correlation between the LTP-II modes of operation and the resulted (extracted) groove density variations. We also speculate on possible origin of the correlation.

42 ENGINEERING↗
DOE OSTI2026

SpaceNet 9—Cross-Sensor Alignment of Optical and SAR Imagery

Precise registration of high-resolution synthetic aperture radar (SAR) and optical imagery is necessary for realizing the full potential and benefits of multimodal image analysis. However, two significant challenges presently exist. First, there is a lack of annotated datasets and benchmarks available for high-resolution SAR–optical image registration. Second, an assessment of efficient and reliable image registration methods that can precisely align these modalities is lacking. Here, we present a holistic description of the SpaceNet 9 Challenge and its results. We present a description of the dataset and baseline algorithm along with the results of the challenge, including a description of the winning algorithms. We release the SpaceNet 9 dataset along with open-sourcing the winning algorithms and baseline. The objective of SpaceNet 9 was to compute a dense displacement map that indicates the shift needed to align pixels in an optical image to the pixels in a SAR image. The challenge launched in April 2025 and was active for approximately two months. The top five solutions reduced image alignment error from approximately 34 m to under 13 m for public and private test data, with the best results obtaining a registration error of only 8.5 and 6.7 m on the public testing and private testing dataset, respectively. Usage of pretrained image matching models, robust outlier rejection with RANSAC, and estimating local displacement were common among the top solutions. The results of this challenge provide insight into high-resolution SAR–optical image registration and offer opportunities for future benchmarking in this domain. The baseline algorithm, winning solutions, and datasets are available at https://spacenet.ai/sn9-challenge/.

benchmark datasets↗
DOE OSTI2022

Element differentiation with a Hartmann- based X-ray phase imaging system

Significant efforts are currently ongoing in X-ray imaging to provide multimodal imaging systems, targeting better sensitivity and specificity for both biomedical or non-destructive testing applications. Knowing the elemental composition of specific structures, such as breast microcalcifications in mammography, would help to differentiate malign and benign tumours. Standard X-ray Phase Contrast Imaging techniques provide only qualitative information on elements with similar absorption properties. However, their chemical composition can be determined from the measurement of the phase as it is directly related to the optical index of elemental materials. We will show new experimental results obtained with an X-ray phase imaging system based on a Hartmann mask. Early data treatment succeeded in retrieving both the real and imaginary parts of the refractive index. The system demonstrates the capability to discriminate materials based on elemental composition.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Addition of transient kinetics capabilities to an infrared reflection absorption spectroscopy system through synchronized gas pulsing and data acquisition

Surface science methodologies for understanding thermodynamic aspects of surface processes are at an advanced level. However, instrumentation and approaches for extracting kinetic parameters from elementary steps are far less accessible. In this work, we present an approach combining the use of a fast gas pulsing valve synchronized with data acquisition to enable surface transient kinetics studies using infrared reflection absorption spectroscopy. This methodology applies to the study of reversible processes and borrows concepts and ideas from molecular beam scattering and temporal analysis of products. Here, a temporal resolution of ~67 ms is achieved, and this is illustrated through the study of CO adsorption and desorption on a Pd(111) crystal in the presence and absence of background O 2 . The same approach can be extended to other surface spectroscopies, such as X-ray photoelectron spectroscopy, to obtain spectra with high temporal resolution and signal-to-noise ratio and enable future multimodal surface transient kinetic studies aiming at elucidating reaction mechanisms.

36 MATERIALS SCIENCE↗