Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “multimodal data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Generalist multimodal AI: A review of architectures, challenges and opportunities

Multimodal models are expected to be a critical component to future advances in artificial intelligence. Here, this field is starting to grow rapidly with a surge of new design elements motivated by the success of foundation models in natural language processing (NLP) and vision. It is widely hoped that further extending the foundation models to multiple modalities (e.g., text, image, video, sensor, time series, graph, etc.) will ultimately lead to generalist multimodal models, i.e. one model across different data modalities and tasks. However, there is little research that systematically analyzes recent multimodal models (particularly the ones that work beyond text and vision) with respect to the underling architecture proposed. Therefore, this work provides a fresh perspective on generalist multimodal models (GMMs) via a novel architecture and training configuration specific taxonomy. This includes factors such as Unifiability, Modularity, and Adaptability that are pertinent and essential to the wide adoption and application of GMMs. The review further highlights key challenges and prospects for the field and guide the researchers into the new advancements.

Artificial intelligence (AI)↗
DOE OSTI2024

The POINTER Imaging baseline cohort: Associations between multimodal neuroimaging biomarkers, cardiovascular health, and cognition

Abstract INTRODUCTION The U.S. Study to Protect Brain Health Through Lifestyle Intervention to Reduce Risk (U.S. POINTER) is evaluating lifestyle interventions in older adults at risk for cognitive decline and dementia. Here we characterize the baseline data set of the POINTER Imaging ancillary study. METHODS Participants underwent health and cognitive assessments and neuroimaging with multimodal positron emission tomography (PET) (beta‐amyloid [Aβ] and tau) and magnetic resonance imaging (MRI). Framingham risk score (FRS) was used to quantify cardiovascular disease (CVD) risk. RESULTS A total of 1052 participants (31% from underrepresented ethnoracial groups) were enrolled. Compared to Aβ−, Aβ+ (29%) participants were older, had higher apolipoprotein E (APOE) ε4 carriage rate and white matter hyperintensity volume, and greater temporal tau. FRS was related to MRI measures, but not AD biomarkers. FRS and tau had independent effects on cognition. DISCUSSION In this heterogenous, at‐risk cohort, CVD risk was related to more abnormal brain structure and poorer cognition, representing a putative non‐AD (Alzheimer's disease) pathway to brain injury and cognitive decline. Highlights The U.S. Study to Protect Brain Health Through Lifestyle Intervention to Reduce Risk (U.S. POINTER) cohort is enriched for cardiovascular disease (CVD) and poor lifestyle POINTER Imaging collected multimodal neuroimaging data in this unique, at‐risk cohort Amyloid burden was related to age, apolipoprotein E (APOE) ε4 carriage, and measures of disease progression Associations between amyloid and tau, and tau and cognition, were relatively weak CVD risk and tau pathology were independently related to memory

Neurosciences & Neurology↗
DOE OSTI2024

From Data to Insights: A Covariate Analysis of the IARPA BRIAR Dataset for Multimodal Biometric Recognition Algorithms at Altitude and Range

This paper examines covariate effects on fused whole body biometrics performance in the IARPA BRIAR dataset, specifically focusing on UAV platforms, elevated positions, and distances up to 1000 meters. The dataset includes outdoor videos compared with indoor images and controlled gait recordings. Normalized raw fusion scores relate directly to predicted false accept rates (FAR), offering an intuitive means for interpreting model results. A linear model is developed to predict biometric algorithm scores, analyzing their performance to identify the most influential covariates on accuracy at altitude and range. Weather factors like temperature, wind speed, solar loading, and turbulence are also investigated in this analysis. The study found that resolution and camera distance best predicted accuracy and findings can guide future research and development efforts in long-range/elevated/UAV biometrics and support the creation of more reliable and robust systems for national security and other critical domains.

Bolme, David↗
DOE OSTI2025

Unsupervised anomaly clustering via offset alignment in multivariate grid sensing data

Modern industries increasingly rely on multi-sensor technologies to acquire complex, high-dimensional data streams, enabling advanced monitoring and control systems. One critical application is online anomaly detection in electrical smart grids, where multivariate and multimodal sensing technologies play a vital role. However, detecting anomalies in such time-series data is challenging due to their inherent temporal dependencies and stochastic behavior. Traditional approaches based on supervised and semi-supervised learning methods depend on labeled datasets, which are often unavailable in real-world scenarios. While unsupervised methods have emerged as promising alternatives, these methods are highly susceptible to noise and outliers commonly present in sensing applications. Furthermore, deep learning-based anomaly detection methods, despite their performance, are often criticized for their black-box nature, limiting their applicability in safety-critical and online environments where interpretability and explainability are paramount. In this work, we propose an unsupervised anomaly clustering method leveraging a cyclic alignment-based offset detection algorithm for multivariate time-series signals. The proposed method is applied to multivariate data collected from vibrational, voltage, and magnetic field sensors deployed in a local grid substation. Our results demonstrate the robustness of the algorithm in accurately clustering various anomalies/events across different sensing modalities. Additionally, we compare the effectiveness of the proposed approach against a simple pattern-based anomaly detection method, which performs well for univariate data but fails to generalize to multivariate and multimodal time-series data.

Mukherjee, Subrata [ORNL] (ORCID:0000000309930338)↗
DOE OSTI2023

Sputter-Deposited Mo Thin Films: Multimodal Characterization of Structure, Surface Morphology, Density, Residual Stress, Electrical Resistivity, and Mechanical Response

Multimodal datasets of materials are rich sources of information which can be leveraged for expedited discovery of process–structure–property relationships and for designing materials with targeted structures and/or properties. For this data descriptor article, we provide a multimodal dataset of magnetron sputter-deposited molybdenum (Mo) thin films, which are used in a variety of industries including high temperature coatings, photovoltaics, and microelectronics. In this dataset we explored a process space consisting of 27 unique combinations of sputter power and Ar deposition pressure. Here, the phase, structure, surface morphology, and composition of the Mo thin films were characterized by x-ray diffraction, scanning electron microscopy, atomic force microscopy, and Rutherford backscattering spectrometry. Physical properties—namely, thickness, film stress and sheet resistance—were also measured to provide additional film characteristics and behaviors. Additionally, nanoindentation was utilized to obtain mechanical load-displacement data. The entire dataset consists of 2072 measurements including scalar values (e.g., film stress values), 2D linescans (e.g., x-ray diffractograms), and 3D imagery (e.g., atomic force microscopy images). An additional 1889 quantities, including film hardness, modulus, electrical resistivity, density, and surface roughness, were derived from the experimental datasets using traditional methods. Minimal analysis and discussion of the results are provided in this data descriptor article to limit the authors’ preconceived interpretations of the data. Overall, the data modalities are consistent with previous reports of refractory metal thin films, ensuring that a high-quality dataset was generated. The entirety of this data is committed to a public repository in the Materials Data Facility.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Heterogeneous Multi-Domain Dataset Synthesis to Facilitate Privacy and Risk Assessments in Smart City IoT

The emergence of the Smart Cities paradigm and the rapid expansion and integration of Internet of Things (IoT) technologies within this context have created unprecedented opportunities for high-resolution behavioral analytics, urban optimization, and context-aware services. However, this same proliferation intensifies privacy risks, particularly those arising from cross-modal data linkage across heterogeneous sensing platforms. To address these challenges, this paper introduces a comprehensive, statistically grounded framework for generating synthetic, multimodal IoT datasets tailored to Smart City research. The framework produces behaviorally plausible synthetic data suitable for preliminary privacy risk assessment and as a benchmark for future re-identification studies, as well as for evaluating algorithms in mobility modeling, urban informatics, and privacy-enhancing technologies. As part of our approach, we formalize probabilistic methods for synthesizing three heterogeneous and operationally relevant data streams—cellular mobility traces, payment terminal transaction logs, and Smart Retail nutrition records—capturing the behaviors of a large number of synthetically generated urban residents over a 12-week period. The framework integrates spatially explicit merchant selection using K-Dimensional (KD)-tree nearest-neighbor algorithms, temporally correlated anchor-based mobility simulation reflective of daily urban rhythms, and dietary-constraint filtering to preserve ecological validity in consumption patterns. In total, the system generates approximately 116 million mobility pings, 5.4 million transactions, and 1.9 million itemized purchases, yielding a reproducible benchmark for evaluating multimodal analytics, privacy-preserving computation, and secure IoT data-sharing protocols. To show the validity of this dataset, the underlying distributions of these residents were successfully validated against reported distributions in published research. We present preliminary uniqueness and cross-modal linkage indicators; comprehensive re-identification benchmarking against specific attack algorithms is planned as future work. This framework can be easily adapted to various scenarios of interest in Smart Cities and other IoT applications. By aligning methodological rigor with the operational needs of Smart City ecosystems, this work fills critical gaps in synthetic data generation for privacy-sensitive domains, including intelligent transportation systems, urban health informatics, and next-generation digital commerce infrastructures.

IoT↗
DOE OSTI2025

Foundation Models for Zero-Shot Segmentation of Scientific Images without AI-Ready Data

Zero-shot and prompt-based models have excelled at visual reasoning tasks by leveraging large-scale natural image corpora, but they often fail on sparse and domain-specific scientific image data. We introduce Zenesis, a no-code interactive computer vision platform designed to reduce data readiness bottlenecks in scientific imaging workflows. Zenesis integrates lightweight multimodal adaptation for zero-shot inference on raw scientific data, human-in-the-loop refinement, and heuristic-based temporal enhancement. We validate our approach on Focused Ion Beam Scanning Electron Microscopy (FIB-SEM) datasets of catalyst-loaded membranes. Zenesis outperforms baselines, achieving an average accuracy of 0.947, Intersection over Union (IoU) of 0.858, and Dice score of 0.923 on amorphous catalyst samples; and 0.987 accuracy, 0.857 IoU, and 0.923 Dice on crystalline samples. These results represent a significant performance gain over conventional methods such as Otsu thresholding and standalone models like the Segment Anything Model (SAM). Zenesis enables effective image segmentation in domains where annotated datasets are limited, offering a scalable solution for scientific discovery.

Mukherjee, Shubhabrata↗
DOE OSTI2022

pnnl/Multimodal-Few-Shot

This project combines few-shot machine learning with a multimodal framework for electron microscope data quantification.

Doty, Christina↗
NASA NTRS1991

Magellan imaging radar mission to Venus

The Magellan imaging-radar mapping mission has collected and processed data from the spacecraft in an elliptical orbit around Venus. A brief description is given of the mission and the spacecraft, followed by a more detailed description of the radar system design, which used earth-orbiting SAR experience and several innovations in its design to operate from an orbit around another planet. The radar sensor, ground processing, and data products are described. This multimode radar is the only science instrument on the mission and has the objective of mapping at least 70 percent of the planet surface. It has three modes: SAR, altimetry, and passive radiometry. The radar system has produced maps of almost all of the Venusian surface with a resolution better than 600-m equivalent optical line pair, and the best resolution obtained is equivalent to less than 300 m. Some of the early radar images are shown.

Johnson, William T. K.↗
DOE OSTI2021

Heterogeneous multiphase flow properties of volcanic rocks and implications for noble gas transport from underground nuclear explosions

Of interest to the Underground Nuclear Explosion Signatures Experiment are patterns and timing of explosion-generated noble gases that reach the land surface. The impact of potentially simultaneous flow of water and gas on noble gas transport in heterogeneous fractured rock is a current scientific knowledge gap. This article presents field and laboratory data to constrain and justify a triple continua conceptual model with multimodal multiphase fluid flow constitutive equations that represents host rock matrix, natural fractures, and induced fractures from past underground nuclear explosions (UNEs) at Aqueduct and Pahute Mesas, Nevada National Security Site, Nevada, USA. Capillary pressure from mercury intrusion and direct air–water measurements on volcanic tuff core samples exhibit extreme spatial heterogeneity (i.e., variation over multiple orders of magnitude). Petrographic observations indicate that heterogeneity derives from multimodal pore structures in ash-flow tuff components and post-depositional alteration processes. Comparisons of pre- and post-UNE samples reveal different pore size distributions that are due in part to microfractures. Capillary pressure relationships require a multimodal van Genuchten (VG) constitutive model to best fit the data. Relative permeability estimations based on unimodal VG fits to capillary pressure can be different from those based on bimodal VG fits, implying the choice of unimodal vs. bimodal fits may greatly affect flow and transport predictions of noble gas signatures. The range in measured capillary pressure and predicted relative permeability curves for a given lithology and between lithologies highlights the need for future modeling to consider spatially distributed properties.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
NASA NTRSDate not supplied

AI Foundation Models for Science: An Open Collaborative Initiative

Foundation Models (FMs), AI models designed to replace task-specific models, are increasingly being recognized for their versatility across numerous downstream applications. These models, trained using self-supervised techniques on any type of sequence data, circumvent the need for large annotated datasets, a major bottleneck in traditional AI model development. FMs can be applied to downstream tasks using few-shot learning and fine-tuning, significantly reducing the need for large labeled training datasets and computational resources. However, the development of FMs requires substantial resources, including access to data and compute power, expertise in the latest models, and specialized scientific knowledge for systematic evaluation. It is challenging for a single group to possess all these capabilities. To address this, NASA IMPACT has initiated an open collaborative effort, leveraging partnerships with the private sector and other groups within and outside NASA, to jointly build FMs. The overarching goal is to develop a consistent and collaborative approach to building FMs for high-value science datasets. This initiative has fostered collaboration within NASA and with external partners, including IBM Research, Clark University, DOE’s ORNL, ESA, and USGS. The effort focuses on identifying key datasets with a wide range of downstream applications, pretraining and building FMs using modified transformer architectures, evaluating compute infrastructure needs, and sharing models, pretraining and fine-tuning code, and data with the community. Furthermore, it aims to train the Earth science community to fine-tune these models for various downstream applications. Our initial effort resulted in the creation of a 100 million parameter HLS Geospatial Model within six months, which was released on HuggingFace. We are now expanding our scope to include data from weather and climate models and investigating multimodal models. We invite those interested in participating in this effort to join us by sharing their use cases, expertise, or data.

Rahul Ramachandran↗
DOE OSTI2026

Revealing Local Structures through Machine-Learning-Fused Multimodal Spectroscopy

Atomistic structures of materials offer valuable insights into their functionality. Determining these structures remains a fundamental challenge in materials science, especially for systems with defects. While both experimental and computational methods exist, each has limitations in resolving nanoscale structures. Core-level spectroscopies, such as X-ray absorption (XAS) or electron energy-loss spectroscopies (EELS), have been used to determine the local bonding environment and structure of materials. Recently, machine learning (ML) methods have been applied to extract structural and bonding information from XAS/EELS data. However, frameworks relying solely on a single data stream, defined as characterization data derived from a single element using one technique, are often insufficient because multiple local environments can yield similar spectral features, making it challenging to differentiate between competing structural hypotheses. Here, in this work, we address this challenge by integrating multimodal ab initio simulations, experimental data acquisition, and ML techniques for structure characterization. Our goal is to determine local structures and properties using EELS and XAS data from multiple elements and edges. To showcase our approach, we use various lithium nickel manganese cobalt (NMC) oxide compounds which are used for lithium ion batteries, including those with oxygen vacancies and antisite defects, as the sample material system. We successfully inferred local element content, ranging from lithium to transition metals, with quantitative agreement with experimental data. Beyond local element inference, we find that ML model based on multimodal spectroscopic data is able to determine whether local defects such as oxygen vacancy and antisites are present, a task which is impossible for single mode spectra or other experimental techniques. Furthermore, our framework is able to provide physical interpretability, bridging spectroscopy with the local atomic and electronic structures.

battery↗
NASA NTRS1991

THz dichroic plates for use at high angles of incidence

The design of a high frequency dichroic plate consisting of an electrically thick self supporting metallic mesh that can be used in a linearly polarized quasi-optical system at high angles of incidence is described. The measured and computed performance of a 2.5-inch aperture mesh is given. This mesh has a 3-dB cutoff frequency of 875 GHz and less than 0.75 dB of transmission loss from 950-1350 GHz at incidence angles of 0, 30, and 45 degrees. The results of a multimode waveguide analysis corroborate the measured data.

Siegel, Peter H.↗
DOE OSTI2022

Development of Multimodal Few-Shot Analytics for Electron Micrographs

Recent advances in materials data analytics have provided new avenues for determining process-structure-property (PSP) linkages in a variety of materials. Machine learning techniques including few-shot learning have increased the efficiency of classifying microscopy images for the purposes of material characterization. Attempts at creating a multimodal approach can provide further improvements to current models and help extract more salient features from data. In this vein, raw spectrum data was taken to provide an additional modality to our current pyCHIP classifier. Modifications in segmentation also show potential in improving the accuracy of the pyCHIP classifier. Classifier output was analyzed using network graphs and unsupervised clustering algorithms such as spectral clustering to detect better segmentation methods than the current “chipping” approach. We suggest that the chip selection process can be automated in the future using a combination of these techniques to enable high-throughput analyses.

36 MATERIALS SCIENCE↗
NASA NTRS2005

A least-squares parameter estimation algorithm for switched hammerstein systems with applications to the VOR

A "Multimode" or "switched" system is one that switches between various modes of operation. When a switch occurs from one mode to another, a discontinuity may result followed by a smooth evolution under the new regime. Characterizing the switching behavior of these systems is not well understood and, therefore, identification of multimode systems typically requires a preprocessing step to classify the observed data according to a mode of operation. A further consequence of the switched nature of these systems is that data available for parameter estimation of any subsystem may be inadequate. As such, identification and parameter estimation of multimode systems remains an unresolved problem. In this paper, we 1) show that the NARMAX model structure can be used to describe the impulsive-smooth behavior of switched systems, 2) propose a modified extended least squares (MELS) algorithm to estimate the coefficients of such models, and 3) demonstrate its applicability to simulated and real data from the Vestibulo-Ocular Reflex (VOR). The approach will also allow the identification of other nonlinear bio-systems, suspected of containing "hard" nonlinearities.

Evaluation Studies↗
DOE OSTI2024

ATAT: Astronomical Transformer for time series and Tabular data

Context. The advent of next-generation survey instruments, such as theVera C. RubinObservatory and its Legacy Survey of Space and Time (LSST), is opening a window for new research in time-domain astronomy. The Extended LSST Astronomical Time-Series Classification Challenge (ELAsTiCC) was created to test the capacity of brokers to deal with a simulated LSST stream. Aims. Our aim is to develop a next-generation model for the classification of variable astronomical objects. We describe ATAT, the Astronomical Transformer for time series And Tabular data, a classification model conceived by the ALeRCE alert broker to classify light curves from next-generation alert streams. ATAT was tested in production during the first round of the ELAsTiCC campaigns. Methods. ATAT consists of two transformer models that encode light curves and features using novel time modulation and quantile feature tokenizer mechanisms, respectively. ATAT was trained on different combinations of light curves, metadata, and features calculated over the light curves. We compare ATAT against the current ALeRCE classifier, a balanced hierarchical random forest (BHRF) trained on human-engineered features derived from light curves and metadata. Results. When trained on light curves and metadata, ATAT achieves a macro F1 score of 82.9 ± 0.4 in 20 classes, outperforming the BHRF model trained on 429 features, which achieves a macro F1 score of 79.4 ± 0.1. Conclusions. The use of transformer multimodal architectures, combining light curves and tabular data, opens new possibilities for classifying alerts from a new generation of large etendue telescopes, such as theVera C. RubinObservatory, in real-world brokering scenarios.

Astronomy & Astrophysics↗
DOE OSTI2023

Applying Multimodal Mass Spectrometry to Image Tumors Undergoing Ferroptosis Following In Vivo Treatment with a Ferroptosis Inducer

Epithelial ovarian cancer (EOC) is the most common form of ovarian cancer. The poor prognosis generally associated with this disease has led to the search for improved therapies such as ferroptosis-inducing agents. Ferroptosis is a form of regulated cell death that is dependent on iron and is characterized by lipid peroxidation. Precise mapping of lipids and iron within tumors exposed to ferroptosis-inducing agents may provide insight into processes of ferroptosis in vivo and ultimately assist in the optimal deployment of ferroptosis inducers in cancer therapy. In this work, we present a method for combining matrix-assisted laser desorption/ionization (MALDI) mass spectrometry imaging (MSI) with secondary ion mass spectrometry (SIMS) to analyze changes in spatial lipidomics and metal composition, respectively, in ovarian tumors following exposure to a ferroptosis inducer. Tumors were obtained by injecting human ovarian cancer tumor-initiating cells into mice, followed by treatment with the ferroptosis inducer erastin. SIMS imaging detected iron accumulation in the tumor tissue, and sequential MALDI-MS imaging of the same tissue section displayed two chemically distinct regions of lipids. Additionally, one region was associated with the iron-rich area detected with SIMS, and the other region encompassed the remainder of the tissue section. Bulk lipidomics confirmed the lipid assignments putatively assigned from the MALDI-MS data. Overall, we demonstrate the ability of multimodal MSI to identify the spatial locations of iron and lipids in the same tissue section and associate these regions with clinical pathology.

60 APPLIED LIFE SCIENCES↗