Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “multimodal network”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Accurate, Uncertainty-Aware Classification of Molecular Chemical Motifs from Multimodal X-ray Absorption Spectroscopy

Accurate classification of molecular chemical motifs from experimental measurement is an important problem in molecular physics, chemistry, and biology. In this work, we present neural network ensemble classifiers for predicting the presence (or lack thereof) of 41 different chemical motifs on small molecules from simulated C, N, and O K-edge X-ray absorption near-edge structure (XANES) spectra. Our classifiers not only achieve class-balanced accuracies of more than 0.95 but also accurately quantify uncertainty. Here, we also show that including multiple XANES modalities improves predictions notably on average, demonstrating a “multimodal advantage” over any single modality. In addition to structure refinement, our approach can be generalized to broad applications with molecular design pipelines.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Ultrafast CMOS image sensors and data-enabled super-resolution for multimodal radiographic imaging and tomography

We summarize recent progress in ultrafast Complementary Metal Oxide Semiconductor (CMOS) image sensor development and the application of neural networks for post-processing of CMOS and charge-coupled device (CCD) image data to achieve sub-pixel resolution (thus ‘super-resolution’). The combination of novel CMOS pixel designs and data-enabled image post-processing provides a promising path towards ultrafast high-resolution multi-modal radiographic imaging and tomography applications.

47 OTHER INSTRUMENTATION↗
NASA NTRS1976

Multimode and multistate ladder oscillator and frequency recognition device

A ladder oscillator composed of capacitive and inductive impedances connected together to form a ladder network which has a chosen number N oscillation modes at N different frequencies. Each oscillation mode is characterized by a unique standing wave voltage pattern along the nodes of the ladder oscillator, with the mode in which the ladder oscillator is oscillating being determinable from the amplitudes or phase of the oscillations at the nodes. A logic circuit may be connected to the nodes of the oscillator to compare the phases of selected nodes and thereby determine which mode the oscillator is oscillating in. A ladder oscillator composed of passive capacitive and inductive impedances can be utilized as a frequency recognition device, since the passive ladder oscillator will display the characteristic standing wave patterns if an input signal impressed upon the ladder oscillator is close to one of the mode frequencies of the oscillator. A CL ladder oscillator having series capacitive impedances and shunt inductive impedances can exhibit sustained and autonomous oscillations if active nonlinear devices are connected in parallel with the shunt inductive impedances. The active CL ladder oscillator can be synchronized to input frequencies impressed upon the oscillator, and will continue to oscillate after the input signal has been removed at a mode frequency which is, in general, nearest to the input signal frequency. Autonomous oscillations may also be obtained as desired from the active CL ladder oscillator at the mode frequencies.

Aumann, Herbert M.↗
DOE OSTI2023

Ultrafast CMOS image sensors and data-enabled super-resolution for multimodal radiographic imaging and tomography

We summarize recent progress in ultrafast Complementary Metal Oxide Semiconductor (CMOS) image sensor development and the application of neural networks for post-processing of CMOS and charge-coupled device (CCD) image data to achieve sub-pixel resolution (thus $super$-$resolution$). The combination of novel CMOS pixel designs and data-enabled image post-processing provides a promising path towards ultrafast high-resolution multi-modal radiographic imaging and tomography applications.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

A guide to the BRAIN Initiative Cell Census Network data ecosystem

Characterizing cellular diversity at different levels of biological organization and across data modalities is a prerequisite to understanding the function of cell types in the brain. Classification of neurons is also essential to manipulate cell types in controlled ways and to understand their variation and vulnerability in brain disorders. The BRAIN Initiative Cell Census Network (BICCN) is an integrated network of data-generating centers, data archives, and data standards developers, with the goal of systematic multimodal brain cell type profiling and characterization. Emphasis of the BICCN is on the whole mouse brain with demonstration of prototype feasibility for human and nonhuman primate (NHP) brains. Here, we provide a guide to the cellular and spatial approaches employed by the BICCN, and to accessing and using these data and extensive resources, including the BRAIN Cell Data Center (BCDC), which serves to manage and integrate data across the ecosystem. We illustrate the power of the BICCN data ecosystem through vignettes highlighting several BICCN analysis and visualization tools. Finally, we present emerging standards that have been developed or adopted toward Findable, Accessible, Interoperable, and Reusable (FAIR) neuroscience. The combined BICCN ecosystem provides a comprehensive resource for the exploration and analysis of cell types in the brain.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

Advancing Industry 4.0: Multimodal Sensor Fusion for AI-Based Fault Detection in 3D Printing

Additive manufacturing, particularly fused deposition modeling, is transforming modern production by enabling rapid prototyping and complex part fabrication. However, its layer-by-layer process remains vulnerable to faults such as nozzle clogging, filament runout, and layer misalignment, which compromise print quality and reliability. Traditional inspection methods are costly, time-intensive, and often limited to post-process analysis, making them unsuitable for real-time intervention. In this current study, the authors developed a novel, low-cost, and portable faultdetection system that leverages multimodal sensor fusion and artificial intelligence for real-time monitoring in FDM-based 3D printing. The system integrates acoustic, vibration, and thermal sensing into a non-intrusive architecture, capturing complementary data streams that reflect both mechanical and process-related anomalies. Acoustic and thermal sensors operate in a fully contactless manner, while the vibration sensor requires minimal attachment such that it will not interfere with printer hardware, thereby preserving portability and ease of deployment. The multimodal signals are processed into spectrograms and time-frequency features, which are classified using convolutional neural networks for intelligent fault detection. The proposed system advances Industry 4.0 objectives by offering an affordable, scalable, and practical monitoring solution that improves faultdetection accuracy, reduces waste, and supports sustainable, adaptive manufacturing.

42 ENGINEERING↗
DOE OSTI2023

In planta single-molecule imaging and holographic force spectroscopy to study real-time, multimodal turnover dynamics of polysaccharides and associated carbohydrate metabolites

Cellular production of polysaccharides from simple sugar molecules serves critical roles in a variety of metabolic processes essential to the survival of every living organism. In the context of photosynthetically fixed carbon, cell wall polysaccharides synthesis is particularly of interest for increased biomass accumulation for human food, animal feed, and bioenergy related applications. Despite a long history of research in plant cell biology, our understanding of in planta cell wall biosynthesis and its regulation is far from complete due to lack of a comprehensive microscopy toolkit that encompass the multiple spatial and temporal scales in which cell wall polysaccharides fibrils are synthesized and assembled into intricate cell wall networks. In this project, a multidisciplinary team of scientists from Rutgers University, Vanderbilt University, and Oak Ridge National Laboratory carried out innovative multimodal, multiscale microscopy studies of cell wall synthesis using single-molecule force-spectroscopy, super-resolution fluorescence microscopy, and in vivo live cell imaging. Our novel multimodal and holistic imaging approach revealed plant cell wall polysaccharides synthesis processes in unprecedented detail across multiple spatiotemporal scales, from a single-molecule to a single-cell. Besides, the team has made technical and scientific innovations across multiple research fields—microscopy, bioengineering, single-molecule biophysics, and plant biology, etc.—to accomplish the goals. The results from this project will greatly advance the mechanistic and holistic understanding of in planta cell wall synthesis, which will accelerate the development of better transgenic crops for bioenergy applications. Moreover, the new toolbox, combining powerful advanced microscopy assays with cell/protein engineering, will have broader impacts on molecular and cellular biology fields by paving the way for studying cellular processes occurring across multiple physical scales with multimodal microscopy methodologies.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Multimodal sensor fusion framework for residential building occupancy detection

For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This work proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine's (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms' performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Automated Grain Boundary (GB) Segmentation and Microstructural Analysis in 347H Stainless Steel Using Deep Learning and Multimodal Microscopy

Austenitic 347H stainless steel offers superior mechanical properties and corrosion resistance required for extreme operating conditions such as high temperature. The change in microstructure due to composition and process variations is expected to impact material properties. Identifying microstructural features such as grain boundaries thus becomes an important task in the process-microstructure-properties loop. Applying convolutional neural network (CNN)-based deep learning models is a powerful technique to detect features from material micrographs in an automated manner. In contrast to microstructural classification, supervised CNN models for segmentation tasks require pixel-wise annotation labels. However, manual labeling of the images for the segmentation task poses a major bottleneck for generating training data and labels in a reliable and reproducible way within a reasonable timeframe. Microstructural characterization especially needs to be expedited for faster material discovery by changing alloy compositions. Here, in this study, we attempt to overcome such limitations by utilizing multimodal microscopy to generate labels directly instead of manual labeling. We combine scanning electron microscopy images of 347H stainless steel as training data and electron backscatter diffraction micrographs as pixel-wise labels for grain boundary detection as a semantic segmentation task. The viability of our method is evaluated by considering a set of deep CNN architectures. We demonstrate that despite producing instrumentation drift during data collection between two modes of microscopy, this method performs comparably to similar segmentation tasks that used manual labeling. Additionally, we find that naïve pixel-wise segmentation results in small gaps and missing boundaries in the predicted grain boundary map. By incorporating topological information during model training, the connectivity of the grain boundary network and segmentation performance is improved. Finally, our approach is validated by accurate computation on downstream tasks of predicting the underlying grain morphology distributions which are the ultimate quantities of interest for microstructural characterization.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Attention-based explainability for structure–property relationships

Machine learning methods are emerging as a universal paradigm for constructing correlative structure–property relationships in materials science based on multimodal characterization. However, this necessitates the development of methods for the physical interpretability of the resulting correlative models. Here, we demonstrate the potential of attention-based neural networks for revealing structure–property relationships and the underlying physical mechanisms, using the ferroelectric properties of PbTiO3 thin films as a case study. Through the analysis of attention scores, we disentangle the influence of distinct domain patterns on the polarization switching process. The attention-based Transformer model is explored both as a direct interpretability tool and as a surrogate for explaining representations learned via unsupervised machine learning, enabling the identification of physically grounded correlations. We compare attention-derived interpretability scores with classical SHapley Additive exPlanations analysis and show that, in contrast to applications in natural language processing, attention mechanisms in materials science exhibit high efficiency in highlighting meaningful structural features.

Slautin, Boris [Independent Researcher]↗
DOE OSTI2021

Striving to translate shale physics across ten orders of magnitude: What have we learned?

Shales will play an important role in the successful transition of energy from fossil-based resources to renewables in the coming decades. Aside from being a significant source of low-carbon intensity fuels, like natural gas, they also serve as geologic seals of subsurface formations that may be used to isolate nuclear waste, sequester CO 2 , or store intermittent energy (e.g., solar hydrogen). Despite their importance, shales pose significant engineering and environmental challenges due to their nanoporous structure and extreme heterogeneity that spans at least ~10 orders of magnitude in spatial scale. Two challenges inhibit a system-level understanding: (1) the physics of fluid flow and phase behavior in shales are poorly understood due to the dominant molecular interactions between minerals and fluids under confinement, and (2) the apparent lack of scale separation that prevents a reliable (closed) description of the physics at any single scale of observation. In this review, we focus on the latter issue and discuss scale translation, which in its broadest sense is transforming data or simulations from one spatiotemporal scale to another. While effective scale translation is not exclusive to shales, but all geologic porous media, the need for it is especially acute in shales given their high degree of heterogeneity. Classical theories like homogenization, while indispensable, fail when scales are not separated. Other methods, like numerical upscaling, scale-translate in only one direction: small to large, but not the reverse, called downscaling. However, the confluence of advances in three areas are bringing challenging problems such as shales within reach: increased computational power and scalable algorithms; high-resolution imaging and multi-modal data acquisition; and machine learning to process massive amounts of data. While these advances equip geoscientists with a wide array of experimental and computational tools, no individual tool can probe the entire gamut of heterogeneity in shales. Their effective use, therefore, requires an ability to bridge between various data types obtained at different scales. The aim of this review is to present a coherent account of computational and experimental methods that may be used to achieve just that, i.e., to perform scale translation. We provide a broader definition of scale translation, one that transcends classical homogenization and upscaling methods, but is consistent with them and accommodates notions like downscaling and data translation. After a brief introduction to homogenization, we review hybrid methods, numerical upscaling and its recent extensions, multiscale computing, high-resolution imaging, and machine learning. We place particular emphasis on multiscale computing and propose an algorithmic framework to bridge between the pore (micro) and Darcy (macro) scales. Throughout the paper, we draw comparisons between the various methods and highlight their (often hidden) similarities, differences, benefits, and pitfalls. We finally conclude with two case studies on shales that exemplify some of the methods presented.

58 GEOSCIENCES↗
NASA NTRS1984

Magneto-optic garnet and liquid crystal optical switches

Magnetic stripe domain and liquid crystal devices are being developed and evaluated as fiber optic switches that can be utilized for nonblocking type nxm optical matrix switches in networking and optical processing. Liquid crystal switches are characterized by very low insertion loss and crosstalk, while stripe domain switches commutate in less than one microsecond. Both switches operate on multimode, randomly polarized fiber light with potentially large values for (n,m). The applications of these magnetic stripe domain and liquid crystal devices are discussed.

Krawczak, J. A.↗
DOE OSTI2023

Parallel measurement of transcriptomes and proteomes from same single cells using nanodroplet splitting

Single-cell multiomics provides comprehensive insights into gene regulatory networks, cellular diversity, and temporal dynamics. While tools for co-profiling single-cell genomes, transcriptomes, and epigenomes are available, accessing proteomes in parallel is more challenging. We developed nanoSPLITS (nanodroplet SPlitting for Linked-multimodal Investigations of Trace Samples), an integrated platform that enables global profiling of the transcriptome and proteome from same single cells using RNA sequencing and mass spectrometry-based proteomics, respectively. nanoSPLITS can precisely quantify over 5000 genes, 2000 proteins, and 140 phosphopeptides per single cell and identify candidate cell markers from these modalities. By exploring Cdk1-mediated cell cycle arrest, we demonstrate how nanoSPLITS single-cell multiomics can provide comprehensive cellular characterization with insights into covarying protein/gene clusters, unique phosphorylation events, and mitotic pathways.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRSDate not supplied

Concept-based Analysis of Neural Networks via Vision-Language Models

The analysis of vision-based deep neural networks (DNNs) is highly desirable but it is very challenging due to the difficulty of expressing formal specifications for vision tasks and the lack of efficient verification procedures. In this paper, we propose to leverage emerging multimodal, vision-language, foundation models (VLMs) as a lens through which we can reason about vision models. VLMs have been trained on a large body of images accompanied by their textual description, and are thus implicitly aware of high-level, human-understandable concepts describing the images. We describe a logical specification language Con spec designed to facilitate writing specifications in terms of these concepts. To define and formally check Con spec specifications, we build a map between the internal representations of a given vision model and a VLM, leading to an efficient verification procedure of natural-language properties for vision models. We demonstrate our techniques on a ResNet-based classifier trained on the RIVAL-10 dataset using CLIP as the multimodal model.

Large Vision Language Models↗
DOE OSTI2022

Machine‐Learning Spectral Indicators of Topology

Abstract Topological materials discovery has emerged as an important frontier in condensed matter physics. While theoretical classification frameworks have been used to identify thousands of candidate topological materials, experimental determination of materials’ topology often poses significant technical challenges. X‐ray absorption spectroscopy (XAS) is a widely used materials characterization technique sensitive to atoms’ local symmetry and chemical bonding, which are intimately linked to band topology by the theory of topological quantum chemistry (TQC). Moreover, as a local structural probe, XAS is known to have high quantitative agreement between experiment and calculation, suggesting that insights from computational spectra can effectively inform experiments. In this work, computed X‐ray absorption near‐edge structure (XANES) spectra of more than 10 000 inorganic materials to train a neural network (NN) classifier that predicts topological class directly from XANES signatures, achieving F 1 scores of 89% and 93% for topological and trivial classes, respectively is leveraged. Given the simplicity of the XAS setup and its compatibility with multimodal sample environments, the proposed machine‐learning‐augmented XAS topological indicator has the potential to discover broader categories of topological materials, such as non‐cleavable compounds and amorphous materials, and may further inform field‐driven phenomena in situ, such as magnetic field‐driven topological phase transitions.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Solving multiphysics-based inverse problems with learned surrogates and constraints

Abstract Solving multiphysics-based inverse problems for geological carbon storage monitoring can be challenging when multimodal time-lapse data are expensive to collect and costly to simulate numerically. We overcome these challenges by combining computationally cheap learned surrogates with learned constraints. Not only does this combination lead to vastly improved inversions for the important fluid-flow property, permeability, it also provides a natural platform for inverting multimodal data including well measurements and active-source time-lapse seismic data. By adding a learned constraint, we arrive at a computationally feasible inversion approach that remains accurate. This is accomplished by including a trained deep neural network, known as a normalizing flow, which forces the model iterates to remain in-distribution, thereby safeguarding the accuracy of trained Fourier neural operators that act as surrogates for the computationally expensive multiphase flow simulations involving partial differential equation solves. By means of carefully selected experiments, centered around the problem of geological carbon storage, we demonstrate the efficacy of the proposed constrained optimization method on two different data modalities, namely time-lapse well and time-lapse seismic data. While permeability inversions from both these two modalities have their pluses and minuses, their joint inversion benefits from either, yielding valuable superior permeability inversions and CO 2 plume predictions near, and far away, from the monitoring wells.

Yin, Ziyi (ORCID:0000000250248771)↗
DOE OSTI2023

Scaling Subspace-Driven Approaches Using Information Fusion

In this work, we seek to exploit the deep structure of multi-modal data to robustly exploit the group subspace distribution of the information using the Convolutional Neural Networks (CNNs) formalism. Upon unfolding the set of subspaces constituting each data modality, and learning their corresponding encoders, an optimized integration of the generated inherent information is carried out to yield a characterization of various classes. Referred to as deep Multimodal Robust Group Subspace Clustering (DRoGSuRe), this approach is compared against the independently developed state-of-the-art approach named Deep Multimodal Subspace Clustering (DMSC). Experiments on different multimodal datasets show that our approach is competitive and more robust in the presence of noise.

Ghanem, Sally↗
NASA NTRS1974

Design, fabrication, test and delivery of a K-band antenna breadboard model

The results of a research effort to develop a Ku-Band single channel monopulse antenna with significant improvements in efficiency and bandwidth are reported. A single aperture, multimode horn, utilized in a near field Cassegrainian configuration, was the technique selected for achieving the desired efficiency and bandwidth performance. In order to provide wide polarization flexibility, a wire grid, space filter polarizer was developed. A solid state switching network with appropriate driving electronics provides the receive channel sum and difference signal interface with an existing Apollo type tracking electronics subsystem. A full scale breadboard model of the antenna was fabricated and tested. Performance of the model was well within the requirements and goals of the contract.

Source record↗