SEARCH · Engineering Papers

Results for “multimodal imaging”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Identification of Blue Horizontal Branch Stars with Multimodal Fusion

Blue Horizontal Branch stars (BHBs) are ideal tracers to probe the global structure of the milky Way (MW), and the increased size of the BHB star sample could be helpful to accurately calculate the MW's enclosed mass and kinematics. Large survey telescopes have produced an increasing number of astronomical images and spectra. However, traditional methods of identifying BHBs are limited in dealing with the large scale of astronomical data. A fast and efficient way of identifying BHBs can provide a more significant sample for further analysis and research. Therefore, in order to fully use the various data observed and further improve the identification accuracy of BHBs, we have innovatively proposed and implemented a Bi-level attention mechanism-based Transformer multimodal fusion model, called Bi-level Attention in the Transformer with Multimodality (BATMM). The model consists of a spectrum encoder, an image encoder, and a Transformer multimodal fusion module. The Transformer enables the effective fusion of data from two modalities, namely image and spectrum, by using the proposed Bi-level attention mechanism, including cross-attention and self-attention. As a result, the information from the different modalities complements each other, thus improving the accuracy of the identification of BHBs. The experimental results show that the F1 score of the proposed BATMM is 94.78%, which is 21.77% and 2.76% higher than the image and spectral unimodality, respectively. It is therefore demonstrated that higher identification accuracy of BHBs can be achieved by means of using data from multiple modalities and employing an efficient data fusion strategy.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Development of Multimodal Few-Shot Analytics for Electron Micrographs

Recent advances in materials data analytics have provided new avenues for determining process-structure-property (PSP) linkages in a variety of materials. Machine learning techniques including few-shot learning have increased the efficiency of classifying microscopy images for the purposes of material characterization. Attempts at creating a multimodal approach can provide further improvements to current models and help extract more salient features from data. In this vein, raw spectrum data was taken to provide an additional modality to our current pyCHIP classifier. Modifications in segmentation also show potential in improving the accuracy of the pyCHIP classifier. Classifier output was analyzed using network graphs and unsupervised clustering algorithms such as spectral clustering to detect better segmentation methods than the current “chipping” approach. We suggest that the chip selection process can be automated in the future using a combination of these techniques to enable high-throughput analyses.

36 MATERIALS SCIENCE↗

DOE OSTI2020

Covalent Surface Modification Effects on Single‐Walled Carbon Nanotubes for Targeted Sensing and Optical Imaging

Abstract Optical nanoscale technologies often implement covalent or noncovalent strategies for the modification of nanoparticles, whereby both functionalizations are leveraged for multimodal applications but can affect the intrinsic fluorescence of nanoparticles. Specifically, single‐walled carbon nanotubes (SWCNTs) can enable real‐time imaging and cellular delivery; however, the introduction of covalent SWCNT sidewall functionalizations often attenuates SWCNT fluorescence. Recent advances in SWCNT covalent functionalization chemistries preserve the SWCNT's pristine graphitic lattice and intrinsic fluorescence, and here, such covalently functionalized SWCNTs maintain intrinsic fluorescence‐based molecular recognition of neurotransmitter and protein analytes. The covalently modified SWCNT nanosensor preserves its fluorescence response towards its analyte for certain nanosensors, presumably dependent on the intermolecular interactions between SWCNTs or the steric hindrance introduced by the covalent functionalization that hinders noncovalent interactions with the SWCNT surface. These SWCNT nanosensors are further functionalized via their covalent handles with a targeting ligand, biotin, to self‐assemble on passivated microscopy slides, and these dual‐functionalized SWCNT materials are explored for future use in multiplexed sensing and imaging applications.

Chio, Linda↗

DOE OSTI2024

Single-molecule imaging of lignocellulose deconstruction by SCATTIRSTORM microscopy

The goal of this project is to build a multimodal optical microscope to measure the binding, processive degradation, and pausing behaviors of cellulases as they interact with and degrade both synthetic and naturally occurring lignocellulosic walls. To achieve this, we will use high spatio-temporal single-molecule imaging to track cellulases, while visualizing specific molecular components of cellulose, lignin and hemicellulose, that make up their lignocellulose substrate. The microscope will combine Interferometric Scattering (iSCAT), which provides unprecedented spatiotemporal resolution; Total Internal Reflection Fluorescence (TIRF), which provides single-molecule resolution of multiple fluorophore-labeled molecules; and Stochastic Reconstruction (STORM), which allows for three-dimensional super-resolution imaging of intact plant cell walls during degradation. Initial studies will investigate cellulase dynamics on in vitro-assembled cell wall analogs, and later work will progress to using native plant cell walls.

09 BIOMASS FUELS↗

DOE OSTI2020

Future trends in synchrotron science at NSLS-II

In this paper, we summarize briefly some of the future trends in synchrotron science as seen at the National Synchrotron Light Source II, a new, low emittance source recently commissioned at Brookhaven National Laboratory. We touch upon imaging techniques, the study of dynamics, the increasing use of multimodal approaches, the vital importance of data science, and finally, other enabling technologies. Each are presently undergoing a time of rapid change, driving the field of synchrotron science forward at an ever increasing pace. It is truly an exciting time and one in which Roger Cowley, to whom this journal issue is dedicated, would surely be both invigorated by, and at the heart of.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2020

Using Neural Architecture Search for Improving Software Flaw Detection in Multimodal Deep Learning Models

Software flaw detection using multimodal deep learning models has been demonstrated as a very competitive approach on benchmark problems. In this work, we demonstrate that even better performance can be achieved using neural architecture search (NAS) combined with multimodal learning models. We adapt a NAS framework aimed at investigating image classification to the problem of software flaw detection and demonstrate improved results on the Juliet Test Suite, a popular benchmarking data set for measuring performance of machine learning models in this problem domain.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

AC Magnetometry Using Nano-ferrofluid Cladded Multimode Interferometric Fiber Optic Sensors for Power Grid Monitoring Applications

The AC magnetic field response of the superparamagnetic nano-ferrofluid is an interplay between the Neel and Brownian relaxation processes and is generally quantified via the susceptibility measurements at high frequencies. The high frequency limit is dictated by these relaxation times which need to be shorter than the time scale of the time varying magnetic field for the nano-ferrofluid to be considered in an equilibrium state at each time instant. Even though the high frequency response of ferrofluid has been extensively investigated for frequencies up to GHz range by non-optical methods, harnessing dynamic response by optical means for AC magnetic field sensing in fiber-optic-based sensors-field remains unexplored. Instead, the incorporation of nano-ferrofluid as sensing materials has been only limited to DC magnetic field sensing, often citing their long response time as a limiting factor to AC field sensing. This work reports the finding of high frequency (up to 15 kHz) AC magnetic field sensing capability of nanomagnetic fluid as the cladding material of a fiber-optic multimode interferometry (MMI) structure optimized for the fourth self-imaging spectral response. The key parameter enabling high frequency response is the short response time (<1 ms) achieved by optimizing both the sensing structure and nano-ferrofluid solution. Focus has been imparted on 60 Hz line-frequency profiles of various current/magnetic fields to test the efficacy of these sensors in metering and monitoring current and current-induced magnetic fields in the electrical power grid systems. The magnetic field sensitivity of 240 mV/Gauss per dBm of transmitted power was achieved for 60 Hz field applied via Helmholtz coil, whereas the 60 Hz AC current sensitivity of 2.83 mV/A was measured due to magnetic field induced by current in a straight conducting wire.

42 ENGINEERING↗

DOE OSTI2026

Generalist multimodal AI: A review of architectures, challenges and opportunities

Multimodal models are expected to be a critical component to future advances in artificial intelligence. Here, this field is starting to grow rapidly with a surge of new design elements motivated by the success of foundation models in natural language processing (NLP) and vision. It is widely hoped that further extending the foundation models to multiple modalities (e.g., text, image, video, sensor, time series, graph, etc.) will ultimately lead to generalist multimodal models, i.e. one model across different data modalities and tasks. However, there is little research that systematically analyzes recent multimodal models (particularly the ones that work beyond text and vision) with respect to the underling architecture proposed. Therefore, this work provides a fresh perspective on generalist multimodal models (GMMs) via a novel architecture and training configuration specific taxonomy. This includes factors such as Unifiability, Modularity, and Adaptability that are pertinent and essential to the wide adoption and application of GMMs. The review further highlights key challenges and prospects for the field and guide the researchers into the new advancements.

Artificial intelligence (AI)↗

NASA NTRS1986

Instrumentation for optical remote sensing from space; Proceedings of the Meeting, Cannes, France, November 27-29, 1985

Papers are presented on such topics as the development of the Imaging Spectrometer for Shuttle and space platform applications; the in-flight calibration of pushbroom remote sensing instruments for the SPOT program; buttable detector arrays for 1.55-1.7 micron imaging; the design of the Improved Stratospheric and Mesospheric Sounder on the Upper Atmosphere Research Satellite; and SAGE II design and in-orbit performance. Consideration is also given to the Shuttle Imaging Radar-B/C instruments; the Venus Radar Mapper multimode radar system design; various ISO instruments (ISOCAM, ISOPHOT, and SWS and LWS); and instrumentation for the Space Infrared Telescope Facility.

Seeley, John S.↗

DOE OSTI2025

Cryosectioning-enhanced super-resolution microscopy for single-protein imaging across cells and tissues

DNA-points accumulation for imaging in nanoscale topography (DNA-PAINT) enables nanoscale imaging with virtually unlimited multiplexing and molecular counting. Here, we address challenges, such as variable imaging performance and target accessibility, that can limit its broader applicability. Specifically, we enhance its capacity for robust single-protein imaging and molecular counting by optimizing the integration of total internal reflection fluorescence microscopy with physical sectioning, in particular, Tokuyasu cryosectioning. Our method, tomographic and kinetically enhanced DNA-PAINT (tkPAINT), achieves 3 nm localization precision across diverse samples, enhanced imager binding, and improved cellular integrity. tkPAINT can facilitate molecular counting with DNA-PAINT inside the nucleus, as demonstrated through its quantification of the in situ abundance of RNA Polymerase II in both HeLa cells as well as mouse tissues. Anticipating that tkPAINT could become a versatile tool for the exploration of biomolecular organization and interactions across cells and tissues, we also demonstrate its capacity to support multiplexing, multimodal targeting of proteins and nucleic acids, and three-dimensional (3D) imaging.

Science & Technology - Other Topics↗

DOE OSTI2025

Label-free structural imaging of plant roots and microbes using third-harmonic generation microscopy

Root biology is pivotal in addressing global challenges including sustainable agriculture and climate change. However, roots have been relatively understudied among plant organs, partly due to the difficulties in imaging root structures in their natural environment. Here we used microfabricated ecosystems (EcoFABs) to establish growing environments with optical access and employed nonlinear multimodal microscopy of third-harmonic generation (THG) and three-photon fluorescence (3PF) to achieve label-free, in situ imaging of live roots and microbes at high spatiotemporal resolution. THG enabled us to observe key plant root structures including the vasculature, Casparian strips, dividing meristematic cells, and root cap cells, as well as subcellular features including nuclear envelopes, nucleoli, starch granules, and putative stress granules. THG from the cell walls of bacteria and fungi also provides label-free contrast for visualizing these microbes in the root rhizosphere. With simultaneously recorded 3PF signal, we demonstrated our ability to investigate root-microbe interactions by achieving single-bacterium tracking and subcellular imaging of fungal spores and hyphae in the rhizosphere.

Pan, Daisong [University of California, Berkeley, ↗

DOE OSTI2020

Using Neural Architecture Search for Improving Software Flaw Detection in Multimodal Deep Learning Models

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Dose Summation Strategies for External Beam Radiation Therapy and Brachytherapy in Gynecologic Malignancy: A Review from the NRG Oncology and NCTN Medical Physics Subcommittees

Definitive, nonsurgical management of gynecologic malignancies involves external beam radiation therapy (EBRT) and/or brachytherapy (BT). Summation of the cumulative dose is critical to assess the total biologic effective dose to targets and organs at risk. Cumulative dose calculation from EBRT and BT can be performed with or without image registration (IR) and biologic dose summation. Among these dose summation strategies, linear addition of dose-volume histogram (DVH) parameters without IR is the global standard for composite dose reporting. This approach stems from an era without image guidance and simple external beam and brachytherapy treatment approaches. With technological advances, EBRT and high-dose-rate BT have evolved to allow for volume-based treatment planning and delivery. Modern conformal therapeutic radiation involves volumetric or intensity modulated EBRT, capable of simultaneously treating multiple targets at different specified dose levels. Therefore, given the complexity of modern radiation treatment, the linear addition of DVH parameters from EBRT and high-dose-rate BT is challenging to represent the combined dose distribution. Deformable image registration (DIR) between EBRT and image guided brachytherapy (IGBT) data sets may provide a more nuanced calculation of multimodal dose accumulation. However, DIR is still nascent in this regard, and needs further development for accuracy and efficiency for clinical use. Biologic dose summation can combine physical dose maps from EBRT and each IGBT fraction, thereby generating a composite DVH from the biologic effective dose. However, accurate radiobiologic parameters are tissue-dependent and not well characterized. A combination of voxel-based DIR and biologic weighted dose maps may be the best approximation of dose accumulation but remains invalidated. The purpose of this report is to review dose summation strategies for EBRT and BT, including conventional equivalent dose in 2-Gy fractions dose summation without image registration, physical dose summation using 3-dimensional rigid IR and DIR, and biologic dose summation. We also provide general clinical workflows for IGBT with a focus on cervical cancer.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

NASA NTRS2013

Mixture-Tuned, Clutter Matched Filter for Remote Detection of Subpixel Spectral Signals

Mapping localized spectral features in large images demands sensitive and robust detection algorithms. Two aspects of large images that can harm matched-filter detection performance are addressed simultaneously. First, multimodal backgrounds may thwart the typical Gaussian model. Second, outlier features can trigger false detections from large projections onto the target vector. Two state-of-the-art approaches are combined that independently address outlier false positives and multimodal backgrounds. The background clustering models multimodal backgrounds, and the mixture tuned matched filter (MT-MF) addresses outliers. Combining the two methods captures significant additional performance benefits. The resulting mixture tuned clutter matched filter (MT-CMF) shows effective performance on simulated and airborne datasets. The classical MNF transform was applied, followed by k-means clustering. Then, each cluster s mean, covariance, and the corresponding eigenvalues were estimated. This yields a cluster-specific matched filter estimate as well as a cluster- specific feasibility score to flag outlier false positives. The technology described is a proof of concept that may be employed in future target detection and mapping applications for remote imaging spectrometers. It is of most direct relevance to JPL proposals for airborne and orbital hyperspectral instruments. Applications include subpixel target detection in hyperspectral scenes for military surveillance. Earth science applications include mineralogical mapping, species discrimination for ecosystem health monitoring, and land use classification.

Thompson, David R.↗

DOE OSTI2021

Light Propagation Considerations For Internally Clad Sapphire Optical Fiber Using the 6Li(n,)3H Reaction

Optical frequency domain reflectometry measurements in internally clad single crystal sapphire fiber have received attention in recent years due to their high temperature distributed sensing potential. As work with these fibers has proceeded, there have been some inconsistencies in the results. Deeper investigation and testing has identified two critical considerations for the proper functioning of these fibers. First, users must address the propagation of multimode light along the outer surface of the fiber. Here, by observing the far-field image of an internally clad sapphire fiber when adding index matching fluid to the outer fiber surface, we demonstrate the effects of removing the higher order modes from these fibers. The addition of index matching fluid resulted in nearly single mode performance where multimode performance was previously observed. Second, users must address the effect that coupling the fiber to the interrogator via silica based fiber has on the internally clad sapphire fibers performance. Direct fusion splicing of silica to sapphire, as has been used in the recent work with these fibers, has a mode filtering effect which can be beneficial towards the modal behavior of the fibers. However, in this paper we demonstrate that the splicing can cause a sensing failure due to little or no low order mode light, that is useful for sensing, returning to the detector. The positive results from recent years have demonstrated that optical frequency domain reflectometry sensing performance will be successful in clad sapphire fiber; but only when the considerations described herein are addressed properly.

47 OTHER INSTRUMENTATION↗

DOE OSTI2024

Unsupervised multimodal fusion of in-process sensor data for advanced manufacturing process monitoring

Effective monitoring of manufacturing processes is crucial for maintaining product quality and operational efficiency. Modern manufacturing environments often generate vast amounts of complementary multimodal data, including visual imagery from various perspectives and resolutions, hyperspectral data, and machine health monitoring information such as actuator positions, accelerometer readings, and temperature measurements. However, fusing and interpreting this complex, high-dimensional data presents significant challenges, particularly when labeled datasets are unavailable or impractical to obtain. This paper presents a novel approach to multimodal sensor data fusion in manufacturing processes, inspired by the Contrastive Language-Image Pre-training (CLIP) model. We leverage contrastive learning techniques to correlate different data modalities without the need for labeled data, overcoming limitations of traditional supervised machine learning methods in manufacturing contexts. Our proposed method demonstrates the ability to handle and learn encoders for five distinct modalities: visual imagery, audio signals, laser position (x and y coordinates), and laser power measurements. By compressing these high-dimensional datasets into low-dimensional representational spaces, our approach facilitates downstream tasks such as process control, anomaly detection, and quality assurance. The unsupervised nature of our method makes it broadly applicable across various manufacturing domains, where large volumes of unlabeled sensor data are common. We evaluate the effectiveness of our approach through a series of experiments, demonstrating its potential to enhance process monitoring capabilities in advanced manufacturing systems. This research contributes to the field of smart manufacturing by providing a flexible, scalable framework for multimodal data fusion that can adapt to diverse manufacturing environments and sensor configurations. The proposed method paves the way for more robust, data-driven decision-making in complex manufacturing processes.

Contrastive Learning↗

DOE OSTI2021

User defined intensity profile laser beam

A high-power laser beam with an arbitrary intensity profile is produced. Such beam has a variety of uses including for laser materials processing such as powder bed fusion additive manufacturing. Several challenges in additive manufacturing are mitigated with the present non-uniform intensity laser profiles. Nonuniform shapes include a set of intensity pixels in a line that could print a wide stripe area instead of just a single line. One example uses the multimode interference pattern from the output of a ribbon fiber which is imaged onto a work piece. The interference pattern is controlled to allow turning on or off of ‘pixels’ along a line which can be used to shape the beam and form the additively manufactured part.

Drachenberg, Derrek R.↗

DOE OSTI2024

Automated electrosynthesis reaction mining with multimodal large language models (MLLMs)

Leveraging the chemical data available in legacy formats such as publications and patents is a significant challenge for the community. Automated reaction mining offers a promising solution to unleash this knowledge into a learnable digital form and therefore help expedite materials and reaction discovery. However, existing reaction mining toolkits are limited to single input modalities (text or images) and cannot effectively integrate heterogeneous data that is scattered across text, tables, and figures. In this work, we go beyond single input modalities and explore multimodal large language models (MLLMs) for the analysis of diverse data inputs for automated electrosynthesis reaction mining. We compiled a test dataset of 65 articles (MERMES-T24 set) and employed it to benchmark five prominent MLLMs against two critical tasks: (i) reaction diagram parsing and (ii) resolving cross-modality data interdependencies. The frontrunner MLLM achieved ≥96% accuracy in both tasks, with the strategic integration of single-shot visual prompts and image pre-processing techniques. We integrate this capability into a toolkit named MERMES (multimodal reaction mining pipeline for electrosynthesis). Our toolkit functions as an end-to-end MLLM-powered pipeline that integrates article retrieval, information extraction and multimodal analysis for streamlining and automating knowledge extraction. This work lays the groundwork for the increased utilization of MLLMs to accelerate the digitization of chemistry knowledge for data-driven research.

Leong, Shi Xuan↗