Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “multimodal imaging”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Covalent Surface Modification Effects on Single‐Walled Carbon Nanotubes for Targeted Sensing and Optical Imaging

Abstract Optical nanoscale technologies often implement covalent or noncovalent strategies for the modification of nanoparticles, whereby both functionalizations are leveraged for multimodal applications but can affect the intrinsic fluorescence of nanoparticles. Specifically, single‐walled carbon nanotubes (SWCNTs) can enable real‐time imaging and cellular delivery; however, the introduction of covalent SWCNT sidewall functionalizations often attenuates SWCNT fluorescence. Recent advances in SWCNT covalent functionalization chemistries preserve the SWCNT's pristine graphitic lattice and intrinsic fluorescence, and here, such covalently functionalized SWCNTs maintain intrinsic fluorescence‐based molecular recognition of neurotransmitter and protein analytes. The covalently modified SWCNT nanosensor preserves its fluorescence response towards its analyte for certain nanosensors, presumably dependent on the intermolecular interactions between SWCNTs or the steric hindrance introduced by the covalent functionalization that hinders noncovalent interactions with the SWCNT surface. These SWCNT nanosensors are further functionalized via their covalent handles with a targeting ligand, biotin, to self‐assemble on passivated microscopy slides, and these dual‐functionalized SWCNT materials are explored for future use in multiplexed sensing and imaging applications.

Chio, Linda↗
DOE OSTI2024

Single-molecule imaging of lignocellulose deconstruction by SCATTIRSTORM microscopy

The goal of this project is to build a multimodal optical microscope to measure the binding, processive degradation, and pausing behaviors of cellulases as they interact with and degrade both synthetic and naturally occurring lignocellulosic walls. To achieve this, we will use high spatio-temporal single-molecule imaging to track cellulases, while visualizing specific molecular components of cellulose, lignin and hemicellulose, that make up their lignocellulose substrate. The microscope will combine Interferometric Scattering (iSCAT), which provides unprecedented spatiotemporal resolution; Total Internal Reflection Fluorescence (TIRF), which provides single-molecule resolution of multiple fluorophore-labeled molecules; and Stochastic Reconstruction (STORM), which allows for three-dimensional super-resolution imaging of intact plant cell walls during degradation. Initial studies will investigate cellulase dynamics on in vitro-assembled cell wall analogs, and later work will progress to using native plant cell walls.

09 BIOMASS FUELS↗
DOE OSTI2020

Future trends in synchrotron science at NSLS-II

In this paper, we summarize briefly some of the future trends in synchrotron science as seen at the National Synchrotron Light Source II, a new, low emittance source recently commissioned at Brookhaven National Laboratory. We touch upon imaging techniques, the study of dynamics, the increasing use of multimodal approaches, the vital importance of data science, and finally, other enabling technologies. Each are presently undergoing a time of rapid change, driving the field of synchrotron science forward at an ever increasing pace. It is truly an exciting time and one in which Roger Cowley, to whom this journal issue is dedicated, would surely be both invigorated by, and at the heart of.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2020

Using Neural Architecture Search for Improving Software Flaw Detection in Multimodal Deep Learning Models

Software flaw detection using multimodal deep learning models has been demonstrated as a very competitive approach on benchmark problems. In this work, we demonstrate that even better performance can be achieved using neural architecture search (NAS) combined with multimodal learning models. We adapt a NAS framework aimed at investigating image classification to the problem of software flaw detection and demonstrate improved results on the Juliet Test Suite, a popular benchmarking data set for measuring performance of machine learning models in this problem domain.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

AC Magnetometry Using Nano-ferrofluid Cladded Multimode Interferometric Fiber Optic Sensors for Power Grid Monitoring Applications

The AC magnetic field response of the superparamagnetic nano-ferrofluid is an interplay between the Neel and Brownian relaxation processes and is generally quantified via the susceptibility measurements at high frequencies. The high frequency limit is dictated by these relaxation times which need to be shorter than the time scale of the time varying magnetic field for the nano-ferrofluid to be considered in an equilibrium state at each time instant. Even though the high frequency response of ferrofluid has been extensively investigated for frequencies up to GHz range by non-optical methods, harnessing dynamic response by optical means for AC magnetic field sensing in fiber-optic-based sensors-field remains unexplored. Instead, the incorporation of nano-ferrofluid as sensing materials has been only limited to DC magnetic field sensing, often citing their long response time as a limiting factor to AC field sensing. This work reports the finding of high frequency (up to 15 kHz) AC magnetic field sensing capability of nanomagnetic fluid as the cladding material of a fiber-optic multimode interferometry (MMI) structure optimized for the fourth self-imaging spectral response. The key parameter enabling high frequency response is the short response time (<1 ms) achieved by optimizing both the sensing structure and nano-ferrofluid solution. Focus has been imparted on 60 Hz line-frequency profiles of various current/magnetic fields to test the efficacy of these sensors in metering and monitoring current and current-induced magnetic fields in the electrical power grid systems. The magnetic field sensitivity of 240 mV/Gauss per dBm of transmitted power was achieved for 60 Hz field applied via Helmholtz coil, whereas the 60 Hz AC current sensitivity of 2.83 mV/A was measured due to magnetic field induced by current in a straight conducting wire.

42 ENGINEERING↗
DOE OSTI2026

Generalist multimodal AI: A review of architectures, challenges and opportunities

Multimodal models are expected to be a critical component to future advances in artificial intelligence. Here, this field is starting to grow rapidly with a surge of new design elements motivated by the success of foundation models in natural language processing (NLP) and vision. It is widely hoped that further extending the foundation models to multiple modalities (e.g., text, image, video, sensor, time series, graph, etc.) will ultimately lead to generalist multimodal models, i.e. one model across different data modalities and tasks. However, there is little research that systematically analyzes recent multimodal models (particularly the ones that work beyond text and vision) with respect to the underling architecture proposed. Therefore, this work provides a fresh perspective on generalist multimodal models (GMMs) via a novel architecture and training configuration specific taxonomy. This includes factors such as Unifiability, Modularity, and Adaptability that are pertinent and essential to the wide adoption and application of GMMs. The review further highlights key challenges and prospects for the field and guide the researchers into the new advancements.

Artificial intelligence (AI)↗
DOE OSTI2025

Cryosectioning-enhanced super-resolution microscopy for single-protein imaging across cells and tissues

DNA-points accumulation for imaging in nanoscale topography (DNA-PAINT) enables nanoscale imaging with virtually unlimited multiplexing and molecular counting. Here, we address challenges, such as variable imaging performance and target accessibility, that can limit its broader applicability. Specifically, we enhance its capacity for robust single-protein imaging and molecular counting by optimizing the integration of total internal reflection fluorescence microscopy with physical sectioning, in particular, Tokuyasu cryosectioning. Our method, tomographic and kinetically enhanced DNA-PAINT (tkPAINT), achieves 3 nm localization precision across diverse samples, enhanced imager binding, and improved cellular integrity. tkPAINT can facilitate molecular counting with DNA-PAINT inside the nucleus, as demonstrated through its quantification of the in situ abundance of RNA Polymerase II in both HeLa cells as well as mouse tissues. Anticipating that tkPAINT could become a versatile tool for the exploration of biomolecular organization and interactions across cells and tissues, we also demonstrate its capacity to support multiplexing, multimodal targeting of proteins and nucleic acids, and three-dimensional (3D) imaging.

Science & Technology - Other Topics↗
DOE OSTI2025

Label-free structural imaging of plant roots and microbes using third-harmonic generation microscopy

Root biology is pivotal in addressing global challenges including sustainable agriculture and climate change. However, roots have been relatively understudied among plant organs, partly due to the difficulties in imaging root structures in their natural environment. Here we used microfabricated ecosystems (EcoFABs) to establish growing environments with optical access and employed nonlinear multimodal microscopy of third-harmonic generation (THG) and three-photon fluorescence (3PF) to achieve label-free, in situ imaging of live roots and microbes at high spatiotemporal resolution. THG enabled us to observe key plant root structures including the vasculature, Casparian strips, dividing meristematic cells, and root cap cells, as well as subcellular features including nuclear envelopes, nucleoli, starch granules, and putative stress granules. THG from the cell walls of bacteria and fungi also provides label-free contrast for visualizing these microbes in the root rhizosphere. With simultaneously recorded 3PF signal, we demonstrated our ability to investigate root-microbe interactions by achieving single-bacterium tracking and subcellular imaging of fungal spores and hyphae in the rhizosphere.

Pan, Daisong [University of California, Berkeley, ↗
DOE OSTI2020

Using Neural Architecture Search for Improving Software Flaw Detection in Multimodal Deep Learning Models

Software flaw detection using multimodal deep learning models has been demonstrated as a very competitive approach on benchmark problems. In this work, we demonstrate that even better performance can be achieved using neural architecture search (NAS) combined with multimodal learning models. We adapt a NAS framework aimed at investigating image classification to the problem of software flaw detection and demonstrate improved results on the Juliet Test Suite, a popular benchmarking data set for measuring performance of machine learning models in this problem domain.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Dose Summation Strategies for External Beam Radiation Therapy and Brachytherapy in Gynecologic Malignancy: A Review from the NRG Oncology and NCTN Medical Physics Subcommittees

Definitive, nonsurgical management of gynecologic malignancies involves external beam radiation therapy (EBRT) and/or brachytherapy (BT). Summation of the cumulative dose is critical to assess the total biologic effective dose to targets and organs at risk. Cumulative dose calculation from EBRT and BT can be performed with or without image registration (IR) and biologic dose summation. Among these dose summation strategies, linear addition of dose-volume histogram (DVH) parameters without IR is the global standard for composite dose reporting. This approach stems from an era without image guidance and simple external beam and brachytherapy treatment approaches. With technological advances, EBRT and high-dose-rate BT have evolved to allow for volume-based treatment planning and delivery. Modern conformal therapeutic radiation involves volumetric or intensity modulated EBRT, capable of simultaneously treating multiple targets at different specified dose levels. Therefore, given the complexity of modern radiation treatment, the linear addition of DVH parameters from EBRT and high-dose-rate BT is challenging to represent the combined dose distribution. Deformable image registration (DIR) between EBRT and image guided brachytherapy (IGBT) data sets may provide a more nuanced calculation of multimodal dose accumulation. However, DIR is still nascent in this regard, and needs further development for accuracy and efficiency for clinical use. Biologic dose summation can combine physical dose maps from EBRT and each IGBT fraction, thereby generating a composite DVH from the biologic effective dose. However, accurate radiobiologic parameters are tissue-dependent and not well characterized. A combination of voxel-based DIR and biologic weighted dose maps may be the best approximation of dose accumulation but remains invalidated. The purpose of this report is to review dose summation strategies for EBRT and BT, including conventional equivalent dose in 2-Gy fractions dose summation without image registration, physical dose summation using 3-dimensional rigid IR and DIR, and biologic dose summation. We also provide general clinical workflows for IGBT with a focus on cervical cancer.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2021

Light Propagation Considerations For Internally Clad Sapphire Optical Fiber Using the 6Li(n,)3H Reaction

Optical frequency domain reflectometry measurements in internally clad single crystal sapphire fiber have received attention in recent years due to their high temperature distributed sensing potential. As work with these fibers has proceeded, there have been some inconsistencies in the results. Deeper investigation and testing has identified two critical considerations for the proper functioning of these fibers. First, users must address the propagation of multimode light along the outer surface of the fiber. Here, by observing the far-field image of an internally clad sapphire fiber when adding index matching fluid to the outer fiber surface, we demonstrate the effects of removing the higher order modes from these fibers. The addition of index matching fluid resulted in nearly single mode performance where multimode performance was previously observed. Second, users must address the effect that coupling the fiber to the interrogator via silica based fiber has on the internally clad sapphire fibers performance. Direct fusion splicing of silica to sapphire, as has been used in the recent work with these fibers, has a mode filtering effect which can be beneficial towards the modal behavior of the fibers. However, in this paper we demonstrate that the splicing can cause a sensing failure due to little or no low order mode light, that is useful for sensing, returning to the detector. The positive results from recent years have demonstrated that optical frequency domain reflectometry sensing performance will be successful in clad sapphire fiber; but only when the considerations described herein are addressed properly.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Unsupervised multimodal fusion of in-process sensor data for advanced manufacturing process monitoring

Effective monitoring of manufacturing processes is crucial for maintaining product quality and operational efficiency. Modern manufacturing environments often generate vast amounts of complementary multimodal data, including visual imagery from various perspectives and resolutions, hyperspectral data, and machine health monitoring information such as actuator positions, accelerometer readings, and temperature measurements. However, fusing and interpreting this complex, high-dimensional data presents significant challenges, particularly when labeled datasets are unavailable or impractical to obtain. This paper presents a novel approach to multimodal sensor data fusion in manufacturing processes, inspired by the Contrastive Language-Image Pre-training (CLIP) model. We leverage contrastive learning techniques to correlate different data modalities without the need for labeled data, overcoming limitations of traditional supervised machine learning methods in manufacturing contexts. Our proposed method demonstrates the ability to handle and learn encoders for five distinct modalities: visual imagery, audio signals, laser position (x and y coordinates), and laser power measurements. By compressing these high-dimensional datasets into low-dimensional representational spaces, our approach facilitates downstream tasks such as process control, anomaly detection, and quality assurance. The unsupervised nature of our method makes it broadly applicable across various manufacturing domains, where large volumes of unlabeled sensor data are common. We evaluate the effectiveness of our approach through a series of experiments, demonstrating its potential to enhance process monitoring capabilities in advanced manufacturing systems. This research contributes to the field of smart manufacturing by providing a flexible, scalable framework for multimodal data fusion that can adapt to diverse manufacturing environments and sensor configurations. The proposed method paves the way for more robust, data-driven decision-making in complex manufacturing processes.

Contrastive Learning↗
DOE OSTI2021

User defined intensity profile laser beam

A high-power laser beam with an arbitrary intensity profile is produced. Such beam has a variety of uses including for laser materials processing such as powder bed fusion additive manufacturing. Several challenges in additive manufacturing are mitigated with the present non-uniform intensity laser profiles. Nonuniform shapes include a set of intensity pixels in a line that could print a wide stripe area instead of just a single line. One example uses the multimode interference pattern from the output of a ribbon fiber which is imaged onto a work piece. The interference pattern is controlled to allow turning on or off of ‘pixels’ along a line which can be used to shape the beam and form the additively manufactured part.

Drachenberg, Derrek R.↗
DOE OSTI2024

Automated electrosynthesis reaction mining with multimodal large language models (MLLMs)

Leveraging the chemical data available in legacy formats such as publications and patents is a significant challenge for the community. Automated reaction mining offers a promising solution to unleash this knowledge into a learnable digital form and therefore help expedite materials and reaction discovery. However, existing reaction mining toolkits are limited to single input modalities (text or images) and cannot effectively integrate heterogeneous data that is scattered across text, tables, and figures. In this work, we go beyond single input modalities and explore multimodal large language models (MLLMs) for the analysis of diverse data inputs for automated electrosynthesis reaction mining. We compiled a test dataset of 65 articles (MERMES-T24 set) and employed it to benchmark five prominent MLLMs against two critical tasks: (i) reaction diagram parsing and (ii) resolving cross-modality data interdependencies. The frontrunner MLLM achieved ≥96% accuracy in both tasks, with the strategic integration of single-shot visual prompts and image pre-processing techniques. We integrate this capability into a toolkit named MERMES (multimodal reaction mining pipeline for electrosynthesis). Our toolkit functions as an end-to-end MLLM-powered pipeline that integrates article retrieval, information extraction and multimodal analysis for streamlining and automating knowledge extraction. This work lays the groundwork for the increased utilization of MLLMs to accelerate the digitization of chemistry knowledge for data-driven research.

Leong, Shi Xuan↗
DOE OSTI2021

Utilizing distributed acoustic sensing and ocean bottom fiber optic cables for submarine structural characterization

The sparsity of permanent seismic instrumentation in marine environments often limits the availability of subsea information on geohazards, including active fault systems, in both time and space. One sensing resource that provides observational access to the seafloor environment are existing networks of ocean bottom fiber optic cables; these cables, coupled to modern distributed acoustic sensing (DAS) systems, can provide dense arrays of broadband seismic observations capable of recording both seismic events and the ambient noise wavefield. Here, we report a marine DAS application which demonstrates the strength and limitation of this new technique on submarine structural characterization. Based on ambient noise DAS records on a 20 km section of a fiber optic cable offshore of Moss Landing, CA, in Monterey Bay, we extract Scholte waves from DAS ambient noise records using interferometry techniques and invert the resulting multimodal dispersion curves to recover a high resolution 2D shear-wave velocity image of the near seafloor sediments. We show for the first time that the migration of coherently scattered Scholte waves observed on DAS records can provide an approach for resolving sharp lateral contrasts in subsurface properties, particularly shallow faults and depositional features near the seafloor. Our results provide improved constraints on shallow submarine features in Monterey Bay, including fault zones and paleo-channel deposits, thus highlighting one of many possible geophysical uses of the marine cable network.

47 OTHER INSTRUMENTATION↗
DOE OSTI2026

Quantifying Operational Drivers of Multimodal Biometric Verification in Aerial Surveillance

Multimodal biometric verification is increasingly applied across operational contexts ranging from close-range security cameras and building-mounted surveillance to long-range ground sensors and unmanned aerial system (UAS) imagery. Variations in acquisition conditions—such as image resolution, viewing geometry, and motion artifacts—pose significant challenges for cross-domain algorithmic generalization. This study evaluates two independent multimodal biometric verification systems developed under the Intelligence Advanced Research Projects Activity (IARPA) Biometric Recognition and Identification at Altitude and Range (BRIAR) program, comparing performance on close-range and aerial datasets. Close-range video served as a baseline to quantify the decline in verification performance on aerial footage. The dataset included six UAS platforms, spanning small quadcopters at 10m altitude to medium-sized fixed-wing aircraft at 360m. Mixed-effects logistic regression identified image resolution (head and body pixel counts), head height, sensor characteristics, and algorithm selection as primary determinants of verification success, whereas demographic attributes and mission gait were not significant predictors. Activity type and collection site influenced performance in close-range data but had negligible impact on UAS imagery. These results clarify modality-specific strengths and limitations and highlight opportunities to enhance cross-domain biometric verification.

Peluso, Alina [ORNL] (ORCID:0000000328950406)↗
DOE OSTI2025

Using multimodal X-ray computed tomography to advance 3D petrography: A non-destructive investigation of olivine inside a carbonaceous chondrite

Rocks form in three dimensions through time and studying them provides information from inside dynamic systems we cannot otherwise observe. Yet how we typically access the interior of the rocks themselves to gain that information may limit our understanding and influence how we reconstruct the processes that formed them. Here, we demonstrate combined non-destructive 3D X-ray imaging techniques that produce quantitative densitometric and crystallographic maps of entire individual grains inside a rock. Olivine grains throughout a sample of the carbonaceous chondrite Northwest Africa (NWA) 11346 were each characterized by size, shape, composition, zoning intensity, and crystallographic orientation. The addition of 3D crystallographic mapping to calibrated 3D densitometric analysis—used to calculate chemical composition—demonstrates a fully non-destructive petrographic method and provides unique insight. For instance, in our case, using crystallographic data to delineate individual grains and then measuring the 3D size, shape, and composition of each distinguishes variably reset relict grains from those later crystallized after a melting event. Intersection in a 2D slice could not have led to this interpretation because the integration of three-dimensional size, rounding, composition, location, and crystallographic orientation measured from each grain forms the key patterns. Multimodal laboratory X-ray imaging has strong potential to advance 3D petrography.

36 MATERIALS SCIENCE↗