Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “multimodal imaging”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Multimodal photon and neutron imaging devices with fast event-by-event multisampling readout and optimized particle type discrimination

A particle imaging method for distinguishing between types of incident particles, such as neutrons, photons, and alphas, and improving the position resolution of particle imaging devices with matrix readout. The method includes high frequency multisampling readout electronics that provides the sequences of multiple measurements for each detected event, resulting in recorded detailed waveform information describing the signals. Such detailed information is used to approximate each signal waveform with a parameterized function in which the extracted parameter sets determine the type of the incident particle in an optimized fashion. The detailed event-by-event multisampling information for each signal readout channel in the matrix readout of the radiation imaging devices improves and optimizes the position resolution for variable shapes of the signals. Such devices can be used in mixed radiation fields, creating a new class of multimodal photon and neutron imagers.

Degtiarenko, Pavel V.↗
DOE OSTI2025

Exploring Capability of Multimodal Foundation Model for Image-based Fault Detection of Photovoltaic Modules

Multimodal Foundation Model (MFM), like ChatGPT and Gemini, have emerged as powerful tools for their exceptional natural language processing capabilities and their emerging potential in image analysis. This paper investigates the application of MFMs for photovoltaic (PV) fault detection through image analysis, focusing on ChatGPT 4.0 and Gemini 1.5 Pro. Three types of PV images and the corresponding common PV faults are detected: bird droppings using visible images, cell cracks via electroluminescence (EL) images, and hotspots using infrared (IR) images. Among the two models, Gemini 1.5 Pro demonstrated superior performance, achieving near-perfect results with an average F1 score of 0.97, consistently outperforming ChatGPT 4.0 in accuracy and reliability. Unlike traditional machine learning (ML) models, MFMs can operate in a zero shot manner that does not require additional training by the user, and the input images are not limited by size, angle, scope, or PV technology. The strong adaptability and user-friendliness make MFM a promising tool for analyzing PV images and advancing health monitoring for PV modules.

Li, Baojie↗
DOE OSTI2021

Universal image representation based on a multimodal graph

A system for classifying a target image with segments having attributes is provided. The system generates a graph for the target image that includes vertices representing segments of the image and edges representing relationships between the connected vertices. For each vertex, the system generates a subgraph that includes the vertex as a home vertex and neighboring vertices representing segments of the target image within a neighborhood of the segment represented by the home vertex. The system applies an autoencoder to each subgraph to generate latent variables to represent the subgraph. The system applies a machine learning algorithm to a feature vector comprising a universal image representation of the target image that is derived from the generated latent variables of the subgraphs to generate a classification for the target image.

Bremer, Peer-Timo↗
DOE OSTI2023

Universal image representation based on a multimodal graph

A system for classifying a target image with segments having attributes is provided. The system generates a graph for the target image that includes vertices representing segments of the image and edges representing relationships between the connected vertices. For each vertex, the system generates a subgraph that includes the vertex as a home vertex and neighboring vertices representing segments of the target image within a neighborhood of the segment represented by the home vertex. The system applies an autoencoder to each subgraph to generate latent variables to represent the subgraph. The system applies a machine learning algorithm to a feature vector comprising a universal image representation of the target image that is derived from the generated latent variables of the subgraphs to generate a classification for the target image.

Bremer, Peer-Timo↗
DOE OSTI2023

Deep Learning on Multimodal Chemical and Whole Slide Imaging Data for Predicting Prostate Cancer Directly from Tissue Images

Prostate cancer is one of the most common cancers globally and is the second most common cancer in the male population in the US. Here we develop a study based on correlating the hematoxylin and eosin (H&E)-stained biopsy data with MALDI mass-spectrometric imaging data of the corresponding tissue to determine the cancerous regions and their unique chemical signatures and variations of the predicted regions with original pathological annotations. We obtain features from high-resolution optical micrographs of whole slide H&E stained data through deep learning and spatially register them with mass spectrometry imaging (MSI) data to correlate the chemical signature with the tissue anatomy of the data. We then use the learned correlation to predict prostate cancer from observed H&E images using trained coregistered MSI data. This multimodal approach can predict cancerous regions with ~80% accuracy, which indicates a correlation between optical H&E features and chemical information found in MSI. Further, we show that such paired multimodal data can be used for training feature extraction networks on H&E data which bypasses the need to acquire expensive MSI data and eliminates the need for manual annotation saving valuable time. Two chemical biomarkers were also found to be predicting the ground truth cancerous regions. This study shows promise in generating improved patient treatment trajectories by predicting prostate cancer directly from readily available H&E-stained biopsy images aided by coregistered MSI data.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

Direct Multimodal Nanoscale Visualization of Early Phosphorus-Based Antiwear Tribofilm Formation

Understanding the mechanism of antiwear (AW) tribofilm formation and how to tune surface chemistry to control functionality is essential for the development of the next generation of oil lubricants. In particular, understanding and optimizing early AW tribofilm formation can increase the energy efficiency of mechanical systems. However, the mechanism for how these films form is not well understood. The majority of prior work has focused on analyzing only end-of-test surfaces long after the film has formed. In this work, we develop an in situ multimodal chemical imaging methodology to directly visualize the early formation of AW films on steel surfaces. We investigate an oil formulation containing a phosphorus-based additive commonly used to protect surfaces from wear and fatigue processes in machine elements, such as gears, bearings, and sliding contacts. Using nanoscale multimodal chemical imaging on combined platforms of atomic force microscopy (AFM) coupled directly with in situ nano-infrared (nano-IR) spectroscopy, and further combined ex situ with time-of-flight secondary ion mass spectrometry (ToF-SIMS), we demonstrate a direct correlation between changes in friction and local surface chemistry. In these experiments, the AFM probe acts as a single asperity contact to generate the tribofilm as well as a tool to analyze it in situ as it is forming. To verify our in situ measurements, we compare these results to the ex situ ToF-SIMS of macroscale block-on-ring tribometer-formed samples. We report the understanding gained here on how AW films form and how film properties can be modified by tuning the chemistry of the additives will facilitate developing transmission fluids to meet increasing demands for vehicle performance and efficiency.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Applying Multimodal Mass Spectrometry to Image Tumors Undergoing Ferroptosis Following In Vivo Treatment with a Ferroptosis Inducer

Epithelial ovarian cancer (EOC) is the most common form of ovarian cancer. The poor prognosis generally associated with this disease has led to the search for improved therapies such as ferroptosis-inducing agents. Ferroptosis is a form of regulated cell death that is dependent on iron and is characterized by lipid peroxidation. Precise mapping of lipids and iron within tumors exposed to ferroptosis-inducing agents may provide insight into processes of ferroptosis in vivo and ultimately assist in the optimal deployment of ferroptosis inducers in cancer therapy. In this work, we present a method for combining matrix-assisted laser desorption/ionization (MALDI) mass spectrometry imaging (MSI) with secondary ion mass spectrometry (SIMS) to analyze changes in spatial lipidomics and metal composition, respectively, in ovarian tumors following exposure to a ferroptosis inducer. Tumors were obtained by injecting human ovarian cancer tumor-initiating cells into mice, followed by treatment with the ferroptosis inducer erastin. SIMS imaging detected iron accumulation in the tumor tissue, and sequential MALDI-MS imaging of the same tissue section displayed two chemically distinct regions of lipids. Additionally, one region was associated with the iron-rich area detected with SIMS, and the other region encompassed the remainder of the tissue section. Bulk lipidomics confirmed the lipid assignments putatively assigned from the MALDI-MS data. Overall, we demonstrate the ability of multimodal MSI to identify the spatial locations of iron and lipids in the same tissue section and associate these regions with clinical pathology.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2024

The POINTER Imaging baseline cohort: Associations between multimodal neuroimaging biomarkers, cardiovascular health, and cognition

Abstract INTRODUCTION The U.S. Study to Protect Brain Health Through Lifestyle Intervention to Reduce Risk (U.S. POINTER) is evaluating lifestyle interventions in older adults at risk for cognitive decline and dementia. Here we characterize the baseline data set of the POINTER Imaging ancillary study. METHODS Participants underwent health and cognitive assessments and neuroimaging with multimodal positron emission tomography (PET) (beta‐amyloid [Aβ] and tau) and magnetic resonance imaging (MRI). Framingham risk score (FRS) was used to quantify cardiovascular disease (CVD) risk. RESULTS A total of 1052 participants (31% from underrepresented ethnoracial groups) were enrolled. Compared to Aβ−, Aβ+ (29%) participants were older, had higher apolipoprotein E (APOE) ε4 carriage rate and white matter hyperintensity volume, and greater temporal tau. FRS was related to MRI measures, but not AD biomarkers. FRS and tau had independent effects on cognition. DISCUSSION In this heterogenous, at‐risk cohort, CVD risk was related to more abnormal brain structure and poorer cognition, representing a putative non‐AD (Alzheimer's disease) pathway to brain injury and cognitive decline. Highlights The U.S. Study to Protect Brain Health Through Lifestyle Intervention to Reduce Risk (U.S. POINTER) cohort is enriched for cardiovascular disease (CVD) and poor lifestyle POINTER Imaging collected multimodal neuroimaging data in this unique, at‐risk cohort Amyloid burden was related to age, apolipoprotein E (APOE) ε4 carriage, and measures of disease progression Associations between amyloid and tau, and tau and cognition, were relatively weak CVD risk and tau pathology were independently related to memory

Neurosciences & Neurology↗
DOE OSTI2020

Toward nanoscale molecular mass spectrometry imaging via physically constrained machine learning on co-registered multimodal data

Mass spectrometry imaging (MSI) plays a pivotal role in investigating the chemical nature of complex systems that underly our understanding in biology and medicine. Multiple fields of life science such as proteomics, lipidomics and metabolomics benefit from the ability to simultaneously identify molecules and pinpoint their distribution across a sample. However, achieving the necessary submicron spatial resolution to distinguish chemical differences between individual cells and generating intact molecular spectra is still a challenge with any single imaging approach. Here, we developed an approach that combines two MSI techniques, matrix-assisted laser desorption/ionization (MALDI) and time-of-flight secondary ion mass spectrometry (ToF-SIMS), one with low spatial resolution but intact molecular spectra and the other with nanometer spatial resolution but fragmented molecular signatures, to predict molecular MSI spectra with submicron spatial resolution. The known relationships between the two MSI channels of information are enforced via a physically constrained machine-learning approach and directly incorporated in the data processing. We demonstrate the robustness of this method by generating intact molecular MALDI-type spectra and chemical maps at ToF-SIMS resolution when imaging mouse brain thin tissue sections. This approach can be readily adopted for other types of bioimaging where physical relationships between methods have to be considered to boost the confidence in the reconstruction product.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Small animal PET: a review of what we have done and where we are going

Abstract Small animal research is an essential tool in studying both pharmaceutical biodistribution and disease progression over time. Furthermore, through the rapid development of in vivo imaging technology over the last few decades, small animal imaging (also referred to as preclinical imaging) has become a mainstay for all fields of biologic research and a center point for most preclinical cancer research. Preclinical imaging modalities include optical, MRI and MRS, microCT, small animal PET, ultrasound, and photoacoustic, each with their individual strengths. The strong points of small animal PET are its translatability to the clinic; its quantitative imaging capabilities; its whole-body imaging ability to dynamically trace functional/biochemical processes; its ability to provide useful images with only nano- to pico- molar concentrations of administered compounds; and its ability to study animals serially over time. This review paper gives an overview of the development and evolution of small animal PET imaging. It provides an overview of detector designs; system configurations; multimodality PET imaging systems; image reconstruction and analysis tools; and an overview of research and commercially available small animal PET systems. It concludes with a look toward developing technologies/methodologies that will further enhance the impact of small animal PET imaging on medical research in the future.

Engineering↗
DOE OSTI2023

Multimodal single-Cell/ Particle imaging and engineering for energy conversion in bacteria (Final Technical Report)

Hybrid inorganic-microbial systems have emerged as a potentially transformative approach to combine the light-harvesting capability of inorganic semiconductors and the ability of microbes to orchestrate complex chemical transformations. The objective of this collaborative research is to combine quantum materials synthesis, bacterial synthetic biology, and multimodal single-entity imaging to quantitatively study how hybrid QD-bacteria systems convert light to value chemicals at the single-to-sub cell level, with the ultimate goal of gaining insights to guide the engineering of QDs and bacterial genetics for more efficient bioenergy conversion. The final technical report summarizes our achievements toward this objective.

09 BIOMASS FUELS↗
DOE OSTI2025

Editorial overview: Unlocking the secrets of nongenetic plasticity, one cell at a time

Cellular noise, the non-genetic variability observed among isogenic cells, arises from factors such as growth conditions, aging, and stochastic gene expression, influencing cell stress-response, metabolism, morphology, and size. Here, such plasticity, while critical for adaptation, often goes unnoticed with traditional population-averaging biotechnologies that inevitably mask cell-specific variations and prompting the question, "What else might we be missing". However, recent breakthroughs in optical imaging, microfluidics, and omics, are beginning to uncover the complexity of cellular plasticity. This special is-sue highlights some of these breakthroughs, with key contributions including innovations in multimodal chemical imaging, label-free microscopy, spatial and temporal omics, and droplet-based microfluidics. Collectively, these cutting-edge tools provide unprecedent-ed insights into non-genetic cell-to-cell variability, enhancing our understanding of cellu-lar plasticity and its implications for health, energy, and ecology.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

X-ray microscopy enables multiscale high-resolution 3D imaging of plant cells, tissues, and organs

Capturing complete internal anatomies of plant organs and tissues within their relevant morphological context remains a key challenge in plant science. While plant growth and development are inherently multiscale, conventional light, fluorescence, and electron microscopy platforms are typically limited to imaging of plant microstructure from small flat samples that lack a direct spatial context to, and represent only a small portion of, the relevant plant macrostructures. We demonstrate technical advances with a lab-based X-ray microscope (XRM) that bridge the imaging gap by providing multiscale high-resolution three-dimensional (3D) volumes of intact plant samples from the cell to the whole plant level. Serial imaging of a single sample is shown to provide sub-micron 3D volumes co-registered with lower magnification scans for explicit contextual reference. High-quality 3D volume data from our enhanced methods facilitate sophisticated and effective computational segmentation. Advances in sample preparation make multimodal correlative imaging workflows possible, where a single resin-embedded plant sample is scanned via XRM to generate a 3D cell-level map, and then used to identify and zoom in on sub-cellular regions of interest for high-resolution scanning electron microscopy. In total, we present the methodologies for use of XRM in the multiscale and multimodal analysis of 3D plant features using numerous economically and scientifically important plant systems.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Amorphous Ag 2-x Cu x S quantum dots: “all-in-one” theranostic nanomedicines for near-infrared fluorescence/photoacoustics dual-modal-imaging-guided photothermal therapy

Integrating dual-modal imaging and photothermal effect within a single nanosystem is an enormous challenge for current nanomedicine application. In this study, non-toxic amorphous Ag 2-x Cu x S quantum dots (QDs) possessing near-infrared fluorescence (NIRF), enhanced photothermal and photoacoustic (PA) performance, as “one-in-all” nanomedicines, are designed and obtained. Cu doping in amorphous Ag 2 S QDs causes red-shift of NIRF to the lowest tissue absorption wavelength in so called “first biowindow” (~820 nm) and improves photothermal conversion efficiency to 44.0% simultaneously. Density functional theory simulation results suggest the fluorescence red-shift and enhanced photothermal/photoacoustics properties can be attributed to the generation of intragap states introduced by Cu doping. The amorphous Ag 2-x Cu x S QDs exhibit high long-term biocompatibility. Under a single laser irradiation with relatively low power density, an effective tumor ablation induced by high photothermal effect of amorphous Ag 2-x Cu x S QDs is achieved in vivo. Simultaneously, the PTT process can be guided by high quality NIRF/PA dual-modal-imaging.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Electron Energy-Loss Spectroscopy and Differential Phase Contrast Imaging with Active Decision in Multimodal Electron Microscopy: Isotopic detection at the atomic scale

Isotopic engineering provides a powerful route to control phonon behavior in crystalline solids, enabling fundamental studies of lattice dynamics and heat transport at the atomic scale. Here, we directly visualize isotope-dependent phonon propagation in epitaxial Cr 2 O 3 using aberration-corrected scanning transmission electron microscopy (STEM) combined with monochromated, high-energy-resolution electron energy-loss spectroscopy (EELS). Guided by ab initio phonon calculations, we demonstrate that optical phonon modes above 70 meV are predominantly oxygen-derived and exhibit measurable redshifts upon substitution of natural 16 O by enriched 18 O. Spatially resolved vibrational spectrum imaging reveals isotope-enriched tracer layers within Cr 2 O 3 thin films, correlating isotope concentration with phonon intensity variations and vibrational energy shifts. At the nanometer and atomic scales, vibrational EELS mapping uncovers coherent phonon propagation across isotopic interfaces, consistent with theoretical phonon density of states and dispersion relations. These results establish vibrational EELS as a quantitative probe for isotope-dependent phonon transport in materials, opening new possibilities for studying energy dissipation and lattice dynamics.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Graphene Electric Field Sensor Enables Single Shot Label-Free Imaging of Bioelectric Potentials

The measurement of electrical activity across systems of excitable cells underlies current progress in neuroscience, cardiac pharmacology, and neurotechnology. However, bioelectricity spans orders of magnitude in intensity, space, and time, posing substantial technological challenges. The development of methods permitting network-scale recordings with high spatial resolution remains key to studies of electrogenic cells, emergent networks, and bioelectric computation. Here, for this work, we demonstrate single-shot and label-free imaging of extracellular potentials with high resolution across a wide field-of-view. The critically coupled waveguide-amplified graphene electric field (CAGE) sensor leverages the field-sensitive optical transitions in graphene to convert electric potentials into the optical regime. As a proof-of-concept, we use the CAGE sensor to detect native electrical activity from cardiac action potentials with tens-of-microns resolution, simultaneously map the propagation of these potentials at tissue-scale, and monitor their modification by pharmacological agents. This platform is robust, scalable, and compatible with existing microscopy techniques for multimodal correlative imaging.

47 OTHER INSTRUMENTATION↗