Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “multimodal network”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Resilient information and inference networks under mixed-trust sensing

With ubiquitous digitization, sensing, and computational intelligence deployed in increasingly more and broader domains, including critical infrastructure, potentially misleading and destabilizing effects of multimodal anomalies and adversarial behavior are growing in importance. Here, we develop randomized and reinforcement learning-based strategies for strategically recruiting and utilizing deployed (and, thus, vulnerable and potentially faulty and/or compromised) nodes from information and inference networks, while defending against adversaries that attempt to misguide assessments of inferred variables. Recognizing that, besides communication and other costs, sampling from any observable node can either provide true data or dangerously expose our inference to misinformation (without being easily distinguishable what actually happens), the proposed strategies proceed by progressively recruiting nodes and cautiously scaling their information contribution based on assumed, or, in our reinforcement learning approach, intelligently weighed trustworthiness, with the learning approach also considering network-wide, threat-inclusive risk/value tradeoffs. While avoiding the hardware, communication, analytical and computational burden of explicit redundancy, the proposed defensive schemes enable on-the-fly assessments of underlying processes, and system-wide situational awareness with demonstrable resilience against adversarial activities.

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Multimode Strong Coupling in Cavity Optomechanics

Optomechanical systems show great potential as quantum transducers and information storage devices for use in future hybrid quantum networks. In this context, optomechanical strong coupling can enable efficient, high-bandwidth, and deterministic transfer of quantum states. While optomechanical strong coupling has been realized at optical frequencies, it has proven difficult to identify a robust optomechanical system that features the low loss and high coupling rates required for more sophisticated control of mechanical motion. In this paper, we demonstrate strong coupling in a Brillouin-based bulk cavity optomechanical system in both the single-mode and the multimode strong-coupling regime, which leads to a useful device both for applications in quantum information and for investigating decoherence phenomena in bulk acoustic wave resonators. Using nontrivial mode hybridizations in the strong-coupling regime, we create hybridized photonic-phononic modes with lifetimes that are significantly longer than those of the uncoupled system. This surprising lifetime enhancement, which results from the interference of decay channels, showcases the use of multimode strong coupling as a general strategy to control extrinsic decoherence mechanisms. Moreover, phonons supported by such bulk-acoustic-wave resonators have a collection of properties, including high frequencies, long coherence times, and robustness against thermal decoherence, that make this optomechanical system particularly enticing for applications such as quantum transduction and memories. Hence, this system provides access to phenomena in a previously unexplored regime of optomechanical interactions and could serve as an important building block for future quantum devices.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Investigating permafrost carbon dynamics in Alaska with artificial intelligence

Abstract Positive feedbacks between permafrost degradation and the release of soil carbon into the atmosphere impact land–atmosphere interactions, disrupt the global carbon cycle, and accelerate climate change. The widespread distribution of thawing permafrost is causing a cascade of geophysical and biochemical disturbances with global impacts. Currently, few earth system models account for permafrost carbon feedback (PCF) mechanisms. This research study integrates artificial intelligence (AI) tools and information derived from field-scale surveys across the tundra and boreal landscapes in Alaska. We identify and interpret the permafrost carbon cycling links and feedback sensitivities with GeoCryoAI, a hybridized multimodal deep learning (DL) architecture of stacked convolutionally layered, memory-encoded recurrent neural networks (NN). This framework integratesin-situmeasurements and flux tower observations for teacher forcing and model training. Preliminary experiments to quantify, validate, and forecast permafrost degradation and carbon efflux across Alaska demonstrate the fidelity of this data-driven architecture. More specifically, GeoCryoAI logs the ecological memory and effectively learns covariate dynamics while demonstrating an aptitude to simulate and forecast PCF dynamics—active layer thickness (ALT), carbon dioxide flux (CO 2 ), and methane flux (CH 4 )—with high precision and minimal loss (i.e. ALT RMSE : 1.327 cm [1969–2022]; CO 2 RMSE : 0.697µmolCO 2 m −2 s −1 [2003–2021]; CH 4 RMSE : 0.715 nmolCH 4 m −2 s −1 [2011–2022]). ALT variability is a sensitive harbinger of change, a unique signal characterizing the PCF, and our model is the first characterization of these dynamics across space and time.

Environmental Sciences & Ecology↗
DOE OSTI2023

Pan-cancer evaluation of clinical value of mitotic network activity index (MNAI) and its predictive value for immunotherapy

Increased mitotic activity is associated with the genesis and aggressiveness of many cancers. To assess the clinical value of mitotic activity as prognostic biomarker, we performed a pan-cancer study on the mitotic network activity index (MNAI) constructed based on 54-gene mitotic apparatus network. Our pan-cancer assessment on TCGA (33 tumor types, 10,061 patients) and validation on other publicly available cohorts (23 tumor types, 9,209 patients) confirmed the significant association of MNAI with overall survival, progression-free survival, and other prognostic endpoints in multiple cancer types, including lower-grade gliomas (LGG), breast invasive carcinoma (BRCA), as well as many others. We also showed significant association between MNAI and genetic instability, which provides a biological explanation of its prognostic impact at pan-cancer landscape. Our association analysis revealed that patients with high MNAI benefitted more from anti-PD-1 and Anti-CTLA-4 treatment. In addition, we demonstrated that multimodal integration of MNAI and the AI-empowered Cellular Morphometric Subtypes (CMS) significantly improved the predictive power of prognosis compared to using MNAI and CMS alone. Our results suggest that MNAI can be used as a potential prognostic biomarker for different tumor types toward different clinical endpoints, and multimodal integration of MNAI and CMS exceeds individual biomarker for precision prognosis.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2024

Parallel measurement of transcriptomes and proteomes from same single cells using nanodroplet splitting

Single-cell multiomics provides comprehensive insights into gene regulatory networks, cellular diversity, and temporal dynamics. Here, we introduce nanoSPLITS (nanodroplet SPlitting for Linked-multimodal Investigations of Trace Samples), an integrated platform that enables global profiling of the transcriptome and proteome from same single cells via RNA sequencing and mass spectrometry-based proteomics, respectively. Benchmarking of nanoSPLITS demonstrates high measurement precision with deep proteomic and transcriptomic profiling of single-cells. We apply nanoSPLITS to cyclin-dependent kinase 1 inhibited cells and found phospho-signaling events could be quantified alongside global protein and mRNA measurements, providing insights into cell cycle regulation. We extend nanoSPLITS to primary cells isolated from human pancreatic islets, introducing an efficient approach for facile identification of unknown cell types and their protein markers by mapping transcriptomic data to existing large-scale single-cell RNA sequencing reference databases. Accordingly, we establish nanoSPLITS as a multiomic technology incorporating global proteomics and anticipate the approach will be critical to furthering our understanding of biological systems.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

Data Fusion for the Development of a Multimodal Freight Transload Facilities Dataset in the U.S.

To withstand the growing demand of commodity volume and its strain on the transportation infrastructure, it is necessary to identify the flow of commodities by route and mode. However, a national multimodal freight routing model does not exist for the U.S. The development of such model requires multiple building blocks, such as virtual representations of roadway, railway, and waterway networks, transload facilities (TFs), and access/egress links. Most of these blocks have a robust database in the U.S., except for the TFs. Here, this paper presents the fusion of dispersed and heterogeneous representations of multimodal TFs into a single, comprehensive, geospatial freight TF dataset. The TF dataset is derived from several sources, including the U.S. Army Corps of Engineers Master Docks Plus, the National Transportation Atlas Database, the Intermodal Association of North America, industry publications, and other public information. First, individual datasets were queried and reconciled. A geocoding/reverse geocoding process was applied to get the best street address and latitude/longitude location for each terminal. Then, duplicate terminals were identified by a fuzzy match algorithm based on terminal name and location, and removed. Validation was performed by visual inspection of random facilities. The main contributions of this work are: a publicly available version of the TF dataset, including facility location and multimodal transfer capability of 9,003 facilities, and an enterprise-version with the same facilities but including commodity handling capabilities. The main purpose of developing the TF dataset is to inform multimodal routing algorithms. The proposed TF dataset allows for credibly modeling the multimodal transfer of commodities within shipment routes.

Commodity Routing↗
DOE OSTI2025

Phylogenomic discovery and engineering of nitrogen fixation into the bioenergy woody crop poplar

Biological nitrogen fixation (BNF) is a key process enabling plants in specific lineages to convert atmospheric dinitrogen (N₂) into bioavailable ammonia through symbioses with diazotrophic microbes. Expanding this capability beyond native nitrogen-fixing clades into non-nodulating crops would reduce synthetic fertilizer use, lowering energy inputs and environmental impacts in agriculture. Supported by DOE Funding Award DE-SC0018247, the NitFix project advanced foundational knowledge required to engineer root-nodule symbioses in new host species. The team generated the most comprehensive phylogenomic analysis to date of all known nodulating lineages, resolving the evolutionary history of nitrogen-fixing symbiosis and identifying core gene suites retained across nodulating taxa. Through multimodal genomics, transcriptomics, and functional analyses in Medicago truncatula and related species, the project mapped regulatory networks underlying nodule organogenesis, bacterial infection, and nitrogen-fixation efficiency. Key discoveries include the identification of conserved signaling modules for rhizobial recognition, transcription factors controlling nodule differentiation, and metabolic pathways integrating fixed nitrogen into plant growth. The project also developed enabling tools—including optimized transformation pipelines, gene-editing workflows, and imaging-based phenotyping—to accelerate engineering efforts in emerging models. Together, these results refine the mechanistic framework of symbiotic nitrogen fixation and highlight transferable components essential for rewiring these traits into non-nodulating crops.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Accurate, Uncertainty-Aware Classification of Molecular Chemical Motifs from Multimodal X-ray Absorption Spectroscopy

Accurate classification of molecular chemical motifs from experimental measurement is an important problem in molecular physics, chemistry, and biology. In this work, we present neural network ensemble classifiers for predicting the presence (or lack thereof) of 41 different chemical motifs on small molecules from simulated C, N, and O K-edge X-ray absorption near-edge structure (XANES) spectra. Our classifiers not only achieve class-balanced accuracies of more than 0.95 but also accurately quantify uncertainty. Here, we also show that including multiple XANES modalities improves predictions notably on average, demonstrating a “multimodal advantage” over any single modality. In addition to structure refinement, our approach can be generalized to broad applications with molecular design pipelines.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Ultrafast CMOS image sensors and data-enabled super-resolution for multimodal radiographic imaging and tomography

We summarize recent progress in ultrafast Complementary Metal Oxide Semiconductor (CMOS) image sensor development and the application of neural networks for post-processing of CMOS and charge-coupled device (CCD) image data to achieve sub-pixel resolution (thus ‘super-resolution’). The combination of novel CMOS pixel designs and data-enabled image post-processing provides a promising path towards ultrafast high-resolution multi-modal radiographic imaging and tomography applications.

47 OTHER INSTRUMENTATION↗
DOE OSTI2023

Ultrafast CMOS image sensors and data-enabled super-resolution for multimodal radiographic imaging and tomography

We summarize recent progress in ultrafast Complementary Metal Oxide Semiconductor (CMOS) image sensor development and the application of neural networks for post-processing of CMOS and charge-coupled device (CCD) image data to achieve sub-pixel resolution (thus $super$-$resolution$). The combination of novel CMOS pixel designs and data-enabled image post-processing provides a promising path towards ultrafast high-resolution multi-modal radiographic imaging and tomography applications.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

A guide to the BRAIN Initiative Cell Census Network data ecosystem

Characterizing cellular diversity at different levels of biological organization and across data modalities is a prerequisite to understanding the function of cell types in the brain. Classification of neurons is also essential to manipulate cell types in controlled ways and to understand their variation and vulnerability in brain disorders. The BRAIN Initiative Cell Census Network (BICCN) is an integrated network of data-generating centers, data archives, and data standards developers, with the goal of systematic multimodal brain cell type profiling and characterization. Emphasis of the BICCN is on the whole mouse brain with demonstration of prototype feasibility for human and nonhuman primate (NHP) brains. Here, we provide a guide to the cellular and spatial approaches employed by the BICCN, and to accessing and using these data and extensive resources, including the BRAIN Cell Data Center (BCDC), which serves to manage and integrate data across the ecosystem. We illustrate the power of the BICCN data ecosystem through vignettes highlighting several BICCN analysis and visualization tools. Finally, we present emerging standards that have been developed or adopted toward Findable, Accessible, Interoperable, and Reusable (FAIR) neuroscience. The combined BICCN ecosystem provides a comprehensive resource for the exploration and analysis of cell types in the brain.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

Advancing Industry 4.0: Multimodal Sensor Fusion for AI-Based Fault Detection in 3D Printing

Additive manufacturing, particularly fused deposition modeling, is transforming modern production by enabling rapid prototyping and complex part fabrication. However, its layer-by-layer process remains vulnerable to faults such as nozzle clogging, filament runout, and layer misalignment, which compromise print quality and reliability. Traditional inspection methods are costly, time-intensive, and often limited to post-process analysis, making them unsuitable for real-time intervention. In this current study, the authors developed a novel, low-cost, and portable faultdetection system that leverages multimodal sensor fusion and artificial intelligence for real-time monitoring in FDM-based 3D printing. The system integrates acoustic, vibration, and thermal sensing into a non-intrusive architecture, capturing complementary data streams that reflect both mechanical and process-related anomalies. Acoustic and thermal sensors operate in a fully contactless manner, while the vibration sensor requires minimal attachment such that it will not interfere with printer hardware, thereby preserving portability and ease of deployment. The multimodal signals are processed into spectrograms and time-frequency features, which are classified using convolutional neural networks for intelligent fault detection. The proposed system advances Industry 4.0 objectives by offering an affordable, scalable, and practical monitoring solution that improves faultdetection accuracy, reduces waste, and supports sustainable, adaptive manufacturing.

42 ENGINEERING↗
DOE OSTI2023

In planta single-molecule imaging and holographic force spectroscopy to study real-time, multimodal turnover dynamics of polysaccharides and associated carbohydrate metabolites

Cellular production of polysaccharides from simple sugar molecules serves critical roles in a variety of metabolic processes essential to the survival of every living organism. In the context of photosynthetically fixed carbon, cell wall polysaccharides synthesis is particularly of interest for increased biomass accumulation for human food, animal feed, and bioenergy related applications. Despite a long history of research in plant cell biology, our understanding of in planta cell wall biosynthesis and its regulation is far from complete due to lack of a comprehensive microscopy toolkit that encompass the multiple spatial and temporal scales in which cell wall polysaccharides fibrils are synthesized and assembled into intricate cell wall networks. In this project, a multidisciplinary team of scientists from Rutgers University, Vanderbilt University, and Oak Ridge National Laboratory carried out innovative multimodal, multiscale microscopy studies of cell wall synthesis using single-molecule force-spectroscopy, super-resolution fluorescence microscopy, and in vivo live cell imaging. Our novel multimodal and holistic imaging approach revealed plant cell wall polysaccharides synthesis processes in unprecedented detail across multiple spatiotemporal scales, from a single-molecule to a single-cell. Besides, the team has made technical and scientific innovations across multiple research fields—microscopy, bioengineering, single-molecule biophysics, and plant biology, etc.—to accomplish the goals. The results from this project will greatly advance the mechanistic and holistic understanding of in planta cell wall synthesis, which will accelerate the development of better transgenic crops for bioenergy applications. Moreover, the new toolbox, combining powerful advanced microscopy assays with cell/protein engineering, will have broader impacts on molecular and cellular biology fields by paving the way for studying cellular processes occurring across multiple physical scales with multimodal microscopy methodologies.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Multimodal sensor fusion framework for residential building occupancy detection

For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This work proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine's (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms' performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Automated Grain Boundary (GB) Segmentation and Microstructural Analysis in 347H Stainless Steel Using Deep Learning and Multimodal Microscopy

Austenitic 347H stainless steel offers superior mechanical properties and corrosion resistance required for extreme operating conditions such as high temperature. The change in microstructure due to composition and process variations is expected to impact material properties. Identifying microstructural features such as grain boundaries thus becomes an important task in the process-microstructure-properties loop. Applying convolutional neural network (CNN)-based deep learning models is a powerful technique to detect features from material micrographs in an automated manner. In contrast to microstructural classification, supervised CNN models for segmentation tasks require pixel-wise annotation labels. However, manual labeling of the images for the segmentation task poses a major bottleneck for generating training data and labels in a reliable and reproducible way within a reasonable timeframe. Microstructural characterization especially needs to be expedited for faster material discovery by changing alloy compositions. Here, in this study, we attempt to overcome such limitations by utilizing multimodal microscopy to generate labels directly instead of manual labeling. We combine scanning electron microscopy images of 347H stainless steel as training data and electron backscatter diffraction micrographs as pixel-wise labels for grain boundary detection as a semantic segmentation task. The viability of our method is evaluated by considering a set of deep CNN architectures. We demonstrate that despite producing instrumentation drift during data collection between two modes of microscopy, this method performs comparably to similar segmentation tasks that used manual labeling. Additionally, we find that naïve pixel-wise segmentation results in small gaps and missing boundaries in the predicted grain boundary map. By incorporating topological information during model training, the connectivity of the grain boundary network and segmentation performance is improved. Finally, our approach is validated by accurate computation on downstream tasks of predicting the underlying grain morphology distributions which are the ultimate quantities of interest for microstructural characterization.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Attention-based explainability for structure–property relationships

Machine learning methods are emerging as a universal paradigm for constructing correlative structure–property relationships in materials science based on multimodal characterization. However, this necessitates the development of methods for the physical interpretability of the resulting correlative models. Here, we demonstrate the potential of attention-based neural networks for revealing structure–property relationships and the underlying physical mechanisms, using the ferroelectric properties of PbTiO3 thin films as a case study. Through the analysis of attention scores, we disentangle the influence of distinct domain patterns on the polarization switching process. The attention-based Transformer model is explored both as a direct interpretability tool and as a surrogate for explaining representations learned via unsupervised machine learning, enabling the identification of physically grounded correlations. We compare attention-derived interpretability scores with classical SHapley Additive exPlanations analysis and show that, in contrast to applications in natural language processing, attention mechanisms in materials science exhibit high efficiency in highlighting meaningful structural features.

Slautin, Boris [Independent Researcher]↗
DOE OSTI2021

Striving to translate shale physics across ten orders of magnitude: What have we learned?

Shales will play an important role in the successful transition of energy from fossil-based resources to renewables in the coming decades. Aside from being a significant source of low-carbon intensity fuels, like natural gas, they also serve as geologic seals of subsurface formations that may be used to isolate nuclear waste, sequester CO 2 , or store intermittent energy (e.g., solar hydrogen). Despite their importance, shales pose significant engineering and environmental challenges due to their nanoporous structure and extreme heterogeneity that spans at least ~10 orders of magnitude in spatial scale. Two challenges inhibit a system-level understanding: (1) the physics of fluid flow and phase behavior in shales are poorly understood due to the dominant molecular interactions between minerals and fluids under confinement, and (2) the apparent lack of scale separation that prevents a reliable (closed) description of the physics at any single scale of observation. In this review, we focus on the latter issue and discuss scale translation, which in its broadest sense is transforming data or simulations from one spatiotemporal scale to another. While effective scale translation is not exclusive to shales, but all geologic porous media, the need for it is especially acute in shales given their high degree of heterogeneity. Classical theories like homogenization, while indispensable, fail when scales are not separated. Other methods, like numerical upscaling, scale-translate in only one direction: small to large, but not the reverse, called downscaling. However, the confluence of advances in three areas are bringing challenging problems such as shales within reach: increased computational power and scalable algorithms; high-resolution imaging and multi-modal data acquisition; and machine learning to process massive amounts of data. While these advances equip geoscientists with a wide array of experimental and computational tools, no individual tool can probe the entire gamut of heterogeneity in shales. Their effective use, therefore, requires an ability to bridge between various data types obtained at different scales. The aim of this review is to present a coherent account of computational and experimental methods that may be used to achieve just that, i.e., to perform scale translation. We provide a broader definition of scale translation, one that transcends classical homogenization and upscaling methods, but is consistent with them and accommodates notions like downscaling and data translation. After a brief introduction to homogenization, we review hybrid methods, numerical upscaling and its recent extensions, multiscale computing, high-resolution imaging, and machine learning. We place particular emphasis on multiscale computing and propose an algorithmic framework to bridge between the pore (micro) and Darcy (macro) scales. Throughout the paper, we draw comparisons between the various methods and highlight their (often hidden) similarities, differences, benefits, and pitfalls. We finally conclude with two case studies on shales that exemplify some of the methods presented.

58 GEOSCIENCES↗
DOE OSTI2023

Parallel measurement of transcriptomes and proteomes from same single cells using nanodroplet splitting

Single-cell multiomics provides comprehensive insights into gene regulatory networks, cellular diversity, and temporal dynamics. While tools for co-profiling single-cell genomes, transcriptomes, and epigenomes are available, accessing proteomes in parallel is more challenging. We developed nanoSPLITS (nanodroplet SPlitting for Linked-multimodal Investigations of Trace Samples), an integrated platform that enables global profiling of the transcriptome and proteome from same single cells using RNA sequencing and mass spectrometry-based proteomics, respectively. nanoSPLITS can precisely quantify over 5000 genes, 2000 proteins, and 140 phosphopeptides per single cell and identify candidate cell markers from these modalities. By exploring Cdk1-mediated cell cycle arrest, we demonstrate how nanoSPLITS single-cell multiomics can provide comprehensive cellular characterization with insights into covarying protein/gene clusters, unique phosphorylation events, and mitotic pathways.

59 BASIC BIOLOGICAL SCIENCES↗