Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “multi-modal learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

68 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Learning From Failure: Boosting Cycling Endurance of Optical Phase Change Materials

Chalcogenide phase change materials (PCMs) are a unique class of compounds whose switchable optical and electronic properties have fueled an explosion of emerging applications in microelectronics and microphotonics. Key to any application is the ability of PCMs to reliably switch between crystalline and amorphous states over a large number of cycles. While this issue has been extensively studied in the case of microelectronic memories, current PCM-based optical devices suffer from much inferior endurance. To understand the failure mechanisms limiting endurance of PCMs specifically in microphotonic devices, we have developed an on-chip resistive micro-heater platform and an automatic multi-modal characterization system to analyze cycling performance of optical PCMs. Reversible switching of large-area PCM devices over 50,000 cycles was demonstrated.

Optical phase change material↗
DOE OSTI2024

Machine-Learning-Based Mapping and Modeling of Solar Energy with Ultra-High Spatiotemporal Granularity

Despite the rapid growth of solar energy, we still lack a dynamic, high-fidelity database that tracks the spatiotemporal variations of solar PVs and their associated infrastructures across different places at a spatially resolved scale. The absence of such data presents a barrier to various applications such as solar PV growth projection, solar energy integration, solar incentive design, and climate risk assessment. In this project, we aim to bridge this gap by developing AI-based algorithms to extract granular information about solar PV installations and their associated infrastructures (i.e., distribution grids) from widely available unstructured data like remote sensing images and street views. As a result, we have built the Solar Energy Atlas, a fine-grained, large-scale geospatial overlay of distributed solar PVs and distribution grids. On top of it, we have advanced the understanding of solar adoption and distribution grid vulnerability to climate-induced extremes. Our major contributions can be summarized as follow: (1) By developing new AI algorithms, we have built the most comprehensive solar PV spatiotemporal database covering the entire US. This is the first time we obtained the exact GPS locations, size, subtype, and installation year information for rooftop solar PVs across the US. This database can be used for solar PV growth projection, solar energy integration, solar energy policy analysis and design, and spatially-resolved climate risk assessment. (2) Leveraging this database, we have uncovered the socioeconomic driving factors that are correlated with earlier onset of solar adoption and higher saturated adoption levels. We have identified the heterogeneity in the effects of different types of financial incentives on solar adoption and provided implications for tailoring incentive design based on local income levels to promote equitable solar adoption. (3) We have developed a distribution grid GIS mapping algorithm which can obtain granular geospatial and topology information about distribution grids using multi-modal open data, reducing the dependency on hard-to-obtain smart meter data of conventional approaches. It shows effectiveness in both the U.S. and Sub-Saharan Africa. Using this algorithm, we have uncovered the non-uniform vulnerability of distribution grids to wildfires in California in the aspects of undergrounding protection and Distributed Energy Resources (DER) preparedness. This has provided important implications for improving the affordability and equity of grid adaptation approaches. (3) We have made our produced database publicly available and provided user-friendly interface to enable various stakeholders and the general public to interact with the data. We have also integrated the produced data into the Data Commons platform to enable the public to access the data and correlate it with other location-specific characteristics simply using natural language as queries. The impact of our project is three-fold: (1) New algorithms for mapping solar PVs and distribution grids across space and time, which are open source to facilitate researchers and industry; (2) New databases of solar PVs and distribution grids that have been made publicly available for engineering, social, and policy applications; (3) New understandings and actionable insights on the potential approaches to promoting solar adoption and reducing energy infrastructure vulnerabilities. In this report, we start by discussing the project background and motivation (section 5), followed by the overview of project objectives (section 6). Results and discussion for each task are presented in section 7. Significant accomplishments are summarized in section 8. This report will be concluded by discussing the paths forwards (section 9), products (section 10), and team roles (section 11).

14 SOLAR ENERGY↗
DOE OSTI2025

Unveiling the Hidden Evolution of Crystal Defects and Disorder in Energy Materials

Control of point defects and disorder in functional thin films and 2D materials is critical to realizing their full potential in applications ranging from energy storage to advanced electronics. However, these phenomena are often poorly understood, difficult to characterize, and challenging to direct with precision. This presentation explores emerging multi-modal computer vision to decipher and predict order in materials across multiple length scales in the electron microscope, from the atomic to the nanoscale. By fusing data from diverse sources, these powerful models provide unprecedented insights into materials' lifecycles, enabling the control of defects and their associated properties at a fundamental level. This capability promises to transform materials design and accelerate the development of next-generation technologies.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2010

Space Flight Resource Management for ISS Operations

Although the astronaut training flow for the International Space Station (ISS) spans 2 years, each astronaut or cosmonaut often spends most of their training alone. Rarely is it operationally feasible for all six ISS crewmembers to train together, even more unlikely that crewmembers can practice living together before launch. Likewise, ISS Flight Controller training spans 18 months of learning to manage incredibly complex systems remotely in plug-and-play ground teams that have little to no exposure to crewmembers before a mission. How then do all of these people quickly become a team - a team that must respond flexibly yet decisively to a variety of situations? The answer implemented at NASA is Space Flight Resource Management (SFRM), the so-called "soft skills" or team performance skills. Based on Crew Resource Management, SFRM was developed first for shuttle astronauts and focused on managing human errors during time-critical events (Rogers, et al. 2002). Given the nature of life on ISS, the scope of SFRM for ISS broadened to include teamwork during prolonged and routine operations (O'Keefe, 2008). The ISS SFRM model resembles a star with one competency for each point: Communication, Cross-Culture, Teamwork, Decision Making, Team Care, Leadership/Followership, Conflict Management, and Situation Awareness. These eight competencies were developed with international participation by the Human Behavior and Performance Training Working Group. Over the last two years, these competencies have been used to build a multi-modal SFRM training flow for astronaut candidates and flight controllers that integrates team performance skills into the practice of technical skills. Preliminary results show trainee skill increases as the flow progresses; and participants find the training invaluable to performing well and staying healthy during ISS operations. Future development of SFRM training will aim to help support indirect handovers as ISS operations evolve further with the retirement of the Space Shuttle Program.

Schmidt, Lacey L.↗
NASA NTRSDate not supplied

Earth Independent Medical Operations (EIMO) Datascope: Challenges and Potential Solutions

Data flows and storage/retrieval capacity are severely constrained during missions in space and challenges will become even greater during exploration class missions. There is a need for an artificial intelligence (AI)-based clinical decision support system (CDSS) to monitor and analyze data to provide real-time consultative support for crew medical officer (CMO) decision-making. EIMO is defined as the gradual transition of medical care and decision making from terrestrial to space-based assets, enabling support of astronaut health and performance and reducing overall mission risk. While a hallmark of this paradigm shift from low-earth orbit is that on-board care will increasingly become the responsibility of the astronauts for primary management and decision making, terrestrial assets will continue to be paramount in pre-mission screening and planning, as well as prevention, health maintenance and long-term care contingencies. New capabilities and systems that enable progressively more robust and resilient systems and crews will be necessary to reduce risk and increase probability of deep space exploration mission success. An aspiration for EIMO is to develop AI-enhanced solutions for analysis of crew health & performance data and to facilitate clinical decision support for autonomous medical operations. A “system of systems” approach is envisioned whereby EIMO will deploy AI-supported natural language processing and machine learning (ML) techniques to utilize embedded reference databases and real-time data streams [input vectors] from multiple data sources. Constituent input vectors may include environmental controls, countermeasures data, behavioral data, physiologic wearables, point-of-care laboratory tests, personalized medical records, inventory trade space risk assessments, COTS medical databases, and ground support inputs. An ideal AI capability would possess trained fusion algorithms to cross reference input vectors with medical ‘knowledge’ [cultivated database] to stratify relevant data streams for predictive and actionable capabilities. In addition, EIMO will feature mobility, in that it can be accessed and can push/pull data within and between multiple vehicles/habitats. Large amounts and variable sources of data can be leveraged to diagnose, inform treatment strategies, and potentially predict medical events and performance decrements. Inclusion of advanced training tools using extended reality will enable increasingly autonomous medical care to aid a CMO when ground support is unavailable or time-delayed beyond required action window, e.g., emergent medical situations. EIMO CDSS would require very large datasets to train pre-flight and significant amounts of data are needed to support ML via in-flight CDSS operations. An additional challenge will be to find sufficient data to train a model relevant to astronaut demographics. The rapid, accelerating evolution of this field creates a propitious solution space to leverage multi-modal AI through public-private partnership(s). The status of multi-modal AI systems today would preclude their use for long duration missions as they remain unreliable and are subject to “digital hallucinations” and other errors that could pose operational risk. A federated labs structure is being considered to test and optimize data flow from the multiple input vectors leading to field testing in suitable ground/flight analogs. Critical to the success of an EIMO CDSS will be integration and interoperability and success will be defined by a system that can serve as an in-flight medical consult for the CMO providing critical support during medical contingencies. Benefits to terrestrial medicine may be significant as an outflow of the EIMO medical system, particularly for remote areas and communities lacking significant infrastructure, personnel and resources.

J Lemery↗
NASA NTRS2024

Earth Independent Medical Operations (EIMO) Datascope: Challenges and Potential Solutions

Data flows and storage/retrieval capacity are severely constrained during missions in space and challenges will become even greater during exploration class missions. There is a need for an artificial intelligence (AI)-based clinical decision support system (CDSS) to monitor and analyze data to provide real-time consultative support for crew medical officer (CMO) decision-making. EIMO is defined as the gradual transition of medical care and decision making from terrestrial to space-based assets, enabling support of astronaut health and performance and reducing overall mission risk. While a hallmark of this paradigm shift from low-earth orbit is that on-board care will increasingly become the responsibility of the astronauts for primary management and decision making, terrestrial assets will continue to be paramount in pre-mission screening and planning, as well as prevention, health maintenance and long-term care contingencies. New capabilities and systems that enable progressively more robust and resilient systems and crews will be necessary to reduce risk and increase probability of deep space exploration mission success. An aspiration for EIMO is to develop AI-enhanced solutions for analysis of crew health & performance data and to facilitate clinical decision support for autonomous medical operations. A “system of systems” approach is envisioned whereby EIMO will deploy AI-supported natural language processing and machine learning (ML) techniques to utilize embedded reference databases and real-time data streams [input vectors] from multiple data sources. Constituent input vectors may include environmental controls, countermeasures data, behavioral data, physiologic wearables, point-of-care laboratory tests, personalized medical records, inventory trade space risk assessments, COTS medical databases, and ground support inputs. An ideal AI capability would possess trained fusion algorithms to cross reference input vectors with medical ‘knowledge’ [cultivated database] to stratify relevant data streams for predictive and actionable capabilities. In addition, EIMO will feature mobility, in that it can be accessed and can push/pull data within and between multiple vehicles/habitats. Large amounts and variable sources of data can be leveraged to diagnose, inform treatment strategies, and potentially predict medical events and performance decrements. Inclusion of advanced training tools using extended reality will enable increasingly autonomous medical care to aid a CMO when ground support is unavailable or time-delayed beyond required action window, e.g., emergent medical situations. EIMO CDSS would require very large datasets to train pre-flight and significant amounts of data are needed to support ML via in-flight CDSS operations. An additional challenge will be to find sufficient data to train a model relevant to astronaut demographics. The rapid, accelerating evolution of this field creates a propitious solution space to leverage multi-modal AI through public-private partnership(s). The status of multi-modal AI systems today would preclude their use for long duration missions as they remain unreliable and are subject to “digital hallucinations” and other errors that could pose operational risk. A federated labs structure is being considered to test and optimize data flow from the multiple input vectors leading to field testing in suitable ground/flight analogs. Critical to the success of an EIMO CDSS will be integration and interoperability and success will be defined by a system that can serve as an in-flight medical consult for the CMO providing critical support during medical contingencies. Benefits to terrestrial medicine may be significant as an outflow of the EIMO medical system, particularly for remote areas and communities lacking significant infrastructure, personnel and resources.

Medical Operations↗
DOE OSTI2026

Multi-modal dynamic radiography using short-pulse laser-generated probe beams

Radiography is an important tool for the interrogation of dynamic experiments in the fields of dynamic properties of materials, and in condensed matter, high explosive, and high-energy-density physics. Multi-modal radiography advances the hypothesis that combining the information delivered by multiple radiographic modalities can lead to more constrained (improved) “reconstruction” of the scene than can be obtained from a single probe. We identify four modalities: multi-probe, time sequence, multi-view, and multi-messenger. Multi-probe radiography is a promising candidate for a next-generation dynamic radiographic facility. High-energy X-rays are the most frequently used probe for dynamic radiography, although recent developments show the utility of proton (pRad), electron (eRad), and neutron probe beams. Because each probing species interacts with material in the radiographic scene through quantitatively different mechanisms, each returns independent information about the scene, which can add extra constraints to the reconstruction process. How to conduct detailed, quantitative “co-analysis” of multiple data streams remains an area of active research. Multi-beam, short-pulse, laser-generated probes offer sufficient dose, an appropriate spectrum, and appropriate spatio-temporal resolution to produce high-quality dynamic radiographs. This paper reports on technology development to advance the state of the art of multi-modal/multi-probe radiography and the pursuit of both deterministic and inferential (AI/ML assisted) co-analysis methodologies to produce more constrained reconstructions from multi-modal data.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

Object detection with deep learning for rare event search in the GADGET II TPC

In the pursuit of identifying rare two-particle events within the GADGET II Time Projection Chamber (TPC), this paper presents a comprehensive approach for leveraging Convolutional Neural Networks (CNNs) and various data processing methods. To address the inherent complexities of 3D TPC track reconstructions, the data is expressed in 2D projections and 1D quantities. This approach capitalizes on the diverse data modalities of the TPC, allowing for the efficient representation of the distinct features of the 3D events, with no loss in topology uniqueness. Additionally, it leverages the computational efficiency of 2D CNNs and benefits from the extensive availability of pre-trained models. Given the scarcity of real training data for the rare events of interest, simulated events are used to train the models to detect real events. To account for potential distribution shifts when predominantly depending on simulations, significant perturbations are embedded within the simulations. This produces a broad parameter space that works to account for potential physics parameter and detector response variations and uncertainties. These parameter-varied simulations are used to train sensitive 2D CNN object detectors. When combined with 1D histogram peak detection algorithms, this multi-modal detection framework is highly adept at identifying rare, two-particle events in data taken during experiment 21072 at the Facility for Rare Isotope Beams (FRIB), demonstrating a 100% recall for events of interest. Here, we present the methods and outcomes of our investigation and discuss the potential future applications of these techniques.

Convolutional neural network↗
NASA NTRSDate not supplied

Kinematic Sensors Evaluation for Spaceflight Exercise Data Collections

INTRODUCTION: On the International Space Station (ISS), exercise feedback from astronauts is very important to diagnose and mitigate any form-related injuries and ensure efficacious exercise prescriptions and systems. Going forward, exploration exercise efforts seek to gain further quantitative data of human and system performance. Currently, methods of collecting in-flight exercise data on the ISS are limited to marker-based motion capture (MoCap) where astronauts must wear reflective markers over their clothes and specialized cameras are used. The main objective of this work was to investigate the following alternative tracking options: markerless video-based MoCap and inertial measurement units (IMUs). These were compared against traditional marker-based MoCap to evaluate kinematic accuracy and inform feasible methods for future exercise data collections on the ISS, especially in support of future Vibration Isolation and Stabilization (VIS) system development. METHODS: Three test subjects performed a variety of flight-like resistance and aerobic exercises using the Miniature Exercise Device (MED-2), Concept-2 rowing ergometer, barbell mockup, bench (e.g., for bench press, hip thruster, and cycling), and a custom structure for dips. These were intended also to represent exercises which could be performed on the multi-modality European Enhanced Exploration Exercise Device (E4D) [1]. The marker-based MoCap data, collected through a 16-camera OptiTrack MoCap system, was regarded as the gold standard to compare the data against. Passive markers were affixed to each subject according to a modified full body Plug-in Gait marker set [2] with 46 total markers. The markerless MoCap data was collected using two GoPro Hero7 cameras and one GoPro Hero11 camera. For the IMU data, a full body set of 17 Xsens DOTs were placed on the subject: 10 upper body and 7 lower body IMUs. Biomechanical modeling and evaluation was conducted through OpenSim [3] (MoCap), OpenSense [4] (IMU), OpenCap [5] (markerless), ENABLE [6] (markerless), and other modeling software. Secondary objectives included comparing the volume of equipment, reducing mass and crew set-up time. RESULTS AND DISCUSSION: While there were issues with initial processing for the IMUs and markerless MoCap, the results aided in the understanding of each sensor, developing end-to-end processes, and identifying future needs. Some observed concerns with the markerless MoCap approaches included being cognizant of a cluttered background, number of people in field of view, camera number and placement. Some challenges with the IMUs included possible sliding, early deactivation possibly due to exercise pose, and large quantity sensor synchronization. Overall, the markerless MoCap option may be the preferred method of data collection and processing as it provides a solution for certain IMU shortcomings and may be least in equipment volume, upmass, and crew setup time. CONCLUSIONS: While this work was mainly focused on ISS data collection, these sensor data along with continued evaluation and development efforts will help to establish best methods for exercise data collection on Gateway, for other Artemis missions, and beyond. Details on the latest end-to-end processing of the data and results will be presented, along with lessons learned and recommended sensor selection and methods.

S. Faragalla↗
NASA NTRS2024

Kinematic Sensors Evaluation for Spaceflight Exercise Data Collections

INTRODUCTION: On the International Space Station (ISS), exercise feedback from astronauts is very important to diagnose and mitigate any form-related injuries and ensure efficacious exercise prescriptions and systems. Going forward, exploration exercise efforts seek to gain further quantitative data of human and system performance. Currently, methods of collecting in-flight exercise data on the ISS are limited to marker-based motion capture (MoCap) where astronauts must wear reflective markers over their clothes and specialized cameras are used. The main objective of this work was to investigate the following alternative tracking options: markerless video-based MoCap and inertial measurement units (IMUs). These were compared against traditional marker-based MoCap to evaluate kinematic accuracy and inform feasible methods for future exercise data collections on the ISS, especially in support of future Vibration Isolation and Stabilization (VIS) system development. METHODS: Three test subjects performed a variety of flight-like resistance and aerobic exercises using the Miniature Exercise Device (MED-2), Concept-2 rowing ergometer, barbell mockup, bench (e.g., for bench press, hip thruster, and cycling), and a custom structure for dips. These were intended also to represent exercises which could be performed on the multi-modality European Enhanced Exploration Exercise Device (E4D) [1]. The marker-based MoCap data, collected through a 16-camera OptiTrack MoCap system, was regarded as the gold standard to compare the data against. Passive markers were affixed to each subject according to a modified full body Plug-in Gait marker set [2] with 46 total markers. The markerless MoCap data was collected using two GoPro Hero7 cameras and one GoPro Hero11 camera. For the IMU data, a full body set of 17 Xsens DOTs were placed on the subject: 10 upper body and 7 lower body IMUs. Biomechanical modeling and evaluation was conducted through OpenSim [3] (MoCap), OpenSense [4] (IMU), OpenCap [5] (markerless), ENABLE [6] (markerless), and other modeling software. Secondary objectives included comparing the volume of equipment, reducing mass and crew set-up time. RESULTS AND DISCUSSION: While there were issues with initial processing for the IMUs and markerless MoCap, the results aided in the understanding of each sensor, developing end-to-end processes, and identifying future needs. Some observed concerns with the markerless MoCap approaches included being cognizant of a cluttered background, number of people in field of view, camera number and placement. Some challenges with the IMUs included possible sliding, early deactivation possibly due to exercise pose, and large quantity sensor synchronization. Overall, the markerless MoCap option may be the preferred method of data collection and processing as it provides a solution for certain IMU shortcomings and may be least in equipment volume, upmass, and crew setup time. CONCLUSIONS: While this work was mainly focused on ISS data collection, these sensor data along with continued evaluation and development efforts will help to establish best methods for exercise data collection on Gateway, for other Artemis missions, and beyond. Details on the latest end-to-end processing of the data and results will be presented, along with lessons learned and recommended sensor selection and methods.

S Faragalla↗
DOE OSTI2026

Ripening of Rh Nanoparticle Catalysts in Reverse Water–Gas Shift via a Data-Driven Model Combining Physics, Theory, and Experiment

Degradation via sintering is an ongoing challenge that impedes the broad commercial success of supported metallic nanoparticle catalysts. To mitigate degradation via informed catalyst design and process operations, here we aim to disambiguate the underlying mechanisms of sintering by combining theory and experiment in a quantitative framework. While mechanistic sintering models exist, they only model a single sintering pathway, even though multiple sintering mechanisms can occur simultaneously or dominate at different stages of the process. Data-driven machine learning models have emerged as a means to represent complex processes through data regression. However, machine learning models have very large data needs and lack mechanistic insights due to their black-box encoding. To develop an interpretive model of catalyst degradation via sintering, we constructed a hybrid model combining mechanistic “physics-based” models and data-driven methods to obtain both reliable predictions and mechanistic insights regarding experimentally observed sintering phenomena. Focusing on nanoparticle sintering in the Rh–TiO 2 catalyst for the reverse water–gas shift (RWGS) reaction, the hybrid model couples a mechanistic term for Ostwald ripening with energy values calculated via density functional theory (DFT) with a parametric, data-driven discrepancy function term for unmodeled mechanisms. The hybrid model is trained using Bayesian inference with data collected from small-angle X-ray scattering (SAXS) in situ experiments wherein average nanoparticle diameter versus time was measured at three relevant operating temperatures. The calibrated hybrid model results show that an Ostwald ripening-only model parameterized with fixed DFT energies does not fully capture the time and temperature dependence of the SAXS-observed sintering kinetics, and that an additional functional contribution, or DFT energy calibration, is required to reconcile simulation and experiment. Analysis of the hybrid-model error confirms that the hybrid model outperforms both the purely mechanistic and purely data-driven alternatives in terms of expected predictive accuracy for time-evolving average particle sizes. Furthermore, the results support the hypothesis that the Ostwald ripening mechanism is less important for explaining the sintering phenomena as operating temperature increases under an assumed fixed DFT parameterization. This could be explained in one of two ways: either latent, unmodeled sintering mechanisms dominate at higher temperatures, or the DFT uncertainty increases with temperature. The proposed modeling approach directly links theory to experiments and simulations via a statistical hybrid modeling framework and can be extended to other catalytic systems to improve predictive models and mechanistic understanding.

Bayesian hybrid modeling↗
DOE OSTI2025

Enhancing Automotive Intrusion Detection Through Multi-Modal Fusion: A CAN FD-LiDAR Approach

As vehicles become smarter and more autonomous, they increasingly depend on advanced sensors and communication technologies to operate securely. However, such growing dependence on technology—whether it’s CAN (Controller Area Network) for internal communication or LiDAR (Light Detection and Ranging) for sensing the world around them—also expands the attack surface for the types of cyber attacks. Traditional intrusion detection systems (IDS) typically monitor these systems in isolation, limiting their ability to detect sophisticated, crosssystem attacks. To address this, we propose a multi-modal fusion approach that combines real-world CAN FD signals (from the HCRL dataset) with LiDAR features (from the nuScenes dataset) to enhance attack detection. Our method employs a twostage ensemble approach. Calibrated XGBoost and LightGBM models initially process CAN FD (Fuzzing Data) and LiDAR data independently, detecting timing anomalies and space abnormalities. They are subsequently logarithmically combined with a logistic regression meta-model along with 17 engineered features capturing cross-modal behavior, prediction conflicts, and nonlinear interactions. This approach achieves an AUC of 0.87 and an F1-score of 0.82, surpassing single-modality baselines and early fusion methods, at merely 2 ms inference latency. Compared with deep learning competitors, it is 3 times more efficient, providing a lightweight, interpretable, and real time solution to automotive cybersecurity.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Cell to Ecosystem: Understanding Methane and Associated Nutrient Cycling by Sediment Hosted Syntrophic Consortia and Their Viral Predators

The anaerobic oxidation of methane (AOM) is a significant worldwide microbial process in anoxic lake and ocean sediments, responsible for sequestering up to 80% of this greenhouse gas. Often considered a metabolism on the edge of thermodynamic probability, the impact of ANaerobic MEthane-oxidizing ‘ANME’ archaea on carbon and nutrient cycling in sediment ecosystems is far reaching. They not only serve as a sink for methane coupled to diverse electron acceptors, but also catalyze the transformation of many important nutrients including nitrogen, phosphate, and iron. While information about the potential mechanisms supporting metabolism in AOM is now available, remarkably little has been learned about their nutritional requirements, their dependencies on bacterial partners, and consequentially their ultimate impact on nutrient transformation and bioavailability within sedimentary ecosystems, and beyond. Further, the role of viruses within sediment ecosystems represents an essential but vastly understudied aspect of the transformation of carbon and nutrients by AOM. Viruses are now widely appreciated as central players in biogeochemical cycles across diverse ecosystems. These nanoscale predators have been shown to enhance the turnover of essential nutrients, thus stimulating microbial growth and environmental viruses themselves may constitute an important reservoir of nitrogen and phosphorous. Little is known about the role of viruses in methane-impacted sedimentary ecosystems, however prior genomic and microscopy evidence suggests that AOM consortia are susceptible to phage infection. The overarching scientific goal of this multi-disciplinary research proposal is to build on these recent discoveries and expand our understanding of interactions and fundamental activities involved in cycling of carbon and nutrients by syntrophic methanotrophic archaeal-bacterial consortia and associated viruses in anoxic sedimentary environments. Our three specific objectives are to 1) Quantify energy and nutrient exchange (e.g. N, P, Fe and vitamins) within AOM consortia and between ANME-bacterial partners; 2) Identify virus-host interactions associated with AOM and assess C and N transfer through viruses in methane-impacted sediment ecosystems; 3) Model energy and nutrient exchange in AOM consortia and viral-host interactions (i.e. viral activity), and their environmental distribution patterns. Our experimental emphasis cuts across scales that are important for understanding microbial and viral interactions and activities within their habitats, as well as community wide biogeochemical transformations. These research goals will be accomplished through the application of novel molecular techniques targeting DNA, RNA, proteins, and metabolites combined with a unique multi-modal analytical imaging pipeline. We will then model the ecophysiological capabilities of diverse sediment-hosted methanotrophic consortia to develop a more comprehensive understanding of the energetic and nutritional interactions between different AOM partner couplings that occur in sediments.

03 NATURAL GAS↗
DOE OSTI2024

Hiding-in-Plain-Sight (HiPS) Attack on CLIP for Targetted Object Removal from Images

Machine learning models are known to be vulnerable to adversarial attacks, but prior works have mostly focused on single-modalities. With the rise of large multi-modal models (LMMs) like CLIP, which combine vision and language capabilities, new vulnerabilities have emerged. However, these multimodal targeted attacks aim to completely change the model's output to what the adversary wants. In many realistic scenarios, an adversary might seek to make only subtle modifications to the output, so that the changes go unnoticed by downstream models or even by humans. We introduce Hiding-in-Plain-Sight (HiPS) attacks, a novel class of adversarial attacks that subtly modifies model predictions by selectively concealing target object(s), as if the target object was absent from the scene. We propose two HiPS attack variants, HiPS-cls and HiPS-cap, and demonstrate their effectiveness in transferring to downstream image captioning models, such as CLIP-Cap, for targeted object removal from image captions.

Daw, Arka [ORNL] (ORCID:0009000633191271)↗