Results for “multi-modal learning” — Page 2

NASA NTRS2017

NeMO-Net: The Neural Multi-Modal Observation and Training Network for Global Coral Reef Assessment

In the past decade, coral reefs worldwide have experienced unprecedented stresses due to climate change, ocean acidification, and anthropomorphic pressures, instigating massive bleaching and die-off of these fragile and diverse ecosystems. Furthermore, remote sensing of these shallow marine habitats is hindered by ocean wave distortion, refraction and optical attenuation, leading invariably to data products that are often of low resolution and signal-to-noise (SNR) ratio. However, recent advances in UAV and Fluid Lensing technology have allowed us to capture multispectral 3D imagery of these systems at sub-cm scales from above the water surface, giving us an unprecedented view of their growth and decay. Exploiting the fine-scaled features of these datasets, machine learning methods such as MAP, PCA, and SVM can not only accurately classify the living cover and morphology of these reef systems (below 8 percent error), but are also able to map the spectral space between airborne and satellite imagery, augmenting and improving the classification accuracy of previously low-resolution datasets. We are currently implementing NeMO-Net, the first open-source deep convolutional neural network (CNN) and interactive active learning and training software to accurately assess the present and past dynamics of coral reef ecosystems through determination of percent living cover and morphology. NeMO-Net will be built upon the QGIS platform to ingest UAV, airborne and satellite datasets from various sources and sensor capabilities, and through data-fusion determine the coral reef ecosystem makeup globally at unprecedented spatial and temporal scales. To achieve this, we will exploit virtual data augmentation, the use of semi-supervised learning, and active learning through a tablet platform allowing for users to manually train uncertain or difficult to classify datasets. The project will make use of Pythons extensive libraries for machine learning, as well as extending integration to GPU and High-End Computing Capability (HECC) on the Pleiades supercomputing cluster, located at NASA Ames. The project is being supported by NASAs Earth Science Technology Office (ESTO) Advanced Information Systems Technology (AIST-16) Program.

NeMO-Net↗

NASA NTRSDate not supplied

Learning From Failure: Boosting Cycling Endurance of Optical Phase Change Materials

Chalcogenide phase change materials (PCMs) are a unique class of compounds whose switchable optical and electronic properties have fueled an explosion of emerging applications in microelectronics and microphotonics. Key to any application is the ability of PCMs to reliably switch between crystalline and amorphous states over a large number of cycles. While this issue has been extensively studied in the case of microelectronic memories, current PCM-based optical devices suffer from much inferior endurance. To understand the failure mechanisms limiting endurance of PCMs specifically in microphotonic devices, we have developed an on-chip resistive micro-heater platform and an automatic multi-modal characterization system to analyze cycling performance of optical PCMs. Reversible switching of large-area PCM devices over 50,000 cycles was demonstrated.

Optical phase change material↗

NASA NTRS2010

Space Flight Resource Management for ISS Operations

Although the astronaut training flow for the International Space Station (ISS) spans 2 years, each astronaut or cosmonaut often spends most of their training alone. Rarely is it operationally feasible for all six ISS crewmembers to train together, even more unlikely that crewmembers can practice living together before launch. Likewise, ISS Flight Controller training spans 18 months of learning to manage incredibly complex systems remotely in plug-and-play ground teams that have little to no exposure to crewmembers before a mission. How then do all of these people quickly become a team - a team that must respond flexibly yet decisively to a variety of situations? The answer implemented at NASA is Space Flight Resource Management (SFRM), the so-called "soft skills" or team performance skills. Based on Crew Resource Management, SFRM was developed first for shuttle astronauts and focused on managing human errors during time-critical events (Rogers, et al. 2002). Given the nature of life on ISS, the scope of SFRM for ISS broadened to include teamwork during prolonged and routine operations (O'Keefe, 2008). The ISS SFRM model resembles a star with one competency for each point: Communication, Cross-Culture, Teamwork, Decision Making, Team Care, Leadership/Followership, Conflict Management, and Situation Awareness. These eight competencies were developed with international participation by the Human Behavior and Performance Training Working Group. Over the last two years, these competencies have been used to build a multi-modal SFRM training flow for astronaut candidates and flight controllers that integrates team performance skills into the practice of technical skills. Preliminary results show trainee skill increases as the flow progresses; and participants find the training invaluable to performing well and staying healthy during ISS operations. Future development of SFRM training will aim to help support indirect handovers as ISS operations evolve further with the retirement of the Space Shuttle Program.

Schmidt, Lacey L.↗

NASA NTRSDate not supplied

Earth Independent Medical Operations (EIMO) Datascope: Challenges and Potential Solutions

Data flows and storage/retrieval capacity are severely constrained during missions in space and challenges will become even greater during exploration class missions. There is a need for an artificial intelligence (AI)-based clinical decision support system (CDSS) to monitor and analyze data to provide real-time consultative support for crew medical officer (CMO) decision-making. EIMO is defined as the gradual transition of medical care and decision making from terrestrial to space-based assets, enabling support of astronaut health and performance and reducing overall mission risk. While a hallmark of this paradigm shift from low-earth orbit is that on-board care will increasingly become the responsibility of the astronauts for primary management and decision making, terrestrial assets will continue to be paramount in pre-mission screening and planning, as well as prevention, health maintenance and long-term care contingencies. New capabilities and systems that enable progressively more robust and resilient systems and crews will be necessary to reduce risk and increase probability of deep space exploration mission success. An aspiration for EIMO is to develop AI-enhanced solutions for analysis of crew health & performance data and to facilitate clinical decision support for autonomous medical operations. A “system of systems” approach is envisioned whereby EIMO will deploy AI-supported natural language processing and machine learning (ML) techniques to utilize embedded reference databases and real-time data streams [input vectors] from multiple data sources. Constituent input vectors may include environmental controls, countermeasures data, behavioral data, physiologic wearables, point-of-care laboratory tests, personalized medical records, inventory trade space risk assessments, COTS medical databases, and ground support inputs. An ideal AI capability would possess trained fusion algorithms to cross reference input vectors with medical ‘knowledge’ [cultivated database] to stratify relevant data streams for predictive and actionable capabilities. In addition, EIMO will feature mobility, in that it can be accessed and can push/pull data within and between multiple vehicles/habitats. Large amounts and variable sources of data can be leveraged to diagnose, inform treatment strategies, and potentially predict medical events and performance decrements. Inclusion of advanced training tools using extended reality will enable increasingly autonomous medical care to aid a CMO when ground support is unavailable or time-delayed beyond required action window, e.g., emergent medical situations. EIMO CDSS would require very large datasets to train pre-flight and significant amounts of data are needed to support ML via in-flight CDSS operations. An additional challenge will be to find sufficient data to train a model relevant to astronaut demographics. The rapid, accelerating evolution of this field creates a propitious solution space to leverage multi-modal AI through public-private partnership(s). The status of multi-modal AI systems today would preclude their use for long duration missions as they remain unreliable and are subject to “digital hallucinations” and other errors that could pose operational risk. A federated labs structure is being considered to test and optimize data flow from the multiple input vectors leading to field testing in suitable ground/flight analogs. Critical to the success of an EIMO CDSS will be integration and interoperability and success will be defined by a system that can serve as an in-flight medical consult for the CMO providing critical support during medical contingencies. Benefits to terrestrial medicine may be significant as an outflow of the EIMO medical system, particularly for remote areas and communities lacking significant infrastructure, personnel and resources.

J Lemery↗

NASA NTRS2024

Earth Independent Medical Operations (EIMO) Datascope: Challenges and Potential Solutions

Data flows and storage/retrieval capacity are severely constrained during missions in space and challenges will become even greater during exploration class missions. There is a need for an artificial intelligence (AI)-based clinical decision support system (CDSS) to monitor and analyze data to provide real-time consultative support for crew medical officer (CMO) decision-making. EIMO is defined as the gradual transition of medical care and decision making from terrestrial to space-based assets, enabling support of astronaut health and performance and reducing overall mission risk. While a hallmark of this paradigm shift from low-earth orbit is that on-board care will increasingly become the responsibility of the astronauts for primary management and decision making, terrestrial assets will continue to be paramount in pre-mission screening and planning, as well as prevention, health maintenance and long-term care contingencies. New capabilities and systems that enable progressively more robust and resilient systems and crews will be necessary to reduce risk and increase probability of deep space exploration mission success. An aspiration for EIMO is to develop AI-enhanced solutions for analysis of crew health & performance data and to facilitate clinical decision support for autonomous medical operations. A “system of systems” approach is envisioned whereby EIMO will deploy AI-supported natural language processing and machine learning (ML) techniques to utilize embedded reference databases and real-time data streams [input vectors] from multiple data sources. Constituent input vectors may include environmental controls, countermeasures data, behavioral data, physiologic wearables, point-of-care laboratory tests, personalized medical records, inventory trade space risk assessments, COTS medical databases, and ground support inputs. An ideal AI capability would possess trained fusion algorithms to cross reference input vectors with medical ‘knowledge’ [cultivated database] to stratify relevant data streams for predictive and actionable capabilities. In addition, EIMO will feature mobility, in that it can be accessed and can push/pull data within and between multiple vehicles/habitats. Large amounts and variable sources of data can be leveraged to diagnose, inform treatment strategies, and potentially predict medical events and performance decrements. Inclusion of advanced training tools using extended reality will enable increasingly autonomous medical care to aid a CMO when ground support is unavailable or time-delayed beyond required action window, e.g., emergent medical situations. EIMO CDSS would require very large datasets to train pre-flight and significant amounts of data are needed to support ML via in-flight CDSS operations. An additional challenge will be to find sufficient data to train a model relevant to astronaut demographics. The rapid, accelerating evolution of this field creates a propitious solution space to leverage multi-modal AI through public-private partnership(s). The status of multi-modal AI systems today would preclude their use for long duration missions as they remain unreliable and are subject to “digital hallucinations” and other errors that could pose operational risk. A federated labs structure is being considered to test and optimize data flow from the multiple input vectors leading to field testing in suitable ground/flight analogs. Critical to the success of an EIMO CDSS will be integration and interoperability and success will be defined by a system that can serve as an in-flight medical consult for the CMO providing critical support during medical contingencies. Benefits to terrestrial medicine may be significant as an outflow of the EIMO medical system, particularly for remote areas and communities lacking significant infrastructure, personnel and resources.

Medical Operations↗

NASA NTRSDate not supplied

Kinematic Sensors Evaluation for Spaceflight Exercise Data Collections

INTRODUCTION: On the International Space Station (ISS), exercise feedback from astronauts is very important to diagnose and mitigate any form-related injuries and ensure efficacious exercise prescriptions and systems. Going forward, exploration exercise efforts seek to gain further quantitative data of human and system performance. Currently, methods of collecting in-flight exercise data on the ISS are limited to marker-based motion capture (MoCap) where astronauts must wear reflective markers over their clothes and specialized cameras are used. The main objective of this work was to investigate the following alternative tracking options: markerless video-based MoCap and inertial measurement units (IMUs). These were compared against traditional marker-based MoCap to evaluate kinematic accuracy and inform feasible methods for future exercise data collections on the ISS, especially in support of future Vibration Isolation and Stabilization (VIS) system development. METHODS: Three test subjects performed a variety of flight-like resistance and aerobic exercises using the Miniature Exercise Device (MED-2), Concept-2 rowing ergometer, barbell mockup, bench (e.g., for bench press, hip thruster, and cycling), and a custom structure for dips. These were intended also to represent exercises which could be performed on the multi-modality European Enhanced Exploration Exercise Device (E4D) [1]. The marker-based MoCap data, collected through a 16-camera OptiTrack MoCap system, was regarded as the gold standard to compare the data against. Passive markers were affixed to each subject according to a modified full body Plug-in Gait marker set [2] with 46 total markers. The markerless MoCap data was collected using two GoPro Hero7 cameras and one GoPro Hero11 camera. For the IMU data, a full body set of 17 Xsens DOTs were placed on the subject: 10 upper body and 7 lower body IMUs. Biomechanical modeling and evaluation was conducted through OpenSim [3] (MoCap), OpenSense [4] (IMU), OpenCap [5] (markerless), ENABLE [6] (markerless), and other modeling software. Secondary objectives included comparing the volume of equipment, reducing mass and crew set-up time. RESULTS AND DISCUSSION: While there were issues with initial processing for the IMUs and markerless MoCap, the results aided in the understanding of each sensor, developing end-to-end processes, and identifying future needs. Some observed concerns with the markerless MoCap approaches included being cognizant of a cluttered background, number of people in field of view, camera number and placement. Some challenges with the IMUs included possible sliding, early deactivation possibly due to exercise pose, and large quantity sensor synchronization. Overall, the markerless MoCap option may be the preferred method of data collection and processing as it provides a solution for certain IMU shortcomings and may be least in equipment volume, upmass, and crew setup time. CONCLUSIONS: While this work was mainly focused on ISS data collection, these sensor data along with continued evaluation and development efforts will help to establish best methods for exercise data collection on Gateway, for other Artemis missions, and beyond. Details on the latest end-to-end processing of the data and results will be presented, along with lessons learned and recommended sensor selection and methods.

S. Faragalla↗

NASA NTRS2024

Kinematic Sensors Evaluation for Spaceflight Exercise Data Collections

INTRODUCTION: On the International Space Station (ISS), exercise feedback from astronauts is very important to diagnose and mitigate any form-related injuries and ensure efficacious exercise prescriptions and systems. Going forward, exploration exercise efforts seek to gain further quantitative data of human and system performance. Currently, methods of collecting in-flight exercise data on the ISS are limited to marker-based motion capture (MoCap) where astronauts must wear reflective markers over their clothes and specialized cameras are used. The main objective of this work was to investigate the following alternative tracking options: markerless video-based MoCap and inertial measurement units (IMUs). These were compared against traditional marker-based MoCap to evaluate kinematic accuracy and inform feasible methods for future exercise data collections on the ISS, especially in support of future Vibration Isolation and Stabilization (VIS) system development. METHODS: Three test subjects performed a variety of flight-like resistance and aerobic exercises using the Miniature Exercise Device (MED-2), Concept-2 rowing ergometer, barbell mockup, bench (e.g., for bench press, hip thruster, and cycling), and a custom structure for dips. These were intended also to represent exercises which could be performed on the multi-modality European Enhanced Exploration Exercise Device (E4D) [1]. The marker-based MoCap data, collected through a 16-camera OptiTrack MoCap system, was regarded as the gold standard to compare the data against. Passive markers were affixed to each subject according to a modified full body Plug-in Gait marker set [2] with 46 total markers. The markerless MoCap data was collected using two GoPro Hero7 cameras and one GoPro Hero11 camera. For the IMU data, a full body set of 17 Xsens DOTs were placed on the subject: 10 upper body and 7 lower body IMUs. Biomechanical modeling and evaluation was conducted through OpenSim [3] (MoCap), OpenSense [4] (IMU), OpenCap [5] (markerless), ENABLE [6] (markerless), and other modeling software. Secondary objectives included comparing the volume of equipment, reducing mass and crew set-up time. RESULTS AND DISCUSSION: While there were issues with initial processing for the IMUs and markerless MoCap, the results aided in the understanding of each sensor, developing end-to-end processes, and identifying future needs. Some observed concerns with the markerless MoCap approaches included being cognizant of a cluttered background, number of people in field of view, camera number and placement. Some challenges with the IMUs included possible sliding, early deactivation possibly due to exercise pose, and large quantity sensor synchronization. Overall, the markerless MoCap option may be the preferred method of data collection and processing as it provides a solution for certain IMU shortcomings and may be least in equipment volume, upmass, and crew setup time. CONCLUSIONS: While this work was mainly focused on ISS data collection, these sensor data along with continued evaluation and development efforts will help to establish best methods for exercise data collection on Gateway, for other Artemis missions, and beyond. Details on the latest end-to-end processing of the data and results will be presented, along with lessons learned and recommended sensor selection and methods.

S Faragalla↗