SEARCH · Engineering Papers

Results for “Augmented Reality”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

All sources NASA NTRS DOE OSTI

NASA NTRS2025

Limitations and Feasibility of Mini X-Ray Devices in Space Environments

LIMITATIONS AND FEASIBILITY OF MINI X-RAY DEVICES IN SPACE ENVIRONMENTS As space exploration advances toward long-duration missions, reliable medical diagnostic tools become increasingly critical. The miniature x-ray (XR) technology demonstrations by the Exploration Medical Capability (ExMC) and the Exploration Medical Integrated Product Team (XMIPT) aim to assess the feasibility and utility of miniature XR devices in spaceflight. This abstract explores the limitations of current miniature XR systems, the challenges of training crew members, the potential role of clinical decision support systems (CDSS), and the feasibility of ground-based image interpretation. We also propose the integration of miniature XR into other ExMC efforts aimed at identifying the capabilities and resources needed for future exploration class missions. One of the primary challenges with miniature XR devices is the ability to achieve specific anatomical views, particularly in the confined and weightless conditions of a spacecraft. Operators may struggle to acquire diagnostic-quality images when space is limited for proper patient positioning and the volume of the imaging device. Since space radiation and detector limitations may further impact image quality, the flexibility of the operating procedures of these devices will be critical for their success in space applications. CHALLENGES IN TRAINING CREW TO OPERATE IMAGING DEVICES Training in the skills necessary to acquire diagnostic-quality scans may be a barrier for non-clinician crewmembers. The curriculum developed for crew medical officers (CMOs) will require simplification and adaptation to fit into the highly truncated pre-flight training period. Therefore, hands-on familiarization and simulation, both pre-flight and just-in-time training during missions, will be crucial to ensuring the crew can operate the devices in real-life situations. The ability to adjust acquisition parameters must be simplified or made automatic through exam selections on equipment user interfaces, and subject and operator positioning should be assisted with laser guidance and pictorial guides. POTENTIAL FOR CDSS OR ARTIFICIAL INTELLIGENCE (AI)-ASSISTED CDSS CDSS and AI-assisted CDSS offer significant promise in assisting crew members with limited medical training. These systems could provide real-time feedback on image quality and interpretation, helping to mitigate the risks of human error during space missions. Integrating procedural guidance tools, such as virtual and augmented reality, will support crewmembers in accurately positioning patients and obtaining high-quality images. However, the success of such systems will depend on the development of robust training datasets, integration with spaceflight-rated hardware, and the medical decision-making capabilities of operators. FEASIBILITY OF GROUND INTERPRETATION AND DATA TRANSMISSION Reliance on ground-based interpretation may prove difficult for acute care during exploration class-missions due to delays in transmission with increasing distance from Earth or complete communication blackout periods. In such instances where immediate interpretation for clinical intervention is required, crew must be able to interpret the images independently or utilize AI-based assistance to do so. File sizes for XR exams can also be large if numerous images are acquired and bandwidth constraints may limit data transmissions for both radiography and ultrasound exams. FUTURE WORK AND INTEGRATION INTO THE EVIDENCE LIBRARY Future work proposes integrating miniature XR devices into NASA’s Evidence Library to address medical conditions identified as significant contributors to crew morbidity and mortality. The possibility of combining miniature XR with other imaging modalities, such as ultrasound devices, is also under investigation. In conclusion, while miniature XR technology holds potential for extraterrestrial medical systems, there are significant challenges to overcome. Training, integration of AI tools, dedicated exam protocols for microgravity, and improved data transmission systems will be key to realizing the full benefits of miniature XR technology in space.

A M Nelson↗

NASA NTRSDate not supplied

Discovering Digital Engineering Needs Through Human-Centered Approaches

Digital engineering has the power to transform organizations in profound ways, but digitizing legacy workflows and infusing new technologies into existing practices may not yield the intended results or address the underlying problems or bottlenecks. This paper describes the human-centered approaches used to understand stakeholder challenges and identify digital engineering needs during the design-build-test portion of the systems development lifecycle. Through a series of interactive stakeholder workshops and need-finding activities, key stakeholder needs were identified to help guide the integration of digital engineering technologies. The overarching need was for downstream system development steps to be considered further upstream in the project lifecycle in a manner that allows for proactive and interactive co-constructive idea exploration, problem resolution, and requirements considerations in a collective engagement with engineers and technicians from multiple disciplines to increase efficiency and reduce risks while offering opportunities to improve the final engineered system. Supporting this need was a wide-ranging desire from stakeholders for digital and non-digital support structures for efficient rapid iteration during system development. This work yielded several improvements for system development processes and for digital engineering infusion.

Design Science↗

NASA NTRS2017

The NASA Augmented/Virtual Reality Lab: The State of the Art at KSC

The NASA Augmented Virtual Reality (AVR) Lab at Kennedy Space Center is dedicated to the investigation of Augmented Reality (AR) and Virtual Reality (VR) technologies, with the goal of determining potential uses of these technologies as human-computer interaction (HCI) devices in an aerospace engineering context. Begun in 2012, the AVR Lab has concentrated on commercially available AR and VR devices that are gaining in popularity and use in a number of fields such as gaming, training, and telepresence. We are working with such devices as the Microsoft Kinect, the Oculus Rift, the Leap Motion, the HTC Vive, motion capture systems, and the Microsoft Hololens. The focus of our work has been on human interaction with the virtual environment, which in turn acts as a communications bridge to remote physical devices and environments which the operator cannot or should not control or experience directly. Particularly in reference to dealing with spacecraft and the oftentimes hazardous environments they inhabit, it is our hope that AR and VR technologies can be utilized to increase human safety and mission success by physically removing humans from those hazardous environments while virtually putting them right in the middle of those environments.

Little, William↗

NASA NTRS2017

Incorporating Speech Recognition into a Natural User Interface

The Augmented/ Virtual Reality (AVR) Lab has been working to study the applicability of recent virtual and augmented reality hardware and software to KSC operations. This includes the Oculus Rift, HTC Vive, Microsoft HoloLens, and Unity game engine. My project in this lab is to integrate voice recognition and voice commands into an easy to modify system that can be added to an existing portion of a Natural User Interface (NUI). A NUI is an intuitive and simple to use interface incorporating visual, touch, and speech recognition. The inclusion of speech recognition capability will allow users to perform actions or make inquiries using only their voice. The simplicity of needing only to speak to control an on-screen object or enact some digital action means that any user can quickly become accustomed to using this system. Multiple programs were tested for use in a speech command and recognition system. Sphinx4 translates speech to text using a Hidden Markov Model (HMM) based Language Model, an Acoustic Model, and a word Dictionary running on Java. PocketSphinx had similar functionality to Sphinx4 but instead ran on C. However, neither of these programs were ideal as building a Java or C wrapper slowed performance. The most ideal speech recognition system tested was the Unity Engine Grammar Recognizer. A Context Free Grammar (CFG) structure is written in an XML file to specify the structure of phrases and words that will be recognized by Unity Grammar Recognizer. Using Speech Recognition Grammar Specification (SRGS) 1.0 makes modifying the recognized combinations of words and phrases very simple and quick to do. With SRGS 1.0, semantic information can also be added to the XML file, which allows for even more control over how spoken words and phrases are interpreted by Unity. Additionally, using a CFG with SRGS 1.0 produces a Finite State Machine (FSM) functionality limiting the potential for incorrectly heard words or phrases. The purpose of my project was to investigate options for a Speech Recognition System. To that end I attempted to integrate Sphinx4 into a user interface. Sphinx4 had great accuracy and is the only free program able to perform offline speech dictation. However it had a limited dictionary of words that could be recognized, single syllable words were almost impossible for it to hear, and since it ran on Java it could not be integrated into the Unity based NUI. PocketSphinx ran much faster than Sphinx4 which would've made it ideal as a plugin to the Unity NUI, unfortunately creating a C# wrapper for the C code made the program unusable with Unity due to the wrapper slowing code execution and class files becoming unreachable. Unity Grammar Recognizer is the ideal speech recognition interface, it is flexible in recognizing multiple variations of the same command. It is also the most accurate program in recognizing speech due to using an XML grammar to specify speech structure instead of relying solely on a Dictionary and Language model. The Unity Grammar Recognizer will be used with the NUI for these reasons as well as being written in C# which further simplifies the incorporation.

Chapa, Nicholas↗

NASA NTRS2016

Hybrid Reality Lab Capabilities - Video 2

Our Hybrid Reality and Advanced Operations Lab is developing incredibly realistic and immersive systems that could be used to provide training, support engineering analysis, and augment data collection for various human performance metrics at NASA. To get a better understanding of what Hybrid Reality is, let's go through the two most commonly known types of immersive realities: Virtual Reality, and Augmented Reality. Virtual Reality creates immersive scenes that are completely made up of digital information. This technology has been used to train astronauts at NASA, used during teleoperation of remote assets (arms, rovers, robots, etc.) and other activities. One challenge with Virtual Reality is that if you are using it for real time-applications (like landing an airplane) then the information used to create the virtual scenes can be old (i.e. visualized long after physical objects moved in the scene) and not accurate enough to land the airplane safely. This is where Augmented Reality comes in. Augmented Reality takes real-time environment information (from a camera, or see through window, and places digitally created information into the scene so that it matches with the video/glass information). Augmented Reality enhances real environment information collected with a live sensor or viewport (e.g. camera, window, etc.) with the information-rich visualization provided by Virtual Reality. Hybrid Reality takes Augmented Reality even further, by creating a higher level of immersion where interactivity can take place. Hybrid Reality takes Virtual Reality objects and a trackable, physical representation of those objects, places them in the same coordinate system, and allows people to interact with both objects' representations (virtual and physical) simultaneously. After a short period of adjustment, the individuals begin to interact with all the objects in the scene as if they were real-life objects. The ability to physically touch and interact with digitally created objects that have the same shape, size, location to their physical object counterpart in virtual reality environment can be a game changer when it comes to training, planning, engineering analysis, science, entertainment, etc. Our Project is developing such capabilities for various types of environments. The video outlined with this abstract is a representation of an ISS Hybrid Reality experience. In the video you can see various Hybrid Reality elements that provide immersion beyond just standard Virtual Reality or Augmented Reality.

Delgado, Francisco J.↗

NASA NTRS2018

VECTR: Virtual Environment Computational Training Resource

The Westridge Middle School Curriculum and Community Night is an annual event designed to introduce students and parents to potential employers in the Central Florida area. NASA participated in the event in 2017, and has been asked to come back for the 2018 event on January 25. We will be demonstrating our Microsoft Hololens Virtual Rovers project, and the Virtual Environment Computational Training Resource (VECTR) virtual reality tool.

Augmented Reality↗

NASA NTRS2005

Data visualization for effective rover sequencing

The Rover Sequencing and Visualization Program is a suite of toos for the commanding of planetary rovers which are subject to significant light time delay and thus are unsuitable for tele-operation.

surface operations↗

NASA NTRS2014

The Evolution of Three Dimensional Visualization for Commanding the Mars Rovers

NASA's Jet Propulsion Laboratory has built and operated four rovers on the surface of Mars. Two and three dimensional visualization has been extensively employed to command both the mobility and robotic arm operations of these rovers. Stereo visualization has been an important component in this set of visualization techniques. This paper discusses the progression of the implementation and use of visualization techniques for in-situ operations of these robotic missions. Illustrative examples will be drawn from the results of using these techniques over more than ten years of surface operations on Mars.

stereo visualization↗

NASA NTRS2019

Deep Learning Applications in Manned Spaceflight

This presentation discusses a basic overview of Deep Machine Learning (DML) fundamentals, and two JSC applications of DML to create an Intelligent Personal Coach for exercise applications on deep-space missions, and the training of a neural network using the SingleShotPose algorithm from Microsoft to detect object 6 degree of freedom pose information from 2D image data for use in an Intelligent Procedure Assistant. The presentation concludes with a discussion about conceptual future uses of DML for space missions.

augmented reality↗

NASA NTRS2023

Deep Learning for Space Applications

Explore the source record for details and available documents.

Augmented Reality↗

NASA NTRSDate not supplied

Discovering Digital Engineering Needs Through Human-Centered Approaches

Human-centered Design↗

NASA NTRS2020

Modeling and Simulation - Virtual

The purpose of this report is to describe the body of work I have produced as a NASA Office of Science Technology Engineering and Math (OSTEM) intern in the Fall 2020 semester. My mentor during this session was Antonio Pego and my main task was to research and develop solutions in augmented and virtual reality technologies for use in training simulations. A main focus of this project was the development of human models for use in the game engine, with the purpose of creating simulations that would help with realistic analysis of tasks that will be performed by personnel. A steep learning curve had to be tackled in order to learn and catch up with current MR technologies. Along the way I created documentation of the different techniques utilized to support future continuation of the project.

Peter Leroy Santana Rodriguez↗

NASA NTRSDate not supplied

NASA's Research & Development in Airspace Management for Drones, Air Taxis, and Beyond

Today's National Airspace System is extremely safe. Multiple people and organizations are involved in the handling of flights at all phases. We are, however, at the dawn of a new era in aviation where new entrants are emerging from multiple domains. These new entrants will need to be integrated into the airspace safely and given the flexibility to scale. With that integration and flexibility comes the need and responsibility to develop concepts and systems that will work with the regulatory and operational needs of the Federal Aviation Administration while accounting for the needs of industry. NASA is at the leading edge of research, development, and testing across multiple domains to enable the realization of new entrants such as drones, air cargo and taxis, and high altitude vehicles. Addressing the potential for public good coming from these new capabilities is also a NASA focus area. While there has been considerable effort in the engineering aspects of this research, another thrust that is of importance is that of data integration and visualization. With respect to visualization, NASA Ames Research Center has been leading the advancement in data visualization for these new domains and has spent considerable efforts in the exploration of incorporating Augmented and Mixed Reality as a situation awareness platform. This presentation will outline the new developments in airspace integration and tie in NASA's efforts in visualizing the associated operational data.

airspace management↗

NASA NTRSDate not supplied

Immersive Technologies for Human-in-the-Loop Lunar Surface Simulations

NASA, the National Aeronautics and Space Administration, continually seeks innovative solutions to enhance its operations, particularly in the realms of testing, evaluation, and training for future missions. Immersive technologies, such as virtual, augmented, and mixed reality have proven to be powerful tools for realistic, interactive, and engaging environments. This paper explores how the Simulation and Graphics Branch at NASA’s Johnson Space Center (JSC) leverages immersive technology, modern commercial rendering engines, and physics-based systems simulations to develop human-in-the-loop systems for humanity’s return to the Moon through the Artemis program. When NASA returns to the Moon, astronauts will travel to the Moon’s South Pole where lighting conditions will cause a more complex operational environment. Human-in-the-loop testing plays a crucial role in NASA's mission planning, spacecraft and space systems development, and evaluation of operational scenarios. The development of immersive environments such as a lunar rover mockup at a video wall enables engineers and astronauts to simulate and experience mission scenarios, integrated spacecraft systems, and operational procedures in a relevant environment before deployment. By integrating realistic virtual environments, immersive technology allows for the visualization and interaction with virtual spacecraft models, mission landscapes, and complex operational tasks. This approach helps identify potential design flaws, operational challenges, and safety considerations. It also provides valuable insights for risk reduction and helps improve mission efficiency and effectiveness. With advanced motion tracking systems and custom virtual environments data can be gathered and evaluated to help NASA refine training protocols, develop specialized training procedures and optimize human-robotic interactions for future space missions. Furthermore, immersive technology offers opportunities for future training initiatives at NASA. The Virtual Reality Laboratory at JSC has pioneered training with Virtual Reality (VR) since the Hubble Space Telescope repair missions in the early 1990’s. Extended Reality (XR) simulations enable astronauts to rehearse complex spacewalks, spacecraft maneuvers, and extravehicular activities in a safe and controlled environment. By replicating the physical and cognitive challenges of space missions, immersive training experiences enhance astronauts' situational awareness, decision-making abilities, and adaptability to unexpected scenarios. Additionally, immersive technology facilitates collaborative training, allowing geographically dispersed crew and mission control personnel to engage in synchronized simulations, fostering teamwork and effective communication. The adoption of immersive technology in NASA's testing, evaluation, and future training programs has yielded significant benefits. By incorporating human-in-the-loop testing for studies involving Extra Vehicular Activities (EVA), surface mobility and landing systems, NASA can identify and mitigate risks, optimize operational procedures, and enhance mission success. Ultimately, immersive training experiences can empower astronauts to better navigate the complexities of space missions, ensuring their safety, productivity, and success in the dynamic and challenging environments they will experience at the Lunar South Pole.

Simulation Modeling Virtual Reality Immersive Tech↗

NASA NTRSDate not supplied

Immersive Technologies for Human-in-the-Loop Lunar Surface Simulations

NASA, the National Aeronautics and Space Administration, continually seeks innovative solutions to enhance its operations, particularly in the realms of testing, evaluation, and training for future missions. Immersive technologies, such as virtual, augmented, and mixed reality have proven to be powerful tools for immersing users in realistic, interactive, and engaging environments. This paper explores how the Simulation and Graphics Branch at NASA’s Johnson Space Center (JSC) leverages immersive technology, modern commercial rendering engines, and physics-based systems simulations to develop human-in-the-loop systems for humanity’s return to the Moon through the Artemis program. When NASA returns to the Moon, astronauts will travel to the Moon’s South Pole where lighting conditions will cause a more complex operational environment. Human-in-the-loop simulations play a crucial role in NASA’s mission planning, spacecraft and space systems development, and evaluation of operational scenarios. The development of immersive environments such as a lunar rover mockup at a video wall enables engineers and astronauts to simulate and experience mission scenarios, integrated spacecraft systems, and operational procedures in a relevant environment before deployment. By integrating realistic virtual environments, immersive technology allows for the visualization and interaction with virtual spacecraft models, mission landscapes, and complex operational tasks. This approach helps identify potential design flaws, operational challenges, and safety considerations. It also provides valuable insights for risk reduction and helps improve mission efficiency and effectiveness. With advanced motion tracking systems and custom virtual environments, data can be gathered and evaluated to help NASA refine training protocols, develop specialized training procedures, and optimize human-robotic interactions for future space missions. Furthermore, immersive technology offers opportunities for future training initiatives at NASA. The Virtual Reality Laboratory at JSC has pioneered training with Virtual Reality (VR) since the Hubble Space Telescope repair missions in the early 1990’s. Extended Reality (XR) simulations enable astronauts to rehearse complex spacewalks, spacecraft maneuvers, and extravehicular activities in a safe and controlled environment. By replicating the physical and cognitive challenges of space missions, immersive training experiences enhance astronauts’ situational awareness, decision-making abilities, and adaptability to unexpected scenarios. Additionally, immersive technology facilitates collaborative training, allowing geographically dispersed crew and mission control personnel to engage in synchronized simulations, fostering teamwork and effective communication. The adoption of immersive technology in NASA’s testing, evaluation, and future training programs has yielded significant benefits. By incorporating human-in-the-loop simulations for studies involving Extra Vehicular Activities (EVA), surface mobility and landing systems, NASA can identify and mitigate risks, optimize operational procedures, and enhance mission success. Ultimately, immersive simulation experiences can empower astronauts to better navigate the complexities of space missions, ensuring their safety, productivity, and success in the dynamic and challenging environments they will experience at the Lunar South Pole.

HITL↗

NASA NTRS1992

Virtual reality and planetary exploration

NASA-Ames is intensively developing virtual-reality (VR) capabilities that can extend and augment computer-generated and remote spatial environments. VR is envisioned not only as a basis for improving human/machine interactions involved in planetary exploration, but also as a medium for the more widespread sharing of the experience of exploration, thereby broadening the support-base for the lunar and planetary-exploration endeavors. Imagery representative of Mars are being gathered for VR presentation at such terrestrial sites as Antarctica and Death Valley.

Mcgreevy, Michael W.↗

NASA NTRS1992

Virtual reality and planetary exploration

Exploring planetary environments is central to NASA's missions and goals. A new computing technology called Virtual Reality has much to offer in support of planetary exploration. This technology augments and extends human presence within computer-generated and remote spatial environments. Historically, NASA has been a leader in many of the fundamental concepts and technologies that comprise Virtual Reality. Indeed, Ames Research Center has a central role in the development of this rapidly emerging approach to using computers. This ground breaking work has inspired researchers in academia, industry, and the military. Further, NASA's leadership in this technology has spun off new businesses, has caught the attention of the international business community, and has generated several years of positive international media coverage. In the future, Virtual Reality technology will enable greatly improved human-machine interactions for more productive planetary surface exploration. Perhaps more importantly, Virtual Reality technology will democratize the experience of planetary exploration and thereby broaden understanding of, and support for, this historic enterprise.

Mcgreevy, Michael W.↗

NASA NTRSDate not supplied

Improving Sim-to-Real Transfer in Vision-Based Robot Navigation Via Instance-Level GAN-Based Data Augmentation

Achieving robust vision-based robotic tasks requires large amounts of data, which are often difficult to obtain in real-world scenarios. Simulators and synthetic data offer a cost-effective alternative, but the visual gap between simulation and reality hinders the performance of models when deployed in real-world environments. In this paper, we present a data augmentation pipeline that integrates a foundation model (Segment Anything Model) with an unsupervised image-to-image translation model (CycleGAN) for instance-level domain transfer from simulation to reality. This pipeline enables the generation of realistic labeled data from synthetic images for training supervised machine learning models in vision-based navigation tasks. We evaluate our approach on real-world data for ego-vehicle pose estimation, a critical autonomous navigation task involving the prediction of cross-track position and heading angle relative to road center line markings. The results of our tests show that our GAN-based data augmentation pipeline significantly outperforms models trained solely on simulation data or on data processed with standard image augmentation methods for sim-to-real transfer, enhancing model robustness and generalizability in real-world scenarios. Our method provides a scalable and flexible data augmentation tool for leveraging large synthetic datasets to enhance vision-based robotic navigation tasks.

artificial intelligence↗