Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “natural user interfaces”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Exploring Multimodal Interactions in Human-Autonomy Teaming Using a Natural User Interface

The creation of a multimodal, natural user interface to facilitate multi-agent interaction is essential to establishing trust among human and machine teammates in multi-agent systems. Trust is being researched, along with trustworthiness, as a path to certification of autonomous systems by the Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR) project at NASA. The Autonomous Mission Experimental Logistics Interactive Assistant (AMELIA) is a natural user interface that enables multimodal interaction and is designed for rapid mission planning. AMELIA is an intelligent system that considers the user’s preferred communication strategies, as well as the time-critical aspect of the multi-agent system decision-making process. Twenty-four participants planned a multi-agent search and rescue mission, with the aid of an intelligent assistant. The results show that while the combined use of touch and speech was faster than speech alone, the single modality, touch, was still the most efficient. Future research should investigate additional input technologies.

Lisa R Le Vie↗

Exploring Multimodal Interactions in Human-Autonomy Teaming Using a Natural User Interface

The creation of a multimodal, natural user interface to facilitate multi-agent interaction is essential to establishing trust among human and machine teammates in multi-agent systems. Trust is being researched, along with trustworthiness, as a path to certification of autonomous systems by the Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR) project at NASA. The Autonomous Mission Experimental Logistics Interactive Assistant (AMELIA) is a natural user interface that enables multimodal interaction and is designed for rapid mission planning. AMELIA is an intelligent system that considers the user’s preferred communication strategies, as well as the time-critical aspect of the multi-agent system decision-making process. Twenty-four participants planned a multi-agent search and rescue mission, with the aid of an intelligent assistant. The results show that while the combined use of touch and speech was faster than speech alone, the single modality, touch, was still the most efficient. Future research should investigate additional input technologies.

Lisa Renee Le Vie↗

Incorporating Speech Recognition into a Natural User Interface

The Augmented/ Virtual Reality (AVR) Lab has been working to study the applicability of recent virtual and augmented reality hardware and software to KSC operations. This includes the Oculus Rift, HTC Vive, Microsoft HoloLens, and Unity game engine. My project in this lab is to integrate voice recognition and voice commands into an easy to modify system that can be added to an existing portion of a Natural User Interface (NUI). A NUI is an intuitive and simple to use interface incorporating visual, touch, and speech recognition. The inclusion of speech recognition capability will allow users to perform actions or make inquiries using only their voice. The simplicity of needing only to speak to control an on-screen object or enact some digital action means that any user can quickly become accustomed to using this system. Multiple programs were tested for use in a speech command and recognition system. Sphinx4 translates speech to text using a Hidden Markov Model (HMM) based Language Model, an Acoustic Model, and a word Dictionary running on Java. PocketSphinx had similar functionality to Sphinx4 but instead ran on C. However, neither of these programs were ideal as building a Java or C wrapper slowed performance. The most ideal speech recognition system tested was the Unity Engine Grammar Recognizer. A Context Free Grammar (CFG) structure is written in an XML file to specify the structure of phrases and words that will be recognized by Unity Grammar Recognizer. Using Speech Recognition Grammar Specification (SRGS) 1.0 makes modifying the recognized combinations of words and phrases very simple and quick to do. With SRGS 1.0, semantic information can also be added to the XML file, which allows for even more control over how spoken words and phrases are interpreted by Unity. Additionally, using a CFG with SRGS 1.0 produces a Finite State Machine (FSM) functionality limiting the potential for incorrectly heard words or phrases. The purpose of my project was to investigate options for a Speech Recognition System. To that end I attempted to integrate Sphinx4 into a user interface. Sphinx4 had great accuracy and is the only free program able to perform offline speech dictation. However it had a limited dictionary of words that could be recognized, single syllable words were almost impossible for it to hear, and since it ran on Java it could not be integrated into the Unity based NUI. PocketSphinx ran much faster than Sphinx4 which would've made it ideal as a plugin to the Unity NUI, unfortunately creating a C# wrapper for the C code made the program unusable with Unity due to the wrapper slowing code execution and class files becoming unreachable. Unity Grammar Recognizer is the ideal speech recognition interface, it is flexible in recognizing multiple variations of the same command. It is also the most accurate program in recognizing speech due to using an XML grammar to specify speech structure instead of relying solely on a Dictionary and Language model. The Unity Grammar Recognizer will be used with the NUI for these reasons as well as being written in C# which further simplifies the incorporation.

Chapa, Nicholas↗

Developing a Natural User Interface and Facial Recognition System With OpenCV and the Microsoft Kinect

The task for this project was to design, develop, test, and deploy a facial recognition system for the Kennedy Space Center Augmented/Virtual Reality Lab. This system will serve as a means of user authentication as part of the NUI of the lab. The overarching goal is to create a seamless user interface that will allow the user to initiate and interact with AR and VR experiences without ever needing to use a mouse or keyboard at any step in the process.

Human Interfaces↗

Force Control and Nonlinear Master-Slave Force Profile to Manage an Admittance Type Multi-Fingered Haptic User Interface

Natural movements and force feedback are important elements in using teleoperated equipment if complex and speedy manipulation tasks are to be accomplished in remote and/or hazardous environments, such as hot cells, glove boxes, decommissioning, explosives disarmament, and space to name a few. In order to achieve this end the research presented in this paper has developed an admittance type exoskeleton like multi-fingered haptic hand user interface that secures the user’s palm and provides 3-dimensional force feedback to the user’s fingertips. Atypical to conventional haptic hand user interfaces that limit themselves to integrating the human hand’s characteristics just into the system’s mechanical design this system also perpetuates that inspiration into the designed user interface’s controller. This is achieved by manifesting the property differences of manipulation and grasping activities as they pertain to the human hand into a nonlinear master-slave force relationship. The results presented in this paper show that the admittance-type system has sufficient bandwidth that it appears nearly transparent to the user when the user is in free motion and when the system is subjected to a manipulation task, increased performance is achieved using the nonlinear force relationship compared to the traditional linear scaling techniques implemented in the vast majority of systems.

42 ENGINEERING↗

The Next Wave: Humans, Computers, and Redefining Reality

The Augmented/Virtual Reality (AVR) Lab at KSC is dedicated to " exploration into the growing computer fields of Extended Reality and the Natural User Interface (it is) a proving ground for new technologies that can be integrated into future NASA projects and programs." The topics of Human Computer Interface, Human Computer Interaction, Augmented Reality, Virtual Reality, and Mixed Reality are defined; examples of work being done in these fields in the AVR Lab are given. Current new and future work in Computer Vision, Speech Recognition, and Artificial Intelligence are also outlined.

Extended Reality↗

The Virtual Tablet: Virtual Reality as a Control System

In the field of human-computer interaction, Augmented Reality (AR) and Virtual Reality (VR) have been rapidly growing areas of interest and concerted development effort thanks to both private and public research. At NASA, a number of groups have explored the possibilities afforded by AR and VR technology, among which is the IT Advanced Concepts Lab (ITACL). Within ITACL, the AVR (Augmented/Virtual Reality) Lab focuses on VR technology specifically for its use in command and control. Previous work in the AVR lab includes the Natural User Interface (NUI) project and the Virtual Control Panel (VCP) project, which created virtual three-dimensional interfaces that users could interact with while wearing a VR headset thanks to body- and hand-tracking technology. The Virtual Tablet (VT) project attempts to improve on these previous efforts by incorporating a physical surrogate which is mirrored in the virtual environment, mitigating issues with difficulty of visually determining the interface location and lack of tactile feedback discovered in the development of previous efforts. The physical surrogate takes the form of a handheld sheet of acrylic glass with several infrared-range reflective markers and a sensor package attached. Using the sensor package to track orientation and a motion-capture system to track the marker positions, a model of the surrogate is placed in the virtual environment at a position which corresponds with the real-world location relative to the user's VR Head Mounted Display (HMD). A set of control mechanisms is then projected onto the surface of the surrogate such that to the user, immersed in VR, the control interface appears to be attached to the object they are holding. The VT project was taken from an early stage where the sensor package, motion-capture system, and physical surrogate had been constructed or tested individually but not yet combined or incorporated into the virtual environment. My contribution was to combine the pieces of hardware, write software to incorporate each piece of position or orientation data into a coherent description of the object's location in space, place the virtual analogue accordingly, and project the control interface onto it, resulting in a functioning object which has both a physical and a virtual presence. Additionally, the virtual environment was enhanced with two live video feeds from cameras mounted on the robotic device being used as an example target of the virtual interface. The working VT allows users to naturally interact with a control interface with little to no training and without the issues found in previous efforts.

virtual control interface↗

Teleoperation of Robonaut Using Finger Tracking

With the advent of new finger tracking systems, the idea of a more expressive and intuitive user interface is being explored and implemented. One practical application for this new kind of interface is that of teleoperating a robot. For humanoid robots, a finger tracking interface is required due to the level of complexity in a human-like hand, where a joystick isn't accurate. Moreover, for some tasks, using one's own hands allows the user to communicate their intentions more effectively than other input. The purpose of this project was to develop a natural user interface for someone to teleoperate a robot that is elsewhere. Specifically, this was designed to control Robonaut on the international space station to do tasks too dangerous and/or too trivial for human astronauts. This interface was developed by integrating and modifying 3Gear's software, which includes a library of gestures and the ability to track hands. The end result is an interface in which the user can manipulate objects in real time in the user interface. then, the information is relayed to a simulator, the stand in for Robonaut, at a slight delay.

robotics↗

Crowdsourcing, citizen sensing and Sensor Web technologies for public and environmental health surveillance and crisis management: trends, OGC standards and application examples

The PIE Activity Awareness Environment is designed to be an adaptive data triage and decision support tool that allows role and activity based situation awareness through a dynamic, trainable filtering system. This paper discusses the process and methodology involved in the application as well as some of its capabilities. 'Wikification of GIS by the masses' is a phrase-term first coined by Kamel Boulos in 2005, two years earlier than Goodchild's term 'Volunteered Geographic Information'. Six years later (2005-2011), OpenStreetMap and Google Earth (GE) are now full-fledged, crowdsourced 'Wikipedias of the Earth' par excellence, with millions of users contributing their own layers to GE, attaching photos, videos, notes and even 3-D (three dimensional) models to locations in GE. From using Twitter in participatory sensing and bicycle-mounted sensors in pervasive environmental sensing, to creating a 100,000-sensor geo-mashup using Semantic Web technology, to the 3-D visualisation of indoor and outdoor surveillance data in real-time and the development of next-generation, collaborative natural user interfaces that will power the spatially-enabled public health and emergency situation rooms of the future, where sensor data and citizen reports can be triaged and acted upon in real-time by distributed teams of professionals, this paper offers a comprehensive state-of-the-art review of the overlapping domains of the Sensor Web, citizen sensing and 'human-in-the-loop sensing' in the era of the Mobile and Social Web, and the roles these domains can play in environmental and public health surveillance and crisis/disaster informatics. We provide an in-depth review of the key issues and trends in these areas, the challenges faced when reasoning and making decisions with real-time crowdsourced data (such as issues of information overload, 'noise', misinformation, bias and trust), the core technologies and Open Geospatial Consortium (OGC) standards involved (Sensor Web Enablement and Open GeoSMS), as well as a few outstanding project implementation examples from around the world.

59 BASIC BIOLOGICAL SCIENCES↗

Advanced data management system architectures testbed

The objective of the Architecture and Tools Testbed is to provide a working, experimental focus to the evolving automation applications for the Space Station Freedom data management system. Emphasis is on defining and refining real-world applications including the following: the validation of user needs; understanding system requirements and capabilities; and extending capabilities. The approach is to provide an open, distributed system of high performance workstations representing both the standard data processors and networks and advanced RISC-based processors and multiprocessor systems. The system provides a base from which to develop and evaluate new performance and risk management concepts and for sharing the results. Participants are given a common view of requirements and capability via: remote login to the testbed; standard, natural user interfaces to simulations and emulations; special attention to user manuals for all software tools; and E-mail communication. The testbed elements which instantiate the approach are briefly described including the workstations, the software simulation and monitoring tools, and performance and fault tolerance experiments.

Grant, Terry↗

White Paper for Virtual Control Room

The Virtual Control Room (VCR) Proof of Concept (PoC) project is the result of an award given by the Fourth Annual NASA T&I Labs Challenge Project Call. This paper will outline the work done over the award period to build and enhance the capabilities of the Augmented/Virtual Reality (AVR) Lab at NASA's Kennedy Space Center (KSC) to create the VCR.

Virtual Reality↗

A common path forward for the immersive visualization community

Immersive visualization makes use of the medium of virtual reality (VR) - it is a subset of virtual reality focused on the application of VR technologies to scientific and information visualization. As the name implies, there is a particular focus on the physically immersive aspect of VR that more fully engages the perceptual and kinesthetic capabilities of the scientist with the goal of producing greater insight. The immersive visualization community is uniquely positioned to address the analysis needs of the wide spectrum of domain scientists who are becoming increasingly overwhelmed by data. The outputs of computational science simulations and high-resolution sensors are creating a data deluge. Data is coming in faster than it can be analyzed, and there are countless opportunities for discovery that are missed as the data speeds by. By more fully utilizing the scientists visual and other sensory systems, and by offering a more natural user interface with which to interact with computer-generated representations, immersive visualization offers great promise in taming this data torrent. However, increasing the adoption of immersive visualization in scientific research communities can only happen by simultaneously lowering the engagement threshold while raising the measurable benefits of adoption. Scientists time spent immersed with their data will thus be rewarded with higher productivity, deeper insight, and improved creativity. Immersive visualization ties together technologies and methodologies from a variety of related but frequently disjoint areas, including hardware, software and human-computer interaction (HCI) disciplines. In many ways, hardware is a solved problem. There are well established technologies including large walk-in systems such as the CAVE{trademark} and head-based systems such as the Wide-5{trademark}. The advent of new consumer-level technologies now enable an entirely new generation of immersive displays, with smaller footprints and costs, widening the potential consumer base. While one would be hard-pressed to call software a solved problem, we now understand considerably more about best practices for designing and developing sustainable, scalable software systems, and we have useful software examples that illuminate the way to even better implementations. As with any research endeavour, HCI will always be exploring new topics in interface design, but we now have a sizable knowledge base of the strengths and weaknesses of the human perceptual systems and we know how to design effective interfaces for immersive systems. So, in a research landscape with a clear need for better visualization and analysis tools, a methodology in immersive visualization that has been shown to effectively address some of those needs, and vastly improved supporting technologies and knowledge of hardware, software, and HCI, why hasn't immersive visualization 'caught on' more with scientists? What can we do as a community of immersive visualization researchers and practitioners to facilitate greater adoption by scientific communities so as to make the transition from 'the promise of virtual reality' to 'the reality of virtual reality'.

97 MATHEMATICS AND COMPUTING↗

A VR-based volumetric medical image segmentation and visualization system with natural human interaction

Volume rendering produces informative two-dimensional (2D) images from a 3-dimensional (3D) volume. It highlights the region of interest and facilitates a good comprehension of the entire data set. However, volume rendering faces a few challenges. First, a high-dimensional transfer function is usually required to differentiate the target from its neighboring objects with subtle variance. Unfortunately, designing such a transfer function is a strenuously trial-and-error process. Second, manipulating/visualizing a 3D volume with a traditional 2D input/output device suffers dimensional limitations. To address all the challenges, we design NUI-VR 2 , a natural user interface-enabled volume rendering system in the virtual reality space. NUI-VR 2 marries volume rendering and interactive image segmentation. It transforms the original volume into a probability map with image segmentation. A simple linear transfer function will highlight the target well in the probability map. More importantly, we set the entire image segmentation and volume rendering pipeline in an immersive virtual reality environment with a natural user interface. NUI-VR 2 eliminates the dimensional limitations in manipulating and perceiving 3D volumes and dramatically improves the user experience.

42 ENGINEERING↗

Interactive Quantum Chemistry Enabled by Machine Learning, Graphical Processing Units, and Cloud Computing

Modern quantum chemistry algorithms are increasingly able to accurately predict molecular properties that are useful for chemists in research and education. Despite this progress, performing such calculations is currently unattainable to the wider chemistry community, as they often require domain expertise, computer programming skills, and powerful computer hardware. In this review, we outline methods to eliminate these barriers using cutting-edge technologies. We discuss the ingredients needed to create accessible platforms that can compute quantum chemistry properties in real time, including graphical processing units–accelerated quantum chemistry in the cloud, artificial intelligence–driven natural molecule input methods, and extended reality visualization. We end by highlighting a series of exciting applications that assemble these components to create uniquely interactive platforms for computing and visualizing spectra, 3D structures, molecular orbitals, and many other chemical properties.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Anthropomorphic teleoperation: Controlling remote manipulators with the DataGlove

A two phase effort was conducted to assess the capabilities and limitations of the DataGlove, a lightweight glove input device that can output signals in real-time based on hand shape, orientation, and movement. The first phase was a period for system integration, checkout, and familiarization in a virtual environment. The second phase was a formal experiment using the DataGlove as input device to control the protoflight manipulator arm (PFMA) - a large telerobotic arm with an 8-ft reach. The first phase was used to explore and understand how the DataGlove functions in a virtual environment, build a virtual PFMA, and consider and select a reasonable teleoperation control methodology. Twelve volunteers (six males and six females) participated in a 2 x 3 (x 2) full-factorial formal experiment using the DataGlove to control the PFMA in a simple retraction, slewing, and insertion task. Two within-subjects variables, time delay (0, 1, and 2 seconds) and PFMA wrist flexibility (rigid/flexible), were manipulated. Gender served as a blocking variable. A main effect of time delay was found for slewing and total task times. Correlations among questionnaire responses, and between questionnaire responses and session mean scores and gender were computed. The experimental data were also compared with data collected in another study that used a six degree-of-freedom handcontroller to control the PFMA in the same task. It was concluded that the DataGlove is a legitimate teleoperations input device that provides a natural, intuitive user interface. From an operational point of view, it compares favorably with other 'standard' telerobotic input devices and should be considered in future trades in teleoperation systems' designs.

Hale, J. P., II↗

Affinity+: Semi-Structured Brainstorming on Large Displays

Affinity diagraming is a powerful method for encouraging and capturing lateral thinking in a group environment. The Affinity+ Concept was designed to improve the collaborative brainstorm process through the use of large display surfaces in conjunction with mobile devices like smart phones and tablets. The system works by capturing the ideas digitally and allowing users to sort and group them on a large touch screen manually. Additionally, Affinity+ incorporates theme detection, topic clustering, and other processing algorithms that help bring structured analytic techniques to the process without requiring explicit leadership roles and other overhead typically involved in these activities.

large display, natural user interface, ubiquitous ↗

Bias by default? A means for a priori interface measurement

Systems have biases. Their interfaces naturally guide a user toward specific patterns of action. For example, modern word-processors and spreadsheets are both capable of taking word wrapping, checking spelling, storing tables, and calculating formulas. You could write a paper in a spreadsheet or could do simple business modeling in a word-processor. However, their interfaces naturally communicate which function they are designed for. Visual analytic interfaces also have biases. In this paper, we outline why simple Markov models are a plausible tool for investigating that bias and how they might be applied. We also discuss some anticipated difficulties in such modeling and touch briefly on what some Markov model extensions might provide.

Cottam, Joseph A.↗

Sentiment of Search: KM and IT for User Expectations

User perceived value is the number one indicator of a successful implementation of KM and IT collaborations. The system known as "Search" requires more strategy and workflow that a mere data dump or ungoverned infrastructure can provide. Monitoring of user sentiment can be a driver for providing objective measures of success and justifying changes to the user interface. The dynamic nature of information technology makes traditional usability metrics difficult to identify, yet easy to argue against. There is little disagreement, however, on the criticality of adapting to user needs and expectations. The Systems Usability Scale (SUS), developed by John Brook in 1986 has become an industry standard for usability engineering. The first phase of a modified SUS, polls the sentiment of representative users of the JSC Search system. This information can be used to correlate user determined value with types of information sought and how the system is (or is not) meeting expectations. Sentiment analysis by way of the SUS assists an organization in identification and prioritization of the KM and IT variables impacting user perceived value. A secondary, user group focused analysis is the topic of additional work that demonstrates the impact of specific changes dictated by user sentiment.

Berndt, Sarah Ann↗