Engineering topics
Matthies, Larry
Publications and source records attributed to Matthies, Larry.
Gaussian Mixture Models for Temporal Depth Fusion
Sensing the 3D environment of a moving robot is essential for collision avoidance. Most 3D sensors produce dense depth maps, which are subject to imperfections due to various environmental factors. Temporal fusion of depth maps is crucial to overcome those. Temporal fusion is traditionally done in 3D space with voxel data structures, but it can be approached by temporal fusion in image space, with potential benefits in reduced memory and computational cost for applications like reactive collision avoidance for micro air vehicles. In this paper, we present an efficient Gaussian Mixture Models based depth map fusion approach, introducing an online update scheme for dense representations. The environment is modeled from an ego-centric point of view, where each pixel is represented by a mixture of Gaussian inverse-depth models. Consecutive frames are related to each other by transformations obtained from visual odometry. This approach achieves better accuracy than alternative image space depth map fusion techniques at lower computational cost.
NIAC Phase 1 Final Study Report on Titan Aerial Daughtercraft
Saturns giant moon Titan has become one of the most fascinating bodies in the Solar System. Even though it is a billion miles from Earth, data from the Cassini mission reveals that Titan has a very diverse, Earth-like surface, with mountains, fluvial channels, lakes, evaporite basins, plains, dunes, and seas [Lopes 2010] (Figure 1). But unlike Earth, Titans surface likely is composed of organic chemistry products derived from complex atmospheric photochemistry [Lorenz 2008]. In addition, Titan has an active meteorological system with observed storms and precipitation-induced surface darkening suggesting a hydrocarbon cycle analogous to Earths water cycle [Turtle 2011].Titan is the richest laboratory in the solar system for studying prebiotic chemistry, which makes studying its chemistry from the surface and in the atmosphere one of the most important objectives in planetary science [Decadal 2011]. The diversity of surface features on Titan related to organic solids and liquids makes long-range mobility with surface access important [Decadal 2011]. This has not been possible to date, because mission concepts have had either no mobility (landers), no surface access (balloons and airplanes), or low maturity, high risk, and/or high development costs for this environment (e,g. large, self-sufficient, long-duration helicopters). Enabling in situ mobility could revolutionize Titan exploration, similarly to the way rovers revolutionized Mars exploration. Recent progress on several fronts has suggested that small-scale rotorcraft deployed as daughtercraft from a lander or balloon mothercraft may be an effective, affordable approach to expanding Titan surface access. This includes rapid progress on autonomous navigation capabilities of such aircraft for terrestrial applications and on miniaturization, driven by the consumer mobile electronics market, of high performance of sensors, processors, and other avionics components needed for such aircraft. Chemical analysis, for example with a mass spectrometer, will be important to any Titan surface mission. Anticipating that it may be more practical to host chemical analysis instruments on a mothership than a daughtercraft, we defined system and mission concepts that deploy a small rotorcraft, termed a Titan Aerial Daughtercraft (TAD), from a lander or balloon to perform high-resolution imaging and mapping, potentially land to acquire microscopic images or other in situ measurements, and acquire samples to return to analytical instruments on the mothership. In principle, the ability to recharge batteries in TAD from a radioisotope or other long-lived power source on the mothership could enable multiple sorties. For a lander-based mission, a variety of landing sites is conceivable, including near lake margins, in dry lake beds, or in regions of plains, dunes, or putative cryovolanic or impact melt features. Such missions may require landing with greater precision than in previous missions (Huygens) and mission studies; this could also enhance the ability of TAD to reach interesting terrain from the landing site. Precision descent may also benefit balloon missions, with or without a daughtercraft, by increasing the probability that the balloon will drift over desired terrain early in its mission. Given these potential benefits, the overall concept studied here includes brief consideration of precision descent for landing or balloon deployment, followed by one or more sorties by a rotorcraft deployed from the mothership, with the ability to return to the mothership.
4DoF Drift Free Navigation Using Inertial Cues and Optical Flow
In this paper, we describe a novel approach in fusing optical flow with inertial cues (3D acceleration and 3D angular velocities) in order to navigate a Micro Aerial Vehicle (MAV) drift free in 4DoF and metric velocity. Our approach only requires two consecutive images with a minimum of three feature matches. It does not require any (point) map nor any type of feature history. Thus it is an inherently failsafe approach that is immune to map and feature-track failures. With these minimal requirements we show in real experiments that the system is able to navigate drift free in all angles including yaw, in one metric position axis, and in 3D metric velocity. Furthermore, it is a power-on-and-go system able to online self-calibrate the inertial biases, the visual scale and the full 6DoF extrinsic transformation parameters between camera and IMU.
End-to-End Dexterous Manipulation with Deliberate Interactive Estimation
No abstract available
Fully Self-Contained Vision-Aided Navigation and Landing of a Micro Air Vehicle Independent from External Sensor Inputs
Direct-lift micro air vehicles have important applications in reconnaissance. In order to conduct persistent surveillance in urban environments, it is essential that these systems can perform autonomous landing maneuvers on elevated surfaces that provide high vantage points without the help of any external sensor and with a fully contained on-board software solution. In this paper, we present a micro air vehicle that uses vision feedback from a single down looking camera to navigate autonomously and detect an elevated landing platform as a surrogate for a roof top. Our method requires no special preparation (labels or markers) of the landing location. Rather, leveraging the planar character of urban structure, the landing platform detection system uses a planar homography decomposition to detect landing targets and produce approach waypoints for autonomous landing. The vehicle control algorithm uses a Kalman filter based approach for pose estimation to fuse visual SLAM (PTAM) position estimates with IMU data to correct for high latency SLAM inputs and to increase the position estimate update rate in order to improve control stability. Scale recovery is achieved using inputs from a sonar altimeter. In experimental runs, we demonstrate a real-time implementation running on-board a micro aerial vehicle that is fully self-contained and independent from any external sensor information. With this method, the vehicle is able to search autonomously for a landing location and perform precision landing maneuvers on the detected targets.
Stereo and IMU Assisted Visual Odometry on an OMAP3530 for Small Robots
No abstract available
Feature and Pose Constrained Visual Aided Inertial Navigation for Computationally Constrained Aerial Vehicles
A Feature and Pose Constrained Extended Kalman Filter (FPC-EKF) is developed for highly dynamic computationally constrained micro aerial vehicles. Vehicle localization is achieved using only a low performance inertial measurement unit and a single camera. The FPC-EKF framework augments the vehicle's state with both previous vehicle poses and critical environmental features, including vertical edges. This filter framework efficiently incorporates measurements from hundreds of opportunistic visual features to constrain the motion estimate, while allowing navigating and sustained tracking with respect to a few persistent features. In addition, vertical features in the environment are opportunistically used to provide global attitude references. Accurate pose estimation is demonstrated on a sequence including fast traversing, where visual features enter and exit the field-of-view quickly, as well as hover and ingress maneuvers where drift free navigation is achieved with respect to the environment.
Daytime Water Detection Based on Sky Reflections
A water body s surface can be modeled as a horizontal mirror. Water detection based on sky reflections and color variation are complementary. A reflection coefficient model suggests sky reflections dominate the color of water at ranges > 12 meters. Water detection based on sky reflections: (1) geometrically locates the pixel in the sky that is reflecting on a candidate water pixel on the ground (2) predicts if the ground pixel is water based on color similarity and local terrain features. Water detection has been integrated on XUVs.
Unmanned Ground Vehicle Perception Using Thermal Infrared Cameras
TIR cameras can be used for day/night Unmanned Ground Vehicle (UGV) autonomous navigation when stealth is required. The quality of uncooled TIR cameras has significantly improved over the last decade, making them a viable option at low speed Limiting factors for stereo ranging with uncooled LWIR cameras are image blur and low texture scenes TIR perception capabilities JPL has explored includes: (1) single and dual band TIR terrain classification (2) obstacle detection (pedestrian, vehicle, tree trunks, ditches, and water) (3) perception thru obscurants
Unmanned Ground Vehicle Perception Using Thermal Infrared Cameras
The ability to perform off-road autonomous navigation at any time of day or night is a requirement for some unmanned ground vehicle (UGV) programs. Because there are times when it is desirable for military UGVs to operate without emitting strong, detectable electromagnetic signals, a passive only terrain perception mode of operation is also often a requirement. Thermal infrared (TIR) cameras can be used to provide day and night passive terrain perception. TIR cameras have a detector sensitive to either mid-wave infrared (MWIR) radiation (3-5?m) or long-wave infrared (LWIR) radiation (8-12?m). With the recent emergence of high-quality uncooled LWIR cameras, TIR cameras have become viable passive perception options for some UGV programs. The Jet Propulsion Laboratory (JPL) has used a stereo pair of TIR cameras under several UGV programs to perform stereo ranging, terrain mapping, tree-trunk detection, pedestrian detection, negative obstacle detection, and water detection based on object reflections. In addition, we have evaluated stereo range data at a variety of UGV speeds, evaluated dual-band TIR classification of soil, vegetation, and rock terrain types, analyzed 24 hour water and 12 hour mud TIR imagery, and analyzed TIR imagery for hazard detection through smoke. Since TIR cameras do not currently provide the resolution available from megapixel color cameras, a UGV's daytime safe speed is often reduced when using TIR instead of color cameras. In this paper, we summarize the UGV terrain perception work JPL has performed with TIR cameras over the last decade and describe a calibration target developed by General Dynamics Robotic Systems (GDRS) for TIR cameras and other sensors.
Autonomous Landing and Ingress of Micro-Air-Vehicles in Urban Environments Based on Monocular Vision
Unmanned micro air vehicles (MAVs) will play an important role in future reconnaissance and search and rescue applications. In order to conduct persistent surveillance and to conserve energy, MAVs need the ability to land, and they need the ability to enter (ingress) buildings and other structures to conduct reconnaissance. To be safe and practical under a wide range of environmental conditions, landing and ingress maneuvers must be autonomous, using real-time, onboard sensor feedback. To address these key behaviors, we present a novel method for vision-based autonomous MAV landing and ingress using a single camera for two urban scenarios: landing on an elevated surface, representative of a rooftop, and ingress through a rectangular opening, representative of a door or window. Real-world scenarios will not include special navigation markers, so we rely on tracking arbitrary scene features; however, we do currently exploit planarity of the scene. Our vision system uses a planar homography decomposition to detect navigation targets and to produce approach waypoints as inputs to the vehicle control algorithm. Scene perception, planning, and control run onboard in real-time; at present we obtain aircraft position knowledge from an external motion capture system, but we expect to replace this in the near future with a fully self-contained, onboard, vision-aided state estimation algorithm. We demonstrate autonomous vision-based landing and ingress target detection with two different quadrotor MAV platforms. To our knowledge, this is the first demonstration of onboard, vision-based autonomous landing and ingress algorithms that do not use special purpose scene markers to identify the destination.
FPGA Implementation of Stereo Disparity with High Throughput for Mobility Applications
No abstract available
FPGA Implementation of Stereo Disparity with High Throughput for Mobility Applications
High speed stereo vision can allow unmanned robotic systems to navigate safely in unstructured terrain, but the computational cost can exceed the capacity of typical embedded CPUs. In this paper, we describe an end-to-end stereo computation co-processing system optimized for fast throughput that has been implemented on a single Virtex 4 LX160 FPGA. This system is capable of operating on images from a 1024 x 768 3CCD (true RGB) camera pair at 15 Hz. Data enters the FPGA directly from the cameras via Camera Link and is rectified, pre-filtered and converted into a disparity image all within the FPGA, incurring no CPU load. Once complete, a rectified image and the final disparity image are read out over the PCI bus, for a bandwidth cost of 68 MB/sec. Within the FPGA there are 4 distinct algorithms: Camera Link capture, Bilinear rectification, Bilateral subtraction pre-filtering and the Sum of Absolute Difference (SAD) disparity. Each module will be described in brief along with the data flow and control logic for the system. The system has been successfully fielded upon the Carnegie Mellon University's National Robotics Engineering Center (NREC) Crusher system during extensive field trials in 2007 and 2008 and is being implemented for other surface mobility systems at JPL.
Implementation of Pin Point Landing Vision Components in an FPGA System
Pin-point landing is required to enable missions to land close, typically within 10 meters, to scientifically important targets in generally hazardous terrain. In Pin Point Landing both high accuracy and high speed estimation of position and orientation is needed to provide input to the control system to safely choose and navigate to a safe landing site. A proposed algorithm called VISion aided Inertial NAVigation (VISINAV) has shown that the accuracy requirements can be met. [2][3] VISINAV was shown in software only, and was expected to use FPGA enhancements in the future to improve the computational speed needed for pin point landing during Entry Descent and Landing (EDL). Homography, feature detection and spatial correlation are computationally intensive parts of VISINAV. Homography aligns the map image with the descent image so that small correlation windows can be used, and feature detection provides regions that spatial correlation can track from frame to frame in order to estimate vehicle motion. On MER the image Homography, Feature Detection and Correlation would take approximately 650ms tracking 75 features between frames. We implemented Homography, Feature detection and Correlation on a Virtex 4 LX160 FPGA to run in under 25ms while tracking 500 features to improve algorithm reliability and throughput.
Head Pose Estimation Using Multilinear Subspace Analysis for Robot Human Awareness
Mobile robots, operating in unconstrained indoor and outdoor environments, would benefit in many ways from perception of the human awareness around them. Knowledge of people's head pose and gaze directions would enable the robot to deduce which people are aware of the its presence, and to predict future motions of the people for better path planning. To make such inferences, requires estimating head pose on facial images that are combination of multiple varying factors, such as identity, appearance, head pose, and illumination. By applying multilinear algebra, the algebra of higher-order tensors, we can separate these factors and estimate head pose regardless of subject's identity or image conditions. Furthermore, we can automatically handle uncertainty in the size of the face and its location. We demonstrate a pipeline of on-the-move detection of pedestrians with a robot stereo vision system, segmentation of the head, and head pose estimation in cluttered urban street scenes.
Robust Landing Using Time-to-Collision Measurement with Actuator Saturation
This paper considers a landing problem for an MAV that uses only a monocular camera for guidance. Although this sensor cannot measure the absolute distance to the target, by using optical flow algorithms, time-to-collision to the target is obtained. Existing work has applied a simple proportional feedback control to simple dynamics and demonstrated its potential. However, due to the singularity in the time-to-collision measurement around the target, this feedback could require an infinite control action. This paper extends the approach into nonlinear dynamics. In particular, we explicitly consider the saturation of the actuator and include the effect of the aerial drag. It is shown that the convergence to the target is guaranteed from a set of initial conditions, and the boundaries of such initial conditions in the state space are numerically obtained. The paper then introduces parametric uncertainties in the vehicle model and in the time-to-collision measurements. Using an argument similar to the nominal case, the robust convergence to the target is proven, but the region of attraction is shown to shrink due to the existence of uncertainties. The numerical simulation validates these theoretical results.
Vision System Measures Motions of Robot and External Objects
A prototype of an advanced robotic vision system both (1) measures its own motion with respect to a stationary background and (2) detects other moving objects and estimates their motions, all by use of visual cues. Like some prior robotic and other optoelectronic vision systems, this system is based partly on concepts of optical flow and visual odometry. Whereas prior optoelectronic visual-odometry systems have been limited to frame rates of no more than 1 Hz, a visual-odometry subsystem that is part of this system operates at a frame rate of 60 to 200 Hz, given optical-flow estimates. The overall system operates at an effective frame rate of 12 Hz. Moreover, unlike prior machine-vision systems for detecting motions of external objects, this system need not remain stationary: it can detect such motions while it is moving (even vibrating). The system includes a stereoscopic pair of cameras mounted on a moving robot. The outputs of the cameras are digitized, then processed to extract positions and velocities. The initial image-data-processing functions of this system are the same as those of some prior systems: Stereoscopy is used to compute three-dimensional (3D) positions for all pixels in the camera images. For each pixel of each image, optical flow between successive image frames is used to compute the two-dimensional (2D) apparent relative translational motion of the point transverse to the line of sight of the camera. The challenge in designing this system was to provide for utilization of the 3D information from stereoscopy in conjunction with the 2D information from optical flow to distinguish between motion of the camera pair and motions of external objects, compute the motion of the camera pair in all six degrees of translational and rotational freedom, and robustly estimate the motions of external objects, all in real time. To meet this challenge, the system is designed to perform the following image-data-processing functions: The visual-odometry subsystem (the subsystem that estimates the motion of the camera pair relative to the stationary background) utilizes the 3D information from stereoscopy and the 2D information from optical flow. It computes the relationship between the 3D and 2D motions and uses a least-mean-squares technique to estimate motion parameters. The least-mean-squares technique is suitable for real-time implementation when the number of external-moving-object pixels is smaller than the number of stationary-background pixels.