Engineering topics

Mulligan, Jeffrey B.

Publications and source records attributed to Mulligan, Jeffrey B..

At least 37 records · Page 2

NASA NTRS2016

Measuring and Modeling Shared Visual Attention

Multi-person teams are sometimes responsible for critical tasks, such as flying an airliner. Here we present a method using gaze tracking data to assess shared visual attention, a term we use to describe the situation where team members are attending to a common set of elements in the environment. Gaze data are quantized with respect to a set of N areas of interest (AOIs); these are then used to construct a time series of N dimensional vectors, with each vector component representing one of the AOIs, all set to 0 except for the component corresponding to the currently fixated AOI, which is set to 1. The resulting sequence of vectors can be averaged in time, with the result that each vector component represents the proportion of time that the corresponding AOI was fixated within the given time interval. We present two methods for comparing sequences of this sort, one based on computing the time-varying correlation of the averaged vectors, and another based on a chi-square test testing the hypothesis that the observed gaze proportions are drawn from identical probability distributions. We have evaluated the method using synthetic data sets, in which the behavior was modeled as a series of "activities," each of which was modeled as a first-order Markov process. By tabulating distributions for pairs of identical and disparate activities, we are able to perform a receiver operating characteristic (ROC) analysis, allowing us to choose appropriate criteria and estimate error rates. We have applied the methods to data from airline crews, collected in a high-fidelity flight simulator (Haslbeck, Gontar & Schubert, 2014). We conclude by considering the problem of automatic (blind) discovery of activities, using methods developed for text analysis.

attention↗

NASA NTRS2016

A Method for Rapid Measurement of Contrast Sensitivity on Mobile Touch-Screens

Touch-screen displays in cell phones and tablet computers are now pervasive, making them an attractive option for vision testing outside of the laboratory or clinic. Here we de- scribe a novel method in which subjects use a finger swipe to indicate the transition from visible to invisible on a grating which is swept in both contrast and frequency. Because a single image can be swiped in about a second, it is practical to use a series of images to zoom in on particular ranges of contrast or frequency, both to increase the accuracy of the measurements and to obtain an estimate of the reliability of the subject. Sensitivities to chromatic and spatio-temporal modulations are easily measured using the same method. A proto- type has been developed for Apple Computer's iPad/iPod/iPhone family of devices, implemented using an open-source scripting environment known as QuIP (QUick Image Processing, http://hsi.arc.nasa.gov/groups/scanpath/research.php). Preliminary data show good agreement with estimates obtained from traditional psychophysical methods as well as newer rapid estimation techniques. Issues relating to device calibration are also discussed.

contrast sensitivity↗

NASA NTRS2015

Measuring and Modeling Shared Visual Attention

Multi-person teams are sometimes responsible for critical tasks, such as flying an airliner. Here we present a method using gaze tracking data to assess shared visual attention, a term we use to describe the situation where team members are attending to a common set of elements in the environment. Gaze data are quantized with respect to a set of N areas of interest (AOIs); these are then used to construct a time series of N dimensional vectors, with each vector component representing one of the AOIs, all set to 0 except for the component corresponding to the currently fixated AOI, which is set to 1. The resulting sequence of vectors can be averaged in time, with the result that each vector component represents the proportion of time that the corresponding AOI was fixated within the given time interval.We present two methods for comparing sequences of this sort, one based on computing the time varying correlation of the averaged vectors, and another based on a chi-square test testing the hypothesis that the observed gaze proportions are drawn from identical probability distributions.We have evaluated the method using synthetic data sets, in which the behavior was modeled as a series of activities, each of which was modeled as a first-order Markov process. By tabulating distributions for pairs of identical and disparate activities, we are able to perform a receiver operating characteristic (ROC) analysis, allowing us to choose appropriate criteria and estimate error rates. Using these criteria, we have applied the methods to data from airline crews, collected in a high-fidelity flight simulator (Gontar Hoermann, 2014). We conclude by considering the problem of automatic (blind) discovery of activities, using methods developed for text analysis.

Mulligan, Jeffrey B.↗

NASA NTRS2015

Measurement of Visual Reaction Times Using Hand-held Mobile Devices

Modern mobile devices provide a convenient platform for collecting research data in the field. But,because the working of these devices is often cloaked behind multiple layers of proprietary system software, it can bedifficult to assess the accuracy of the data they produce, particularly in the case of timing. We have been collecting datain a simple visual reaction time experiment, as part of a fatigue testing protocol known as the Psychomotor Vigilance Test (PVT). In this protocol, subjects run a 5-minute block consisting of a sequence of trials in which a visual stimulus appears after an unpredictable variable delay. The subject is required to tap the screen as soon as possible after the appearance of the stimulus. In order to validate the reaction times reported by our program, we had subjects perform the task while a high-speed video camera recorded both the display screen, and a side view of the finger (observed in a mirror). Simple image-processing methods were applied to determine the frames in which the stimulus appeared and disappeared, and in which the finger made and broke contact with the screen. The results demonstrate a systematic delay between the initial contact by the finger and the detection of the touch by the software, having a value of 80 +- 20 milliseconds.

reaction time↗

NASA NTRS2015

A Metric to Quantify Shared Visual Attention in Two-Person Teams

Introduction: Critical tasks in high-risk environments are often performed by teams, the members of which must work together efficiently. In some situations, the team members may have to work together to solve a particular problem, while in others it may be better for them to divide the work into separate tasks that can be completed in parallel. We hypothesize that these two team strategies can be differentiated on the basis of shared visual attention, measured by gaze tracking. 2) Methods: Gaze recordings were obtained for two-person flight crews flying a high-fidelity simulator (Gontar, Hoermann, 2014). Gaze was categorized with respect to 12 areas of interest (AOIs). We used these data to construct time series of 12 dimensional vectors, with each vector component representing one of the AOIs. At each time step, each vector component was set to 0, except for the one corresponding to the currently fixated AOI, which was set to 1. This time series could then be averaged in time, with the averaging window time (t) as a variable parameter. For example, when we average with a t of one minute, each vector component represents the proportion of time that the corresponding AOI was fixated within the corresponding one minute interval. We then computed the Pearson product-moment correlation coefficient between the gaze proportion vectors for each of the two crew members, at each point in time, resulting in a signal representing the time-varying correlation between gaze behaviors. We determined criteria for concluding correlated gaze behavior using two methods: first, a permutation test was applied to the subjects' data. When one crew member's gaze proportion vector is correlated with a random time sample from the other crewmember's data, a distribution of correlation values is obtained that differs markedly from the distribution obtained from temporally aligned samples. In addition to validating that the gaze tracker was functioning reasonably well, this also allows us to compute probabilities of coordinated behavior for each value of the correlation. As an alternative, we also tabulated distributions of correlation coefficients for synthetic data sets, in which the behavior was modeled as a first-order Markov process, and compared correlation distributions for identical processes with those for disparate processes, allowing us to choose criteria and estimate error rates. 3) Discussion: Our method of gaze correlation is able to measure shared visual attention, and can distinguish between activities involving different instruments. We plan to analyze whether pilots strategies of sharing visual attention can predict performance. Possible measurements of performance include expert ratings from instructors, fuel consumption, total task time, and failure rate. While developed for two-person crews, our approach can be applied to larger groups, using intra-class correlation coefficients instead of the Pearson product-moment correlation.

Gontar, Patrick↗

NASA NTRS2015

A Metric to Quantify Shared Visual Attention in Two-Person Teams

Critical tasks in high-risk environments are often performed by teams, the members of which must work together efficiently. In some situations, the team members may have to work together to solve a particular problem, while in others it may be better for them to divide the work into separate tasks that can be completed in parallel. We hypothesize that these two team strategies can be differentiated on the basis of shared visual attention, measured by gaze tracking.

Gontar, Patrick↗

NASA NTRS2015

Psychophysical Calibration of Mobile Touch-Screens for Vision Testing in the Field

The now ubiquitous nature of touch-screen displays in cell phones and tablet computers makes them an attractive option for vision testing outside of the laboratory or clinic. Accurate measurement of parameters such as contrast sensitivity, however, requires precise control of absolute and relative screen luminances. The nonlinearity of the display response (gamma) can be measured or checked using a minimum motion technique similar to that developed by Anstis and Cavanagh (1983) for the determination of isoluminance. While the relative luminances of the color primaries vary between subjects (due to factors such as individual differences in pre-retinal pigment densities), the gamma nonlinearity can be checked in the lab using a photometer. Here we compare results obtained using the psychophysical method with physical measurements for a number of different devices. In addition, we present a novel physical method using the device's built-in front-facing camera in conjunction with a mirror to jointly calibrate the camera and display. A high degree of consistency between devices is found, but some departures from ideal performance are observed. In spite of this, the effects of calibration errors and display artifacts on estimates of contrast sensitivity are found to be small.

gamma correction↗

NASA NTRS2014

Smooth Pursuit of Flicker-Defined Motion

We examined the pursuit response to stimuli defined by space-variant flicker of a dense random dot carrier pattern. On each frame, every element of the pattern could change polarity, with a probability given by a two-dimensional Gaussian distribution. A normal distribution produces a circular region of twinkle, while inverting the distribution results in a spot of static texture in a twinkling surround. In this latter case, the carrier texture could be stationary, or could move with the twinkle modulator,thereby producing first-order motion in the region of the spot. While the twinkle-defined spot produces a strong sensation of motion, the complementary stimulus defined by the absence of twinkle does not;when viewed peripherally, it appears to move in steps even when the generating distribution moves smoothly. We examined pursuit responses to these stimuli using two techniques: 1) the eye movement correlogram, obtained by cross-correlating eye velocity with the velocity of a randomly-moving stimulus; and 2) delayed visual feedback, where transient stabilization of a target can produce spontaneous oscillations of the eye, with a period empirically observed to vary linearly with the applied delay. Both techniques provide an estimate of the internal processing time, which can be as short as100 milliseconds for a first-order target. Assessed by the correlogram method, the response to flicker defined motion is delayed by more than 100 milliseconds, and significantly weaker (especially in the vertical dimension). When initially presented in the delayed feedback condition, purely saccadic oscillation is observed. One subject eventually developed smooth oscillations (albeit with significant saccadic intrusions), showing a period-versus-delay slope similar to that observed for first-order targets. This result is somewhat surprising, given that we interpret the slope of the period-versus-delayfunction as reflecting the balance between position- and velocity-sensitive inputs to pursuit.

smooth pursuit↗

NASA NTRS2014

Smooth Pursuit of Flicker-Defined Motion

We examined the pursuit response to stimuli defined by space-variant flicker of a dense random dot carrier pattern. On each frame, every element of the pattern could change polarity, with a probability given by a two-dimensional Gaussian distribution. A normal distribution produces a circular region of twinkle, while inverting the distribution results in a spot of static texture in a twinkling surround. In this latter case, the carrier texture could be stationary, or could move with the twinkle modulator, thereby producing first-order motion in the region of the spot. While the twinkle-defined spot produces a strong sensation of motion, the complementary stimulus defined by the absence of twinkle does not, when viewed peripherally, it appears to move in steps even when the generating distribution moves smoothly. We examined pursuit responses to these stimuli using two techniques: 1) the eye movement correlogram, obtained by cross-correlating eye velocity with the velocity of a randomly-moving stimulus; and 2) delayed visual feedback, where transient stabilization of a target can produce spontaneous oscillations of the eye, with a period empirically observed to vary linearly with the applied delay. Both techniques provide an estimate of the internal processing time, which can be as short as 100 milliseconds for a first-order target. Assessed by the correlogram method, the response to flicker-defined motion is delayed by more than 100 milliseconds, and significantly weaker (especially in the vertical dimension). When initially presented in the delayed feedback condition, purely saccadic oscillation is observed. One subject eventually developed smooth oscillations (albeit with significant saccadic intrusions), showing a period-versus-delay slope similar to that observed for first-order targets. This result is somewhat surprising, given that we interpret the slope of the period-versus-delay-function as reflecting the balance between position- and velocity-sensitive inputs to pursuit.

eye movements↗

NASA NTRS2013

Rapid Assessment of Contrast Sensitivity with Mobile Touch-screens

The availability of low-cost high-quality touch-screen displays in modern mobile devices has created opportunities for new approaches to routine visual measurements. Here we describe a novel method in which subjects use a finger swipe to indicate the transition from visible to invisible on a grating which is swept in both contrast and frequency. Because a single image can be swiped in about a second, it is practical to use a series of images to zoom in on particular ranges of contrast or frequency, both to increase the accuracy of the measurements and to obtain an estimate of the reliability of the subject. Sensitivities to chromatic and spatio-temporal modulations are easily measured using the same method. We will demonstrate a prototype for Apple Computer's iPad-iPod-iPhone family of devices, implemented using an open-source scripting environment known as QuIP (QUick Image Processing,

mobile devices↗

NASA NTRS2012

Assessing Visual Delays using Pupil Oscillations

Stark (1962) demonstrated vigorous pupil oscillations by illuminating the retina with a beam of light focussed to a small spot near the edge of the pupil. Small constrictions of the pupil then are sufficient to completely block the beam, amplifying the normal relationship between pupil area and retinal illuminance. In addition to this simple and elegant method, Stark also investigated more complex feedback systems using an electronic "clamping box" which provided arbitrary gain and phase delay between a measurement of pupil area and an electronically controlled light source. We have replicated Stark's results using a video-based pupillometer to control the luminance of a display monitor. Pupil oscillations were induced by imposing a linear relationship between pupil area and display luminance, with a variable delay. Slopes of the period-vs-delay function for 3 subjects are close to the predicted value of 2 (1.96-2.39), and the implied delays range from 254 to 376 508 to 652 milliseconds. Our setup allows us to extend Stark's work by investigating a broader class of stimuli.

Mulligan, Jeffrey B.↗

NASA NTRS2010

Factors Influencing Scanning for Alerts

Aircraft pilots (like operators in many domains) are required to monitor locations for rare events, while concurrently performing their everyday tasks. In many cases, the visual parameters of the alert are such that it is not visible unless directly fixated. For this reason, critical alerts should be designed to be visible in peripheral vision and/or augmented by an audio alarm. We use the term "conspicuity" to distinguish the attention-getting power of a visual stimulus from simple visibility in a single-task context. We have measured conspicuity in an experimental paradigm designed to test the N-SEEV model of attention and noticing (Steelman-Allen et al., HFES 2009). The subject performed a demanding central task while monitoring four peripheral locations for color change events. Visibility of the alerting stimuli was measured separately in a control experiment in which the subject maintained steady fixation without the central task. Thresholds in the dual-task experiments were lower than would be expected based on the results of the control experiment, due to the fact that the subjects actively sampled the alert locations with fixations while performing the central task. Locations of high-frequency alerts are generally sampled more often than locations of low-frequency alerts, and alert location sampling in general increases with practice, presumably because the demands of the central task are reduced.

Mulligan, Jeffrey B.↗

NASA NTRS1998

Pursuit Latency for Chromatic Targets

The temporal dynamics of eye movement response to a change in direction of stimulus motion has been used to compare the processing speeds of different types of stimuli (Mulligan, ARVO '97). In this study, the pursuit response to colored targets was measured to test the hypothesis that the slow response of the chromatic system (as measured using traditional temporal sensitivity measures such as contrast sensitivity) results in increased eye movement latencies. Subjects viewed a small (0.4 deg) Gaussian spot which moved downward at a speed of 6.6 deg/sec. At a variable time during the trajectory, the dot's direction of motion changed by 30 degrees, either to the right or left. Subjects were instructed to pursue the spot. Eye movements were measured using a video ophthalmoscope with an angular resolution of approximately 1 arc min and a temporal sampling rate of 60 Hz. Stimuli were modulated in chrominance for a variety of hue directions, combined with a range of small luminance increments and decrements, to insure that some of the stimuli fell in the subjects' equiluminance planes. The smooth portions of the resulting eye movement traces were fit by convolving the stimulus velocity with an exponential having variable onset latency, time constant and amplitude. Smooth eye movements with few saccades were observed for all stimuli. Pursuit responses to stimuli having a significant luminance component are well-fit by exponentials having latencies and time constants on the order of 100 msec. Increases in pursuit response latency on the order of 100-200 msec are observed in response to certain stimuli, which occur in pairs of complementary hues, corresponding to the intersection of the stimulus section with the subjects' equiluminant plane. Smooth eye movements can be made in response to purely chromatic stimuli, but are slower than responses to stimuli with a luminance component.

Mulligan, Jeffrey B.↗

NASA NTRS1997

Recovery of motion parameters from distortions in scanned images

Scanned images, such as those produced by the scanning-laser ophthalmoscope (SLO), show distortions when there is target motion. This is because pixels corresponding to different image regions are acquired sequentially, and so, in essence, are slices of different snapshots. While these distortions create problems for image registration algorithms, they are potentially useful for recovering target motion parameters at temporal frequencies above the frame rate. Stetter, Sendtner and Timberlake measured large distortions in SLO images to recover the time course of rapid horizontal saccadic eye movements. Here, this work is extended with the goal of automatically recovering small eye movements in two dimensions. Eye position during the frame interval is modeled using a low dimensional parametric description, which in turn is used to generate predicted distortions of a reference template. The input image is then registered to the distorted template using normalized cross correlation. The motion parameters are then varied, and the correlation recomputed, to find the motion which maximizes the peak value of the correlation. The location and value of the correlation maximum are determined with sub-pixel precision using biquadratic interpolation, yielding eye position resolution better than 1 arc minute. This method of motion parameter estimation is tested using actual SLO images as well as simulated images. Motion parameter estimation might also be applied to individual video lines in order to reduce pipeline delays for a near real-time system.

Mulligan, Jeffrey B.↗

NASA NTRS1997

Eye Movement Latencies to Direction Change for Different Classes of Motion

In the analysis of visual motion, local features such as orientation are analyzed early in the cortical processing stream (V1), while integration across orientation and space is thought to occur in higher cortical areas such as MT, MST, etc. If all areas provide inputs to eye movement control centers, we would expect that local properties would drive eye movements with relatively short latencies, while global properties would require longer latencies. When such latencies are observed, they can provide information about when (and where?) various stimulus properties are analyzed. To this end, a stimulus was employed in which local and global properties determining perceived direction-of-motion could be manipulated independently: an elliptical Gabor patch with a drifting carrier, with variable orientation of the carrier grating and the contrast window. We have previously demonstrated that the directional percepts evoked by this stimulus vary between the "grating direction" (the normal to the grating's orientation) and the "window direction" (ARVO 91, 94), and that similar effects can be observed in reflexive eye movements (ARVO 95). Subjects viewed such a stimulus while attempting to maintain steady fixation on the center of the pattern, and the small reflexive eye movements ("stare OKN") were recorded. In the middle of the trial, the orientation of either the grating or the window was rotated smoothly by 30 degrees. Responses to the shift of both grating orientation and window orientation are seen in the average OKN slow phase velocity. Grating rotations produce a rapid OKN rotation to the grating direction (100 ms latency, 300 ms time constant), followed by a slower rebound to the steady state perceived direction midway between the grating and window directions. Window rotations, on the other hand, evoke a slower response (200 ms latency, 500 ms time constant). The results demonstrate multiple cortical inputs to eye movement control: a taste early input driven by orientation, and a slower input from higher areas sensitive to global stimulus properties.

Mulligan, Jeffrey B.↗

NASA NTRS1997

Image Rotation Does Not Rotate Smooth Eye Movements

Subjects viewing a drifting noise pattern make reflexive smooth eye movements in the direction of motion, which follow rapid changes in movement direction. These responses are unaffected by rotations of the pattern, suggesting that there is no coupling between visually sensed rotation and the direction of ocular following.

Mulligan, Jeffrey B.↗

NASA NTRS1996

Eye Movements Reveal Hierarchical Motion Processing

Purpose: In the analysis of visual motion, local features such as orientation are analyzed early in the cortical processing stream (V1), while integration across orientation and space is thought to occur in higher cortical areas such as MT, MST, etc. If all areas provide inputs to eye movement control centers, we would expect that local properties would drive eye movements with relatively short latencies, while global properties would require longer latencies. When such latencies are observed, they can provide information about when (and where?) various stimulus properties are analyzed. Methods: The stimulus employed was an elliptical Gabor patch with a drifting carrier, in which the orientations of the carrier grating and the contrast window were varied independently. We have previously demonstrated that the directional percepts evoked by this stimulus vary between the "grating direction" (the normal to the grating's orientation) and the "window direction", and that similar effects can be observed in reflexive eye movements. Subjects viewed such a stimulus while attempting to maintain steady fixation on the center of the pattern, and the small reflexive eye movements ("stare OKN") were recorded. In the middle of the trial, the orientation of either the grating or the window was rotated smoothly by 30 degrees. Results: Responses to the shift of both grating orientation and window orientation are seen in the average OKN slow phase velocity. Grating rotations produce a rapid OKN rotation to the grating direction (100 ms latency, 300 ms time constant), followed by a slower rebound to the steady state perceived direction midway between the grating and window directions. Window rotations, on the other hand, evoke a slower response (200 ms latency, 500 ms time constant). Conclusions: The results demonstrate multiple cortical inputs to eye movement control: a fast, early input driven by orientation, and a slower input from higher areas sensitive to global stimulus properties.

Mulligan, Jeffrey B.↗

NASA NTRS1996

High-Procession Eye Tracking Using Fundus Images

Fundus images provide high optical gain for eye movement tracking, i.e. large image displacements occur as a result of small eye rotations. Subpixel registration techniques can provide resolution better than 1 arc minute using images acquired with a CCD camera. Ocular torsion may also be estimated, with a precision of approximately 0.1 degree. This talk will discuss the software algorithms used to attain this performance.

Mulligan, Jeffrey B.↗