Engineering topics
Publications and source records attributed to Mulligan, Jeffrey B..
Three psychophysical methods for measurement of visual contrast sensitivity on a tablet computer were evaluated and compared. A novel rapid method involving "swiping" a test image produced results comparable to more traditional methods in a fraction of the time.
Some astronauts have suffered degradation of vision during long-duration space flight, suffering from a condition that has come to be known as Spaceflight Associated Neuro-ocular Syndrome (SANS). While related morphological changes can be observed with imaging technologies such as optical coherence tomography (OCT), it may be useful to have a rapid method for functional vision assessment. In this paper, we compare three tablet-based methods for rapid assessment of contrast sensitivity. First, a relatively novel method developed expressly for touch screens, in which the subject ”swipes” a frequency/contrast sweep grating to indicate the boundary between visible and invisible patterns; second, a method-of-adjustment task in which the subject adjusts the contrast of a grating patch up and down to bracket the visual threshold; and third, a traditional temporal two-alternative forced choice (2AFC) task, in which the subject is presented with a near-threshold stimulus in one of two intervals, and must report the interval containing the stimulus. The swipe method shows variability comparable to the 2AFC method, and shows good agreement in estimates of the spatial frequency of peak sensitivity. The absolute sensitivity estimated with the swipe method is higher than that of the other methods, perhaps because subjects are biased to trace outside of the visible pattern region, or perhaps due to stimulus differences.
During terrestrial activities, sensation of pressure on the skin and tension in muscles and joints provides information about how the body is oriented relative to gravity and how the body is moving relative to the surrounding environment. In contrast, in aquatic environments when suspended in a state of neutral buoyancy, the weight of the body and limbs is offloaded, rendering these cues uninformative. It is not yet known how this altered sensory environment impacts virtual reality experiences. To investigate this question, we converted a full-face SCUBA mask into an underwater head-mounted display and developed software to simulate jetpack locomotion outside the International Space Station. Our goal was to emulate conditions experienced by astronauts during training at NASA's Neutral Buoyancy Lab. A user study was conducted to evaluate both sickness and presence when using virtual reality in this altered sensory environment. We observed an increase in nausea related symptoms underwater, but we cannot conclude that this is due to VR use. Other measures of sickness and presence underwater were comparable to measures taken above water. We conclude with suggestions for improved underwater VR systems and improved methods for evaluation of these systems based on our experience.
Short-haul flights are associated with irregular work schedules and increased workload, due to frequent takeoffs and landings. We examined the relationship between pilot workload, performance and subjective fatigue during normal short-haul operations. Ninety airline pilots (8 female), mean age 33 (±8 years) completed a NASA-Task Load Index (NASA-TLX), a Psychomotor Vigilance Task (PVT; a reaction time test sensitive to sleep loss) and a Samn-Perelli (SP) fatigue scale, over a period of 20 duty days at top-of-descent on 2762 short-haul flights. The duty days included either 2 or 4 flights per day starting at different times as scheduled during normal operations. Workload was measured using the six NASA-TLX scales: mental demand, physical demand, temporal demand, effort, performance and frustration. Lapses (reaction times [RT] > 500ms) were calculated for the PVT. Spearman correlations were calculated to identify relationships between the NASA-TLX, PVT lapses, and SP. The six scales of NASA-TLX were positively correlated with the PVT lapses (p < 0.01) showing an increase in workload when lapses increased. There was a positive correlation between subjective fatigue as measured by the SP fatigue scale and each of the six scales of NASA-TLX (p < 0.001) suggesting that pilots reported higher workload when perceived levels of fatigue were higher. Of the six workload scales, mental demand and performance were rated the highest (mental: M = 40.99, SD = 20.32; performance: M = 41.61, SD = 20.71) and effort was rated the lowest (M = 15.59, SD = 8.98). Preliminary analyses suggest that high workload is associated with poorer PVT performance and increased self-reported fatigue in this population of short-haul pilots. Future studies should explore how other workload factors (i.e. flight hours, time of day) influence self-reported and objective fatigue measures.
Human behavior often consists of a series of distinct activities, each characterized by a unique signature of visual behavior. This is true even in a restricted domain, such as piloting an aircraft, where patterns of visual signatures might represent activities like communicating, navigating, and monitoring. We propose a novel analysis method for gaze-tracking data, to perform blind discovery of these activities based on their behavioral signatures. The method is in some respects similar to recurrence analysis, but here we compare not individual fixations, but groups of fixations aggregated over a fixed time interval. The duration of this interval is a parameter that we will refer to as τ. We assume that the environment has been divided into a set of N different areas-of-interest (AOIs). For a given interval of time of duration τ, we compute the proportion of time spent fixating each AOI, resulting in an N-dimensional vector. These proportions can be converted to counts by multiplying by τ divided by the average fixation duration (another parameter that we fix at 280 milliseconds). We compare different intervals by computing the chi-square statistic. The p-value associated with the statistic is the likelihood of observing the data under the hypothesis that the data in the two intervals were generated by a single process with a single set of probabilities governing the fixation of each AOI. We have investigated the method using a set of 10 synthetic "activities," that sample 4 AOIs. Four of these activities visit 3 of the 4 AOIs, with equal probability; as there are four different ways to leave-one- out, there are four such activities. Similarly, there are six different activities that leave-two-out. Sequences of simulated behavior were generated by running each activity for 40 seconds, in sequence, for a total of 6.7 minutes. The figure to the right shows the matrix of chi-square statistics, using a value of 2.8 seconds for τ, corresponding to 10 fixations. Low values (dark) indicate poor evidence for activity differences, while high values (bright) indicate strong evidence. The dark squares along the main diagonal each correspond to the forty second intervals in which the activity was held constant; the 4x4 block at the lower left corresponds to the four leave-one-out activities, while the 6x6 block in the upper right corresponds to the leave-two-out activities. (The anti-diagonal pattern of white squares indicates those activity pairs that share no AOIs.) The chi-square values can be binarized by choosing a particular significance level; we are interested in grouping bins that represent the same activity, effectively accepting the null hypothesis. Therefore, we may adopt a relatively lax criterion; for example, choosing a p-value of 0.2 means that two behaviors that have only a 1-in-5 chance of being produced by a single activity might nevertheless be clustered together. We have explored several methods to perform clustering on the data and solving for the activity probabilities. Greedy methods begin by selecting the time bin that is similar to the most (or least) other bins, and then forming a cluster from it and all other non-discriminable bins. These methods show mediocre performance, as they do not take into account temporal contiguity. Preliminary results indicate that methods that "grow" clusters in time from seed points perform better.
Human behavior often consists of a series of distinct activities, each characterized by a unique pattern of interaction with the visual environment. This is true even in a restricted domain, such as a pilot flying an airplane; in this case, activities with distinct visual signatures might be things like communicating, navigating, monitoring, etc. We propose a novel analysis method for gaze-tracking data, to perform blind discovery of these hypothetical activities. We compare, not individual fixations, but groups of fixations aggregated over a fixed time interval (Tau). We assume that the environment has been divided into a finite set of discrete areas-of-interest (AOIs). For a given time interval, we compute the proportion of time spent fixating each AOI, resulting in an N-dimensional vector, where N is the number of AOIs. These proportions can be converted to integer counts by multiplying by Tau divided by the average fixation duration, a parameter that we fix at 283 milliseconds. We compare different intervals by computing the chi-squared statistic. The p-value associated with the statistic is the likelihood of observing the data under the hypothesis that the data in the two intervals were generated by a single process with a single set of probabilities governing the fixation of each AOI. We cluster the intervals, first by merging adjacent intervals that are sufficiently similar, optionally shifting the boundary between non-merged intervals to maximize the difference. Then we compare and cluster non-adjacent intervals. The method is evaluated using synthetic data generated by a hand-crafted set of activities. While the method generally finds more activities than put into the simulation, we have obtained agreement as high as 80 percent between the inferred activity labels and ground truth.
As early as 1980, Donald MacLeod coined the term "optoretinogram" to describe a change in the near-infrared reflectance of the retina following visual stimulation. The hope that such signals might exist was stimulated by an observation of a change in the infrared transmission of isolated toad retinae. This signal showed a time course of several seconds, much slower than the electroretinogram, and was thought to be related to some chemical step in the phototransduction cascade. During the early 1980's, we made several attempts to observe such signals in vivo in humans, without success. In the intervening decades, however, researchers have succeeded in observing what are now known as "intrinsic optical signals" or IOSs. Responses from single photoreceptors (obtained using adaptive optics) are surprisingly heterogenous, with some cells increasing their reflectance in response to visual stimulation, while others decrease. This talk will review our early work, survey recent findings, and consider various theories for the origin of the effect.
Human behavior often consists of a series of distinct activities, each characterized by a unique pattern of interaction with the visual environment. This is true even in a restricted domain, such as a piloting an aircraft, where activities with distinct visual signatures might be things like communicating, navigating, and monitoring. We propose a novel analysis method for gaze-tracking data, to perform blind discovery of these hypothetical activities. The method is in some respects similar to recurrence analysis, but here we compare not individual fixations, but groups of fixations aggregated over a fixed time interval. The duration of this interval is a parameter that we will refer to as delta. We assume that the environment has been divided into a set of N different areas-of-interest (AOIs). For a given interval of time of duration delta, we compute the proportion of time spent fixating each AOI, resulting in an N-dimensional vector. These proportions can be converted to integer counts by multiplying by delta divided by the average fixation duration (another parameter that we fix at 280 milliseconds). We compare different intervals by computing the chi-square statistic. The p-value associated with the statistic is the likelihood of observing the data under the hypothesis that the data in the two intervals were generated by a single process with a single set of probabilities governing the fixation of each AOI. The method has been applied to approximately 100 hours of eye movement data collected from pilots in a high-fidelity B747 flight simulator, and the results have been compared to synthetic data in which the each activity is represented as first-order Markov process with random probabilities assigned to the AOIs. Randomly-generated synthetic activities can require thousands of fixations to be discriminated with statistical significance, while the human data can be clustered using averaging windows of some 10's of seconds, suggesting that the actual activities are much more narrowly focused than random Markov models.
Saccadic and pupillary responses are reliable indices of arousal decrement (e.g. fatigue), that might be exploited to improve work schedule guidelines. In this study, we tested the sensitivity of a short 30-s oculomotor test to detect diurnal arousal variations. Twelve participants (5 females, 7 males, 37.7+-11.9 years) volunteered to be assessed every hour (66+-20 min) for three consecutive working days, during their regular office-hours. We used a fully automated testing system, the FIT 2000 Fitness Impairment Tester (Pulse Medical Instruments Inc., Rockville, MD, USA), to measure and record saccadic peak velocity, pupil diameter, and latency and amplitude of the pupillary light reflex. In addition, we collected subjective levels of arousal using the Stanford Sleepiness Scale, and body core temperature. We analyzed the data using a linear mixed model approach for longitudinal data. Both saccadic velocity and subjective alertness decreased over the course of a day, while body core temperature increased (all p-values.05). The data also weakly suggested an increase of the pupil diameter (p 07). The findings support the use of oculomotor indices in the assessment of arousal and fatigue in applied settings.
A color algebra refers to a system for computing sums and products of colors, analogous to additive and subtractive color mixtures. The difficulty addressed here is the fact that, because of metamerism, we cannot know with certainty the spectrum that produced a particular color solely on the basis of sensory data. Knowledge of the spectrum is not required to compute additive mixture of colors, but is critical for subtractive (multiplicative) mixture. Therefore, we cannot predict with certainty the multiplicative interactions between colors based solely on sensory data. There are two potential applications of a color algebra: first, to aid modeling phenomena of human visual perception, such as color constancy and transparency; and, second, to provide better models of the interactions of lights and surfaces for computer graphics rendering.
A color algebra refers to a system for computing sums and products of colors, analogous to additive and subtractive color mixtures. We would like it to match the well-defined algebra of spectral functions describing lights and surface reflectances, but an exact correspondence is impossible after the spectra have been projected to a three-dimensional color space, because of metamerism physically different spectra can produce the same color sensation. Metameric spectra are interchangeable for the purposes of addition, but not multiplication, so any color algebra is necessarily an approximation to physical reality. Nevertheless, because the majority of naturally-occurring spectra are well-behaved (e.g., continuous and slowly-varying), color algebras can be formulated that are largely accurate and agree well with human intuition. Here we explore the family of algebras that result from associating each color with a member of a three-dimensional manifold of spectra. This association can be used to construct a color product, defined as the color of the spectrum of the wavelength-wise product of the spectra associated with the two input colors. The choice of the spectral manifold determines the behavior of the resulting system, and certain special subspaces allow computational efficiencies. The resulting systems can be used to improve computer graphic rendering techniques, and to model various perceptual phenomena such as color constancy.
When solving problems, multi-person airline crews can choose whether to work together, or to address different aspects of a situation with a divide and conquer strategy. Knowing which of these strategies is most effective may help airlines develop better procedures and training. This paper concentrates on joint attention as a measure of crew coordination. We report results obtained by applying cross recurrence analysis to eye movement data from two-person crews, collected in a flight simulator experiment. The analysis shows that crews exhibit coordinated gaze roughly one sixth of the time, with a tendency for the captain to lead the first officers visual attention. The degree to which crews coordinate their gaze is not significantly correlated with performance ratings assigned by instructors; further research questions and approaches are discussed.
Complex user interfaces (such as those found in an aircraft cockpit) may be designed from first principles, but inevitably must be evaluated with real users. User gaze data can provide valuable information that can help to interpret other actions that change the state of the system. However, care must be taken to ensure that any conclusions drawn from gaze data are well supported. Through a combination of empirical and simulated data, we identify several considerations and potential pitfalls when measuring gaze behavior in high-fidelity simulators. We show that physical layout, behavioral differences, and noise levels can all substantially alter the quality of fit for algorithms that segment gaze measurements into individual fixations. We provide guidelines to help investigators ensure that conclusions drawn from gaze tracking data are not artifactual consequences of data quality or analysis techniques.
When solving problems, multi-person airline crews can choose whether to work together, or to address different aspects of a situation with a divide and conquer strategy. Knowing which of these strategies is most effective may help airlines develop better procedures and training. This paper concentrates on joint attention as a measure of crew coordination. We report results obtained by applying cross recurrence analysis to eye movement data from two-person crews, collected in a flight simulator experiment. The analysis shows that crews exhibit coordinated gaze roughly 1/6th of the time, with a tendency for the captain to lead the first officer’s visual attention. The degree to which crews coordinate their gaze is not significantly correlated with performance ratings assigned by instructors; further research questions and approaches are discussed.
When solving problems, multi-person airline crews can choose whether to work together, or to address different aspects of a situation with a divide and conquer strategy. Knowing which of these strategies is most effective may help airlines develop better procedures and training. This paper concentrates on joint attention as a measure of crew coordination. We report results obtained by applying cross recurrence analysis to eye movement data from two-person crews, collected in a flight simulator experiment. The analysis shows that crews exhibit coordinated gaze roughly 16th of the time, with a tendency for the captain to lead the first officers visual attention. The degree to which crews coordinate their gaze is not significantly correlated with performance ratings assigned by instructors; further research questions and approaches are discussed.
Eye movements generally have both reflexive and voluntary aspects, but torsional eye movements are usually thought of as a reflexive response to image rotation around the line of sight (torsional OKN) or to head roll (torsional VOR). In this study we asked whether torsional responses could be modulated by attention in a case where two stimuli rotated independently, and whether attention would influence the latency of responses. The display consisted of rear-projected radial "pinwheel" gratings, with an inner annulus segment extending from the center to 22 degrees eccentricity, and an outer annulus segment extending from 22 degrees out to 45 degrees eccentricity. The two segments rotated around the center in independent random walks, stepping randomly 4 degrees clockwise or counterclockwise at 60 Hz. Subjects were asked to attend to one or the other while keeping fixation steady at the center of the display. To encourage attention on one or the other segment of the display, subjects were asked to move a joystick in synchrony with the back and forth rotations of one part of the image while ignoring the other. Eye torsion was recorded with the scleral search coil technique, sampled at 500 Hz. All four subjects showed roughly 50% stronger torsion responses to the attended compared to unattended segments. Latency varied from 100 to 150 msec across subjects and was unchanged by attention. These findings suggest that attention can influence eye movement responses that are not typically under voluntary control.
Eye movements generally have both reflexive and voluntary aspects, but torsional eye movements are usually thought of as a reflexive response to image rotation around the line of sight (torsional OKN) or to head roll (torsional VOR). In this study we asked whether torsional responses could be modulated by attention in a case where two stimuli rotated independently, and whether attention would influence the latency of responses. The display consisted of rear-projected radial pinwheel gratings, with an inner annulus segment extending from the center to 22 degrees eccentricity, and an outer annulus segment extending from 22 degrees out to 45 degrees eccentricity. The two segments rotated around the center in independent random walks, stepping randomly 4 degrees clockwise or counterclockwise at 60 Hz. Subjects were asked to attend to one or the other while keeping fixation steady at the center of the display. To encourage attention on one or the other segment of the display, subjects were asked to move a joystick in synchrony with the back and forth rotations of one part of the image while ignoring the other. Eye torsion was recorded with the scleral search coil technique, sampled at 500 Hz. All four subjects showed roughly 50 stronger torsion responses to the attended compared to unattended segments. Latency varied from 100 to 150 msec across subjects and was unchanged by attention. These findings suggest that attention can influence eye movement responses that are not typically under voluntary control.
Multi-person teams are sometimes responsible for critical tasks, such as flying an airliner. Here we present a method using gaze tracking data to assess shared visual attention, a term we use to describe the situation where team members are attending to a common set of elements in the environment. Gaze data are quantized with respect to a set of N areas of interest (AOIs); these are then used to construct a time series of N dimensional vectors, with each vector component representing one of the AOIs, all set to 0 except for the component corresponding to the currently fixated AOI, which is set to 1. The resulting sequence of vectors can be averaged in time, with the result that each vector component represents the proportion of time that the corresponding AOI was fixated within the given time interval. We present two methods for comparing sequences of this sort, one based on computing the time-varying correlation of the averaged vectors, and another based on a chi-square test testing the hypothesis that the observed gaze proportions are drawn from identical probability distributions.We have evaluated the method using synthetic data sets, in which the behavior was modeled as a series of activities, each of which was modeled as a first-order Markov process. By tabulating distributions for pairs of identical and disparate activities, we are able to perform a receiver operating characteristic (ROC) analysis, allowing us to choose appropriate criteria and estimate error rates.We have applied the methods to data from airline crews, collected in a high-fidelity flight simulator (Haslbeck, Gontar Schubert, 2014). We conclude by considering the problem of automatic (blind) discovery of activities, using methods developed for text analysis.