AN EYE ON REACTOR AND COMPUTER CONTROL
This article also appears in the Oak Ridge National Laboratory
Review (Vol. 25, No. 2), a quarterly research and development
magazine. If you'd like more information about the research
discussed in the article or about the Review, or if you have any
helpful comments, drop us a line. Thanks for reading the Review.
A nuclear power plant operator stared at the computer screen and
yawned. As the end of a long shift approached, he was feeling very
tired as he looked at icons of pumps, pipes, and turbines. Even so,
it was still early in the reactor startup sequence. When an
indication that the reactor had gone critical appeared on the
screen, the operator tried to rouse himself. There was no need for
him or anyone else to be alarmed, however. Based on the way the
operator looked at the screen, the computer had already sensed that
he was tiring and had begun to assist the operator in reactor
control. The computer zoomed in on the displays the operator needed
to monitor reactivity and highlighted the important data. It would
even be capable of assuming partial reactor control if necessary.
At a research laboratory, a cognitive scientist put a diskette in
her computer and studied the screen. The data indicated that a
reactor operator had been glancing at a number of widely scattered
points in a short time on the computer screen in the reactor
control room. "All these eye movements," said the scientist, "tell
me the operator is experiencing excessive mental work load. It
seems that several displays associated with reactor control must be
redesigned to make it easier for the operator to understand quickly
what's going on."
A teenager who lost the use of both hands as a result of an
automobile accident had found a way to write again. His father
bought him an "eye typewriter"--a computer that displays and prints
letters from a display of the alphabet on the screen in the order
in which they are stared at. The boy enjoyed his ability to control
a computer simply by looking at it.
These futuristic scenarios suggest that information on eye
gazes--the way people look at an object--can be put to use to
determine a person's mental work load and level of fatigue, to
guide the design of computer displays to speed human processing of
information, and to control computers. Other applications include
controlling camera positions on robots and guiding an artificial
intelligence system in recognizing enemy targets.
At ORNL computer software has been developed to make possible an
improved eye-gaze measurement technology. Such an innovation could
be the basis for advanced eye-gaze systems that may have
applications such as those mentioned above.
THE PROBLEM OF REACTOR CONTROL
The nuclear power plant, like many other process control
environments, is an extremely complex system. Control rooms of
conventional nuclear power plants reflect the complexity of the
systems they control. Visitors are usually overwhelmed by the
hundreds of meters, gauges, and switches that are arranged on vast
control panels. Even a trained operator finds it difficult to be
totally aware of all the information available in the control room
and to make decisions about plant operation under time pressure.
Thus, it is not surprising that human error in the control of
nuclear power plants is often implicated in incident reports
concerning a compromise of plant safety or unnecessary plant
shutdown.
The physical functions of the nuclear power plant are relatively
well understood by designers. However, little is understood about
mental tasks performed by operators and the associated types of
cognitive error. Furthermore, little is known about the types of
information actually required to perform these cognitive operations
and about the form in which they should be represented to support
the operator's mental model of the plant. A central concern is how
to provide critical cognitive support to the operator to ease the
effects of excessive mental work load, which can lead to human
error. These concerns are also important for other complex
human-machine systems, including teleoperated systems, chemical
processing plants, spacecraft, airplanes, and other transportation
vehicles. One approach to reducing operator error is the use of
advanced eye-gaze measurement technology.
A WINDOW TO THE MIND
It has been said that you can sometimes tell what a person is
thinking "by the look in his eye." The look, or eye gaze, can be
measured by an eyetracker, an unobtrusive device that can determine
the spot on a computer monitor at which a person is looking.
Eyetrackers can measure eye-gaze direction 30 times per second.
Cognitive psychologists have been using eye-gaze measurement
methods for over 15 years to capture "direct" snapshots of mental
processes, including how people read. According to the "eye-mind
hypothesis," the eye fixates on the symbols currently being
processed by the brain.
Several experiments have demonstrated that the eye can be a window
to the mind. In a typical experiment, human subjects were shown a
small array of simple drawings of common objects. When the subjects
were asked, "What makes of car can you name?", they tended to look
at the drawing of a car while responding. Furthermore, if the
subjects were asked the same question after the display was
removed, they still fixated on the same position in space where the
drawing of the car had been located. These results suggest that eye
fixations play an important organizational or place-keeping role in
cognition.
In ORNL's Engineering Physics and Mathematics Division, the
Cognitive Systems and Human Factors Group is now using eye-gaze
data to increase our understanding of how operators view and use
data shown in information displays for complex systems. Eye-gaze
data are enabling us to gain a better understanding of how
information should be presented so that it can be easily understood
and used.
Curiously, the operator is the only control element in the plant
whose behavior cannot be described in terms of a known set of
equations, because little is known about the cognitive activity of
operators. Because the cognitive activity of the brain cannot be
observed directly, little attempt was made for many years to
understand what operators were actually thinking while controlling
the plant. System designers most likely assumed that a plant could
be operated safely and efficiently if operators were trained to
follow procedures and that, consequently, it was not necessary to
understand the mental processes of the operator.
An obvious alternative approach in the study of cognitive behavior
is to ask operators what they are thinking and use their answers to
suggest ways to better support cognitive tasks. Verbal protocol
analysis techniques have been used with some success, but they
suffer from several weaknesses. Psychological research has shown
that verbal reports are biased toward the expectations of the
questioner, or what the respondent believes are questioner
expectations. Answers often conform to textbook statements of
written procedures. Verbal explanations usually reveal the tip of
the iceberg, but a vast store of tacit knowledge often remains
hidden during the fast pace of events.
On the other hand, eye-gaze data are quantitative and can be
treated as any other quantitative data. The scan path the eye
traces over a display is objective and unbiased. Finally, eye
movements can occur rapidly and keep pace with high-speed cognitive
processes.
THE EYE-GAZE SYSTEM
Eyetracking technology has undergone a slow evolution over the past
20 years. The new eye-gaze system marketed by LC Technologies of
Fairfax, Virginia, offers state-of-the-art eyetracking. Most of the
hardware is off the shelf.
Eyetracking is done with a standard personal computer equipped with
a video frame-grabber board. An infrared-sensitive high-speed
camera is mounted under a standard high-resolution computer screen.
A small light-emitting diode (LED) is positioned at the center of
the 75-mm camera lens.
When the human eye is focused in the camera lens, infrared light
from the LED is reflected off the eye. A simple algorithm takes
advantage of two optical phenomena resulting from infrared
illumination. One of these effects is the bright-eye effect, an
illumination of the pupil similar to the red-eye effect that
photographers try to avoid. The other is a bright spot created on
the surface of the cornea, known as corneal glint. If the vector
relationship between the pupil center and corneal glint can be
determined, then the direction of the eye gaze can be calculated.
Furthermore, if someone is calibrated to a particular display
screen (a simple procedure involving fixating on a series of
circles presented on the screen), then the point being looked at
can also be calculated.
This simple method of determining eye gaze was discovered by Tom
Hutchinson of the University of Virginia in the early 1980s, and it
was licensed for commercial production to LC Technologies in 1988.
LC Technologies developed some very efficient image-processing
software to locate the pupil center and corneal glint in real time.
This software is able to discriminate these phenomena from
extraneous effects, such as reflections from spectacles and other
bright spots on the eyeball surface.
APPLICATION SOFTWARE DEVELOPED AT ORNL
At ORNL we have developed application software for the eye-gaze
system to fulfill two broad purposes. First, we want to provide the
researcher with a visualization tool to immediately show the scan
path taken by the operator's eyes while inspecting a display screen
for process control. The trace playback software displays the
graphical interface, and a drawing of connected line segments is
overlaid on the monitor. Each new point in the connected line
drawing represents a new gaze point in the protocol. Playback speed
is user-selectable and the playback may be paused at any point. The
total number of line segments visible in the drawing is also
user-selectable to avoid excessive clutter on the monitor for long
protocols. When the trace reaches the maximum length, old
gazepoints and line segments are erased and replaced by new
gazepoints and segments.
Using the trace playback tool, the analyst may test a new graphical
user interface and immediately view the trace laid over the screen.
It is possible to see successive fixations at the desired speed and
correlate the trace with meaningful objects in the underlying
display. This tool can help the analyst build intuitions about
connections between eye-gaze activity and graphical user
interfaces. It is also useful for quickly evaluating displays for
human factors engineering.
Second, we are also building a library of quantitative tools for
analysis of eye-gaze protocols. The data analysis tools use an
output file containing records of successive gazepoints collected
while the operator inspects a graphical user-interface. These tools
can be used off-line to analyze multiple data sets. We have
identified at least three layers in eye-gaze protocols to reduce
the large number of data that must be analyzed. The gaze point--the
spot on the screen where the user is looking--is the lowest level
of data; it is sampled at a rate of 30 Hz by the eye-gaze system.
At this rate of sampling, 1800 gaze points will be determined for
every minute of data collection.
An intermediate level of analysis is the fixation--a sequence of
gaze points within a small area that occurs over a short time. Eye
fixations are closely related to detailed visual processing of
objects in the scene, and they occur in the foveal or central
region. Virtually every object is recognized when the eye fixates
on it, and not while the eye is engaged in rapid, or saccadic,
movement. The fixation lasts from 200 ms to several seconds. Even
while the eye is fixated, gaze-point activity is characterized by
slight jitters and perhaps some drift. New fixations are
characterized by sudden movement. At ORNL we have developed a
fixation post-processing routine to analyze raw data on gaze points
and identify the location and duration of fixations. This procedure
reduces the number of data by up to an order of magnitude.
The highest level of analysis is the area of interest, which
corresponds to a meaningful region in the graphical user interface.
For example, if the display shows a plant schematic, the area of
interest might correspond to a region containing drawings of
valves, pumps, or turbines as well as connections such as pipes.
Eye-fixation protocols can be analyzed to determine how much scan
time is given to each area of interest. This measure indicates the
relative depth of cognitive processing given each area on the
display.
We can also study the transfer of attention from one area of
interest to another by assembling an area-of-interest transition
matrix. This matrix shows the number of fixations between each
area-of-interest pair, indicating that the user is combining
information from several areas of interest before making a
decision. The complexity of the scan path is reflected in the
number of successive fixations between areas of interest as
compared with the number of successive fixations within areas of
interest. We are trying to determine whether the complexity of
transitions among several areas of interest, based on eye-fixation
data, is related to the mental work load of inspecting a graphical
user interface.
COGNITIVE ENGINEERING EXPERIMENTS PLANNED
We are preparing to conduct cognitive engineering experiments using
a UNIX workstation at ORNL. We selected a UNIX platform to separate
the development platform from the eye-gaze system. The goal of much
of our development work on the UNIX workstation has been to create
experimental tasks for which eye-gaze data can be captured. In
addition, we want to store keyboard strokes and mouse clicks to
correlate with eye-gaze protocols. Some of our original graphical
user-interfaces were created using the X Window System on the UNIX
platform to ensure portability of the software.
Controlled experiments are needed to standardize procedures for
collecting eye-gaze protocols. In the standard experimental setup,
a volunteer is seated at the workstation and an infrared-sensitive
camera, calibrated to the monitor, is mounted underneath. A series
of graphical user interfaces to various human-machine systems are
presented to the volunteer subject, who is charged with a specific
task or responsibility. In a prediction task, for example, each
volunteer is shown initial conditions of a transient event, such as
loss of a primary pump, and a candidate outcome state, such as
elevation of coolant temperature, is displayed in another window.
The subject's task is to verify the plausibility of the outcome
state.
A related task is diagnosis. Each volunteer is shown an outcome
state and is asked to choose among alternative initiating events
that could lead to this state. In this example, to get the right
answer, the subject must select "loss of power to pump" instead of
"turbine trip" or "feedwater valve fails to open." In fault
detection, a normal transient is depicted. The subject monitors the
transient and is asked to respond appropriately to the fault when
it is detected. In this example, the subject first notices the
temperature change or pump coastdown. For implausibility detection
a subject views a display showing dynamic data from a simulated
system and attempts to determine the validity of the data.
EVALUATION OF DISPLAYS
Scan path traces from early eyetracking instruments in
human-machine systems were first used to analyze the arrangement of
displays. The aviation industry used eyetracking in the cockpit
because each pilot is faced with a cluttered arrangement of
displays on an instrument panel. When monitoring the status of the
aircraft, the pilot views the displays in a relatively fixed order.
If the pilot is required to repeatedly scan back and forth across
the panel to locate and read instruments, monitoring becomes quite
cumbersome. This situation will be reflected in an extended and
convoluted scan path. The eye scan path reveals the order in which
the pilot fixates on the instruments. From this information,
instruments that are fixated on in succession can be grouped to
simplify scan paths during monitoring.
Scan paths can reveal more than the best way to arrange instrument
displays. The eye-mind hypothesis suggests that stages of an
operator's cognitive processing can be inferred while this
individual inspects graphical user interfaces. The actual stages of
processing inferred from the data can then be compared with an
ideal cognitive model to determine how actual processing departs
from the intended use of the display. This comparison provides the
analyst with a powerful basis upon which to evaluate the
effectiveness of advanced interface concepts.
In addition, types of cognitive error and their frequency will be
observable in eye-gaze protocols. Such errors may include reading
a display incorrectly, forgetting to perform an action, and
reaching the wrong conclusion. Models of cognitive error during
inspection of operator interfaces offer a powerful new dimension of
evaluation. Eye-gaze protocols both suggest and validate sources of
cognitive error. We believe that "natural" interfaces that conform
to the mental model of the operator ought to minimize opportunities
for cognitive error.
PREDICTION OF MENTAL WORK LOAD
The complexity of control rooms of conventional nuclear power
plants is well-documented. The need to integrate diverse
information simultaneously can produce periods of peak overload for
the operator. Therefore, it is important to be able to predict
those situations that may result in excessive mental work load for
individual operators. During the onset of an accident such as loss
of coolant, the operator would be inundated by a large number of
alarms and annunciators. Identifying the relevant alarms is a
difficult pattern-recognition problem that may induce high work
load and stress.
Indications of mental work load can be found in eye-gaze protocols
as operators extract information from displays showing relevant
data about plant status. Psychologists have long known that pupil
diameter is related to cognitive arousal or the need for
information, and hence, the degree of mental work load. Scan path
complexity is another estimator of mental work load. We have
performed pilot testing with problems in mental multiplication of
numbers. Our results indicate that difficult problems tend to
generate longer and more complex scan paths. We believe that
eye-gaze data reflect not only the degree of effectiveness of
interfaces but also task complexity.
A correlation may exist between mental work load and scanning
strategies employed for different tasks. We suspect that
low-work-load activity such as passive monitoring leaves a distinct
eye-gaze signature. Eye movements during passive monitoring are
referred to as open-loop scanning; the next fixation does not
depend on the information content of the present fixation. Displays
may be scanned according to a simple predetermined sequence. On the
other hand, fault diagnosis is probably characterized by
closed-loop scanning, in which the location of the next fixation is
not determined until cognitive processing of the present fixation
is completed. If this hypothesis is true, closed-loop scanning
should be detectable in the correlations between eye fixations and
the information content of displays.
As suggested in the introduction, a possible future application of
eye-gaze measurement technology is determining when reactor
operators are too fatigued to do their jobs effectively. In such
cases, detection of excessive mental work load by a computer could
activate it to assist the operator in reactor control.
Intelligent interfaces could guide the operator's attention to
relevant displays by highlighting, blinking, or zooming in on the
appropriate information. Real-time expert systems could present the
operator with an estimate of current plant status together with an
explanation of what happened. Many other kinds of cognitive
assistance are possible under conditions of peak work load.
TELEROBOTICS AND HUMAN-IN-THE-LOOP TARGET RECOGNITION
The human eye may also be used to control a computer. In this case,
eye-gaze measurements can be entered into the computer as a
substitute for keystrokes and mouse clicks. An individual facing a
graphical user interface can issue a specific command to the
computer merely by looking at a specific region on the monitor.
Controlling a computer by looking at it promotes a feeling of power
in the user. It is almost as if the user commands the computer
directly with the mind. The interface, if properly designed, can
become so natural that the user forgets it is present, leaving the
impression of mind control.
LC Technologies has produced several impressive applications for
the disabled. One of the most useful programs is an eye typewriter.
A graphical keyboard is presented to the user who types by
focusing, or "dwelling," on the region containing the next letter
of text to be generated. When dwell time exceeds the user-selected
threshold, the letter appears in the text at the top of the
monitor. A special "key" is used to delete mistaken picks. One
problem with eye input is the so-called "Midas touch" situation.
When the user is searching an area for a particular letter, many
false positives can be created: such items are evaluated and then
rejected. The incidence of false positives can be lowered by
increasing the dwell threshold for selection, but the trade-off for
increased accuracy is decreased processing speed.
Teleoperated vehicles and robots that have many degrees of freedom
that may be manually controlled may benefit from eye input. For
example, a legged vehicle under manual control while climbing
stairs may require both hands of the teleoperator to control leg
motions. Eye input could be used to control additional degrees of
freedom, such as video camera position.
We are now engaged in a proof-of-principle project that will
demonstrate eye-gaze control of the perspective view of a 3-D
interactive graphical interface. The new interface is "natural"
because object rotation responds to the user's active search for
information. When the user shifts the eyes to see hidden parts
around the edge of an object such as an airplane or robot arm, it
automatically rotates toward the user and reveals the areas of
interest.
Data on human eye gazes may also enhance computer vision, one of
the most difficult problem areas in robotics and automatic target
recognition (ATR). ATR systems have not achieved the reliability
and robustness required for this difficult task. The human visual
system is highly advanced and able to use the parallel architecture
of the brain to perform visual tasks unequaled by computer systems.
Human-assisted ATR may be able to improve the performance of such
systems.
For example, if a simplistic ATR computer is asked to search for
enemy tanks, it may scan the entire scene, including the sky, for
such objects. A human would look at the ground, not the sky, for
such targets. In human-assisted ATR, real-time eye-gaze data taken
as a person surveys the scene would be used to guide the ATR in
identifying regions of that scene where targets most likely will be
located. As a result, computing time is saved because the ATR can
avoid scanning regions not considered likely to have potential
targets.
FUTURE DIRECTIONS FOR EYE-GAZE SYSTEMS
A limiting feature of the eye-gaze system is intolerance to head
motion. The head must be kept stationary within a tight volume to
keep the eye focused in the center of the camera lens. It is
possible to mount a camera on specially constructed headgear to
ensure that the eye is always in focus. But this sort of gear is
obtrusive and can provide eye-gaze direction relative only to the
head. It is difficult to correlate the eye-gaze data to the spatial
coordinates of a display. Some systems have a headtracker that
allows the camera to follow the eye as the head moves through a
larger volume of space. The camera remains fixed in its mounted
position, but a motorized rotating mirror and motorized focus ring
track the eye to keep it in focus. Image processing, ultrasound,
and a second camera can all be used to predict the direction of
head movement to control the motorized mirror. Such systems are in
various stages of development, but none, as yet, are very reliable.
The headtracker described above operates in an environment
comparable to the volume occupied by a user seated before a
workstation monitor. Future eye-gaze systems will be able to
perform unobtrusive computation of head position and eye-gaze
direction in a small control room. Because future systems will
simultaneously calibrate several displays, it will then be possible
to collect eye-gaze data from an entire control room crew while
using multiple displays to control a complex system. As eye-gaze
systems evolve from the laboratory to real-life situations, they
will become valuable tools for understanding human-machine systems,
improving communication of information, and easing the mental work
loads of operators of process systems, including nuclear power
plants. Biographical SketchesJack C. Schryver is a research staff
member of the Cognitive Systems and Human Factors Group in the
Intelligent Systems Section of ORNL's Engineering Physics and
Mathematics Division. He has a B. A. degree from the University of
California at Los Angeles and a Ph.D. degree in cognitive and
experimental psychology from the University of California at
Irvine. His research interests include experimental cognitive
science, cognitive modeling, simulation, and human-computer
interaction and visualization.Helmut E. (Bill) Knee is leader of
the Cognitive Systems and Human Factors Group. He has an M.S.
degree in nuclear engineering from UCLA and an M.S. degree in
business administration from the University of Tennessee. His
research interests include human performance and visualization in
complex control environments, human and system reliability, and
behavioral and cognitive modeling.
Jack Schryver and Bill Knee
(keywords: eye gaze, human factors, human-machine systems, reactor
control)
------------------------------------------------------------------------
Please send us your comments.
Date Posted: 2/7/94 (ktb)