&EPA
           United
           Environmental Protection
           Agency
           Environmental Research
           laboratory
           Harragansett Rl 02882
   3-80-010
January 1980
           Research and Development
Development of an
Interactive System to
Study Sub-Lethal
Effects of Pollutants
on the Behavior of
Organisms

-------
                RESEARCH REPORTING SERIES

Research reports of the Office of Research and Development, U.S. Environmental
Protection Agency, have been grouped into nine series. These nine broad cate-
gories were established to facilitate further development and application of en-
vironmental technology.  Elimination  of traditional grouping was consciously
planned to foster technology transfer and a maximum interface in related fields.
The nine series are:

      1.   Environmental  Health Effects Research
      2.   Environmental  Protection Technology
      3.   Ecological Research
      4.   Environmental  Monitoring
      5.   Socioeconomic Environmental Studies
      6.   Scientific and Technical  Assessment Reports (STAR)
      7.   Interagency  Energy-Environment Research and Development
      8.   "Special" Reports
      9.   Miscellaneous Reports

This report has been assigned to the ECOLOGICAL RESEARCH series. This series
describes research on  the effects of pollution on humans, plant and animal spe-
cies, and materials. Problems are assessed for their long- and short-term influ-
ences. Investigations include formation, transport, and pathway studies to deter-
mine the fate of pollutants and their effects. This work provides the technical basis
for setting standards to minimize undesirable changes in living organisms in the
aquatic, terrestrial, and atmospheric environments.
This document is available to the public through the National Technical Informa-
tion Service, Springfield, Virginia 22161.

-------
                                          EPA-600/3-80-010
                                          January 1980
DEVELOPMENT OF AN  INTERACTIVE SYSTEM TO
STUDY SUB-LETHAL  EFFECTS OF POLLUTANTS
      ON THE BEHAVIOR OF ORGANISMS
                   by
           John 0. B. Greaves
  Department of Electrical Engineering
                  and
            Robert S. Wilson
         Department of Biology
 Southeastern Massachusetts University
   N. Dartmouth, Massachusetts  02747
           Grant No. R803244
            Project Officer

             Don C. Miller
   Environmental Research Laboratory
   Narragansett, Rhode Island  02882
   ENVIRONMENTAL RESEARCH LABORATORY
   OFFICE OF RESEARCH AND DEVELOPMENT
  U.S. ENVIRONMENTAL PROTECTION AGENCY
   NARRAGANSETT, RHODE ISLAND  02882

-------
                                  DISCLAIMER

     This report has been reviewed by the Environmental Research Laboratory,
Narragansett, U.S. Environmental Protection Agency, and approved for
publication.  Approval does not signify that the contents necessarily
reflect the views and policies of the U.S. Environmental Protection Agency,
nor does mention of trade names or commercial products constitute
endorsement or recommendation for use.
                                       ii

-------
                                   FOREWORD

     The Environmental Research Laboratory of the U.S. Environmental
Protection Agency is located on the shore of Narragansett Bay, Rhode
Island.  In order to assure the protection of marine resources, the
laboratory is charged with providing a scientifically sound basis for Agency
decisions on the environmental safety of various uses of marine systems.  To
a great extent, this requires research on the tolerance of marine organisms
and their life stages as well as of ecosystems to many forms of pollution
stress.  In addition, a knowledge of pollutant transport and fate is needed.

     This report describes a three-year development project culminating  in
the design and full  implementation of a computerized data acquisition and
analysis system for  the study of animal behavior.  Movements of organisms
are recorded on video tape and later played back into a television computer
interface.  The user, with full interactive graphics support, extracts
quantitative information on the movements of large numbers of organisms.
The work cited in this report was essentially complete as of June, 1977.
However the system allows new capabilities to be added in the form of
software modules, and its repertoire continues to grow.  An account of this
project has also been presented in an invited paper (10) at the "State of
Marine Environmental Research" Symposium, June 1977, EPA Environmental
Research Lab in Narragansett, Rhode Island.
                                      iii

-------
                                   ABSTRACT

     Experimental investigation of the movements of organisms often entails
the acquisition and processing of large samples of spatio-temporal data.  An
interactive, interpretive, on-line computer television system (viz., the
Bugsystem) was initially developed at the University of California at Santa
Barbara in order to expedite such analyses.  Aspects of the structure of
this prototype system are outlined.  Its effectiveness is evaluated with
regard to the problems confronting the bio-behavioral researcher.

     A second generation system has been developed under a research grant
from the Environmental Protection Agency.  Utilizing new hardware and soft-
ware, it in many ways constitutes a generalization of its prototype.  We
describe features of the refined system which provide for the following:  a
large degree of machine-independence, significant expansion of the size of
data records, inclusion of experimental parameters and variables within the
data structure, investigation of rotational and flectional movement, statis-
tical analysis, advanced graphics and tracking of organisms in three
dimensional space.  Current utilization of the Bugsystem for research in
behavioral physiology .and. the potential for future development are discussed.
                                      iv

-------
                              TABLE OF CONTENTS
Foreword	iii
Abstract	   iv
List of Figures	   vi
Acknowledgements 	  vii
       I.  Introduction  	    1
      II.  Coupling to Research Environments 	    8
     III.  Processing and Display	   14
      IV.  Discussion	   33
       V.  References	   36

-------
                                   FIGURES

No.                                                                   Page

 1.  Block diagram of Bugsystem 	 	     9

 2.  Apparatus used in collecting data	    10

 3.  Polarotatic orientation of Daphnia pulex paths 	    19

 4.  Polarotatic orientation of Daphnia pulex-wedge
       histograms	    20

 5.  Merged path files of Balanus improvisus  	    21

 6.  Linear velocity vs. time for EL improvisus	    23

 7.  Average linear velocity of J3. improvisus vs. copper
       concentration  	    26

 8.  Distributions of instantaneous linear velocity of
       ]3. improvisus at two levels of copper exposure	    27

 9.  Phototactic orientation of ]3. improvisus at two
       levels of copper exposure  	    28

10.  Dependence of phototactic orientation in J3. improvisus
       upon copper concentration  	    30

11.  Dependence of rate of change of direction of EL improvisus
       upon copper concentration  	    31

12.  Dependence of rate of change of direction of travel
       in Daphnia pulex in unpolarized and polarized light   ....    32

-------
                               ACKNOWLEDGMENTS

     The authors would like to thank Dr. William Lang for his friendly
assistance in debugging the system and guiding  it from  its development
stages to its fully operational status.  Dr.  Don C. Miller as Project
Officer provided the initiative and support to  start and manage the research
grant development of the Bugsystem.  He superivsed  and  coordinated the
merging of technological skills with the needs  of the animal behavior
research team and the goals of the Environmental Protection Agency.  Dr.
Miller and Dr. Lang also provided some of the data  shown in the figures of
this report.  Dr. William Hand offered valuable suggestions during the early
development of the system as the first real biological  user and contributed
a user's manual.  The programmers who helped  create the extensive software
package include:  Steve Fyfe, Richard Boucher,  Paul Messier, Bob Souza,
Peter Roy, Dr. Hans van den Biggelaar, Wallace  Shubach, Barbara Blake and
David Sleczkowski.  Their help and patience was appreciated.  Mr. Jay
Sinnett was always available for consultation in the electronics area and he
provided refinements to the Bugwatcher hardware and built the video
multiplexing circuitry for the dual camera data collection scheme.  And
finally, we would like to thank Dr. Eric Schneider, who as the Director of
the Narragansett laboratory, recognized this  project as a potentially
valuable asset for the Agency and committed himself and the laboratory
resources to the successful completion of the project.
                                     vii

-------
                                  SECTION I

                                 INTRODUCTION
     The fundamental focus of behavioral research  is the description and
explication of what individual organisms do.  Because those biological
activities most often classified as "behavTor" consist  largely  (although not
exclusively) of the movements of organisms, quantitative investigation of
behavior is often dependent upon the collection of temporal sequences of
spatial information.  Cinematography is the classical method used to gather
such information.  Behavioral variables such as the position of an organism,
its orientation and angles of flection of  its appendages are extracted from
the motion pictures by means of frame-by-frame analysis.  An important
advantage of the technique is its flexibility; it may,  in principle, be
employed in the investigation of any overt behavior which can be photo-
graphed.  However, two factors preclude the widespread  use of this method:
(1) manual quantitization via frame-by-frame analysis is a lengthy and
tedious process; and (2) once the data are obtained, a  substantial amount of
subsequent data processing may be required in their analysis.

     The Bugsystem (3, 5, 6) was designed  to enable the acquisition and
processing of video data by a minicomputer.  Behavioral data are initially
recorded using standard closed-circuit television  equipment.  These data are
analyzed by replaying a video tape into a  specially designed
video-to-digital processor (christened the "Bugwatcher").  This device acts
as an edge detector, greatly reducing the  information flow to the computer
and thereby allowing the real-time collection of spatial coordinates delin-
eating the outlines of moving organisms.   Frame-by-frame analysis of
digitized video data is achieved through the use of specially designed pro-
grams tailored to the task of quantifying  behavioral variables  of interest
to the researcher.

     The original version of the Bugsystem was developed by Greaves and
implemented on an IBM 1800 computer at the University of California, Santa
Barbara.  This prototye system, previously described by Greaves (5, 6), has
been utilized by Hand and Schmidt (7) and  by Wilson (9) to investigate the
photokineses and phototaxes of marine dinoflagellates.  However, several
features of this system severely limited the domain of  its application.
Supported by a research grant from the Environmental Protection Agency, we
have developed a second generation Bugsystem.  Our explicit goal in the
design of this system was to provide a flexible tool for the quantitative
investigation of behavior, a system capable of realizing much of the poten-
tial of frame-by-frame analytical techniques.

-------
     The purpose of this paper is to describe this second generation Bug-
system, emphasizing the way in which certain hardware and software refine-
ments have expanded the scope of questions which may be conveniently
answered by means of "bugwatching".  We discuss the way in which the user
interacts with the system via a specially formulated "Behavioral Research
Language" and the way in which this language has been implemented upon
physical machines.  We also described the input of data to the system, the
processing and display of behavioral data and a variety of experimental
strategies accommodated by the system.  Finally, we outline work now in pro-
gress to provide for the analysis of movement in three dimensions.


DEFINING A BEHAVIORAL RESEARCH LANGUAGE (BRL)

     The Behavioral Research Language, or BRL, is a high level operator
based language that is tailored to the unique problems associated with the
input, scaling, analysis and display of video images of moving objects.  BRL
is an interpretive language which runs as an application program on a good
sized minicomputer and relies heavily upon user interaction with a storage
graphics terminal to input, plot, edit and transform the data through image
processing functions.  Image processing generally culminates in the com-
putation of paths or trajectories of the objects moving before the video
camera.  Sets of trajectories may be merged together and the data may be
transformed to yield time series of behavioral variables (e.g., linear vel-
ocity, angular velocity, direction of travel, etc).  These results may then
be analyzed statistically and the resultant data sets either listed numer-
ically or plotted on the graphics terminal or plotter.  Thus, the Bugsystem
consists of two basic subsystems:  (1) an unique image processing system for
the frame-by-frame analysis of video data, and (2) a signal processing
system for the statistical analysis of equispaced time series.

     The key element to understanding and using the BRL lies in grasping the
operator-operand-resultant nature of specifying functions or commands to the
system on the graphics terminal keyboard.  The general command syntax is as
fo11ows:

    *0perator/sw/sw Operand-name/sw/sw Resultant-name/sw/sw nl, . . ., n5

Where "*" is the prompting character, "Operator" is one of the available
functions (of which there are currently 96, with the list still growing),
"Operand-name" is the name of the input (or operand) data set,
"Resultant-name" is the name of the output (or resultant) data set, and the
numeric constants "nl" through "n5" are optional numeric constants which
govern details of the function of certain operators.  The "/sw" denote
optional "switches" (the word is taken from computer jargon), which are used
as operator modifiers or to supply special information to the operator being
used.  All data sets are disk resident and are specified by a four-letter
name that can be used to denote an experimental condition, a two-letter
extention that specifies the type of data represented (e.g., "VI"-video,
"PA"-path, "LV"-linear velocity, "CA"-categorized, etc.) and a six-letter

-------
front name that can be used to indentify the species studied and/or the date
of the experiment.  The front name must be specified only when  starting the
system and remains unchanged unless  it is explicitly modified by the LOAD
operator.  Operator names will henceforth be underlined  in the  text.

     For example, to plot all of the data in the video file called BUGS.VI,
the command would appear as "*PLOT BUGS.VI".  To plot only the  first twenty
frames of this file, one would~enTer "*PLOT BUGS.VI 1, 20".  To calculate
the linear velocities for the paths  in tfhT~file BUGS.PA, the command line  is
"*LVEL BUGS.PA BUGS.LV".  Note that  in this language the loops  required to
process all data elements within a data set are not explicitly  stated.  The
operator automatically processes all of the elements of  the operand data
file unless directed to a particular subset (e.g., "*PLOT 1, 20", as illus-
trated above).

     Three other aspects of BRL are worth including here.  The  first of
these concerns the way in which data are represented within the Bugsystem:
a data set consists of one or more vectors of variable length.  While per-
forming image processing operations, each vector represents one video frame;
one element of such a vector represents a single point in two dimensional
space.  As the analysis of the data proceeds through successive application
of operators to operand data sets, a single vector may represent an or-
ganism's path (i.e., a time series of Cartesian coordinates in  two space as
in the file "BUGS.PA" illustrated above) or a real function defined over the
length of such a path (e.g., the estimate of instantaneous linear velocity
as previously illustrated by the file "BUGS.LV").  Finally, in  the
statistical analysis of such data a  vector may correspond to a  set of sta-
tistical parameters, a collection of "bins" or categories established for
histograming, an estimate of an autocorrelation function, etc.

     The second aspect of the language to be considered  here are the
self-documenting aspects of BRL.  No one can be expected to memorize all of
the 96+ operator names, what they do in detail and the various  switches and
numeric constants which they expect.  To help in this regard, the NAMES
operator lists on the terminal the names of all the keyboard operators.
Moreover, entering "*Operator/HELP" for any of the available operators will
cause the system to print a full page of information describing what the
operator does, the types of operands for which the operation is defined,
what constants and switches are expected and an example  of the  operator's
use.

     The third aspect of BRL to be considered involves the construction of
user programs.  BRL was designed primarily to be an interactive language:
the user normally enters commands at the terminal one at a time and thereby
directs the analysis to its desired end.  It is also possible to create a
disk file of commands consisting of the operators and operand specifications
and to direct the system to execute this stored sequence of operations (a
computer program written in BRL) via the USER operator.

-------
THE IMPLEMENTATION OF BRL

     The implementation of BRL by means of the second generation  Bugsystem
differs significantly (both in hardware and in software).from that of the
prototype system.  This section outlines the main features of the new
system, giving reasons for their importance.

A New Bugwatcher

     The Bugwatcher hardware was redesigned to utilize medium scale  inte-
gration (MSI) circuitry.  Algorithmic state machine  (ASM) charts were
employed to formulate and to document the new design.  Functionally, the new
Bugwatcher is similar to its earlier counterpart:  it extracts digital X-Y
coordinate pairs representing points belonging to image  outlines from the
video raster scan and stuffs these coordinates into  the  computer's direct
memory access (DMA) channel.  The computer program to input these data
buffers them and writes them to disk, thereby allowing the collection of
extremely long data records.  Within such a record each  frame of video data
is represented as a vector.  Each video vector sent  by the Bugwatcher to the
computer contains not only a variable length list of coordinate data, but
also includes a leading header of fixed length.  The elements of this header
are refered to as "vector attributes" and are employed to encode  information
associated with each vector.  Video data possess four 16-bit words of attri-
bute information supplied by the Bugwatcher hardware:  (1)  a unique word
consisting entirely of zeroes used by the software to delimit frame
boundaries, (2) a descriptor word which contains an  encoding of the frame
rate at which the video data were digitized and the  on-off status of tone
stimulus markers, (3) a total frame counter that can be  used to determine
relative or absolute time intervals between data segments recorded at vary-
ing time intervals, and (4) auxiliary digital input  which allows an encoding
of an experimental variable (e.g., the direction of  the  source of stimu-
lation) to be automatically associated with each video frame.  Represen-
tation of stimulus conditions is discussed in more detail below (see
"Coupling to Research Environments").  This division of  vectors into attri-
butes and data applies to all vectors manipulated by Bugsystem software.
However, the meanings of each attribute depends on the type of data; e.g.,
one of the attributes of a path (represented as a single vector)  is the
starting frame number.


The Implementation Language—FORTRAN IV

     The prototype BRL system was implemented entirely in assembler  language
on an IBM 1800 computer and ran as a stand-alone system  with no operating
system support.  This made it non-portable and difficult to maintain and
expand.  The new BRL system was designed to overcome these major
shortcomings.  We chose FORTRAN IV as the language with  which to  implement
the Bugsystem.  FORTRAN IV has become a standard language among mini-
computers and has simplified the tasks of maintaining Bugsystem software and
training new programmers to implement new BRL operators.  The importance of

-------
these aspects cannot be overestimated since, all  in all, ten different
programmers have added software to the system during its three years of
development.  Each required instruction on the software conventions and use
of system utililty subroutine packages, but at least they knew the FORTRAN
language.

     The software was first operational on a POP  11/45 computer at South-
eastern Massachusetts University under the DOS-9  operating system.  It was
then implemented on a Data General ECLIPSE S/200  under the RDOS operating
system.  Some assembler language subroutines had  to be receded, including
the software drivers that handle the direct memory channel to the Bug-
watcher.  But the transition to the new computer  went fairly smoothly.  The
POP 11 system was maintained for development purposes after the ECLIPSE was
moved to the Narragansett EPA Laboratory.  Due to malfunctioning DOS-9
software, the POP 11 operating system was changed to RT 11—a change that
required as much or more development effort than  changing computers!

Virtual File Structure

     Certainly one of the major disadvantages to  using minicomputers for
large application software projects is the address space  limitation imposed
by its small word size.  This system was no exception.  To overcome this
limitation so as to allow both programs and data  to fit within allocated
memory, both program structures and data structures were  designed so that
only pieces of either resided in memory at any given time.  Manufacturers of
minicomputers recognize this problem and provide  software support for manip-
ulating overlayed programs; i.e., programs consisting of  parts which are
swapped in and out of main memory.  However, software support was not
available for similarly overlaying data sets.  The earlier prototype system
did not allow data sets to be any larger than the memory  buffer on the IBM
1800.  This was not acceptable in the newer system.  Certain Bugsystem
applications require the acquisition and analysis of behavioral records
consisting of many frames of video data or the trajectories of hundreds of
organisms.  The system required the potential to  manipulate data structures
ten to hundreds of times the size of main memory  available for data.

     One of the early software design goals was to simplify the process of
adding new BRL operators.  Each opertor was to be implemented in FORTRAN as
a program overlay.  An operator would be required to access as many as three
simultaneous data sets (e.g., the arithmetic operators PLUS, SUBTRACT,
MULTIPLY AND DIVIDE each required two operands and generate one resultant)
within a labeTicF common buffer of 8192 integer values (or 4096 single pre-
cision values).  Within the code that implements  a new operator, files are
accessed with a complete set of virtual file handlers coded in FORTRAN.
These routines provide services for opening an existing file, creating a new
file, reading a vector into the buffer from disk, writing a vector from the
buffer to the disk and closing a file.  To minimize disk  access time and
thereby insure optimum response time to the user, techniques employed in
managing other virtual memory systems were adapted to the Bugsystem.  All
data sets, stored on disk as contiguous files, are accessed directly using

-------
multiple block transfers.  If, for example, a command  is  issued from  an
applications program to read a given vector within a file, then the software
first determines if the vector is resident within the  buffer.  If  it  is
resident, then the routine immediately returns to the  calling program pro-
viding the length of the vector and a pointer into the buffer to the  first
element of the vector.  If it is not resident, its logical address within
the file is computed and it—and the vectors which succeed it—are read  into
the labeled common area.  A similar algorithm is employed when writing a
vector into a file.  A disk write is not required unless the output buffer
area is full.

     Funneling all file input/output through a common  set of routines has
significant advantages.  The development of new BRL operators is simplified
insofar as the underlying applications programs do not each separately (and
redundantly) require the algorithms needed to manipulate large files.
Another advantage lies in the increased portability of the software.  Ver-
sions of FORTRAN supported by different machines and operating systems vary
most markedly in their non-standardized methods of accessing files.   Machine
and operating system dependencies are thus isolated in a manageable number
of software modules, yielding more portable and more easily maintainable
software.
SYSTEM BLOCK DIAGRAM

     The major components needed to collect and analyze behavioral data  are
shown outlined in Figures I and 2.  Figure 1 shows the major components  of
the minicomputer system that were purchased on this research grant.  An
explanation of the components in Figure 1 follows.  The video tape deck
plays back the images of moving organisms of a given experiment  into the
video preprocessor (Bugwatcher), which renders the image into X-Y outline
coordinates and transfers these coordinates at direct memory access (DMA)
speeds (microseconds) into the ECLIPSE'S memory.  The TV monitor is used to
monitor the quality of the ordinary and quantized images and thus provides
operator feedback to ensure that the equipment settings are correct.  A  TV
camera may be fed directly into the Bugwatcher, but it is more convenient to
record the data at another site and bring in the video tape for  a "computer
session".  The audio channel of the tape deck can be used for recording  and
playing back tones used to indicate a change in experimental conditions
(e.g. initiation of a light stimulus).

     The Bugsystem is the name of the complete and rather large  network  of
software subsystems that runs as an interactive application program under
the RDOS operating system.  The Bugsystem uses a 32 K Word memory partition
and has built into it support for alphanumeric output on the terminal
(Tektronix 4010) or the line printer and graphics output to either the ter-
minal or the CALCOMP 1038 plotter.  Examples of the plotter output are
included in this paper.  The Bugsystem run time software executes from one
of the fixed disks and the users have their own removable disks  (5 M Bytes
each) for storing data.  The continuing software development is  conducted

-------
from another removable disk.  Every week the disks  are  dumped to mag  tape
for backup (9 track 800 BPI, RDOS dump format)  onto one of  three tapes  that
are rotated.  Every six months an archive tape  is made  of the development
software, and as often as requested by Bugsystem users  for  archiving  their
data onto mag tape.
     The equipment shown  in Figure 2  is  located in  the  biological
laboratory.  Organisms to be recorded are placed in a dish  or deep  well
slide and images of their movements are  captured on video tape.  The  TV
monitor  is present to insure that the recorded  images are free from spurious
noise.   Even though most  of its  circuitry is not required  in this context,
the Bugwatcher  is also present for the following reasons.   First, it  shows
what the quantized images look like as they  are being recorded.  This
insures  that no valuable  behavioral observations will be lost because of
improper sync or video levels when the tape  is  replayed during  input  to the
computer.  Second, the Bugwatcher has the tone  oscillator  and detector
circuits for recording tones on  the audio channel of the tape deck  to denote
a change in some experimental condition.

-------
                                  SECTION II

                      COUPLING TO RESEARCH ENVIRONMENTS
IMAGES
     The primary input to the Bugsystem consists of digitized video data.
If the video images are sufficiently "clean" (i.e., possess high contrast
and lack structural complexity), input of data to the computer is
accomplished automatically in real time.  Video tapes are replayed into the
Bugwatcher which compares the incoming video signal to a "video threshold"
set by the user.  Those points within the image where the video signal
crosses the video threshold are displayed on a video monitor.  The user
adjusts the video threshold to make these points coincide with the outlines
of moving organisms and selects a frame rate (ranging from sixty frames/sec
to one frame/min) appropriate for the relative speed of the organisms.  When
the BUGWATCHER JNPUT operator is executed, the computer accepts digitized
image information from the Bugwatcher at the selected frame rate.  Each
threshold point is represented in Cartesian .coordinates with 8-bit
resolution for each of two orthogonal components.  A video frame is
represented within the resultant data structure as a data vector with a
variable number of such points as its data elements.  An entire record (or
"video file") consists of a temporally ordered sequence of such vectors.

     The Bugsystem was originally developed for the investigation of the
behavior of motile microorganisms (3).  In this application, the organisms
are viewed under dark-field illumination swimming within a well slide upon
the stage of a compound microscope.  However, automatic digitization of data
is possible for any study of moving objects for which "clean" video records
are available.  One of us (Wilson) is currently using the second generation
system to study the effects of plane polarized light upon the behavior of
aquatic arthropods (Daphnia).  The animals move freely within a cylindrical
aquarium (diameter  =20 cm) under bright-field illumination and are viewed
using a macro lens attached to the television camera.  Use of video tape as
a storage medium allows experiments to be conducted in a laboratory remote
from the site at which the data are analyzed.

     Occasionally video recordings are not "clean" enough to allow fully
automated digitization (e.g., data collected in the field) or the digitized
images are too crude to provide information about details of an organism's
anatomical structure (e.g, the orientation of its eyes).  A technique has
been developed to expedite manual analysis of such data.  Using the PICK
operator, the video tape is examined frame-by-frame.  The user selecTs
                                      8

-------
                              "Bug watcher"
                            Video Preprocessor
    O 00 OOO
     o oo oo
   L  O O O O
Graphics
Console
7
    TV Monitor
Audio
                                   Video Tape Deck
                        Minicomputer
                       ECLIPSE S/200
                        72 KWORDS
   Figure 1.  Block diagram of major components of the "Bugsystem",

-------
              Video
  "Bugwatcher"
Video  Preprocessor
   Organisms
   under study
                                   Video
                                   Audio
                    Video Tape Deck
                                                TV Monitor
Figure 2.   Apparatus used in collecting data in the biological laboratory.
                              10

-------
points upon the screen of a video monitor using a video cursor controlled by
a joystick.  A synthetic video signal representing a tiny bright dot is sent
to the Bugwatcher which, in turn, sends digitized video information to the
computer.  Because averaging algorithms are employed, this method affords
higher spatial resolution than does fully automated input:  each coordinate
in the resultant data structure may have 8-9 significant bits in comparison
to the 7-8 bits of the normally digitized input data.  This is, of course,
orders of magnitude slower than automatic collection of clean images;
consequently, manual specification of points through this process is avoided
whenever possible.

     Input of video data to the computer effects a mapping of the image as
seen upon a television screen into a two dimensional representational
space.  Distances within this space ("Bugspace") are measured in arbitrary
internal units (or "Bugwatcher units") and, therefore, must be scaled.
Using the LIVE INPUT operator, the Bugwatcher processed image of a
centimeter ruler (or any object of known length)—recorded under the same
conditions used in gathering behavioral data—is displayed on the screen of
a CRT terminal and two points on the rule are selected using the terminal
cursors.  Thus, the ratio between the distances separating the points  in
Bugspace and in physical space yields a factor of proportionality between
Bugwatcher units and conventional units of spatial measurement (e.g.,  m,
mm, cm, etc.).
EXPERIMENTAL PARAMETERS AND EXPERIMENTAL  VARIABLES

     Biological interpretation of behavioral  data requires  that the behavior
of organisms be related to experimental conditions prevailing at the time of
observation.  Consequently, we have developed methods to  associate experi-
mental parameters and experimental variables  with sets of data.  An experi-
mental parameter is a quantity characterizing a condition which is constant
throughout any given record but may vary  from record to record  (e.g., tem-
perature, concentration of a pollutant, etc.).  Using the PARAMETER
operator, such numerical constants may be inserted into the set of attri-
butes belonging to each data vector.  Parameters may be deleted, modified or
listed.  They may be used to organize graphical displays  (e.g., average
linear velocity as a function of copper concentration as  illustrated in
Figure 7) or to modify the execution of certain operations.

     An experimental variable is a quantity whose value changes during a
single record.  Times at which simple step changes in stimulus  conditions
(e.g., switching a light on and off) occur may be indicated by  the presence
or absence of tones stored on the audio track of a video  tape.  The Bug-
watcher possesses four tone generators to produce such temporal markers dur-
ing the course of an experiment; it also  possesses external  connections
which allow the simultaneous actuation of laboratory apparatus.  When the
video tape is replayed into the Bugwatcher, these tones are detected.  As
noted above, the second attribute in each data vector sent  to the computer
contains four bits dedicated to representing  the presence or absence of the
                                      11

-------
four tones.

     We are presently developing a technique which provides for the repre-
sentation of stimuli which vary continuously over time.  The  stimulus  level
will be encoded by means of frequency modulation (FM) on the  audio track of
the video tape.  When the tape is replayed into the Bugwatcher, the FM  sig-
nal will be digitized and represented, with 10-bit precision, by the fourth
attribute of each data vector sent to the computer.  The APPEND operator
will be employed to extract this information from each data vector, scale it
and store as an equispaced time series.  Again these data may be used  to
organize graphical displays or they may enter into computations involving
time series of behavioral variables.  One of us (Wilson) is preparing  to
employ this method to investigate behavior evoked by rotation of the plane
of polarized light.  The organisms swim beneath a Polaroid filter whose
angle is controlled by an analog servomechanism.  The filter may be rotated
so as to describe quick jumps, ramps, harmonics, etc.  A signal directly
proportional to the angle of the filter will be encoded and analyzed.


THREE DIMENSIONAL IMAGES

     At the time of this writing (summer of 1978), there have been no  three
dimensional images successfully processed on this system.  There is progress
in this direction and this section will describe the approach being taken to
solve the general problem of tracking fish in X-Y-Z space in a four-foot
diameter tank.

     The experimental system employs two stationary television cameras
mounted approximately 120 cm above a fishtank 120 cm in diameter and 100 cm
deep.  The cameras are positioned to point to the center of the water
surface in the tank and are thus fitted about 25° away from the normal.
It was necessary to reverse the direction of the Y-scan on one of the
cameras so that the image viewed on two monitors would both have the "top"
of the picture be the same general location in the fish tank.  Without  this
reversal, which was accomplished electronically, one picture appeared
"upside down" with respect to the other.

     The goal of this 3-D processing system is to use the same general  Bug-
watcher hardware and software system to analyze motion in three dimensions.
Of course, we foresaw the development of special 3-D operators to generate
and interpret triads (rather than pairs) of data values as well as some
special hardware to enable analog recording and digital input of such  data.
For example, data are recorded using a specially built video multiplexer
called the J-Box (after its creator, Jay Sinnett) which multiplexes—or
alternately switches between—the two cameras to produce a single video
output that can be recorded on a conventional tape deck.  Thus, the maximum
effective frame rate is reduced from 60 to 30 f/sec for stereo images.
Nevertheless, the video data produced in this manner can be entered into the
computer automatically using the existing BUGWATCHER ^NPUT operator.   This
presupposes that the illumination of the fTsh in the tank is  designed  and
                                      12

-------
adjusted to yield high contrast (i.e., "clean") images as required by the
Bugwatcher for automatic input.  Consequently, stereo images are input to
the Bugwatcher as is done with a normal single image video recording.
Likewise centroids can be computed using existing routines and the software
is oblivious to the fact that alternate frames were fish generated from two
different views of the same scene.  These stereo mode centroids are  input to
the MAP3 operator which converts all of the X-Y points into corrected water
surface coordinates, and, using alternate frame and camera position
information, generates three dimensional positions of the fish centroids.
The PAT3 operator connects these centroids into continuous path trajectories
in 3-space.  With the data organized in this way, it is not difficult to
generate time series functions of the path data using operators such as LIN3
for 3-D linear velocity and PIS3 to compute the continuously varying
distance of a fish to its nearest neighbor.

     Improperly functioning hardware and failure to obtain high contrast
video images of the fish swimming against a uniformly illuminated background
proved to be major stumbling blocks in the analysis of 3-D data.
Unfortunately, the grant period ended before any real behavioral data could
be analyzed using this system.  However, as of this writing, the prospects
look reasonably good that the system will be utilized to track organisms
swimming in three dimensions.  But due to a lack of clean stereo images, the
automated input of video data—a technique which has been very successful in
investigations of two dimensional patterns of movement exhibited by
organisms swimming within smaller containers—has been abandoned.  In a
recent attempt to image fish swimming within the tank as distinct dark
objects moving against a uniformly bright field, the fish appeared as dark
gray entities moving across a light gray to dark gray background.
Therefore, the analysis of such stereo images will entail using the  manual
(or semi-manual) data input procedure implemented via the PICK operator:
the video tape must be stepped through manually using the still frame
control on the video tape playback unit until the desired data records have
been collected.  Thus obtained, the analysis of these data will proceed as
outlined above, through application of the MAP3 and PAT3 operators.  It
should be noted that the Bugsystem still greatly facilitates the
quantification of the behavior of organisms moving freely through a  three
dimensional space.  It provides a fairly accurate means of measuring spatial
relationships of motile organisms over time.  Although the data collection
process is much more time consuming as compared to the automatic collection
of two dimensional high contrast data, subsequent investigation can  proceed
with all of the available analytic tools such as interactive graphics,
mathematical manipulations, and statistical analyses developed for such
data.
                                      13

-------
                                 SECTION III

                            PROCESSING AND DISPLAY
IMAGE PROCESSING

     As discussed above, input of video 2-D data to the computer entails
substantial preprocessing of pictorial information.  A data vector within  a
resultant video file is a list of contemporaneous points, with an organism's
outline represented within this data structure as a localized set of
points.  The user can display such data graphically (using the PLOT
operator) or alpha-numerically (using the LIST or EXAMINE operators).   Video
files may be edited both in time and in space.  The EDIT operator allows one
to save or delete temporally contiguous sets of data vectors.  Thus, the
user could EXAMINE the data to ascertain the frame at which the status  of  a
tone had changed (indicating a change in stimulus conditions, e.g.,
switching on a blue light) and then EDIT the data to insure that this change
occurs on frame number 100.  The MASK operator allows one to save or delete
points within rectangular or circular regions of the image plane.  Thus, the
user could MASK out all points within a video file which correspond to  a
particle of detritus within the experimental preparation.  Finally, the user
may APPEND additional information to a video file (e.g. text describing the
conditions of the experiment, numerical constants, time series of tone
states or time series of experimental variables).

     Analysis of video data by means of the Bugsystem proceeds by
abstracting one (or more) points from each point set which delineats an
organism's outline.  In an investigation of translational movement this task
is easily defined.  Unlike either rotational or flectional movement,
quantitative description of the translational component of an individual
organism's behavior does not require detailed knowledge of the organism's
external anatomy.  The body of the organism is represented by a single   ,'^^
point, viz., its "center of mass".  Translational movement is defined as
displacement of this point from one position in space to another.

     Mapping outlines into centrally located points is usually achieved by
means of the CENTROID operator whose command syntax is exemplified by the
entry

                      *CENT BUGS.VI BUGS.CE nl, n2, n3.

Each vector in the resultant file "BUGS.CE" corresponds to a vector in  the
operand file "BUGS.VI".  Each element of a resultant data vector  is a "cen-
                                       14

-------
troid":  a point in Bugspace whose X and Y coordinates are, respectively,
the average X and Y coordinates of an "acceptable set" of points in the
corresponding operand data vector.  The numerical parameters "nl", "n2" and
"n3" are required to characterize an "acceptable set" of operand data
points.  An initial member (having non-zero coordinates) of such a set is
chosen from the operand data vector, the coordinates of this point are set
equal to zero in the operand data and a mask (width = 2-nl, and height =
2'n2) is centered upon this point.  The set is augmented if an operand
point (having non-zero coordinates) falls within the mask.  The mask is then
centered upon the new point and the search continues.  The search terminates
when the set can no longer by augmented; the set is deemed "acceptable" if
it possesses at least n3 members.  Ideally, this process yields one centroid
corresponding to each outline—unless the outlines of two organisms are
merged.  The user can PLOT the centreids over the original data to confirm
this correspondence and recompute the centroids using new parameters if the
correspondence is not adequate.  The structure of "centroid data" is quite
similar to that of video data except that each coordinate of a centroid is
represented with 15-bit (rather than 8-bit) precision in attempt to exploit
the greater accuracy resulting from the averaging procedure.  Centroid data
is displayed and edited in the same fashion as video data.

     The PICK operator provides an alternative method to abstract points of
interest from video data.  The use of this operator for direct analysis of a
video tape, thereby bypassing the acquisition and processing of video files,
is discussed above (see "COUPLING TO RESEARCH ENVIRONMENTS").  The user may
also PICK points associated with each outline by means of the terminal
cursors.  The PICK operator is indispensable for investigations of
rotational and flectional movement.  For example, the user might elect to
study the orientations of the longitudinal body axes of a group of organisms
(i.e., simple rotation in a single plane).  The longitudinal axis may be
defined as a vector extending from the tail to the head of an organism.
Thus, the user picks an ordered pair of points corresponding to each outline
by entering

                           *PICK/VI BUGS.VI BUGS.TH

and using the cursors to specify first the "tail" and then the "head"
associated with each outline.  The resultant file "BUGS.TH" possesses the
structure of centroid data with each "tail point" immediately followed by
the correlated "head point".  These points are then segregated using the
PLUCK operator.  For example, the two commands

                           *PLUC BUGS.TH BUGS.T 1,2
                           *PLUC BUGS.TH BUGS.H 2,2

respectively produces the files "BUGS.T"~containing all first elements of
the ordered pairs (viz., the "tail points")--and "BUGS.H"—containing all
second elements of the ordered pairs (viz., the "head points").  This type
of analysis may be extended to encompass larger collections of points (or
n-tuples), thereby providing for the study of flectional movement (e.g., the
                                      15

-------
angles of propulsive appendages with respect to the longitudinal axis).

     Regardless of the type of movement under investigation, the next stage
of image processing entails the compuation of paths or trajectories through
Bugspace.  A path or trajectory is a time-ordered set of points represented
as a single data vector and characterized as follows:  all points  in a path
are selected from an operand file consisting of centroids or having the
structure of centroid data.  The manner in which points are represented  in a
path is identical to the way they are represented in a frame of centroid
data.  Each path starts within a specific frame, with its starting frame
number represented as the fourth attribute of the vector.  Over the temporal
interval during which a path is defined, one (and only one) point  is
selected from each corresponding vector (or frame) of the operand file.
Adjacent pairs of points within a path are selected on the basis of their
spatial contiguity within adjacent frames.

     The search for paths is performed by the PATH operator whose command
syntax is exemplified by the entry

                   PATH BUGS.CE BUGS.PA nl, n2, n3, n4, n5.

The numerical parameters "nl" through "n5" control various details of the
search,  nl specifies the width (in Bugwatcher units) of a square mask used
as a criterion of spatial contiguity.  n2 is the maximum number of times to
expand the mask if no contiguous point is located within the frame being
searched.  n3 is the minimum number of points to be accepted as a  valid
path.  n4 is the minimum average displacement in Bugspace between
consecutive frames for a set of points to be accepted as a valid path; this
parameter may be used to "weed out" stationary artifacts (e.g., a particle
of detritus).  n5 is the maximum number of frames to "look ahead"; i.e., if
no contiguous point is found in the current frame, then the next frame may
be searched.  If a point within the new frame qualifies, the missing point
is computed by linear interpolation.

     Ideally, every path would correspond to the movement of a single point
(associated with the outline of each individual organism) through Bugspace
and each such movement would be respresented by one path.  In practice,  the
PATH operator may commit two types of errors.  The operator may overlook
certain segments of continuous movement.  Such omissions may yield an abnor-
mally short path, or they may result in a one-to-many (even a one-to-none)
correspondence between real word trajectories and paths.  Alternatively, the
operator may confound certain segments of the continuous movements of two
(or more) organisms.  If there were but one organism in the field  of view at
any time, then errors of omission could be abolished by using a large mask
(why not let it include all of Bugspace?) and allowing the program to look
ahead several frames.  But with several organisms represented within each
frame there is clearly a tradeoff in choosing parameters so as to  reduce the
incidence of the two types of errors.  The user may PLOT the paths over  the
centroids (or over the original video data) to check their fit and then
recompute the paths using new parameters.  Often several iterations of this
                                      16

-------
process are required to obtain an optimum combination of parameters.  Since
the parameters depend largely upon the magnification, the density of
organisms and the way in which they move, the same "optimum set" of para-
meters (retained in memory for the user's convenience) are generally used to
compute paths for "all replicates of an experiment.

     Proper selection of path finding parameters can significantly reduce
the number of erroneous paths, but cannot be relied upon to eliminate all
errors.  Consequently, we have developed programs which allow the user to
interactively detect and correct mistakes within path files.  Path editing
programs (inducing CHOZ, EDIT, PEDI and JOIN) allow the user to perform the
following basic operations!(1) delete a path, (2) truncate a path, (3) cut
a path into two smaller paths and (4) join two paths (assuming they do not
overlap in time).

     Once a valid collection of paths has been obtained, the user may pro-
ceed directly to the extraction of time series of behavioral variables (as
discussed below) from the path files.  Before doing so, however, there are
several additional procedures which the user may choose to apply to the path
data.  Since the use of these procedures (and the order in which they are
applied) is dependent upon the overall design of the experiment, we will
first  illustrate them by means of specific examples.

     Wilson video taped the behavior of Daphnia pulex (a small freshwater
crustacean, commonly known as a "water flea") in Talbot Waterman's  labora-
tory at Yale University.  Twelve animals were observed from below swimming
against a brightly and uniformly illuminated background.  A variable
polarizer/depolarizer was interposed between the chamber containing the
animals and the light source.  The tape consisted of 23 separate
experimental records.  Before each recording the polarizing filter was
rotated to a randomly chosen angle and, also in random sequence, the device
was adjusted so as to polarize or depolarize the illumination.  Each record
began with the image of a strip of plastic attached to the filter to
indicate the angle of the filter and the magnification of the  image.  The
plastic strip was then removed and the animals were observed swimming under
constant conditions.  In all, 13 records were obtained under polarized light
and ten records were obtained under unpolarized light.  Each recording
lasted two minutes.
                              i
     The video tape was analyzed with the aid of the Bugsystem at
Narragansett, R.I.  For each record the LIVE INPUT operator was used to
determine the angle of the filter with respect to the Bugsystem reference
frame  (which is fixed with respect to the raster scan of the video  signal).
As discussed above, the appropriate spatial scale factor (approximately 0.40
mm/Bugwatcher unit) was also computed.  The J5UGWATCHER JMNPUT operator was
then used to generate one video file (480 frames of data at five frames/sec
or 96 sec of data) for each record on the tape.  Wilson then MASKed the
data, saving only those points within a centered circular region of Bug-
space.  The centroids and then the paths were separately computed for each
masked video file using the same set of centroid and pathfinding para-
                                       17

-------
meters.  Very few pathfinding errors resulted from these computations, but
these were corrected by editing the separate path files.  Wilson then
employed the PARAMETER operator to associate the angle of the filter with
respect to the Bugsystem reference frame (an experimental parameter) with
each path.  This angle was the same for every path within a given path
file.  By repeated use of the MERGE operator, Wilson condensed the data to
produce two exceedingly large Tiles:  one containing all paths observed
under polarized light (154 paths, 13210 data points) and another containing
all paths observed under unpolarized light (166 paths, 13691 data points).
These paths were still represented in Cartesian coordinates relative to the
Bugsystem reference frame.  However, orientation with respect to the plane
of polarization should only be manifest with respect to the reference frame
of the filter.  Using the ROTATE operator, the two frames of reference were
made to coincide:  every path within each merged file was rotated in Bug-
space (about an axis passing through the center of the space)through an
angle obtained by negating the appropriate experimental parameter for each
path.  All resultant paths for Daphm'a swimming under both polarized and
unpolarized light are PLOTed in Figure 3.  As shown in Figure 4, many of
these paths are aligned approximately orthogonal to the axis of the filter
(i.e., perpendicular to the ^-vector) in the presence of polarized light.

     Drs. William Lang and Don Miller, Sally Lawrence, Martha Marcy and Anne
Ackenhausen have employed the Bugsystem to conduct preliminary studies on
the effects of copper exposure upon swimming behavior of barnacle larvae
(viz., second stage nauplii of Balanus improvisus).  The nauplii were
exposed to six different levels of copper for 24 hours prior to ,
observation:  3.2, 25, 60, 115, 147 and 205 ppb.  No significant mortality
resulted from such exposure.  They viewed the larvae with the aid of a
disecting microscope using 830 nm substage illumination and obtained video
tapes of larval behavior in "darkness" (i.e., stage illumination only) and
when stimulated with 488 nm light (0.06 W/MZ).  Three replicates were
performed for each experimental condition (36 observations in all).  These
data were digitized at 10 frames per second.  Photic stimulation was
indicated as the Bugwatcher detected a tone denoting the onset of 480 nm
illumination.  Video files for stimulated nauplii were EDITed to start 0.50
seconds after the light was turned on and to last 2.00 seconds.  For all
files centroids and paths were computed, then three path files corresponding
to each experimental condition were MERGEd to form a single file.
Unfortunately, the computed trajectories of larvae exposed to high levels of
copper (150 ppb) were quite ragged and irregular due to the low linear
velocities of the larvae.  Consequently, all path data were SMOOTHed using a
linear low-pass filter:  the SMOOTH operator computes moving average of X
and Y components with weights as specified by the Tukey window (2).
Smoothed paths of barnacle nauplii are shown in Figure 5.

     The preceeding analyses illustrate the use of operators to organize
path data prior to the computation of behavioral variables.  The ROTATE
operator was implemented to expedite studies of animal orientation.  Using
this operator, all spatial data in a file may be rotated through a constant
angle, each path may be rotated through a constant angle associated with
                                      18

-------
         UNPOLARIZED
              LIGHT
         POLARIZED
             LIGHT
 ~ 50
 o
 u
 c
 B
 •+•
 •> 25

 O
            	1	»-
              25         50
            Distance  (mm)
75  o
	1	H
  25         50
D I stance  (mm)
Figure 3.  Polarotactic orientation of Daphnia pulex as illustrated by merged and rotated paths in
        unpolarized (166 paths) and polarized (154 paths) light.  The sequence of operations
        required to generate these data are discussed in the text.

-------
       UNPOLARIZED
             LIGHT
180
                                          180
POLARIZED
    LIGHT
                  270
                                                             270
 Figure 4.  Polarotactic orientation of Daphnia pulex as illustrated by polar wedge histograms of
          instantaneous direction of travel in unpolarized  (13691 data points) and polarized (13210
          data points) light.  Percent of sample is plotted on the radius and the dashed circle
          denotes the expectation for a circular uniform distribution.  The e-vector of the plane
          polarized light is horizontal (viz., 0 and 180 degrees) in the graph on the right.

-------
DARK
                                                     LIGHT
   oc
   UJ
   Q.
   0.
   o
   o

   CD
   Q.
   Q.

   in
   CM
   a:
   ui
   a.
   Q.
   O
   u

   CO
   0.
   OL

   IX)
Figure 5.  Merged path files of Balanus improvisus  nauplii  (barnacle  larvae)
           in darkness (column on left) and stimulated  by 480  nm
           illumination (column on right)  at two  different  levels  of  copper
           exposure (top row,  25 ppb Cu; bottom row,  115  ppb Cu).   Control
           records are approximately 35 sec in  duration.  Experimental
           records begin 0.5 sec after  stimulus onset and last 2.0 sec.
           Photic stimulation  was aligned  horizontally, originating from the
           right side of the above plot.
                                      21

-------
that path or each point may be rotated through an angle associated with a
corresponding moment in time.  The last option enables the investigator to
study orientation with respect to a moving stimulus.  Files may be MERGED
whenever they are taken to be replicates of the same experiment.  Not only
does this simplify the bookkeeping tasks associated with subseqent analysis,
but, in addition, allows a set of similar data to be treated as a single
sample by statistical operators.  The fundamental advantage of an
operator-based interactive system for the analysis of behavioral data is its
flexibility:  the operators which the user chooses to apply to the
data—and—the order in which they are applied can be selected to correspond
to the design of the original experiment.


GENERATING AND TRANSFORMING TIME SERIES DATA

     Time series of X and Y coordinate values are generated from path data
by means of the SPLIT operator, whose command syntax is illustrated by the
entry

                         *SPLI BUGS.PA BUGS.X BUGS.Y.

Each data vector in the resultant files "BUGS.X" and "BUGS.Y" is, respect-
ively, a time series of X and Y coordinate values.  The resultant data sets
are in one-to-one correspondence and each element is represented as a single
precision floating point number.  Like the paths from which they are
derived, resultant time series may start and end at arbitrary points in
time.  Therefore, the number of series defined at any given moment is also
arbitrary.  Other computer systems have been developed for the anlysis of
equispaced time series (8).  After image processing has been completed, the
operators available on the Bugsystem are unique only insofar they possess
the sophistication required to manage large collections of arbitrarily
derived series (e.g. Figure 6).

     In principle, all behavioral variables which may be investigated using
the Bugsystem can be generated by simple arithmetic transformation of series
of X and Y coordinates.  Greaves (5,6) has discussed the computation of cer-
tain behavioral variables (viz., linear velocity, net to gross displacement
ratio, direction of travel and angular velocity) using arithematic operators
implemented within the prototype system.  Such operators transform every
element of every data vector within a file, treating each vector as a
separate unit of data.  The present Bugsystem is also provided with a wide
selection of simple arithmetical operators.  However, we have condensed the
computation of certain frequently calculated behavioral variables into sin-
gle operators which require path data as input.  As an example, let us con-
sider the RATE OF CHANGE OF DIRECTION operator whose command syntax is
illustrated by the entry

                          *RCDI BUGS.PA BUGS.RD nl.
                                      22

-------
N>
                       LINEAR  VELOCITY   VS.   TIME
                  4.0
                                        Time  (sec)
     Figure 6.
Linear velocity as a function of time for §. improvisus nauplii exposed to 25 ppb copper in
the absence of photic stimulation.  Sample consists of 37 paths (2369 data points); the
dashed line denotes the estimate of the mean (1.20 + 0.01 mm/sec).

-------
Every element in the resultant file "BUGS.RD" is the (unsigned) rate at
which the corresponding path changed its direction of travel at a given
moment.  RCDI is the absolute value of angular velocity expressed in degrees
per second (The parameter "nl" is the frame rate, requried to convert from
degrees per frame.)  The importance of this variable in ascertaining mechan-
isms responsible for certain animal aggregations has been widely discussed
(4).  RCDI may also be computed by a sequence of simple arithmetical
operations as follows:

           (a)  *SPLIT BUGS. PA BUGS.X BUGS.Y
           (b)  *CW BUGS.X BUGS.DX
           (c)  *C1)TF BUGS.Y. BUGS.DY
           (d)  *TTTA¥ BUGS.DX BUGS.DY BUGS.DI
           (e)  *ZrpTF BUGS.DI BUGS. AC
           (f)  *MT/CO BUGS. AC BUGS.AV nl
                *M5Y BUGS.AV BUGS.RD
Where "CDIF" if the CENTRAL DIFFERENCE operator (a discrete approximation to
the differential operator), "CTAN" is the CONTINUOUS ARCTANGENT operator,
"MULT/CO"denotes multiplicatiorTBy a constant and "AB5VirTs the ABSOLUTE
VATUE operator.  The resultant "BUGS.DI" of step (cfpTs the direcTTon of
travel measured in degrees with respect to the Bugsystem reference frame; it
could have been generated from the original path data using the DIRECTION OF
TRAVEL operator.  The resultant "BUGS.AV" of step (f) contains angular
velocities (measured in degrees per second); this file could have been pro-
duced using the ANGULAR VELOCITY operator.  Other operators have been
developed to evaluate ILIREHR VELOCITY and j^ET TO GROSS £ISPLACEMENT RATIO
functions defined upon path data.


SIMPLE STATISTICAL PROCESSING

     For the purpose of statistical analysis two different types of data
structure—representing two levels in a structural hierarchy— may be distin-
guished as "samples":  vectors and files.  Many statistical operators recog-
nize this distinction.  For example the STATISTICAL PARAMETER operator esti-
mates parameters such as the mean, variance, standard deviation, skewness,
kurtosis, etc.

The command

           STAT/VE BUGS.LV BUGS.ST

produces the resultant file "BUGS.ST," containing one data vector (i.e., a
list of statistical parameters) for each data vector in the operand, whereas
the command

           *STAT/FI BUGS.LV BUGS.ST

produces only one resultant data vector characterizing the entire file.  In
                                      24

-------
either case, the user can LIST the resultant parameters.  Similary, the SLOT
operator provides for estimation of density and distribution functions vTa
histograms both for individual vectors and entire files.  These data may be
displayed graphically (Figures 4, 8 and 9) or LISTed on the terminal or the
line printer.

     We chose to store statistical parameters within data vectors, rather
than merely computing and displaying them, in order to allow them to enter
into subsequent calculations.  For example, assume the user had computed
statistical parameters for each data vector in a file of instantaneous
linear velocities (e.g., the "STAT/VE" example given above); each estimate
of the mean is thereby an average for each path.  The user may then investi-
gate the distribution of these path averages.  The estimated means are first
isoloated using the STRIP operator:

           *STRI/MN BUGS.ST BUGS.MN.

The resultant file "BUGS.MN" contains a single data vector; each element of
this vector is a mean ("/MM") stripped from one operand data vector.  The
user may wish to compute statistical parameters for the new set of data, or
explore its frequency distribution via histograms.  The user may also MERGE
such files so that each data vector corresponds to a single experimental
condition.  The vectors may then be compared with one another (e.g., one-way
analysis of variance with the ANOV operator, chi-square tests with the CHI2
operator, etc) or graphically displayed as shown in Figure 7.

     We have also made special provision within the Bugsystem for the
manipulation and statistical analysis of angular data.  For example, the
command

           *STAT/AN/FI BUGS.DI BUGS.ST

produces a single data vector whose elements are statistical parameters
appropriate for circular distributions.  These include the length and
direction of the mean vector (and related measures) as discussed by
Batschelet (1).  Moreover, because the Bugsystem is well suited for investi-
gations of animal orientation we have implemented an extensive polar
graphics package within the Bugsystem.  Figures 4 and 9 illustrate the use
of polar wedge histograms to represent angular density functions.


MORE ADVANCED OPERATIONS

     As is illustrated in Figure 6, a typical behavioral record consists of
multiple time series which begin and end at arbitrary points in time.  We
have implemented operations to discern time-varying trends within such
collections of data.  One approach we have taken is to develop so-called
"running" statistical operators:  operators which provide estimates of
statistical quantities for all points within vectors defined at each frame
or instant of time.  For example, the RUNNING AVERAGE computes an ensemble
                                      25

-------
               LINEAR  VELOCITY
                          VS.
            COPPER   CONCENTRATION
        2.00
                    I	1	1	1
Figure 7.
        o.oo
                   so
               75   100  125  150  175  200  225
           Copper Concentration
                   (ppb)

Average linear velocity of B. improvisus nauplii for each sample
(+_ standard deviation) plotted as a function of level of copper
exposure in the absence of photic stimulation.  The number of
instantaneous linear velocity values for each copper
concentration ranged between 2055 and 3560. Copper exposure
time = 24 hr.
                            26

-------
          25  PPB   COPPER
205   PPB  COPPER

25-
0
J- 20-
«>
«- 15-
o
+i
c
0 10-
o
V
°~ S
ft








•
^^•a








^MB










-l_


































—







••p






_t_


• »•









• IS
»••$









• •
• •









2369
1 .20 ••/••

*




-TT~r-

• •
> •
• •




























f-i

















— i 	 1 i i

««*pl« «ii« • 3560
•v«r*g« • 0.45 ••/«•






-I",
      0.0   0.5    1.0   1.5   2.0    2.5   3.00.0   0.5    1.0   1.5   2.0   2.5   3-0
      Instantaneous  Linear Velocity   Instantaneous  Linear  Velocity
                   (mm/sec)                           (mm/sec)
Figure 8.  Distributions of Instantaneous linear velocity of B. improvistis nauplii at two different
         levels of 24 hr. copper exposure:  25 ppb (left) and 205 ppb (right);  no photic stimulus
         present.  Vertical dashed lines denote estimates of the mean of each sample.

-------
                  25  PPB  COPPER
          180
N>
oo
115  PPB  COPPER

            90
                                                      180
                                                                          270
     Figure 9.   Fhotactic orientation oJSB. \improvisas nauplii elicited by 480 nm illumination at two
                different levels of 24 hrT copper efcpfcsure:   25 ppb (sample size = 854)  and 115 ppb
                (sample size = 598).  The angle denotes instantaneous direction of travel with respect
                to the light source; percent qf sample is represented on the radius.  The stimulation
                is directed from the right of each polar graph (viz., 0.0 degrees).  The dashed circles
                denote the expectation for circular uniform distributions.

-------
average for all simultaneously defined operand values, storing the resultant
values within a single data vector.  Similar routines provide for estimates
of ensemble standard error of the mean and direction of ensemble mean
vector.  We have also developed a RUNNING SLOT operator which computes a
separate discrete" density function for eacFTrame of data thereby providing
separate histograms for each instant in within a behavioral record.

     Often an investigator may wish to explore the temporal structure of
behavioral data in greater detail.  We have taken several approaches to
enable such analyses.  Certain simple behaviors may be classified as being
merely present or absent and may be denoted by time series of ones and
zeros.  Several operators (e.g., the LOCAL MAXIMUM, LOCAL MINIMUM, DELTA,
INTERVAL DELTA and SPIKE operators) convert real-valued time series~TnTo
Finary valued series.  The TIME INTERVAL operator computes the number of
frames separating contiguous non-zero elements of such series, thereby
allowing the user to determine the temporal intervals which separate dis-
crete events.  These time intervals may then be analyzed statistically,
resulting in estimates of statistical parameters (using the STATISTICAL
PARAMETER operator) and interval histograms (using the SLOT operator).

     The analysis of discrete time intervals is a relatively simple
procedure inspired by the widespread use of interval histograms among
electrophysiologists studying trains of spikes (viz., nerve impulses).  We
have implemented more complex operators which provide for analysis in the
time domain and produce resultant data vectors whose elements are serial
correlation (AUTO CORRELATION and CROSS-CORRELATION) coefficients (2).  Such
operators yield quantitative measures of statistical independence/dependence
among neighboring data values within series.  They are also helpful  in
detecting rhythmic or periodic behavior.  In addition, we have planned for,
but not yet implemented, operators which enable analysis in the frequency
domain using Fourier methods (2).

     Finally, many behavioral studies entail the investigation of relation-
ships between two or more variables.  Figure 7 illustrates the relationship
between a simple scalar statistical quantity (viz., sample average linear
velocity, a dependent variable) and a real parameter (viz., level of copper
exposure, an independent variable).  It is possible to study histogrammed
distributions in a similar way; however, this requires three-dimensional
graphics capability.  We have implemented an interactive graphics package
(viz., the BRICK operator) which produces fully labeled rectalinear
three-dimensional graphs on the plotter.  Figures 10 and 11 illustrate the
display of separately computed histograms (for direction of travel and rate
of change of direction respectively) as functions of copper exposure in J3.
improvisus.  This technique can also be used to study the relationship
between two behavioral (i.e., inter-dependent) variables such as direction
of travel and rate of change of direction (see Figure 12).
                                      29

-------
               DEPENDENCE  OF  ORIENTATION
               UPON  COPPER  CONCENTRATION
Figure 10.  Dependence of phototactic orientation in B. improvisus nauplii
          upon level of copper concentration:  the Tarvae exhibit positive
          phototaxis at low concentrations of copper (<100 ppb) and
          negative phototaxis at intermediate levels (100 ppb).  No
          orientation is manifest at high levels of copper exposure C>150
          ppb).  Sample sizes range between 521 and 854 data points for
          each copper concentration.  Copper exposure time = 24 hr.
                                 30

-------
                       DEPENDENCE  OF
            RATE  OF  CHANGE  OF  DIRECTION
              UPON  COPPER  CONCENTRATION
Figure 11. Dependence of rate of change of direction of IB. improvisus
         nauplii upon level of copper exposure.  The larvae tend to turn
         less with increasing concentrations of copper.  Sample sizes
         range between 2055 and 3560 data points. No photic stimulation
         was present. Copper exposure time = 24 hr.
                               31

-------
to
                      UNPOLARIZED
                           LIGHT
                              «/
                                         v <  *
                                   so!x <»»'
0.
3
O
L
O
_
c
o
^
o
L.
O
A


70-

60
SO

40

30
1C


~T<

4-
i

^

->
^
                                                        0
                                                       180
   POLARIZED
      LIGHT
^..r r>
                                                                '9J  '»„
     Figure 12.  Dependence of rate of change of direction upon direction of travel in  Daphnia pulex in
               unpolarized and polarized liqht.  The animals in this  experiment orient orthogonal to the
               ^-vector of polarized light (as shown in Figure 4). When they deviate from this preferred
               orientation thay also turn rapidly, often executing abrupt U-turns.

-------
                                  SECTION  IV

                                  DISCUSSION
ORIGINAL PROJECT OBJECTIVES
     At the onset of the project, the project goals were  to  develop  a
responsive system that would execute on  a minicomputer, be as portable  as
possible, be built from available components using standard  television
equipment and have one specially built piece of  hardware, the Bugwatcher.
Also, the system was to be usuable by people with little  or  no computer
background.  A discussion of how these goals were met follows.

System responsiveness

     When the user of the system sits down to input or  analyze data, he or
she can collect data at a phenomenal rate through the BUGWATCHER  INPUT
routine.  This alone represents the greatest time savings, with tFe  ability
to play back say two minutes of video data and have the Bugwatcher and
computer collect and store thousands of  data points as  the tape is being
played back.  The software system then responds  with a  "*" indicating it is
ready for the next command.  The delay between keyboard operations on the
system is usually less than one half second and  rarely more  than  two
seconds, which indicates that the system is indeed responsive.

     The time required to input and process data is estimated by  Wilson as
follows.  To input and process one hour  of video tape, which might typically
contain two experimental conditons with  twelve replicates each, would take
about 15 hours of time on the computer.  This analysis  includes two  sets of
experimental conditions (e.g. control and another experimental condition)
each having perhaps 10,000 data points of some 200 path trajectories.  The
processing would include rotating the data points to be aligned with an
angular parameter, calculating the respective directions  of  travel,  linear
velocities, rates of change of direction, calculating the statistical
parameters for the above, listing them on the line printer,  generating
statistical distributions and plotting these results on the  plotter.

Portability

     Another design goal was to develop  a system that could  be exported to
other potential users.  This objective has been  met largely  through  the
choice of FORTRAN as an implementation language.  Although there  are often
"special" features in the FORTRAN language that  can be found on one  machine
but not another, we have resisted temptations to employ them when possible.


                                      33

-------
The Bugsystem software fully occupies the maximum 32 K-Word memory partition
under the RDOS operating system and has several overlays that are swapped  in
and out at execution time.  Such overlay calls are compiler dependent, but
are also fairly well isolated in the source files.  In addition, there are
some features in "the software that utilize the 16 bit architecture of the
minicomputer; e.g. integers require a single word of storage and real
numbers require two such words.  The software system was first developed on
a POP11 under the DOS-8 operating system and was made to execute on the same
machine under the DOS-9 and, later, the RT11 operating systems.  Due to
repeated PDP11 hardware failures during the last six months of the project,
we stopped supporting the POP11 software system and put our full effort into
the Data General ECLIPSE S/200 under RDOS, now Revision 6.2.  To support
this software on another PDP11 operating system would take three to four
months of programming effort, depending upon programmer experience and
familiarity with the Bugsystem software structures.  Programming needs might
be equally divided into three areas:  (1)  implementing the driver to
communicate with the DMA channel and the Bugwatcher, (2) formulating the
appropriate program overlay structures for the new system, and (3) smoothing
out differences between the FORTRAN language processors.

Standard Parts

     The image collection and storage medium has been standard one-half inch
video tape.  The RTMA standard 525-line interlaced raster scan cameras have
been employed instead of the lower cost random scan cameras, which are not
as good in terms of resolution and stability.  The best and cleanest image
for computer input is recovered from a video deck if the playback unit is
manufactured by the same company as the initial video recorder.

     The graphics terminal is another standard component.  The Tektronix
4010 and 4012 have been used.  We did add a special feature to the 4010
terminal on both the ECLIPSE and the PDP11, which we name the "Fastpoint"
option.  This modification was added to speed up the point displays of video
data (which might be 100 pages each with 100 points).  The Fastpoint option
uses a digital to analog converter on the computer to store a point directly
on the face of the storage display screen bypassing the communications
procedures.  For point displays, this increased the speed of plotting a
video data set about an order of magnitude—from about two minutes down to
about twelve seconds.


CONCLUSION

     In addition to the specific project goals addressed above, a more
general objective has guided our development of the Bugsystem:  to
facilitate highly automated, yet flexible, frame-by-frame analysis of
bio-behavioral data.  Moreover, we wished to engineer a comprehensive
system; i.e., data aquisition, image processing, pathway analysis,
statistical analysis and, ultimately, polished tabular and graphical
presentation of behavioral data by means of a single system, implemented as
an interactive computer language (BRL).  We feel that our development


                                      34

-------
efforts have been exceedingly successful in meeting these broader
objectives.  By means of the Bugsystem, the user is able to collect and
analyze behavioral records consisting of tens of thousands of data points,
tracking the movement of multiple organisms through space and time.  The
rich detail inherent to such behavioral data provides direct measures of
motility; viz., quantitative representations of trans!ational, rotational
and flectional movement.  Accordingly, we believe that the importance of the
Bugsystem is that it makes possible programs of behavioral research which
were formerly not physically realizable for "mere" practical reasons.  Thus,
the modern digital computer promises to offer one route of escape from an
old dilemma of behavioral research:  the tradeoff between qualitative
investigation of biologically significant behavior and quantitative study of
activities of dubious biological importance (e.g., interruption of light
beams or tripping microswitches).  We have little doubt that systems similar
to the Bugsystem will become indispensible tools in future investigations of
animal behavior.

     Development of the Bugsystem at the Environmental Research Laboratory
has provided EPA investigators with the technical means to easily record  and
rapidly quantify the motile behavior of aquatic organisms.  Studies are now
in progress to determine swimming patterns (both spontaneous and elicited by
controlled stimulation) of various marine invertebrate larvae and
zooplankton.  Modification of behavior  in the presence of pollutants is also
being compared with results obtained by more traditional assay methods
(e.g., LC-50).  Significant changes in  swimming behavior are manifest when
they are subjected to sub-lethal levels of stress  (Lang, personal
communication).  Currently, research is being conducted with the Bugsystem
to evaluate the utility of behavioral bioassays to assess water quality and
to screen for toxic substances.
                                       35

-------
                                  SECTION V

                                  REFERENCES
Batschelet, E.  1965.  Statistical Methods for the Analysis of Problems
     in Animal Orientation and Certain Biological Rhythms.  AIBS Monograph.
     Washington, DC.  57 pp.

Chatfield, C.  1975.  The Analysis of Time Series:  Theory and
     Practice.  Chapman and Hall, London.  263 pp.

Davenport, D., 6.J. Culler, J.O.B. Greaves, R.B. Forward and W.6. Hand.
     1970.  The Investigation of the Behavior of Microorganisms by
     Computerized Television.  IEEE Trans.  BME 17:  230-237.

Fraenkel, 6.S. and D.L. Gunn.  1961.  The Orientation of Animals.
     Dover.  New York.  367 pp.

Greaves, J.O.B.  1971.  An On-line Television Computer System for the
     Study of the Behavior of Microorganisms.  PH.D.  Dissertation.  Dept.
     Elec. Eng., Univ. of California, Santa Barbara.

Greaves, J.O.B.  1975.  The Bugsystem:  The Software Structure for the
     Reduction of Quantized Video Data of Moving Organisms.  IEEE Proc. 63:
     1415-1425.                                                         ~

Hand, W.G. and J.A. Schmidt.  1975.  Phototactic Orientation by the
     Marine Dinoflagellate Gyrodinium dorsum Kofoid.  II.  Flagellar
     Activity and Over-all Response Mechanism, J. Protozool. 22:  494-498.

Martin, W. and K. Brinkman.  1976.  A Computer Program System for the
     Analysis of Equispaced Time Series.  J. Interdiscipl. Cycle Res. 7:
     251-258.

Wilson, R.S.  1977.  Light Elicited Behavior of the Marine
     Dinoflagellate, Ceratium dens.  Ph.D. Dissertation.  Dept. Biol. Sci.,
     Univ. of California, Santa Barbara.

Wilson, R.S. and J.O.B. Greaves.  1979.  The Evolution of the Bugsystem:
     Recent Progress in the Analysis of Bio-behavioral Data.  In:
     Proceedings of the symposium on the State of Marine Environmental
     Research, F. Jacoff, ed.  In Press.
                                      36

-------
                                   TECHNICAL REPORT DATA
                            (Please read Instructions on the reverse before completing)
1. REPORT NO.
  EPA-600/3-80-010
             3. RECIPIENT'S ACCESSION-NO.
4. TITLE AND SUBTITLE
     Development  of an interactive system  to  study
 sub-lethal effects of pollutants on the behavior of
 organisms
             5. REPORT DATE
              January 1980 issuing date
             6. PERFORMING ORGANIZATION CODE
7. AUTHOR(S)
 John 0. B. Greaves and Robert S. Wilson
             8. PERFORMING ORGANIZATION REPORT NO.
9. PERFORMING ORGANIZATION NAME AND ADDRESS
 Department of  Electrical Engineering & Dept. of  Biology
 Southeastern Massachusetts University
 North Dartmouth,  Massachusetts  02747
             10. PROGRAM ELEMENT NO.

               1BA819
             11. CONTRACT/GRANT NO.

               R803244
12. SPONSORING AGENCY NAME AND ADDRESS
 Environmental  Research Laboratory - Narragansett,  RI
 Office of Research and Development
 U.S. Environmental Protection Agency
 Narragansett,  Rhode Island  02882
             13. TYPE OF REPORT AND PERIOD COVERED
               Final    	
             14. SPONSORING AGENCY CODE
               EPA/600/05
15. SUPPLEMENTARY NOTES
is. ABSTRACT Experimental investigation of the movements of  organisms often entails the
acquisition and processing of  large samples of spatio-temporal data.   An interactive,
interpretive, on-line computer television system (viz., the  Bugsystem) was initially
developed at the University  of California at Santa Barbara in order to expedite such
analyses.   Aspects of the structure of this prototype system are outlined.  Its
effectiveness is evaluated with regard to the problems confronting the bio-behavioral
researcher.
     A second generation system has been developed under  a research grant from EPA.
Utilizing new hardware and software, it in many ways constitutes a generalization of
its prototype.  Features of  the refined system are described which provide for the
following:   a large degree of  machine-independence, significant expansion of the size
of data records, inclusion of  experimental parameters and variables within the data
structure,  investigation of  rotational and flectional movement,  statistical analysis,
advanced graphics and tracking of  organisms in three dimentional space.  Current
utilization of the Bugsystem for future development are discussed.
17.
                                KEY WORDS AND DOCUMENT ANALYSIS
                  DESCRIPTORS
                                              b.lDENTIFIERS/OPEN ENDED TERMS
                          c.  COSATI Field/Group
 Animal behavior 0510  063
 Television systems 1702
 Data Processing 0902
 Toxicology 0620
 Motion       2011
 Pollution   1302
 Closed circuit televisioi
 Animal locomotion
57H
18
                TEMENT
19. SECURITY CLASS (ThisReport)
UNCLASSIFIED
                                                                         21. NO. OF PAGES
                                                                                 45
                                              20. SECURITY CLASS (Thispage)
                                              UNCLASSIFIED
                                                                         22. PRICE
EPA Form 2220-1 (9-73)
                                            37
                                                                    « U.S. mVBMHEKT MUMS Of«£ «• -4S7-146/5S36

-------