AWBERG LIBRARY U.S. EPA
         DEVELOPING A SYSTEM FOR EVALUATING
       THE EFFECTIVENESS OF INSPECTOR TRAINING

       PHASE 1: IDENTIFICATION OF EVALUATION TECHNIQUES
AND DEVELOPMENT OF MODEL FOR TRAINING PROGRAM OPERATIONS
                            Prepared for:
                United States Environmental Protection Agency
                       Office of Air and Radiation
                   Stationary Source Compliance Division
                          Washington, DC
                            Prepared by:
                          Kathleen M. Reed
                       School of the Environment
                       Fuqua School of Business
                           Duke University

                           September 1991

-------

-------
                           DISCLAIMER

This report was  furnished to the U.S. Environmental Protection
Agency by  the student identified  on the cover page,  under a National
Network for Environmental Management Studies  fellowship.

The  contents are essentially as  received  from  the  author.  The
opinions, findings,  and conclusions  expressed  are  those  of the author
and  not necessarily those of  the  U.S. Environmental  Protection
Agency.  Mention,  if any, of company, process, or product names is
not to be considered as an endorsement by the U.S.  Environmental
Protection  Agency.

-------

-------
                                    ACKNOWLEDGEMENTS


This study was performed by Kathleen Reed under the National Network for Environmental Management
Studies program.  The project was listed as #91-1103, with Mark Siegler, Stationary Source Compliance
Division, Environmental Protection Agency, acting as Project Sponsor.  I would also like to thank Kirk Foster,
Stationary Source Compliance Division, Environmental Protection  Agency, for  providing a  historical
background on the Air Inspector Training Program activities, as well as  Martha Strobel,  Air Quality
Management Division, Environmental Protection Agency, and Laurel Sneed, Apprend Associates for their
assistance with providing a better understanding of the mechanics of training.

-------

-------
                       TABLE OF CONTENTS
EXECUTIVE SUMMARY	-	
INTRODUCTION		—•
      PURPOSE
      BACKGROUND
METHODS OF CONTINUALLY EVALUATING TRAINING EFFORTS'
EFFECTIVENESS	»»-
      DURING THE SESSION OR COURSE...
      AT THE END OF A SESSION OR COURSE...
      SOME TIME AFTER THE COURSE OR PROGRAM-
      INDEPENDENT OF THE TRAINING PROCESS...
A MODEL FOR A TRAINING GROUPS OPERATIONAL
CYCLE	
      MISSION STATEMENT
      IDENTIFY GENERAL TRAINING GOALS AND NEEDS
      DETERMINE SPECIFIC OBJECTIVES
      DETERMINE TRAINING SEQUENCE
      PREPARE MEDIA
      CONDUCT PILOT STUDY
      COORDINATE ALL MEDIA AND FACILITIES TO PREPARE
        FOR TRAINING DELIVERY
      TRAINING DELIVERY
      EVALUATION
POSSIBLE EFFECTS OF TRAINING ON INSPECTOR QUALITY	
RECOMMENDATIONS	
EXHIBITS	•	
      EVALUATION BY TRAINEES AT END OF COURSE
      EVALUATION BY TRAINERS AT END OF COURSE
      DELAYED EVALUATION BY TRAINEES
      DELAYED EVALUATION BY TRAINEES' SUPERVISORS
        OR BY INSPECTORS SERVING AS ON-THE-JOB
        TRAINERS
      A MODEL FOR A TRAINING GROUP'S OPERATIONAL
        CYCLE
      MISSION STATEMENT LEADING TO SPECIFIC COURSE
        OBJECTIVES
      CRITICAL INCIDENT SURVEY
      A BEHAVIORALLY ANCHORED RATING SCALE FOR
        BEHAVIORS USED IN OBTAINING A SOURCE'S
        COMPLIANCE DOCUMENTATION
      A QUICK METHOD OF EVALUATING THE AIR STATIONARY
        SOURCE COMPLIANCE INSPECTORS' TRAINING
        PROGRAM
BIBLIOGRAPHY	
ABSTRACT	...»
          .4
          .5
5
5
9
9
10
11
          13
13
14
15
17
19
20

20
21
21
          22
         .24
         .28
28
30
32
34

35

36
37


38


39
         ..41
         ...45

-------

-------
                                    EXECUTIVE SUMMARY

This document begins by focusing upon methods of continuously evaluating training program effectiveness.
It is important to recognize that training developers have only recently begun to use evaluations to target
training efforts, and have not yet found any generic techniques which are applicable in all cases.  Each
situation must be assessed individually in order to determine the best evaluation techniques to employ.
Evaluation  methodologies which directly investigate the educational  process can be  used during,
immediately after, or some  time after conducting the training  effort.  Methodologies which indirectly
investigate training's  effectiveness and appropriateness are also explored.

After investigating evaluation techniques, a model of an operational structure for developing, delivering, and
evaluating training efforts is discussed. This structure must actively involve interactions with environmental
influences external to the training group in order to proactively identify and eventually fulfill prioritized training
needs.  The described operation flows from the mission statement through training delivery, with feedback
from later steps, including evaluation, being used to revise and better target previous activities. This model
emphasizes using proactive techniques and adjusting the training continuously in order to best apply limited
resources.

Next, possible effects of effective training on air stationary source compliance inspections are examined.
Quite simply, the training is likely to result in better compliance in sources inspected by trained inspectors
as the sources recognize that the improved enforcement of regulations. Although training provides the skills
and knowledge necessary for job completion, these skills may not be employed  if factors within the
organization discourage their use. An accurate model including all possible influences on inspection quality
must be developed in order to determine exactly what role training plays  in the system.

Finally, several actions are recommended and suggested for further investigation or implementation.

  RECOMMENDED METHODS OF CONTINUALLY EVALUATING TRAINING PROGRAM EFFECTIVENESS
         •  implement additional quantitative methods of continually evaluating training effectiveness
         •  conduct a preliminary evaluation

  RECOMMENDATIONS FROM THE MODEL OF A  TRAINING FUNCTION
         •  document specific objectives for educational  efforts
         •  promptly meet demand for all types of courses

  RECOMMENDED OVERALL IDEAS
         •  incorporate feedback from newly implemented evaluation techniques when designing training.
         •  document specific, quantitative objectives of  each training session, course, and program.
         •  operate in the long run.

 One should remember that inspectors facilitate effective environmental problem resolution. When all factors
 influencing job performance function correctly, well trained inspectors are very likely to help the EPA to meet
 its goals.  Training seems to represent one  of the best methods of  ensuring that inspectors are able to
 complete their duties.  The training cycle should result in the highest quality, most efficient, most effective
 training possible. Once the  inspectors are given the tools necessary to complete their duties, they will be
 likely to do a better job of surveying for violations.  That leads to better enforcement when necessary. The
 potential violators recognize that they are better off complying with  the requirements than trying to hide
 excess pollution rates.  A greater compliance rate with  environmental requirements leads to a cleaner
 environment, directly furthering the EPA's goals.

-------

-------
                                       INTRODUCTION
PURPOSE

This document explores NNEMS (National Network for Environmental Management Studies) project #91-
1103, which is stated below:

        How can air stationary source compliance inspections be improved through effective
        training and how can these improvements be measured?

This topic was modified slightly during the course of the project.  The primary focus became describing
methods of evaluating educational effectiveness, while the secondary focus involved creating and describing
a model of an operational structure for a training group in order to best deliver effective training. Possible
effects of training on inspection quality are also discussed.  One should note that while the information
presented in this document is tailored to the air stationary source compliance inspector training program,
it contains information that could be relevant to other programs as well.

The project was conducted primarily through searches of available literature and interviews with training
experts. Although the project's  scope was limited, it effectively served as a screening mechanism, focusing
upon broad aspects of effective training design and thus highlighting areas which may warrant further study
or action.   Later projects will investigate and/or implement program modifications recommended by the
initial  project.

The project's primary sponsor was the Stationary Source Compliance Division, and the secondary sponsor
was the Air Pollution Training Institute. Both sponsors are part of the Office of Air and Radiation, of the
United States Environmental Protection Agency. This topic involves one of the training programs mandated
by EPA Order 3500.1.
BACKGROUND

The Environmental  Protection Agency was created to protect and  preserve the American and world
environment from excessive changes caused by pollution, increases in technology, and the general effects
of increasing  populations.  The EPA  completes this mission through various mechanisms.  One key
mechanism involves the following problem resolution cycle:

  -> discovery of problem
     -> creation of law by Congress to minimize or resolve a problem
        -> targeting and refinement of law by the Environmental Protection Agency through
           regulations
           -> surveillance monitoring  conducted to check for compliance with the laws and
               regulations
               -> enforcement of the requirements
                 -> recognition by regulated parties of the penalties of noncompliance
                    -> compliance of regulated parties with requirements
                       -> positive changes
                        ->  goals of problem minimization or problem resolution achieved

-------
 One should note that inspectors play a key role in this process. The inspectors investigate whether or not
 the regulated parties are in compliance situations.  This monitoring greatly enhances the requirements'
 enforceability.  Although continuous emission monitoring systems, new analytical tools, record keeping
 requirements, and reporting requirements are being implemented, personal inspection still provides the
 backbone of compliance determination. Once the new techniques are fully implemented, inspections will
 probably still be used to verify the devices' accuracy and the facilities' compliance.

 The inspectors are important for other reasons as well.  For one, competent, credible inspectors seem to
 serve as an enforcement deterrent. The "Virginia Report" describes how the EPA Air program investigated
 the influence of training on improved compliance reporting.  The findings indicate that various sources'
 compliance statuses improved after the well trained inspectors found more compliance violations.1  The
 inspector is important because the  inspection data provides the basis for actions and cases against
 noncomplying sources.  Quite simply, the better the data, the better the case, and thus, the greater the
 likelihood that an enforcement action will  occur.  If the data is weak (for  example, if it is incomplete or
 inaccurate), the case will be weak, and the enforcement probably will not occur. Without enforcement, the
 incentives for affected parties to comply with the requirements decrease substantially. While the inspectors'
 roles have been substantial in the past, the inspectors' responsibilities may soon increase to also include
 more enforcement, when the permitting process that the Clean Air Act Amendments  of 1990 provided is
 activated.

 Not only do the inspectors play a major role in facilitating the EPA's goals, they also represent a significant
 annual investment for control agencies.  Stationary source compliance monitoring represented 32% of state
 agencies' reported annual compliance expenditures, and 37% of local agencies' reported annual compliance
 expenditures.

 When one considers both the critical role that inspectors play in the process of implementing programs  and
 also the enormous annual investments made in the inspectors, one recognizes that great care must be given
 to guaranteeing that the inspectors fill their role to the peak of their abilities. A deficiency in skills  and
 knowledge requisite to completing duties may pose the most basic potential problem affecting inspector
 performance. Any inability of inspectors to  conduct their duties would  impair the problem resolution cycle.
 If the inspectors are unable to survey, then enforcement is not likely to happen, compliance is even less
 likely, and problem resolution may not occur. Training provides one mechanism for rectifying a lack of skills
 necessary to perform the duties.  If the agency is able to increase the inspectors' ability to complete their
job requirements, it will most likely improve  the quality of inspections,  thereby facilitating the program
 implementation cycle and increasing the likelihood of meeting the EPA's goals.
   1  Pedco.

   2  Radian Corporation, E-5.

-------
                METHODS OF CONTINUALLY EVALUATING TRAINING EFFORTS-
                                       EFFECTIVENESS
Evaluations are frequently used in order to determine the training effort's effectiveness and targeting.  This
is because training groups and departments need to verify that their efforts effectively teach the appropriate
skills, knowledge, values, and behaviors.  As one training expert states,

               "If you are happily training hundreds of employees yet have no idea whether they
       perform to the standards they've been taught- or even if they use the skills youVe  given
       them- you are failing in your mandate. Remember, the purpose of training is to bring about
       change.  If you dont measure the change that has taken place, you cant know if you are
       performing the task."3

Quite simply, an evaluation's purpose is to determine whether the program is meeting its goals or not. The
goals merely provide a basis for comparison.  For example, if one were asked whether or not slide  rules
were quick calculating devices, the answer would depend on what slide rules were being  compared to.
They are fast when compared to working the problem with pencil and paper, but are slow when compared
to using a calculator. In the same manner, the interpretation of the training evaluation's results depends on
what the goals were. If  the training is meeting its goals, then all is well.  However, if it is not, then some
changes need to take place.  Feedback must occur and necessary changes must be made in order for the
evaluation to have any purpose.  This helps to ensure that scarce resources are  allocated to areas where
they will have the greatest impacts.

Training courses and their required amounts of targeting can be compared to shotguns and rifles, as very
similar targeting processes are used both in training and in firing a gun.  Shotguns require some focusing;
however, the shot disperses over a relatively large area.  This dispersion permits one to hit the target even
if the aim is slightly  inaccurate. The likelihood that at  least some of the shot will  hit the target is high, but
there is also waste, as  many of the pellets will miss the bull's eye.  On the other hand, one must be
extremely precise when aiming a rifle, as  the single bullet does not forgive mistakes. If the aim is incorrect,
there is extremely high waste, and there  is no chance of hitting the target; however, as long as the rifle is
aimed correctly, the bullet efficiently hits the target. Just as these two firearms require different amounts
of aiming because  their shot disperses differently, various types of training require different evaluative
procedures to ensure that their dispersed or targeted educational efforts fulfill their purposes.  Some training
efforts are similar to a shotgun, in that the training efforts are assembled quickly and cover a wide range of
topics in the hope that at least a few  of these widely dispersed topics will teach what the trainees need  to
know. These do not require  many mechanisms to verify that they teach the appropriate materials.  On the
other hand, one can develop extensively planned, efficient, precisely targeted training efforts which have
been designed to complete only very specific objectives. This type of an effort is analogous to a rifle; the
tightly packaged, well focused training materials are highly efficient but leave little room for inaccuracy.

Evaluations can be used for several purposes.  In all  cases, they help to employ limited resources in the
most effective manner possible. The evaluations are commonly grouped as either summative or formative.
Summative evaluations are used to determine whether or not to continue the program. These have been
used extensively  in  the past.  Elizabeth Hawthorne, a training expert, reviewed all of the training program
evaluations published between 1966 and 1984 in  order to provide broad models  for later  evaluations  to
follow. She found that most of the evaluations were used to determine whether to continue or terminate the
      Mitchell, p. 141.

-------
 training program in question."  The second group of evaluations, the formative evaluations, are somewhat
 different.  They are used to identify areas of a current program which may require modification. Hawthorne
 also states that:

        Few of the evaluations conducted and reported in the literature were formative evaluations,
        that is evaluations that are intended to modify and improve the program in process.5

 While the training industry recognizes the need for formative evaluations, there are no proven methods of
 conducting those evaluations.  Each evaluator must determine an appropriate method from ground zero.

 Formative and summative evaluations may also  be better explained by continuing the gun  analogy.
 Summative evaluations involve a process similar to that of determining whether or not to stop shooting the
 gun. On the other hand, formative evaluations can be compared to the process of aiming a gun, evaluating
 whether or not the firing was accurate, and determining the adjustments necessary to increase the  next
 round's accuracy.

 A logical path for any evaluation to take involves comparing  actual results to ideal  results. The training
 effort's ideal results should be determined in order to provide a basis for comparison.  Perhaps the simplest
 and most comprehensive source for these projections is the statement of specific objectives for a given
 course, session, or program.   If a model which accurately forecasts the programs effects exists, its  results
 could also be used as the basis.  Next, the actual results of the training effort should  be obtained. Simply
 matching the actual to the ideal results can then provide an evaluation of the program's effectiveness.

 It is important to recognize that all training sessions, courses, and programs  should be evaluated.  This
 includes short run courses even though the evaluation may be completed after the  training group stops
 offering the effort. It is important for the course designers to see the effectiveness of previous courses so
 that they can either avoid similar mistakes in the future or continue to use very effective techniques.  The
 need for evaluations of training efforts with long  life spans is more obvious.  A lack  of evaluations could
 result in very costly mistakes.
 Although a variety of techniques exist for obtaining information necessary when conducting evaluations,
 evaluators often use surveys.  These surveys can take the form of either questionnaires or interviews.  The
 interviews may be conducted either in person or over the telephone, and can be structured or unstructured.
 Structured interviews contain  a set, unchanging series of questions.6   Unstructured interviews  are  also
 based on a list of questions; however, the unstructured interviews maintain the flexibility to investigate areas
 of interest as they unfold.7  A less common method of evaluating a course involves observing actual course
 sessions.  As noted, this is rarely used, but allows evaluators to obtain an outside opinion of a course.  A
third well known technique is to use focus groups.  This consensus building technique often serves either
to explore an unknown or to measure a past occurrence. Focus groups consist of a panel of either subject
 experts or a representative cross section of a group targeted by the issue in question. The panel's diverse
 opinions are debated  until a consensus is reached.

 Evaluations for sessions, courses, and programs can be  employed at several times.. Some evaluations may
 be conducted during the course session, both to slightly modify the course effectiveness and also  to make
   4 Hawthorne, p.26.

   5 ibid.

   6 Philips, p. 81.

   7 ibid.

-------
long term changes to the course. Others are more appropriate at the courses' or sessions' conclusion, and
affect future presentations. The evaluations conducted during and immediately following the session, course,
or program can be thought of as short- run evaluations in that they measure only immediate features of the
training effort. These features include the abilities both of the training to communicate and of the trainees
to assimilate the educational objectives. In contrast, other evaluations investigate long- run issues. One long
run issue could be the effects that the training effort has on job performance. Evaluations conducted several
months after the course or program was presented can investigate long-run issues. The delay facilitates an
investigation of how the training  effort actually affected trainees' job performances. A fourth and final group
of evaluation techniques investigates factors other than  the actual training.  We will now investigate these
four groups of evaluations.


DURING THE SESSION OR COURSE...

There are several uses for these methods of evaluation. Their primary purpose is to ensure that a course
already in session is  achieving its goals.  By monitoring the course as it progresses, trainers can tailor the
course to the trainees'  needs. These techniques can also  identify necessary modifications to the course's
basic format; repeated revisions of the same problems in the course may indicate that the course's outline
should be changed.  A trained observer may be helpful in objectively employing  these techniques.

These  in-session  evaluations have several benefits and weaknesses.  One benefit is that they are very
inexpensive to implement, requiring only the trainer's willingness to observe the session, modify the course
if necessary, and  report any findings back to the training development group. However, these evaluations
also have  a number of  drawbacks due to their superficial nature.  They cannot show the long  term
effectiveness of the course; only its ability to communicate the material while in session.  They can fail to
recognize areas that require attention.  Finally,  they only look at material currently in the courses and
therefore cannot determine whether the topics are appropriate or if other topics should also be addressed.

One method of  evaluating  a course in session involves  monitoring questions to determine trainees'
comprehension of course materials.8 The trainer should monitor trainer initiated  and also trainee initiated
questions. If trainees neither answer nor ask questions, some  probing may be  necessary to determine the
cause  of their lack of involvement- be it that the material is too basic, too advanced, too dull, or any other
reasons.

In class exercises present an excellent method of determining whether or not trainees are assimilating the
material.9  Practice simulations, including cases, discussions, and problems, can show whether or not the
trainees have fully understood the material, and highlight which areas require more emphasis.


AT THE END OF  THE SESSION OR COURSE...

Evaluations conducted at this time allow participants  to reflect back on the entire session and present an
overall opinion.  One  should note that  participants may  forget information about the early parts of the
training effort, especially if it is long. These evaluations  can only measure the perceived applicability of the
educational materials to future job requirements, and can investigate only the perceived appropriateness and
completeness of the topics included in the course. Examinations, trainee evaluations, trainer evaluations,
and summary reports are frequently used  methods.
    8 Mitchell, p. 126.

    9 Mitchell, p. 128.

-------
 Examinations generally come as either a set of pre- and post- tests, or a single test given at the end of the
 exam. The post-test method provides a picture of the trainees' abilities at the end of the course.  This
 enables one to show overall competencies, but not to determine the exact effects that the course had.  A
 better method involves the pre- and post- set, which measures not only overall competencies but also gains
 from  the course.  This method facilitates comparing abilities  both before and after  the course; this
 comparison can be used to show the precise effects that the course had.10  Quite simply, the course is
 effective if trainees learn necessary material during the course.

 Examinations can also be  used to determine both course or session and media effectiveness.  Raw data
 from each session's or course's tests should  be gathered and assimilated according to question number.
 Poor wording of questions or a failure to achieve the objectives could cause  large numbers of trainees to
 have difficulty with certain questions.  The data can also differentiate between problems caused by the
 instructor's inability to communicate certain information or by the class design.  If multiple types of media
 are used to teach the same information, then a comparison of results between media can determine if the
 difficulty is caused by the media. For example, if trainees under only one trainer miss a given question, and
 trainees instructed  by other trainers do not have trouble with that question, then the instructor is probably
 causing the problem. If all  of the trainees have difficulty regardless of who the trainer was, then it indicates
 a problem with the course or session design. One should be very careful to use proper statistical evaluation
 techniques when looking at test data, as oftentimes casual observations can  lead to statistically incorrect
 conclusions.

 In  addition to final course examinations, trainee evaluations  can  also determine apparent training
 effectiveness as  well as training acceptability.  These evaluations generally investigate areas such as
 satisfaction with the course, the facilities, and the course materials.  Different areas of the course can be
 rated on a set scale. Opinions regarding the best and worst parts of the course are often gathered, as are
 suggestions for improving the course. An example of this type of an evaluation can be found in Exhibit 1.

 Trainer evaluations may investigate the course's design, teaching materials, course content, and other areas.
 They can also be used to gather results from in-class evaluations, such as the questions and class exercises
 described earlier. An example of a trainer evaluation is located in Exhibit 2.

 Finally, summary reports can evaluate sessions, courses, or programs.  The Air program currently employs
 this method.  Once the  training effort is completed, the trainees fill  out evaluations. The trainer sorts,
 reviews, and summarizes these evaluations. The trainer then evaluates the training effort and documents
 any recommended  modifications to the course.  The summary report  includes the trainee examinations,
 trainee evaluations, the summary of the trainee evaluations, the trainer evaluation, and the recommendations.
SOME TIME AFTER THE COURSE OR PROGRAM...

These evaluations enable one to investigate long term results and issues of training efforts.  The evaluations
are often conducted between 6 and 18 months after taking a course. This method can investigate items
such  as effects on job performance, whether or not general topics were useful, or whether other topics
should be added to the training effort. These evaluations' strength resides in their ability to determine actual
and perceived impacts of the training on job performance.

One of these methods involves surveying the trainees. Two identical surveys can be conducted; one a few
months after the training effort was completed, the other can be conducted  1-2 years after completing the
effort.   Areas to investigate include perceived usefulness of the course,  perceived  impact on job
   10
      Philips, p. 102.
                                               10

-------
performance, subjects that were most helpful to the job, as well as areas that should have been covered,
such as new technologies and techniques.  These can also be used to determine whether the job has
changed since the course was originally designed.  If so, the course or program should be modified to
reflect the changing work environment.  An example of this survey can be found in Exhibit 3.

Another evaluation technique entails distributing questionnaires to trainees who have returned for additional
courses.11 The questionnaires could investigate any previous courses that the inspectors may have taken.
This form may be similar to the previous survey, with the  only difference being that it could request
information about a number of courses. Although surveying returning trainees provides valuable information,
this method should not be the sole evaluation method. This is because it surveys only the returning trainees
(those trainees satisfied enough with the previous course to return for additional instruction) rather than the
entire set of trainees.

A third evaluation technique involves surveying either the trainees' first line supervisors, or the inspectors
serving as on-the-job trainers, or both. Again, two identical surveys could be conducted at different intervals.
They could investigate the perceived usefulness of the course, the perceived  changes in the inspectors'
performances, and also the changing requirements for the inspectors' jobs. Exhibit 4 contains an example.
A list of course objectives or other basis for comparison could be sent with the survey if specific skills and
abilities are being investigated. In addition, the evaluation to the inspector/on-the-job trainers could ask
them to describe any areas that consistently present a problem for the trainees.


INDEPENDENT OF THE TRAINING PROCESS...

The following areas can be continuously observed in order to monitor the training program's actual and
perceived effectiveness. One should note that these evaluation techniques can be used regardless of when
the training occurred.  Many of these are qualitative rather than quantitative. Although this section describes
how to  use these methods for Air Stationary Source Inspector training programs, their general ideas can
be applied elsewhere.

To begin, evaluators can collect information about inspectors. This information should reflect overall course
goals (possibly key course objectives) and may include absenteeism, numbers and types of mistakes made,
inspector turnover, and  information from the exit interviews.  Mistakes could be analyzed to determine (1)
whether they were caused by failure to meet a course objective, and (2) whether they should be included
in the objectives to prevent others from having the same difficulties. Evaluators may wish to determine why
the inspectors leave the EPA, as stated in exit interviews. These results could be analyzed according to the
reasons for leaving and the extent to which the inspector was trained.

Information  about the  actual inspections can also be  used to evaluate training  effectiveness.  This
information can be obtained either from EPA records or from the inspected sites.  There is some reason to
believe that personnel at the sites would be responsive to requests for this information, as  it parallels the
trend in the business world to become ever more customer driven. While the act of requesting information
from inspected  sources contains some immediate risk that the feedback  may be  negative,  requesting
information from the sources actually decreases total  risk. By learning of and correcting problems in the
present, problems can be minimized and resources can be used more effectively in the long run.   Even if
the evaluation used only EPA records, it would still carry some risk. By only using EPA documentation, the
risk would  involve  obtaining incomplete information and  therefore failing to respond  appropriately.
Inspection information that can be monitored includes, but is not limited to, the list in Figure 1.
       Mitchell, p. 142.
                                                11

-------
        Figure 1
                  : EXAMPLES OF INSPECTION INFORMATION VWICH COULD BE
                                 INVESTIGATED CONTINUOUSLY
              the inspector's  Image (whether or not the inspector appeared competent  and
              professional)            -'                                     ,,
              the disruption that the inspection caused to the source's operations
              numbers of repeat Inspections required to collect additional information
              information thai required repeat visits to fully obtain
              the numbers of contested versus uncontested Inspection conclusions
              ff lawsuits arise, the adequacy of the inspectors' evidence to prove noncomptiance
This information should be grouped according to whether or not the inspectors had taken the training, and
if so, whether or not they passed the course.

A third method could be to monitor informal anecdotal feedback on the courses and programs.  This
includes feedback from regional, state, and local contacts, as well as comments made at meetings, such
as national meetings.  Informal comments from trainee supervisors and on-the-job trainers should also be
noted. Complaints, compliments, and suggestions should be noted.  Although this is helpful, it is neither
comprehensive nor complete and therefore should be employed only in conjunction with other evaluation
methods.

Forma^feedback should be followed if at all possible.  Formal letters of complaint or compliment should be
noted.    These may be sent by training recipients, from inspected units, or from public interest groups.
Complaints should result in either further evaluation or immediate changes, depending upon the complaints
severity and validity.

The demand for the courses should be monitored. Total increases or decreases in both the numbers of
potential trainees and in demand should be noted.  Demand should be compared to the estimated or actual
figures for competitive training courses and programs. The worst scenario would  be for the total demand
for the training program  to decrease  substantially while  the number of perspective trainees and  the
competition's  demand  skyrocketed.

Finally, periodic surveys in media such as newspapers or bulletins that training customers would receive can
be used to determine perceptions of the training courses and programs.
   12
     Mitchell, p. 108.
                                             12

-------
                  A MODEL FOR A TRAINING GROUPS OPERATIONAL CYCLE
This section will examine one method of organizing the operation of developing and delivering training.
Exhibit 5, which contains a model of a training group's operational cycle, illustrates the ideas that will be
discussed in rest of this section.  The  box in that Exhibit represents the border between the training
department and the rest of the world. Everything within the box represents the training operations; all that
occurs and exists outside of the training group is shown outside of the box. The arrows show information
flowing in the direction that the arrows point.  It is important to notice that information flows throughout the
process, not just from the top down. Each major portion of the process will be described separately.


MISSION STATEMENT

The mission statement concisely states the training group's purpose and guiding principles.  This mission
must help to satisfy the missions and goals of the organizational structure above the group.  Because the
group is part of an entire organization, factors external to the group also influence the groups resources and
future.  In order to ensure that the group  survives and thrives, it must respond to  the needs  of the
organization and must help to fulfill the missions of the organizational  structure above it.  In the EPA, the
air stationary source compliance inspector training development group's actual  superior structure includes
the Stationary Source Compliance Division, the Office of Air Quality Standards and Planning, the Office of
Air and Radiation, and, of course, the parent Environmental Protection Agency.

Figure 2 shows a sample mission statement.  This Figure is actually part of Exhibit 6, which illustrates how
the mission influences the training department's activities through identifying the specific objectives of the
training effort.


        Figure 2	   	__________^__

                           MISSION STATEMENT SHOWN IN  EXHIBIT 6

       To satisfy changing air educational requirements In a timely, effective, and  efficient manner
       in order to further the Environmental Protection Agency's goals. This includes:

       1) continuously searching for new training requirements,
       2) operating the process as efficiently as possible.
       3) complying with legislation, regulations, and EPA Orders that mandate training and
               associated standards.
       4) training EPA, Regional, and State personnel to determine compliance with legislation and
               regulations.


 The mission statement should  be sufficiently global so that it can guide the training  group throughout
 changing conditions; however, if conditions change so drastically that it no longer serves to appropriately
 guide the group, the statement should be changed.
                                                13

-------
 IDENTIFY GENERAL TRAINING GOALS AND NEEDS
 This involves determining the general need for training, be it a session, course, or program, and also
 preliminarily determining its subject matter. Figure 3 shows several possible course needs identified from
 the Mission  Statement. The specific course content should not be determined at this point.  Instead,
 decisions should be made on items such as subject matter, whether  it  should provide background
 knowledge or hands-on skills, and whether it will be used as training or development.

 The training group could determine the need for a new course or program from a number of sources. The
 mission statement provides the most basic source of information.  For example, Section 4 of the Mission
 Statement found in  Figure 3 states that the group will train inspectors to determine compliance with new
 legislation and regulations.  Note that one of the educational needs identified in this example involves
 preparing personnel to issue permits as described in the Clean Air Act Amendments of 1990.

        Figure 3	


                                    MISSION STATEMENT                     \

      To satisfy changing air educational requirements In a timely, effective, and efficient
      manner in order to further the Environmental Protection Agency s goals. This includes:

      1) continuously searching for new training requirements.
      2) operating  the process as efficiently as possible.
      3) complying with legislation, regulations, and EPA Orders that mandate training and
              associated standards.                                            !
      4) training EPA, Regional, and State personnel to investigate and manage source
              compliance with legislation and regulations.


                           POSSIBLE TRAINING GOALS AND NEEDS

      • training for evaluating air sources'compliance with the Clean Air Act Amendments of
              1990  and the resultant regulations.                                •
      • materials to help prepare individuals to become inspectors'first line supervisors.
      • training to  enable personnel to Issue permits according to the Clean Air Act
             Amendments of 1990.                        -
Courses will also need to be developed to comply with mandates including legislation, regulations, or EPA
orders, all of which may specify training requirements. These mandates currently drive the training cycle.
One undesirable operational method of determining these stated needs involves watching for applicable
mandates to be officially published.  This eleventh hour, reactionary method prevents long term planning
and  job scheduling.  This results in a fast-or-famine mode of operation with  only limited flexibility in
responding to educational needs. A better method of operation would involve scanning the environment
for clues that any requirements relevant to the training department's  operations would soon be emerging.
This  desirable long run operational mode would enable the training group to plan and schedule jobs ahead
of time and thus to avoid backups and to best utilize slack periods.
                                             14

-------
Courses may also  be developed to fulfill superiors' direct requests.  These requests  may occur if
management believes that a problem could best be resolved through training. The training group should
respond to these requests as quickly as possible.

Although congress currently drives the system, changes in inspection technology, manufacturing techniques,
emission control equipment, or other similar indicators should be used to investigate the needs for course
development or modification.  The training group could learn about these technological breakthroughs from
technical trade journals, from contacts at universities, contacts in industry, or from other EPA  personnel
(inspectors, perhaps). The group should also watch for scientific discoveries, such as determinations that
compounds which had been thought to  be harmless actually are hazardous.  Such a discovery would
indicate that the industry will eventually be required to reduce that compound's emissions and that the air
stationary source inspectors will be required to check for that compound.  Both new technologies and
scientific discoveries could also  indicate that new types  of stationary sources would soon be  emerging,
requiring new methods of inspection.  Keeping the courses current maximizes their value to the trainees,
provides the most impact for their resources, and helps to justify the training group's  existence.

Specific requests may inspire course modification or creation to meet set needs.  One method of handling
these requests would involve processing them as they emerge. As discussed previously, a long run vision
provides better operational results than does a short run viewpoint. The long term vision requires that the
training group nurture ongoing relationships with the offices  which may request training.  Staff from the
training department should regularly contact customers (in the air stationary source program, this includes
regional, state, and  local offices) to discover possible future needs.  Once these have been identified, the
training group may create a preliminary plan showing  how courses meet the requirements.  This plan then
becomes a marketing tool. The training group may use the plan to convince the potential client that the it
the only one that can meet the clients needs. Besides customers, those affected by training results (ie- the
sources inspected by stationary source compliance inspectors) may request that inspectors be  trained on
specific technologies. These requests would only occur if the inspections began to cause severe problems,
and would most likely occur in the form of registered  complaints to the EPA.

Finally, feedback from training evaluations may either verify that the appropriate needs were addressed or
may indicate that other needs exist. Quite simply, trainees may require additional skills. Also, some needs
may no longer be valid as conditions change.


DETERMINE SPECIFIC OBJECTIVES

Once the training effort's  general goals have been determined, the specific objectives necessary to fulfill
those goals must be determined.  Figure 4 shows examples of specific course objectives which could have
been identified according  to a possible course need.  In  this instance, a partial list of objectives has been
created to train  personnel to issue permits to stationary sources.

These objectives have many purposes. One is to shape the training; identifying course objectives facilitates
the selection of appropriate training media such as videos or documentation. Trainee examinations should
investigate whether or not trainees have met the objectives. These will be discussed in more detail in the
section describing media preparation. If available funding would prohibit the training group's ability to meet
stated needs, the group could request increased funds.   The objectives also can be used as inspectors'
performance standards if the training is for specific jobs.
                                                15

-------
         Figure 4
                                 ONE POSSIBLE TRAINING NEED
         training to enable personnel to Issue permits according to the Clean Air Act Amendments "
               of 1990.             '   ,            ,„",„>       ,*\                   '•   '   '
                  POSSIBLE SPECIFIC OBJECTIVES OP A TRAINING COURSE1   '
                                           ..     %  f f&ff^fff S f  "*                   1 ""

       » to be able to determine whether sources emit 10 tons £er year or more of a hazardous air
              pollutant.               -—  ^    ' ™"     "_  "  , ,/"          -I-
       • to be able to determine whether sources emit 25 tons per year or more of a combination
              of hazardous pollutants.
       * to know where to obtain a current fist of sources required to obtain permits.
       • to know how to determine the current tonnage fee to be collected,
       • to be able to complete 100% of the processing procedures for permit application.
       • to be able to obtain a historical record of modifications that would revise previous
              legislation, regulations, or standards.                                 ;
       * etc.               .,        ,,,,,/,,'  ,_*$*"?,   "/,",              I   -
        •                                            . t,.,_  ^  "f, ,„              , ^

       1 based on Pytte, p. 3955.                            '''   ''""~              \
The objectives can also be used to modify existing courses.  If the objectives serve this purpose, they can
then guide the changes necessary to improve the course.  It is important to note that while investigating
specific course objectives, the course designers may discover previously unstated goals and needs for the
course.  These should be verified, and either included in the same course or used as needs for a new
course.

Although the discussion has been theoretically sound to this point, practical applications have not yet been
discussed. Three practical methods of determining course objectives are  described below.

One method of determining specific course objectives  would involve analyzing  a regulation or law's
requirements.1   Take, for example, the Clean Air Act  Amendments of 1990.  The training group  in
coordination with subject experts or program managers could analyze the  law and resulting regulations  to
determine exactly what skills, abilities, and knowledge would be necessary  to inspect for regulated parties'
compliance. The program managers often are both knowledgeable about these issues and in control  of
funds,  some of which  could be used to facilitate training development.  Again, if the funding prevents
effective training, requests for additional resources would be justified. The objectives for a course or a series
of courses would then consist of necessary competencies.

Utilizing critical incident reports is another method of determining course objectives.14  This method  is
typically used for jobs that require employee discretion, such as that of an  inspector.  The critical incident
technique,  described  in  Figure  5, determines  both  productive,  desirable  behaviors as  well  as
counterproductive, undesirable behaviors which may prevent employees from fulfilling job requirements.
   13 Fletcher.

   14 Zemke. Figuring Things Out, p. 134-8.
                                              16

-------
              Figure 5                                                         	

                                THE CRITICAL INCIDENT TECHNIQUE""

           1)     Information about the job is gathered on critical incident forms, such as the
                 example shown in Exhibit 7. These forms generally investigate three areas:
                  (1)    identification of the reporter (name, project title, office, etc.) in case
                        additional information must be obtained.
                  (2)    a description of an incident, including the situation, the objective, the
                        technique used, and the positive or negative results.
                  (3)    evaluator commentary regarding  the  usefulness of the behavior,
                        alternative behaviors which could have been used, etc.
           2)     The critical incident reports are gathered and a Behavlorally Anchored Rating
                 Scale (BARS) Is  created.  This is completed by first  grouping reports
                 containing similar incidents.  The behaviors are then arranged on  a scale,
                 from most productive to feast beneficial.  An example of a BARS can be seen
                 in Exhibit 8.
           3)     Additional information required to fill in the gaps can then be obtained.
           4)     The final results should be verified for thoroughness and accuracy by experts.
Focus groups of job experts can facilitate this process.15  These experts could include highly competent,
experienced inspectors, supervisors, knowledgeable individuals who  were once inspectors, or other
individuals. The focus group could generate the initial group of critical incident reports, sort the reports
between beneficial and detrimental behaviors, and complete any missing areas.

The critical incident technique can also be used to determine whether or not the program needs revision.
If the program has been running for a period long enough that the job requirements may have changed, the
critical incident technique can highlight areas that should be included in future courses.

A  third method for determining course objectives involves  basing  them on job  descriptions.  The
Environmental Protection Agency's CERCLA program currently uses this method.16  A panel  analyzes GS
job descriptions to determine what the specific training requirements are. The panel first reviews the current
GS descriptions to determine what the different activities are and to estimate the percentage of time that is
spent on each activity.  Surveying actual inspectors verifies the panel's findings and determines whether
other activities should be  included in the job descriptions.  The results are used to update the GS job
descriptions.  The course objectives then involve preparing inspectors to complete those identified activities.


DETERMINE TRAINING SEQUENCE

The training sequence should respect both logical order and resource limitations. The objectives' priorities
should be determined, they should be arranged according to their priorities, and then as many of the high
priority items as possible should  be included in the  sequence as resources permit.
    15 Zemke. Figuring Things Out, p. 134.

    16 Thomas.
                                               17

-------
 The objectives and their requisite tasks and behavior may be prioritized in a number of ways. One basis
 involves  determining which objectives form conceptual bases for later,  more advanced requirements.
 Another method assigns priorities according to pre determined penalties for noncompliance.17 This could
 be appropriate if the course teaches skills necessary for specific types of inspections, Finally, objectives
 could be prioritized according to relative importance to the job.

 The following describes one way to prioritize items according to job importance or compliance penalties.
 A questionnaire may be sent to knowledgeable parties to determine the item's frequency of occurrence and
 its absolute importance, either on a scale of 0 (never used) to 1  (critical) as shown in Figure 6 or in dollar
 amounts.  A chart similar to the one below could then be created.

        Figure 6
- - RELATIVE IMPORTANCE
ITEM
A
B
C
FREQUENCY
1.00
.20
.30
IMPORTANCE (FREQUENCY X IMPORTANCES
1.00 % ,
1.00
.70
, 1.00
,20
>21
 An illustration may help to better explain this method.  Please refer back to Figure 6.  Imagine that this
 technique was being  used to determine the relative importance of different behaviors for inspectors.
 Questionnaires could be sent to highly competent, experienced inspectors and other inspection experts.
 Let's say that item A represented the behaviors required to gain entry to a facility.  Because the fictional
 questionnaires indicated that this behaviors occurs for every inspection, the frequency is 1. Also, because
 it is critical to completing the inspection, its importance is also 1. Now, say that B represents the behaviors
 necessary to  successfully obtain  a warrant.  If the questionnaires indicated that  these behaviors are
 necessary only 20% of the time, the assigned frequency is  .2.  Now imagine that when required, these
 warrants are absolutely critical to completing the inspection.  Based on this information, their importance
 is assigned a  1.  Finally, imagine that behavior C represents the behaviors necessary to  learn about new
 technologies while on the job.  This time the questionnaires indicate that the behavior is used in 30% of the
 inspections (thus a frequency of .3) and that while important, it is not critical to completing the inspection.
 Item  Cs  importance is therefore  assigned  as .7.   Each frequency is multiplied  by the corresponding
 importance to obtain the relative importance.  In this instance, behavior A would be the highest priority,
 behavior C the second priority, and behavior B the lowest priority.

 One should note that the training sequence may occasionally require modification. The pilot study's results
 may indicate that there is a much more effective sequence of presenting the materials. Also, priorities may
 change with time.  Therefore, the evaluations' feedback  concerning  the training  sequence should  be
 reviewed on a regular basis.

 In addition to the relative priorities  of different items, the  training developers should determine what limits
there are on the resources. These  could include time, finances, or any others.  If the resources are limited
to such an extent that there is only  a questionable ability to create the necessary training, then requests to
increase the resources would be appropriate. The discrepancy between the need for education and the
inability to provide training could be used to justify these requests. The final course would include as many
of the highest  priority items as resources permit.
   17
      Fletcher.
                                               18

-------
PREPARE MEDIA

Once objectives have been determined, the media for communication must be prepared.  One should
remember that each type of media is appropriate in different circumstances and for different purposes. Each
objective should be analyzed to determine the most appropriate media for teaching that item. For example,
swimming classes are held both in a swimming pool and on dry land. Strokes and techniques are taught
in the water, where  students can  learn through experience.  In contrast, issues of safety tend to be
presented as lectures. It is better to hear about the dangers of running on deck or diving into shallow water
than to learn through experience!  The objectives of swimming lessons are taught using training media
appropriate for each particular objective. Unfortunately, traditional training tends to present even the most
practical information as theoretical documentation. This poses the same problem as would attempting to
teach people to swim by discussing swimming theory. The trainees may know the theory, but not how to
apply it. Regardless of the type of media used, the course developers must make certain that the presented
information is consistent within the  course and with other courses presented by the training department.

Examinations should be developed at this time. They should be used to measure the trainees'  achievement
of course objectives.  If exams are based on anything other than the objectives, then there  is no way to
determine whether or not the objectives have been met.  Because training in particular is used to enable
trainees to better conduct their current job requirements, training exams should be very real world oriented.
Quite simply, if the objectives are met in the classroom, then they are more likely to be achieved in the real
world. If the objectives are to provide skills for use on the job, then the best examinations would include
performance tests  or simulations of performance tests.

Examinations can  be used for several purposes.  One is to provide an objective criteria for determining
exemptions from the course.  Quite simply, if the exam is based on the course objectives and the trainee
demonstrates mastery of those objectives, then there is no need for the potential trainee to take the course.
The exams can also be used to determine who "passed" the course and who "failed" it. If someone did not
demonstrate mastery of the objectives, several  options present themselves. One is to have the trainee
retake the entire course.  Another option is first to have the trainee study the trouble areas, and then to
demonstrate mastery of those areas. If the cause for failure appears to be that the trainee has difficultly with
written examinations, other less intimidating methods of investigating and documenting competence (such
as personal, non threatening interviews) should be used. A third option for managing "failed"  exams would
be to ignore the incompetencies.  This option should not be tolerated. The entire purpose of providing
education  is to ensure that trainees are qualified and competent. Lowering the  standards to  allow anyone
to claim that they received the benefit of the training lowers the perceived value of the training.

This is especially true for the EPA's inspector training programs. The EPA Order 3500.1 mandates that
inspectors meet certain standards before taking positions of responsibility  in the field.  "Passing" inspectors
that do not have the necessary skills neither helps the  inspectors  nor  does it  further the EPA's  goals  of
having a qualified inspector staff. It also damages the training program's reputation, as inspectors will not
be able to do what the training  program was supposedly preparing them to do.

As mentioned previously,  exams can be used as evaluation tools. In this sense, they can determine both
instructor and course effectiveness.
As a final word  about this phase of developing training, one should modify media as feedback indicates.
This  feedback could be from the  pilot study or from evaluations conducted after the  course has  been
 implemented. Cost effective changes in media selection should be made as soon as possible.
                                                19

-------
 CONDUCT PILOT STUDY

 The training group should determine whether or not to conduct pilot studies for media, sessions, or courses
 on an individual basis before implementing them on a large scale. A pilot study is simply a dry run before
 fully implementing new material.  If the course is only going to be offered once the pilot study may be
 excessive preparation for limited returns.  However, if the course is going to be offered many times over a
 long period of time, the pilot study would be very appropriate for checking to see whether there are any
 problems with the media  or the course before full implementation.  In this case, it would tend to be more
 cost effective to run a pilot study than to revise the educational materials once implemented.

 The first step in conducting a pilot study involves finding a location to hold the study. The location should
 be similar to if not the same as the actual facility where the course will be taught. Next, a representative
 audience must be picked. The audience could consist of a group which would have received the training
 anyway.  In any case, the audience must mirror the actual trainees in regards to mix of gender, background,
 job classifications, ages, etc., as much as possible.  The pilot run should then be conducted.

 The pilot test must be evaluated. The evaluation should include quantitative information, such as the results
 of the final exams. It should also include qualitative data- the audiences feelings, opinions, and attitudes
 about the course.  Suggestions should be collected.  Focus groups could  be used to further delve into
 problem areas.

 Feedback from the pilot study should be incorporated as appropriate.  The training materials should then
 be revised to  rectify any problems.  If extensive problems were noted, another pilot study could be run to
 ensure that everything works effectively.


 COORDINATE ALL MEDIA AND FACILITIES TO PREPARE FOR TRAINING DELIVERY

 Once the course is ready,  the training  group must focus upon actually bringing the training to the trainees.
 Facilitating media communication involves finding presenters and trainers, finding facilities, determining when
 to offer the course, determining how often to offer it, and removing any barriers which may prevent trainees
 from  experiencing the course.  The training group should  remember that just as it experiences resource
 constraints, the trainees also have limits on available time, travel funds,  and educational funds. Therefore,
 every effort possible must be made to minimize the costs of training.

 The presenters and trainers must be able to communicate to an audience. They should have both technical
 expertise and also teaching skills. Slight deficiencies can be corrected if the potential trainer has otherwise
 strong qualifications; however, a great number of deficiencies pose  a barrier to trainer effectiveness. The
 trainer usually will  not need to be a world expert in both the technical and teaching fields; however, some
 moderate degree of competency in both areas is requisite.                            •

 The facilities must be conducive to learning. There must  be good lighting,  resources necessary for the
 media used (slide projector, overhead projector, etc.), adequate support facilities, comfortable seating, and
 so on.

The timing of the course should be convenient for trainees. The literature suggest that the ideal time to train
 is  just before  the  skills and abilities are  used on the job.18  That way, the real world experience can
 reinforce the largely theoretical training materials. Courses should  not be scheduled  if there are known
conflicts with other events including key national seminars or religious holidays. Self study materials should


   18  Troy, p.  19.
                                               20

-------
be mailed quickly to the trainees.  In order to deliver training as soon as possible, there should be sufficient
supply to meet the demand for the training.

Once again, feedback from evaluations should be used to determine whether any changes should be made.
Different facilities could be selected, different trainers and instructors could be used, other aspects of timing
could be watched, and other changes could be made.


TRAINING DELIVERY

Once it has reached this point, the course is ready for implementation.  This is where all of the training
group's planning and preparation combine to meet the needs for the course. One should remember that
the course can still  be.modified for the specific audience  during delivery by using  some  evaluation
techniques.   Hopefully,  in session targeting will be minimized as the  extensive preparation  accurately
prepared the materials.


EVALUATION

As discussed earlier, the courses must be evaluated in order to determine whether or not they are meeting
their goals. If they are found to be missing the targets, the courses should be modified until they do achieve
their goals.  Continuous evaluation techniques have been discussed  more comprehensively in an earlier
portion of this paper and can be reviewed for further information.
                                               21

-------
                   POSSIBLE EFFECTS OF TRAINING ON INSPECTION QUALITY
 Effective training may improve inspection quality in a number of manners.  First, it provides the inspectors
 with ail of the requirements necessary for proper inspections.  Quite simply,  inspectors that do not know
 either how or what to investigate are not likely to gather the necessary information. Training may also help
 inspectors to prioritize their inspection activities, therefore enabling them to have the greatest impact for the
 limited amount of time that they are able to inspect. Competent inspectors are much more likely to be able
 to find the necessary depth and type of information to investigate a situation. Well educated inspectors are
 more likely to be able to determine whether the sources are in compliance or noncompliance. They would
 also be more likely to be able to determine the magnitude or extent of any violations. All of this may serve
 to encourage the sources to attain states of compliance. This hypothesis is verified by the "Virginia Report,"
 which determined that inspectors who were better trained in fact found more  compliance violations. The
 findings indicate that various sources' compliance statuses improved after the well trained inspectors found
 more compliance violations.18

 Training EPA inspectors may also improve Regional,  State,  and Local inspections, as these inspectors
 frequently work together on teams. The competent EPA inspector can bring the necessary skills to the work
 at hand and under  review.  The  well-trained EPA inspector could prevent  mistakes and improve the
 inspection's quality if the other inspectors were not yet fully trained.  The well-trained EPA inspector may
 also serve as an on-the-job trainer by showing other inspectors areas that they may not have been aware
 of.  This would improve the other inspectors' abilities to conduct high quality independent inspections.

 It is intellectually appealing to believe that after teaching someone how to do something, they would then
 do the job better.  Unfortunately, it does not seem to be that simple.  One national training expert states
 that:

               A well-known but less practiced maxim is that post-training support is as important
        as the  instruction itself.  New ways of acting  or thinking must be reinforced in the work
        setting  if training is to affect business functioning according to strategic initiations.20

 Unfortunately, a number of influences besides training may also diminish the inspectors' job performances.
 If the only hinderance to high inspection quality is that an individual lacks the necessary knowledge and
 skills, then effective training can rectify the problem. However, it appears that other factors may also hinder
job performance.  Incentive systems which do not both  establish priorities and reward high> work standards
 may actually encourage  inspectors to focus upon less beneficial activities,  thus diminishing inspector
 effectiveness.   In addition, the  incentive systems for  the inspectors' supervisors may conflict  with the
 inspectors' priorities.  This conflict may distract subordinates from their primary duties by encouraging them
to focus upon their supervisor's  priorities rather than their own.  Other influential factors may include a
supervisor's  managerial skills, personal interest in the job, and peer support,  to name a few.  Rectifying
problems  in  these areas would require changes in the way that the Agency handles human resources.

One  must develop a model in  order to know  exactly how all of these possible influences can affect
inspection quality.   This requires first developing  a preliminary  model, possibly  based  on models
representing similar situations. One must then analyze the actual situation to verify that the model includes
   19 Pedco.

   20 Gherson, p.6.
                                               22

-------
all possible influences on job performance. Next, actual data for the variables must be collected. This may
require creative attempts to quantify qualitative information. The data should then be incorporated into the
model and results obtained. The model should  be revised until it predicts the real world fairly accurately.

There are many benefits to developing  such a  model. One can forecast the effects of changes in any
influential factor in order to determine how these changes  may affect the end result.  It allows the group to
monitor itself by providing a basis for comparison.  The forecasted results can be compared with actual
results to determine whether or not they are in an acceptable range. It can predict training's impact on the
inspection quality of the trainee offices.  It can be used as a basis for determining the appropriate size for
the training group and  effort.  The training group should not  be too small, therefore  unable to provide
requisite training, nor too  large, therefore improperly allocating resources.  It is important to note that the
group should not grow indefinitely, as the diminishing returns to scale could eventually damage the group's
performance.  This is the same idea as in the phrase "too many cooks spoil the broth." Finally, by showing
the impact that effective training can have on inspection  quality, the model enables the group to protect
itself. The model can justify requests for resource increases or can defend against reductions in resources
by forecasting those actions' effects.
                                                23

-------
             RECOMMENDATIONS FOR THE AIR INSPECTOR TRAINING PROGRAM

This section contains recommendations and  suggestions  for  activities which  either require further
investigation or implementation. Although the recommendations and suggestions are slanted for the Air
Stationary Source inspector training program to use in later phases of this project, their ideas may be
applicable to many situations.  The recommendations should be considered carefully.  While helpful, the
suggestions are not crucial to the training program's continued ability to function. One should also consider
previous comments made throughout this document as a targets for which to strive.


METHODS OF CONTINUALLY EVALUATING TRAINING PROGRAM EFFECTIVENESS

The current Air Inspector Training Program uses several methods to estimate training effectiveness. While
providing some feedback, the current processes may not present complete information.

 Recommendations:

 •     Implement additional quantitative methods of continually evaluating training effectiveness.

        •     Perhaps the simplest method to implement would  involve statistically analyzing data from
              course examinations.  The exams indicate not only trainee competencies but, when
              statistically analyzed, they also indicate overall course and media effectiveness. As a word
              of caution, the exams must test the course objectives in order to use them in evaluating
              training effectiveness.

        •     Another easily  implemented quantitative  evaluation technique would  involve surveying
              trainees 6 months after completing the course. This could be done through questionnaire
              in order to minimize the resources required to complete this technique.  The 6 month time
              period appears  to be as short as possible while providing enough time for the inspectors
              to really use the training and determine its effectiveness and completeness. The 6 month
              period is  only  an estimate-  the  appropriate time period  should be verified  before
              implementing this method.

•    conduct a preliminary evaluation.

      There will be a delay before results of newly implemented evaluation techniques can be collected
      and analyzed.   A preliminary evaluation would provide  an overall picture of whether or not the Air
      Stationary Source Compliance Inspector  Program appears to be meeting its goals. Conducting a
      quick evaluation will provide immediate feedback which can be used to prepare for possible results
      of other evaluation techniques or to  begin to correct likely problem areas..  Exhibit 9 describes a
      method that could be used  for a quick, goal-based evaluation.

Suggestions:

•     Implement other continuous training effectiveness evaluation techniques.

      These techniques could  include:

        •    following inspector information, including information from exit interviews
                                             24

-------
        •     following inspection results

        •     obtaining copies of complaints and compliments relevant to the trainees or the training
              itself, and

        •     following the demand for and opinion of this training programs courses versus competitive
              programs' courses.


THE MODEL OF A TRAINING GROUPS OPERATIONAL CYCLE

 Recommendations:

 •     document specific objectives for educational efforts.

       Quite simply, without a set of objectives, it is impossible to know what the program is supposed to
       achieve, when it requires modification, when to start over, or even when the educational effort has
       been successful.

 •     promptly meet demand for all types of courses.

       If demand is not met, chances are that the inspectors will attempt to meet their needs elsewhere.
       Enough classroom courses must  be scheduled and enough self  study course materials must  be
       developed to meet the demand. In addition, other media of presentation could be investigated to
       determine whether or not they would facilitate meeting this demand.

 Suggestions:

 •     consider using previously described methods of determining specific training objectives.

       These methods could be used as follows:

         •     It may be very beneficial for course designers to analyze appropriate portions of the Clean
               Air Act Amendments  of  1990 (CAAA '90) and their respective regulations  in order to
               determine new training needs, goals and specific objectives.

         •     The critical incident technique could be very helpful in determining how to train individuals
               to fulfill the permitting responsibilities  under CAAA '90.

         •     Employing CERCLA's method of verifying GS descriptions could help the department in two
               ways: first, it would provide information about  course needs and specific objectives and
               second, it would update the inspector profile.

 •    prioritize the objectives.

        Determining both the frequency and the importance of the objectives not only helps in developing
       the courses, but also facilitates the  process of reprioritizing the objectives when needs change.

 •    base end-of-course examinations on  previously  determined specific  objectives of the
       training effort.

       This allows one to evaluate not only the trainees, but also the training media and the program itself.

                                               25

-------
       Trainees who fail to show the requisite competencies should review the problem areas until they can
       demonstrate the necessary abilities.  If the exam itself poses too intimidating a threat, other, more
       friendly methods could be used as long as trainee competency is demonstrated. Achievement of
       acceptable competency levels should be documented.

 •     consider using pilot studies.

       Pilot studies will probably be most cost effective for media, sessions, and courses that will be used
       repeatedly over long periods of time.


OVERALL IDEAS

 Recommendations:

 •     Incorporate feedback from  newly  implemented evaluation techniques when designing
       training.

       Evaluation feedback should be incorporated into the training if the resulting gains would exceed
       their implementation costs.  If the feedback is ignored then the evaluation was  not likely to have
       served any useful purposes.

 •     document specific, quantitative  objectives of each training session, course, and program.

       A set of specific objectives for  each session, course,  and program  to achieve  provides the
       foundation requisite for first designing training efforts and later evaluating their effectiveness. These
       objectives provide the course and  program developers with set goals to strive for, to teach, and to
       test.  Checking for completion of the goals not only determines whether the students mastered the
       objectives, but also indicates the effectiveness of various instructors, media, and the course design.

 •      operate In the long run.

       While organizations tend to respond only to issues as they  emerge,  it is also important to look
       ahead for future trends. Some energy required  to look to the future will be expended without any
       tangible results. Despite this risk,  planning ahead will allow the group to use the  limited resources
       more in the activities of greatest importance.


 Suggestions:

 •      develop a model describing various factors' influences on effective  inspections.

       This model will forecast the effects of changes in any of the factors influencing inspection quality,
       including effective and appropriate inspector training.

 •     obtain the following documents:

              Mitchell, Garry. The Trainer's Handbook. USA: American Management Association, 1987.

              Phillips, Jack J.  Handbook of Training Evaluation and Measurement Methods.  Gulf
                     Publishing Qompany: Houston, 1983.
                                             26

-------
Troy, Kathryn L  "Quality Training: What Top Companies Have Learned." Report No. 959,
       ISBN No. 0-8237-0405-X.  New York:  The Conference Board, 1991.

Zemke, Ron and Thomas Kramlinger.  Figuring Things Out A Trainer's Guide to Needs and
       Task Analysis.  Reading, MA: Addison Wesley Publishing Company, 1982.
                               27

-------
                                          EXHIBITS

                                          EXHIBIT 1

                       EVALUATION BY TRAINEES AT END OF COURSE
 Course:
 Date of course:
COURSE DESIGN
Was the course appropriately designed for someone with your background?
               too advanced   1       2      3      4      5      too elementary
Were the course objectives clearly stated?
If they were stated, were they met?
If they were not met, which ones weren't?
                                          yes
                                          yes
              no
              no
FACILITIES
How were the facilities that the course was held in?
              they met
              my needs

              conducive
              to learning
1
they were
inadequate

distracting
COURSE CONTENT
perceived applicability of course to current work
              right on       123
              target

perceived usefulness of course to current work
           .   critical         1      2      3

Did you find the course to be stimulating?
              fascinating     1      2      3
                     4      5      missed
                                   the mark
                                   not useful
                                  deadly!
How challenging was the course?
              too           1
              challenging
                                  not challenging
                                  at all
                                            28

-------
How were the videos, slides, or any other such materials?
               clarified        1234
               topics and ideas

How were the in class exercises?
               very helpful    1234
confusing
or unclear
pointless
Did the course units follow a logical sequence?
Did later units build upon previous units?
Were the instructors'  presentations well coordinated?
Were the training materials consistent?
       yes
       yes
       yes
       yes
no
no
no
no
TRAINERS
Were the trainers enthusiastic?
Did they seem to be knowledgeable about their subjects?
Did they appear to be prepared for their sessions?
Were they able to answer questions satisfactorily?
       yes
       yes
       yes
       yes
no
no
no
no
EXAMINATION
Did the examination test the course objectives?
Did the examination seem to be fair?
        yes
        yes
no
no
OVERALL
How was this course compared to all others that you have ever taken?
               one of         12345
               the best

Did the course meet your expectations?
        If not, why not?
one of
the worst
        yes
               no
                                               29

-------
                                         EXHIBIT 2

                       EVALUATION BY TRAINERS AT END OF COURSE
 Course:
 Date of course:
TRAINERS MATERIALS
Did the materials coordinate with other units?                              yes    no
Did the materials follow a logical sequence?                               yes    no
Did they provide adequate coverage for you to teach the course?
                                                                     yes    no
Did they contain current issues, examples, etc.?                            yes    no
Please comment on any suggestions that you may have for improving these materials
COURSE CONTENT
Was there appropriate timing for the units?
              great timing    1      2
Which units required more time?

Which units required less time?
all the timing
was off
Was the information up to date?

What topics should be

       added?
       yes
no
       omitted?
       emphasized more?
                                            30

-------
       emphasizedess?_
What changes would make the course more effective?.
STUDENT QUESTIONS
How many of the questions were regarding clarification of the materials?
              all             12345      none

Did the questions indicate that the material was too advanced?

IN CLASS EXERCISES
Did these appear to help the students understand the material?
       yes    no
                                                                       yes    no
Exercises that the students had difficulty with:
Were exercises difficult because of problems with the course material preceding the exercises?
                                                                       yes    no
FACILITIES
How were the facilities?
               they met my   1
               needs

               conducive to   1
               learning
they were
inadequate

distracting
 Comments:
 OVERALL
 How was this course designed compared to others that you have taught?
               one of the     1      2      3      4      5      one of
               best                                             the worst

 Suggestions:	
                                              31

-------
                                          EXHIBIT 3

                             DELAYED EVALUATION BY TRAINEES
 Course:
 Date that the course was taken:
 I found this course-
               helpful         1       2

               to improve my  1       2
               job performance

               to be of        1       2
               appropriate depth

       if too shallow or too detailed, which?
3

3
4      5

4      5
not helpful

hurt my
job performance

too shallow    '•
or too detailed
The information and techniques in the course were
               current        123

Overall, the course
               met my        123
               expectations
If it did not meet your expectations, why not?
                     out of date
                     did not meet
                     my expectations
What topics were most helpful for you?
What would have been more helpful for you?
Do you have any suggestions for new technologies to add to the course?
                                            32

-------
How would you rate the course compared to all others that you have taken?
              one of        12345      one of
              the best            .                               the worst
Have your job responsibilities changed since you took the course?

If so, please describe the changes:	
yes
       no
                                               33

-------
                                         EXHIBIT 4


                     DELAYED EVALUATION BY TRAINEES'SUPERVISORS
                  OR BY INSPECTORS SERVING AS ON-THE-JOB TRAINERS
 Course:
 Date that the course was taken:
 Did the inspector's performance seem to improve after taking the course?      yes    no


 Did the inspector who took the course seem to perform better than comparable inspectors who have not
 taken the course?                                                      yes    no


 Did the results of the course fulfill your expectations?


        if you answered "no" to any of the above, please explain:
yes    no
Have the inspectors' duties changed since taking the course?
       If so, please describe the changes:
                                                                     yes    no
Do you or your subordinates have any training needs that are not currently met? yes    no

       If no, please describe:
                                            34

-------
                                      EXHIBIT 521

              A MODEL FOR A TRAINING GROUPS OPERATIONAL CYCLE


                       External to the Training Group	
Legislation,
Regulations, £-
EPA Orders
Changes,
 including
 technological,
 manufacturing, /	
 inspecting,     \
 new sources, and
 scientific
 discoveries
Resources ^-
                      , Within the Training Group
                       •>
                           ^
                                  "  Mission Statement
    JderrtSy General
Training Needs and Goafs
       Determine
    Specific Objectives
                        -^     Determine Training Sequence
                             71
                              Coordinate Media and Facilities
                                   for Training Delivery
                                     Training Delivery
                                           /K
                                        Evaluation
                                                                 4
                                                             
-------
                                           EXHIBIT 6

              MISSION STATEMENT LEADING TO SPECIFIC COURSE OBJECTIVES
 MISSION STATEMENT

        To satisfy changing air training requirements in a timely, effective, and efficient manner in
        order to further the Environmental Protection Agency's goals. This includes:

        1) continuously searching for new training requirements
        2) operating the process as efficiently as possible
        3) complying with legislation, regulations, and EPA Orders that mandate training and associated
               standards.
        4) training EPA, Regional, and State personnel to investigate and manage source compliance with
               legislation and regulations
GENERAL TRAINING GOALS AND NEEDS

       • training for evaluating air sources' compliance with the Clean Air Act Amendments of 1990 and
              the resultant regulations.
       • materials to help prepare individuals to become inspectors' first line supervisors.
       • training to enable personnel to issue permits according to the Clean Air Act Amendments of 1990
POSSIBLE SPECIFIC OBJECTIVES OF A TRAINING COURSE22

       • to be able to determine whether sources emit 10 tons per year or more of a hazardous air
              pollutant
       • to be able to determine whether sources emit 25 tons per year or more of a combination of
              hazardous pollutants
       • to know where to obtain a current list of sources required to obtain permits
       • to know how to determine the current tonnage fee to be collected
       • to be able to complete 100% of the processing procedures for permit application
       • to be able to obtain a historical  record of modifications that would revise previous legislation,
              regulations, or standards
       • etc.
     based on Pytte.
                                             36

-------
                                          EXHIBIT 7




                                CRITICAL INCIDENT SURVEY2
name:



office:
Please think of an incident that you have observed which reflects effective or ineffective job performance.
What was the objective?
What were the results?
 What specific behaviors led to the results?
 What attitudes, values, abilities, knowledge, and skills seemed to cause the success or failure?
 What alternative behaviors could have improved the results?
     23 based on Zemke. Figuring Things Out, p. 130.



                                               37

-------
                                   EXHIBIT 8

                   A BEHAVIORALLY ANCHORED RATING SCALE
               WHICH COULD BE USED FOR BEHAVIORS EMPLOYED
            IN OBTAINING A SOURCES COMPLIANCE DOCUMENTATION24
   EFFECTIVE
   BEHAVIORS
 To request in a polite and non threatening
 manner the source's compliance information
 documenting the issues that the inspector
 had planned to investigate.
  MODERATELY
  EFFECTIVE
  BEHAVIORS
 To demand that the source turn over all
 of the compliance information documenting
 the issues that the inspector had planned
 to investigate.
                                          Aggressively demanding the necessary
                                          documentation, stating that the documents
                                          will enable the inspector to discover how
                                          the source is cheating, failing to obtain
                                          all of the necessary documentation.
  INEFFECTIVE
  BEHAVIORS
Failing to obtain any of the source's
compliance documentation necessary for
completing the investigation.
based on Zemke. Bguring Things Out, p. 130.

                                     38

-------
                                          EXHIBIT 9

                           A QUICK METHOD OF EVALUATING THE
         AIR STATIONARY SOURCE COMPLIANCE INSPECTORS TRAINING PROGRAM
This document outlines one method of evaluating the Air Stationary Source Compliance Inspectors' Training
Program (ASSCITP) as provided by the Air Pollution Training Institute (APTI) at a point  in time.  The
evaluation will determine whether or not the Program is meeting its goals, and, if not, the areas that would
require modification. One should note that this should not be the only evaluation of the program- it should
only be used as a mechanism to determine areas which may cause problems so that planning can occur.
Data from other, more comprehensive evaluation techniques should be obtained to verify and extend the
quick evaluation's results.

Unfortunately,  there are no set models to follow in formulating this type of an evaluation.   One training
expert, Elizabeth Hawthorne, reviewed all of the training program evaluations published between 1966 and
1984 in order to provide broad models for later evaluations to follow. She found that:

        Few of the evaluations conducted and reported in the literature were formative evaluations,
        that is evaluations that are intended to modify and improve the program in process.25

Instead, most  of the evaluations were used to determine  whether to continue or discontinue the training
program in question.26  Laurel Sneed, a nationally recognized consultant specializing in training programs,
agrees with this observation27 While the training industry recognizes the need for formative evaluations,
there are no  proven  methods  of  conducting these  evaluations.  Each evaluator must  determine  an
appropriate path from scratch.

The process chosen for this evaluation of the ASSCITP involves three steps:
         1)
         2)
         3)
Determining the program's broadest goals
Investigating the ASSCITP training deliverables
If possible, delving into the discrepancies between the ideal and actual deliverables.
 This method was chosen because it provides a basis for comparison; quite simply, one cannot evaluate an
 object, process or idea without comparing it against a standard. For example, if one were asked whether
 or not slide rules were quick calculating devices, the answer would depend on what slide rules were being
 compared to. They are fast when compared to working the problem with pencil and paper,  but are slow
 when  compared to using a calculator.  In the same manner, the interpretation of the training program
 evaluation's results depends on what is used as the basis for comparison.
 The EPA Order 3500.1 should be this evaluation's basis for comparison for two reasons. First, that Order
 now drives the training program. Second, this project is a screening mechanism; therefore, it should deal
    25 Hawthorne, p.26.

    26 ibid.

    27 Sneed.
                                               39

-------
 with the overall program rather than focusing on more specific purposes.  The EPA Order 3500.1 provides
 the most general and far reaching goals for the Program.

 These goals could be determined both by examining appropriate documentation and also by interviewing
 key personnel. Specific goals stated in both the EPA Order 3500.1 and corresponding documentation
 should be noted. Interviews with personnel involved in creating the Order will verify that all goals had been
 properly  identified.  Ideally, these interviews should be in person rather than by telephone in order to
 facilitate communication between the interviewer and the interviewee. However, these interviews could also
 be conducted over the telephone without critically affecting the quality of the information obtained if resource
 constraints prohibit the use of personal interviews.

 A similar process  may  be used to complete the next step- that of investigating  the  actual ASSCITP
 deliverables. This investigation should focus on those products which directly relate to the Order's goals.
 This process involves reviewing  program documentation and  interviewing appropriate  personnel.  The
 documentation review should include general descriptions of the program as well as the results of previous
 program evaluations. Interviews  could be conducted both by telephone and in person.  Again, personal
 interviews should  be conducted whenever possible; telephone interviews may substitute  for personal
 interviews when necessary. The interviewees could include trained inspectors, regional contacts, trainers,
 and personnel in the APTI.  Ideally, the trained inspectors will have completed all of their training so that they
 can reflect back upon the entire program.  The inspectors and trainers should be able to provide insight into
 a relatively small picture; that of their own experience.  Regional coordinators should  provide a much
 broader view of the actual program deliverables. Training program designers should provide their views of
 the goals as well as the process that they use to meet those goals.

 Once any discrepancies between the stated goals and actual results are identified, these may be further
 examined. As the discrepancies have not yet been uncovered, one cannot say with certainty how best to
 further scrutinize them.   However,  one may speculate that interviews and document  reviews would be
 suitable. Interviewees could include inspectors' first line supervisors, inspectors yet in training, trainers, and
 well as additional persons in the groups previously interviewed.

 In summary, the evaluation will consist of up to three steps. These will all involve interviews with persons
 knowledgeable about specific areas, and may involve reviewing pertinent documentation  The first activity
will involve determining the goals of the EPA Order 3500.1.  Next, the actual training products  will be
determined. Finally, any discrepancies between the two will be investigated further as time permits
                                               40

-------
                                       BIBLIOGRAPHY
Air Compliance Inspector Training Informal Advisory Group Members. Memorandum from Seitz, John S.,
       Director, Stationary Source Compliance Division, Office of Air Quality Planning and Standards, U.S.
       Environmental Protection Agency:  May 26, 1989.

Air Pollution Training Institute. "Student Course Critique."

Air Pollution Training Institute. "Instructor's Course Summary." R 12/86.

Air-Specific Inspector Training Implementation Plan for Headquarters Stationary Source Compliance Program
       (OAQPS/OARVNarrative.   Memorandum from Seitz, John S.,  Director, Stationary  Source
       Compliance Division, Office of Air Quality Planning and Standards, U.S. Environmental Protection
       Agency:  February 17, 1989.

Barclay, Becky, Office of Compliance Analysis and Program Operations, Environmental Protection Agency,
       Washington D.C. Personal Interview.  June 14, 1991.

Carnevale, Anthony P. and Eric R.  Schulz.  "Return on Investment: Accounting for Training." Training &
       Development Journal. Vol. 44, No. 7: 41-72.

Centifonti, Gary J. and Richard M.  Ritota. "New Jersey Cleans Up Its Permit Process."  Asbestos Issues.
       Vol. 4, No. 5: 50-53.

Compliance Policy and Planning Branch, Office of Compliance Analysis and Program Operations, Office of
       Enforcement.  Building the Enforcement Infrastructure:  Compliance Inspector Training.  10/90.

Compliance Policy and Planning Branch, Office of Compliance Analysis and Program Operations, Office of
       Enforcement.  Report on Regional Status of Compliance Inspector Training. 10/90.

Evaluating Business and Industry Training. Ed. Leslie S. May, Carol A. Moore, Stephen J. Zammit, Kluwer
       Academic  Publishers:  Boston, 1987.

Fletcher, Donna, Office of Cooperative Environmental Management, Environmental Protection Agency,
       Washington D.C. Personal Interview.  June 13, 1991.

Gherson, Diane J.  and Carol Anne Moore. "The Role of Training in Implementing Strategic Change."
        Evaluating Business and Industry Training. Ed. Leslie S. May, Carol A. Moore, Stephen J. Zammit,
        Kluwer Academic Publishers: Boston, 1987, 3-18.

Hanlin, James R. and Nancy J. Johns. "Championship Training." Training & Development Journal. Vol. 45,
        No. 2: 57-62.

 Hannum, Wallace and Carol  Hansen. Instructional Systems Development in Large Organizations.
        Englewood Cliffs: Educational Technology Publications, 1989: Ch. 13.

 Hawthorne, Elizabeth M. Evaluating Employee Training Programs. New York: Quorum Books, 1987.
                                               41

-------
 Inspector Training and Development Workgroup. Inspector Profile. U.S. Environmental Protection Aaencv
        July, 1987.                                                                      a   y'

 Jean, Paul, Office of Human Resources Management, Environmental Protection Agency, Washington DC
        Personal Interview. June 21, .1991.

 Kovalick, Walter W. Jr., Ph.D., Director, Technology Innovation Office, Environmental Protection Agency.
        Memorandum, Directors Waste Management Division, Environmental Services Division, Emergency
        & Remedial  Response  Division, Hazardous Waste  Management Division,  Toxic and  Waste
        Management Division, Hazardous Waste Division: Dec. 20, 1990.

 Lai, Larry. Project Manager, SAIC. McCleaton, VA. Personal Interview.  June 24, 1991. :

 Mayo, C. Douglas and Philip H. DuBois.  The Complete Book of Training: Theory. Principles, and
        Techniques. San Diego:  University Associates Inc., 1987, 32.

 Mitchell, Garry.- The Trainer's Handbook. USA: American Management Association, 1987.

 Nadler, Leonard.  Designing Training Programs, the Critical Events Model.  Reading, Mass: Addison-Wesley
        Publishing Company, 1982.

 Office of Air and Radiation, Office of Air Quality Planning and Standards. Air Specific Inspector Training Plan
        for the  Stationary Source Compliance Program. SSCD SR-89-002.  U.S. Environmental Protection
        Agency:  February 1989.

 Office of Cooperative Environmental Management.  EPA Inspector Profile. Draft.  9/10/89.

 OSC/RPM Basic Training Academy. "Session A- Agenda."  6/90.

 OSC/RPM Training Academy. "A Comprehensive Program For Incident/ Site Management."

 OSC/RPM Training Academy. "Overall Objectives for Session B."                      '

 Phillips, Jack J. Handbook of Training Evaluation and Measurement Methods. Gulf Publishing Company-
        Houston,  1983.                                                           :

 Pedco, for Stationary Source Compliance Division, Environmental Protection Agency.  Development of Pilot
        Inspection System for Virginia State Air Pollution Control Board.  EPA 340/1 -83-021: 1983.

 Polvi, Gary, Office of Water Enforcement and Permits, Washington, D.C. Personal Interview. June 17,1991.

 Pytte, Alyson. "Clean Air Act Amendments."  Congressional Quarterly. 11/24/90, Vol. 48, No. 47: 3934-63.

 Radian Corporation, for Regional Operations Branch, Environmental Protection Aaencv. Survey of State and
        Local Air Pollution Agencies' Activities and Costs. EPA-450/2-90-006: February 1990.

 Rassett, Allison. Training Needs Assessment. Techniques in Training & Performance Development Series.
        Englewood Cliffs, NJ: Educational Technology Publications, 1987.  133-55.

Rae, Renelle, Office of Human Resources Management, Washington, D.C. Personal Interview June 17
        1991.                                                                    ;
                                             42

-------
Rtfkin, K. I., R. L Dueker, W. F. Diggins, F. C. Foss, and Michael Senew. Task Analysis of State and Local
       Air Pollution Control Agencies and Development of Staffing Guidelines. Volume C: Detailed Task
       Data, and Staffing Guidance FIELD ENFORCEMENT. U.S. Environmental Protection Agency, Office
       of Air Programs, Manpower Development Staff: November 1972.

Rogoff, Rosalind L The Training Wheel.  New York: John Wiley & Sons, 1987.

Schwartz,  Charles G. "Training Providers Look to New Challenges.' Asbestos Issues. Vol. 4, No. 5:16,18,
        59.

Seitz, John S., Director, Stationary Source Compliance Division, Office of Air Quality Planning and Standards,
       Environmental Protection Agency. Memorandum, Air-specific Inspector Training and Implementation
       Plans for the Stationary Source Compliance Program: February 17, 1989.

Seitz, John S., Director, Stationary Source Compliance Division, Office of Air Quality Planning and Standards,
       Environmental Protection Agency.  Memorandum, Air Compliance Inspector Training Informal
       Advisory Group: May 26,1989.

Siegler, Mark S. and Kirk Foster. Stationary Source Compliance Inspector Training Program.  U.S.
       Environmental Protection Agency, Stationary Source Compliance Division.

Smith, W.  Douglas, Senior Compliance Investigator Engineering and Inspection Section, EDO.
       Memorandum, "Draft" Inspector's KISS Manual: June 27, 1991.

Sneed, Laurel,  President, Apprend Associates, Research Triangle Park, NC. Personal assistance. May 24,
       June 7, July 9 1991.

Stationary Source Compliance Division.  Interim Guidance on the Enforcement Provisions of the Clean Air
       Act Amendments.  U.S. Environmental Protection Agency: October 19, 1990.

Thomas, Debbie, Office of Waste Programs Enforcement,  Crystal City, VA. Personal Interview. June 17,
        1991.

Townsend, Ron, Office of Air Quality Planning and  Standards, Durham, NC.  Personal Interview.  June 27,
        1991.

Tracey, William R.  Designing Training and Development Systems. Revised Edition.  New York: American
        Management Associations, 1984.

Training and Development Journal. Vol.  43, No. 10.

Training and Development Journal. Vol.  44, No. 2.

Training and Development Journal. Vol.  44, No. 7.

Training and Development Journal. Vol.  45, No. 2.

Troy, Kathryn L  "Quality Training: What Top Companies Have Learned." Report No. 959, ISBN No.
        0-8237-0405-X. New York: The  Conference Board, 1991.
                                              43

-------
Zemke, Ron. "Measuring the Impact of Management Development." Designing and Delivering Cost-Effective
       Training and Measuring the Results. Ed. Ron Zemke, Linda Standke, and Philip Jones, Minneapolis,
       MN: Training Books, 1981. 267-268.

Zemke, Ron.  "Are Sales Training Results Measurable?"  Designing and Delivering Cost-Effective Training
       and Measuring the Results.  Ed. Ron Zemke, Linda Standke, and Philip Jones, Minneapolis, MN:
       Training Books, 1981.  269-272.                                          •

Zemke, Ron and Thomas Kramlinger.  Figuring Things Out 4 Trainer's Guide to Needs and Task Analysis.
       Reading, MA: Addison-Wesley Publishing Company, 1982.
                                            44

-------
                                         TECHNICAL REPORT DATA
                                 (Please read Instructions on the reverse before competing/
1. REPORT NO.
                                                                       RECI
4. TITLE AND SUBTITLE                                      ,r-kirroo
    DEVELOPING A SYSTEM FOR EVALUATING THE EFFECTIVENESS
    OF INSPECTOR TRAINING
    Phase 1:  Identification of Evaluation Techniques and Development
    of Model  for Training Program Operations 	•	
7. AUTHOR. PERFORMING ORGANIZATION CODE
                                                                     8. PERFORMING ORGANIZATION REPOI
    Stationary Source Compliance Division
    US EPA (EN-341W)
    401 M Street SW
    Washington, DC 20460
                                                                     11. CONTRACT/GRANT NO.

                                                                     U-913530-01-0
12. SPONSORING AGENCY NAME AND ADDRESS

    Stationary Source Compliance Division
    US EPA (EN-341W)
    401  M Street SW
    Washinaton, DC 20460	
                                                                      13. TYPE OF REPOR
                                                                      Final Draft
                                                                      14. SPONSORING AGENCY CODE
 15. SUPPLEMENTARY NOTES
     Project Officer: Mark Siegler                  FTS: 398-8673         Comm.#: (703) 308-8673
     Stationary Source Compliance Division, US Environmental Protection Agency	
 16. ABSTRACT


    This study was performed under the NNEMS Grant Program for the Stationary Source Compliance Division
    of the US Environmental Protection Agency. The study was initiated in order to identify general approaches
    and techniques for evaluating inspector training program effectiveness.

    This document discusses the following:
      •     methods of evaluating training appropriateness and effectiveness
      •     operational activities for developing and delivering effective, targeted training
      •     possible effects of training on inspection quality, and
      •     recommended and suggested courses of action.

    The research was conducted through literature searches and interviews.  Only practical techniques have
    been described  in the  document.  These  include both well  established methods as well as  newer
    procedures. It was noted that formative evaluations have no standard formulas to follow;  they must be
    tailored to each  unique  situation.  Although the document frequently refers to the EPA's  Air Stationary
    Source inspector training program, the ideas and techniques are applicable to many other programs.
  7.
                                       KEY WORDS AND DOCUMENT ANALYSIS
                       DESCRIPTORS
                                                       b.lDENTlFIERS,OPEN ENDED TERMS
                                                                                          COSATI Field.'Group
     Training Development
     Inspector Training
     Training
     Training Effectiveness
     Training Evaluations
  18. DISTRIBUTION STATEMENT


     Release Unlimited
                                                         jnclassified
-45-
                                                       20. SECURITY CLASS , Tim page I

                                                       i Unclassified          	
                                                                                       22. PRICE
  EPA Form 2220-1 (Rev. 4-77)    PREVIOUS EDITION is OBSOLETE

-------

-------