AWBERG LIBRARY U.S. EPA
DEVELOPING A SYSTEM FOR EVALUATING
THE EFFECTIVENESS OF INSPECTOR TRAINING
PHASE 1: IDENTIFICATION OF EVALUATION TECHNIQUES
AND DEVELOPMENT OF MODEL FOR TRAINING PROGRAM OPERATIONS
Prepared for:
United States Environmental Protection Agency
Office of Air and Radiation
Stationary Source Compliance Division
Washington, DC
Prepared by:
Kathleen M. Reed
School of the Environment
Fuqua School of Business
Duke University
September 1991
-------
-------
DISCLAIMER
This report was furnished to the U.S. Environmental Protection
Agency by the student identified on the cover page, under a National
Network for Environmental Management Studies fellowship.
The contents are essentially as received from the author. The
opinions, findings, and conclusions expressed are those of the author
and not necessarily those of the U.S. Environmental Protection
Agency. Mention, if any, of company, process, or product names is
not to be considered as an endorsement by the U.S. Environmental
Protection Agency.
-------
-------
ACKNOWLEDGEMENTS
This study was performed by Kathleen Reed under the National Network for Environmental Management
Studies program. The project was listed as #91-1103, with Mark Siegler, Stationary Source Compliance
Division, Environmental Protection Agency, acting as Project Sponsor. I would also like to thank Kirk Foster,
Stationary Source Compliance Division, Environmental Protection Agency, for providing a historical
background on the Air Inspector Training Program activities, as well as Martha Strobel, Air Quality
Management Division, Environmental Protection Agency, and Laurel Sneed, Apprend Associates for their
assistance with providing a better understanding of the mechanics of training.
-------
-------
TABLE OF CONTENTS
EXECUTIVE SUMMARY -
INTRODUCTION
PURPOSE
BACKGROUND
METHODS OF CONTINUALLY EVALUATING TRAINING EFFORTS'
EFFECTIVENESS »»-
DURING THE SESSION OR COURSE...
AT THE END OF A SESSION OR COURSE...
SOME TIME AFTER THE COURSE OR PROGRAM-
INDEPENDENT OF THE TRAINING PROCESS...
A MODEL FOR A TRAINING GROUPS OPERATIONAL
CYCLE
MISSION STATEMENT
IDENTIFY GENERAL TRAINING GOALS AND NEEDS
DETERMINE SPECIFIC OBJECTIVES
DETERMINE TRAINING SEQUENCE
PREPARE MEDIA
CONDUCT PILOT STUDY
COORDINATE ALL MEDIA AND FACILITIES TO PREPARE
FOR TRAINING DELIVERY
TRAINING DELIVERY
EVALUATION
POSSIBLE EFFECTS OF TRAINING ON INSPECTOR QUALITY
RECOMMENDATIONS
EXHIBITS
EVALUATION BY TRAINEES AT END OF COURSE
EVALUATION BY TRAINERS AT END OF COURSE
DELAYED EVALUATION BY TRAINEES
DELAYED EVALUATION BY TRAINEES' SUPERVISORS
OR BY INSPECTORS SERVING AS ON-THE-JOB
TRAINERS
A MODEL FOR A TRAINING GROUP'S OPERATIONAL
CYCLE
MISSION STATEMENT LEADING TO SPECIFIC COURSE
OBJECTIVES
CRITICAL INCIDENT SURVEY
A BEHAVIORALLY ANCHORED RATING SCALE FOR
BEHAVIORS USED IN OBTAINING A SOURCE'S
COMPLIANCE DOCUMENTATION
A QUICK METHOD OF EVALUATING THE AIR STATIONARY
SOURCE COMPLIANCE INSPECTORS' TRAINING
PROGRAM
BIBLIOGRAPHY
ABSTRACT ...»
.4
.5
5
5
9
9
10
11
13
13
14
15
17
19
20
20
21
21
22
.24
.28
28
30
32
34
35
36
37
38
39
..41
...45
-------
-------
EXECUTIVE SUMMARY
This document begins by focusing upon methods of continuously evaluating training program effectiveness.
It is important to recognize that training developers have only recently begun to use evaluations to target
training efforts, and have not yet found any generic techniques which are applicable in all cases. Each
situation must be assessed individually in order to determine the best evaluation techniques to employ.
Evaluation methodologies which directly investigate the educational process can be used during,
immediately after, or some time after conducting the training effort. Methodologies which indirectly
investigate training's effectiveness and appropriateness are also explored.
After investigating evaluation techniques, a model of an operational structure for developing, delivering, and
evaluating training efforts is discussed. This structure must actively involve interactions with environmental
influences external to the training group in order to proactively identify and eventually fulfill prioritized training
needs. The described operation flows from the mission statement through training delivery, with feedback
from later steps, including evaluation, being used to revise and better target previous activities. This model
emphasizes using proactive techniques and adjusting the training continuously in order to best apply limited
resources.
Next, possible effects of effective training on air stationary source compliance inspections are examined.
Quite simply, the training is likely to result in better compliance in sources inspected by trained inspectors
as the sources recognize that the improved enforcement of regulations. Although training provides the skills
and knowledge necessary for job completion, these skills may not be employed if factors within the
organization discourage their use. An accurate model including all possible influences on inspection quality
must be developed in order to determine exactly what role training plays in the system.
Finally, several actions are recommended and suggested for further investigation or implementation.
RECOMMENDED METHODS OF CONTINUALLY EVALUATING TRAINING PROGRAM EFFECTIVENESS
implement additional quantitative methods of continually evaluating training effectiveness
conduct a preliminary evaluation
RECOMMENDATIONS FROM THE MODEL OF A TRAINING FUNCTION
document specific objectives for educational efforts
promptly meet demand for all types of courses
RECOMMENDED OVERALL IDEAS
incorporate feedback from newly implemented evaluation techniques when designing training.
document specific, quantitative objectives of each training session, course, and program.
operate in the long run.
One should remember that inspectors facilitate effective environmental problem resolution. When all factors
influencing job performance function correctly, well trained inspectors are very likely to help the EPA to meet
its goals. Training seems to represent one of the best methods of ensuring that inspectors are able to
complete their duties. The training cycle should result in the highest quality, most efficient, most effective
training possible. Once the inspectors are given the tools necessary to complete their duties, they will be
likely to do a better job of surveying for violations. That leads to better enforcement when necessary. The
potential violators recognize that they are better off complying with the requirements than trying to hide
excess pollution rates. A greater compliance rate with environmental requirements leads to a cleaner
environment, directly furthering the EPA's goals.
-------
-------
INTRODUCTION
PURPOSE
This document explores NNEMS (National Network for Environmental Management Studies) project #91-
1103, which is stated below:
How can air stationary source compliance inspections be improved through effective
training and how can these improvements be measured?
This topic was modified slightly during the course of the project. The primary focus became describing
methods of evaluating educational effectiveness, while the secondary focus involved creating and describing
a model of an operational structure for a training group in order to best deliver effective training. Possible
effects of training on inspection quality are also discussed. One should note that while the information
presented in this document is tailored to the air stationary source compliance inspector training program,
it contains information that could be relevant to other programs as well.
The project was conducted primarily through searches of available literature and interviews with training
experts. Although the project's scope was limited, it effectively served as a screening mechanism, focusing
upon broad aspects of effective training design and thus highlighting areas which may warrant further study
or action. Later projects will investigate and/or implement program modifications recommended by the
initial project.
The project's primary sponsor was the Stationary Source Compliance Division, and the secondary sponsor
was the Air Pollution Training Institute. Both sponsors are part of the Office of Air and Radiation, of the
United States Environmental Protection Agency. This topic involves one of the training programs mandated
by EPA Order 3500.1.
BACKGROUND
The Environmental Protection Agency was created to protect and preserve the American and world
environment from excessive changes caused by pollution, increases in technology, and the general effects
of increasing populations. The EPA completes this mission through various mechanisms. One key
mechanism involves the following problem resolution cycle:
-> discovery of problem
-> creation of law by Congress to minimize or resolve a problem
-> targeting and refinement of law by the Environmental Protection Agency through
regulations
-> surveillance monitoring conducted to check for compliance with the laws and
regulations
-> enforcement of the requirements
-> recognition by regulated parties of the penalties of noncompliance
-> compliance of regulated parties with requirements
-> positive changes
-> goals of problem minimization or problem resolution achieved
-------
One should note that inspectors play a key role in this process. The inspectors investigate whether or not
the regulated parties are in compliance situations. This monitoring greatly enhances the requirements'
enforceability. Although continuous emission monitoring systems, new analytical tools, record keeping
requirements, and reporting requirements are being implemented, personal inspection still provides the
backbone of compliance determination. Once the new techniques are fully implemented, inspections will
probably still be used to verify the devices' accuracy and the facilities' compliance.
The inspectors are important for other reasons as well. For one, competent, credible inspectors seem to
serve as an enforcement deterrent. The "Virginia Report" describes how the EPA Air program investigated
the influence of training on improved compliance reporting. The findings indicate that various sources'
compliance statuses improved after the well trained inspectors found more compliance violations.1 The
inspector is important because the inspection data provides the basis for actions and cases against
noncomplying sources. Quite simply, the better the data, the better the case, and thus, the greater the
likelihood that an enforcement action will occur. If the data is weak (for example, if it is incomplete or
inaccurate), the case will be weak, and the enforcement probably will not occur. Without enforcement, the
incentives for affected parties to comply with the requirements decrease substantially. While the inspectors'
roles have been substantial in the past, the inspectors' responsibilities may soon increase to also include
more enforcement, when the permitting process that the Clean Air Act Amendments of 1990 provided is
activated.
Not only do the inspectors play a major role in facilitating the EPA's goals, they also represent a significant
annual investment for control agencies. Stationary source compliance monitoring represented 32% of state
agencies' reported annual compliance expenditures, and 37% of local agencies' reported annual compliance
expenditures.
When one considers both the critical role that inspectors play in the process of implementing programs and
also the enormous annual investments made in the inspectors, one recognizes that great care must be given
to guaranteeing that the inspectors fill their role to the peak of their abilities. A deficiency in skills and
knowledge requisite to completing duties may pose the most basic potential problem affecting inspector
performance. Any inability of inspectors to conduct their duties would impair the problem resolution cycle.
If the inspectors are unable to survey, then enforcement is not likely to happen, compliance is even less
likely, and problem resolution may not occur. Training provides one mechanism for rectifying a lack of skills
necessary to perform the duties. If the agency is able to increase the inspectors' ability to complete their
job requirements, it will most likely improve the quality of inspections, thereby facilitating the program
implementation cycle and increasing the likelihood of meeting the EPA's goals.
1 Pedco.
2 Radian Corporation, E-5.
-------
METHODS OF CONTINUALLY EVALUATING TRAINING EFFORTS-
EFFECTIVENESS
Evaluations are frequently used in order to determine the training effort's effectiveness and targeting. This
is because training groups and departments need to verify that their efforts effectively teach the appropriate
skills, knowledge, values, and behaviors. As one training expert states,
"If you are happily training hundreds of employees yet have no idea whether they
perform to the standards they've been taught- or even if they use the skills youVe given
them- you are failing in your mandate. Remember, the purpose of training is to bring about
change. If you dont measure the change that has taken place, you cant know if you are
performing the task."3
Quite simply, an evaluation's purpose is to determine whether the program is meeting its goals or not. The
goals merely provide a basis for comparison. For example, if one were asked whether or not slide rules
were quick calculating devices, the answer would depend on what slide rules were being compared to.
They are fast when compared to working the problem with pencil and paper, but are slow when compared
to using a calculator. In the same manner, the interpretation of the training evaluation's results depends on
what the goals were. If the training is meeting its goals, then all is well. However, if it is not, then some
changes need to take place. Feedback must occur and necessary changes must be made in order for the
evaluation to have any purpose. This helps to ensure that scarce resources are allocated to areas where
they will have the greatest impacts.
Training courses and their required amounts of targeting can be compared to shotguns and rifles, as very
similar targeting processes are used both in training and in firing a gun. Shotguns require some focusing;
however, the shot disperses over a relatively large area. This dispersion permits one to hit the target even
if the aim is slightly inaccurate. The likelihood that at least some of the shot will hit the target is high, but
there is also waste, as many of the pellets will miss the bull's eye. On the other hand, one must be
extremely precise when aiming a rifle, as the single bullet does not forgive mistakes. If the aim is incorrect,
there is extremely high waste, and there is no chance of hitting the target; however, as long as the rifle is
aimed correctly, the bullet efficiently hits the target. Just as these two firearms require different amounts
of aiming because their shot disperses differently, various types of training require different evaluative
procedures to ensure that their dispersed or targeted educational efforts fulfill their purposes. Some training
efforts are similar to a shotgun, in that the training efforts are assembled quickly and cover a wide range of
topics in the hope that at least a few of these widely dispersed topics will teach what the trainees need to
know. These do not require many mechanisms to verify that they teach the appropriate materials. On the
other hand, one can develop extensively planned, efficient, precisely targeted training efforts which have
been designed to complete only very specific objectives. This type of an effort is analogous to a rifle; the
tightly packaged, well focused training materials are highly efficient but leave little room for inaccuracy.
Evaluations can be used for several purposes. In all cases, they help to employ limited resources in the
most effective manner possible. The evaluations are commonly grouped as either summative or formative.
Summative evaluations are used to determine whether or not to continue the program. These have been
used extensively in the past. Elizabeth Hawthorne, a training expert, reviewed all of the training program
evaluations published between 1966 and 1984 in order to provide broad models for later evaluations to
follow. She found that most of the evaluations were used to determine whether to continue or terminate the
Mitchell, p. 141.
-------
training program in question." The second group of evaluations, the formative evaluations, are somewhat
different. They are used to identify areas of a current program which may require modification. Hawthorne
also states that:
Few of the evaluations conducted and reported in the literature were formative evaluations,
that is evaluations that are intended to modify and improve the program in process.5
While the training industry recognizes the need for formative evaluations, there are no proven methods of
conducting those evaluations. Each evaluator must determine an appropriate method from ground zero.
Formative and summative evaluations may also be better explained by continuing the gun analogy.
Summative evaluations involve a process similar to that of determining whether or not to stop shooting the
gun. On the other hand, formative evaluations can be compared to the process of aiming a gun, evaluating
whether or not the firing was accurate, and determining the adjustments necessary to increase the next
round's accuracy.
A logical path for any evaluation to take involves comparing actual results to ideal results. The training
effort's ideal results should be determined in order to provide a basis for comparison. Perhaps the simplest
and most comprehensive source for these projections is the statement of specific objectives for a given
course, session, or program. If a model which accurately forecasts the programs effects exists, its results
could also be used as the basis. Next, the actual results of the training effort should be obtained. Simply
matching the actual to the ideal results can then provide an evaluation of the program's effectiveness.
It is important to recognize that all training sessions, courses, and programs should be evaluated. This
includes short run courses even though the evaluation may be completed after the training group stops
offering the effort. It is important for the course designers to see the effectiveness of previous courses so
that they can either avoid similar mistakes in the future or continue to use very effective techniques. The
need for evaluations of training efforts with long life spans is more obvious. A lack of evaluations could
result in very costly mistakes.
Although a variety of techniques exist for obtaining information necessary when conducting evaluations,
evaluators often use surveys. These surveys can take the form of either questionnaires or interviews. The
interviews may be conducted either in person or over the telephone, and can be structured or unstructured.
Structured interviews contain a set, unchanging series of questions.6 Unstructured interviews are also
based on a list of questions; however, the unstructured interviews maintain the flexibility to investigate areas
of interest as they unfold.7 A less common method of evaluating a course involves observing actual course
sessions. As noted, this is rarely used, but allows evaluators to obtain an outside opinion of a course. A
third well known technique is to use focus groups. This consensus building technique often serves either
to explore an unknown or to measure a past occurrence. Focus groups consist of a panel of either subject
experts or a representative cross section of a group targeted by the issue in question. The panel's diverse
opinions are debated until a consensus is reached.
Evaluations for sessions, courses, and programs can be employed at several times.. Some evaluations may
be conducted during the course session, both to slightly modify the course effectiveness and also to make
4 Hawthorne, p.26.
5 ibid.
6 Philips, p. 81.
7 ibid.
-------
long term changes to the course. Others are more appropriate at the courses' or sessions' conclusion, and
affect future presentations. The evaluations conducted during and immediately following the session, course,
or program can be thought of as short- run evaluations in that they measure only immediate features of the
training effort. These features include the abilities both of the training to communicate and of the trainees
to assimilate the educational objectives. In contrast, other evaluations investigate long- run issues. One long
run issue could be the effects that the training effort has on job performance. Evaluations conducted several
months after the course or program was presented can investigate long-run issues. The delay facilitates an
investigation of how the training effort actually affected trainees' job performances. A fourth and final group
of evaluation techniques investigates factors other than the actual training. We will now investigate these
four groups of evaluations.
DURING THE SESSION OR COURSE...
There are several uses for these methods of evaluation. Their primary purpose is to ensure that a course
already in session is achieving its goals. By monitoring the course as it progresses, trainers can tailor the
course to the trainees' needs. These techniques can also identify necessary modifications to the course's
basic format; repeated revisions of the same problems in the course may indicate that the course's outline
should be changed. A trained observer may be helpful in objectively employing these techniques.
These in-session evaluations have several benefits and weaknesses. One benefit is that they are very
inexpensive to implement, requiring only the trainer's willingness to observe the session, modify the course
if necessary, and report any findings back to the training development group. However, these evaluations
also have a number of drawbacks due to their superficial nature. They cannot show the long term
effectiveness of the course; only its ability to communicate the material while in session. They can fail to
recognize areas that require attention. Finally, they only look at material currently in the courses and
therefore cannot determine whether the topics are appropriate or if other topics should also be addressed.
One method of evaluating a course in session involves monitoring questions to determine trainees'
comprehension of course materials.8 The trainer should monitor trainer initiated and also trainee initiated
questions. If trainees neither answer nor ask questions, some probing may be necessary to determine the
cause of their lack of involvement- be it that the material is too basic, too advanced, too dull, or any other
reasons.
In class exercises present an excellent method of determining whether or not trainees are assimilating the
material.9 Practice simulations, including cases, discussions, and problems, can show whether or not the
trainees have fully understood the material, and highlight which areas require more emphasis.
AT THE END OF THE SESSION OR COURSE...
Evaluations conducted at this time allow participants to reflect back on the entire session and present an
overall opinion. One should note that participants may forget information about the early parts of the
training effort, especially if it is long. These evaluations can only measure the perceived applicability of the
educational materials to future job requirements, and can investigate only the perceived appropriateness and
completeness of the topics included in the course. Examinations, trainee evaluations, trainer evaluations,
and summary reports are frequently used methods.
8 Mitchell, p. 126.
9 Mitchell, p. 128.
-------
Examinations generally come as either a set of pre- and post- tests, or a single test given at the end of the
exam. The post-test method provides a picture of the trainees' abilities at the end of the course. This
enables one to show overall competencies, but not to determine the exact effects that the course had. A
better method involves the pre- and post- set, which measures not only overall competencies but also gains
from the course. This method facilitates comparing abilities both before and after the course; this
comparison can be used to show the precise effects that the course had.10 Quite simply, the course is
effective if trainees learn necessary material during the course.
Examinations can also be used to determine both course or session and media effectiveness. Raw data
from each session's or course's tests should be gathered and assimilated according to question number.
Poor wording of questions or a failure to achieve the objectives could cause large numbers of trainees to
have difficulty with certain questions. The data can also differentiate between problems caused by the
instructor's inability to communicate certain information or by the class design. If multiple types of media
are used to teach the same information, then a comparison of results between media can determine if the
difficulty is caused by the media. For example, if trainees under only one trainer miss a given question, and
trainees instructed by other trainers do not have trouble with that question, then the instructor is probably
causing the problem. If all of the trainees have difficulty regardless of who the trainer was, then it indicates
a problem with the course or session design. One should be very careful to use proper statistical evaluation
techniques when looking at test data, as oftentimes casual observations can lead to statistically incorrect
conclusions.
In addition to final course examinations, trainee evaluations can also determine apparent training
effectiveness as well as training acceptability. These evaluations generally investigate areas such as
satisfaction with the course, the facilities, and the course materials. Different areas of the course can be
rated on a set scale. Opinions regarding the best and worst parts of the course are often gathered, as are
suggestions for improving the course. An example of this type of an evaluation can be found in Exhibit 1.
Trainer evaluations may investigate the course's design, teaching materials, course content, and other areas.
They can also be used to gather results from in-class evaluations, such as the questions and class exercises
described earlier. An example of a trainer evaluation is located in Exhibit 2.
Finally, summary reports can evaluate sessions, courses, or programs. The Air program currently employs
this method. Once the training effort is completed, the trainees fill out evaluations. The trainer sorts,
reviews, and summarizes these evaluations. The trainer then evaluates the training effort and documents
any recommended modifications to the course. The summary report includes the trainee examinations,
trainee evaluations, the summary of the trainee evaluations, the trainer evaluation, and the recommendations.
SOME TIME AFTER THE COURSE OR PROGRAM...
These evaluations enable one to investigate long term results and issues of training efforts. The evaluations
are often conducted between 6 and 18 months after taking a course. This method can investigate items
such as effects on job performance, whether or not general topics were useful, or whether other topics
should be added to the training effort. These evaluations' strength resides in their ability to determine actual
and perceived impacts of the training on job performance.
One of these methods involves surveying the trainees. Two identical surveys can be conducted; one a few
months after the training effort was completed, the other can be conducted 1-2 years after completing the
effort. Areas to investigate include perceived usefulness of the course, perceived impact on job
10
Philips, p. 102.
10
-------
performance, subjects that were most helpful to the job, as well as areas that should have been covered,
such as new technologies and techniques. These can also be used to determine whether the job has
changed since the course was originally designed. If so, the course or program should be modified to
reflect the changing work environment. An example of this survey can be found in Exhibit 3.
Another evaluation technique entails distributing questionnaires to trainees who have returned for additional
courses.11 The questionnaires could investigate any previous courses that the inspectors may have taken.
This form may be similar to the previous survey, with the only difference being that it could request
information about a number of courses. Although surveying returning trainees provides valuable information,
this method should not be the sole evaluation method. This is because it surveys only the returning trainees
(those trainees satisfied enough with the previous course to return for additional instruction) rather than the
entire set of trainees.
A third evaluation technique involves surveying either the trainees' first line supervisors, or the inspectors
serving as on-the-job trainers, or both. Again, two identical surveys could be conducted at different intervals.
They could investigate the perceived usefulness of the course, the perceived changes in the inspectors'
performances, and also the changing requirements for the inspectors' jobs. Exhibit 4 contains an example.
A list of course objectives or other basis for comparison could be sent with the survey if specific skills and
abilities are being investigated. In addition, the evaluation to the inspector/on-the-job trainers could ask
them to describe any areas that consistently present a problem for the trainees.
INDEPENDENT OF THE TRAINING PROCESS...
The following areas can be continuously observed in order to monitor the training program's actual and
perceived effectiveness. One should note that these evaluation techniques can be used regardless of when
the training occurred. Many of these are qualitative rather than quantitative. Although this section describes
how to use these methods for Air Stationary Source Inspector training programs, their general ideas can
be applied elsewhere.
To begin, evaluators can collect information about inspectors. This information should reflect overall course
goals (possibly key course objectives) and may include absenteeism, numbers and types of mistakes made,
inspector turnover, and information from the exit interviews. Mistakes could be analyzed to determine (1)
whether they were caused by failure to meet a course objective, and (2) whether they should be included
in the objectives to prevent others from having the same difficulties. Evaluators may wish to determine why
the inspectors leave the EPA, as stated in exit interviews. These results could be analyzed according to the
reasons for leaving and the extent to which the inspector was trained.
Information about the actual inspections can also be used to evaluate training effectiveness. This
information can be obtained either from EPA records or from the inspected sites. There is some reason to
believe that personnel at the sites would be responsive to requests for this information, as it parallels the
trend in the business world to become ever more customer driven. While the act of requesting information
from inspected sources contains some immediate risk that the feedback may be negative, requesting
information from the sources actually decreases total risk. By learning of and correcting problems in the
present, problems can be minimized and resources can be used more effectively in the long run. Even if
the evaluation used only EPA records, it would still carry some risk. By only using EPA documentation, the
risk would involve obtaining incomplete information and therefore failing to respond appropriately.
Inspection information that can be monitored includes, but is not limited to, the list in Figure 1.
Mitchell, p. 142.
11
-------
Figure 1
: EXAMPLES OF INSPECTION INFORMATION VWICH COULD BE
INVESTIGATED CONTINUOUSLY
the inspector's Image (whether or not the inspector appeared competent and
professional) -' ,,
the disruption that the inspection caused to the source's operations
numbers of repeat Inspections required to collect additional information
information thai required repeat visits to fully obtain
the numbers of contested versus uncontested Inspection conclusions
ff lawsuits arise, the adequacy of the inspectors' evidence to prove noncomptiance
This information should be grouped according to whether or not the inspectors had taken the training, and
if so, whether or not they passed the course.
A third method could be to monitor informal anecdotal feedback on the courses and programs. This
includes feedback from regional, state, and local contacts, as well as comments made at meetings, such
as national meetings. Informal comments from trainee supervisors and on-the-job trainers should also be
noted. Complaints, compliments, and suggestions should be noted. Although this is helpful, it is neither
comprehensive nor complete and therefore should be employed only in conjunction with other evaluation
methods.
Forma^feedback should be followed if at all possible. Formal letters of complaint or compliment should be
noted. These may be sent by training recipients, from inspected units, or from public interest groups.
Complaints should result in either further evaluation or immediate changes, depending upon the complaints
severity and validity.
The demand for the courses should be monitored. Total increases or decreases in both the numbers of
potential trainees and in demand should be noted. Demand should be compared to the estimated or actual
figures for competitive training courses and programs. The worst scenario would be for the total demand
for the training program to decrease substantially while the number of perspective trainees and the
competition's demand skyrocketed.
Finally, periodic surveys in media such as newspapers or bulletins that training customers would receive can
be used to determine perceptions of the training courses and programs.
12
Mitchell, p. 108.
12
-------
A MODEL FOR A TRAINING GROUPS OPERATIONAL CYCLE
This section will examine one method of organizing the operation of developing and delivering training.
Exhibit 5, which contains a model of a training group's operational cycle, illustrates the ideas that will be
discussed in rest of this section. The box in that Exhibit represents the border between the training
department and the rest of the world. Everything within the box represents the training operations; all that
occurs and exists outside of the training group is shown outside of the box. The arrows show information
flowing in the direction that the arrows point. It is important to notice that information flows throughout the
process, not just from the top down. Each major portion of the process will be described separately.
MISSION STATEMENT
The mission statement concisely states the training group's purpose and guiding principles. This mission
must help to satisfy the missions and goals of the organizational structure above the group. Because the
group is part of an entire organization, factors external to the group also influence the groups resources and
future. In order to ensure that the group survives and thrives, it must respond to the needs of the
organization and must help to fulfill the missions of the organizational structure above it. In the EPA, the
air stationary source compliance inspector training development group's actual superior structure includes
the Stationary Source Compliance Division, the Office of Air Quality Standards and Planning, the Office of
Air and Radiation, and, of course, the parent Environmental Protection Agency.
Figure 2 shows a sample mission statement. This Figure is actually part of Exhibit 6, which illustrates how
the mission influences the training department's activities through identifying the specific objectives of the
training effort.
Figure 2 __________^__
MISSION STATEMENT SHOWN IN EXHIBIT 6
To satisfy changing air educational requirements In a timely, effective, and efficient manner
in order to further the Environmental Protection Agency's goals. This includes:
1) continuously searching for new training requirements,
2) operating the process as efficiently as possible.
3) complying with legislation, regulations, and EPA Orders that mandate training and
associated standards.
4) training EPA, Regional, and State personnel to determine compliance with legislation and
regulations.
The mission statement should be sufficiently global so that it can guide the training group throughout
changing conditions; however, if conditions change so drastically that it no longer serves to appropriately
guide the group, the statement should be changed.
13
-------
IDENTIFY GENERAL TRAINING GOALS AND NEEDS
This involves determining the general need for training, be it a session, course, or program, and also
preliminarily determining its subject matter. Figure 3 shows several possible course needs identified from
the Mission Statement. The specific course content should not be determined at this point. Instead,
decisions should be made on items such as subject matter, whether it should provide background
knowledge or hands-on skills, and whether it will be used as training or development.
The training group could determine the need for a new course or program from a number of sources. The
mission statement provides the most basic source of information. For example, Section 4 of the Mission
Statement found in Figure 3 states that the group will train inspectors to determine compliance with new
legislation and regulations. Note that one of the educational needs identified in this example involves
preparing personnel to issue permits as described in the Clean Air Act Amendments of 1990.
Figure 3
MISSION STATEMENT \
To satisfy changing air educational requirements In a timely, effective, and efficient
manner in order to further the Environmental Protection Agency s goals. This includes:
1) continuously searching for new training requirements.
2) operating the process as efficiently as possible.
3) complying with legislation, regulations, and EPA Orders that mandate training and
associated standards. !
4) training EPA, Regional, and State personnel to investigate and manage source
compliance with legislation and regulations.
POSSIBLE TRAINING GOALS AND NEEDS
training for evaluating air sources'compliance with the Clean Air Act Amendments of
1990 and the resultant regulations.
materials to help prepare individuals to become inspectors'first line supervisors.
training to enable personnel to Issue permits according to the Clean Air Act
Amendments of 1990. -
Courses will also need to be developed to comply with mandates including legislation, regulations, or EPA
orders, all of which may specify training requirements. These mandates currently drive the training cycle.
One undesirable operational method of determining these stated needs involves watching for applicable
mandates to be officially published. This eleventh hour, reactionary method prevents long term planning
and job scheduling. This results in a fast-or-famine mode of operation with only limited flexibility in
responding to educational needs. A better method of operation would involve scanning the environment
for clues that any requirements relevant to the training department's operations would soon be emerging.
This desirable long run operational mode would enable the training group to plan and schedule jobs ahead
of time and thus to avoid backups and to best utilize slack periods.
14
-------
Courses may also be developed to fulfill superiors' direct requests. These requests may occur if
management believes that a problem could best be resolved through training. The training group should
respond to these requests as quickly as possible.
Although congress currently drives the system, changes in inspection technology, manufacturing techniques,
emission control equipment, or other similar indicators should be used to investigate the needs for course
development or modification. The training group could learn about these technological breakthroughs from
technical trade journals, from contacts at universities, contacts in industry, or from other EPA personnel
(inspectors, perhaps). The group should also watch for scientific discoveries, such as determinations that
compounds which had been thought to be harmless actually are hazardous. Such a discovery would
indicate that the industry will eventually be required to reduce that compound's emissions and that the air
stationary source inspectors will be required to check for that compound. Both new technologies and
scientific discoveries could also indicate that new types of stationary sources would soon be emerging,
requiring new methods of inspection. Keeping the courses current maximizes their value to the trainees,
provides the most impact for their resources, and helps to justify the training group's existence.
Specific requests may inspire course modification or creation to meet set needs. One method of handling
these requests would involve processing them as they emerge. As discussed previously, a long run vision
provides better operational results than does a short run viewpoint. The long term vision requires that the
training group nurture ongoing relationships with the offices which may request training. Staff from the
training department should regularly contact customers (in the air stationary source program, this includes
regional, state, and local offices) to discover possible future needs. Once these have been identified, the
training group may create a preliminary plan showing how courses meet the requirements. This plan then
becomes a marketing tool. The training group may use the plan to convince the potential client that the it
the only one that can meet the clients needs. Besides customers, those affected by training results (ie- the
sources inspected by stationary source compliance inspectors) may request that inspectors be trained on
specific technologies. These requests would only occur if the inspections began to cause severe problems,
and would most likely occur in the form of registered complaints to the EPA.
Finally, feedback from training evaluations may either verify that the appropriate needs were addressed or
may indicate that other needs exist. Quite simply, trainees may require additional skills. Also, some needs
may no longer be valid as conditions change.
DETERMINE SPECIFIC OBJECTIVES
Once the training effort's general goals have been determined, the specific objectives necessary to fulfill
those goals must be determined. Figure 4 shows examples of specific course objectives which could have
been identified according to a possible course need. In this instance, a partial list of objectives has been
created to train personnel to issue permits to stationary sources.
These objectives have many purposes. One is to shape the training; identifying course objectives facilitates
the selection of appropriate training media such as videos or documentation. Trainee examinations should
investigate whether or not trainees have met the objectives. These will be discussed in more detail in the
section describing media preparation. If available funding would prohibit the training group's ability to meet
stated needs, the group could request increased funds. The objectives also can be used as inspectors'
performance standards if the training is for specific jobs.
15
-------
Figure 4
ONE POSSIBLE TRAINING NEED
training to enable personnel to Issue permits according to the Clean Air Act Amendments "
of 1990. ' , ,",> ,*\ ' ' '
POSSIBLE SPECIFIC OBJECTIVES OP A TRAINING COURSE1 '
.. % f f&ff^fff S f "* 1 ""
» to be able to determine whether sources emit 10 tons £er year or more of a hazardous air
pollutant. - ^ ' " "_ " , ,/" -I-
to be able to determine whether sources emit 25 tons per year or more of a combination
of hazardous pollutants.
* to know where to obtain a current fist of sources required to obtain permits.
to know how to determine the current tonnage fee to be collected,
to be able to complete 100% of the processing procedures for permit application.
to be able to obtain a historical record of modifications that would revise previous
legislation, regulations, or standards. ;
* etc. ., ,,,,,/,,' ,_*$*"?, "/,", I -
. t,.,_ ^ "f, , , ^
1 based on Pytte, p. 3955. ''' ''""~ \
The objectives can also be used to modify existing courses. If the objectives serve this purpose, they can
then guide the changes necessary to improve the course. It is important to note that while investigating
specific course objectives, the course designers may discover previously unstated goals and needs for the
course. These should be verified, and either included in the same course or used as needs for a new
course.
Although the discussion has been theoretically sound to this point, practical applications have not yet been
discussed. Three practical methods of determining course objectives are described below.
One method of determining specific course objectives would involve analyzing a regulation or law's
requirements.1 Take, for example, the Clean Air Act Amendments of 1990. The training group in
coordination with subject experts or program managers could analyze the law and resulting regulations to
determine exactly what skills, abilities, and knowledge would be necessary to inspect for regulated parties'
compliance. The program managers often are both knowledgeable about these issues and in control of
funds, some of which could be used to facilitate training development. Again, if the funding prevents
effective training, requests for additional resources would be justified. The objectives for a course or a series
of courses would then consist of necessary competencies.
Utilizing critical incident reports is another method of determining course objectives.14 This method is
typically used for jobs that require employee discretion, such as that of an inspector. The critical incident
technique, described in Figure 5, determines both productive, desirable behaviors as well as
counterproductive, undesirable behaviors which may prevent employees from fulfilling job requirements.
13 Fletcher.
14 Zemke. Figuring Things Out, p. 134-8.
16
-------
Figure 5
THE CRITICAL INCIDENT TECHNIQUE""
1) Information about the job is gathered on critical incident forms, such as the
example shown in Exhibit 7. These forms generally investigate three areas:
(1) identification of the reporter (name, project title, office, etc.) in case
additional information must be obtained.
(2) a description of an incident, including the situation, the objective, the
technique used, and the positive or negative results.
(3) evaluator commentary regarding the usefulness of the behavior,
alternative behaviors which could have been used, etc.
2) The critical incident reports are gathered and a Behavlorally Anchored Rating
Scale (BARS) Is created. This is completed by first grouping reports
containing similar incidents. The behaviors are then arranged on a scale,
from most productive to feast beneficial. An example of a BARS can be seen
in Exhibit 8.
3) Additional information required to fill in the gaps can then be obtained.
4) The final results should be verified for thoroughness and accuracy by experts.
Focus groups of job experts can facilitate this process.15 These experts could include highly competent,
experienced inspectors, supervisors, knowledgeable individuals who were once inspectors, or other
individuals. The focus group could generate the initial group of critical incident reports, sort the reports
between beneficial and detrimental behaviors, and complete any missing areas.
The critical incident technique can also be used to determine whether or not the program needs revision.
If the program has been running for a period long enough that the job requirements may have changed, the
critical incident technique can highlight areas that should be included in future courses.
A third method for determining course objectives involves basing them on job descriptions. The
Environmental Protection Agency's CERCLA program currently uses this method.16 A panel analyzes GS
job descriptions to determine what the specific training requirements are. The panel first reviews the current
GS descriptions to determine what the different activities are and to estimate the percentage of time that is
spent on each activity. Surveying actual inspectors verifies the panel's findings and determines whether
other activities should be included in the job descriptions. The results are used to update the GS job
descriptions. The course objectives then involve preparing inspectors to complete those identified activities.
DETERMINE TRAINING SEQUENCE
The training sequence should respect both logical order and resource limitations. The objectives' priorities
should be determined, they should be arranged according to their priorities, and then as many of the high
priority items as possible should be included in the sequence as resources permit.
15 Zemke. Figuring Things Out, p. 134.
16 Thomas.
17
-------
The objectives and their requisite tasks and behavior may be prioritized in a number of ways. One basis
involves determining which objectives form conceptual bases for later, more advanced requirements.
Another method assigns priorities according to pre determined penalties for noncompliance.17 This could
be appropriate if the course teaches skills necessary for specific types of inspections, Finally, objectives
could be prioritized according to relative importance to the job.
The following describes one way to prioritize items according to job importance or compliance penalties.
A questionnaire may be sent to knowledgeable parties to determine the item's frequency of occurrence and
its absolute importance, either on a scale of 0 (never used) to 1 (critical) as shown in Figure 6 or in dollar
amounts. A chart similar to the one below could then be created.
Figure 6
- - RELATIVE IMPORTANCE
ITEM
A
B
C
FREQUENCY
1.00
.20
.30
IMPORTANCE (FREQUENCY X IMPORTANCES
1.00 % ,
1.00
.70
, 1.00
,20
>21
An illustration may help to better explain this method. Please refer back to Figure 6. Imagine that this
technique was being used to determine the relative importance of different behaviors for inspectors.
Questionnaires could be sent to highly competent, experienced inspectors and other inspection experts.
Let's say that item A represented the behaviors required to gain entry to a facility. Because the fictional
questionnaires indicated that this behaviors occurs for every inspection, the frequency is 1. Also, because
it is critical to completing the inspection, its importance is also 1. Now, say that B represents the behaviors
necessary to successfully obtain a warrant. If the questionnaires indicated that these behaviors are
necessary only 20% of the time, the assigned frequency is .2. Now imagine that when required, these
warrants are absolutely critical to completing the inspection. Based on this information, their importance
is assigned a 1. Finally, imagine that behavior C represents the behaviors necessary to learn about new
technologies while on the job. This time the questionnaires indicate that the behavior is used in 30% of the
inspections (thus a frequency of .3) and that while important, it is not critical to completing the inspection.
Item Cs importance is therefore assigned as .7. Each frequency is multiplied by the corresponding
importance to obtain the relative importance. In this instance, behavior A would be the highest priority,
behavior C the second priority, and behavior B the lowest priority.
One should note that the training sequence may occasionally require modification. The pilot study's results
may indicate that there is a much more effective sequence of presenting the materials. Also, priorities may
change with time. Therefore, the evaluations' feedback concerning the training sequence should be
reviewed on a regular basis.
In addition to the relative priorities of different items, the training developers should determine what limits
there are on the resources. These could include time, finances, or any others. If the resources are limited
to such an extent that there is only a questionable ability to create the necessary training, then requests to
increase the resources would be appropriate. The discrepancy between the need for education and the
inability to provide training could be used to justify these requests. The final course would include as many
of the highest priority items as resources permit.
17
Fletcher.
18
-------
PREPARE MEDIA
Once objectives have been determined, the media for communication must be prepared. One should
remember that each type of media is appropriate in different circumstances and for different purposes. Each
objective should be analyzed to determine the most appropriate media for teaching that item. For example,
swimming classes are held both in a swimming pool and on dry land. Strokes and techniques are taught
in the water, where students can learn through experience. In contrast, issues of safety tend to be
presented as lectures. It is better to hear about the dangers of running on deck or diving into shallow water
than to learn through experience! The objectives of swimming lessons are taught using training media
appropriate for each particular objective. Unfortunately, traditional training tends to present even the most
practical information as theoretical documentation. This poses the same problem as would attempting to
teach people to swim by discussing swimming theory. The trainees may know the theory, but not how to
apply it. Regardless of the type of media used, the course developers must make certain that the presented
information is consistent within the course and with other courses presented by the training department.
Examinations should be developed at this time. They should be used to measure the trainees' achievement
of course objectives. If exams are based on anything other than the objectives, then there is no way to
determine whether or not the objectives have been met. Because training in particular is used to enable
trainees to better conduct their current job requirements, training exams should be very real world oriented.
Quite simply, if the objectives are met in the classroom, then they are more likely to be achieved in the real
world. If the objectives are to provide skills for use on the job, then the best examinations would include
performance tests or simulations of performance tests.
Examinations can be used for several purposes. One is to provide an objective criteria for determining
exemptions from the course. Quite simply, if the exam is based on the course objectives and the trainee
demonstrates mastery of those objectives, then there is no need for the potential trainee to take the course.
The exams can also be used to determine who "passed" the course and who "failed" it. If someone did not
demonstrate mastery of the objectives, several options present themselves. One is to have the trainee
retake the entire course. Another option is first to have the trainee study the trouble areas, and then to
demonstrate mastery of those areas. If the cause for failure appears to be that the trainee has difficultly with
written examinations, other less intimidating methods of investigating and documenting competence (such
as personal, non threatening interviews) should be used. A third option for managing "failed" exams would
be to ignore the incompetencies. This option should not be tolerated. The entire purpose of providing
education is to ensure that trainees are qualified and competent. Lowering the standards to allow anyone
to claim that they received the benefit of the training lowers the perceived value of the training.
This is especially true for the EPA's inspector training programs. The EPA Order 3500.1 mandates that
inspectors meet certain standards before taking positions of responsibility in the field. "Passing" inspectors
that do not have the necessary skills neither helps the inspectors nor does it further the EPA's goals of
having a qualified inspector staff. It also damages the training program's reputation, as inspectors will not
be able to do what the training program was supposedly preparing them to do.
As mentioned previously, exams can be used as evaluation tools. In this sense, they can determine both
instructor and course effectiveness.
As a final word about this phase of developing training, one should modify media as feedback indicates.
This feedback could be from the pilot study or from evaluations conducted after the course has been
implemented. Cost effective changes in media selection should be made as soon as possible.
19
-------
CONDUCT PILOT STUDY
The training group should determine whether or not to conduct pilot studies for media, sessions, or courses
on an individual basis before implementing them on a large scale. A pilot study is simply a dry run before
fully implementing new material. If the course is only going to be offered once the pilot study may be
excessive preparation for limited returns. However, if the course is going to be offered many times over a
long period of time, the pilot study would be very appropriate for checking to see whether there are any
problems with the media or the course before full implementation. In this case, it would tend to be more
cost effective to run a pilot study than to revise the educational materials once implemented.
The first step in conducting a pilot study involves finding a location to hold the study. The location should
be similar to if not the same as the actual facility where the course will be taught. Next, a representative
audience must be picked. The audience could consist of a group which would have received the training
anyway. In any case, the audience must mirror the actual trainees in regards to mix of gender, background,
job classifications, ages, etc., as much as possible. The pilot run should then be conducted.
The pilot test must be evaluated. The evaluation should include quantitative information, such as the results
of the final exams. It should also include qualitative data- the audiences feelings, opinions, and attitudes
about the course. Suggestions should be collected. Focus groups could be used to further delve into
problem areas.
Feedback from the pilot study should be incorporated as appropriate. The training materials should then
be revised to rectify any problems. If extensive problems were noted, another pilot study could be run to
ensure that everything works effectively.
COORDINATE ALL MEDIA AND FACILITIES TO PREPARE FOR TRAINING DELIVERY
Once the course is ready, the training group must focus upon actually bringing the training to the trainees.
Facilitating media communication involves finding presenters and trainers, finding facilities, determining when
to offer the course, determining how often to offer it, and removing any barriers which may prevent trainees
from experiencing the course. The training group should remember that just as it experiences resource
constraints, the trainees also have limits on available time, travel funds, and educational funds. Therefore,
every effort possible must be made to minimize the costs of training.
The presenters and trainers must be able to communicate to an audience. They should have both technical
expertise and also teaching skills. Slight deficiencies can be corrected if the potential trainer has otherwise
strong qualifications; however, a great number of deficiencies pose a barrier to trainer effectiveness. The
trainer usually will not need to be a world expert in both the technical and teaching fields; however, some
moderate degree of competency in both areas is requisite.
The facilities must be conducive to learning. There must be good lighting, resources necessary for the
media used (slide projector, overhead projector, etc.), adequate support facilities, comfortable seating, and
so on.
The timing of the course should be convenient for trainees. The literature suggest that the ideal time to train
is just before the skills and abilities are used on the job.18 That way, the real world experience can
reinforce the largely theoretical training materials. Courses should not be scheduled if there are known
conflicts with other events including key national seminars or religious holidays. Self study materials should
18 Troy, p. 19.
20
-------
be mailed quickly to the trainees. In order to deliver training as soon as possible, there should be sufficient
supply to meet the demand for the training.
Once again, feedback from evaluations should be used to determine whether any changes should be made.
Different facilities could be selected, different trainers and instructors could be used, other aspects of timing
could be watched, and other changes could be made.
TRAINING DELIVERY
Once it has reached this point, the course is ready for implementation. This is where all of the training
group's planning and preparation combine to meet the needs for the course. One should remember that
the course can still be.modified for the specific audience during delivery by using some evaluation
techniques. Hopefully, in session targeting will be minimized as the extensive preparation accurately
prepared the materials.
EVALUATION
As discussed earlier, the courses must be evaluated in order to determine whether or not they are meeting
their goals. If they are found to be missing the targets, the courses should be modified until they do achieve
their goals. Continuous evaluation techniques have been discussed more comprehensively in an earlier
portion of this paper and can be reviewed for further information.
21
-------
POSSIBLE EFFECTS OF TRAINING ON INSPECTION QUALITY
Effective training may improve inspection quality in a number of manners. First, it provides the inspectors
with ail of the requirements necessary for proper inspections. Quite simply, inspectors that do not know
either how or what to investigate are not likely to gather the necessary information. Training may also help
inspectors to prioritize their inspection activities, therefore enabling them to have the greatest impact for the
limited amount of time that they are able to inspect. Competent inspectors are much more likely to be able
to find the necessary depth and type of information to investigate a situation. Well educated inspectors are
more likely to be able to determine whether the sources are in compliance or noncompliance. They would
also be more likely to be able to determine the magnitude or extent of any violations. All of this may serve
to encourage the sources to attain states of compliance. This hypothesis is verified by the "Virginia Report,"
which determined that inspectors who were better trained in fact found more compliance violations. The
findings indicate that various sources' compliance statuses improved after the well trained inspectors found
more compliance violations.18
Training EPA inspectors may also improve Regional, State, and Local inspections, as these inspectors
frequently work together on teams. The competent EPA inspector can bring the necessary skills to the work
at hand and under review. The well-trained EPA inspector could prevent mistakes and improve the
inspection's quality if the other inspectors were not yet fully trained. The well-trained EPA inspector may
also serve as an on-the-job trainer by showing other inspectors areas that they may not have been aware
of. This would improve the other inspectors' abilities to conduct high quality independent inspections.
It is intellectually appealing to believe that after teaching someone how to do something, they would then
do the job better. Unfortunately, it does not seem to be that simple. One national training expert states
that:
A well-known but less practiced maxim is that post-training support is as important
as the instruction itself. New ways of acting or thinking must be reinforced in the work
setting if training is to affect business functioning according to strategic initiations.20
Unfortunately, a number of influences besides training may also diminish the inspectors' job performances.
If the only hinderance to high inspection quality is that an individual lacks the necessary knowledge and
skills, then effective training can rectify the problem. However, it appears that other factors may also hinder
job performance. Incentive systems which do not both establish priorities and reward high> work standards
may actually encourage inspectors to focus upon less beneficial activities, thus diminishing inspector
effectiveness. In addition, the incentive systems for the inspectors' supervisors may conflict with the
inspectors' priorities. This conflict may distract subordinates from their primary duties by encouraging them
to focus upon their supervisor's priorities rather than their own. Other influential factors may include a
supervisor's managerial skills, personal interest in the job, and peer support, to name a few. Rectifying
problems in these areas would require changes in the way that the Agency handles human resources.
One must develop a model in order to know exactly how all of these possible influences can affect
inspection quality. This requires first developing a preliminary model, possibly based on models
representing similar situations. One must then analyze the actual situation to verify that the model includes
19 Pedco.
20 Gherson, p.6.
22
-------
all possible influences on job performance. Next, actual data for the variables must be collected. This may
require creative attempts to quantify qualitative information. The data should then be incorporated into the
model and results obtained. The model should be revised until it predicts the real world fairly accurately.
There are many benefits to developing such a model. One can forecast the effects of changes in any
influential factor in order to determine how these changes may affect the end result. It allows the group to
monitor itself by providing a basis for comparison. The forecasted results can be compared with actual
results to determine whether or not they are in an acceptable range. It can predict training's impact on the
inspection quality of the trainee offices. It can be used as a basis for determining the appropriate size for
the training group and effort. The training group should not be too small, therefore unable to provide
requisite training, nor too large, therefore improperly allocating resources. It is important to note that the
group should not grow indefinitely, as the diminishing returns to scale could eventually damage the group's
performance. This is the same idea as in the phrase "too many cooks spoil the broth." Finally, by showing
the impact that effective training can have on inspection quality, the model enables the group to protect
itself. The model can justify requests for resource increases or can defend against reductions in resources
by forecasting those actions' effects.
23
-------
RECOMMENDATIONS FOR THE AIR INSPECTOR TRAINING PROGRAM
This section contains recommendations and suggestions for activities which either require further
investigation or implementation. Although the recommendations and suggestions are slanted for the Air
Stationary Source inspector training program to use in later phases of this project, their ideas may be
applicable to many situations. The recommendations should be considered carefully. While helpful, the
suggestions are not crucial to the training program's continued ability to function. One should also consider
previous comments made throughout this document as a targets for which to strive.
METHODS OF CONTINUALLY EVALUATING TRAINING PROGRAM EFFECTIVENESS
The current Air Inspector Training Program uses several methods to estimate training effectiveness. While
providing some feedback, the current processes may not present complete information.
Recommendations:
Implement additional quantitative methods of continually evaluating training effectiveness.
Perhaps the simplest method to implement would involve statistically analyzing data from
course examinations. The exams indicate not only trainee competencies but, when
statistically analyzed, they also indicate overall course and media effectiveness. As a word
of caution, the exams must test the course objectives in order to use them in evaluating
training effectiveness.
Another easily implemented quantitative evaluation technique would involve surveying
trainees 6 months after completing the course. This could be done through questionnaire
in order to minimize the resources required to complete this technique. The 6 month time
period appears to be as short as possible while providing enough time for the inspectors
to really use the training and determine its effectiveness and completeness. The 6 month
period is only an estimate- the appropriate time period should be verified before
implementing this method.
conduct a preliminary evaluation.
There will be a delay before results of newly implemented evaluation techniques can be collected
and analyzed. A preliminary evaluation would provide an overall picture of whether or not the Air
Stationary Source Compliance Inspector Program appears to be meeting its goals. Conducting a
quick evaluation will provide immediate feedback which can be used to prepare for possible results
of other evaluation techniques or to begin to correct likely problem areas.. Exhibit 9 describes a
method that could be used for a quick, goal-based evaluation.
Suggestions:
Implement other continuous training effectiveness evaluation techniques.
These techniques could include:
following inspector information, including information from exit interviews
24
-------
following inspection results
obtaining copies of complaints and compliments relevant to the trainees or the training
itself, and
following the demand for and opinion of this training programs courses versus competitive
programs' courses.
THE MODEL OF A TRAINING GROUPS OPERATIONAL CYCLE
Recommendations:
document specific objectives for educational efforts.
Quite simply, without a set of objectives, it is impossible to know what the program is supposed to
achieve, when it requires modification, when to start over, or even when the educational effort has
been successful.
promptly meet demand for all types of courses.
If demand is not met, chances are that the inspectors will attempt to meet their needs elsewhere.
Enough classroom courses must be scheduled and enough self study course materials must be
developed to meet the demand. In addition, other media of presentation could be investigated to
determine whether or not they would facilitate meeting this demand.
Suggestions:
consider using previously described methods of determining specific training objectives.
These methods could be used as follows:
It may be very beneficial for course designers to analyze appropriate portions of the Clean
Air Act Amendments of 1990 (CAAA '90) and their respective regulations in order to
determine new training needs, goals and specific objectives.
The critical incident technique could be very helpful in determining how to train individuals
to fulfill the permitting responsibilities under CAAA '90.
Employing CERCLA's method of verifying GS descriptions could help the department in two
ways: first, it would provide information about course needs and specific objectives and
second, it would update the inspector profile.
prioritize the objectives.
Determining both the frequency and the importance of the objectives not only helps in developing
the courses, but also facilitates the process of reprioritizing the objectives when needs change.
base end-of-course examinations on previously determined specific objectives of the
training effort.
This allows one to evaluate not only the trainees, but also the training media and the program itself.
25
-------
Trainees who fail to show the requisite competencies should review the problem areas until they can
demonstrate the necessary abilities. If the exam itself poses too intimidating a threat, other, more
friendly methods could be used as long as trainee competency is demonstrated. Achievement of
acceptable competency levels should be documented.
consider using pilot studies.
Pilot studies will probably be most cost effective for media, sessions, and courses that will be used
repeatedly over long periods of time.
OVERALL IDEAS
Recommendations:
Incorporate feedback from newly implemented evaluation techniques when designing
training.
Evaluation feedback should be incorporated into the training if the resulting gains would exceed
their implementation costs. If the feedback is ignored then the evaluation was not likely to have
served any useful purposes.
document specific, quantitative objectives of each training session, course, and program.
A set of specific objectives for each session, course, and program to achieve provides the
foundation requisite for first designing training efforts and later evaluating their effectiveness. These
objectives provide the course and program developers with set goals to strive for, to teach, and to
test. Checking for completion of the goals not only determines whether the students mastered the
objectives, but also indicates the effectiveness of various instructors, media, and the course design.
operate In the long run.
While organizations tend to respond only to issues as they emerge, it is also important to look
ahead for future trends. Some energy required to look to the future will be expended without any
tangible results. Despite this risk, planning ahead will allow the group to use the limited resources
more in the activities of greatest importance.
Suggestions:
develop a model describing various factors' influences on effective inspections.
This model will forecast the effects of changes in any of the factors influencing inspection quality,
including effective and appropriate inspector training.
obtain the following documents:
Mitchell, Garry. The Trainer's Handbook. USA: American Management Association, 1987.
Phillips, Jack J. Handbook of Training Evaluation and Measurement Methods. Gulf
Publishing Qompany: Houston, 1983.
26
-------
Troy, Kathryn L "Quality Training: What Top Companies Have Learned." Report No. 959,
ISBN No. 0-8237-0405-X. New York: The Conference Board, 1991.
Zemke, Ron and Thomas Kramlinger. Figuring Things Out A Trainer's Guide to Needs and
Task Analysis. Reading, MA: Addison Wesley Publishing Company, 1982.
27
-------
EXHIBITS
EXHIBIT 1
EVALUATION BY TRAINEES AT END OF COURSE
Course:
Date of course:
COURSE DESIGN
Was the course appropriately designed for someone with your background?
too advanced 1 2 3 4 5 too elementary
Were the course objectives clearly stated?
If they were stated, were they met?
If they were not met, which ones weren't?
yes
yes
no
no
FACILITIES
How were the facilities that the course was held in?
they met
my needs
conducive
to learning
1
they were
inadequate
distracting
COURSE CONTENT
perceived applicability of course to current work
right on 123
target
perceived usefulness of course to current work
. critical 1 2 3
Did you find the course to be stimulating?
fascinating 1 2 3
4 5 missed
the mark
not useful
deadly!
How challenging was the course?
too 1
challenging
not challenging
at all
28
-------
How were the videos, slides, or any other such materials?
clarified 1234
topics and ideas
How were the in class exercises?
very helpful 1234
confusing
or unclear
pointless
Did the course units follow a logical sequence?
Did later units build upon previous units?
Were the instructors' presentations well coordinated?
Were the training materials consistent?
yes
yes
yes
yes
no
no
no
no
TRAINERS
Were the trainers enthusiastic?
Did they seem to be knowledgeable about their subjects?
Did they appear to be prepared for their sessions?
Were they able to answer questions satisfactorily?
yes
yes
yes
yes
no
no
no
no
EXAMINATION
Did the examination test the course objectives?
Did the examination seem to be fair?
yes
yes
no
no
OVERALL
How was this course compared to all others that you have ever taken?
one of 12345
the best
Did the course meet your expectations?
If not, why not?
one of
the worst
yes
no
29
-------
EXHIBIT 2
EVALUATION BY TRAINERS AT END OF COURSE
Course:
Date of course:
TRAINERS MATERIALS
Did the materials coordinate with other units? yes no
Did the materials follow a logical sequence? yes no
Did they provide adequate coverage for you to teach the course?
yes no
Did they contain current issues, examples, etc.? yes no
Please comment on any suggestions that you may have for improving these materials
COURSE CONTENT
Was there appropriate timing for the units?
great timing 1 2
Which units required more time?
Which units required less time?
all the timing
was off
Was the information up to date?
What topics should be
added?
yes
no
omitted?
emphasized more?
30
-------
emphasizedess?_
What changes would make the course more effective?.
STUDENT QUESTIONS
How many of the questions were regarding clarification of the materials?
all 12345 none
Did the questions indicate that the material was too advanced?
IN CLASS EXERCISES
Did these appear to help the students understand the material?
yes no
yes no
Exercises that the students had difficulty with:
Were exercises difficult because of problems with the course material preceding the exercises?
yes no
FACILITIES
How were the facilities?
they met my 1
needs
conducive to 1
learning
they were
inadequate
distracting
Comments:
OVERALL
How was this course designed compared to others that you have taught?
one of the 1 2 3 4 5 one of
best the worst
Suggestions:
31
-------
EXHIBIT 3
DELAYED EVALUATION BY TRAINEES
Course:
Date that the course was taken:
I found this course-
helpful 1 2
to improve my 1 2
job performance
to be of 1 2
appropriate depth
if too shallow or too detailed, which?
3
3
4 5
4 5
not helpful
hurt my
job performance
too shallow '
or too detailed
The information and techniques in the course were
current 123
Overall, the course
met my 123
expectations
If it did not meet your expectations, why not?
out of date
did not meet
my expectations
What topics were most helpful for you?
What would have been more helpful for you?
Do you have any suggestions for new technologies to add to the course?
32
-------
How would you rate the course compared to all others that you have taken?
one of 12345 one of
the best . the worst
Have your job responsibilities changed since you took the course?
If so, please describe the changes:
yes
no
33
-------
EXHIBIT 4
DELAYED EVALUATION BY TRAINEES'SUPERVISORS
OR BY INSPECTORS SERVING AS ON-THE-JOB TRAINERS
Course:
Date that the course was taken:
Did the inspector's performance seem to improve after taking the course? yes no
Did the inspector who took the course seem to perform better than comparable inspectors who have not
taken the course? yes no
Did the results of the course fulfill your expectations?
if you answered "no" to any of the above, please explain:
yes no
Have the inspectors' duties changed since taking the course?
If so, please describe the changes:
yes no
Do you or your subordinates have any training needs that are not currently met? yes no
If no, please describe:
34
-------
EXHIBIT 521
A MODEL FOR A TRAINING GROUPS OPERATIONAL CYCLE
External to the Training Group
Legislation,
Regulations, £-
EPA Orders
Changes,
including
technological,
manufacturing, /
inspecting, \
new sources, and
scientific
discoveries
Resources ^-
, Within the Training Group
>
^
" Mission Statement
JderrtSy General
Training Needs and Goafs
Determine
Specific Objectives
-^ Determine Training Sequence
71
Coordinate Media and Facilities
for Training Delivery
Training Delivery
/K
Evaluation
4
-------
EXHIBIT 6
MISSION STATEMENT LEADING TO SPECIFIC COURSE OBJECTIVES
MISSION STATEMENT
To satisfy changing air training requirements in a timely, effective, and efficient manner in
order to further the Environmental Protection Agency's goals. This includes:
1) continuously searching for new training requirements
2) operating the process as efficiently as possible
3) complying with legislation, regulations, and EPA Orders that mandate training and associated
standards.
4) training EPA, Regional, and State personnel to investigate and manage source compliance with
legislation and regulations
GENERAL TRAINING GOALS AND NEEDS
training for evaluating air sources' compliance with the Clean Air Act Amendments of 1990 and
the resultant regulations.
materials to help prepare individuals to become inspectors' first line supervisors.
training to enable personnel to issue permits according to the Clean Air Act Amendments of 1990
POSSIBLE SPECIFIC OBJECTIVES OF A TRAINING COURSE22
to be able to determine whether sources emit 10 tons per year or more of a hazardous air
pollutant
to be able to determine whether sources emit 25 tons per year or more of a combination of
hazardous pollutants
to know where to obtain a current list of sources required to obtain permits
to know how to determine the current tonnage fee to be collected
to be able to complete 100% of the processing procedures for permit application
to be able to obtain a historical record of modifications that would revise previous legislation,
regulations, or standards
etc.
based on Pytte.
36
-------
EXHIBIT 7
CRITICAL INCIDENT SURVEY2
name:
office:
Please think of an incident that you have observed which reflects effective or ineffective job performance.
What was the objective?
What were the results?
What specific behaviors led to the results?
What attitudes, values, abilities, knowledge, and skills seemed to cause the success or failure?
What alternative behaviors could have improved the results?
23 based on Zemke. Figuring Things Out, p. 130.
37
-------
EXHIBIT 8
A BEHAVIORALLY ANCHORED RATING SCALE
WHICH COULD BE USED FOR BEHAVIORS EMPLOYED
IN OBTAINING A SOURCES COMPLIANCE DOCUMENTATION24
EFFECTIVE
BEHAVIORS
To request in a polite and non threatening
manner the source's compliance information
documenting the issues that the inspector
had planned to investigate.
MODERATELY
EFFECTIVE
BEHAVIORS
To demand that the source turn over all
of the compliance information documenting
the issues that the inspector had planned
to investigate.
Aggressively demanding the necessary
documentation, stating that the documents
will enable the inspector to discover how
the source is cheating, failing to obtain
all of the necessary documentation.
INEFFECTIVE
BEHAVIORS
Failing to obtain any of the source's
compliance documentation necessary for
completing the investigation.
based on Zemke. Bguring Things Out, p. 130.
38
-------
EXHIBIT 9
A QUICK METHOD OF EVALUATING THE
AIR STATIONARY SOURCE COMPLIANCE INSPECTORS TRAINING PROGRAM
This document outlines one method of evaluating the Air Stationary Source Compliance Inspectors' Training
Program (ASSCITP) as provided by the Air Pollution Training Institute (APTI) at a point in time. The
evaluation will determine whether or not the Program is meeting its goals, and, if not, the areas that would
require modification. One should note that this should not be the only evaluation of the program- it should
only be used as a mechanism to determine areas which may cause problems so that planning can occur.
Data from other, more comprehensive evaluation techniques should be obtained to verify and extend the
quick evaluation's results.
Unfortunately, there are no set models to follow in formulating this type of an evaluation. One training
expert, Elizabeth Hawthorne, reviewed all of the training program evaluations published between 1966 and
1984 in order to provide broad models for later evaluations to follow. She found that:
Few of the evaluations conducted and reported in the literature were formative evaluations,
that is evaluations that are intended to modify and improve the program in process.25
Instead, most of the evaluations were used to determine whether to continue or discontinue the training
program in question.26 Laurel Sneed, a nationally recognized consultant specializing in training programs,
agrees with this observation27 While the training industry recognizes the need for formative evaluations,
there are no proven methods of conducting these evaluations. Each evaluator must determine an
appropriate path from scratch.
The process chosen for this evaluation of the ASSCITP involves three steps:
1)
2)
3)
Determining the program's broadest goals
Investigating the ASSCITP training deliverables
If possible, delving into the discrepancies between the ideal and actual deliverables.
This method was chosen because it provides a basis for comparison; quite simply, one cannot evaluate an
object, process or idea without comparing it against a standard. For example, if one were asked whether
or not slide rules were quick calculating devices, the answer would depend on what slide rules were being
compared to. They are fast when compared to working the problem with pencil and paper, but are slow
when compared to using a calculator. In the same manner, the interpretation of the training program
evaluation's results depends on what is used as the basis for comparison.
The EPA Order 3500.1 should be this evaluation's basis for comparison for two reasons. First, that Order
now drives the training program. Second, this project is a screening mechanism; therefore, it should deal
25 Hawthorne, p.26.
26 ibid.
27 Sneed.
39
-------
with the overall program rather than focusing on more specific purposes. The EPA Order 3500.1 provides
the most general and far reaching goals for the Program.
These goals could be determined both by examining appropriate documentation and also by interviewing
key personnel. Specific goals stated in both the EPA Order 3500.1 and corresponding documentation
should be noted. Interviews with personnel involved in creating the Order will verify that all goals had been
properly identified. Ideally, these interviews should be in person rather than by telephone in order to
facilitate communication between the interviewer and the interviewee. However, these interviews could also
be conducted over the telephone without critically affecting the quality of the information obtained if resource
constraints prohibit the use of personal interviews.
A similar process may be used to complete the next step- that of investigating the actual ASSCITP
deliverables. This investigation should focus on those products which directly relate to the Order's goals.
This process involves reviewing program documentation and interviewing appropriate personnel. The
documentation review should include general descriptions of the program as well as the results of previous
program evaluations. Interviews could be conducted both by telephone and in person. Again, personal
interviews should be conducted whenever possible; telephone interviews may substitute for personal
interviews when necessary. The interviewees could include trained inspectors, regional contacts, trainers,
and personnel in the APTI. Ideally, the trained inspectors will have completed all of their training so that they
can reflect back upon the entire program. The inspectors and trainers should be able to provide insight into
a relatively small picture; that of their own experience. Regional coordinators should provide a much
broader view of the actual program deliverables. Training program designers should provide their views of
the goals as well as the process that they use to meet those goals.
Once any discrepancies between the stated goals and actual results are identified, these may be further
examined. As the discrepancies have not yet been uncovered, one cannot say with certainty how best to
further scrutinize them. However, one may speculate that interviews and document reviews would be
suitable. Interviewees could include inspectors' first line supervisors, inspectors yet in training, trainers, and
well as additional persons in the groups previously interviewed.
In summary, the evaluation will consist of up to three steps. These will all involve interviews with persons
knowledgeable about specific areas, and may involve reviewing pertinent documentation The first activity
will involve determining the goals of the EPA Order 3500.1. Next, the actual training products will be
determined. Finally, any discrepancies between the two will be investigated further as time permits
40
-------
BIBLIOGRAPHY
Air Compliance Inspector Training Informal Advisory Group Members. Memorandum from Seitz, John S.,
Director, Stationary Source Compliance Division, Office of Air Quality Planning and Standards, U.S.
Environmental Protection Agency: May 26, 1989.
Air Pollution Training Institute. "Student Course Critique."
Air Pollution Training Institute. "Instructor's Course Summary." R 12/86.
Air-Specific Inspector Training Implementation Plan for Headquarters Stationary Source Compliance Program
(OAQPS/OARVNarrative. Memorandum from Seitz, John S., Director, Stationary Source
Compliance Division, Office of Air Quality Planning and Standards, U.S. Environmental Protection
Agency: February 17, 1989.
Barclay, Becky, Office of Compliance Analysis and Program Operations, Environmental Protection Agency,
Washington D.C. Personal Interview. June 14, 1991.
Carnevale, Anthony P. and Eric R. Schulz. "Return on Investment: Accounting for Training." Training &
Development Journal. Vol. 44, No. 7: 41-72.
Centifonti, Gary J. and Richard M. Ritota. "New Jersey Cleans Up Its Permit Process." Asbestos Issues.
Vol. 4, No. 5: 50-53.
Compliance Policy and Planning Branch, Office of Compliance Analysis and Program Operations, Office of
Enforcement. Building the Enforcement Infrastructure: Compliance Inspector Training. 10/90.
Compliance Policy and Planning Branch, Office of Compliance Analysis and Program Operations, Office of
Enforcement. Report on Regional Status of Compliance Inspector Training. 10/90.
Evaluating Business and Industry Training. Ed. Leslie S. May, Carol A. Moore, Stephen J. Zammit, Kluwer
Academic Publishers: Boston, 1987.
Fletcher, Donna, Office of Cooperative Environmental Management, Environmental Protection Agency,
Washington D.C. Personal Interview. June 13, 1991.
Gherson, Diane J. and Carol Anne Moore. "The Role of Training in Implementing Strategic Change."
Evaluating Business and Industry Training. Ed. Leslie S. May, Carol A. Moore, Stephen J. Zammit,
Kluwer Academic Publishers: Boston, 1987, 3-18.
Hanlin, James R. and Nancy J. Johns. "Championship Training." Training & Development Journal. Vol. 45,
No. 2: 57-62.
Hannum, Wallace and Carol Hansen. Instructional Systems Development in Large Organizations.
Englewood Cliffs: Educational Technology Publications, 1989: Ch. 13.
Hawthorne, Elizabeth M. Evaluating Employee Training Programs. New York: Quorum Books, 1987.
41
-------
Inspector Training and Development Workgroup. Inspector Profile. U.S. Environmental Protection Aaencv
July, 1987. a y'
Jean, Paul, Office of Human Resources Management, Environmental Protection Agency, Washington DC
Personal Interview. June 21, .1991.
Kovalick, Walter W. Jr., Ph.D., Director, Technology Innovation Office, Environmental Protection Agency.
Memorandum, Directors Waste Management Division, Environmental Services Division, Emergency
& Remedial Response Division, Hazardous Waste Management Division, Toxic and Waste
Management Division, Hazardous Waste Division: Dec. 20, 1990.
Lai, Larry. Project Manager, SAIC. McCleaton, VA. Personal Interview. June 24, 1991. :
Mayo, C. Douglas and Philip H. DuBois. The Complete Book of Training: Theory. Principles, and
Techniques. San Diego: University Associates Inc., 1987, 32.
Mitchell, Garry.- The Trainer's Handbook. USA: American Management Association, 1987.
Nadler, Leonard. Designing Training Programs, the Critical Events Model. Reading, Mass: Addison-Wesley
Publishing Company, 1982.
Office of Air and Radiation, Office of Air Quality Planning and Standards. Air Specific Inspector Training Plan
for the Stationary Source Compliance Program. SSCD SR-89-002. U.S. Environmental Protection
Agency: February 1989.
Office of Cooperative Environmental Management. EPA Inspector Profile. Draft. 9/10/89.
OSC/RPM Basic Training Academy. "Session A- Agenda." 6/90.
OSC/RPM Training Academy. "A Comprehensive Program For Incident/ Site Management."
OSC/RPM Training Academy. "Overall Objectives for Session B." '
Phillips, Jack J. Handbook of Training Evaluation and Measurement Methods. Gulf Publishing Company-
Houston, 1983. :
Pedco, for Stationary Source Compliance Division, Environmental Protection Agency. Development of Pilot
Inspection System for Virginia State Air Pollution Control Board. EPA 340/1 -83-021: 1983.
Polvi, Gary, Office of Water Enforcement and Permits, Washington, D.C. Personal Interview. June 17,1991.
Pytte, Alyson. "Clean Air Act Amendments." Congressional Quarterly. 11/24/90, Vol. 48, No. 47: 3934-63.
Radian Corporation, for Regional Operations Branch, Environmental Protection Aaencv. Survey of State and
Local Air Pollution Agencies' Activities and Costs. EPA-450/2-90-006: February 1990.
Rassett, Allison. Training Needs Assessment. Techniques in Training & Performance Development Series.
Englewood Cliffs, NJ: Educational Technology Publications, 1987. 133-55.
Rae, Renelle, Office of Human Resources Management, Washington, D.C. Personal Interview June 17
1991. ;
42
-------
Rtfkin, K. I., R. L Dueker, W. F. Diggins, F. C. Foss, and Michael Senew. Task Analysis of State and Local
Air Pollution Control Agencies and Development of Staffing Guidelines. Volume C: Detailed Task
Data, and Staffing Guidance FIELD ENFORCEMENT. U.S. Environmental Protection Agency, Office
of Air Programs, Manpower Development Staff: November 1972.
Rogoff, Rosalind L The Training Wheel. New York: John Wiley & Sons, 1987.
Schwartz, Charles G. "Training Providers Look to New Challenges.' Asbestos Issues. Vol. 4, No. 5:16,18,
59.
Seitz, John S., Director, Stationary Source Compliance Division, Office of Air Quality Planning and Standards,
Environmental Protection Agency. Memorandum, Air-specific Inspector Training and Implementation
Plans for the Stationary Source Compliance Program: February 17, 1989.
Seitz, John S., Director, Stationary Source Compliance Division, Office of Air Quality Planning and Standards,
Environmental Protection Agency. Memorandum, Air Compliance Inspector Training Informal
Advisory Group: May 26,1989.
Siegler, Mark S. and Kirk Foster. Stationary Source Compliance Inspector Training Program. U.S.
Environmental Protection Agency, Stationary Source Compliance Division.
Smith, W. Douglas, Senior Compliance Investigator Engineering and Inspection Section, EDO.
Memorandum, "Draft" Inspector's KISS Manual: June 27, 1991.
Sneed, Laurel, President, Apprend Associates, Research Triangle Park, NC. Personal assistance. May 24,
June 7, July 9 1991.
Stationary Source Compliance Division. Interim Guidance on the Enforcement Provisions of the Clean Air
Act Amendments. U.S. Environmental Protection Agency: October 19, 1990.
Thomas, Debbie, Office of Waste Programs Enforcement, Crystal City, VA. Personal Interview. June 17,
1991.
Townsend, Ron, Office of Air Quality Planning and Standards, Durham, NC. Personal Interview. June 27,
1991.
Tracey, William R. Designing Training and Development Systems. Revised Edition. New York: American
Management Associations, 1984.
Training and Development Journal. Vol. 43, No. 10.
Training and Development Journal. Vol. 44, No. 2.
Training and Development Journal. Vol. 44, No. 7.
Training and Development Journal. Vol. 45, No. 2.
Troy, Kathryn L "Quality Training: What Top Companies Have Learned." Report No. 959, ISBN No.
0-8237-0405-X. New York: The Conference Board, 1991.
43
-------
Zemke, Ron. "Measuring the Impact of Management Development." Designing and Delivering Cost-Effective
Training and Measuring the Results. Ed. Ron Zemke, Linda Standke, and Philip Jones, Minneapolis,
MN: Training Books, 1981. 267-268.
Zemke, Ron. "Are Sales Training Results Measurable?" Designing and Delivering Cost-Effective Training
and Measuring the Results. Ed. Ron Zemke, Linda Standke, and Philip Jones, Minneapolis, MN:
Training Books, 1981. 269-272.
Zemke, Ron and Thomas Kramlinger. Figuring Things Out 4 Trainer's Guide to Needs and Task Analysis.
Reading, MA: Addison-Wesley Publishing Company, 1982.
44
-------
TECHNICAL REPORT DATA
(Please read Instructions on the reverse before competing/
1. REPORT NO.
RECI
4. TITLE AND SUBTITLE ,r-kirroo
DEVELOPING A SYSTEM FOR EVALUATING THE EFFECTIVENESS
OF INSPECTOR TRAINING
Phase 1: Identification of Evaluation Techniques and Development
of Model for Training Program Operations
7. AUTHOR. PERFORMING ORGANIZATION CODE
8. PERFORMING ORGANIZATION REPOI
Stationary Source Compliance Division
US EPA (EN-341W)
401 M Street SW
Washington, DC 20460
11. CONTRACT/GRANT NO.
U-913530-01-0
12. SPONSORING AGENCY NAME AND ADDRESS
Stationary Source Compliance Division
US EPA (EN-341W)
401 M Street SW
Washinaton, DC 20460
13. TYPE OF REPOR
Final Draft
14. SPONSORING AGENCY CODE
15. SUPPLEMENTARY NOTES
Project Officer: Mark Siegler FTS: 398-8673 Comm.#: (703) 308-8673
Stationary Source Compliance Division, US Environmental Protection Agency
16. ABSTRACT
This study was performed under the NNEMS Grant Program for the Stationary Source Compliance Division
of the US Environmental Protection Agency. The study was initiated in order to identify general approaches
and techniques for evaluating inspector training program effectiveness.
This document discusses the following:
methods of evaluating training appropriateness and effectiveness
operational activities for developing and delivering effective, targeted training
possible effects of training on inspection quality, and
recommended and suggested courses of action.
The research was conducted through literature searches and interviews. Only practical techniques have
been described in the document. These include both well established methods as well as newer
procedures. It was noted that formative evaluations have no standard formulas to follow; they must be
tailored to each unique situation. Although the document frequently refers to the EPA's Air Stationary
Source inspector training program, the ideas and techniques are applicable to many other programs.
7.
KEY WORDS AND DOCUMENT ANALYSIS
DESCRIPTORS
b.lDENTlFIERS,OPEN ENDED TERMS
COSATI Field.'Group
Training Development
Inspector Training
Training
Training Effectiveness
Training Evaluations
18. DISTRIBUTION STATEMENT
Release Unlimited
jnclassified
-45-
20. SECURITY CLASS , Tim page I
i Unclassified
22. PRICE
EPA Form 2220-1 (Rev. 4-77) PREVIOUS EDITION is OBSOLETE
-------
------- |