Results of Benchmarking Study
                      for
       Environmental Protection Agency
        Air Pollution Training Institute
               FINAL REPORT
                        January 7, 2005
                            Submitted by:
                            Hay Group
                            4301 N. Fairfax Drive, Suite 500
                            Arlington, VA 22203
                            (703) 841-3100
roup

-------
         X,        UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
o vx-ivy *                 RESEARCH TRIANGLE PARK, NC 27711
%;
                                    February 7, 2005
                                                                            OFFICE OF
                                                                        AIR QUALITY PLANNING
                                                                          AND STANDARDS
 Dear STAPPA/ALAPCO Joint Training Committee:

        The EPA's Office of Air Quality Planning and Standards has conducted a benchmark

 study of its training program. This study is part of our effort to improve the overall value and

 effectiveness of the program. The report entitled "Results of the Benchmarking Study" offers

 insights as to what others in the industry of training are doing and recommends a number of

 quick fixes and long-term changes to the current training program. At the upcoming

 STAPPA/ALAPCO Joint Training Committee Meeting of March 2005 in Florida, we will use

 the "Results of the Benchmarking Study" as the framework for further discussions on the training

 program. We are looking forward to your comments  and suggestions to help us focus and

 strengthen the program.

                                              Sinaerely,
                                                         Harnett
                                                   Director
                                        Information Transfer and Program
                                               Integration Division
 Enclosure

-------
                    TABLE OF CONTENTS
I. INTRODUCTION.	1
  Background	1
  Methodology	1
  Report Overview	4
II. OPTIONS FOR IMPROVING APTI's NEEDS ASSESSMENT PROCESS	5
///. OPTIONS FOR IMPROVING APTI's COURSE DESIGN AND UPDATING PROCESS	9
IV. OPTIONS FOR IMPROVING APTI's TRAINING DELIVERY METHODS	16
V. OPTIONS FOR IMPROVING APTI's TRAINING EVALUATION PROCESS	24
VI. NEXT STEPS.	28
APPENDIX A  Task 1 Report: Evaluation of Current Air Polution Training Program.. 33
APPENDIX B Task 2 Report: Best Practices of Leading Training Programs	60
APPENDIX c Summary of Quick Fix Recommendations.	128
HayGroup

-------
EPA: Results of Benchmarking Study
                                /.  Introduction

Background

In May 2004, the Education and Outreach Group (EOG) of the Environmental Protection
Agency (EPA) contracted with the Hay Group to conduct a benchmarking study of its Air
Pollution Training program.  The ultimate goal  of the project was to provide options for
improving the current Air Pollution Training program, and ultimately, EOG's business
performance. The following three tasks were aimed at accomplishing this goal:
       >   Evaluate the current Air Pollution Training program (Task 1)
       >   Identify best practices of leading training programs (Task 2)
       >   Develop a  plan  of action for improving the current Air Pollution Training
          program (Task 3)

This final report  provides options  for improving the current Air Pollution Training
program and next steps for doing so.  Throughout this report, we present our specific
conclusions  and recommendations  regarding  the direction that we believe  the Air
Pollution Training Institute (APTI) should take to improve its training program.

The options  and recommendations described in this report are based on our evaluation of
the current Air Pollution Training program and the best practices of training  industry
leaders that we identified.  We present options for narrowing the gap between the current
APTI training program and industry leading training programs.
Methodology

This section provides a high-level  summary of the  methodology  used during the
benchmarking study.

Review of Current APTI Training Program

The first phase in the benchmarking study was  to evaluate the  current Air Pollution
Training program.  Hay interviewed a total of 22 individuals (in person or by telephone)
who have a good understanding of the current training program, its objectives and desired
results, its future direction, and overall effectiveness.  Additionally, we spoke to several
other individuals in  group  settings  about the Air  Pollution  Training program.  The
following types of individuals were interviewed:

    •  Members of the STAPPA/ALAPCO Joint Training Committee (JTC)
    •  Members of the MARAMA Regional Consortium
    •  Individuals who run area training centers
    •  APTI course instructors
    •  Supervisors in state and local agencies who have sent employees to APTI training
       courses
HayGroup                                                                  Page 1

-------
EPA: Results of Benchmarking Study
    •   Contractors who  design  APTI course content  and conduct the training needs
       assessment
    •   APTI management and staff

The interviews were aimed at gaining an understanding of the practices, procedures and
processes currently used by APTI to  design  and deliver  air pollution training, and
determining the effectiveness of the APTI training program  in meeting the  needs of air
professionals and other stakeholders. We focused on evaluating four key aspects of the
current training program:  1) needs assessment process; 2) course design and content;  3)
training delivery methods; and 4) training evaluation process.

In  addition to conducting  subject matter  expert interviews, Hay reviewed several
documents, Web sites, and APTI training materials as part of the evaluation of the current
APTI training program. More specifically, we reviewed the following:

    •   APTFs Web site  (e.g., course  schedules,  training providers, APTFs mission,
       course registration)
    •   Site Coordinators  Resource Center Web site
    •   EOG FY03 Highlights and FY04 Midyear Accomplishments
    •   Training Needs Assessment Survey and results
    •   Training evaluation instruments and results
    •   Training  materials (self-instructional  workbooks,  Web-based training courses,
       satellite broadcasts)
    •   Statistics on training course attendance and certificates issued

Hay summarized the information collected during the subject  matter interviews and
document/Web site reviews and provided an evaluation of the current APTI training
program. The findings and conclusions are reported in the Task 1 Report: Evaluation of
Current Air Pollution Training Program, which can be found in Appendix A.

Identification of Best Practices  of Industry Leaders in Training

The second phase in the  benchmarking  study was  to conduct site visits and a literature
review to identify best practices of industry leaders in training, particularly in the area of
distance  learning.   The best  practices  were  identified based  on  site  visits with
organizations with innovative training programs, as well as a literature review of current
training trends and best practices.

Site Visits.  Hay conducted  site visits  with  five organizations recognized as industry
leaders in technical training  and distance learning.  We identified these organizations
based on awards (e.g., Training Magazine's Top 100 list, American Society  for Training
and Development BEST Award,  Government e-learning award), accolades, participation
in other training benchmarking studies (e.g., Society for Human Resource Management
Consortium Benchmarking  Study   on  Training and  Development),   and  expert
recommendations.  We  particularly  looked  for organizations that  use innovative  e-
learning  and distance learning  training  delivery methods. Table  1  shows the  five
HayGroup                                                                   Page 2

-------
EPA: Results of Benchmarking Study
organizations that agreed to participate in the benchmarking study and our rationale for
selecting each of them as a benchmarking partner.

Table 1: Participating Organizations in the Site Visits
Organization
Occupational Safety
and Health
Administration
(OSHA), Office of
Training and Education
Center for Disease
Control (CDC), Public
Health Training
Network (PHTN)
Georgia Tech
University, Distance
Learning and
Professional Education
Department (DLPE)
SAS Institute (SAS)
GMAC Commercial
Mortgage (GMAC),
Staff Development
Division
Rationale for Selection as a Benchmarking Partner
> Provides technical training to a large, geographically-dispersed
population
> Blends Web-based training and live satellite broadcasts with more
traditional classroom instruction (uses innovative distance learning
training media)
> Provides technical training to a large, geographically-dispersed
population
> Provides diverse training delivery methods, including distance learning
> Has state-of-the-art training facilities
> Trains professionals in engineering, business, and other hard sciences
> Recognized as providing an exceptional distance learning program and
successfully incorporating an interactive component into distance
learning
> Has state-of-the-art training facilities
> Provides extensive technical training to customers around the world
> Blends e-learning with more traditional classroom training (uses
innovative e-learning training media)
> Delivers training primarily via e-learning methodologies, including
videoconferencing, live Webcasts, videotapes and C-ROMs/DVDs
The site visits focused on identifying best practices around:
       >  The needs assessment process
       >  Course design and content
       >  Training  delivery methods,  with a special  emphasis  on distance  learning
          methods
       >  Strategies for incorporating an interactive component into distance learning
       >  Training evaluation process

During site visits, we reviewed training program documentation and materials, observed
training facilities and training programs (e.g., a live Webcast, an interactive Web-based
course), and  conducted interviews with members of the training group.  Each site visit
lasted two to  six hours.
HayGroup
Page3

-------
EPA: Results of Benchmarking Study
Literature Review.  In addition to conducting site visits, Hay reviewed several reports,
articles, books and Web  sites to identify best practices in e-learning  and distance
learning. The primary documents/Web sites that were reviewed are identified in the Task
2 Report: Best Practices of Leading Training Programs., which can be found in Appendix
B.

Summary of "Best Practices". Hay summarized our findings from the site visits and
literature review to identify trends  and best practices  of industry  leaders in training.
These findings are also presented in Appendix B.

Discussions with APTI about Task 2 Report.  After completing the Task 2 report, we
met with APTI management and staff to obtain their input into those best practices that
are most cost effective and feasible to implement at APTI.  We facilitated a discussion
with APTI around the following:

       >  Extent to which the best practices can be incorporated into APTFs culture,
          processes, and procedures

       >  Whether APTI  has the resources (e.g., dollars, staff) to  implement the best
          practices

       >  Barriers that may impede successful implementation of the needed changes to
          the program

Information collected  during these meetings  served as additional input for determining
the options for improving APTI's training program,  as well as next steps for moving
APTI forward.
Report Overview

In this report, we highlight where we  see the biggest gaps between  APTI's current
training program and the ideal training program (one that incorporates best practices of
industry leaders).  We also provide options and recommendations for making APTI more
successful, and next steps for doing so.  We provide options for improving the following
aspects of APTI's training program:
       >  Training Needs Assessment Process
       >  Training Design and Content
       >  Training Delivery Methods
       >  Training Evaluation Process

Throughout this report, we discuss options that we believe will have the biggest impact
on the overall effectiveness of APTI. We recognize that some of these options are long-
term.  However, we also present some "quick fix" suggestions or "low hanging fruit" for
immediately improving the Air Pollution Training program.  A summary of all "quick
fix" suggestions can be found in Appendix C.
HayGroup                                                                  Page 4

-------
        EPA: Results of Benchmarking Study
                //. Options for Improving APTFs Needs Assessment Process

        The benchmarking study revealed that APTFs current needs assessment process does not
        provide  sufficient or the  "right" types  of information  to  make  decisions  about the
        direction APTI should take to meet the needs of its most critical customers.  To become a
        more effective  training  program, APTI should  expand its  current needs  assessment
        process beyond simply predicting future attendance at current classroom courses, and
        incorporate a competency/skill-based assessment in this process.

        Table 2  highlights the current state of APTI's needs assessment process compared to
        what industry leaders are doing  in this  area,  and presents options for creating a  more
        value-added training needs  assessment process.

        Table 2:  Gaps in Needs Assessment Process
  Current APTI Training Program
 Industry Leader Training Program
 Options for Making APTI an Industry
	Leading Training Program	
Needs assessment primarily focuses on
determining whether air professionals
are likely to attend existing APTI
classroom courses. There is no
competency/skill component of this
assessment.

There has not been a recent broad-
based needs assessment that takes a
strategic look at whether current
courses/offerings and training delivery
methods meet the needs of air
professional and other key customers.

Needs assessment data are not
consistently used to develop and update
courses and offerings.
A competency/skill assessment is
included as part of the needs
assessment process. That is, methods
are used to identify the
competencies/skills required for
success on the job and gaps in the
competency/skill level of the trainee
population.

The needs assessment process is a
critical source of information for
making decisions about training content
and delivery methods. For example,
whether current courses should be
updated or eliminated or new courses
should be designed.

Decisions about courses and offerings
are communicated  to the trainee
population.
As a first step, conduct a broad-based
needs assessment to determine the future
role and scope of APTI, including
courses/offerings and delivery methods
most needed by air professionals and
other customers.  This assessment should
include a competency/skill component.

Continue to conduct needs assessments
on a regular basis. Make decisions about
the frequency and breadth of future needs
assessments.

Use needs assessment results to refine
APTI's training program (e.g., decisions
about which courses to maintain or
update, new courses to design, delivery
methods best suited for course content),
and communicate key decisions to the air
pollution community.
        Below we further discuss our recommended options for enhancing the needs assessment
        process at APTI.

            1.  Conduct a  Broad-based Needs Assessment to Redefine  APTI's Role and
               Scope

        We recommend that APTI conduct a broad-based needs assessment in the near future to
        determine the extent to which  it is providing the right mix  of courses  and offerings to
        meet the needs of its most critical customers.  This is particularly important because  we
        found  that some key customers/stakeholders  who  were interviewed in Task  1  do not
        believe that APTI is currently meeting the needs of state and local agencies.
        HayGroup
                                                      Page 5

-------
EPA: Results of Benchmarking Study
This needs assessment will help to define APTFs key customer contingency, what types
of services (e.g., outreach, technical training) APTI should be providing to its customers,
what courses should be  delivered, and how courses should be delivered (e.g., live Web,
satellite,  classroom).  Ultimately,  this needs assessment  will help APTI to define its
mission and scope so that it can be successful at meeting the most critical needs of its
customers. This mission should be communicated to all APTI staff, as well as external
customers/stakeholders,  so that everyone has a good understanding of APTI's role and
scope.

The  current needs assessment  process focuses on  identifying those current classroom
courses that air professionals are likely to attend in the future, and solicits input about
potential  new training topics. While this approach provides some useful information for
planning  future APTI courses/offerings, it does not provide direct insight into whether the
current courses are teaching the skills/competencies required by  air professionals (and
other customers), or whether new courses should be developed to meet air professional
training needs.  Additionally, the current needs assessment provides limited information
for determining the best  way to deliver training to air professionals.

The   broad-based   needs  assessment  should   solicit   input  from  key   APTI
customers/stakeholders on:
       >  The competencies/skills  required  for  air  professionals (and  other  key
           customers) to be successful now and in the future
       >  The current  competency/skill level of air professionals  and key gaps  from
           those required for success
       >  Whether critical competencies/skills can be obtained  through other sources
           than APTI (e.g., universities)  or APTI courses/offerings should focus on
           developing these competencies/skills
       >  The most effective  ways to deliver training to air professionals  (based on
           considerations such as cost, convenience, learning ability/style)

Information collected from the needs  assessment process  should be used  to review
current APTI courses and offerings to determine  which should be retained as is; which
should be updated; which should be eliminated; and what new courses/offerings should
be developed. Additionally, this information will help APTI make decisions about the
best method for delivering different types of courses/offerings to  air professionals  (e.g.,
should APTI make  more  use of live Web-based training?).   Finally,  and  most
importantly, this broad-based needs assessment will provide APTI  with the information it
needs to  redefine its vision, role, and scope so it is meeting the needs of its most critical
customers. That is, what types of training and outreach should APTI be delivering and to
whom? By limiting its  scope to those courses/offerings that are most critical for success
on the job, APTI will  be able to  focus its limited resources on  ensuring  that these
courses/offerings are of the highest quality and meeting the needs of air professionals and
other key customers.
HayGroup                                                                    Page 6

-------
EPA: Results of Benchmarking Study
Once APTI conducts the needs assessment and can prioritize the courses and offerings it
provides, it will be easier to implement many of the other recommendations that will be
discussed in this report.  For example, by reducing the number of courses/offerings APTI
provides to those that are most needed by air professionals, it will be less  challenging to
conduct systematic reviews and updates on a periodic basis of all APTI courses/offerings.

                 Tips for Making the Needs Assessment a Success

•  Create a detailed  plan  for conducting the needs assessment,  including the study's
   objectives, timelines, individuals to participate in  the  assessment,  types of  data
   collection methods, how data will be used, who will be accountable for conducting
   the needs assessment, etc.

•  Publicize the needs assessment throughout the air pollution community (through the
   Internet, JTC  and site coordinators, newsletters, etc.).  Communicate that APTI is
   making an effort to identify the needs of its most critical customers and redefine the
   Air  Pollution Training  program to  meet these needs.   This will  also increase
   participation in the needs assessment process now and in the future.

•  Collect data from multiple sources,  including subject  matter experts,  current and
   potential  trainees,  supervisors  of trainees,  air directors,  course instructors  and
   developers, among others.  Sources should have insight into the competencies/skills
   required for the success of air professionals (and other key customers), the current
   competency/skill level in the air professional population, types of training available
   outside of APTI, and so forth.

•  Consider using multiple data collection  methods, such as  reviews  of position
   descriptions, SME panels, interviews, and short, targeted surveys with a small sample
   of air professionals and other customers/stakeholders.

   2.  Continue to Conduct Regular Needs Assessments

Recognizing that the broad-based needs assessment is a one-time or very infrequent
activity, APTI (in conjunction with the JTC) should continue to conduct regular (every 1-
3 years) needs assessments to keep a pulse  on whether it is meeting air professional and
other customers'  needs  and whether any new training  needs arise.  This  will require
making modifications to  the current  needs assessment instrument  (e.g., to include
competencies/skills; the  desire for different training delivery methods). The broad-based
needs assessment process may  help  shape the future  needs assessment instrument.  It
should be noted that a comprehensive competency/skill-based needs  assessment  may
need to be conducted on a less frequent basis, potentially every three to five years.
HayGroup                                                                   Page 7

-------
            EPA: Results of Benchmarking Study
I.
We recognize that completing a broad-based needs assessment will take approximately five to six
months.  In the short-term, APTI should perform a high-level review of its current and planned
courses/offerings to make "quick" improvements. This could be accomplished through meetings
with APTI staff, informal discussions with the JTC and representatives from state, local, and
tribal agencies, and reviews  of prior needs assessments and training evaluations.  Information
collected during this  initial review could be used to make immediate improvements to APTFs
training program.  For example, this  review  will  provide input into decisions about those
courses/offerings that should be:

    >   Updated first (highest priority for updating)
    >   Enhanced through incorporating  more opportunities for  interaction,  more  visually
        stimulating graphics, and so forth

    >   Transferred to a new medium or technology
    >   Shelved because the material is no longer relevant or up-to-date
    >   Retooled into  a  blended delivery  approach  to minimize the amount of time air
        professionals have to spend away from the job

In the  short-term, APTI should focus on  improving those  courses/offerings that meet its core
mission and fulfill the immediate needs  of air professionals and other key customers.  For
example, APTI does not want to waste valuable resources updating a course if it is not viewed as
currently filling an important need.

Note:  This initial "quick" review has a more tactical focus - making immediate changes and
improvements  to APTFs training program.  The broad-based needs assessment can help validate
the information collected during this initial review, and strategically shape APTFs future mission
and scope.	
            HayGroup
                                                                                 PageS

-------
        EPA: Results of Benchmarking Study
         ///. Options for Improving APTFs Course Design and Updating Process

        Based on the results of the benchmarking study, it appears that the APTI training group
        may not encompass all of the key roles required to effectively design and update training
        and outreach courses/offerings.  The  results of a future broad-based needs assessment
        will provide APTI  with  critical information for determining whether its current training
        group should  be supplemented with  other key roles in  order to design  the  types of
        courses/offering needed by air professionals and other customers.

        Additionally, APTI does not currently apply  consistent course  design standards  and
        procedures and does not incorporate many of the industry best practices (e.g., use script
        writers for live broadcasts; require rehearsals for all live broadcasts) into its course design
        process. Finally, there is no systematic process  or timeline for reviewing and updating all
        existing courses/offerings, regardless of the media in which they are delivered.

        Table 3 highlights the current state of APTI's training design/updating process compared
        to what industry leaders are doing in this  area,  and presents options for creating a more
        effective course design and updating process.

        Table 3: Gaps in Training Design and Updating Process
  Current APTI Training Program
  Industry Leader Training Program
    Options for Making APTI an
Industry Leading Training Program
The APTI training group may not
encompass all of the key roles required
for course design.
Course development is the responsibility
of a team made up of key roles such as
subject matter experts, script writers,
instructional design experts, technical
experts, editors, and project managers.
Examine the competency/skill level of
the current APTI team and bring in
individuals to perform key roles that
are not filled.

Use contractors to fill key roles, when
needed.
APTI does not use consistent standards
and processes to design its courses and
offerings. Rather, the design usually
follows the structure created by the
individual developing the
course/offering.
There is no systematic process for
updating APTI courses/offerings on a
regular basis.

Some APTI courses are viewed as out-
of-date or of poor quality.	
There are consistent standards and
processes for designing courses. For
example:
    •  Collaboration between subject
       matter experts and script writers
       to design live broadcasts
    •  Application of specific criteria
       for determining delivery media
    •  Minimum standards for
       frequency of interactions during
       distance learning courses
    •  Rehearsals before all live
       broadcasts
    •  Pilot tests before courses go live
Develop consistent standards and
processes for designing courses. Adopt
many of the course design "best
practices."
There is a regular process for reviewing
and updating all courses to ensure they
are meeting trainee needs.
Implement a systematic process and
schedule for reviewing and updating
courses/offerings.
        HayGroup
                                                        Page 9

-------
    EPA: Results of Benchmarking Study
        1. Create Course Content Development and Design Teams

    We recommend that APTI create a course development team that encompasses all of the
    key roles required to design training courses/offerings that will be delivered in the media
    required to meet trainee needs. As a first step, APTI should examine the roles and skills
    of individuals in the current training group in relation to the results from the broad-based
    needs assessment to determine whether training group competency/skill gaps exist.  For
    example,  if  APTI  elects to move towards  more synchronous Web-based delivery
    methods, there should be members  of the course development team with expertise in
    Web-based technologies  and experience in migrating courses from one  medium to
    another.
    We recommend that at a minimum, APTI consider creating a course development team
    with the roles outlined in Table 4 below (which may require additional training of APTI
    staff, hiring new talent, developing partnerships with other organizations or experts, or
    using outside contractors).  The number of individuals required for each role will depend
    on the scope of APTFs services (e.g., number of courses/offerings delivered by media).

    Note: We understand that  many of the benchmarking partners we spoke to have  larger
    training staffs and budgets  than APTI. As a result, APTI may not be able to fill each role
    internally, and may need to rely on contractors, on an as-needed basis, to fill some roles.
    APTI  should create a process for filling key roles, regardless  of whether  it involves
    acquiring internal staff, contractors, or partners.

    Table 4: Key Roles of Training Design Team
   Key Role
                         Responsibilities
EOG
Program/Project
Manager
Develops  the budget and  plans  the  overall  training  program (e.g.,  determines
courses/offerings, training delivery media).
Oversees the day-to-day work for a particular course/offering and ensures objectives are
being met.
Takes responsibility for the training program and interfaces with key stakeholders.
Promotes and evaluates the training program.
Subject Matter
Expert
Expertise in the subject of the course/offering (e.g., professors, external consultants).
Provides an  outline  of the  information  that should  be included  in  the  training
course/offering.
Ensures that information included in the training course/offering is accurate.
Instructional
Designer
Expertise in the design of training courses to maximize adult learning.
Creates the training objectives and the overall framework of the course/offering.
Creates the look and feel of the course/offering; makes decisions about how to best present
course materials, regardless of the media.
Script Writer
Takes the information provided by the SME and writes a full script for the satellite or live
Web broadcast.
Creates a more conversational broadcast out of the technical information provided by the
subject matter experts.
    HayGroup
                                                          Page 10

-------
            EPA: Results of Benchmarking Study
        Technical Experts
                •     Expertise in the technology used to deliver courses/offerings (e.g.,  Web  developers,
                     camera crews, satellite technicians).
                •     Develops Web pages, creates satellite broadcasts or live Webcasts, and so forth.
                •     Reviews the technical aspects of the course, including delivery methods.
              Note: The type of medium used (e.g., Web) will determine the type of technical expertise that is
              needed.
        Editor/Producer
                •     Makes sure that course content, regardless of the training delivery medium, is clear and
                     accurate.
                •     Produces  the course/offering and makes sure it is incorporated in the overall training
                                 program.
        Training
        Evaluation Expert
                     Collects and analyzes training evaluation data.
                     Creates  feedback  reports  for training  design  teams and  others  responsible for
                     modifying/updating courses.
                     Works with key stakeholders to evaluate the impact of the APTI training program and
                     makes recommendations for improvements.	
            Note:  One individual may assume multiple roles, and some roles may be filled by
            contractors.
            Note:  The subject matter expert will most likely be a partner versus an APTI team
            member.

            Below we present  two  options for  developing a  training  design  team(s) and we
            recommend  that APTI consider the feasibility of doing each  within given resource
            constraints before deciding on the appropriate approach.

                •   Option 1 - Develop a cross-functional training design team which includes all of
                    the core  roles that are required regardless of the training  delivery media.  These
                    core positions would be  responsible for the design of all training courses,  with
                    media-specific roles (e.g., Web  developer, camera crew) filled on an  as-needed
                    basis.  That is, a core team would remain intact with media-specific roles filled ad
                    hoc.

                •   Option 2 - Develop training media-related teams which include  all the roles
                    required  for a specific training delivery medium.  That is, a  continuously intact
                    team would exist for each training medium that is responsible for the development
                    of courses within that training  medium. For example, there  would be one team
                    responsible for designing all  Web courses.
X.
APTI can immediately begin to take an inventory of the roles and skill sets of its current team to
determine where  gaps  exist.   For example, if the  team is missing  a critical role (e.g., an
instructional designer), APTI can begin to search for an individual (within or outside of the EPA)
to fill this role.  As another example, APTI may send one of its team members to training to
develop a key skill currently missing from the group.

APTI should consider either hiring or contracting for a script writer to work with subject matter
experts to develop live broadcasts.  Almost all the industry leaders we met with use script writers
to improve the effectiveness of their live broadcasts.	
            HayGroup
                                                                                Page 11

-------
EPA: Results of Benchmarking Study
    2. Create a Systematic Process for Designing APTI Courses/Offerings

To become more effective in the design of training courses/offerings, APTI should adopt
a systematic and consistent process for designing all courses/offerings regardless of the
media in which they are delivered.  APTI should develop (based on "best practices" and
input from APTI  staff and other training design  experts)  and adhere to minimum
standards for designing all  courses/offerings.  We recommend that at a minimum, APTI
adopt the following standards and processes for designing its courses/offerings.

    •  Apply  criteria  to determine the best  media  for delivering  particular training
       courses or offerings.  These decisions should be based on the needs assessment
       process, expert opinions, and  other customer feedback.  Table 5  provides some
       factors to consider when making decisions about methods for delivering training.

Table 5: Criteria for Determining Training Delivery Methods

  •     Size of Trainee Audience - For larger, geographically-dispersed audiences, think about
       using live Web or self-paced Web courses if suitable for the content and objectives of the
       course. For smaller audiences, it is probably  not cost effective to develop live or self-
       paced Web courses. A classroom course may work best.
  •     Trainee Characteristics - Consider the learning styles and "technical savvy" of trainees.
       That is, how does your trainee population learn best?
  •     Complexity of the Course Materials - More complex material is better taught in the
       classroom (e.g.,  laboratory)  or through interactive synchronous training methods.   For
       less complex material, it may not be worth the cost to use synchronous training delivery
       methods; a self-paced course may be sufficient (e.g., policy information, background
       information for a classroom course).
  •     Travel Restrictions - Use distance learning  training delivery methods when travel is
       limited or consider using a blended learning approach to minimize time away from the
       office.
  •     Need for Just-in-Time Information  - Critical information that must be disseminated
       quickly is best suited for a live Webcast or satellite broadcast.
  •     Standard Messages - When training  material should be delivered in  a consistent (or
       standard) way, non-classroom-based methods work best.
  •     Need for Interaction - Training  content that requires extensive student discussion and
       question-and-answer sessions are better taught via synchronous training methods (e.g.,
       classroom, live Web training with an audio component).
  •     Requirements for Certification - Classroom training or other synchronous methods that
       can track student progress work best for courses that require certification.
  •     Frequency of Updates - Courses that require  regular updates may be best suited for the
       self-paced Web training delivery method.
  •     Budget - For smaller budgets, use a delivery  method with  an established  infrastructure
       (e.g., satellite) or one that does not require extensive development time and cost.
Note: This is not an  exhaustive list, it is meant to outline some of the more common training
situations/needs and the methods best suited for them.
HayGroup                                                                      Page 12

-------
EPA: Results of Benchmarking Study
Provide critical information to course designers in the early stage of the process,
including objectives of the course/offering, intended delivery methods, audience
characteristics, technological constraints, and timelines and resources.  This will
ensure that courses/offerings that are developed meet stated  objectives and
maximize learning for air professionals and other customers.
                                                                                       "\^*
                                                                                        JL
                                                                                       *
    •   Define the frequency and types of interactions that should be incorporated into all  M||ck fL*.
       courses/offerings.  For example, for satellite broadcasts  or live Webcasts,  the
       standard may be to ensure that there is an opportunity for interaction at least every
       10-15 minutes.  For self-paced Web courses, the standard may be to provide an
       opportunity for interaction (e.g.,  through a short quiz or simulation) every 3 to 5
       screens.

    •   Require rehearsals for all broadcasts (satellite or live Web), including full script
       and use of teleprompters. This will ensure that the instructor presents the material
       in an engaging way to the audience, and final glitches and inaccuracies are caught
       before the broadcast is delivered live.

    •   Pilot test all  self-paced Web and classroom courses to ensure that the course
       content is  clear, easy to understand,  and  meets  training objectives,  and  the
       technology itself  is  easy to navigate,  provides  sufficient opportunities  for
       interaction, and  so forth.  (Note:  It is our understanding  that APTI is currently
       pilot testing its courses before they go live).

These processes and  standards should be developed and captured in a policy manual that
outlines the guidelines for designing each new APTI course/offering.  This will help to
ensure consistency and  quality of all courses/offering that are developed by APTI.  It is
also important to communicate the importance of following these guidelines to all APTI
staff and those outside of APTI involved in the course design process (e.g., contractors,
subject matter experts).

    3. Develop a Systematic Process and Schedule for Reviewing/Updating Courses

We recommend that APTI create a  systematic process and schedule for reviewing and
updating all courses/offerings on a  periodic basis.   One of the biggest complaints we
heard about the current APTI training program (during  interviews conducted during
Task 1) was that many courses are outdated and contain inaccurate information, and that
course materials that are sent out are not of the highest quality (e.g.,  pages are upside
down  and missing,  graphics are   outdated).    A  periodic review schedule  for all
courses/offerings will help to make  sure that APTI courses provide the most up-to-date
information using the most innovative delivery methods and graphics possible.

As  a first step, APTI should review the information collected from the needs assessment
process and review/update only those courses that are critical for meeting the needs of air
professionals  and  other customers  (courses that don't  meet critical  needs should be
eliminated  or shelved for a finite period of time).  It is important to incorporate this
HayGroup                                                                   Page 13

-------
EPA: Results of Benchmarking Study
review process into all new courses or offerings.  The course review and update process
should include the following:

   •   Timeline -  One of the most important steps is to develop a clear schedule for
       updating  courses/offerings.   APTI should review  current courses/offerings to
       determine the "shelf life" of information provided during trainings, and develop
       review schedules based on the average time  it takes for information to become
       outdated.   For example,  courses  that  provide  information  about frequently
       changing air pollution regulations may need to be reviewed every 12 months,
       while courses that deal with long-standing technologies may require less frequent
       reviews (every 24-36 months).

   •   Review Team - Because the course design teams  are intimately  involved in
       developing  the courses/offerings, they may also prove useful during the course
       review process. For example, subject matter experts involved in the design of a
       particular course may be asked to also be involved in the continual review/update
       of the course.  The key point is to appoint a team(s) that is responsible for one or a
       group of courses/media so that different teams are not updating the same courses.
       This reviewer role could be built into the contracts for subject matter experts and
       others who are working for APTI on the design of courses/offerings.

   •   Automatic Review Notifications - We recommend that  APTI adopt methods that
       automatically  notify team members when courses/offerings are  scheduled for
       review.  These technologies may also be used to track the review and update
       process.  For example, the reviewer may receive an electronic notification 30 days
       prior to a scheduled  course review. This gives the reviewer enough time to notify
       other team members and make final amendments to the review process.

   •   Systematic Steps - APTI should create systematic steps for both  reviewing and
       updating courses.   For each step, APTI  should  determine who is  responsible,
       when the review should be completed, and what should be the final product. It
       may prove beneficial to develop and disseminate a timeline that highlights major
       activities and the responsible parties.

   •   Communication Plan  - Changes made  to APTI  courses/offerings should  be
       communicated to  air  professionals  and  other customers  to  ensure  that  they
       continue to enroll in and find the courses useful.  Also,  communicating about
       course updates shows that APTI takes the review and feedback process seriously.

As with the course design guidelines, processes and standards for reviewing and updating
courses should be developed and incorporated in a policy manual that is shared with all
parties (internal and external to APTI) involved in updating courses/offerings.
HayGroup                                                                  Page 14

-------
            EPA: Results of Benchmarking Study
I.
APTI should conduct a quality review of all current course materials before they are distributed to
trainees. Reviewers should check for:
    •  Typos
    •  Incorrect information
    •  Missing, upside down, or out-of-order pages
    •  Whether the right materials are sent out on time, and with the right courses	
            HayGroup
                                                                              Page 15

-------
        EPA: Results of Benchmarking Study
               IV.  Options for Improving APTPs Training Delivery Methods

        Our benchmarking research indicates that APTI may not currently be delivering training
        in ways that best meet the needs of air professionals and other customers.  Courses are
        primarily offered and delivered using the classroom medium (with some use  of satellite
        broadcasts and self-paced Web  courses; paper-based self-instructional courses are going
        away).  APTI is not currently taking advantage  of live Web-based methodologies to
        deliver training and information to air professionals.  Thus, a  large portion  of APTFs
        training methods require air professionals (or trainers) to travel to classroom  or satellite
        downlink sites to complete training.

        It was clear from Task 1 interviews that classroom training is  the dominant, and often
        preferred, delivery method for many types of courses and we believe there will always be
        a need for  classroom courses.  However, the consensus among interviewees, and  our
        recommendation, is  that APTI  must adopt  a complement of distance  learning training
        vehicles, in addition to  classroom courses, to ensure that training is  accessible to air
        professionals and other customers. Our research shows that many types of training and
        outreach are amenable to Web-based delivery methods.

        There is also the perception that APTFs  current distance learning courses/offerings may
        not be as effective as they could be.  One of the biggest complaints is that there are not
        enough opportunities  for  interaction during  satellite broadcasts and  self-paced Web
        courses,  and   trainees  don't   take  advantage of the  opportunities  that   do   exist.
        Additionally, some instructors may not have the skills and experience to deliver training
        in particular media (e.g., deliver an effective satellite broadcast).

        Our broad recommendation is that APTI develop the infrastructure necessary to deliver a
        larger proportion of its courses/offerings via distance learning methods (e.g.,  live Web),
        while continuing to use the classroom to deliver courses best suited for this medium (e.g.,
        laboratory  courses,  courses only  offered  to  a small  trainee population).   Table  6
        highlights the key gaps in training delivery methods that we  see between the current
        APTI training program and industry leading training programs, as well as options for
        closing the gaps.

        Table 6: Gaps in Training Delivery Methods
  Current APTI Training Program
  Industry Leader Training Program
    Options for Making APTI an
Industry Leading Training Program
There is an emphasis on classroom
courses, and satellite broadcasts are the
primary method used for distance
learning.

Satellite broadcasts are not viewed as
effective by some stakeholders and
trainees (e.g., not enough opportunity
for interaction, ineffective speakers,
some broadcasts are poorly attended).
Industry leaders are making more use of
distance learning methods versus using
the classroom as the primary training
delivery medium.

Industry leaders tend to make more use of
live Web training than satellite broadcasts
for distance learning.

Satellite broadcast are typically used for
Move from primarily classroom-based
delivery methods to distance learning
methods.

Evaluate the feasibility, benefits, and
cost effectiveness of making more use
of live Web training versus other
delivery methods. If feasible, begin to
move towards more live Web training.
        HayGroup
                                                    Page 16

-------
        EPA: Results of Benchmarking Study
                                   dissemination of critical, real-time
                                   information.
                                     Determine the best use of satellite
                                     broadcasts.
APTI has not fully adopted a blended
learning approach. APTI is currently
using only one medium to deliver each
course/offering.
Many courses are delivered via multiple
media (e.g., classroom and Web).
Determine whether some APTI courses
can be delivered via multiple media to
maximize the effectiveness of training,
minimize travel costs, and make
training more convenient and
accessible to all air professionals.	
There is the perception that
opportunities for interaction during
APTI satellite broadcasts are limited,
and trainees don't take advantage of
those opportunities that do exist.

Serf-paced Web courses are not
interactive enough to maximize
learning potential.
Innovative methods are used to
incorporate opportunities for interaction
into distance learning courses (interaction
with the technology, instructor, other
students).
Review current and past methods for
incorporating interaction into satellite
broadcasts. Determine whether these
practices are effective and ways to
improve their effectiveness (e.g.,
encourage air professionals to ask
questions during satellite broadcasts).

Look for ways to incorporate
opportunities for interaction into serf-
paced Web courses (e.g., chat rooms,
real-world simulations, case studies).
Instructors are not always effective at
delivering training in the selected
medium.
Instructors are required to complete
training before delivering a course in a
particular medium (e.g., live Web).

A train-the-trainer approach is often used.
Ensure that all instructors are qualified
to deliver training in the appropriate
medium.
        For the remainder of this section, we further discuss our options for improving the way
        APTI training courses/offerings are delivered.

            1. Make More Use of Synchronous Distance Learning Methods, with a Primary
               Focus on Live Web

        As indicated previously, many industry leaders are moving away from classroom courses
        to live Web courses that trainees can complete at their work stations.  Furthermore, in the
        most recent APTI needs assessment, participants indicated that the  biggest factors that
        influence  sending  staff to  training  are cost, travel restrictions, workload, and  staff
        shortages. This seems to indicate that APTI must look for ways other than the classroom
        and satellite broadcast methods to deliver synchronous training to air  professionals.  Live
        Web  courses offer all the advantages of satellite courses, with  additional capabilities for
        incorporating interactions into training  courses.  Also, live Web meets the "my  desktop,
        at my time"  needs  of most busy air professionals.   That is, live Web allows trainees to
        access courses wherever they have a Web connection, rather than traveling to a classroom
        or  satellite  capable site.  Our benchmarking research also found that many  industry
        leaders are successfully delivering training via the Web that was previously delivered in a
        classroom setting.

        As a first  step, APTI  should  conduct a feasibility  study  (to  supplement the  "best
        practices" research which supports the use of live Web training) to determine the types of
        courses/offerings best suited for live Web delivery, and the cost and benefits of using live
        Web  versus classroom or satellite media.  APTI must weigh the cost of developing live
        HayGroup
                                                       Page 17

-------
EPA: Results of Benchmarking Study
Web  courses  (and adopting this  technology) against  the  potential benefits  that this
technology will provide air professionals and other customers.  For example, are more air
professionals likely to attend a training course or outreach activity if it is delivered via the
Web versus a classroom session or satellite broadcast?  Is live Web  training equal to  or
better than  the  classroom or satellite  in  engaging air professionals, teaching critical
skills/competencies, and offering  opportunities for interaction? How does the cost  of
delivering live Web training (once this technology is adopted) compare to the classroom
or satellite media?

As part of the feasibility study, APTI may  want to select a specific course and deliver it
through different media (e.g., classroom, satellite, live Web) and evaluate how effectively
the course is delivered through the selected media (as a pilot).  Feedback could be sought
from instructors, instructional design experts, and trainees as to the effectiveness of each
medium in keeping students engaged, teaching required competencies/skills, providing
opportunities  for  interaction,  and so  forth.   Additionally, if  a  broad-based needs
assessment is  conducted, APTI can also seek input into whether air professionals prefer
one delivery method over the others; whether supervisors are more likely to support their
staff in training if it is delivered via the Web versus the classroom or satellite broadcast;
and so forth. That is, the broad-based needs assessment will provide critical information
about whether the needs of APTI's key customers would be better served through a live
Web training delivery method.

Tips for Conducting a Feasibility Study

A feasibility study will  help determine whether the live Web technology is feasible and
cost  effective to  implement at APTI.   It will provide  input  into the new APTI
courses/offering that would be best delivered using the live  Web medium,  and which
current courses are most suitable for migration from the classroom or satellite broadcast
to live Web media.  As part of this  assessment, APTI will want to study the following:
•  The cost to develop a course/offering using live Web versus other media
•  The cost to deliver a course/offering using live Web versus other media
•  The number of participants likely to attend a live Web course versus a course
   delivered via other media
•  Extent to which opportunities for interaction can be incorporated into the Web versus
   other media
•  Effectiveness of live Web versus other media in meeting the stated objectives of the
   course/offering (e.g., teaching the required competencies/skills)

In sum, we believe that APTI should deliver a larger proportion of its courses via distance
learning methods, particularly live Web. Most of the industry leaders we spoke with are
moving away  from the classroom and satellite to the live Web training delivery method.
However, because  APTI has invested significant resources in classroom training (and
satellite broadcasts), our recommendation  is to determine how  to best use its existing
training methodologies while  following  the industry shift  towards live Web-based
distance learning programs.
HayGroup                                                                   Page 18

-------
EPA: Results of Benchmarking Study
   2.  Create a Blended Learning Approach

Like many industry leaders, APTI should consider adopting a blended learning approach
for some of its key training courses/offerings.  By incorporating two or more training
delivery media (e.g., classroom and Web), APTI may improve the effectiveness of its
current and future courses/offerings and reduce travel cost and time away from the job for
air professionals.  We are not recommending that APTI offer the same course/offering in
two different media, but use different media for different portions of the course/offering
(e.g., Web  for prerequisite materials; classroom for portions of the course that require a
lab or extensive discussions).

As a first step, APTI should determine which of its current and future courses/offerings
are best suited for a blended learning  approach.  Data collected  from the broad-based
needs assessment will also provide insight into where a blended approach may work best.

Below are  a  couple scenarios where we believe the blended learning approach would
work well for APTI:

   •  A multiple day classroom course that is offered multiple times throughout the
      year - APTI may be able to  reduce the amount of time air professionals spend in
       the classroom and away from their jobs through a blended learning approach. For
       example,  a self-study Web course  could be  developed  to  provide  critical
       knowledge (e.g.,  regulations,  policies)  before  the classroom  course  (versus
       spending time lecturing about basic knowledge in the classroom  course), and to
       get  all air professionals up  to a similar knowledge level before the classroom
       sessions.  This  could reduce  a  significant portion  of classroom  time.  The
       classroom session would then focus on interactive exercises,  discussions,  case
       studies,  and simulations  or actual  real-world  job  activities versus delivering
       information in a lecture format.  Following  a few days of classroom sessions, air
       professionals could  also complete  additional self-paced Web  sessions  or
       participate in a live Webcast back at their own office  to complete the course.

   •  A self-paced Web course that requires discussion - APTI may be able to reduce
       the time spent developing self-paced Web courses (most notably the interactivity
       aspects) and improve interactions by incorporating  a blended learning approach.
       For example, live Web course sessions could be developed to supplement self-
       paced Web courses in an effort to augment interactions and foster discussions.
       Trainees would still access the self-paced Web course at their own convenience.
       However, they would be required to attend periodic live Web sessions during the
       overall training course, where trainees would be given the opportunity to discuss
       and clarify  previous course content. This  could reduce  the  time  required  to
       develop the interactive discussion aspects of the self-paced Web course, while
       providing for real-time discussions among trainees (and instructors). Further, the
       self-paced Web sessions could focus solely on providing necessary information
       rather  than  alternating   between   information dissemination  and  delayed
       discussions.   Note:  Satellite  or  classroom-based course  sessions  may be
       substituted for or augment further live Web sessions.
HayGroup                                                                  Page 19

-------
            EPA: Results of Benchmarking Study
I.
               3.  Ensure Live Distance Learning Courses Provide Sufficient Opportunities for
                   Interaction

            As  indicated  previously, the  perception is  that  APTI  does not  provide  enough
            opportunities for interaction in its current distance learning methods, and trainees do not
            maximize the  opportunities that are provided.  Industry experts agree that providing
            opportunities for interaction is critical to keeping trainees engaged in the course and most
            importantly,  ensuring  that they  develop the critical skills  and  competencies that the
            course is intended to teach.  To become more effective at delivering  distance learning
            training, APTI needs to:  a) examine the opportunities for interaction that are provided in
            its broadcasts;  b) determine new ways to incorporate opportunities  for  interaction into
            broadcasts; and c) explore ways to  ensure that trainees  take advantage of opportunities
            for interaction.  These same principles will apply if APTI decides to adopt live Web
            broadcasts.

                   a) Examine opportunities for interaction in satellite broadcasts
In the short-term, APTI should examine current opportunities for interaction used during satellite
broadcasts, as well as methods that have been used in the past.  As part of this review, APTI
should consider the effectiveness of each type of interaction opportunity, and why some methods
are  not as effective  as others.  Below are some  criteria for determining the effectiveness of
methods of interaction:
    •   Is there an opportunity for interaction at least every 10-15 minutes?
    •   Do methods for interaction interrupt the flow of the session?
    •   Do trainees take advantage of opportunities for interaction (e.g., ask questions, participate
       in discussions)?
    •   Do trainees receive answers to all questions that are not addressed during the broadcast?
    •   Can trainees hear and respond to others' questions?

This information will help APTI to determine which methods for interaction should be retained
(and used during satellite  or live  Web broadcasts), and whether new methods for interaction
should be adopted.

APTI  also needs to ensure that it is consistently providing opportunities for interaction in all its
distance learning courses. _
                   b) Determine new  ways  to provide opportunities for  interaction  during
                      broadcasts

            Based  on our benchmarking research, we believe that new methods for interaction are
            needed to enhance the effectiveness of APTI' s satellite broadcasts (and live Web training,
            if it is adopted).   We recommend that APTI consider using some  of the  following
            methods to enhance interactions during broadcasts.

                   >  Incorporate an  open audio line (phone  line) during broadcasts so  that air
                      professionals  can ask questions and  discuss issues with the instructor and
                      among each other.
            HayGroup
                                                                             Page 20

-------
            EPA: Results of Benchmarking Study
                   >  Create a chat  room  that  air professionals can visit  after the broadcast to
                       discuss  issues  and ask questions.   The instructor or  subject matter  experts
                       should periodically participate in chat room discussions and answer trainee
                       questions.

                   >  Provide trainees with an e-mail address for sending questions to the instructor
                       or other subject matter experts  during a live  Web session (if live Web is
                       adopted).   These questions could be answered during the live broadcast or
                       questions and answers could be posted on a Web site in the future.

                   c)  Adopt practices for ensuring that opportunities for interaction are effective

            APTI  needs to go beyond simply incorporating methods for interaction into  its  live
            distance learning courses.  These methods for  interaction  must be effective at soliciting
            questions from trainees and generating discussions during the  broadcast. We recommend
            several strategies for ensuring that methods for incorporating interaction into broadcasts
            are effective.  Many of these strategies can be adopted in the short term.
I.
Enhancing Methods of Interaction

    •   Publicize opportunities to interact. As part of publicizing the satellite (or live Web)
       broadcast, inform potential participants about the process  for asking  questions or
       participating in a discussion during an APTI broadcast and encourage them to do so.
       This may include broadcasting call-in  numbers and e-mail addresses, instructions for
       using the various  methods, and the benefits of actively participating in the broadcast.

    •   Establish proper etiquette. Develop proper interaction etiquette for trainees to follow.
       This etiquette  should be  communicated to  participants before each training session
       begins.  Participants should be reminded that they may be asked to leave the session if
       they repeatedly violate the etiquette.

    •   Use screeners and  technology monitors to review incoming questions.   APTI or
       state/local agencies should provide an individual who can act as a question screener and
       technology monitor for each broadcast.  This  individual would be tasked with accepting,
       reviewing, and facilitating questions, and making sure  that all technology  is working
       properly.

    •   Develop discussion questions prior to the broadcast.  Develop a set of 5-10 potential
       questions for each training session. When  trainees are reluctant to ask questions, a
       training team member can  initiate discussions  by asking one of the questions.

    •   Follow up on unanswered questions. Provide trainees with the answers to all questions
       that are asked, regardless of whether they are answered during the broadcast.  This  may
       be done by posting questions and answers to a Website. _
            HayGroup
                                                                               Page 21

-------
EPA: Results of Benchmarking Study
   4.  Improve the Effectiveness of Self-paced Web Courses

We recommend that APTI enhance its current self-paced Web courses, particularly by
incorporating opportunities  for interaction with the  technology  itself, instructor, and
among trainees.  The most recent needs assessment found that more self-training is
needed, especially those related to interactive Web-based methods.  It  is important that
the self-paced Web courses that are offered provide an engaging learning experience and
are not simply a .PDF document or Power Point slides put on the Web.

Below are several strategies for increasing the effectiveness of Self-paced Web courses.

   •   Incorporate real-world simulations and case studies  Our findings from Task
       1  indicated that air professionals would like more practical, real-world examples
       and applications in training courses so they could see how the training applies to
       the "real world."  By incorporating simulations and case studies into self-paced
       Web courses, air professionals can actually practice the skills they need on their
       jobs (e.g., testing air quality) and solve real-world problems. Additionally, these
       tools will increase the likelihood that trainees will be more engaged in the training
       than   if  the  course  simply  presented information  or focused   on theory.
       Incorporating real-world simulations and case studies will require sophisticated
       Web development resources and should only be done for courses that are taught
       to a relatively large audience and meet critical needs.

   •   Incorporate quizzes and assessments. We  recommend incorporating periodic
       quizzes  and assessments  into  self-paced Web  courses  to  help keep  trainees
       engaged and focused  on the material.   Also,  by  incorporating  quizzes and
       assessments into the  courses, APTI can effectively track short-term  learning and
       begin to  move towards  consistent  level  2 evaluations.    Quizzes  can be
       programmed so that trainees are blocked from continuing  the training until they
       provide  correct responses.  However, participants do not need to be blocked from
       continuing; it may be more effective to provide brief explanations after each quiz
       or assessment of the correct responses.

   •   Create  engaging material.  APTI should develop engaging visual materials for
       every Web course.  As a first step, APTI should review existing courses to make
       sure that the  material is presented in an engaging manner.   Creating  engaging
       material is one of the most important aspects of delivering an effective self-paced
       Web training session.  Our research shows that  individuals learn about 75% of
       what they know through vision. Also, trainees will "tune out" after long training
       sessions, especially if they don't find the visual material stimulating or feel that
       the material is not relevant to the course.

   •   Use chat rooms or Web boards to facilitate interactions  To foster self-paced ^      *b
       Web course interactions, we recommend that APTI create chat rooms or Web
       boards where trainees can post questions to instructors, receive course updates
       from instructors, and have e-mail discussions  with each other.  At the beginning
       of the training course or offering, each trainee should be provided  with  the
HayGroup                                                                  Page 22

-------
            EPA: Results of Benchmarking Study
                   location  of and instructions  for using  the  chat room or Web board.  Also,
                   instructors should review the chat room or  Web board periodically  and be
                   encouraged to use these as a medium to interact with trainees.

               •   Facilitate cohort-based  training.   APTI may want to consider facilitating the
                   assembly  of cohorts  of approximately five to seven  trainees to complete the
                   training program as a group rather than individually.  Cohorts may, for example,
                   come from the same  state or local agency or be distributed across the  country
                   based on function.  Participants would complete the self-paced modules at their
                   own pace  but  within a  specified  time frame (e.g.,  one week  to complete a
                   module).  At a  set time, participants would then access chat rooms or  participate
                   in a conference call to discuss the course and ask each other questions in between
                   modules.  This is  a more effective and efficient method  because trainees have
                   more opportunities to  interact and  they can provide each  other with more
                   assistance and support.

               5.  Ensure That Instructors Have the Tools and  Skills to Teach

            Our  benchmarking research  and interviews from Task 1  of the benchmarking study
            revealed  that  some  instructors  do not have  the  skills required  to  deliver training in
            particular media. For example, some potential instructors may be experts in the subject
            of the course, but are not effective  at delivering  a satellite broadcast (e.g.,  are  not
            engaging).   APTI should focus on ensuring that instructors have not only the subject
            matter expertise required to teach,  but also  the  technical tools  and skills and "stage
            presence" needed to teach using various distance learning methods.  Instructors with the
            appropriate  skills will  conduct more effective courses, thus increasing attendance and
            making the overall training program more valuable.
X.
>  Select instructors who have subject matter expertise, applied experience in the field, and
    experience in the medium of course delivery.
>  Consider using professional  speakers to  deliver training in conjunction with subject
    matter experts who can answer specific questions raised by air professionals.
>  Provide instructors  with  standard  guidelines  for  training  in  a particular  medium.
    Instructors should be required  to  participate  in  a  short training  session designed to
    provide them with basic, standard guidelines for conducting a training session in a given
    medium (before delivering APTI training in a particular medium for the first time).  Not
    only will this provide instructors with tips on how to provide training, but it will also help
    to "brand" EPA instructors and training programs by showing instructors the "EPA way"
    to train. Some of the topics to cover include:
         •   Where to stand and how much to move during training
         •   When and how to look into the camera
         •   How to use basic technologies
         •   How to encourage interactions
         •   How to deal with difficult students
         •   Tips for holding the attention of students
Note:  Several private companies and educational institutions offer "training the trainer"
courses that APTI may consider using before developing its own program.	
                                                                                    X.
            HayGroup
                                                                          Page 23

-------
EPA: Results of Benchmarking Study
V. Options for Improving APTFs Training Evaluation Process

APTFs current training evaluation process focuses on end-of-course level  1 evaluations
("smile sheets").  For some classroom courses, air professionals complete a pretest before
the course begins to assess their level of knowledge and a posttest at the end of the course
to see  how much they have learned (level 2).  To become an industry leading training
program, APTI needs to reexamine its training evaluation process to  ensure  that it is
measuring the "right things" - the extent to which  the training is adding value  to air
professionals  and the agency as a whole.  Measuring  the "right things" means expanding
APTI's training evaluation process to at least include  level 3 evaluations (extent to which
training impacts air professionals' performance on the job).

There is also room for improving the level 1 forms that are currently used to assess APTI
training by including questions to better understand why training courses were effective
or not. Finally, the perception is that feedback from the training evaluation process is
rarely used to improve training courses and offerings.

Table 7 highlights the current state of APTFs training evaluation  process compared to
what industry leaders are doing  in this area,  and presents options for creating a more
effective training evaluation process.

Table 7:  Gaps in Training Evaluation Process
Current APTI Training Program
Level 1 evaluations are not as effective
as they could be.
The evaluation of APTI training
courses focuses on level 1 - End of
course evaluations (also called "smile
sheets"). There is some use of level 2
evaluations.
Training evaluation data are not
consistently used to improve courses
and offerings.
Industry Leader Training Program
The training evaluation process assesses
the impact of training on: 1) trainee
knowledge gained during the course
(level 2); 2) future performance on the job
(level 3); and 3) overall organizational
performance (level 4).
Training evaluation data are used to
improve training course content and
delivery methods.
Options for Making APTI an
Industry Leading Training Program
Improve the instruments used to
provide level 1 evaluations of APTI
courses/offerings.
Expand the training evaluation process
beyond end-of-course level 1
evaluations.
Create a process for tracking,
monitoring and using training
evaluation data to make improvements
to training courses and offerings.
Below we expand on the recommended options for improving the training evaluation
process at APTF

    1.  Increase the Effectiveness of Level 1 Evaluations

There are several  quick fixes that APTI can make to improve its process for conducting
level 1 evaluations, which are described on the following page.
HayGroup
Page 24

-------
            EPA: Results of Benchmarking Study
X.
a) Improve level 1 evaluation instruments.  Level  1 evaluations provide useful information for
making immediate changes to training course content and delivery methods (e.g., there is a need
to add more opportunities for interaction into a Web-based course).  Based on our review of the
current APTI  training  program, we  believe that  there is room for improving the level  1
evaluations ("smile sheets") used to evaluate classroom courses and satellite broadcasts. That is,
the current course evaluation forms do not provide enough information for understanding why the
classroom course/satellite broadcast was  effective, whether the delivery method is the  best
medium  for  delivering training, and  more  importantly, how  to  improve  APTI  training
courses/offerings. At a minimum, we recommend  adding open-ended questions to the training
evaluation forms.  For example:
    •   What about the course/broadcast did you like best?
    •   What about the course/broadcast would you  change?
    •   How would you improve the current training course/broadcast?
    •   Were there enough opportunities for interaction in the course/broadcast? If not, what are
       your recommendations for enhancing interactions during the course/broadcast?

b) Evaluate all courses/offerings, regardless of the delivery method.  It is important to  solicit
feedback  about  all APTI courses and  offerings, including self-paced  Web  courses.   Air
professionals should be given the opportunity to complete a course evaluation on-line after they
finish the self-paced Web course.  The evaluation should focus on the course content itself; how it
was presented; the effectiveness of any quizzes, simulations, or case studies; the ease of accessing
the course and navigation through the Web site; and so forth. The evaluation should also include
overall questions about the effectiveness of the Web  course at teaching required skills  and
knowledge, whether it engages air professionals to  begin and complete the course, and whether
the Web is the best method for teaching required competencies/skills.

c) Follow up on less favorable evaluations.  APTI should consider conducting follow-up  data
collection when level  1  evaluations  do not meet a  minimum rating.   For example,  if air
professionals rate a particular aspect of a training course below a minimum rating, APTI may
want  to contact air professionals to  obtain more insight into why the course  was rated  less
favorably than expected. This could be done through a telephone interview.	
               2.  Expand the Training Evaluation Process Beyond Level 1 Evaluations

            APTI should  create  a training  evaluation process that not only determines how air
            professionals feel about courses/offerings and whether they learned during the courses,
            but whether the training improved their performance on the job (level 3  evaluation) and
            helped them contribute to  the  mission of their agency.  As indicated  previously, the
            current APTI training evaluation process focuses on end-of-course level 1 evaluations,
            the "smile sheet" evaluations.  These evaluations are necessary to  obtain trainees' initial
            reactions to  courses/offerings,  but  they are far from sufficient for establishing the
            organizational impact of training courses or offerings.  Level 3 and 4 evaluations provide
            valuable  information for modifying training courses/offerings (and potentially training
            objectives), as well as for demonstrating the value of the Air Pollution  Training program.

            We  recommend   the  following  steps  for  creating  a more  comprehensive  training
            evaluation process:
            HayGroup
                                                                               Page 25

-------
EPA: Results of Benchmarking Study
Plan for long-term evaluations.  Developing a training evaluation process that assesses
overall organizational impact will take time and planning.  Training experts agree that
higher level evaluations are very difficult to do because it takes several months before an
organization may see any change in performance as a result of training.  Additionally, it
is often difficult to discern the effects of a training course on trainee performance; there
are so many other factors that could impact performance.

As  a first step, APTI should design a plan for evaluating  the long-term impact  of its
courses/offerings.  This plan should include:
    •   Timelines for evaluating training (e.g., transfer of training will be evaluated three
       and six months after the air professionals complete a training course/offering).
    •   Who will participate in the assessments (e.g., air professionals, supervisors of air
       professionals, agency leaders)?
    •   How many assessment instruments will be required?  For example, can the same
       data collection instrument be used for all types of training media or will different
       types of instruments be required?
    •   How will data be collected (e.g., on-line surveys, interviews with key
       stakeholders)?
    •   Which courses/offerings will be evaluated beyond level 1 (e.g., level 3
       evaluations take time and resources; APTI may want to conduct these evaluations
       only for its most critical courses/offerings)?
    •   What will be measured by the assessment? There is a need to define the desired
       changes in air professional behavior and how they will be measured.
    •   What are the minimum ratings required for an evaluation (e.g., a 3.5 on a 5 point
       scale)? What is the follow up process if ratings fall below the minimum rating
       (e.g., a follow up interview to gain more insight into areas rated lower than
       expected; development of an action plan for improving the course/offering)?
    •   Who is accountable for evaluating APTI training?
    •   How will training evaluation data be used to improve  APTI training courses and
       offerings?

Gain buy-in from key stakeholders. A higher level training evaluation will  require the
support and participation  of state and local agencies and other key stakeholders.   APTI
should communicate its plans for expanding its training evaluation process and how this
will benefit state and local agencies and other key stakeholders. Additionally, it would be
helpful to involve  key stakeholders in  the design and conduct of the training evaluation
process.

Begin  to conduct evaluations  with  trainees  and supervisors  to evaluate learning
transfer (level 3).  APTI should conduct follow-up evaluations 3 to 6 months post-course
to measure whether trainees have effectively transferred the knowledge/skills learned
during training  to their jobs.  Evaluations should follow a 360-degree methodology that
obtains data from  the trainee as well  as supervisors and subordinates. For example, a
HayGroup                                                                   Page 26

-------
EPA: Results of Benchmarking Study
sample of air professionals who have completed a critical classroom course may receive
an on-line survey three and six months after they complete the course,  asking  for their
opinions about how well they have applied the APTI training to their jobs and how it has
contributed to the overall effectiveness of their agency.  Their supervisors would also
receive  a similar on-line survey to obtain their opinions about the extent to which the
training has improved their employee's performance on the job.

To the extent possible, APTI should compare evaluation data  (e.g., performance of key
tasks on the job) with air professionals not yet participating in training.  This will help
demonstrate that changes are due to the training and not some other outside source (e.g.,
implementation of a new process or procedure).

   3.  Create  a Process for Monitoring  and Using Training  Evaluation  Data to
       Make Improvements

We recommend that  APTI use the evaluation  data not only  as a measure of training
effectiveness but also to make improvements to training courses and offerings. The level
1 training evaluations are likely to highlight aspects of courses/offerings that participants
find useful/not useful as well as ideas they may have to improve courses/offerings (e.g.,
technology,  exercises, instructors).   The higher level  evaluations should further show
areas where the training courses and offering are not effective; that  is, where they are not
meeting  stated  objectives.   By reviewing these data,  APTI should be able to  pinpoint
areas that need to be  modified to meet these training objectives.  It is also important to
communicate to the air professional community the changes that are being made to the
training as a result of the evaluation.

In order to do this, APTI should develop a systematic feedback process to ensure that the
appropriate evaluation data are monitored, processed, and used to improve courses.  For
example, instructors  or designated training  team  members should be  responsible for
collecting evaluation data  and  transferring it to a  central  database  for processing.
Transferring the data  to a central location will allow the APTI training team to  evaluate
trends (i.e., from year to year) and make comparisons to other courses. Whether courses
meet these needs can be determined by setting minimum standards that each course must
meet (e.g., at least 70%  satisfaction with the  effectiveness of training).   The results
(positive and negative)  and any suggested changes should then be fed back to  both the
team responsible for designing and updating courses as well as current and future course
participants. Even if the data are used to improve courses, participants will not  feel that
evaluations are effective if they are not actively shown results and subsequent changes.
HayGroup                                                                   Page 27

-------
EPA: Results of Benchmarking Study
                                 VI. Next Steps

In this  section,  we  outline our recommended next steps for  APTI to identify and
implement needed  changes to  the  Air Pollution Training program.   We  present
suggestions   for  translating   the  options  presented  throughout  this report  into
improvements to the APTI training program.  The ultimate goal of this change effort is to
ensure  that the  APTI  training program is  continually  meeting  the  needs  of air
professionals and other key customers.

We  recognize  that  APTI may  not be  able  to  immediately  implement  all  of the
recommended options outlined in this report,  and some may not be beneficial or feasible
to implement.  Also,  we understand that there may be some reluctance to  change  certain
practices or to move away from specific delivery  media  (e.g., classroom and satellite
broadcasts).

   1.   Plan for a Successful Change Effort

As a first step, we recommend that APTI define its desired end state, make decisions
about those changes that should be made to the APTI training program (which options to
implement), create a plan of action, and determine strategies for overcoming resistance to
change.

Define desired end state. APTI leadership, staff, and other key stakeholders should meet
to discuss the recommended options outlined in this report.  We believe  that the APTI
team and key stakeholders must have buy-in and ownership of the recommended changes
to ensure they  are successfully implemented.   That is, this group must have a clear
understanding of not only the changes APTI  chooses to  make, but also the end point or
desired  state that APTI hopes to attain.

These meetings  will allow the APTI team and other key  stakeholders  to voice their
concerns and opinions about the recommend options, identify potential new options for
improving the training program,  and come to consensus on the  recommended changes
that APTI should pursue. During these meetings, the APTI team should focus on:

   •  Translating recommended  changes  (options)  into actionable initiatives  and
       accountabilities

   •  Illustrating the benefits, feasibility, and cost of the recommended options
   •  Prioritizing short-term and long-term actions

As indicated previously, we believe that APTI should also conduct a broad-based needs
assessment in the near future to redefine its mission and  scope, and ultimately determine
the courses/offerings that should be provided to meet the  needs of its key  customers.
This broad-based needs assessment will validate and further define the direction APTI
should  take  in  the  future,  and which long-term options should be  adopted.  We
recommend  holding off on major changes to the APTI training program (e.g., those that
require  extensive resources) until after the broad-based  needs assessment is  conducted.
That is, customer and stakeholder feedback from the broad-based needs assessment will
HayGroup                                                                  Page 28

-------
EPA: Results of Benchmarking Study
be instrumental for determining whether major options/changes meet a critical customer
need, and thus should be adopted. On the other hand, APTI should begin immediately to
work on options that are easier to implement, less resource intensive, and that will meet a
stated or known need of a key customer group (e.g.,  a need revealed in a prior needs
assessment or training evaluation).

Collect additional information.  APTI may need to  collect additional  information to
make decisions about the feasibility of some of the recommended options in this report.
For example, in determining whether to deliver more courses through live Web (versus
the classroom  or  satellite  broadcasts), APTI  should conduct a  feasibility study to
determine whether adopting the live Web  technology:  1) is cost effective; 2) would
increase attendance in courses; and 3) is more effective than the classroom or satellite in
teaching required skills and competencies (see section IV: Options for Improving APTFs
Training Delivery Methods).

Create change plan.  After identifying the priority options that APTI will implement to
improve  its  current training program, we recommend creating a change plan  that
describes the changes to be made and why they will  be made, who is accountable for
each change, timelines for implementing short- and long-term change, resources required
to implement the changes, and how success will be measured.  Below,  we provide a
sample template for a change plan.
                             Change Plan
Air Pollution Training Program
Date:

Area for
Improvement

Goals


Recommended
Actions (Options)

Responsible
Party(s)


Support
Required

Performance
Measures

Target
Dates

Overcoming resistance to change.  Resistance to change is one of the most common
barriers to overcome at all levels of an organization and during all phases of the change
process.  Creating buy-in from leadership and individuals who will be affected by the
change (e.g., APTI staff, state  and local agencies) is an effective method to  help
individuals overcome this resistance. When key individuals are included in the decision
process, and when they endorse mutually agreed upon changes, they are more likely to
HayGroup
Page 29

-------
EPA: Results of Benchmarking Study
work to ensure that change happens as planned.  Once an individual publicly supports a
plan, it becomes more difficult for the individual to resist the change.  For example, it
may be relatively easy for someone to resist migrating a training course from a satellite to
Web-based medium if they feel that the decision to change has been dumped on them.
However, APTI staff (or air professionals) will be less likely to resist this change when
the need for change is explained and they have endorsed at least some aspect of that need.

   2.  Implement "Quick Wins"

Throughout this report we highlight some "quick fixes" or short-term changes that APTI
may take to immediately improve its training program (a summary of all "quick fixes" is
presented in Appendix  C). By implementing some of these short-term changes, APTI
can begin to establish  some "quick wins".   Quick wins are immediate  successes that
demonstrate a commitment to and seriousness about making  improvements.

We recommend that APTI focus on foundational quick wins. That is, quick wins that
form the basis for additional wins.  For example,  before incorporating  standards for
interaction  into  all future distance  learning offerings, APTI  should  first  focus on
developing the standards and using them to review existing courses/offerings.

Quick wins are most effective when they are  timely, focused, and visible.
   •   Timely. APTI should begin to implement quick wins as soon after announcing
       the  change initiative as possible.  By waiting too long,  APTI runs the risk of
       appearing uncommitted or unable to make effective decisions.

   •   Focused.   The  best quick wins  are focused.   They are implemented with a
       strategic focus, not as  a part of 50 separate initiatives.  APTI should show that
       each quick win is part of a  strategic focus that will be used  as a  bridge to
       continued change efforts.

   •   Visible. As with having a strategic focus, the best quick wins stand out and are
       visible to all key stakeholders.  Stakeholders need to  be able to see the wins to
       effectively  understand  what  changes are taking place.   Even  if change is
       occurring,  stakeholders are not likely to be motivated if they cannot see what is
       happening.

   3.  Implement Long-Term Changes

Long-term change (e.g., adopting a blended learning approach) requires preparation and a
long-term commitment to success, even in the face of periodic setbacks.  The remaining
section of this report describes factors that APTI should consider to help prepare for and
implement long-term changes to the Air Pollution Training program.

       a)  Create capability and capacity to change

With leadership commitment and support and some quick wins in place, the APTI team
should begin to develop  the capability and capacity for long-term change.   This will
HayGroup                                                                  Page 30

-------
EPA: Results of Benchmarking Study
require identifying and building the key change infrastructure and partnerships needed for
successful change implementation.  Some of the questions that the APTI team will need
to ask during this stage are:

   •   Are the right people in place? For example, does APTI have the right mix of
       roles to effectively design courses/offerings?

   •   Does APTI have the right technology?

   •   Have key stakeholders been receptive to the quick wins?

   •   What are the likely resource constraints?

   •   Does APTI need and can it get additional resources?

We recognize that limited resources is one of APTFs biggest barriers to implementing
needed changes to the training program.  However, once APTFs scope and strategic
direction are clearly defined (e.g., through the  broad-based needs assessment), APTI will
be able to make the most effective use of its limited resources. For example, updating all
courses/offerings on a periodic basis will be less resource intensive if APTI reduces its
course offerings to those that best  meet  customer needs (and can  not be  obtained
elsewhere).

       b) Design and implement solutions

Once the APTI team has a framework in  place for developing the capability and capacity
to change, it should focus on designing the right solutions for change. For example, how
is APTI going to implement live Web training or  create a  systematic  process for
designing courses?  Or, what should be the standards for ensuring interactions in self-
paced Web courses and how should these standards be regulated?

Fortunately, by this stage  in the  process, the APTI team should have  at least the
beginning designs to answer these and  other important questions.  That is, leadership
support should be in place, the APTI change  team should have a vision and buy-in for
change,  some quick wins  should be accomplished,  ongoing partnerships should be
developed,  and there should be an overall  capacity  and  commitment to  change.   As  a
result, the APTI team can focus on designing solutions rather than garnering support, and
may even be able to build off of short-term solutions developed for the quick wins.

       c) Evaluate success

APTI should continually monitor the effectiveness of short- and long-term changes to its
Air Pollution Training program. This can be accomplished through:
   •   Informal feedback from customers and stakeholders (e.g., as to the effectiveness
       of a live Web training method or  a blended learning approach)
   •   Informal feedback  from subject matter experts,  instructional designers, and
       instructors
HayGroup                                                                 Page 31

-------
EPA: Results of Benchmarking Study
        Data from the training evaluation process (e.g., air professional opinions about a
        blended  learning offering;  impact  of a blended  learning  offering  on  future
        performance on the job)
Key Factors for Change

In addition to the steps outlined in this section, we believe that the following factors are critical to
ensure that APTI implements a successful change effort:
Leadership commitment and support. Clear and consistent commitment and support from APTI
(and EPA) leadership will be a critical factor in successfully implementing change in the Air
Pollution Training program.  During our benchmarking  research, we found that the industry
leaders in training have leadership that shows a strong commitment not only to training but also
to implementing new and innovative distance learning delivery methods. Leadership can set the
example of ongoing commitment and support for the entire organization to follow.
Communicate the vision and change plan.  Communicating APTI's vision and desired end state
throughout the  process will help  motivate and  provide  all stakeholders  with a  common
understanding of where APTI wants to take the Air Pollution Training program, and why changes
are needed to the program.  Sending clear and consistent  messages about change helps develop
buy-in, informs  people of courses of action, and gives them the real-time information they need to
act in accordance with the vision.
Partner with JTC and state and local agencies. The JTC and state and local  agencies are key
stakeholders in the Air Pollution Training program and any change initiative that APTI develops.
As with the benchmarking study, we recommend that APTI communicate its plans and seek input
from these stakeholders throughout the change process.	
HayGroup                                                                       Page 32

-------
EPA: Results of Benchmarking Study
                              Appendix A

                             Task 1 Report:
          Evaluation of Current Air Pollution Training Program
HayGroup                                                          Page 33

-------
EPA: Results of Benchmarking Study
                                /.  Introduction

Background
In May 2004, the Education and Outreach Group (EOG) of the Environmental Protection
Agency (EPA) contracted with the Hay Group to conduct a Benchmarking Study of its
Air Pollution Training program. The ultimate goal of the project is to provide options for
improving the current Air Pollution  Training program, and ultimately, EOG's business
performance. The following three tasks are aimed at accomplishing this goal:

    •   Evaluate the current Air Pollution Training program (Task 1)

    •   Identify best practices of leading training programs (Task 2)
    •   Develop a  plan of action for  improving  the current  Air Pollution  Training
       program (Task 3)

This report summarizes our findings from Task 1 - evaluation of the current Air Pollution
Training program.   The findings  in  this report provide an assessment of the practices,
procedures and processes currently used by the Air Pollution Training Institute (APTI) to
develop, deliver, and evaluate training. In this report, we focus on evaluating the current
training program based on input from numerous sources.

In subsequent reports, we will present our conclusions regarding the direction that we
believe the Air Pollution Training program should take.

Methodology
This section summarizes  the methodology used to evaluate the current Air Pollution
Training program.  The steps for evaluating the  current training program are described
below.

Identify evaluation criteria.   The first step  was to identify the  critical air pollution
training practices,  procedures, and processes that should be  evaluated (i.e., evaluation
criteria). These evaluation criteria were identified based on discussions with  APTI staff,
reviews of the training literature, and the Hay Group's  extensive research and experience
in training delivery and evaluation.   The table on the following  page shows the key
aspects of the current training program that were evaluated during Task 1.
HayGroup                                                                  Page 34

-------
EPA: Results of Benchmarking Study
Evaluation Criteria
Needs Assessment
Process
             Types of Information Collected
   Effectiveness of Needs Assessment Survey
   Uses of needs assessment results
Course Content
•  Process for determining course content and delivery methods
•  Availability of courses
•  Extent to which courses are offered that meet student needs
•  Quality of course content
Course Delivery
Methods
•  Effectiveness of classroom courses and instructors
•  Effectiveness of satellite broadcasts
•  Effectiveness of on-line training
•  Effectiveness of self-instructional training
Training Evaluation
Process
   Effectiveness of training evaluation instruments
   Uses of training evaluation results
Conduct subject matter expert interviews.  Hay designed an interview protocol (based on
the evaluation criteria) to capture information that is  critical to understanding  and
assessing the current Air Pollution Training program.  The interview protocol included an
introductory  paragraph to read to participants at the beginning of the interview  and
interview questions.

Hay worked  with APTI staff to identify appropriate individuals to  interview about the
current training program.  An effort was made to interview a diverse group of individuals
who have a good understanding of the training program, its objectives and desired results,
its future direction, and its overall effectiveness. The  following types of individuals were
interviewed:

    •   Members of the Joint Training Committee (JTC)

    •   Members of the MARAMA Regional Consortium

    •   Individuals who run area training centers

    •   APTI course instructors

    •   Supervisors in state and local agencies who have sent employees to APTI training
       courses (some of these supervisors solicited feedback from their employees about
       the effective of APTI training)

    •   Contractors who design  APTI course content and conduct the  training  needs
       assessment

    •   APTI management and staff
HayGroup
                                                  Page 35

-------
EPA: Results of Benchmarking Study
A total of 22 individuals participated in one-on-one interviews in person  at Research
Triangle Park or via the telephone.  Additionally, we spoke to several other individuals in
group  settings (e.g.,  APTI staff members,  JTC members, members of the MARAMA
Regional Consortium) about the Air Pollution Training program.

An experienced  Hay Group interviewer  conducted all interviews using the standardized
protocol.  The interviews focused on:

 •  Gaining  a better understanding of the practices, procedures and processes used by
    APTI to design and deliver air pollution training; and

 •  Obtaining opinions on the effectiveness of these practices/procedures/processes and
    ways  to improve the current training program.

We also used these  interviews to  identify  potential organizations to  participate in  the
benchmarking study that will be conducted in the late July/August time frame. Extensive
notes were taken during each 30 to 60 minute interview.   Information collected from the
interviews was reviewed, and we  identified common themes (e.g.,  many interviewees
indicated  that "the  training  materials  are outdated").   These themes  are presented
throughout this report.

Review key documents.  In addition to conducting subject matter expert interviews, Hay
reviewed  several documents, web  sites, and  APTI training materials  as  part  of  the
evaluation of the current APTI training program.  More specifically, we reviewed  the
following:

    •   APTI web site (e.g., course schedules, training providers, APTFs mission, course
       registration)

    •   Site Coordinators Resource Center web site

    •   EOG FY03 Highlights and FY04  Midyear Accomplishments

    •   Training  Needs Assessment Survey and results

    •   Training  evaluation instrument and results (i.e.,  feedback from students  about
       courses)

    •   Training  materials  (self-instructional  workbooks, web-based  training courses,
       satellite broadcasts)

    •   Statistics on training course attendance and certificates issued
HayGroup                                                                  Page 36

-------
EPA: Results of Benchmarking Study
Report Overview
In this report, we summarize our findings from the subject matter expert interviews and
document/web site review in an effort to provide an assessment of the current training
program.  We provide an evaluation of each of the following aspects of the Air Pollution
Training program:

   •   Program Mission and Direction

   •   Training Needs Assessment Process

   •   Course Content

   •   Effectiveness  of Training  Delivery Methods  (classroom  courses,  satellite
       broadcasts, on-line interactive courses, self-instructional training)
   •   Training Evaluation Process

When describing our findings, we first provide a short overview of what the current APTI
program  looks like.   We then present  a summary of interviewee opinions about the
current APTI training program. Finally, we provide options, identified by interviewees,
for improving the APTI training program.

We conclude the report with  a discussion of our main conclusions about the state of the
current Air Pollution Training program and next steps in the benchmarking study.
HayGroup                                                                  Page 37

-------
EPA: Results of Benchmarking Study
                    //. Program Mission and Direction

Many interviewees praise APTI on its classroom training courses, particularly those with
a laboratory  component; the breadth  of courses  provided; and its efforts to create a
blended approach to training in response to  the  current budget and travel restrictions
faced by state and local agencies. However, some believe that APTI is no longer the
premier training group that it was in the past and that it is not currently meeting the needs
of state and local agencies.

Several interviewees feel that training  is not  a priority at EPA and that EPA needs to
recommit to its original function - training the people who do the "nuts and bolts" work
at state and local agencies (i.e., air professionals).  The perception is that APTI focuses
too much on outreach and not enough on providing technical training to air professionals.
As one interviewee put it,  "EPA puts  good lip service to training  but doesn't put the
dollars into training."  Additionally, APTI has dramatically reduced its staff over time.
As  a result,  some  of the regional  consortiums (e.g., NESCAM, MARAMA)  have
established their own training institutes  to supplement APTI training. Some big states no
longer rely  on APTI  as a training provider.  Additionally,  some feel that APTI  is
currently too focused on providing air pollution training internationally.  Interviewees
believe that there is a need for APTI to take care of the state and local agencies before
expanding its training to other countries.

It should be noted that many interviewees commented that APTI leadership (particularly
Lourdes Morales) is very committed to providing the best training to  state  and local
agencies.  Additionally, interviewees recognize that  APTI has been faced with budget
cuts over the past few years, which hinder its ability to satisfy all of the needs of state and
local agencies.

One interviewee recommended that EPA create one group (versus separate groups, such
as APTI) that is responsible for training across EPA.  This training group would get one
budget and be held responsible for delivering all  EPA  training (e.g., air quality, water,
land).
HayGroup                                                                  Page 38

-------
EPA: Results of Benchmarking Study
               ///. Training Needs Assessment Process
EPA contracts with SYSTANI, Inc. to assist in administering a bi-annual Training Needs
Assessment Survey.  The last survey was completed in March 2004 with data collected in
2003  for projecting training needs into  FY2005/2006.   The data are collected  from
members of the State and Territorial Air Pollution Program Administrators (STAPPA)
and the Association of Local Air Pollution Control Officials  (ALAPCO).  The training
needs survey is designed, administered and reported with heavy involvement from the
Joint Training Committee (JTC).

The survey has two primary purposes: 1) to assess training  needs related to currently
available courses  and 2) to assess future training needs.  The  survey covers  only
classroom-based courses, including those  provided by institutions other than APTI  (e.g.,
NETI, CARS, and RACC). A total of 88 agencies completed the FY 2005/2006 Training
Needs Assessment Survey.   Thirty-eight responses were from state agencies, 46  were
from local agencies, and four were from tribal agencies. Responses are also broken down
by regional consortium (e.g., CENSARA, LADCO, MARAMA).  The majority of the
respondents work as training coordinators  or managers.

For each classroom course, respondents are asked to project how many people from their
areas will be attending if the course is given "In the State," "In the Region," or "Outside
of the Region."  Twenty-nine classroom-based courses tied for the top ten ranked  courses
in terms of prospective number of attendees in FY2005/2006. Of these courses:

   •   15 were APTI courses;

   •   10 were CARB courses;

   •   3 were RACC courses; and

   •   1 was a NETI course.

This list of training providers  represents a significant change from the  last  survey
administered  (FY2002/2003  report administered in 2001).  At that time,  APTI  courses
comprised all of the top ten ranked courses in terms of prospective numbers of attendees.
As can be seen above, in the most recent survey, CARB RACC, and NETI had  courses
ranked in the top ten (in terms of prospective number of attendees in FY2005/2006).

Of the 13 potential new training topics that appeared in the top ten list, nine of the topics
were  also listed in  the  FY2002/2003 survey  results.  Twenty respondents specified
"Other" potential new training topics that were not included on the pick list.

Projected attendance for courses if offered "In the Region" is down by  approximately
63% from the FY2002/2003 survey results. Projected attendance for classroom courses if
held "Outside of  the Region"  is down  by approximately 88% from two years ago.
Similar results were found  for  staff projected to  attend potential classroom courses.
Attendance for  potential classroom courses if offered "In  the Region" is  down by

HayGroup                                                                 Page 39

-------
EPA: Results of Benchmarking Study
approximately 52% from the previous survey two years earlier. Attendance for potential
classroom courses if held "Outside of the Region" is down by approximately 82% from
two years ago.

The  Training Needs Assessment Survey also contains a question related to the factors
that  influence sending  staff  to training.   In  other words,  what factors  would either
discourage or encourage managers from sending staff to training.  Respondents indicated
that  the biggest  factors that would  discourage them from sending staff to training are
costs, travel restrictions, workload,  and staff shortages.  Factors that would encourage
them to send  staff to  training include  course relevance,  staff turnover,  and  staff
development.

At the end of the Training Needs Assessment Survey, respondents are asked whether they
have any additional  comments or suggestions related to their agencies' training needs.
Reponses cover several different areas and are bulleted below:

   •  Greater outreach by APTI is required to market and publicize training courses.

   •  Agencies need to be given at least 30 days notice that a classroom training course
       is coming up if they are to have enough time to process travel requests.  Some
       agencies require up to  6 weeks.

   •  Attendance at classroom training  is largely determined by  budget.   Training
       attendance must be prioritized based on the relevance of the course to the work
       being carried out in the job.  Training methods should take into  consideration
       budget/travel restrictions.

   •  Workload also dictates whether or not staff can take the time to attend classroom
       training courses.  Budget restrictions have resulted  in fewer people to do the
       work.

   •  More use of self-training is needed,  especially interactive web-based methods.

Current Views of Training Needs Assessment
Few interviewees commented on the effectiveness  of the Training Needs Assessment
process.  Those who did comment believe that, in general, the needs assessment is  a
useful tool for forecasting the future training needs of air professionals.  They  view the
needs assessment as a snapshot in time and as a good technique for identifying trends and
peaks (e.g., when a large group of people will need a basic air pollution training course).
As one interviewee put it, "the needs assessment is a great tool and an effective way for
making training needs known."
HayGroup                                                                  Page 40

-------
EPA: Results of Benchmarking Study
However, the following aspects of the needs assessment process were identified as areas
for improvement:

Other feedback mechanisms are used to  supplement the  Training Needs Assessment
Survey.  Some interviewees indicated that in addition to the Training Needs Assessment
Survey, they solicit other feedback to make decisions  about course content and delivery
methods.  For example, one interviewee reported that he/she conducts interviews with
directors who run state and local agencies and with training providers to determine the
training needs of the consortium.   This finding may  indicate that the current Training
Needs  Assessment  Survey is  not broad  enough to  provide  comprehensive data  for
decision-making about future training content and delivery methods.

Not  enough is done with  needs  assessment  results.   Some  interviewees  questioned
whether APTI uses the results  of the Training Needs  Assessment Survey in its internal
decision-making regarding course  content.  As one interviewee put it, "I am not  sure if
APTI is using it (the needs assessment) for anything more than scheduling based on the
'number of staff  likely to attend  training' question."   Interviewees believe  that other
organizations (e.g., CARB) are  using the needs  assessment results to develop  new
training topics/courses.

Some decisions about training  are not well understood. Some interviewees (particularly
members of the JTC)  indicated that they are not always clear about how decisions
regarding training content are made by APTI.  As one  interviewee put it, "we sometimes
wonder where some of the training comes from and why APTI picked a certain course or
satellite broadcast."

Options for Improvement:
Interviewees provided  some suggestions  for  improving the current needs  assessment
process.
    1.  Include a skills  assessment in the Training Needs Assessment Survey in addition
       to projecting the numbers  of attendees.  For example, have respondents assess
       their group's overall skill  level related to each Potential New Training  Topic.
       This information  would  help  course  developers  determine at  what  level
       (beginning, intermediate, or advanced) training  should be targeted.

   2.  APTI should clearly communicate what changes to training are being made as a
       result of the needs assessment survey and link the results to course development
       efforts.

   3.  Expand the needs assessment beyond classroom courses.  For example, use the
       assessment to attempt to gauge what types of satellite and web-based courses will
       be needed by air professionals in the future.
HayGroup                                                                 Page 41

-------
EPA: Results of Benchmarking Study
                             IV.  Course Content

APTI currently develops few new courses from scratch each year (0-2 new classroom
courses per year).   The focus  is on updating current courses (which is a key concern
raised  by interviewees) and converting  course material  to be delivered in a  different
medium, such  as the web.  In  2003,  APTI updated four courses (e.g., Combustion
Evaluation, Principles and Practices  of  Air Pollution Control).  All  development and
updating of APTI courses is done by contractors with insight from EPA subject matter
experts.

Current Views of Course Content
We consistently heard from interviewees that in cases where APTI course content is up-
to-date, the content itself is very good. APTI courses are considered to be comprehensive
and cover a wide breadth of subject matter.

However, some interviewees stated that  they prefer attending CARB, NETI, or RACC
courses over APTI courses because courses provided by  these agencies are consistently
more up to date, more specialized, and have more knowledgeable instructors than APTI
courses.  Additionally,  many consortia members are developing their own courses  over
time to supplement APTI courses.  As described below,  interviewees identified several
opportunities for improving APTI course  content.

Need to update course materials.  By far the most prevalent comment regarding course
content and design  is that many  APTI courses are  in need of updating to the point  of
being obsolete, and APTI is not updating these courses fast enough. For example, some
interviewees believe that the Introduction to Air Pollutants course is so outdated that it is
not even worth attending.  Some  courses have  not been updated for over  15 years.
Additionally, some APTI course materials contain inaccurate data.  Interviewees  also
commented that in the course evaluations, students complain about course materials more
than anything else.  The FY2005/2006 Training Needs Assessment also reveals that many
students are disappointed with the quality of APTI courses and believe that many courses
are out-of-date.

Because APTI course materials are out-of-date, instructors often must update the course
material prior  to delivering the course.  As one participant put it, "they (EPA)  expect
industry to be  up to date; I think it is reasonable that they keep themselves up  to date."
Some  interviewees believe that  updating  existing courses should be a priority  over
developing new courses.

Outdated  course content  is  also  an issue for paper-based and  .pdf  on-line  self-
instructional courses.  That is, on-line  .pdf courses are  not looked upon favorably by
those interviewed  primarily because course materials are out of  date.   Interviewees
indicated that if course materials are old and contain out-of-date or incorrect information,
then it could be detrimental to make them easily accessible through the web.   As one
interviewee put it, "this serves to propagate the inaccuracies."

HayGroup                                                                 Page 42

-------
EPA: Results of Benchmarking Study
One interviewee stated  that outdated APTI  course materials  actually influence  the
selection of contractors (instructors) in the sense that contractors who have taken their
own time to update APTI course materials will be selected over contractors who teach the
course content as it  is.  Interviewees reported that APTI courses have to be modified
before they are  delivered and that these modifications require a significant amount of
instructor resources.  Related to this, some individuals who have updated APTI courses
out of necessity are unwilling to share their updates with APTI as they see the revised
course materials as their own intellectual property. One implication of the need to update
course content is that there is a lack of consistency in how APTI courses are delivered
(i.e., course instructors modify the course content in different ways).

Other interviewee comments related to the need to update APTI course content include:

    •   There is a  large  need for introductory courses to be updated  because  of staff
       turnover in state and local agencies due to retirement.

    •   Policies and case laws change, so training must be modified to account for these
       changes.

    •   In  addition to  the need for  course  content to be updated,  some interviewees
       believe that laboratory equipment/sampling equipment needs to be updated.

    •   There is a need to update lab books used in classroom training (e.g., to reflect new
       instruments).

It  should be noted that many interviewees  recognize that budget restrictions limit the
number of course updates that EPA can handle.

Some course material is of poor quality.  In addition to needing to be updated, the overall
quality of the course materials  is  viewed as being poor.  Booklets are sent out  that are
difficult to  read; pages are missing, upside down, or out of order; materials arrive late; or
the wrong materials are sent out. Interviewees believe that basic quality assurance is not
carried out on  a consistent basis.   Some interviewees reported that  it is frustrating to
either have to fix the  same typos every time the materials are received or continue to
receive materials with the same errors after repeatedly reporting  them to EPA.  As one
interviewee put it, "the content may not be out of date, but it looks out  of date."

There is overlap in course content.  Course content is also seen by some interviewees as
overlapping too much in certain courses.  Supervisors who have sent  their employees to
training indicated that their students complain that they sometimes take a course that is
too similar to another course they have already taken. The FY2005/2006 Training Needs
Assessment also found that  many  students believe that there is too much overlap in
content among APTI courses. Interviewees agree that with training dollars  and  training
time being  stretched to the limit, there is little tolerance for learning about the same topic
in two or more different courses.
HayGroup                                                                  Page 43

-------
EPA: Results of Benchmarking Study
There is a need for more specialized courses.  Several interviewees commented that more
specialized courses  need to be  added  to  the  course schedule each  year.   Some
interviewees perceive that EPA is not spending enough time teaching courses that have
specialized content.  Additionally, APTI is not  offering enough  courses that are  at a
higher, more strategic level.   One interviewee  stated that  they hired contractors to
develop a specialized course because the need was not being met by APTI.

Incorporate more real-world applications to training content.  Some of those interviewed
believe that while the APTI courses offer good technical information, they do not always
cover how the information relates to industry or the "real world".  Some courses focus
too much on theory and not  enough on practical  applications in the field.  Many APTI
courses give air professionals background information but not the process skills they need
to do their jobs (although, lab courses do develop these types of skills). As one  course
instructor who was interviewed put it, "the course needs to tell people what they need to
know to do their jobs in the field." One supervisor also stated that "meetings put on by
local  groups/associations are more meaningful  than  APTI courses  because there  are
people attending from industry; you get more dialogue, different perspectives."

Need for more courses and better scheduling.  Some interviewees indicated that APTI
needs to  make more courses available each year, and that the number of courses APTI
provides  has declined over the past few years (due to budget restrictions). Additionally,
comments were  made about the importance  of course timing and scheduling, and that
courses should be staggered throughout the year.

Options for Improving Course Content
Interviewee recommendations for improving APTI course content are summarized below.
    1. Provide clear direction about the process for updating course content.  Questions
      that should be addressed by EPA include:
      a) Will EPA take the lead in updating APTI courses?
      b) How will consistency be ensured if different contractors/groups are updating
          the same APTI courses themselves?
      c) Who  owns  the intellectual  property  when  an outside group  spends time
          updating out-of-date APTI courses?
      d) How many courses will APTI be able to update per year?
      e) Who will determine which courses are updated and in what order (e.g., update
          highest volume course first)?
      f)  What process will be used to update specific course content?
      g) Given the fact that many APTI courses need to be updated, should certain
          courses  be "shelved" as opposed to having inaccurate information going out
          over the web site?
      h) Should  APTI  turn over certain outdated courses to other institutions (e.g.,
          RACC)?


HayGroup                                                                 Page 44

-------
EPA: Results of Benchmarking Study
       i)  What is the  process for certifying an outside course?   Is there a standard
          process?

   2.  Put into place regular review cycles to keep courses up to date.  For example,
       review course content, objectives, and resources on a three-year review cycle.
       Ensure that course materials are relevant to what is going on in today's regulatory
       world  and incorporate  the use  of  modern technology (e.g., some  sampling
       equipment has changed drastically).

   3.  At a minimum, there should be a process for checking course materials before
       they are sent out.  Missing pages, upside down  pages, and pages out of order
       should be  eliminated. When typos are found in course materials, they should be
       corrected before being  sent out  again.  Interviewees feel  strongly that APTI
       should take responsibility for correcting errors in its courses.

   4.  Review and redesign courses to minimize content overlap.  However, given the
       need to update course content, this is likely not a big priority. A better alternative
       may be to review course overviews and objectives with the goal that prospective
       course participants can clearly  see whether or not the material covers topics they
       have already learned.  This will  enable participants to make  a judgment as to
       whether they should attend the course or download the materials from the web.

   5.  Review the look and feel of the course content. Interviewees suggested updating
       the course  content to be much more visually oriented by adding pictures and
       making the science aspect much more interactive by putting courses on the web.

   6.  Centralize  the  development  of  air pollution training  (e.g., one  interviewee
       indicated that there are two  agencies that are  spending time and resources
       developing the same course separately).  EPA should provide more centralized
       support in course development.
HayGroup                                                                  Page 45

-------
EPA: Results of Benchmarking Study
                             V.  Course Delivery

There  are four types  of courses  provided by  APTI:  classroom,  satellite,  on-line
interactive, and self-instructional (.pdf on the web  and paper-based).   In the past few
years, APTI has put an emphasis on moving more training to an  on-line and satellite
format (distance learning techniques) to meet the needs of a largely dispersed customer
population faced with travel and budget restrictions.  Additionally, on-line and satellite
training allow state and local agencies to train new hires  and others who require air
pollution training in a timely manner (just-in-time training).

For the purposes of this report, on-line courses will refer to web-based interactive courses
only; self-instructional courses will  refer to  hard copy manuals and .pdf files that are
available for download on-line but include no interaction.

In the remainder of this section, we discuss and evaluate each of the four training delivery
methods.

Classroom Courses
Currently,  classroom  courses  are  those which  are  presented   live by recognized
professionals (e.g., consultants,  university faculty)  and held  either  at an Area Training
Center,  a related  university,  or on-site  at  a state or local agency. APTI is actively
involved in updating and developing courses.  In 2003, four APTI courses were updated
and thirty-seven courses were delivered nationwide.  Students can receive certification for
courses  by completing  class  assignments, passing  a final exam,  attending  and
participating in class sessions, and submitting a course evaluation.

Current Views about Classroom Courses
By  and large, most people interviewed prefer the classroom delivery vehicle when time
and travel is not an issue.  This perception is held by interviewees, as well  as by students
who have completed APTI  courses  (as indicated by student evaluations and feedback
from supervisors who have sent their employees to training).  Student evaluations of
classroom training courses tend to be very favorable, and many students feel that the
course materials will be useful  to their jobs.  Classroom training  allows for the most
interactions with the instructor, personal attention to questions and students learning from
one another. Classroom training is seen as the most  effective delivery method, especially
for higher level technical classes and those with a laboratory component.

However, most interviewees agree that classroom learning is not appropriate for all APTI
courses, especially in light of recent budget cuts. Despite the fact that classroom training
was the preferred  method in the past, most  agencies are under budget constraints and
travel restrictions  that prevent  them from getting  to the classes.   Therefore,  agency
training professionals recognize the need to have a complement of vehicles, although
some interviewees believe that classroom courses should be mandatory for classes with
HayGroup                                                                   Page 46

-------
EPA: Results of Benchmarking Study
laboratory  components.  Additionally, many interviewees indicated that the quality of
APTI classroom courses vary greatly and primarily depend on the instructors.

 A summary of comments made by interviewees and students (in response to student
evaluation forms) regarding the APTI classroom courses is provided below.

Overall,  classroom is  best of the four delivery methods.  While there  is a need for a
variety of training vehicles, classroom delivery of courses is viewed as the most effective.
One interviewee said, "students are completely dedicated to learning in that environment
and the diversity of students allows for different perspectives and better learning."

However, most interviewees indicated that while classroom  interaction is effective,  it
should not be used for all courses. The perception is that while it is a good technique, the
classroom is not the most efficient way to  deliver  training, especially for introductory
classes or classes that need to be attended by large numbers of people.

Laboratory component is very valuable.  There is agreement among interviewees that the
laboratory component (included as part of some APTI classroom training courses) is very
effective as a learning tool. Students are given the opportunity to actually learn the skills
that  they need to do their jobs effectively.   Student  evaluations  of laboratory training
courses echo this finding; students feel that the laboratory component is the best part of
the classroom training experience.  Students believe that the lab helps them understand
the course  materials, and they like the  hands-on learning component of the classroom
training.  As one student indicated in the student evaluation, "the best part of the course
was  hands-on modeling and the exercises."

Students want more practical applications in non-laboratory courses. Some interviewees
commented that lecture only courses  need to be supplemented with more "real world"
examples to emphasize how, for example, sampling techniques, equipment, mathematical
equations, will actually be used in the field.  Student  feedback from course  evaluations
also suggests that they want more hands-on  applications. Some courses focus too much
on theory and "put people to sleep." Instead students  want to know "what do I do with
this  information?" "The course needs to tell people what they need to know to do their
jobs in the field."

Mixed opinions about the effectiveness of instructors.  While the classroom environment
allows people to remain engaged at all times, the effectiveness of the instructor can have
a huge effect on whether the training is viewed as worthwhile.  Some interviewees feel
that there needs to be a process for ensuring that instructors are effective (e.g., monitoring
student feedback on the effectiveness  of instructors, observing  instructors delivering
classroom  courses).  Interviewees view instructors as having a thorough grasp of the
technical information  but do  not see some instructors as effective in delivering the
material. The general feeling is that  the best instructors are  those who have "worn all
hats" (for example, worked for EPA, worked for private industry in a monitoring capacity
and worked on the design or update of APTI courses).  It should be noted that, overall,
the  student  evaluation  feedback that  we reviewed  was very favorable regarding


HayGroup                                                                  Page 47

-------
EPA: Results of Benchmarking Study
instructors.   Instructors are viewed as dynamic and interesting,  and students  like the
interactions among the instructors.

Lack of consistency in how classroom courses are delivered.  Interviewees believe that
classroom  courses are delivered  inconsistently  when  APTI materials  are  out-of-date
because instructors must supplement APTI materials with their own. Inconsistencies can
also occur when instructors travel to states that need the training delivered on-site. While
traveling instructors are viewed  positively in  light of the travel  restrictions that  most
states  are  under,  the travel  can  lead  to a less integrated  training curriculum  and
inconsistent learnings.

Options for Improving the Classroom Courses
Interviewees  had several recommendations for improving APTI classroom courses.
1.  Consider  a "train-the-trainer"  approach so that  there will be  people on site at the
    agencies that are able to teach the course materials locally.
2.  Increase the number of real world examples and hands-on applications included in
    classroom courses that  are primarily in a lecture format.  Supplement the lecture with
    practical illustrations when applicable.
3.  Increase the prevalence of group exercises in order to expose attendees to one another
    and their  respective issues and diverse perspectives.  This practice can increase group
    learning for the class.
4.  Ensure that instructors are not only knowledgeable about the materials but also skilled
    at teaching. Use course instructors  who not only have  educational experience but
    who have "done it in the field."
5.  Make  sure  classes (including supplemental materials)  are  consistent across  all
    courses, and then take the instructors on the road (one interviewee indicated that
    "EPA should travel more to the states to deliver training").
6.  Shorten the length of classroom courses (e.g., 1 day training as opposed to 3 to 5
    days).  Shorter courses  reduce costs and time away from the job.
7.  Reserve classroom courses for those  having laboratory assignments  and  covering
    more technical, specialized topics.
8.  Ensure that classroom locations are easily accessible to most of the country.
Satellite Broadcasts
APTI uses its Air Pollution Distance Learning Network to deliver satellite broadcasts to
air professionals (and other interested parties) throughout the country in a cost effective
manner.  The satellite broadcast is used to deliver informational broadcasts (e.g., to keep
air professionals up to date on  new regulations), as well as technical telecourses.  For
example,  APTI course 427 (Combustion Evaluation)  has been converted to a satellite
course to be delivered in 4-hour blocks over a four-day period.  Some satellite broadcasts
allow  participants  to fax in questions and  selected questions are  answered by the

HayGroup                                                                   Page 48

-------
EPA: Results of Benchmarking Study
presenters.  Participants are not required to register for satellite broadcasts, except for
300- and 400-level telecourses.

 In 2003, APTI delivered  a total  of 14 broadcasts covering  66.5  broadcast hours.
Additionally, several broadcasts have  already been delivered or are in development for
2004.

There are approximately 100  downlink sites throughout the country.  Site coordinators
are responsible for advertising upcoming  broadcasts, arranging the room where the
broadcast will take place, and other logistical duties. Videotapes of past satellite courses
can  also  be used  as  self-instructional courses. A limited number of broadcasts are
simulcast and can be viewed on a PC.

Current Views  about the Satellite Broadcast
There are mixed opinions about the  appropriateness of using the satellite broadcast as a
training delivery method.  Some interview participants do not like the satellite broadcast
as a training delivery  method. This  camp  believes that APTI has placed too  much
emphasis, and budget, on the delivery of training via satellite broadcasts at the expense of
other delivery methods (e.g., many classroom courses have not been updated).  As one
interviewee indicated, "the direction towards satellite is a  complete waste  of time and
resources."

Most interviewees, however,  recognize that state  and local  agencies have budget
limitations,  travel restrictions, and the need to minimize employee time away from the
job, making it necessary for APTI to deliver training in other ways besides the traditional
classroom approach.  This finding is  echoed by the 2004  Training Needs  Assessment
through  which participants urged APTI to consider budget/travel restrictions  when
making decisions about training delivery  methods.  Satellite broadcasts are  seen as one
cost-effective way to deliver training to a large, geographically dispersed audience.

Another benefit of the satellite broadcast is that it can provide  timely and consistent
information to  air professionals.  For example, satellite broadcasts allow 300- and 400-
level training courses to be delivered in a consistent manner, which is not always the case
with classroom training. Additionally, because satellite broadcasts are taped and can be
re-shown in a video format at any time, state and local agencies can provide just-in-time
training to new hires.

Although many interviewees recognize the potential value of satellite broadcasts, there is
agreement that the current APTI satellite broadcast delivery method is not effective and is
in need of improvement.  Many satellite broadcasts play to a very small audience or no
audience at all. Proponents of the satellite training method feel that APTI should focus
its limited resources on improving its  satellite broadcasts so that air professionals have
easy access to  APTI training (particularly those who are impacted by travel restrictions
and  can not attend classroom training).   It should be noted  that some interviewees
indicated that  the  satellite  broadcasts have improved over the  past few  years (e.g.,
instructors are more effective, satellite productions are better).

HayGroup                                                                   Page 49

-------
EPA: Results of Benchmarking Study
The following is  a summary  of the  comments that were made about  APTI satellite
broadcasts:
Speakers are not effective presenters.  Many interviewees indicated that the speakers who
deliver the satellite broadcast  are often dull  (speak in a monotone voice)  and do not
present the information well. Participants do not find many of these speakers engaging,
and have  difficulty paying attention to broadcasts that last for two  hours or more (and
thus,  do not get much out of the  broadcast).  It is particularly important  to  have an
engaging  speaker because many of the satellite broadcasts are presented in  a lecture
format.   Participants in the satellite  broadcast often end  up reading the  text that  is
provided because they do not learn enough from the speaker (due to the poor presentation
style). As one  interviewee put it,  "the material was good but the  presenter got in the
way."  Another interviewee  stated,  "you need to  pick  the right people who  can
communicate and work the medium to deliver the broadcast."

It should be noted that some interviewees believe that APTI has improved in this area and
is now using  some of its best instructors to deliver the satellite broadcasts.

Limited opportunity for  interaction.  Perhaps the biggest complaint about the satellite
broadcast is that it provides limited opportunity for student interaction (e.g., discussion
among participants, interactions with the instructors).  Most interviewees believe that it is
very  difficult to sit through a long television broadcast in a lecture format without
opportunities  for  interaction.   Furthermore,  a lecture-style  delivery method  where
students passively watch the television (versus one that facilitates student interaction) is
not an effective adult learning technique (i.e.,  not  effective for retaining information,
particularly of a technical nature).  One interviewee  indicated that "the satellite  can be a
bit boring, not because of the instructors but because of the mode."

Even when  the broadcasts  allow  for participants  to fax  in  questions, there  is the
perception that few participants actually get their questions answered (e.g., speakers are
often reluctant  to  answer  some questions,  such as  about  policy issues).   As  one
interviewee put  it, "in theory, you can ask questions during the satellite broadcast, but in
application, you can't." Others feel that participants in a satellite broadcast are less likely
to fax in  questions than  they would be to  ask a question in a classroom  setting.
Additionally, the process of faxing in questions is not viewed as being "real time."

Length of broadcasts are too long.  Some believe that the satellite broadcasts are too long
(considering  that there is little interaction and many speakers are  not dynamic).  For
example, one interviewee indicated that it is not effective to do four-day satellite courses
(which occurs with 400-level courses) because "you will bore students."

Location of satellite link is not always convenient.  Some interviewees reported that the
location  of the  satellite link may limit  attendance for  some air  professionals.   For
example, some air professionals have to travel to the downlink location (in a time when
state and local agencies have travel restrictions).
HayGroup                                                                    Page 50

-------
EPA: Results of Benchmarking Study
Topics do not meet customer needs.  Some interviewees commented that the information
covered by the satellite broadcasts does not meet the needs of air professionals.  For
example,  many broadcasts focus  on national topics versus local issues.   As one
interviewee  put it, "one broadcast focused on air toxics, but we don't have toxics."
Another interviewee believes that the satellite broadcasts are not really training but PR
for EPA.  One interviewee stated that "the broadcasts are good touchy-feely stuff versus
training."  Another commented that the satellite is sometimes used to provide information
about topics that have nothing to do with air pollution.

Satellite was down for several months.   Many interviewees expressed frustration that
EPA  lost  its satellite for six  months.  They  also indicated that EPA  did  not provide
adequate  communication  when  the satellite  was  down.   As  a result,  many site
coordinators lost interest in the satellite as a training delivery method. While the satellite
was down for six  months, for two of those six months, APDLN was able  to show re-
broadcasts.

Options for Improving the Satellite Broadcasts
Interviewees provided several recommendations for improving the satellite broadcasts
delivered by APTI:
   1.  Use the satellite broadcasts primarily to:
          a.  Provide information about subjects of interest to a broad audience (not just
              air professionals)
          b.  Deliver refresher training and less technical courses
          c.  Show a panel discussion of topics covered in other courses
   2.  Enhance the  satellite technology  so that air professionals can view  the satellite
       broadcasts from their PC (if not already done)
   3.  Do more marketing and outreach about satellite broadcasts
   4.  Incorporate opportunities for interaction into  the satellite broadcast format,  such
       as:
          a.  Include a small group exercise facilitated via conference call by EPA
              representatives
          b.  Include a workbook that is referred to and utilized during the course
          c.  Use a state/local agency representative to facilitate discussion and answer
              questions  during  the  sessions (with support from an APTI  facilitation
              guide)
          d.  Include a studio audience that asks questions of the presenters as part of
              the broadcast
   5.  Require  read ahead materials to supplement the  broadcast  (similar to  self-
       instructional course materials).
HayGroup                                                                   Page 51

-------
EPA: Results of Benchmarking Study
   6.  Shorten the length of broadcasts (e.g., one hour) to increase attendance and the
       potential for learning.
On-line Interactive Courses
In the past few years, APTI has placed a greater emphasis on moving training to a web-
based format.  Both classroom and self-instructional courses are being converted to this
medium. Results  of  the FY2005/2006 Training Needs  Assessment  highlighted the
importance of providing students with opportunities for self-training,  especially through
interactive web-based methods.

For FY2004, APTI will select the last group of courses as virtual  classroom courses.
These  courses  should be  operational by October 2004.  Several other courses being
developed as web-based courses include: Emissions Inventory Introduction; Orientation
to Air Pollution  (Orientation to Air Pollution  is on the web  now);  Title V Citizen
Training; and Ozone and Your Patient's Health.  Examples of on-line interactive courses
currently  used by APTI include  Risk Based  Air Toxics;  Air   Pollution  Control
Orientation; and Introduction to Air Pollution Control.   On-line, interactive courses do
not require registration.  There is a final  exam for web-based courses and participants can
now receive CEU credits.

Current Views about On-line Interactive Courses
Although APTI considers two types of courses  to be web-based courses -  .pdf files of
paper materials and interactive web-based course materials - we will focus on  only the
interactive courses in this section (the .pdf courses are reviewed in the next section, Self-
Instructional Courses).

Interviewees consider APTFs on-line interactive courses to be very good, especially for
courses covering introductory material.  The  on-line interactive medium is viewed as the
best medium (better than the satellite broadcasts) for mass distribution of information and
teaching of foundational materials.  Interviewees believe that it is very easy to access on-
line training courses (typically at one's PC), which is important due to budget and travel
restrictions. On-line courses are also viewed favorably because:  1) students have the
ability to  go at their own pace;  2) technical information is presented in  a consistent
manner (which is not always the case with classroom courses); and 3) there is an element
of interaction which holds participants'  attention.  Classes that are currently on-line and
interactive  are also viewed as  being more up-to-date  (e.g., than  classroom  or self-
instructional, paper-based courses).

The on-line training delivery method is an area that interviewees view as having the most
potential to make a positive impact for  the least amount of money (i.e., a cost effective
way to enhance the effectiveness of APTI training).  Most interviewees agree that more
on-line, self-instructional courses needed to be made available. However, the consensus
is that these types of training must be more interactive if they are to be effective.
HayGroup                                                                   Page 52

-------
EPA: Results of Benchmarking Study
Below is a summary of the  specific  comments that were  made about APTFs on-line
interactive courses:

On-line interactive courses are working.  Most interviewees had positive comments to
make about  the  APTI  courses that are  on-line and  interactive.   One interviewee
commented that, "APTI  has  added interactive computer courses which are very good;
most students provide feedback that the courses are good."  Another interviewee stated,
"APTI is going in the right direction with computer-based courses."

On-line  interactive  courses work  well  for  introductory materials.    Interviewees
mentioned multiple times that the accessibility of these types of classes was great for
introductory classes or those in which the EPA needs to  distribute information to a large
number of people.   One interviewee  indicated that the  "virtual classroom is great but
should be used for introductory courses only.  This  allows everyone  to have a certain
level of background."  "This is a priority because  it  is most cost-effective and [the
student] can  stop and go at their own pace." However, some interviewees  questioned
whether web-based courses are  as useful for intermediate or advanced  course content as
they are for introductory course content.

There is a need to incorporate more interaction into on-line courses.  As with the  satellite
broadcast delivery method, interviewees believe that one way to improve the APTI web-
based  courses  is to add  more opportunities for  student interaction (e.g.,  through
simulations, chat rooms). As indicated previously, training methods that facilitate student
interaction are more effective for retaining information delivered in training courses than
are methods without an interactive component.

Options for Improving the On-line Interactive Courses
Interviewees  provided  several  recommendations for  improving  on-line  interactive
courses.
1.  Use on-line interactive courses for  introductory  classes and classes that are taken
   most frequently.
2.  Add additional opportunities for interaction to the on-line courses including regularly
   scheduled "chat rooms" where participants can have their questions answered and
   interact with instructors. This could be done at designated times in  order for  students
   to ask questions and discuss issues simultaneously.
3.  Incorporate interactive video conferencing with the on-line classes.
4.  Use more dynamic authoring tools for web-based training.
5.  Add more computer-based simulation in order to demonstrate the equipment that the
   students will have to use in the field.

Self-instructional Courses (Non-Interactive)
APTI currently has two forms  of self-instructional (non-interactive)  courses, a paper-
based course  where APTI mails out the hard copies  of materials in response to student

HayGroup                                                                  Page 53

-------
EPA: Results of Benchmarking Study
requests, and  an on-line version where the same  materials are downloaded onto the
Internet in a .pdf format. APTI is currently committed to putting all hard copy materials
on-line, so mailings of self-instructional workbooks will not continue in the future.
In 2003, 1,583 students registered for self-instructional courses. It is expected that this
number will grow in the next year due to continued budget restraints.

Current Views about Self-instructional Courses
Interviewees have mixed opinions regarding the APTI self-instructional delivery method.
When  course materials are out of date, self-instructional courses are viewed very
negatively.  Additionally, some interviewees believe that these courses have minimal
benefit because there is no interaction, and no opportunity to have direct communication
with instructors or other students.  This camp sees self-instructional training as simply
"feeding information" to students.  As a result, many students do not ever finish the self-
instructional courses and probably do not learn much from these courses.

Some interviewees, however, believe that the self-instructional method  of training  is
valuable for certain types  of classes and students.   One  interviewee commented  that
"these courses are good for students who are just starting to learn about air quality.  The
materials provide a good, basic understanding."  This person also noted,  however,  that
the materials need to be updated first before putting them on-line.  Another interviewee
mentioned that many of the engineers taking these classes are visual learners, so they like
the self-instructional materials. Most interviewees believe that the two self-instructional
media (paper-based and .pdf)  are equivalent, although access to the .pdf files is seen  as
more convenient.

Below is a summary of specific comments that were made about APTI self-instructional,
non-interactive courses (i.e., courses that are paper-based or on-line  in a .pdf format):

Participants want more interactions. Interviewees indicated that students would like their
questions answered while they are taking the self-instructional course, not after. Students
also desire more practice problems, and would like to  see what questions they missed on
the final exam rather than just whether they passed.  In this way, they can learn from their
mistakes. One supervisor who has sent employees to training questioned whether any
lasting learning had taken place although the supervisor's staff had passed their exams.
Some interviewees indicate that adding more visual cues, and including interaction would
help students retain the course  information.

Course  materials are not always  reviewed by students.   Some interviewees  state  that
those who download the courses off the web rarely read all (or in some cases, any) of the
course material. Instead they focus on responding to test questions.  One  interviewee
indicated, "this problem is  especially prevalent with courses in a .pdf format instead  of
the interactive web-site."

Supervisors  want more controls around testing.   One interviewee  (a supervisor who
sends employees to APTI training) is having problems with staff simply completing the

HayGroup                                                                   Page 54

-------
EPA: Results of Benchmarking Study
open book test and not reading through the materials when .pdf courses are downloaded
from the web.
Options for Improving the Self-Instructional Courses
Interviewee  recommendations for improving self-instructional courses  are summarized
below.

1.  Update the materials before putting them on-line.
2.  Add  some visuals  to the materials in  order to make them more appealing and
   memorable.
3.  Do not make the test available on the  web or have more  controls built into the
   process.  Do not send out the test until after participants notify APTI that they have
   read through the self-instructional materials.
4.  Develop one comprehensive  manual that users can reference  in the  future regarding
   all foundational air quality materials.
5.  Put all hard copy manuals on-line for easier accessibility.
HayGroup                                                                  Page 55

-------
EPA: Results of Benchmarking Study
                     VI. Training Evaluation Process

APTI uses several different forms for evaluating its training courses.  Typically, students
are asked to complete these evaluations at the end of the course (or satellite broadcast).
Many classroom courses also give students a pretest before the course begins to assess
their level of knowledge and a posttest at the end of the course to see how much the
students learned.

Some of the supervisors we interviewed believe that students learn a significant amount
in APTI courses and this knowledge helps them to do their jobs more effectively.  As one
supervisor put it, "APTI courses really enhance peoples' learning."  However, the current
training evaluation process does not allow for an evaluation of the extent to which APTI
training actually helps air professionals to be more effective in their jobs.

Additionally, many  interviewees indicated that they have developed their own surveys to
evaluate  APTI  training.  This may suggest that the current APTI training  evaluation
process is not  effectively  meeting stakeholder  needs to  evaluate and improve upon
training courses.

The comments below  summarize  what  interviewees  said  about APTFs  training
evaluation process,  as well as the Hay  Group's assessment of the training  evaluation
process (based  on  our extensive experience in this area).  It should be noted that  few
interviewees actually commented on the training evaluation process.

Too much focus on end-of-course  evaluations.  APTI  focuses its training  evaluation
efforts solely on end-of-course evaluations (some call these surveys "smile surveys").
This type of evaluation provides limited information for determining the extent to which
each  APTI  training  course  achieved its  ultimate objective  -  to  improve  the  job
performance of air professionals.

Evaluation form does not provide enough useful information for improving courses.
The current course evaluations do not provide enough information for understanding why
the training  course  was or was not effective, whether the delivery method is the best
medium for delivering the training, and more importantly, how to improve APTI training
courses.  As one interviewee indicated, "the evaluation focuses mostly on the instructors
(are they boring or not) and classroom logistics (coffee or snacks)."

Feedback is rarely used to improve courses.  Many interviewees feel that APTI does not
take feedback about its training program seriously, and rarely makes improvements to its
training based on this feedback.  When feedback about training courses is given to APTI,
the typical response is "we will try to do better next time."
HayGroup                                                                  Page 56

-------
EPA: Results of Benchmarking Study
Options for Improving the Training Evaluation Process

Interviewees provided  a few recommendations  for  improving  the current training
evaluation process.

    1.  Expand the  training evaluation  process  beyond end-of-course evaluations.  In
       order to truly assess  the  effectiveness of APTI  courses,  it is important to
       determine the  extent to which the training  actually  helps air  professionals to do
       their jobs effectively and contribute to the mission of their agency. It is critical to
       collect feedback about  the effectiveness of the training at different points of time
       and  from different  sources (e.g.,   students themselves,  supervisors, training
       coordinators).  Suggestions include:
          a.  Solicit  feedback (e.g., through surveys, interviews) from supervisors of
              attendees of training to determine the  extent  to which they believe the
              training has helped their air  professionals to  be more effective in their
              jobs.
          b.  Survey course participants 3  to 6 months after the training to assess the
              extent to which they have improved their job performance (not just after
              the course is complete)
          c.  Survey site coordinators  to determine why satellite courses are not well
              attended and how to improve them.

    2.  Add  open-ended questions  to  the  training evaluation  form that will provide
       feedback for  enhancing  the effectiveness  of the training courses.  Examples
       questions are:  What should we do differently and what should we do in the same
       way? How would you improve the current training course?

    3.  Solicit feedback about web-based  courses (not just  satellite  and  classroom
       courses) to assess their effectiveness and ways to improve them and make them
       more interactive.

    4.  Review feedback from  training  evaluations  on  a  regular basis  and make
       modifications to training courses, where appropriate. Communicate what changes
       to training are  being made as a result  of the training evaluation and link the results
       to efforts to update or  redesign courses  and make decisions  about  ways to
       improve courses.
HayGroup                                                                   Page 57

-------
EPA: Results of Benchmarking Study
                     VII.  Conclusions and Next Steps

In this report, we provide an evaluation of the current Air Pollution Training program
based on data from numerous sources: 1) interviews with stakeholders who have in-depth
knowledge of the APTI training program and its effectiveness; 2) reviews of relevant
documents and web sites about the APTI training program; and  3) reviews of actual
training courses (e.g., self-instructional manuals, web-based courses, satellite broadcasts).
In subsequent reports, after we identify the best practices around training (through site
visits and benchmarking  research),  we will provide more insight into ways to improve
APTFs training  program, and where APTI should focus its limited training resources
(how to  deliver training  to  a large, geographically   dispersed population  of air
professionals in the most effective and cost efficient manner).

In summary, our findings show that there are mixed opinions about the effectiveness of
the Air Pollution Training program.  An initial concern raised by the study is the extent to
which EPA (and APTI) is committed to meeting its  core objective of training air quality
professionals. There is a perception among interviewees, whether it is valid or not, that
EPA  does not have a commitment to  providing technical  training to  state and  local
agencies.

However, APTI was often  praised  for its classroom training, particularly courses  with
laboratory components; its  breadth of courses; and its course content (for those courses
that are up-to-date).  Additionally, many interviewees  believe that APTI is taking a step
in the right direction by adopting a blended approach to training - providing easy access
to courses (e.g., via web or  satellite) when state and local agencies are facing budget cuts
and travel restrictions.

Stakeholders are most concerned with the quality of course materials (i.e., many course
materials are outdated and of poor quality).  Additionally, there are several potential areas
for improving APTI distance learning training methods (satellite and web-based training)
that are the "wave of the future" for APTI, particularly  increasing the  opportunity for
interaction.   There is also  a  need to ensure that APTI courses  (particularly classroom
courses) are being delivered consistently and by instructors who have technical expertise,
practical experience in the field, and "teaching ability." Finally, stakeholders would like
to see APTI place more emphasis on reviewing and acting on feedback obtained through
the needs assessment and training evaluation process in an effort to  ensure that the needs
of air professionals are being met through its training courses.

While there were numerous  suggestions from the various constituents  about what the
EOG (and APTI) could do to improve its effectiveness, the following were the  most
prevalent and noteworthy:
   •   Reestablish  and  communicate  the mission  of   APTI  and  its  goals  to all
       stakeholders
HayGroup                                                                  Page 58

-------
EPA: Results of Benchmarking Study
    •   Determine which courses are most important to foundational learning and update
       them

    •   Put more course materials on-line (those that are relevant and up-to-date)

    •   Move  toward remote,  interactive  classes,  either through  web-based tools  or
       satellite, especially for introductory courses

    •   Respond to needs assessments and course evaluations, even if to say the issue will
       not be addressed at this time

    •   Increase the use of practical, hands-on applications for all courses

Our next step in the benchmarking study is to identify the best practices of organizations
that have  been successful at delivering training to a geographically dispersed audience,
and that  use  innovative training  delivery  methods.  We  will  be selecting  potential
benchmarking partners and  conducting site visits to learn about their training practices,
processes, and procedures (which will  be summarized in a separate report).  In the final
report, we will provide options for making APTI more successful going forward.
HayGroup                                                                   Page 59

-------
 EPA: Results of Benchmarking Study
                              Appendix B

                             Task 2 Report:
              Best Practices of Leading Training Programs
HayGroup                                                          Page 60

-------
EPA: Results of Benchmarking Study
                              /.  Introduction
BACKGROUND
In May 2004, the Education and Outreach Group (EOG) of the Environmental Protection
Agency (EPA) contracted with the Hay Group to conduct a Benchmarking Study of its
Air Pollution Training program. The ultimate goal of the project is to provide options for
improving the current Air Pollution Training program, and ultimately, EOG's business
performance. The following three tasks are aimed at accomplishing this goal:
       >  Evaluate the current Air Pollution Training program (Task 1)
       >  Identify best practices of leading training programs (Task 2)
       >  Develop a plan of action for improving the current Air  Pollution Training
          program (Task 3)

This report  summarizes our findings from Task 2 - identify best practices of leading
training programs.  The findings  in this report provide a summary of the  practices,
procedures and processes used by industry leaders in training. In this report, we focus on
these best practices and how the Air Pollution Training  Institute (APTI) may incorporate
some of these practices into its current training program.

In the final report (Task 3),  we will present  our specific conclusions regarding the
direction that we believe the Air Pollution Training program should take.

METHODOLOGY
This section summarizes the methodology used  to identify the best practices of industry
leaders in training and distance learning.  The primary methods used to identify the best
practices were:   1) site visits with public and private sector  organizations that are
recognized as  industry leaders in training; and  2) literature review on innovative ways
other organizations are delivering training, particularly distance learning.

Site Visits
Hay conducted site visits with five organizations to identify industry best practices in
training, particularly distance  learning.  Below, we describe the methodology used to
conduct the site visits.

Select  organizations for site visits and elicit participation. The first step was to select a
list of  potential organizations to participate in site visits.  Through reviews of published
sources and the Internet, we  developed a list of organizations  considered leaders  in
technical training and distance learning by  professional organizations (e.g.,  American
Society for Training and  Development,  ASTD;  American Productivity and Quality
Center, APQC), publications  (e.g., Training Magazine), and academic and professional
experts.  We based our  selection on awards (e.g., Training Magazine's Top 100 list,
ASTD BEST Award, Government e-learning award);  accolades, participation in other
HayGroup                                                                  Page 61

-------
   EPA: Results of Benchmarking Study
  training  benchmarking  studies  (e.g.,  Society  for  Human Resource  Management
  Consortium  Benchmarking  Study   on  Training  and  Development),  and   expert
  recommendations.  We particularly  looked for organizations that use innovative  e-
  learning and distance learning training delivery methods.

  We then contacted each of the organizations to determine whether they would be an
  appropriate benchmark organization and interested in participating in the study. We sent
  potential organizations a short description  of the benchmarking  study and the level  of
  effort required to participate. We  then qualified each  organization  through a short
  conference call and cut the initial  list down to a select group of organizations.  Table  1
  shows the five organizations that agreed to participate in the benchmarking study and our
  rationale for selecting each  of them as a benchmarking partner.  We refer to these five
  organizations throughout the study as a "benchmarking partner".
                  Table 1: Participating Organizations in the Site Visits
    Organization
Occupational Safety
and Health
Administration
(OSHA), Office of
Training and Education
      Rationale for Selection as a Benchmarking Partner
>  Provides  technical  training  to  a  large, geographically-dispersed
   population
>  Blends  Web-based training and live satellite  broadcasts with more
   traditional  classroom  instruction (uses  innovative distance learning
   training media)
Center for Disease
Control (CDC), Public
Health Training
Network (PHTN)
>  Provides  technical  training  to  a  large, geographically-dispersed
   population
>  Provides diverse training delivery methods, including distance learning

>  Has state-of-the-art training facilities
Georgia Tech
University, Distance
Learning and
Professional  Education
Department (DLPE)
   Trains professionals in engineering, business, and other hard sciences
   Recognized as providing an exceptional distance learning program and
   successfully incorporating an  interactive  component into  distance
   learning
   Has state-of-the-art training facilities
SAS Institute (SAS)
>  Provides extensive technical training to customers around the world
>  Blends  e-learning with more traditional  classroom  training  (uses
                            innovative e-learning training media)
GMAC Commercial
Mortgage (GMAC),
Staff Development
Division
   Delivers training primarily via e-learning  methodologies, including
   videoconferencing, live Webcasts, videotapes and C-ROMs/DVDs
  Prepare for site visits.  Hay designed a site visit protocol (based on the evaluation criteria
  used in Task  1) to capture the best practices of industry leaders in training. The protocol
  included  an introductory paragraph to  read  to  participants  at the beginning of the site
  visits and interview questions.  A copy of the site visit protocol is presented in Appendix
  Bl.
   HayGroup
                                                        Page 62

-------
EPA: Results of Benchmarking Study
Hay worked with a point  of contact  at  each participating  organization to  identify
appropriate individuals to interview about the training program. An effort was made to
interview a diverse group of individuals who have a good understanding of the  training
program,  its  objectives  and  desired  results,  its  future  direction,  and its  overall
effectiveness.  We typically  spoke to 3 - 10 individuals (e.g., training director,  training
content  developers,   graphics  specialists,  studio  staff)  from   each  participating
organization.

Conduct site visits.  We  conducted site visits with five  organizations recognized as
industry leaders in training.  An experienced Hay facilitator conducted all the site visits
using the standardized protocol. An EPA representative also attended all site visits. The
site visits focused on identifying best practices around:
       >   The needs assessment process (e.g., how decisions are  made  about course
           content and delivery methods)
       >   Course design and content (e.g., process for designing/updating courses)
       >   Training  delivery  methods,  with  a special  emphasis  on distance learning
           methods (e.g., live Webcasts,  self-paced Web courses, satellite broadcasts)
       >   Strategies for incorporating an interactive component into distance learning
       >   Training evaluation process

During site visits, we reviewed training  program documentation and materials, observed
training facilities and training programs  (e.g., a live Webcast; an interactive Web-based
course), and conducted interviews with  members of the training group.  Each site visit
lasted two to six hours.

Extensive notes were taken during the site visits.   Information collected  from  the site
visits was reviewed, and we identified existing training best practices. These best
practices are presented throughout this report.

Literature Review
In addition to conducting  site visits, Hay reviewed several reports, articles, books and
web sites to identify best practices in e-learning  and distance learning.   The  primary
documents/web sites that were reviewed  are shown in Table 2.
HayGroup                                                                    Page 63

-------
EPA: Results of Benchmarking Study
  Table 2: Primary Documents/Web Sites Reviewed for the Benchmarking Study
  >   American  Society for Training and Development (ASTD). State of the Industry
     Report (2004).
  >   American  Society for Training and Development (ASTD). State of the Industry
     Report (2003).
  >   American  Society for Training and Development. E-Learning Handbook:  Best
     Practices, Strategies, and Case Studies for an Emerging Field (2002).
  >   Bersin and Associates. Blended Learning: What Works (May 2003).
  >   Bersin, Josh.  The Blended Learning Book:  Best Practices, Proven Methodologies,
     and Lessons Learned (2004).
  >   Daly, David, and Scott, Amy. Best Practices for Advanced Distributed Learning.
     www.jointadlcolab. org.
  >   Galagan, Patricia A. Mission E-Possible, the Cisco E-Learning Story. Training and
     Development (February 2001).
  >   Galvin, Tammy. Training Magazine.  The 22nd Annual Industry Report (October,
     2003).
  >   General Accounting Office.  Human  Capital:  A  Guide  for Assessing  Strategic
     Training and Development Efforts in the Federal Government (March 2004).
  >   General Accounting Office.   Information Technology  Training:   Practices of
     Leading Public Sector Companies (2003).
  >   Hall, Brandon. Six Steps to Developing a Successful E-Learning Initiative (2001).
  >   Hall,  Brandon and  LeCavalier,  Jacques. E-Learning across the Enterprise:  The
     Benchmarking Study of Best Practices (2000).
  >   Hequet, Marc. Training Magazine.  The State of E-Learning (September, 2003).
  >   Hofman, Jennifer. Blended Learning Case Study. Learning Circuits  (2001).
  >   Johnson, Gail.  Training Magazine.  Blended Learning: How to Brew the Perfect
     Blend (December, 2003).
  >   Kilpatrick, D.  (1994). Evaluating training programs: The four levels.  San Francisco:
     Berrett-Koehler.
  >   Kiser, Kim. E-learning Evangelism.  Online Learning (2001).
  >   Singh, Henry. Building Effective Blended Learning Programs. Educational
     Technology (November/December 2003).	
HayGroup                                                                 Page 64

-------
EPA: Results of Benchmarking Study
REPORT OVERVIEW
In this report, we summarize our findings from the site visits and literature review to
identify trends  and best practices  of industry  leaders  in training. We  provide  an
assessment of the training best practices in each of the following areas:
       >  Training Needs Assessment Process
       >  Course Design and Content
       >  Training Delivery Methods and Ways to Incorporate Interaction into Distance
          Learning
       >  Training Evaluation Process

Summary of the findings from each  of the five site visits is presented in Appendix B2.
Appendix B2 also includes a short description of the characteristics of the training groups
that participated in the site visits (e.g., size of staff, facilities, mission).

We conclude the report with a discussion of our main conclusions about best practices in
the training industry and how they may be incorporated into APTI's training program.
HayGroup                                                                   Page 65

-------
EPA: Results of Benchmarking Study
             //.  Training Needs Assessment Process


Our research shows that conducting a needs assessment is often the first, and possibly the
most important,  step in developing an effective training course or an entire training
program.  Training needs  assessments provide information about the types  of training
courses needed by potential participants and the manner in  which training should be
delivered.  More  specifically, training needs assessments  help determine,  among other
things:
       >  Types of training programs that participants want and need
       >  Gaps in current and needed course offerings and participant skill sets
       >  Goals and objectives for training programs
       >  Steps needed to achieve training objectives
       >  Training areas in need of additional resources
       >  Best methods for delivering training courses (e.g., classroom, self-paced Web,
          satellite)

Our benchmarking research shows that training needs assessments are being conducted at
some level by most training programs. However, across the broad range of organizations
that  conduct training  programs,  the  overall  status and effectiveness of  the  needs
assessments  are  not  consistent.    Some   organizations  conduct  systematic  needs
assessments for both overall training programs  and individual courses,  while  others
appear to make somewhat  haphazard attempts at determining what potential participants
want and need.  What  is clear,  however,  is that training experts agree that  conducting
systematic needs assessments is an important contributor to the overall effectiveness  of
training programs (the extent to which they ultimately improve trainee performance and
overall organizational effectiveness).

TYPES OF NEEDS ASSESSMENTS CONDUCTED BY INDUSTRY LEADERS
We found through our benchmarking research that there are different types of needs
assessments that training organizations conduct: 1) overall training needs assessment;  2)
market needs for a specific course; and 3) content design and delivery needs assessment.
Each of these needs assessments has different objectives and processes.

Overall training needs assessment.  Overall training needs assessments are conducted by
most  of the  benchmarking partners  and  other  training  industry  leaders.    These
assessments examine the courses that current, former, and potential training participants
think they will need over a one- or two-year time period. This type of needs  assessment
typically involves an annual survey of the trainee population (and sometimes, supervisors
of trainees).  In the best cases,  the training group (or the organization as a whole) has
identified a set of overall objectives that the  trainee population should achieve over the
specified time period (e.g., employees should develop a  specific set of IT skills).  These
objectives then form the basis of the needs assessment. For example, a needs  assessment
HayGroup                                                                  Page 66

-------
EPA: Results of Benchmarking Study
may provide the trainee population with the list of objectives and ask them to list courses
they think are needed to meet the objectives (e.g., which courses will help them develop
the specific set of IT skills required for future success on the job?).  Regardless of the
actual approach, overall training needs assessments attempt to determine programmatic
needs over a one- or two-year period.

                  Example of Overall Training Needs Assessment
SAS conducts an extensive annual needs assessment survey to assess what courses should
be offered to  customers in the upcoming year. The  survey consists of approximately 400
items designed to determine whether and how customers use software (importance),
when they last used specific software (recency), and how often they use the  software
(frequency).

Market needs for a specific course.   Some organizations develop particular training
courses on an ad-hoc basis; that is, courses are developed to meet specific needs or at the
specific request of a participant, a customer, or  even an internal subject matter expert.
Because these  courses are  not  often identified during the overall needs assessment
process,  some  training programs will  conduct a  course-specific needs  assessment to
determine whether the "market" will  support them.   For example, an internal subject
matter expert may want to provide a training course for  a specific topic that was not
identified as  a need during the  overall needs assessment.  Before taking the  time to
develop the course, a market needs assessment is conducted for the course to determine
whether participants believe there is a need for the training  and whether they are likely to
attend.

Some of the questions that market needs assessments typically ask are:
       >  Is  there a need in the industry or organization for the specific course?
       >  Does this course fit in within the organization's overall programmatic goals?
       >  Are participants likely to attend?

          Example of a Market Needs Assessment for Individual Courses
Georgia Tech develops courses that are requested by several sources including customers,
professors, and  other  internal  or  external  subject  matter  experts.  Georgia  Tech
subsequently  conducts a market needs assessment for selected courses to make sure that a
similar course does not already exist and that potential participants feel there is a need for
the course.

Content design and delivery  needs  assessment.  Content design  and delivery needs
assessments  are  typically conducted as  part of  either  an overall or  market needs
assessment. Because distance learning is a rapidly developing field, content design and
delivery needs assessments help  determine  not only what types of delivery methods are
being used by the industry but also what types of delivery methods potential participants
need and expect.  For example, SAS found that customers were less willing to travel to
training sites  after September 11, 2001.  A content design and delivery needs assessments
was used to  determine the  types of training delivery methods that could supplement
HayGroup                                                                  Page 67

-------
EPA: Results of Benchmarking Study
classroom training in light of a reduced willingness to travel on the part of customers.
This type of needs assessment may also be used to determine the best methods to reach
participants in remote locations, or the extent to which participants are willing to watch
satellite broadcasts, participate in live Webcasts, complete a self-paced Web course, and
so forth.

NEEDS ASSESSMENT BEST PRACTICES
Through our site visits and  literature review,  we identified several best practices  in
conducting needs assessments.

1. Incorporate  skill/competency  assessments  into  the  needs  assessment  process.
   Industry  leaders go beyond simply asking trainees to evaluate their need for future
   training courses or their desire for different delivery methods by incorporating skill or
   competency assessments into the needs assessment process.  For example, trainees
   may be asked to indicate, from a list of skills, knowledge, and competencies, those
   they feel are necessary for future success on the job, and to rate their current level of
   proficiency with those knowledge/skills/competencies.   The training group  would
   then develop training courses aimed  at teaching the critical skills, knowledge, and
   competencies needed by trainees (those that  are important to future success and need
   further  development).  As another example,  some organizations have  developed
   competency models for certain positions which outline the key competencies required
   for the job.   Training  courses  are then  reviewed  to determine  whether they
   develop/teach the competencies required for success on the job, and determinations
   are made about the need for updating current courses or developing new ones.

               Example of Incorporating a Competency Assessment
                        into the Needs Assessment Process
OSHA recently developed a competency model  for its  compliance officers (to determine
the competencies required for success on the  job).    A consultant  was  then hired  to
perform a gap analysis to determine the gaps between the current training curriculum and
the competency model.  As a result of this analysis,  OSHA made  decisions to update
current courses and develop new courses to ensure that its training program was teaching
the competencies required for success on the compliance officer job.

2. Encourage trainees to participate in the needs assessment process. For training needs
   assessment data to  be useful for making decisions  about training  courses  and
   programs, it is critical to obtain feedback from  a representative sample of current and
   potential trainees. To enhance response rates, industry leaders  publicize the needs
   assessment approximately 30  days  in advance  using  a  variety of outlets  (e.g.,
   newsletters,  the Intranet,  supervisors,   flyers  posted  on  walls throughout  an
   organization, message boards, e-mail).

   It is  also  helpful to  send potential participants   an introductory letter (often  in
   conjunction with a paper or Web survey) from an organizational leader explaining the
   importance of the needs assessment process.  Additional follow-ups further help to
   ensure that as many participants as possible complete the needs assessment.
HayGroup                                                                  Page 68

-------
EPA: Results of Benchmarking Study
3.  Collect data from multiple  sources.   In  addition to collecting  information  from
   trainees about their specific training needs, industry leaders also seek input from other
   sources that may have insight into the needs of the trainee population.  For example,
   some training programs conduct surveys of supervisors to obtain their feedback on
   the types of skills and knowledge their employees will need in the upcoming year to
   be  successful on the job.  Additionally, course instructors or developers  are  often
   included in the needs assessment process.  Through  their experience with trainees,
   course instructors and developers often have a good understanding of the types of
   courses that will be needed in the future.

4.  Use several  data collection  methods.  A needs  assessment does not  have to be
   conducted  solely as a paper-and-pencil or an on-line survey.   Other data collection
   methods used  by industry  leaders  are information  interviews,  focus  groups, and
   comment cards.  The more data  collection methods that are used, the more likely a
   representative sample of participants will respond.  For example, conducting only on-
   line needs  assessment  surveys  may limit the participant  sample to  those  most
   comfortable  with the Internet.   Using additional  methods  also results in  more
   insightful and actionable data.  While surveys provide objective data about who is
   likely to participant in a particular training course, focus groups or interviews provide
   more qualitative information about why they are likely to participate.

5.  Use consistent methods and processes.  Needs assessment methods and  questions
   should remain stable, to the extent possible, over time. The ultimate goal of a needs
   assessment  is to  determine what training and skills are needed  to meet specific
   organizational goals.  To measure progress and training gaps over time, the methods
   and questions should be consistent.   For example, asking participants to simply list
   courses they need in the future may provide different results than offering a list of
   courses and asking participants to rate the importance of each course.

6.  Use the needs assessment results during the course development process.  Industry
   leaders consistently monitor the data collected from  needs  assessments and make
   decisions about training  content and delivery methods based on these data.  Training
   needs  assessment data provide valuable input into whether the "right"  courses are
   currently available to meet trainee needs, whether there  is a need to update current
   courses or transfer them  to other delivery methods, or whether new courses  should be
   developed. Using needs assessment data shows that the process is taken seriously and
   helps in planning how training resources will be used in the future.

7.  Provide  feedback  about results.  Participants take the  time to complete  needs
   assessments and expect to learn about the results and decisions about training courses
   that are made based on these results.  Industry leaders  publicize the results of training
   needs assessments (and decisions made based on these results) to all participants.  As
   a result, participants gain an understanding of how training decisions are made (e.g.,
   why a particular course was eliminated or why a classroom course was translated to a
   live Web course).  Also, participants may be willing to provide additional insights
HayGroup                                                                   Page 69

-------
EPA: Results of Benchmarking Study
   into the needs assessment results, which may add another level of understanding to
   their needs.   For example,  participants  may  find  the needs  assessment results
   surprising and following up on this may reveal  that they did not understand certain
   questions or that the results were misinterpreted.

SUMMARY

The needs assessment process is one of the first steps to developing an excellent training
program or course. The value of the needs assessment is that it allows an organization to
discover where the gaps are in terms of courses offered, delivery methods used, and the
skill  sets required for the trainee population to be effective in their jobs. Industry leaders
regularly conduct needs assessments which  incorporate skill/competency assessments
into  the process, and solicit input  from multiple  sources  such as potential trainees,
industry leaders, and supervisors.  Most importantly, needs assessment data are used to
modify existing training courses, develop new courses, and eliminate courses that do not
add value.
HayGroup                                                                   Page 70

-------
EPA: Results of Benchmarking Study
                  ///.  Course Design and Content


The content of a training course is the most important determinant of its effectiveness.
Regardless of how  training  programs  are  delivered, they are not valuable unless
participants learn and take away something that can be applied to their day-to-day jobs.
Even the most sophisticated delivery method will not teach trainees if the content is of
little value, outdated,  or not appropriate for the intended audience.

The benchmarking partners and other industry leaders place a great deal of emphasis on
developing training content and keeping it up-to-date and of the highest quality. Several
use internal and external subject matter experts  (SMEs)  to develop course content,
expend  considerable  resources developing  programs, and conduct  systematic course
reviews to ensure that content is up-to-date.

The benchmarking partners agree that developing high-quality courses takes time and
resources. For example, Georgia Tech estimates that three hours of post-production time
are required for each hour of instruction.  The  CDC  estimates  that a self-paced Web
course consisting of 120 pages can take about 3-5 months and $120,000 to develop from
scratch.

                    SAS: Estimated Time to Complete Training
SAS indicated that one of the  biggest lessons it has learned is that good training takes
time to develop. It takes:
12 hours to develop 1  hour of classroom delivery content
20 hours to develop 1  hour of live Web delivery content
60 hours to develop 1  hour of self-paced Web delivery content

In this section we present best practices around course development and updating that we
identified during the site visits and literature review.

COURSE DEVELOPMENT
Most of the benchmarking partners use a systematic process for designing new courses or
converting existing courses to a  different medium (e.g., a classroom course to  a live
Webcast; a paper self-study course to a self-paced Web course). There is wide variety in
the number of new  courses  developed each year by  the benchmarking partners (and
number converted to  other media).   For example, on average, OSHA designs only two
new courses per year and transforms an additional  12 courses from the  classroom to a
Web-based format.   CDC develops  approximately 30  to 40 new courses (satellite and
Web-based) per year,  while Georgia Tech developed over 100 courses in 2004.
We found that initial  ideas for new courses or new course delivery methods come from a
variety of sources  including reviews of needs assessments  and industry trends, subject
matter experts, external clients, and organizational leaders.  Additionally, we discovered
HayGroup                                                                  Page 71

-------
 EPA: Results of Benchmarking Study
 that some organizations have  a  specific training philosophy or specific resources that
 guide their course development process.  For example, SAS follows at least four rules
 each time it develops a Web-based course, and Georgia Tech has enormous  academic
 resources at its disposal to assist with course development (e.g., SMEs/facuity, libraries,
 students).

 Regardless of where the course ideas originate or the planned delivery method  (e.g., live
 Web,  satellite,  classroom),  we identified  several best practices for developing training
 courses.

 1.  Form a course development  team.  The benchmarking  partners indicated  that each
    course development project requires an experienced team covering a variety of roles
    including, program manager, project manager, instructional  designer, subject matter
    expert,  technical  experts,  technical review  team, editor, and producer.   Table 3
    describes the basic roles that should be filled for each course development project;
    some of these roles may overlap and be completed by one person, and additional roles
    may be required for specific courses.

	Table 3: Roles of the Content Development Team	
 >  Program manager - Develops the budget and plans the overall course; takes responsibility
    for the course and interfaces with executives; promotes and evaluates the program
 >  Project manager - Reviews plans and oversees day-to-day work and goals (may also be the
    program manager for larger projects)
 >  Instructional designer - Creates the  training objectives and the overall framework, and the
    look and feel of the course
 >  Subject matter expert - Understands the subject and drafts the technical aspects of the course
 >  Technical experts - Understand the technology to deliver the course (e.g., Web developers,
    camera crews, satellite technicians)
 >  Technical review team - Reviews  the technical aspects of the  course including delivery
    methods, training/teaching techniques, and the feasibility of the overall design
 >  Editor - Makes sure content is clear and accurate
 >  Producer -  Produces the  course and makes sure it is incorporated into the overall training
    program
 Primarily taken from Bersin, Josh.  The Blended Learning Book: Best Practices, Proven Methodologies,
 and Lessons Learned (2004).	
 HayGroup                                                                     Page 72

-------
EPA: Results of Benchmarking Study
                     Example of Course Development Process
The  CDC typically  initiates course development  at an internal  request.  It  provides
instructional design expertise and expects that content will be developed by an external
subject matter expert.  Subject matter experts provide content information including an
outline of the course  so that CDC can review and make changes.  Then, depending on the
type of delivery method, the content  development process follows a specific path.

Web-based Development
CDC provides a Web  developer (usually a contractor) with  the  course  outline and
content.  The  Web developer takes  the content and develops a course based on it and
other design specifications.  CDC staff review  the process along the way and conduct
pilot tests to ensure the effectiveness of the course.

Satellite Broadcast Development
CDC uses the following process to develop its satellite broadcasts:   1)  subject matter
experts create  an outline of the material to be covered by the broadcast (the CDC asks to
see the  script at least a few days in advance); and 2) a hired script  writer takes the
information  provided by  the subject matter experts  and writes a full  script for the
broadcast (to make the material conversational). Because subject matter experts are just
that, the script writers help create  more conversational programs out  of the technical
information provides by the SMEs. CDC then requires an extensive practice session (full
script and use of teleprompters) as a final check for glitches or inaccuracies before the
broadcast is delivered live.

2.  Work with subject matter experts.   Our benchmarking research suggests that most
   courses require SMEs to develop technical  content.  Subject matter experts can be
   anyone with special expertise and  may include, among others, internal  specialists,
   professors, or external consultants. For example, Georgia Tech often uses professors,
   and GMAC has brought in experts from financial ratings agencies such as Moodys to
   help develop course content.  Although SMEs are excellent resources, they are often
   very  busy people and working with them can prove to  be one of the biggest
   challenges to completing course  content design in a timely fashion.  Through the site
   visits and literature review, we identified the following guidelines for working with
   SMEs.
       >  Ask questions. Asking specific questions about how a SME develops courses,
          and about his/her philosophy and  time constraints,  may help determine
          whether or not the SME is a good match for the course design project.
       >  Provide SMEs with information about plans for the course.  SMEs are often
          brought into a course design project after  plans for the course have been
          discussed. The benchmarking  partners recommend providing  SMEs with
          specific  details about the  course  including overall objectives;  intended
          delivery methods; time, resource, and technological constraints; and audience
          characteristics. Unless SMEs are provided with information  about the course
          up-front, they may develop content that does  not meet the stated objectives
HayGroup                                                                  Page 73

-------
EPA: Results of Benchmarking Study
       >  Review SMEs work. Although most SMEs have extensive knowledge of the
          course subject matter, they may not have a good understanding of the intended
          audience or even the course objectives.  Therefore, it is important to review
          the SMEs work to ensure it fits the intended audience and  meets course
          objectives.

3.  Decide whether to outsource.  Many of the benchmarking partners and other industry
   leaders are not  able to  perform all  of the course development roles and  must
   outsource  certain  tasks.    Most often  these are  technical  tasks such  as  Web
   development.  Although outsourcing is often a good short-term solution, it can also be
   more expensive.  For example, the CDC initially contracted with Web  developers to
   help design Web-based  courses. They found, however, that contracting out this work
   was more expensive than hiring a full-time Web developer.   On the other hand,
   Georgia  Tech provides a full range  of content development services and many
   organizations  contract with them to develop and conduct their training.

4.  Conduct a pilot test before courses go live.  The  benchmarking partners stressed the
   importance of pilot testing courses  before  they  are delivered to the  entire trainee
   population. Pilot testing is particularly important for self-study Web-based courses,
   where there is a need to not only determine whether the content of the course is clear,
   easy to understand, and meets training  objectives, but whether the technology itself is
   easy to navigate, provides sufficient opportunities for interaction, etc.

                           Example of Course Pilot Test
CDC pilot tests all its asynchronous Web-based courses before they are  delivered to the
trainee population. The purpose of the pilot test is to ensure that the course content and
design are appropriate for the intended audience. Pilot test participants (potential training
participants, training coordinators, site facilitators) are  asked to review the Web-based
course and  provide  feedback about the  length of  the  course (which  determines the
number of continuing  education  credits), degree to  which they understand the course
content, whether the  technology is easy to navigate, extent to which the  Web is the best
method for  delivering the specific training, and  so forth.   Feedback  from pilot test
participants is incorporated  into the final version of the Web training.

                           Example of Course Pilot Test
OSHA conducts an extensive pilot test of its Web courses before going live.  Volunteers
take  a pre-test (to determine  their knowledge/skill level before taking the  course),
complete the Web course,  and then take a post-test to evaluate whether they learned
required knowledge/skills by completing the course. Additionally, participants fill out an
evaluation form to provide their  feedback on any concerns or errors with the training,
questions, and so  forth. OSHA also holds a one-hour conference call with the pilot test
participants to go  over their evaluations and gather additional information. An evaluation
report is then developed and required changes are made to the Web course before it goes
live.
HayGroup                                                                   Page 74

-------
EPA: Results of Benchmarking Study
Special Considerations for Designing Self-Study Web Courses.  We found through the
site visits and literature review that designing self-study Web courses present additional
challenges.  To be effective, these courses must be designed to facilitate self-learning
(i.e., keep participants involved and engaged; monitor participant progress in  completing
the course).  Best practices in designing self-study Web courses include:
1.  Organize courses into learning modules.  Industry leaders agree that self-study Web
   courses should be  organized into learning modules or "learning chunks" based on
   major topic areas.  A specific module includes similar content items, practice items,
   and quizzes. Modules should  take 20 minutes or less to complete.  The American
   Society for Training and Development suggests that the benefits of learning modules
   is that they allow trainees to learn small sections of content at a time, learn skills on
   an as-needed basis, and skip modules that they have already mastered (cited in ASTD
   E-learning Handbook).  Learning modules also cut down development costs and
   allow organizations to personalize training for employees.

                          Example of Learning Modules
When Oracle designs Web-based courses, the company breaks the course content into
segments (or modules).  Each  segment includes pre-recorded streaming video lecture,
demonstrations,  exercises,  and  quizzes,  all  focused  on a  similar topic  or learning
objective. Each segment takes trainees 10-15 minutes to complete.

Taken from: Kiser, Kim. E-Learning Evangelism.  Online Learning (2001).

2.  Provide learning objectives at the  beginning of each session or module. Effective self-
   study Web sessions begin with an  overview of the purpose of the session and  the
   learning goals.  Trainees must see the course as relevant or they will not complete it.

3.  Re-package course materials when converting to self-study Web medium.  One of the
   biggest challenges faced by the benchmarking partners and other industry leaders is
   converting classroom courses (or paper-based self-study courses) to effective Web-
   based courses.   Converting classroom courses to the Web involves much more than
   simply cutting and pasting an existing classroom course onto the Web.  A large
   percentage of  development time is spent transferring  classroom content to more
   visually stimulating material and incorporating opportunities for student-to-instructor,
   student-to-student, and student-to-technology interaction into the Web-based training
   course.  For example,  the  American  Society  for  Training and Development
   recommended in its E-Learning Handbook  that content transferred from classroom
   courses to the Web must be reorganized and packaged for flow. For example:
       >  Graphics must be recreated (versus simply using existing Power Point slides)
          to visually represent or enhance the content of Web courses
       >  Exercises,  case studies,  simulations,  quizzes and other opportunities  for
          interaction should be incorporated into the Web course
HayGroup                                                                  Page 75

-------
EPA: Results of Benchmarking Study
REVIEWING AND UPDATING COURSE CONTENT
The  benchmarking partners  have an extensive  process for continually reviewing and
updating training courses to  ensure that they are up-to-date and of the highest quality.
For example, Georgia Tech reviews and updates all its courses at least on an annual basis.
OSHA uses course chairs to monitor courses and make sure they are up-to-date.  The
course chair typically reviews the courses approximately every three years to ensure they
are up-to-date, and makes changes as needed.

It should be noted that some training delivery methods are easier to update than others.
For example, updating self-paced Web courses may simply require removing the out-of-
date material  and replacing it (e.g., with a new  set of Power Point slides).  Live Web-
based courses, on the  other hand, may  be more difficult to update because the process
often requires re-taping the entire episode or trying to synch-up the new information with
the old.

The following are best practices in reviewing and updating training courses:

1.  Update continuously  and periodically.  Updating  course content can be  a timely
   process, which is a very  important reason for updating as often as possible. Much
   like cleaning a house, it is easier to update (or clean) periodically than wait until the
   course  is  such a mess that it needs a complete overhaul (i.e., extensive  "spring"
   cleaning).  However, there may be a point where a course does need a complete
   overhaul and not just periodic updates. For example, a new methodology or policy
   may be developed that completely changes an industry.  Rather than  update the
   course content, it may make more sense to develop a brand new course based on the
   new methodology/policy.

                        Examples of Content Update Tools
•  OSHA  uses special  software  (Workforce  Connections) that allows Web-based
   courses to be updated and changed "just-in-time".

•  Cisco has a system in place to alert SMEs or training designers when the content of
   Web-based courses is aging or becoming out-of-date.

2.  Plan for updates. Rather than waiting until course content  is old or outdated, industry
   leaders plan ahead for updates. For example, it may be necessary to plan a systematic
   review every 6 or 12 months to ensure that the content is appropriate for the intended
   trainee  audience.   Also,  because course updates take  time  and  resources, it  is
   important, when developing a course, to consider how much time and  money will be
   needed to update the course, and plan resources accordingly.

3.  Explain updates.   The benchmarking partners make it a point to communicate to
   trainees why and how course content is updated.  Some trainees may not agree that
   content should be updated,  so it is important to explain to them why the content has
   changed (e.g., a new method has been developed).
HayGroup                                                                  Page 76

-------
EPA: Results of Benchmarking Study
4.  Use updates as teaching points. Course updates can be used as a learning opportunity
   for trainees.  For example,  some benchmarking partners indicated that they use the
   content updates as a teaching point by detailing why the previous content is outdated.
   They compare and contrast  the new  and old information  and explain how the
   information has progressed.

SUMMARY

The  content of  a course  is  the  most  important determinant  of its  effectiveness.
Regardless of how sophisticated the delivery method,  a course that  has out-of-date
content or is inappropriate for a specific audience will  not teach trainees skills that they
can apply to their day-to-day jobs. We found that our  benchmarking partners and other
industry leaders use systematic  processes to develop course content, and rely  on subject
matter experts  to  provide  the  foundation  for the course  content. They  pilot test the
majority of their courses before they go live to ensure that the content is up-to-date and
appropriate, interactive exercises and simulations are effective, and the delivery method
is  appropriate  for the  course  content. Most importantly, industry leaders regularly
monitor the quality of their training courses and update courses when necessary.
HayGroup                                                                   Page 77

-------
EPA: Results of Benchmarking Study
                  IV.    Training Delivery Methods


All of the benchmarking partners we visited are challenged with providing effective
training to large populations of individuals who are geographically dispersed.  Although
classroom-based courses continue to play a very  large role in training programs, these
organizations realize that they can no longer rely  on the classroom as their primary
training delivery method. Instead, they are using innovative distance learning training
methods to ensure that they reach a  diverse audience,  and more importantly, provide
stimulating learning opportunities for those  they train.  As  further evidence  of the
increased reliance on distance learning training methods, ASTD cited in its 2003 State of
the Industry Report that classroom training (as a percentage of delivery methods)  has
decreased over the past four years and is projected to continue to decline.

Training  Magazine's  22nd  Annual  Industry Report (2003)  stated  that 72% of
organizations surveyed had training budgets in 2003 that were similar to or greater than
2002 budgets.  This report also notedd  other training trends from 2002 to 2003, including:
       >  Instructor-led classroom accounted  for  74% of all  training in  2002,  but
          dropped to 69% in 2003
       >  Computer-delivered training with no instructor rose from 12% of all training
          provided in 2002 to 16% in 2003
       >  Instructor-led training from a remote location rose from 7% of all training
          provided in 2002 to 10% in 2003
       >  Separate technology-based training budgets  rose from 24% of responding
          organizations in 2002 to 29% in 2003

While the Web is a powerful distance learning tool and is the primary distance learning
medium used by  industry leaders, it is by no means the sole medium used to deliver
distance learning  programs.  In fact,  distance learning programs  can  range from  highly
interactive videoconferences to  programs that require students  to review PowerPoint
slides that are delivered by inserting a CD-Rom into one's personal computer.

The biggest challenge  in distance learning  is to incorporate  both  interpersonal and
person-to-technology interactions into training delivery  methods.  These interactions are
critical  to ensuring that trainees actually complete the training,  and  more importantly,
learn required knowledge and skills.  Not unexpectedly, a common theme among the
benchmarking partners  was this very concern: how to integrate effective interactions into
the various media employed in distance learning.  During our site visits, we  discovered
that the benchmarking  partners are using cutting-edge techniques to foster interaction,
which is seen as a critical component of an effective training program.

In this  section, we summarize training delivery trends  and best practices, particularly
those used to promote  distance learning.  Although each program delivers,  and most
experts  agree there will  always be a place for classroom-based training, we focus  on the
distance learning  delivery methods because they represent the direction the industry is
HayGroup                                                                  Page 78

-------
EPA: Results of Benchmarking Study
headed.  The best practices summarized below can  be adapted for classroom-based
courses.

DISTANCE LEARNING DELIVERY CATEGORIES
Most experts  categorize distance learning delivery  methods as either synchronous or
asynchronous.  Synchronous training methods are conducted live and delivered real-time
in a classroom setting, over the Web, or through a video (typically satellite) broadcast.
Asynchronous training methods are sometimes referred to as self-study methods because
they are delivered "flat" (i.e., not live) and participants  can access them at any  time.
Table 4 presents a brief overview of the pros and cons of synchronous and asynchronous
delivery methods

    Table 4: Pros and Cons of Synchronous and Asynchronous Delivery Methods
Synchronous
Training
                                  Pros
>  Can be highly interactive;
   promotes relationships and
   students gain from live
   discussions and feedback
>  Easier to evaluate whether
   learning occurred
>  The speaker's tone and style
   come through well
>  Often easier to develop
                                           Cons
>  Can be very expensive to
   deliver
>  Scheduling is difficult and
   may take several sessions to
   reach everyone
>  Can require complex
   technologies such as
   satellite receivers and may
   require large bandwidth
Asynchronous
Training
   Easier to distribute to large
   audiences
   "Desktop, anytime"; students
   can access where and when
   they want
   Conveys a standard message
   Ability to skip topics that are
   already understood or
   mastered
>  Lacks inherent interactions
>  Can have a high dropout
   rate
>  Can be difficult to evaluate
   learning/monitor trainee' s
   progress
>  Can be costly to develop
The benchmarking partners we spoke with, and other training industry leaders, tend to
use a variety  of synchronous and  asynchronous methods to deliver distance learning
programs to trainees (classroom training is not the focus of this report).  Synchronous
methods include live Webcasts/Webinars, satellite broadcasts, and videoconferences. We
found  that the  industry trend  is  towards Web-based  courses (Webcasts/Webinars).
Asynchronous  methods  include self-paced Web  courses,  CD-Rom/DVD  self-study
courses,  and paper-based self-study courses.  Similar to synchronous courses,  we found
that Web-based courses are quickly replacing paper-based self-study courses.
HayGroup
                                                     Page 79

-------
EPA: Results of Benchmarking Study
SELECTING TRAINING DELIVERY METHODS
The benchmarking partners we spoke with do not have hard and fast rules for making
decisions about which training delivery media to use.  Often the decision is a pragmatic
one that is based on the time available and money or other resources needed to develop
and deliver the training program.  Some benchmarking partners, however, did use some
rules-of-thumb to guide them in selecting a training delivery method:
       >  Complexity of the material - For example, more complex material is better
          taught in the classroom (e.g., laboratory) or through interactive synchronous
          training methods. For less complex material, it may not be worth the cost to
          use synchronous training  delivery methods;  a  self-paced  course may be
          sufficient.  For  example,  self-paced courses work well  for the delivery of
          information, while  the classroom or  interactive  Webcasts work well when
          discussion is needed.
       >  Need for interaction — Training  content that requires student interactions is
          better   taught   via  synchronous   training  methods   (e.g.,  classroom;
          Webcasts/Webinars that incorporate methods for interaction).
       >  Requirements for  validation and/or certification  - Classroom training and
          other synchronous  training methods that can track student  progress  and
          incorporate quizzes and tests  work best for courses that require certification.
          Certification  programs require  that participants  meet some industry or test
          standard, have strict score reporting protocols,  and  often  have  expiration
          dates.  It is  difficult to meet  these standards with  an asynchronous program
          because it is not always possible to determine who took the test, participants
          cannot be expected to report their own results, and certification courses may
          expire but still be available in Web or paper format.
       >  Training audience - Participants skill levels may impact the effectiveness of
          different training delivery methods.  For example, trainees may not all have
          the same  understanding or  experience  with  certain technologies.    If  the
          audience is likely to have lower technological savvy, it makes more sense to
          use less technologically advanced delivery methods. The  size of the audience
          should also impact  decisions about training delivery methods.   As a rule of
          thumb, larger audiences (>3000) are more suited for asynchronous methods or
          live Webcasts (versus classroom training, etc.). Finally, if the audience has a
          varied background, this is an area where asynchronous training  could be used
          to provide a common foundation of knowledge  prior to the  synchronous
          portion of a course (i.e., a blended learning approach).
       >  Goals of the program - Programs that are designed to foster culture-building,
          such as orientation programs, are best delivered using synchronous methods.
          Training programs that focus on delivering instructions or an introduction to a
          task may best be suited to an asynchronous method.
HayGroup                                                                  Page 80

-------
EPA: Results of Benchmarking Study
       >  Budget and time - Training programs  with  smaller budgets need to think
          about how best to use their limited resources in delivering training.  While
          synchronous methods  may require a lot of personnel resources, asynchronous
          courses can often require  a lot of up-front time  and money to develop.
          Additionally, asynchronous courses often take a long time to develop and are
          not the best method to use when a course has a rapid window between design,
          development, and delivery unless appropriate technology is available.

In sum, when the outcomes of training are to analyze, synthesize, or evaluate, or when
poorly defined problems  are the  focus, more synchronous  delivery methods work best.
When the outcomes of training are to provide procedural  or well-defined information, or
when providing regulation updates, the training methods should be more asynchronous.

TRAINING DELIVERY METHODS USED BY BENCHMARKING PARTNERS
It  is clear that several  delivery methods have been  developed  for  distance learning
training programs.  The term blended learning is often used to define the combined use of
two  or more  delivery methods, and  is becoming the  approach of choice for  many
organizations (discussed later in this section). While many  of the benchmarking partners
use and recommend the blended learning approach,  we  provide here an overview of each
method as if it is the sole delivery method. At the end  of this Training Delivery section,
we will discuss ways benchmarking partners and other training leaders are using blended
learning to increase the effectiveness of their training programs.

Synchronous Delivery Methods
Most of the benchmarking partners agree that synchronous training  delivery methods
should be used when live interpersonal interactions  are  necessary.   These methods, when
done "right", provide participants with the opportunity to have real-time interactions with
instructors, other participants, and the technology (e.g., simulations, quizzes).  All  of the
benchmarking partners and many industry leaders  in training  use non-classroom-based
synchronous  training delivery methods,  in addition to  classroom courses.  The most
commonly used, and preferred, non-classroom-based synchronous training medium is the
Web   (e.g.,   live   Webcasts/Webinars),   followed  by   satellite   broadcasts  and
vi doeconference s.

Satellite broadcasts and live Webcasts are often similar in design  and course developers
face many of the same challenges with the two methods. The primary challenge shared
by the  two methods is  incorporating interactivity into courses.  In fact,  all  of the
benchmarking partners stressed the  importance of  integrating  interactivity into training
programs.  Integrating interactivity into live Web courses seems to be the focus of most
of the benchmarking partners because many have already or are beginning to move away
from satellite broadcasts.

There  are several  reasons why organizations are focusing on live Web training (e.g.,
Webcasts) rather than satellite broadcasts.  First, as training staff at S AS indicated, people
tend to physiologically tune out during training, and the Web offers exciting and varied
HayGroup                                                                  Page 81

-------
EPA: Results of Benchmarking Study
opportunities to enhance learning programs.  Additionally, busy professionals often do
not have the time or the ability to travel to a satellite-equipped site for training, and the
Web offers training that can be accessed on one's desktop.

For the remainder of this section, we present our findings on the primary synchronous
delivery  methods   (non-classroom)   used  by  the  benchmarking  partners:  Live
Webcasts/Webinars,  Satellite Broadcasts, and Videoconferences.  Because our research
shows that the trend in  synchronous delivery is toward live Web-based delivery methods,
we focus on this medium when describing the best practices. However, because there are
some similarities between the functionality and  challenges faced  for  each delivery
medium, many of the best  practices  described for  the Web-based medium may be
applicable  to  other  synchronous training methods.   For example,  the  benchmarking
partners are using question monitors for both live Webcasts and satellite broadcasts, but
we focus on how this is being adapted for Web use.

Live  Webcasts  or  Webinars.    As  indicated  previously,   our  research  shows that
synchronous distance learning programs are moving  toward  the Web.   Several of the
benchmarking partners are  conducting live Webcast or Webinar  programs  over  the
Internet (or Intranet). The terms Webcast and Webinar are often used  interchangeably by
training industry experts and  our benchmarking partners as they refer to similar delivery
methods (i.e.,  live Web-based training  delivery).  For this reason, we will use the term
Webcast to refer to any live Web-based training method.

Through these programs, participants can hear and see instructors on-line, in real-time.
Live Webcasts allow participants to interact with each other and can be programmed to
work with simulations and other interactive media.   Also, the benchmarking partners
strongly believe  that  these  programs  have  proven  effective,   particularly  when
opportunities for interaction are provided to participants.

The  benchmarking partners  have spent considerable  time and resources to make live
Webcasts interactive,  interesting,  and effective at  training  individuals to be  more
productive in their jobs.  For example, SAS began delivering live Web courses in July,
2001.  But before SAS would deliver a live Web course, the training staff made certain
that they could incorporate and encourage interactions without interrupting  the overall
flow of the program.   SAS created a set of rules they follow  when developing any live
Webcast course to ensure they meet their initial program goals.

                             SAS Live Webcast Rules
    1)  Participants must be given the opportunity to interact at least every  10 minutes.
   2)  Instructors must have specific training.
   3)  Instructors must have two rehearsals before they can teach a course.
   4)  Moderators must be present for each course session.

Through our site visits and benchmarking research, we identified several best practices
for delivering live Webcasts, which are  highlighted throughout this section.
HayGroup                                                                  Page 82

-------
EPA: Results of Benchmarking Study
1.  Record all  live  Webcasts.   The  benchmarking partners  recognize  that  not all
individuals can  watch the Webcast when it is delivered  live (e.g., have  scheduling
conflicts). As a result, they record all the live sessions and place links for these sessions
on their Web sites (or Intranets) so that individuals can watch the Webcast at their own
time and location (essentially  transforming into asynchronous delivery).  Additionally,
this allows new  employees to  complete needed training whenever they are able, rather
than waiting for the next live session.

                         Example of Recording Webcasts
Most of GMAC's programs are broadcast live and taped using Windows Media Encoder
for later viewing.  The training  staff found that live audiences were often smaller than
anticipated and decided that taping and re-broadcasting the sessions would allow them to
reach more people. The programs are posted to the Intranet and employees can access
them at their leisure.

2.  Incorporate  opportunities  for interaction into  the Webcast. The benchmarking
   partners recognize that for successful learning to occur, participants must have the
   opportunity to ask questions and discuss the broadcast  with others  (e.g., instructor,
   moderator, other participants).  Simply watching a "talking head"  or an electronic
   "page turner" does not engage participants; many participants will simply tune out.
   To promote interactions between participants and instructors during live Webcasts,
   many benchmarking partners give participants the opportunity to ask questions during
   the session.   We found that the most common methods used to elicit questions
   include:
       >  Call-in questions (e.g., participants call in to a centralized 800 number that is
          publicized before the Webcast)
       >  Fax-in questions
       >  E-mail questions
       >  Voice Over Internet Protocol  (VOIP) (VOIP is a  technology  that allows
          participants to  make telephone calls using a broadband Internet  connection
          instead of a regular phone line - our partners disagree as to whether VOIP
          technology is reliable enough for most training purposes)

For these  interactive  methods to  be effective,  there must  be  communication  to
participants before the broadcast about the process for asking questions  (e.g., fax-in,
telephone) and  participants  must be encouraged to  do  so. The above processes have
proven relatively effective in  enhancing interactions during Webcasts.  However, the
benchmarking partners have experienced some challenges, which are discussed below
along with strategies  that have been used to overcome them.

       >  Many participant  questions  go  unanswered  during the  Webcast  - To
          overcome this challenge, most of the benchmarking  partners capture all the
          participant questions during a Webcast and post them  to an asynchronous site.
          Although  all questions are not answered live,  all  questions are acknowledged
HayGroup                                                                  Page 83

-------
EPA: Results of Benchmarking Study
          and participants have access to the answers (participants are provided links to
          the site).
       >  Participants  do not always ask  questions  during the  Webcast  - Many
          benchmarking partners develop a set of approximately 10 questions prior to
          the Webcast  in the event that participants do not ask questions during the
          broadcast.  For example, if participants are not asking questions, a member of
          the broadcast team  can act as a participant and ask one of the pre-set
          questions.    This  helps to  fill  the  time  and  often  initiates  participant
          interactions.
       >  Participants  do not follow proper "netiquette"  -- Finally, to overcome the
          third  challenge,  benchmarking  partners  such   as  SAS  have  developed
          "netiquette" rules  that are provided to all participants prior to  and at the
          beginning  of each session.  For  example, because there  is limited  time,
          questions should be asked only  if they help clarify a  point  or  continue
          discussion. Questions about scheduling should be taken off-line.

   Another strategy  used by benchmarking partners  to enhance interactions during
   Webcasts is the incorporation  of audio and visual means of communication into the
   training delivery method.  Audio communications typically take place over  two-way
   telephone lines (e.g., an open  telephone line  where participants in the Webcast can
   ask  questions during the  session), but some benchmarking partners are also using
   VOIP to  provide  for  total communication over one's  personal  computer.  Video
   communication  can  also  occur over personal computers by mounting a small video
   camera on the computer (often used during Webcasts). Larger video cameras can be
   effective  when  a group  of people are  gathered in one location,  but for  more
   individualized locations, the smaller PC-mounted cameras work best.

3.  Use screeners to review participant questions.  Some of the benchmarking partners
   find that instructors  receive too many questions to respond to during the Webcasts
   and  that reading each question individually takes time away from teaching.  To meet
   this  challenge, a form of question screening and monitoring has  been incorporated
   where one or more individuals are available to receive, read, and review questions to
   determine whether the instructor should respond to  them during the live Webcast.
   For  example, Georgia Tech pairs  a screener with an instructor and uses the screener
   to monitor the questions  and  pass  along  to the instructor those questions  that will
   improve the training  and help students to learn.

   SAS also uses technology to help monitor and review questions.  Through Microsoft
   Live Meeting, participants are able to "raise their hands" when they have a question.
   The program indicates a potential question by changing a participant's "seat" color on
   the on-line screen visible  to the instructor and screener.   In addition, the screener is
   able to mute one or all participants if the conversation gets out of control.

4.  Use technology monitors.   The benchmarking partners have  found  that many
   participants and instructors  have trouble  with and questions  about the  technology
   used during the  live Webcasts.   Because providing technology assistance  to
HayGroup                                                                  Page 84

-------
EPA: Results of Benchmarking Study
   participants is not the role of the instructor and can take away from other participants'
   experiences, Georgia Tech provides each instructor with a technology expert.  The
   technology expert is there to:
       >  Respond to questions about technology
       >  Troubleshoot technology challenges
       >  Operate technologies for instructors

             Example of Question Screeners and Technology Monitors
Georgia Tech provides at least one question screener and one technology monitor for
each live Webcast.  The question screener can focus on receiving and determining which
questions are suitable to be addressed  during the live Webcast. The technology monitor
makes sure that the instructor can focus on teaching the course rather than worrying about
whether  internal or participant  technologies  are  working.   Each member of  the
presentation team has  a  specific  role that  serves to provide an uninterrupted training
session to participants.

5.  Ensure  that individuals who  deliver  the Webcast  have  teaching  abilities.   The
   benchmarking partners that use live Webcasts  indicate that not all instructors  are
   effective  at delivering training using this medium.   For  example, GMAC  hired  a
   university professor as the subject matter expert to deliver a live Webcast training
   session.   Although the information he presented was  very  well  received,  he was
   unable to stay within range of the camera and would walk in-and-out of view at  a
   rapid rate.  As mentioned earlier, SAS will  not allow  an instructor to teach a live
   Webcast  course  until  he/she  has received  specific training  and  conducted  two
   rehearsals before the first session.  In  fact,  SAS has 77 instructors, with only 20
   qualified to teach live Webcasts.

6.  Publicize broadcasts.  Because live Webcasts can be difficult to  schedule and  the
   training will not be  effective if there is no one participating in the session, publicizing
   them is  very important.   A  rule-of-thumb is to begin publicizing each Webcast
   approximately 30 days in advance by using a variety of media.  These may include
   newsletters, the  Internet, the Intranet,  and supervisors who can help remind  the
   participants of the training.

7.  Limit the length of programs.  People tend to get bored and physiologically tune  out
   during  long training  sessions, especially when opportunities  for  interaction  are
   limited.  A key recommendation by our benchmarking partners is to limit the amount
   of time  spent in any one  Webcast training  session.  One benchmarking partner
   suggested that individual  training sessions should be scheduled for no longer than  a
   half-day, and the training session should be broken-up into 1-2 hour segments.

8.  Make informed decisions about technology. The benchmarking partners use a range
   of software programs for their courses including Microsoft Live Meeting through
   PlaceWhere,  WebEx, and  Microsoft  Windows Media  Encoder.   Each  partner
   considered using several programs before deciding on one of those listed above. For
HayGroup                                                                  Page 85

-------
EPA: Results of Benchmarking Study
    example, GMAC has used WebEx but has had difficulty recording and synchronizing
    both audio and video communications.  Media Encoder is now used  to record
    programs.  SAS makes the  assumption that  most of the top products have similar
    capabilities and makes decisions primarily based on cost and whether the vendor will
    be around in a few years. The assumptions made by SAS is that each top product will
    have:
       >   Application sharing
       >   PowerPoint capabilities
       >   Chat capabilities
       >   Registration capabilities

Satellite Broadcasts.  Satellite broadcasts were not the focal point of any  benchmarking
partner's  training  program,  although  some  benchmarking  partners  are  using  the
technology effectively to disseminate  critical  information and train  geographically
dispersed individuals.  As indicated previously,  our  research shows that most training
programs are moving toward more live Web-based delivery methods that  share many of
the benefits of satellite broadcasts.  One benchmarking partner  explained the  shift  by
saying that satellite courses do not meet the "my desktop, at my time" needs of most busy
professionals.  Another  benchmarking partner indicated that "live Web  courses have
taken over satellite courses because people have better access to them."

We discovered during the site visits some definite downsides of using satellite broadcasts
to deliver training programs, which are presented in table 5.

Table 5:  Limitations of Satellite  Broadcast
>  Facilities  - Satellite  broadcasts require  facilities that have  video production and
   broadcasting capabilities, and training  sites to have satellite reception capabilities.
>  Location - Satellite broadcasts  require participants to travel to specific locations (with
   satellite downlinks) to participate in the training session.
>  Scheduling -  Satellite courses  are difficult to schedule — courses must be  scheduled
   when the infrastructure and all participants are available.
>  Personnel - Significant personnel is  required to maintain the facilities, troubleshoot
   technical problems, and schedule and conduct courses.
>  Cost - The overall initial cost of developing the infrastructure to deliver  satellite
   programs can be high.  However, satellite broadcasts may be a cost effective training
   medium for organizations that have already developed the infrastructure.
>  Interactions - Satellite broadcasts are  typically one-way communications from the
   instructor with limited opportunities for participant interactions.	

Despite the  downsides  to  using  satellite  broadcasts, we did  find that two of  our
benchmarking partners, the CDC  and Georgia Tech, use satellite broadcasts (typically
recorded live) effectively for certain types of material.  One reason  they are able to use
the satellite technology  successfully for training is  that both sites have an extensive
satellite broadcast capability and maintain a relatively large staff that can develop training
HayGroup                                                                   Page 86

-------
EPA: Results of Benchmarking Study
programs using various media.  For example, the CDC has a staff of 50 professionals
dedicated to maintaining its training programs and Georgia Tech has three satellite dishes
and nine satellite capable classrooms.  OSHA  is using satellite broadcast minimally for
outreach or just-in-time updates.

Through the site visits and benchmarking research, we identified several best practices in
the delivery of satellite broadcasts.  Because satellite broadcasts  and live Web-based
courses have  similar designs,  capabilities,  and challenges,  many  of the  satellite best
practices  are  similar to those discussed previously in the Webcast/Webinar training
delivery  sections.   We present each  best practice  briefly  and expand  only on  those
practices that are unique to satellite broadcasts.

1.  Training goals  and content should dictate the use  of  satellite broadcasts.  Many
   experts and  the benchmarking  partners agree  that satellite  broadcasts  are  most
   effective  for delivering informational  programs  to  a  geographically  dispersed
   population.   Satellite broadcasts  are  also  effective  for providing just-in-time
   information and can be  used to deliver  technical programs that are broken up into
   short courses.   For example, the  CDC  uses the satellite broadcast  to disseminate
   critical information in  a timely manner (i.e., in less than 48 hours).  Some rapid turn-
   around satellite broadcasts sponsored  by the  CDC have focused  on  smallpox,
   bioterrorism, and SARS.

2.  Record satellite broadcasts.  When possible, our benchmarking partners  suggest
   recording at least the instructor in a live satellite broadcasts. One of the challenges of
   satellite broadcasts is  scheduling them so  that all  interested parties can participate.
   Recorded broadcasts can be placed on  asynchronous sites for later use, which is
   especially  helpful for  rapid turn-around  programs  such  as the CDC conducts.  The
   benchmarking partners stress that there are two important points to remember  when
   recording satellite broadcasts: 1) video broadcast require a lot of bandwidth and may
   be difficult to store on some sites or send  out as a CD-Rom; and 2) synching video
   and audio can be difficult.

3.  Incorporate opportunities for interaction into the satellite broadcast.  Similar to the
   live Webcasts, our benchmarking partners recognize that for successful learning to
   occur, participants in a satellite broadcast must have the  opportunity to ask questions
   and discuss the  broadcast with others. To  promote interactions  between participants
   and  instructors  during  satellite broadcasts,  many benchmarking  partners  give
   participants the opportunity to  ask questions during the broadcast.  Similar to the
   Webcasts, we found that the most common  methods used to elicit questions  include:
       >  Call-in questions
       >  Fax-in questions
       >  E-mail questions

4.  Use screeners to review participant questions.   Some of the benchmarking partners
   find that instructors receive too many questions to respond to during the satellite
HayGroup                                                                   Page 87

-------
EPA: Results of Benchmarking Study
   broadcasts and that reading each question individually takes time away from teaching.
   To meet this  challenge, a form of question screening and  monitoring has been
   incorporated where one or more individuals are available to receive, read, and review
   questions to determine whether the instructor should respond to them during the live
   broadcast.

            Example of Question Screening During Satellite Broadcasts
The CDC maintains a call center staffed with  4 to 5 individuals that accept  and  monitor
phone-in questions during satellite broadcasts.  Their job is to evaluate questions before
deciding whether to patch them into the live broadcast.  Questions that do  not promote
topical discussions are saved and later posted to an asynchronous site, while those that do
promote the discussion are patched in live to the instructor.  By using the telephone
monitors, the instructor is free to teach the course without interrupting the  flow to wait
for questions.

5.  Publicize broadcasts.   Because satellite  broadcasts can be  difficult to schedule,
   publicizing them is very important.  A rule-of-thumb  is to begin publicizing each
   broadcast  approximately 30  days in  advance  by using a variety of  media (e.g.,
   Intranet, newsletters, supervisor communications). The CDC has distance learning
   coordinators in every state to promote  the satellite broadcasts (e.g., sending fliers
   about the broadcast to health care workers) and assist with registration for broadcasts.

6.  Ensure  potential  participants  can participate in the  satellite broadcast.  Before
   developing a satellite broadcast, it is critical to ensure that all participants will have
   access to it.  Access may be at participants' home office or they may need to travel to
   a regional center.

Other best practices that apply to both live Web-based training and satellite broadcasts
are:
       >   Train instructors in  satellite  training  methods  so  they are  effective  at
           delivering the broadcast
       >   Use technology monitors
       >   Limit the length of broadcasts to 1 - 2 hours per session

Videoconferences.  Both  the   CDC  and  Georgia Tech  have   also  incorporated
videoconference capabilities into their training programs.  Videoconferences are similar
to  satellite broadcasts but have the capability to better incorporate two-way  interactions.
Instead of a camera focusing primarily on the instructor, Videoconferences  may have
several cameras set up in different locations that are able to show the participants at each
of the  locations.  Videoconferences allow participants to ask questions of the instructor
and each other and discuss the training material throughout the session.

Many of the downsides that we found with satellite broadcasts are also  associated with
Videoconferences,   and some  may  be  more  extreme.    One   example  is  that  for
Videoconferences to be a viable training method, training  sites must have not only the
HayGroup                                                                   Page 88

-------
EPA: Results of Benchmarking Study
capability to receive  broadcasts but also the capability to send broadcasts.  Another
example is that recording the sessions is even more difficult because both the audio and
visual   communications  are  two-way  and  capturing  this  requires  an  extensive
infrastructure that most training programs cannot afford.

Videoconferences  are best used when the training must take on  the feel of a more
intimate meeting and where extensive interactions are required.  Because the media is
used to promote  continuous  two-way conversations,  the  number of participants or
participant sites  should be limited to a number that is effective for action meetings (i.e.,
10 or fewer).  In addition to the best practices we identified for satellite broadcasts, we
found the following best practices for videoconferences.

1.  Establish etiquette.  Before beginning a videoconference, all participants should be
   reminded  of proper etiquette.   Videoconferences  provide  all  participants  the
   opportunity to see and hear each other, and thus interact in an intimate and possibly
   informal atmosphere. Unfortunately, participants sometimes forget that and say or do
   things that would be considered inappropriate in  a  regular meeting.  For example,
   participants  in one  site  may  constantly  move around  and  disrupt  the visual
   communications for others, and participants in another location  may forget to press
   mute while  having  discussions  amongst  themselves  about the program,  other
   participants,  or even lunch plans.

2.  Contract with a vendor.  Videoconferences often require facilities and technology that
   most training programs  cannot afford and many industry leaders  are not  sold that the
   costs outweigh the benefits of videoconferences.  One solution is to contract with a
   vendor such as Georgia Tech that maintains a staff dedicated to maintaining the
   videoconference facilities.  Georgia Tech has, for example, the capability to centralize
   all  communications and can even convert dissimilar videoconference formats into a
   common one.

Asynchronous Delivery Methods
Asynchronous training does not require live instructors  and is commonly referred to as
self-paced training. Most of the benchmarking partners agree that asynchronous training
delivery methods should be used when live interpersonal interactions are not necessary,
when trainings are informational-based or include simulations, and when participants are
geographically dispersed. Asynchronous delivery methods provide participants with the
opportunity to access training when and where they want.  All five of the benchmarking
partners use one or more asynchronous training delivery methods, with self-paced Web
training being used most frequently, followed by CD-Rom/DVD  courses.  Paper-based
(self-paced) training courses are not as commonly used as other  asynchronous  training
delivery methods.

As with synchronous training methods, the primary challenge faced by organizations that
use asynchronous methods is incorporating interactivity into courses.  Because there is no
live interactions with these methods, this challenge  takes on  a  different meaning for
asynchronous courses. Not only must the course provide opportunities for participants to
HayGroup                                                                   Page 89

-------
EPA: Results of Benchmarking Study
interact with each other, but it must incorporate methods for participants to interact with
the technology.   Also, while impromptu  interactions may be likely in synchronous
courses, all interactions must be specifically designed into an asynchronous course.

For the remainder of this section, we present our findings on the best practices and trends
associated with asynchronous training delivery methods: Web-based  (self-paced), CD-
Rom/DVD, and paper-based (self-paced).   Similar to the synchronous training delivery
methods, we focus on Web-based delivery because we feel this is the direction that most
of our benchmarking partners and the industry are moving. However, many of the best
practices associated with self-paced Web courses can be applied to other asynchronous
training methods.

Self-Paced Web Courses.   Industry experts agree that self-paced Web courses are better
for some course content than other.  For example, Hall (2001) suggests that  the Web
works best for courses that:
       >   Focus on content and information and are fact-based
       >   Do not require experiential learning that closely mirrors job situations
       >   Require minimal  interactions among  students   and  instructors and are not
          intended to develop interpersonal skills

Through our site visits and benchmarking research, we learned that there is a great deal of
variability in the effectiveness of self-paced Web courses.  Many organizations are in the
rudimentary stages of developing  asynchronous  Web courses, simply putting PowerPoint
slides on the Web site or putting material from written self-study training courses on the
Web. Further, the American Society for Training and Development (ASTD) claims that
most on-line training lacks interaction and quality instruction  (from ASTD E-Learning
Handbook).  The industry  leaders, however, are creating interactive, Web-based courses
that  engage participants through simulations,  quizzes, chat rooms, Web boards, and so
forth.

We identified several best practices in self-paced Web courses:
1. Provide opportunities for interactions with other students and instructors.   Many of
   the benchmarking  partners and other  training industry leaders have incorporated
   several methods for allowing students to ask an instructor, TA,  or other students
   questions  while  they  are completing a self-paced  Web  course.   Examples  of
   interactive methods used are:
       >   Create a link within the Web  site  where students can email questions to
          instructors or set up electronic office hours.
       >   Create a Web board where students can post questions that are answered by an
          instructor or others students (also known as threaded discussions).  Instructors
          can also post updates and new slides and materials to the Web board.
       >   Incorporate  chat rooms  where  students can discuss course  issues and ask
          questions of each other and the instructor.
HayGroup                                                                  Page 90

-------
EPA: Results of Benchmarking Study
       >  Encourage cohorts of participants to complete the course together at the same
          time. It is easier and more efficient to provide interactive opportunities to a
          group of participants taking a course at the same time.  For example Georgia
          Tech sets up chat rooms and bulletin/Web boards so cohorts can discuss the
          course and ask each other questions while they are taking  the course (e.g.,
          between sessions).     Another  organization uses a course moderator  who
          introduces the self-paced Web course, asks questions throughout the course to
          involve students, answers student questions, and facilitates interactions among
          students taking the course at the same time.
       >  Use audioconferences as a de-briefing after students complete the  self-paced
          course.   This provides  students the  opportunity  to ask questions  about the
          course and discuss course content with other students.
2.  Provide  opportunities for  interactions  with the technology.   Asynchronous Web
   courses do not inherently  require interactions.   However, interactions are  deemed
   necessary for training to be effective, and advancements  in technology have made it
   easy to incorporate strategies for interacting with the technology.  For asynchronous
   courses, it is critical to keep participants involved in the course so they will actually
   complete the course and learn the required skills/knowledge. By incorporating some
   of these interaction strategies, a self-paced Web course can be an effective  method of
   providing technical types of training.   Some examples of how interactions with the
   technology are incorporated are presented below.
       >  Quizzes/assessments - Some of the benchmarking partners use short quizzes
          or assessments to foster interactions with the technology. For example, during
          a Web-based PowerPoint presentation, SAS presents short quizzes  (often just
          one question) every few slides to keep  participants interested and engaged
          with the course.  The course then provides feedback to trainees on whether
          they answered the  questions correctly  and explanations for those that are
          answered incorrectly.
       >  Simulations - Some intricate simulations have been developed and used to
          augment self-based Web training. Effective simulations reflect the  real world
          and allow participants to learn by doing and practice a skill taught in the
          training session.  The primary types  of simulations are software application,
          scenario-based, and business and financial simulations.  For example,  SAS
          provides  instruction and then has  participants  practice key  tasks  before
          returning  to the main  training  session. Other  organizations  show actual
          equipment used  on  the job and use three dimensional graphics to instruct on
          correct usage of the equipment.
       >  Case studies  - Many  industry  leaders  are incorporating  interactive  case
          studies into Web-based training by providing trainees with problems to solve.
          Case studies serve much the  same purpose as simulations; they provide real
          world examples to augment the training.   Interactions with case studies can be
          enhanced by presenting  them as points  that  cohorts can discuss on a Web
          board between sessions.
HayGroup                                                                  Page 91

-------
EPA: Results of Benchmarking Study
                         Example of Interactive Quizzes
During a self-paced Web program, SAS often presents quizzes.  The quizzes  can be
programmed  so that participants providing wrong answers  may be  blocked from
continuing with the training until a correct answer is provided.

                        Examples of Interactive Simulation
Bank of America uses streaming video and audio in its self-paced Web training course to
allow loan officer trainees to interact with a prospective simulated client.  Trainees ask a
series of questions to determine whether the client would be eligible for a loan.  (Taken
from Brandon Hall,  Six Steps to Developing a Successful E-Learning Initiative, 2001).

Cisco provides on-line access to equipment labs so that the trainee can, for example, try
configuring a switch or router before actually doing it on the company site.

3.  Create  engaging material. Although  the  course content  is  the most important
   determinant of the effectiveness of a self-paced Web course, the look and feel of the
   training (Web site) is a close second.  Research cited by David Daly and Amy Scott
   (Best Practices Handbook:  Best Practices for Advanced Distributed Learning) shows
   that individuals  learn 75% of what they know through vision.  They indicated  that
   "pictures and words together are six times more effective than words alone."

   Industry leaders use graphics and pictures to make self-paced Web training more
   interesting.  However, graphics and pictures should be relevant to the training  and
   visually represent or enhance the content of the Web course (not just "nice to  have"),
   and be  easy to  read.  Another recommendation made by industry leaders  is  that
   presentations should maintain the look and feel of the organization (branding).

4.  Make it easy for participants to access the training.   Some self-paced Web courses
   require participants to download programs such as Flash or WebEx, or update their
   existing Web browsers.  However, many organizations  limit  what employees  can
   download from  the Web to maintain security and prevent viruses from  entering a
   network. It is critical for organizations to provide trainees with technical  support for
   accessing Web-based courses.  For example, to assist participants, SAS  sends  CDs
   with downloading materials to those who are unable to download the materials from
   the Web.

5.  Demonstrate a commitment to Web-based training.  Industry  leaders demonstrate a
   strong commitment to Web-based training by encouraging employees to complete the
   self-paced course on company  time.   Unlike employees who complete training  at a
   location away from the office, self-paced Web training introduces the challenge of
   ensuring that employees  are free to complete the training without distractions (e.g.,
   supervisors  interrupting trainees and  asking them to attend a meeting;  a customer
   calling a trainee). For example, at Cisco Systems, employees who are completing an
   on-line class can put up yellow police tape to signal that they are in the  process of
   completing a Web-based course.
HayGroup                                                                  Page 92

-------
EPA: Results of Benchmarking Study
6.  Be thorough with content and delivery methods.  Because self-paced Web courses are
   essentially stand-alone programs, it is even more important to make sure that content
   is up-to-date and that delivery techniques work.  While  synchronous courses have
   almost instant feedback loops, asynchronous courses do not. Participants may not be
   able to access or work within the program and  it may take weeks or longer for the
   training developer to find out that students are having difficulty with the training.

CD-Rom/DVD and paper-based self-paced courses.  Completing courses by inserting a
CD-Rom/DVD into one's personal computer and completing a paper-based  self-study
training course are two other forms of asynchronous learning.  None of the benchmarking
partners make extensive use  of CD-Rom/DVD or paper-based training materials,  but
some did augment training with these delivery methods.  The most common method used
was to send already developed training materials to participants (in a CD-Rom/DVD or
paper format) who could not access a  self-paced Web  course.  The materials sent are
essentially the same as those posted to the Web site.  One of the benchmarking partners,
OSHA,  indicated that it does  not use  CD-Rom because  the data can get outdated easily
and there is  not a way to update it quickly.  OSHA also feels that with CD-Rom, it  can
lose control of the course content.

However, many  organizations we learned about in our  literature  review are using CD-
Rom or DVDs to replace Web-based training when potential trainees are on the road or in
foreign  countries and do not have Internet access.  Other  organizations have created
electronic libraries with hundreds of CD-Rom and DVDs that were created to capture live
satellite broadcasts, classroom courses, or Webcasts.  These libraries provide just-in-time
training for individuals who missed the live course, new employees, and so forth.   For
example, Georgia Tech records all videoconferences on CDs and DVDs so that they  can
be used to train individuals that could not attend the live sessions.

Blended Learning
Another trend we identified through the site visits and benchmarking research is  the
introduction of blended learning programs. Blended learning programs incorporate two
or more training delivery media to provide not only a more holistic learning experience
but also multiple learning situations for the busy professional.  Many training experts  and
industry leaders make the claim that blended learning is more effective than any single
training delivery  method.  For example,  OSHA  is moving toward a blended learning
approach.  OSHA uses the Web-based portion of a blended course to bring all students up
to the same level by the time they get to the classroom.

One  interesting trend in blended learning is the  shift  back to using  classroom-based
experiences  to augment distance learning programs.  That  is, many  organizations  are
combining classroom and Web-based training.  For example, a self-study Web  course
may be used to provide critical knowledge before the classroom session (as a prerequisite
to the course) or as a wrap  up after the classroom session.  The classroom session then
focuses on  interactive  exercises, discussions,  case studies,  and  simulations  versus
delivering information in a lecture format. Other organizations are combining self-paced
materials with live  Webcasts.  For example, Stanford University has  trainees review
HayGroup                                                                  Page 93

-------
EPA: Results of Benchmarking Study
regulations in a self-study Web or paper format and then participate in a live moderated
Webcast to discuss the implications of the regulation to their jobs, ask questions, and so
forth.

Blending learning programs  have the  opportunity to  provide a  more  exciting and
enriching training experience.  Rather than being limited to one training medium, blended
learning programs utilize the best practices of two or even several  training methods.
These types of programs also offer greater flexibility to both instructors and participants.
Classroom-based  or even satellite broadcasts often require that participants travel to a
training site.  If a training program is  scheduled to last five  days, participants must travel
all five days.   However, the blended learning approach can help reduce travel time by
presenting day 1 introductory and day 5 wrap-up information over the Web.  This reduces
travel costs and potentially frees-up participants'  time so they can work on other tasks
rather than spend all day at a training site.

Some benefits of blended learning programs cited by industry leaders are:
       >  Greater flexibility  - instructors and  participants are not  constrained  by  a
          specific delivery method
       >  Improved learning effectiveness - the use of a variety  of delivery methods
          provides a better match between delivery and participant learning styles
       >  Greater reach - training  can better reach  participants  with scheduling  or
          technological constraints
       >  More time spent training and less time spent lecturing - more instructor time
          can be spent on hands-on training rather than lecturing.
       >  Greater opportunities  to  cut costs - expensive delivery methods  can be
          replaced or augmented with  other methods, rather than eliminating an entire
          program

                       Example of Blended Learning Program

One  organization recently converted a 5-week long  satellite broadcast training  program to a
blended learning program.  Before it is converted to the blended learning course, 20 participants
participated in the  training watching a satellite broadcast as a  cohort (4 training sites with 5
participants per site). Participants watched one satellite broadcast per week for 5 weeks.

The  organization decided to  convert this  course to  a  blended learning program because
conducting training solely through satellite broadcasts required a great deal of scheduling, forced
participants to  attend pre-set sessions at a site away from their office, and required technical
support staff to be available each week.  By using the following blended learning approach, the
instructor  could  limit the time he/she and  the participants  spent in satellite broadcasts  while
providing a more robust training program for participants:

Week 1 - The instructor presents an introduction to the training program via a satellite broadcast.
He/she demonstrates some key  tasks that participants need and provides  an opportunity for
participants to see and meet each other.
HayGroup                                                                      Page 94

-------
EPA: Results of Benchmarking Study
Week 2 -  Information is delivered via the  self-paced Web method.   This session  is more
informational and does not require any demonstrations. However, while participating in the Web
session, trainees practice the skills  demonstrated in week 1  by  completing a short simulation.
Participants are free to access this session anytime during the week and are provided a Web board
to interact with other participants.

Week 3 - This session is conducted via a live Webcast.  Similar to week 1, the instructor
demonstrates some skills via a live  session. The Webcast is also taped so that participants can
access it during the week.

Week 4 - The training is once again conducted via the self-paced Web method. Participants are
provided the opportunity to practice the skills demonstrated in week 3.

Week 5 -  The wrap-up session is presented via live Webcast.  This session is  conducted
primarily so that the instructor can  provide final thoughts and answer any remaining questions
that participants have.

                      Example of Blended Learning Program

Georgia Tech uses a blended learning method for some of its training courses (typically
courses that require student interactions and group work).  Students complete a self-study
Web course and then get together in local classrooms after the course to discuss issues or
work in labs.

SUMMARY
Classroom-based  delivery methods  still  account for a large  proportion of training
programs, and many training  experts believe that there will always be a need for these
types of programs.   However,  live  Web-based training  (e.g.,  Webcasts) is  quickly
replacing  satellite broadcasts  as  the primary method  for delivering interactive training
programs.   Satellite broadcasts can  be expensive to develop  and deliver, difficult  to
schedule, and require participants to travel to a site with satellite downlink capabilities.
Live Webcasts, on the other  hand,  provide busy professionals with desktop  access  to
courses,  and technological advances  have led to even greater opportunities for both
interpersonal and person-to-technology interactions.  Additionally, live Webcasts can be
easily recorded and posted to a Web site for later viewing.

Many industry leaders also use a variety  of self-paced  training methods such as Web-
based courses and CDs/DVDs.  The trend is toward providing self-paced Web  courses
because they allow participants to access the materials at "their desktop and their time."

Incorporating methods for interactions into distance learning  training courses  is an
important best practice for enhancing their effectiveness.  Our results show that many
organizations  are implementing innovative strategies for increasing interactions  in both
synchronous and asynchronous training courses. Some of the methods include  providing
opportunities for participants to ask questions during the training session (e.g., live call-in
questions, Web boards), providing question screeners, and using simulations, case studies
and quizzes.
HayGroup                                                                    Page 95

-------
EPA: Results of Benchmarking Study
                   V.  Training Evaluation Process


Industry experts in the training field stress the importance of evaluating training programs
to ensure they add value to organizations and training participants. Training evaluation
data are critical for determining the extent to which the content of the training program
and the way in which the training is delivered results in increased skills and knowledge
for trainees, and positively impacts the organization's performance.  It is only through
this evaluation process that organizations can gain insights into ways to improve training
programs, and demonstrate that they get a good return on investment  from their training
efforts.

TRAINING EVALUATION LEVELS
Our research shows that despite the stated emphasis on evaluating training programs, few
organizations actually evaluate their programs beyond the commonly referred to "smile
sheets" or level 1 evaluations (see Table 6 for a brief description of  commonly cited
training evaluation levels).  ASTD stated in its 2003  State of the Industry Report that
75% of organizations use "smile sheets" (level 1) to evaluate their  training programs.
The  results from our site visits  reveal a similar trend - a  few training programs  are
evaluated at level 2 (learning) but most training  programs are not evaluated beyond level
1  (participant reactions).   Level  1 evaluations, when  done appropriately  (e.g.,  are
systematically developed, use standardized questions, include a balance of multiple-
choice and open-ended  questions), provide information for assessing the structure and
content of the course.  This type of evaluation  typically  provides an assessment of the
course content, format, teacher  effectiveness,  delivery  method, and course materials.
Type 1 evaluations are a necessary but not sufficient component of the overall training
evaluation process.
HayGroup                                                                  Page 96

-------
EPA: Results of Benchmarking Study
Evaluation
   Level
Level 1 -
Reactions
                        Table 6: Training Evaluation Levels
                                 Description
Often referred to as "smile sheets", a level 1 evaluation measures participant reactions and
satisfaction with the training course.  For example, did trainees like the training program
and was it what they expected? While any training program should be evaluated at least at
this  level,  it is  rarely a sufficient  measure  of the training program's impact or of
participants' learning.  For example, although a negative reaction reduces the possibility that
the training was effective, a positive reaction in no way guarantees effectiveness.	
Level 2 -
Learning
A level 2  evaluation moves  beyond  reactions  and attempts  to  measure changes in
knowledge, skills, and attitudes as a result of the training.  The most effective way to
determine whether learning has occurred is to conduct pre- and post-evaluations to measure
absolute change in skills/knowledge.	
Level 3 -
Transfer
A level 3 evaluation measures not only whether participants have learned but also whether
the learned skills are being used (transferred) to everyday work situations. Does the training
lead to changes in behavior on the job?	
Level 4 -
Results
A level 4 evaluation measures the bottom line results; did the training improve quality or
increase productivity?   These  are the  results that  top  line managers  and executives
understand and are looking for.	
See: Kilpatrick, D. (1994). Evaluating training programs: The four levels. San Francisco:
Berrett-Koehler.

As indicated previously, a few of the benchmarking partners do evaluate their training
using level 2 measures, which assess the extent to which   participants have learned
required skills/knowledge.  Examples  of level  2 evaluations used  by benchmarking
partners include: 1) quizzes or examinations to determine proficiency (many use pre- and
post-test exams to measure changes in skills/knowledge due to the training course); and
2) course certification or continuing education units (CEUs) for completing courses.

However, the benchmarking partners have found that providing certifications  or CEUs is
difficult in a distance learning format because it is often  difficult to monitor course
attendance.  For example,  SAS does  not conduct any certification  programs through
distance learning.  They believe that there is no way to truly monitor the process unless it
is administered through classroom-based training.   Georgia Tech  has attempted  to
alleviate this problem by requiring distance learning participants to use registered exam
proctors such as work supervisors, managers, or FIR representatives. These proctors are
responsible for ensuring  that the person taking the exam is the same person that is taking
the course.

While level 1  and 2 evaluations are important  and  can help  design better training
programs,  they do not  provide any substantive  information relating to the ultimate
outcomes or value of the program. The evaluations typically take place at the very end of
a course when  participants are tired  and less likely to take the additional time to provide
insightful feedback. Also, because they are conducted solely at the end of a course, there
are  no  opportunities to  track  whether  the   training has led  to  any   substantive
improvements in key organizational metrics
HayGroup
                                                              Page 97

-------
EPA: Results of Benchmarking Study
BEST PRACTICES IN TRAINING EVALUATION

In this section, we summarize training evaluation trends and best practices based on the
site visits and benchmarking research.

1.  Evaluate the impact of training on job and organizational performance.  Experts in the
   field recommend evaluating training programs  beyond levels 1 (reactions) and 2
   (learning) to demonstrate the overall value of training programs.  By  showing that
   participants are able to transfer their newly gained knowledge and skills to everyday
   work tasks and  connecting this knowledge  transfer to  improved  employee and
   organizational performance, a training department is able to quantify, not just say,
   why training is important.

   While  in theory the benchmarking partners understand the importance  of evaluating
   training programs beyond  levels 1 and 2, none are systematically  measuring the
   extent  to  which  their training  programs  impact  individual  and  organizational
   performance.   The benchmarking  partners  are  still  focusing on  end-of-course
   evaluations —  they typically ask all trainees to complete a course evaluation form at
   the end of the training course, satellite broadcast, Webcast, etc. The primary focus of
   the evaluations seems to be on helping improve the overall design  of the training
   course (e.g., instructor effectiveness, flow of training, training materials).

   Most of the benchmarking partners  agree that higher level evaluations are very
   difficult to do because it often takes several months  before an organization may see
   any change in performance and unless the training is focused on a specific skill set, it
   is often difficult to demonstrate a cause-and-effect relationship between training and
   performance.

   However, through our benchmarking research (e.g., literature review), we identified a
   few training industry leaders that are conducting level 3 and 4  evaluations  of their
   training programs.   Below  we describe some of the evaluation practices  of these
   industry leaders.

                     Example of Level 3 Training Evaluation
One organization has implemented an extensive process for evaluating the impact of its
training programs  on employee performance.  Each month,  the training organization
selects courses for further evaluation.  Training participants receive an  electronic survey
approximately three months after course completion to assess how well they have applied
the training on the job.  The participant's supervisor receives a similar survey asking for
an assessment of the extent  to  which the  training has  improved their employee's
performance on the job.  These surveys are used to refine and improve training courses.
HayGroup                                                                  Page 98

-------
EPA: Results of Benchmarking Study
                     Example of Level 4 Training Evaluation
Cisco Systems is  using  several methods to assess the overall  effectiveness  of its E-
Learning efforts. Examples include:

Cisco examined a sample of resellers taking a certification course (1/2 through classroom
training and  !/2 through  Web-based training).    The  pass  rates for  each group were
compared, and the  e-learners had a 10% better pass rate than the classroom learners.

Cisco also looks at cost savings for e-learning courses.  Cisco has found that Web-based
training saves  millions of dollars per year because  it reduces  the  amount of time
employees  take to  learn a  desired skill/knowledge,  and ultimately improves their
performance  on the job.   Other cost  savings  include reduced travel cost  to  attend
classroom training and less time away from the customer (which increases productivity).

Taken from:   Mission E-Possible:  The Cisco E-Learning Story (Patricia A. Galagan,
February 2001).

Training evaluation experts provide a good piece of advice to help move to level 3 and 4
evaluations:  begin the planning  process early.  Organizations must define the  desired
changes in employee or  organizational performance  and quantify how  they will be
measured before developing the training program.  These defined changes should help
shape the overall evaluation process so that a clear connection can be made between the
training program and desired results.  Level 3 and 4 evaluations can be accomplished, but
they take time and  planning.

2.   Conduct pre-and post-course evaluations to track learning as a result of the training
    course.   Many of the benchmarking partners conduct pre-course  evaluations as a
    baseline to measure learning throughout the courses. Pre-course evaluations provide
    an excellent measure of the knowledge, skills, and abilities that participants bring to a
    course.  Understanding what participants bring to a course not only allows instructors
    to tailor their messages, but it also provides a way to measure change due to the
    course (a level  2 assessment of the effectiveness of the training),

3.   Collect feedback from multiple sources. Industry leaders collect data from multiple
    sources  when  evaluating  training  programs.    Sources  may   include  training
    participants,  participants'  supervisors,  training developers  and  instructors,  site
    coordinators, and organizational  leaders.  Taken together,  these data can provide a
    complete  picture of the effectiveness of the training content and structure,  and most
    importantly, the impact of the training on participant job performance and  overall
    organizational results.

4.   Regularly review and use evaluation data to improve the effectiveness of training
    courses.  Although  the  benchmarking partners  typically  focus on level 1  and 2
    evaluations, they are  taking these evaluations seriously by monitoring the feedback
    provided by participants in training courses, and most importantly, using the feedback
HayGroup                                                                   Page 99

-------
EPA: Results of Benchmarking Study
   to make improvements to training materials, the way training is delivered, instructor
   effectiveness,  the technology,  and so forth.  They are using these data to make
   decisions  about  whether to modify  or redesign training  programs or eliminate
   ineffective programs.

         Example of Using Evaluation Data to Improve Training Programs
SAS has one  of the most extensive evaluation procedures of the benchmarking partners,
but the  evaluation still  does  not go  much beyond a typical "smile sheet."   SAS
evaluations occur immediately following a course. Courses that  do not meet a minimum
average rating of 3.5 out of 4 on key evaluation questions are further examined to assess
why the ratings are low. SAS will directly contact customers to elicit further information
about why courses were reviewed less favorably than expected.

5.  Extend training evaluation beyond classroom courses.   Organizations tend to focus
   on evaluating classroom training and do not conduct extensive evaluations of their
   Web-based or satellite courses.   Industry  experts recommend that organizations
   moving away  from classroom training conduct evaluations to ensure that Web-based
   training:  1)  is as effective as classroom training at teaching  required skills and
   knowledge; 2) engages trainees to begin and complete  courses;  and  3) provides a
   cost-effective alternative to classroom training.

                  Examples of Evaluations of Web-based Courses
•  In its E-learning Handbook, the American  Society for Training and Development
   recommended the following types  of measures  for evaluating the  effectiveness of
   self-paced Web training:

•  Total number  of training hours - to  determine whether Web training reduces training
   time (which saves the organization money)

•  Attendance and retention rates (Did  employees  actually  complete the Web-based
   course?)

•  End-of-course questionnaires to obtain participant feedback on the effectiveness of
   the Web  course (Was the Web the best method for teaching the skill/knowledge?
   Was  the Web site easy to access and navigate?   Were  the quizzes and simulations
   effective?)

•  Pre- and post-tests to determine whether participants increased their knowledge as a
   result of the Web training course

•  Cost reduction - Did the Web course reduce training delivery  cost (e.g., travel)?

•  Productivity - Did the Web course  help to prepare new employees to be productive
   on their jobs quicker than more traditional training methods?

6.  Use  standard evaluation procedures.   The  benchmarking  partners  revealed  that
   another key area to consider in training evaluation is standardization of evaluation
   instruments.  For example, although the CDC has five standard evaluation questions,
   its clients are  encouraged  to  include  additional questions.   While this may help
HayGroup                                                                 Page 100

-------
EPA: Results of Benchmarking Study
   customize the process, the overall result is a lack of standardization beyond the five
   questions.  Additionally there is a lack of standardization for when evaluations are
   conducted.  While the CDC evaluates most asynchronous courses,  it is not standard
   practice to evaluate synchronous courses.

   Without a standard process, it is not only extremely difficult to monitor the overall
   impact  of training courses, but also to develop a  common goal  for all courses.
   Standardization of the process will help connect overall training goals and needs by
   focusing on a core set of measures that are changed only after all training goals have
   been met.

7.  Consider unique  ways to evaluate training programs.   The benchmarking partners
   demonstrate  some  unique  methods  of evaluating  their training  programs.   Both
   GMAC and SAS believe if training participants find the courses valuable, they will
   continue to  enroll in, and in SAS' case, purchase the courses.  For example, SAS
   monitors the  18-month re-buy rate of its training programs.  SAS believes that a
   successful training program  is one  in which  60% or more of  customers re-buy
   training.

SUMMARY

APTI is not alone in the challenge  to  conduct effective evaluations of its training
programs. However, like most of the benchmarking partners, APTI needs to  reexamine
its training evaluation process to ensure that it is measuring the "right things" - the extent
to  which the training is adding value to air professionals and their agencies as a whole.
To conduct such best practice evaluations, APTI should move its evaluation level up to
transfer and results.  That is, focus on evaluating the extent to which training programs
affect the performance of air professionals and key organizational metrics.
HayGroup                                                                 Page 101

-------
EPA: Results of Benchmarking Study
                   VL Conclusions and Next Steps


In this report, we identify best practices of leading training programs, particularly in the
areas of  needs assessment, course  design  and  content, training delivery  methods
(specifically  distance  learning),  and training  evaluation.   The  best  practices were
identified based on site visits with five organizations with innovative training programs,
as well as a literature review of current training trends and best practices.

We believe that there are several best practices that APTI should consider incorporating
into its Air Pollution Training Program to reduce the gaps that were identified during the
evaluation of the current training program (during Task 1). For example, in the Task 1
report, we concluded that  APTFs  distance learning training courses  (e.g.,  satellite
broadcasts, self-paced Web courses) did not include enough opportunities for interactions
between participants and instructors, among participants, or with the technology itself.

In this section,  we present a high level summary of those best practices that, if adopted,
may improve the overall effectiveness of the Air Pollution Training program.  The Task
3 report will provide more detailed options for improving the APTI training program and
a plan of action for doing so.

Training Needs Assessment
    1.  Incorporate a  skill/competency assessment into the needs  assessment  process
       versus simply  projecting the number of air professionals likely to attend various
       classroom training courses.    This type of assessment  will  provide  valuable
       information for determining the gap between the skills/competencies required for
       success  in the job and the levels currently possessed  by air professionals. This
       should be followed by a thorough review of the current APTI training courses to
       ensure they are meeting the training needs of air professionals.
    2.  Collect  data from multiple  sources (e.g.,  air professionals,  supervisors of air
       professionals,  regional consortium members, training  instructors  and  subject
       matter experts) about the need for different  types of air pollution training courses
       (both course content and delivery methods).
    3.  Use the results of training needs assessments to design more effective  training
       programs, and communicate  decisions about training courses (e.g., eliminating a
       course, adding a new course, translating a course from the classroom to a Web-
       based medium) to the air professionals  community.
Course Design and Content
   4.  Conduct regular reviews of training courses (e.g., every 1-3 years) and update
       courses as needs.   For example,  some updates may  require simply replacing
       content in a self-paced Web course. Others may require major overhauls or total
       redesign of a course.
HayGroup                                                                 Page 102

-------
EPA: Results of Benchmarking Study
    5.  Incorporate real-world  examples, simulations, and case studies into self-study
       Web courses.
    6.  Pilot test all courses before they go live.  For  example, pilot test participants
       should complete and evaluate all self-paced Web courses (e.g., for content, ease
       of using  the  technology,  value of interactive  exercises)  and feedback from
       participants should be used to improve the Web  courses.  As another example,
       live  Webcasts and satellite  broadcasts  should  be rehearsed and the  scripts
       reviewed before being delivered to air professionals.

Training Delivery Methods
    1.  Begin to make more use of live Webcasts (versus satellite broadcasts) to allow air
       professionals to complete the training at their desktops versus travel to a satellite
       downlink site.
    8.  Incorporate methods for interaction into distance learning courses (e.g., self-paced
       Web courses,  live Webcasts, satellite broadcasts) to enhance their effectiveness.
       For  example, provide  opportunities for participants to  ask questions  of the
       instructor and use question screeners during live Webcasts or satellite broadcasts
       (e.g.,  phone  or fax in questions).   For self-paced Web courses,  incorporate
       methods for:   1)  student-to-instructor  and  student-to-student interactions (e.g.,
       chat rooms, Web boards); and 2) interactions with the technology (e.g., real-world
       simulations, case studies, quizzes).
    9.  Ensure  that  individuals who  deliver  training (e.g., satellite broadcasts,  live
       Webcasts) not only are subject matter experts  but effective at teaching with the
       delivery medium being used.
Training Evaluation
    10. Expand the training evaluation process beyond end-of-course level 1 evaluations
       ("smile sheets").   Determine the extent  to which the training actually helps air
       professionals to do their jobs effectively and contributes to the mission of their
       agency.    For example,  solicit  feedback  from  air professionals  and  their
       supervisors 3-6 months after the training to determine the extent to which the
       training helped air professionals to be more effective on their jobs.
    11. Monitor and use evaluation data to make improvements to training courses (e.g.,
       training materials, the way training is delivered, effectiveness of simulations or
       case studies, ease of use of the technology).  Decisions about whether to modify,
       update, redesign, or eliminate training courses should be based on  data collected
       via training evaluations.
HayGroup                                                                   Page 103

-------
EPA: Results of Benchmarking Study
Our next step in the benchmarking study is to obtain input from APTI into those best
practices that  are most cost effective and feasible to implement.  We will facilitate a
discussion with the APTI team to determine:
       >  Extent to which the best practices can be incorporated into APTFs culture,
          processes, and procedures
       >  Whether APTI  has the resources (e.g., dollars, staff) to implement the best
          practices
       >  Barriers that may impede successful implementation of the needed changes to
          the program
       >  Key accountabilities for needed changes

After these discussions with the APTI team, we will write the final Task 3 report which
will provide options for making  APTI more successful moving forward, and a plan of
action for incorporating relevant best practices into the Air Pollution Training program.
HayGroup                                                                 Page 104

-------
EPA: Results of Benchmarking Study
              Appendix El: Site Visit Protocol
HayGroup                                                    Page 105

-------
EPA: Results of Benchmarking Study
        Questions for Benchmarking Partners who Participate in Site Visits

                        Environmental Protection Agency
                              Benchmarking Study
Background

    1.  What is the overall mission of your group?
    2.  Describe your trainee population.
    3.  How large is your staff?
    4.  What is your annual budget?
    5.  How would  you characterize the organization's commitment to your group's
       mission?

Overview of Training

    6.  How many courses do you deliver per year?
          a. By delivery method (e.g., classroom, Web-based, satellite)
          b. By type of course content
    7.  Describe the  different modes used to deliver training.
Course Design & Content

   8.  For each type of training (classroom, Web-based, satellite),
          a.  What is the process for course design and development?
          b.  How long does it take to design and develop the course?
          c.  What is the cost to design and develop the course?
   9.  Within a given year, how many new courses would typically be developed?
          a.  Do you primarily develop new  courses from scratch or update current
             courses?
          b.  How do you make decisions about whether to develop a new course or
             update an existing one?
   10. What is the process for developing course content for new courses?
   11. Describe the process(es) used to update courses.
          a.  How often do you update courses?
   12. What  are  best practices  for the development of content for satellite training
       course s/program s?
   13. What are best practices for the development of content for Web-based training
       courses?
   14. What  actions have you taken to incorporate  interactive  components into the
       design of distance learning courses?
HayGroup                                                                 Page 106

-------
EPA: Results of Benchmarking Study
Needs Assessment

    15. How do you determine what type of courses are needed?
    16. Do you perform a formal training needs assessment?
          a. If so, how often?  Who is involved?
    17. How do you use the needs assessment to make  decisions about training content
       and delivery methods?

Delivery Methods

    18. How do you determine what type of delivery method to use?  How do you make
       decisions about the mix of classroom and distance learning courses?
    19. In your  opinion, what  are  some  of the innovative ways  you  are  currently
       delivering training?
    20. What have you done to ensure that distance learning courses (e.g., satellite, Web-
       based) are effective?
    21. Do your distance learning courses have an interactive component?  If so, please
       describe how you make your distance learning courses more interactive.
    22. What do you see as the primary cost-benefit issues for EPA to consider when
       deciding whether or not to offer course content using delivery modes other than
       classroom training?
    23. How do you ensure that training is delivered in a timely manner to new hires or
       those who require just-in-time training?
    24. Do  you believe that modes of delivery other  than  classroom training are as
       effective at facilitating learning? Why or why not?
Instructors

   25. Do  you use outside vendors  to deliver training courses?   If so, what is your
       process for selecting vendors?
   26. Where do the course instructors come from (e.g., university, organizations)?
   27. How  do you measure the effectiveness of the instructor  (for  classroom  and
       distance learning courses)?
Course Evaluation

   28. How do you ensure course content and delivery are kept to the highest possible
       standard?
   29. What types of training evaluation have you undertaken?  Describe the course
       evaluation process.
   30. How do you assess whether your training is effective or that you have received a
       return on investment from training?
   31. How do you use training evaluation data?
HayGroup                                                                  Page 107

-------
EPA: Results of Benchmarking Study
Future
   32. What have been some of your "lessons learned" with respect to distance learning?
   33. What makes some training courses more effective than others?
   34. Are you involved in any type of train-the-trainer program  (i.e., training outside
       groups to take over some of the training currently being done by your group)?
   35. What  do you think the  future holds  for  distance  learning training  versus
       classroom training?
HayGroup                                                                  Page 108

-------
EPA: Results of Benchmarking Study
    Appendix B2:  Summaries of Individual Site Visits
HayGroup                                                 Page 109

-------
EPA: Results of Benchmarking Study
 Site 1: Occupational Safety and Health Administration (OSHA) Office of Training
                                  & Education
                                     Site Summary

The OSHA Office of Training and Education (OTE) is responsible for providing occupational
safety  and health training programs for Federal and  State  OSHA  compliance personnel,
consultation staff, other Federal agency personnel, and private sector employers and workers.
OSHA training programs emphasize the recognition, avoidance, prevention, and abatement of
unsafe and unhealthful working conditions.

With a staff of approximately 50 professionals, eighteen of which are instructors, the OSHA
Training  Institute  focuses  on  professional development  courses that  encompass  either  a
classroom or a blended approach (i.e. classroom and Web-based). OSHA offers approximately
80 courses per year  and has  ten education centers around the U.S.  which provide additional
training to other federal agencies as well  as the private sector.   OSHA uses recommendations
from staff or changes to regulations in order  to determine the needs  assessment  for new or
revised courses.   A functional competency model  has been developed to help  in  needs
assessments going forward.	
Needs Assessment

OSHA uses feedback from instructors, field personnel and changes in regulations to keep
up to date on what training is needed.

Course Content, Design, and Training Facilities

The OSHA Training Institute (OTI) will have delivered approximately 90 classroom or
blended courses in 2004.  To ensure that information is up-to-date, "Course Chairs" are
assigned to each course. A "Course Chair" is someone within the department who takes
ownership of a course.  They are responsible for keeping information current.  Typically
they review information every three years unless there is an immediate change or need to
be addressed.

New  courses are  rare at OSHA.   However, OSHA has  developed a seven  phase
development process to assist Subject Matter Experts in the development of new web-
based courses and transferring classroom content onto the web.  This is done through
utilizing a software package called Workforce Connections.  OSHA offers courses both
on the road and at their facility.  As many courses include some laboratory component
and OSHA has many of these labs set up for experimentation, these typically need to be
held on-site.

Delivery Methods and Participant Interaction

OSHA employs three primary delivery methods: 1) Classroom, 2) Web-based, and 3)
Blended. All three provide specific opportunities for participant interaction. In its Web
HayGroup                                                                 Page 110

-------
EPA: Results of Benchmarking Study
Based  Training  Style Guide,  OSHA recognizes  four  levels of interactivity:  passive,
limited, complex and real-time.

Classroom.  Classroom courses are held either on-site at the OSHA Training Institute or
on the  road. Classes at OSHA typically include some laboratory portion. These courses
provide real-time participation.

Web-based.   Internet  courses  are  delivered  in a  traditional  web-based  format.
Participation levels for web based training must include limited and complex levels.

Blended. Blended courses include a Web-based portion  of the course prior to the student
coming to  the  classroom.   The  Web-based  courses  include  limited  and  complex
participation while the classroom includes hands-on instruction.

Course Evaluations

Evaluations of both course content, and delivery and technology are typically conducted
at the end of each course. Evaluations focus primarily on content and teacher instruction.
However, Web-based and blended  courses include an evaluation  in  the development
stage, prior to going live.  Volunteers take a pre-test, the course and a post test.  Then
they fill out an evaluation including errors, concerns, questions and suggestions.  They
have a one-hour conference call with all volunteers to review the evaluations  and gather
additional information.  After this is taken into account and any changes are  made, this
portion of the course goes live.

Lesson Learned/Future Directions

       >  Blending  courses with a web-based component  allows students to review
          general content prior to the class meeting.   This  also allows all students to
          come to into  this portion of the course with the same level of knowledge and
          more time for hands-on instruction.
       >  Workforce Connections  has  been  a  significant  time and  cost  savings
          improvement on web-based course development.   It allows subject matter
          experts to design and maintain the courses in a user friendly manner.
       >  Some courses can not be solely web-based as  they require a hands-on  portion.
       >  Competency-based training is seen as an important initiative.
HayGroup                                                                 Page 111

-------
EPA: Results of Benchmarking Study
      Site 2: The Centers for Disease Control Public Health Training Network
                                 Site Summary

The  CDC Public Health Training Network (PHTN) is  a network of public, private,
academic, and  business  organizations  with the mission to develop a  public  health
workforce that is able and motivated to apply the current knowledge of disease control to
reduce human suffering, improve community health, and  control health care costs.  The
primary CDC customers are public health officials including, among others, physicians,
nurses,  emergency response personnel, mental health facilities,  veterans affairs,  and
school of public health.

With a staff of approximately 50 professionals, the CDC focuses on programs concerned
with all aspects of public health.  The CDC has the capacity and ability to deliver long-
term courses as well as react to immediate needs and produce an entire course in less than
48 hours.  Some examples of rapid turn-around issues are smallpox, bioterrorism,  and
SARS.  CDC  programs have reached an  estimated audience of 4,806,680 individuals.
Finally, the  CDC  supports  the  "learner at a  distance"  by  establishing  learning
communities  and  self-study  group leaders to  provide  guidance to CDC program
participants.

Courses are delivered  in both synchronous and asynchronous formats using a variety of
media  including the  Web,  satellite, and videoconference.   The CDC continuously
conducts needs assessments  to determine  individual course  needs and pre-tests Web-
based  courses to ensure applicability and usability.  All  courses are  evaluated  and
designed to promote interactions.	
Needs Assessment

The CDC currently conducts needs assessments to determine whether a specific course is
needed.  However, these assessments are not standardized and are not conducted with
every course.  The CDC is working to become more goal-focused and plan to conduct
more needs assessment to ensure that training programs meet stated goals.

Course Content, Design, and Delivery Facilities

The CDC develops approximately 30 - 40 new courses per year and has produced 853
products  focused on terrorism and emergency response programming.  The technology
utilized for these courses includes:
       >   Internet,
       >   Satellite transmission,
       >   Cable TV,
       >   Audio conferences,
       >   CD-Rom/DVD,
       >   Video tape, and
       >   Print-based self learning.
HayGroup                                                                 Page 112

-------
EPA: Results of Benchmarking Study
Course development is primarily initiated by a client outside the CDC, and clients have
primary responsibility for updating course content.  Clients may  also ask the CDC to
update the content, which results in a new project rather than a continuation  of the
existing project.   CDC has  established relationships with  several external  distance
learning networks that help distribute CDC programs at little or no cost.

A project typically begins when a client (various  CDC divisions) contacts  the  PHTN
about developing a course.  The CDC typically provides instructional design expertise
and expects the client to provide content expertise.  Although most projects are initiated
by a client, the CDC has established different processes for developing and delivering
courses for the different types of  media.   The different  development and  delivery
processes are described below.

Internet Course Development.
       >   Course design and the look  of the training are  developed in-house  by  an
           instructional  design expert and graphics specialist.
       >   The CDC provides a  web developer (contractor) with a  Power Point briefing.
           The web developer then posts the training to the Internet.
       >   The CDC found that  using a contractor is more expensive than hiring an in-
           house web developer.
       >   Courses with continuing education credits are pilot tested to determine length
           and applicability to assess the number of credits to be offered. Pilot testers are
           composed of at least five representative of the target audience.
       >   A clearance process is initiated to finalize the program.
       >   A web-based training  course of 120 pages takes about 3-5 months and
           $120,000 to develop.

Satellite Course Development.
       >   The client and subject matter experts (SMEs) provide the  CDC with content
           information.  The CDC requests that clients also provide an outline of course
           content.
       >   The CDC hires a script writer to develop a broadcast.
           o  SMEs often don't want the entire broadcast to be scripted; however, this is
              necessary to make certain the broadcast flows as seamlessly as possible.
       >   A full dress rehearsal  is conducted the day before the broadcast.

Online Registration System

CDC  has  an online registration system that serves as one  source for  marketing,
registration, testing, evaluation, continuing education certificates, and  learner transcripts.
This  system  allows  students to  search and register for  courses,  complete  course
evaluations and tests, and view and print transcripts and education certificates.
HayGroup                                                                  Page 113

-------
EPA: Results of Benchmarking Study
Delivery Methods and Participant Interaction

The CDC employs two primary training delivery methods: 1) Internet, and 2) Satellite.
Each  delivery method provides specific opportunities for participant interaction.  The
methods for interaction are described below.

Internet. Internet courses are delivered in either a traditional web-based (asynchronous)
or a  Webcast (synchronous)  format.   Methods of enhancing  student interaction  are
limited and include:
       >  Web boards and threaded discussions, and
       >  E-mail links to send instructors questions.

Satellite.    Satellite   courses  include  live broadcasts  that  provide  participants  the
opportunity  to interact  directly  with the instructor.   Communication  can occur either
before or during the broadcast. Methods of interaction include:
       >  Call-in questions,
       >  E-mail questions and
       >  Fax-in questions.

Satellite Course Delivery.
       >  Distance learning coordinators  are available  in every state to  promote and
          assist with broadcasts.
       >  Moderators are available during each broadcast to help maintain flow and
          screen questions.
       >  Most broadcasts  are  done live  and  the CDC tries to get subject experts to
          deliver the broadcasts.
       >  The CDC  asks clients to develop at least 10 questions  in case there  are no
          audience questions.
       >  Some broadcasts are also delivered via Webcasts.
       >  Continuing education credits are often available.
       >  Some broadcasts are recorded for later delivery via CD-Rom or the Internet.
       >  Participants are invited to provide comments about the program.

The CDC has a call center used  to accept calls through a posted 1-800 number. Four to
five individuals are typically used to accept calls and screen  questions.  If a question is
accepted the question can  be broadcast  live.   Questions that  are e-mailed may be
answered live or posted to an asynchronous web site.

Course Evaluations

Evaluations  occur before and during course development and following course delivery.
For each web-based course, there is a formal evaluation during the development phase to
ensure that the content and design are appropriate for the intended audience. These pilot
test evaluations collect important information about the appropriateness of course content
and style from those most likely affected by the training; potential  participants, training
coordinators, and site facilitators.  Satellite courses are not subjected to as  rigorous of a
HayGroup                                                                  Page 114

-------
EPA: Results of Benchmarking Study
pilot test procedure because satellite courses are synchronous, easier to modify, and more
difficult to script.

Both  synchronous and  asynchronous  courses  are  evaluated by  participants upon
completion for content and delivery.  The CDC uses standard level  1 evaluations ("smile
sheets"), and has five standard evaluation questions that clients may add to (e.g., Did you
learn  what you  expected?   Did the training meet your objectives?).  Because clients
typically add their own questions the CDC has had difficulty  standardizing the entire
evaluation process.  Satellite courses are not evaluated as regularly as other courses
because they are typically one-time sessions.  However,  many clients will evaluate them
to help improve future broadcasts.

Course Examinations

The  CDC encourages  clients to  administer examinations to students in asynchronous
courses to help determine  whether the  entire  course  was completed by the same
individual.  Because  synchronous courses  are  typically one-time  events, a  course
evaluation is typically  deemed sufficient to track  completion. Examinations are further
used to award continuing education credits.

Lesson Learned/Future Directions

       >  It can take a long time to get training content cleared/approved.
       >  It is best to have one primary client representative.
       >  "Don't buy it when you can borrow it."
       >  Publish locations of satellite links on your website.
       >  Hire a script writer to make satellite content more conversational.
       >  Do a  dress rehearsal when possible for satellite broadcasts.
       >  It is more expensive to hire outside contractors than use internal resources.
HayGroup                                                                  Page 115

-------
EPA: Results of Benchmarking Study
  Site 3: Georgia Tech Distance Learning and Professional Education Department
                                  Site Summary

The primary mission of the Georgia Tech Distance Learning and Professional Education
(DLPE) Department is to provide professional  education  courses  for  both internal
Georgia Tech and external clients, including both public and private organizations. The
Department offers programs that are designed to strengthen and update existing skills and
also teach new skills to help individuals achieve their own and organizational goals. The
Department further helps to facilitate Georgia Tech's distance learning capabilities by
taping conventional Georgia Tech academic courses for use in distance learning.

With  a staff of approximately 60 professionals, Georgia Tech focuses on professional
development, non-credit courses that range between one  and five days in duration.  In
addition, over 40 Certificate Programs are offered for which participants are eligible to
receive Continuing  Education Units (CEUs).  Finally, Georgia Tech offers conferences
and seminars, has the capability to  coordinate remote meetings via teleconference, and
provides instructional design and consultation.

Courses are delivered in both synchronous and asynchronous formats using a variety of
media including the Web,  satellite, and videoconference. Georgia Tech continuously
conducts needs assessments to determine individual  course needs as  well as general
training needs.  All courses are evaluated and designed to promote  interactions between
participants.	
Needs Assessment

Georgia Tech continuously conducts needs assessments to determine the market's needs
for proposed courses as well as overall industry training needs.

Course Content, Design, and Training Facilities

Georgia Tech will  have delivered 113  courses  in the 2004  academic year (about 45
courses per semester  plus  summer),  including  1000 course enrollments, 3000  student
credit  hours, and  95  faculty  members.   To  ensure  that information is up-to-date,
productions are updated with each subsequent course.  The technology utilized for these
courses includes:
       >  Internet,
       >  Satellite transmission,
       >  Cable TV,
       >  ITFS - "Wireless Cable",
       >  MPEG2,
       >  Voice Over Internet Protocol (VOIP),
       >  CD-Rom/DVD, and
       >  Video tape.
HayGroup                                                                 Page 116

-------
EPA: Results of Benchmarking Study
Course development is primarily initiated by either a Georgia Tech faculty member or an
external customer, and course content is primarily the responsibility of the customer.
Georgia Tech will contract with faculty to assist with the development of course content;
however, the primary roles of Georgia Tech are course design and delivery.  In addition,
Georgia Tech  assists students with course registration, textbook purchases, and receipt
and delivery of homework, reports, and examinations. The services provided by Georgia
Tech include:
       >  Video production.  Video production is primarily  geared toward producing
          videotapes for distance learning.  Georgia Tech has 9 classroom/studios, each
          with  a  high  bandwidth  Internet connection,  computer,  four  remotely
          controlled cameras, two front displays, two rear monitors, one portable control
          panel, and a VHS handheld camera.
       >  Tape  duplication and video editing.  Georgia Tech has approximately 80 VHS
          duplication decks, and linear and non-linear editing capabilities.
       >  Optical Media duplication.  Georgia Tech can duplicate and print labels for up
          for  1000 CD-Rom and DVDs.
       >  Teleconferencing/Satellite.   Georgia Tech has 9 Tandberg teleconferencing
          units, a  12-site MCU used to combine  dissimilar VTC formats into a common
          one, and three satellite dishes.
       >  Streaming video.  Georgia Tech can encode video for streamlining format at
          56kbps or with a DSL, Cable modem, ISDN, or Tl line.

Delivery Methods and Participant Interaction

Georgia Tech  employs three primary delivery methods: 1) Internet,  2) Satellite, and 3)
Videoconferencing. Each delivery method provides specific  opportunities for participant
interaction. The  methods for interaction are described below.

Internet.  Internet courses are delivered in either a traditional web-based or a Webcast
format.  Methods for incorporating interaction include:

   Traditional Web-based
       >  Web boards and threaded discussions,
       >  E-mail links to send instructors questions, and
       >  Face-to-face classroom discussion (when possible);

   Webcasts
       >  Chat rooms,
       >  Voice Over Internet Protocol (VOIP), and
       >  Web boards and threaded discussions.
HayGroup                                                                  Page 117

-------
EPA: Results of Benchmarking Study
Satellite.    Satellite  courses  include live broadcasts  that  provide  participants  the
opportunity to  interact  directly  with the instructor.  Communication can occur  either
before or during the broadcast.  Typically a moderator is available to take and filter
questions before giving them  to  instructor.   Methods for incorporating  interaction
include:
       >   Call-in questions, and
       >   Fax-in questions.

Videoconferencing.   Videoconferencing  provides  the  best  opportunity  for  direct
interactions between instructors  and students.  Because both parties are able to hear and
see each other interactions  most  typically  take place in the form of a  continuous
discussion.

Course Evaluations

Evaluations of both course content, and delivery and technology are typically conducted
at or near the end of each course.  Evaluations focus primarily on content and teacher
effectiveness and are often referred to as  "smile sheets". However, evaluations of course
delivery and technology are also conducted to help Georgia Tech further  understand
current trends in distance learning.  Sample evaluation items include:
       >   The video signal is clear and easy to see
       >   The video operator shows what the instructor is pointing at
       >   Course materials are received  in a timely manner
       >   The website is comprehensive and easy to use

Course Examinations

Georgia Tech  encourages the  integration of student  examinations  into all courses.
Registered proctors are made available and students are required to select a local proctor
for  examination administration.  One issue with distance learning, because of a lack of
oversight, is that it is difficult to ensure that the person taking the exam is the person
taking the  course.  Georgia Tech has  attempted to alleviate  this problem by using  and
requiring registered exam  proctors.  Proctors  must be  a  supervisor, manager,  or  HR
representative.

Lesson Learned/Future Directions

      >  Distance learning is going desktop, on demand.
      >  Presenters/instructors must have teaching abilities.
      >  Just because the capacity  to do the biggest and best is available, you don't have
         to use it.  Target your programs to your audience and their needs.
      >  Make training actionable.
      >  Make sure people have the resources to receive training (e.g., Internet, satellite
         receivers, CD players).
      >  Provide continuous technical support to end users.
HayGroup                                                                  Page 118

-------
EPA: Results of Benchmarking Study
      >  Be aware of firewalls that will prevent end users from accessing the Internet
         (especially in the government).
      >  Do your best to make the program interactive.
      >  Package materials so they are inviting.
      >  Don't send anything you don't want copied (e.g., CDs).
      >  It is easier to support class cohorts than individual students.
      >  For every 1 hour of instruction expect 3 hours of post-production time.
HayGroup                                                                 Page 119

-------
EPA: Results of Benchmarking Study
     Site 4: GMAC Commercial Mortgage (GMAC) Staff Development Division
                                 Site Summary

GMAC Commercial Mortgage (GMAC)  began  developing  and providing  distance
learning programs in February, 2002 and considers itself "young at what we do".  The
mission of the Division  is to provide programs that focus on  1)  servicing-based
knowledge;  2)  the  mortgage  business in  general; and 3) personal  and  leadership
development.  GMAC sees it role as not only teaching/training, but as communicating
information to employees.  The teaching it provides is vendor based, GMAC does not
have a large staff.

With  a staff of approximately three  professionals and one student intern, the  Staff
Development Division  uses  the  following  training  delivery  methods:    Intranet,
Videoconferencing, videotapes, and classroom-based courses. The Division is looking to
add "the wow factor" to its courses; GMAC agreed to participate in the benchmarking
meeting partly as a way  to learn  from us  what others  are doing in  distance learning.
Examples of programs that the Division has provided include book studies, teachings by
ratings agencies (e.g., Moodys), and leadership development.  Students can earn credit
for some GMAC courses.

Courses are delivered in both synchronous and asynchronous formats using a variety of
media including the Web and videoconference.  GMAC conducts  informal needs
assessments to determine  individual course  needs as well as general training needs.  All
courses are evaluated and designed to promote interactions between participants.	
Needs Assessment

GMAC conducts some informal needs assessments to determine what courses are needed
and what people like. This is accomplished primarily through talking with employees to
determine the types of training they need/the types of training courses GMAC should
deliver.  The Division has done some needs assessment surveys in the past to determine
what types of delivery methods students prefer.

Course Content, Design, and Training Facilities

The  GMAC Staff  Development  Division  is  relatively young  and  has done  some
impressive work with a small staff.  It delivers between 20 and 30 training sessions each
year  and  use  Web-based,  videoconferencing,  videotapes,  and  classroom-based
technologies including the following:
      >   Intranet,
      >   Live streaming,
      >   Web ex,
      >   Windows Media Encode,
      >   CD-Rom/DVD (non-interactive), and
      >   Videotape (GMAC has a large library of videotapes).
HayGroup                                                                 Page 120

-------
EPA: Results of Benchmarking Study
Course  development is  primarily initiated by  internal  GMAC when there is a stated
interest  in a specific topic.  GMAC then either contracts with an  outside subject matter
expert (e.g., professor) or uses internal staff to develop the course.  Courses have been
conducted  as  panel discussions, small group  studies, and  traditional presentations.
Previous courses have covered topics such as real estate, property management, asset
management, commercial mortgages, and leadership development.

Delivery Methods and Participant Interaction

GMAC  uses two primary delivery media: 1) Web, and 2) Videoconferencing. Most of its
programs are broadcast live and taped (audio and video) for later viewing using Windows
Media Encoder. GMAC is moving towards doing more videotaping of live sessions
because it  is finding  that live audiences  are often smaller  than anticipated.   Some
programs are conducted  using Webex; however, GMAC has had trouble recording both
audio and video with  Webex.  Finally, GMAC  has the capacity to record programs to
CD-Rom/DVD for delivery to  individuals  who were  unable to  attend  the original
broadcast. Each delivery method and methods for interaction are described below.

Web. GMAC  uses its Intranet, rather than the Internet, to broadcast courses using Web-
based technology.  GMAC does not have interactive Web-based (e.g., Webcast) courses,
but rather use  the  Intranet as a means to stream live videos (one-way) to participants.
Although  streaming occurs only one-way, GMAC has developed some  methods  for
participant interaction that include:
      >  E-mailing questions (that the Division monitors)
      >  Calling  in questions that are answered live (via an open telephone line)
HayGroup                                                                 Page 121

-------
EPA: Results of Benchmarking Study
Videoconferencing.    Videoconferencing provides  the  best  opportunity  for  direct
interactions between instructors and students. Because both parties are able to hear and
see each other, interactions primarily take place in the form of a continuous  discussion.
Communication can occur either before or during the broadcast. Typically a moderator is
available to take and filter questions before giving them to the instructor.  Methods for
incorporating interaction include:
       >  Call-in questions
       >  Fax-in questions

Course Evaluations

Some level 1  evaluations are conducted, but GMAC conducts relatively few evaluations.
The Division has the philosophy that "if they find value, they will come." And, because
there is continued enrollment, GMAC believes that the teachings are successful.  When
the Division does survey, it often uses online survey tools such as Zoomerang.

Course Examinations

GMAC does  conduct some quizzes; however, there is little focus on examinations.  To
prevent participants from simply viewing part of a course the Division has programmed,
asynchronous sessions are used so that participants cannot fast-forward through the entire
session. Once a student completes the session, they are provided a code and asked  to log-
in the code as proof they completed the course.
HayGroup                                                                 Page 122

-------
EPA: Results of Benchmarking Study
                       Site 5: SAS Education to Customers
Site Summary

The primary mission of the SAS Education to Customers (SAS) division is to provide
technical training to  SAS  customers.  Approximately 90% of the training  provided is
task-oriented (how to use SAS products) with the remaining 10% focused on technical
overviews (why a product is used), often as a precursor to the task-oriented training. All
the training provided is fee-based; however, the SAS Education to Customers division is
not a profit center.  It seeks to cover costs while supporting SAS customers to ensure they
are able to effectively use, and continue to purchase, SAS products.

SAS provides classroom-based, live Web, and self-paced (Internet, CD  Rom) training.
Classroom-based training accounts for about 80% of all  training and typically occurs at
SAS'  28 training facilities.  Thus far in 2004, SAS has provided 120 unique training
episodes (courses) and 250 training sessions to about 1500 customers.

SAS conducts an annual needs assessment survey to  determine what courses to  deliver in
the upcoming year.  The survey consists  of about 400 items designed to determine
whether and how customers use software (importance), when they last used the software
(recency), and how often they use the software (frequency).  In addition, SAS monitors
product sales and other data to determine courses that customers are likely to need.  All
courses are evaluated using "smile sheets" and require an average rating  of 3.5 out of 4.
SAS also monitors the re-buy rates and believes that customers will not re-buy if they
don't feel that training is effective.

As a  policy,  SAS will only  conduct certification  programs through classroom-based
training.  They have determined that classroom-based training is the only way  to ensure
that the person taking the certification test is the person signed up for the course.	
Needs Assessment

SAS conducts an annual needs assessment survey to assess what courses to deliver in the
upcoming year. The survey consists of about 400 items designed to determine whether
and  how  customers use  software (importance),  when they last used the software
(recency), and how often they use the software (frequency).  In addition, SAS monitors
product sales and other data to determine courses that customers are likely to need.

Course Content, Design, and Delivery Facilities

SAS is primarily a software development company and SAS is responsible for providing
technical training to SAS customers. Therefore, SAS develops courses and uses training
delivery methods  that are  customer-driven.   The technology utilized for these courses
includes:
       >   Internet (MS Live Meeting),
       >   CD-Rom/DVD,
HayGroup                                                                 Page 123

-------
EPA: Results of Benchmarking Study
       >  Videotape,
       >  Print-based self learning, and
       >  Classroom-based technologies.

SAS maintains 28 classroom-based training facilities throughout the United States where
customers and  SAS instructors can  meet  for  face-to-face  trainings  (about 30% of
classroom-based instruction is conducted at customer sites).  All live Web and self-paced
training  courses  are developed  and  delivered  using facilities maintained at SAS
headquarters in Gary, NC.  SAS has 77 instructors, 20 of which are certified by SAS to
teach one or more live Web courses.  Because the nature of teaching is different for live
Web and classroom-based courses,  SAS requires instructors  to take courses on providing
training  over  the  Web  before they  are  allowed  to provide  live Web instruction.
Instructors must also be certified in a course before they are allowed to teach it.

Course development is primarily initiated after review of customer needs as determined
by an annual needs assessment survey and software sales and usage data. Because SAS'
existence depends on expanding and retaining its customer  base, courses are developed
that  provide existing and potential customers the know-how to effectively use SAS
products.

SAS has developed both absolute rules and rules-of-thumb that must be considered when
developing any course.   These rules were developed because SAS acknowledged that
people are likely to physiologically tune-out during training.  Four absolute rules have
been developed for courses that are not classroom-based. They include:
   1)  Participants must be given the opportunity to interact  at least every 10 minutes
       >  Ask questions about things just completed
       >  Ask open-ended questions about general training subjects
       >  Provide and complete short quizzes
   2)  Instructors must have specific training
       >  Web delivery is different than classroom
   3)  Instructors must have two rehearsals before they can teach a course
       >  Instructors must demonstrate they can effectively  instruct over the web
       >  Instructors must show they understand the material
   4)  Moderators must be present for each course session
       >  Each course has at least one instructor, moderator, and technical support staff

Some additional rules-of-thumb that SAS considers when designing a course are:
   1)  Course design and delivery depend on  complexity of the material, the need for
       interactivity, validation for certification purposes, and sensitivity of issues
   2)  No course session should be longer than a half-day
   3)  Classroom-based courses are best for interactions
       >  Best for validation
       >  Best for sensitive issues
   4)  Self-paced is best for informational courses
       >  Easy to scale
       >  Cheap to develop and deploy
HayGroup                                                                 Page 124

-------
EPA: Results of Benchmarking Study
       >  HTML has low standards - doesn't need much bandwidth and can be recorded
          easily to CD-Rom/DVD

SAS currently uses MS Live Meeting and Place Where, and is considering using Webex,
to deliver live Web training.  They  believe that content delivery methods are more
important than tool selection.  SAS makes tool selection decisions based a lot on cost and
whether  the  vendor will  be around  in a  few years,  with a few assumptions.  The
assumptions are that each tool has:
       >  Application sharing
       >  PowerPoint capabilities
       >  Chat capabilities
       >  Registration capabilities

Delivery Methods and Participant Interaction

SAS employs three primary delivery methods:  1) Classroom-based, 2) Live Web, and 3)
Self-paced.   Each delivery  method provides  specific  opportunities for  participant
interaction. The delivery methods and strategies for interaction are described below.

Classroom-based.  Classroom-based courses account for about 82% of SAS courses and
are delivered at either one of SAS' 28 training centers or at a customer site.  Training is
provided by  one  of SAS'   77  instructors.   The  courses are designed for  optimal
interactivity using the latest in face-to-face training skills.

Live Web.  Live Web courses account for about 10% of SAS courses. When  SAS began
delivering live Web courses (June, 2001) they believed that a key to live Web success
would be providing students the ability to interact, without  stopping training.  To do this,
SAS incorporated the following:
       >  Text questioning - Participants can e-mail questions during the training
       >  Phone questions - All questions come in privately before decisions are made
          about whether the question  should "go live"
       >  Participants can "raise their hand" by indicating they have  a question - The
          delivery  tool indicates  a potential question by changing  the participant's
          "seat" color on the online screen visible to the moderator and instructor

All questions come in privately and are captured so they may be sent to participants as a
training supplement.  A moderator is available during every course session to determine
which questions should be addressed during the session.   Before any session begins,
participants are e-mailed rules about live Web course etiquette. Participants are reminded
of the etiquette during the course and  phone lines can be muted  by the moderator if one
or more participants fails to use the proper etiquette.

Self-paced.  Self-paced courses account for about 8% of SAS courses  and are delivered
primarily over the  Internet.  Customers may also request CD-Rom versions that  can be
distributed to participants as a CD or may be loaded onto a customer's Intranet site.
HayGroup                                                                 Page 125

-------
EPA: Results of Benchmarking Study
Interactions in self-paced  courses are focused  on participant interactions with  the
technology, not with other participants or the instructor.  Examples include:
       >   Short quizzes that participants must get right before continuing the training
       >   Instructions to go practice using the software before returning to training
       >   Questions that direct participants  down different paths  depending on their
           responses

Course Evaluations

Evaluations occur  immediately following a course and consist  primarily  of  "smile
sheets".  Courses that do not get an average rating of at least 3.5 out of 4 are  further
examined to determine why the ratings are low.  SAS will directly contact customers to
inquire further as to why courses were reviewed less favorably than expected.  SAS also
monitors the re-buy rate of trainings to help determine whether customers are willing to
continue to participate in training.  It is believed that customers will not continue to buy
training if they feel it is not worthwhile. Therefore, SAS feels that a re-buy rate of 60%
over a period of 18 months indicates that customers have evaluated the course positively
enough to continue purchasing it.

SAS also uses short quizzes during self-paced  courses primarily as an interaction method.
Quizzes occur every few slides to ensure that  participants are interacting with the  media;
they are rarely used as a measure of whether participants are learning. However, courses
can be designed so that participants are blocked from continuing with the  training until
they provide correct responses to the quizzes.

Lesson Learned/Future Directions

       >   There is a definite need for training that is not classroom-based
           o   Willingness to travel reduced dramatically after September 11, 2001
           o   Travel more  than 100 miles dropped by 50%
       >   It took 12-18 months for constituents to truly buy-into web training
       >   It takes time to develop courses
           o   Classroom: 12 hours to develop 1 hour of delivery content
           o   Live Web: 20 hours to develop  1 hour of delivery content
           o   Self-paced: 60 hours to develop 1 hour of delivery content
       >   There will always be a need for classroom-based training
       >   Make it east for participants
           o   SAS will  send  CDs with downloading  materials if participants  cannot
              download from the web (Flash,  Webex, etc.)
       >   HTML has low standards
           o   It takes bandwidth to add video, audio, etc.
           o   Can store hundreds of hours on CD
           o   Streaming video is tough to get on CD - about lOmegs/minute
       >   Transition from classroom-based to Web-based courses takes time
           o   Most of the time is taken up redesigning interactivity
           o   Add 30-45 days if development team has never migrated before
HayGroup                                                                 Page 126

-------
EPA: Results of Benchmarking Study
          o   For each project include a project manager, instructional designed, subject
              matter expert, technical review team, editor, and producer
       >  SAS's average class size is about 12 people
          o   They try to limit most classes to 20 people; highly technical class to 10
              people; and very general classes to 40 people
HayGroup                                                                  Page 127

-------
EPA: Results of Benchmarking Study
                            Appendix C

             Summary of Quick Fix Recommendations for
                Improving theAPTI Training Program
HayGroup                                                     Page 128

-------
EPA: Results of Benchmarking Study
Below,  we present an overview  of all the  "quick fix" suggestions that are presented
throughout this report.  Please refer to the main body of the report for a more detailed
description of each "quick fix".

Quick Fix Summary
Needs Assessment Process

Course Design and
Updating Process
> Perform a high-level review of current and planned courses/offerings to
make "quick" improvements to courses that meet APTFs core mission
and fulfill the immediate needs of air professionals and other key
customers. For example, APTI does not want to waste valuable resources
updating a course if it is not viewed as currently filling an important need.
This high-level review may be accomplished through meetings with APTI
staff, informal discussions with the JTC and representatives from state,
local, and tribal agencies, and reviews of prior needs assessments and
training evaluations.
This review will provide input into decisions about those
courses/offerings that should be:
• Updated first
• Enhanced
• Transferred to a new medium or technology
• Shelved
• Retooled

> Take inventory of the roles and skill sets of the current training team to
determine where gaps exist. For example, if the team is missing a critical
role (e.g., instructional designer), APTI can begin to search for an
individual (within or outside EPA) to fill this role.
• Consider sending team members to training or hiring contractors
to fill skill gaps
• Consider hiring or contracting with a script writer to work with
subject matter experts to develop live broadcasts
> Provide critical information to course designers early in the process,
including objectives of the course/offering, intended delivery methods,
audience characteristics, technological constraints, and timelines and
resources. This will ensure that courses/offerings that are developed meet
stated objectives and maximize learning for air professionals and other
customers.
> Define the frequency and types of interactions that should be incorporated
into all courses/offerings. For example, for live Web courses, the
standards may be to ensure that there is an opportunity for interaction at
least every 10-15 minutes. For self-paced Web courses, the standard may
be to provide an opportunity for interaction (e.g., through a short quiz or
simulation) every 3 to 5 screens.
HayGroup
Page 129

-------
      EPA: Results of Benchmarking Study
Course Design and
Updating Process
(continued)
>   Require rehearsals for all live broadcasts, including full script and use of
    teleprompters and other technology.  This will ensure that the instructor
    presents the material in an engaging way to the audience, and  final
    glitches and inaccuracies are caught before the broadcast is delivered live.
                               >  Conduct a quality review of all current course materials before they are
                                  distributed to trainees to check for:
                                     •   Typos
                                     •   Incorrect information
                                     •   Missing, upside down, or out-of-order pages
                                     •   Whether the right materials are sent on time, and with the
                                          right courses.
Training Delivery Methods
>   Examine current  opportunities for  interaction  used  during  satellite
    broadcasts.    Consider the  effectiveness of  each type  of interaction
    opportunity, and why some methods are not as effective as others, using
    the following criteria:
       •   Is there  an  opportunity for  interaction  at least  every  10-15
          minutes?
       •   Do methods for interaction interrupt the flow of the session?
       •   Do trainees take advantage of opportunities for interaction?
       •   Do trainees receive answers to all questions that are not addressed
          during the broadcast?
       •   Can trainees hear and respond to others' questions?
    Ensure that these opportunities for interaction are provided consistently in
    all distance  learning courses.
                               >  Enhance methods of interaction used in distance learning courses.
                                     •   Publicize opportunities to interact - publicize contact information,
                                         instructions, and benefits of interacting
                                     •   Establish proper etiquette and communicate prior to each session
                                     •   Use screeners and technology  monitors  to  review  incoming
                                         questions - provide at least one individual other than the instructor
                                         to accept, review, and facilitate questions
                                     •   Develop discussion questions prior to the broadcast - develop a set
                                         of 5-10 questions to use when participants  are hesitant to interact
                                     •   Follow-up on unanswered questions  -  provide trainees  with
                                         answers to all questions asked regardless if they  were answered
                                         during the broadcast
                               >   Select instructors with subject matter expertise, applied experience in the
                                   field, and experience in the medium of course delivery.
                                  Consider using professional speakers to  deliver training in conjunction
                                  with subject matter experts who can answer specific questions raised by
                                  the air professionals.
      HayGroup
                                                         Page 130

-------
      EPA: Results of Benchmarking Study
Training Delivery Methods
(continued)
>   Determine new ways to  provide  opportunities for interaction  during
    broadcasts.
       •  Use chat rooms or Web boards to facilitate  interactions.  These
          should be used to post questions, send course updates, and foster
          e-mail discussions.  Provide each trainee with the location of and
          instructions for using the chat room or Web board.  Instructors
          should also review the chat room or Web board and be encouraged
          to use these as a medium to interact with trainees.
       •  Consider facilitating the assembly of cohorts  of 5 to  7 trainees to
          complete the training program as a group, rather than  individually.
          Participants may complete self-paced modules  at their own  pace
          (within a specified time frame),  and then access  chat rooms or
          participate  in  conference calls to discuss  the course and ask
          questions.
                              >  Provide instructors with standard guidelines for training in a particular
                                  medium, including:
                                     •   Where to stand and how much to move during training
                                     •   When and how to look into the camera
                                     •   How to use basic technologies
                                     •   How to encourage interactions
                                     •   How to deal with difficult students
                                     •   Tips for holding the attention of students
Training Evaluation
Process
>   Improve level  1 evaluation instruments to enhance an understanding of
    why  the  training  course/offering was effective,  whether the delivery
    method is the best for delivering training, and how to improve training
    courses/offerings.  At a minimum, ask some open-ended questions such
    as:
       •   What about the course/offering did you like best?
       •   What about the course/offering would you change?
       •   How would you improve the current training course/offering?
       •   Were there enough opportunities  for interaction?  What are your
          recommendations for enhancing interactions?
                              >  Evaluate all courses/offerings, regardless of the delivery method.  Give
                                  participants the opportunity to complete a course evaluation on-line after
                                  they finish a self-paced Web course. Focus the evaluation on the content;
                                  how it  was presented; the effectiveness of  interactions; the ease of
                                  accessing the course;  and so forth.  The evaluation should also  include
                                  overall questions about the effectiveness of the Web course, whether it
                                  engages air professionals, and whether the Web is the best method for
                                  teaching this course.
                              >  Follow-up on less favorable evaluations. For example, if participants rate
                                  a course below the minimum rating,  APTI may contact participants to
                                  obtain insight into why the course was rated less favorably.
      HayGroup
                                                         Page 131

-------