Results of Benchmarking Study
for
Environmental Protection Agency
Air Pollution Training Institute
FINAL REPORT
January 7, 2005
Submitted by:
Hay Group
4301 N. Fairfax Drive, Suite 500
Arlington, VA 22203
(703) 841-3100
roup
-------
X, UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
o vx-ivy * RESEARCH TRIANGLE PARK, NC 27711
%;
February 7, 2005
OFFICE OF
AIR QUALITY PLANNING
AND STANDARDS
Dear STAPPA/ALAPCO Joint Training Committee:
The EPA's Office of Air Quality Planning and Standards has conducted a benchmark
study of its training program. This study is part of our effort to improve the overall value and
effectiveness of the program. The report entitled "Results of the Benchmarking Study" offers
insights as to what others in the industry of training are doing and recommends a number of
quick fixes and long-term changes to the current training program. At the upcoming
STAPPA/ALAPCO Joint Training Committee Meeting of March 2005 in Florida, we will use
the "Results of the Benchmarking Study" as the framework for further discussions on the training
program. We are looking forward to your comments and suggestions to help us focus and
strengthen the program.
Sinaerely,
Harnett
Director
Information Transfer and Program
Integration Division
Enclosure
-------
TABLE OF CONTENTS
I. INTRODUCTION. 1
Background 1
Methodology 1
Report Overview 4
II. OPTIONS FOR IMPROVING APTI's NEEDS ASSESSMENT PROCESS 5
///. OPTIONS FOR IMPROVING APTI's COURSE DESIGN AND UPDATING PROCESS 9
IV. OPTIONS FOR IMPROVING APTI's TRAINING DELIVERY METHODS 16
V. OPTIONS FOR IMPROVING APTI's TRAINING EVALUATION PROCESS 24
VI. NEXT STEPS. 28
APPENDIX A Task 1 Report: Evaluation of Current Air Polution Training Program.. 33
APPENDIX B Task 2 Report: Best Practices of Leading Training Programs 60
APPENDIX c Summary of Quick Fix Recommendations. 128
HayGroup
-------
EPA: Results of Benchmarking Study
/. Introduction
Background
In May 2004, the Education and Outreach Group (EOG) of the Environmental Protection
Agency (EPA) contracted with the Hay Group to conduct a benchmarking study of its Air
Pollution Training program. The ultimate goal of the project was to provide options for
improving the current Air Pollution Training program, and ultimately, EOG's business
performance. The following three tasks were aimed at accomplishing this goal:
> Evaluate the current Air Pollution Training program (Task 1)
> Identify best practices of leading training programs (Task 2)
> Develop a plan of action for improving the current Air Pollution Training
program (Task 3)
This final report provides options for improving the current Air Pollution Training
program and next steps for doing so. Throughout this report, we present our specific
conclusions and recommendations regarding the direction that we believe the Air
Pollution Training Institute (APTI) should take to improve its training program.
The options and recommendations described in this report are based on our evaluation of
the current Air Pollution Training program and the best practices of training industry
leaders that we identified. We present options for narrowing the gap between the current
APTI training program and industry leading training programs.
Methodology
This section provides a high-level summary of the methodology used during the
benchmarking study.
Review of Current APTI Training Program
The first phase in the benchmarking study was to evaluate the current Air Pollution
Training program. Hay interviewed a total of 22 individuals (in person or by telephone)
who have a good understanding of the current training program, its objectives and desired
results, its future direction, and overall effectiveness. Additionally, we spoke to several
other individuals in group settings about the Air Pollution Training program. The
following types of individuals were interviewed:
• Members of the STAPPA/ALAPCO Joint Training Committee (JTC)
• Members of the MARAMA Regional Consortium
• Individuals who run area training centers
• APTI course instructors
• Supervisors in state and local agencies who have sent employees to APTI training
courses
HayGroup Page 1
-------
EPA: Results of Benchmarking Study
• Contractors who design APTI course content and conduct the training needs
assessment
• APTI management and staff
The interviews were aimed at gaining an understanding of the practices, procedures and
processes currently used by APTI to design and deliver air pollution training, and
determining the effectiveness of the APTI training program in meeting the needs of air
professionals and other stakeholders. We focused on evaluating four key aspects of the
current training program: 1) needs assessment process; 2) course design and content; 3)
training delivery methods; and 4) training evaluation process.
In addition to conducting subject matter expert interviews, Hay reviewed several
documents, Web sites, and APTI training materials as part of the evaluation of the current
APTI training program. More specifically, we reviewed the following:
• APTFs Web site (e.g., course schedules, training providers, APTFs mission,
course registration)
• Site Coordinators Resource Center Web site
• EOG FY03 Highlights and FY04 Midyear Accomplishments
• Training Needs Assessment Survey and results
• Training evaluation instruments and results
• Training materials (self-instructional workbooks, Web-based training courses,
satellite broadcasts)
• Statistics on training course attendance and certificates issued
Hay summarized the information collected during the subject matter interviews and
document/Web site reviews and provided an evaluation of the current APTI training
program. The findings and conclusions are reported in the Task 1 Report: Evaluation of
Current Air Pollution Training Program, which can be found in Appendix A.
Identification of Best Practices of Industry Leaders in Training
The second phase in the benchmarking study was to conduct site visits and a literature
review to identify best practices of industry leaders in training, particularly in the area of
distance learning. The best practices were identified based on site visits with
organizations with innovative training programs, as well as a literature review of current
training trends and best practices.
Site Visits. Hay conducted site visits with five organizations recognized as industry
leaders in technical training and distance learning. We identified these organizations
based on awards (e.g., Training Magazine's Top 100 list, American Society for Training
and Development BEST Award, Government e-learning award), accolades, participation
in other training benchmarking studies (e.g., Society for Human Resource Management
Consortium Benchmarking Study on Training and Development), and expert
recommendations. We particularly looked for organizations that use innovative e-
learning and distance learning training delivery methods. Table 1 shows the five
HayGroup Page 2
-------
EPA: Results of Benchmarking Study
organizations that agreed to participate in the benchmarking study and our rationale for
selecting each of them as a benchmarking partner.
Table 1: Participating Organizations in the Site Visits
Organization
Occupational Safety
and Health
Administration
(OSHA), Office of
Training and Education
Center for Disease
Control (CDC), Public
Health Training
Network (PHTN)
Georgia Tech
University, Distance
Learning and
Professional Education
Department (DLPE)
SAS Institute (SAS)
GMAC Commercial
Mortgage (GMAC),
Staff Development
Division
Rationale for Selection as a Benchmarking Partner
> Provides technical training to a large, geographically-dispersed
population
> Blends Web-based training and live satellite broadcasts with more
traditional classroom instruction (uses innovative distance learning
training media)
> Provides technical training to a large, geographically-dispersed
population
> Provides diverse training delivery methods, including distance learning
> Has state-of-the-art training facilities
> Trains professionals in engineering, business, and other hard sciences
> Recognized as providing an exceptional distance learning program and
successfully incorporating an interactive component into distance
learning
> Has state-of-the-art training facilities
> Provides extensive technical training to customers around the world
> Blends e-learning with more traditional classroom training (uses
innovative e-learning training media)
> Delivers training primarily via e-learning methodologies, including
videoconferencing, live Webcasts, videotapes and C-ROMs/DVDs
The site visits focused on identifying best practices around:
> The needs assessment process
> Course design and content
> Training delivery methods, with a special emphasis on distance learning
methods
> Strategies for incorporating an interactive component into distance learning
> Training evaluation process
During site visits, we reviewed training program documentation and materials, observed
training facilities and training programs (e.g., a live Webcast, an interactive Web-based
course), and conducted interviews with members of the training group. Each site visit
lasted two to six hours.
HayGroup
Page3
-------
EPA: Results of Benchmarking Study
Literature Review. In addition to conducting site visits, Hay reviewed several reports,
articles, books and Web sites to identify best practices in e-learning and distance
learning. The primary documents/Web sites that were reviewed are identified in the Task
2 Report: Best Practices of Leading Training Programs., which can be found in Appendix
B.
Summary of "Best Practices". Hay summarized our findings from the site visits and
literature review to identify trends and best practices of industry leaders in training.
These findings are also presented in Appendix B.
Discussions with APTI about Task 2 Report. After completing the Task 2 report, we
met with APTI management and staff to obtain their input into those best practices that
are most cost effective and feasible to implement at APTI. We facilitated a discussion
with APTI around the following:
> Extent to which the best practices can be incorporated into APTFs culture,
processes, and procedures
> Whether APTI has the resources (e.g., dollars, staff) to implement the best
practices
> Barriers that may impede successful implementation of the needed changes to
the program
Information collected during these meetings served as additional input for determining
the options for improving APTI's training program, as well as next steps for moving
APTI forward.
Report Overview
In this report, we highlight where we see the biggest gaps between APTI's current
training program and the ideal training program (one that incorporates best practices of
industry leaders). We also provide options and recommendations for making APTI more
successful, and next steps for doing so. We provide options for improving the following
aspects of APTI's training program:
> Training Needs Assessment Process
> Training Design and Content
> Training Delivery Methods
> Training Evaluation Process
Throughout this report, we discuss options that we believe will have the biggest impact
on the overall effectiveness of APTI. We recognize that some of these options are long-
term. However, we also present some "quick fix" suggestions or "low hanging fruit" for
immediately improving the Air Pollution Training program. A summary of all "quick
fix" suggestions can be found in Appendix C.
HayGroup Page 4
-------
EPA: Results of Benchmarking Study
//. Options for Improving APTFs Needs Assessment Process
The benchmarking study revealed that APTFs current needs assessment process does not
provide sufficient or the "right" types of information to make decisions about the
direction APTI should take to meet the needs of its most critical customers. To become a
more effective training program, APTI should expand its current needs assessment
process beyond simply predicting future attendance at current classroom courses, and
incorporate a competency/skill-based assessment in this process.
Table 2 highlights the current state of APTI's needs assessment process compared to
what industry leaders are doing in this area, and presents options for creating a more
value-added training needs assessment process.
Table 2: Gaps in Needs Assessment Process
Current APTI Training Program
Industry Leader Training Program
Options for Making APTI an Industry
Leading Training Program
Needs assessment primarily focuses on
determining whether air professionals
are likely to attend existing APTI
classroom courses. There is no
competency/skill component of this
assessment.
There has not been a recent broad-
based needs assessment that takes a
strategic look at whether current
courses/offerings and training delivery
methods meet the needs of air
professional and other key customers.
Needs assessment data are not
consistently used to develop and update
courses and offerings.
A competency/skill assessment is
included as part of the needs
assessment process. That is, methods
are used to identify the
competencies/skills required for
success on the job and gaps in the
competency/skill level of the trainee
population.
The needs assessment process is a
critical source of information for
making decisions about training content
and delivery methods. For example,
whether current courses should be
updated or eliminated or new courses
should be designed.
Decisions about courses and offerings
are communicated to the trainee
population.
As a first step, conduct a broad-based
needs assessment to determine the future
role and scope of APTI, including
courses/offerings and delivery methods
most needed by air professionals and
other customers. This assessment should
include a competency/skill component.
Continue to conduct needs assessments
on a regular basis. Make decisions about
the frequency and breadth of future needs
assessments.
Use needs assessment results to refine
APTI's training program (e.g., decisions
about which courses to maintain or
update, new courses to design, delivery
methods best suited for course content),
and communicate key decisions to the air
pollution community.
Below we further discuss our recommended options for enhancing the needs assessment
process at APTI.
1. Conduct a Broad-based Needs Assessment to Redefine APTI's Role and
Scope
We recommend that APTI conduct a broad-based needs assessment in the near future to
determine the extent to which it is providing the right mix of courses and offerings to
meet the needs of its most critical customers. This is particularly important because we
found that some key customers/stakeholders who were interviewed in Task 1 do not
believe that APTI is currently meeting the needs of state and local agencies.
HayGroup
Page 5
-------
EPA: Results of Benchmarking Study
This needs assessment will help to define APTFs key customer contingency, what types
of services (e.g., outreach, technical training) APTI should be providing to its customers,
what courses should be delivered, and how courses should be delivered (e.g., live Web,
satellite, classroom). Ultimately, this needs assessment will help APTI to define its
mission and scope so that it can be successful at meeting the most critical needs of its
customers. This mission should be communicated to all APTI staff, as well as external
customers/stakeholders, so that everyone has a good understanding of APTI's role and
scope.
The current needs assessment process focuses on identifying those current classroom
courses that air professionals are likely to attend in the future, and solicits input about
potential new training topics. While this approach provides some useful information for
planning future APTI courses/offerings, it does not provide direct insight into whether the
current courses are teaching the skills/competencies required by air professionals (and
other customers), or whether new courses should be developed to meet air professional
training needs. Additionally, the current needs assessment provides limited information
for determining the best way to deliver training to air professionals.
The broad-based needs assessment should solicit input from key APTI
customers/stakeholders on:
> The competencies/skills required for air professionals (and other key
customers) to be successful now and in the future
> The current competency/skill level of air professionals and key gaps from
those required for success
> Whether critical competencies/skills can be obtained through other sources
than APTI (e.g., universities) or APTI courses/offerings should focus on
developing these competencies/skills
> The most effective ways to deliver training to air professionals (based on
considerations such as cost, convenience, learning ability/style)
Information collected from the needs assessment process should be used to review
current APTI courses and offerings to determine which should be retained as is; which
should be updated; which should be eliminated; and what new courses/offerings should
be developed. Additionally, this information will help APTI make decisions about the
best method for delivering different types of courses/offerings to air professionals (e.g.,
should APTI make more use of live Web-based training?). Finally, and most
importantly, this broad-based needs assessment will provide APTI with the information it
needs to redefine its vision, role, and scope so it is meeting the needs of its most critical
customers. That is, what types of training and outreach should APTI be delivering and to
whom? By limiting its scope to those courses/offerings that are most critical for success
on the job, APTI will be able to focus its limited resources on ensuring that these
courses/offerings are of the highest quality and meeting the needs of air professionals and
other key customers.
HayGroup Page 6
-------
EPA: Results of Benchmarking Study
Once APTI conducts the needs assessment and can prioritize the courses and offerings it
provides, it will be easier to implement many of the other recommendations that will be
discussed in this report. For example, by reducing the number of courses/offerings APTI
provides to those that are most needed by air professionals, it will be less challenging to
conduct systematic reviews and updates on a periodic basis of all APTI courses/offerings.
Tips for Making the Needs Assessment a Success
• Create a detailed plan for conducting the needs assessment, including the study's
objectives, timelines, individuals to participate in the assessment, types of data
collection methods, how data will be used, who will be accountable for conducting
the needs assessment, etc.
• Publicize the needs assessment throughout the air pollution community (through the
Internet, JTC and site coordinators, newsletters, etc.). Communicate that APTI is
making an effort to identify the needs of its most critical customers and redefine the
Air Pollution Training program to meet these needs. This will also increase
participation in the needs assessment process now and in the future.
• Collect data from multiple sources, including subject matter experts, current and
potential trainees, supervisors of trainees, air directors, course instructors and
developers, among others. Sources should have insight into the competencies/skills
required for the success of air professionals (and other key customers), the current
competency/skill level in the air professional population, types of training available
outside of APTI, and so forth.
• Consider using multiple data collection methods, such as reviews of position
descriptions, SME panels, interviews, and short, targeted surveys with a small sample
of air professionals and other customers/stakeholders.
2. Continue to Conduct Regular Needs Assessments
Recognizing that the broad-based needs assessment is a one-time or very infrequent
activity, APTI (in conjunction with the JTC) should continue to conduct regular (every 1-
3 years) needs assessments to keep a pulse on whether it is meeting air professional and
other customers' needs and whether any new training needs arise. This will require
making modifications to the current needs assessment instrument (e.g., to include
competencies/skills; the desire for different training delivery methods). The broad-based
needs assessment process may help shape the future needs assessment instrument. It
should be noted that a comprehensive competency/skill-based needs assessment may
need to be conducted on a less frequent basis, potentially every three to five years.
HayGroup Page 7
-------
EPA: Results of Benchmarking Study
I.
We recognize that completing a broad-based needs assessment will take approximately five to six
months. In the short-term, APTI should perform a high-level review of its current and planned
courses/offerings to make "quick" improvements. This could be accomplished through meetings
with APTI staff, informal discussions with the JTC and representatives from state, local, and
tribal agencies, and reviews of prior needs assessments and training evaluations. Information
collected during this initial review could be used to make immediate improvements to APTFs
training program. For example, this review will provide input into decisions about those
courses/offerings that should be:
> Updated first (highest priority for updating)
> Enhanced through incorporating more opportunities for interaction, more visually
stimulating graphics, and so forth
> Transferred to a new medium or technology
> Shelved because the material is no longer relevant or up-to-date
> Retooled into a blended delivery approach to minimize the amount of time air
professionals have to spend away from the job
In the short-term, APTI should focus on improving those courses/offerings that meet its core
mission and fulfill the immediate needs of air professionals and other key customers. For
example, APTI does not want to waste valuable resources updating a course if it is not viewed as
currently filling an important need.
Note: This initial "quick" review has a more tactical focus - making immediate changes and
improvements to APTFs training program. The broad-based needs assessment can help validate
the information collected during this initial review, and strategically shape APTFs future mission
and scope.
HayGroup
PageS
-------
EPA: Results of Benchmarking Study
///. Options for Improving APTFs Course Design and Updating Process
Based on the results of the benchmarking study, it appears that the APTI training group
may not encompass all of the key roles required to effectively design and update training
and outreach courses/offerings. The results of a future broad-based needs assessment
will provide APTI with critical information for determining whether its current training
group should be supplemented with other key roles in order to design the types of
courses/offering needed by air professionals and other customers.
Additionally, APTI does not currently apply consistent course design standards and
procedures and does not incorporate many of the industry best practices (e.g., use script
writers for live broadcasts; require rehearsals for all live broadcasts) into its course design
process. Finally, there is no systematic process or timeline for reviewing and updating all
existing courses/offerings, regardless of the media in which they are delivered.
Table 3 highlights the current state of APTI's training design/updating process compared
to what industry leaders are doing in this area, and presents options for creating a more
effective course design and updating process.
Table 3: Gaps in Training Design and Updating Process
Current APTI Training Program
Industry Leader Training Program
Options for Making APTI an
Industry Leading Training Program
The APTI training group may not
encompass all of the key roles required
for course design.
Course development is the responsibility
of a team made up of key roles such as
subject matter experts, script writers,
instructional design experts, technical
experts, editors, and project managers.
Examine the competency/skill level of
the current APTI team and bring in
individuals to perform key roles that
are not filled.
Use contractors to fill key roles, when
needed.
APTI does not use consistent standards
and processes to design its courses and
offerings. Rather, the design usually
follows the structure created by the
individual developing the
course/offering.
There is no systematic process for
updating APTI courses/offerings on a
regular basis.
Some APTI courses are viewed as out-
of-date or of poor quality.
There are consistent standards and
processes for designing courses. For
example:
• Collaboration between subject
matter experts and script writers
to design live broadcasts
• Application of specific criteria
for determining delivery media
• Minimum standards for
frequency of interactions during
distance learning courses
• Rehearsals before all live
broadcasts
• Pilot tests before courses go live
Develop consistent standards and
processes for designing courses. Adopt
many of the course design "best
practices."
There is a regular process for reviewing
and updating all courses to ensure they
are meeting trainee needs.
Implement a systematic process and
schedule for reviewing and updating
courses/offerings.
HayGroup
Page 9
-------
EPA: Results of Benchmarking Study
1. Create Course Content Development and Design Teams
We recommend that APTI create a course development team that encompasses all of the
key roles required to design training courses/offerings that will be delivered in the media
required to meet trainee needs. As a first step, APTI should examine the roles and skills
of individuals in the current training group in relation to the results from the broad-based
needs assessment to determine whether training group competency/skill gaps exist. For
example, if APTI elects to move towards more synchronous Web-based delivery
methods, there should be members of the course development team with expertise in
Web-based technologies and experience in migrating courses from one medium to
another.
We recommend that at a minimum, APTI consider creating a course development team
with the roles outlined in Table 4 below (which may require additional training of APTI
staff, hiring new talent, developing partnerships with other organizations or experts, or
using outside contractors). The number of individuals required for each role will depend
on the scope of APTFs services (e.g., number of courses/offerings delivered by media).
Note: We understand that many of the benchmarking partners we spoke to have larger
training staffs and budgets than APTI. As a result, APTI may not be able to fill each role
internally, and may need to rely on contractors, on an as-needed basis, to fill some roles.
APTI should create a process for filling key roles, regardless of whether it involves
acquiring internal staff, contractors, or partners.
Table 4: Key Roles of Training Design Team
Key Role
Responsibilities
EOG
Program/Project
Manager
Develops the budget and plans the overall training program (e.g., determines
courses/offerings, training delivery media).
Oversees the day-to-day work for a particular course/offering and ensures objectives are
being met.
Takes responsibility for the training program and interfaces with key stakeholders.
Promotes and evaluates the training program.
Subject Matter
Expert
Expertise in the subject of the course/offering (e.g., professors, external consultants).
Provides an outline of the information that should be included in the training
course/offering.
Ensures that information included in the training course/offering is accurate.
Instructional
Designer
Expertise in the design of training courses to maximize adult learning.
Creates the training objectives and the overall framework of the course/offering.
Creates the look and feel of the course/offering; makes decisions about how to best present
course materials, regardless of the media.
Script Writer
Takes the information provided by the SME and writes a full script for the satellite or live
Web broadcast.
Creates a more conversational broadcast out of the technical information provided by the
subject matter experts.
HayGroup
Page 10
-------
EPA: Results of Benchmarking Study
Technical Experts
• Expertise in the technology used to deliver courses/offerings (e.g., Web developers,
camera crews, satellite technicians).
• Develops Web pages, creates satellite broadcasts or live Webcasts, and so forth.
• Reviews the technical aspects of the course, including delivery methods.
Note: The type of medium used (e.g., Web) will determine the type of technical expertise that is
needed.
Editor/Producer
• Makes sure that course content, regardless of the training delivery medium, is clear and
accurate.
• Produces the course/offering and makes sure it is incorporated in the overall training
program.
Training
Evaluation Expert
Collects and analyzes training evaluation data.
Creates feedback reports for training design teams and others responsible for
modifying/updating courses.
Works with key stakeholders to evaluate the impact of the APTI training program and
makes recommendations for improvements.
Note: One individual may assume multiple roles, and some roles may be filled by
contractors.
Note: The subject matter expert will most likely be a partner versus an APTI team
member.
Below we present two options for developing a training design team(s) and we
recommend that APTI consider the feasibility of doing each within given resource
constraints before deciding on the appropriate approach.
• Option 1 - Develop a cross-functional training design team which includes all of
the core roles that are required regardless of the training delivery media. These
core positions would be responsible for the design of all training courses, with
media-specific roles (e.g., Web developer, camera crew) filled on an as-needed
basis. That is, a core team would remain intact with media-specific roles filled ad
hoc.
• Option 2 - Develop training media-related teams which include all the roles
required for a specific training delivery medium. That is, a continuously intact
team would exist for each training medium that is responsible for the development
of courses within that training medium. For example, there would be one team
responsible for designing all Web courses.
X.
APTI can immediately begin to take an inventory of the roles and skill sets of its current team to
determine where gaps exist. For example, if the team is missing a critical role (e.g., an
instructional designer), APTI can begin to search for an individual (within or outside of the EPA)
to fill this role. As another example, APTI may send one of its team members to training to
develop a key skill currently missing from the group.
APTI should consider either hiring or contracting for a script writer to work with subject matter
experts to develop live broadcasts. Almost all the industry leaders we met with use script writers
to improve the effectiveness of their live broadcasts.
HayGroup
Page 11
-------
EPA: Results of Benchmarking Study
2. Create a Systematic Process for Designing APTI Courses/Offerings
To become more effective in the design of training courses/offerings, APTI should adopt
a systematic and consistent process for designing all courses/offerings regardless of the
media in which they are delivered. APTI should develop (based on "best practices" and
input from APTI staff and other training design experts) and adhere to minimum
standards for designing all courses/offerings. We recommend that at a minimum, APTI
adopt the following standards and processes for designing its courses/offerings.
• Apply criteria to determine the best media for delivering particular training
courses or offerings. These decisions should be based on the needs assessment
process, expert opinions, and other customer feedback. Table 5 provides some
factors to consider when making decisions about methods for delivering training.
Table 5: Criteria for Determining Training Delivery Methods
• Size of Trainee Audience - For larger, geographically-dispersed audiences, think about
using live Web or self-paced Web courses if suitable for the content and objectives of the
course. For smaller audiences, it is probably not cost effective to develop live or self-
paced Web courses. A classroom course may work best.
• Trainee Characteristics - Consider the learning styles and "technical savvy" of trainees.
That is, how does your trainee population learn best?
• Complexity of the Course Materials - More complex material is better taught in the
classroom (e.g., laboratory) or through interactive synchronous training methods. For
less complex material, it may not be worth the cost to use synchronous training delivery
methods; a self-paced course may be sufficient (e.g., policy information, background
information for a classroom course).
• Travel Restrictions - Use distance learning training delivery methods when travel is
limited or consider using a blended learning approach to minimize time away from the
office.
• Need for Just-in-Time Information - Critical information that must be disseminated
quickly is best suited for a live Webcast or satellite broadcast.
• Standard Messages - When training material should be delivered in a consistent (or
standard) way, non-classroom-based methods work best.
• Need for Interaction - Training content that requires extensive student discussion and
question-and-answer sessions are better taught via synchronous training methods (e.g.,
classroom, live Web training with an audio component).
• Requirements for Certification - Classroom training or other synchronous methods that
can track student progress work best for courses that require certification.
• Frequency of Updates - Courses that require regular updates may be best suited for the
self-paced Web training delivery method.
• Budget - For smaller budgets, use a delivery method with an established infrastructure
(e.g., satellite) or one that does not require extensive development time and cost.
Note: This is not an exhaustive list, it is meant to outline some of the more common training
situations/needs and the methods best suited for them.
HayGroup Page 12
-------
EPA: Results of Benchmarking Study
Provide critical information to course designers in the early stage of the process,
including objectives of the course/offering, intended delivery methods, audience
characteristics, technological constraints, and timelines and resources. This will
ensure that courses/offerings that are developed meet stated objectives and
maximize learning for air professionals and other customers.
"\^*
JL
*
• Define the frequency and types of interactions that should be incorporated into all M||ck fL*.
courses/offerings. For example, for satellite broadcasts or live Webcasts, the
standard may be to ensure that there is an opportunity for interaction at least every
10-15 minutes. For self-paced Web courses, the standard may be to provide an
opportunity for interaction (e.g., through a short quiz or simulation) every 3 to 5
screens.
• Require rehearsals for all broadcasts (satellite or live Web), including full script
and use of teleprompters. This will ensure that the instructor presents the material
in an engaging way to the audience, and final glitches and inaccuracies are caught
before the broadcast is delivered live.
• Pilot test all self-paced Web and classroom courses to ensure that the course
content is clear, easy to understand, and meets training objectives, and the
technology itself is easy to navigate, provides sufficient opportunities for
interaction, and so forth. (Note: It is our understanding that APTI is currently
pilot testing its courses before they go live).
These processes and standards should be developed and captured in a policy manual that
outlines the guidelines for designing each new APTI course/offering. This will help to
ensure consistency and quality of all courses/offering that are developed by APTI. It is
also important to communicate the importance of following these guidelines to all APTI
staff and those outside of APTI involved in the course design process (e.g., contractors,
subject matter experts).
3. Develop a Systematic Process and Schedule for Reviewing/Updating Courses
We recommend that APTI create a systematic process and schedule for reviewing and
updating all courses/offerings on a periodic basis. One of the biggest complaints we
heard about the current APTI training program (during interviews conducted during
Task 1) was that many courses are outdated and contain inaccurate information, and that
course materials that are sent out are not of the highest quality (e.g., pages are upside
down and missing, graphics are outdated). A periodic review schedule for all
courses/offerings will help to make sure that APTI courses provide the most up-to-date
information using the most innovative delivery methods and graphics possible.
As a first step, APTI should review the information collected from the needs assessment
process and review/update only those courses that are critical for meeting the needs of air
professionals and other customers (courses that don't meet critical needs should be
eliminated or shelved for a finite period of time). It is important to incorporate this
HayGroup Page 13
-------
EPA: Results of Benchmarking Study
review process into all new courses or offerings. The course review and update process
should include the following:
• Timeline - One of the most important steps is to develop a clear schedule for
updating courses/offerings. APTI should review current courses/offerings to
determine the "shelf life" of information provided during trainings, and develop
review schedules based on the average time it takes for information to become
outdated. For example, courses that provide information about frequently
changing air pollution regulations may need to be reviewed every 12 months,
while courses that deal with long-standing technologies may require less frequent
reviews (every 24-36 months).
• Review Team - Because the course design teams are intimately involved in
developing the courses/offerings, they may also prove useful during the course
review process. For example, subject matter experts involved in the design of a
particular course may be asked to also be involved in the continual review/update
of the course. The key point is to appoint a team(s) that is responsible for one or a
group of courses/media so that different teams are not updating the same courses.
This reviewer role could be built into the contracts for subject matter experts and
others who are working for APTI on the design of courses/offerings.
• Automatic Review Notifications - We recommend that APTI adopt methods that
automatically notify team members when courses/offerings are scheduled for
review. These technologies may also be used to track the review and update
process. For example, the reviewer may receive an electronic notification 30 days
prior to a scheduled course review. This gives the reviewer enough time to notify
other team members and make final amendments to the review process.
• Systematic Steps - APTI should create systematic steps for both reviewing and
updating courses. For each step, APTI should determine who is responsible,
when the review should be completed, and what should be the final product. It
may prove beneficial to develop and disseminate a timeline that highlights major
activities and the responsible parties.
• Communication Plan - Changes made to APTI courses/offerings should be
communicated to air professionals and other customers to ensure that they
continue to enroll in and find the courses useful. Also, communicating about
course updates shows that APTI takes the review and feedback process seriously.
As with the course design guidelines, processes and standards for reviewing and updating
courses should be developed and incorporated in a policy manual that is shared with all
parties (internal and external to APTI) involved in updating courses/offerings.
HayGroup Page 14
-------
EPA: Results of Benchmarking Study
I.
APTI should conduct a quality review of all current course materials before they are distributed to
trainees. Reviewers should check for:
• Typos
• Incorrect information
• Missing, upside down, or out-of-order pages
• Whether the right materials are sent out on time, and with the right courses
HayGroup
Page 15
-------
EPA: Results of Benchmarking Study
IV. Options for Improving APTPs Training Delivery Methods
Our benchmarking research indicates that APTI may not currently be delivering training
in ways that best meet the needs of air professionals and other customers. Courses are
primarily offered and delivered using the classroom medium (with some use of satellite
broadcasts and self-paced Web courses; paper-based self-instructional courses are going
away). APTI is not currently taking advantage of live Web-based methodologies to
deliver training and information to air professionals. Thus, a large portion of APTFs
training methods require air professionals (or trainers) to travel to classroom or satellite
downlink sites to complete training.
It was clear from Task 1 interviews that classroom training is the dominant, and often
preferred, delivery method for many types of courses and we believe there will always be
a need for classroom courses. However, the consensus among interviewees, and our
recommendation, is that APTI must adopt a complement of distance learning training
vehicles, in addition to classroom courses, to ensure that training is accessible to air
professionals and other customers. Our research shows that many types of training and
outreach are amenable to Web-based delivery methods.
There is also the perception that APTFs current distance learning courses/offerings may
not be as effective as they could be. One of the biggest complaints is that there are not
enough opportunities for interaction during satellite broadcasts and self-paced Web
courses, and trainees don't take advantage of the opportunities that do exist.
Additionally, some instructors may not have the skills and experience to deliver training
in particular media (e.g., deliver an effective satellite broadcast).
Our broad recommendation is that APTI develop the infrastructure necessary to deliver a
larger proportion of its courses/offerings via distance learning methods (e.g., live Web),
while continuing to use the classroom to deliver courses best suited for this medium (e.g.,
laboratory courses, courses only offered to a small trainee population). Table 6
highlights the key gaps in training delivery methods that we see between the current
APTI training program and industry leading training programs, as well as options for
closing the gaps.
Table 6: Gaps in Training Delivery Methods
Current APTI Training Program
Industry Leader Training Program
Options for Making APTI an
Industry Leading Training Program
There is an emphasis on classroom
courses, and satellite broadcasts are the
primary method used for distance
learning.
Satellite broadcasts are not viewed as
effective by some stakeholders and
trainees (e.g., not enough opportunity
for interaction, ineffective speakers,
some broadcasts are poorly attended).
Industry leaders are making more use of
distance learning methods versus using
the classroom as the primary training
delivery medium.
Industry leaders tend to make more use of
live Web training than satellite broadcasts
for distance learning.
Satellite broadcast are typically used for
Move from primarily classroom-based
delivery methods to distance learning
methods.
Evaluate the feasibility, benefits, and
cost effectiveness of making more use
of live Web training versus other
delivery methods. If feasible, begin to
move towards more live Web training.
HayGroup
Page 16
-------
EPA: Results of Benchmarking Study
dissemination of critical, real-time
information.
Determine the best use of satellite
broadcasts.
APTI has not fully adopted a blended
learning approach. APTI is currently
using only one medium to deliver each
course/offering.
Many courses are delivered via multiple
media (e.g., classroom and Web).
Determine whether some APTI courses
can be delivered via multiple media to
maximize the effectiveness of training,
minimize travel costs, and make
training more convenient and
accessible to all air professionals.
There is the perception that
opportunities for interaction during
APTI satellite broadcasts are limited,
and trainees don't take advantage of
those opportunities that do exist.
Serf-paced Web courses are not
interactive enough to maximize
learning potential.
Innovative methods are used to
incorporate opportunities for interaction
into distance learning courses (interaction
with the technology, instructor, other
students).
Review current and past methods for
incorporating interaction into satellite
broadcasts. Determine whether these
practices are effective and ways to
improve their effectiveness (e.g.,
encourage air professionals to ask
questions during satellite broadcasts).
Look for ways to incorporate
opportunities for interaction into serf-
paced Web courses (e.g., chat rooms,
real-world simulations, case studies).
Instructors are not always effective at
delivering training in the selected
medium.
Instructors are required to complete
training before delivering a course in a
particular medium (e.g., live Web).
A train-the-trainer approach is often used.
Ensure that all instructors are qualified
to deliver training in the appropriate
medium.
For the remainder of this section, we further discuss our options for improving the way
APTI training courses/offerings are delivered.
1. Make More Use of Synchronous Distance Learning Methods, with a Primary
Focus on Live Web
As indicated previously, many industry leaders are moving away from classroom courses
to live Web courses that trainees can complete at their work stations. Furthermore, in the
most recent APTI needs assessment, participants indicated that the biggest factors that
influence sending staff to training are cost, travel restrictions, workload, and staff
shortages. This seems to indicate that APTI must look for ways other than the classroom
and satellite broadcast methods to deliver synchronous training to air professionals. Live
Web courses offer all the advantages of satellite courses, with additional capabilities for
incorporating interactions into training courses. Also, live Web meets the "my desktop,
at my time" needs of most busy air professionals. That is, live Web allows trainees to
access courses wherever they have a Web connection, rather than traveling to a classroom
or satellite capable site. Our benchmarking research also found that many industry
leaders are successfully delivering training via the Web that was previously delivered in a
classroom setting.
As a first step, APTI should conduct a feasibility study (to supplement the "best
practices" research which supports the use of live Web training) to determine the types of
courses/offerings best suited for live Web delivery, and the cost and benefits of using live
Web versus classroom or satellite media. APTI must weigh the cost of developing live
HayGroup
Page 17
-------
EPA: Results of Benchmarking Study
Web courses (and adopting this technology) against the potential benefits that this
technology will provide air professionals and other customers. For example, are more air
professionals likely to attend a training course or outreach activity if it is delivered via the
Web versus a classroom session or satellite broadcast? Is live Web training equal to or
better than the classroom or satellite in engaging air professionals, teaching critical
skills/competencies, and offering opportunities for interaction? How does the cost of
delivering live Web training (once this technology is adopted) compare to the classroom
or satellite media?
As part of the feasibility study, APTI may want to select a specific course and deliver it
through different media (e.g., classroom, satellite, live Web) and evaluate how effectively
the course is delivered through the selected media (as a pilot). Feedback could be sought
from instructors, instructional design experts, and trainees as to the effectiveness of each
medium in keeping students engaged, teaching required competencies/skills, providing
opportunities for interaction, and so forth. Additionally, if a broad-based needs
assessment is conducted, APTI can also seek input into whether air professionals prefer
one delivery method over the others; whether supervisors are more likely to support their
staff in training if it is delivered via the Web versus the classroom or satellite broadcast;
and so forth. That is, the broad-based needs assessment will provide critical information
about whether the needs of APTI's key customers would be better served through a live
Web training delivery method.
Tips for Conducting a Feasibility Study
A feasibility study will help determine whether the live Web technology is feasible and
cost effective to implement at APTI. It will provide input into the new APTI
courses/offering that would be best delivered using the live Web medium, and which
current courses are most suitable for migration from the classroom or satellite broadcast
to live Web media. As part of this assessment, APTI will want to study the following:
• The cost to develop a course/offering using live Web versus other media
• The cost to deliver a course/offering using live Web versus other media
• The number of participants likely to attend a live Web course versus a course
delivered via other media
• Extent to which opportunities for interaction can be incorporated into the Web versus
other media
• Effectiveness of live Web versus other media in meeting the stated objectives of the
course/offering (e.g., teaching the required competencies/skills)
In sum, we believe that APTI should deliver a larger proportion of its courses via distance
learning methods, particularly live Web. Most of the industry leaders we spoke with are
moving away from the classroom and satellite to the live Web training delivery method.
However, because APTI has invested significant resources in classroom training (and
satellite broadcasts), our recommendation is to determine how to best use its existing
training methodologies while following the industry shift towards live Web-based
distance learning programs.
HayGroup Page 18
-------
EPA: Results of Benchmarking Study
2. Create a Blended Learning Approach
Like many industry leaders, APTI should consider adopting a blended learning approach
for some of its key training courses/offerings. By incorporating two or more training
delivery media (e.g., classroom and Web), APTI may improve the effectiveness of its
current and future courses/offerings and reduce travel cost and time away from the job for
air professionals. We are not recommending that APTI offer the same course/offering in
two different media, but use different media for different portions of the course/offering
(e.g., Web for prerequisite materials; classroom for portions of the course that require a
lab or extensive discussions).
As a first step, APTI should determine which of its current and future courses/offerings
are best suited for a blended learning approach. Data collected from the broad-based
needs assessment will also provide insight into where a blended approach may work best.
Below are a couple scenarios where we believe the blended learning approach would
work well for APTI:
• A multiple day classroom course that is offered multiple times throughout the
year - APTI may be able to reduce the amount of time air professionals spend in
the classroom and away from their jobs through a blended learning approach. For
example, a self-study Web course could be developed to provide critical
knowledge (e.g., regulations, policies) before the classroom course (versus
spending time lecturing about basic knowledge in the classroom course), and to
get all air professionals up to a similar knowledge level before the classroom
sessions. This could reduce a significant portion of classroom time. The
classroom session would then focus on interactive exercises, discussions, case
studies, and simulations or actual real-world job activities versus delivering
information in a lecture format. Following a few days of classroom sessions, air
professionals could also complete additional self-paced Web sessions or
participate in a live Webcast back at their own office to complete the course.
• A self-paced Web course that requires discussion - APTI may be able to reduce
the time spent developing self-paced Web courses (most notably the interactivity
aspects) and improve interactions by incorporating a blended learning approach.
For example, live Web course sessions could be developed to supplement self-
paced Web courses in an effort to augment interactions and foster discussions.
Trainees would still access the self-paced Web course at their own convenience.
However, they would be required to attend periodic live Web sessions during the
overall training course, where trainees would be given the opportunity to discuss
and clarify previous course content. This could reduce the time required to
develop the interactive discussion aspects of the self-paced Web course, while
providing for real-time discussions among trainees (and instructors). Further, the
self-paced Web sessions could focus solely on providing necessary information
rather than alternating between information dissemination and delayed
discussions. Note: Satellite or classroom-based course sessions may be
substituted for or augment further live Web sessions.
HayGroup Page 19
-------
EPA: Results of Benchmarking Study
I.
3. Ensure Live Distance Learning Courses Provide Sufficient Opportunities for
Interaction
As indicated previously, the perception is that APTI does not provide enough
opportunities for interaction in its current distance learning methods, and trainees do not
maximize the opportunities that are provided. Industry experts agree that providing
opportunities for interaction is critical to keeping trainees engaged in the course and most
importantly, ensuring that they develop the critical skills and competencies that the
course is intended to teach. To become more effective at delivering distance learning
training, APTI needs to: a) examine the opportunities for interaction that are provided in
its broadcasts; b) determine new ways to incorporate opportunities for interaction into
broadcasts; and c) explore ways to ensure that trainees take advantage of opportunities
for interaction. These same principles will apply if APTI decides to adopt live Web
broadcasts.
a) Examine opportunities for interaction in satellite broadcasts
In the short-term, APTI should examine current opportunities for interaction used during satellite
broadcasts, as well as methods that have been used in the past. As part of this review, APTI
should consider the effectiveness of each type of interaction opportunity, and why some methods
are not as effective as others. Below are some criteria for determining the effectiveness of
methods of interaction:
• Is there an opportunity for interaction at least every 10-15 minutes?
• Do methods for interaction interrupt the flow of the session?
• Do trainees take advantage of opportunities for interaction (e.g., ask questions, participate
in discussions)?
• Do trainees receive answers to all questions that are not addressed during the broadcast?
• Can trainees hear and respond to others' questions?
This information will help APTI to determine which methods for interaction should be retained
(and used during satellite or live Web broadcasts), and whether new methods for interaction
should be adopted.
APTI also needs to ensure that it is consistently providing opportunities for interaction in all its
distance learning courses. _
b) Determine new ways to provide opportunities for interaction during
broadcasts
Based on our benchmarking research, we believe that new methods for interaction are
needed to enhance the effectiveness of APTI' s satellite broadcasts (and live Web training,
if it is adopted). We recommend that APTI consider using some of the following
methods to enhance interactions during broadcasts.
> Incorporate an open audio line (phone line) during broadcasts so that air
professionals can ask questions and discuss issues with the instructor and
among each other.
HayGroup
Page 20
-------
EPA: Results of Benchmarking Study
> Create a chat room that air professionals can visit after the broadcast to
discuss issues and ask questions. The instructor or subject matter experts
should periodically participate in chat room discussions and answer trainee
questions.
> Provide trainees with an e-mail address for sending questions to the instructor
or other subject matter experts during a live Web session (if live Web is
adopted). These questions could be answered during the live broadcast or
questions and answers could be posted on a Web site in the future.
c) Adopt practices for ensuring that opportunities for interaction are effective
APTI needs to go beyond simply incorporating methods for interaction into its live
distance learning courses. These methods for interaction must be effective at soliciting
questions from trainees and generating discussions during the broadcast. We recommend
several strategies for ensuring that methods for incorporating interaction into broadcasts
are effective. Many of these strategies can be adopted in the short term.
I.
Enhancing Methods of Interaction
• Publicize opportunities to interact. As part of publicizing the satellite (or live Web)
broadcast, inform potential participants about the process for asking questions or
participating in a discussion during an APTI broadcast and encourage them to do so.
This may include broadcasting call-in numbers and e-mail addresses, instructions for
using the various methods, and the benefits of actively participating in the broadcast.
• Establish proper etiquette. Develop proper interaction etiquette for trainees to follow.
This etiquette should be communicated to participants before each training session
begins. Participants should be reminded that they may be asked to leave the session if
they repeatedly violate the etiquette.
• Use screeners and technology monitors to review incoming questions. APTI or
state/local agencies should provide an individual who can act as a question screener and
technology monitor for each broadcast. This individual would be tasked with accepting,
reviewing, and facilitating questions, and making sure that all technology is working
properly.
• Develop discussion questions prior to the broadcast. Develop a set of 5-10 potential
questions for each training session. When trainees are reluctant to ask questions, a
training team member can initiate discussions by asking one of the questions.
• Follow up on unanswered questions. Provide trainees with the answers to all questions
that are asked, regardless of whether they are answered during the broadcast. This may
be done by posting questions and answers to a Website. _
HayGroup
Page 21
-------
EPA: Results of Benchmarking Study
4. Improve the Effectiveness of Self-paced Web Courses
We recommend that APTI enhance its current self-paced Web courses, particularly by
incorporating opportunities for interaction with the technology itself, instructor, and
among trainees. The most recent needs assessment found that more self-training is
needed, especially those related to interactive Web-based methods. It is important that
the self-paced Web courses that are offered provide an engaging learning experience and
are not simply a .PDF document or Power Point slides put on the Web.
Below are several strategies for increasing the effectiveness of Self-paced Web courses.
• Incorporate real-world simulations and case studies Our findings from Task
1 indicated that air professionals would like more practical, real-world examples
and applications in training courses so they could see how the training applies to
the "real world." By incorporating simulations and case studies into self-paced
Web courses, air professionals can actually practice the skills they need on their
jobs (e.g., testing air quality) and solve real-world problems. Additionally, these
tools will increase the likelihood that trainees will be more engaged in the training
than if the course simply presented information or focused on theory.
Incorporating real-world simulations and case studies will require sophisticated
Web development resources and should only be done for courses that are taught
to a relatively large audience and meet critical needs.
• Incorporate quizzes and assessments. We recommend incorporating periodic
quizzes and assessments into self-paced Web courses to help keep trainees
engaged and focused on the material. Also, by incorporating quizzes and
assessments into the courses, APTI can effectively track short-term learning and
begin to move towards consistent level 2 evaluations. Quizzes can be
programmed so that trainees are blocked from continuing the training until they
provide correct responses. However, participants do not need to be blocked from
continuing; it may be more effective to provide brief explanations after each quiz
or assessment of the correct responses.
• Create engaging material. APTI should develop engaging visual materials for
every Web course. As a first step, APTI should review existing courses to make
sure that the material is presented in an engaging manner. Creating engaging
material is one of the most important aspects of delivering an effective self-paced
Web training session. Our research shows that individuals learn about 75% of
what they know through vision. Also, trainees will "tune out" after long training
sessions, especially if they don't find the visual material stimulating or feel that
the material is not relevant to the course.
• Use chat rooms or Web boards to facilitate interactions To foster self-paced ^ *b
Web course interactions, we recommend that APTI create chat rooms or Web
boards where trainees can post questions to instructors, receive course updates
from instructors, and have e-mail discussions with each other. At the beginning
of the training course or offering, each trainee should be provided with the
HayGroup Page 22
-------
EPA: Results of Benchmarking Study
location of and instructions for using the chat room or Web board. Also,
instructors should review the chat room or Web board periodically and be
encouraged to use these as a medium to interact with trainees.
• Facilitate cohort-based training. APTI may want to consider facilitating the
assembly of cohorts of approximately five to seven trainees to complete the
training program as a group rather than individually. Cohorts may, for example,
come from the same state or local agency or be distributed across the country
based on function. Participants would complete the self-paced modules at their
own pace but within a specified time frame (e.g., one week to complete a
module). At a set time, participants would then access chat rooms or participate
in a conference call to discuss the course and ask each other questions in between
modules. This is a more effective and efficient method because trainees have
more opportunities to interact and they can provide each other with more
assistance and support.
5. Ensure That Instructors Have the Tools and Skills to Teach
Our benchmarking research and interviews from Task 1 of the benchmarking study
revealed that some instructors do not have the skills required to deliver training in
particular media. For example, some potential instructors may be experts in the subject
of the course, but are not effective at delivering a satellite broadcast (e.g., are not
engaging). APTI should focus on ensuring that instructors have not only the subject
matter expertise required to teach, but also the technical tools and skills and "stage
presence" needed to teach using various distance learning methods. Instructors with the
appropriate skills will conduct more effective courses, thus increasing attendance and
making the overall training program more valuable.
X.
> Select instructors who have subject matter expertise, applied experience in the field, and
experience in the medium of course delivery.
> Consider using professional speakers to deliver training in conjunction with subject
matter experts who can answer specific questions raised by air professionals.
> Provide instructors with standard guidelines for training in a particular medium.
Instructors should be required to participate in a short training session designed to
provide them with basic, standard guidelines for conducting a training session in a given
medium (before delivering APTI training in a particular medium for the first time). Not
only will this provide instructors with tips on how to provide training, but it will also help
to "brand" EPA instructors and training programs by showing instructors the "EPA way"
to train. Some of the topics to cover include:
• Where to stand and how much to move during training
• When and how to look into the camera
• How to use basic technologies
• How to encourage interactions
• How to deal with difficult students
• Tips for holding the attention of students
Note: Several private companies and educational institutions offer "training the trainer"
courses that APTI may consider using before developing its own program.
X.
HayGroup
Page 23
-------
EPA: Results of Benchmarking Study
V. Options for Improving APTFs Training Evaluation Process
APTFs current training evaluation process focuses on end-of-course level 1 evaluations
("smile sheets"). For some classroom courses, air professionals complete a pretest before
the course begins to assess their level of knowledge and a posttest at the end of the course
to see how much they have learned (level 2). To become an industry leading training
program, APTI needs to reexamine its training evaluation process to ensure that it is
measuring the "right things" - the extent to which the training is adding value to air
professionals and the agency as a whole. Measuring the "right things" means expanding
APTI's training evaluation process to at least include level 3 evaluations (extent to which
training impacts air professionals' performance on the job).
There is also room for improving the level 1 forms that are currently used to assess APTI
training by including questions to better understand why training courses were effective
or not. Finally, the perception is that feedback from the training evaluation process is
rarely used to improve training courses and offerings.
Table 7 highlights the current state of APTFs training evaluation process compared to
what industry leaders are doing in this area, and presents options for creating a more
effective training evaluation process.
Table 7: Gaps in Training Evaluation Process
Current APTI Training Program
Level 1 evaluations are not as effective
as they could be.
The evaluation of APTI training
courses focuses on level 1 - End of
course evaluations (also called "smile
sheets"). There is some use of level 2
evaluations.
Training evaluation data are not
consistently used to improve courses
and offerings.
Industry Leader Training Program
The training evaluation process assesses
the impact of training on: 1) trainee
knowledge gained during the course
(level 2); 2) future performance on the job
(level 3); and 3) overall organizational
performance (level 4).
Training evaluation data are used to
improve training course content and
delivery methods.
Options for Making APTI an
Industry Leading Training Program
Improve the instruments used to
provide level 1 evaluations of APTI
courses/offerings.
Expand the training evaluation process
beyond end-of-course level 1
evaluations.
Create a process for tracking,
monitoring and using training
evaluation data to make improvements
to training courses and offerings.
Below we expand on the recommended options for improving the training evaluation
process at APTF
1. Increase the Effectiveness of Level 1 Evaluations
There are several quick fixes that APTI can make to improve its process for conducting
level 1 evaluations, which are described on the following page.
HayGroup
Page 24
-------
EPA: Results of Benchmarking Study
X.
a) Improve level 1 evaluation instruments. Level 1 evaluations provide useful information for
making immediate changes to training course content and delivery methods (e.g., there is a need
to add more opportunities for interaction into a Web-based course). Based on our review of the
current APTI training program, we believe that there is room for improving the level 1
evaluations ("smile sheets") used to evaluate classroom courses and satellite broadcasts. That is,
the current course evaluation forms do not provide enough information for understanding why the
classroom course/satellite broadcast was effective, whether the delivery method is the best
medium for delivering training, and more importantly, how to improve APTI training
courses/offerings. At a minimum, we recommend adding open-ended questions to the training
evaluation forms. For example:
• What about the course/broadcast did you like best?
• What about the course/broadcast would you change?
• How would you improve the current training course/broadcast?
• Were there enough opportunities for interaction in the course/broadcast? If not, what are
your recommendations for enhancing interactions during the course/broadcast?
b) Evaluate all courses/offerings, regardless of the delivery method. It is important to solicit
feedback about all APTI courses and offerings, including self-paced Web courses. Air
professionals should be given the opportunity to complete a course evaluation on-line after they
finish the self-paced Web course. The evaluation should focus on the course content itself; how it
was presented; the effectiveness of any quizzes, simulations, or case studies; the ease of accessing
the course and navigation through the Web site; and so forth. The evaluation should also include
overall questions about the effectiveness of the Web course at teaching required skills and
knowledge, whether it engages air professionals to begin and complete the course, and whether
the Web is the best method for teaching required competencies/skills.
c) Follow up on less favorable evaluations. APTI should consider conducting follow-up data
collection when level 1 evaluations do not meet a minimum rating. For example, if air
professionals rate a particular aspect of a training course below a minimum rating, APTI may
want to contact air professionals to obtain more insight into why the course was rated less
favorably than expected. This could be done through a telephone interview.
2. Expand the Training Evaluation Process Beyond Level 1 Evaluations
APTI should create a training evaluation process that not only determines how air
professionals feel about courses/offerings and whether they learned during the courses,
but whether the training improved their performance on the job (level 3 evaluation) and
helped them contribute to the mission of their agency. As indicated previously, the
current APTI training evaluation process focuses on end-of-course level 1 evaluations,
the "smile sheet" evaluations. These evaluations are necessary to obtain trainees' initial
reactions to courses/offerings, but they are far from sufficient for establishing the
organizational impact of training courses or offerings. Level 3 and 4 evaluations provide
valuable information for modifying training courses/offerings (and potentially training
objectives), as well as for demonstrating the value of the Air Pollution Training program.
We recommend the following steps for creating a more comprehensive training
evaluation process:
HayGroup
Page 25
-------
EPA: Results of Benchmarking Study
Plan for long-term evaluations. Developing a training evaluation process that assesses
overall organizational impact will take time and planning. Training experts agree that
higher level evaluations are very difficult to do because it takes several months before an
organization may see any change in performance as a result of training. Additionally, it
is often difficult to discern the effects of a training course on trainee performance; there
are so many other factors that could impact performance.
As a first step, APTI should design a plan for evaluating the long-term impact of its
courses/offerings. This plan should include:
• Timelines for evaluating training (e.g., transfer of training will be evaluated three
and six months after the air professionals complete a training course/offering).
• Who will participate in the assessments (e.g., air professionals, supervisors of air
professionals, agency leaders)?
• How many assessment instruments will be required? For example, can the same
data collection instrument be used for all types of training media or will different
types of instruments be required?
• How will data be collected (e.g., on-line surveys, interviews with key
stakeholders)?
• Which courses/offerings will be evaluated beyond level 1 (e.g., level 3
evaluations take time and resources; APTI may want to conduct these evaluations
only for its most critical courses/offerings)?
• What will be measured by the assessment? There is a need to define the desired
changes in air professional behavior and how they will be measured.
• What are the minimum ratings required for an evaluation (e.g., a 3.5 on a 5 point
scale)? What is the follow up process if ratings fall below the minimum rating
(e.g., a follow up interview to gain more insight into areas rated lower than
expected; development of an action plan for improving the course/offering)?
• Who is accountable for evaluating APTI training?
• How will training evaluation data be used to improve APTI training courses and
offerings?
Gain buy-in from key stakeholders. A higher level training evaluation will require the
support and participation of state and local agencies and other key stakeholders. APTI
should communicate its plans for expanding its training evaluation process and how this
will benefit state and local agencies and other key stakeholders. Additionally, it would be
helpful to involve key stakeholders in the design and conduct of the training evaluation
process.
Begin to conduct evaluations with trainees and supervisors to evaluate learning
transfer (level 3). APTI should conduct follow-up evaluations 3 to 6 months post-course
to measure whether trainees have effectively transferred the knowledge/skills learned
during training to their jobs. Evaluations should follow a 360-degree methodology that
obtains data from the trainee as well as supervisors and subordinates. For example, a
HayGroup Page 26
-------
EPA: Results of Benchmarking Study
sample of air professionals who have completed a critical classroom course may receive
an on-line survey three and six months after they complete the course, asking for their
opinions about how well they have applied the APTI training to their jobs and how it has
contributed to the overall effectiveness of their agency. Their supervisors would also
receive a similar on-line survey to obtain their opinions about the extent to which the
training has improved their employee's performance on the job.
To the extent possible, APTI should compare evaluation data (e.g., performance of key
tasks on the job) with air professionals not yet participating in training. This will help
demonstrate that changes are due to the training and not some other outside source (e.g.,
implementation of a new process or procedure).
3. Create a Process for Monitoring and Using Training Evaluation Data to
Make Improvements
We recommend that APTI use the evaluation data not only as a measure of training
effectiveness but also to make improvements to training courses and offerings. The level
1 training evaluations are likely to highlight aspects of courses/offerings that participants
find useful/not useful as well as ideas they may have to improve courses/offerings (e.g.,
technology, exercises, instructors). The higher level evaluations should further show
areas where the training courses and offering are not effective; that is, where they are not
meeting stated objectives. By reviewing these data, APTI should be able to pinpoint
areas that need to be modified to meet these training objectives. It is also important to
communicate to the air professional community the changes that are being made to the
training as a result of the evaluation.
In order to do this, APTI should develop a systematic feedback process to ensure that the
appropriate evaluation data are monitored, processed, and used to improve courses. For
example, instructors or designated training team members should be responsible for
collecting evaluation data and transferring it to a central database for processing.
Transferring the data to a central location will allow the APTI training team to evaluate
trends (i.e., from year to year) and make comparisons to other courses. Whether courses
meet these needs can be determined by setting minimum standards that each course must
meet (e.g., at least 70% satisfaction with the effectiveness of training). The results
(positive and negative) and any suggested changes should then be fed back to both the
team responsible for designing and updating courses as well as current and future course
participants. Even if the data are used to improve courses, participants will not feel that
evaluations are effective if they are not actively shown results and subsequent changes.
HayGroup Page 27
-------
EPA: Results of Benchmarking Study
VI. Next Steps
In this section, we outline our recommended next steps for APTI to identify and
implement needed changes to the Air Pollution Training program. We present
suggestions for translating the options presented throughout this report into
improvements to the APTI training program. The ultimate goal of this change effort is to
ensure that the APTI training program is continually meeting the needs of air
professionals and other key customers.
We recognize that APTI may not be able to immediately implement all of the
recommended options outlined in this report, and some may not be beneficial or feasible
to implement. Also, we understand that there may be some reluctance to change certain
practices or to move away from specific delivery media (e.g., classroom and satellite
broadcasts).
1. Plan for a Successful Change Effort
As a first step, we recommend that APTI define its desired end state, make decisions
about those changes that should be made to the APTI training program (which options to
implement), create a plan of action, and determine strategies for overcoming resistance to
change.
Define desired end state. APTI leadership, staff, and other key stakeholders should meet
to discuss the recommended options outlined in this report. We believe that the APTI
team and key stakeholders must have buy-in and ownership of the recommended changes
to ensure they are successfully implemented. That is, this group must have a clear
understanding of not only the changes APTI chooses to make, but also the end point or
desired state that APTI hopes to attain.
These meetings will allow the APTI team and other key stakeholders to voice their
concerns and opinions about the recommend options, identify potential new options for
improving the training program, and come to consensus on the recommended changes
that APTI should pursue. During these meetings, the APTI team should focus on:
• Translating recommended changes (options) into actionable initiatives and
accountabilities
• Illustrating the benefits, feasibility, and cost of the recommended options
• Prioritizing short-term and long-term actions
As indicated previously, we believe that APTI should also conduct a broad-based needs
assessment in the near future to redefine its mission and scope, and ultimately determine
the courses/offerings that should be provided to meet the needs of its key customers.
This broad-based needs assessment will validate and further define the direction APTI
should take in the future, and which long-term options should be adopted. We
recommend holding off on major changes to the APTI training program (e.g., those that
require extensive resources) until after the broad-based needs assessment is conducted.
That is, customer and stakeholder feedback from the broad-based needs assessment will
HayGroup Page 28
-------
EPA: Results of Benchmarking Study
be instrumental for determining whether major options/changes meet a critical customer
need, and thus should be adopted. On the other hand, APTI should begin immediately to
work on options that are easier to implement, less resource intensive, and that will meet a
stated or known need of a key customer group (e.g., a need revealed in a prior needs
assessment or training evaluation).
Collect additional information. APTI may need to collect additional information to
make decisions about the feasibility of some of the recommended options in this report.
For example, in determining whether to deliver more courses through live Web (versus
the classroom or satellite broadcasts), APTI should conduct a feasibility study to
determine whether adopting the live Web technology: 1) is cost effective; 2) would
increase attendance in courses; and 3) is more effective than the classroom or satellite in
teaching required skills and competencies (see section IV: Options for Improving APTFs
Training Delivery Methods).
Create change plan. After identifying the priority options that APTI will implement to
improve its current training program, we recommend creating a change plan that
describes the changes to be made and why they will be made, who is accountable for
each change, timelines for implementing short- and long-term change, resources required
to implement the changes, and how success will be measured. Below, we provide a
sample template for a change plan.
Change Plan
Air Pollution Training Program
Date:
Area for
Improvement
Goals
Recommended
Actions (Options)
Responsible
Party(s)
Support
Required
Performance
Measures
Target
Dates
Overcoming resistance to change. Resistance to change is one of the most common
barriers to overcome at all levels of an organization and during all phases of the change
process. Creating buy-in from leadership and individuals who will be affected by the
change (e.g., APTI staff, state and local agencies) is an effective method to help
individuals overcome this resistance. When key individuals are included in the decision
process, and when they endorse mutually agreed upon changes, they are more likely to
HayGroup
Page 29
-------
EPA: Results of Benchmarking Study
work to ensure that change happens as planned. Once an individual publicly supports a
plan, it becomes more difficult for the individual to resist the change. For example, it
may be relatively easy for someone to resist migrating a training course from a satellite to
Web-based medium if they feel that the decision to change has been dumped on them.
However, APTI staff (or air professionals) will be less likely to resist this change when
the need for change is explained and they have endorsed at least some aspect of that need.
2. Implement "Quick Wins"
Throughout this report we highlight some "quick fixes" or short-term changes that APTI
may take to immediately improve its training program (a summary of all "quick fixes" is
presented in Appendix C). By implementing some of these short-term changes, APTI
can begin to establish some "quick wins". Quick wins are immediate successes that
demonstrate a commitment to and seriousness about making improvements.
We recommend that APTI focus on foundational quick wins. That is, quick wins that
form the basis for additional wins. For example, before incorporating standards for
interaction into all future distance learning offerings, APTI should first focus on
developing the standards and using them to review existing courses/offerings.
Quick wins are most effective when they are timely, focused, and visible.
• Timely. APTI should begin to implement quick wins as soon after announcing
the change initiative as possible. By waiting too long, APTI runs the risk of
appearing uncommitted or unable to make effective decisions.
• Focused. The best quick wins are focused. They are implemented with a
strategic focus, not as a part of 50 separate initiatives. APTI should show that
each quick win is part of a strategic focus that will be used as a bridge to
continued change efforts.
• Visible. As with having a strategic focus, the best quick wins stand out and are
visible to all key stakeholders. Stakeholders need to be able to see the wins to
effectively understand what changes are taking place. Even if change is
occurring, stakeholders are not likely to be motivated if they cannot see what is
happening.
3. Implement Long-Term Changes
Long-term change (e.g., adopting a blended learning approach) requires preparation and a
long-term commitment to success, even in the face of periodic setbacks. The remaining
section of this report describes factors that APTI should consider to help prepare for and
implement long-term changes to the Air Pollution Training program.
a) Create capability and capacity to change
With leadership commitment and support and some quick wins in place, the APTI team
should begin to develop the capability and capacity for long-term change. This will
HayGroup Page 30
-------
EPA: Results of Benchmarking Study
require identifying and building the key change infrastructure and partnerships needed for
successful change implementation. Some of the questions that the APTI team will need
to ask during this stage are:
• Are the right people in place? For example, does APTI have the right mix of
roles to effectively design courses/offerings?
• Does APTI have the right technology?
• Have key stakeholders been receptive to the quick wins?
• What are the likely resource constraints?
• Does APTI need and can it get additional resources?
We recognize that limited resources is one of APTFs biggest barriers to implementing
needed changes to the training program. However, once APTFs scope and strategic
direction are clearly defined (e.g., through the broad-based needs assessment), APTI will
be able to make the most effective use of its limited resources. For example, updating all
courses/offerings on a periodic basis will be less resource intensive if APTI reduces its
course offerings to those that best meet customer needs (and can not be obtained
elsewhere).
b) Design and implement solutions
Once the APTI team has a framework in place for developing the capability and capacity
to change, it should focus on designing the right solutions for change. For example, how
is APTI going to implement live Web training or create a systematic process for
designing courses? Or, what should be the standards for ensuring interactions in self-
paced Web courses and how should these standards be regulated?
Fortunately, by this stage in the process, the APTI team should have at least the
beginning designs to answer these and other important questions. That is, leadership
support should be in place, the APTI change team should have a vision and buy-in for
change, some quick wins should be accomplished, ongoing partnerships should be
developed, and there should be an overall capacity and commitment to change. As a
result, the APTI team can focus on designing solutions rather than garnering support, and
may even be able to build off of short-term solutions developed for the quick wins.
c) Evaluate success
APTI should continually monitor the effectiveness of short- and long-term changes to its
Air Pollution Training program. This can be accomplished through:
• Informal feedback from customers and stakeholders (e.g., as to the effectiveness
of a live Web training method or a blended learning approach)
• Informal feedback from subject matter experts, instructional designers, and
instructors
HayGroup Page 31
-------
EPA: Results of Benchmarking Study
Data from the training evaluation process (e.g., air professional opinions about a
blended learning offering; impact of a blended learning offering on future
performance on the job)
Key Factors for Change
In addition to the steps outlined in this section, we believe that the following factors are critical to
ensure that APTI implements a successful change effort:
Leadership commitment and support. Clear and consistent commitment and support from APTI
(and EPA) leadership will be a critical factor in successfully implementing change in the Air
Pollution Training program. During our benchmarking research, we found that the industry
leaders in training have leadership that shows a strong commitment not only to training but also
to implementing new and innovative distance learning delivery methods. Leadership can set the
example of ongoing commitment and support for the entire organization to follow.
Communicate the vision and change plan. Communicating APTI's vision and desired end state
throughout the process will help motivate and provide all stakeholders with a common
understanding of where APTI wants to take the Air Pollution Training program, and why changes
are needed to the program. Sending clear and consistent messages about change helps develop
buy-in, informs people of courses of action, and gives them the real-time information they need to
act in accordance with the vision.
Partner with JTC and state and local agencies. The JTC and state and local agencies are key
stakeholders in the Air Pollution Training program and any change initiative that APTI develops.
As with the benchmarking study, we recommend that APTI communicate its plans and seek input
from these stakeholders throughout the change process.
HayGroup Page 32
-------
EPA: Results of Benchmarking Study
Appendix A
Task 1 Report:
Evaluation of Current Air Pollution Training Program
HayGroup Page 33
-------
EPA: Results of Benchmarking Study
/. Introduction
Background
In May 2004, the Education and Outreach Group (EOG) of the Environmental Protection
Agency (EPA) contracted with the Hay Group to conduct a Benchmarking Study of its
Air Pollution Training program. The ultimate goal of the project is to provide options for
improving the current Air Pollution Training program, and ultimately, EOG's business
performance. The following three tasks are aimed at accomplishing this goal:
• Evaluate the current Air Pollution Training program (Task 1)
• Identify best practices of leading training programs (Task 2)
• Develop a plan of action for improving the current Air Pollution Training
program (Task 3)
This report summarizes our findings from Task 1 - evaluation of the current Air Pollution
Training program. The findings in this report provide an assessment of the practices,
procedures and processes currently used by the Air Pollution Training Institute (APTI) to
develop, deliver, and evaluate training. In this report, we focus on evaluating the current
training program based on input from numerous sources.
In subsequent reports, we will present our conclusions regarding the direction that we
believe the Air Pollution Training program should take.
Methodology
This section summarizes the methodology used to evaluate the current Air Pollution
Training program. The steps for evaluating the current training program are described
below.
Identify evaluation criteria. The first step was to identify the critical air pollution
training practices, procedures, and processes that should be evaluated (i.e., evaluation
criteria). These evaluation criteria were identified based on discussions with APTI staff,
reviews of the training literature, and the Hay Group's extensive research and experience
in training delivery and evaluation. The table on the following page shows the key
aspects of the current training program that were evaluated during Task 1.
HayGroup Page 34
-------
EPA: Results of Benchmarking Study
Evaluation Criteria
Needs Assessment
Process
Types of Information Collected
Effectiveness of Needs Assessment Survey
Uses of needs assessment results
Course Content
• Process for determining course content and delivery methods
• Availability of courses
• Extent to which courses are offered that meet student needs
• Quality of course content
Course Delivery
Methods
• Effectiveness of classroom courses and instructors
• Effectiveness of satellite broadcasts
• Effectiveness of on-line training
• Effectiveness of self-instructional training
Training Evaluation
Process
Effectiveness of training evaluation instruments
Uses of training evaluation results
Conduct subject matter expert interviews. Hay designed an interview protocol (based on
the evaluation criteria) to capture information that is critical to understanding and
assessing the current Air Pollution Training program. The interview protocol included an
introductory paragraph to read to participants at the beginning of the interview and
interview questions.
Hay worked with APTI staff to identify appropriate individuals to interview about the
current training program. An effort was made to interview a diverse group of individuals
who have a good understanding of the training program, its objectives and desired results,
its future direction, and its overall effectiveness. The following types of individuals were
interviewed:
• Members of the Joint Training Committee (JTC)
• Members of the MARAMA Regional Consortium
• Individuals who run area training centers
• APTI course instructors
• Supervisors in state and local agencies who have sent employees to APTI training
courses (some of these supervisors solicited feedback from their employees about
the effective of APTI training)
• Contractors who design APTI course content and conduct the training needs
assessment
• APTI management and staff
HayGroup
Page 35
-------
EPA: Results of Benchmarking Study
A total of 22 individuals participated in one-on-one interviews in person at Research
Triangle Park or via the telephone. Additionally, we spoke to several other individuals in
group settings (e.g., APTI staff members, JTC members, members of the MARAMA
Regional Consortium) about the Air Pollution Training program.
An experienced Hay Group interviewer conducted all interviews using the standardized
protocol. The interviews focused on:
• Gaining a better understanding of the practices, procedures and processes used by
APTI to design and deliver air pollution training; and
• Obtaining opinions on the effectiveness of these practices/procedures/processes and
ways to improve the current training program.
We also used these interviews to identify potential organizations to participate in the
benchmarking study that will be conducted in the late July/August time frame. Extensive
notes were taken during each 30 to 60 minute interview. Information collected from the
interviews was reviewed, and we identified common themes (e.g., many interviewees
indicated that "the training materials are outdated"). These themes are presented
throughout this report.
Review key documents. In addition to conducting subject matter expert interviews, Hay
reviewed several documents, web sites, and APTI training materials as part of the
evaluation of the current APTI training program. More specifically, we reviewed the
following:
• APTI web site (e.g., course schedules, training providers, APTFs mission, course
registration)
• Site Coordinators Resource Center web site
• EOG FY03 Highlights and FY04 Midyear Accomplishments
• Training Needs Assessment Survey and results
• Training evaluation instrument and results (i.e., feedback from students about
courses)
• Training materials (self-instructional workbooks, web-based training courses,
satellite broadcasts)
• Statistics on training course attendance and certificates issued
HayGroup Page 36
-------
EPA: Results of Benchmarking Study
Report Overview
In this report, we summarize our findings from the subject matter expert interviews and
document/web site review in an effort to provide an assessment of the current training
program. We provide an evaluation of each of the following aspects of the Air Pollution
Training program:
• Program Mission and Direction
• Training Needs Assessment Process
• Course Content
• Effectiveness of Training Delivery Methods (classroom courses, satellite
broadcasts, on-line interactive courses, self-instructional training)
• Training Evaluation Process
When describing our findings, we first provide a short overview of what the current APTI
program looks like. We then present a summary of interviewee opinions about the
current APTI training program. Finally, we provide options, identified by interviewees,
for improving the APTI training program.
We conclude the report with a discussion of our main conclusions about the state of the
current Air Pollution Training program and next steps in the benchmarking study.
HayGroup Page 37
-------
EPA: Results of Benchmarking Study
//. Program Mission and Direction
Many interviewees praise APTI on its classroom training courses, particularly those with
a laboratory component; the breadth of courses provided; and its efforts to create a
blended approach to training in response to the current budget and travel restrictions
faced by state and local agencies. However, some believe that APTI is no longer the
premier training group that it was in the past and that it is not currently meeting the needs
of state and local agencies.
Several interviewees feel that training is not a priority at EPA and that EPA needs to
recommit to its original function - training the people who do the "nuts and bolts" work
at state and local agencies (i.e., air professionals). The perception is that APTI focuses
too much on outreach and not enough on providing technical training to air professionals.
As one interviewee put it, "EPA puts good lip service to training but doesn't put the
dollars into training." Additionally, APTI has dramatically reduced its staff over time.
As a result, some of the regional consortiums (e.g., NESCAM, MARAMA) have
established their own training institutes to supplement APTI training. Some big states no
longer rely on APTI as a training provider. Additionally, some feel that APTI is
currently too focused on providing air pollution training internationally. Interviewees
believe that there is a need for APTI to take care of the state and local agencies before
expanding its training to other countries.
It should be noted that many interviewees commented that APTI leadership (particularly
Lourdes Morales) is very committed to providing the best training to state and local
agencies. Additionally, interviewees recognize that APTI has been faced with budget
cuts over the past few years, which hinder its ability to satisfy all of the needs of state and
local agencies.
One interviewee recommended that EPA create one group (versus separate groups, such
as APTI) that is responsible for training across EPA. This training group would get one
budget and be held responsible for delivering all EPA training (e.g., air quality, water,
land).
HayGroup Page 38
-------
EPA: Results of Benchmarking Study
///. Training Needs Assessment Process
EPA contracts with SYSTANI, Inc. to assist in administering a bi-annual Training Needs
Assessment Survey. The last survey was completed in March 2004 with data collected in
2003 for projecting training needs into FY2005/2006. The data are collected from
members of the State and Territorial Air Pollution Program Administrators (STAPPA)
and the Association of Local Air Pollution Control Officials (ALAPCO). The training
needs survey is designed, administered and reported with heavy involvement from the
Joint Training Committee (JTC).
The survey has two primary purposes: 1) to assess training needs related to currently
available courses and 2) to assess future training needs. The survey covers only
classroom-based courses, including those provided by institutions other than APTI (e.g.,
NETI, CARS, and RACC). A total of 88 agencies completed the FY 2005/2006 Training
Needs Assessment Survey. Thirty-eight responses were from state agencies, 46 were
from local agencies, and four were from tribal agencies. Responses are also broken down
by regional consortium (e.g., CENSARA, LADCO, MARAMA). The majority of the
respondents work as training coordinators or managers.
For each classroom course, respondents are asked to project how many people from their
areas will be attending if the course is given "In the State," "In the Region," or "Outside
of the Region." Twenty-nine classroom-based courses tied for the top ten ranked courses
in terms of prospective number of attendees in FY2005/2006. Of these courses:
• 15 were APTI courses;
• 10 were CARB courses;
• 3 were RACC courses; and
• 1 was a NETI course.
This list of training providers represents a significant change from the last survey
administered (FY2002/2003 report administered in 2001). At that time, APTI courses
comprised all of the top ten ranked courses in terms of prospective numbers of attendees.
As can be seen above, in the most recent survey, CARB RACC, and NETI had courses
ranked in the top ten (in terms of prospective number of attendees in FY2005/2006).
Of the 13 potential new training topics that appeared in the top ten list, nine of the topics
were also listed in the FY2002/2003 survey results. Twenty respondents specified
"Other" potential new training topics that were not included on the pick list.
Projected attendance for courses if offered "In the Region" is down by approximately
63% from the FY2002/2003 survey results. Projected attendance for classroom courses if
held "Outside of the Region" is down by approximately 88% from two years ago.
Similar results were found for staff projected to attend potential classroom courses.
Attendance for potential classroom courses if offered "In the Region" is down by
HayGroup Page 39
-------
EPA: Results of Benchmarking Study
approximately 52% from the previous survey two years earlier. Attendance for potential
classroom courses if held "Outside of the Region" is down by approximately 82% from
two years ago.
The Training Needs Assessment Survey also contains a question related to the factors
that influence sending staff to training. In other words, what factors would either
discourage or encourage managers from sending staff to training. Respondents indicated
that the biggest factors that would discourage them from sending staff to training are
costs, travel restrictions, workload, and staff shortages. Factors that would encourage
them to send staff to training include course relevance, staff turnover, and staff
development.
At the end of the Training Needs Assessment Survey, respondents are asked whether they
have any additional comments or suggestions related to their agencies' training needs.
Reponses cover several different areas and are bulleted below:
• Greater outreach by APTI is required to market and publicize training courses.
• Agencies need to be given at least 30 days notice that a classroom training course
is coming up if they are to have enough time to process travel requests. Some
agencies require up to 6 weeks.
• Attendance at classroom training is largely determined by budget. Training
attendance must be prioritized based on the relevance of the course to the work
being carried out in the job. Training methods should take into consideration
budget/travel restrictions.
• Workload also dictates whether or not staff can take the time to attend classroom
training courses. Budget restrictions have resulted in fewer people to do the
work.
• More use of self-training is needed, especially interactive web-based methods.
Current Views of Training Needs Assessment
Few interviewees commented on the effectiveness of the Training Needs Assessment
process. Those who did comment believe that, in general, the needs assessment is a
useful tool for forecasting the future training needs of air professionals. They view the
needs assessment as a snapshot in time and as a good technique for identifying trends and
peaks (e.g., when a large group of people will need a basic air pollution training course).
As one interviewee put it, "the needs assessment is a great tool and an effective way for
making training needs known."
HayGroup Page 40
-------
EPA: Results of Benchmarking Study
However, the following aspects of the needs assessment process were identified as areas
for improvement:
Other feedback mechanisms are used to supplement the Training Needs Assessment
Survey. Some interviewees indicated that in addition to the Training Needs Assessment
Survey, they solicit other feedback to make decisions about course content and delivery
methods. For example, one interviewee reported that he/she conducts interviews with
directors who run state and local agencies and with training providers to determine the
training needs of the consortium. This finding may indicate that the current Training
Needs Assessment Survey is not broad enough to provide comprehensive data for
decision-making about future training content and delivery methods.
Not enough is done with needs assessment results. Some interviewees questioned
whether APTI uses the results of the Training Needs Assessment Survey in its internal
decision-making regarding course content. As one interviewee put it, "I am not sure if
APTI is using it (the needs assessment) for anything more than scheduling based on the
'number of staff likely to attend training' question." Interviewees believe that other
organizations (e.g., CARB) are using the needs assessment results to develop new
training topics/courses.
Some decisions about training are not well understood. Some interviewees (particularly
members of the JTC) indicated that they are not always clear about how decisions
regarding training content are made by APTI. As one interviewee put it, "we sometimes
wonder where some of the training comes from and why APTI picked a certain course or
satellite broadcast."
Options for Improvement:
Interviewees provided some suggestions for improving the current needs assessment
process.
1. Include a skills assessment in the Training Needs Assessment Survey in addition
to projecting the numbers of attendees. For example, have respondents assess
their group's overall skill level related to each Potential New Training Topic.
This information would help course developers determine at what level
(beginning, intermediate, or advanced) training should be targeted.
2. APTI should clearly communicate what changes to training are being made as a
result of the needs assessment survey and link the results to course development
efforts.
3. Expand the needs assessment beyond classroom courses. For example, use the
assessment to attempt to gauge what types of satellite and web-based courses will
be needed by air professionals in the future.
HayGroup Page 41
-------
EPA: Results of Benchmarking Study
IV. Course Content
APTI currently develops few new courses from scratch each year (0-2 new classroom
courses per year). The focus is on updating current courses (which is a key concern
raised by interviewees) and converting course material to be delivered in a different
medium, such as the web. In 2003, APTI updated four courses (e.g., Combustion
Evaluation, Principles and Practices of Air Pollution Control). All development and
updating of APTI courses is done by contractors with insight from EPA subject matter
experts.
Current Views of Course Content
We consistently heard from interviewees that in cases where APTI course content is up-
to-date, the content itself is very good. APTI courses are considered to be comprehensive
and cover a wide breadth of subject matter.
However, some interviewees stated that they prefer attending CARB, NETI, or RACC
courses over APTI courses because courses provided by these agencies are consistently
more up to date, more specialized, and have more knowledgeable instructors than APTI
courses. Additionally, many consortia members are developing their own courses over
time to supplement APTI courses. As described below, interviewees identified several
opportunities for improving APTI course content.
Need to update course materials. By far the most prevalent comment regarding course
content and design is that many APTI courses are in need of updating to the point of
being obsolete, and APTI is not updating these courses fast enough. For example, some
interviewees believe that the Introduction to Air Pollutants course is so outdated that it is
not even worth attending. Some courses have not been updated for over 15 years.
Additionally, some APTI course materials contain inaccurate data. Interviewees also
commented that in the course evaluations, students complain about course materials more
than anything else. The FY2005/2006 Training Needs Assessment also reveals that many
students are disappointed with the quality of APTI courses and believe that many courses
are out-of-date.
Because APTI course materials are out-of-date, instructors often must update the course
material prior to delivering the course. As one participant put it, "they (EPA) expect
industry to be up to date; I think it is reasonable that they keep themselves up to date."
Some interviewees believe that updating existing courses should be a priority over
developing new courses.
Outdated course content is also an issue for paper-based and .pdf on-line self-
instructional courses. That is, on-line .pdf courses are not looked upon favorably by
those interviewed primarily because course materials are out of date. Interviewees
indicated that if course materials are old and contain out-of-date or incorrect information,
then it could be detrimental to make them easily accessible through the web. As one
interviewee put it, "this serves to propagate the inaccuracies."
HayGroup Page 42
-------
EPA: Results of Benchmarking Study
One interviewee stated that outdated APTI course materials actually influence the
selection of contractors (instructors) in the sense that contractors who have taken their
own time to update APTI course materials will be selected over contractors who teach the
course content as it is. Interviewees reported that APTI courses have to be modified
before they are delivered and that these modifications require a significant amount of
instructor resources. Related to this, some individuals who have updated APTI courses
out of necessity are unwilling to share their updates with APTI as they see the revised
course materials as their own intellectual property. One implication of the need to update
course content is that there is a lack of consistency in how APTI courses are delivered
(i.e., course instructors modify the course content in different ways).
Other interviewee comments related to the need to update APTI course content include:
• There is a large need for introductory courses to be updated because of staff
turnover in state and local agencies due to retirement.
• Policies and case laws change, so training must be modified to account for these
changes.
• In addition to the need for course content to be updated, some interviewees
believe that laboratory equipment/sampling equipment needs to be updated.
• There is a need to update lab books used in classroom training (e.g., to reflect new
instruments).
It should be noted that many interviewees recognize that budget restrictions limit the
number of course updates that EPA can handle.
Some course material is of poor quality. In addition to needing to be updated, the overall
quality of the course materials is viewed as being poor. Booklets are sent out that are
difficult to read; pages are missing, upside down, or out of order; materials arrive late; or
the wrong materials are sent out. Interviewees believe that basic quality assurance is not
carried out on a consistent basis. Some interviewees reported that it is frustrating to
either have to fix the same typos every time the materials are received or continue to
receive materials with the same errors after repeatedly reporting them to EPA. As one
interviewee put it, "the content may not be out of date, but it looks out of date."
There is overlap in course content. Course content is also seen by some interviewees as
overlapping too much in certain courses. Supervisors who have sent their employees to
training indicated that their students complain that they sometimes take a course that is
too similar to another course they have already taken. The FY2005/2006 Training Needs
Assessment also found that many students believe that there is too much overlap in
content among APTI courses. Interviewees agree that with training dollars and training
time being stretched to the limit, there is little tolerance for learning about the same topic
in two or more different courses.
HayGroup Page 43
-------
EPA: Results of Benchmarking Study
There is a need for more specialized courses. Several interviewees commented that more
specialized courses need to be added to the course schedule each year. Some
interviewees perceive that EPA is not spending enough time teaching courses that have
specialized content. Additionally, APTI is not offering enough courses that are at a
higher, more strategic level. One interviewee stated that they hired contractors to
develop a specialized course because the need was not being met by APTI.
Incorporate more real-world applications to training content. Some of those interviewed
believe that while the APTI courses offer good technical information, they do not always
cover how the information relates to industry or the "real world". Some courses focus
too much on theory and not enough on practical applications in the field. Many APTI
courses give air professionals background information but not the process skills they need
to do their jobs (although, lab courses do develop these types of skills). As one course
instructor who was interviewed put it, "the course needs to tell people what they need to
know to do their jobs in the field." One supervisor also stated that "meetings put on by
local groups/associations are more meaningful than APTI courses because there are
people attending from industry; you get more dialogue, different perspectives."
Need for more courses and better scheduling. Some interviewees indicated that APTI
needs to make more courses available each year, and that the number of courses APTI
provides has declined over the past few years (due to budget restrictions). Additionally,
comments were made about the importance of course timing and scheduling, and that
courses should be staggered throughout the year.
Options for Improving Course Content
Interviewee recommendations for improving APTI course content are summarized below.
1. Provide clear direction about the process for updating course content. Questions
that should be addressed by EPA include:
a) Will EPA take the lead in updating APTI courses?
b) How will consistency be ensured if different contractors/groups are updating
the same APTI courses themselves?
c) Who owns the intellectual property when an outside group spends time
updating out-of-date APTI courses?
d) How many courses will APTI be able to update per year?
e) Who will determine which courses are updated and in what order (e.g., update
highest volume course first)?
f) What process will be used to update specific course content?
g) Given the fact that many APTI courses need to be updated, should certain
courses be "shelved" as opposed to having inaccurate information going out
over the web site?
h) Should APTI turn over certain outdated courses to other institutions (e.g.,
RACC)?
HayGroup Page 44
-------
EPA: Results of Benchmarking Study
i) What is the process for certifying an outside course? Is there a standard
process?
2. Put into place regular review cycles to keep courses up to date. For example,
review course content, objectives, and resources on a three-year review cycle.
Ensure that course materials are relevant to what is going on in today's regulatory
world and incorporate the use of modern technology (e.g., some sampling
equipment has changed drastically).
3. At a minimum, there should be a process for checking course materials before
they are sent out. Missing pages, upside down pages, and pages out of order
should be eliminated. When typos are found in course materials, they should be
corrected before being sent out again. Interviewees feel strongly that APTI
should take responsibility for correcting errors in its courses.
4. Review and redesign courses to minimize content overlap. However, given the
need to update course content, this is likely not a big priority. A better alternative
may be to review course overviews and objectives with the goal that prospective
course participants can clearly see whether or not the material covers topics they
have already learned. This will enable participants to make a judgment as to
whether they should attend the course or download the materials from the web.
5. Review the look and feel of the course content. Interviewees suggested updating
the course content to be much more visually oriented by adding pictures and
making the science aspect much more interactive by putting courses on the web.
6. Centralize the development of air pollution training (e.g., one interviewee
indicated that there are two agencies that are spending time and resources
developing the same course separately). EPA should provide more centralized
support in course development.
HayGroup Page 45
-------
EPA: Results of Benchmarking Study
V. Course Delivery
There are four types of courses provided by APTI: classroom, satellite, on-line
interactive, and self-instructional (.pdf on the web and paper-based). In the past few
years, APTI has put an emphasis on moving more training to an on-line and satellite
format (distance learning techniques) to meet the needs of a largely dispersed customer
population faced with travel and budget restrictions. Additionally, on-line and satellite
training allow state and local agencies to train new hires and others who require air
pollution training in a timely manner (just-in-time training).
For the purposes of this report, on-line courses will refer to web-based interactive courses
only; self-instructional courses will refer to hard copy manuals and .pdf files that are
available for download on-line but include no interaction.
In the remainder of this section, we discuss and evaluate each of the four training delivery
methods.
Classroom Courses
Currently, classroom courses are those which are presented live by recognized
professionals (e.g., consultants, university faculty) and held either at an Area Training
Center, a related university, or on-site at a state or local agency. APTI is actively
involved in updating and developing courses. In 2003, four APTI courses were updated
and thirty-seven courses were delivered nationwide. Students can receive certification for
courses by completing class assignments, passing a final exam, attending and
participating in class sessions, and submitting a course evaluation.
Current Views about Classroom Courses
By and large, most people interviewed prefer the classroom delivery vehicle when time
and travel is not an issue. This perception is held by interviewees, as well as by students
who have completed APTI courses (as indicated by student evaluations and feedback
from supervisors who have sent their employees to training). Student evaluations of
classroom training courses tend to be very favorable, and many students feel that the
course materials will be useful to their jobs. Classroom training allows for the most
interactions with the instructor, personal attention to questions and students learning from
one another. Classroom training is seen as the most effective delivery method, especially
for higher level technical classes and those with a laboratory component.
However, most interviewees agree that classroom learning is not appropriate for all APTI
courses, especially in light of recent budget cuts. Despite the fact that classroom training
was the preferred method in the past, most agencies are under budget constraints and
travel restrictions that prevent them from getting to the classes. Therefore, agency
training professionals recognize the need to have a complement of vehicles, although
some interviewees believe that classroom courses should be mandatory for classes with
HayGroup Page 46
-------
EPA: Results of Benchmarking Study
laboratory components. Additionally, many interviewees indicated that the quality of
APTI classroom courses vary greatly and primarily depend on the instructors.
A summary of comments made by interviewees and students (in response to student
evaluation forms) regarding the APTI classroom courses is provided below.
Overall, classroom is best of the four delivery methods. While there is a need for a
variety of training vehicles, classroom delivery of courses is viewed as the most effective.
One interviewee said, "students are completely dedicated to learning in that environment
and the diversity of students allows for different perspectives and better learning."
However, most interviewees indicated that while classroom interaction is effective, it
should not be used for all courses. The perception is that while it is a good technique, the
classroom is not the most efficient way to deliver training, especially for introductory
classes or classes that need to be attended by large numbers of people.
Laboratory component is very valuable. There is agreement among interviewees that the
laboratory component (included as part of some APTI classroom training courses) is very
effective as a learning tool. Students are given the opportunity to actually learn the skills
that they need to do their jobs effectively. Student evaluations of laboratory training
courses echo this finding; students feel that the laboratory component is the best part of
the classroom training experience. Students believe that the lab helps them understand
the course materials, and they like the hands-on learning component of the classroom
training. As one student indicated in the student evaluation, "the best part of the course
was hands-on modeling and the exercises."
Students want more practical applications in non-laboratory courses. Some interviewees
commented that lecture only courses need to be supplemented with more "real world"
examples to emphasize how, for example, sampling techniques, equipment, mathematical
equations, will actually be used in the field. Student feedback from course evaluations
also suggests that they want more hands-on applications. Some courses focus too much
on theory and "put people to sleep." Instead students want to know "what do I do with
this information?" "The course needs to tell people what they need to know to do their
jobs in the field."
Mixed opinions about the effectiveness of instructors. While the classroom environment
allows people to remain engaged at all times, the effectiveness of the instructor can have
a huge effect on whether the training is viewed as worthwhile. Some interviewees feel
that there needs to be a process for ensuring that instructors are effective (e.g., monitoring
student feedback on the effectiveness of instructors, observing instructors delivering
classroom courses). Interviewees view instructors as having a thorough grasp of the
technical information but do not see some instructors as effective in delivering the
material. The general feeling is that the best instructors are those who have "worn all
hats" (for example, worked for EPA, worked for private industry in a monitoring capacity
and worked on the design or update of APTI courses). It should be noted that, overall,
the student evaluation feedback that we reviewed was very favorable regarding
HayGroup Page 47
-------
EPA: Results of Benchmarking Study
instructors. Instructors are viewed as dynamic and interesting, and students like the
interactions among the instructors.
Lack of consistency in how classroom courses are delivered. Interviewees believe that
classroom courses are delivered inconsistently when APTI materials are out-of-date
because instructors must supplement APTI materials with their own. Inconsistencies can
also occur when instructors travel to states that need the training delivered on-site. While
traveling instructors are viewed positively in light of the travel restrictions that most
states are under, the travel can lead to a less integrated training curriculum and
inconsistent learnings.
Options for Improving the Classroom Courses
Interviewees had several recommendations for improving APTI classroom courses.
1. Consider a "train-the-trainer" approach so that there will be people on site at the
agencies that are able to teach the course materials locally.
2. Increase the number of real world examples and hands-on applications included in
classroom courses that are primarily in a lecture format. Supplement the lecture with
practical illustrations when applicable.
3. Increase the prevalence of group exercises in order to expose attendees to one another
and their respective issues and diverse perspectives. This practice can increase group
learning for the class.
4. Ensure that instructors are not only knowledgeable about the materials but also skilled
at teaching. Use course instructors who not only have educational experience but
who have "done it in the field."
5. Make sure classes (including supplemental materials) are consistent across all
courses, and then take the instructors on the road (one interviewee indicated that
"EPA should travel more to the states to deliver training").
6. Shorten the length of classroom courses (e.g., 1 day training as opposed to 3 to 5
days). Shorter courses reduce costs and time away from the job.
7. Reserve classroom courses for those having laboratory assignments and covering
more technical, specialized topics.
8. Ensure that classroom locations are easily accessible to most of the country.
Satellite Broadcasts
APTI uses its Air Pollution Distance Learning Network to deliver satellite broadcasts to
air professionals (and other interested parties) throughout the country in a cost effective
manner. The satellite broadcast is used to deliver informational broadcasts (e.g., to keep
air professionals up to date on new regulations), as well as technical telecourses. For
example, APTI course 427 (Combustion Evaluation) has been converted to a satellite
course to be delivered in 4-hour blocks over a four-day period. Some satellite broadcasts
allow participants to fax in questions and selected questions are answered by the
HayGroup Page 48
-------
EPA: Results of Benchmarking Study
presenters. Participants are not required to register for satellite broadcasts, except for
300- and 400-level telecourses.
In 2003, APTI delivered a total of 14 broadcasts covering 66.5 broadcast hours.
Additionally, several broadcasts have already been delivered or are in development for
2004.
There are approximately 100 downlink sites throughout the country. Site coordinators
are responsible for advertising upcoming broadcasts, arranging the room where the
broadcast will take place, and other logistical duties. Videotapes of past satellite courses
can also be used as self-instructional courses. A limited number of broadcasts are
simulcast and can be viewed on a PC.
Current Views about the Satellite Broadcast
There are mixed opinions about the appropriateness of using the satellite broadcast as a
training delivery method. Some interview participants do not like the satellite broadcast
as a training delivery method. This camp believes that APTI has placed too much
emphasis, and budget, on the delivery of training via satellite broadcasts at the expense of
other delivery methods (e.g., many classroom courses have not been updated). As one
interviewee indicated, "the direction towards satellite is a complete waste of time and
resources."
Most interviewees, however, recognize that state and local agencies have budget
limitations, travel restrictions, and the need to minimize employee time away from the
job, making it necessary for APTI to deliver training in other ways besides the traditional
classroom approach. This finding is echoed by the 2004 Training Needs Assessment
through which participants urged APTI to consider budget/travel restrictions when
making decisions about training delivery methods. Satellite broadcasts are seen as one
cost-effective way to deliver training to a large, geographically dispersed audience.
Another benefit of the satellite broadcast is that it can provide timely and consistent
information to air professionals. For example, satellite broadcasts allow 300- and 400-
level training courses to be delivered in a consistent manner, which is not always the case
with classroom training. Additionally, because satellite broadcasts are taped and can be
re-shown in a video format at any time, state and local agencies can provide just-in-time
training to new hires.
Although many interviewees recognize the potential value of satellite broadcasts, there is
agreement that the current APTI satellite broadcast delivery method is not effective and is
in need of improvement. Many satellite broadcasts play to a very small audience or no
audience at all. Proponents of the satellite training method feel that APTI should focus
its limited resources on improving its satellite broadcasts so that air professionals have
easy access to APTI training (particularly those who are impacted by travel restrictions
and can not attend classroom training). It should be noted that some interviewees
indicated that the satellite broadcasts have improved over the past few years (e.g.,
instructors are more effective, satellite productions are better).
HayGroup Page 49
-------
EPA: Results of Benchmarking Study
The following is a summary of the comments that were made about APTI satellite
broadcasts:
Speakers are not effective presenters. Many interviewees indicated that the speakers who
deliver the satellite broadcast are often dull (speak in a monotone voice) and do not
present the information well. Participants do not find many of these speakers engaging,
and have difficulty paying attention to broadcasts that last for two hours or more (and
thus, do not get much out of the broadcast). It is particularly important to have an
engaging speaker because many of the satellite broadcasts are presented in a lecture
format. Participants in the satellite broadcast often end up reading the text that is
provided because they do not learn enough from the speaker (due to the poor presentation
style). As one interviewee put it, "the material was good but the presenter got in the
way." Another interviewee stated, "you need to pick the right people who can
communicate and work the medium to deliver the broadcast."
It should be noted that some interviewees believe that APTI has improved in this area and
is now using some of its best instructors to deliver the satellite broadcasts.
Limited opportunity for interaction. Perhaps the biggest complaint about the satellite
broadcast is that it provides limited opportunity for student interaction (e.g., discussion
among participants, interactions with the instructors). Most interviewees believe that it is
very difficult to sit through a long television broadcast in a lecture format without
opportunities for interaction. Furthermore, a lecture-style delivery method where
students passively watch the television (versus one that facilitates student interaction) is
not an effective adult learning technique (i.e., not effective for retaining information,
particularly of a technical nature). One interviewee indicated that "the satellite can be a
bit boring, not because of the instructors but because of the mode."
Even when the broadcasts allow for participants to fax in questions, there is the
perception that few participants actually get their questions answered (e.g., speakers are
often reluctant to answer some questions, such as about policy issues). As one
interviewee put it, "in theory, you can ask questions during the satellite broadcast, but in
application, you can't." Others feel that participants in a satellite broadcast are less likely
to fax in questions than they would be to ask a question in a classroom setting.
Additionally, the process of faxing in questions is not viewed as being "real time."
Length of broadcasts are too long. Some believe that the satellite broadcasts are too long
(considering that there is little interaction and many speakers are not dynamic). For
example, one interviewee indicated that it is not effective to do four-day satellite courses
(which occurs with 400-level courses) because "you will bore students."
Location of satellite link is not always convenient. Some interviewees reported that the
location of the satellite link may limit attendance for some air professionals. For
example, some air professionals have to travel to the downlink location (in a time when
state and local agencies have travel restrictions).
HayGroup Page 50
-------
EPA: Results of Benchmarking Study
Topics do not meet customer needs. Some interviewees commented that the information
covered by the satellite broadcasts does not meet the needs of air professionals. For
example, many broadcasts focus on national topics versus local issues. As one
interviewee put it, "one broadcast focused on air toxics, but we don't have toxics."
Another interviewee believes that the satellite broadcasts are not really training but PR
for EPA. One interviewee stated that "the broadcasts are good touchy-feely stuff versus
training." Another commented that the satellite is sometimes used to provide information
about topics that have nothing to do with air pollution.
Satellite was down for several months. Many interviewees expressed frustration that
EPA lost its satellite for six months. They also indicated that EPA did not provide
adequate communication when the satellite was down. As a result, many site
coordinators lost interest in the satellite as a training delivery method. While the satellite
was down for six months, for two of those six months, APDLN was able to show re-
broadcasts.
Options for Improving the Satellite Broadcasts
Interviewees provided several recommendations for improving the satellite broadcasts
delivered by APTI:
1. Use the satellite broadcasts primarily to:
a. Provide information about subjects of interest to a broad audience (not just
air professionals)
b. Deliver refresher training and less technical courses
c. Show a panel discussion of topics covered in other courses
2. Enhance the satellite technology so that air professionals can view the satellite
broadcasts from their PC (if not already done)
3. Do more marketing and outreach about satellite broadcasts
4. Incorporate opportunities for interaction into the satellite broadcast format, such
as:
a. Include a small group exercise facilitated via conference call by EPA
representatives
b. Include a workbook that is referred to and utilized during the course
c. Use a state/local agency representative to facilitate discussion and answer
questions during the sessions (with support from an APTI facilitation
guide)
d. Include a studio audience that asks questions of the presenters as part of
the broadcast
5. Require read ahead materials to supplement the broadcast (similar to self-
instructional course materials).
HayGroup Page 51
-------
EPA: Results of Benchmarking Study
6. Shorten the length of broadcasts (e.g., one hour) to increase attendance and the
potential for learning.
On-line Interactive Courses
In the past few years, APTI has placed a greater emphasis on moving training to a web-
based format. Both classroom and self-instructional courses are being converted to this
medium. Results of the FY2005/2006 Training Needs Assessment highlighted the
importance of providing students with opportunities for self-training, especially through
interactive web-based methods.
For FY2004, APTI will select the last group of courses as virtual classroom courses.
These courses should be operational by October 2004. Several other courses being
developed as web-based courses include: Emissions Inventory Introduction; Orientation
to Air Pollution (Orientation to Air Pollution is on the web now); Title V Citizen
Training; and Ozone and Your Patient's Health. Examples of on-line interactive courses
currently used by APTI include Risk Based Air Toxics; Air Pollution Control
Orientation; and Introduction to Air Pollution Control. On-line, interactive courses do
not require registration. There is a final exam for web-based courses and participants can
now receive CEU credits.
Current Views about On-line Interactive Courses
Although APTI considers two types of courses to be web-based courses - .pdf files of
paper materials and interactive web-based course materials - we will focus on only the
interactive courses in this section (the .pdf courses are reviewed in the next section, Self-
Instructional Courses).
Interviewees consider APTFs on-line interactive courses to be very good, especially for
courses covering introductory material. The on-line interactive medium is viewed as the
best medium (better than the satellite broadcasts) for mass distribution of information and
teaching of foundational materials. Interviewees believe that it is very easy to access on-
line training courses (typically at one's PC), which is important due to budget and travel
restrictions. On-line courses are also viewed favorably because: 1) students have the
ability to go at their own pace; 2) technical information is presented in a consistent
manner (which is not always the case with classroom courses); and 3) there is an element
of interaction which holds participants' attention. Classes that are currently on-line and
interactive are also viewed as being more up-to-date (e.g., than classroom or self-
instructional, paper-based courses).
The on-line training delivery method is an area that interviewees view as having the most
potential to make a positive impact for the least amount of money (i.e., a cost effective
way to enhance the effectiveness of APTI training). Most interviewees agree that more
on-line, self-instructional courses needed to be made available. However, the consensus
is that these types of training must be more interactive if they are to be effective.
HayGroup Page 52
-------
EPA: Results of Benchmarking Study
Below is a summary of the specific comments that were made about APTFs on-line
interactive courses:
On-line interactive courses are working. Most interviewees had positive comments to
make about the APTI courses that are on-line and interactive. One interviewee
commented that, "APTI has added interactive computer courses which are very good;
most students provide feedback that the courses are good." Another interviewee stated,
"APTI is going in the right direction with computer-based courses."
On-line interactive courses work well for introductory materials. Interviewees
mentioned multiple times that the accessibility of these types of classes was great for
introductory classes or those in which the EPA needs to distribute information to a large
number of people. One interviewee indicated that the "virtual classroom is great but
should be used for introductory courses only. This allows everyone to have a certain
level of background." "This is a priority because it is most cost-effective and [the
student] can stop and go at their own pace." However, some interviewees questioned
whether web-based courses are as useful for intermediate or advanced course content as
they are for introductory course content.
There is a need to incorporate more interaction into on-line courses. As with the satellite
broadcast delivery method, interviewees believe that one way to improve the APTI web-
based courses is to add more opportunities for student interaction (e.g., through
simulations, chat rooms). As indicated previously, training methods that facilitate student
interaction are more effective for retaining information delivered in training courses than
are methods without an interactive component.
Options for Improving the On-line Interactive Courses
Interviewees provided several recommendations for improving on-line interactive
courses.
1. Use on-line interactive courses for introductory classes and classes that are taken
most frequently.
2. Add additional opportunities for interaction to the on-line courses including regularly
scheduled "chat rooms" where participants can have their questions answered and
interact with instructors. This could be done at designated times in order for students
to ask questions and discuss issues simultaneously.
3. Incorporate interactive video conferencing with the on-line classes.
4. Use more dynamic authoring tools for web-based training.
5. Add more computer-based simulation in order to demonstrate the equipment that the
students will have to use in the field.
Self-instructional Courses (Non-Interactive)
APTI currently has two forms of self-instructional (non-interactive) courses, a paper-
based course where APTI mails out the hard copies of materials in response to student
HayGroup Page 53
-------
EPA: Results of Benchmarking Study
requests, and an on-line version where the same materials are downloaded onto the
Internet in a .pdf format. APTI is currently committed to putting all hard copy materials
on-line, so mailings of self-instructional workbooks will not continue in the future.
In 2003, 1,583 students registered for self-instructional courses. It is expected that this
number will grow in the next year due to continued budget restraints.
Current Views about Self-instructional Courses
Interviewees have mixed opinions regarding the APTI self-instructional delivery method.
When course materials are out of date, self-instructional courses are viewed very
negatively. Additionally, some interviewees believe that these courses have minimal
benefit because there is no interaction, and no opportunity to have direct communication
with instructors or other students. This camp sees self-instructional training as simply
"feeding information" to students. As a result, many students do not ever finish the self-
instructional courses and probably do not learn much from these courses.
Some interviewees, however, believe that the self-instructional method of training is
valuable for certain types of classes and students. One interviewee commented that
"these courses are good for students who are just starting to learn about air quality. The
materials provide a good, basic understanding." This person also noted, however, that
the materials need to be updated first before putting them on-line. Another interviewee
mentioned that many of the engineers taking these classes are visual learners, so they like
the self-instructional materials. Most interviewees believe that the two self-instructional
media (paper-based and .pdf) are equivalent, although access to the .pdf files is seen as
more convenient.
Below is a summary of specific comments that were made about APTI self-instructional,
non-interactive courses (i.e., courses that are paper-based or on-line in a .pdf format):
Participants want more interactions. Interviewees indicated that students would like their
questions answered while they are taking the self-instructional course, not after. Students
also desire more practice problems, and would like to see what questions they missed on
the final exam rather than just whether they passed. In this way, they can learn from their
mistakes. One supervisor who has sent employees to training questioned whether any
lasting learning had taken place although the supervisor's staff had passed their exams.
Some interviewees indicate that adding more visual cues, and including interaction would
help students retain the course information.
Course materials are not always reviewed by students. Some interviewees state that
those who download the courses off the web rarely read all (or in some cases, any) of the
course material. Instead they focus on responding to test questions. One interviewee
indicated, "this problem is especially prevalent with courses in a .pdf format instead of
the interactive web-site."
Supervisors want more controls around testing. One interviewee (a supervisor who
sends employees to APTI training) is having problems with staff simply completing the
HayGroup Page 54
-------
EPA: Results of Benchmarking Study
open book test and not reading through the materials when .pdf courses are downloaded
from the web.
Options for Improving the Self-Instructional Courses
Interviewee recommendations for improving self-instructional courses are summarized
below.
1. Update the materials before putting them on-line.
2. Add some visuals to the materials in order to make them more appealing and
memorable.
3. Do not make the test available on the web or have more controls built into the
process. Do not send out the test until after participants notify APTI that they have
read through the self-instructional materials.
4. Develop one comprehensive manual that users can reference in the future regarding
all foundational air quality materials.
5. Put all hard copy manuals on-line for easier accessibility.
HayGroup Page 55
-------
EPA: Results of Benchmarking Study
VI. Training Evaluation Process
APTI uses several different forms for evaluating its training courses. Typically, students
are asked to complete these evaluations at the end of the course (or satellite broadcast).
Many classroom courses also give students a pretest before the course begins to assess
their level of knowledge and a posttest at the end of the course to see how much the
students learned.
Some of the supervisors we interviewed believe that students learn a significant amount
in APTI courses and this knowledge helps them to do their jobs more effectively. As one
supervisor put it, "APTI courses really enhance peoples' learning." However, the current
training evaluation process does not allow for an evaluation of the extent to which APTI
training actually helps air professionals to be more effective in their jobs.
Additionally, many interviewees indicated that they have developed their own surveys to
evaluate APTI training. This may suggest that the current APTI training evaluation
process is not effectively meeting stakeholder needs to evaluate and improve upon
training courses.
The comments below summarize what interviewees said about APTFs training
evaluation process, as well as the Hay Group's assessment of the training evaluation
process (based on our extensive experience in this area). It should be noted that few
interviewees actually commented on the training evaluation process.
Too much focus on end-of-course evaluations. APTI focuses its training evaluation
efforts solely on end-of-course evaluations (some call these surveys "smile surveys").
This type of evaluation provides limited information for determining the extent to which
each APTI training course achieved its ultimate objective - to improve the job
performance of air professionals.
Evaluation form does not provide enough useful information for improving courses.
The current course evaluations do not provide enough information for understanding why
the training course was or was not effective, whether the delivery method is the best
medium for delivering the training, and more importantly, how to improve APTI training
courses. As one interviewee indicated, "the evaluation focuses mostly on the instructors
(are they boring or not) and classroom logistics (coffee or snacks)."
Feedback is rarely used to improve courses. Many interviewees feel that APTI does not
take feedback about its training program seriously, and rarely makes improvements to its
training based on this feedback. When feedback about training courses is given to APTI,
the typical response is "we will try to do better next time."
HayGroup Page 56
-------
EPA: Results of Benchmarking Study
Options for Improving the Training Evaluation Process
Interviewees provided a few recommendations for improving the current training
evaluation process.
1. Expand the training evaluation process beyond end-of-course evaluations. In
order to truly assess the effectiveness of APTI courses, it is important to
determine the extent to which the training actually helps air professionals to do
their jobs effectively and contribute to the mission of their agency. It is critical to
collect feedback about the effectiveness of the training at different points of time
and from different sources (e.g., students themselves, supervisors, training
coordinators). Suggestions include:
a. Solicit feedback (e.g., through surveys, interviews) from supervisors of
attendees of training to determine the extent to which they believe the
training has helped their air professionals to be more effective in their
jobs.
b. Survey course participants 3 to 6 months after the training to assess the
extent to which they have improved their job performance (not just after
the course is complete)
c. Survey site coordinators to determine why satellite courses are not well
attended and how to improve them.
2. Add open-ended questions to the training evaluation form that will provide
feedback for enhancing the effectiveness of the training courses. Examples
questions are: What should we do differently and what should we do in the same
way? How would you improve the current training course?
3. Solicit feedback about web-based courses (not just satellite and classroom
courses) to assess their effectiveness and ways to improve them and make them
more interactive.
4. Review feedback from training evaluations on a regular basis and make
modifications to training courses, where appropriate. Communicate what changes
to training are being made as a result of the training evaluation and link the results
to efforts to update or redesign courses and make decisions about ways to
improve courses.
HayGroup Page 57
-------
EPA: Results of Benchmarking Study
VII. Conclusions and Next Steps
In this report, we provide an evaluation of the current Air Pollution Training program
based on data from numerous sources: 1) interviews with stakeholders who have in-depth
knowledge of the APTI training program and its effectiveness; 2) reviews of relevant
documents and web sites about the APTI training program; and 3) reviews of actual
training courses (e.g., self-instructional manuals, web-based courses, satellite broadcasts).
In subsequent reports, after we identify the best practices around training (through site
visits and benchmarking research), we will provide more insight into ways to improve
APTFs training program, and where APTI should focus its limited training resources
(how to deliver training to a large, geographically dispersed population of air
professionals in the most effective and cost efficient manner).
In summary, our findings show that there are mixed opinions about the effectiveness of
the Air Pollution Training program. An initial concern raised by the study is the extent to
which EPA (and APTI) is committed to meeting its core objective of training air quality
professionals. There is a perception among interviewees, whether it is valid or not, that
EPA does not have a commitment to providing technical training to state and local
agencies.
However, APTI was often praised for its classroom training, particularly courses with
laboratory components; its breadth of courses; and its course content (for those courses
that are up-to-date). Additionally, many interviewees believe that APTI is taking a step
in the right direction by adopting a blended approach to training - providing easy access
to courses (e.g., via web or satellite) when state and local agencies are facing budget cuts
and travel restrictions.
Stakeholders are most concerned with the quality of course materials (i.e., many course
materials are outdated and of poor quality). Additionally, there are several potential areas
for improving APTI distance learning training methods (satellite and web-based training)
that are the "wave of the future" for APTI, particularly increasing the opportunity for
interaction. There is also a need to ensure that APTI courses (particularly classroom
courses) are being delivered consistently and by instructors who have technical expertise,
practical experience in the field, and "teaching ability." Finally, stakeholders would like
to see APTI place more emphasis on reviewing and acting on feedback obtained through
the needs assessment and training evaluation process in an effort to ensure that the needs
of air professionals are being met through its training courses.
While there were numerous suggestions from the various constituents about what the
EOG (and APTI) could do to improve its effectiveness, the following were the most
prevalent and noteworthy:
• Reestablish and communicate the mission of APTI and its goals to all
stakeholders
HayGroup Page 58
-------
EPA: Results of Benchmarking Study
• Determine which courses are most important to foundational learning and update
them
• Put more course materials on-line (those that are relevant and up-to-date)
• Move toward remote, interactive classes, either through web-based tools or
satellite, especially for introductory courses
• Respond to needs assessments and course evaluations, even if to say the issue will
not be addressed at this time
• Increase the use of practical, hands-on applications for all courses
Our next step in the benchmarking study is to identify the best practices of organizations
that have been successful at delivering training to a geographically dispersed audience,
and that use innovative training delivery methods. We will be selecting potential
benchmarking partners and conducting site visits to learn about their training practices,
processes, and procedures (which will be summarized in a separate report). In the final
report, we will provide options for making APTI more successful going forward.
HayGroup Page 59
-------
EPA: Results of Benchmarking Study
Appendix B
Task 2 Report:
Best Practices of Leading Training Programs
HayGroup Page 60
-------
EPA: Results of Benchmarking Study
/. Introduction
BACKGROUND
In May 2004, the Education and Outreach Group (EOG) of the Environmental Protection
Agency (EPA) contracted with the Hay Group to conduct a Benchmarking Study of its
Air Pollution Training program. The ultimate goal of the project is to provide options for
improving the current Air Pollution Training program, and ultimately, EOG's business
performance. The following three tasks are aimed at accomplishing this goal:
> Evaluate the current Air Pollution Training program (Task 1)
> Identify best practices of leading training programs (Task 2)
> Develop a plan of action for improving the current Air Pollution Training
program (Task 3)
This report summarizes our findings from Task 2 - identify best practices of leading
training programs. The findings in this report provide a summary of the practices,
procedures and processes used by industry leaders in training. In this report, we focus on
these best practices and how the Air Pollution Training Institute (APTI) may incorporate
some of these practices into its current training program.
In the final report (Task 3), we will present our specific conclusions regarding the
direction that we believe the Air Pollution Training program should take.
METHODOLOGY
This section summarizes the methodology used to identify the best practices of industry
leaders in training and distance learning. The primary methods used to identify the best
practices were: 1) site visits with public and private sector organizations that are
recognized as industry leaders in training; and 2) literature review on innovative ways
other organizations are delivering training, particularly distance learning.
Site Visits
Hay conducted site visits with five organizations to identify industry best practices in
training, particularly distance learning. Below, we describe the methodology used to
conduct the site visits.
Select organizations for site visits and elicit participation. The first step was to select a
list of potential organizations to participate in site visits. Through reviews of published
sources and the Internet, we developed a list of organizations considered leaders in
technical training and distance learning by professional organizations (e.g., American
Society for Training and Development, ASTD; American Productivity and Quality
Center, APQC), publications (e.g., Training Magazine), and academic and professional
experts. We based our selection on awards (e.g., Training Magazine's Top 100 list,
ASTD BEST Award, Government e-learning award); accolades, participation in other
HayGroup Page 61
-------
EPA: Results of Benchmarking Study
training benchmarking studies (e.g., Society for Human Resource Management
Consortium Benchmarking Study on Training and Development), and expert
recommendations. We particularly looked for organizations that use innovative e-
learning and distance learning training delivery methods.
We then contacted each of the organizations to determine whether they would be an
appropriate benchmark organization and interested in participating in the study. We sent
potential organizations a short description of the benchmarking study and the level of
effort required to participate. We then qualified each organization through a short
conference call and cut the initial list down to a select group of organizations. Table 1
shows the five organizations that agreed to participate in the benchmarking study and our
rationale for selecting each of them as a benchmarking partner. We refer to these five
organizations throughout the study as a "benchmarking partner".
Table 1: Participating Organizations in the Site Visits
Organization
Occupational Safety
and Health
Administration
(OSHA), Office of
Training and Education
Rationale for Selection as a Benchmarking Partner
> Provides technical training to a large, geographically-dispersed
population
> Blends Web-based training and live satellite broadcasts with more
traditional classroom instruction (uses innovative distance learning
training media)
Center for Disease
Control (CDC), Public
Health Training
Network (PHTN)
> Provides technical training to a large, geographically-dispersed
population
> Provides diverse training delivery methods, including distance learning
> Has state-of-the-art training facilities
Georgia Tech
University, Distance
Learning and
Professional Education
Department (DLPE)
Trains professionals in engineering, business, and other hard sciences
Recognized as providing an exceptional distance learning program and
successfully incorporating an interactive component into distance
learning
Has state-of-the-art training facilities
SAS Institute (SAS)
> Provides extensive technical training to customers around the world
> Blends e-learning with more traditional classroom training (uses
innovative e-learning training media)
GMAC Commercial
Mortgage (GMAC),
Staff Development
Division
Delivers training primarily via e-learning methodologies, including
videoconferencing, live Webcasts, videotapes and C-ROMs/DVDs
Prepare for site visits. Hay designed a site visit protocol (based on the evaluation criteria
used in Task 1) to capture the best practices of industry leaders in training. The protocol
included an introductory paragraph to read to participants at the beginning of the site
visits and interview questions. A copy of the site visit protocol is presented in Appendix
Bl.
HayGroup
Page 62
-------
EPA: Results of Benchmarking Study
Hay worked with a point of contact at each participating organization to identify
appropriate individuals to interview about the training program. An effort was made to
interview a diverse group of individuals who have a good understanding of the training
program, its objectives and desired results, its future direction, and its overall
effectiveness. We typically spoke to 3 - 10 individuals (e.g., training director, training
content developers, graphics specialists, studio staff) from each participating
organization.
Conduct site visits. We conducted site visits with five organizations recognized as
industry leaders in training. An experienced Hay facilitator conducted all the site visits
using the standardized protocol. An EPA representative also attended all site visits. The
site visits focused on identifying best practices around:
> The needs assessment process (e.g., how decisions are made about course
content and delivery methods)
> Course design and content (e.g., process for designing/updating courses)
> Training delivery methods, with a special emphasis on distance learning
methods (e.g., live Webcasts, self-paced Web courses, satellite broadcasts)
> Strategies for incorporating an interactive component into distance learning
> Training evaluation process
During site visits, we reviewed training program documentation and materials, observed
training facilities and training programs (e.g., a live Webcast; an interactive Web-based
course), and conducted interviews with members of the training group. Each site visit
lasted two to six hours.
Extensive notes were taken during the site visits. Information collected from the site
visits was reviewed, and we identified existing training best practices. These best
practices are presented throughout this report.
Literature Review
In addition to conducting site visits, Hay reviewed several reports, articles, books and
web sites to identify best practices in e-learning and distance learning. The primary
documents/web sites that were reviewed are shown in Table 2.
HayGroup Page 63
-------
EPA: Results of Benchmarking Study
Table 2: Primary Documents/Web Sites Reviewed for the Benchmarking Study
> American Society for Training and Development (ASTD). State of the Industry
Report (2004).
> American Society for Training and Development (ASTD). State of the Industry
Report (2003).
> American Society for Training and Development. E-Learning Handbook: Best
Practices, Strategies, and Case Studies for an Emerging Field (2002).
> Bersin and Associates. Blended Learning: What Works (May 2003).
> Bersin, Josh. The Blended Learning Book: Best Practices, Proven Methodologies,
and Lessons Learned (2004).
> Daly, David, and Scott, Amy. Best Practices for Advanced Distributed Learning.
www.jointadlcolab. org.
> Galagan, Patricia A. Mission E-Possible, the Cisco E-Learning Story. Training and
Development (February 2001).
> Galvin, Tammy. Training Magazine. The 22nd Annual Industry Report (October,
2003).
> General Accounting Office. Human Capital: A Guide for Assessing Strategic
Training and Development Efforts in the Federal Government (March 2004).
> General Accounting Office. Information Technology Training: Practices of
Leading Public Sector Companies (2003).
> Hall, Brandon. Six Steps to Developing a Successful E-Learning Initiative (2001).
> Hall, Brandon and LeCavalier, Jacques. E-Learning across the Enterprise: The
Benchmarking Study of Best Practices (2000).
> Hequet, Marc. Training Magazine. The State of E-Learning (September, 2003).
> Hofman, Jennifer. Blended Learning Case Study. Learning Circuits (2001).
> Johnson, Gail. Training Magazine. Blended Learning: How to Brew the Perfect
Blend (December, 2003).
> Kilpatrick, D. (1994). Evaluating training programs: The four levels. San Francisco:
Berrett-Koehler.
> Kiser, Kim. E-learning Evangelism. Online Learning (2001).
> Singh, Henry. Building Effective Blended Learning Programs. Educational
Technology (November/December 2003).
HayGroup Page 64
-------
EPA: Results of Benchmarking Study
REPORT OVERVIEW
In this report, we summarize our findings from the site visits and literature review to
identify trends and best practices of industry leaders in training. We provide an
assessment of the training best practices in each of the following areas:
> Training Needs Assessment Process
> Course Design and Content
> Training Delivery Methods and Ways to Incorporate Interaction into Distance
Learning
> Training Evaluation Process
Summary of the findings from each of the five site visits is presented in Appendix B2.
Appendix B2 also includes a short description of the characteristics of the training groups
that participated in the site visits (e.g., size of staff, facilities, mission).
We conclude the report with a discussion of our main conclusions about best practices in
the training industry and how they may be incorporated into APTI's training program.
HayGroup Page 65
-------
EPA: Results of Benchmarking Study
//. Training Needs Assessment Process
Our research shows that conducting a needs assessment is often the first, and possibly the
most important, step in developing an effective training course or an entire training
program. Training needs assessments provide information about the types of training
courses needed by potential participants and the manner in which training should be
delivered. More specifically, training needs assessments help determine, among other
things:
> Types of training programs that participants want and need
> Gaps in current and needed course offerings and participant skill sets
> Goals and objectives for training programs
> Steps needed to achieve training objectives
> Training areas in need of additional resources
> Best methods for delivering training courses (e.g., classroom, self-paced Web,
satellite)
Our benchmarking research shows that training needs assessments are being conducted at
some level by most training programs. However, across the broad range of organizations
that conduct training programs, the overall status and effectiveness of the needs
assessments are not consistent. Some organizations conduct systematic needs
assessments for both overall training programs and individual courses, while others
appear to make somewhat haphazard attempts at determining what potential participants
want and need. What is clear, however, is that training experts agree that conducting
systematic needs assessments is an important contributor to the overall effectiveness of
training programs (the extent to which they ultimately improve trainee performance and
overall organizational effectiveness).
TYPES OF NEEDS ASSESSMENTS CONDUCTED BY INDUSTRY LEADERS
We found through our benchmarking research that there are different types of needs
assessments that training organizations conduct: 1) overall training needs assessment; 2)
market needs for a specific course; and 3) content design and delivery needs assessment.
Each of these needs assessments has different objectives and processes.
Overall training needs assessment. Overall training needs assessments are conducted by
most of the benchmarking partners and other training industry leaders. These
assessments examine the courses that current, former, and potential training participants
think they will need over a one- or two-year time period. This type of needs assessment
typically involves an annual survey of the trainee population (and sometimes, supervisors
of trainees). In the best cases, the training group (or the organization as a whole) has
identified a set of overall objectives that the trainee population should achieve over the
specified time period (e.g., employees should develop a specific set of IT skills). These
objectives then form the basis of the needs assessment. For example, a needs assessment
HayGroup Page 66
-------
EPA: Results of Benchmarking Study
may provide the trainee population with the list of objectives and ask them to list courses
they think are needed to meet the objectives (e.g., which courses will help them develop
the specific set of IT skills required for future success on the job?). Regardless of the
actual approach, overall training needs assessments attempt to determine programmatic
needs over a one- or two-year period.
Example of Overall Training Needs Assessment
SAS conducts an extensive annual needs assessment survey to assess what courses should
be offered to customers in the upcoming year. The survey consists of approximately 400
items designed to determine whether and how customers use software (importance),
when they last used specific software (recency), and how often they use the software
(frequency).
Market needs for a specific course. Some organizations develop particular training
courses on an ad-hoc basis; that is, courses are developed to meet specific needs or at the
specific request of a participant, a customer, or even an internal subject matter expert.
Because these courses are not often identified during the overall needs assessment
process, some training programs will conduct a course-specific needs assessment to
determine whether the "market" will support them. For example, an internal subject
matter expert may want to provide a training course for a specific topic that was not
identified as a need during the overall needs assessment. Before taking the time to
develop the course, a market needs assessment is conducted for the course to determine
whether participants believe there is a need for the training and whether they are likely to
attend.
Some of the questions that market needs assessments typically ask are:
> Is there a need in the industry or organization for the specific course?
> Does this course fit in within the organization's overall programmatic goals?
> Are participants likely to attend?
Example of a Market Needs Assessment for Individual Courses
Georgia Tech develops courses that are requested by several sources including customers,
professors, and other internal or external subject matter experts. Georgia Tech
subsequently conducts a market needs assessment for selected courses to make sure that a
similar course does not already exist and that potential participants feel there is a need for
the course.
Content design and delivery needs assessment. Content design and delivery needs
assessments are typically conducted as part of either an overall or market needs
assessment. Because distance learning is a rapidly developing field, content design and
delivery needs assessments help determine not only what types of delivery methods are
being used by the industry but also what types of delivery methods potential participants
need and expect. For example, SAS found that customers were less willing to travel to
training sites after September 11, 2001. A content design and delivery needs assessments
was used to determine the types of training delivery methods that could supplement
HayGroup Page 67
-------
EPA: Results of Benchmarking Study
classroom training in light of a reduced willingness to travel on the part of customers.
This type of needs assessment may also be used to determine the best methods to reach
participants in remote locations, or the extent to which participants are willing to watch
satellite broadcasts, participate in live Webcasts, complete a self-paced Web course, and
so forth.
NEEDS ASSESSMENT BEST PRACTICES
Through our site visits and literature review, we identified several best practices in
conducting needs assessments.
1. Incorporate skill/competency assessments into the needs assessment process.
Industry leaders go beyond simply asking trainees to evaluate their need for future
training courses or their desire for different delivery methods by incorporating skill or
competency assessments into the needs assessment process. For example, trainees
may be asked to indicate, from a list of skills, knowledge, and competencies, those
they feel are necessary for future success on the job, and to rate their current level of
proficiency with those knowledge/skills/competencies. The training group would
then develop training courses aimed at teaching the critical skills, knowledge, and
competencies needed by trainees (those that are important to future success and need
further development). As another example, some organizations have developed
competency models for certain positions which outline the key competencies required
for the job. Training courses are then reviewed to determine whether they
develop/teach the competencies required for success on the job, and determinations
are made about the need for updating current courses or developing new ones.
Example of Incorporating a Competency Assessment
into the Needs Assessment Process
OSHA recently developed a competency model for its compliance officers (to determine
the competencies required for success on the job). A consultant was then hired to
perform a gap analysis to determine the gaps between the current training curriculum and
the competency model. As a result of this analysis, OSHA made decisions to update
current courses and develop new courses to ensure that its training program was teaching
the competencies required for success on the compliance officer job.
2. Encourage trainees to participate in the needs assessment process. For training needs
assessment data to be useful for making decisions about training courses and
programs, it is critical to obtain feedback from a representative sample of current and
potential trainees. To enhance response rates, industry leaders publicize the needs
assessment approximately 30 days in advance using a variety of outlets (e.g.,
newsletters, the Intranet, supervisors, flyers posted on walls throughout an
organization, message boards, e-mail).
It is also helpful to send potential participants an introductory letter (often in
conjunction with a paper or Web survey) from an organizational leader explaining the
importance of the needs assessment process. Additional follow-ups further help to
ensure that as many participants as possible complete the needs assessment.
HayGroup Page 68
-------
EPA: Results of Benchmarking Study
3. Collect data from multiple sources. In addition to collecting information from
trainees about their specific training needs, industry leaders also seek input from other
sources that may have insight into the needs of the trainee population. For example,
some training programs conduct surveys of supervisors to obtain their feedback on
the types of skills and knowledge their employees will need in the upcoming year to
be successful on the job. Additionally, course instructors or developers are often
included in the needs assessment process. Through their experience with trainees,
course instructors and developers often have a good understanding of the types of
courses that will be needed in the future.
4. Use several data collection methods. A needs assessment does not have to be
conducted solely as a paper-and-pencil or an on-line survey. Other data collection
methods used by industry leaders are information interviews, focus groups, and
comment cards. The more data collection methods that are used, the more likely a
representative sample of participants will respond. For example, conducting only on-
line needs assessment surveys may limit the participant sample to those most
comfortable with the Internet. Using additional methods also results in more
insightful and actionable data. While surveys provide objective data about who is
likely to participant in a particular training course, focus groups or interviews provide
more qualitative information about why they are likely to participate.
5. Use consistent methods and processes. Needs assessment methods and questions
should remain stable, to the extent possible, over time. The ultimate goal of a needs
assessment is to determine what training and skills are needed to meet specific
organizational goals. To measure progress and training gaps over time, the methods
and questions should be consistent. For example, asking participants to simply list
courses they need in the future may provide different results than offering a list of
courses and asking participants to rate the importance of each course.
6. Use the needs assessment results during the course development process. Industry
leaders consistently monitor the data collected from needs assessments and make
decisions about training content and delivery methods based on these data. Training
needs assessment data provide valuable input into whether the "right" courses are
currently available to meet trainee needs, whether there is a need to update current
courses or transfer them to other delivery methods, or whether new courses should be
developed. Using needs assessment data shows that the process is taken seriously and
helps in planning how training resources will be used in the future.
7. Provide feedback about results. Participants take the time to complete needs
assessments and expect to learn about the results and decisions about training courses
that are made based on these results. Industry leaders publicize the results of training
needs assessments (and decisions made based on these results) to all participants. As
a result, participants gain an understanding of how training decisions are made (e.g.,
why a particular course was eliminated or why a classroom course was translated to a
live Web course). Also, participants may be willing to provide additional insights
HayGroup Page 69
-------
EPA: Results of Benchmarking Study
into the needs assessment results, which may add another level of understanding to
their needs. For example, participants may find the needs assessment results
surprising and following up on this may reveal that they did not understand certain
questions or that the results were misinterpreted.
SUMMARY
The needs assessment process is one of the first steps to developing an excellent training
program or course. The value of the needs assessment is that it allows an organization to
discover where the gaps are in terms of courses offered, delivery methods used, and the
skill sets required for the trainee population to be effective in their jobs. Industry leaders
regularly conduct needs assessments which incorporate skill/competency assessments
into the process, and solicit input from multiple sources such as potential trainees,
industry leaders, and supervisors. Most importantly, needs assessment data are used to
modify existing training courses, develop new courses, and eliminate courses that do not
add value.
HayGroup Page 70
-------
EPA: Results of Benchmarking Study
///. Course Design and Content
The content of a training course is the most important determinant of its effectiveness.
Regardless of how training programs are delivered, they are not valuable unless
participants learn and take away something that can be applied to their day-to-day jobs.
Even the most sophisticated delivery method will not teach trainees if the content is of
little value, outdated, or not appropriate for the intended audience.
The benchmarking partners and other industry leaders place a great deal of emphasis on
developing training content and keeping it up-to-date and of the highest quality. Several
use internal and external subject matter experts (SMEs) to develop course content,
expend considerable resources developing programs, and conduct systematic course
reviews to ensure that content is up-to-date.
The benchmarking partners agree that developing high-quality courses takes time and
resources. For example, Georgia Tech estimates that three hours of post-production time
are required for each hour of instruction. The CDC estimates that a self-paced Web
course consisting of 120 pages can take about 3-5 months and $120,000 to develop from
scratch.
SAS: Estimated Time to Complete Training
SAS indicated that one of the biggest lessons it has learned is that good training takes
time to develop. It takes:
12 hours to develop 1 hour of classroom delivery content
20 hours to develop 1 hour of live Web delivery content
60 hours to develop 1 hour of self-paced Web delivery content
In this section we present best practices around course development and updating that we
identified during the site visits and literature review.
COURSE DEVELOPMENT
Most of the benchmarking partners use a systematic process for designing new courses or
converting existing courses to a different medium (e.g., a classroom course to a live
Webcast; a paper self-study course to a self-paced Web course). There is wide variety in
the number of new courses developed each year by the benchmarking partners (and
number converted to other media). For example, on average, OSHA designs only two
new courses per year and transforms an additional 12 courses from the classroom to a
Web-based format. CDC develops approximately 30 to 40 new courses (satellite and
Web-based) per year, while Georgia Tech developed over 100 courses in 2004.
We found that initial ideas for new courses or new course delivery methods come from a
variety of sources including reviews of needs assessments and industry trends, subject
matter experts, external clients, and organizational leaders. Additionally, we discovered
HayGroup Page 71
-------
EPA: Results of Benchmarking Study
that some organizations have a specific training philosophy or specific resources that
guide their course development process. For example, SAS follows at least four rules
each time it develops a Web-based course, and Georgia Tech has enormous academic
resources at its disposal to assist with course development (e.g., SMEs/facuity, libraries,
students).
Regardless of where the course ideas originate or the planned delivery method (e.g., live
Web, satellite, classroom), we identified several best practices for developing training
courses.
1. Form a course development team. The benchmarking partners indicated that each
course development project requires an experienced team covering a variety of roles
including, program manager, project manager, instructional designer, subject matter
expert, technical experts, technical review team, editor, and producer. Table 3
describes the basic roles that should be filled for each course development project;
some of these roles may overlap and be completed by one person, and additional roles
may be required for specific courses.
Table 3: Roles of the Content Development Team
> Program manager - Develops the budget and plans the overall course; takes responsibility
for the course and interfaces with executives; promotes and evaluates the program
> Project manager - Reviews plans and oversees day-to-day work and goals (may also be the
program manager for larger projects)
> Instructional designer - Creates the training objectives and the overall framework, and the
look and feel of the course
> Subject matter expert - Understands the subject and drafts the technical aspects of the course
> Technical experts - Understand the technology to deliver the course (e.g., Web developers,
camera crews, satellite technicians)
> Technical review team - Reviews the technical aspects of the course including delivery
methods, training/teaching techniques, and the feasibility of the overall design
> Editor - Makes sure content is clear and accurate
> Producer - Produces the course and makes sure it is incorporated into the overall training
program
Primarily taken from Bersin, Josh. The Blended Learning Book: Best Practices, Proven Methodologies,
and Lessons Learned (2004).
HayGroup Page 72
-------
EPA: Results of Benchmarking Study
Example of Course Development Process
The CDC typically initiates course development at an internal request. It provides
instructional design expertise and expects that content will be developed by an external
subject matter expert. Subject matter experts provide content information including an
outline of the course so that CDC can review and make changes. Then, depending on the
type of delivery method, the content development process follows a specific path.
Web-based Development
CDC provides a Web developer (usually a contractor) with the course outline and
content. The Web developer takes the content and develops a course based on it and
other design specifications. CDC staff review the process along the way and conduct
pilot tests to ensure the effectiveness of the course.
Satellite Broadcast Development
CDC uses the following process to develop its satellite broadcasts: 1) subject matter
experts create an outline of the material to be covered by the broadcast (the CDC asks to
see the script at least a few days in advance); and 2) a hired script writer takes the
information provided by the subject matter experts and writes a full script for the
broadcast (to make the material conversational). Because subject matter experts are just
that, the script writers help create more conversational programs out of the technical
information provides by the SMEs. CDC then requires an extensive practice session (full
script and use of teleprompters) as a final check for glitches or inaccuracies before the
broadcast is delivered live.
2. Work with subject matter experts. Our benchmarking research suggests that most
courses require SMEs to develop technical content. Subject matter experts can be
anyone with special expertise and may include, among others, internal specialists,
professors, or external consultants. For example, Georgia Tech often uses professors,
and GMAC has brought in experts from financial ratings agencies such as Moodys to
help develop course content. Although SMEs are excellent resources, they are often
very busy people and working with them can prove to be one of the biggest
challenges to completing course content design in a timely fashion. Through the site
visits and literature review, we identified the following guidelines for working with
SMEs.
> Ask questions. Asking specific questions about how a SME develops courses,
and about his/her philosophy and time constraints, may help determine
whether or not the SME is a good match for the course design project.
> Provide SMEs with information about plans for the course. SMEs are often
brought into a course design project after plans for the course have been
discussed. The benchmarking partners recommend providing SMEs with
specific details about the course including overall objectives; intended
delivery methods; time, resource, and technological constraints; and audience
characteristics. Unless SMEs are provided with information about the course
up-front, they may develop content that does not meet the stated objectives
HayGroup Page 73
-------
EPA: Results of Benchmarking Study
> Review SMEs work. Although most SMEs have extensive knowledge of the
course subject matter, they may not have a good understanding of the intended
audience or even the course objectives. Therefore, it is important to review
the SMEs work to ensure it fits the intended audience and meets course
objectives.
3. Decide whether to outsource. Many of the benchmarking partners and other industry
leaders are not able to perform all of the course development roles and must
outsource certain tasks. Most often these are technical tasks such as Web
development. Although outsourcing is often a good short-term solution, it can also be
more expensive. For example, the CDC initially contracted with Web developers to
help design Web-based courses. They found, however, that contracting out this work
was more expensive than hiring a full-time Web developer. On the other hand,
Georgia Tech provides a full range of content development services and many
organizations contract with them to develop and conduct their training.
4. Conduct a pilot test before courses go live. The benchmarking partners stressed the
importance of pilot testing courses before they are delivered to the entire trainee
population. Pilot testing is particularly important for self-study Web-based courses,
where there is a need to not only determine whether the content of the course is clear,
easy to understand, and meets training objectives, but whether the technology itself is
easy to navigate, provides sufficient opportunities for interaction, etc.
Example of Course Pilot Test
CDC pilot tests all its asynchronous Web-based courses before they are delivered to the
trainee population. The purpose of the pilot test is to ensure that the course content and
design are appropriate for the intended audience. Pilot test participants (potential training
participants, training coordinators, site facilitators) are asked to review the Web-based
course and provide feedback about the length of the course (which determines the
number of continuing education credits), degree to which they understand the course
content, whether the technology is easy to navigate, extent to which the Web is the best
method for delivering the specific training, and so forth. Feedback from pilot test
participants is incorporated into the final version of the Web training.
Example of Course Pilot Test
OSHA conducts an extensive pilot test of its Web courses before going live. Volunteers
take a pre-test (to determine their knowledge/skill level before taking the course),
complete the Web course, and then take a post-test to evaluate whether they learned
required knowledge/skills by completing the course. Additionally, participants fill out an
evaluation form to provide their feedback on any concerns or errors with the training,
questions, and so forth. OSHA also holds a one-hour conference call with the pilot test
participants to go over their evaluations and gather additional information. An evaluation
report is then developed and required changes are made to the Web course before it goes
live.
HayGroup Page 74
-------
EPA: Results of Benchmarking Study
Special Considerations for Designing Self-Study Web Courses. We found through the
site visits and literature review that designing self-study Web courses present additional
challenges. To be effective, these courses must be designed to facilitate self-learning
(i.e., keep participants involved and engaged; monitor participant progress in completing
the course). Best practices in designing self-study Web courses include:
1. Organize courses into learning modules. Industry leaders agree that self-study Web
courses should be organized into learning modules or "learning chunks" based on
major topic areas. A specific module includes similar content items, practice items,
and quizzes. Modules should take 20 minutes or less to complete. The American
Society for Training and Development suggests that the benefits of learning modules
is that they allow trainees to learn small sections of content at a time, learn skills on
an as-needed basis, and skip modules that they have already mastered (cited in ASTD
E-learning Handbook). Learning modules also cut down development costs and
allow organizations to personalize training for employees.
Example of Learning Modules
When Oracle designs Web-based courses, the company breaks the course content into
segments (or modules). Each segment includes pre-recorded streaming video lecture,
demonstrations, exercises, and quizzes, all focused on a similar topic or learning
objective. Each segment takes trainees 10-15 minutes to complete.
Taken from: Kiser, Kim. E-Learning Evangelism. Online Learning (2001).
2. Provide learning objectives at the beginning of each session or module. Effective self-
study Web sessions begin with an overview of the purpose of the session and the
learning goals. Trainees must see the course as relevant or they will not complete it.
3. Re-package course materials when converting to self-study Web medium. One of the
biggest challenges faced by the benchmarking partners and other industry leaders is
converting classroom courses (or paper-based self-study courses) to effective Web-
based courses. Converting classroom courses to the Web involves much more than
simply cutting and pasting an existing classroom course onto the Web. A large
percentage of development time is spent transferring classroom content to more
visually stimulating material and incorporating opportunities for student-to-instructor,
student-to-student, and student-to-technology interaction into the Web-based training
course. For example, the American Society for Training and Development
recommended in its E-Learning Handbook that content transferred from classroom
courses to the Web must be reorganized and packaged for flow. For example:
> Graphics must be recreated (versus simply using existing Power Point slides)
to visually represent or enhance the content of Web courses
> Exercises, case studies, simulations, quizzes and other opportunities for
interaction should be incorporated into the Web course
HayGroup Page 75
-------
EPA: Results of Benchmarking Study
REVIEWING AND UPDATING COURSE CONTENT
The benchmarking partners have an extensive process for continually reviewing and
updating training courses to ensure that they are up-to-date and of the highest quality.
For example, Georgia Tech reviews and updates all its courses at least on an annual basis.
OSHA uses course chairs to monitor courses and make sure they are up-to-date. The
course chair typically reviews the courses approximately every three years to ensure they
are up-to-date, and makes changes as needed.
It should be noted that some training delivery methods are easier to update than others.
For example, updating self-paced Web courses may simply require removing the out-of-
date material and replacing it (e.g., with a new set of Power Point slides). Live Web-
based courses, on the other hand, may be more difficult to update because the process
often requires re-taping the entire episode or trying to synch-up the new information with
the old.
The following are best practices in reviewing and updating training courses:
1. Update continuously and periodically. Updating course content can be a timely
process, which is a very important reason for updating as often as possible. Much
like cleaning a house, it is easier to update (or clean) periodically than wait until the
course is such a mess that it needs a complete overhaul (i.e., extensive "spring"
cleaning). However, there may be a point where a course does need a complete
overhaul and not just periodic updates. For example, a new methodology or policy
may be developed that completely changes an industry. Rather than update the
course content, it may make more sense to develop a brand new course based on the
new methodology/policy.
Examples of Content Update Tools
• OSHA uses special software (Workforce Connections) that allows Web-based
courses to be updated and changed "just-in-time".
• Cisco has a system in place to alert SMEs or training designers when the content of
Web-based courses is aging or becoming out-of-date.
2. Plan for updates. Rather than waiting until course content is old or outdated, industry
leaders plan ahead for updates. For example, it may be necessary to plan a systematic
review every 6 or 12 months to ensure that the content is appropriate for the intended
trainee audience. Also, because course updates take time and resources, it is
important, when developing a course, to consider how much time and money will be
needed to update the course, and plan resources accordingly.
3. Explain updates. The benchmarking partners make it a point to communicate to
trainees why and how course content is updated. Some trainees may not agree that
content should be updated, so it is important to explain to them why the content has
changed (e.g., a new method has been developed).
HayGroup Page 76
-------
EPA: Results of Benchmarking Study
4. Use updates as teaching points. Course updates can be used as a learning opportunity
for trainees. For example, some benchmarking partners indicated that they use the
content updates as a teaching point by detailing why the previous content is outdated.
They compare and contrast the new and old information and explain how the
information has progressed.
SUMMARY
The content of a course is the most important determinant of its effectiveness.
Regardless of how sophisticated the delivery method, a course that has out-of-date
content or is inappropriate for a specific audience will not teach trainees skills that they
can apply to their day-to-day jobs. We found that our benchmarking partners and other
industry leaders use systematic processes to develop course content, and rely on subject
matter experts to provide the foundation for the course content. They pilot test the
majority of their courses before they go live to ensure that the content is up-to-date and
appropriate, interactive exercises and simulations are effective, and the delivery method
is appropriate for the course content. Most importantly, industry leaders regularly
monitor the quality of their training courses and update courses when necessary.
HayGroup Page 77
-------
EPA: Results of Benchmarking Study
IV. Training Delivery Methods
All of the benchmarking partners we visited are challenged with providing effective
training to large populations of individuals who are geographically dispersed. Although
classroom-based courses continue to play a very large role in training programs, these
organizations realize that they can no longer rely on the classroom as their primary
training delivery method. Instead, they are using innovative distance learning training
methods to ensure that they reach a diverse audience, and more importantly, provide
stimulating learning opportunities for those they train. As further evidence of the
increased reliance on distance learning training methods, ASTD cited in its 2003 State of
the Industry Report that classroom training (as a percentage of delivery methods) has
decreased over the past four years and is projected to continue to decline.
Training Magazine's 22nd Annual Industry Report (2003) stated that 72% of
organizations surveyed had training budgets in 2003 that were similar to or greater than
2002 budgets. This report also notedd other training trends from 2002 to 2003, including:
> Instructor-led classroom accounted for 74% of all training in 2002, but
dropped to 69% in 2003
> Computer-delivered training with no instructor rose from 12% of all training
provided in 2002 to 16% in 2003
> Instructor-led training from a remote location rose from 7% of all training
provided in 2002 to 10% in 2003
> Separate technology-based training budgets rose from 24% of responding
organizations in 2002 to 29% in 2003
While the Web is a powerful distance learning tool and is the primary distance learning
medium used by industry leaders, it is by no means the sole medium used to deliver
distance learning programs. In fact, distance learning programs can range from highly
interactive videoconferences to programs that require students to review PowerPoint
slides that are delivered by inserting a CD-Rom into one's personal computer.
The biggest challenge in distance learning is to incorporate both interpersonal and
person-to-technology interactions into training delivery methods. These interactions are
critical to ensuring that trainees actually complete the training, and more importantly,
learn required knowledge and skills. Not unexpectedly, a common theme among the
benchmarking partners was this very concern: how to integrate effective interactions into
the various media employed in distance learning. During our site visits, we discovered
that the benchmarking partners are using cutting-edge techniques to foster interaction,
which is seen as a critical component of an effective training program.
In this section, we summarize training delivery trends and best practices, particularly
those used to promote distance learning. Although each program delivers, and most
experts agree there will always be a place for classroom-based training, we focus on the
distance learning delivery methods because they represent the direction the industry is
HayGroup Page 78
-------
EPA: Results of Benchmarking Study
headed. The best practices summarized below can be adapted for classroom-based
courses.
DISTANCE LEARNING DELIVERY CATEGORIES
Most experts categorize distance learning delivery methods as either synchronous or
asynchronous. Synchronous training methods are conducted live and delivered real-time
in a classroom setting, over the Web, or through a video (typically satellite) broadcast.
Asynchronous training methods are sometimes referred to as self-study methods because
they are delivered "flat" (i.e., not live) and participants can access them at any time.
Table 4 presents a brief overview of the pros and cons of synchronous and asynchronous
delivery methods
Table 4: Pros and Cons of Synchronous and Asynchronous Delivery Methods
Synchronous
Training
Pros
> Can be highly interactive;
promotes relationships and
students gain from live
discussions and feedback
> Easier to evaluate whether
learning occurred
> The speaker's tone and style
come through well
> Often easier to develop
Cons
> Can be very expensive to
deliver
> Scheduling is difficult and
may take several sessions to
reach everyone
> Can require complex
technologies such as
satellite receivers and may
require large bandwidth
Asynchronous
Training
Easier to distribute to large
audiences
"Desktop, anytime"; students
can access where and when
they want
Conveys a standard message
Ability to skip topics that are
already understood or
mastered
> Lacks inherent interactions
> Can have a high dropout
rate
> Can be difficult to evaluate
learning/monitor trainee' s
progress
> Can be costly to develop
The benchmarking partners we spoke with, and other training industry leaders, tend to
use a variety of synchronous and asynchronous methods to deliver distance learning
programs to trainees (classroom training is not the focus of this report). Synchronous
methods include live Webcasts/Webinars, satellite broadcasts, and videoconferences. We
found that the industry trend is towards Web-based courses (Webcasts/Webinars).
Asynchronous methods include self-paced Web courses, CD-Rom/DVD self-study
courses, and paper-based self-study courses. Similar to synchronous courses, we found
that Web-based courses are quickly replacing paper-based self-study courses.
HayGroup
Page 79
-------
EPA: Results of Benchmarking Study
SELECTING TRAINING DELIVERY METHODS
The benchmarking partners we spoke with do not have hard and fast rules for making
decisions about which training delivery media to use. Often the decision is a pragmatic
one that is based on the time available and money or other resources needed to develop
and deliver the training program. Some benchmarking partners, however, did use some
rules-of-thumb to guide them in selecting a training delivery method:
> Complexity of the material - For example, more complex material is better
taught in the classroom (e.g., laboratory) or through interactive synchronous
training methods. For less complex material, it may not be worth the cost to
use synchronous training delivery methods; a self-paced course may be
sufficient. For example, self-paced courses work well for the delivery of
information, while the classroom or interactive Webcasts work well when
discussion is needed.
> Need for interaction — Training content that requires student interactions is
better taught via synchronous training methods (e.g., classroom;
Webcasts/Webinars that incorporate methods for interaction).
> Requirements for validation and/or certification - Classroom training and
other synchronous training methods that can track student progress and
incorporate quizzes and tests work best for courses that require certification.
Certification programs require that participants meet some industry or test
standard, have strict score reporting protocols, and often have expiration
dates. It is difficult to meet these standards with an asynchronous program
because it is not always possible to determine who took the test, participants
cannot be expected to report their own results, and certification courses may
expire but still be available in Web or paper format.
> Training audience - Participants skill levels may impact the effectiveness of
different training delivery methods. For example, trainees may not all have
the same understanding or experience with certain technologies. If the
audience is likely to have lower technological savvy, it makes more sense to
use less technologically advanced delivery methods. The size of the audience
should also impact decisions about training delivery methods. As a rule of
thumb, larger audiences (>3000) are more suited for asynchronous methods or
live Webcasts (versus classroom training, etc.). Finally, if the audience has a
varied background, this is an area where asynchronous training could be used
to provide a common foundation of knowledge prior to the synchronous
portion of a course (i.e., a blended learning approach).
> Goals of the program - Programs that are designed to foster culture-building,
such as orientation programs, are best delivered using synchronous methods.
Training programs that focus on delivering instructions or an introduction to a
task may best be suited to an asynchronous method.
HayGroup Page 80
-------
EPA: Results of Benchmarking Study
> Budget and time - Training programs with smaller budgets need to think
about how best to use their limited resources in delivering training. While
synchronous methods may require a lot of personnel resources, asynchronous
courses can often require a lot of up-front time and money to develop.
Additionally, asynchronous courses often take a long time to develop and are
not the best method to use when a course has a rapid window between design,
development, and delivery unless appropriate technology is available.
In sum, when the outcomes of training are to analyze, synthesize, or evaluate, or when
poorly defined problems are the focus, more synchronous delivery methods work best.
When the outcomes of training are to provide procedural or well-defined information, or
when providing regulation updates, the training methods should be more asynchronous.
TRAINING DELIVERY METHODS USED BY BENCHMARKING PARTNERS
It is clear that several delivery methods have been developed for distance learning
training programs. The term blended learning is often used to define the combined use of
two or more delivery methods, and is becoming the approach of choice for many
organizations (discussed later in this section). While many of the benchmarking partners
use and recommend the blended learning approach, we provide here an overview of each
method as if it is the sole delivery method. At the end of this Training Delivery section,
we will discuss ways benchmarking partners and other training leaders are using blended
learning to increase the effectiveness of their training programs.
Synchronous Delivery Methods
Most of the benchmarking partners agree that synchronous training delivery methods
should be used when live interpersonal interactions are necessary. These methods, when
done "right", provide participants with the opportunity to have real-time interactions with
instructors, other participants, and the technology (e.g., simulations, quizzes). All of the
benchmarking partners and many industry leaders in training use non-classroom-based
synchronous training delivery methods, in addition to classroom courses. The most
commonly used, and preferred, non-classroom-based synchronous training medium is the
Web (e.g., live Webcasts/Webinars), followed by satellite broadcasts and
vi doeconference s.
Satellite broadcasts and live Webcasts are often similar in design and course developers
face many of the same challenges with the two methods. The primary challenge shared
by the two methods is incorporating interactivity into courses. In fact, all of the
benchmarking partners stressed the importance of integrating interactivity into training
programs. Integrating interactivity into live Web courses seems to be the focus of most
of the benchmarking partners because many have already or are beginning to move away
from satellite broadcasts.
There are several reasons why organizations are focusing on live Web training (e.g.,
Webcasts) rather than satellite broadcasts. First, as training staff at S AS indicated, people
tend to physiologically tune out during training, and the Web offers exciting and varied
HayGroup Page 81
-------
EPA: Results of Benchmarking Study
opportunities to enhance learning programs. Additionally, busy professionals often do
not have the time or the ability to travel to a satellite-equipped site for training, and the
Web offers training that can be accessed on one's desktop.
For the remainder of this section, we present our findings on the primary synchronous
delivery methods (non-classroom) used by the benchmarking partners: Live
Webcasts/Webinars, Satellite Broadcasts, and Videoconferences. Because our research
shows that the trend in synchronous delivery is toward live Web-based delivery methods,
we focus on this medium when describing the best practices. However, because there are
some similarities between the functionality and challenges faced for each delivery
medium, many of the best practices described for the Web-based medium may be
applicable to other synchronous training methods. For example, the benchmarking
partners are using question monitors for both live Webcasts and satellite broadcasts, but
we focus on how this is being adapted for Web use.
Live Webcasts or Webinars. As indicated previously, our research shows that
synchronous distance learning programs are moving toward the Web. Several of the
benchmarking partners are conducting live Webcast or Webinar programs over the
Internet (or Intranet). The terms Webcast and Webinar are often used interchangeably by
training industry experts and our benchmarking partners as they refer to similar delivery
methods (i.e., live Web-based training delivery). For this reason, we will use the term
Webcast to refer to any live Web-based training method.
Through these programs, participants can hear and see instructors on-line, in real-time.
Live Webcasts allow participants to interact with each other and can be programmed to
work with simulations and other interactive media. Also, the benchmarking partners
strongly believe that these programs have proven effective, particularly when
opportunities for interaction are provided to participants.
The benchmarking partners have spent considerable time and resources to make live
Webcasts interactive, interesting, and effective at training individuals to be more
productive in their jobs. For example, SAS began delivering live Web courses in July,
2001. But before SAS would deliver a live Web course, the training staff made certain
that they could incorporate and encourage interactions without interrupting the overall
flow of the program. SAS created a set of rules they follow when developing any live
Webcast course to ensure they meet their initial program goals.
SAS Live Webcast Rules
1) Participants must be given the opportunity to interact at least every 10 minutes.
2) Instructors must have specific training.
3) Instructors must have two rehearsals before they can teach a course.
4) Moderators must be present for each course session.
Through our site visits and benchmarking research, we identified several best practices
for delivering live Webcasts, which are highlighted throughout this section.
HayGroup Page 82
-------
EPA: Results of Benchmarking Study
1. Record all live Webcasts. The benchmarking partners recognize that not all
individuals can watch the Webcast when it is delivered live (e.g., have scheduling
conflicts). As a result, they record all the live sessions and place links for these sessions
on their Web sites (or Intranets) so that individuals can watch the Webcast at their own
time and location (essentially transforming into asynchronous delivery). Additionally,
this allows new employees to complete needed training whenever they are able, rather
than waiting for the next live session.
Example of Recording Webcasts
Most of GMAC's programs are broadcast live and taped using Windows Media Encoder
for later viewing. The training staff found that live audiences were often smaller than
anticipated and decided that taping and re-broadcasting the sessions would allow them to
reach more people. The programs are posted to the Intranet and employees can access
them at their leisure.
2. Incorporate opportunities for interaction into the Webcast. The benchmarking
partners recognize that for successful learning to occur, participants must have the
opportunity to ask questions and discuss the broadcast with others (e.g., instructor,
moderator, other participants). Simply watching a "talking head" or an electronic
"page turner" does not engage participants; many participants will simply tune out.
To promote interactions between participants and instructors during live Webcasts,
many benchmarking partners give participants the opportunity to ask questions during
the session. We found that the most common methods used to elicit questions
include:
> Call-in questions (e.g., participants call in to a centralized 800 number that is
publicized before the Webcast)
> Fax-in questions
> E-mail questions
> Voice Over Internet Protocol (VOIP) (VOIP is a technology that allows
participants to make telephone calls using a broadband Internet connection
instead of a regular phone line - our partners disagree as to whether VOIP
technology is reliable enough for most training purposes)
For these interactive methods to be effective, there must be communication to
participants before the broadcast about the process for asking questions (e.g., fax-in,
telephone) and participants must be encouraged to do so. The above processes have
proven relatively effective in enhancing interactions during Webcasts. However, the
benchmarking partners have experienced some challenges, which are discussed below
along with strategies that have been used to overcome them.
> Many participant questions go unanswered during the Webcast - To
overcome this challenge, most of the benchmarking partners capture all the
participant questions during a Webcast and post them to an asynchronous site.
Although all questions are not answered live, all questions are acknowledged
HayGroup Page 83
-------
EPA: Results of Benchmarking Study
and participants have access to the answers (participants are provided links to
the site).
> Participants do not always ask questions during the Webcast - Many
benchmarking partners develop a set of approximately 10 questions prior to
the Webcast in the event that participants do not ask questions during the
broadcast. For example, if participants are not asking questions, a member of
the broadcast team can act as a participant and ask one of the pre-set
questions. This helps to fill the time and often initiates participant
interactions.
> Participants do not follow proper "netiquette" -- Finally, to overcome the
third challenge, benchmarking partners such as SAS have developed
"netiquette" rules that are provided to all participants prior to and at the
beginning of each session. For example, because there is limited time,
questions should be asked only if they help clarify a point or continue
discussion. Questions about scheduling should be taken off-line.
Another strategy used by benchmarking partners to enhance interactions during
Webcasts is the incorporation of audio and visual means of communication into the
training delivery method. Audio communications typically take place over two-way
telephone lines (e.g., an open telephone line where participants in the Webcast can
ask questions during the session), but some benchmarking partners are also using
VOIP to provide for total communication over one's personal computer. Video
communication can also occur over personal computers by mounting a small video
camera on the computer (often used during Webcasts). Larger video cameras can be
effective when a group of people are gathered in one location, but for more
individualized locations, the smaller PC-mounted cameras work best.
3. Use screeners to review participant questions. Some of the benchmarking partners
find that instructors receive too many questions to respond to during the Webcasts
and that reading each question individually takes time away from teaching. To meet
this challenge, a form of question screening and monitoring has been incorporated
where one or more individuals are available to receive, read, and review questions to
determine whether the instructor should respond to them during the live Webcast.
For example, Georgia Tech pairs a screener with an instructor and uses the screener
to monitor the questions and pass along to the instructor those questions that will
improve the training and help students to learn.
SAS also uses technology to help monitor and review questions. Through Microsoft
Live Meeting, participants are able to "raise their hands" when they have a question.
The program indicates a potential question by changing a participant's "seat" color on
the on-line screen visible to the instructor and screener. In addition, the screener is
able to mute one or all participants if the conversation gets out of control.
4. Use technology monitors. The benchmarking partners have found that many
participants and instructors have trouble with and questions about the technology
used during the live Webcasts. Because providing technology assistance to
HayGroup Page 84
-------
EPA: Results of Benchmarking Study
participants is not the role of the instructor and can take away from other participants'
experiences, Georgia Tech provides each instructor with a technology expert. The
technology expert is there to:
> Respond to questions about technology
> Troubleshoot technology challenges
> Operate technologies for instructors
Example of Question Screeners and Technology Monitors
Georgia Tech provides at least one question screener and one technology monitor for
each live Webcast. The question screener can focus on receiving and determining which
questions are suitable to be addressed during the live Webcast. The technology monitor
makes sure that the instructor can focus on teaching the course rather than worrying about
whether internal or participant technologies are working. Each member of the
presentation team has a specific role that serves to provide an uninterrupted training
session to participants.
5. Ensure that individuals who deliver the Webcast have teaching abilities. The
benchmarking partners that use live Webcasts indicate that not all instructors are
effective at delivering training using this medium. For example, GMAC hired a
university professor as the subject matter expert to deliver a live Webcast training
session. Although the information he presented was very well received, he was
unable to stay within range of the camera and would walk in-and-out of view at a
rapid rate. As mentioned earlier, SAS will not allow an instructor to teach a live
Webcast course until he/she has received specific training and conducted two
rehearsals before the first session. In fact, SAS has 77 instructors, with only 20
qualified to teach live Webcasts.
6. Publicize broadcasts. Because live Webcasts can be difficult to schedule and the
training will not be effective if there is no one participating in the session, publicizing
them is very important. A rule-of-thumb is to begin publicizing each Webcast
approximately 30 days in advance by using a variety of media. These may include
newsletters, the Internet, the Intranet, and supervisors who can help remind the
participants of the training.
7. Limit the length of programs. People tend to get bored and physiologically tune out
during long training sessions, especially when opportunities for interaction are
limited. A key recommendation by our benchmarking partners is to limit the amount
of time spent in any one Webcast training session. One benchmarking partner
suggested that individual training sessions should be scheduled for no longer than a
half-day, and the training session should be broken-up into 1-2 hour segments.
8. Make informed decisions about technology. The benchmarking partners use a range
of software programs for their courses including Microsoft Live Meeting through
PlaceWhere, WebEx, and Microsoft Windows Media Encoder. Each partner
considered using several programs before deciding on one of those listed above. For
HayGroup Page 85
-------
EPA: Results of Benchmarking Study
example, GMAC has used WebEx but has had difficulty recording and synchronizing
both audio and video communications. Media Encoder is now used to record
programs. SAS makes the assumption that most of the top products have similar
capabilities and makes decisions primarily based on cost and whether the vendor will
be around in a few years. The assumptions made by SAS is that each top product will
have:
> Application sharing
> PowerPoint capabilities
> Chat capabilities
> Registration capabilities
Satellite Broadcasts. Satellite broadcasts were not the focal point of any benchmarking
partner's training program, although some benchmarking partners are using the
technology effectively to disseminate critical information and train geographically
dispersed individuals. As indicated previously, our research shows that most training
programs are moving toward more live Web-based delivery methods that share many of
the benefits of satellite broadcasts. One benchmarking partner explained the shift by
saying that satellite courses do not meet the "my desktop, at my time" needs of most busy
professionals. Another benchmarking partner indicated that "live Web courses have
taken over satellite courses because people have better access to them."
We discovered during the site visits some definite downsides of using satellite broadcasts
to deliver training programs, which are presented in table 5.
Table 5: Limitations of Satellite Broadcast
> Facilities - Satellite broadcasts require facilities that have video production and
broadcasting capabilities, and training sites to have satellite reception capabilities.
> Location - Satellite broadcasts require participants to travel to specific locations (with
satellite downlinks) to participate in the training session.
> Scheduling - Satellite courses are difficult to schedule — courses must be scheduled
when the infrastructure and all participants are available.
> Personnel - Significant personnel is required to maintain the facilities, troubleshoot
technical problems, and schedule and conduct courses.
> Cost - The overall initial cost of developing the infrastructure to deliver satellite
programs can be high. However, satellite broadcasts may be a cost effective training
medium for organizations that have already developed the infrastructure.
> Interactions - Satellite broadcasts are typically one-way communications from the
instructor with limited opportunities for participant interactions.
Despite the downsides to using satellite broadcasts, we did find that two of our
benchmarking partners, the CDC and Georgia Tech, use satellite broadcasts (typically
recorded live) effectively for certain types of material. One reason they are able to use
the satellite technology successfully for training is that both sites have an extensive
satellite broadcast capability and maintain a relatively large staff that can develop training
HayGroup Page 86
-------
EPA: Results of Benchmarking Study
programs using various media. For example, the CDC has a staff of 50 professionals
dedicated to maintaining its training programs and Georgia Tech has three satellite dishes
and nine satellite capable classrooms. OSHA is using satellite broadcast minimally for
outreach or just-in-time updates.
Through the site visits and benchmarking research, we identified several best practices in
the delivery of satellite broadcasts. Because satellite broadcasts and live Web-based
courses have similar designs, capabilities, and challenges, many of the satellite best
practices are similar to those discussed previously in the Webcast/Webinar training
delivery sections. We present each best practice briefly and expand only on those
practices that are unique to satellite broadcasts.
1. Training goals and content should dictate the use of satellite broadcasts. Many
experts and the benchmarking partners agree that satellite broadcasts are most
effective for delivering informational programs to a geographically dispersed
population. Satellite broadcasts are also effective for providing just-in-time
information and can be used to deliver technical programs that are broken up into
short courses. For example, the CDC uses the satellite broadcast to disseminate
critical information in a timely manner (i.e., in less than 48 hours). Some rapid turn-
around satellite broadcasts sponsored by the CDC have focused on smallpox,
bioterrorism, and SARS.
2. Record satellite broadcasts. When possible, our benchmarking partners suggest
recording at least the instructor in a live satellite broadcasts. One of the challenges of
satellite broadcasts is scheduling them so that all interested parties can participate.
Recorded broadcasts can be placed on asynchronous sites for later use, which is
especially helpful for rapid turn-around programs such as the CDC conducts. The
benchmarking partners stress that there are two important points to remember when
recording satellite broadcasts: 1) video broadcast require a lot of bandwidth and may
be difficult to store on some sites or send out as a CD-Rom; and 2) synching video
and audio can be difficult.
3. Incorporate opportunities for interaction into the satellite broadcast. Similar to the
live Webcasts, our benchmarking partners recognize that for successful learning to
occur, participants in a satellite broadcast must have the opportunity to ask questions
and discuss the broadcast with others. To promote interactions between participants
and instructors during satellite broadcasts, many benchmarking partners give
participants the opportunity to ask questions during the broadcast. Similar to the
Webcasts, we found that the most common methods used to elicit questions include:
> Call-in questions
> Fax-in questions
> E-mail questions
4. Use screeners to review participant questions. Some of the benchmarking partners
find that instructors receive too many questions to respond to during the satellite
HayGroup Page 87
-------
EPA: Results of Benchmarking Study
broadcasts and that reading each question individually takes time away from teaching.
To meet this challenge, a form of question screening and monitoring has been
incorporated where one or more individuals are available to receive, read, and review
questions to determine whether the instructor should respond to them during the live
broadcast.
Example of Question Screening During Satellite Broadcasts
The CDC maintains a call center staffed with 4 to 5 individuals that accept and monitor
phone-in questions during satellite broadcasts. Their job is to evaluate questions before
deciding whether to patch them into the live broadcast. Questions that do not promote
topical discussions are saved and later posted to an asynchronous site, while those that do
promote the discussion are patched in live to the instructor. By using the telephone
monitors, the instructor is free to teach the course without interrupting the flow to wait
for questions.
5. Publicize broadcasts. Because satellite broadcasts can be difficult to schedule,
publicizing them is very important. A rule-of-thumb is to begin publicizing each
broadcast approximately 30 days in advance by using a variety of media (e.g.,
Intranet, newsletters, supervisor communications). The CDC has distance learning
coordinators in every state to promote the satellite broadcasts (e.g., sending fliers
about the broadcast to health care workers) and assist with registration for broadcasts.
6. Ensure potential participants can participate in the satellite broadcast. Before
developing a satellite broadcast, it is critical to ensure that all participants will have
access to it. Access may be at participants' home office or they may need to travel to
a regional center.
Other best practices that apply to both live Web-based training and satellite broadcasts
are:
> Train instructors in satellite training methods so they are effective at
delivering the broadcast
> Use technology monitors
> Limit the length of broadcasts to 1 - 2 hours per session
Videoconferences. Both the CDC and Georgia Tech have also incorporated
videoconference capabilities into their training programs. Videoconferences are similar
to satellite broadcasts but have the capability to better incorporate two-way interactions.
Instead of a camera focusing primarily on the instructor, Videoconferences may have
several cameras set up in different locations that are able to show the participants at each
of the locations. Videoconferences allow participants to ask questions of the instructor
and each other and discuss the training material throughout the session.
Many of the downsides that we found with satellite broadcasts are also associated with
Videoconferences, and some may be more extreme. One example is that for
Videoconferences to be a viable training method, training sites must have not only the
HayGroup Page 88
-------
EPA: Results of Benchmarking Study
capability to receive broadcasts but also the capability to send broadcasts. Another
example is that recording the sessions is even more difficult because both the audio and
visual communications are two-way and capturing this requires an extensive
infrastructure that most training programs cannot afford.
Videoconferences are best used when the training must take on the feel of a more
intimate meeting and where extensive interactions are required. Because the media is
used to promote continuous two-way conversations, the number of participants or
participant sites should be limited to a number that is effective for action meetings (i.e.,
10 or fewer). In addition to the best practices we identified for satellite broadcasts, we
found the following best practices for videoconferences.
1. Establish etiquette. Before beginning a videoconference, all participants should be
reminded of proper etiquette. Videoconferences provide all participants the
opportunity to see and hear each other, and thus interact in an intimate and possibly
informal atmosphere. Unfortunately, participants sometimes forget that and say or do
things that would be considered inappropriate in a regular meeting. For example,
participants in one site may constantly move around and disrupt the visual
communications for others, and participants in another location may forget to press
mute while having discussions amongst themselves about the program, other
participants, or even lunch plans.
2. Contract with a vendor. Videoconferences often require facilities and technology that
most training programs cannot afford and many industry leaders are not sold that the
costs outweigh the benefits of videoconferences. One solution is to contract with a
vendor such as Georgia Tech that maintains a staff dedicated to maintaining the
videoconference facilities. Georgia Tech has, for example, the capability to centralize
all communications and can even convert dissimilar videoconference formats into a
common one.
Asynchronous Delivery Methods
Asynchronous training does not require live instructors and is commonly referred to as
self-paced training. Most of the benchmarking partners agree that asynchronous training
delivery methods should be used when live interpersonal interactions are not necessary,
when trainings are informational-based or include simulations, and when participants are
geographically dispersed. Asynchronous delivery methods provide participants with the
opportunity to access training when and where they want. All five of the benchmarking
partners use one or more asynchronous training delivery methods, with self-paced Web
training being used most frequently, followed by CD-Rom/DVD courses. Paper-based
(self-paced) training courses are not as commonly used as other asynchronous training
delivery methods.
As with synchronous training methods, the primary challenge faced by organizations that
use asynchronous methods is incorporating interactivity into courses. Because there is no
live interactions with these methods, this challenge takes on a different meaning for
asynchronous courses. Not only must the course provide opportunities for participants to
HayGroup Page 89
-------
EPA: Results of Benchmarking Study
interact with each other, but it must incorporate methods for participants to interact with
the technology. Also, while impromptu interactions may be likely in synchronous
courses, all interactions must be specifically designed into an asynchronous course.
For the remainder of this section, we present our findings on the best practices and trends
associated with asynchronous training delivery methods: Web-based (self-paced), CD-
Rom/DVD, and paper-based (self-paced). Similar to the synchronous training delivery
methods, we focus on Web-based delivery because we feel this is the direction that most
of our benchmarking partners and the industry are moving. However, many of the best
practices associated with self-paced Web courses can be applied to other asynchronous
training methods.
Self-Paced Web Courses. Industry experts agree that self-paced Web courses are better
for some course content than other. For example, Hall (2001) suggests that the Web
works best for courses that:
> Focus on content and information and are fact-based
> Do not require experiential learning that closely mirrors job situations
> Require minimal interactions among students and instructors and are not
intended to develop interpersonal skills
Through our site visits and benchmarking research, we learned that there is a great deal of
variability in the effectiveness of self-paced Web courses. Many organizations are in the
rudimentary stages of developing asynchronous Web courses, simply putting PowerPoint
slides on the Web site or putting material from written self-study training courses on the
Web. Further, the American Society for Training and Development (ASTD) claims that
most on-line training lacks interaction and quality instruction (from ASTD E-Learning
Handbook). The industry leaders, however, are creating interactive, Web-based courses
that engage participants through simulations, quizzes, chat rooms, Web boards, and so
forth.
We identified several best practices in self-paced Web courses:
1. Provide opportunities for interactions with other students and instructors. Many of
the benchmarking partners and other training industry leaders have incorporated
several methods for allowing students to ask an instructor, TA, or other students
questions while they are completing a self-paced Web course. Examples of
interactive methods used are:
> Create a link within the Web site where students can email questions to
instructors or set up electronic office hours.
> Create a Web board where students can post questions that are answered by an
instructor or others students (also known as threaded discussions). Instructors
can also post updates and new slides and materials to the Web board.
> Incorporate chat rooms where students can discuss course issues and ask
questions of each other and the instructor.
HayGroup Page 90
-------
EPA: Results of Benchmarking Study
> Encourage cohorts of participants to complete the course together at the same
time. It is easier and more efficient to provide interactive opportunities to a
group of participants taking a course at the same time. For example Georgia
Tech sets up chat rooms and bulletin/Web boards so cohorts can discuss the
course and ask each other questions while they are taking the course (e.g.,
between sessions). Another organization uses a course moderator who
introduces the self-paced Web course, asks questions throughout the course to
involve students, answers student questions, and facilitates interactions among
students taking the course at the same time.
> Use audioconferences as a de-briefing after students complete the self-paced
course. This provides students the opportunity to ask questions about the
course and discuss course content with other students.
2. Provide opportunities for interactions with the technology. Asynchronous Web
courses do not inherently require interactions. However, interactions are deemed
necessary for training to be effective, and advancements in technology have made it
easy to incorporate strategies for interacting with the technology. For asynchronous
courses, it is critical to keep participants involved in the course so they will actually
complete the course and learn the required skills/knowledge. By incorporating some
of these interaction strategies, a self-paced Web course can be an effective method of
providing technical types of training. Some examples of how interactions with the
technology are incorporated are presented below.
> Quizzes/assessments - Some of the benchmarking partners use short quizzes
or assessments to foster interactions with the technology. For example, during
a Web-based PowerPoint presentation, SAS presents short quizzes (often just
one question) every few slides to keep participants interested and engaged
with the course. The course then provides feedback to trainees on whether
they answered the questions correctly and explanations for those that are
answered incorrectly.
> Simulations - Some intricate simulations have been developed and used to
augment self-based Web training. Effective simulations reflect the real world
and allow participants to learn by doing and practice a skill taught in the
training session. The primary types of simulations are software application,
scenario-based, and business and financial simulations. For example, SAS
provides instruction and then has participants practice key tasks before
returning to the main training session. Other organizations show actual
equipment used on the job and use three dimensional graphics to instruct on
correct usage of the equipment.
> Case studies - Many industry leaders are incorporating interactive case
studies into Web-based training by providing trainees with problems to solve.
Case studies serve much the same purpose as simulations; they provide real
world examples to augment the training. Interactions with case studies can be
enhanced by presenting them as points that cohorts can discuss on a Web
board between sessions.
HayGroup Page 91
-------
EPA: Results of Benchmarking Study
Example of Interactive Quizzes
During a self-paced Web program, SAS often presents quizzes. The quizzes can be
programmed so that participants providing wrong answers may be blocked from
continuing with the training until a correct answer is provided.
Examples of Interactive Simulation
Bank of America uses streaming video and audio in its self-paced Web training course to
allow loan officer trainees to interact with a prospective simulated client. Trainees ask a
series of questions to determine whether the client would be eligible for a loan. (Taken
from Brandon Hall, Six Steps to Developing a Successful E-Learning Initiative, 2001).
Cisco provides on-line access to equipment labs so that the trainee can, for example, try
configuring a switch or router before actually doing it on the company site.
3. Create engaging material. Although the course content is the most important
determinant of the effectiveness of a self-paced Web course, the look and feel of the
training (Web site) is a close second. Research cited by David Daly and Amy Scott
(Best Practices Handbook: Best Practices for Advanced Distributed Learning) shows
that individuals learn 75% of what they know through vision. They indicated that
"pictures and words together are six times more effective than words alone."
Industry leaders use graphics and pictures to make self-paced Web training more
interesting. However, graphics and pictures should be relevant to the training and
visually represent or enhance the content of the Web course (not just "nice to have"),
and be easy to read. Another recommendation made by industry leaders is that
presentations should maintain the look and feel of the organization (branding).
4. Make it easy for participants to access the training. Some self-paced Web courses
require participants to download programs such as Flash or WebEx, or update their
existing Web browsers. However, many organizations limit what employees can
download from the Web to maintain security and prevent viruses from entering a
network. It is critical for organizations to provide trainees with technical support for
accessing Web-based courses. For example, to assist participants, SAS sends CDs
with downloading materials to those who are unable to download the materials from
the Web.
5. Demonstrate a commitment to Web-based training. Industry leaders demonstrate a
strong commitment to Web-based training by encouraging employees to complete the
self-paced course on company time. Unlike employees who complete training at a
location away from the office, self-paced Web training introduces the challenge of
ensuring that employees are free to complete the training without distractions (e.g.,
supervisors interrupting trainees and asking them to attend a meeting; a customer
calling a trainee). For example, at Cisco Systems, employees who are completing an
on-line class can put up yellow police tape to signal that they are in the process of
completing a Web-based course.
HayGroup Page 92
-------
EPA: Results of Benchmarking Study
6. Be thorough with content and delivery methods. Because self-paced Web courses are
essentially stand-alone programs, it is even more important to make sure that content
is up-to-date and that delivery techniques work. While synchronous courses have
almost instant feedback loops, asynchronous courses do not. Participants may not be
able to access or work within the program and it may take weeks or longer for the
training developer to find out that students are having difficulty with the training.
CD-Rom/DVD and paper-based self-paced courses. Completing courses by inserting a
CD-Rom/DVD into one's personal computer and completing a paper-based self-study
training course are two other forms of asynchronous learning. None of the benchmarking
partners make extensive use of CD-Rom/DVD or paper-based training materials, but
some did augment training with these delivery methods. The most common method used
was to send already developed training materials to participants (in a CD-Rom/DVD or
paper format) who could not access a self-paced Web course. The materials sent are
essentially the same as those posted to the Web site. One of the benchmarking partners,
OSHA, indicated that it does not use CD-Rom because the data can get outdated easily
and there is not a way to update it quickly. OSHA also feels that with CD-Rom, it can
lose control of the course content.
However, many organizations we learned about in our literature review are using CD-
Rom or DVDs to replace Web-based training when potential trainees are on the road or in
foreign countries and do not have Internet access. Other organizations have created
electronic libraries with hundreds of CD-Rom and DVDs that were created to capture live
satellite broadcasts, classroom courses, or Webcasts. These libraries provide just-in-time
training for individuals who missed the live course, new employees, and so forth. For
example, Georgia Tech records all videoconferences on CDs and DVDs so that they can
be used to train individuals that could not attend the live sessions.
Blended Learning
Another trend we identified through the site visits and benchmarking research is the
introduction of blended learning programs. Blended learning programs incorporate two
or more training delivery media to provide not only a more holistic learning experience
but also multiple learning situations for the busy professional. Many training experts and
industry leaders make the claim that blended learning is more effective than any single
training delivery method. For example, OSHA is moving toward a blended learning
approach. OSHA uses the Web-based portion of a blended course to bring all students up
to the same level by the time they get to the classroom.
One interesting trend in blended learning is the shift back to using classroom-based
experiences to augment distance learning programs. That is, many organizations are
combining classroom and Web-based training. For example, a self-study Web course
may be used to provide critical knowledge before the classroom session (as a prerequisite
to the course) or as a wrap up after the classroom session. The classroom session then
focuses on interactive exercises, discussions, case studies, and simulations versus
delivering information in a lecture format. Other organizations are combining self-paced
materials with live Webcasts. For example, Stanford University has trainees review
HayGroup Page 93
-------
EPA: Results of Benchmarking Study
regulations in a self-study Web or paper format and then participate in a live moderated
Webcast to discuss the implications of the regulation to their jobs, ask questions, and so
forth.
Blending learning programs have the opportunity to provide a more exciting and
enriching training experience. Rather than being limited to one training medium, blended
learning programs utilize the best practices of two or even several training methods.
These types of programs also offer greater flexibility to both instructors and participants.
Classroom-based or even satellite broadcasts often require that participants travel to a
training site. If a training program is scheduled to last five days, participants must travel
all five days. However, the blended learning approach can help reduce travel time by
presenting day 1 introductory and day 5 wrap-up information over the Web. This reduces
travel costs and potentially frees-up participants' time so they can work on other tasks
rather than spend all day at a training site.
Some benefits of blended learning programs cited by industry leaders are:
> Greater flexibility - instructors and participants are not constrained by a
specific delivery method
> Improved learning effectiveness - the use of a variety of delivery methods
provides a better match between delivery and participant learning styles
> Greater reach - training can better reach participants with scheduling or
technological constraints
> More time spent training and less time spent lecturing - more instructor time
can be spent on hands-on training rather than lecturing.
> Greater opportunities to cut costs - expensive delivery methods can be
replaced or augmented with other methods, rather than eliminating an entire
program
Example of Blended Learning Program
One organization recently converted a 5-week long satellite broadcast training program to a
blended learning program. Before it is converted to the blended learning course, 20 participants
participated in the training watching a satellite broadcast as a cohort (4 training sites with 5
participants per site). Participants watched one satellite broadcast per week for 5 weeks.
The organization decided to convert this course to a blended learning program because
conducting training solely through satellite broadcasts required a great deal of scheduling, forced
participants to attend pre-set sessions at a site away from their office, and required technical
support staff to be available each week. By using the following blended learning approach, the
instructor could limit the time he/she and the participants spent in satellite broadcasts while
providing a more robust training program for participants:
Week 1 - The instructor presents an introduction to the training program via a satellite broadcast.
He/she demonstrates some key tasks that participants need and provides an opportunity for
participants to see and meet each other.
HayGroup Page 94
-------
EPA: Results of Benchmarking Study
Week 2 - Information is delivered via the self-paced Web method. This session is more
informational and does not require any demonstrations. However, while participating in the Web
session, trainees practice the skills demonstrated in week 1 by completing a short simulation.
Participants are free to access this session anytime during the week and are provided a Web board
to interact with other participants.
Week 3 - This session is conducted via a live Webcast. Similar to week 1, the instructor
demonstrates some skills via a live session. The Webcast is also taped so that participants can
access it during the week.
Week 4 - The training is once again conducted via the self-paced Web method. Participants are
provided the opportunity to practice the skills demonstrated in week 3.
Week 5 - The wrap-up session is presented via live Webcast. This session is conducted
primarily so that the instructor can provide final thoughts and answer any remaining questions
that participants have.
Example of Blended Learning Program
Georgia Tech uses a blended learning method for some of its training courses (typically
courses that require student interactions and group work). Students complete a self-study
Web course and then get together in local classrooms after the course to discuss issues or
work in labs.
SUMMARY
Classroom-based delivery methods still account for a large proportion of training
programs, and many training experts believe that there will always be a need for these
types of programs. However, live Web-based training (e.g., Webcasts) is quickly
replacing satellite broadcasts as the primary method for delivering interactive training
programs. Satellite broadcasts can be expensive to develop and deliver, difficult to
schedule, and require participants to travel to a site with satellite downlink capabilities.
Live Webcasts, on the other hand, provide busy professionals with desktop access to
courses, and technological advances have led to even greater opportunities for both
interpersonal and person-to-technology interactions. Additionally, live Webcasts can be
easily recorded and posted to a Web site for later viewing.
Many industry leaders also use a variety of self-paced training methods such as Web-
based courses and CDs/DVDs. The trend is toward providing self-paced Web courses
because they allow participants to access the materials at "their desktop and their time."
Incorporating methods for interactions into distance learning training courses is an
important best practice for enhancing their effectiveness. Our results show that many
organizations are implementing innovative strategies for increasing interactions in both
synchronous and asynchronous training courses. Some of the methods include providing
opportunities for participants to ask questions during the training session (e.g., live call-in
questions, Web boards), providing question screeners, and using simulations, case studies
and quizzes.
HayGroup Page 95
-------
EPA: Results of Benchmarking Study
V. Training Evaluation Process
Industry experts in the training field stress the importance of evaluating training programs
to ensure they add value to organizations and training participants. Training evaluation
data are critical for determining the extent to which the content of the training program
and the way in which the training is delivered results in increased skills and knowledge
for trainees, and positively impacts the organization's performance. It is only through
this evaluation process that organizations can gain insights into ways to improve training
programs, and demonstrate that they get a good return on investment from their training
efforts.
TRAINING EVALUATION LEVELS
Our research shows that despite the stated emphasis on evaluating training programs, few
organizations actually evaluate their programs beyond the commonly referred to "smile
sheets" or level 1 evaluations (see Table 6 for a brief description of commonly cited
training evaluation levels). ASTD stated in its 2003 State of the Industry Report that
75% of organizations use "smile sheets" (level 1) to evaluate their training programs.
The results from our site visits reveal a similar trend - a few training programs are
evaluated at level 2 (learning) but most training programs are not evaluated beyond level
1 (participant reactions). Level 1 evaluations, when done appropriately (e.g., are
systematically developed, use standardized questions, include a balance of multiple-
choice and open-ended questions), provide information for assessing the structure and
content of the course. This type of evaluation typically provides an assessment of the
course content, format, teacher effectiveness, delivery method, and course materials.
Type 1 evaluations are a necessary but not sufficient component of the overall training
evaluation process.
HayGroup Page 96
-------
EPA: Results of Benchmarking Study
Evaluation
Level
Level 1 -
Reactions
Table 6: Training Evaluation Levels
Description
Often referred to as "smile sheets", a level 1 evaluation measures participant reactions and
satisfaction with the training course. For example, did trainees like the training program
and was it what they expected? While any training program should be evaluated at least at
this level, it is rarely a sufficient measure of the training program's impact or of
participants' learning. For example, although a negative reaction reduces the possibility that
the training was effective, a positive reaction in no way guarantees effectiveness.
Level 2 -
Learning
A level 2 evaluation moves beyond reactions and attempts to measure changes in
knowledge, skills, and attitudes as a result of the training. The most effective way to
determine whether learning has occurred is to conduct pre- and post-evaluations to measure
absolute change in skills/knowledge.
Level 3 -
Transfer
A level 3 evaluation measures not only whether participants have learned but also whether
the learned skills are being used (transferred) to everyday work situations. Does the training
lead to changes in behavior on the job?
Level 4 -
Results
A level 4 evaluation measures the bottom line results; did the training improve quality or
increase productivity? These are the results that top line managers and executives
understand and are looking for.
See: Kilpatrick, D. (1994). Evaluating training programs: The four levels. San Francisco:
Berrett-Koehler.
As indicated previously, a few of the benchmarking partners do evaluate their training
using level 2 measures, which assess the extent to which participants have learned
required skills/knowledge. Examples of level 2 evaluations used by benchmarking
partners include: 1) quizzes or examinations to determine proficiency (many use pre- and
post-test exams to measure changes in skills/knowledge due to the training course); and
2) course certification or continuing education units (CEUs) for completing courses.
However, the benchmarking partners have found that providing certifications or CEUs is
difficult in a distance learning format because it is often difficult to monitor course
attendance. For example, SAS does not conduct any certification programs through
distance learning. They believe that there is no way to truly monitor the process unless it
is administered through classroom-based training. Georgia Tech has attempted to
alleviate this problem by requiring distance learning participants to use registered exam
proctors such as work supervisors, managers, or FIR representatives. These proctors are
responsible for ensuring that the person taking the exam is the same person that is taking
the course.
While level 1 and 2 evaluations are important and can help design better training
programs, they do not provide any substantive information relating to the ultimate
outcomes or value of the program. The evaluations typically take place at the very end of
a course when participants are tired and less likely to take the additional time to provide
insightful feedback. Also, because they are conducted solely at the end of a course, there
are no opportunities to track whether the training has led to any substantive
improvements in key organizational metrics
HayGroup
Page 97
-------
EPA: Results of Benchmarking Study
BEST PRACTICES IN TRAINING EVALUATION
In this section, we summarize training evaluation trends and best practices based on the
site visits and benchmarking research.
1. Evaluate the impact of training on job and organizational performance. Experts in the
field recommend evaluating training programs beyond levels 1 (reactions) and 2
(learning) to demonstrate the overall value of training programs. By showing that
participants are able to transfer their newly gained knowledge and skills to everyday
work tasks and connecting this knowledge transfer to improved employee and
organizational performance, a training department is able to quantify, not just say,
why training is important.
While in theory the benchmarking partners understand the importance of evaluating
training programs beyond levels 1 and 2, none are systematically measuring the
extent to which their training programs impact individual and organizational
performance. The benchmarking partners are still focusing on end-of-course
evaluations — they typically ask all trainees to complete a course evaluation form at
the end of the training course, satellite broadcast, Webcast, etc. The primary focus of
the evaluations seems to be on helping improve the overall design of the training
course (e.g., instructor effectiveness, flow of training, training materials).
Most of the benchmarking partners agree that higher level evaluations are very
difficult to do because it often takes several months before an organization may see
any change in performance and unless the training is focused on a specific skill set, it
is often difficult to demonstrate a cause-and-effect relationship between training and
performance.
However, through our benchmarking research (e.g., literature review), we identified a
few training industry leaders that are conducting level 3 and 4 evaluations of their
training programs. Below we describe some of the evaluation practices of these
industry leaders.
Example of Level 3 Training Evaluation
One organization has implemented an extensive process for evaluating the impact of its
training programs on employee performance. Each month, the training organization
selects courses for further evaluation. Training participants receive an electronic survey
approximately three months after course completion to assess how well they have applied
the training on the job. The participant's supervisor receives a similar survey asking for
an assessment of the extent to which the training has improved their employee's
performance on the job. These surveys are used to refine and improve training courses.
HayGroup Page 98
-------
EPA: Results of Benchmarking Study
Example of Level 4 Training Evaluation
Cisco Systems is using several methods to assess the overall effectiveness of its E-
Learning efforts. Examples include:
Cisco examined a sample of resellers taking a certification course (1/2 through classroom
training and !/2 through Web-based training). The pass rates for each group were
compared, and the e-learners had a 10% better pass rate than the classroom learners.
Cisco also looks at cost savings for e-learning courses. Cisco has found that Web-based
training saves millions of dollars per year because it reduces the amount of time
employees take to learn a desired skill/knowledge, and ultimately improves their
performance on the job. Other cost savings include reduced travel cost to attend
classroom training and less time away from the customer (which increases productivity).
Taken from: Mission E-Possible: The Cisco E-Learning Story (Patricia A. Galagan,
February 2001).
Training evaluation experts provide a good piece of advice to help move to level 3 and 4
evaluations: begin the planning process early. Organizations must define the desired
changes in employee or organizational performance and quantify how they will be
measured before developing the training program. These defined changes should help
shape the overall evaluation process so that a clear connection can be made between the
training program and desired results. Level 3 and 4 evaluations can be accomplished, but
they take time and planning.
2. Conduct pre-and post-course evaluations to track learning as a result of the training
course. Many of the benchmarking partners conduct pre-course evaluations as a
baseline to measure learning throughout the courses. Pre-course evaluations provide
an excellent measure of the knowledge, skills, and abilities that participants bring to a
course. Understanding what participants bring to a course not only allows instructors
to tailor their messages, but it also provides a way to measure change due to the
course (a level 2 assessment of the effectiveness of the training),
3. Collect feedback from multiple sources. Industry leaders collect data from multiple
sources when evaluating training programs. Sources may include training
participants, participants' supervisors, training developers and instructors, site
coordinators, and organizational leaders. Taken together, these data can provide a
complete picture of the effectiveness of the training content and structure, and most
importantly, the impact of the training on participant job performance and overall
organizational results.
4. Regularly review and use evaluation data to improve the effectiveness of training
courses. Although the benchmarking partners typically focus on level 1 and 2
evaluations, they are taking these evaluations seriously by monitoring the feedback
provided by participants in training courses, and most importantly, using the feedback
HayGroup Page 99
-------
EPA: Results of Benchmarking Study
to make improvements to training materials, the way training is delivered, instructor
effectiveness, the technology, and so forth. They are using these data to make
decisions about whether to modify or redesign training programs or eliminate
ineffective programs.
Example of Using Evaluation Data to Improve Training Programs
SAS has one of the most extensive evaluation procedures of the benchmarking partners,
but the evaluation still does not go much beyond a typical "smile sheet." SAS
evaluations occur immediately following a course. Courses that do not meet a minimum
average rating of 3.5 out of 4 on key evaluation questions are further examined to assess
why the ratings are low. SAS will directly contact customers to elicit further information
about why courses were reviewed less favorably than expected.
5. Extend training evaluation beyond classroom courses. Organizations tend to focus
on evaluating classroom training and do not conduct extensive evaluations of their
Web-based or satellite courses. Industry experts recommend that organizations
moving away from classroom training conduct evaluations to ensure that Web-based
training: 1) is as effective as classroom training at teaching required skills and
knowledge; 2) engages trainees to begin and complete courses; and 3) provides a
cost-effective alternative to classroom training.
Examples of Evaluations of Web-based Courses
• In its E-learning Handbook, the American Society for Training and Development
recommended the following types of measures for evaluating the effectiveness of
self-paced Web training:
• Total number of training hours - to determine whether Web training reduces training
time (which saves the organization money)
• Attendance and retention rates (Did employees actually complete the Web-based
course?)
• End-of-course questionnaires to obtain participant feedback on the effectiveness of
the Web course (Was the Web the best method for teaching the skill/knowledge?
Was the Web site easy to access and navigate? Were the quizzes and simulations
effective?)
• Pre- and post-tests to determine whether participants increased their knowledge as a
result of the Web training course
• Cost reduction - Did the Web course reduce training delivery cost (e.g., travel)?
• Productivity - Did the Web course help to prepare new employees to be productive
on their jobs quicker than more traditional training methods?
6. Use standard evaluation procedures. The benchmarking partners revealed that
another key area to consider in training evaluation is standardization of evaluation
instruments. For example, although the CDC has five standard evaluation questions,
its clients are encouraged to include additional questions. While this may help
HayGroup Page 100
-------
EPA: Results of Benchmarking Study
customize the process, the overall result is a lack of standardization beyond the five
questions. Additionally there is a lack of standardization for when evaluations are
conducted. While the CDC evaluates most asynchronous courses, it is not standard
practice to evaluate synchronous courses.
Without a standard process, it is not only extremely difficult to monitor the overall
impact of training courses, but also to develop a common goal for all courses.
Standardization of the process will help connect overall training goals and needs by
focusing on a core set of measures that are changed only after all training goals have
been met.
7. Consider unique ways to evaluate training programs. The benchmarking partners
demonstrate some unique methods of evaluating their training programs. Both
GMAC and SAS believe if training participants find the courses valuable, they will
continue to enroll in, and in SAS' case, purchase the courses. For example, SAS
monitors the 18-month re-buy rate of its training programs. SAS believes that a
successful training program is one in which 60% or more of customers re-buy
training.
SUMMARY
APTI is not alone in the challenge to conduct effective evaluations of its training
programs. However, like most of the benchmarking partners, APTI needs to reexamine
its training evaluation process to ensure that it is measuring the "right things" - the extent
to which the training is adding value to air professionals and their agencies as a whole.
To conduct such best practice evaluations, APTI should move its evaluation level up to
transfer and results. That is, focus on evaluating the extent to which training programs
affect the performance of air professionals and key organizational metrics.
HayGroup Page 101
-------
EPA: Results of Benchmarking Study
VL Conclusions and Next Steps
In this report, we identify best practices of leading training programs, particularly in the
areas of needs assessment, course design and content, training delivery methods
(specifically distance learning), and training evaluation. The best practices were
identified based on site visits with five organizations with innovative training programs,
as well as a literature review of current training trends and best practices.
We believe that there are several best practices that APTI should consider incorporating
into its Air Pollution Training Program to reduce the gaps that were identified during the
evaluation of the current training program (during Task 1). For example, in the Task 1
report, we concluded that APTFs distance learning training courses (e.g., satellite
broadcasts, self-paced Web courses) did not include enough opportunities for interactions
between participants and instructors, among participants, or with the technology itself.
In this section, we present a high level summary of those best practices that, if adopted,
may improve the overall effectiveness of the Air Pollution Training program. The Task
3 report will provide more detailed options for improving the APTI training program and
a plan of action for doing so.
Training Needs Assessment
1. Incorporate a skill/competency assessment into the needs assessment process
versus simply projecting the number of air professionals likely to attend various
classroom training courses. This type of assessment will provide valuable
information for determining the gap between the skills/competencies required for
success in the job and the levels currently possessed by air professionals. This
should be followed by a thorough review of the current APTI training courses to
ensure they are meeting the training needs of air professionals.
2. Collect data from multiple sources (e.g., air professionals, supervisors of air
professionals, regional consortium members, training instructors and subject
matter experts) about the need for different types of air pollution training courses
(both course content and delivery methods).
3. Use the results of training needs assessments to design more effective training
programs, and communicate decisions about training courses (e.g., eliminating a
course, adding a new course, translating a course from the classroom to a Web-
based medium) to the air professionals community.
Course Design and Content
4. Conduct regular reviews of training courses (e.g., every 1-3 years) and update
courses as needs. For example, some updates may require simply replacing
content in a self-paced Web course. Others may require major overhauls or total
redesign of a course.
HayGroup Page 102
-------
EPA: Results of Benchmarking Study
5. Incorporate real-world examples, simulations, and case studies into self-study
Web courses.
6. Pilot test all courses before they go live. For example, pilot test participants
should complete and evaluate all self-paced Web courses (e.g., for content, ease
of using the technology, value of interactive exercises) and feedback from
participants should be used to improve the Web courses. As another example,
live Webcasts and satellite broadcasts should be rehearsed and the scripts
reviewed before being delivered to air professionals.
Training Delivery Methods
1. Begin to make more use of live Webcasts (versus satellite broadcasts) to allow air
professionals to complete the training at their desktops versus travel to a satellite
downlink site.
8. Incorporate methods for interaction into distance learning courses (e.g., self-paced
Web courses, live Webcasts, satellite broadcasts) to enhance their effectiveness.
For example, provide opportunities for participants to ask questions of the
instructor and use question screeners during live Webcasts or satellite broadcasts
(e.g., phone or fax in questions). For self-paced Web courses, incorporate
methods for: 1) student-to-instructor and student-to-student interactions (e.g.,
chat rooms, Web boards); and 2) interactions with the technology (e.g., real-world
simulations, case studies, quizzes).
9. Ensure that individuals who deliver training (e.g., satellite broadcasts, live
Webcasts) not only are subject matter experts but effective at teaching with the
delivery medium being used.
Training Evaluation
10. Expand the training evaluation process beyond end-of-course level 1 evaluations
("smile sheets"). Determine the extent to which the training actually helps air
professionals to do their jobs effectively and contributes to the mission of their
agency. For example, solicit feedback from air professionals and their
supervisors 3-6 months after the training to determine the extent to which the
training helped air professionals to be more effective on their jobs.
11. Monitor and use evaluation data to make improvements to training courses (e.g.,
training materials, the way training is delivered, effectiveness of simulations or
case studies, ease of use of the technology). Decisions about whether to modify,
update, redesign, or eliminate training courses should be based on data collected
via training evaluations.
HayGroup Page 103
-------
EPA: Results of Benchmarking Study
Our next step in the benchmarking study is to obtain input from APTI into those best
practices that are most cost effective and feasible to implement. We will facilitate a
discussion with the APTI team to determine:
> Extent to which the best practices can be incorporated into APTFs culture,
processes, and procedures
> Whether APTI has the resources (e.g., dollars, staff) to implement the best
practices
> Barriers that may impede successful implementation of the needed changes to
the program
> Key accountabilities for needed changes
After these discussions with the APTI team, we will write the final Task 3 report which
will provide options for making APTI more successful moving forward, and a plan of
action for incorporating relevant best practices into the Air Pollution Training program.
HayGroup Page 104
-------
EPA: Results of Benchmarking Study
Appendix El: Site Visit Protocol
HayGroup Page 105
-------
EPA: Results of Benchmarking Study
Questions for Benchmarking Partners who Participate in Site Visits
Environmental Protection Agency
Benchmarking Study
Background
1. What is the overall mission of your group?
2. Describe your trainee population.
3. How large is your staff?
4. What is your annual budget?
5. How would you characterize the organization's commitment to your group's
mission?
Overview of Training
6. How many courses do you deliver per year?
a. By delivery method (e.g., classroom, Web-based, satellite)
b. By type of course content
7. Describe the different modes used to deliver training.
Course Design & Content
8. For each type of training (classroom, Web-based, satellite),
a. What is the process for course design and development?
b. How long does it take to design and develop the course?
c. What is the cost to design and develop the course?
9. Within a given year, how many new courses would typically be developed?
a. Do you primarily develop new courses from scratch or update current
courses?
b. How do you make decisions about whether to develop a new course or
update an existing one?
10. What is the process for developing course content for new courses?
11. Describe the process(es) used to update courses.
a. How often do you update courses?
12. What are best practices for the development of content for satellite training
course s/program s?
13. What are best practices for the development of content for Web-based training
courses?
14. What actions have you taken to incorporate interactive components into the
design of distance learning courses?
HayGroup Page 106
-------
EPA: Results of Benchmarking Study
Needs Assessment
15. How do you determine what type of courses are needed?
16. Do you perform a formal training needs assessment?
a. If so, how often? Who is involved?
17. How do you use the needs assessment to make decisions about training content
and delivery methods?
Delivery Methods
18. How do you determine what type of delivery method to use? How do you make
decisions about the mix of classroom and distance learning courses?
19. In your opinion, what are some of the innovative ways you are currently
delivering training?
20. What have you done to ensure that distance learning courses (e.g., satellite, Web-
based) are effective?
21. Do your distance learning courses have an interactive component? If so, please
describe how you make your distance learning courses more interactive.
22. What do you see as the primary cost-benefit issues for EPA to consider when
deciding whether or not to offer course content using delivery modes other than
classroom training?
23. How do you ensure that training is delivered in a timely manner to new hires or
those who require just-in-time training?
24. Do you believe that modes of delivery other than classroom training are as
effective at facilitating learning? Why or why not?
Instructors
25. Do you use outside vendors to deliver training courses? If so, what is your
process for selecting vendors?
26. Where do the course instructors come from (e.g., university, organizations)?
27. How do you measure the effectiveness of the instructor (for classroom and
distance learning courses)?
Course Evaluation
28. How do you ensure course content and delivery are kept to the highest possible
standard?
29. What types of training evaluation have you undertaken? Describe the course
evaluation process.
30. How do you assess whether your training is effective or that you have received a
return on investment from training?
31. How do you use training evaluation data?
HayGroup Page 107
-------
EPA: Results of Benchmarking Study
Future
32. What have been some of your "lessons learned" with respect to distance learning?
33. What makes some training courses more effective than others?
34. Are you involved in any type of train-the-trainer program (i.e., training outside
groups to take over some of the training currently being done by your group)?
35. What do you think the future holds for distance learning training versus
classroom training?
HayGroup Page 108
-------
EPA: Results of Benchmarking Study
Appendix B2: Summaries of Individual Site Visits
HayGroup Page 109
-------
EPA: Results of Benchmarking Study
Site 1: Occupational Safety and Health Administration (OSHA) Office of Training
& Education
Site Summary
The OSHA Office of Training and Education (OTE) is responsible for providing occupational
safety and health training programs for Federal and State OSHA compliance personnel,
consultation staff, other Federal agency personnel, and private sector employers and workers.
OSHA training programs emphasize the recognition, avoidance, prevention, and abatement of
unsafe and unhealthful working conditions.
With a staff of approximately 50 professionals, eighteen of which are instructors, the OSHA
Training Institute focuses on professional development courses that encompass either a
classroom or a blended approach (i.e. classroom and Web-based). OSHA offers approximately
80 courses per year and has ten education centers around the U.S. which provide additional
training to other federal agencies as well as the private sector. OSHA uses recommendations
from staff or changes to regulations in order to determine the needs assessment for new or
revised courses. A functional competency model has been developed to help in needs
assessments going forward.
Needs Assessment
OSHA uses feedback from instructors, field personnel and changes in regulations to keep
up to date on what training is needed.
Course Content, Design, and Training Facilities
The OSHA Training Institute (OTI) will have delivered approximately 90 classroom or
blended courses in 2004. To ensure that information is up-to-date, "Course Chairs" are
assigned to each course. A "Course Chair" is someone within the department who takes
ownership of a course. They are responsible for keeping information current. Typically
they review information every three years unless there is an immediate change or need to
be addressed.
New courses are rare at OSHA. However, OSHA has developed a seven phase
development process to assist Subject Matter Experts in the development of new web-
based courses and transferring classroom content onto the web. This is done through
utilizing a software package called Workforce Connections. OSHA offers courses both
on the road and at their facility. As many courses include some laboratory component
and OSHA has many of these labs set up for experimentation, these typically need to be
held on-site.
Delivery Methods and Participant Interaction
OSHA employs three primary delivery methods: 1) Classroom, 2) Web-based, and 3)
Blended. All three provide specific opportunities for participant interaction. In its Web
HayGroup Page 110
-------
EPA: Results of Benchmarking Study
Based Training Style Guide, OSHA recognizes four levels of interactivity: passive,
limited, complex and real-time.
Classroom. Classroom courses are held either on-site at the OSHA Training Institute or
on the road. Classes at OSHA typically include some laboratory portion. These courses
provide real-time participation.
Web-based. Internet courses are delivered in a traditional web-based format.
Participation levels for web based training must include limited and complex levels.
Blended. Blended courses include a Web-based portion of the course prior to the student
coming to the classroom. The Web-based courses include limited and complex
participation while the classroom includes hands-on instruction.
Course Evaluations
Evaluations of both course content, and delivery and technology are typically conducted
at the end of each course. Evaluations focus primarily on content and teacher instruction.
However, Web-based and blended courses include an evaluation in the development
stage, prior to going live. Volunteers take a pre-test, the course and a post test. Then
they fill out an evaluation including errors, concerns, questions and suggestions. They
have a one-hour conference call with all volunteers to review the evaluations and gather
additional information. After this is taken into account and any changes are made, this
portion of the course goes live.
Lesson Learned/Future Directions
> Blending courses with a web-based component allows students to review
general content prior to the class meeting. This also allows all students to
come to into this portion of the course with the same level of knowledge and
more time for hands-on instruction.
> Workforce Connections has been a significant time and cost savings
improvement on web-based course development. It allows subject matter
experts to design and maintain the courses in a user friendly manner.
> Some courses can not be solely web-based as they require a hands-on portion.
> Competency-based training is seen as an important initiative.
HayGroup Page 111
-------
EPA: Results of Benchmarking Study
Site 2: The Centers for Disease Control Public Health Training Network
Site Summary
The CDC Public Health Training Network (PHTN) is a network of public, private,
academic, and business organizations with the mission to develop a public health
workforce that is able and motivated to apply the current knowledge of disease control to
reduce human suffering, improve community health, and control health care costs. The
primary CDC customers are public health officials including, among others, physicians,
nurses, emergency response personnel, mental health facilities, veterans affairs, and
school of public health.
With a staff of approximately 50 professionals, the CDC focuses on programs concerned
with all aspects of public health. The CDC has the capacity and ability to deliver long-
term courses as well as react to immediate needs and produce an entire course in less than
48 hours. Some examples of rapid turn-around issues are smallpox, bioterrorism, and
SARS. CDC programs have reached an estimated audience of 4,806,680 individuals.
Finally, the CDC supports the "learner at a distance" by establishing learning
communities and self-study group leaders to provide guidance to CDC program
participants.
Courses are delivered in both synchronous and asynchronous formats using a variety of
media including the Web, satellite, and videoconference. The CDC continuously
conducts needs assessments to determine individual course needs and pre-tests Web-
based courses to ensure applicability and usability. All courses are evaluated and
designed to promote interactions.
Needs Assessment
The CDC currently conducts needs assessments to determine whether a specific course is
needed. However, these assessments are not standardized and are not conducted with
every course. The CDC is working to become more goal-focused and plan to conduct
more needs assessment to ensure that training programs meet stated goals.
Course Content, Design, and Delivery Facilities
The CDC develops approximately 30 - 40 new courses per year and has produced 853
products focused on terrorism and emergency response programming. The technology
utilized for these courses includes:
> Internet,
> Satellite transmission,
> Cable TV,
> Audio conferences,
> CD-Rom/DVD,
> Video tape, and
> Print-based self learning.
HayGroup Page 112
-------
EPA: Results of Benchmarking Study
Course development is primarily initiated by a client outside the CDC, and clients have
primary responsibility for updating course content. Clients may also ask the CDC to
update the content, which results in a new project rather than a continuation of the
existing project. CDC has established relationships with several external distance
learning networks that help distribute CDC programs at little or no cost.
A project typically begins when a client (various CDC divisions) contacts the PHTN
about developing a course. The CDC typically provides instructional design expertise
and expects the client to provide content expertise. Although most projects are initiated
by a client, the CDC has established different processes for developing and delivering
courses for the different types of media. The different development and delivery
processes are described below.
Internet Course Development.
> Course design and the look of the training are developed in-house by an
instructional design expert and graphics specialist.
> The CDC provides a web developer (contractor) with a Power Point briefing.
The web developer then posts the training to the Internet.
> The CDC found that using a contractor is more expensive than hiring an in-
house web developer.
> Courses with continuing education credits are pilot tested to determine length
and applicability to assess the number of credits to be offered. Pilot testers are
composed of at least five representative of the target audience.
> A clearance process is initiated to finalize the program.
> A web-based training course of 120 pages takes about 3-5 months and
$120,000 to develop.
Satellite Course Development.
> The client and subject matter experts (SMEs) provide the CDC with content
information. The CDC requests that clients also provide an outline of course
content.
> The CDC hires a script writer to develop a broadcast.
o SMEs often don't want the entire broadcast to be scripted; however, this is
necessary to make certain the broadcast flows as seamlessly as possible.
> A full dress rehearsal is conducted the day before the broadcast.
Online Registration System
CDC has an online registration system that serves as one source for marketing,
registration, testing, evaluation, continuing education certificates, and learner transcripts.
This system allows students to search and register for courses, complete course
evaluations and tests, and view and print transcripts and education certificates.
HayGroup Page 113
-------
EPA: Results of Benchmarking Study
Delivery Methods and Participant Interaction
The CDC employs two primary training delivery methods: 1) Internet, and 2) Satellite.
Each delivery method provides specific opportunities for participant interaction. The
methods for interaction are described below.
Internet. Internet courses are delivered in either a traditional web-based (asynchronous)
or a Webcast (synchronous) format. Methods of enhancing student interaction are
limited and include:
> Web boards and threaded discussions, and
> E-mail links to send instructors questions.
Satellite. Satellite courses include live broadcasts that provide participants the
opportunity to interact directly with the instructor. Communication can occur either
before or during the broadcast. Methods of interaction include:
> Call-in questions,
> E-mail questions and
> Fax-in questions.
Satellite Course Delivery.
> Distance learning coordinators are available in every state to promote and
assist with broadcasts.
> Moderators are available during each broadcast to help maintain flow and
screen questions.
> Most broadcasts are done live and the CDC tries to get subject experts to
deliver the broadcasts.
> The CDC asks clients to develop at least 10 questions in case there are no
audience questions.
> Some broadcasts are also delivered via Webcasts.
> Continuing education credits are often available.
> Some broadcasts are recorded for later delivery via CD-Rom or the Internet.
> Participants are invited to provide comments about the program.
The CDC has a call center used to accept calls through a posted 1-800 number. Four to
five individuals are typically used to accept calls and screen questions. If a question is
accepted the question can be broadcast live. Questions that are e-mailed may be
answered live or posted to an asynchronous web site.
Course Evaluations
Evaluations occur before and during course development and following course delivery.
For each web-based course, there is a formal evaluation during the development phase to
ensure that the content and design are appropriate for the intended audience. These pilot
test evaluations collect important information about the appropriateness of course content
and style from those most likely affected by the training; potential participants, training
coordinators, and site facilitators. Satellite courses are not subjected to as rigorous of a
HayGroup Page 114
-------
EPA: Results of Benchmarking Study
pilot test procedure because satellite courses are synchronous, easier to modify, and more
difficult to script.
Both synchronous and asynchronous courses are evaluated by participants upon
completion for content and delivery. The CDC uses standard level 1 evaluations ("smile
sheets"), and has five standard evaluation questions that clients may add to (e.g., Did you
learn what you expected? Did the training meet your objectives?). Because clients
typically add their own questions the CDC has had difficulty standardizing the entire
evaluation process. Satellite courses are not evaluated as regularly as other courses
because they are typically one-time sessions. However, many clients will evaluate them
to help improve future broadcasts.
Course Examinations
The CDC encourages clients to administer examinations to students in asynchronous
courses to help determine whether the entire course was completed by the same
individual. Because synchronous courses are typically one-time events, a course
evaluation is typically deemed sufficient to track completion. Examinations are further
used to award continuing education credits.
Lesson Learned/Future Directions
> It can take a long time to get training content cleared/approved.
> It is best to have one primary client representative.
> "Don't buy it when you can borrow it."
> Publish locations of satellite links on your website.
> Hire a script writer to make satellite content more conversational.
> Do a dress rehearsal when possible for satellite broadcasts.
> It is more expensive to hire outside contractors than use internal resources.
HayGroup Page 115
-------
EPA: Results of Benchmarking Study
Site 3: Georgia Tech Distance Learning and Professional Education Department
Site Summary
The primary mission of the Georgia Tech Distance Learning and Professional Education
(DLPE) Department is to provide professional education courses for both internal
Georgia Tech and external clients, including both public and private organizations. The
Department offers programs that are designed to strengthen and update existing skills and
also teach new skills to help individuals achieve their own and organizational goals. The
Department further helps to facilitate Georgia Tech's distance learning capabilities by
taping conventional Georgia Tech academic courses for use in distance learning.
With a staff of approximately 60 professionals, Georgia Tech focuses on professional
development, non-credit courses that range between one and five days in duration. In
addition, over 40 Certificate Programs are offered for which participants are eligible to
receive Continuing Education Units (CEUs). Finally, Georgia Tech offers conferences
and seminars, has the capability to coordinate remote meetings via teleconference, and
provides instructional design and consultation.
Courses are delivered in both synchronous and asynchronous formats using a variety of
media including the Web, satellite, and videoconference. Georgia Tech continuously
conducts needs assessments to determine individual course needs as well as general
training needs. All courses are evaluated and designed to promote interactions between
participants.
Needs Assessment
Georgia Tech continuously conducts needs assessments to determine the market's needs
for proposed courses as well as overall industry training needs.
Course Content, Design, and Training Facilities
Georgia Tech will have delivered 113 courses in the 2004 academic year (about 45
courses per semester plus summer), including 1000 course enrollments, 3000 student
credit hours, and 95 faculty members. To ensure that information is up-to-date,
productions are updated with each subsequent course. The technology utilized for these
courses includes:
> Internet,
> Satellite transmission,
> Cable TV,
> ITFS - "Wireless Cable",
> MPEG2,
> Voice Over Internet Protocol (VOIP),
> CD-Rom/DVD, and
> Video tape.
HayGroup Page 116
-------
EPA: Results of Benchmarking Study
Course development is primarily initiated by either a Georgia Tech faculty member or an
external customer, and course content is primarily the responsibility of the customer.
Georgia Tech will contract with faculty to assist with the development of course content;
however, the primary roles of Georgia Tech are course design and delivery. In addition,
Georgia Tech assists students with course registration, textbook purchases, and receipt
and delivery of homework, reports, and examinations. The services provided by Georgia
Tech include:
> Video production. Video production is primarily geared toward producing
videotapes for distance learning. Georgia Tech has 9 classroom/studios, each
with a high bandwidth Internet connection, computer, four remotely
controlled cameras, two front displays, two rear monitors, one portable control
panel, and a VHS handheld camera.
> Tape duplication and video editing. Georgia Tech has approximately 80 VHS
duplication decks, and linear and non-linear editing capabilities.
> Optical Media duplication. Georgia Tech can duplicate and print labels for up
for 1000 CD-Rom and DVDs.
> Teleconferencing/Satellite. Georgia Tech has 9 Tandberg teleconferencing
units, a 12-site MCU used to combine dissimilar VTC formats into a common
one, and three satellite dishes.
> Streaming video. Georgia Tech can encode video for streamlining format at
56kbps or with a DSL, Cable modem, ISDN, or Tl line.
Delivery Methods and Participant Interaction
Georgia Tech employs three primary delivery methods: 1) Internet, 2) Satellite, and 3)
Videoconferencing. Each delivery method provides specific opportunities for participant
interaction. The methods for interaction are described below.
Internet. Internet courses are delivered in either a traditional web-based or a Webcast
format. Methods for incorporating interaction include:
Traditional Web-based
> Web boards and threaded discussions,
> E-mail links to send instructors questions, and
> Face-to-face classroom discussion (when possible);
Webcasts
> Chat rooms,
> Voice Over Internet Protocol (VOIP), and
> Web boards and threaded discussions.
HayGroup Page 117
-------
EPA: Results of Benchmarking Study
Satellite. Satellite courses include live broadcasts that provide participants the
opportunity to interact directly with the instructor. Communication can occur either
before or during the broadcast. Typically a moderator is available to take and filter
questions before giving them to instructor. Methods for incorporating interaction
include:
> Call-in questions, and
> Fax-in questions.
Videoconferencing. Videoconferencing provides the best opportunity for direct
interactions between instructors and students. Because both parties are able to hear and
see each other interactions most typically take place in the form of a continuous
discussion.
Course Evaluations
Evaluations of both course content, and delivery and technology are typically conducted
at or near the end of each course. Evaluations focus primarily on content and teacher
effectiveness and are often referred to as "smile sheets". However, evaluations of course
delivery and technology are also conducted to help Georgia Tech further understand
current trends in distance learning. Sample evaluation items include:
> The video signal is clear and easy to see
> The video operator shows what the instructor is pointing at
> Course materials are received in a timely manner
> The website is comprehensive and easy to use
Course Examinations
Georgia Tech encourages the integration of student examinations into all courses.
Registered proctors are made available and students are required to select a local proctor
for examination administration. One issue with distance learning, because of a lack of
oversight, is that it is difficult to ensure that the person taking the exam is the person
taking the course. Georgia Tech has attempted to alleviate this problem by using and
requiring registered exam proctors. Proctors must be a supervisor, manager, or HR
representative.
Lesson Learned/Future Directions
> Distance learning is going desktop, on demand.
> Presenters/instructors must have teaching abilities.
> Just because the capacity to do the biggest and best is available, you don't have
to use it. Target your programs to your audience and their needs.
> Make training actionable.
> Make sure people have the resources to receive training (e.g., Internet, satellite
receivers, CD players).
> Provide continuous technical support to end users.
HayGroup Page 118
-------
EPA: Results of Benchmarking Study
> Be aware of firewalls that will prevent end users from accessing the Internet
(especially in the government).
> Do your best to make the program interactive.
> Package materials so they are inviting.
> Don't send anything you don't want copied (e.g., CDs).
> It is easier to support class cohorts than individual students.
> For every 1 hour of instruction expect 3 hours of post-production time.
HayGroup Page 119
-------
EPA: Results of Benchmarking Study
Site 4: GMAC Commercial Mortgage (GMAC) Staff Development Division
Site Summary
GMAC Commercial Mortgage (GMAC) began developing and providing distance
learning programs in February, 2002 and considers itself "young at what we do". The
mission of the Division is to provide programs that focus on 1) servicing-based
knowledge; 2) the mortgage business in general; and 3) personal and leadership
development. GMAC sees it role as not only teaching/training, but as communicating
information to employees. The teaching it provides is vendor based, GMAC does not
have a large staff.
With a staff of approximately three professionals and one student intern, the Staff
Development Division uses the following training delivery methods: Intranet,
Videoconferencing, videotapes, and classroom-based courses. The Division is looking to
add "the wow factor" to its courses; GMAC agreed to participate in the benchmarking
meeting partly as a way to learn from us what others are doing in distance learning.
Examples of programs that the Division has provided include book studies, teachings by
ratings agencies (e.g., Moodys), and leadership development. Students can earn credit
for some GMAC courses.
Courses are delivered in both synchronous and asynchronous formats using a variety of
media including the Web and videoconference. GMAC conducts informal needs
assessments to determine individual course needs as well as general training needs. All
courses are evaluated and designed to promote interactions between participants.
Needs Assessment
GMAC conducts some informal needs assessments to determine what courses are needed
and what people like. This is accomplished primarily through talking with employees to
determine the types of training they need/the types of training courses GMAC should
deliver. The Division has done some needs assessment surveys in the past to determine
what types of delivery methods students prefer.
Course Content, Design, and Training Facilities
The GMAC Staff Development Division is relatively young and has done some
impressive work with a small staff. It delivers between 20 and 30 training sessions each
year and use Web-based, videoconferencing, videotapes, and classroom-based
technologies including the following:
> Intranet,
> Live streaming,
> Web ex,
> Windows Media Encode,
> CD-Rom/DVD (non-interactive), and
> Videotape (GMAC has a large library of videotapes).
HayGroup Page 120
-------
EPA: Results of Benchmarking Study
Course development is primarily initiated by internal GMAC when there is a stated
interest in a specific topic. GMAC then either contracts with an outside subject matter
expert (e.g., professor) or uses internal staff to develop the course. Courses have been
conducted as panel discussions, small group studies, and traditional presentations.
Previous courses have covered topics such as real estate, property management, asset
management, commercial mortgages, and leadership development.
Delivery Methods and Participant Interaction
GMAC uses two primary delivery media: 1) Web, and 2) Videoconferencing. Most of its
programs are broadcast live and taped (audio and video) for later viewing using Windows
Media Encoder. GMAC is moving towards doing more videotaping of live sessions
because it is finding that live audiences are often smaller than anticipated. Some
programs are conducted using Webex; however, GMAC has had trouble recording both
audio and video with Webex. Finally, GMAC has the capacity to record programs to
CD-Rom/DVD for delivery to individuals who were unable to attend the original
broadcast. Each delivery method and methods for interaction are described below.
Web. GMAC uses its Intranet, rather than the Internet, to broadcast courses using Web-
based technology. GMAC does not have interactive Web-based (e.g., Webcast) courses,
but rather use the Intranet as a means to stream live videos (one-way) to participants.
Although streaming occurs only one-way, GMAC has developed some methods for
participant interaction that include:
> E-mailing questions (that the Division monitors)
> Calling in questions that are answered live (via an open telephone line)
HayGroup Page 121
-------
EPA: Results of Benchmarking Study
Videoconferencing. Videoconferencing provides the best opportunity for direct
interactions between instructors and students. Because both parties are able to hear and
see each other, interactions primarily take place in the form of a continuous discussion.
Communication can occur either before or during the broadcast. Typically a moderator is
available to take and filter questions before giving them to the instructor. Methods for
incorporating interaction include:
> Call-in questions
> Fax-in questions
Course Evaluations
Some level 1 evaluations are conducted, but GMAC conducts relatively few evaluations.
The Division has the philosophy that "if they find value, they will come." And, because
there is continued enrollment, GMAC believes that the teachings are successful. When
the Division does survey, it often uses online survey tools such as Zoomerang.
Course Examinations
GMAC does conduct some quizzes; however, there is little focus on examinations. To
prevent participants from simply viewing part of a course the Division has programmed,
asynchronous sessions are used so that participants cannot fast-forward through the entire
session. Once a student completes the session, they are provided a code and asked to log-
in the code as proof they completed the course.
HayGroup Page 122
-------
EPA: Results of Benchmarking Study
Site 5: SAS Education to Customers
Site Summary
The primary mission of the SAS Education to Customers (SAS) division is to provide
technical training to SAS customers. Approximately 90% of the training provided is
task-oriented (how to use SAS products) with the remaining 10% focused on technical
overviews (why a product is used), often as a precursor to the task-oriented training. All
the training provided is fee-based; however, the SAS Education to Customers division is
not a profit center. It seeks to cover costs while supporting SAS customers to ensure they
are able to effectively use, and continue to purchase, SAS products.
SAS provides classroom-based, live Web, and self-paced (Internet, CD Rom) training.
Classroom-based training accounts for about 80% of all training and typically occurs at
SAS' 28 training facilities. Thus far in 2004, SAS has provided 120 unique training
episodes (courses) and 250 training sessions to about 1500 customers.
SAS conducts an annual needs assessment survey to determine what courses to deliver in
the upcoming year. The survey consists of about 400 items designed to determine
whether and how customers use software (importance), when they last used the software
(recency), and how often they use the software (frequency). In addition, SAS monitors
product sales and other data to determine courses that customers are likely to need. All
courses are evaluated using "smile sheets" and require an average rating of 3.5 out of 4.
SAS also monitors the re-buy rates and believes that customers will not re-buy if they
don't feel that training is effective.
As a policy, SAS will only conduct certification programs through classroom-based
training. They have determined that classroom-based training is the only way to ensure
that the person taking the certification test is the person signed up for the course.
Needs Assessment
SAS conducts an annual needs assessment survey to assess what courses to deliver in the
upcoming year. The survey consists of about 400 items designed to determine whether
and how customers use software (importance), when they last used the software
(recency), and how often they use the software (frequency). In addition, SAS monitors
product sales and other data to determine courses that customers are likely to need.
Course Content, Design, and Delivery Facilities
SAS is primarily a software development company and SAS is responsible for providing
technical training to SAS customers. Therefore, SAS develops courses and uses training
delivery methods that are customer-driven. The technology utilized for these courses
includes:
> Internet (MS Live Meeting),
> CD-Rom/DVD,
HayGroup Page 123
-------
EPA: Results of Benchmarking Study
> Videotape,
> Print-based self learning, and
> Classroom-based technologies.
SAS maintains 28 classroom-based training facilities throughout the United States where
customers and SAS instructors can meet for face-to-face trainings (about 30% of
classroom-based instruction is conducted at customer sites). All live Web and self-paced
training courses are developed and delivered using facilities maintained at SAS
headquarters in Gary, NC. SAS has 77 instructors, 20 of which are certified by SAS to
teach one or more live Web courses. Because the nature of teaching is different for live
Web and classroom-based courses, SAS requires instructors to take courses on providing
training over the Web before they are allowed to provide live Web instruction.
Instructors must also be certified in a course before they are allowed to teach it.
Course development is primarily initiated after review of customer needs as determined
by an annual needs assessment survey and software sales and usage data. Because SAS'
existence depends on expanding and retaining its customer base, courses are developed
that provide existing and potential customers the know-how to effectively use SAS
products.
SAS has developed both absolute rules and rules-of-thumb that must be considered when
developing any course. These rules were developed because SAS acknowledged that
people are likely to physiologically tune-out during training. Four absolute rules have
been developed for courses that are not classroom-based. They include:
1) Participants must be given the opportunity to interact at least every 10 minutes
> Ask questions about things just completed
> Ask open-ended questions about general training subjects
> Provide and complete short quizzes
2) Instructors must have specific training
> Web delivery is different than classroom
3) Instructors must have two rehearsals before they can teach a course
> Instructors must demonstrate they can effectively instruct over the web
> Instructors must show they understand the material
4) Moderators must be present for each course session
> Each course has at least one instructor, moderator, and technical support staff
Some additional rules-of-thumb that SAS considers when designing a course are:
1) Course design and delivery depend on complexity of the material, the need for
interactivity, validation for certification purposes, and sensitivity of issues
2) No course session should be longer than a half-day
3) Classroom-based courses are best for interactions
> Best for validation
> Best for sensitive issues
4) Self-paced is best for informational courses
> Easy to scale
> Cheap to develop and deploy
HayGroup Page 124
-------
EPA: Results of Benchmarking Study
> HTML has low standards - doesn't need much bandwidth and can be recorded
easily to CD-Rom/DVD
SAS currently uses MS Live Meeting and Place Where, and is considering using Webex,
to deliver live Web training. They believe that content delivery methods are more
important than tool selection. SAS makes tool selection decisions based a lot on cost and
whether the vendor will be around in a few years, with a few assumptions. The
assumptions are that each tool has:
> Application sharing
> PowerPoint capabilities
> Chat capabilities
> Registration capabilities
Delivery Methods and Participant Interaction
SAS employs three primary delivery methods: 1) Classroom-based, 2) Live Web, and 3)
Self-paced. Each delivery method provides specific opportunities for participant
interaction. The delivery methods and strategies for interaction are described below.
Classroom-based. Classroom-based courses account for about 82% of SAS courses and
are delivered at either one of SAS' 28 training centers or at a customer site. Training is
provided by one of SAS' 77 instructors. The courses are designed for optimal
interactivity using the latest in face-to-face training skills.
Live Web. Live Web courses account for about 10% of SAS courses. When SAS began
delivering live Web courses (June, 2001) they believed that a key to live Web success
would be providing students the ability to interact, without stopping training. To do this,
SAS incorporated the following:
> Text questioning - Participants can e-mail questions during the training
> Phone questions - All questions come in privately before decisions are made
about whether the question should "go live"
> Participants can "raise their hand" by indicating they have a question - The
delivery tool indicates a potential question by changing the participant's
"seat" color on the online screen visible to the moderator and instructor
All questions come in privately and are captured so they may be sent to participants as a
training supplement. A moderator is available during every course session to determine
which questions should be addressed during the session. Before any session begins,
participants are e-mailed rules about live Web course etiquette. Participants are reminded
of the etiquette during the course and phone lines can be muted by the moderator if one
or more participants fails to use the proper etiquette.
Self-paced. Self-paced courses account for about 8% of SAS courses and are delivered
primarily over the Internet. Customers may also request CD-Rom versions that can be
distributed to participants as a CD or may be loaded onto a customer's Intranet site.
HayGroup Page 125
-------
EPA: Results of Benchmarking Study
Interactions in self-paced courses are focused on participant interactions with the
technology, not with other participants or the instructor. Examples include:
> Short quizzes that participants must get right before continuing the training
> Instructions to go practice using the software before returning to training
> Questions that direct participants down different paths depending on their
responses
Course Evaluations
Evaluations occur immediately following a course and consist primarily of "smile
sheets". Courses that do not get an average rating of at least 3.5 out of 4 are further
examined to determine why the ratings are low. SAS will directly contact customers to
inquire further as to why courses were reviewed less favorably than expected. SAS also
monitors the re-buy rate of trainings to help determine whether customers are willing to
continue to participate in training. It is believed that customers will not continue to buy
training if they feel it is not worthwhile. Therefore, SAS feels that a re-buy rate of 60%
over a period of 18 months indicates that customers have evaluated the course positively
enough to continue purchasing it.
SAS also uses short quizzes during self-paced courses primarily as an interaction method.
Quizzes occur every few slides to ensure that participants are interacting with the media;
they are rarely used as a measure of whether participants are learning. However, courses
can be designed so that participants are blocked from continuing with the training until
they provide correct responses to the quizzes.
Lesson Learned/Future Directions
> There is a definite need for training that is not classroom-based
o Willingness to travel reduced dramatically after September 11, 2001
o Travel more than 100 miles dropped by 50%
> It took 12-18 months for constituents to truly buy-into web training
> It takes time to develop courses
o Classroom: 12 hours to develop 1 hour of delivery content
o Live Web: 20 hours to develop 1 hour of delivery content
o Self-paced: 60 hours to develop 1 hour of delivery content
> There will always be a need for classroom-based training
> Make it east for participants
o SAS will send CDs with downloading materials if participants cannot
download from the web (Flash, Webex, etc.)
> HTML has low standards
o It takes bandwidth to add video, audio, etc.
o Can store hundreds of hours on CD
o Streaming video is tough to get on CD - about lOmegs/minute
> Transition from classroom-based to Web-based courses takes time
o Most of the time is taken up redesigning interactivity
o Add 30-45 days if development team has never migrated before
HayGroup Page 126
-------
EPA: Results of Benchmarking Study
o For each project include a project manager, instructional designed, subject
matter expert, technical review team, editor, and producer
> SAS's average class size is about 12 people
o They try to limit most classes to 20 people; highly technical class to 10
people; and very general classes to 40 people
HayGroup Page 127
-------
EPA: Results of Benchmarking Study
Appendix C
Summary of Quick Fix Recommendations for
Improving theAPTI Training Program
HayGroup Page 128
-------
EPA: Results of Benchmarking Study
Below, we present an overview of all the "quick fix" suggestions that are presented
throughout this report. Please refer to the main body of the report for a more detailed
description of each "quick fix".
Quick Fix Summary
Needs Assessment Process
Course Design and
Updating Process
> Perform a high-level review of current and planned courses/offerings to
make "quick" improvements to courses that meet APTFs core mission
and fulfill the immediate needs of air professionals and other key
customers. For example, APTI does not want to waste valuable resources
updating a course if it is not viewed as currently filling an important need.
This high-level review may be accomplished through meetings with APTI
staff, informal discussions with the JTC and representatives from state,
local, and tribal agencies, and reviews of prior needs assessments and
training evaluations.
This review will provide input into decisions about those
courses/offerings that should be:
• Updated first
• Enhanced
• Transferred to a new medium or technology
• Shelved
• Retooled
> Take inventory of the roles and skill sets of the current training team to
determine where gaps exist. For example, if the team is missing a critical
role (e.g., instructional designer), APTI can begin to search for an
individual (within or outside EPA) to fill this role.
• Consider sending team members to training or hiring contractors
to fill skill gaps
• Consider hiring or contracting with a script writer to work with
subject matter experts to develop live broadcasts
> Provide critical information to course designers early in the process,
including objectives of the course/offering, intended delivery methods,
audience characteristics, technological constraints, and timelines and
resources. This will ensure that courses/offerings that are developed meet
stated objectives and maximize learning for air professionals and other
customers.
> Define the frequency and types of interactions that should be incorporated
into all courses/offerings. For example, for live Web courses, the
standards may be to ensure that there is an opportunity for interaction at
least every 10-15 minutes. For self-paced Web courses, the standard may
be to provide an opportunity for interaction (e.g., through a short quiz or
simulation) every 3 to 5 screens.
HayGroup
Page 129
-------
EPA: Results of Benchmarking Study
Course Design and
Updating Process
(continued)
> Require rehearsals for all live broadcasts, including full script and use of
teleprompters and other technology. This will ensure that the instructor
presents the material in an engaging way to the audience, and final
glitches and inaccuracies are caught before the broadcast is delivered live.
> Conduct a quality review of all current course materials before they are
distributed to trainees to check for:
• Typos
• Incorrect information
• Missing, upside down, or out-of-order pages
• Whether the right materials are sent on time, and with the
right courses.
Training Delivery Methods
> Examine current opportunities for interaction used during satellite
broadcasts. Consider the effectiveness of each type of interaction
opportunity, and why some methods are not as effective as others, using
the following criteria:
• Is there an opportunity for interaction at least every 10-15
minutes?
• Do methods for interaction interrupt the flow of the session?
• Do trainees take advantage of opportunities for interaction?
• Do trainees receive answers to all questions that are not addressed
during the broadcast?
• Can trainees hear and respond to others' questions?
Ensure that these opportunities for interaction are provided consistently in
all distance learning courses.
> Enhance methods of interaction used in distance learning courses.
• Publicize opportunities to interact - publicize contact information,
instructions, and benefits of interacting
• Establish proper etiquette and communicate prior to each session
• Use screeners and technology monitors to review incoming
questions - provide at least one individual other than the instructor
to accept, review, and facilitate questions
• Develop discussion questions prior to the broadcast - develop a set
of 5-10 questions to use when participants are hesitant to interact
• Follow-up on unanswered questions - provide trainees with
answers to all questions asked regardless if they were answered
during the broadcast
> Select instructors with subject matter expertise, applied experience in the
field, and experience in the medium of course delivery.
Consider using professional speakers to deliver training in conjunction
with subject matter experts who can answer specific questions raised by
the air professionals.
HayGroup
Page 130
-------
EPA: Results of Benchmarking Study
Training Delivery Methods
(continued)
> Determine new ways to provide opportunities for interaction during
broadcasts.
• Use chat rooms or Web boards to facilitate interactions. These
should be used to post questions, send course updates, and foster
e-mail discussions. Provide each trainee with the location of and
instructions for using the chat room or Web board. Instructors
should also review the chat room or Web board and be encouraged
to use these as a medium to interact with trainees.
• Consider facilitating the assembly of cohorts of 5 to 7 trainees to
complete the training program as a group, rather than individually.
Participants may complete self-paced modules at their own pace
(within a specified time frame), and then access chat rooms or
participate in conference calls to discuss the course and ask
questions.
> Provide instructors with standard guidelines for training in a particular
medium, including:
• Where to stand and how much to move during training
• When and how to look into the camera
• How to use basic technologies
• How to encourage interactions
• How to deal with difficult students
• Tips for holding the attention of students
Training Evaluation
Process
> Improve level 1 evaluation instruments to enhance an understanding of
why the training course/offering was effective, whether the delivery
method is the best for delivering training, and how to improve training
courses/offerings. At a minimum, ask some open-ended questions such
as:
• What about the course/offering did you like best?
• What about the course/offering would you change?
• How would you improve the current training course/offering?
• Were there enough opportunities for interaction? What are your
recommendations for enhancing interactions?
> Evaluate all courses/offerings, regardless of the delivery method. Give
participants the opportunity to complete a course evaluation on-line after
they finish a self-paced Web course. Focus the evaluation on the content;
how it was presented; the effectiveness of interactions; the ease of
accessing the course; and so forth. The evaluation should also include
overall questions about the effectiveness of the Web course, whether it
engages air professionals, and whether the Web is the best method for
teaching this course.
> Follow-up on less favorable evaluations. For example, if participants rate
a course below the minimum rating, APTI may contact participants to
obtain insight into why the course was rated less favorably.
HayGroup
Page 131
------- |