United States
Environmental Protection
Agency
Office of Environmental
Information
Washington, DC 20460
EPA/240/B-00/004
December 2000
<&EPA Guidance for Developing a
Training Program for Quality
Systems
EPA QA/G-10
-------
-------
FOREWORD
This guidance presents approaches to assessing quality assurance training needs and
developing a training program for quality systems to support implementation and management of
the Agency-wide quality system. This document is one of the U.S. Environmental Protection
Agency Quality System Series documents. These documents describe the EPA policies and
procedures for planning, implementing, and assessing the effectiveness of a quality system.
Questions regarding this document or other Quality System Series documents should be directed
to the Quality Staff at:
U.S. EPA
Quality Staff (2811R)
1200 Pennsylvania Avenue, NW
Washington, DC 20460
Phone: (202)564-6830
Fax: (202)565-2441
e-mail: quality@epa.gov
Copies of the EPA Quality System Series documents may be obtained from the Quality Staffer by
downloading them from the Quality Staff Home Page:
http://www.epa.gov/quality
EPAQA/G-10 l December 2000
-------
(This page has been left blank intentionally.)
EPAQA/G-10 11 December 2000
-------
TABLE OF CONTENTS
Page
CHAPTER 1. INTRODUCTION 1
1.1 BACKGROUND 1
1.2 BENEFITS OF QUALITY SYSTEMS TRAINING 1
1.3 TRAINING AND THE EPA QUALITY SYSTEM 2
1.4 POLICY AND RESPONSIBILITIES 2
1.5 SUPERSESSION 4
1.6 PERIOD OF APPLICABILITY 4
1.7 ADDITIONAL RESOURCES 4
CHAPTER 2. A DETAILED APPROACH TO BUILDING A
QUALITY SYSTEMS TRAINING PROGRAM 5
2.1 OVERVIEW 5
2.2 PHASE 1: UP-FRONT REVIEW 6
2.3 PHASE 2: NEEDS ASSESSMENT 8
2.4 PHASE 3: ANALYZING THE NEEDS ASSESSMENT DATA 12
2.5 PHASE 4: DEVELOPING A QUALITY SYSTEMS
TRAINING PROGRAM 14
2.6 PHASES: CONSTRUCTING THE QUALITY SYSTEMS
TRAINING PROGRAM 18
2.7 PHASE 6: EVALUATING THE QUALITY SYSTEMS
TRAINING PROGRAM 21
CHAPTER 3. A SIMPLIFIED APPROACH TO BUILDING A
QUALITY SYSTEMS TRAINING PROGRAM 23
3.1 OVERVIEW 23
3.2 ALTERNATE PHASE 1: GET INFORMED 24
3.3 ALTERNATE PHASE 2: ORGANIZE THE WORKLOAD 25
3.4 ALTERNATE PHASE 3: PERFORM GAP ANALYSIS 27
3.5 ALTERNATE PHASE 4: IDENTIFY METHODS TO ADDRESS GOALS . . 28
3.6 ALTERNATE PHASE 5: FORMULATE THE TRAINING PROGRAM 29
3.7 EVALUATING THE APPROACH TO BUILDING
QUALITY SYSTEMS TRAINING PROGRAMS 30
CHAPTER 4. QUALITY SYSTEMS TRAINING RESOURCES 35
4.1 AGENCY-WIDE QUALITY SYSTEMS TRAINING RESOURCES 35
4.2 IN-HOUSE COURSE DEVELOPMENT 35
4.3 USE OF CONTRACTORS/CONSULTANTS 39
BIBLIOGRAPHY 43
APPENDIX: FORMS FOR DEVELOPING A TRAINING PROGRAM 45
EPAQA/G-10 iii December 2000
-------
LIST OF FIGURES
Page
Figure 1. EPA Quality System Components 3
Figure 2. Needs Assessment Model 8
Figure 3. Example Unit Assessment Form 9
Figure 4. Example of Flierarchical Job Assessment Form 11
Figure 5. Gap Analysis Model 13
LIST OF TABLES
Page
Table 1. Example of a Gap Analysis Matrix 13
Table 2. Example of a Ranking Matrix 16
Table 3. Example of a Course Evaluation Matrix 17
Table 4. Training Media 20
Table 5. Example of Resources to Meet Training Goals 20
Table 6. Rationale of the Approach to Building Quality Systems Training Programs 31
Table 7. Example of Quality Systems Training 36
Table 8. Course Outline Template 38
LIST OF BOXES
Page
Box 1. Quality System Series Documents 4
Box 2. Flierarchical Job Assessment 10
Box 3. Objectives Form 37
EPAQA/G-10 IV December 2000
-------
CHAPTER 1
INTRODUCTION
1.1 BACKGROUND
This document describes a process for developing a training program that will assist users
in meeting the requirements of EPA Order 5360.1 A2. The process outlined in this document
represents a method for building a quality systems training program that uses a focused needs
assessment. It is not intended for use outside of this purpose. There are two approaches
described in this document: the detailed approach in Chapter 2 and the simplified approach in
Chapter 3. Users should read Chapter 2 to become familiar with the overall process, even if they
follow the approach described in Chapter 3.
The steps described in this document are designed for a general audience with limited
experience in training program development but can be used by anyone involved in quality
assurance- and quality control-related training issues. Each organization should evaluate the steps
described in this document and determine how they relate to the organization's needs. Each step
should be reviewed for applicability and adhered to, as appropriate, to ensure thorough training
program development takes place.
The intended audience for this document includes managers, quality assurance (QA)
personnel, and other potential training providers that participate in environmental programs
governed by the EPA Quality System. Users of this document should have primary or
contributory roles and responsibilities for assessing the need for quality-related training and for
developing a quality systems training program for a specific Region, National Center, Laboratory,
or Program Office. Although this document focuses on EPA organizations, it can be tailored to
the needs of a variety of other Federal agencies, State, local or Tribal agencies, and non-
governmental organizations, as well as contractors and financial assistance recipients.
1.2 BENEFITS OF QUALITY SYSTEMS TRAINING
An effective quality systems training program is necessary to ensure that personnel who
plan, implement, and assess environmental programs have the skills needed to perform their
responsibilities. Quality systems training can:
• help institutionalize quality system requirements through learning,
• provide guidance on how to comply with quality system policies and requirements,
• enhance individual performance by developing proficiencies in the use of QA and
quality control (QC) tools and related technical skills, and
• standardize quality systems policy throughout an organization.
The depth of each employee's knowledge and skills contributes to an agency's ability to
achieve its mission and goals. Thus, all EPA personnel should be trained to a level that advances
EPAQA/G-10 1 December 2000
-------
the Agency's commitment to protect human health and safeguard the natural environment by
providing environmental data that are credible, scientifically sound, and of sufficient and adequate
quality. This document provides a systematic method for defining quality system training needs
and developing a training program to help organizations incorporate the necessary QA and QC
skills into the full training curriculum.
1.3 TRAINING AND THE EPA QUALITY SYSTEM
Since environmental data operations can be complex, a systematic structure for quality
must be established to provide confidence in the quality of information used to support decisions.
Such a structure is provided by a quality system and is defined by the organization conducting
environmental data operations. The EPA Quality System (Figure 1) consists of three structural
levels: Policy, Organization, and Project. Effective implementation of the EPA Quality System
requires that training occur at each level of the organization that performs activities affecting
environmental data quality. Training needs at the various levels of the organization are task
specific. Senior and line managers can benefit from training to help understand the structure,
concepts, and operating principles of the quality system. Technical personnel can benefit from
training to help understand their organization's quality system and the QA and QC tools and
techniques necessary to fulfill the requirements of the system.
The mandatory EPA Quality System is based on the national consensus standard,
Specifications and Guidelines for Quality Systems for Environmental Data Collection and
Environmental Technology Programs, ANSI/ASQC E4-1994. This standard provides the
management and technical elements necessary for developing and implementing a quality system.
This standard also contains specifications for identifying training needs to provide the necessary
level of skills in QA and QC practices for effective implementation of environmental programs.
Quality systems training ensures that all personnel involved in the planning, management,
and implementation of data collection activities have the knowledge and skills to complete their
tasks according to the policies and procedures in their organization's quality system. An
organization typically specifies the process for determining training requirements in its Quality
Management Plan (EPA, 2000a).
1.4 POLICY AND RESPONSIBILITIES
EPA Order 5360.1 A2, Policy and Program Requirements for the Mandatory Agency-
Wide Quality System (2000), specifies that the collection of environmental data by, funded by, or
provided to the U.S. Environmental Protection Agency (EPA) be supported by an effective
quality system. This Order also requires all EPA organizations supporting environmental
programs to identify program-specific QA and QC training needs for all levels of management and
staff, and to provide resources for this training. The goal of this requirement is to assure that QA
and QC responsibilities are understood at every stage of project implementation.
EPAQA/G-10 2 December 2000
-------
O
LU
O
o:
o_
OL
O
D.
§
o
a:
o
o
_i
o
D.
Consensus Standards
ANSI/ASQC E4
ISO 9000 Series
Internal EPA Policies
EPA Order 5360.1
EPA Manual 5360
External Policies
Contracts - 48 CFR 46
Assistance Agreements -
CFR 30, 31, and 35
Quality System
Documentation
Supporting System Elements
Training/Communication
Annual Review
and Planning
Systems
Assessments
Conduct Study/
v Experiment j
1
i
Standard
Operating
Procedures
i
i
Technical
Assessments
PLANNING
A.
IMPLEMENTATION
ASSESSMENT
Defensible Products and Decisions
Figure 1. EPA Quality System Components
EPA QA/G-10
December 2000
-------
These requirements are externalized through 48 CFR 46 for contractors; 40 CFR 30, 31,
and 35 for assistance agreement recipients; and through other mechanisms, such as consent
agreements in enforcement actions.
EPA Order 5360.1 A2 states that the ultimate responsibility for ensuring that training
needs are identified and appropriately addressed rests with the management of the organization.
EPA's Senior Management Official for Quality is responsible for developing generic training
programs for EPA management and staff. The national quality systems training program was
developed in response to this requirement (see Chapter 4). The task of developing and
implementing the national training program has been delegated to the Quality Staff.
1.5 SUPERSESSION
This document represents a component of the EPA Quality System for which guidance has
not been previously developed. Therefore, it does not supersede any other documents.
1.6 PERIOD OF APPLICABILITY
Per EPA Order 5360 Al, this document will be valid for five years from the official date
of publication. After five years this document will either be reissued without change, revised, or
withdrawn from the EPA Quality System.
1.7 ADDITIONAL RESOURCES
Several other documents have been developed to provide suggestions on developing
suitable and effective quality systems for environmental programs. These documents establish
criteria and mandatory specifications for QA and QC activities and provide recommendations for
using the various components of the EPA Quality System. These documents are listed in Box 1.
Since these documents contain guidance on activities critical to successful environmental data
operations, they should be considered as resources when developing a quality systems training
program. In addition, program-specific documents may be used as a resource.
Box 1. Quality System Series Documents
EPA Requirements for Quality Management Plans (QA/R-2) (EPA 1999b)
EPA Requirements for Quality Assurance Project Plans for Environmental Data
Operations (QA/R-5) (EPA 1999a)
Guidance for the Data Quality Objectives Process (QA/G-4) (EPA 2000b)
EPA Guidance for Quality Assurance Project Plans (QA/G-5) (EPA 1998)
Guidance for the Preparation of Standard Operating Procedures for Quality-Related
Operations (QA/G-6) (EPA 1995)
Guidance for Data Quality Assessment: Practical Methods for Data Analysis
(QA/G-9) (EPA 2000a)
EPAQA/G-10 4 December 2000
-------
CHAPTER 2
A DETAILED APPROACH TO BUILDING A
QUALITY SYSTEMS TRAINING PROGRAM
2.1 OVERVIEW
C UP-f RONT REVIEW"^ A quality systems training program should facilitate the achievement
^ r-i ^ of an organization's mission, goals, and objectives. Once the details of the
quality systems training program are established, they are documented in a
(~x (^uaiiuy a^autina naming jjivjgiain ait tauauiiantu, unt^ ait uvjtuintnutu 11
ASSESSMENT! j training plan. To be properly aligned with the organization, the training
_
-------
Proi ect Level
defining project scope and requirements;
• managing project time, cost, and performance;
ensuring project quality management; and
• project procurement management.
The process outlined in the following pages is iterative in nature. It should involve
participation of stakeholders with the organization that is developing the training program. The
complexity of each phase will vary with the type and size of the organization or unit being
addressed. The end result should be a quality systems training program that yields enhanced work
performance by knowledgeable and skilled staff and a strengthened quality system.
2.2 PHASE 1: UP-FRONT REVIEW
^ UP-FRONT REVIEW^ The purpose of the up-front review is to:
* Describe reasons for builiding
program
* Determine the best solution
* Create a team and assign roles
1. describe reasons for building a quality systems training
program,
2. determine the best solution, and
C ASSESSMENT ) 3. create a team and assign roles.
ANALYSIS
Reasons for Building a Quality Systems Training Program
DEVELOPMENT j Training programs are developed for a variety of reasons. Such
programs may be used to meet directives from senior management or to
--
CONSTRUCTION j address a performance issue. Understanding the need for the training
program provides insight into the expectations the program must meet. If a
_ _ .
f EVALUATING THE ) directive from senior management provides the impetus for creating the
V PROGRAM J & f ... ... ,, f ^ A- f T7
program, one should become familiar with the reason tor the directive. For
example, is the organization ensuring that mission critical skills are available or does a
performance issue exist?
A new job description, the introduction of new technology, or a change in responsibilities
may result in a lack of knowledge, skills, or abilities needed to successfully perform a job.
Training is often the most appropriate method for addressing these types of problems.
Performance issues, including communication, management, and attitude, can sometimes benefit
from training, but training alone rarely solves these types of performance problems.
Determine the Best Solution
In order to determine if training or some other method is the most appropriate method for
addressing a job performance problem, the issue needs to be researched. Start with the initial
source of the information. Try to find out:
EPAQA/G-10 6 December 2000
-------
• how the job performance issue was identified (i.e., internal or external audits, work
product review, etc.),
• the criteria the proposed solution is expected to meet (cost versus benefit, time
lines, and management's expectations),
• the scope of the job performance issue (individual, team, or organization), and
from whose perspective the job performance issue exists (individual or
management).
When at least part of the problem is that the worker does not know how to meet
expectations for job performance, training is the best solution. When a lack of knowledge, skill,
or ability is not the cause of the problem, then a non-training solution should be developed.
Occasionally, the cause of a problem is determined to be a lack of training when in reality the
correct solution is to make changes within a unit or organization. If a non-training solution is
required, the organization's Human Resources Department may be able to provide assistance.
Create a Team and Assign Roles
The work involved in building a quality systems training program, including the up-front
review, can be labor intensive. It should not be assumed that this process is the primary role of
the QA Manager or the QA support staff. However, once the responsibility for developing the
quality systems training program has been delegated, the QA Manager or designee should provide
their opinion and validate results throughout the process.
The development and implementation of a training program can be simplified by pulling
together a team of key individuals in the organization. This critical step helps to gain support for
the project throughout the organization. Early on, those working on the process should be
identified and their roles and responsibilities clearly communicated. Those roles and
responsibilities should be reviewed and revised as necessary throughout the development process.
During the initial stages of this process, it is important to keep stakeholders and decision
makers within the organization informed about the progress of the assessment and involve them
where appropriate. It is also important to explain to stakeholders how the needs assessment
process assists the organization in complying with the order. These results should be documented
in a report and approved according to the appropriate chain of command.
Phase 1 Tasks
Complete these tasks to perform an up-front review.
Identify the performance issue.
Form a project team, as appropriate.
Assign roles and responsibilities.
Inform stakeholders and decision makers.
EPA QA/G-10
December 2000
-------
2.3 PHASE 2: NEEDS ASSESSMENT
C
UP-FRONT REVIEW
ASSESSMENT
* Decide which jobs to assess
» Identify QA-related KSAs
» Determine existing level of
knowledge and expertise
o
C
ANALYSIS
o
C
DEVELOPMENT
•u-
C
CONSTRUCTION
4J.
C
EVALUATING THE
PROGRAM
A Needs Assessment is the process used to identify and understand
the particular quality-related training goals of a specific job, team,
organization, or agency (Figure 2). Identifying the training goals helps to
ensure that the quality systems training program makes the best use of
available resources. The needs assessment process involves identifying the
gaps in KSAs that an employee needs to perform his or her job.
Existing knowledge,
skills, and abilities (KSAs)
Necessary knowledge,
skills, and abilities (KSAs)
Figure 2. Needs Assessment Model
A training program outlines in detail the training needs and how
they will be met. A quality systems training program outlines the particular
needs of a unit or organization for which quality assurance related tasks are
a primary focus.
There are three basic steps to performing a needs assessment:
1. decide which jobs to assess,
2. identify the quality-related KSAs required to perform the jobs, and
3. determine the existing level of knowledge and experience.
During the Needs Assessment process, it may also be helpful to review existing documentation.
Additional information to support the process may be found in reports from management
assessments, internal assessment studies, or Inspector General reports.
Decide Which Jobs to Assess
Deciding which jobs to assess will define the scope of the needs assessment and ensure its
validity. The decision begins with the identification of an appropriate unit for assessment. The
results of the up-front review may logically suggest a certain unit of jobs within an organization.
A unit will contain related jobs that facilitate an organization's quality systems work. It can be a
cross section of the organization or a sub-section within. It should cover all positions involved
with environmental data operations, including data users, data generators, and decision makers.
Because the quality system operates at the project, organizational, and policy levels, it is critical
that the training program address the quality systems training needs of each level.
Once the unit of jobs has been defined, identify and document its associated
responsibilities and activities. Most likely, this information already exists and can be found
through the Human Resources Department, work plans, and staffing plans. The responsibilities
and activities of the job(s) should be documented in the Unit Assessment Form (see Appendix A).
EPA QA/G-10
December 2000
-------
The focus of this phase is on the actual job roles or functions in the unit. The information
will be used later to identify which KSAs personnel need to perform their jobs effectively. The
Unit Assessment Form should result in a list of responsibilities and activities that can be examined
in terms of what is needed to perform them. See Figure 3 for an example of a completed Unit
Assessment Form.
Date: January 24
Manager's Name: Jane Doe
Unit Description: Program Oversight Section
Job List (include all functions in the unit listed above)
Work Assignment Manager (WAM)
Hydrologist
Database Managers
Technical Liaison
Job Title
Environmental Scientist
Responsibilities
WAM
Technical Liaison
Staff Size: 8-10 full-time
employees
Primary Tasks
Assure effective
performance
Evaluate performance
Manage communications
networks for local
governments
Figure 3. Example Unit Assessment Form
Identify the Quality-Related KSAs Required to Perform the Jobs
The list generated by the Unit Assessment Form provides the framework for performing
the job assessment. Job assessment is the process of breaking a job down into specific tasks and
evaluating the knowledge and skills needed to perform each task.
There are several methods for performing a job assessment, but the simplest method
creates a hierarchy of tasks and sub-tasks necessary to perform a job. The Hierarchical Job Form
can be used as a guide throughout the process. The completed form results in a list of quality-
related KSAs required to perform each job within a unit. A model for identifying quality-related
KSAs is contained in Box 2.
A job assessment requires several levels of detail. It is appropriate to break down the
skills needed to perform a job task until entry-level knowledge is determined. Figure 4 contains
an example Hierarchical Job Assessment Form. It provides an example of a job assessment
conducted on a portion of a Work Assignment Manager's responsibilities. Appendix A contains a
copy of this form.
EPA QA/G-10
December 2000
-------
Box 2. Hierarchical Job Assessment
c
1. Identify the primary tasks of a job.
2. Identify the general KSAs.
3. Identify the Quality-related KSAs.
Step 1
Step 1
Step 1
General
KSAs
General
KSAs
General
KSAs
C
(auality-Relate
KSAs
0
(auality-Relate
KSAs
0
(auality-Relate
KSAs
The information from the job assessment should be validated with staff members who
perform the tasks and their feedback should be implemented where appropriate. The product of
this form should be a list of quality-related KSAs required to perform a specific job.
Determine the Existing Level of Knowledge and Experience
The final step in the needs assessment is to identify the existing KSAs of employees who
are currently performing or will potentially perform quality-related functions. This step should
validate the information collected so far and identify a baseline for the training goals. There are
three steps involved in determining existing KSAs:
1. conduct research,
2. confirm findings, and
3. document findings.
To start this process, decide whether the unit will be assessed in its entirety, or whether it
will be broken into smaller groups (for example, by job category or organizational level). It is
inadvisable to break the unit down by individual employee since it is likely the quality systems
training program will apply beyond a given employee's tenure in that particular position.
Examination of employee performance reviews and training records as well as interviews
conducted with members of the unit are useful in identifying existing knowledge and experience
within a job category or organizational level. Conduct interviews with the entire unit or a
representative sample, depending on its size and the availability of resources needed to conduct
the interviews.
Sample interview questions may include the following:
Does the employee perform the task?
EPA QA/G-10
10
December 2000
-------
A.
B.
Job Title Work Assignment Manager
Primary Job Tasks
1 . Assure effective performance
2. Evaluate performance
C.
List the Sub-tasks related to each task
Task 1 Assure effective performance
Sub-task 1 Inspect work
Sub-task 2 Review progress reports
Sub-task 3 Review outputs
D.
Task 2 Evaluate performance
Sub-task 1 Conduct on-site inspection
Sub-task 2 Evaluate outputs/deliverables
List KSAs needed to perform each sub-task
Task 1 Assure effective performance
Sub-task 1 Inspect work
KSA: Ability to evaluate and analyze
technical and/or administrative issues
involved in implementation of
environmental programs
Sub-task 2 Review program reports
KSA: Ability to communicate
environmental information orally and in
writing
Sub-task 3 Review outputs
KSA: Knowledge of work plan and QA
Project Plan development and review
processes
E.
Task 2 Evaluate performance
Sub-task 1 Conduct on-site inspection
KSA: Ability to evaluate environmental
monitoring aspects of the work
assignment
Sub-task 2 Evaluation of outputs
KSA: Ability to review, analyze, and
recommend modifications to outputs
including QA Project Plans or data
reports
List Quality-Related KSAs
Taskl
• Ability to apply QA and QC criteria to technical directives as appropriate
• Knowledge of basic quality-related concepts and terms
• Knowledge of QA Project Plan review skills criteria and procedures
Task 2
• Ability to perform technical systems audits
• Ability to perform environmental data evaluation
Figure 4. Example of Hierarchical Job Assessment Form
EPA QA/G-10
11
December 2000
-------
• How frequently does the employee perform the task?
What steps does the employee use to perform the task?
• What can the employee suggest for improvement?
• How would the employee rate his or her ability to perform the task?
• What are the employee's reasons for his or her rating?
Where practical, observe employees on the job, talk with those who are outside the job's scope
but work with those who are within it, and review customer feedback records.
After current skills data has been collected, have members of the unit and/or stakeholders
review the results and provide feedback to verify that all existing KSAs are noted and that none
are there that do not belong. Ensure that information collected is that which was intended.
Developing training goals based on inaccurate or non-representative information can be
counterproductive to the organization.
Phase 2 Tasks
Complete these tasks to perform the needs assessment.
Identify the jobs to be assessed.
Document the unit's roles and
responsibilities in a Unit Assessment Form.
Complete a Hierarchical Job Assessment for
each job to be assessed.
Identify the KSAs for identified jobs.
2.4 PHASES: ANALYZING THE NEEDS ASSESSMENT DATA
Needs assessment information analysis is the process of identifying
the differences or gaps between what is needed to perform a job and what
performance criteria are currently being met. This process includes the
following steps:
1. compare existing skills and required skills and
2. verify the results.
Compare Existing and Required Skills
The Gap Analysis Matrix (Table 1) was designed to assist in making
the comparison between existing and required skills. To develop this
matrix, enter a job category or unit description on the top row as indicated
below. Enter the KSAs required to perform each job in the first column.
C
UP-FRONT REVIEW
4J.
C
V
ASSESSMENT
u
ANALYSIS
* Compare existing and
required skills
* Verify the results
4.;
C
DEVELOPMENT
o
C
(
CONSTRUCTION
U
EVALUATING THE
PROGRAM
EPA QA/G-10
12
December 2000
-------
Table 1. Example of a Gap Analysis Matrix
Required KSAs
Ability to apply quality-related concepts to environmental study plans and
reports
Ability to apply QA and QC criteria to technical directives as appropriate
Knowledge of basic quality -related concepts and terms
Knowledge of QA Project Plan review skills
Ability to perform technical systems audits
Ability to perform environmental data evaluation
Requirement
Met?
•
•
For each job category, indicate where the existing level of KSAs are adequate to achieve
satisfactory job performance by placing a mark in the corresponding box.
Any KSAs that are not currently met, either by job category or unit, should be
documented. The resulting list of KSAs is the gap that needs to be bridged in the development of
the quality systems training program and should be translated into training goals (see Figure 5). A
training goal should be stated in as much detail as the KSA gap indicates, along with any other
criteria the training program should meet. The training goals should describe what would change,
how work might flow, how quickly it might be completed and to what result.
Training Goals
Existing knowledge,
skills, and abilities (KSAs)
Necessary knowledge,
skills, and abilities (KSAs)
Figure 5. Gap Analysis Model
Depending on the size of the unit, the quality-related KSAs may differ widely among job
categories. If this is the case, a separate analysis should be made of each. If no job category
breakdown is necessary, the entire unit should be analyzed together.
If accurate and representative information are collected, the results will accurately reflect
the quality-related training needs. This information should be described in a detailed report that
explains how the process was performed, describes the results, and provides an interpretation of
the results.
EPA QA/G-10
13
December 2000
-------
If the results indicate that there are no quality-related training needs, then the process ends
with the analysis. If the results show a need for quality-related training needs, the information will
be used during subsequent phases of this process.
Verify the Results
Obtain feedback on the results of the needs assessment from the stakeholders, regardless
of what the results indicate. If the stakeholders cannot verify the results, their concerns should be
taken seriously and discrepancies should be resolved prior to continuing the process. The person
who originally requested that this project be completed may be a source for resolving any
discrepancies.
Phase 3 Tasks
Complete these tasks to perform a needs assessment
information analysis.
Complete a Gap Analysis Matrix for each job
category or group of jobs.
Develop training goals based on the gap
analysis.
Verify the findings from the gap analysis.
(^UP-FRONT REVIEW
•U-
C
ASSESSMENT
4J,
c
ANALYSIS
4J-
2.5 PHASE 4: DEVELOPING A QUALITY SYSTEMS TRAINING PROGRAM
The purpose of the development stage is to create an approach for
meeting the quality systems training goals. There are four steps:
1. rank the training goals,
2. define success,
3. compare the current quality-related training resources to
training goals, and
4. verify and document the results.
Rank the Training Goals
There are many ways to rank training goals to establish priorities.
Any selected method should involve an understanding of the stakeholder
needs and the use of consistent criteria. An explanation of one process to
rank training goals using weighted criteria follows (see Table 2).
DEVELOPMENT
* Rank the training goals
* Define success
» Compare resources to goals
* Verify and document results
c
C
CONSTRUCTION
EVALUATING THE
PROGRAM
EPA QA/G-10
14
December 2000
-------
First, list the training goals identified by the gap analysis. Link each training goal to the
level of the organization where the training need exists (either policy, organization, or project).
Discuss the training goals and their priority in the organization with the stakeholders until
consensus is reached on the list to be ranked. This discussion should not revisit whether the goal
is valid, but should determine how important it is to the organization. The example of a Ranking
Matrix (see Table 2) lists the training goals on the left side.
Second, identify the criteria you will use to measure the relative importance of the training
goals. The criteria should be selected by consensus with the stakeholder organization. Only the
stakeholders can really determine the priority. The example of a Ranking Matrix (see Table 2)
identifies the criteria for determining the priority of the training goals across the top.
Third, give each criterion a weight depending on how important it is to the organization.
Criteria of lower relative importance should receive a lesser weight and criteria of higher
importance should receive a greater weight. Assign the weight as a percentage of 100. (The total
weight of all the criteria should equal 100.) Table 2 shows four criteria of varying weights.
Fourth, rate each training goal against each criterion, using a scale selected by the group.
In Table 2 the scale is 1 to 5 with 5 being the highest (or most important). This rating can be
done individually or by the group. If done individually, a composite rating must be calculated
before moving on to Step 5. To calculate a composite rating, average the scores of all the group
members by totaling all the ratings for a particular criterion and dividing the total by the number
of people in the group.
Fifth, multiply the rating by the weight given for the criterion. In Table 2, the third
training goal (ability to perform technical systems audits) rating is 3 when evaluated against the
Agency Requirement criterion. When this rating (3) is multiplied by the weight for the Agency
Requirement criterion (30%), the score is .90. The scores are shown in parentheses in each cell.
Finally, determine the total score for each training goal by adding together the scores for
the training goal across all criteria. Enter that number in the last column. The totals for each row
represent the relative priority the group gives to each training goal. The Ranking Matrix yields a
list of prioritized training goals. In Table 2, the fourth training goal (Ability to perform
environmental data evaluation) has the lowest priority and the third training goal (Ability to
Perform Technical Systems Audits) has the highest priority.
Define Success
Defining success will:
• provide a unified understanding of each training goal,
help to avoid confusion as the program is being developed, and
• describe the desired results so it is clear when success has been achieved.
EPAQA/G-10 15 December 2000
-------
Table 2. Example of a Ranking Matrix
TRAINING
GOALS
Ability to apply
quality-related
concepts to
environmental
study plans and
reports
Ability to apply QA
and QC criteria to
technical directives
as appropriate
Ability to perform
technical systems
audits
Ability to perform
environmental data
evaluation
CRITERIA
Job
Related
Weight
15%
1
(.15)
1
(.15)
5
(.75)
1
(.15)
Mission
Essential
Weight
30%
O
(.90)
3
(.90)
1
(.30)
1
(.30)
Agency
Requirement
Weight
30%
2
(.60)
2
(.60)
3
(.90)
1
(.30)
Career
Development
Weight
25%
1
(.25)
1
(.25)
3
(.75)
2
(.25)
Total
Weight
100%
1.9
1.9
2.7
1.0
In simplest terms, the quality systems training program will be a success if it results in a better
trained workforce with improved job performance. The first measure of success is whether or not
the training goals have been met. Other measures may include how well the training goals were
met and the degree of job performance improvement. Any additions to the description of success
should be made at this time and should be agreed upon by the stakeholders.
Compare the Current Quality Systems Training Resources to Training Goals
The next step is to research the quality-related training that is currently being offered and
the training goal this training meets. Develop a Training Resource Course Evaluation Matrix
(see example in Table 3) to document if an existing training resource meets each training goal.
The current quality-related training resources should be listed on the left side of the matrix.
Chapter 4 in this document discusses resources for generating such a list.
The training goals of highest priority should be listed across the top of the matrix. Each
course should be evaluated in terms of how well it meets each training goal. A checkmark should
be placed in the box where a course meets a training goal.
EPA QA/G-10
16
December 2000
-------
Table 3. Example of a Training Resource Evaluation Matrix
Resource
Introduction to EPA Quality
System Requirements Training
Course
Management Systems Review
Workshop
Introduction to DQOs Training
Course
Integrating QA into Project
Development (Advanced)
Workshop
Sampling Designs for QA
Project Plans (Advanced)
Workshop
Data Quality Assessment
Training Course
Apply quality-
related
concepts to
environmental
study plans and
reports
•
•
•
•
Met
Apply QA and
QC criteria to
technical
directives as
appropriate
•
•
Met
Perform
technical
systems audits
Unmet
•= Course meets training goal.
The Training Resource Evaluation Matrix will determine which training goals can be
adequately met by current training resources. The matrix will also distinguish these training goals
that cannot adequately be met by current courses. An approach for meeting the latter group of
training goals will need to be designed and information about this can be found in Chapter 4 of
this document.
Verify and Document the Results
All information should be verified by people familiar with the courses and the training
goals. The final results of this phase should be documented in a report and shared with the
appropriate staff.
EPA QA/G-10
17
December 2000
-------
The completion of the Training Resource Evaluation Matrix concludes the information
collection and evaluation portion of building the training program. The next portion focuses on
constructing and evaluating the quality systems training program prior to its implementation.
Phase 4 Tasks
Complete these tasks to develop the quality systems
training program.
Rank the training goals using a Ranking
Matrix.
Identify gaps in resources using a Training
Resource Matrix.
Verify the results.
Document the results.
2.6 PHASE 5: CONSTRUCTING THE QUALITY SYSTEMS TRAINING
PROGRAM
C
UP-FRONT REVIEW
c
c
ASSESSMENT
ANALYSIS
Phase 5 of building a quality systems training program is the last
portion of developing the program. The Construction phase puts together
the conclusions reached during the information collection and analysis
phases to attain the training goals. The process of constructing the training
program includes:
c
DEVELOPMENT
1 .
2.
3 .
4.
create a time line for achieving each training goal,
identify how each training goal will be met,
identify redundancy, and
identify roles and responsibilities.
CONSTRUCTION
* Create a time line
* Identify how each training goal
will be met
* Identify redundancy
^dentifyrdes and ^0^1^ Create a Time Line fof Achieving Each Training Goal
C
EVALUATING THE
PROGRAM
The purpose of a time line is to determine which goals will be met at
what point during the program's life cycle. This determination will be made
according to two pieces of information — the priority of the training goal
(Section 2.5) and the availability of resources needed to achieve the goal. This priority was
identified in Section 2.5. Availability of resources refers to both the amount of resources that
exist and the likelihood of obtaining the necessary resources. If resources are not available to
fulfill a high priority training goal, it might be necessary to either identify alternative resources for
meeting the training need or revisit its priority for the organization. Resolving any conflicts
between priority level and availability of resources requires flexibility and management support.
EPA QA/G-10
18
December 2000
-------
The basic information needed to make these decisions comes from the results of the
Training Resource Evaluation Matrix and the Ranking Matrix from Phase 3.
Identify How Each Training Goal Will Be Met
There are several considerations when deciding how to achieve an organization's training
goals:
1. financial resources,
2. available time,
3. available personnel and their skills,
4. existing programs that can be adapted for use,
5. costs of these resources, including administrative costs,
6. types of training that are needed, and
7. personality, attitude, learning capabilities, and size of the audience.
These items will help determine if the match between the training goal and the existing
courses is strong. These considerations will also begin the process of addressing the needs not
met by current offerings. Some familiarity with the six basic training methods will also be helpful.
Table 4 lists a variety of training platforms including formal training (classroom) and computer-
based, video-based, print-based instruction, and informal on-the-job training . These platforms
can be mixed and matched to suit a particular need.
Table 5 provides a template for documenting how the training goals will be met.
For each Training Goal, list its priority, when it should be met, the method or methods that should
be used, the resources needed, and the availability of resources.
Identify Redundancy
After reviewing the quality systems training program, there may be some organization-
wide training goals that, because of their topics or methods of achieving the goal, can be linked
together. For example, a Needs Assessment may establish that all Project Officers have quality-
related tasks. An organization may have a training need or requirement for all of its employees to
participate in Basic Project Officers Training. It may be possible that Basic Project Officers
Training can fulfill some or all of the quality-related training requirements. Regardless of the
extent to which the requirements can be met, making use of appropriate existing courses, rather
than creating new ones, creates a cost-effective solution.
Identify Roles and Responsibilities
After the quality systems training program is developed, delegating implementation and
management responsibilities to organizations, units, or individuals may be appropriate. There
should still be designated individual(s) who coordinates the implementation plan. That individual
may be the same person who coordinated the development of the training program, or it may be a
EPAQA/G-10 19 December 2000
-------
Table 4. Training Media
Medium
Advantages
Disadvantages
Classroom
Inexpensive to develop
Easy to produce/reproduce
Allows student participation
Travel costs
Limited availability of qualified
instructors
Computer-Based
Multimedia
Interactive
Individualized
Realistic simulation capabilities
Consistent instruction
High up-front development costs
Hardware access
Expensive to revise
Not flexible when operating
system changes
Distance Learning
Reaches geographically-
dispersed audience
High delivery costs
Video-Based
Excellent for presenting
overviews of programs
Easy to distribute
Attention getting
Not easy to revise
High up-front costs
Print-Based
Inexpensive to develop
Easy to produce/reproduce
Unlimited capacity for
information
Not as aesthetically pleasing as
multimedia
On-the-job
Low cost
Easy to implement
Flexible
Provides practicing opportunities
May slow down pace of work
and interfere with time limits for
job performance
Table 5. Example of Resources to Meet Training Goals
Training Goal
Apply quality-related
concepts to study
plans and reports
Apply QA and QC
criteria to technical
directives as
appropriate
Perform technical
systems audits
Priority
2
2
1
Due
Date
09/30
09/30
06/01
Training
Method(s)
Classroom
OJT
Classroom
OJT
CBT
Resources Needed
• Training dollars
• Personnel availability
• Training support
• Training dollars
• Personnel availability
• Training support
• Training dollars
• Personnel availability
• Training support
Resource Availability
• Personnel availability
• Training dollars
• Personnel availability
• Training support
• Training support
EPA QA/G-10
20
December 2000
-------
completely different individual. If someone not involved in the initial development of the training
program is assigned responsibility for coordination, the new coordinator should be familiar with
how the training program was developed, its objectives, and the desired outcomes of the process.
Maintaining open lines of communication with all stakeholders during the program's
development is still important in this phase. Stakeholders should review the progress of the
training program throughout the development process so that they remain involved and
supportive of the plan.
Phase 5 Tasks
Complete these tasks to construct the training program.
Create a time line.
Choose training methods for achieving the
training goals.
Discuss the program with stakeholders.
PHASE 6: EVALUATING THE QUALITY SYSTEMS TRAINING PROGRAM
The last phase of building a quality systems training program is
evaluating the program to make sure it is sufficient in scope and will meet
the needs of its users. Evaluate the program to determine if it addresses
training needs and priorities. The following three areas are important when
evaluating a training program:
1. review the original request,
2. review the training goals, and
3. get stakeholder feedback.
Review the Original Request
Before considering the quality systems training program to be
complete, the original request or directive that started this process should be
reviewed for consistency. If the program does not meet the original criteria,
there are two options. The first option is to alter the program to meet any
criteria that are not met. This option is viable if the missing criteria are fairly simple to include
and will not make major changes in the program's approach. If this is not viable, the second
option is to meet with the originator of the request to obtain their input on the program. With
some creative thinking, some pieces of the program may be salvaged or revised without having to
repeat the entire process.
2.
C
7 PHASE (
UP-FRONT REVIEW
o
C
ASSESSMENT
o
C
ANALYSIS
4.J.
C
DEVELOPMENT
O
C
X
V
CONSTRUCTION
.Q-
EVALUATINGTHE
PROGRAM
» Review the original request
» Review the training goals
* Get stakeholder feedback
EPA QA/G-10
21
December 2000
-------
Review the Training Goals
It should be determined if the program has the reasonable potential to meet the original
training goals and if any revisions might increase the likelihood of its success. To make this
determination, refer to the training goals identified in Phase 3. Review the training goals and the
plan to meet the goals, which are listed in Phase 5. If the suggestions in Phase 5 do not correlate
with the training goals, revisions to one or both are necessary. Most likely the revisions need to
be made in the approach rather than the goals, but both items should be considered. To make
these revisions, refer back to the process recommended in the pertinent phase.
Get Stakeholder Feedback
If the stakeholders of the training program have been kept up-to-date on the various
milestones of the project, this final review will not produce many surprises. To encourage
stakeholder review, package the program in a format that is easy to read, and provide the
stakeholders with guidance on the type of feedback being sought and a deadline for providing it.
Once their feedback has been received, implement the stakeholders' suggestions wherever
possible. Their feedback should represent the needs and concerns of the average user and will
increase the support for the program.
After the first review, stakeholders' feedback should be implemented and the program
should be sent out for subsequent reviews before it is finalized. While the same people can review
the plan more than once, new reviewers might pick up different details than those who have seen
it before. The number of reviews should be limited to only that which will enhance the quality of
the plan.
Additionally, it is important to track the quality systems training program's contribution to
the organization's mission and goals. This task involves getting feedback from the students,
managers, and customers, some of whom may be included in the stakeholder review (described
previously).
EPAQA/G-10 22 December 2000
-------
CHAPTER 3
A SIMPLIFIED APPROACH TO BUILDING A
QUALITY SYSTEMS TRAINING PROGRAM
3.1 OVERVIEW
A successful training program facilitates the achievement of an organization's mission,
goals, and objectives. Chapter 2 outlines one method of building a training program that works
well when resources are not a limiting factor. However, limited resources require a simplified
approach.
This chapter describes a simplified approach to developing a quality systems training
program that can be used when time, personnel, or other resources are extremely limited. This
approach offers flexibility, allowing the user to choose those recommendations that best meet the
current needs. However, it should be understood that the closer the original process (Chapter 2)
is followed, the more thorough the result will be. Steps in this approach include:
1. Get Informed (Section 3.2),
2. Organize the Workload (Section 3.3),
3. Perform Gap Analysis (Section 3.4),
4. Identify Methods to Address Goals (Section 3.5), and
5. Formulate the Plan (Section 3.6).
This process uses an outline similar to the more detailed process, but the techniques for
completing each phase have been simplified. (Remember that by abbreviating the training
program development, some of the benefits of a thorough detailed process may be lost.) Section
3.7 presents the steps of the detailed process and the impact of their removal.
Each step in the simplified process builds on the step that precedes it, with the exception
of Organize the Workload. In both the detailed and simplified approaches to developing a
training program, training needs will be specific to the tasks and functions at each level of the
quality system.
EPAQA/G-10 23 December 2000
-------
3.2 ALTERNATE PHASE 1: GET INFORMED
The purpose of the Get Informed phase of the process is to identify
GET INFORMED ] two things: (1) the reason for developing a training program and (2) to
what unit within the organization the training program should apply. The
following tasks are included in this phase:
ORGANIZE THE
WORKLOAD / , . . . . „
1. review existing information,
2. conduct observations, and
PERFORM GAP ^ 3. survey stakeholders.
ANALYSIS
The first task is to obtain and review existing documents or
IDENTIFY "METHODS^ information. Examples of such documentation include:
TO ADDRESS GOALS )
• policies or directives,
FORMULATE A J°b descriptions,
THE PLAN J • work plans,
assessment reports, and
• Inspector General reports.
Assessment reports may not attribute findings to a specific training need, but may point
out where performance issues exist within an organization. The other documents may indicate
skills needed by employees to fulfill either a new policy or a particular job. This information will
help determine if training is the most appropriate solution for the performance issue. When a lack
of knowledge, skill, or ability is not the cause of the performance problem, then a non-training
solution should be developed.
The next step in this phase is to conduct on-the-job observations. The observer should
have some familiarity with the tasks at hand or the job being observed. Performance issues can be
validated or reasons for developing the training refocused by observing employees as they go
about their work. The person making the observation should pay attention to an employee's:
1. understanding of the task/j ob,
2. ability to perform it within a reasonable amount of time,
3. method of approaching the task/j ob, and
4. quality of work.
The observations should be documented and included in the full report developed at the
end of this phase. If you are operating under time constraints, ask the employee to give you a
detailed description of his or her work.
The final step in this phase is to survey stakeholders, defined as those who have a
significant interest in the outcome of the training program, to determine to what extent current
quality systems training meets the needs of the organization. This information will contribute to
EPAQA/G-10 24 December 2000
-------
understanding of the reasons for developing the training needs identified in the first two tasks. It
can also be a means for performing a customer service check of the training currently provided to
the unit. The survey may indicate that the training topic or format can be altered to provide
maximum benefit to the employee.
At the completion of the first three activities in Phase I, document your findings in a
report. The report should include:
• the results from the information review,
observations of employees' work, and
• the results of the stakeholder survey.
Alternative Phase 1 Tasks
Complete the following tasks to get informed.
Review existing documents that indicate
training needs.
Observe employees on the job, or review
detailed work descriptions.
Survey stakeholders to determine to what
extent current training meets their needs.
Record conclusions in a report.
3.3 ALTERNATE PHASE 2: ORGANIZE THE WORKLOAD
GET INFORMED
ORGANIZE THE
WORKLOAD
PERFORM GAP
ANALYSIS
IDENTIFY METHODS
TO ADDRESS GOALS
FORMULATE
THE PLAN
Under optimal circumstances, a team will be responsible for
attending to all the details of developing a training program. The
development team can be as small as two individuals or as large as
resources allow. For more information on working as part of a team, refer
to Section 3.2, Phase 1. If these resources do not exist, an individual can
develop a successful training program as long as he or she is well organized
and communicates with stakeholders frequently. The following tasks are
useful in identifying the best individual(s) to develop the plan and for
assigning the project to them:
1. identify skills and experience needed to develop the plan,
2. identify the roles required to develop the plan,
3. list the possible candidates, and
4. select the best choice(s).
EPA QA/G-10
25
December 2000
-------
The first task is to identify what skills and/or experience will facilitate the training program
development process. Such skills or experience might include:
• familiarity with work of the unit,
• understanding of the relationship between the work of the unit and EPA's quality
assurance requirements, and
• familiarity with training principles or plans.
Once the necessary skills and/or experience to perform the job have been identified, a list of
candidates to work on developing the training plan should be prepared.
The second task of this phase is to consider the roles of the team or individual creating the
plan. Roles might include those knowledgeable in information collection, fiscal resources, or
organizational work processes.
After responsibility has been assigned, the following must also be done:
list the tasks to be completed using this document for guidance,
• establish specific guidelines for completing these tasks,
• identify time frames in which the tasks should be completed, and
• assign the tasks to a team member if the team approach is being used.
Before beginning Alternate Phase 3, the responsible parties should have a clear understanding of
their assignment and the roles they are expected to fill.
Alternate Phase 2 Tasks
Complete these tasks to organize the workload.
Identify the people or person responsible for
developing the training program.
Ensure that those responsible understand
the project requirements.
Make decisions about task assignments and
time lines.
EPA QA/G-10
26
December 2000
-------
3.4 ALTERNATE PHASE 3: PERFORM GAP ANALYSIS
The purpose of performing a gap analysis is to identify the
GET INFORMED J difference between what skills a unit requires to perform optimally and
what skills currently exist. This is similar to the needs assessment done in
Section 2.3. To identify the gap, the following tasks should be completed:
ORGANIZE THE
WORKLOAD
1. identify job roles and responsibilities,
2. identify required quality systems skills,
( PERFORM GAP ^ 3 identify existing skill levels,
^ — ^ 4. describe gaps, and
5. verify results.
IDENTIFY METHODS
TO ADDRESS GOALS ) Interviews of selected staff managers will provide information for
J L, the first three activities within this phase — identifying tasks of the unit,
FORMULATE ^ quality-related skills required to perform them, and existing skill levels
THE PLAN J within a unit. The focus of the first two tasks is on the jobs in the unit,
rather than the individual employees. Training needs to be targeted toward
specific jobs because its shelf life will be longer than the average employee's tenure in any one
job.
Interviews for the first task can be structured into two parts:
1. Identify the required skills of the unit:
• What tasks should be performed in the unit?
• What skills are required to perform the tasks?
• Which of those required skills are quality-related?
2. Identify existing quality-related skill levels:
• What quality-related tasks are performed in the unit?
• What skills are used to perform the quality-related tasks?
The fourth task is to compare the list of required skills against the list of existing skills to
see where the gaps exist. Refer to the Gap Analysis Matrix in Section 2.4 for a more structured
approach to the review. Describe those gaps in as much detail as possible, making note of
questions that arise during the review.
Finally, confirm the information and obtain answers to questions by meeting with
managers, stakeholders, and/or employees. Wherever possible, implement feedback from these
individuals.
EPAQA/G-10 21 December 2000
-------
Alternate Phase 3 Tasks
Complete this task to perform a gap analysis.
Identify, describe, and validate the gap
3.5 ALTERNATE PHASE 4: IDENTIFY METHODS TO ADDRESS GOALS
GET INFORMED
ORGANIZE THE
WORKLOAD
PERFORM GAP
ANALYSIS
IDENTIFY METHODS
TO ADDRESS GOALS
FORMULATE
THE PLAN
To identify methods to address goals., it is necessary to:
1. rank quality systems training needs,
2. determine what progress should look like, and
3. identify available training that meets high priority needs.
If all of the training needs cannot be met at once, it is useful to know
which needs are critical to the organization's mission and which are lower
priorities. Discuss the training needs with the stakeholders and try to obtain
some agreement on their ranking.
For some organizations, diagraming these priorities showing
secondary and tertiary needs might be necessary, since some secondary and
tertiary needs may support a primary need. A more detailed approach for
ranking the quality systems training goals can be found in Section 2.5.
After ranking the quality systems training needs, develop a list of milestones so that
progress can be identified and reported. This provides checkpoints for success.
There may be existing training that will meet some training needs in-house. Discuss this
possibility with training coordinators or managers. Chapter 4 contains more information about
quality systems training resources.
List the training needs and any existing training that can meet them, understanding that
some training may meet several needs and that others may meet partial needs. Refer to the
Training Resource Evaluation Matrix (Table 3) for assistance. All of this information should be
noted in detail so it can be included in the training program. Include the following: (1) training
goal, (2) its priority, (3) how it will be addressed, (4) the resources required to address it, and (5)
the availability of the necessary resources.
EPA QA/G-10
28
December 2000
-------
Alternate Phase 4 Tasks
Complete this task to identify methods to address goals.
Identify and document which of the currently
available training meets quality systems
training needs and to what extent.
3.6 ALTERNATE PHASE 5: FORMULATE THE TRAINING PROGRAM
GET INFORMED
ORGANIZE THE
WORKLOAD
PERFORM GAP
ANALYSIS
The final step in developing a quality systems training program is to
take the information gathered in the first four phases of development and
formulate the training program. There are four steps to this phase:
1. write a draft quality systems training program,
2. evaluate the program,
3. obtain feedback from stakeholders, and
4. develop a final quality systems training program.
The draft quality systems training program should include the
FORMULATE
THE PLAN
IDENTIFY METHODS > „ „ . . „
TO ADDRESS GOALS J following information:
Reasons for developing a training program: A detailed
account of the conclusions drawn from Alternate Phase 1:
Get Informed.
Gaps between required and existing skills: The results of
Alternate Phase 3: Perform Gap Analysis.
Plan of action: The results of Alternate Phase 4: Identify Methods to Address
Goals, a time line for addressing each goal, and assignments for roles and
responsibilities for completing each piece of the plan.
After the draft program is written, make sure that the plan will:
• address the original issues,
meet the training goals identified in Alternate Phase 3, and
• identify quality-related training needs that overlap with other organizational
training requirements.
The draft training program should be shared with the stakeholder group. Include their
feedback in the final training program. Upon completion and approval of the final training
EPA QA/G-10
29
December 2000
-------
program, training should be implemented as quickly as possible to maintain the momentum built
by the development process.
Alternate Phase 5 Tasks
Complete these tasks to formulate the training program.
Create a draft training program.
Evaluate the draft training program.
Validate the draft training program.
Develop the final training program.
3.7 EVALUATING THE APPROACH TO BUILDING QUALITY SYSTEMS
TRAINING PROGRAMS
Resource constraints may prompt the user to choose those aspects of the process that best
meet the needs of the organization. Careful evaluation of the impact of eliminating steps is
warranted in such cases. Table 6 chart explains the approach to building training programs. It
outlines the steps and purpose of each sub-process in Chapter 2, and the effect of eliminating each
step. The phases and steps listed here match the those in Chapter 2, however some steps have
been combined for streamlining purposes.
EPA QA/G-10
30
December 2000
-------
Table 6. Rationale of the Approach to Building a Quality Systems Training Program
Phase
Steps
Purpose
Effect of Elimination
1. Up-front
Review
Describe the reasons
for building a training
program
To provide insight into the
expectations the plan should meet; to
identify the problem the training
program will address
Unclear expectations, confusion, lack of
coordination of efforts, scope creep
Determine the best
solution
To research the problem and
determine if training or some other
method is the most appropriate
solution
Lack of information, unclear expectations,
lack of mission, lack of perspective, waste
of time
Create a team and
assign roles
To identify who will be responsible for
what; to gather a dedicated staff to
work on issues; to create a group of
stakeholders and supporters
Lack of support from organization, no
dedicated staff, unclear expectations and
no accountability, ambiguous leadership
roles and unwieldy responsibilities
2. Needs
Assessment
Decide what jobs to
assess
To identify the unit of jobs needing
assessment (Unit Assessment)
Lack of clear starting point, no consensus
on where to begin
Identify the KSAs
required to perform
the jobs
To identify responsibilities and
activities required for each role; to
break down the jobs into specific tasks
and evaluate the knowledge and skills
needed to perform them. (Main tasks,
decisions and options, sequence in
order of performance, create
flowchart, identify quality-related
KSAs) (Hierarchical Job Assessment)
No identification of desired state. If tasks
are not broken down, it's hard to see what
skills are required. This step helps create a
systematic approach; otherwise the large
amount of information may become
unwieldy, causing confusion and
overlooking important skills
EPA QA/G-10
31
December 2000
-------
Table 6. Rationale of the Approach to Building a Quality Systems Training Program
Phase
Steps
Purpose
Effect of Elimination
Identify those KSAs
that are quality-related
To identify those tasks that require
quality-related KSAs (Hierarchical Job
Assessment)
Lack of knowledge of required quality-
related skills; won't have a clear idea of
what needs to be accomplished
Determine the existing
knowledge and
experience
To examine the KSAs currently held
by groups of employees who are/will
be performing quality-related functions
(identify unit of assessment, conduct
research, confirm findings, document)
Won't have good idea of the level of
knowledge, skills, and abilities that already
exist. Impacts the ability to target training
3. Analyzing the
Needs Assessment
Information
Compare existing
skills with required
skills
To determine the gap between desired
and existing skills
Won't have good idea of what needs to be
accomplished; need to know what is in
place in order to figure out what needs to
be done
Validate the results
To get feedback from stakeholders and
resolve discrepancies
May be missing important feedback, input,
news, developments, support, perspective,
buy-in
4. Developing a
Quality Systems
Training Program
Rank the training
goals
To prioritize needs and goals to make
sure the program will address most
important issues; to gather input from
representatives of each level and assess
how important each goal is to the unit
Won't have priorities in place; won't
know what goals should be accomplished
first; may start on a path that isn't optimal
or sanctioned; won't have the big picture
EPA QA/G-10
32
December 2000
-------
Table 6. Rationale of the Approach to Building a Quality Systems Training Program
Phase
Steps
Purpose
Effect of Elimination
Define success
To provide a unified understanding of
goals; to specify how people will know
that the quality systems training needs
have been achieved (how it would
look if the goal were achieved, how
work would flow, speed of work
completion, results)
Won't know when real progress has been
made or where changes need to be made
Compare the current
quality systems
training resources to
training goals
To determine what quality-related
training is currently being offered and
how it could be used to meet training
objectives
May cause redundant work, wasting time,
money, and effort
Validate and
document results
To get feedback from people familiar
with the courses and training needs
Won't have the full information on a
subject or the right perspective; might
think that a course achieves a goal that it
doesn't
5. Constructing
the Quality
Systems Training
Program
Create a time line for
achieving each training
goal
To determine when goals will be met
(based on priority and available
resources); to provide a framework for
determining success and progress
Won't have a clear schedule; might cause
confusion, slow down the pace, and cause
missed deadlines
Identify how each
training goal will be
met
To balance the pros and cons of each
approach to addressing training needs;
to detail priorities, deadlines,
approaches, resources needed, and
availability of resources for each goal
Might choose the wrong approach and not
know it until it is too late; might cause
waste and rework
EPA QA/G-10
33
December 2000
-------
Table 6. Rationale of the Approach to Building a Quality Systems Training Program
Phase
Steps
Purpose
Effect of Elimination
Consolidate programs;
Identify redundancy
To determine when needs or efforts to
address them are being duplicated, to
save time, resources, fill needs
Redundancy, waste of time, money and
effort
Identify roles and
responsibilities
To determine who should be doing
what; to familiarize new members with
the background of the program
Confusion and irritation for involved staff;
unrealistic expectations; low group
cohesion, lack of accountability
6. Evaluating the
Quality Systems
Training Program
Compare the program
to the original request
To review original reasons for this
training program; to make sure the
project is on track; if not, to correct;
to get input from originator of request
Won't know if the training program is
correctly focused; waste of time, money,
and effort if later it is determined that the
training program doesn't address concerns
Compare the program
to training goals
To determine if program is likely to
meet original goals; to determine the
need for revisions to meet goals
Waste of time, money, effort if later it is
determined that the training program does
address goals
Obtain stakeholder
feedback (should
happen throughout -
not at end)
To get feedback from stakeholders; to
implement suggestions from
stakeholders
Lack of support, perspective, help, buy-
in; leading to an incomplete effort
EPA QA/G-10
34
December 2000
-------
CHAPTER 4
QUALITY SYSTEMS TRAINING RESOURCES
A single generic training program cannot address all possible EPA-wide quality systems
training needs. Organizations and programs are complex and diverse. However, core training
courses should be able to address and satisfy basic organization-specific training needs.
Changing roles and priorities as well as new organizational needs all contribute to the
challenge of maintaining skills and developing new areas. An organization chooses resources
based on identified training needs, selected training strategies, and budget allowances. Some
training resources are discussed below. Other possible resources include Human Resources
Departments and other Federal and State agencies. This list is not all inclusive, but provides a
starting point in evaluating resources for the quality systems training program.
4.1 AGENCY-WIDE QUALITY SYSTEMS TRAINING RESOURCES
The Quality Staff is one resource for the EPA Quality System that provides training
materials, guidance documents, trained instructors, and assistance to EPA organizations. Two
types of training resources have been developed by the Quality Staff:
generic training materials for Agency-wide tailoring and use, and
• advanced training courses delivered by the Quality Staff.
Generic courses developed by the Quality Staff are intended to provide users with a working
understanding of the EPA Quality System and its components. These training courses enhance an
employee's ability to address tasks associated with planning, implementation, and assessment of
the EPA Quality System. The generic courses may be tailored to fit the specific requirements of
EPA organizations. Staff and management must be trained to adequately perform quality-related
tasks defined in EPA Order 5360.1 A2.
While there is no Agency-wide quality systems training requirement, Table 7 shows
examples of quality-related functions and suggested training to support those functions. For a
complete listing of Quality Staff training, contact the Quality Staff (see Foreword).
4.2 IN-HOUSE COURSE DEVELOPMENT
There will likely be instances when current quality systems course offerings and contractor
services are not readily available and in-house development is the only option. This section
provides information about the general steps involved with course development. More
information can be found in the reference section of this document. The general steps involved in
creating a course include:
EPAQA/G-10 35 December 2000
-------
Table 7. Quality Systems Training Matrix Example
QA and QC Functions
Suggested Quality Systems
Training
Quality
System Level
Establish, document, and periodically
revise Agency policies and procedures
for planning, implementing, and
assessing the effectiveness of the
mandatory, Agency-wide Quality
System.
Introduction to EPA Quality
System Requirements
Policy
Ensure that all organizational
components and applicable programs
fully comply with the requirements of
EPA Order 5360.1 Al.
Introduction to EPA Quality
System Requirements
Program
Perform periodic assessments of
environmental programs to determine
the effectiveness of their mandatory
quality systems.
Management Systems Review
Workshop
Planning and Implementing
Quality System Audits
Program
Ensure that environmental data are of
sufficient quantity and adequate
quality for their intended use and are
used consistent with such intentions.
Introduction to DQOs
Integrating QA into Project
Development (Advanced)
Sampling Designs for QA
Project Plans (Advanced)
Project
Ensure that all Agency funded
environmental programs implemented
through extramural agreements
comply fully with applicable QA and
QC requirements.
Introduction to EPA Quality
System Requirements
Introduction to DQOs
Integrating QA into Project
Development (Advanced)
Data Quality Assessment
Project
develop course objectives,
• select training methods,
select practice activities,
• develop a course outline,
develop a training schedule, and
• develop a training syllabus.
4.2.1 Develop Course Objectives
The course objective describes what an individual will know or be able to do after
completing the course. A course objective should include three components:
EPA QA/G-10
36
December 2000
-------
• what KSAs are to be demonstrated,
who is to demonstrate the KSAs, and
• when the KSAs are to be demonstrated.
In addition, a course objective can include two more components:
• what tasks the individual may be given to demonstrate the KSAs and
• what level of achievement is required for the KSAs.
Each course should have at least one objective. Each course objective should be
supported by a series of enabling objectives that outline the KSAs one must acquire in order to
achieve the course objective.
The Objectives Form (Box 3) provides a guide for creating course objectives and their
enabling objectives. Enabling objectives provide order to training presentations because they lead
the training to the course objective. Enabling objectives consist of two components:
the action to be taken and
• the object of the action.
Box 3. Objectives Form
Course Objective: After completing this course, the student will be able to apply
quality-related concepts to environmental study plans and reports
Enabling Objectives: To attain the course objective, the student will:
1 . Identify quality-related concepts
2. Identify the purpose of environmental study plans and reports
3 . Understand the impact of quality-related concepts on plans and reports
_
4.2.2 Select Training Methods
The selection of the training method relies heavily on the objectives that are to be taught
in the course. There are several considerations to be made in making this selection.
• Is the method appropriate for the objectives?
Are the resources available?
• What facilities are required?
What is the projected size of the group to be trained?
• What is the background of those attending?
EPAQA/G-10 37 December 2000
-------
• What are the most common training methods (illustrated lecture, demonstration,
discussion, case studies, guided practice, self-paced learning, distance learning,
coaching, on-the-job)?
4.2.3 Select Practice Activities
Individuals receiving information or developing skills should be able to practice their new
knowledge and receive feedback on their applications. These practice activities should be
dynamic and engaging, and they should require the individual to interact with the content and/or
other people in the course.
Feedback should occur as soon as possible after the activity and be clearly stated so that
the student can integrate it with their new knowledge.
Here are some examples of practice activities.
• Solve a series of problems.
Answer a series of questions.
• Locate information in a manual.
Perform a series of calculations.
• React to a case study.
• Practice a skill following instructions in a technical manual.
• Participate in a role play.
Conduct an interview.
4.2.4 Develop Course Outline
The course outline is a plan of the training that will be delivered. It is divided into five
columns as illustrated in Table 8.
Table 8. Course Outline Template
Topics
Primary
concepts to
be covered
during the
course
Sub-topic
Concepts that
support each
topic. There will
likely be more
than one sub-
topic for each
topic
Teaching
Points
A list of items
to be discussed
during the
discussion of
each sub -topic
Instructional
Strategy
Method for
conveying the
concepts (for
example, lecture,
exercise, video,
experiential
activity)
Duration
How long it will
take to complete
each Instructional
Strategy. The
time can be
totaled at the end
of each topic
EPA QA/G-10
38
December 2000
-------
4.2.5 Develop a Training Schedule
From the course outline, a training schedule should be created to distribute to participants
on the first day of training. The schedule will help the developer ensure that all training objectives
and activities are completed in the allotted time.
4.2.6 Develop a Training Syllabus
A training syllabus provides information about a course and is often given to participants
in advance of the course. It can include the following information:
course title and description,
• course objectives,
course topics,
• description of the target audience,
length of the course,
• course prerequisites,
location of the course,
• information about the instructor,
description of materials the participant will need or receive, and
• description of any items the participant should bring to the course.
In conclusion, developing a course can be resource intensive and should not be attempted
without careful consideration of the costs and benefits. Prior familiarity with the topic, the use of
existing documentation, and effective collaboration among the principals involved in design and
delivery of training materials will expedite the process.
4.3 USE OF CONTRACTORS/CONSULTANTS
Many situations may warrant procurement of training expertise from outside sources.
Quality-related responsibilities may require highly technical and specialized expertise.
Additionally, the most cost effective approach may be to use the services of contractors or
consultants. Using consultants may be the most cost-effective way to meet training needs when
you need unique skills, specialized experience, or knowledge that is not readily available within
your organization.
Organizations such as the American Society of Quality, the American Chemical Society,
and others offer comprehensive lists of quality-related training courses that cover a wide range of
topics. Private firms also offer courses in quality management and related quality topics.
4.3.1 Inherently Government Functions
Contractors and consultants often can be used to create or to support the production of
training materials. However, many quality system activities involving environmental data
EPAQA/G-10 39 December 2000
-------
operations are inherently governmental functions and must be performed only by EPA personnel
or by personnel explicitly authorized by EPA based on statute, regulation, or by the terms of an
extramural agreement. Such representatives may include other governmental personnel and with
specific authorization, contractor personnel. When such quality management tasks are performed
by a contractor, the contract must be appropriately managed and must remain under the control of
the authorized EPA contracting representatives. EPA cannot use cooperative agreements or
grants to provide quality management activities such as QA and QC services for EPA
because it is an inappropriate use of financial assistance (Office of General Counsel
memorandum, August 2,1994).
Technical direction or other instructions to an extramural organization, relating to
performance of an extramural agreement, shall be provided only by authorized EPA or other
Government representatives in accordance with the terms of the applicable extramural agreement.
Only authorized EPA or other Government representatives are to provide direction or instructions
to an extramural organization providing quality systems support for environmental programs.
This is to avoid such actions as:
• providing directions or instructions that are inconsistent with the terms of an
extramural agreement,
• unauthorized access to confidential business information, or
unauthorized access to information that may allow an extramural organization to
gain an unfair competitive advantage.
4.3.2 Prepare and Present Quality Systems Training Materials and Courses
Exclusively EPA Functions are defined as inherently governmental work which must be
performed only by responsible EPA officials, including the QA Managers, or authorized EPA
representatives. Exclusively EPA training related functions include developing and presenting
detailed guidance and training for QA and QC activities based on interpretation of Agency-wide
requirements and guidance.
Discretionary Functions are defined as activities that may be performed either by EPA
personnel or by non-EPA personnel under the specific technical direction of and performance
monitoring by the QA Manager or other responsible EPA or Government official under an
approved contract, work assignment, delivery order, task order, etc. Discretionary training-
related functions include:
• providing or coordinating quality-related training for the organization in special
skill areas identified by the Agency and not generally available to the organization;
and
• providing allowable technical and/or logistical assistance in preparing and
presenting quality-related technical training (within the Agency's implementation
of special management and control measures and the constraints of potential for
EPAQA/G-10 40 December 2000
-------
conflict of interest, of revealing confidential business information, or of appearing
to be interpreting or representing Agency policy).
4.3.3 Additional Considerations
When using contractor services for training needs, the process of articulating
requirements, setting project parameters, and defining outputs so that specific training needs are
met can be complex. Accordingly, it is important to:
ensure that the organization's needs are explicitly defined;
• define the task completely before beginning the procurement process;
agree on checkpoints, outputs, and expected results; and
• carefully monitor progress and provide direction as needed.
If the primary role of the contractor is to design and deliver instructional materials, it is
helpful to ensure that the selected firm has a working knowledge of the Instructional Systems
Design process, various training methods, and the application of technology in training practices.
It can also be helpful for the firm to have some knowledge and understanding of the topic being
taught. When considering contractor/consultant services for training instructional design services,
it is important to determine that the firm has:
an understanding of the EPA Quality System;
• a demonstrated capacity to assess organizational training needs;
a strategy to involve the organization in assessment, planning, and critical stages of
decision making;
an approach that will adapt established methodologies for training strategies to the
specific organization with which they work; and
the ability to change from a content-based technique to a process orientation as
organizational needs and issues arise.
Finally, contractor/consultant services should enhance the overall training effort without
imposing a consultant-created plan. The organization's contributions to the training program are
critical to its success.
EPAQA/G-10 41 December 2000
-------
EPAQA/G-10 42 December 2000
-------
BIBLIOGRAPHY
40 CFR 30, Code of Federal Regulations, "Grants and Agreements With Institutions of
Higher Education, Hospitals, and Other Non-Profit Organizations."
40 CFR Part 31, Code of Federal Regulations, "Uniform Administrative Requirements for Grants
and Cooperative Agreement to State and Local Governments."
40 CFR Part 35, Code of Federal Regulations, "State and Local Assistance."
40 CFR Part 46, Code of Federal Regulations, "Federal Acquisition Regulations."
ANSI/ASQC E4-1994, Specifications and Guidelines for Quality Systems for Environmental
Data Collection and Environmental Technology Programs, American National Standard,
January 1995.
EPA Order 5360.1 A2, Policy and Program Requirements for the Mandatory Agency-wide
Quality System (May 2000).
EPA Order 5360, The EPA Quality Manual for Environmental Programs (May 2000).
EPA Order 1900, Contracts Management Manual (February 1998).
Gordon, Sallie E._( 1994) Systematic Training Plan Design: Maximizing Effectiveness
and Minimizing Liability. Englewood Cliffs, NJ: PTR Prentice Hall.
Smith, Barry J. and Delahaye, Brian L. (1947) How to be an Effective Trainer 2nd Edition.
New York: John Wiley & Sons, Inc.
U.S. Environmental Protection Agency, 2000a. Guidance for Data Quality Assessment:
Practical Methods for Data Analysis (QA/G-9), EPA/600/R-96/084, Office of
Environmental Information.
U.S. Environmental Protection Agency, 2000b. Guidance for the Data Quality Objectives
Process (QA/G-4), EPA/600/R-96/055, Office of Research and Development.
U.S. Environmental Protection Agency, 1999a. EPA Requirements for Quality Assurance Project
Plans (QA/R-5) Interim Final., Office of Research and Development.
U.S. Environmental Protection Agency, 1999b. EPA Requirements for Quality Management
Plans (QA/R-2) Interim Final, Office of Research and Development.
U.S. Environmental Protection Agency, 1998. Guidance for Quality Assurance Project Plans
(QA/G-5), EPA/600/R-98/018, Office of Research and Development.
EPAQA/G-10 43 December 2000
-------
U.S. Environmental Protection Agency, 1995. Guidance for the Preparation of Standard
Operating Procedures (SOPs) for Quality-Related Documents (QA/G-6), EPA/600/R-
96/027, Office of Research and Development.
Office of Federal Procurement Policy, 1992. OFPP Policy Letter 92-1, "Inherently
Governmental Functions," 57 FR 45096, Washington, DC.
EPAQA/G-10 44 December 2000
-------
APPENDIX
FORMS FOR DEVELOPING A TRAINING PROGRAM
EPAQA/G-10 45 December 2000
-------
Unit Assessment Form
Date:
Manager's Name:
Unit Description:
Job List (include all functions in the unit listed above)
Job Title
Responsibilities
Staff Size:
Primary Tasks
EPA QA/G-10
46
December 2000
-------
Hierarchical Job Assessment Form
A. Job Title
B. Primary Job Tasks
1.
2.
C. List the Sub-tasks related to each task
Taskl
Sub-task 1
Sub-task 2
Sub-task 3
Task 2
Sub-task 1
Sub-task 2
D. List KSAs needed to perform each sub-task
Taskl
Sub-task 1
Sub-task 2
Sub-task 3
Task 2
Sub-task 1
Sub-task 2
E. List Quality-Related KSAs
Taskl
Task 2
EPA QA/G-10
47
December 2000
-------
Ranking Matrix Form
TRAINING
GOALS
CRITERIA
Weight %
Weight %
Weight %
Weight %
Total
Weight
%100
EPA QA/G-10
48
December 2000
-------
Training Resource Matrix Form
Courses
Training Goals
EPA QA/G-10
49
December 2000
-------
Quality Systems Training Program Form
Training Goal
Priority
Due
Date
Training
Method(s)
Resources
Needed
Resource
Availability
EPA QA/G-10
50
December 2000
-------
Course Objectives Form
Course Objective 1: After completing this course, will be able to
Enabling Objectives: To attain the course objective, will:
1.
2.
3.
4.
Course Objective 2: After completing this course, will be able to
Enabling Objectives: To attain the course objective, will:
1.
2.
3.
4.
Course Objective 3: After completing this course, will be able to
Enabling Objectives: To attain the course objective, will:
1.
2.
3.
4.
Course Objective 4: After completing this course, will be able to
Enabling Objectives: To attain the course objective, will:
1.
2.
3.
4.
EPAQA/G-10 51 December 2000
-------
Course Outline Template Form
Topic
Sub-topic
Teaching
Points
Instructional
Strategy
Duration
EPA QA/G-10
52
December 2000
------- |