United States
Environmental Protection
Agency
Office of Environmental
Information
Washington, DC 20460
EPA/240/R-03/002
March 2003
Guidance on Assessing
Quality Systems
EPA QA/G-3
-------
FOREWORD
The U.S. Environmental Protection Agency (EPA) has developed an Agency-wide program of
quality assurance for environmental data that includes documentation of both management and technical
activities. This guidance document, Guidance on Assessing Quality Systems., provides methods and
tools for assessing quality systems and provides information about assessments for those who use them.
It is pertinent to organizations that carry out environmental data operations within or for EPA.
This document provides guidance to program managers and assessment teams within EPA and
extramural organizations with quality systems based on EPA policies. The EPA's Quality System has
been built to ensure that environmental programs are supported by the type and quality of data
appropriate for their intended use. This document is valid for up to five years from the official date of
publication per EPA Quality Manual for Environmental Programs, Order 5360 Al (EPA, 2000a).
After five years, this document will be reissued without change, revised, or withdrawn from the U.S.
Environmental Protection Agency Quality System Series documents.
EPA works every day to produce quality information products. The information used in these
products are based on Agency processes to produce quality data, such as the quality system described
in this document. Therefore, implementation of the activities described in this document is consistent
with EPA's Information Quality Guidelines and promotes the dissemination of quality technical,
scientific, and policy information and decisions.
This document does not impose legally binding requirements on EPA or the public and may not
apply to a particular situation based on the circumstances. EPA retains the discretion to adopt
approaches on a case-by-case basis that differ from this guidance where appropriate. Interested
parties are free to raise questions about the recommendations in this document and the appropriateness
of using them in a particular situation, and EPA and other parties should consider whether the
recommendations in the document are appropriate for the particular situation. EPA may periodically
revise this guidance without public notice.
This document is one of the EPA Quality System Series documents, which describe policies
and procedures for planning, implementing, and assessing the effectiveness of a quality system.
Questions regarding this document or other EPA Quality System Series documents can be directed to:
U.S. EPA
Quality Staff (2811R)
1200 Pennsylvania Ave., NW
Washington, D.C. 20460
Phone: (202)564-6830
e-mail: quality@epa.gov
EPAQA/G-3 1 March 2003
-------
EPAQA/G-3 11 March 2003
-------
TABLE OF CONTENTS
Page
CHAPTER 1. ASSESSMENTS IN THE QUALITY SYSTEM 1
1.1 PURPOSE 1
1.2 QUALITY SYSTEM CONTEXT 1
1.3 ASSESSMENTS OF QUALITY SYSTEMS 4
1.4 SPECIFICATIONS FOR ASSESSMENTS OF QUALITY SYSTEMS 5
1.5 GRADED APPROACH 6
1.6 INTENDED AUDIENCE AND PERIOD OF APPLICABILITY 7
1.7 ADDITIONAL REFERENCES 8
CHAPTER 2. MANAGING ASSESSMENTS 9
2.1 ASSESSMENT ROLES AND RESPONSIBILITIES 9
2.2 ASSESSMENT SYSTEMS 10
2.3 DECISION TO CONDUCT THE ASSESSMENT 12
2.4 CRITERIA FOR THE ASSESSMENT 12
2.5 SCOPE OF THE ASSESSMENT 13
2.6 THE ASSESSMENT TEAM 14
2.6.1 Assessment Team Selection 14
2.6.2 Assessment Team Leader Responsibilities 15
2.6.3 Assessor Responsibilities and Qualifications 15
2.7 ASSESSMENT RESOURCES 16
CHAPTERS. PREPARING FOR THE ASSESSMENT 17
3.1 DOCUMENTATION AND TRACKING 19
3.2 ASSESSMENT TEAM PREPARATION 19
3.3 INITIAL CONTACT WITH THE ASSESSES 19
3.4 INFORMATION REVIEW 20
3.5 ISSUE SELECTION 21
3.6 IDENTIFY INTERVIEWEES AND DOCUMENTS 22
3.7 ALTERNATIVES TO ON-SITE INTERVIEWS 24
3.8 PREPARATION OF THE ASSESSMENT PLAN 25
3.9 REVIEW AND APPROVAL OF THE ASSESSMENT PLAN 25
3.10 CONFIDENTIALITY 27
3.11 COORDINATION AND LOGISTICAL ARRANGEMENTS 27
3.12 FORMAL NOTIFICATION 28
3.13 CHECKLISTS AND OTHER AIDS 28
CHAPTER 4. CONDUCTING THE ASSESSMENT 35
4.1 OPENING MEETING 36
EPA QA/G-3 iii March 2003
-------
Page
4.2 DOCUMENTS AND RECORDS REVIEW 36
4.3 INTERVIEWS 37
4.4 ASSESSMENT WORKING PAPERS 39
4.5 PRELIMINARY FINDINGS 39
4.6 CLOSING MEETING 40
CHAPTER 5. REPORTING AND FOLLOW-UP 43
5.1 EVALUATING COLLECTED ASSESSMENT INFORMATION 43
5.2 REPORTING FINDINGS 44
5.3 CORRECTIVE ACTION AND FOLLOW-UP ACTIVITIES 45
5.4 FORMAL CLOSE OUT OF ASSESSMENT 46
5.5 QUALITY IMPROVEMENT 47
CHAPTER 6. REFERENCES AND SUPPLEMENTAL READING 49
6.1 REFERENCES 49
6.2 SUPPLEMENTAL READING 50
GLOSSARY 53
APPENDIX A. GUIDANCE FOR BEING ASSESSED A-l
APPENDIX B. INTERVIEWING SKILLS B-l
APPENDIX C. EXAMPLE ISSUES AND INTERVIEW QUESTIONS C-l
APPENDIX D. EXAMPLE CHECKLIST D-l
EPA QA/G-3 IV March 2003
-------
LIST OF FIGURES
Page
Figure 1. EPA Quality System Components and Tools 3
Figure 2. Systematic Planning Activities for an Assessment 18
Figure 3. Example Job Categories For Interviews 23
Figure 4. Example Documents and Features To Review 24
Figure 5. Example Contents of a Plan for Assessing a Quality System 26
Figure 6. Example Formal Notification Letter 29
Figure 7. Flow Chart for Conducting the Assessment 35
Figure 8. Agenda for the Opening Meeting 36
Figure 9. Agenda for the Closing Meeting 41
Figure 10. Typical Steps for Assessment Reporting and Foilow-Up 43
Figure 11. Example Assessment Report Outline 44
Figure 12. Example Close-Out Letter 47
LIST OF TABLES
Page
Table 1. Examples of Assessment Roles 9
Table 2. Example Row of Working Paper Matrix for Recording Assessment
Observations 33
Table 3. Example of a Corrective Action Plan 45
Table B-l. Types of Questions B-2
EPAQA/G-3 V March 2003
-------
EPAQA/G-3 VI March 2003
-------
CHAPTER 1
ASSESSMENTS IN THE QUALITY SYSTEM
1.1 PURPOSE
This document provides guidance for assessing quality systems, particularly for programs
conducted by or funded by the U.S. Environmental Protection Agency (EPA), including:
intramural environmental programs performed by EPA organizations, and
• environmental programs performed under EPA extramural agreements [i.e., contracts,
grants, cooperative agreements, and interagency agreements (IAGs)].
Non-mandatory guidance is provided to help these organizations plan, conduct, evaluate, and
document assessments of quality systems. It contains advice to help these organizations develop an
assessment program and for conducting assessments of internal and external quality systems.
Establishing and implementing an effective assessment program are integral parts of a quality
system. Assessments of an organization's quality system and technical systems provide management
with information that is needed to evaluate and improve an organization's operation, including:
the organizational progress in reaching strategic goals and objectives,
• the adequacy and implementation of programs developed to achieve the mission,
• the quality of products and services, and
the degree of compliance with contractual and regulatory specifications.
1.2 QUALITY SYSTEM CONTEXT
A quality system is a structured and documented management system which consists of the
policies, objectives, principles, organizational authority, responsibilities, accountability, and
implementation plan of an organization for ensuring quality in its work processes, products, and
services. It provides the framework for planning, implementing, documenting, and assessing the work
performed by the organization and for carrying out quality assurance (QA) and quality control (QC)
activities.
Since 1979, EPA policy has specified participation in an Agency-wide quality system by all
EPA organizations (i.e., offices, regions, national centers, and laboratories) supporting intramural
environmental programs and by non-EPA organizations performing work funded by EPA through
extramural agreements. EPA's quality system operates consistent with Policy and Program
Requirements for the Mandatory Agency-wide Quality System, Order 5360.1 A2 (EPA, 2000a),
hereafter called the Order. Specifications for implementing the Order in EPA organizations are given in
EPAQA/G-3 1 March 2003
-------
EPA Quality Manual for Environmental Programs, 5360 Al (EPA, 2000b), herein called the
Manual. Specifications for extramural organizations are given in 40 Code of Federal Regulations
(CFR) Parts 30, 31, and 35 and EPA Requirements for Quality Management Plans (EPA QA/R-2)
(EPA, 2001). Figure 1 illustrates EPA's quality system. All EPA QA documents mentioned here are
available at http://www.epa.gov/quality.
EPA bases its quality system on Specifications and Guidelines for Quality Systems for
Environmental Data Collection and Environmental Technology Programs (ANSI/ASQC E4-
1994), which was developed by the American National Standards Institute (ANSI) and the American
Society for Quality (formerly the American Society for Quality Control). EPA quality system
components are based on these specifications, so it is not necessary to consult ANSI/ASQC E4 to
follow EPA quality system specifications. Extramural quality systems that demonstrate compliance with
ANSI/ASQC E4 for quality systems are also consistent with EPA policy.l
According to ANSI/ASQC E4, assessments of environmental programs should be conducted
periodically and the assessment findings should be evaluated to measure the effectiveness of the
programs' quality systems. The types of assessments that can be conducted include management self-
assessments, management independent assessments, technical self-assessments, and technical
independent assessments. The specific type of assessment that is used is determined by management.
Every EPA organization or extramural organization performing work funded by EPA should
document its quality system in an approved Quality Management Plan (QMP). Under a QMP, all
steps associated with the generation of environmental data should be documented and such documenta-
tion should be verifiable and defensible. Because Agency decisions rely on the quality of environmental
data, it is imperative that the effectiveness of the quality systems that support the collection and use of
environmental data be periodically assessed.
Section 7.a(3) of the Order specifies that the Agency Senior Management Official for Quality
perform periodic management assessments of all EPA organizations, and Section 6.a(4) specifies that
EPA organizations perform assessments of the effectiveness of their quality system at least annually.
Federal regulations governing extramural agreements addressed in 48 CFR Part 1546 and 40 CFR
Parts 30, 31, and 35 mention the assessment of extramural organizations by EPA.
Extramural organizations, which include financial assistance agreement recipients, cooperative
agreement recipients, contractors, and grantees (States, tribal governments, local governments,
universities, contractors, etc.), should also conduct periodic internal assessments of their own quality
systems. An extramural organization's assessment process is described in its
'The References and Supplemental Reading sections in Chapter 6 list documents from the American Society
for Quality and the International Organization for Standardization, which may be helpful to the reader.
EPAQA/G-3 2 March 2003
-------
O
LU
O
a:
a.
a:
o
o
01
a.
N
o
a:
o
w
o
s
O
LLI
O
_l
o
0.
Consensus Standards
ANSI/ASQC E4
ISO 9000 Series
External Regulations
Contracts - 48 CFR 46
Assistance Agreements -
40 CFR 30, 31, and 35
Internal EPA Policies
EPA Order 5360.1
EPA Manual 5360
EPA Program &
Regional Policy
rj
Quality System
Documentation
(e.g., Quality Management Plan)
Supporting System Elements
(e.g., Procurements,
Computer Hardware/Software)
I
Training/Communication
(e.g., Training Plan,
Conferences)
Annual Review and Planning
(e.g., QA Annual Report
and Work Plan)
System Assessment
(e.g., Quality System Audit)
Systematic
Planning
(e.g., DQO Process)
QA
Project Plan
Conduct
Study/
Experiment
Data Verification
& Validation
Technical
Assessments
PLANNING
IMPLEMENTATION
ASSESSMENT
Defensible Products and Decisions
Figure 1. EPA Quality System Components and Tools
EPA QA/G-3 3
March 2003
-------
QMP. Additionally, assessments play an important role in the continuous improvement process.
EPA QMP specifications are generally implemented through grants and enforcement decrees,
and orders. The particular plan elements will depend on the terms of the particular order.
1.3 ASSESSMENTS OF QUALITY SYSTEMS
An assessment of a quality system is a systematic, independent, and documented examination
that uses specified assessment criteria to answer one or more of the following questions about an
organization's quality system:
• If an organization is developing a quality system, what QA activities remain to be
implemented and what technical assistance by the assessors will promote the
development and implementation of this quality system?
Is the organization's quality system documented and fully implemented?
• Has the organization effectively implemented external quality specifications?
• Do the activities that are being performed by the organization follow its quality system
documentation, particularly the QMP?
Are the quality system procedures implemented effectively?
• Does the quality system support environmental decision making with processes that
ensure that data are sufficient in quantity and quality appropriate for their intended
purpose?
An assessment is designed to provide objective feedback about the quality system. It evaluates
and documents the management policies and procedures that are used to plan, implement, assess, and
correct the technical activities for environmental programs. It includes quality system document review,
file examination and review, and interviews of managers and staff responsible for environmental data
operations. Assessments can be conducted for specific environmental programs within organizations.
Assessments can apply to entire organizations, suborganizational units, and one or more specific
environmental programs within the organization.
This guidance addresses assessments of quality systems at the organization level that focus on
process rather than the quality of data from specific projects. Depending upon which of the previous
questions are addressed and local usage of terms, these assessments also have been referred to as
quality system audits, management assessments, and management systems reviews (MSRs) or
management system audits. For example, the term MSR is used to describe an assessment of a
developing quality system. MSRs may include providing technical assistance for improving a quality
system as an assessment objective.
One purpose of assessments is to improve the quality system, whether it is implemented or
developing. To accomplish this purpose, the objectives of an assessment should be appropriate to the
current stage of the quality system. For a developing quality system, the objectives may be to perform
EPAQA/G-3 4 March 2003
-------
a gap analysis of the quality system and to advise the assessee about any components of the quality
system for which more support and training are needed. For an implemented quality system, the
objective may be to determine whether the quality system is effective as implemented. Systematic
planning should ensure that an assessment's objectives are appropriate.
Another purpose of assessments is to provide valid feedback to management on the adequacy,
implementation, and effectiveness of the quality system. Assessments are helpful because the process
emphasizes noting good practices and suggesting changes for improving the quality system that provides
data for defensible environmental decisions.
In addition, the overall assessment program is beneficial to the Agency-wide quality system.
Assessors are in a good position to gather information on the reasonableness of the quality
specifications and the consistency of their implementation across all organizations and programs.
Assessments could indicate that additional quality policies and procedures, guidance documents, etc.,
need to be developed and implemented, or that additional training needs to be developed and provided.
Findings from assessments may lead to modifications of specific management or technical practices to
improve environmental decision making. Assessments of quality systems benefit the Agency in general
by providing increased confidence in environmental decisions and strengthening its overall credibility.
Assessments of quality systems are similar to technical assessments in many ways; they both
need planning, qualified assessors, and reports, for instance. The focus of the assessments is different,
however. Technical assessments emphasize technical activities, such as chains-of-custody and
analytical measurements, often on a specific project. Assessments of a quality system are at a higher,
more system-oriented level and emphasize organizational activities, such as systematic planning and
training.
1.4 SPECIFICATIONS FOR ASSESSMENTS OF QUALITY SYSTEMS
An organization's QMP spells out roles and responsibilities for implementing assessments, as
well as the uses of assessment tools in the organization. QMPs discuss or address the following items
pertaining to management and technical assessments:
how the process for planning, scheduling, and implementation of assessments works
and how the organization will respond to needed changes
• the responsibilities, levels of participation, and authority for all management and staff
participating in the assessment process
• how, when, and by whom actions will be taken in response to findings of assessments
and how the effectiveness of the response will be determined.
Furthermore, EPA's QMPs generally should describe or reference the processes (i.e., roles,
responsibilities, and authorities) of management and staff for:
EPAQA/G-3 5 March 2003
-------
assessing the adequacy of the quality system at least annually
planning, implementing, and documenting assessments and reporting findings to
management including how to select assessment tools, the expected frequency of their
application, and the roles and responsibilities of assessors
determining the level of competence, experience, and training necessary to ensure that
personnel conducting assessments are technically knowledgeable, with no real or
perceived conflict of interest, and have no direct involvement or responsibility for the
work being assessed
ensuring that personnel conducting assessments have sufficient authority and access to
programs, managers, documents, and records, and organizational freedom to:
identify both quality problems and noteworthy practices
propose recommendations for resolving quality problems
independently confirm implementation and effectiveness of solutions
• having management's review of and respond to findings
• identifying how and when corrective actions are to be taken in response to assessment
findings, ensuring that corrective actions are made promptly, confirming the
implementation and effectiveness of any corrective action, and documenting such
actions that include:
identifying root causes
determining whether the problem is unique or has more generic implications
recommending procedures to prevent recurrence
addressing any disputes encountered as a result of assessments.
1.5 GRADED APPROACH
The EPA quality system is characterized by the principle of the "graded approach," which
allows QA managers to base the QA and QC activities that are implemented in an organizational area
or project on the intended use by the environmental program and on the confidence that is needed and
expected in the quality of the program. The graded approach is also used in developing an assessment
strategy that is appropriate for both the organization that performs the assessments and the quality
system that is assessed. This approach starts with systematic assessment planning and continues
through the assessment's implementation and reporting phases. The graded approach is used to guide
assessment planning decisions and to guide the collection of desired information about the quality
system being assessed.
The use of the quality management components and tools in the organization, program and
project levels is based on a graded approach where components and tools are applied according to the
scope of the program and/or the intended use of the outputs from a process (EPA, 2002). This
approach recognizes that a "one size fits all" approach to quality specifications is not appropriate for an
organization as diverse as EPA. Applying a graded approach means that quality system tools and
EPAQA/G-3 6 March 2003
-------
components for different organizations and programs will vary according to the specific objectives and
needs of the organization.
The graded approach describes the idea that the level of intensity and rigor devoted to an
assessment is commensurate with the scope and risks associated with the quality system being
assessed. In essence, if the consequences of failure are small, then relatively little is at stake, and less
effort and fewer resources may be spent on assessment. On the other hand, if an important or highly
visible quality system is under consideration or if the consequences of poor quality are great (such as
loss of highly valuable work or severe damage to the organization's reputation), then a systematic and
rigorous assessment of the quality may be needed to assure that the risk of failure is acceptably low.
Adherence to the graded approach helps ensure that the assessment is cost effective and valuable to the
organization.
Resources and time needed for assessments of quality systems are scaled to the constraints of
the organization being assessed and may not encompass all of the quality processes of the assessed
organization. Resource and personnel limitations do not eliminate the obligation to comply with quality
specifications, but they may be factors in determining the rate that components of the quality system are
developed and implemented.
The graded approach takes the nature of organizational and/or program/project areas into
consideration in determining the scope and frequency of assessments. For example, a water quality
monitoring program with limited scope and complexity may conduct less frequent and less complex
assessments than a multi-program environmental performance partnership agreement (EnPPA) with a
State. Organizations that are responsible for highly visible enforcement activities may conduct more
extensive assessments than organizations that perform basic research. Assessments of small
organizations may be less extensive than assessments of large organizations.
1.6 INTENDED AUDIENCE AND PERIOD OF APPLICABILITY
This document is intended for all EPA and extramural organizations that have quality systems
based on EPA policies and specifications and that may periodically assess these quality systems for
compliance to the specifications. It is also intended for organizations whose quality systems are
assessed by EPA. In addition, this guidance may be used by other organizations that assess quality
systems applied to specific environmental programs.
As described in the Manual, this document will be valid for five years from the official date of its
publication. After five years, this document will either be reissued without change, revised, or
withdrawn from the EPA Quality System Series.
EPAQA/G-3 7 March 2003
-------
1.7 ADDITIONAL REFERENCES
Other documents are available to provide guidance for developing suitable and effective quality
systems for environmental programs. They provide guidance for QA and QC activities and for
documenting various components of a quality system, such as technical systems audits, standard
operating procedures (SOPs), and QA Project Plans. A list of these documents is provided in the
References and Supplemental Reading sections. Since they contain guidance for activities critical to
successful environmental data collection activities and operations, they serve as important resources for
planning and conducting assessments.
EPAQA/G-3 8 March 2003
-------
CHAPTER 2
MANAGING ASSESSMENTS
2.1 ASSESSMENT ROLES AND RESPONSIBILITIES
The authorizing entity for an assessment is whoever authorizes the assessment and has the
authority to do so. The authorizing entity for an assessment can be the individual ultimately responsible
for the quality system or overseeing the quality system that is being assessed. Examples of authorizing
entities are: unit managers for a specific program such as grant awards, the Regional QA Manager for
an internal assessment performed within a EPA Regional Office, or the Contracting Officer for an
external assessment of a contractor's quality system. The authorizing entity is given the opportunity to
approve the assessment plan, receives the assessment findings, may need to mediate any disputes, and
may monitor responses to and implementation of any corrective actions.
The assessee is the organization being assessed, and an assessor is a person who performs the
assessment. An assessor can be an individual either from part of the organization being assessed but
independent of the specific system being assessed (i.e., an internal assessment) or from an outside
organization (i.e., an external assessment). For external assessments, the assessors should be
independent of the assessed organization. For internal assessments, the assessors should not be directly
involved in performing or managing the environmental program. Table 1 gives examples of the roles
that various organizations may play in internal and external assessments.
Table 1. Examples of Assessment Roles
Assessments of
EPA organization
Assistance agreement
recipient/
contractor
Assessee
Program/Regional
Office/Laboratory
(external assessment)
Program/Regional
Office/Laboratory
(internal assessment)
State or tribal
environmental agency,
nonprofit organization,
or other assistance
agreement recipient/
contractor
(external assessment)
Authorizing Entity
Assistant
Administrator, Office
of Environmental
Information (OEI)
Assistant/Regional
Administrator
EPA Program/
Regional Office,
laboratory, or division
director
Assessors
OEI Quality Staff and
technical experts as
needed
Program/Regional
Office staff and
technical experts as
needed
EPA program office,
laboratory, or
Region/division QA
staff and technical
experts as needed
EPA QA/G-3
March 2003
-------
2.2 ASSESSMENT SYSTEMS
Organizations that conduct multiple assessments may establish a system to ensure that
assessments are performed consistently and according to current quality specifications (see
Worthington, 1998). The assessment system should focus on planning and establishing priorities for
assessments, assessment frequency, scheduling, conducting assessments, procedures and formats for
assessment reports, and assessor qualifications and training. SOPs should be developed that describe
the assessment procedures in sufficient detail to encourage consistency in how assessments are
performed.
Effective assessment systems answer four key questions for assessment system managers:
1. Am I doing the right job? (Do I select those assessments that will make a significant
contribution to the overall quality system?)
2. Am I doing the job right? (Does the assessment system use its personnel and resources
efficiently?)
3. Am I getting the desired results? (Do the assessments have a beneficial effect on the
assessed quality systems?)
4. Does my organization consistently do high-quality work? (Is care taken in the selection,
planning, performance, reporting, and follow-up of assessments? Are assessment
findings given a final quality check before they are sent out?)
A graded approach should be factored into designing an appropriate assessment system. Some
organizations may not have adequate staffing to implement all possible activities of an assessment
system, but all organizations are encouraged to consider these topics and to implement them within their
assessment system when possible and appropriate.
Managers of an assessment system provide administrative support to the assessors, have
practical knowledge of assessment procedures and practices, and should:
• be independent of direct responsibility for implementing the projects being assessed
• clarify the authority to assess within the organization, if necessary
• establish awareness of the assessment system by potential users and potential assessees
emphasize the benefits of a well-established and functional quality system
establish priorities for quality systems to be assessed
ensure that adequate resources are available for the assessment system
establish an assessment QC system
• evaluate assessor training needs regularly and provide appropriate training opportunities
• ensure that procedures are in place for planning, scheduling, conducting, reporting, and
following up on assessments, and that assessments are consistently documented
• select assessment team leaders, approve assessment teams, and ensure that they
receive administrative support
• review assessment findings
resolve any disputes between assessors and assessees concerning assessment findings
transmit assessment findings to authorizing entities
• brief senior management on the status of the assessment system
EPAQA/G-3 10 March 2003
-------
ensure that the experience gained by assessors improves the assessment system and the
quality system.
Management of the assessment system may be a shared responsibility performed by a small
staff instead of one individual. Management of the assessment system is typically not a staff member's
sole responsibility. For instance, the manager for assessments that are conducted by an EPA Regional
Office may be the Regional QA Manager.
Assessing organizations should have an appropriate QC system in place and participate in an
external QC review program (GAO, 1994). This system provides reasonable assurance that the
assessment system has established, and is following, adequate assessment procedures and that it has
adopted, and is following, applicable assessment standards. The nature and extent of this system in any
organization follow the graded approach according to the following factors: the organization's size and
structure; the degree of operating autonomy allowed its assessors; the nature of its work; and
appropriate cost-benefit considerations.
The assessment system will generally include a QC system to ensure the quality of
assessments. An assessment QC system helps to ensure that assessments are effective and that
assessments of similar organizations under similar conditions by different assessment teams arrive at
similar findings. An assessment organization's reputation for performing high-quality assessments
increases the impact of their findings and the likelihood that corrective actions will be implemented.
The assessment system's managers should create the expectation for high-quality assessments,
establish the policies and procedures that will produce high-quality assessments, and determine whether
the assessment QC system has improved the quality system. They should develop procedures and
criteria to compare assessor performance to achieve consistency among assessors to the extent
possible and regularly evaluate assessor performance. Such procedures can include assessor training
workshops, reviews of assessment reports, performance appraisals, and rotation of assessors among
different assessment teams.
Assessment system managers should not only review findings of individual assessments, but also
review the findings in a holistic way. This review process feeds back into planning with an emphasis on
improving both the assessment system and the quality system. For most organizations, assessments are
not just one-time events but are done on a recurring basis with assessments conducted on different
groups and at different locations within the organization. The review may also identify relevant and
emerging quality issues in assessments, perhaps coming from a synthesis of findings from assessments of
multiple organizations. For instance, such a review may reveal areas in a quality system that are prone
to problems or areas that need more controls or more training.
The graded approach should also be factored into the assessment QC system. In small
organizations, a fully developed QC system for the assessment process may not be possible because of
limited staffing. These organizations still can incorporate aspects of a mature assessment QC system
into their own system, within their constraints, to ensure the quality of their assessments.
EPAQA/G-3 11 March 2003
-------
2.3 DECISION TO CONDUCT THE ASSESSMENT
Planning for an assessment will generally begin in response to a direct request from the
authorizing entity or according to a schedule that has been approved previously by the authorizing
entity. The decision process typically includes the identification of some or all of the following items:
• the organization to be assessed
• the authority to conduct the assessment
the criteria for the assessment
the scope of the assessment
the resources available for the assessment
the size of the assessment team
• an approximate date for the assessment
• the assessor qualifications needed to conduct the assessment
• availability of qualified assessors to conduct the assessment
• selection of the assessment team leader
selection of assessment team members.
The assessment team leader addresses any of the above items that have not been decided by
the authorizing entity or the assessment system managers.
2.4 CRITERIA FOR THE ASSESSMENT
For the assessment team to assess the adequacy and effectiveness of a quality system in an
objective manner, the quality system's characteristics should be compared to objective and written
reference standards rather than to the subjective, unwritten expectations of the assessors or other
individuals. These assessment criteria would generally include: (1) the external policies, procedures,
and specifications that are applicable to the assessee and (2) the assessee's internal policies,
procedures, specifications, and quality system planning documents. Specific policies and requirements
relevant to the quality systems of EPA organizations and of extramural organizations performing work
funded by EPA through extramural agreements, enforcement agreements, decrees, or orders may
include the following:
Order 5360.1 (EPA, 2000a)
EPA's Quality Manual (EPA, 2000b)
• EPA specifications for QMPs
ANSI/ASQC E4
• the assessee's QMP
• the assessee's reports [e.g., quarterly progress reports or QA Annual Report and
Work Plan (QAARWP)]
• QA and QC specifications in regulations.
It is important that the authorizing entity, the assessment team, and the assessee all agree on the
assessment criteria prior to the assessment. If the parties involved in the assessment do not have a
common understanding of the criteria beforehand, questions concerning the basis for the subsequent
EPAQA/G-3 12 March 2003
-------
assessment findings may arise. The credibility of the assessment can be diminished if team members
apply inconsistent or subjective assessment criteria.
2.5 SCOPE OF THE ASSESSMENT
The scope of the assessment may be set by the authorizing entity or it may be systematically
developed by the assessment team. The scope can define the limits of the time period and subject
matter or organizational "boundaries," and can be affected by assessor time and resource constraints.
It may also include more specific items, such as the job positions of the people to be interviewed and
what parts of the quality system to examine. Selection of the items may be based on their importance
to the overall quality system or on concern that there might be a problem. Issues for consideration in
the assessment may derive from any part of the quality system (e.g., policy, processes or procedures,
products, or resources). Issues may also be derived from the findings of previous assessments.
Section 3.5 contains more information about issue identification.
The scope for assessing an implemented quality system will generally differ from that for
assessing a developing quality system. For example, a developing quality system might not have an
approved QMP in place to serve as a basis for the assessment and is less likely to have formal QA
tracking systems. For an assessment of a developing quality system, the scope may include may
include providing information helpful in development of specific parts of the quality system. Assessors
should maintain their independence, but may be able to provide useful documents, for instance, existing
or draft documents and procedures that might be helpful to fill gaps in the current quality system.
The scope of the assessment may include the provision for the assessment team to make
recommendations for corrective actions based on their findings. Recommendations may be requested
by the authorizing entity or the assessee. In response to the team's recommendations, the assessee may
propose alternative corrective actions that address the team's findings. The assessee retains the
responsibility to implement corrective actions. All involved organizations (i.e., the assessors, the
assessee, and the authorizing entity) should understand prior to the start of the assessment whether the
assessors will make recommendations. If necessary, the dispute resolution processes discussed in the
assessing organization's QMP should be followed, unless there is an overriding legal constraint.
The graded approach should be applied to recommendations. For assessments of developing
quality systems, the assessment team may be more knowledgeable regarding quality system and
corrective actions than the assessee. The assessee may welcome and may benefit from any technical
assistance that the assessment team can provide. Given that the ultimate goal of the assessment is to
improve the developing quality system, the value of the technical assistance may be more important than
the value of the findings of the assessment.
The scope can be limited by assessment resource constraints, which often preclude assessing
the whole quality system, so specific items are selected for inclusion in the assessment. The use of the
graded approach helps to ensure that assessment resources are used effectively and efficiently where
they are needed most. Because an assessment is closely linked to the assessee's QMP, the scope of
an assessment can be estimated by the complexity and detail of the quality system described in the plan.
EPAQA/G-3 13 March 2003
-------
The scope can also be limited by what can be accomplished on-site. Planning and scheduling
interviews and document reviews should consider both what can realistically be covered within the
allocated resources and what needs to be covered to adequately characterize the assessed system.
After the duration of the on-site portion of the assessment has been decided, the number of interviews
to be conducted during the assessment can be estimated. The time necessary for the opening and
closing meetings, document reviews, and breaks should be taken into account. Perhaps only six or
seven 1-hour interviews can be conducted per day. As is discussed in Section 2.6.1, each interview
should be conducted by one or two assessors.
The authorizing entity approves the assessment plan and by doing so approves the scope of the
assessment. However, the assessment team leader is usually enabled to modify the scope during the
assessment if any relevant, but unforeseen, quality issues are encountered during the assessment. For
instance, it may be necessary to interview staff members who were not identified in the assessment plan.
Section 3.8 contains more information about the assessment plan.
2.6 THE ASSESSMENT TEAM
2.6.1 Assessment Team Selection
The scope of the assessment generally determines the size and composition of the assessment
team. The scope of the assessment should be determined before the assessment team members are
selected. The assessors collectively should have subject matter knowledge in the areas of concern, as
well as assessment knowledge and experience. They should be free of any conflicts of interest.
Training in interviewing skills is usually a prerequisite for performing the assessment. Section 2.6.3
describes the assessment team qualifications in greater detail. Interviewing skills are addressed in
Appendix B.
There are good reasons for an assessment team to consist of two or more members (a team
leader and at least one additional assessor) (see Adams, 2000). In some cases, the assessment team
may need to include additional assessors as well as technical experts. For example, an internal
assessment of an EPA Regional Office may be performed by an assessment team composed of a
leader, two assessors, and a technical consultant with expertise in Regional Office QA programs.
During interviews, two assessors can each document an interviewee's responses. This helps to
ensure that the statements by the interviewees are recorded accurately. If there is any confusion about
what was said in an interview, the two interviewers can discuss the response and come to agreement on
what the interviewee said. One interviewer may recognize an important piece of information that the
other interviewer may have overlooked. This two-assessor approach can allow for "tag team"
questioning. That is, while one assessor asks a question and records the interviewee's response,
another assessor can be preparing to ask the next question. Other advantages of having more that one
person perform an assessment include: complementary expertise and work experience, the ability to
work simultaneously with different interviewees, and cost savings in both the planning and
implementation phases of the assessment.
Assessors from other organizations may be a possible resource. Their management may need
to grant permission for their participation. For assessments of EPA organizations, assessors are usually
EPAQA/G-3 14 March 2003
-------
QA professionals for a different Regional Office, Program Office, or National Laboratory. For
assessments of State agencies, a QA manager from another State in the Region could participate in the
assessment.
2.6.2 Assessment Team Leader Responsibilities
Once the need, authority, and funding for an assessment have been established, an assessment
team leader and other assessment team members may be selected. The assessment team leader should
be responsible for all phases of the assessment. The assessment team leader should have experience
and skill in organizing group efforts and in interpersonal relationships and should have the authority to
make decisions during the assessment and while presenting any assessment findings. The assessment
team leader also generally:
• may assist in selecting other assessment team members
• prepares the assessment plan and submits it for review and approval
• represents the assessment team to the assessee' s management
manages the assessment team during the assessment
submits the assessment report
organizes the response to comments.
2.6.3 Assessor Responsibilities and Qualifications
According to ANSI/ASQC E4, personnel conducting assessments of quality systems should
have the authority, access, and independence to:
identify and report problems that affect quality
identify and cite noteworthy practices
if requested, propose recommendations for correcting problems that affect quality
• independently confirm implementation and effectiveness of corrective actions
• if requested, monitor the work and report to management until the identified problems
have been corrected
• provide documented assurance to management that further work performed by the
organization is monitored until identified problems are corrected.
To establish their competency and credibility, team members should:
be free from personal and external barriers to independence, organizationally
independent, and able to maintain an independent attitude and appearance
• possess integrity and report only what is observed
• collectively possess adequate assessment proficiency and appropriate technical
background
be qualified to perform their duties by virtue of education, training, and/or experience
understand assessment techniques and quality system concepts and principles
have experience appropriate for their duties in the team (leading, for example)
understand their roles and responsibilities in the assessment process and be responsive
to the assessment team leader's directions
EPAQA/G-3 15 March 2003
-------
be familiar with the assessee' s organization and with applicable regulations
have good information-gathering and communication skills, i.e.,be able to assimilate
information, formulate pertinent questions, present questions clearly during interviews,
listen carefully to the information being provided, and verify the information from
documentation
• be even-tempered and keep potentially confrontational circumstances under control
• be organized and able to prepare assessment reports promptly.
Corrective actions are more likely to be initiated in response to assessment findings if the assessment
team is perceived to be competent and credible.
According to a recent General Account Office (GAO) report, assessors have a responsibility to
maintain independence, so that their opinions, conclusions, judgments, and recommendations will be
impartial and viewed as impartial by knowledgeable third parties (GAO, 2002). The GAO report
describes three general classes of potential problems with independence as personal, external, and
organizational. The GAO report concludes that assessors cannot be independent if they perform
management functions or make management decisions for the assessed organization. The assessing
organization should have procedures to help determine if assessors have any personal impairments to
independence. Technical experts that are members of the assessment team should also maintain
independence.
2.7 ASSESSMENT RESOURCES
Knowledge of the resources needed for assessments helps to ensure that adequate resources
can be made available. The budget for an assessment depends on the scope, objectives, duration, and
complexity of the assessment. Resources are affected by the number of assessors needed, and their
associated labor, travel, and lodging costs. Assessors need time to prepare for the assessment,
conduct the assessment, generate the report, and if specified, verify corrective actions. Off-site
activities, such as preparation and reporting, may take more time than the on-site portion of the
assessment.
EPAQA/G-3 16 March 2003
-------
CHAPTER 3
PREPARING FOR THE ASSESSMENT
Planning is the most crucial part of the assessment process and a systematic approach is
recommended. Chapter 2 identified initial planning activities: deciding to conduct the assessment
(Section 2.3); identifying the criteria for the assessment (Section 2.4); determining the scope of the
assessment (Section 2.5); selecting an assessment team to conduct the effort (Section 2.6); and
allocating resources for the assessment (Section 2.7). Once these activities have been performed, the
planning process can proceed to identify:
specific information that is needed from the assessee to identify assessment issues
• specific issues about the quality system to be checked during the assessment
• a point of contact for the assessee and establish communication between the authorizing
entity, the assessment team, and the assessed organization
• the sources, type, and quantity of information to be collected
how collected information will be evaluated to determine if the quality system meets the
assessment criteria.
One product of this process should be a written plan that summarizes what will be done in the
assessment. It should be prepared by the assessment team and approved by the authorizing entity
before being sent to the assessee prior to the assessment. Another product should be a written
assessment checklist that is used by the assessment team to organize the interviews and to document
the information that they will collect. Logistical arrangements for the assessment should be made as
part of the planning process.
The size, complexity, and development status of a quality system do not alter the need for
objective and systematic planning for any assessment of that quality system. The graded approach
should be used during this planning to establish an assessment scope and assessment issues that are
appropriate for the size, complexity, and development status of the quality system. The procedures that
would be used for planning a complex, criteria driven assessment of a fully implemented quality system
are the same for planning an assessment of a developing quality system that needs technical assistance.
In this chapter, planning activities are presented in a particular order. This order does not mean
that the activities have to be performed in this order. Many of the activities can occur concurrently or
iteratively, and the order of the activities will vary for different assessments and for different assessing
organizations. For example, selecting the assessment team leader and assessment team may be the first
step on some assessments, if the organization knows which staff members are the best matched
technically for a particular assessment. In other instances, an assessment team leader may be selected
who will then begin the initial planning and scoping, which will be followed by selection of other
members of the assessment team.
Figure 2 illustrates an approach for planning activities for an assessment. This figure is not to be
considered a chronological flowchart for assessment planning. The four columns in the figure
correspond to four general types of activities that are associated with assessment planning. The
activities in this systematic planning process are described in more detail in the rest of this chapter. The
EPAQA/G-3 17 March 2003
-------
lines between the boxes show some of the logical connections between the activities, rather than a strict
chronological order.
Planning Activities Associated with:
Authorizing
Entity
Assessment Team
Development
Assessment Plan
Development
Assessee
Interactions
Figure 2. Systematic Planning Activities for an Assessment
EPA QA/G-3
18
March 2003
-------
An organization may choose a different systematic planning process or may limit some of these
activities. Application of the graded approach may reduce the importance of some activities. For
example, a quality system that is still being developed may not yet have many quality documents to be
reviewed and an assessment focusing on this activity may not be a productive use of resources.
However, systematic planning should be used for any assessment, regardless of the size or complexity
of the quality system being assessed. A written plan is useful for any assessment as a way to document
the assessment planning, including determining the criteria for and the scope of the assessment.
3.1 DOCUMENTATION AND TRACKING
At the beginning of the planning phase, it is helpful to establish an assessment file, which helps to
track the paperwork from initiation of the assessment through completion. The file may contain all
materials collected before, during, and after the assessment including:
• planning documents, such as the assessment plan and the agenda
• all relevant correspondence, such as notification letters
working papers, such as assessment checklists that record the observations from
interviews and document review
all assessment reports
any other documents collected or arising from the assessment such as corrective action
reports.
The assessment file serves to document the course of the assessment and its outcome. As the
file is prepared, note that it may be possible for the public to obtain assessment files and working
papers through the Freedom of Information Act. Electronic tracking of assessments may be possible in
some organizations. Generally, close-out of the assessment should be tracked or documented with a
formal close-out memorandum or some other type of record.
3.2 ASSESSMENT TEAM PREPARATION
Before going to the site, the assessment team should review information about the quality
system, plan the assessment, divide up responsibility for interviews and document reviews, work out
scheduling and logistical issues, and understand the procedures for note taking, reporting, and follow
up. The roles and responsibilities of individual team members should be discussed. The team
members' expectations for the assessment should be discussed and reconciled as needed. Assessor
responsibilities and qualifications are described in Section 2.6.3.
3.3 INITIAL CONTACT WITH THE ASSESSEE
During initial contact with the assessee, the assessment team leader should relay the authorizing
entity's decision to conduct an assessment. The leader may make contact by telephone, e-mail, or
letter with the assessee's QA manager. The authorizing entity may have previously informed the
assessee of this decision. The leader and the QA manager should discuss possible dates for the
assessment, the assessment criteria, the scope of the assessment, requests for supporting documents,
and potential interviewees who are representative of the of the program areas to be assessed.
Application of the graded approach may reduce the importance of some activities.
EPAQA/G-3 19 March 2003
-------
The formality of the initial contact with the assessee and subsequent contacts will be determined
largely by the organizational relationship between the assessors and the assessee. External assessments
tend toward more formality than internal assessments. After an initial verbal contact, it may be
appropriate for the assessment team leader or the authorizing entity to send a written notification of the
upcoming assessment to the assessee's senior management or to the assessee's QA manager, as
appropriate. Regardless of how the initial contact is made, the assessee's senior management should
be aware that an assessment will be occurring.
The initial contact is as important as any other contacts made during the planning and should
include discussion of the authority, scope, and purpose of the assessment. An open, objective,
cooperative, and professional tone by the assessment team establishes a positive and less stressful tone
for interactions during the entire assessment. This begins with the initial contact and continues
throughout the assessment. If the senior management and the QA manager understand that the
upcoming assessment offers an opportunity to improve their quality system, their attitude will be
communicated to the rest of the organization, and the assessment should proceed more smoothly. A
positive purpose should encourage the organization to implement any corrective actions that are needed
to respond to assessment findings.
3.4 INFORMATION REVIEW
The purpose of reviewing information about the assessee's quality system is to establish the
knowledge base for the assessment. It is essential that the assessment team understands what is
already documented about the assessee's quality system and its environmental programs in order to
formulate relevant questions for the interviews and to identify pertinent case studies, documents, or
reports to be examined.
Helpful information includes the specifications for the assessee's quality system and supporting
documentation, such as the QMP; applicable regulations for environmental programs; reports of
previous assessments of this organization; the QA Annual Report and Work Plan (an EPA-specific
document); and fiscal reports such as Government Performance and Results Act reports (for Federal
organizations). If these documents are not already on hand, the assessment team leader should request
them during the initial contact with the assessee. Organizations with developing quality systems will
generally have less documentation available for review than those with fully implemented quality
systems.
Reviewing these documents will allow the assessment team to consider some or all of the
following items, as appropriate for the scope of the assessment:
• the mission and quality policy of the organization
• the specifications for the quality system that are in the assessment criteria
• the specific roles, authorities, and responsibilities of management and staff with respect
to QA and QC activities
• the means by which effective communication within the organization is assured
the processes used to plan, implement, document, and assess the work performed
• the process by which measures of effectiveness of QA and QC activities will be
established and how frequently effectiveness will be measured
EPAQA/G-3 20 March 2003
-------
• the level of improvement based on lessons learned from previous experience
fiscal reports (for Federal organizations).
The assessee's fiscal reports can provide a window into the quality system because they show
how money was budgeted and spent, which may be an indication of which issues are considered most
important by the organization. If budgetary information is included in an assessment at all, it will be
fairly broad information about environmental data collection activities, similar to the information that
might be included in the QA Annual Report and Work Plan. Some organizations may be
uncomfortable with assessors reviewing budgetary information and, in those cases, the assessment team
should discuss the reason for including such information and may allow the assessee to collect the
needed information in a way that obscures sensitive information, such as salaries.
3.5 ISSUE SELECTION
If the authorizing entity does not specify the assessment issues, the assessment team should
select and document them in the assessment plan. Time and resources can limit an assessment, making
it impossible to evaluate and characterize all aspects of a quality system. If such limitations do exist,
then the assessment team has the opportunity during planning to select the specific quality system
components and associated issues that will be investigated. Priorities can be established using input
from three sources:
• the quality system and associated (e.g., contract) specifications
• documentation about the assessee
• possible knowledge of or experience with similar organizations.
Some Quality System specifications may become assessment issues because:
they have a significant effect on the quality of the environmental data being collected to
support decision making
• they are not easy to implement or fulfill
• they are vague and contradictory or onerous and burdensome
• they are new or have been revised since the last update of the QMP.
Documentation of quality system processes and their effect on end product development may
lead to the selection of assessment issues if the products are of special importance. For example, they
may be used directly for making rules, regulations, or policy or have significant national or
Congressional visibility. Although the products themselves are not assessment issues, the effect of the
processes used to develop them are important because they demonstrate the ability of the quality
system to support rule-making and regulation and policy development.
The reports of previous assessments of the assessee may indicate quality system components
that have had problems in the past and for which corrective action may have been necessary. The
assessment team may decide to determine whether the corrective actions were implemented and
effective. Also, the reports of assessments of other organizations may point to quality system
components with common weaknesses, which may also be present in the organization being assessed.
The assessment team may look for similar weaknesses in the organization being assessed.
EPAQA/G-3 21 March 2003
-------
One technique for selecting issues is to look at the completeness and clarity of written
descriptions of the organization's quality system. If the quality system documentation presents only a
generic description of the quality system, it may be necessary for the assessors to obtain more
information about the quality system as implemented in this organization. If the documentation does not
describe all of the components of the quality system and the specifications for them, the missing
components or specifications may become issues. Although a thorough and lucid description of a
quality system component does not guarantee that this component is being implemented or that it is
effective as implemented, the lack of such a description may point to an area that merits observation
during the assessment.
Assessment issues may also be selected by studying information to trace or reconstruct the
quality system processes affecting a program or activity from its antecedents (e.g., a regulation) to its
end products. A program's quality system can also be traced through the personnel who plan,
implement, and assess it. If the documentation does not reveal the connection between antecedents
and products, the personnel pathway, or the quality-related steps, then the assessment team may wish
to allow time to investigate them during the assessment.
A final consideration is whether including an issue in the assessment can be of benefit to the
organization's quality system. Some issues may highlight problems that are beyond the control of the
assessee, such as inadequate funding for travel or inadequate staffing in some aspect of the program.
The assessors can be realistic about what will be accomplished by including an issue in the assessment.
For example, issues may be included in assessments at times to bring them to the attention of
management above the staff interviewed during the assessment. The graded approach may influence
how such issues are addressed during an assessment.
3.6 IDENTIFY INTERVIEWEES AND DOCUMENTS
After the major assessment issues have been selected, the next step should be to choose an
information collection tool that is appropriate to investigate the assessment issues. The underlying
concept is that the tool should enable the assessment team to understand a quality system and quickly
integrate the collected information. The tool should allow the assessment team to document objective
evidence or observations about the quality system. Even when assessing a developing quality system,
the information collection tool should still be systematic and geared towards collecting objective
evidence.
Generally, face-to-face interviews and document reviews are the preferred tools for collecting
information in assessments because interaction with the interviewee provides the assessment team with
direct information about the quality system. Supporting documentation can be consulted and questions
can be explained and clarified as necessary.
There is a benefit-risk consideration associated with the use of interviews. People can be a
valuable source of information in the right circumstances. However, the human mind is a very complex
and vulnerable observation instrument. If the assessment team does not ask the right people the right
questions, they may not get appropriate answers.
EPAQA/G-3 22 March 2003
-------
The next task should be to identify the
type and number of representative individuals to
be interviewed and the type and number of
documents to be reviewed that will enable the
team to gather sufficient information to address
the issues. Before scheduling interviews and
document reviews, the assessors can consider if
a specific job or document gives them objective
evidence for the issues. They can then consider
how many interviews or document reviews are
needed, relative to the size of the organization,
to make a representative finding. The individuals
and documents may be involved with program-
level or project-level quality activities. Examples
of job category sources for interviews are listed
in Figure 3.
It is not necessary at this point to name
specific individuals to be interviewed; identifying
job titles or job functions may be all the
assessment team can accomplish given the
information on hand. The goal at this point is to
be specific enough in identifying the interviewees
that these individuals can provide the information
Examples of job category sources for
interviews during an assessment:
senior managers (e.g., division directors,
office directors)
• middle managers (e.g., branch chiefs,
section chiefs)
• project managers (e.g., project officers,
principal investigators)
• quality assurance managers (program-level,
branch-level and project-level)
data analysts (e.g., statisticians and
modelers)
• data handling specialists
• laboratory managers/staff
• field support staff/samplers.
Figure 3. Example Job Categories for
Interviews
so that the assessment team has reasonable assurance
that is needed to address the assessment issues.
Some considerations for selecting interviewees who are appropriate for the issues may include:
(1) their availability; (2) their experience; (3) their knowledge of the issues; (4) how long the individuals
have held their positions; and (5) the extent that these individuals represent the entire pool of those in
similar positions.
Individuals to be interviewed should be selected to get adequate coverage of issues, programs,
and job types within the allocated assessment resources. The assessee may recommend specific
individuals to be interviewed. This practice is generally acceptable if the individuals' characteristics
such as on-the-job experience meet the assessment needs. If all of the interviewees are selected by the
assessee and the assessment team is unsure that they are representative of the program, then the final
assessment report may need to include qualifying text such as "If the interviewees are representative of
your program, then . . . ."
As is the case with identifying interviewees, the assessment team should specify the documents
to be reviewed in sufficient detail to ensure that the documents are accessible and the assessment issues
will be addressed effectively. Document selection criteria may include: (1) being representative of the
document types most frequently prepared by the organization; (2) being representative of the work
performed by the organization; and (3) having importance relative to the organization's mission. If an
organization has changed its quality policy or procedures, select documents that reflect the changes
being assessed. When selecting particular projects for document review, make sure these projects
cover a time period and implementation stage that are appropriate for the issue being addressed. For
EPA QA/G-3
23
March 2003
-------
Example documents and their specific features
that may be reviewed:
• QMPs
signature and date
QA Project Plans
review and approval process
signature (QA manager or designee), date
compared to project start date
data quality objectives/systematic planning
process
selected elements relevant to assessment:
(a) training/expertise for field personnel
(b) oversight of field activities
• QA Review Forms
signature (QA manager or designee), date
project title, number
Figure 4. Example Documents and Features To
Review
example, projects just getting underway
would not be appropriate for a review of
data quality assessment procedures.
Figure 4 contains examples of
documents and specific features of the
documents that may be reviewed to prepare
for and during an assessment. Other
documents, such as financial assistance
agreement decision packages and contract
specifications, also may be relevant to the
assessment.
EPA QA Annual Reports and
Work Plans summarize resources available
for QA in EPA programs. As part of the
assessment, these documents can be
compared to the QMP or verified on-site to
ensure that the roles and responsibilities are
covered as described in the QMP. For
example, the number of full-time equivalents
(FTEs) designated for QA staff could be
verified against possible vacancies or
assignments to non-QA activities when on-site or checked against the work reported on QA project
plan reviews, internal/external assessments, and training.
To assess the use of resources for oversight, lists of both external and internal assessments in
the previous year's work plan could be checked to see if they agree with lists of completed reports and
the specifications of the QMP. These documents give indications about the adequacy of resources and
the commitment of the organization (for example, if less work is performed than was planned). They
would be useful in targeting issues (for example, if no internal assessments are reported, why?).
3.7 ALTERNATIVES TO ON-SITE INTERVIEWS
The decision to use interviews or other information collection tools should involve considering
their comparative advantages and disadvantages. Examples of other information collection tools are
videoconferencing, telephone interviews, and return mail questionnaires. More information about these
tools can be found in the literature on survey research methodology (e.g., GAO, 1991 and 1993; De
Leeuw, 1992).
Each tool has its own blend of strengths and weaknesses. Because of their flexibility and
potential, face-to-face interviews have been considered superior to telephone interviews and mail
surveys. Information collected in face-to-face interviews has often been considered to be less suspect
than information obtained by other tools such as telephone interviews. However, the other tools do not
incur the travel costs that are associated with face-to-face interviews.
EPA QA/G-3
24
March 2003
-------
3.8 PREPARATION OF THE ASSESSMENT PLAN
The assessment plan is a short document prepared by the assessment team under the direction
of the assessment team leader. It is a concise summary of the assessment and the manner in which the
assessment will be conducted. It should give adequate information to the assessee about what activities
are expected to occur during the assessment and a schedule for these activities. An example outline of
an assessment plan appears in Figure 5.
The assessment plan should include the authority and criteria for the assessment, the purpose
and scope of the assessment, the assessment issues, and the organizations that will be visited during the
assessment. The plan should also include details, such as a schedule of assessment activities, specific
personnel (or job positions) to be interviewed, and specific files and documentation that will be
reviewed during the assessment. The assessment plan should state clearly what will and will not be
done regarding confidentiality and the dissemination of the assessment findings. The assessment
checklist can be appended to the assessment plan. The checklist contains the specific technical
questions to be asked of specific interviewees and the specific documents to be reviewed, if
appropriate.
An informal discussion is generally held before the assessment with the assessee about the
planned assessment to negotiate schedules, identify needed documents and records, and confirm the
availability of interviewees and meeting space. Planning and scheduling interviews and document
reviews should be considered in view of what can realistically be covered within the allotted time and
what should be covered to adequately characterize the assessed system.
The assessment plan should specify whether the assessment team will present recommended
corrective actions as part of the assessment report or whether the assessee management will develop
these corrective actions based on the assessment findings. If the team makes recommendations, the
assessee may propose alternative corrective actions that address the team's findings and has the
responsibility to implement corrective actions. All involved organizations (i.e., the assessment team, the
assessee, and the authorizing entity) should understand prior to the start of the assessment whether the
team will make recommendations.
3.9 REVIEW AND APPROVAL OF THE ASSESSMENT PLAN
Once the assessment plan has been completed by the assessment team, it is usually submitted to
the authorizing entity for concurrence and approval unless it is routine. Transmitting the plan well before
the assessment date allows:
• the authorizing authority to raise questions about the plan or discuss the rationale of the
proposed approach
• the authorizing entity to be informed explicitly of any nonroutine aspects of the
assessment
the assessment team to revise the plan and to resubmit it for approval if sufficient
concerns or issues are raised by the authorizing authority, which is unlikely to occur if
the assessment team has been thorough in its planning
EPAQA/G-3 25 March 2003
-------
Assesses: Organization:
Location:
Senior Official
QA Manager:
Authorizing Entity:
Review and Concurrence by:
Assessment Team: Leader:
Assessor:
Assessor:
Anticipated Dates of Assessment:
Authority to Conduct Assessment:
Criteria for Assessment:
Purpose and Scope of Assessment:
Issues Selected:
Personnel to Be Interviewed:
Documents to Be Reviewed:
Anticipated Date for Receipt:
Anticipated Opening Meeting:
Opening Meeting Participants:
Anticipated Assessment Schedule:
Anticipated Closing Meeting
Closing Meeting Participants:
Anticipated Reporting Schedule:
Report Routing Pathway:
Confidentiality of Findings Report:
Dissemination of Findings Report:
Assessment Plan
EPA Region 12, Division of Solid Waste (DSW)
Juneau, Alaska
Jim Schnee, Director, Division of Solid Waste
Mary Eulen, Division QA Manager
William Shipley, Regional Administrator (RA)
Pat Pack, Deputy Regional Administrator (DRA)
Susan Davis, Regional QA Manager
Emmanuel Kealeboga, Division of Oil and Gas Remediation
Margaret O'Connor, Division of Arctic Air
January 2-4, 2002
EPA Order 5360.1 A2 (May 2000)
QMP, applicable assistance agreements, contract regulations
Implementation of DSW QMP in Juneau branches
QA project plan review and approval, data quality assessment process, data
quality objective process, training, and record keeping
Branch QA Coordinators, 4 project officers per branch (2 with data
collection/analysis completed, all in branch at least 1 year), DSW QAM, DSW
supervisor, DSW training coordinator, and DSW statistician
Interviewed project officer files including all QA documentation (e.g., QA project
plans, SOPs, oversight records, data analysis records, project reports), QAM
files including QA project plan reviews, project implementation and report
reviews, and training records
December 15,2002
January 2,2002, 8:00 a.m.
DSW managers
9 a.m. to 4 p.m. each day, one branch per day
January 4, 2002, 4:30 p.m.
DSW managers
February 2, 2002
RA, DRA, Jim Schnee, Mary Eulen, DSW managers
External dissemination needs assessee approval
Internal only
Figure 5. Example Contents of a Plan for Assessing a Quality System
EPA QA/G-3
26
March 2003
-------
the assessment team to resolve all concerns or issues before proceeding any further.
The authorizing entity should approve the assessment plan before the assessment proceeds.
The concurrence of the authorizing entity:
• affirms the authority, credibility, and scope of the assessment with the assessee and with
the persons who will receive the final assessment report
• encourages authorizing entity "buy-in" and engenders a sense of ownership of the
process
assures the authorizing entity that the assessment will accomplish the objectives
encourages support from the authorizing entity for any disputed findings and for
implementation of recommended corrective actions.
3.10 CONFIDENTIALITY
The confidentiality and dissemination of the assessment findings and other assessment
documents should be addressed during planning for the assessment and described in the assessment
plan. Disputes over confidentiality issues should be resolved with all of the involved organizations prior
to the start of the assessment. If necessary, the dispute resolution processes discussed in the assessing
organization's QMP can be followed, unless there is an overriding legal reason. Generally, assessment
findings should be released only to the involved parties.
Any information that the assessee claims as confidential business information (CBI) should be
treated as described in the relevant regulations [for instance, Title 40, CFR, Part 2, Subpart B;
Resource Conservation and Recovery Act (42 USC 6901 et seq); Clean Air Act (42 USC 1857 et
seq); Federal Insecticide, Fungicide, and Rodenticide Act (7 USC 136 et seq); and 18 USC Section
1001]. Documents containing CBI should be handled in accordance with applicable regulations
covering the documents in question. Information of concern may include:
• proprietary technical information or trade secrets
• financial information
• personnel records.
Assessors may also have access to enforcement-sensitive information, which should be treated with
appropriate confidentiality. The Freedom of Information Act, in some cases, may be used to obtain
assessment findings and other assessment documents. Personnel records may include records of
training and proficiency demonstrations. Fiscal reports may be reviewed during an assessment and may
need special confidentiality approaches.
3.11 COORDINATION AND LOGISTICAL ARRANGEMENTS
Scheduling an assessment involves coordination between the assessment team and the
assessee. Both parties should understand each other's time constraints and develop a schedule that
does not put an undue burden on either party. The schedule may include the dates of the opening and
closing meetings, assessment activities, assessment report transmittal, and the assessee's response to the
report. The schedule should be included in the assessment plan.
EPAQA/G-3 27 March 2003
-------
Many assessments involve travel and thus need a fairly tight, workable schedule. Adherence to
a workable schedule means good coordination between the assessment team and the assessee. The
assessment team leader usually makes logistical arrangements, such as finalizing the assessment dates
with the assessee, and arranging for transportation and lodging. The assessment team typically should
ask the assessee to provide a meeting room on-site. Security clearances, special site passes, access to
the assessee's facility, and parking passes should be arranged in advance. Health and safety concerns
will be considerations if the assessment enters laboratory or mechanical areas. For assessments
involving travel, the assessment team leader should inform the assessee's QA manager of the team's
itinerary with a telephone number where the assessment team can be reached.
The QA manager usually arranges for a meeting space for interviews and document reviews,
ensures that requested documents will be available to the assessment team, arranges interviewee
participation and logistics for the assessment, and coordinates the on site activities with the leader.
Interviews and document reviews are best conducted in a quiet place, away from potential interruptions
in offices and laboratories.
3.12 FORMAL NOTIFICATION
After the assessment plan has been approved by the authorizing entity, it should be formally
transmitted to assessee management. If not done in previous communication (see Section 3.3), this
document should establish the authority for the assessment, identify the assessment team members and
their affiliations, define the assessment scope and criteria, and include a tentative schedule. An example
of a formal notification letter is provided in Figure 6.
A no-surprises approach of keeping the assessee informed may improve cooperation during the
assessment. Sending the plan to the assessee at least two weeks before the assessment should allow
the assessee some time to prepare for the assessment and helps to generate a positive attitude towards
the assessment. Because any necessary corrective actions will be implemented by the assessee's
management, management's involvement from the start of the assessment should be a priority. The
assessment team may also elect to send a copy of the assessment checklist to the assessee prior to the
assessment. After the assessee acknowledges the notification, the schedule for the interviews and
document reviews should be finalized and the assessment team's logistical arrangements should be
completed.
3.13 CHECKLISTS AND OTHER ASSESSMENT AIDS
The planning process usually includes the development of assessment checklists and other
written assessment aids, which incorporate all of the issues that were identified. They should be used
by the assessment team to organize the interviews and the document reviews and to record the
information collected.
These aids should be specific to the scope and issues of the assessment and individually tailored
for each assessment. They should be a systematic means to obtain and record objective evidence
about the quality system that is not, but could have been, documented by the assessee. They help to
ensure the objectivity, reliability, consistency, and completeness of the assessment. They can be used
to help the assessor track some basic questions:
EPAQA/G-3 28 March 2003
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
Region 12
Juneau, Alaska 99801
December 15, 2002
Julia Bennett, Commissioner
Alaska Department of Environmental Conservation
410 Willoughby Avenue, Suite 303
Juneau, AK 99801-1795
Dear Commissioner Bennett:
EPA Order 5360.1 A2 (2000), Policy and Program Requirements for the Mandatory Agency-wide Quality System,
specifies that all EPA-funded organizations collecting and using environmental data to develop and implement
adequate quality assurance (QA) and quality control (QC) practices to ensure that the data are of the type and quality
needed for EPA decisions. These practices are documented in Quality Management Plans (QMPs) that are reviewed
by Regional quality assurance staff and approved for implementation by the Regional Administrators.
One of the quality management responsibilities of the Region is to provide periodic oversight and assessment of
the implementation of the Quality System in Region 12. In compliance with this responsibility, the Region will
conduct an assessment of DEC's quality system to determine:
(1) compliance with the DEC QMP or, in the absence of this plan, compliance with EPA QA specifications for the
QA and QC practices in support of EPA-funded environmental data collection and use, and
(2) the suitability and effectiveness of the quality practices actually being implemented by DEC.
The assessment process will include interviews of DEC managers and staff and related document reviews
regarding QMP implementation. The criteria for the assessment are EPA QA specifications, DEC's QMP, referenced
procedures, and DEC's annual QA Report. The team plans to conduct the assessment during the week of January 27,
2003. Logistical details and the schedule for interviews and document reviews are under discussion with the DEC QA
Manager, Mark Zimmerman. The assessment plan will be sent to you at least two weeks before the assessment.
The assessment team will be composed of Susan Davis, RQAM, who will serve as team leader, and Michael
O'Brien of the Quality Assurance Staff. Marsha Brown of the Frozen Waste Division will provide expertise in frozen
waste programs. They plan to brief DEC management on the scope of the assessment during an opening meeting on
January 27, if that is convenient.
I intend for this assessment to be helpful to your organization so that we may leam from our experience and
improve the DEC's quality system. I look forward to the successful completion of this assessment.
Respectfully,
William Shipley
Regional Administrator
cc: Mark Zimmerman, DEC QA Manager
Figure 6. Example Formal Notification Letter
EPAQA/G-3 29 March 2003
-------
In what manner is the issue being addressed?
What implementation processes are in place ?
Is there evidence to support the assessee's statements?
Does it work? Is it a noteworthy practice, just okay, or a serious problem?
• How does what the assessee is actually doing on this issue compare to what the
assessee says is being done?
• Is enough of the assessee's staff doing this to allow something definitive to be said?
In interviews, assessment issues should be discussed with an interviewee. The interviewee's
responses can be recorded in a checklist tailored for that interview and in supplemental notes. The goal
of the interview is not to complete the checklist, but to obtain objective information that addresses the
issues. The questions generally are tied to the audit criteria to simplify report preparation and to
achieve the goals of the assessment. See Appendix C for example interview questions for developing
and mature quality systems and for different job classifications. An example assessment checklist
appears in Appendix D. When EPA is determining which questions to ask in an assessment of a non-
EPA organization, be aware that the Paperwork Reduction Act (44 U.S.C. 3501 et seq..) requires EPA
to obtain OMB approval before posing identical questions to ten or more persons in a 12-month
period.
The questions to be asked in interviews or investigated in document reviews should be
formulated to fill gaps in the previously collected information about the quality system and assessment
issues, and to verify this information. It may be helpful for the assessors to note previously studied
information that needs to be verified. Checklist questions generally have the following characteristics,
some of which are applicable only to interviews:
The questions are specific to the quality system being assessed.
They are relevant to the assessment being conducted and have a good probability of
yielding useful information.
They are relatively easy to answer and do not cause undue burden or discomfort to the
interviewee.
• They concern a single piece of information. (It is better to have more questions with a
narrow focus than fewer broad questions that may be difficult to answer succinctly.)
• They address objective, measurable characteristics of the quality system.
• They are clear and comprehensible to the intended interviewees.
• They have real answers, even if some answers may be "I don't know" or "I do not
have enough information to answer."
They do not lead the interviewee toward a particular answer by the use of biased
language.
Typically, open-ended questions are preferable to close-ended (i.e., yes/no) questions for
interviews because they allow the interviewee to explain the answer more completely. The questions
may be qualitative or quantitative as needed. They should address quality practices that are described
in the assessee's QMP or other quality documents or specifications. They should address specific,
observable activities that are to be performed, rather than the more general principles that may be hard
to define in practice. For example, if the assessee's quality documents state that records will be kept in
a central, locked file, "How are the quality records stored?" is a better question than "Are good
EPAQA/G-3 30 March 2003
-------
record-keeping procedures being followed?" Time spent in the planning phase developing appropriate
assessment questions can save time while on site.
The use of generic checklists for assessments should be discouraged. A "one size fits all"
checklist may overlook unique features of the specific quality system being assessed. Although a
checklist from one assessment may serve as the basis for developing a checklist for a subsequent
assessment, it is not appropriate to reuse unrevised checklists. General types of questions (e.g.,
responsibilities, training, and planning) may be similar among assessments, but the specific questions in a
checklist depend on the specific criteria for that assessment, which will vary among organizations. The
process of developing a checklist that is tailored to a specific quality system helps the assessment team
to develop a more complete understanding of this quality system and to be better prepared to conduct
the assessment. Assessment team members can share their expertise on specific issues if they devise
the questions and note information for the rest of the team. Under one possible format for assessment
checklists, the questions would include a citation of the specific section of the quality document that is
the basis for the question. They may also include the quality document's specifications for acceptable
performance or compliance.
To ensure that the appropriate source for the information is used, the team should prepare
different assessment questions for each different job category of interviewee (senior manager, line
manager, QA staff member, project officer, etc.) and for each different document type (QMP, QA
Project Plan, SOP, etc.). A question may be relevant to an assessment issue and yet be useless if the
wrong person is asked or the wrong document is examined. For more efficiency, the team should
remove redundant questions caused by addressing issues that use some of the same information from
interviews and document reviews.
If an assessment has many issues, interviewees, and documents, it may be helpful to prepare a
matrix, which is a variation of a checklist. The matrix is an information collection tool used to increase
the understanding of the quality system by the assessor and to keep track of all of the information
gathered during assessment planning, on-site interviews, and document review. A matrix can be used
more directly than can a checklist to help prepare a complex assessment report.
A matrix presents the important assessment issues in a format that consolidates the findings from
various interviewees in one place. It could list, at least: (1) the issue with the assessment criterion or
justification; (2) the information discovered in preassessment document reviews with any notation of
things to verify; and (3) space for the summary of on-site interviews and document reviews. It may also
be helpful to include space for comments from the analysis of the evidence, whether there is a negative
or positive finding and/or a noteworthy effective practice. An example matrix appears in Table 2.
During the assessment, this can become a visual aid for the assessment team to see how wide spread an
issue is in the assessed organization. The column for indicating which documents were reviewed can
contain a reference (e.g., QA Project Plan title, date, etc.) and whether or not copies were made. The
matrix can also be used for referencing, a technique that is discussed in Section 4.4, and for focusing
the interview notes on the issues and information needing verification. Additional information regarding
interviewing skills is given in Appendix B.
When completed, the checklists and other assessment aids can demonstrate that the assessment
was conducted, that it was conducted in an orderly and complete manner, and that all assessment
EPAQA/G-3 31 March 2003
-------
issues were addressed. Assessors will probably find it difficult to retain in memory the details of every
interview or document review so it is important to record the information while it is still fresh in the
assessor's mind. Completed checklists and other assessment aids also provide an information base for
assembling findings for the closing meeting and the assessment report.
EPAQA/G-3 32 March 2003
-------
Table 2. Example Row of Working Paper Matrix for Recording Assessment Observations
Issue from audit plan
or other source with
description/rationale
I. QA
Documentation in
EPA Contracts:
A. Effectiveness in
identifying data
collection activities,
B. Verifying process
for reviewing and
approving QA Project
Plan before data
collection begins,
C. Verifying that any
Agency report
(resulting from
contract in this case)
includes the requisite
QA section on
limitations on the use
of the data.
Specification
(cite references)
A. QA Review Form
attached to each scope of
work for contract to
identify data collection
activities.
For contracts over
S500K, QA officer
should be on proposal
technical evaluation panel
(TEP) (Order 5360. land
E4).
B. QA Project Plan
review and approval by
QA manager or delegated
person, dated before data
collection activities begin
(Quality Manual).
C. Report review by QA
manager or delegated
person, dated before
publication (Quality
Manual).
Organization subunit
being reviewed as
part of the
assessment
To be filled in during
the assessment. This
column becomes a
visual aid for the
assessment team to
see how wide spread
an issue is within the
organization.
Specific Documents
that are reviewed
(cite references) and
specific individuals
that are interviewed
To be filled in during
the assessment.
Indicate whether
copies of documents
are made and whether
they are in the
assessors' possession
at the completion of
the assessment.
Quality Management
Plan (QMP)
compliance with
specifications
(cite references)
To be filled in during
planning with
information on how
the processes in
the available
documentation
(QMP, for example)
compare to
specifications.
Interview/documents
compliance with
QMP? (cite notes,
copies, and
references)
To be filled in from
interviews and
document reviews,
although it may be
useful beforehand to
list the expected
evidence from
documents (contracts,
QA Review Forms,
QA Project Plans,
review tools,
oversight
documentation,
reports) and
interview topics
(processes and
qualifications,
participation
on TEPs).
Analysis and positive
or negative finding
(Effective practice?)
To be filled out after
completing interviews
and document
reviews.
EPA QA/G-3
33
March 2003
-------
EPAQA/G-3 34 March 2003
-------
CHAPTER 4
CONDUCTING THE ASSESSMENT
Conduct the
Opening
Meeting
Conduct
Interviews
Review Documents
and Records
Reference Results
of Information
Collection Activity
Compile
Preliminary
Findings
Conduct the
Closing Meeting
After the assessment planning is
complete (as described in Chapter 3), the
assessment can be conducted. Figure 7
indicates the steps for actually conducting the
assessment, which are described in more
detail in this chapter.
The on-site portion of an assessment
may last for a few days. The assessment
team should remain aware that the
assessment is disruptive of the normal
activities of the assessee and use due
professional care in conducting the
assessment. The assessment team should be
considerate of the interviewees' scheduling
constraints and be as professional and
efficient as possible.
At least once each day during a
multiday assessment, the assessment team
should meet privately to share information
gathered so far and to discuss potential
findings and possible problem areas. If
contradictory information has been gathered, more information may need to be collected to resolve the
contradiction. The assessment team may need to discuss and possibly revise the assessment schedule.
In the discussions of assessment schedule, the assessment team leader should make sure that the
assessment stays on track and that team members are not distracted by minor issues. The team
members should be able to contact the leader between the daily meetings in case they encounter a
problem they cannot address.
Similarly, the team may need close contact with the assessee to facilitate scheduling changes. If
needed, the assessee should provide staff to escort the assessment team and see to their needs for
communications, photocopying, etc. If possible to do so within the assessment schedule and if
appropriate for the particular assessment, daily briefings between assessors and assessees can be held.
These provide an opportunity to map out the next day's schedule and to ask for additional documents.
Daily briefings also provide an early opportunity to resolve any misunderstandings between the
assessors and assessees. If daily briefings are not possible because of scheduling constraints, it may be
appropriate to contact the assessee's QA manager and to establish a time to talk with the QA manager.
Many assessors also find it useful to meet with the assessee's QA manager before the closing meeting
to discuss the findings.
Figure 7. Flow Chart for Conducting the
Assessment
EPA QA/G-3
35
March 2003
-------
4.1 OPENING MEETING
A successful opening meeting with the assessee's senior management, QA manager, and other
staff as appropriate is critical to the success of the assessment. The assessment team should keep a list
of attendees with name, titles, affiliations, phone numbers, and mailing and e-mail addresses for post-
assessment contacts. If some of the assessees are anxious or irritated at having to spend time on the
assessment, the assessment team leader can make every effort to reduce the anxiety level by focusing
on the purpose of the assessment and by emphasizing that the team will minimize disruptions of the
organization's normal activities.
Opening Meeting Agenda with Senior
Management and QA Manager
1.
2.
3.
The opening meeting is an
opportunity to describe what will be done,
why, when, and how during the
assessment. An example agenda for this
meeting is presented in Figure 8. The
meeting generally starts with introductions
and thanks for ongoing cooperation with
the assessment. The assessment team
leader should introduce the assessment
team members and review the objectives
of and authority for the assessment,
assessment scope, and criteria. This can
be followed by detailing the principal
questions to be asked during interviews,
the expectations for the reviews and
reports, and the process for assessment
report review. If this assessee organization
has been assessed previously by the same
organization, any changes and additions to
the process since the previous assessment
can be noted. Afterward, the assessee
management should be invited to ask
questions about the assessment. There should be no hidden agenda and surprises. Questions should
be answered directly, truthfully, and without hesitation. The entire meeting usually lasts no longer than
30 to 45 minutes. The assessee should provide any last-minute changes to the list of interviewees and
the schedule for the interviews.
4.2 DOCUMENTS AND RECORDS REVIEW
Information is gathered by reviewing written documentation, such as documents and records,
during the assessment. Assessments typically verify records for evidence of compliance with the quality
system specifications, as stated in the QMP. Generally, documents are examined to find relevant data
and records and to supplement information collected in interviews. Planning documents, prior
assessment reports, and SOPs are examples of the types of documentation that are included in the
document review. Some of these documents may have been reviewed by the assessment team during
the planning phase of the assessment and do not need to be reviewed on-site. As discussed below,
Introductions
Authority for and purpose of the assessment
(if needed, for repeat assessment)
Assessment scope, criteria, and schedule for
interviews and document reviews
Assessment reporting process (with any
differences from previous assessment)
• Closing meeting
• Report content and schedule
• Report review process and schedule
• Corrective action plan and implementation
tracking
• Final report and distribution
Questions and answers
Conclusion
Figure 8. Agenda for the Opening Meeting
EPA QA/G-3
36
March 2003
-------
working papers, such as completed assessment checklists and matrices, should be prepared during the
assessment to keep track of the sources of all information.
During the document review, the assessment team should collect information that answers
specific questions and topics in the assessment plan. The documents to be examined have been
identified in the assessment plan to allow the assessee to assemble the documents before the
assessment, making the assessment team's review more efficient. Including the preliminary list of
documents to be examined in the assessment plan helps the assessment team track the document
review process during the on-site portion of the assessment. It is possible that additional documents
will be identified and requested for review during the assessment. The assessment plan should also list
the documents to be provided to the assessment team prior to the assessment and the time frame for
receiving those documents. However, organizations may document their quality system and its
components differently, or may use different titles for their documents. The list of quality records and
documents in Section 3.6 can be a starting point for the types of documents to review.
The following are some generic questions for documents being reviewed:
If the document is needed for the quality system, does it actually exist? If not, do plans
exist to prepare the document? Does the assessee need assistance in preparing this
document?
• Is a copy of the document readily available for review by the assessor?
• Is the document stored in an organized fashion?
• Is the document accessible to the staff who need to use it? Do they use it?
• Is there evidence (e.g., signature page entries) that the document has been reviewed
and approved in the manner specified for the quality system?
Is the document up to date? If it has to be updated periodically, is this being updated
according to schedule?
Is the document in a format that is reasonable for its intended purpose? Is it readable?
Does the document cite the appropriate quality system specification?
• Does the document fulfill its intended purpose?
• Does the document present evidence that the quality system is functioning as needed?
4.3 INTERVIEWS
The basics of an assessment interview are described in this section. Appendix B gives more
information on interviewing techniques and skills. Examples of interview questions are presented in
Appendix C, and Appendix D is an example assessment checklist. Assessors are not limited to
checklists and can include observations on issues uncovered during document reviews and interviews.
During interviews, emphasize that the quality system is being assessed, rather than the
individuals in the organization. Interviewees can be reassured that their job performance is not being
judged. The organization's management can set an example for the staff by projecting a positive
attitude toward the assessment and the assessors.
Assessment interviews are generally limited to one hour. As is discussed in Appendix B, many
assessment teams prefer to have two assessors participate in all interviews. The assessment team
EPAQA/G-3 37 March 2003
-------
should remain flexible during the interviewing process to accommodate last-minute changes resulting
from scheduling conflicts, retrieval of documentation, and so on. All of the interviewers should be
introduced at the start of the interview. One of the interviewers should briefly discuss the purpose of
the assessment, how and why the interviewees were selected, what information is needed from the
interview, and what will be done with the information. One useful point to make at the start of the
interview is that the assessment report will not attribute specific comments to specific interviewees. The
interviewees should be given an opportunity to ask questions. Assessors generally use the assessment
checklist as a guide, not a script to be rigidly followed. Assessors can record observations on new
issues that are uncovered during document reviews and interviews. They should be open to information
that is supplied by interviewees about needed improvements to the quality system. However, they
should be careful not to allow the assessment issues to be manipulated by interviewees who have goals
that are not related to the improving quality system. Assessors should not be drawn into political
maneuvering within the assessed organization.
Generally, only one interviewee is included in each interview, but there are circumstances in
which more than one interviewee will be included. Also, when there is a team working together on a
project, it may be expedient to interview the team together so that all of the questions can be answered
at one time. For instance, some assessee managers may insist on having their QA manager or a
management representative attend selected interviews. During the interviews, the interviewer should be
careful to ensure that the information is provided by the interviewee without prompting by the manager
or any other management representative who may be present. While the manager or a management
representative may be welcome to attend interviews, they should not be allowed to direct or signal the
interviewee what the acceptable answer is from their perspective.
Assessment questions should not lead the interviewee toward a specific response that the
assessor expects to hear. Leading questions would bias the assessment findings. They can be
avoided by making them open-ended rather than closed-ended. The following close-ended and open-
ended questions illustrate how poorly designed assessment questions may produce a biased response:
Close-ended: Are appropriate technical experts involved in the project planning
process?
Open-ended: What is the role of technical experts in planning your office's projects?
Because an open-ended question provides no structure for the answer, the interviewee may
provide information that is not directly relevant to the issue at hand. The interviewer should keep the
interviewee focused on the issue. Often it is useful to ask the same question of several staff
members in different positions. This can help gauge the degree of implementation of the quality
system.
At the end of each interview, the interviewer generally repeats how the assessment findings will
be used, mentions the possibility of follow-up, asks if there is anything more that the interviewee would
like to add, and then thanks the interviewee for his or her time. Interview times should be structured so
that after each interview, the interviewer(s) has sufficient time to review and complete notes before the
next interview.
EPAQA/G-3 38 March 2003
-------
4.4 ASSESSMENT WORKING PAPERS
Working papers are the written record of the assessors' observations. They provide the link
between objective evidence obtained during the assessment and the findings presented in the
assessment report. They may include checklist questions, assessment criteria, the assessor's
observations, and cross-references to the source of the objective evidence supporting the observations.
They should be legible, accurate, complete, concise, and understandable without oral explanation.
Assessment organizations should have procedures for the preparation and maintenance of working
papers, including their storage and retention duration. All documents received during an assessment
should be tracked by the assessment team. In some organizations, a tracking number is assigned; in
others, receipt of documents is tracked in the assessors' notes.
During an assessment, each assessor should compile working papers that record observations
from interviews and document reviews as well as the sources of these observations. These working
papers may be retained by the assessors as objective evidence for all statements made in the
assessment result reports. Objective evidence is any documented statement of fact, other information,
or record, either quantitative or qualitative, pertaining to the quality of an item or activity, based on
observations, measurements, or tests which can be verified (ASQ, 1994).
The technique of referencing the assessment findings to the working papers can impose a high
standard for note taking (GAO, 1994). The assessment team should understand the importance of
recording all information accurately during interviews and document reviews. The assessment
checklists can function as working papers provided that sufficient space for notes is available on the
form. Taking good, organized notes during the assessment, particularly notes geared to the assessment
issues, will make preparing the report easier and will substantiate findings in case of any disputes.
4.5 PRELIMINARY FINDINGS
Assessment findings are statements of importance that are based on a comparison of objective
evidence obtained during the assessment to the assessment criteria. They are the result of information
development; a logical pulling together of information to arrive at conclusions (or a response to an audit
objective on the basis of the sum of the information) about an organization, program, activity, function,
condition, or other matter which was analyzed or evaluated (GAO, 1994). They will form the basis for
conclusions and any recommendations for corrective action.
Assessment findings can generally be divided into three categories:
1. noteworthy practices or conditions (i.e., strengths) - positive;
2. observations, which are neither positive nor negative - neutral; and
3. nonconformances, which are deviations from standards and documented practices -
negative. They can be divided into two subcategories:
a. deficiencies, which adversely impact quality, and
b. weaknesses, which do not necessarily (but could) result in unacceptable quality.
To ensure the relevance of the findings, the "so what" test can be applied. This test helps to
determine whether a finding is significant relative to the overall goals of the quality system. The
EPAQA/G-3 39 March 2003
-------
credibility of the assessment will largely rest on how the findings are perceived by the assessee.
Frivolous or irrelevant findings can easily destroy credibility. It is essential that the findings reflect only
significant issues because insignificant findings obscure those that really matter.
Generally, the graded approach is applied to the development and presentation of assessment
findings. For internal assessments in small organizations, the findings may be presented less formally
than for external assessments in large organizations. Nevertheless, the written documentation of
assessments and their findings should be objective evidence that a quality system is or is not effective
and implemented as planned.
Before the closing meeting, the assessment team should review and summarize its observations
from the interviews and document reviews and discuss the preliminary findings. One approach to
compiling team findings is to have each team member nominate candidate findings and then discuss the
specific observations supporting each candidate finding. This approach allows the team to resolve any
uncertainties or inconsistencies regarding individual findings and to determine the relative importance of
individual findings. The assessment team leader has final authority for decisions on the findings, but all
team members are expected to have input. It is important that findings be prioritized according to their
significance so that important findings are not lost within a list of trivial concerns. Remember that the
assessment plan can assist with the interpretation of observations and also aids in identifying findings.
The initial findings are usually presented in the closing meeting. It may be a good idea to meet
with the assessee's QA manager or his or her designated point of contact before that meeting to gain
the organization's perspective on issues. This would also present an opportunity to share details about
other issues identified during the assessment that may not be of interest to management.
The initial findings may have limitations. For example, notes may not have been completely
studied and discussed. If documents are being taken for further study, new information may be
identified that will change the interpretation and lead to different or additional findings. The assessment
team should commit to contacting the assessee if findings change.
The assessment team leader should develop a summary or overview of the assessment and the
preliminary findings. The assessment team would then be well prepared to discuss the findings at the
closing meeting. Development and implementation of corrective actions remain the responsibility of the
assessee, but the assessment team can provide technical assistance, when appropriate.
4.6 CLOSING MEETING
Generally the closing meeting is attended by the same group that attended the opening meeting.
Important talking points to be stressed during the closing meeting include that:
findings from the assessment are preliminary
findings may change if the review is incomplete (particularly, if the findings of individual
assessors have not yet been generalized to the findings for the entire quality system -
this may not be available at the time of the closing meeting)
the assessee will be contacted if the findings change or if more information is needed;
findings will be made available to the assessee before the report is finalized
EPAQA/G-3 40 March 2003
-------
findings can be addressed by the assessee before the report is finalized
technical assistance and/or recommendations can be provided by the assessment team
(if specified by the authorizing entity)
confidentiality and dissemination of assessment findings, and the schedule for reports
are discussed.
As noted above, findings should still be preliminary during the closing meeting until the assessment team
has had an opportunity to compare notes, think
about the ramifications of any problems, and
complete the review of documents. Technical
assistance and/or recommendations generally are
provided only if requested by the authorizing
entity or the assessee. Technical assistance or a
recommendation should be offered carefully, if at
all, because it is important that the assessed
organization retains ownership of corrective
actions. The assessment team may be experts in
quality systems and assessments, but the
assessed organization knows more about their
own quality system and how to best implement
corrective actions. Both identifying findings and
offering assistance or recommendations should
be performed only within the scope of the
assessment plan and in agreement with any
ground rules established prior to the assessment.
The participants should be thanked for
their cooperation, time, and help. An example
agenda for the closing meeting is presented in
Figure 9.
2.
Closing Meeting (Same attendees as
Opening Meeting, or assessee's choice)
1. Introductions (if needed) and appreciation
for assistance and cooperation
Brief discussion of deviations from the
assessment plan (if needed)
Preliminary findings with discussion of
corrective action process (if needed)
• Addressing findings before final report
• Technical assistance and
recommendations
Procedure for contact if findings change
Assessment reporting process review
(if needed)
Questions and answers
Conclusion
4.
5.
Figure 9. Agenda for the Closing Meeting
EPA QA/G-3
41
March 2003
-------
EPAQA/G-3 42 March 2003
-------
CHAPTER 5
REPORTING AND FOLLOW-UP
After the assessment is complete, the
assessment team should summarize the findings in a
timely manner. The report will have more of an
effect on the quality system if it is received while the
assessment is still fresh in the assessee's mind. An
assessment report may be prepared, reviewed, and
then submitted to the assessee, typically to check the
report for accuracy. After comments by the
assessee are resolved, the final assessment report
should be prepared. Figure 10 presents the steps for
reporting and follow-up.
5.1 EVALUATING COLLECTED
ASSESSMENT INFORMATION
Corrective Action
and Follow-Up
Activitities
Close-Out
Letter
Quality
Improvement
Soon after completing the on-site portion of
the assessment, each team member should review all
of his or her collected materials, working papers, and
notes, and prepare preliminary findings. The
assessment team leader then would consolidate the
preliminary findings and circulates them to team
members, who add more material and can suggest
new findings based on additional review of their
notes and other materials obtained during the
assessment. Findings are generally tested against the
evidence, such as the documents and records
reviewed. At this point, it is critical that the team
determine whether the findings are relevant to the
assessment goals. The team should reach consensus
on the message and format, and determine if the
findings are clear, coherent, and persuasive. To
avoid surprises, the assessee is generally contacted
to discuss any new findings, as had been previously
arranged during the on-site portion of the assessment. The assessee's QA manager or designee can be
contacted, if additional information is needed, with copies of any requests sent to the assessee's
management.
As discussed in Section 4.4, referencing is a technique for controlling the quality of assessment
reports. Using this technique, all statements are substantiated by notes taken during interviews or
review of documentation. Assessment team members could provide highlighted notes and relevant
pages of reviewed documents to the report writer to support findings.
Evaluate
Collected
Assessment
Information
Report Findings
Draft Assessment Report Reviewed by:
1. All assessment teams members,
2. Assessing organization's management,
3. Authorizing entity, and
4. Assessee.
Final Audit Report Prepared and Submited
Figure 10. Typical Steps for Assessment
Reporting and Follow-Up
EPA QA/G-3
43
March 2003
-------
5.2 REPORTING FINDINGS t „ ,
1. Background, Purpose, and Scope
The objective of an assessment report is
to communicate assessment findings to the proper
levels of management. Different organizations use
different formats, but many of these formats
clearly state the type of assessment, the assessor,
the assessee, what was assessed, the findings,
and, if requested by the authorizing entity, the
2. Summary and Findings
3. Corrective/Response Actions Process and
Recommendations (if previously specified)
References
Appendices
A. Assessment Plan
B. List of Personnel Interviewed
conclusions and recommendations. An example ^ T • ± r-r\ ^ j ™ j ™ • j
~ . • • • r. 11 T,, . C. List or Documents and Records Reviewed
format is given in Figure 11. The assessment
team leader is primarily responsible for producing D Correctlve Actlon plan
the assessment report, but it should be a
collaborative effort Figure 11. Example Assessment Report
Outline
Many organizations prepare a draft
assessment report for review by the assessee, while others present an oral report at the end of the on-
site portion of the assessment in lieu of a written draft report. A draft report, when that approach is
used, provides the assessee with an opportunity to comment on the written document before it is
finalized, but the approach does take additional time. For some assessments, the criteria and issues
may be so straightforward as to permit concluding the assessment on-site with a presentation of a
streamlined report, which does not need additional explanation. This method, like more conventional
reporting, would warrant up-front agreement with both the authorizing entity and the assessee.
Using a standard report format with boilerplate text, when appropriate, can make report
preparation easier. Clear and concise writing, without unsubstantiated generalizations or ambiguous
remarks, facilitates understanding and appropriate action by the assessee. Try to avoid words that
could be misinterpreted. To achieve the goal of quality improvement, significant deficiencies are best
addressed in a constructive manner. The report should include both positive and negative observations,
when appropriate. In the report, the organization's actions should be discussed, but not the actions of
specific individuals, because individual interviewees are not quoted in the report.
Assessment findings and any recommendations should be considered in the context of the
assessee's overall goals. A higher priority can be accorded to findings that might affect more important
aspects of the assessee's quality system. Any recommendations in the findings should be clearly
presented and provided to the assessee only if specified in the approved assessment plan or upon
request by the authorizing entity or assessee. An unsolicited recommendation carries a risk of being
accepted and implemented, but then leading to unanticipated negative consequences. Any
recommendations that are not specifically linked to any nonconformances should be identified and
justified.
A recommendation that is not convincing will not be implemented (GAO, 1991b). The GAO
explains that when adequately implemented by the assessee, it accomplishes a defined and worthwhile
result. It states a clear, convincing, and worthwhile basis for implementation. One that does not
correct the root cause of a nonconformance may not achieve the desired result. The utility and
continued relevance of a recommendation should be reevaluated during follow-up activities.
EPAQA/G-3 44 March 2003
-------
When the need for corrective actions is identified, it is helpful to attach a chart for the assessee
to fill in that gives a corrective action plan with a proposed schedule. The assessment team may
provide a template of the corrective action plan in the draft report that includes the specific findings; an
example of this template is given in Table 3. If no corrective actions are identified, the report with
recommendations can be recorded and sent to the assessee to check for accuracy.
Table 3. Example of a Corrective Action Plan
Finding
Number
1
Report Finding
Oversight of field and laboratory
activities is not routinely implemented
as described in the QMP
Corrective
Action
(To be added
by assessee)
Responsible
Official
(To be added
by assessee)
Due Date
(To be
negotiated)
Typically, the draft report is reviewed by all team members, then by the assessing organization's
internal management, then by the authorizing entity. Finally, the report is transmitted to the assessee
with a transmittal memorandum or letter. When the report is sent to the assessee for comment, a
specific date for receiving comments is often stated in the transmittal memorandum or letter.
The assessee should complete the corrective action plan and submit it to the assessment team
for approval, generally along with any other comments on the draft report. This submission sets the
stage for follow-up with specific commitments by management. The corrective action plan can specify
the organizational positions of the individuals who are responsible for implementing the corrective
actions. If agreed upon, the completed corrective action plan may be sent back from the assessee later
than the comments on the draft assessment report.
To finalize the report, the assessment team should incorporate any relevant comments from the
assessee when appropriate, correct any identified factual errors, and resolve any disputes if possible.
Any disputes are usually resolved at the lowest administrative level possible and in accordance with the
dispute resolution process for the assessment system. If the assessee does not respond in a timely
fashion, the assessment team leader should contact the assessee QA manager or designated point of
contact. It may be necessary for the assessment program manager or the authorizing entity to play a
role in dispute resolution. After final approval for the report is received from the authorizing entity, it
should be distributed as previously agreed in the assessment plan.
5.3 CORRECTIVE ACTION AND FOLLOW-UP ACTIVITIES
The development and implementation of corrective actions are an integral part of the
assessment process, but they typically are the responsibility of the assessee. Development of corrective
actions generally addresses the following points:
• measures to correct each nonconformance;
identification of all root causes for significant deficiencies;
• determination of the existence of similar deficiencies;
corrective actions to preclude recurrence of like or similar deficiencies;
EPA QA/G-3
45
March 2003
-------
• assignment of responsibility for implementing each corrective action; and
completion dates for each corrective action.
The assessment team can provide technical assistance to the assessee in developing appropriate
corrective actions, but they may not have detailed enough knowledge of the assessee's organization to
identify the most appropriate and effective corrective actions. If they participate in implementing the
corrective actions, their objectivity during subsequent assessments of the quality system may be
compromised. Although the corrective actions will be developed and implemented by the assessee
after the assessors have submitted their report, they are still an integral part of the assessment process.
It is important that the assessee establish ownership of the corrective actions to help ensure that the
promised corrective actions will be implemented. This ownership will also help to ensure that the
corrective actions will be effective in resolving the root cause of the assessors' findings, rather than only
addressing the symptoms of the problem.
After the corrective actions have been completed, the assessors may conduct follow-up
activities. These activities can range from a review of documentation submitted by the assessee about
the corrective actions to an on-site follow-up assessment to determine the effectiveness of the
corrective actions. The authorizing entity should be informed of planned follow-up activities and
approve them beforehand.
As noted in the previous section, identifying both a deadline and responsible person for
implementing corrective actions will facilitate appropriate actions being completed. Regular reporting
may be established, or the assessors may be assigned to periodically contact the organization's QA
manager. Another means of follow-up can be to have a designated assessor check progress with the
assessee within a designated time frame on a particular issue. The assessors also should make sure that
they provide any promised assistance or reviews.
Documenting the follow-up activities should ensure that a subsequent assessment team will be
able to track activities. Subsequent assessments are often performed by a different assessment team.
Often, this follow up is accomplished during subsequent assessments. In addition, it may be done by
receiving and reviewing reports summarizing the corrective actions or by tracking them in routine
reports, such as the QA Annual Reports and Work Plans that are submitted by EPA Program Offices,
Regional Offices, and National Research Laboratories. These reports document activities of the quality
system or revisions to the QMP.
5.4 FORMAL CLOSE OUT OF ASSESSMENT
After all assessment activities are complete, the assessment can be closed. This generally
occurs after a response from the assessed organization, an acceptable corrective action plan (if
necessary), and verification of completion of corrective action are received. The assessment file should
be reviewed to ensure that it is complete before it is archived. The assessment team leader may issue a
close-out letter stating that all actions associated with the assessment are complete. The close-out letter
should be added to the file. Figure 12 presents an example of a close-out letter. Not all organizations
use a formal close-out letter; some prefer to document close out of assessments in QA Annual Reports
and Work Plans.
EPAQA/G-3 46 March 2003
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
Region 12
Juneau, Alaska 99801
April 15, 2003
Julia Bennett, Commissioner
Alaska Department of Environmental Conservation
410 Willoughby Avenue, Suite 303
Juneau, AK 99801-1795
Dear Commissioner Bennett:
This letter confirms the close-out of the assessment of the Alaska Department of Environmental
Conservation's quality system conducted by Region 12 during the week of January 27, 2003.
Based on our evaluation of your response to the draft assessment report, we have determined that
all deficiencies have been resolved. This is reflected in the final assessment report, which is
enclosed.
Thank you very much for your cooperation and assistance during the assessment. Please contact
me if you have any further questions about the assessment.
Respectfully,
William Shipley
Regional Administrator
Enclosure: Final Assessment Report
cc: Mark Zimmerman, DEC Quality Assurance Manager
Figure 12. Example Close-Out Letter
5.5 QUALITY IMPROVEMENT
After an assessment, the team may review the experience and identify what went well and what
needs improvement in the assessment process. They also may consider how the process is supporting
EPA's environmental decision making. The goals for the assessment can be revisited. The findings
from one assessment of an organization can be used in planning for its next assessment. In the future,
the assessment team may decide to concentrate on areas where deficiencies were identified, areas with
significant staff turnover between assessments, areas added to the program since the last assessment, or
areas that were not previously assessed.
EPAQA/G-3 47 March 2003
-------
EPAQA/G-3 48 March 2003
-------
CHAPTER 6
REFERENCES AND SUPPLEMENTAL READING
6.1 REFERENCES
Adams, N. H. 2000. "Never Audit Alone-The Case for Audit Teams." Presented at the 19th
Annual National Conference on Managing Environmental Quality Systems, Albuquerque, NM.
American Society for Quality. 1994. Specifications and Guidelines for Quality Systems for
Environmental Data Collection and Environmental Technology Programs, ANSI/ASQC
E4-1994. American Society for Quality, Milwaukee, WI.
DeLeeuw, E.D. 1992. "Data Quality in Mail, Telephone and Face-to-Face Surveys" TT-
Publikaties, ISBN 90-801073-1-X. Amsterdam, Netherlands.
U.S. Environmental Protection Agency. 2000a. EPA Order 5360.1. A2: Policy and Program
Requirements for the Mandatory Agency-wide Quality System. Washington, DC.
U.S. Environmental Protection Agency. 2000b. EPA Order 5360 Al: EPA Quality Manual for
Environmental Programs. Washington, DC.
U.S. Environmental Protection Agency, 2001. EPA Requirements for Quality Management Plans
(QA/R-2). EPA/240/B-01/002. Office of Environmental Information, Washington, DC.
U.S. Environmental Protection Agency. 2002. Overview of the EPA Quality System for
Environmental Data and Technology. EPA/240/R-02/003. Washington, DC.
U.S. Government Accounting Office (GAO). 1991. Using Structured Interviewing Techniques.
GAO/PEMD-10.1.5. Washington, DC.
U.S. Government Accounting Office (GAO). 1991b. How to get Action on Audit
Recommendations. GAO/OP-9.2.1. Washington, DC.
U.S. Government Accounting Office (GAO). 1993. Developing and Using Questionnaires.
GAO/PEMD-10.1.7. Washington, DC.
U.S. Government Accounting Office (GAO). 1993b. An Audit Quality Control System: Essential
Elements. GAO/OP-4.1.6. Washington, DC.
U.S. Government Accounting Office (GAO). 1994. Government Auditing Standards. 1994
Revisions. GAO/OCG-94-4. Washington, DC.
U.S. Government Accounting Office (GAO). 2002. Government Auditing Standards. Amendment
No. 3. Independence. GAO-02-388G Washington, DC.
EPAQA/G-3 49 March 2003
-------
Worthington, J.C. 1998. "Continuous Improvement in Quality Audit Systems." Environmental
Testing Analysis, 7(l):23-26.
6.2 SUPPLEMENTAL READING
EPA Documents (http://www.epa.gov/quality l/qa_docs.html)
U.S. Environmental Protection Agency. 2000. EPA Guidance for Quality Assurance Project Plans
(QA/G-5). EPA/600/R-98/018. Washington, DC.
U.S. Environmental Protection Agency. 2000. Guidance for Data Quality Assessment: Practical
Methods for Data Analysis (QA/G-9). EPA/600/R-96/084. Washington, DC.
U.S. Environmental Protection Agency. 2000. Guidance for the Data Quality Objectives Process
(QA/G-4). EPA/600/R-96/055. Washington, DC.
U.S. Environmental Protection Agency. 2000. Guidance on Technical Audits and Related
Assessments (QA/G-7). EPA/600/R-99/080. Washington, DC.
U.S. Environmental Protection Agency. 2001. EPA Requirements for Quality Assurance Project
Plans for Environmental Data Operations (QA/R-5). EPA/240/B-01/003. Washington,
DC.
U.S. Environmental Protection Agency. 2001. Guidance for the Preparation of Standard
Operating Procedures for Quality-Related Operations (QA/G-6). EPA/240/B-01/004.
Washington, DC.
Not available electronically:
U.S. Department of Energy, Office of Environmental Policy & Assistance and U.S. Environmental
Protection Agency, Office of Federal Facilities, undated. Environmental Management
Systems Primer for Federal Facilities. DOE/EH-0573. Washington, DC.
EPA Regulations (http://access.gpo.gov/nara/cfr/cfr-table-search.html)
40 CFR 2, Subpart B, Code of Federal Regulations. Confidentiality of Business Information.
40 CFR 30, Code of Federal Regulations. Grants and Agreements with Institutions of Higher
Education, Hospitals, and Other Non-Profit Organizations.
40 CFR 31, Code of Federal Regulations. Uniform Administrative Requirements for Grants and
Cooperative Agreements to State and Local Governments.
40 CFR 35, Code of Federal Regulations. State and Local Assistance.
EPAQA/G-3 50 March 2003
-------
International Organization for Standardization (ISO) Documents (http://www.iso.ch)
Guidelines for Auditing Quality Systems-Auditing, ISO Standard 10011-1-1994.
Guidelines for Auditing Quality Systems-Management of Audit Programs, ISO Standard 10011-
3-1994.
Guidelines for Auditing Quality Systems-Qualification Criteria for Quality System Auditors, ISO
Standard 10011-2-1994.
American Society for Quality (ASQ) Publications (http://www.qualitypress.asq.org and
http://www.asq.org)
American Society for Quality. 1987. How to Plan an Audit, ASQC Quality Audit Technical
Committee, C.B. Robinson, ed. American Society for Quality, Milwaukee, WI.
American Society for Quality Standards Committee. 1994. American National Standard.
Guidelines for Auditing Quality Systems. ANSI/ISO/ASQC Q10011 -1 -1994, Q10011 -2-
1994, Q10011-3-1994. American Society for Quality, Milwaukee, WI.
Arter, D.R. 1994. Quality Audits for Improved Performance, Second Edition. American Society
for Quality, Milwaukee, WI.
Beeler, DeWitt 1998. "Internal Auditing: The Big Lies." Quality Progress, 31(5):73-78.
Mills, CA. 1989. The Quality Audit: A Management Evaluation Tool. American Society for
Quality, Milwaukee, WI.
Sayle, A.J. 1997. Management Audits. The Assessment of Quality Management Systems, Third
Edition. American Society for Quality, Milwaukee, WI.
Smith, J.L. 2000. The Quality Audit Handbook, Second Edition. ASQ Quality Audit Division, IP.
Russell, ed. ASQ Quality Press, Milwaukee, WI.
EPAQA/G-3 51 March 2003
-------
EPAQA/G-3 52 March 2003
-------
GLOSSARY
assessee-the organization being assessed.
assessment - the evaluation process used to measure the performance or effectiveness of a system
and its elements.
assessment checklist- a document for systematically recording objective evidence from interviews.
It is useful as a means to obtain information that has not been documented by the assessee. It consists
of a series of specific questions about the quality system. When completed, the assessment checklist
demonstrates that the assessment was conducted, that it was conducted in an orderly and complete
manner, and that all relevant aspects of the quality system were addressed during the assessment.
assessment criteria - objective and written reference standards to which the assessed quality
system's characteristics are compared. These documents may be external specifications coming from
outside the assessee as well as the assessee's own specifications and quality system planning
documents.
assessment findings - statements of importance that are based on a comparison of objective
evidence obtained during the assessment to the assessment criteria. They should be the result of
information development; a logical pulling together of information to arrive at conclusions (or a response
to an audit objective on the basis of the sum of the information) about an organization, program,
activity, function, condition, or other matter which was analyzed or evaluated. They may be positive,
neutral, or negative. They are normally accompanied by specific examples of the observed condition.
They will be the basis for conclusions and any recommendations for corrective action.
assessment issues - the specific components of a quality system or organization that will be assessed.
A quality system or organization may be too large or complex to be assessed completely within the
resources that are available for the assessment. Specific components of the quality system may be
selected to be assessed to narrow the focus of the assessment to a manageable scale. Assessment
issues should not be selected on a subjective basis, but should be selected after an objective analysis of
the assessment criteria, the assessment scope, and the information about the assessee that the
assessment team has reviewed. Examples of assessment issues are QA officer independence, the QA
project plan review process, and QA training for staff. Systematic planning should be used to select
assessment issues that will yield the greatest benefit to the quality system within the available assessment
resources.
assessment of a quality system- a process for assessing an organization's practices as they relate to
its quality system. The focus of the assessment process should be on the quality system rather than the
quality of data to support an individual decision. Assessments should be designed to assess the
organization's quality system and to provide a relatively unbiased and objective source of feedback
about the quality system. The assessment seeks to determine if a quality system is implemented and is
operating within an organization in the manner prescribed by the approved QMP and consistent with
current specifications.
EPAQA/G-3 53 March 2003
-------
assessment plan - a written document prepared by the assessment team under the direction of the
assessment team leader. It should include the authority and assessment criteria for the assessment, the
purpose and scope of the assessment, and a description of organizations that will be visited during the
assessment. The plan should include details, such as a schedule of assessment activities, specific
personnel (or job positions) to be interviewed, and specific files and documentation that will be
reviewed during the assessment.
assessment scope - the depth and coverage of the assessment. It concerns such questions as: Is the
whole quality system or part of it going to be assessed? What programs, projects, laboratories or
offices are to be assessed? How many documents are going to be reviewed and how many individuals
are going to be interviewed?
assessment team leader - the person responsible for all phases of the assessment. The assessment
team leader should have management ability and experience and be given authority to make final
decisions regarding the conduct of the assessment and any assessment findings.
assessor - the person or team of people who perform the assessment. The assessor can be either
internal (part of the organization being assessed) or external.
audit - a systematic and independent examination to determine whether activities and related results
comply with planned arrangements and whether these arrangements are implemented effectively and are
suitable to achieve objectives.
authorizing entity - whoever authorizes the assessment and has the authority to do so. This is often
the individual responsible for the quality system that is being assessed.
confidential business information - any information, in any form, received by EPA from a person,
firm, partnership, corporation, association, or local, state, or federal agency that related to trade secrets
or commercial or financial information and that has been claimed as confidential by the person
submitting it under the procedures in 40 CFR, Part 2, Subpart B.
corrective action - any measures taken to rectify conditions adverse to quality and, where
possible, in order to prevent recurrence.
deficiency - a negative assessment finding (i.e., a nonconformance) that renders the quality of an item
or activity unacceptable or indeterminate; nonfulfillment of a specification or standard.
documentation - comprises documents and records.
environmental data - any measurements or information that describe environmental processes,
locations, or conditions; ecological or health effects and consequences; or the performance of
environmental technology. For EPA, environmental data include information collected directly from
measurements, produced from models, and compiled from other sources such as data bases or the
literature.
EPAQA/G-3 54 March 2003
-------
environmental data operation - work performed to obtain, use, or report information pertaining to
environmental processes and conditions.
external assessment - see management independent assessment.
extramural agreement - a legal agreement between EPA and an organization outside EPA for items
or services to be provided. Such agreements include contracts, work assignments, delivery orders,
task orders, cooperative agreements, research grants, state and local grants, and EPA-funded
interagency agreements.
financial assistance - the process by which funds are provided by one organization (usually the
government) to another organization for the purpose of performing work or furnishing services or items.
Financial assistance mechanisms include grants, cooperative agreements, and government interagency
agreements.
graded approach - the process of applying managerial controls to an item or work according to the
intended use of the results and the degree of confidence needed in the quality of the results.
independence - freedom from bias and external influences that could affect the assessor's objectivity.
independent assessment - see management independent assessment.
internal assessment - see management self-assessment.
management - those individuals directly responsible and accountable for planning, implementing, and
assessing.
management independent assessment - the qualitative evaluation of a particular program operation
and/or organization(s) by someone other than the group performing the work (either internal or external
to the organization) to establish whether the prevailing management structure, policies, practices, and
procedures are adequate for ensuring that the type and quality of results needed are obtained.
management self-assessment - the qualitative evaluation of a particular program operation and/or
organization(s) by those immediately responsible for overseeing and/or performing the work to establish
whether the prevailing management structure, policies, practices, and procedures are adequate for
ensuring that the type and quality of results needed are obtained.
management system audit - see management independent assessment.
management system review- an assessment of a developing quality system, including technical
assistance in developing the quality system, as well as evaluation of the quality system.
nonconformance - a negative assessment finding of a deviation from standards, specifications, and
documented practices, which may be either a deficiency or a weakness.
noteworthy practice or condition - a positive assessment finding; a strength.
EPAQA/G-3 55 March 2003
-------
observation - an assessment finding that identifies a neutral condition that does not represent a
significant impact (either positive or negative) on the quality of an item or activity, based on
observations, measurements, or tests that can be verified.
organization - a company, corporation, firm, enterprise, or institution, or part thereof, whether
incorporated or not, public or private, that has its own functions and administration. In the context of
EPA Order 5360.1, an EPA organization is an office, region, national center, or laboratory.
procedure - written instructions for performing a tasks, not the actions themselves.
quality assurance - an integrated system of management activities involving planning, implementation,
documentation, assessment, reporting, and quality improvement to ensure that a process, item, or
service is of the type and quality needed and expected by the customer.
quality control - the overall system of technical activities that measures the attributes and performance
of a process, item, or service against defined standards to verify that they meet the stated specifications
established by the customer; operational techniques and activities that are used to fulfill specifications
for quality.
quality management plan - a document that describes a quality system in terms of the organizational
structure, policy and procedures, functional responsibilities of management and staff, lines of authority,
and needed interfaces for those planning, implementing, documenting, and assessing all activities
conducted.
quality procedures - written instructions for planning, implementing, documenting, or assessing
specific activities associated with the quality system.
quality system- a structured and documented management system describing the policies, objectives,
principles, organizational authority, responsibilities, accountability, and implementation plan of an
organization for ensuring quality in its work processes, products (items), and services. The quality
system provides the framework for planning, implementing, documenting, and assessing work
performed by the organization and for carrying out specified QA and QC activities.
record - a completed document that provides objective evidence of an item or process. Records may
include photographs, drawings, magnetic tape, and other data recording media.
self assessment - see management self-assessment.
strength - a positive assessment finding; a strong attribute or inherent asset.
weakness - a negative assessment finding (i.e., a nonconformance) that does not necessarily result in
unacceptable data.
working papers - documents such as checklists that are used to record information during the
assessment.
EPAQA/G-3 56 March 2003
-------
APPENDIX A
GUIDANCE FOR BEING ASSESSED
This appendix provides guidance for those being assessed, who may not be interested in all of
the details provided in the main document.
What is a quality system?
A quality system is the set of management policies and procedures and related technical
procedures that an organization has developed and documented to ensure that its products and services
(e.g., environmental data collection, environmental technology) attain some specified quality objectives.
The top management of the organization initiated the development of these policies and procedures and
now stands behind them as the expected way of doing things in the organization. The quality system
may have been developed in response to internal initiatives or external specifications.
What is an assessment of a quality system?
An assessment of a quality system is a systematic, independent, and documented examination
that uses specified assessment criteria to answer one or more of the following questions about an
organization's quality system:
• If an organization is developing a quality system, what QA activities remain to be
implemented and what technical assistance by the assessors will promote the
development and implementation of this quality system?
Is the organization's quality system documented and fully implemented?
Does the organization understand external quality specifications?
Does the quality system comply with external quality specifications?
Do the activities that are being performed by the organization comply with its quality
system documentation, particularly the QMP?
• Are the quality system procedures implemented effectively?
• Does the quality system support environmental decision making with processes that
ensure that data are sufficient in quantity and quality appropriate for their intended
purpose?
An assessment of a quality system has a different focus from a technical systems audit, which
determines whether the organization's technical procedures are being followed and whether they
generate work products of a specified quality. Rather, it looks at the management policy and
procedures that are used to plan, implement, assess, and correct the technical activities.
The assessment strives to be objective and should be performed by assessors who are
independent of doing or managing the technical activities. The assessors should not have a vested
interest in the quality system being assessed.
EPAQA/G-3 A-l March 2003
-------
Why is an assessment being conducted?
EPA organizations should perform assessments of their quality systems at least annually. EPA
regulations governing extramural agreements cover assessment of extramural organizations by EPA.
Extramural organizations should perform periodic internal assessments of their own quality systems. An
extramural organization's use of assessments is described in its QMP.
One purpose of an assessment is to improve the assessee's quality system, whether it is
implemented or developing. Another purpose of an assessment is to provide valid feedback to
management on the adequacy, implementation, and effectiveness of the quality system.
Who are the assessors? Who do they represent?
Assessors may either be from part of the organization being assessed (internal) or from outside
the organization being assessed (external). They should be trained for their assessing responsibilities
and should have reviewed relevant materials to prepare for the particular assessment. They represent
the authorizing entity; that is, the organization that authorized the assessment. Often the authorizing
entity is the individual responsible for the quality system in an organization.
What are the criteria for the assessment?
Assessment criteria are objective and written reference standards to which the assessed quality
system's characteristics are compared. These documents may be external specifications that are
applicable to the assessee as well as the assessee's own policies and quality system planning
documents. Assessment criteria should be agreed upon by the assessors, the authorizing entity, and the
assessee before the assessment begins. Documents that are relevant to quality systems for work
performed by or for EPA may include the following:
Order 5360.1A2
EPA's Quality Manual
• EPA specifications for QMPs
• ANSI/ASQC specifications and guidelines for quality systems
• the assessee's QMP
the assessee's reports (e.g., quarterly progress reports or Quality Assurance Annual
Report and Work Plan)
QA and QC aspects of regulations.
What can I expect to happen during an assessment?
In addition to determining compliance with quality system specifications, an assessment is an
opportunity for the assessed organization to obtain independent feedback about the suitability and
effectiveness of its own quality system. An assessment is an opportunity for recognition of the
assessee's commendable practices and a chance to "showcase their talents." Assessments also
provide an opportunity for two-way communication between the assessee and the assessment team.
Assessees are encouraged to keep a spirit of cooperation through the assessment process.
Assessments emphasize quality improvement.
EPAQA/G-3 A-2 March 2003
-------
What does "no surprises" mean?
Assessments should be performed in an open and collegial manner, and every effort should be
made to avoid surprises. The "no surprises" approach means that the assessee should be made fully
aware of the scope of the assessment and how the findings will be used before the assessment takes
place. The assessee should be invited to contribute to assessment planning to help assure that they
understand what will be done. Moreover, the draft report should not introduce any issues that were not
discussed at the closing meeting or in later discussions.
What logistical arrangements should be made for an assessment?
The assessment team generally will make initial contact with the assessee to announce its
intention to conduct an assessment, discuss possible dates, describe the criteria and scope of the
assessment, request necessary documents, and reserve space for document reviews and interviews.
The assessee should be candid in the discussion about the personnel and program schedules so that the
assessment does not occur at a time when the needed staff members are unavailable. The assessee is
encouraged to respond to requests for information in a timely manner because making information
available before the assessment will reduce disruptions during the assessment. The assessee should
designate a point of contact, usually the organization's QA manager, for the assessment. The assessee
should inform the assessment team of any necessary procedures for admittance to the assessment site
and any safety procedures. If the assessment will involve Confidential Business Information (CBI), the
assessee should notify the assessment team leader so that the CBI process can be initiated. The
assessee may also provide information about travel logistics and local accommodations.
The assessee should arrange for appropriate personnel to be present at the opening and closing
meetings and available for interviews. Assessment interviews generally last for one hour. The assessee
should have adequate space available for the meetings, interviews, and document reviews. While some
documents, records, and files may be sent to the assessment team ahead of time, others may need to be
readily accessible during the on-site portion of the assessment. It may be appropriate for the
assessee's QA manager or other designated point of contact to brief the assessee's senior management
prior to the on-site assessment.
The assessee generally will want to inform their personnel of the impending assessment and
arrange for their participation in the assessment. Ideally, the assessee should convey a positive attitude
about the assessment and the assessors. Staff members should understand that the quality system, not
the interviewees, is being assessed. The assessee may want to perform self-assessments in preparation
for an independent assessment. These self-assessments should keep the staff aware of assessment
procedures and encourage maintenance of necessary documents and records.
Occasionally, an unexpected event occurs, and there is a sudden change of plans. The
assessment agenda may need rearrangement, or there may be a substitution of personnel resulting from
illness. The assessment team should be notified of these changes as soon as possible.
EPAQA/G-3 A-3 March 2003
-------
What should the assessment notification and assessment plan contain?
The notification memorandum generally will identify the assessment team members and their
affiliations and define the assessment scope, the assessment criteria, assessment authority, and a
tentative schedule. The assessment plan should specify the authority for the assessment, the assessment
criteria, and the purpose and scope of the assessment. Details such as a schedule of assessment
activities, specific personnel to be interviewed, and documentation to be reviewed should be included in
the assessment plan. The assessment plan should clearly state the rules for dissemination of assessment
findings and confidentiality for the particular assessment. Ideally, the assessee should receive written
notification and the formal assessment plan at least two weeks before the assessment or in enough time
to schedule the interviews and to collect the documents to be reviewed.
Will the assessment cover only the points specified in the assessment plan?
The assessment plan provides a comprehensive approach to the assessment, based on the
assessment team's understanding from reviewing relevant quality system documents before the
assessment. During the assessment, however, the assessment team may realize that there are other
aspects of the quality system that need additional attention. Minor changes may need to be made to the
assessment plan, which will be documented by the assessment team and discussed with the assessee's
management. If the organization's quality system is not fully implemented, the assessment may be
focused on promoting its development, rather than listing its deficiencies.
What can I expect to occur during the opening meeting? What do we talk about during the
meeting? Who is coming to the meeting from the assessor side?
The opening meeting is generally attended by the assessee's QA manager, senior staff, other
staff as appropriate, and the assessment team. At the opening meeting, all assessee personnel and the
assessment team should introduce themselves. Typically, the assessment team will briefly discuss the
assessment scope and criteria. The assessee should be prepared to ask any questions that they have
and to respond to questions from the assessment team. Although an assessee may feel anxious about
the assessment, the assessment should be approached as something that will benefit the assessed
organization. The assessors can look at the quality system objectively and provide assistance to the
organization based on experiences from other assessments. This approach helps to ensure that the
assessment will promote improvements in the quality system.
What can I expect to happen during the assessment?
During the assessment, the staff will be interviewed as specified in the assessment plan. The
quality system is the focus of the assessment, rather than the individuals in the organization. There is no
need for the interviewees to feel that their job performance is being judged. Management can set an
example for the staff by projecting a positive attitude toward the assessment and the assessors.
Staff members should cooperate with the assessment team during the assessment. They should
respond truthfully and fully to the assessor's questions. Their responses should remain focused on the
topic of the question and not include tangential material. It is possible that an assessor may
misunderstand a particular response. In that event, a respondent should correct any apparent errors in
EPAQA/G-3 A-4 March 2003
-------
the assessor's understanding. An appropriate question to the assessor may help to clarify the
assessor's understanding. Remember that the interviewees are more familiar with their quality system
than the assessor, who is attempting to cover a lot of material in a short time.
Documents and records, as specified in the assessment plan, should be reviewed to verify
evidence of compliance with the quality system specifications. Files are generally examined to find
relevant data and records and to confirm information collected during interviews.
During the assessment, the organization's quality manager can act as liaison with the assessment
team and can address any logistical needs that arise. If needed, the quality manager can provide an
escort for the assessment team while they are on site. As was discussed in Chapter 4, daily briefings
can be held when appropriate to encourage on-going communication between the assessment team and
the assessee.
What can I expect to occur during the closing meeting?
The closing meeting is generally attended by the same staff that attended the opening meeting.
At this meeting, the assessment team leader discusses the team's findings. If contrary evidence exists of
which the assessors are unaware, this is the time to present it. If the assessors have misunderstood
anything, this is an opportunity to offer correction. If the assessors have requested information during
the assessment that was not immediately available, the assessee should note this request and provide
the information on a realistic timetable. If the information will not be available when needed by the
assessors, the assessee should state candidly why it is not available.
How will the assessment be reported?
Many assessment organizations prepare a written draft assessment report for review by the
assessee, while others present an oral report at the end of the on-site portion of the assessment in lieu of
a written draft report. In either case, the assessment team should prepare a written final report, which
incorporates any relevant comments from the assessee when appropriate, corrects any identified factual
errors, and resolves any disputes if possible. After final approval for the report is received from the
authorizing entity, it should be distributed as previously agreed in the assessment plan
A written draft report, when that approach is used, provides the assessee with an opportunity
to comment on the report before it is finalized, but this approach does take additional time. The
assessment team will generally send the draft report to the assessee for review after it has been
reviewed by the assessing organization and authorizing entity. This is an opportunity for the assessee to
correct any factual errors in the report. The assessee's review can be thorough, but timely. If the
assessee does not respond in a timely fashion, the assessment team leader should contact the assessee
QA manager or designated point of contact. The assessee should complete the corrective action plan
(if one is attached to the draft report) and include the planned corrective action, responsible party, and
due date. The corrective action plan may be submitted after the assessee's comments on the draft
report, particularly if some issues still need resolution. The confidentiality and dissemination of
assessment findings and reports should have been decided and agreed to during the assessment
planning process, and the agreement should be documented in the assessment plan.
EPAQA/G-3 A-5 March 2003
-------
For some assessments, the assessment criteria and issues may be so straightforward as to
permit concluding the assessment on site with a presentation of a streamlined report, which does not
need additional explanation. This method, like more conventional reporting, would warrant up-front
agreement with both the authorizing entity and the assessee.
How do I address any problems with the assessment findings? What happens if I disagree
with them?
Any disputes over the assessment findings and the draft report are usually resolved at the lowest
administrative level possible and in accordance with the dispute resolution process for the assessment
program. It may be necessary for the assessment program manager or the authorizing entity to play a
role in dispute resolution. If any serious problems are noted by the assessee during the assessment,
such as inappropriate assessor behavior or release of confidential information, the assessee should
notify the management of the assessing organization.
What do I do after the assessment?
In addition to reviewing the assessment report, the assessee is responsible for developing,
implementing, following up on, and tracking corrective actions. The assessment team may provide
assistance and check with the assessee to follow up, but the assessee is responsible for the quality
system and any improvements to it.
How will the assessment findings be distributed? What about confidentiality?
Procedures for distribution and confidentiality of the assessment report should be agreed to
ahead of time by the assessment team, the assessee, and the authorizing entity and documented in the
assessment plan.
EPAQA/G-3 A-6 March 2003
-------
APPENDIX B
INTERVIEWING SKILLS
Communication skills can be easily overlooked or underappreciated, but in conducting
assessments they may be as important as technical skills. The goal of the assessment interviews is to
generate data that are reliable, unambiguous, and of the type, quality, and quantity needed to meet the
objectives of the assessment. During an assessment, interviews will help the assessment team
understand if, how, and to what extent the policies and procedures have been communicated,
understood, and implemented. Interviews should be supplemented by documentation reviews, which
aid in verifying the existence, implementation, and effectiveness of the actual policies, processes, and
procedures.
Barriers to effective communication include:
personal or collective biases toward particular people, ideas, or procedures
lack of feedback
poor listening skills
misunderstanding of nonverbal clues
distractions
personality conflicts.
Nonverbal behaviors, such as facial expressions, posture, tone, inflection, position in the room,
gestures, and silence, make a difference in the interviewee's perception so it is important that the
interviewer be aware of his or her own nonverbal behavior and the messages that are being sent to the
interviewee. The interviewer should also observe the nonverbal behaviors of the interviewee, but only
within the context of the interview. It is important to neither dismiss nor overinterpret any nonverbal
communication and to note that interpretation of body language is not objective evidence.
Active listening is an important part of interviewing. Compared to simply listening, active
listening takes a great deal of effort. Active listening involves verbally responding, with the listener
mirroring back the speaker's message to further clarify understanding. This lessens the possibility of
false assumptions and leads to more accurate interview notes. Active listening has physical, mental, and
motivational aspects. The physical aspects may include making eye contact with the interviewee and
nodding to indicate understanding. The mental aspect forces the interviewer to pay attention to what
the interviewee is saying. Examples of the motivational aspects of active listening include responses
such as "I understand" and "That's interesting, could you elaborate further?" An important aspect of
active listening is to stop talking and to position yourself to direct your attention to what the speaker is
saying.
Appropriate feedback during active listening can include:
neutral ("I see. Please go on . . .")
clarifying ("I'm not sure I understand . . .")
paraphrasing ("So in other words, you are saying that. . .")
EPA QA/G-3 B -1 March 2003
-------
impression checking ("I get the impression that. . .")
summarizing ("Okay. To sum up . . .").
Feedback can be very important, especially if you are receiving nonverbal cues that do not match the
verbal message that you hear.
Interviews generally consist of three steps: opening, questions and comments, and summation
and closing. The opening includes introductions, small talk, explanations (for instance, an explanation of
the assessment objective), and agreement to continue with the interview. The point of the opening step
is to help the interviewee feel at ease and to keep the process a "no surprises" one. During the
interview process, the interviewer should ensure that the interviewee understands the meaning of the
questions as intended. Additional explanation or checking may be needed for assurance of
understanding. After asking all of the interview questions, the interviewer should summarize the main
issues and close the meeting, allowing the interviewee to ask any final questions, ask for clarification of
any points, and make any closing statement.
During the planning step of the assessment, as described in Chapter 3, careful consideration of
the types of information that is needed leads to a decision about the types of questions to ask. Four
types of questions, which may be appropriate for use during an assessment, are summarized in
Table B-l.
Table B-l. Types of Questions
Type
Open-ended
Directive
Leading
Hypothetical
Description
Designed to prompt the speaker to
provide detailed information
Leads the speaker to one of two
choices
Hints at the answer the interviewer
is seeking
Questions that place the interviewee
in a hypothetical situation
Example
"What is the role of technical experts in
planning your office's projects?"
"If you had to choose a method, would you
choose the EPA method or another one?"
"Working with too little QA support doesn't
bother you, does it?"
"If you were in charge of the support
contract, how would you change the
specifications for QA?"
No matter which type of question is selected, the questions used in the interview phase of an
assessment should be simple and understandable, brief, thought-provoking, limited in scope, and
unbiased. It is important to remember that the way a question is phrased will greatly influence how it is
received by the interviewee.
When possible, given the staffing, time, and other resource constraints, many assessment
programs prefer to have two assessors participate in all interviews. One person can ask questions and
lead the discussion including thinking of follow-up questions, while the other assessor can listen more
EPA QA/G-3
B-2
March 2003
-------
carefully and record responses. The two assessors can switch between these roles. They can ask
questions in a "tag team" alternation in which the listener for one question is preparing to pose the next
question to the interviewee. It is important to remember that the goal is not to complete the checklist,
but to use the checklist to obtain the desired information. The interviewee should be allowed time to
reflect and answer the question fully.
There are a variety of difficult interview situations that the assessor might encounter. One of
these is an apprehensive interviewee. The characteristics of this interviewee may include an unsteady
voice or a "frozen" look. It is human nature to be apprehensive in an assessment situation, which is why
it is important to include introductions and small talk in the interviews. This behavior does not
particularly mean that the person "has something to hide" and usually the interviewee will become
helpful after their apprehensions have passed.
Another potential problem is a defensive interviewee. This person may give apprehensive
responses, short comments, and may seem concerned about impressions. It is important that the
interviewer recognize his or her effect on the interviewee.
Sometimes, the interviewee may be too talkative, with many digressions and long-winded
responses. After the first digression, wait and allow the interviewee to talk for a while before
rephrasing the question and trying again. After the second digression, interrupt and clarify. While the
interviewer should be careful to not harm trust or risk any established rapport, no further digression
should be allowed.
Another potential problem can be a disorganized interviewee. This person might seem easily
confused or distracted. The first step is to determine if the interviewee is disorganized by nature or if he
or she is confused by the topic or the way the question is asked. If the latter is the case, further
explanation or rewording of the question may resolve the problem.
An arrogant interviewee is characterized by short and sharp answers, acting too busy, and glib
or cute responses. This person may be motivated by fear or nervousness. It is important that the
interviewer keep control of his or her ego and not lose control of the situation. The goal of the
interview is to obtain quality information.
A hostile interviewee may withhold information or provide worthless information. The hostile
interviewee may show open fear or anger or may seem impatient. If possible, the interviewer should
determine the reason for the hostility and if there are "hidden objectives" on the interviewee's part. If a
particular topic seems to evoke hostile behavior, the interviewer should leave this topic until rapport has
been reestablished. The interviewer may decide to end the interview if the hostility does not end, after
consulting with the assessment team leader, if possible.
EPAQA/G-3 B-3 March 2003
-------
EPAQA/G-3 B-4 March 2003
-------
APPENDIX C
EXAMPLE ISSUES WITH INTERVIEW QUESTIONS
The following six sets of example interview questions are representative of the questions that
might be asked about assessment issues. The sets alternate between questions that are appropriate for
a developing quality system and those that are appropriate for an implemented, and therefore
documented, quality system. Because more information about implemented quality systems is available
to the assessor before the interviews, the questions about these quality systems reflect more of the need
to confirm existing quality processes rather than to gather information about them. These questions are
tailored for three example quality system roles: senior manager/QA staff supervisor; manager/staff; and
field sampler. It is expected that an appropriate number and appropriate types of personnel would be
selected to assure adequate coverage of the assessment issues. When EPA is determining which
questions to ask in an assessment of a non-EPA organization, be aware that the Paperwork Reduction
Act (44 U.S.C. 3501et seg.) requires EPA to obtain OMB approval before posing identical questions
to ten or more persons in a 12-month period.
A. Senior Manager/QA Staff Supervisor for a Developing Quality System
Interviewee's Background and Role in the Quality System
Verify the interviewee's name, title, and organizational unit, if necessary. Note the
date and time of the interview.
How do you ensure the quality of environmental data collected and used by your
organization?
Quality System Context, Resources and Documentation Status
• What quality system functions, for example, project planning, oversight, and record
keeping, are critical to your organization's data collection and use?
• What resources have been allocated for the development of the quality system?
Where is the QA manager/staff in your organizational structure?
What functions are being performed by your organization's QA manager/staff?
What is the current status of the documentation of the quality system?
Training Policy and Resources
• Describe your background in QA principles and procedures.
• How do you assure that your staff is familiar with your quality system?
• How are the needs of the staff for QA training assessed and met?
Systematic Project Planning and Documentation
Describe your organization's systematic process for project planning.
Who participates in the planning process?
• How is the planning process documented?
• What is the process for review and approval of QA project plans?
EPA QA/G-3 C -1 March 2003
-------
Additional question areas could include project implementation and oversight, project- and
system-level assessments, etc., based on the assessment objectives and issues.
B. Senior Manager/QA Staff Supervisor for an Implemented Quality System
Interviewee's Background and Role in the Quality System
Verify the interviewee's name, title, and organizational unit, if necessary. Note the
date and time of the interview.
What is your role in the quality system?
Quality System Communications and Resources
• How (and how often) do you communicate with the QA manager/staff?
• What input on the quality system do you receive?
• How is the adequacy of QA resources assessed?
What input from the QA staff is considered in resource planning?
Quality System Assessment
How are internal assessments planned and scheduled?
• How are assessments reported?
• Who develops and implements corrective actions in response to assessment findings?
• How are disputes handled?
• How are corrective actions tracked to completion?
Quality improvement
How do you assure ongoing improvement of your quality system?
Additional question areas could include oversight of assistance agreement holders, and
contractors, resource issues concerning compliance, or other issues within the scope of the
senior manager's direct responsibilities.
C. QA Manager/Staff in a Developing Quality System
Interviewee's Background and Role in the Quality System
Verify the interviewee's name, title, and organizational unit, if necessary. Note the
date and time of the interview.
Describe your training and experience in quality assurance.
What additional QA training would be helpful to you?
• What is your role in the organization's planning for, collecting, and using
environmental data?
• To whom do you report in the organization?
• What portion of your job is in quality assurance?
Quality System Implementation Status
What is the current status of the development of the quality system?
What functions do the quality system now perform in the data collection and use
process?
EPAQA/G-3 C-2 March 2003
-------
• What quality system functions are critical to the data collection and use process?
• What QA support do you provide to managers/decision makers and to staff? What
additional QA support are you developing for them?
• What is your role in writing the QMP?
How does the organization's management support the development of the quality
system?
What resources have been allocated for the development of the quality system?
What external support, if any, would aid the development of the quality system?
Training
• How are the training needs of your organization assessed?
• What QA training is provided currently to project officers and staff?
What additional QA training for project officers and staff are you developing?
What additional QA training would you like to see made available?
What is the organization's policy regarding training the staff in QA principles and
procedures?
Systematic planning
• Describe the process used in research program and project planning.
• How does the organization address the needs of data users and decision makers
during planning?
What technical support, tools, or expertise (e.g., statistical, field, laboratory) are
available or needed for planners?
Additional question areas could include project implementation and oversight, project- and
system-level assessments, record keeping, etc., based on the assessment objectives and
issues.
D. Quality Assurance Manager/Staff in an Implemented Quality System
Interviewee's Background and Role in the Quality System
Verify the interviewee's name, title, and organizational unit, if necessary. Note the
date and time of the interview.
• Describe your training and experience in QA.
QA Line of Reporting and Independence
• To whom do you report on QA matters in the organization and who appraises your
performance?
• Describe your current position, especially any duties that relate to environmental data
collection or use.
What portion of your job is devoted to QA duties?
How is your QA responsibility reflected in your performance agreement?
Training and Communications
• How and how often are the needs of the staff for QA training evaluated?
EPAQA/G-3 C-3 March 2003
-------
• How is QA training being tracked?
• How are the QA training needs satisfied?
• How are new or changes to QA policies and procedures disseminated to the
organization?
Quality System Assessments
Describe the management support for, the process for, and the frequency of internal
assessments of the quality system.
• How have the assessments improved the quality system?
• How are corrective actions tracked?
• Have there been instances in which the quality of environmental data has been
challenged? If so, what was done to investigate the quality of the data and to
respond to the challenge? What was learned about the quality system?
It is expected that documentation would be produced and examined to substantiate
responses where appropriate.
Additional question areas could include quality system documentation, project planning,
implementation and oversight, etc., based on the assessment objectives and issues.
E. Field Sampler in a Developing Quality System
Interviewee's Background and Role in the Quality System
Verify interviewee's name, title, and organizational unit, if necessary. Note the date
and time of the interview.
What is your role in the process of planning for, collecting, and using environmental
data?
• What training have you received in QA principles and procedures?
• What additional QA training would be helpful?
Quality System Support
What support is provided currently by your organization's QA manager/staff?
How can your organization's QA manager/staff help you further?
Quality System Documentation
• How do you plan field sampling?
• What QA and QC activities in your field sampling are documented? Explain any
process for writing, reviewing, approving, modifying, and controlling the version of
these documents.
Additional question areas could include oversight, record keeping, etc., based on the
assessment objectives and issues.
EPAQA/G-3 C-4 March 2003
-------
F. Field Sampler in an Implemented Quality System
Interviewee's Background and Role in Quality System
• Verify the interviewee's name, title, and organizational unit, if necessary. Note the
date and time of the interview.
Training and Communications
• What QA training have you received?
• How are your training needs assessed and satisfied?
• Where and how are training records kept?
• How do you receive updates to or new QA policies and procedures?
• Describe your access to and/or support from QA staff.
Quality System Documentation and Record Keeping
Describe your role in developing and implementing QA project plans or SOPs for
field sampling.
• What is the process for review and approval and/or changes?
• What other QA documentation do you use (e.g. field notebooks, chain-of-custody
forms, etc.)?
• What is the process for review and approval and/or changes?
What is the process for record keeping of these documents during a project and
after completion?
Implementation and Oversight
• How do you get instructions and training for field sampling for a new project?
• What QC checks are done as part of field sampling?
• How and by whom is the information used?
• What type of oversight is done of your field work?
• Who decides what will be done and when?
• Who oversees the field operation and what is done with the information?
Assessments
Have there been assessments of the field sampling program? When was the most
recent assessment? Who did the assessment?
EPA QA/G-3 C - 5 March 2003
-------
EPAQA/G-3 C-6 March 2003
-------
APPENDIX D
EXAMPLE CHECKLIST
Interviewee:
Assessment of a Quality System
Job Category:
Interview Date:
Time
Organization:
Assessor:
Assessing Organization:
Issues and Questions
I. Management and Organization
A . How is management' s commitment
to the quality system
demonstrated?
B. How are the quality policies that
describe the organization's attitude
towards quality defined and
documented?
C. How is the structure that
management will need to manage the
quality system defined and
documented?
D. How are the procedures that
program managers and supervisors
can use to review the effectiveness
of the quality system defined and
documented?
E. How do you oversee the quality
system?
F. How do you document
identification of verification
specifications and provision of
adequate resources including trained
personnel for all verification
activities?
G. How do you ensure that quality
assurance (QA) activities are
included in employees' job
descriptions?
II. Quality System Components
A. What is the status of development
of your quality system and a
manual that describes it?
Source of
Assessment
Criteria in QMP
Response/Comments
EPA QA/G-3
D-l
March 2003
-------
Assessment of a Quality System
Interviewee:
Interview Date:
Time
Job Category: _
Organization:
Assessor:
Assessing Organization:
Issues and Questions
Source of
Assessment
Criteria in QMP
Response/Comments
B. How do implemented quality
system procedures compare to the
quality policy?
C. Describe the preparation, review,
and approval process of the Quality
Management Plan (QMP). What
was your role in this process?
D. Describe how you developed,
designed, and documented QA
project plans.
E. How do you ensure that your QA
project plans are submitted are
submitted to EPA for review and
approval prior to initiation of any
data collection?
F. How do you ensure that the
standard operating procedures
(SOPs) are consistent with the
quality elements of the activities
and operational specifications?
G. How do you communicate the
QMP roles and responsibilities to
employees and supervisors?
H. How do you ensure that assigned
QA responsibilities are understood
and implemented?
I. Who has approved the QMP?
J. How do you conduct periodic
assessments of programs' quality
systems to assure compliance with
U.S. EPA specifications?
K. How do you ensure that
administration directors, program
managers, and quality coordinators
address all areas of concern in the
report of the self-assessment?
L. What have you submitted as a
Quality Assurance Annual Report
and Work Plan?
EPA QA/G-3
D-2
March 2003
-------
Assessment of a Quality System
Interviewee:
Interview Date:
Time
Job Category: _
Organization:
Assessor:
Assessing Organization:
Issues and Questions
Source of
Assessment
Criteria in QMP
Response/Comments
M. How do you ensure that
administration directors, program
managers, and quality coordinators
approved of the annual report?
N. Please describe the preparation,
review, and internal approval
process for the self-assessment.
O. Have you implemented the
following financial reports as
specified in the QMP:
1. Financial Reconciliation
(Control) report or the
Undrawn Analysis
Report?
2. Federal Grant Inventory
Report (FGIR)?
P. How do you identify and document
your managers',supervisors', and
employees' support for the
implementation of the quality
system described in the QMP?
Q. Describe how you identify, define,
and document the quality
information needed to monitor the
QMP's effective implementation?
EPA QA/G-3
D-3
March 2003
------- |