DRAFT #2 JANUARY 1994
GUIDANCE FOR
PREPARING, CONDUCTING, AND REPORTING
THE RESULTS OF
MANAGEMENT SYSTEMS REVIEWS
EPA QA/G-3
United States Environmental Protection Agency
Quality Assurance Management Staff
Washington, DC 20460
-------
950R94010
DRAFT #2 JANUARY 1994
TABLE OF CONTENTS
Page
Foreword ii
I. Introduction '. . 1
II. Planning the Review 8
III. Conducting the Review 23
IV. Evaluating the Results 31
V. Reporting the Findings and Recommendations 36
References 42
Appendices
A. Use of Process Flow Models in MSRs A-l
B. Flow Model of MSR Process , B-l
C. Terms and Definitions C-l
U.S. EPA Region HI
QA/G-3 MSR Guidance ii Regional Center for Environmental
Information
1650 Arch Street (3PM52)
Philadelphia, PA 19103 >
-------
DRAFT #2 JANUARY 1994
FOREWORD
The U.S. Environmental Protection Agency (EPA) has developed the Manage-
ment Systems Review (MSR) as an important tool for managers to measure the effec-
tiveness of Quality Systems applied to environmental data collection activities. Data
produced by air toxics and hazardous waste sampling or monitoring activities are used by
EPA extensively in problem definition, rule-making, and enforcement decisions. These
activities are supported through implementation of the Agency-wide Quality System,
which requires all organizations to develop and operate management processes and
structures for assuring that the data collected are of the needed and expected type and
quality for their desired use. The MSR has emerged as an effective means by which
senior managers may measure the effectiveness of the quality management processes
encompassing quality assurance (QA) and quality control (QC) applied to environmental
data operations, and which may provide both recognition of noteworthy accomplishments
and identification of needed improvements. The MSR is a process in which the planning,
implementation, and retrospective review phases of a project are examined and the
effectiveness of the quality management activities applied is measured. These elements
are consistent with the fundamental principles of Quality Management which have been
successfully applied in business and which form the foundation of the Agency's Quality
System for environmental data operations.
The purpose of this document is to provide general guidance to organizations on
evaluating the effectiveness of their quality system of QA/QC activities being applied to
environmental data collection. The guidance assumes that approved management
practices have been established and are operational.
This document is one of a series of Quality System requirements and guidance
documents that have been prepared by the EPA Quality Assurance Management Staff
(QAMS) to assist users in implementing appropriate quality management practices.
QA/G-3 MSR Guidance iii
-------
DRAFT #2 JANUARY 1994
CHAPTER I
INTRODUCTION
The Environmental Protection Agency (EPA) spends annually about $500 million
in the collection of environmental data for scientific research and regulatory decision-
making. In addition, the regulated community may spend as much as an order of
magnitude more each year to respond to Agency compliance requirements. In order to
assure that the collection and evaluation of environmental data will produce the type and
quality of data needed and expected for critical decisions, EPA has established the
requirement that quality management practices be applied to these activities and that
such practices be defined by a quality system1" for planning, implementing, and assessing
the effectiveness of such practices. EPA policy further requires that each EPA Quality
System be documented in a Quality Management Plan (QMP)^ that has been reviewed
and approved for implementation in Agency environmental programs.
The success of a quality system depends on its continuing effectiveness in meeting
the mission requirements of the organization to which it is applied. In order to ensure
the on-going effectiveness of the quality system, periodic management assessments are
necessary to provide managers with an evaluation of the program. EPA policy01 requires
that such management assessments be performed routinely as part of Agency-wide
oversight of environmental programs.
Quality Management Practices
Since the late 1980s, the application of recognized quality management principles to the
Agency's Quality System has produced new and effective tools to assist senior managers
in planning, implementing, and evaluating the results of environmental data operations.
One of these tools is the Management Systems Review (MSR) process, which enables
senior management to determine whether an organization's quality system is operating as
designed.
Continuous Improvement is a process used extensively in some areas of the. private
sector in which management philosophy and operating methodology are completely
committed to quality improvement in the organization or program. The elements of
continuous improvement are straight forward and embrace a common-sense approach to
QA/G-3 MSR Guidance i
-------
DRAFT #2 JANUARY 1994
management. These elements are key to an effective quality system and include:
a focus on the needs of the customer or client;
effective communication of customer needs among
all participants;
top management commitment, support, and direction; and
reliance on standards and measures of performance
to demonstrate satisfaction of customer needs.
In the case of EPA, every environmental data operation has a customer or
decision maker who uses the data collected, whether the customer is outside the Agency,
such as a regulated industry, or inside the Agency, such as a Region or Program Office.
Satisfaction of the needs of the customer largely determines the success of the data
operation. The use of measures of performance provides the basis for what is needed to
satisfy the customer. The other elements - effective communication, management
commitment, and reliance on performance measures - are reflected in the process and
structure used to implement the data collection operation and to measure its success. In
order to demonstrate how these elements have been institutionalized in the Management
Systems Review process, it is appropriate to briefly examine the origins of the EPA
Quality System and then the evolution of the MSR process itself.
Management Assessments in EPA's Quality System
Over a decade ago, EPA recognized the need to ensure that the data being used
for important decisions were of the needed and expected quality. EPA management
directed that all organizations engaged in any aspect of environmental data operations
should develop, implement, and periodically review the process and structure for
determining the quality of data produced by those operations(3>. This process and
structure is called the Quality System and is documented by EPA organizations in Quality
Management Plans (QMP).
Since its inception, the EPA Quality Assurance Management Staff (QAMS) has
conducted management assessments of the quality systems in various organizations to
ensure the effectiveness of the programs. From 1982 to 1986, QAMS conducted several
Management Systems Audits (MSA) of different organizations within the Agency. The
MSAs were largely guided by the organization's approved Quality Assurance Program
Plan, the predecessor of QMPs. These MSAs focused on single organizations and
QA/G-3 MSR Guidance 2
-------
DRAFT #2 JANUARY 1994
covered all of the major data collection activities in which the organization was involved.
QAMS found that the audits were very time and resource intensive, and, while they did
allow for a comprehensive examination of the audited organization, the MSAs do not
examine the critical QA linkages among different organizations participating in a single
data collection program. For example, an MSA conducted on a single Region would not
tell the National Program Manager of a major air monitoring program how effectively
quality management practices had been implemented in this Agency-wide program.
Consequently, the manager may not know how consistently the quality assurance (QA)
and quality control (QC) processes were applied by the participating Regions, thereby
creating uncertainty in the decision process. Moreover, the time required to conduct an
MSA, which carefully examined all of the principal data collection programs, was often
several months, frequently causing the findings to be outdated. For these reasons,
QAMS believed that the MSA concept should be revised to ensure greater timeliness and
usefulness to management.
To address these concerns, QAMS developed the MSR as an alternative manage-
ment assessment tool. The difference in philosophy between the MSA and the MSR
may be seen in the definition of "audits" and "reviews," as follows:
AUDITS: Assessments of the conformance of systems to quantitative specifica-
tions.
REVIEWS: Assessments of the conformance of systems to qualitative requirements
or specifications.
The experience of doing MSAs has shown that management systems deal largely with
qualitative criteria; that is, management systems do not generally have quantitative
performance specifications to provide a measure of their effectiveness. The effectiveness
of a management system is generally measured using judgement based on non-technical
information assembled and analyzed. On the other hand, Technical Systems Audits
(TSAs) and Performance Evaluations (PEs) may utilize such performance specifications
defined for a particular technical activity or project to determine quantitatively the
success of the project in meeting technical and quality objectives. That is, given a set of
specifications, conformance of the technical measurement system to those specification
can be measured quantitatively. Consequently, the term Management Systems Review
has been selected to describe the process of qualitatively assessing the effectiveness of
management practices in applying QA/QC to environmental data operations.
QA/G-3 MSR Guidance
-------
DRAFT #2 JANUARY 1994
Approach to Management Systems Reviews
The MSR process is designed to provide more accurate and complete assessments of
how well quality management practices are functioning. Through cooperative interac-
tions and discussions with the organizations involved in data operations, the MSR
process:
identifies the critical linkages among the participants in the data collection
program which are necessary to assure that the quality of the data meets
established requirements,
does not evaluate data; it does not examine or pass judgement on the
quality of the environmental data or on the decisions based on those data,
and
establishes where the Quality System is working well and where improve-
ments should be considered by management.
The MSR process will study only the STRUCTURES of the management systems and the
PROCESSES by which they are implemented. The guiding principles for MSRs,
consistent with accepted quality management principles, are listed in Table I.
The MSR process is a management tool and is designed to be used by managers
to understand and evaluate the technical activities that they must manage. This guidance
document describes the application to the MSR process to quality systems supporting the
collection and evaluation of environmental data, but it can be applied generally to any
management system that needs to be assessed.
For EPA Quality Systems, the MSR process used by QAMS as part of its Agency-
wide oversight responsibility to perform periodic management assessments of EPA
organizations and data collection programs. In addition, the MSR process is used by
senior managers to assess the effectiveness of the quality system(s) applied to their
organization or to major data collection programs involving multiple organizations,
including the National Program Offices, Regions, States, or Office of Research and
Development (ORD) laboratories.
The use of Process Flow Models in the MSR has been shown to significantly
improve the understanding of the process being reviewed. As a result, Flow Models are
encouraged as an integral element of the MSR. The use of Flow Models in MSRs is
described in Appendix A. The complete MSR process described in this guidance is
shown in a Flow Model in Appendix B.
QA/G-3 MSR Guidance 4
-------
DRAFT #2 JANUARY 1994
TABLE I
GUIDING PRINCIPLES FOR MSRs
No surprises - keep all participants informed
through open and effective communications:
Apply the "So What" Test to every finding - be sure
that the finding is significant and that value and
benefit are added as a result of the finding.
Seek to present everything from the positive.
Provide written drafts for comments.
Seek reviewee ownership in the review, the deter-
mination of findings, and in the development of
recommendations.
MSRs have successfully employed the principles listed in Table I. In each MSR,
the findings identified and applauded those aspects of the program being done well in
addition to those areas. This was equally as important as identifying any areas where
some improvement might be warranted. Herein lies the key difference between the
MSR and its predecessor, the MSA: MSRs presume that every operation or process can be
improved. Identifying areas of improvement is intended to add value or benefits to the
process, not to assess blame for any problems. No one "flunks" in an MSR. The MSR is
designed to be a learning experience, not a "report card."
Perhaps the greatest obstacle to the success of the MSR process is the perception
by those being reviewed that it is an audit and that it will seek to identify problems and
where to assign blame for the problems. Such a perception, not surprisingly, creates
considerable anxiety and defensiveness. This is why the MSR approach emphasizes
ownership by the reviewees and a "no surprises" attitude by the reviewers. Results are
presented in a positive tone so that no one will feel threatened or blamed. As noted
earlier, identifying those parts of the reviewed program that are being done well is
equally as important as noting where improvements are needed. Again, the MSR is an
QA/G-3 MSR Guidance 5
-------
DRAFT #2 JANUARY 1994
assessment tool and its' purpose is quality improvement: rinding better ways to do a job
the next time.
Types of MSRs
There are two types of Management Systems Reviews generally performed. They
are:
Institutional MSRs
Reviews which focus on the entire QA program of a single organization and
measure the effectiveness of the management systems applied to all major
environmental programs undertaken by that organization.
Programmatic MSRs
Reviews which focus on a single environmental program and measure the
effectiveness of the management systems applied to that program by multiple
organizations.
When Are MSRs Needed
EPA policy requires that periodic management assessments of Agency Quality
Systems be conducted. The type, scope, and frequency of an assessment should be
determined by management in accordance with the organization's approved QMP. As
noted earlier, «QAMS conducts MSRs of Agency Quality Systems at the direction of the
Assistant Administrator for Research and Development (AA/ORD), who is the senior
quality management official for EPA. The AA/ORD has determined that each EPA
organization collecting environmental data should be reviewed at least every three years.
For consistency throughout this guidance, the person or organization for whom the
MSR is performed shall be referred to as the client
The process involved in carrying out the MSR is basically the same for both types.
The MSR process has four distinct phases:
Planning
Data Gathering
Analysis of Findings
Report Preparation
QA/G-3 MSR Guidance 6
-------
DRAFT #2 JANUARY 1994
The time required to plan, conduct, and report the results of the MSR will vary accord-
ing to the complexity of the data operation being studied and the number of participants.
In some cases, it may be necessary to examine only a representative group of participants
due to resource and time constraints. This is particularly true for programs involving the
Regional Offices.
QA/G-3 MSR Guidance
-------
DRAFT #2 JANUARY 1994
CHAPTER II
PLANNING THE REVIEW
The purpose of the Planning Phase is to define the scope of the review and obtain
concurrence on the scope from the requesting office or client. The scope of the study is
determined through discussions with the client. The steps which follow describe the
sequence of planning activities which are recommended for effective MSRs. A Flow
Model of the Planning Phase is given in Figure 1 at the end of this chapter.
Step 1: Identify the MSR Purpose and Client
V
The purpose of an MSR in general is to understand how well a particular quality
system applied to environmental programs is functioning. In practice, however, there is
usually a specific objective for an MSR; that is, what do you wish to learn about a
process from the MSR. Such an objective is usually defined by the client.
It is essential that a specific manager be identified as the principal client for the
MSR. This manager will participate in the scoping of the MSR and will be the recipient
of all reports produced by the MSR. There may be situations in which the client is a
particular group or organization. Here again, it is essential that one individual be
designated as the principal client and the recipient of the products of the MSR. In most
instances, the principal client will be the manager who has requested the MSR.
Step 2: Determine Scope of MSR and Issues/Questions to Be Addressed
There must be a purpose and a subject for the MSR. This step identifies the
scope of the study, including the program (or programs) to be reviewed. Conceptually,
this step is analogous to the scoping steps of the Data Quality Objectives (DQO)
process.<4) The client is asked to formulate specific issues or questions to be addressed by
the MSR. Typical questions which have been used in MSRs include:
How effective is the Headquarters guidance on QA for this program?
QA/G-3 MSR Guidance 8
-------
DRAFT #2 JANUARY 1994
What guidance is being used to plan and implement the sampling program?
How are environmental data being used in program decision making?
What mechanisms are being employed to provide or obtain analytical
services?
Such general questions provide a starting point for discussions with the client (senior
manager) in order to develop more specific questions for the MSR. In many cases, the
client will have a clear set of goals for the MSR. In others, however, the specific
questions may not be obvious. This is particularly true in those cases in which the client
is learning about the program and wants to understand how the program actually
operates. In this situation, the MSR will not likely seek to answer specific questions, but
will attempt to obtain sufficient information about the program and its operation to
enable the client to get a clear understanding of the process. As will be discussed later,
the Process Flow Model is especially helpful in these cases.
Planning for each MSR should be documented in a written plan that details what
the MSR will accomplish and how it will be done. The initial discussions with the client
and determination of the scope and the issues/questions to be addressed by the MSR
provide the beginning of the written MSR plan, which will be discussed in more detail in
Step P-7.
Step 3: Identify Review Team for the MSR
The composition of the review team is critical and its selection must be made
carefully. The necessary characteristics of an effective review team are given in Table II.
It must be noted that individual members of the review team may not possess all of these
characteristics; however, it is strongly recommended that the team as a whole have them.
The review team must be technically qualified in order to establish its credibility with the
reviewee. As before, this does not mean that individual team members must be experts
in the subject program being reviewed; however, team members should be able to
understand the underlying principles of the program. For example, if the program being
reviewed involves extensive field sampling and laboratory analyses, then the review team
should include members knowledgeable about the type of field sampling involved and
about the chemical analyses required.
There should be no conflict of interest, either real or perceived. The review team
should not include anyone from the organization being reviewed, except as an observer.
The size of the review team should reflect the scope of the MSR, the constraints
of resources, and the allowed schedule. Again, considerations analogous to the DQO
QA/G-3 MSR Guidance 9
-------
DRAFT #2 JANUARY 1994
process should be addressed in selecting the membership of the review team. As will be
discussed in more detail in a later chapter, there should be a minimum of two interview-
ers for each interview team. As will be discussed later, having only one interviewer could
introduce undesirable complications and should be avoided. If multiple organizations are
to be visited, multiple interview teams may be appropriate in order to save time by con-
ducting simultaneous interviews. In addition, the number of team members may be
influenced by the magnitude of the MSR; that is, a review of a large data collection
program involving several Regions could require a larger review team than a review of a
small program at a single research laboratory. The resources available to perform the
MSR may place practical limits on the number of team members, and there may be
constraints imposed in terms of schedule. Historically, MSR teams have had as few as
two members and as many as ten.
CHARACTERISTICS OF
REVIEWTEAM
Members must be knowledgeable in technical principles pertaining to the subject;
of the MSR.
Review team should have no real or perceived conflict of interest
Minimum of two members per interview team is needed.
Members should have basic active listening and interviewing skills.
The Planning Phase of the MSR is critical. While the Quality Management Plans
will provide a blueprint for participating organizations' processes and structures, they do
not generally reflect the unique policies and procedures of individual data collection
programs. It is essential, therefore, that the review team become familiar with all
applicable guidance and procedures of the organization responsible for the program
studied. Where possible, the review team should include a representative of the
requesting office, preferably from the QA program. This fosters the spirit of participa-
tion and management "ownership" of the review. Moreover, it tells the reviewees that
senior management takes the review seriously and expects to learn from the MSR
experience.
QA/G-3 MSR Guidance 10
-------
DRAFT #2 JANUARY 1994
Step 4: Identify and Provide Training Needed by the Review Team
In seeking qualified review team members, it is generally not difficult to identify
candidates with the necessary technical skills. Unfortunately, finding candidates with the
needed communications skills may be more difficult. As shown in Table II previously,
team members should have basic active listening and interviewing skills. It is likely,
therefore, that some specialized training in these skills may be needed to prepare team
members for conducting effective interviews. The value of active listening will be
discussed in considerable detail in Chapter III. It will suffice here to emphasize that
communication skills are a critical element in being able to successfully conduct MSRs
and that training may be necessary for the MSR team to ensure that those skills are
present. .
Depending on the nature of the MSR and the experience of the review team
members, other training may be necessary to prepare the team for the MSR. This may
occur when the review team members are not fully knowledgeable about the organization
or program to be reviewed. Such training could encompass briefings to augment
available guidance or documentation on a program. There could also be formal lectures
on the program to provide a full in-depth discussion on the management processes used.
For example, a MSR of the Discharge Monitoring Report Quality Assurance (DMR/QA)
program in the Regions may be preceded by a familiarization briefing by the Headquar-
ters Office of Water on how the process is intended to work. All training should be
completed before commencing the first interviews.
Step 5: Identify Potential Sources of Information
The next step is to identify the possible sources of information for answers to the
issues and questions. For EPA programs, the major sources usually available are:
Existing documentation (e.g., QMPs, program descriptions, guidance
documents)
Interviews of program personnel
Case studies (e.g., examples of program outputs)
Existing documentation is usually the starting point for most MSRs. A listing of
the documents pertaining to the program(s) being reviewed should be compiled and the
means of acquiring the documents identified. While these documents will not actually be
used until a later step in the MSR process, it is wise to begin early to determine which
QA/G-3 MSR Guidance 11
-------
DRAFT #2 JANUARY 1994
documents will be needed and how to get them. It may be necessary for the client to
assist in the acquisition of these documents in some cases. This documentation will
provide the basis for answering the first of several key process questions: What is the
program or organization supposed to do?
Generally, the most useful information will be obtained from interviews of
program personnel. In this step, the personnel to be interviewed should be identified at
least by job titles or functions. Examples include:
Senior managers (e.g., division directors, office directors)
Middle managers (e.g., branch chiefs, section chiefs)
Project managers (e.g., project officers, technical managers)
Technical-level staff (e.g., chemists, engineers, hydrogeologists)
It is not necessary at this point to name specific individuals, rather the need is to relate
the information needed to specific types of people who can likely supply that informa-
tion. The interviews will provide the basis for answering the second key process ques-
tion: What does the program or organization say that it does?
Case studies, examination of related files and records, and examples of products
from the program provide tangible evidence of how the program actually operates. Case
studies also provide important information on trending and consistency in the execution
of the process. MSR experience has shown that the choice of which case studies to be
provided to the reviewers is best left to the reviewee. This will promote their ownership
of the MSR, but there is also a practical aspect in that the reviewers may not know which
case studies are applicable to this MSR. While there may be some potential for bias in
the case studies chosen by the reviewee, this is easily detected from the significant
divergence in story consistency from the interviews.
File reviews are an important means of assessing the completeness of required
documentation and for confirming the implementation of required management activities
as described in the QMP. Similarly, examples of products such as final reports, standard
operating procedures (SOPs), and assessment results are helpful in documenting the
effectiveness of activities prescribed in the quality system. This information will provide
the basis for answering the third key process question: What did the program or organiza-
tion really do?
QA/G-3 MSR Guidance 12
-------
DRAFT #2 JANUARY 1994
Step 6: Compile and Review Existing Guidance and Information
For every MSR, whether institutional or programmatic, there is existing informa-
tion available. Both the quantity and usefulness of the information may vary. In general,
existing documentation will not answer all of the MSR questions posed, but it will
provide two important contributions to the MSR process. They are:
Definition of the knowledge base of the review team; that is,
what is known about the organization or program being re-
viewed.
Provides a basis for developing specific interview questions or
identifying files, case studies, or assessment results that will
help to answer the overriding MSR questions; that is, what in-
formation is needed.
The review of existing documentation establishes the foundation for the study. It
is essential that the review team understand what is already known about the organiza-
tion or program in order to be able to formulate relevant questions for the interviews
and to identify pertinent case studies, files, or reports to be examined. In some situa-
tions, the credibility of the reviewers will be established by the maturity of their knowl-
edge of the organization or program. The organization being reviewed may already feel
threatened or intimidated by the expected visit by the review team. It is essential that
the reviewers have a sufficient understanding of the MSR subject to assure the organiza-
tion being reviewed that the MSR is relevant and is taken seriously. This understanding
comes from a thorough evaluation of the existing information. The questions to be asked
during the interview phase will be formulated to fill the gaps in the data base developed
from the existing information. It is important, therefore, that the review team have a
clear understanding of the existing information. During the interviews, the questions
must focus on the "unknown" elements and not ask for information that is already
available.
A review of the existing information must assess its relevance to the scope of the
review. Many environmental data operations are described by a wealth of documenta-
tion, but much of it may have no bearing whatsoever on the purpose of the MSR. The
review team may need to discuss the quantity and availability of existing documentation
with the client for the MSR. In some cases, such as that of a major program like Super-
fund, there may be too much existing information. Consultation with the client may help
to identify existing documents and information that are really relevant to the study. The
review team should carefully examine the relevance of a document to the MSR and plan
only to use those that are clearly applicable. In saying this, it is also recognized that
QA/G-3 MSR Guidance 13
-------
DRAFT #2 JANUARY 1994
relevance may be difficult to ascertain in some situations. In such cases, questionable
information should be retained until its relevance can be clarified later. When informa-
tion is clearly irrelevant or unusable, it should be labeled so and set aside.
The review team may discover that not all existing information is readily available
and that there may be some information unknown to them. The MSR client may be
instrumental in obtaining the needed documents or information, particularly if they are
not readily available at the review team's "home base."
Since all existing information may not be readily available, the review team must
allow ample time during the planning process to obtain the information and to review the
material. The review of this documentation is admittedly tedious but necessary. The
documentation may include the following for institutional and programmatic MSRs:
Institutional: Quality Management Plans, guidance documents, oper-
ating plans, mission planning documents, and previous
management assessment results.
Programmatic: Quality Management Plans, implementation guidance
documents, program descriptions, work plans, and
previous management assessment results.
During the process of reviewing the existing information on the MSR subject, the
review team should discuss their evaluation of the information among themselves. Often
one reviewer may recognize an important piece of data that another reviewer might
overlook. This point underscores again the value of having multiple reviewers. Such
discussions will also enable the review team to assess the value of the information to the
issues posed in the MSR. Again, not all information is relevant and the information
found not to be pertinent should be discarded from the study. It is usually helpful to
create some working notes to aid in the reviews and discussions. Such notes may include
written summaries of key guidance documents or briefings by the requestor. The notes
provide a means of documenting the source of specific information for reference during
the evaluation and reporting phase and a basis for completion of a formal review plan..
As noted earlier, sufficient time must be set aside to enable the review team to
conduct a thoughtful evaluation of the existing information. This may require several
weeks, depending upon the complexity of the study. If reviewers are used repeatedly and
gain MSR process experience, or are already familiar with the MSR subject, this period
may be shortened.
QA/G-3 MSR Guidance 14
-------
DRAFT #2 JANUARY 1994
Step 7: Develop Draft Plan for Conducting the MSR
When the information needs are fully defined, they are documented in a written
plan which outlines what is to be done, when it will be done, and who will be involved.
The draft MSR plan should contain the initial set of questions to be asked during the
interviews as well as a list of the files, reports, and case studies that the review team
wishes to examine on site. The plan should outline how the data gathered from the
interviews and from files, reports, or case studies are going to be used. Table III lists the
general structure of the MSR Plan. It should clearly present the results of the planning
process so that the reader will understand the subject of the MSR and what the MSR is
to accomplish. The plan serves as a check for the review team to assure that all of the
relevant questions and issues have been addressed during planning. If not, the draft plan
should be revised accordingly.
The proposed schedule in the draft MSR plan reflects the best estimate of the
review team for completing the MSR. The schedule should include the time needed to
conduct the site visits, evaluate the results, and prepare the draft finding report.
TABLE HI
GENERAL STRUCTURE OF MSR PLAN
I. Title and Subject of the MSR
II Purpose and Objectives
IIL Key Questions to be Answered '
IV. Organizations to be Visited and
Personnel to be Interviewed
V. Proposed Schedule
The MSR plan does not have to be a lengthy document. There is no established
criteria for length, but the plan should be sufficiently detailed to give the client a clear
understanding of the scope of the MSR.
QA/G-3 MSR Guidance 15
-------
DRAFT #2 JANUARY 1994
Step 8: Client Review and Approval of MSR Plan
Once the draft MSR plan has been completed by the review team and the team is
satisfied with its content, the plan must be presented to the client for the MSR for
concurrence and approval. This step is essential. The client must be given the opportu-
nity to review the scope of the proposed MSR and to be assured that the MSR will
accomplish the expected objectives.
The preferred approach is to provide the draft MSR plan to the client, followed
by a briefing. This will allow the client the opportunity to ask questions about the plan
or to discuss the rationale for the proposed approach. The client must be satisfied with
the approach before the MSR can proceed. If sufficient concerns or issues are raised, it
may be necessary for the review team to revise the plan and to re-submit it for approval.
In most cases, however, this situation is unlikely to occur if the review team has been
thorough in its planning. The briefing with the client also serves to engage the client in
the MSR process with the review team. All concerns or issues must be resolved with the
client before proceeding any further.
Step 9: Propose Schedule for Data Gathering and Notify the Reviewee(s)
When the MSR plan has been approved by the client, the review team should
propose a schedule for any needed site visits. Input from the client on the schedule is
very helpful so that obvious conflicts may be avoided. The schedule should be realistic
and should allow for some possible delays, particularly when visits to multiple sites are
necessary.
It is preferred that the notification of the organization(s) to be reviewed be
accomplished by a joint memorandum from the client and the review team leader. This
memorandum will:
state the purpose and objectives of the MSR,
list the review team members and their affiliations,
briefly describe the approach to be used and the expected time needed, and
• propose the schedule for site visits.
QA/G-3 MSR Guidance 16
-------
DRAFT #2 JANUARY 1994
The memorandum should also identify any case studies and reports needed, and files to
be examined, so that they can be compiled and be available when the MSR team arrives
at the site. In addition, the memorandum should schedule the entrance briefing with the
senior managers. Last, the memorandum should request that an office or conference
room be reserved for use by the review team during the on-site visit.
The joint memorandum is important. The client's signature establishes the
credibility and authority of the review team. The team leader's signature acknowledges
the independence of the MSR review team. The memorandum should be sent to the
organization(s) to be reviewed at least one month prior to the expected start of the site
visit(s).
At this time, the team leader should again review with the team members the
process for evaluating and reporting the results. This includes identifying to whom the
interim findings reports will be sent in the reviewed organizations for review before there
is any submittal to the client.
Step 10: Resolve Any Conflicts and Finalize Schedule for Data Gathering
After any schedule conflicts have been resolved regarding the site visits, the next
step is to finalize the schedule for the data gathering phase. The final schedule for the
MSR should include the expected numbers of interviews and briefings and the time
required for them. The schedule is intended to indicate to the management of the
organization to be reviewed that a thoughtful, systematic approach has been developed
and that there is a clear rationale for the actions proposed. The schedule will also
enable the reviewee to identify any conflicts or obstacles to the execution of the plan.
The finalization of the schedule for data gathering requires agreement by the
reviewed organization(s) for the timing of the site visits. The joint memorandum
discussed in Step 8 will propose the dates for the site visits and request confirmation of
the dates from the reviewed organizations. If there is a problem with a set of dates, it
may be necessary to re-examine the original schedule and resequence the site visits where
multiple sites are planned. To expedite matters, this should be done by telephone.
When all organizations to be reviewed in the MSR have agreed to the schedule, a
memorandum should be sent by the MSR team leader to confirm the final arrangements.
This memo should also re-confirm the MSR team's expectations of any case studies or
other documentation to be provided by the organization(s) to be reviewed.
QA/G-3 MSR Guidance 17
-------
DRAFT #2 JANUARY 1994
Step 11: Complete Logistical Needs for Data Gathering
When the final schedule has been approved by all organizations to be reviewed, it
will be necessary to complete the arrangements for travel to the site. As discussed in
more detail in the next chapter, a site visit may require a week at the site. Other MSRs
may require fewer days on-site. The actual time on site is determined by the complexity
of the study.
While it has not been mentioned previously, adequate resources must be commit-
ted to enable the MSR plan to be implemented effectively. This is part of management's
responsibility and commitment to the assessment process. It is assumed, therefore, that
sufficient funds shall be available to support required site visits.
QA/G-3 MSR Guidance 18
-------
DRAFT #2 JANUARY 1994
STEP P-l
INPUT: Request/Need for MSR
Identity the MSR
Purpose and Client
I OUTPUT:
Understanding of who is the Client for the MSR and why it is
needed.
STEPP-2
INPUT:
Purpose of the MSR
Determine Scope of
MSR and Issues to
be Addressed by the
MSR and Begin
MSR Draft Plan
STEPP-3
INPUT:
OUTPUT: Definition of Scope of the Study, including Technical Program to
be Reviewed
MSR Scope; Technical Program to be Reviewed
Identify Review
Team for the MSR
4 OUTPUT: Review Team
Figure 1. Flow Model of MSR Planning Phase
QA/G-3 MSR Guidance
19
-------
DRAFT #2 JANUARY 1994
STEP P-4
INPUT: Review Team; Scope of MSR
Identify and Provide
Training Needed by
the Review Team
OUTPUT:
STEPP-5
INPUT: MSR Scope and Issues
Review Team Trained for this MSR
Identify Potential
Sources of Informa-
tion Needed to Re-
solve MSR Issues
OUTPUT:
Sources of Information to Resolve Issues
STEP P-6
INPUT: Sources of Information Relevant to the MSR Subject
Compile and Review
Existing Guidance
and Information to
Determine Its Ap- ,
plkabiUtj to the
MSR
i OUTPUT: Information Relevant to Study
Figure 1 (Continued). Flow Model of MSR Planning Phase
QA/G-3 MSR Guidance 20
-------
DRAFT #2 JANUARY 1994
STEP P-7
INPUT: Relevant Existing Information; Scope of MSR
Complete
Draft
Plan for Conducting
the MSR,
including
Specific Interview
Questions
STEP P-8
INPUT:
OUTPUT:
Draft MSR Plan
Draft MSR Plan
Review MSR
Plan with Client,
Determine if
Recommenda-
tions are Expect-
ed, and Obtain
Client Approval
of MSR Plan
1 APPROVED
I
i OUTPUT: Client-Approved MSR Plan
(IF NOT APPROVED, REPEAT STEPS P-7 AND P-8 AS NEEDED TO ASSURE
CUSTOMER UNDERSTANDING AND APPROVAL)
Figure 1 (Continued). Flow Model of MSR Planning Phase
QA/G-3 MSR Guidance
21
-------
DRAFT #2 JANUARY 1994
STEP P-9
INPUT: Approved MSR Plan
Propose Schedule
for Data Gathering
and Notify Organi-
zation^) to be Vis-
ited
i OUTPUT: Schedule for Data Gathering
STEPP-10
INPUT: MSR Plan and Schedule
Resolve Conflicts
and Finalize Sched-
ule for Data Gath-
ering
OUTPUT: Final Schedule for MSR
STEP P-ll
INPUT: MSR Plan and Final Schedule
Complete Logistical
Needs for Data
Gathering
4 OUTPUT: Completed Arrangements for Site Visits and Data Gathering
Figure 1 (Concluded). Flow Model of MSR Planning Phase
QA/G-3 MSR Guidance 22
-------
DRAFT #2 JANUARY 1994
CHAPTER III
CONDUCTING THE REVIEW
Conducting the MSR at a site will usually involve interviews of participating
personnel and reviews of applicable files, documents, and case studies. In this phase,
interviews may be the most valuable source of information. Consequently, the review
team should be familiar with interviewing techniques so that information is obtained
effectively. MSRs should always be conducted in a collegia! and businesslike manner.
When visiting other organizations, it is very important to conduct entrance and exit
briefings with the senior management of the organization. Tell them what you are going
to do, then tell them what you did. Keep the lines of communication open at all times.
Last, thank the managers and staff for their time. It is courteous and acknowledges your
recognition of the value of their time and their contribution to the MSR. A Flow Model
of the implementation phase of the MSR is given in Figure 2 at the end of this chapter.
Table IV provides the general format for a typical site visit of five days duration.
Step 1: Conduct Entrance Briefing for Management
A successful entrance briefing with the senior accountable managers of the
organization to be reviewed is critical to the success of the MSR. The MSR team should
arrive for the entrance briefing on-time. It should be assumed that the managers
attending the briefing will be anxious about the MSR or perhaps even irritated at having
to spend time on the MSR. The MSR team leader should make every attempt to reduce
the anxiety level by focusing on the purpose of the MSR and by emphasizing that every
possible effort will be made to minimize disruptions.
During the briefing, the team leader will introduce the MSR team members and
will review the objectives of the MSR, the principal questions to be asked during
interviews, and the expectations from file reviews, case studies, and reports. This should
take no longer than 15-20 minutes. Afterwards, the managers of the organization being
reviewed shall be allowed to ask any questions about the MSR. As stated earlier, there
QA/G-3 MSR Guidance 23
-------
DRAFT #2 JANUARY 1994
TABLE IV
TYPICAL ON-SITE SCHEDULE
Day 1. Arrival; Entrance Briefing;
Begin Interviews
Day 2 Conduct Interviews and Recap
Day 3 Conduct Interviews and Recap
Day 4 Conduct Interviews;
Summarize Interviews
Day 5 Exit Briefing; Departure
should be no hidden agenda and no surprises. Questions should be answered truthfully
and without hesitation. The entire briefing should last no longer than 30-45 minutes.
At the conclusion of the entrance briefing, the host organization should provide
the MSR team with a list of prospective interviews and the schedule for the interviews.
It is recommended that the MSR team leader offer to provide the host senior manager
with a daily synopsis of the information obtained from the interviews and case studies.
Step 2: Interview Key Staff
The process of interviewing of the reviewed organization's staff is the most critical
step of the MSR process. Table V lists twelve elements of positive interviewing skills
that should be utilized during the interviews. It is preferred that the interviews be
conducted in a "neutral" location, such as a conference room or vacant office, rather than
in the interviewee's office. Each interview should last no more than one hour unless the
interviewee specifically wishes to continue. Then the interview should continue only for
another fifteen minutes. If more time is needed, then another time should be scheduled
for later. The limit of 75 minutes for an interview has a practical basis. The interviewers
must have some time between interviews to collect their thoughts and notes, and to
"recharge" for the next interview.
QA/G-3 MSR Guidance 24
-------
DRAFT #2 JANUARY 1994
Another practical consideration is that there should always be a minimum of two
interviewers. This enables a "tag-team" approach to the interviews; that is, while one
interviewer is asking a question and recording the response, the partner is able to listen
actively to the response and to formulate a more thoughtful follow-up question based on
the response. As has been emphasized throughout the MSR process, the questions
should be "open-ended" to enable the respondent to answer in a descriptive manner.
"Yes/no" and leading questions should be avoided. The use of two interviewers will help
to ensure that what was said is recorded accurately. The corroboration provided by the
second interviewer reduces the likelihood of anyone claiming later that a particular
answer was never given. Moreover, if there is any confusion about what was heard, the
two interviewers can discuss the response and agree on what was said.
Using only two interviewers helps to put the interviewee at ease. More than two
may cause the interviewee to feel outnumbered or surrounded. However, it may be
appropriate sometimes to use a third person to take notes provided that this person is so
identified at the start of the interview. An alternative may be to request the reviewee to
provide a note taker. This may increase the sense of "ownership" in the MSR by the
reviewee. An advantage of using a note taker is that it enables the interviewers to focus
more intently on the questions posed and responses given.
The attitude and "body language" of the MSR team must convey a non-threaten-
ing demeanor to the host organization. On rare occasions there may be some hostility
expressed by the host, but at no time may any member of the team respond to anger. If
the situation continues to deteriorate, the interview should be suspended. A short break
may enable the participants to calm down and "defuse" the situation. Afterwards, resume
the interview by restating the purpose and objectives of the MSR. If the situation still
does not improve, the review team should seek assistance from the interviewee's
management.
Team members should always remain calm and professional at all times. One of
the most readily perceived qualities by interviewees is attitude. Being in a situation of
potential conflict may require more mental fortitude by the reviewer than factual
knowledge. While a lack of knowledge of a subject may be compensated for by other
team members, attitude problems cannot. Each interview team member must contribute
to a harmonious demeanor in order to ensure an effective interview. They must be
supportive of one another and avoid disagreements in the presence of the reviewee.
At the conclusion of the interview, the discussion should be summarized briefly to
confirm the key points from the interview. Next, the interviewee should be thanked for
taking the time from his or her busy schedule to participate in the MSR. After the
interviewee leaves, the team members who conducted the interview should review their
QA/G-3 MSR Guidance 25
-------
DRAFT #2 JANUARY 1994
notes to identify any areas of question or misinterpretation and to assure than they are in
agreement on the notes compiled. It is recommended that at least 30 minutes be
scheduled between interviews to allow the interviewers to get prepared for the next
interview. After several interviews have been conducted and trends emerge in some of
the responses to the planned questions, this time may be used to revise the questions in
order to capture some different information or to drop some questions altogether.
Step 3: Conduct File Reviews and Obtain Case Studies/Reports
This step can be completed at any time during the site visit but prior to the exit
briefing. The reviews of relevant files may be most effectively conducted during periods
when interviews are not scheduled. This may occur early or late in the work day. If
known, a list of the files to be reviewed should be sent prior to arrival on site so that the
reviewee organization will have ample time to retrieve them.
The nature of reports and case study material needed is documented in the MSR
plan. Often the material is voluminous and will require considerable time to examine,
usually longer than the review team can afford to spend on site. If extra copies are
unavailable and it cannot be copied on site, the MSR team leader should ask to borrow
the materials long enough to return to the team's home base and have the material
copied. In either case, the team leader should make arrangements to have the case study
materials sent to the team home base.
The review team should anticipate that evidence or documentation not anticipated
during planning may be disclosed during the site visit. This additional information may
significantly augment the interviews and other evidence.
Step 4: Summarize Interviews and Compile Initial Impressions ,
Prior to the exit briefing for the organization's senior managers, the review team
should assemble its notes from the interviews and the file reviews, etc., and discuss the
initial impressions from the site visit. The purpose is to be able to provide the managers
with some indication of the results of the MSR. This does not mean that specific findings
should be developed, although it is likely that some impressions will be so strong as to
also become findings. As noted earlier, the presence of the review team creates anxiety
for the organization and it is important to provide them with at least some impressions
before leaving.
QA/G-3 MSR Guidance 26
-------
DRAFT #2 JANUARY 1994
I ABLE V
POSITIVE INTERVIEWING SKILLS
Linne Bourget, Ph.D.
1. Attending to another means turning off our tendency to evaluate, keeping an open mind,
and walking a mile in their mental moccasins.
2. Be willing to set yourself aside and focus totally on the other person. Shelve your negative
emotions.
3. Listen for other's points and feelings and respond to both, verbally or nonverbalry, to show
that you understand.
4. Ask open-ended rather than yes-no questions.
5. Ask why, what if, be more specific, give an example, what would be the best way to types
of questions.
6. Repeat back or paraphrase interviewee's responses, for clarifying or empathizing, including
facts and feelings. .
7. Be responsible for the communication--"! am not sure I understand, could you clarify?"
8. Ask additional questions as follow-up to obtain more information if necessary.
9. Keep as much eye contact as possible even though you are taking notes. Keep a
supportive facial expression.
10. Keep the interview moving, yet allow interviewee time to make her/his points.
11. Be sure to thank interviewee for time and input
12. Prepare an opening statement which explains clearly your purposes and process clearly and
sets a positive climate.
Used with Permission of the Author
Copyright 1991 ARR
Linne Bourget, PhJ>.
Positive Management Communication Systems
QA/G-3 MSR Guidance 27
-------
DRAFT #2 JANUARY 1994
The summary exercise also offers an opportunity for the review team to talk about
what has happened while they are on site. Such discussions may identify differing
interpretations of what was heard in some interviews and enable clarifications to be made.
It is advantageous to do this now while there is still access to the people interviewed in case
some follow-up is needed. For example, two interviewees may have provided conflicting
information on a particular topic. By identifying the conflict on site, it may be possible to
check back with the interviewees to verify the information.
Step 5: Conduct Exit Briefing
The exit briefing enables the review team to summarize for management what was
heard or found during the visit. Preliminary impressions may be given with the understand-
ing that additional review and evaluation of notes, reports, and case studies will be needed
before specific findings can be determined. The review team leader should be the principal
spokesperson for the team, but all members should participate in the discussions.
The briefing should be opened with a restatement of the purpose and objectives of
the MSR and of what information was expected from the interviews, files reviews, etc. Next,
list the people who were interviewed and provide a general summary of what they said in
response to the questions asked. At this point, it is appropriate to make some observations
about the information collected. During the briefing, it is very likely that the review team
will be pressed to provide conclusions; however, the team should be very careful to avoid
giving conclusions or detailed findings that could be inferred as conclusions.
At the end of the discussions, the review team leader should give an indication of
when the draft Findings Report will be available for review and explain what will happen
to the MSR results. Last, the team leader should thank the managers for their time and for
the cooperation of their staff during the review.
The exit briefing concludes the data gathering phase on site and the review team will
depart following the briefing.
QA/G-3 MSR Guidance 28
-------
DRAFT *2 JANUARY 1994
STEP C-l
INPUT: MSR Plan
Conduct Entrance
Briefing for Man-
agement
4. OUTPUT: Understanding of Scope of MSR by Management at Site.
STEPC-2
INPUT: MSR Plan
Interview Key Staff
at Site
I OUTPUT: Notes from Interviews
STEP C-3
INPUT: MSR Plan; Interview Notes
Conduct File Re*
views and
Obtain
Case Studies/-
Reports
I OUTPUT: Relevant File Information, Case Studies, and Other Documentation
Figure 2. Flow Model of MSR Data Gathering Phase
QA/G-3 MSR Guidance
29
-------
DRAFT #2 JANUARY 1994
STEP C-4
INPUT: Interview Notes and Other Information Collected; MSR Plan
Summarize Inter-
views and Compile
Initial Impressions
4. OUTPUT: Initial Impressions of MSR Findings for the Site
STEPC-5
INPUT: Initial Impressions of MSR Findings for the Site; MSR Plan
Conduct Exit Brief-
ing for Management
I OUTPUT: Completion of Data Gathering Phase
Figure 2 (Concluded). Flow Model of MSR Data Gathering Phase
QA/G-3 MSR Guidance
30
-------
DRAFT #2 JANUARY 1994
CHAPTER IV
EVALUATING THE RESULTS
At the conclusion of the data gathering phase of the MSR, the review team returns
home to the task of compiling and evaluating all of the information collected. Depending
on the number of sites visited and amount of case study information accumulated, this can
be a formidable task. The steps in this phase of the MSR process may be the most critical
to the success of the assessment. The data must be compiled, then evaluated according to
the criteria stated in the MSR Plan to produce interim findings. To ensure the relevance
of the findings, the "So What?" test is applied; that is, a test to determine if a finding is
significant and affects the quality of the operation or the data produced. Insignificant
findings should not be included in the reporting of the MSR because they weaken the effect
of the findings that really matter. Moreover, the client should not have to deal with trivial
issues, but should need dnly to focus only on the significant ones.
The time needed to complete the evaluation phase will be determined largely by the
volume of information to be reviewed and analyzed. It is not necessary for all review team
members to be together when reviewing the data, but they should be together when they
reach a consensus on the findings or consider preliminary conclusions or recommendations.
For a complex MSR, it would not be uncommon for this phase to require 10 work days to
adequately review the data and formulate relevant findings. A Flow Model of the evaluation
phase of the MSR process is given in Figure 3 at the end of this chapter.
When the data from interviews, file reviews, documents, and case studies have been
compiled, they .must be analyzed to identify key findings that pertain to the goals of the
MSR. When complex data operations are involved, it is very helpful to develop a flow
model of the data operation to ensure that all key steps are identified and that the inputs
and outputs from each step are understood. Such a flow model will help to define clearly
the critical linkages among the key steps in the process. The flow model should define the
criteria used to execute the step and the approach taken to produce the desired outputs.
Using the flow model, identification of the important findings becomes very straightforward.
More importantly, perhaps, the flow model provides management with a powerful tool which
may be used to evaluate possible changes to the structure and process of the data operation
in the future. The flow modeling technique and its use are described in Appendix A.
QA/G-3 MSR Guidance 31
-------
DRAFT #2 JANUARY 1994
Step 1: Assemble and Review Information According to MSR Plan Criteria
The results from the data gathering phase will include notes from the on-site
interviews, summaries of documentation reviewed prior to and during the site visits, results
of file reviews, case studies pertaining to the subject of the MSR, and other evidence
acquired during the MSR. This step provides for the sorting of this information into a
manageable data base so that an effective evaluation may be performed. The time required
to accomplish this activity will be determined by the volume of material acquired and the
complexity of the MSR.
As noted earlier, the use of a flow model of the process reviewed becomes
particularly helpful by providing a structure for the sorting of the information. In some
cases, gaps in the data base may be identified and the opportunity presented to obtain the
missing information (if it is available) before proceeding with the evaluation. It should be
noted that a information gap may itself be a significant finding from the MSR.
The MSR plan provides the criteria for the review of the MSR data base. The data
base should be organized first to seek answers to the issues and questions posed in the MSR
plan. Often, other material will emerge that may provide helpful information to the client
even though it wasn't specifically within the scope of the MSR. The review may be
accomplished effectively by allowing the review team to work independently of each other
and to identify preliminary findings separately.
As the information is reviewed, the flow model may be updated as additional
evidence is discovered to help define or explain the process. During this step, the original
questions posed in the MSR will be addressed, assuming that sufficient data are available
to answer them.
Step 2: Identity Interim Findings
This step brings into focus all of the preceding activities. The identification of interim
findings represents the principal outputs of the MSR process. The findings should reflect
the original issues and questions posed by the client and should be stated in a manner that
answers those questions whenever possible. Findings should always be based on documented
facts and not on the reviewer's speculation. The supporting documentation for the finding
must be identified and referenced in case such documentation is required later.
In some literature, the term finding refers only to negative impacts on the quality
system. The MSR process takes a broader interpretation. A finding is a statement based
QA/G-3 MSR Guidance 32
-------
DRAFT #2 JANUARY 1994
on the evidence found relative to a specific criterion or question. For example, consider the
following question: "Does the organization perform annual inspections of the monitoring
instruments?" Depending on the evidence discovered either from files or from interviews,
the finding would indicate whether or not the inspections were performed. To address
whether or not a sufficient number of instruments in the network had been inspected would
go beyond the finding and would require a conclusion; that is, a judgement of the adequacy
of the finding.- Conclusions will be discussed in Step 4.
Step 3: Apply the "So What?" Test to the Findings
As stated earlier, the analysis of findings from a MSR must include the "So What?"
test. The "So What?" test helps the reviewer to eliminate findings that really have little or
no relevance to the questions posed by the client. The value of the MSR process lies in
being able to tell the client what he or she needs to know so that effective response actions
may be taken. Accordingly, care must be exercised in applying the "So What?" test to
ensure that the trivial is separated from the significant. This ties back to the planning phase
and the emphasis on gaining a clear understanding of the client's perception of the issues
and what the client regards to be significant. The "So What?" test will also help to prioritize
the findings to aid the client in evaluating the results of the MSR.
It is essential that the findings be important and that they add value to management's
understanding of the data operation. Superficial or trivial findings diminish the value of the
MSR.
Step 4: Formulate Preliminary Conclusions and Recommendations
It is almost impossible to be totally objective when identifying findings. Invariably
findings will lead to conclusions and probably recommendations in the minds of the
reviewers. Such is the nature of most reviewers. This is not entirely undesirable, but
reviewers must be cautioned not to allow their conclusions to overly influence the evaluation
of the findings by the client. In some cases, conclusions and recommendations may have
been specifically requested by the client. It is then proper to develop conclusions based on
the evidence compiled and to prepare recommendations based on those conclusions. As will
be shown in the reporting phase, conclusions and recommendations must be carefully
controlled until the final report is prepared.
QA/G-3 MSR Guidance 33
-------
DRAFT #2 JANUARY 1994
STEP E-l
INPUT: Results from Interviews and Documentation Collected
Assemble Informa-
tion and Assess Its
Completeness Ac-
cording to MSR
Plan Issues
OUTPUT:
Results of Evaluation by Issue
STEPE-2
INPUT:
Results of Evaluation by Issue
Identify Interim
Findings and Group
Them by Issue
STEP E-3
INPUT:
OUTPUT:
Interim MSR Findings
Interim MSR Findings
Apply "So What'
Test to Findings to
Confirm Their Sig-
nificance for the
Study
OUTPUT: Identification of Relevant Findings
Figure 3. Flow Model of MSR Evaluation Phase
QA/G-3 MSR Guidance
34
-------
DRAFT #2 JANUARY 1994
STEP E-4
INPUT: Relevant MSR Findings
Formulate Prelimi-
nary Conclusions
(and Recommenda-
tions, if Requested)
I OUTPUT: Preliminary Conclusions (and Recommendations, if requested)
Figure 3 (Concluded). Flow Model of MSR Evaluation Phase
QA/G-3 MSR Guidance
35
-------
DRAFT #2 JANUARY 1994
CHAPTER V
REPORTING THE FINDINGS AND RECOMMENDATIONS
One of the basic truths of reviews and audits is that the written word has several
orders of magnitude greater impact than the spoken word. As a consequence, great care
and thoughtfulness must go into the preparation of the draft Findings Report and final MSR
Report. The findings should always be presented in a non-pejorative manner. When areas
needing improvement are found, the findings should point toward a solution to the problem,
not toward assigning blame for the problem. Inflammatory language should be avoided in
all reports. Recognize that bad news can be presented in a manner and tone that are
helpful. The discussions should be candid, but dispassionate.
The effectiveness of this approach in creating positive change has been demonstrated
in previous MSRs. The following steps describe the key elements in reporting the findings
and recommendations of the MSR. They are described in a flow model in Figure 4 at the
end of this chapter.
Step 1: Prepare a Written Draft Findings Report
A written draft report on the findings of the MSR must be prepared. The format of
the report is not critical; however, it should contain the following information:
the statement of objectives for the MSR; that is, why was the MSR needed
and what was to be accomplished by the study,
when and where did the study take place,
who was involved in the study, including the review team and the group being
reviewed,
what were the principal findings,
what are the impacts of the findings (i.e., the results of the "so what?" test),
QA/G-3 MSR Guidance 36
-------
DRAFT #2 JANUARY 1994
The review team may also include some recommendations at this stage if they were
requested to do so at the beginning of the MSR by the client. However, the purpose of the
draft Findings Report is to present the fac,ts of the study, not to draw conclusions from those
facts. It is fair to say that some findings will automatically lead the reader to obvious
conclusions and recommendations without having to state them in the report. Generally,
conclusions and recommendations should be brought forward after the reviewee has
concurred with the accuracy of the statements made in the draft Findings Report. In this
regard, conveying the sense of the "So What?" test is very important. The credibility of the
MSR will rest largely on how the findings are received by the reviewee. It is, therefore,
essential that the findings reflect significant issues or impacts relative to the program or
institution being reviewed. Credibility can be easily destroyed if the findings are frivolous
or irrelevant to the program.
If there are multiple reviewees, it may be necessary to prepare separate draft
Findings Reports for each reviewee. This is done to protect the confidentiality of each
reviewee. If a reviewee chooses to share his report with others, he may do so, but, the
reviewer should never reveal or distribute the contents of a draft Findings Report to anyone
other than the client of the MSR.
Step 2: Obtain Comments from the Reviewees
The draft findings should always be shared in writing with the reviewees to assure that
no factual errors have been included. This enables the reviewer and the reviewee to seek
a consensus on the findings and, in some cases, on the recommendations. Where multiple
reviewees are involved, it will be necessary to allow ample time for their thoughtful
examination of the report. This approach enhances the "ownership" of the study by the
reviewee and increases the likelihood that the recommendations will be implemented.
It is absolutely essential that the reviewee see the draft before the MSR client, who
requested the study. The reviewee must be given the maximum opportunity to assure that
the statements in the draft Findings Report accurately describe what was said or
documented during the study. However, there may be some disagreement between the
review team and the reviewee regarding what is Truth. The intent here to be certain that
the reviewees statements, case studies, documentation, etc., are portrayed accurately as they
were presented. Should any disagreements arise regarding the conclusions and recommen-
dations, an accurate draft Findings Report provides a common basis for discussing those
differences.
QA/G-3 MSR Guidance 37
-------
DRAFT #2 JANUARY 1994
Step 3: Reconcile Reviewees Comments and Prepare Draft Final MSR Report
The review team has the responsibility to correct any "errors in fact" in the draft
Findings Report. If the errors alter the preliminary conclusions and recommendations
developed during the analysis of the findings, then they also must be resolved. As noted
above, the issue pertains only to errors which are supported by evidence to the contrary.
Opinions are not valid cause to alter the report.
The conclusions (and recommendations, if they were requested by the client) are
added to the draft Findings Report at this time. As noted earlier, some conclusions and
recommendations may be shared with the reviewee in the draft Findings Report. This is
particularly helpful for those conclusions in the reviewee and the Review Team are in
complete agreement. Where there is (or may be) disagreement, the conclusions should be
presented to the MSR client for resolution. The review team may also consider including
a written response by the reviewee to the draft Findings Report. Any discussions regarding
the conclusions reported should be between the MSR client and the reviewee.
Step 4: Brief Client on MSR and Finalize Conclusions/Recommendations
Where possible, the draft Final MSR Report should be accompanied by a formal
briefing for the client. This provides the client with an opportunity for questions and
discussion. More importantly, the review team has the opportunity, if asked, to describe the
events of the MSR candidly and to present information or perspectives that would be
inappropriate in a written report. Such a subjective discussion should be initiated by the
client, not the review team.
The briefing should include a careful review of the findings and the conclusions. The
presentation should include the rationale for the conclusions reached. The client may or
may not ask for recommendations. If the request is made, the recommendations should be
supported by the rationale for them. Often, the client may have only a general knowledge
of the quality system and the QA/QC activities involved, and may choose to rely significantly
on the expertise of the reviewers. The briefing should, therefore, present the conclusions
and recommendations in a context and using terminology familiar to the client.
Step 5: Prepare Final MSR Report and Issue to Client
When the client has reviewed and concurred with the draft Final MSR Report, the
Final MSR Report will be prepared and issued to the client only. Any further distribution
of the report must be made by the client. The reviewers must never distribute copies of the
QA/G-3 MSR Guidance 38
-------
DRAFT #2 JANUARY 1994
MSR Report. The report is the property of the client.
As in the case of the draft Findings Report, MSRs involving reviews of multiple
organizations will have multiple draft Final MSR Reports. These may be compiled into the
Final MSR Report as individual sections or as appendices, but the report should include an
executive summary containing the MSR objectives and the principal findings and conclusions.
Step 6 (Optional): Identify any Follow-up to the MSR
As an option, the client may want to follow the MSR with further assessments or with
a request for technical assistance. The period immediately following the MSR presents
opportunities for various responses depending on the needs of the client.
Conclusion
The MSR process provides managers with a powerful management assessment tool.
While developed to evaluate the EPA Quality Systems for environmental programs, its field
of application is not limited to this subject. The steps comprising the MSR process are
sufficiently generic such that the MSR process may be applied to any management systems
to be evaluated. It is expected that the MSR process will continue to evolve and be
improved through growing use.
QAyG-3 MSR Guidance 39
-------
DRAFT #2 JANUARY 1994
STEP R-1
INPUT: Results of MSR; Preliminary Findings and Conclusions
Prepare a
Written
Draft Findings
Report for Each
Group Reviewed in
the MSR
STEP R-2
INPUT:
OUTPUT: Draft MSR Findings Report
Draft MSR Findings Report
Obtain Comments
from Each Reviewee
on the Draft Find-
ings Report
STEP R-3
INPUT:
OUTPUT: Reviewee Comments on Draft Findings Report
Reviewee Comments; Draft Findings Report
Resolve Comments
on Draft(s) and
Compile Draft Final
MSR Report
OUTPUT: Draft MSR Report Containing Final Findings and Conclusions (and
Recommendations, if Requested)
Figure 4. Flow Model of MSR Reporting Phase
QA/G-3 MSR Guidance
40
-------
DRAFT #2 JANUARY 1994
STEPR-4
INPUT: Draft Final MSR Report
Brief Client on
MSR Report and
Finalize Conclu-
sions (and Recom-
mendations, if Re-
quested)
OUTPUT:
Client Acceptance of ReporUncluding Final Conclusions (and
Recommendations, if Requested)
STEPR-5
INPUT:
Draft Final MSR Report; Final Conclusions (and Recommendations, if Requested)
Prepare and Issue
Final MSR Report
to Client Only
J. OUTPUT: Final MSR Report
STEPR-* (OPTIONAL)
INPUT: Final MSR Report; Client Request for Additional Assistance
Identify Next Steps
in Follow-up to the
MSR (if Requested)
I OUTPUT: Identification of Next Steps (as required)
END OF MSR PROCESS
Figure 4 (Concluded). Flow Model of MSR Reporting Phase
QA/G-3 MSR Guidance
41
-------
DRAFT #2 JANUARY 1994
REFERENCES
1. U.S. EPA Quality Manual for Environmental Programs, U.S. Environmental Protection
Agency (1994).
2. EPA Requirements for Quality Management Plans, U.S. Environmental Protection
Agency, EPA QA/R-2 (1994).
3. EPA Order 5360.1, Policy and Program Requirements to Implement the Mandatory
Quality Assurance Program, U.S. Environmental Protection Agency (April 1984).
4. Guidance for Planning for Data Collection in Support of Environmental Decision
Making Using the Data Quality Objectives Process, U.S. Environmental Protection
Agency, EPA QA/G-4 (1994).
QA/G-3 MSR Guidance 42
-------
DRAFT #Z JANUARY 1994
APPENDIX A
USE OF PROCESS FLOW MODELS IN MSRs
The use of a process flow model is particularly helpful in enabling management to
gain additional insights on the relationship of all activities in an operation. In addition,
managers can experiment with potential process modifications and observe their impacts.
Traditionally, process flow models have been used in the engineering design of complex
systems to help engineers understand the intricacies of the process, observe how inputs and
outputs from each step were interrelated, and assure that the sequence of the steps provided
the desired output from the process. Likewise, the application of the process flow model
technique to environmental data operations provides added information and clearer
understanding of the activities under review. For example, the model presents the data
operation as a series of interlinked activity and decision steps which describe the sequence
of logic flow and use of data throughout the process.
Components of Process Flow Models
In this application, the process flow model is composed of all of the steps needed to
describe fully an environmental data operation and the quality assurance and quality control
(QA/QC) activities applied. The model covers the entire scope of the data operation,
including planning (or scoping), implementation, and evaluation of the results, and shows the
logical sequence in which actions or decisions must occur in order to produce a desired
result or product. A typical flow model step is given by Figure A-l. 'For each step in the
process, the input and output is identified in terms of specific environmental data used. If
a decision is involved in the step, the decision paths emerging from the decision are shown.
The steps are linked together in the appropriate sequence to show the flow of decisions and
data throughout the process.
For each step in the flow model, there is a detailed Data Sheet that contains the
following information:
the purpose of the step,
the goal or objective of the step,
QA/G-3 MSR Guidance A-l
-------
DRAFT #2 JANUARY 1994
a description of the activity performed in the step and how environmental data
are used,
the criteria for performing the activity in the step, and
4
any implications of the step relative to preceding or succeeding steps.
An example Data Sheet is given in Figure A-2. The Data Sheets provide the necessary "data
base" on each step in order to present the user of the flow model with a clear picture of
what the step involves and how it relates to other steps in the process. The Data Sheets are
not essential to getting benefits from the flow model. The sequencing of steps and decisions
can provide very powerful information on the effectiveness of the process. However, there
are often subtleties in processes which may not be clear until critical relationships among
steps are fully identified. The Data Sheets provide a record of each step that captures the
necessary detail to allow a fuller utilization of the flow model technique.
Use of the Flow Models in Reviews
The principal benefit of the flow model is to make complex processes easier to
understand. Environmental data operations associated with major Agency programs, such
as the Superfund Remedial Investigation/Feasibility Study (RI/FS), involve many complicated
and diverse steps that produce and use environmental data throughout the process.
Frequently, such operations have many users (or clients) of the data generated, and
satisfying the data needs of such a large array of clients becomes increasingly difficult. The
process flow model provides a framework in which the data needs for each step in the
process may be identified and their sequence examined. This information is very helpful to
the planners of the data operations in assuring that data needs are met within the available
resources. In addition, a flow model of a complex process can be an effective training tool
in helping newcomers to understand all of the important activities and the order in which
they should occur.
Perhaps the greatest value of the flow model is the opportunity it allows for
optimizing the process. By ordering the steps in their proper sequence in a flow model, it
is possible to visualize the interrelationships among various steps, which otherwise may not
be obvious. For example, one may find that a particular step produces data that are not
used until much later in the process. Such a finding could allow the step using the data to
occur earlier and possibly save time and resources. Similarly, it may be possible to identify
more effective sequencing of the steps, which again could yield time and cost savings or
provide significant technical improvements to the process.
QA/G-3 MSR Guidance A-2
-------
DRAFT #2 JANUARY 1994
Case Study: The SUPERFUND RI/FS MSR
The value of process flow models to Management Systems Reviews (MSRs) can be
shown best through example. The Office of Emergency and Remedial Response (OERR)
invited the Quality Assurance Management Staff (QAMS) to perform a review of the
Superfund Remedial Investigation/Feasibility Study (RI/FS) program in order to provide an
independent assessment of this important process. The collection and analysis of
environmental data are the most significant cost and time components of the RI/FS. These
data are also key to the efficacy and reliability of important RI/FS decisions, such as
determining if an unacceptable risk is posed by a site and selecting an appropriate remedy.
As part of its ongoing efforts to reduce costs and improve the effectiveness of
Superfund activities, OERR requested that QAMS conduct a comprehensive review of the
RI/FS process, focusing on the role of environmental data. The review had the following
objectives:
identify the RI/FS decisions that rely on environmental data;
determine how data needs are defined and how their collection is planned and
executed; and
examine the impacts of the planning, collection, and use of RI data on the
scheduling and quality of RI/FS outputs, including remedy selection.
The review was conducted by QAMS with the assistance of the OERR Hazardous
Site Control Division and the Office of Program Management, and included participation
by Regional QA Managers. From the outset of the MSR, the process flow model was an
integral element of the study. During the planning of the MSR, documents such as QA
program plans and RI/FS guidance provided a general blueprint of the data collection and
QA operations, and gave a picture of how the RI/FS is supposed to operate. This
information was used to define the first-order process flow model of the major RI/FS
activities, and to assemble these activities or steps into logical groups for data gathering and
analysis. The flow model became a template for obtaining and organizing information
during subsequent interviews with Regional personnel.
Data gathering for the MSR involved interviews of more than 25 Remedial Project
Managers (RPMs) and their management in three Regions. The interviews traced the logic
and decision flow of the RI/FS, with emphasis on:
the types of environmental data collected and how data needs were deter-
mined;
QA/G-3 MSR Guidance A-3
-------
DRAFT #2 JANUARY 1994
the participants in RI/FS scoping arid their roles;
the way in which remedial alternatives were identified, evaluated, and selected;
how the RPM decides that sufficient RI data have been collected; and
factors that facilitated or impeded timely, effective remedial investigations.
Just as the review of Superfund planning documents and guidance helped to
formulate a framework for the flow model and to describe how the RI/FS process was
supposed to operate, the interviews showed how the RI/FS was performed in practice.
There was significant variability among RI/FSs within a Region and across Regions.
However, the planning and site investigation activities were sufficiently similar to identify a
typical or representative RI/FS in the process flow model. The outcome was then used as
a basis for analyzing the process.
In order to validate the process captured in the flow model and to add to the
understanding of how environmental data were being used, case studies were obtained for
eight sites. These sites were identified by the Regions as fairly typical sites and had Records
of Decision (RODs) completed in 1987 or 1988 to ensure that they reflected recent
procedures. The case study documentation generally included, for each site: the work plan,
sampling and analysis plan, quality assurance project plan, Remedial Investigation (RI)
report, Feasibility Study (FS) report, and ROD. This information was critical to understand-
ing what data were typically collected and how the data were used in making site-related
decisions, and was very helpful in validating the process flow model of the RI/FS.
\
When the data from the interviews and case studies had been integrated into the
process flow model, the model and the data were analyzed with respect to the study
objectives. It was found that many of the steps in the RI/FS process depend to some degree
on environmental data. The flow model simplified the identification of the major decisions
that rely on data. These are:
assessment of risk and determining if the no-action alternative is appropriate
for the site;
identification and screening of remedial process options; and
screening, evaluation, and selection of remedial alternatives.
Having confirmed the specific and critical role of environmental data in the RI/FS
(the first objective of the MSR), the flow diagram was used to document the process
typically used by the Regions for defining data needs. Next, the process used for planning
and executing field sampling activities (the second MSR objective) was identified and
QA/G-3 MSR Guidance A-4
-------
DRAFT #2 JANUARY 1994
documented in the flow model.
Further analysis indicated several opportunities for process changes related to the
third objective of the MSR; i.e., how the collection and use of environmental data impacts
the scheduling and quality of RI/FS outputs. These included:
1) knowing when to stop sampling;
2) reducing the number of unplanned sampling episodes;
3) reducing false starts and rework through structured planning;
4) beginning feasibility study planning during scoping;
5) reducing the number of alternatives considered .and evaluated during the
feasibility study; and
6) conducting treatability studies during remedial investigation field work.
While some of the improvements were already known to management, the process flow
model demonstrated the feasibility of additional changes. These changes show significant
promise for improving the effectiveness of the RI/FS process.
QA/G-3 MSR Guidance A-5
-------
DRAFT #2 JANUARY 1994
STEP PP-4
INPUT: PP-1, PP-2 OUTPUT: Determination that removal action is needed or not
DETERMINE IF EXISTING .
SITE CONDITIONS WARRANT
A REMOVAL ACTION
--->! IMPLEMENT REMOVAL ACTION
YES i
l
NO
Figure 1. Sample Step from RI/FS Process Flow Model
QA/G-3 MSR Guidance
A-6
-------
DRAFT #2 JANUARY 1994
STEP PP-2
INPUT:
PA/SI Results
OUTPUT: Assessment of current surface conditions and
potential receptors.
CONDUCT INITIAL
SITE VISIT
PURPOSE: To obtain first-hand observations of current site conditions.
GOAL/OBJECTIVE: Acquire current information about the site through visual inspection and/or limited
field measurements.
. ACTIVITY PERFORMED/DATA USE:
Historical data are used to guide the visual inspection of the site, which may include observations on
the presence and appearance of surface water, and obvious evidence of impacts from contamination
such as stressed vegetation and soil discoloration. Very limited sampling with portable equipment
may be conducted.
CRITERIA/ISSUES:
Information that may be collected include:
Have site surface conditions changed from the historical data? What are the implications of
the change?
Visual evidence of contamination.
Apparent stability of site (e.g., weakened beams, leaking tanks).
Proximity of population or sensitive ecosystems to the site.
IMPLICATIONS:
Visual inspection may identify areas of concern which may require removal action or
short-term mitigation.
Provides the RPM with a subjective view of the site which helps to define the magnitude of
the effort required for the RI/FS (i.e., where to sample, what site preparations are needed,
etc.).
Figure 2. Sample Process Flow Model Data Sheet
QA/G-3 MSR Guidance
A-7
-------
DRAFT #2 JANUARY 1994
APPENDIX B
FLOW MODEL OF MSR PROCESS
STEP P-l
INPUT: Request/Need for MSR
Identify the MSR
Purpose and Client
4 OUTPUT:
Understanding of who is the Client for the MSR and why it is
needed.
STEP P-2
INPUT: Purpose of the MSR
Determine Scope of
MSR and Issues to
be Addressed by the
MSR and Begin
MSR Draft Plan
I OUTPUT:
Definition of Scope of the Study, including Technical Program to
be Reviewed
QA/G-3 MSR Guidance
B-l
-------
DRAFT #2 JANUARY 1994
STEP P-3
INPUT: MSR Scope; Technical Program to be Reviewed
Identify Review
Team for the MSR
I OUTPUT: Review Team
STEP P-4
INPUT: Review Team; Scope of MSR
Identify and Provide
Training Needed by
the Review Team
OUTPUT: Review Team Trained for this MSR
STEPP-5
INPUT: MSR Scope and Issues
Identify Potential
Sources of Infbnna»
tion Needed to Re-
solve MSR Issues
4 OUTPUT: Sources of Information to Resolve Issues
QA/G-3 MSR Guidance
B-2
-------
DRAFT #2 JANUARY 1994
STEP P-6
INPUT:. Sources of Information Relevant to the MSR Subject
Compile and Review
Existing Guidance
and Information to
Determine Its Ap-
plicability to the
MSR .
4. OUTPUT: Information Relevant to Study
STEPP-7
INPUT: Relevant Existing Information; Scope of MSR
Complete
Draft
Plan for Conducting
the MSR,
including
Specific Interview
Questions
OUTPUT:
Draft MSR Plan
QA/G-3 MSR Guidance
B-3
-------
DRAFT #2 JANUARY 1994
STEP P-8
INPUT: Draft MSR Plan
Review MSR Plan
with Client, Deter-
mine if Recommen-
dations Are Expect-
ed, and Obtain
Client Approval of
MSR Plan
APPF
tOVED
OUTPUT:
Client-Approved MSR Plan
(IF NOT APPROVED, REPEAT STEPS P-7 AND P-8 AS NEEDED TO ASSURE
CUSTOMER UNDERSTANDING AND APPROVAL.)
STEP P-9
INPUT:
Approved MSR Plan
Propose Schedule
for Data Gathering
and Notify Organi-
zatton(s) to be Vis-
ited
OUTPUT:
Schedule for Data Gathering
QA/G-3 MSR Guidance
B-4
-------
DRAFT #2 JANUARY 1994
STEP P-10
INPUT: MSR Plan and Schedule
Resolve Any Con-
flicts and Finalize
Schedule for Data
Gathering
I OUTPUT: Final Schedule for MSR
STEPP-U
INPUT: MSR Plan and Final Schedule
Complete Logistical
Needs for Data
Gathering
I OUTPUT: Completed Arrangements for Site Visits and Data Gathering
STEP C-l
INPUT: MSR Plan
Conduct Entrance
Briefing for Man-
agement
OUTPUT: Understanding of Scope of MSR by Management at Site.
QA/G-3 MSR Guidance
B-5
-------
DRAFT #2 JANUARY 1994
STEP C-2
INPUT: MSR Plan
Interview Key Staff
at Site
I OUTPUT: Notes from Interviews
STEPC-3
INPUT: MSR Plan; Interview Notes
Conduct File Re-
views and Obtain
Case Studies/Re-
ports
OUTPUT: Relevant File Information, Case Studies, and Other Documentation
STEPC-4
INPUT: Interview Notes and Other Information Collected; MSR Plan
Summarize Inter-
views and Compile
Initial Impressions
4. OUTPUT: Initial Impressions of MSR Findings for the Site
QA/G-3 MSR Guidance
B-6
-------
DRAFT #2 . JANUARY 1994
STEP C-5
INPUT: Initial Impressions of MSR Findings for the Site; MSR Plan
Conduct Exit Brief-
ing for Management
i OUTPUT: Completion of Data Gathering Phase
STEP E-l
INPUT: Results from Interviews and Documentation Collected
Assemble Informa-
tion and Assess Its
Completeness Ac-
cording to MSR
Plan Issues
4. OUTPUT: Results of Evaluation by Issue
STEP E-2
INPUT: Results of Evaluation by Issue
Identify Interim
Findings and Group
Them by Issue
I OUTPUT: Interim MSR Findings
QA/G-3 MSR Guidance
B-7
-------
DRAFT #2 JANUARY 1994
STEP E-3
INPUT: Interim MSR Findings
Apply "So What"
Test to Findings to
Confirm Their Sig-
nificance for the
Study
OUTPUT:
Identification of Relevant Findings
STEPE-4
INPUT:
Relevant MSR Findings
Formulate Prelimi-
nary Conclusions
(and Recommenda-
tions, if Requested)
OUTPUT:
Preliminary Conclusions (and Recommendations, if requested)
STEP R-l
INPUT: Results of MSR; Preliminary Findings and Conclusions
Prepare a
Written
Draft Findings
Report for Each
Group Reviewed in
the MSR
OUTPUT:
QA/G-3 MSR Guidance
Draft MSR Findings Report
B-8
-------
DRAFT #2 JANUARY 1994
STEP R-2
INPUT: Draft MSR Findings Report
Obtain Comments
from Each Reviewee
on the Draft Find-
ings Report
OUTPUT:
Reviewee Comments on Draft Findings Report
STEP R-3
INPUT:
Reviewee Comments; Draft Findings Report
Reconcile Comme-
nts on Draft(s) and
Compile Draft Final
MSR Report with
Conclusions
OUTPUT:
Draft MSR Report Containing Final Findings and Conclusions (and
Recommendations, if Requested)
STEPR-4
INPUT:
Draft Final MSR Report
Brief Client on
MSR Report and
Finalize Conclu-
sions (and Recom-
mendations, if Re-
quested)
OUTPUT:
Client Acceptance of Reportjncluding Final Conclusions (and
Recommendations, if Requested)
QA/G-3 MSR Guidance
B-9
-------
DRAFT #2 JANUARY 1994
STEPR-5
INPUT: Draft Final MSR Report; Final Conclusions (and Recommendations, if Requested)*
Prepare and Issue
Final MSR Report
to CUent Only
4 OUTPUT: Final MSR Report
STEPR-6 (OPTIONAL)
INPUT: Final MSR Report; Client Request for Additional Assistance
Identify Next Steps
in Follow-up to the
MSR (if Requested)
OUTPUT:
END OF MSR PROCESS
Identification of Next Steps (as required)
QA/G-3 MSR Guidance
B-10
------- |