£ g% *
V PR0^°
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Be the Best in Public Service
Examination of Pilot Peer Review
Process for Inspectors General
That Follow "Blue Book"
Quality Standards for
Inspection	and Ev
Report No. 16-N-0317	September 21, 2016
HflH Council of the
Inspectors General
^^*2! on INTEGRITY and EFFICIENCY
Quality Standards
for
Inspection and Evaluation
January 2012

-------
Report Contributors:
Jeffrey Harris
Kalpana Ramakrishnan
Andre von Hoyer II
Abbreviations
CIGIE	Council of the Inspectors General on Integrity and Efficiency
EPA	U.S. Environmental Protection Agency
EPR	External Peer Review
I&E	Inspection and Evaluation
MOU	Memorandum of Understanding
OIG	Office of Inspector General
Cover photo: Cover of the Council of the Inspectors General on Integrity and Efficiency
Quality Standards for Inspection and Evaluation, also known as the
"Blue Book."
Are you aware of fraud, waste or abuse in an
EPA program?
EPA Inspector General Hotline
1200 Pennsylvania Avenue, NW (2431T)
Washington, DC 20460
(888) 546-8740
(202) 566-2599 (fax)
OIG Hotline@epa.gov
Learn more about our OIG Hotline.
EPA Office of Inspector General
1200 Pennsylvania Avenue, NW (2410T)
Washington, DC 20460
(202) 566-2391
www.epa.gov/oiq
Subscribe to our Email Updates
Follow us on Twitter @EPAoig
Send us your Project Suggestions

-------
^tDsr-%
• B \
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
16-N-0317
September 21, 2016
Why We Did This Review
We reviewed the Council of the
Inspectors General on Integrity
and Efficiency's (CIGIE's)
Inspection and Evaluation (l&E)
pilot External Peer Review
(EPR) process. Specifically, we
evaluated whether the EPR pilot
process (1) provided a basis to
ensure CIGIE Quality Standards
for Inspection and Evaluation
are achieved; and (2) identified
lessons learned and best
practices, and whether
recommendations were
incorporated into subsequent
training and guidance.
CIGIE is establishing an
external peer review process to
provide assurance that Offices
of Inspector General adhere to
the professional standards
established in the 2012 CIGIE
Quality Standards for Inspection
and Evaluation, also called the
"Blue Book." These standards
were developed to help ensure
a level of quality, objectivity and
independence in the work of
offices that adopt the standards.
This report addresses the
following U.S. Environmental
Protection Agency Office of
Inspector General goal:
• Be the best in public
service.
Send all inquiries to our public
affairs office at (202) 566-2391
or visit www.epa.gov/oiq.
Listing of OIG reports.
Examination of Pilot Peer Review Process for
Inspectors General That Follow "Blue Book"
Quality Standards for Inspection and Evaluation
What We Found
The EPR process provides a basis for
determining participants' adherence to
seven of 14 Blue Book standards. All
14 quality standards in the EPR process
are needed to provide a reasonable basis
that Inspectors General who adopt the Blue
Book quality standards are being
adequately evaluated.
A peer review process that
measures adherence to all the
quality standards established for
federal Inspector General
Inspection and Evaluation offices
provides assurance that
participating offices are being
adequately evaluated.
We identified the following best practices for conducting peer reviews of l&E units:
•	Continue using EPR teams comprised of staff from multiple l&E units.
•	Continue to support time management by encouraging offsite access to
documents needed to conduct the EPR.
We found opportunities for EPR improvement, including the following:
•	Develop a policy to conduct periodic reviews of the EPR process.
•	Develop a policy to require that corrective actions are implemented if an
l&E unit is found deficient in adhering to all quality standards.
•	Update the EPR guidance document to include a Blue Book standards
reference guide to promote consistent analysis of quality standards and
language to require that l&E units being reviewed have no input in the EPR
report selection process.
•	Retain documents created to support peer review findings following the
completion of the EPR.
•	Develop a pre-planning tool to assure that review team members possess the
prerequisites, such as security clearance or specific software skills, required
by the l&E unit being reviewed.
Suggestions
We suggest that all quality standards in the CIGIE peer review guidance be
evaluated, or seek a legal opinion concerning the appropriateness of a peer review
process that reviews adherence to only seven of 14 standards; develop a policy to
conduct periodic reviews of the effectiveness of the EPR process; require that
corrective actions are implemented if an l&E unit is found deficient in adhering to
all quality standards; update EPR guidance documents with best practices for peer
reviews; and develop a pre-EPR planning tool. The CIGIE l&E Roundtable agreed
with all the suggestions and has modified its draft Guide for Conducting Peer
Reviews. The Roundtable indicated it will recommend that the CIGIE l&E
Committee present these recommendations to the full CIGIE for consideration.

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
THE INSPECTOR GENERAL
September 21, 2016
The Honorable Daniel R. Levinson
Inspector General, Department of Health and Human Services;
and Committee Chair, Inspection and Evaluation Committee,
Council of the Inspectors General on Integrity and Efficiency
330 Independence Avenue, S.W.
Washington, D.C. 20201
RE: Examination of Pilot Peer Review Process for Inspectors General That Follow
"Blue Book" Quality Standards for Inspection and Evaluation
Report No. 16-N-0317
Dear Mr. Levinson:
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG) of the
U.S. Environmental Protection Agency (EPA). This report contains findings the OIG has identified and
corrective actions the OIG suggests. This report represents the opinion of the EPA OIG and does not
necessarily represent the final Council of the Inspectors General on Integrity and Efficiency position.
Final determinations on matters in this report will be made by the Inspection and Evaluation Committee
and the Council of the Inspectors General on Integrity and Efficiency.
We will post this report to our website at www.epa.gov/oig.
Sincerely.
Arthur A. Elkins Jr.

-------
Examination of Pilot Peer Review Process for
Inspectors General That Follow "Blue Book"
Quality Standards for Inspection and Evaluation
16-N-0317
Table of C
Purpose		1
Background		1
Scope and Methodology		5
Results of Review		6
Seven of 14 Blue Book Standards Reviewed During Pilot EPR		6
Opportunities for EPR Improvement and Best Practices		6
EPR Pilot Program Procedures Generally Followed		7
Conclusion		8
Suggestions		9
CIGIE l&E Comments on EPA OIG Evaluation		10
Appendix
A CIGIE Roundtable Response to Draft Report	 11

-------
Purpose
We reviewed the Council of the Inspectors General on Integrity and Efficiency's
(CIGIE) Inspection and Evaluation pilot External Peer Review (EPR) process.
We evaluated whether CIGIE's EPR pilot process (1) provided a basis to ensure
CIGIE Quality Standards for Inspection and Evaluation are achieved; and
(2) identified lessons learned and best practices, and whether recommendations
were incorporated into subsequent training and guidance.
Background
The Inspector General Reform Act of 20081 statutorily established CIGIE as an
independent entity within the executive branch to:
"(A) address integrity, economy, and effectiveness issues that
transcend individual Government agencies; and (B) increase the
professionalism and effectiveness of personnel by developing
policies, standards, and approaches to aid in the establishment of a
well-trained and highly skilled workforce in the offices of the
Inspectors General."
CIGIE has seven standing committees, including the
Inspection and Evaluation (I&E) Committee. From 2012
through 2015, the I&E Committee piloted external peer
reviews to provide assurance to OIG management and
parent agencies, as well as the Congress, that I&E units
within the OIG adhere to the 2012 CIGIE Quality Standards
for Inspection and Evaluation guidance document, also
known as the "Blue Book." CIGIE expects that an EPR
process will provide OIGs and their stakeholders assurance
of an I&E unit's compliance with the Blue Book standards.2
According to the 2012 Blue Book, external peer reviews ensure a level of objectivity
and independence, while also providing a learning opportunity for both the unit
under review and for those conducting the reviews. Each I&E unit under review
receives feedback on its work products, and the adequacy of its work processes.
These reviews help validate the unit's objectivity and independence. In addition,
those conducting peer reviews are exposed to different approaches for conducting
inspection and evaluation work, which they can share with their own unit, potentially
leading to more robust I&E approaches across the OIG community.
1	The Inspector General Reform Act of 2008. P.L. 110-409.
2	Section 11(c)(2)(A) of the Inspector General Act of 1978, as amended, states: - "(2) Adherence and participation
by members—To the extent permitted under law, and to the extent not inconsistent with standards established by the
Comptroller General of the United States for audits of Federal establishments, organizations, programs, activities,
and functions, each member of the Council, as appropriate, shall—(A) adhere to professional standards developed
by the Council...."
The I&E Committee's mission is to:
•	Provide leadership to improve
agency program effectiveness.
•	Develop protocols to review
management practices.
•	Promote the use of advanced
evaluaton techniques.
•	Foster awareness of I&E
practices in the I&E community.
16-N-0317
1

-------
Quality Standards for Inspection and Evaluation
The 2012 Blue Book states that I&E organizations should strive to conduct their
operations in the most efficient and effective manner possible, which serves to
enhance the credibility of the organizations. The Blue Book identifies
14 professional standards for I&E work performed by member organizations, as
shown in Table 1.
Table 1: Blue Book quality standards
Standard
Definition
1. Competency
The staff assigned to perform inspection work should collectively possess
adequate professional competency for the tasks required.
2. Independence
In all matters relating to inspection work, the inspection organization and
each individual inspector should be free both in fact and appearance from
personal, external, and organizational impairments to independence.
3. Professional
Judgment
Due professional judgment should be used in planning and performing
inspections and in reporting the results.
4. Quality Control
Each OIG organization that conducts inspections should have appropriate
internal quality controls for that work.
5. Planning
Inspections are to be adequately planned.
6. Data Collection
and Analysis
The collection of information and data will be focused on the organization,
program, activity, or function being inspected, consistent with the inspection
objectives, and will be sufficient to provide a reasonable basis for reaching
conclusions.
7. Evidence
Evidence supporting inspection findings, conclusions, and recommendations
should be sufficient, competent, and relevant and should lead a reasonable
person to sustain the findings, conclusions, and recommendations.
8. Records
Maintenance
All relevant documentation generated, obtained, and used in supporting
inspection findings, conclusions, and recommendations should be retained
for an appropriate period of time.
9. Timeliness
Inspections should strive to deliver significant information to appropriate
management officials and other customers in a timely manner.
10. Fraud, Other Illegal
Acts, and Abuse
In conducting inspection work, inspectors should be alert to possible fraud,
other illegal acts, and abuse and should appropriately follow up on any
indicators of such activity and promptly present associated information to
their supervisors for review and possible referral to the appropriate
investigative office.
11. Reporting
Inspection reporting shall present factual data accurately, fairly, and
objectively and present findings, conclusions, and recommendations in a
persuasive manner.
12. Followup
Appropriate followup will be performed to ensure that any inspection
recommendations made to Department/Agency officials are adequately
considered and appropriately addressed.
13. Performance
Measurement
Mechanisms should be in place to measure the effectiveness of inspection
work.
14. Working
Relationships and
Communication
Each inspection organization should seek to facilitate positive working
relationships and effective communication with those entities being
inspected and other interested parties.
Source: CIGIE guidance, Quality Standards for Inspection and Evaluation, January 2012.
16-N-0317

-------
EPR Pilot Oversight and Process
The CIGIE I&E Committee convened a Roundtable and the Process Oversight
Workgroup (Workgroup) to oversee the EPR pilot process. Suggestions to
improve the EPR process are first considered by the Workgroup, and applicable
suggestions are moved forward and vetted by the I&E Roundtable. Suggestions
are then reviewed by the I&E Committee before they are incorporated into future
guidance documents.
Members of the Workgroup developed tools to conduct an EPR, and delivered
training to reviewers and units reviewed. The Workgroup and the CIGIE Training
Institute's Audit, Inspection & Evaluation Academy created the following
guidance documents and tools to conduct an EPR, which were presented during
training for reviewers and those reviewed:
1.	Framework for External Peer Reviews for Offices of Inspector General
Inspection and Evaluation Units: provides guidance for conducting an EPR.
2.	Template for I&E External Peer Review Work Plan: identifies steps to be
completed by reviewers while conducting an EPR.
3.	Process Checklist: step-by-step checklist to complete an EPR.
4.	Standards Checklist: provides definitions and examples of how to assess
the Blue Book standards.
5.	Template for I&E External Peer Review Report: provides template on
items and information to be included in an EPR report.
6.	Memorandum of Understanding (MOU) Template: sample MOU to be
completed by agency being reviewed.
7.	Training presentation: a PowerPoint presentation used during training.
The Workgroup met with the peer reviewers and units reviewed following each of
the three peer review rounds to solicit feedback. This feedback was used to refine
the EPR training and tools. The EPR pilot was designed to be a cumulative
process whereby feedback was incorporated into subsequent EPR training and
guidance materials. In particular, the Framework document, which provides
guidance for conducting an EPR, was updated following feedback sessions, prior
to the next round of peer reviews.
The pilot EPR was conducted in three rounds, in calendar years 2012 through
2015, and resulted in seven EPR reports. Participation in the pilot phase was
voluntary. Each peer review team was comprised of representatives from various
I&E units. The I&E units reviewed and the review team composition are listed in
Table 2.
16-N-0317
3

-------
Table 2: EPR rounds and participating Office of Inspector General (OIG) units
Pilot round
OIG I&E unit reviewed
OIG review team
2012
Department of Health and
Human Services
•	Federal Housing Finance Agency
•	Peace Corps
•	Treasury Department Inspector General for
Tax Administration

Peace Corps
•	Federal Housing Finance Agency
•	Department of Health and Human Services
•	Department of State
•	Federal Reserve System
2013
Department of the Interior
•	Defense Intelligence Agency
•	Department of Veterans Affairs
•	Department of Energy

Treasury Inspector General
for Tax Administration
•	Department of State
•	Department of Defense
•	Department of Health and Human Services
•	Department of Veterans Affairs

Department of Veterans
Affairs
•	Department of Health and Human Services
•	Department of the Interior
•	Central Intelligence Agency
•	Small Business Administration
2015
Department of the Interior
•	Office of Personnel Management
•	Legal Services Corporation
•	Department of State
•	Department of Veterans Affairs

Department of State
•	Department of Justice
•	Defense Intelligence Agency
•	Department of Homeland Security
•	Department of Defense
Source: Generated by U.S. Environmental Protection Agency (EPA) OIG, from documents provided by CIGIE.
External Peer Review Procedures
According to the Round 3 EPR Framework,3 peer reviewers and units reviewed
must attend a 1-day training prior to the EPR. During the training, peer review
teams select team leaders responsible for managing the EPR. At the training, the
I&E unit being reviewed provides the peer review team with all relevant reports,
policies, procedures, guidance documents and annual strategic plans from the
prior fiscal year. The I&E unit being reviewed may request specific reports or
projects to be reviewed by the peer review team, and this fact must be disclosed in
the final EPR report. The I&E unit being reviewed will draft an MOU for
signature by the peer review team and officials from the I&E unit reviewed. The
MOU establishes terms and conditions between the unit being reviewed and the
peer reviewers, such as security clearance that may be needed and peer review
3 "Framework for External Peer Reviews for Offices of Inspector General Inspection and Evaluation Units -
Round 3"; Version for 3rd Round of Pilot and Training on June 17, 2015.
16-N-0317
4

-------
team access to documents. According to the framework, the MOU also identifies
the peer review team lead who will be responsible to respond to any public
requests for information following the peer review.
According to the framework, all peer review team members "must have
significant experience conducting inspection and/or evaluation work under the
Blue Book standards." Peer review team members are expected to spend
approximately 40 hours reviewing documentation4 prior to onsite review, and also
spend 1 week onsite at the I&E unit being reviewed. At the completion of the
onsite portion of the review, the peer review team holds an exit meeting with the
I&E unit being reviewed. The EPR timeframe consists of 96 hours: 8 hours in
training, 40 hours reviewing policies and procedures, 40 hours conducting the
onsite portion of the external peer review, and 8 hours to prepare the final report.
A draft peer review report is also presented at the exit meeting. The I&E unit
being reviewed has one to two weeks to review the draft report and prepare
written comments, which will be attached to the final report.
Scope and Methodology
We conducted our work from October 2015 through March 2016. Our work does
not constitute an audit conducted in accordance with generally accepted
government auditing standards.
Our evaluation focused on the final Round 3 of the EPR. We reviewed all
guidance documents and tools provided to Round 3 EPR participants. We
interviewed the Workgroup staff, Round 3 EPR trainers, peer reviewers and
representatives of units reviewed to evaluate the implementation of EPR guidance
and determine if there were lessons learned or best practices. We interviewed an
EPA OIG Director of Quality Assurance to obtain best practices during an EPR.5
We analyzed the final Round 3 peer review reports to determine to what extent
the report writing template was followed. We reviewed Rounds 1 and 2 feedback
summaries (meeting minutes) to determine whether suggestions were
incorporated into the EPR Round 3 guidance materials. We reviewed Round 3
feedback summaries and the most recent draft CIGIE EPR guidance document.6
4	Documentation includes selected reports, as well as all relevant policies, procedures, guidelines, annual work plans
and manuals related to the processes followed by the office for conducting I&E projects.
5	In conducting its audits and evaluations, the EPA OIG adheres to the Government Auditing Standards, also known
as the "Yellow Book." The Yellow Book has an established EPR process. Agencies are reviewed at least once every
3 years. The peer review concludes with a report and issues a grade of pass, pass with deficiencies, or fail.
6	The most recent Draft 2016 CIGIE EPR Guide at the time of our review was dated February 17, 2016. We
reviewed the draft document to determine whether recommendations and best practices have been incorporated.
16-N-0317
5

-------
Results of Review
The EPR pilot process provided a reasonable basis for evaluating adherence to
seven of the 14 Blue Book standards included in the pilot peer review. Members
of the Workgroup stated that seven of 14 quality standards were selected because
they were considered the most relevant standards and would expedite completing
the pilot rounds within anticipated target dates. The Workgroup held feedback
sessions with reviewers and units reviewed following each round to collect
suggestions and lessons learned. All suggestions were considered by the
Workgroup, and the applicable suggestions were vetted by the Roundtable and
I&E Committee before being incorporated into the Round 3 EPR guidance,
training documents or overall EPR guidance.
Seven of 14 Blue Book Standards Reviewed During Pilot EPR
The pilot included only seven of the 14 Blue Book quality standards (Table 3).
Table 3: CIGIE Blue Book and EPR pilot standards
CIGIE Blue Book standards
EPR pilot standards
1. Competency

2. Independence

3. Professional Judgment

4. Quality Control
X
5. Planning
X
6. Data Collection and Analysis
X
7. Evidence
X
8. Records Maintenance
X
9. Timeliness

10. Fraud, Other Illegal Acts, and Abuse

11. Reporting
X
12. Followup
X
13. Performance Measurement

14. Working Relationships and Communication

Source: CIGIE guidance, Quality Standards for Inspection and Evaluation, January 2012.
The Workgroup staff stated that they considered the seven selected standards the
"crucial and most relevant" standards. The Workgroup staff stated the pilot
process was to determine whether the selected standards could be reviewed within
a defined time (96 hours). However, neither the EPR training materials nor the
framework provide an explanation for selecting only seven standards. In addition,
the framework did not elaborate on when, or how, the unexamined standards
would be included.
Opportunities for EPR Improvement and Best Practices
Based on our interviews and review of feedback from peer review participants,
the following includes both opportunities for EPR improvement and best
practices, and should be included in the final CIGIE guide for conducting peer
reviews of I&E units:
16-N-0317
6

-------
•	Continue using EPR review teams made up of staff from multiple I&E units.
•	Continue to support time management by encouraging offsite access to
documents needed to conduct the EPR and allowing peer review teams to
access documents at their convenience.
•	Create a Blue Book standards reference guide explaining criteria used and
examples of when standards are meeting Blue Book requirements to be
added into the EPR guidance documents to promote a consistent critique
of quality standards.
•	Update guidance documents to emphasize that EPRs are to evaluate
whether an I&E unit operated under its own policies and procedures, as
opposed to a reviewer's home agency's policies and procedures.
•	Provide that peer review teams select reports for review with no input
from the I&E unit being reviewed, to ensure independence and eliminate
any potential perception of bias.
•	Retain documents created to support peer review findings following the
completion of the EPR.
•	Develop a pre-planning tool to assure that review team members possess
the prerequisites required by the I&E unit being reviewed, such as security
clearances or experience with specific software, such as AutoAudit or
Teammate.
Each round of the pilot produced lessons learned. Feedback meetings were held
following each EPR round, and subsequent suggestions deemed relevant were
incorporated into Round 3 EPR documents. We believe that the EPR process would
benefit from periodic evaluations so that it can continue to capture lessons learned.
Currently, the EPR process does not require corrective actions by the I&E unit
being reviewed. To sustain the integrity and intent of an EPR, I&E units that are
found to be deficient in Blue Book standards should be required to implement
corrective actions. Moreover, until the unit can demonstrate that corrective actions
are implemented, we suggest that an I&E unit determined to be deficient not
participate as a peer reviewer in subsequent EPR processes.7
EPR Pilot Program Procedures Generally Followed
As required by the framework, all peer reviewers and units reviewed attended the
mandatory training prior to the start of review. During training, the peer review
7 CIGIE's Guide for Conducting Peer Reviews of Audit Organizations of Federal Offices ofInspector General,
September 2014, provides direction for an OIG unit's eligibility to remain in the peer review rotation. Eligibility is
based upon results of previous peer reviews.
16-N-0317
7

-------
teams selected a team lead. We found that one peer review team leader did not
have prior Blue Book evaluation experience—a requirement to participate on the
review team. However, the peer review team lead did have evaluation experience,
and became familiar with Blue Book standards prior to starting the EPR.
The I&E unit being reviewed created MOUs, which were signed by all
participants. The peer review teams conducted their reviews within specified time
frames, both onsite and offsite. Onsite peer reviews were conducted at the
"hosting" agency, while off-site peer reviews were generally conducted at the
reviewer's normal duty station. Peer review team participants stated that access to
documents offsite allowed for greater flexibility and time management. Peer
reviewers stated having documents offsite allowed them to review documents at
their convenience, not only during the week-long onsite visit. However, some
peer reviewers noted they did not have prior experience with the software used by
the agency being reviewed, making it difficult to access documents at times.
The framework states that a draft report should be created at the end of the onsite
peer review. The draft report should be provided to the I&E unit being reviewed
at the exit meeting. One peer review team provided a draft report in advance of
the exit meeting, providing the I&E unit time to review the document. The other
peer review team provided a findings summary document at the exit meeting, but
not a draft report. Though a draft report was not provided by both peer review
team at the exit meeting, written documents with findings that facilitated exit
meetings. Further, the teams were able to accept comments, and a final report was
created in a timely manner. Round 3 EPR peer review reports demonstrated that
the report writing template was used.
The framework recognizes the importance of retaining EPR documents, and the
template for the I&E External Peer Review MOU states that, "when the Review
Team receives requests or legal demands for peer review documents, the Review
Team leader is responsible for coordinating and providing the response to the
requestor."8 During our review, we found that Round 3 peer review team leads
had not retained documents created during the review. Rather, each peer review
team member kept his or her own EPR and analysis documents.
Conclusion
The CIGIE External Peer Review pilot process continuously improved guidance
documents—such as the peer review framework, report writing template,
checklists and peer review training—through the continuous incorporation of best
practices and lessons-learned over a series of three pilot peer review cycles. The
process and guidance produced from the pilots provided a reasonable basis for
evaluating adherence to seven Blue Book quality standards selected for review.
However, the EPR process does not provide a reasonable basis to ensure that all
8 The Freedom of Information Act is a law that provides a right to request access to information from the federal
government (https://www.foia.gov/).
16-N-0317
8

-------
14 Blue Book quality standards are being adhered to by Inspectors General that
adopt these standards.
Conducting periodic evaluations of the EPR process will support continuous
improvement. Developing policy to establish participation and corrective action
rules for I&E units that are deemed deficient in Blue Book standards will ensure
integrity and effectiveness of the EPR process. Additionally, updating EPR
guidance documents with best practices and developing a pre-EPR planning tool
to identify requisite skills of peer review team members will improve the EPR
process.
Suggestions
We suggest that the CIGIE Inspections and Evaluations Roundtable:
1.	Address all 14 Blue Book quality standards in the CIGIE Peer Review
Guidance, or seek a legal opinion from appropriate counsel concerning
appropriateness of a peer review process that evaluates adherence to only
seven of 14 standards. If review of only seven of the 14 standards is
deemed appropriate, include a disclosure statement regarding the selection
of only seven of the 14 standards. Additionally, develop language for the
final CIGIE EPR guidance document providing the reasoning for the
selection of only seven of the 14 Blue Book standards.
2.	Develop a policy to conduct periodic reviews of the effectiveness of the
EPR process.
3.	Develop a policy to require that corrective actions are implemented if an
I&E unit is found deficient in adhering to all quality standards, and will
not be qualified to participate in the EPR process as a reviewing entity
until corrective actions are implemented.
4.	Update EPR guidance documents to include the following:
a.	A Blue Book standards reference guide to promote consistent
analysis of quality standards.
b.	Procedures to encourage off-site access to EPR documents.
c.	Language to require that I&E units being reviewed have no input
in the EPR report selection process.
5.	Develop a retention policy for EPR documents and records.
6.	Develop a pre-EPR planning tool to assure that review team members
possess the prerequisites necessary to conduct the review, such as security
clearances and applicable software experience.
16-N-0317
9

-------
CIGIE l&E Comments on EPA OIG Evaluation
The Chair of the CIGIE I&E Committee provided the CIGIE I&E Roundtable's
comments on our report. The CIGIE I&E Roundtable agreed with all the
suggestions. The Roundtable has modified its draft Guide for Conducting Peer
Reviews and has recommended additional actions, with milestones, for
consideration by the I&E Committee and the CIGIE. The Roundtable's detailed
response to the EPA OIG suggestions is in Appendix A.
16-N-0317
10

-------
Appendix A
CIGIE Roundtable Response to Draft Report
SERVICE.
AUG 17 2016
TO:	The Honorable Arthur
Elkins Inspector General
Environmental Protection
Agency 1200 Pennsylvania Ave,
NW Washington, DC 20460
FROM: Daniel R. Levinson
Inspector General

SUBJECT: Environmental Protection Agency Office of Inspector General's Draft
Report: Examination of Pilot Peer Review Process for Inspectors General
That Follow "Blue Book" Quality Standards for Inspection and Evaluation
(Project Number OPE-FY 15-G-0062)
The Council of the Inspectors General on Integrity and Efficiency (CIGIE) Inspections and
Evaluations Committee (herein after referred to as the Committee) and its Inspections and
Evaluations Roundtable (herein after referred to as the Roundtable) appreciates the
opportunity to review and comment on the Environmental Protection Agency Office of
Inspector General's Draft Report: Examination of Pilot Peer Review Process for Inspectors
General That Follow "Blue Book" Quality Standards for Inspection and Evaluation (Project
Number OPE-FY15-G- 0062). Thank you for your review of this important issue.
Please find below the EPA OIG suggestions and Roundtable responses. They include
improvements the Roundtable has incorporated into the draft Guide for Conducting Peer
Reviews of Inspection and Evaluation Organizations of Federal Offices of Inspector General
(the Guide) as well as recommendations the Roundtable will make to the Committee for future
action. The Committee will examine the issues and then present its full recommendations to
the CIGIE for consideration.
16-N-0317
11

-------
EPA PIG SUGGESTION
The Roundtable should address all 14 Blue Book quality standards in the CIGIE Peer
Review Guidance, or seek a legal opinion from appropriate counsel concerning the
appropriateness of a peer review process that reviews adherence to only 7 of 14 standards.
The review of only 7 of the 14 standards is deemed appropriate, include a disclosure
statement regarding the selection of only 7 of the 14 standards. Additionally, develop
language for the final CIGIE External Peer Review (EPR) guidance document providing the
reasoning for the selection of only 7 of the 14 Blue Book standards.
ROUNDTABLE RESPONSE
The Roundtable agrees and will recommend that the Committee further evaluate the costs
and benefits of a proposed expansion to include review of all 14 standards. Since the peer
review pilots examined seven standards,1 the Roundtable will recommend that the
Committee use those seven standards in the first round of peer reviews and concurrently
develop information and pilot reviews of the other seven standards.2 We anticipate that pilot
reviews of the full 14 standards will be complete by spring 2020 (or 3 years after the formal
peer review process begins). The Roundtable will also recommend that the Committee
approve the disclosure statement in the draft Guide that explains why currently reviews
assess 7 standards and the Committee plan to consider a 14-standard assessment for future
reviews. Specifically, the current draft Guide states:
For the first 3-year cycle, the scope of the external peer review will
cover the same seven Blue Book standards used in the pilot external
peer reviews. The seven required standards are Quality Control,
Planning, Data Collection and Analysis, Evidence, Records
Maintenance, Reporting, and Follow-up. At the end of the second
year of each 3-year cycle, the external peer review process will be
re-evaluated. Based on the results of the evaluation, revisions and
improvements to the external peer review process and this Guide
for the next cycle will be recommended to the l&E Committee for
approval.
The Roundtable anticipates the Committee and CIGIE will consider this disclosure statement
by mid-2017. The first evaluation of the peer review process should begin in 2019, after the
end of the second year of the first 3-year cycle, and revisions and improvements will be
recommended to the I&E Committee for approval in 2020.
1	The peer review pilots examined the following seven standards: Quality Control, Planning, Data Collection and
Analysis, Evidence, Records Maintenance, Reporting, and Follow-up.
2	The remaining seven standards are Competency; Independence; Professional Judgement; Timeliness; Fraud, Other
Illegal Act, and Abuse; Performance Measurement; and Working Relationships and Communication.
16-N-0317
12

-------
EPA PIG SUGGESTION
The Roundtable should develop a policy to conduct periodic reviews of the effectiveness of
the EPR process.
ROUNDTABLE RESPONSE
The Roundtable agrees and will recommend that the Committee include in the Guide that the
external peer review process be reevaluated at the end of the second year of each 3-year
cycle.
The Roundtable also will recommend that the Committee include a review of the effectiveness of
the EPR process in these evaluations. The Roundtable anticipates that the Committee and CIGIE
will consider including these statements by mid-2017. The first evaluation of the peer review
process should begin in 2019, after the end of the second year of the first 3-year cycle. Subsequent
evaluations should occur every 3 years.
EPA OIG SUGGESTION
The Roundtable should develop a policy to require that corrective actions be implemented if an
I&E unit is found deficient in adhering to all quality standards and will not be qualified to participate
in the EPR process as a reviewing entity until corrective actions are implemented.
ROUNDTABLE RESPONSE
The Roundtable agrees and will recommend that the Committee approve implementation of a policy
to require corrective action. Currently, the draft Guide states:
The Reviewed OIG is responsible for implementing any recommendation in the external peer
review report. The Reviewed OIG's next peer review should include follow up on
implementation of prior recommendations.
However, the Roundtable believes that excluding an OIG found deficient in adhering to all quality
standards from participating in the peer review process may be counterproductive in that it may
hinder learning gained from participation. Therefore, the Roundtable will recommend that the
Committee limit the participation of an OIG with such deficiencies to team members. Further, staff
of such an OIG would not be allowed to lead a peer review until unit deficiencies are remedied. The
Roundtable anticipates the Committee and CIGIE will consider these requirements by mid-2017 and
requirements will be implemented in the first round of the peer review process that should begin in
2017.
EPA OIG SUGGESTION
The Roundtable should update EPR guidance documents to include the following: (a) a Blue
Book standards reference guide to promote consistent analysis of quality standards, (b) procedures to
encourage off-site access to EPR documents, and (c) language to require that I&E units being
reviewed have no input in the EPR report selection process.
ROUNDTABLE RESPONSE
The Roundtable will recommend that the Committee continue to update the draft Guide to fully
address these issues. Specifically, the Roundtable will recommend that the Committee (a) include a
statement in the peer review report that notes that the review was conducted in accordance with the
CIGIE-approved Guide. Consistent with the way Audit conducts peer reviews, the Guide will not
16-N-0317
13

-------
include a statement noting that the review was conducted in accordance with Blue Book standards.
Additionally, (b) the draft Guide language has been modified to more strongly encourage off-site
access to the reviewed agencies' documentation, policies, and procedures, and (c) the draft Guide
does not allow I&E units being reviewed to have input into the EPR report selection process.
However, the draft Guide allows the I&E unit under review to propose a list of reports to be included
in the peer review. The Roundtable anticipates the Committee and CIGIE will consider these
statements by mid-2017 and requirements will be implemented in the first round of the peer review
process that should begin in 2017.
EPA OIG SUGGESTION
The Roundtable should develop a retention policy for EPR documents and records.
ROUNDTABLE RESPONSE
The Roundtable agrees and will recommend that the Committee approve inclusion of the retention
policy for EPR documents and records currently in the draft. The draft Guide specifically states:
the peer review Team Leader's OIG is responsible for storage and maintenance of any review
team-generated documents. The Team Leader's OIG should handle record
retention/archival/destruction responsibilities under its existing policies and procedures for
I&E work. The Team Leader's OIG should apply the same custody and physical and
electronic security practices with respect to the external peer review documentation that it
applies to its own I&E documentation. These policies should include safeguards against
unauthorized use or access to the documentation. At a minimum, the peer review team-
generated documentation should be retained until after the Reviewed OIG's subsequent peer
review is completed. The Team Leader's OIG will provide the subsequent review team with
access to the documentation, when requested.
Further, the External Peer Review Memorandum of Understanding states:
the Team Leader's OIG shall maintain all supporting and original documents created and
used by the Review Team after issuance of the final report and in accordance with their
agency record retention procedures, or at least until a subsequent peer review of the
Reviewed OIG is performed, and that the Reviewed OIG shall maintain all supporting and
original documents used by the Review Team in accordance with their agency record
retention procedures, or at least until a subsequent peer review of the Reviewed OIG is
performed.
The Roundtable anticipates the Committee and CIGIE will consider these statements by mid-2017
and requirements will be implemented in the first round of the peer review process that should begin
in 2017.
EPA OIG SUGGESTION
The Roundtable should develop a pre-EPR planning tool to assure that review team members possess
the prerequisites necessary to conduct the review, such as security clearances and applicable software
experience.
16-N-0317
14

-------
ROUNDTABLE RESPONSE
The Roundtable agrees and will recommend that the Committee approve inclusion of a statement
ensuring that review team members possess the prerequisites to conduct the review in the Guide.
With regard to the assignment of peer review team members, the draft Guide states:
Other factors that should be considered in assigning team members include any specialized
skills that may be needed, such as information technology (IT) specialists and statisticians.
When the Reviewed OIG uses electronic software to document their work, the review team
members should be capable of reviewing such work. Geographic location of working papers
in relation to the geographic location of review team members should be considered to limit
required travel or remote access issues. Reviewed OIGs may also have special security
requirements relating to access to their work space, OIG IT systems, or documents and
records that need to be considered when assigning team members.
Additionally, the Roundtable will recommend that the Committee include in the draft Guide a
statement that the reviewed OIGs responsibilities include informing the Committee (those
scheduling the peer reviews) of the composition of the peer review team and any prerequisites for
staff conducting the review, such as security clearances and applicable software experience.
Further, the Roundtable will recommend that the Committee include a statement noting that the peer
review schedulers must require the reviewed OIG to submit this information before being scheduled
for a peer review. The Roundtable anticipates the Committee and CIGIE will consider these
inclusions and statements by mid-2017 and requirements will be implemented in the first round of
the peer review process that should begin in 2017.
Thank you again for carrying out this study. The findings reinforce the importance of a peer review
process that measures adherence to the quality standards established for Federal Inspector
General Inspection and Evaluation offices and provide assurance that participating offices are
adequately evaluated.
16-N-0317
15

-------