o
w,
I SiUt I U.S. ENVIRONMENTAL PROTECTION AGENCY
VPro^° OFFICE OF INSPECTOR GENERAL
High-performing organization
Quality Assurance Review
of EPA OIG Reports Issued
in Fiscal Year 2017
Report No. 18-N-0219
July 9, 2018

-------
Report Contributors:	Jerri Dorsey-Hall
Janet Kasper
Abbreviations
CMR
Compliance Monitoring Review
EPA
U.S. Environmental Protection Agency
FY
Fiscal Year
GAGAS
Generally Accepted Government Auditing Standards
OA&E
Office of Audit and Evaluation
OIG
Office of Inspector General
PMH
Project Management Handbook
Are you aware of fraud, waste or abuse in an
EPA program?
EPA Inspector General Hotline
1200 Pennsylvania Avenue, NW (2431T)
Washington, DC 20460
(888) 546-8740
(202) 566-2599 (fax)
OIG Hotline@epa.gov
Learn more about our OIG Hotline.
EPA Office of Inspector General
1200 Pennsylvania Avenue, NW (2410T)
Washington, DC 20460
(202) 566-2391
www.epa.gov/oiq
Subscribe to our Email Updates
Follow us on Twitter @EPAoig
Send us your Project Suggestions

-------
^tDsrx
* Q \
\X!
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
18-N-0219
July 9, 2018
Why We Did This Project
The U.S. Environmental
Protection Agency's (EPA's)
Office of Inspector General
(OIG) operates and maintains
a system of quality controls
designed to provide
reasonable assurance that
personnel performing audit or
evaluation functions comply
with all generally accepted
government auditing standards
(GAGAS) and established OIG
policies and procedures.
Quality assurance staff from
the OIG's Office of Audit and
Evaluation report annually on
systemic issues identified
during referencing and
compliance monitoring
reviews, and make
observations on compliance
with GAGAS and OIG policy.
This report addresses the
following EPA OIG goal:
• Contribute to improved
business practices and
accountability.
Quality Assurance Review of EPA OIG
Reports issued in Fiscal Year 2017
What We Found
Internal reviews of OIG reports issued in fiscal
year (FY) 2017 found that the reports complied
with GAGAS and substantially complied with all
OIG policies and procedures. The external
peer review also found that the OIG's system
of quality control was suitably designed and
complied with to provide the EPA OIG with
reasonable assurance of performing and
reporting audits in conformity with GAGAS.
OIG audit reports issued in
FY 2017 demonstrated high
levels of compliance with
OIG quality assurance
procedures and received
average compliance scores
of nearly 94 percent. Issues
identified in FY 2016 have
been addressed.
OIG offices took actions during FY 2017 to address issues identified in the
quality assurance report for FY 2016 reports. For example, a review of FY 2017
reports found that teams are generally holding regular meetings with the
agency to provide updates on audits. Teams have also increased their
compliance with requirements related to the project guide and improved the
quality of indexing. Further, the results of the peer review of the EPA OIG
conducted by the Department of Defense OIG did not identify concerns in any
of these areas. The compliance monitoring and peer review results
demonstrate a high level of compliance with GAGAS and OIG procedures.
The FY 2017 compliance monitoring reviews indicated one systemic issue:
some assignments exceeded estimated staff and calendar days, and revisions
to estimated milestones were not always approved and documented. For
example, six of the 47 reports reviewed exceeded the approved estimated staff
days by more than 30 percent. GAGAS identifies timeliness as one of the
seven report quality elements. When reports are not timely, their relevance and
usefulness can be diminished.
Recommendation for Improvement
We recommend that the Assistant Inspector General for Audit and Evaluation
take action to have the OIG's Project Management Handbook updated to clarify
(a) what key information regarding assignment calendar days and staff days
must be approved, (b) when revisions are needed to the key information,
(c) who can approve revisions to key information, and (d) how key information
and revisions are to be documented.
Send all inquiries to our public
affairs office at (202) 566-2391
or visit www.epa.gov/oia.
Listing of OIG reports.
Assistant Inspector General Response
The Assistant Inspector General for Audit and Evaluation agreed with the
findings and recommendation, and indicated revisions to the Project
Management Handbook will be completed by September 30, 2018.

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
July 9,2018
MEMORANDUM
SUBJECT: Quality Assurance Review of EPA OIG Reports Issued in Fiscal Year 2017
Report No. 18-N-0219
FROM: Richard Eyermann and Christine El-Zoghbi
Deputy Assistant Inspectors General
Office of Audit and Evaluation
TO:
Kevin Christensen, Assistant Inspector General
Office of Audit and Evaluation
This is our report on the U.S. Environmental Protection Agency Office of Inspector General's (OIG's)
adherence to quality control elements and compliance with generally accepted government auditing
standards in fiscal year 2017 OIG reports. This report covers reports issued by the OIG's former Office
of Audit and Office of Program Evaluation.
This report offers observations and a recommendation to enhance and strengthen the OIG's project
execution process, and provides opportunities for improving adherence to internal controls within the
OIG.
cc: Charles Sheehan, Deputy Inspector General

-------
Quality Assurance Review of EPA OIG Reports
Issued in Fiscal Year 2017
18-N-0219
Table of C
Purpose		1
Background		1
Scope and Methodology		2
Issues Addressed in Previous Quality Assurance Report		3
Estimation and Approval of Project Timeframes
and Cost Estimates Need Improvement		4
Recommendation		5
Assistant Inspector General Response		5
Appendices
A OIG CMR Results for FY 2017		6
B Reports Without CMRs		7
C Implementation Plan		8

-------
Purpose
The U.S. Environmental Protection Agency (EPA) Office of Inspector General
(OIG) operates and maintains a system of quality control designed to provide
reasonable assurance that all personnel performing audit or evaluation functions
comply with generally accepted government auditing standards (GAGAS) and
established OIG policies and procedures. Quality Assurance staff from the OIG's
Office of Audit and Evaluation (OA&E)1 analyze and summarize the results of
their monitoring procedures at least annually to identify any systemic or repetitive
issues needing improvement, along with recommendations for corrective action.
This report summarizes our observations from our analysis of compliance
monitoring reviews (CMRs) for 47 audit projects for which reports were issued
from October 1, 2016, through September 30, 2017 (see Appendix A for a
listing)2 It also includes an update on the implementation of recommendations
from the prior quality assurance report.
Background
The Inspector General Act of 1978, as amended, requires that federal Inspectors
General comply with standards established by the Comptroller General of the
United States for audits. The OIG conducts its audits in accordance with these
standards, known as GAGAS. The OIG also maintains an internal system of quality
controls to provide the organization with reasonable assurance that its products,
services and personnel comply with professional standards and applicable legal and
regulatory requirements.
GAGAS Section 3.95 states that an audit organization:
should analyze and summarize the results of its monitoring process
at least annually, with identification of any systemic or repetitive
issues needing improvement, along with recommendations for
corrective action. The audit organization should communicate to
appropriate personnel any deficiencies noted during the monitoring
process and make recommendations for appropriate remedial
action.
A measuring process should provide a mechanism to evaluate individual products
against specific quality criteria. The process should also present the information in
a manner that, over time, will allow the OIG to assess adherence to quality control
elements so that necessary adjustments can be made to policies, procedures and
activities.
1	In December 2017, the OIG's Office of Audit and Office of Program Evaluation combined to become the Office of
Audit and Evaluation.
2	CMRs are prepared for each assignment. Some assignments may have multiple reports but only one CMR is
completed. Appendix B lists reports issued without CMRs and the reason CMRs were not performed.
18-N-0219
1

-------
In July 2014, the Inspector General signed OIG Policy and Procedure 006,
OIG Quality Control and Assurance Program, which identifies the OIG's quality
control and assurance process, including internal and external components such as
the CMR. Our system of quality control includes the use of CMRs as our ongoing
periodic assessment of work completed to determine whether the professional
standards are followed and the OIG is operating according to OIG Policy 101,
Project Management Handbook (PMH).
The CMR encompasses an evaluation of activities from the start of preliminary
research to when a team submits a final report and closes the working papers. The
CMR results, trends and resulting recommendations are summarized in our annual
quality control report. Compliance with general auditing standards—such as
independence, professional judgment, competence and adherence to continuing
professional education requirements—is not part of the CMR. The categories
evaluated in the CMR and associated total points are in Table 1.
Table 1: CMR categories
Category
Point value
Planning and Execution
12
Communication
13
Supervision
30
Report Quality
20
Timeliness
15
Post Reporting/Data Quality
10
Total
100
Source: OIG-generated.
On June 18, 2018, the Department of Defense OIG issued a report on its review of
the system of quality assurance for the EPA OIG in effect as of September 30,
2017. The review included a sample of reports issued in fiscal year (FY) 2017.
The Department of Defense OIG's overall conclusion was that the EPA OIG's
system of quality control was suitably designed and complied with to provide the
EPA OIG with reasonable assurance of performing and reporting in conformity
with applicable professional standards. In its management letter, the Department
of Defense OIG identified some issues that were not of sufficient significance to
affect its overall conclusion. In response, the EPA OIG proposed corrective
actions that are scheduled for completion by December 31, 2018.
Scope and Methodology
We performed this review on projects with final reports issued from October 1,
2016, to September 30, 2017. This review covered GAGAS-compliant reports by
the then OIG Office of Audit and Office of Program Evaluation (now combined
into the OA&E) and were reviewed and scored by the OIG's quality assurance
staff. We did not include any reports with work performed by external auditors.
18-N-0219
2

-------
The work performed in this review does not constitute an audit conducted in
accordance with GAGAS.
Because the peer review conducted by the Department of Defense OIG assessed
compliance with GAGAS for FY 2017 EPA OIG reports, this quality assurance
review was limited in scope to a review of the CMRs for projects with reports
issued in FY 2017. The scores for each report were scheduled, and any project
that received less than 80 percent in a category was reviewed to assess the issues
identified. The issues identified in the CMRs were then summarized and
presented as findings. The results of the peer review and review of the CMRs
were used to determine whether issues identified in past years have been
addressed.
In past years, the quality assurance report included an assessment of compliance
with independence and competency (i.e., training) standards. These areas were
covered in the peer review and so were not included as part of this quality
assurance report.
Issues Addressed in Previous Quality Assurance Report
OIG offices took actions during FY 2017 to address issues identified in the
previous quality assurance report.3 The FY 2016 quality assurance report
identified one significant issue regarding compliance with OIG procedures: that
teams were not conducting status meetings with the agency every 4 to 6 weeks, or
were not documenting those meetings, as required by the PMH. The FY 2017
CMRs found that this issue has been substantially addressed—the majority of the
teams consistently held and documented meetings with the agency. Although we
still found some instances where teams did not consistently hold or document
status meetings with the agency (for six of the 47 reports reviewed), because these
were isolated instances the issue is no longer considered systemic to the OIG and
is not included as an area for improvement. The Deputy Assistant Inspectors
General discussed the issue with appropriate staff as warranted.
The FY 2016 quality assurance report identified two issues of lesser significance
where improvement was needed. The first issue related to approval, revision and
indexing of the project guide, and the FY 2017 CMRs found that the corrective
action taken resulted in compliance. The second issue related to the quality of
report indexing, and the FY 2017 CMRs found substantial improvement in the
quality of indexing. The quality of indexing did not substantially affect the quality
assurance process for any final reports during FY 2017.
3 Quality Control Review of EPA OIG Reports Issued in Fiscal Year 2016 (Report No. 17-N-0295"). issued June 28,
2017.
18-N-0219
3

-------
Estimation and Approval of Project Timeframes and Cost Estimates
Need Improvement
Internal reviews of OIG reports issued in FY 2017 found that the reports complied
with GAGAS and substantially complied with all OIG policies and procedures.
The 47 OIG CMRs reviewed for that year showed average compliance scores of
nearly 94 percent. Details are in Appendix A.
Nonetheless, the FY 2017 CMRs indicated a systemic issue related to timeliness
and cost of projects. The CMRs found that some projects exceeded estimated staff
and calendar days and that revisions to estimated milestones were not always
approved and documented. GAGAS identifies timeliness as a report quality
element, noting in paragraph A.7.02 that "timely issuance of the report is an
important reporting goal for auditors." OIG procedures describe requirements
relating to the estimated project timeframes, approval of revisions to timeframes,
and documentation of approval of timeframes. When reports are not timely and
current, the report's relevance and usefulness can be diminished.
To ensure the timeliness of reports, the PMH identifies certain internal controls.
One of those internal controls is the Deputy Inspector General memorandum
issued for each report, as described in Section 2.10 of the PMH. The Deputy
Inspector General memorandum is prepared at the time of the decision to proceed
with the audit and includes the total project costs and final report issuance date for
the report. If changes are made to the Deputy Inspector General-designated
milestones (the final report date and final project costs), the PMH requires the
team to notify the Assistant Inspector General, who will then inform the Deputy
Inspector General of the need to revise the milestones.4 In the same section, the
PMH states that the approved Deputy Inspector General memorandum and
changes to milestone dates are to be documented in the working papers.
CMRs for FY 2017 reports found issues related to exceeding project estimates for
the final report and documentation of approval for original and revised project
estimates for the final report. For the 47 reports reviewed:
•	Six projects exceeded the approved estimated staff days by more than
30 percent (by 31 to 192 percent).
•	Three projects exceeded approved estimated calendar days by 4 months.
•	Three projects exceeded 90 calendar days for preliminary research.
•	Two projects did not have approved staff day budgets, so timeliness could
not be evaluated.
•	Five projects did not have approved milestone revisions documented in the
working papers.
4 In FY 2018, the OA&E Assistant Inspector General was delegated the authority to approve extensions of up to
60 days.
18-N-0219
4

-------
While the OIG has designed internal controls to ensure timeliness of reports, the
internal controls are not always being implemented. When reports are not issued
timely, the relevance of the information and the overall report quality decrease.
Recommendation
We recommend that the Assistant Inspector General for Audit and Evaluation:
1. Take action to have the Project Management Handbook updated to clarify:
a.	The key information regarding project calendar days and staff days
that must be approved.
b.	When revisions are needed to the key information.
c.	Who can approve revisions to key information.
d.	How key information and revisions are to be documented in
working papers and OIG information systems.
Assistant Inspector General Response
The Assistant Inspector General for Audit and Evaluation agreed with the findings
and recommendation, and met with the quality assurance staff on June 27, 2018, to
discuss the revisions to the PMH. The revised PMH is expected to be issued by
September 30, 2018.
18-N-0219
5

-------
Appendix A
OIG CMR Results for FY 2017
Report no.
Planning and
execution
Communication
Supervision
Report
quality
Timeliness
Data
quality
CMR total
17-F-0046
11.5
13
29.25
20
15
10
98.75
17-F-0228
12
12
27
20
7.5
8
86.5
17-F-0314
12
13
20.25
20
0
10
75.25
17-F-0315
12
13
22.5
20
0
8
75.5
17-F-0363
12
13
17
20
15
10
87
17-F-0364
12
8
30
20
7.5
10
87.5
17-F-0365
12
8
26
20
7.5
8
81.5
17-P-0044
12
12
27.25
20
15
8
94.25
17-P-0045
12
13
30
15
15
10
95
17-P-0050
8.5
5
28.5
20
7.5
10
79.5
17-P-0053
9
10
29.42
17
15
8
88.42
17-P-0062
12
12
28.5
17.5
7.5
10
87.5
17-P-0113
12
13
30
20
15
10
100
17-P-0118
12
13
27.6
20
15
10
97.6
17-P-0119
12
13
30
15
15
9.5
94.5
17-P-0123
12
13
30
20
15
10
100
17-P-0124
12
13
28.8
20
14.5
9.5
97.8
17-P-0362
11.9
10
28.8
18.5
11
8.5
88.7
17-P-0174
12
10.5
28.2
20
6
10
86.7
17-P-0183
12
13
30
20
14
8
97
17-P-0184
11
13
28.5
20
15
8
95.5
17-P-0186
12
12
30
20
15
8
97
17-P-0204
10
13
30
20
11
10
94
17-P-0205
12
11.5
30
20
7.5
9
90
17-P-0212
12
13
30
20
15
10
100
17-P-0249
12
13
27
20
13.5
10
95.5
17-P-0250
11
13
28.8
20
13
9.9
95.7
17-P-0278
11.5
12
27
20
15
10
95.5
17-P-0294
12
13
28.3
20
15
10
98.3
17-P-0303
11
13
30
19
15
10
98
17-P-0326
11.5
13
29.4
20
12.5
10
96.4
17-P-0343
12
13
30
20
15
10
100
17-P-0346
12
13
30
20
13
10
98
17-P-0350
12
12
30
20
15
10
99
17-P-0352
12
13
30
20
14
10
99
17-P-0368
12
13
29.25
17
11
10
92.25
17-P-0374
10
13
30
19
15
10
97
17-P-0377
12
13
26.5
20
15
10
96.5
17-P-0378
12
13
30
20
15
10
100
17-P-0380
12
13
30
20
9
10
94
17-P-0395
12
13
30
20
15
10
100
17-P-0396
12
13
28.2
20
12
9.9
95.1
17-P-0397
10.5
12
30
20
15
10
97.5
17-P-0402
12
13
30
19
15
10
99
17-P-0407
12
13
30
20
14
10
99
17-P-0408
12
13
30
20
15
10
100
17-P-0412
9.5
7
28.8
20
7.5
10
82.8
Average
11.59
12.13
28.53
19.51
12.36
9.58
93.70
Max. Score
12.00
13.00
30.00
20.00
15.00
10.00
100.00
Source: OIG-generated.
18-N-0219

-------
Appendix B
Reports Without CMRs
Report no.
Report title
Reason for no CMR
17-P-0004
Drinking Water Contamination in Flint, Michigan, Demonstrates a
Need to Clarify EPA Authority to Issue Emergency Orders to
Protect the Public
Interim report
17-P-0029
Acquisition Certifications Needed for Managers Overseeing
Development of EPA's Electronic Manifest System
Work evaluated as part of
Report No. 16-F-0251
17-F-0047
Audit of U.S. Chemical Safety and Hazard Investigation Board's
Fiscal Years 2016 and 2015 Financial Statements
Report done by contractor
17-P-0106
Management Alert: Certain State and Tribal Data Processing
Practices Could Impact Suitability of Data for 8-Hour Ozone
Air Quality Determinations
Interim report
17-P-0140
EPA's 2014 Early-Out and Buyout Activities Aided Workforce
Restructuring Goals, and Continued Monitoring of Progress Can
Show Value of Restructuring,
Work evaluated as part of
Report No. 17-P-0362
17-N-0219
EPA's Fiscal Year 2017 Management Challenges
Non-GAGAS report
17-N-0295
Quality Control Review of EPA OIG Reports Issued in
Fiscal Year 2016
Non-GAGAS report
17-N-0342
FY 2017 U.S. Chemical Safety and Hazard Investigation Board
Management Challenge
Non-GAGAS report
17-P-0344
EPA Lacks Processes to Validate Whether Contractors Receive
Specialized Role-Based Training for Network and Data Protection
Work evaluated as part of
Report No. 17-P-0044
17-P-0355
Management Alert: Concerns Over Compliance, Accountability and
Consistency Identified With EPA's Biweekly Pay Cap Waiver
Process
Interim report
17-P-0409
Management Alert: EPA Has Not Initiated Required Background
Investigations for Information Systems Contractor Personnel
Interim report
17-P-0410
Management Alert: Controls Failed to Prevent Employee From
Receiving Payment in Excess of Statutory Limit
Interim report
Source: OIG-generated.
18-N-0219
7

-------
Appendix C
Implementation Plan




Planned


Action

implementation
No.
Recommendation
official
Status
date
1
Take action to have the Project Management Handbook
updated to clarify:
a.	The key information regarding project calendar
days and staff days that must be approved.
b.	When revisions are needed to the key information.
c.	Who can approve revisions to key information.
d.	How key information and revisions are to be
documented in working papers and OIG
information systems.
Assistant
Inspector
General for
Audit and
Evaluation
R
9/30/18
Source: OIG-generated.
18-N-0219
8

-------