o w, I SiUt I U.S. ENVIRONMENTAL PROTECTION AGENCY VPro^° OFFICE OF INSPECTOR GENERAL High-performing organization Quality Assurance Review of EPA OIG Reports Issued in Fiscal Year 2017 Report No. 18-N-0219 July 9, 2018 ------- Report Contributors: Jerri Dorsey-Hall Janet Kasper Abbreviations CMR Compliance Monitoring Review EPA U.S. Environmental Protection Agency FY Fiscal Year GAGAS Generally Accepted Government Auditing Standards OA&E Office of Audit and Evaluation OIG Office of Inspector General PMH Project Management Handbook Are you aware of fraud, waste or abuse in an EPA program? EPA Inspector General Hotline 1200 Pennsylvania Avenue, NW (2431T) Washington, DC 20460 (888) 546-8740 (202) 566-2599 (fax) OIG Hotline@epa.gov Learn more about our OIG Hotline. EPA Office of Inspector General 1200 Pennsylvania Avenue, NW (2410T) Washington, DC 20460 (202) 566-2391 www.epa.gov/oiq Subscribe to our Email Updates Follow us on Twitter @EPAoig Send us your Project Suggestions ------- ^tDsrx * Q \ \X! U.S. Environmental Protection Agency Office of Inspector General At a Glance 18-N-0219 July 9, 2018 Why We Did This Project The U.S. Environmental Protection Agency's (EPA's) Office of Inspector General (OIG) operates and maintains a system of quality controls designed to provide reasonable assurance that personnel performing audit or evaluation functions comply with all generally accepted government auditing standards (GAGAS) and established OIG policies and procedures. Quality assurance staff from the OIG's Office of Audit and Evaluation report annually on systemic issues identified during referencing and compliance monitoring reviews, and make observations on compliance with GAGAS and OIG policy. This report addresses the following EPA OIG goal: • Contribute to improved business practices and accountability. Quality Assurance Review of EPA OIG Reports issued in Fiscal Year 2017 What We Found Internal reviews of OIG reports issued in fiscal year (FY) 2017 found that the reports complied with GAGAS and substantially complied with all OIG policies and procedures. The external peer review also found that the OIG's system of quality control was suitably designed and complied with to provide the EPA OIG with reasonable assurance of performing and reporting audits in conformity with GAGAS. OIG audit reports issued in FY 2017 demonstrated high levels of compliance with OIG quality assurance procedures and received average compliance scores of nearly 94 percent. Issues identified in FY 2016 have been addressed. OIG offices took actions during FY 2017 to address issues identified in the quality assurance report for FY 2016 reports. For example, a review of FY 2017 reports found that teams are generally holding regular meetings with the agency to provide updates on audits. Teams have also increased their compliance with requirements related to the project guide and improved the quality of indexing. Further, the results of the peer review of the EPA OIG conducted by the Department of Defense OIG did not identify concerns in any of these areas. The compliance monitoring and peer review results demonstrate a high level of compliance with GAGAS and OIG procedures. The FY 2017 compliance monitoring reviews indicated one systemic issue: some assignments exceeded estimated staff and calendar days, and revisions to estimated milestones were not always approved and documented. For example, six of the 47 reports reviewed exceeded the approved estimated staff days by more than 30 percent. GAGAS identifies timeliness as one of the seven report quality elements. When reports are not timely, their relevance and usefulness can be diminished. Recommendation for Improvement We recommend that the Assistant Inspector General for Audit and Evaluation take action to have the OIG's Project Management Handbook updated to clarify (a) what key information regarding assignment calendar days and staff days must be approved, (b) when revisions are needed to the key information, (c) who can approve revisions to key information, and (d) how key information and revisions are to be documented. Send all inquiries to our public affairs office at (202) 566-2391 or visit www.epa.gov/oia. Listing of OIG reports. Assistant Inspector General Response The Assistant Inspector General for Audit and Evaluation agreed with the findings and recommendation, and indicated revisions to the Project Management Handbook will be completed by September 30, 2018. ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF INSPECTOR GENERAL July 9,2018 MEMORANDUM SUBJECT: Quality Assurance Review of EPA OIG Reports Issued in Fiscal Year 2017 Report No. 18-N-0219 FROM: Richard Eyermann and Christine El-Zoghbi Deputy Assistant Inspectors General Office of Audit and Evaluation TO: Kevin Christensen, Assistant Inspector General Office of Audit and Evaluation This is our report on the U.S. Environmental Protection Agency Office of Inspector General's (OIG's) adherence to quality control elements and compliance with generally accepted government auditing standards in fiscal year 2017 OIG reports. This report covers reports issued by the OIG's former Office of Audit and Office of Program Evaluation. This report offers observations and a recommendation to enhance and strengthen the OIG's project execution process, and provides opportunities for improving adherence to internal controls within the OIG. cc: Charles Sheehan, Deputy Inspector General ------- Quality Assurance Review of EPA OIG Reports Issued in Fiscal Year 2017 18-N-0219 Table of C Purpose 1 Background 1 Scope and Methodology 2 Issues Addressed in Previous Quality Assurance Report 3 Estimation and Approval of Project Timeframes and Cost Estimates Need Improvement 4 Recommendation 5 Assistant Inspector General Response 5 Appendices A OIG CMR Results for FY 2017 6 B Reports Without CMRs 7 C Implementation Plan 8 ------- Purpose The U.S. Environmental Protection Agency (EPA) Office of Inspector General (OIG) operates and maintains a system of quality control designed to provide reasonable assurance that all personnel performing audit or evaluation functions comply with generally accepted government auditing standards (GAGAS) and established OIG policies and procedures. Quality Assurance staff from the OIG's Office of Audit and Evaluation (OA&E)1 analyze and summarize the results of their monitoring procedures at least annually to identify any systemic or repetitive issues needing improvement, along with recommendations for corrective action. This report summarizes our observations from our analysis of compliance monitoring reviews (CMRs) for 47 audit projects for which reports were issued from October 1, 2016, through September 30, 2017 (see Appendix A for a listing)2 It also includes an update on the implementation of recommendations from the prior quality assurance report. Background The Inspector General Act of 1978, as amended, requires that federal Inspectors General comply with standards established by the Comptroller General of the United States for audits. The OIG conducts its audits in accordance with these standards, known as GAGAS. The OIG also maintains an internal system of quality controls to provide the organization with reasonable assurance that its products, services and personnel comply with professional standards and applicable legal and regulatory requirements. GAGAS Section 3.95 states that an audit organization: should analyze and summarize the results of its monitoring process at least annually, with identification of any systemic or repetitive issues needing improvement, along with recommendations for corrective action. The audit organization should communicate to appropriate personnel any deficiencies noted during the monitoring process and make recommendations for appropriate remedial action. A measuring process should provide a mechanism to evaluate individual products against specific quality criteria. The process should also present the information in a manner that, over time, will allow the OIG to assess adherence to quality control elements so that necessary adjustments can be made to policies, procedures and activities. 1 In December 2017, the OIG's Office of Audit and Office of Program Evaluation combined to become the Office of Audit and Evaluation. 2 CMRs are prepared for each assignment. Some assignments may have multiple reports but only one CMR is completed. Appendix B lists reports issued without CMRs and the reason CMRs were not performed. 18-N-0219 1 ------- In July 2014, the Inspector General signed OIG Policy and Procedure 006, OIG Quality Control and Assurance Program, which identifies the OIG's quality control and assurance process, including internal and external components such as the CMR. Our system of quality control includes the use of CMRs as our ongoing periodic assessment of work completed to determine whether the professional standards are followed and the OIG is operating according to OIG Policy 101, Project Management Handbook (PMH). The CMR encompasses an evaluation of activities from the start of preliminary research to when a team submits a final report and closes the working papers. The CMR results, trends and resulting recommendations are summarized in our annual quality control report. Compliance with general auditing standards—such as independence, professional judgment, competence and adherence to continuing professional education requirements—is not part of the CMR. The categories evaluated in the CMR and associated total points are in Table 1. Table 1: CMR categories Category Point value Planning and Execution 12 Communication 13 Supervision 30 Report Quality 20 Timeliness 15 Post Reporting/Data Quality 10 Total 100 Source: OIG-generated. On June 18, 2018, the Department of Defense OIG issued a report on its review of the system of quality assurance for the EPA OIG in effect as of September 30, 2017. The review included a sample of reports issued in fiscal year (FY) 2017. The Department of Defense OIG's overall conclusion was that the EPA OIG's system of quality control was suitably designed and complied with to provide the EPA OIG with reasonable assurance of performing and reporting in conformity with applicable professional standards. In its management letter, the Department of Defense OIG identified some issues that were not of sufficient significance to affect its overall conclusion. In response, the EPA OIG proposed corrective actions that are scheduled for completion by December 31, 2018. Scope and Methodology We performed this review on projects with final reports issued from October 1, 2016, to September 30, 2017. This review covered GAGAS-compliant reports by the then OIG Office of Audit and Office of Program Evaluation (now combined into the OA&E) and were reviewed and scored by the OIG's quality assurance staff. We did not include any reports with work performed by external auditors. 18-N-0219 2 ------- The work performed in this review does not constitute an audit conducted in accordance with GAGAS. Because the peer review conducted by the Department of Defense OIG assessed compliance with GAGAS for FY 2017 EPA OIG reports, this quality assurance review was limited in scope to a review of the CMRs for projects with reports issued in FY 2017. The scores for each report were scheduled, and any project that received less than 80 percent in a category was reviewed to assess the issues identified. The issues identified in the CMRs were then summarized and presented as findings. The results of the peer review and review of the CMRs were used to determine whether issues identified in past years have been addressed. In past years, the quality assurance report included an assessment of compliance with independence and competency (i.e., training) standards. These areas were covered in the peer review and so were not included as part of this quality assurance report. Issues Addressed in Previous Quality Assurance Report OIG offices took actions during FY 2017 to address issues identified in the previous quality assurance report.3 The FY 2016 quality assurance report identified one significant issue regarding compliance with OIG procedures: that teams were not conducting status meetings with the agency every 4 to 6 weeks, or were not documenting those meetings, as required by the PMH. The FY 2017 CMRs found that this issue has been substantially addressed—the majority of the teams consistently held and documented meetings with the agency. Although we still found some instances where teams did not consistently hold or document status meetings with the agency (for six of the 47 reports reviewed), because these were isolated instances the issue is no longer considered systemic to the OIG and is not included as an area for improvement. The Deputy Assistant Inspectors General discussed the issue with appropriate staff as warranted. The FY 2016 quality assurance report identified two issues of lesser significance where improvement was needed. The first issue related to approval, revision and indexing of the project guide, and the FY 2017 CMRs found that the corrective action taken resulted in compliance. The second issue related to the quality of report indexing, and the FY 2017 CMRs found substantial improvement in the quality of indexing. The quality of indexing did not substantially affect the quality assurance process for any final reports during FY 2017. 3 Quality Control Review of EPA OIG Reports Issued in Fiscal Year 2016 (Report No. 17-N-0295"). issued June 28, 2017. 18-N-0219 3 ------- Estimation and Approval of Project Timeframes and Cost Estimates Need Improvement Internal reviews of OIG reports issued in FY 2017 found that the reports complied with GAGAS and substantially complied with all OIG policies and procedures. The 47 OIG CMRs reviewed for that year showed average compliance scores of nearly 94 percent. Details are in Appendix A. Nonetheless, the FY 2017 CMRs indicated a systemic issue related to timeliness and cost of projects. The CMRs found that some projects exceeded estimated staff and calendar days and that revisions to estimated milestones were not always approved and documented. GAGAS identifies timeliness as a report quality element, noting in paragraph A.7.02 that "timely issuance of the report is an important reporting goal for auditors." OIG procedures describe requirements relating to the estimated project timeframes, approval of revisions to timeframes, and documentation of approval of timeframes. When reports are not timely and current, the report's relevance and usefulness can be diminished. To ensure the timeliness of reports, the PMH identifies certain internal controls. One of those internal controls is the Deputy Inspector General memorandum issued for each report, as described in Section 2.10 of the PMH. The Deputy Inspector General memorandum is prepared at the time of the decision to proceed with the audit and includes the total project costs and final report issuance date for the report. If changes are made to the Deputy Inspector General-designated milestones (the final report date and final project costs), the PMH requires the team to notify the Assistant Inspector General, who will then inform the Deputy Inspector General of the need to revise the milestones.4 In the same section, the PMH states that the approved Deputy Inspector General memorandum and changes to milestone dates are to be documented in the working papers. CMRs for FY 2017 reports found issues related to exceeding project estimates for the final report and documentation of approval for original and revised project estimates for the final report. For the 47 reports reviewed: • Six projects exceeded the approved estimated staff days by more than 30 percent (by 31 to 192 percent). • Three projects exceeded approved estimated calendar days by 4 months. • Three projects exceeded 90 calendar days for preliminary research. • Two projects did not have approved staff day budgets, so timeliness could not be evaluated. • Five projects did not have approved milestone revisions documented in the working papers. 4 In FY 2018, the OA&E Assistant Inspector General was delegated the authority to approve extensions of up to 60 days. 18-N-0219 4 ------- While the OIG has designed internal controls to ensure timeliness of reports, the internal controls are not always being implemented. When reports are not issued timely, the relevance of the information and the overall report quality decrease. Recommendation We recommend that the Assistant Inspector General for Audit and Evaluation: 1. Take action to have the Project Management Handbook updated to clarify: a. The key information regarding project calendar days and staff days that must be approved. b. When revisions are needed to the key information. c. Who can approve revisions to key information. d. How key information and revisions are to be documented in working papers and OIG information systems. Assistant Inspector General Response The Assistant Inspector General for Audit and Evaluation agreed with the findings and recommendation, and met with the quality assurance staff on June 27, 2018, to discuss the revisions to the PMH. The revised PMH is expected to be issued by September 30, 2018. 18-N-0219 5 ------- Appendix A OIG CMR Results for FY 2017 Report no. Planning and execution Communication Supervision Report quality Timeliness Data quality CMR total 17-F-0046 11.5 13 29.25 20 15 10 98.75 17-F-0228 12 12 27 20 7.5 8 86.5 17-F-0314 12 13 20.25 20 0 10 75.25 17-F-0315 12 13 22.5 20 0 8 75.5 17-F-0363 12 13 17 20 15 10 87 17-F-0364 12 8 30 20 7.5 10 87.5 17-F-0365 12 8 26 20 7.5 8 81.5 17-P-0044 12 12 27.25 20 15 8 94.25 17-P-0045 12 13 30 15 15 10 95 17-P-0050 8.5 5 28.5 20 7.5 10 79.5 17-P-0053 9 10 29.42 17 15 8 88.42 17-P-0062 12 12 28.5 17.5 7.5 10 87.5 17-P-0113 12 13 30 20 15 10 100 17-P-0118 12 13 27.6 20 15 10 97.6 17-P-0119 12 13 30 15 15 9.5 94.5 17-P-0123 12 13 30 20 15 10 100 17-P-0124 12 13 28.8 20 14.5 9.5 97.8 17-P-0362 11.9 10 28.8 18.5 11 8.5 88.7 17-P-0174 12 10.5 28.2 20 6 10 86.7 17-P-0183 12 13 30 20 14 8 97 17-P-0184 11 13 28.5 20 15 8 95.5 17-P-0186 12 12 30 20 15 8 97 17-P-0204 10 13 30 20 11 10 94 17-P-0205 12 11.5 30 20 7.5 9 90 17-P-0212 12 13 30 20 15 10 100 17-P-0249 12 13 27 20 13.5 10 95.5 17-P-0250 11 13 28.8 20 13 9.9 95.7 17-P-0278 11.5 12 27 20 15 10 95.5 17-P-0294 12 13 28.3 20 15 10 98.3 17-P-0303 11 13 30 19 15 10 98 17-P-0326 11.5 13 29.4 20 12.5 10 96.4 17-P-0343 12 13 30 20 15 10 100 17-P-0346 12 13 30 20 13 10 98 17-P-0350 12 12 30 20 15 10 99 17-P-0352 12 13 30 20 14 10 99 17-P-0368 12 13 29.25 17 11 10 92.25 17-P-0374 10 13 30 19 15 10 97 17-P-0377 12 13 26.5 20 15 10 96.5 17-P-0378 12 13 30 20 15 10 100 17-P-0380 12 13 30 20 9 10 94 17-P-0395 12 13 30 20 15 10 100 17-P-0396 12 13 28.2 20 12 9.9 95.1 17-P-0397 10.5 12 30 20 15 10 97.5 17-P-0402 12 13 30 19 15 10 99 17-P-0407 12 13 30 20 14 10 99 17-P-0408 12 13 30 20 15 10 100 17-P-0412 9.5 7 28.8 20 7.5 10 82.8 Average 11.59 12.13 28.53 19.51 12.36 9.58 93.70 Max. Score 12.00 13.00 30.00 20.00 15.00 10.00 100.00 Source: OIG-generated. 18-N-0219 ------- Appendix B Reports Without CMRs Report no. Report title Reason for no CMR 17-P-0004 Drinking Water Contamination in Flint, Michigan, Demonstrates a Need to Clarify EPA Authority to Issue Emergency Orders to Protect the Public Interim report 17-P-0029 Acquisition Certifications Needed for Managers Overseeing Development of EPA's Electronic Manifest System Work evaluated as part of Report No. 16-F-0251 17-F-0047 Audit of U.S. Chemical Safety and Hazard Investigation Board's Fiscal Years 2016 and 2015 Financial Statements Report done by contractor 17-P-0106 Management Alert: Certain State and Tribal Data Processing Practices Could Impact Suitability of Data for 8-Hour Ozone Air Quality Determinations Interim report 17-P-0140 EPA's 2014 Early-Out and Buyout Activities Aided Workforce Restructuring Goals, and Continued Monitoring of Progress Can Show Value of Restructuring, Work evaluated as part of Report No. 17-P-0362 17-N-0219 EPA's Fiscal Year 2017 Management Challenges Non-GAGAS report 17-N-0295 Quality Control Review of EPA OIG Reports Issued in Fiscal Year 2016 Non-GAGAS report 17-N-0342 FY 2017 U.S. Chemical Safety and Hazard Investigation Board Management Challenge Non-GAGAS report 17-P-0344 EPA Lacks Processes to Validate Whether Contractors Receive Specialized Role-Based Training for Network and Data Protection Work evaluated as part of Report No. 17-P-0044 17-P-0355 Management Alert: Concerns Over Compliance, Accountability and Consistency Identified With EPA's Biweekly Pay Cap Waiver Process Interim report 17-P-0409 Management Alert: EPA Has Not Initiated Required Background Investigations for Information Systems Contractor Personnel Interim report 17-P-0410 Management Alert: Controls Failed to Prevent Employee From Receiving Payment in Excess of Statutory Limit Interim report Source: OIG-generated. 18-N-0219 7 ------- Appendix C Implementation Plan Planned Action implementation No. Recommendation official Status date 1 Take action to have the Project Management Handbook updated to clarify: a. The key information regarding project calendar days and staff days that must be approved. b. When revisions are needed to the key information. c. Who can approve revisions to key information. d. How key information and revisions are to be documented in working papers and OIG information systems. Assistant Inspector General for Audit and Evaluation R 9/30/18 Source: OIG-generated. 18-N-0219 8 ------- |