£ < o .36. o \^yy 2 U.S. ENVIRONMENTAL PROTECTION AGENCY %pR0^° OFFICE OF INSPECTOR GENERAL Operating efficiently and effectively EPA Needs to Address Internal Control Deficiencies in the Agencywide Quality System Report No. 20-P-0200 June 22, 2020 ~ ~ I I \ \ V Agencywide Quality System ------- Report Contributors: Patrick Gilbride Erin Barnes-Weaver Alicia Buchanan Jenny Drzewiecki Fred Light Abbreviations CIO Chief Information Officer EPA U.S. Environmental Protection Agency EQMD Enterprise Quality Management Division GAO U.S. Government Accountability Office OEIP Office of Enterprise Information Programs OIG Office of Inspector General OMS Office of Mission Support ORD Office of Research and Development QSA Quality System Assessment Cover Image: A graphic representing the agencywide Quality System. (EPA OIG graphic) Are you aware of fraud, waste, or abuse in an EPA program? EPA Inspector General Hotline 1200 Pennsylvania Avenue. NW(2431T) Washington. D.C. 20460 (888) 546-8740 (202) 566-2599 (fax) OIG Hotline@epa.gov Learn more about our OIG Hotline. EPA Office of Inspector General 1200 Pennsylvania Avenue, NW (2410T) Washington, D.C. 20460 (202) 566-2391 www.epa.gov/oiq Subscribe to our Email Updates Follow us on Twitter @EPAoig Send us your Project Suggestions ------- x-^tD Sf/ljv * *. U.S. Environmental Protection Agency 20-P-0200 § mmj \ Office of Inspector General June 22,2020 UsSi At a Glance Why We Did This Project A November 2018 U.S. Environmental Protection Agency Office of Inspector General scientific integrity survey yielded hotline complaints on potential internal control issues with the EPA's Quality System. We conducted this audit to determine whether the Office of Mission Support has controls in place to carry out its responsibility in developing and coordinating the mandatory agencywide Quality System. The Quality System provides requirements for conducting quality management activities for all environmental data collection performed by or for the Agency. Its primary goal is to ensure that environmental data are of sufficient quantity and quality to support intended uses. Each EPA office implements a quality system, and Quality System staff is responsible for its oversight, policies, procedures, training, and tracking. This report addresses the following: • Operating efficiently and effectively. This project addresses a key EPA management challenge: • Improving data quality and filling identified data gaps. Address inquiries to our public affairs office at (202) 566-2391 or OIG WEBCOMMENTS@epa.gov. List of OIG reports. EPA Needs to Address Internal Control Deficiencies in the Agencywide Quality System After five years and $1.3 million towards the development of an agencywide tracking system, the OMS does not know the status of the agencywide Quality System. What We Found The OMS has not fully implemented internal controls for the mandatory agencywide Quality System. We found that the OMS has not reviewed policies, procedures, and guidance within required time frames. For example, reviews of two quality policies were 15 years overdue. We also found that the OMS has not conducted required annual reviews for five years. The OMS has also not conducted regular assessments of program and regional quality systems as more than half of the systems had not had a review in the past six years. Also, the OMS has not assessed staff and resource needs since 2008, and the OMS has not performed a programmatic risk assessment. Additionally, the OMS has not developed a strategic plan, implemented a tracking system, or provided agencywide training. OMS leaders and staff identified four factors that led to control deficiencies: (1) Quality System leaders overtime have had varying priorities; (2) Quality System staff had a backlog of work; (3) Quality System leaders determined that variations in the length, details, and format of annual reviews made them difficult to analyze and compare; and (4) the Quality System lacked resources for its work. The EPA and the public rely upon the quality of the Agency's data, which helps the Agency make reliable, cost-effective, and defensible decisions. Additionally, the EPA uses its Quality System to manage the quality of its environmental data generation, collection, and use. The Quality System covers activities such as determining hazardous or toxic wastes in the environment and establishing health risk levels, supporting enforcement monitoring efforts, and mapping human health risk data. Poor data quality negatively impacts the EPA's effectiveness in monitoring programs that directly impact public health and could also subject the EPA to significant financial and legal risks. OMS leaders agree that the Quality System needs improvement and are taking steps to strengthen its internal control system. However, until the OMS fully implements internal controls, it cannot know the status or health of the agencywide Quality System. Recommendations and Planned Agency Corrective Actions We made 15 recommendations to the assistant administrator for Mission Support to improve internal controls for the agencywide Quality System. The OMS agreed with 13 recommendations, which are either completed or resolved with corrective actions pending. The OMS disagreed with two recommendations which are unresolved with resolution efforts in progress. ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 THE INSPECTOR GENERAL June 22, 2020 MEMORANDUM SUBJECT: EPA Needs to Address Internal Control Deficiencies in the Agencywide Quality System Report No. 20-P-0200 This is our report on the subject audit conducted by the Office of Inspector General of the U.S. Environmental Protection Agency. The project number for this audit was OA&E-FY19-0329. This report contains findings that describe the problem the OIG has identified and corrective actions the OIG recommends. The Office of Enterprise Information Programs' Enterprise Quality Management Division, within the Office of Mission Support, is primarily responsible for the subjects discussed in this report. Action Required While your office provided acceptable corrective actions and estimated milestone dates for most OIG recommendations, which are resolved, this report also contains two unresolved recommendations. In accordance with EPA Manual 2750, the resolution process for the two unresolved recommendations begins immediately with the issuance of this report. We are requesting a meeting within 30 days between the director of the Office of Enterprise Information Programs and the OIG's assistant inspector general for Audit and Evaluation. We also request a written response to the final report within 60 days of this memorandum. Your response will be posted on the OIG's website, along with our memorandum commenting on your response. Your response should be provided as an Adobe PDF file that complies with the accessibility requirements of Section 508 of the Rehabilitation Act of 1973, as amended. The final response should not contain data that you do not want to be released to the public; if your response contains such data, you should identify the data for redaction or removal along with corresponding justification. If resolution is still not reached, the Office of Mission Support is required to complete and submit a dispute resolution request to the chief financial officer. We will post this report to our website at www.epa.gov/oig. FROM: Sean W. O'Donnell TO: Donna J. Vizian, Principal Deputy Assistant Administrator Office of Mission Support ------- EPA Needs to Address Internal Control Deficiencies in the Agencywide Quality System 20-P-0200 Table of C Chapters 1 Introduction 1 Purpose 1 Background 1 Responsible Office 6 Scope and Methodology 6 2 Further Efforts Needed to Implement Controls 8 Control Environment 9 Risk Assessment 13 Control Activities 13 Information and Communication 15 Monitoring 16 Conclusions 18 Agency Response and OIG Assessment 19 Status of Recommendations and Potential Monetary Benefits 20 Appendices A Agency Response to Draft Report 21 B Distribution 25 ------- Chapter 1 Introduction Purpose The U.S. Environmental Protection Agency Office of Inspector General conducted a scientific integrity survey in late 2018, which yielded several hotline complaints that indicated potential internal control issues with the EPA's Quality System overseen by the Agency's Office of Mission Support. The hotline complaints pertained to leadership, organization, staffing, and tracking of Quality System documents. We conducted this audit to determine whether the OMS has controls in place to carry out its responsibility in developing and coordinating the mandatory agencywide Quality System. Background EPA's Agencywide Quality System The EPA's agencywide Quality System provides requirements for conducting quality management activities for all environmental data collected by or for the Agency. According to the EPA's "About EPA's Quality System" webpage, the primary goal of the Quality System is to "ensure that [the Agency's] environmental data are of sufficient quantity and quality to support the data's intended use." The Quality System provides the framework for planning, implementing, documenting, and assessing work performed by the Agency, and for carrying out required quality assurance and quality control activities. Examples of activities covered by the Quality System include: • Characterizing or evaluating ecological systems or human health. • Monitoring effluent discharges from operations. Key Management Challenge This audit addresses the following key management challenge for the Agency, as identified in OIG Report No. 19-N-0235, FY 2019 EPA Management Challenges, issued July 15, 2019: • Improving data quality and filling identified data qaps. Environmental data are any measures or information that describe environmental processes, locations, or conditions; ecological or health effects and consequences; or the performance of environmental technology. For the EPA, environmental data include both primary data (i.e., information collected directly from measurements) and secondary or existing data collected for other purposes or obtained from other sources (e.g., literature, industry surveys, models, databases, and information systems). Source: EPA, Overview of the EPA Quality System for Environmental Data and Technology (November 2002). 20-P-0200 1 ------- • Mapping environmental conditions or human health risk data. • Developing models to characterize environmental processes. • Establishing ambient air conditions. According to the EPA, successful implementation of the Quality System leads to: • Scientific data integrity: The EPA will produce data of known and documented quality based on sound scientific principles. • Reduced or justifiable resource expenditures: The EPA can reduce resource expenditures if the information collected matches the EPA's information needs. Through proper planning, the EPA will collect only the correct type, amount, and quality data for its use. • Proper evaluation of internal and external activities: The EPA Quality System provides documentation of activities and improved oversight for evaluation purposes, which reduces the potential for waste and abuse. • Reliable and defensible decisions: It is easier to determine whether the data can be used for a specific decision when the quality of data is known. This reduces surprises and challenges to, among other things, regulations and permits. • Burden reduction: As the EPA better defines the data needed for a specific application, the burden on other organizations that are required to collect or report data to the EPA may be reduced. The agencywide Quality System covers all EPA offices, regions, and laboratories that collect, evaluate, or use environmental data, or design, construct, or operate environmental technology. The Quality System comprises more than 40 individual quality systems developed and implemented by the various EPA regions, national program offices, and the Office of Research and Development. Because of the diversity and dispersion of programs within the EPA, different organizations within the Agency have individual quality systems that specifically address their needs. Figure 1 depicts the main components of the EPA Quality System. 20-P-0200 2 ------- Figure 1: Components of the EPA Quality System Project Planning £ Training Methods and techniques to ensure responsible personnel have the skills and knowledge to complete tasks in accordance with their organizational quality system's policies and procedures Documented in Quality Assurance Project Plan to develop data performance criteria, a plan to satisfy criteria, and quality control activifes to ensure criteria are satisfied Organizational Assessment Quality System Assessments to determine compfiance with regulations and conformance to policies and plans Annual Reviews & Planning Quality Assurance Annual Report and Wortc Plan submitted by each organization Project Ov j Implementation Agencywide Quality System £ Ensure EPA environmental data are of sufficient quantity and quality to support the data's intended use. Documentation Quality Management Plans describing authorities, policies, procedures, and training specificto the organization Data collected and environmental technology implemented according to the Quality Assurance Project Plan and standard operating procedures Project Assessment Verify, validate, and analyze data to determine if Quality Assurance Project Plan performance criteria and users' needs are met I ~l Policies Internal documents for EPA organizations and Federal Acquisition Regulations for non- EPA organizations Source; EPA OIG graphic based on Quality System documents. Responsibility for the EPA's Quality System The deputy assistant administrator for Environmental Information in the OMS, who is also the chief information officer, serves as the senior quality management official for the Agency and is responsible for assessing and approving each EPA organization's quality system. The OMS is responsible for developing, coordinating, and overseeing the agencywide Quality System. The Environmental Quality Management Division in the Office of Enterprise Information Programs has oversight responsibility of the system. EQMD's Quality System staff conducts program implementation and coordinates with program and regional offices. The EQMD staff were developing the Quality Assurance Enterprise Management System, an agencywide tracking system that the EPA planned to implement in fiscal year 2020. As of November 2019, the EQMD had eight full-time equivalent staff. Table 1 lists Quality System responsibilities for the EQMD. 20-P-0200 3 ------- Table 1: EQMD's responsibilities for the agencywide Quality System Quality System policies Develop and issue policies for agencywide use. Quality Management Plan Review and approve the EPA organization's plan. Quality System procedures and guidance Develop and issue procedures and guidance for the EPA and non-EPA organizations funded by the EPA. Resources for Quality System activities Monitor and balance resource allocation across the Agency and recommend improvements. Quality System Assessments Periodically review each EPA organization, identify agencywide problems, and mandate corrective actions. Quality Assurance Annual Report and Work Plan Compile information in a report to the OMS assistant administrator and the EPA administrator. Communication and outreach Perform outreach by hosting monthly conference calls, participate in the annual National Quality Assurance Conference, and represent the EPA on quality practices and issues. Training Develop and issue training materials and provide generic training on a limited basis. Employee evaluation (performance) standards Develop and issue general standards policy. Source: OIG analysis of Quality System policies, procedures, and guidance. In addition to Quality System staff within the EQMD, quality assurance managers in each program and regional office serve as the primary contacts for their organization's quality system. Oversight responsibility for the agencywide Quality System has switched offices within the EPA. This function was in the ORD before the EPA transferred it to the then-newly formed Office of Environmental Information in 2000. It was renamed and reorganized under the OMS in 2018. All OMS Quality System leadership changed in FY 2019, including a new CIO, OEIP director and deputy, and EQMD director. Policy and Program Requirements Since 1979, the Agency required all EPA offices and all non-EPA organizations performing work on behalf of the Agency through extramural agreements, such as contractors, to participate in the Quality System. Each EPA office is required to develop and implement a quality system that complies with EPA Order CIO 2105.0 Policy and Program Requirements for the Mandatory Agency-Wide Quality System, approved May 5, 2000. That Order establishes the minimum requirements for quality systems that collect, evaluate, and use environmental data by or for the EPA. 20-P-0200 4 ------- The EPA's centralized Quality Staff develops management practices and documents for use agencywide to enable effective planning, implementation, documentation, and assessment of individual quality systems. This includes developing and maintaining numerous policies, procedures, and guidance documents, such as those found on the EPA's Quality System website. The EPA Quality Manual for Environmental Programs, CIO 2105-P-01-0, dated May 5, 2000, makes all EPA quality-related requirements and guidance documents valid for five years from the approval date, at which time the Agency must take action to reaffirm the document's validity, revise it, or delete it from the agencywide Quality System. Federal and Agency Guidance on Internal Controls The U.S. Government Accountability Office defines internal control as: [A] process effected by an entity's oversight body, management, and other personnel that provides reasonable assurance that the objectives of an entity will be achieved. ... Internal control comprises the plans, methods, policies, and procedures used to fulfill the mission, strategic plan, goals, and objectives of the entity.1 An internal control system is defined as: [A] continuous built-in component of operations, effected by people, that provides reasonable assurance, not absolute assurance, that an entity's objectives will be achieved. ... Internal control is not one event, but a series of actions that occur throughout an entity's operations. ... Management is responsible for an effective internal control system. As part of this responsibility, management sets the entity's objectives, implements controls, and evaluates the internal control system.2 Table 2 lists the five components and 17 principles of internal control. The GAO notes that the "17 principles support the effective design, implementation, and operation of the associated components and represent [the] requirements necessary to establish an effective internal control system." 1 GAO, Standards for Internal Control in the Federal Government, GAO-14-704G, at 5 (September 2014). 2 Ibid, at 5-6. 20-P-0200 5 ------- Table 2: Internal control components and principles Components Principles Control Environment 1. Demonstrate commitment to integrity and ethical values. 2. Exercise oversight responsibility. 3. Establish structure, responsibility, and authority. 4. Demonstrate commitment to competence. 5. Enforce accountability. 6. Define objective and risk tolerances. 7. Identify, analyze, and respond to risks. 8. Assess fraud risk. 9. Identify, analyze, and respond to change. 10. Design control activities. 11. Design activities for information systems. 12. Implement control activities. 13. Use quality information. 14. Communicate internally. 15. Communicate externally. 16. Perform monitoring activities. 17. Remediate deficiency. Risk Assessment Control Activities Information and Communication Monitoring Source: OIG summary of the GAO's Standards for Internal Control in the Federal Government, GAO-14-704G, September 2014. The Office of Management and Budget Circular A-123, Management's Responsibility for Enterprise Risk Management and Internal Control, issued July 2016, defines obligations for risk management and internal control in federal agencies. EPA Order 1000.24 CHG 2, Management's Responsibility for Internal Control, approved July 18, 2008, requires all EPA organizations to establish and maintain internal controls to achieve effective and efficient program operations, including evaluating internal controls on an ongoing basis and taking prompt actions to correct any vulnerabilities identified. The OEIP's EQMD is primarily responsible for the subjects discussed in this report. We performed our work from September 2019 through April 2020. We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. We assessed internal controls necessary to satisfy the audit objective. In particular, we assessed all internal control components and underlying principles Responsible Office Scope and Methodology 20-P-0200 6 ------- as each was significant to the audit objective. Table 2 lists internal control components and underlying principles significant to the audit objective. To answer our objective, we reviewed the EPA's quality policies, procedures, and guidance documents as well as other Agency materials including websites, training courses, budget, and resources. We interviewed key OMS staff and managers responsible for the Quality System, including the chief information officer, OEIP director and deputy director, the current and former EQMD director, and EQMD's Quality System staff. We also attended monthly quality community calls and reviewed Quality Management Plans, QSAs, and the new tracking system's test environment. We benchmarked where other federal agencies organizationally place their quality systems. In addition, to obtain feedback on the EPA's Quality System, we interviewed: • Hotline complainants and EPA quality contacts suggested by complainants. • A random sample of ten national Quality System personnel, many of whom serve as quality assurance managers for their organizations. • Four regional quality assurance managers. We reviewed federal and Agency guidance on internal controls as well as OMS's annual management integrity assurance letters for the past four years. Additionally, we reviewed prior relevant reports issued by the OIG, including our annual list of EPA management challenges which identified "data quality" as an Agency challenge for nine years including fiscal years 2001-2007, 2018, and 2019. We also reviewed GAO's prior reports, as well as a 2014 program evaluation of the EPA's Quality System conducted by an outside contractor. 20-P-0200 7 ------- Chapter 2 Further Efforts Needed to Implement Controls The OMS has not fully implemented internal controls for the mandatory agencywide Quality System. GAO's Standards for Internal Control in the Federal Government describes management's responsibility for an effective internal control system. As part of this responsibility, per GAO, management sets the entity's objectives, implements controls, and evaluates the internal control system using the five components of internal control. We identified issues that the OMS as with all five components of the internal control system: • Control Environment: The OMS has not developed a strategic plan, defined objectives, provided oversight via annual reporting, or adequately assessed the workload for the Quality System staff. • Risk Assessment: Absent a strategic plan and objectives, the OMS has not conducted a risk assessment to define risk tolerances or identify risks requiring mitigation. • Control Activities: The OMS has not reviewed policies, procedures, and guidance within required time frames. For example, reviews of two quality policies were 15 years overdue. Additionally, the OMS has not provided agencywide training in over two years. • Information and Communication: The EPA's Quality System website has numerous outdated policies, procedures, and guidance documents, as well as inaccuracies in its list of the EPA's national quality system personnel. Additionally, half of 26 interviewees expressed concerns about the adequacy of OMS's communication and outreach. • Monitoring: The OMS did not address three recommendations from a November 2014 program evaluation. Additionally, the OMS has not implemented a tracking system. OMS leaders and staff identified four factors that led to control deficiencies: (1) Quality System leaders over time have had varying priorities; (2) Quality System staff had a backlog of work; (3) Quality System leaders determined that variations in the length, details, and format of annual reviews made them difficult to analyze and compare; and (4) the Quality System lacked resources for its work. The EPA and the public rely upon the quality of the Agency's data. For the public, data quality directly impacts decision quality, and poor data quality can mask risks to public health. Quality data helps the Agency make reliable, cost- 20-P-0200 8 ------- effective, and defensible decisions. Additionally, the EPA uses its Quality System to manage the quality of its environmental data generation, collection, and use. The Quality System covers activities such as determining hazardous or toxic wastes in the environment and establishing health risk levels, supporting enforcement monitoring efforts, and mapping human health risk data. Poor data quality negatively impacts the EPA's effectiveness in overseeing programs that directly impact public health and could also subject the EPA to significant financial and legal risks. OMS leaders agree that the Quality System needs improvement and are taking steps to strengthen its internal control system. However, until the OMS fully implements internal controls, it cannot know the status or health of the agencywide Quality System. Control Environment Per GAO's federal internal control standards, the control environment is the foundation for an internal control system and includes, among other things, exercising oversight responsibilities and developing a strategic plan and objectives as well as accompanying structure and staffing plans. In reviewing the control environment for the Quality System, we found that the OMS has not developed a strategic plan, defined objectives, provided oversight via annual reporting, or adequately assessed the workload for the Quality System staff. Interviewees noted that responsibility for the EPA's Quality System varies from the Agency's past structure, and we found that placement varies from quality systems at other federal agencies. Absence of Strategic Plan, Objectives, and Required Reporting The OMS has not developed a strategic plan or defined objectives for the Quality System. A strategic plan would help the EPA's national Quality System personnel and other stakeholders understand the goals, objectives, and priorities for the Quality System, and help the OMS evaluate performance against those objectives. The lack of a strategic plan and defined objectives resulted from varying priorities of different Quality System leaders over time. OMS leaders indicated that they are working on strategic planning and setting goals. As of June 2020, the OEIP had finalized a mission statement and the EQMD had developed draft strategic priorities. OEIP Mission Statement: To provide our partners and customers innovative products and services that improve the access, management, and value of the EPA's information in support of its mission to protect human health and the environment. Draft EQMD Strategic Priorities: • Lead the Agency's quality program. • Engage internal customers as strategic partners. • Provide services and tools quickly and cost effectively to states and tribes. 20-P-0200 9 ------- The OMS has not followed the requirements of EPA Order CIO 2105.0. For example: • Required annual reviews, called Quality Assurance Annual Report and Work Plans, have not occurred since fiscal year 2016, and OMS leaders indicated that these reviews will not occur in fiscal year 2020. • Not all organizations have approved Quality Management Plans. These plans document an organization's quality policy, describe its quality system, and identify environmental programs to which the quality system applies. In October 2019, the EQMD extended the expiration date of the Quality Management Plans and provided a sequence for submitting new plans to the OMS for review and approval. • Required periodic assessments, or QSAs, have not occurred. The Agency's senior management official for Quality is required to perform periodic management assessments of all EPA organizations. EPA organizations are required to perform assessments of the effectiveness of their quality system at least annually. The OMS has not completed management assessments of most EPA organizations in more than six years even though they should be completed on a three-year cycle. The OMS uses these assessments to determine the effectiveness of each organization's quality system and recommend corrective actions. OMS leaders indicated that assessments will not resume until fiscal year 2021. OMS leaders said they ceased requiring annual reports because the reports varied in length, detail, and format, making them difficult to review, analyze, and compare reported information. OMS leaders acknowledged issues with the Quality System and stated they want to develop clear goals and objectives prior to reinstituting required reporting. In December 2019, the EQMD had developed a tactical priority to "eliminate overdue reports." OMS leaders noted that absent required reporting, they do not know the status or the health of program or regional quality systems. Decrease in Quality System Staff Resulting in a Backlog of Work Interviewees expressed concern about the decrease in OMS Quality System staff and the impact to workload (Figure 2). In 2000, there were 17 full-time equivalent staff and, as of November 2019, there were eight, a decrease of nearly 53 percent. Figure 2: Quality System Staff 12 of 26 interviewees expressed concerns about the low number of quality system staff Source: OIG data based on interviews. (OIG image) 20-P-0200 10 ------- OMS leaders said they did not realize that the Quality System had higher staff levels in the past. Quality System staff described a significant backlog of work due to lower staffing numbers. While the OMS noted that the Office of the Chief Financial Officer leads the Agency's workload analysis, the OMS plans to conduct its own staff assessment to help ensure that skills and competencies align with Quality System tasks. Office Responsible for the Quality System Varies from the EPA's Past Structure and from Other Federal Agencies Per GAO's control standards, management should establish the organizational structure necessary to enable it to plan, execute, control, and assess the organization in achieving its objectives. As described in Chapter 1, the EPA placed oversight responsibility for the agencywide Quality System in the ORD until 2000 when the EPA transferred it to the Office of Environmental Information, and then to the OMS in 2018. From 2000 onward, the Quality System reported to the assistant administrator for Environmental Information. As shown on Figure 3, the EPA moved the Quality System lower organizationally to the divisional level in the EQMD by 2018. Figure 3: Changes overtime in organizational placement of the EPA Quality System and number of Quality System staff Administrator Level G Assistant Administrator Level tJ Office Level Divisional Level 2000 2018 Source: OIG graphic based on analysis of Quality System organization and staffing. 20-P-0200 11 ------- While we did not find evidence of clear negative impacts from moving the Quality System to the divisional level, interviewees described the agencywide program as less influential and said that internal processes take longer. Additionally, almost half of the interviewees stated that the OMS is not the appropriate office for the agencywide Quality System (Figure 4). In addition to the Quality System, other OMS functions include information technology, contracts and grants management, and human capital. The OMS also houses the CIO. Interviewees expressed concerns that the more immediate needs related to information technology tended to take precedence over the Quality System. We found that nine out of the 14 agencies we benchmarked housed their quality programs and systems separately from information technology. Four interviewees, including some within the EQMD, recommended connecting the Quality System with other components like scientific integrity and peer review, which are under the ORD. OMS and ORD leaders have discussed whether the Quality System should return to the ORD and have agreed to postpone the decision to allow the ORD's reorganization, initiated in September 2019, more time to mature. The chief information officer said that the OMS and the ORD discussions include assessing structurally where the Quality System would best fit. OMS leaders added that they will continue to work to improve the Quality System while it remains OMS's responsibility. Control Environment Recommendations We recommend that the assistant administrator for Mission Support: 1. Develop and implement a strategic plan and objectives for the agencywide Quality System. 2. Develop and implement a standard operating procedure to conduct annual reviews of program and regional quality systems. 3. Determine the skillsets needed to fulfill responsibilities for developing and coordinating the agencywide Quality System. 4. Work with the Office of the Chief Financial Officer to conduct a workload analysis for the agencywide Quality System. Figure 4: Organizational placement 12 of 26 interviewees expressed concerns about Quality System placement in the OMS Source: OIG data based on interviews. (OIG image) 20-P-0200 12 ------- Risk Assessment Per GAO's federal internal control standards, internal control should provide an assessment of the risks that the agency faces from both internal and external sources. Once risks have been identified, they should be analyzed for their possible effect. After setting program objectives, risk assessment requires agencies to identify vulnerabilities that could impede the efficient and effective achievement of those objectives. Management should then formulate an approach for risk management and put controls in place to mitigate those risks, including during times of change. The OMS has not developed a strategic plan or objectives for the agency wide Quality System, nor has the OMS conducted a risk assessment. OMS leaders said that they did not need to conduct a formal risk analysis to recognize problems with the Quality System. They further stated that they want to address other issues first, such as policy updates and training, before conducting an in-depth risk analysis. Until the OMS develops a strategic plan and objectives, it cannot define risk tolerances or identify risks that require mitigation. However, we agree with the OMS on the immediate need to address risks related to policy updates and training. Risk Assessment Recommendation We recommend the assistant administrator for Mission Support: 5. Conduct and document an internal control risk assessment on the agencywide Quality System based on the Office of Mission Support's strategic plan for the Quality System. Control Activities Per GAO's federal internal control standards, control activities are the actions management takes to achieve objectives and respond to risks, such as implementing policies and procedures. Control activities include reviewing actual performance to planned or expected results, such as through annual reporting and managing human capital through training and other tools. We found that the OMS did not review quality policies, procedures, and guidance within the required five-year approval time frame or reaffirm the validity of some policies. We also found that the OMS did not provide agencywide training. 20-P-0200 13 ------- Outdated and Unclear Policies, Procedures, and Guidance Per the EPA Quality Manual for Environmental Programs, all EPA quality-related requirements and guidance documents are valid for a period of five years from the approval date, at which time the OMS must take action to reaffirm the document's validity, revise it, or delete it. The OMS has not reviewed most of its quality policies, procedures, or guidance within the required five-year approval time frame (Figure 5). For example, the OMS began reviewing EPA Order CIO 2105.0 15 years after its review date. We found that 25 out of 26 Quality System documents on the EPA quality website are overdue for review. We also found the validity of the EPA's Quality Policy, CIO 2106.0, approved on October 20, 2008, unclear because the OMS has neither implemented nor rescinded it. Interviewees expressed concerns about the adequacy of Quality System documents. Outdated or unclear policies lead to confusion among national Quality System personnel and inconsistent program implementation across the Agency. OMS leaders agreed that, over the years, core Quality System aspects like policy updates were not completed as they were not a focus of the leaders at the time. According to one OEIP senior leader, the lack of updated policies has weakened the entire Quality System, which is essential to the EPA's mission. OEIP's director added that, anytime a program is not running well, or is perceived to not run well, it exposes the Agency to vulnerability. As of December 2019, the EQMD had developed a tactical priority to update quality directives and guidance to improve the Quality System. Absence of Agencywide National Training EPA Order CIO 2105.0 requires the OMS to provide training for all levels of management and staff so that they understand quality management responsibilities and requirements at every stage of project implementation. Furthermore, EPA Order CIO 2105.0 requires sufficient resources to assure performance of an adequate level of quality assurance and quality control activities. The OMS has not provided agencywide training in over two years. Figure 5: Overdue reviews 25 of 26 ^policies, procedures/ and guidance are outside of the five- year review and approval date Source: OIG data based on EPA documents. (OIG image) 20-P-0200 14 ------- OMS leaders attributed this to a lack of resources and a training approach that did not have leadership buy-in. Interviewees told us that they would like more training, particularly for new staff as well as for grant recipients (Figure 6). Interviewees added that, without needed training, Quality System staff may inconsistently review project-specific plans that require approval, and interviewees said that old training and outdated documents on the EPA website led to confusion. OMS leaders said they are developing five training modules: Overview, Quality Assurance Manager/Management, Project Manager, Senior Executive, and Quality Assurance Field Activities. Control Activities Recommendations We recommend the assistant administrator for Mission Support: 6. Develop and implement a plan and timeline to review and act on all outdated quality policies, procedures, and guidance documents. 7. Develop and deploy agencywide training modules. 8. Develop and require training for new Quality System personnel. Information and Communication Per GAO's federal internal control standards, for an agency to execute and control its operations, it must have relevant and reliable information. An agency should record and communicate information to management and other stakeholders in a form and within a time frame that enables the agency to carry out its operational responsibilities. We found that the Quality System website had numerous outdated policies, procedures, and guidance documents, including the EPA's Quality Policy, CIO 2106.0, which has not been implemented or rescinded. Furthermore, we found inaccuracies in the EPA's list of national Quality System personnel. As noted previously, reductions in the number of Quality System staff, down by 53 percent over the past 20 years, has resulted in a workload backlog, including website updates and timely reviews of policies, procedures, and guidance. Inaccuracies on the Quality System website lead to confusion on applicable policies and current national contacts, thus diminishing the website's value. In February 2020, the EQMD indicated that it began updating the estimated Figure 6: Training concerns interviewees expressed concerns about training Source: OIG data based on interviews. (OIG image) 20-P-0200 15 ------- 400 pages under the Quality System domain, including updating all organization names and Quality System contacts. Additionally, although the EQMD conducts a monthly informational teleconference for the EPA's quality community, half of the interviewees expressed concerns about the adequacy of OMS's communication and outreach. Interviewees indicated that in-depth discussions on the Quality System cannot occur during the monthly calls because of the large number of varied participants, which may lead some personnel to not participate at all. As of December 2019, the EQMD had a tactical priority to strengthen communications with internal quality partners. Information and Communication Recommendations We recommend the assistant administrator for Mission Support: 9. Update the EPA Quality System website to reflect the current status of all policies, procedures, and guidance, as well as Quality System contacts. 10. Query national Quality System personnel on the value and frequency of Quality System communication methods and revise methods as needed, depending on the input received. Monitoring Per GAO's federal internal control standards, monitoring the internal control system is an essential and dynamic process that continually adapts to meet the risks and changes an entity faces. Internal control monitoring should assess the quality of performance over time and promptly resolve findings of audits and other reviews. Additionally, management should build monitoring activities, such as automated tools, into an entity's operations and evaluate results. We found that the OMS did not address three of the six recommendations from a 2014 EPA- contracted program evaluation conducted by an outside contractor. We also found that the OMS has not yet implemented an agencywide tracking system to monitor Quality System activities and evaluate results. Additionally, the OMS has not finalized core metrics or reporting requirements for the new tracking system. Prior Recommendations Not Addressed An outside contractor conducted a program evaluation of the EPA's Quality System and issued a report in November 2014, titled, Evaluating the Effectiveness of the EPA Quality System. The report made six recommendations; however, the OMS did not complete actions on the following three recommendations: • "Work with program partners to define [OMS's] role and clarify Quality System guidance. 20-P-0200 16 ------- • "Develop a comprehensive staffing plan to address vacancies and skills gaps in the Quality System. • "Rebrand the EPA's Quality System to increase support from project personnel and senior managers." OMS leaders said that while they have started some work on the first recommendation to define OMS's role and clarify guidance, they have not produced any updated Quality System guidance in several years. As noted, OMS leaders said they plan to work on a staff skillset assessment and increase support for the Quality System. Per GAO's federal internal control standards, corrective actions from audits and other reviews complement control activities to achieve objectives. Absence of Agencywide Tracking System Since 2015, the OMS has invested $1.3 million in developing an agencywide quality tracking system. The OMS said that the tracking system should simplify and modernize the quality assurance annual reporting process by tracking EPA organizations' quality information and providing a holistic picture of the current state of quality in the Agency. In addition to tracking organizations' quality information, the OMS said that the system should facilitate knowledge exchange through sharing templates, standard operating procedures, and best practices. In order to realize the benefits of efficiency and cost reduction, the OMS said that its new tracking system would have replaced approximately 80 systems used across the Agency. However, some EPA organizations plan to maintain their existing individual tracking systems until they have assurance that the new system will meet organizational needs. Two programs indicated that they would each invest about $100,000 annually to maintain their systems. Although the OMS has developed a measure that tracks the review and approval time of state and tribal Quality Assurance Project Plans, it has not finalized core program metrics or reporting requirements for the new tracking system. OMS leaders said they wanted new metrics developed that had senior leadership buy-in and wanted to ensure that the metrics could measure program requirements, such as Quality Management Plans and QSAs. Developing new metrics and data migration from existing systems delayed the rollout and implementation of the new agencywide tracking system. Without an agencywide tracking system, the OMS could not readily provide us with a list of required and completed QSAs. Without QSAs, the OMS cannot monitor deficiencies or determine needed corrective actions. Absent these monitoring activities, the OMS cannot assess the quality of performance over time and promptly resolve issues. Furthermore, if multiple tracking systems continue to exist, the OMS will not realize the benefits of efficiency and cost reduction. 20-P-0200 17 ------- In March 2020, the OMS formed a team to review the planned tracking system and other quality assurance system capabilities and make recommendations to the CIO. In May 2020, after we issued our draft report, the OMS team met with the CIO to discuss the path forward on developing a reporting tool for tracking annual reporting. The CIO decided to disinvest in the agencywide tracking system and to leave in place the approximately 80 tracking tools used across the Agency. The CIO agreed with the OMS team that requirements for annual reporting need to occur and be reflected in quality documents prior to developing any enterprisewide system. Monitoring Recommendations We recommend the assistant administrator for Mission Support: 11. Address the three unimplemented recommendations from the 2014 program evaluation by the outside contractor to work with program partners to define the role of the Office of Mission Support and clarify Quality System guidance, develop a comprehensive staffing plan to address vacancies and skill gaps in the Quality System, and rebrand the EPA's Quality System to increase support from project personnel and senior managers. 12. Develop and implement a means to track Quality System Assessments. 13. Complete Quality System Assessments for organizations that are outside of the required three-year assessment time frame. 14. Complete development and rollout of an agencywide tracking system that includes finalized core metrics. 15. Coordinate with program and regional offices to eliminate redundant or duplicative tracking systems. Conclusions The OMS had not implemented needed internal control components intended to provide reasonable assurance that the OMS can carry out its responsibility in developing and coordinating the mandatory agencywide Quality System. OMS's leaders have recognized that they could improve controls and have initiated actions to strengthen the agency's Quality System. These improvements, along with our recommendations, will help the OMS achieve the primary goal of the Quality System: To ensure that the EPA's environmental data are of sufficient quantity and quality to support the data's intended use. 20-P-0200 18 ------- Agency Response and OIG Assessment The OMS concurred with Recommendations 1-8 and 10-13 and provided estimated corrective action dates, and those recommendations are resolved with corrective actions pending. The OMS completed Recommendation 9 and we corresponded with the OMS to verify website updates and corrections. Recommendations 14 and 15 are unresolved with resolution efforts in progress. In its response to Recommendation 14, the OMS said that "the CIO determined that a single enterprise approach does not meet the unique tracking and programmatic needs of the Regions, National Programs and the Office of Research and Development." We note that the CIO made this determination despite the OMS's five-year and $1.3 million investment in developing the agencywide tracking system. We continue to believe that the OMS needs a monitoring approach that meets the goals of the planned agencywide tracking system, which includes simplifying annual reporting, providing a holistic picture on the state of quality in the Agency, and exchanging knowledge through shared procedures and practices. Absent an enterprisewide monitoring or tracking system, it is unclear how the OMS will achieve these goals. Moreover, the CIO's decision to leave the tracking systems in place in program and regional offices means the Agency continues to have redundant and duplicative systems, a risk which Recommendation 15 is designed to address. The Agency's full response to the draft report is in Appendix A. 20-P-0200 19 ------- Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS Rec. No. Page No. Subject Status1 Action Official 1 12 Develop and implement a strategic plan and objectives for the agencywide Quality System. 2 12 Develop and implement a standard operating procedure to conduct annual reviews of program and regional quality systems. 3 12 Determine the skillsets needed to fulfill responsibilities for developing and coordinating the agencywide Quality System. 4 12 Work with the Office of the Chief Financial Officer to conduct a workload analysis for the agencywide Quality System. 5 13 Conduct and document an internal control risk assessment on the agencywide Quality System based on the Office of Mission Support's strategic plan for the Quality System. 6 15 Develop and implement a plan and timeline to review and act on all outdated quality policies, procedures, and guidance documents. 7 15 Develop and deploy agencywide training modules. 8 15 Develop and require training for new Quality System personnel. 9 16 Update the EPA Quality System website to reflect the current status of all policies, procedures, and guidance, as well as Quality System contacts. 10 16 Query national Quality System personnel on the value and frequency of Quality System communication methods and revise methods as needed, depending on the input received. 11 18 Address the three unimplemented recommendations from the 2014 program evaluation by the outside contractor to work with program partners to define the role of the Office of Mission Support and clarify Quality System guidance, develop a comprehensive staffing plan to address vacancies and skill gaps in the Quality System, and rebrand the EPA's Quality System to increase support from project personnel and senior managers. 12 18 Develop and implement a means to track Quality System Assessments. 13 18 Complete Quality System Assessments for organizations that are outside of the required three-year assessment time frame. 14 18 Complete development and rollout of an agencywide tracking system that includes finalized core metrics. 15 18 Coordinate with program and regional offices to eliminate redundant or duplicative tracking systems. Planned Completion Date Potential Monetary Benefits (in $000s) Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support Assistant Administrator for Mission Support 12/31/21 6/30/22 12/31/21 12/31/21 12/31/21 6/30/22 6/30/22 1/15/21 5/14/20 12/31/20 12/31/21 12/31/21 6/30/25 1C = Corrective action completed. R = Recommendation resolved with corrective action pending. U = Recommendation unresolved with resolution efforts in progress. 20-P-0200 20 ------- Appendix A Agency Response to Draft Report UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF MISSION SUPPORT MEMORANDUM SUBJECT: Response to Office of Inspector General Draft Report No. OA&E-FY 19-0329 "EPA Needs to Address Internal Control Deficiencies in the Agencywide Quality Systemdated April 13,2020 FROM: Vaughn Noga, Chief Information Officer VAUGHN NOGA 13^2020.05.14 07:29:19 Thank you for the opportunity to respond to the subject audit report. The following summarizes the agency's overall position. For those report recommendations with which the agency agrees, we have provided high-level intended corrective actions with completion dates. For the report recommendation that the agency does not agree, we have explained our position. AGENCY'S OVERALL POSITION The Office of Mission Support (OMS) supports thirteen (13) of the report recommendations and disagrees with two. Successful resolution for several of the corrective actions is dependent upon the resolution of other corrective actions. OMS' Enterprise Quality Management Division (OMS/EQMD) depends on the cooperation of the entire agency's Quality Assurance Community to successfully resolve the proposed corrective actions. The agency's Quality System is instituted all across the agency with OMS/EQMD providing oversight, guidance and training. It is important to recognize that different agency programs and regions approach quality differently. Therefore, we want to avoid instituting a one-size-fits-all approach to Quality at EPA because it will not be successful. AGENCY'S RESPONSE TO REPORT RECOMMENDATIONS The chart below provides corrective actions and target completion dates for each of the report recommendations that the agency agrees with. Digitally signed by VAUGHN TO: Patrick Gilbride, Director, Environmental Research Programs Office of Inspector General 20-P-0200 21 ------- Agreements No. Recommendation High-level Corrective Action(s) Target Completion Date 1 Develop and implement a strategic plan and objectives for the agencywide Quality System. Develop and implement a strategic plan and objectives for the agency- wide Quality System. December 31, 2021 2 Develop and implement a standard operating procedure to conduct annual reviews of program and regional quality systems. Develop and implement a standard operating procedure to conduct annual reviews of program and regional quality systems. June 30, 2022 3 Determine the skill sets needed to fulfill responsibilities for developing and coordinating the agencywide Quality System. Determine the skill sets needed to fulfill responsibilities for developing and coordinating the agencywide Quality System. December 31, 2021 4 Work with the Office of the Chief Financial Officer to conduct a workload analysis for the agencywide Quality System. Work with the Office of the Chief Financial Officer to conduct a workload analysis for the agencywide Quality System. December 31, 2021 5 Conduct and document an internal control risk assessment on the agencywide Quality System based on the Office of Mission Support's strategic plan for the Quality System. Conduct and document an internal control risk assessment on the agencywide Quality System based on the Office of Mission Support's strategic plan for the Quality System. December 31, 2021 6 Develop and implement a plan and timeline to review and act on all outdated quality policies, procedures, and guidance documents. Develop and implement a plan and timeline to review and act on all outdated quality policies, procedures, and guidance documents. June 30, 2022 7 Develop and deploy agencywide training modules. Develop and deploy agencywide training modules. June 30, 2022 8 Develop and require training for new Quality System personnel. OMS is finalizing a Quality Program course and will deploy this in FedTalent before the end of the calendar year. January 15, 2021 9 Update the EPA Quality System website to reflect the current status of all policies, procedures, and guidance as well as Quality System contacts. Update the EPA Quality System website to reflect the current status of all policies, procedures, and guidance as well as Quality System contacts each quarter. Complete 10 Query national Quality System personnel on the value and frequency of Quality System communication methods and revise Query national Quality System personnel on the value and frequency of Quality System communication methods and revise December 31, 2020 20-P-0200 22 ------- No. Recommendation High-level Corrective Action(s) Target Completion Date methods as needed, depending on the input received. methods as needed, depending on the input received. 11 Address the three unimplemented recommendations from the 2014 program evaluation by Industrial Economics to work with program partners to define the role of the Office of Mission Support and clarify Quality System guidance, develop a comprehensive staffing plan to address vacancies and skill gaps in the Quality System, and rebrand the EPA's Quality System to increase support from project personnel and senior managers. Address the three unimplemented recommendations from the 2014 program evaluation by Industrial Economics to work with program partners to define the role of the Office of Mission Support and clarify Quality System guidance, develop a comprehensive staffing plan to address vacancies and skill gaps in the Quality System, and rebrand the EPA's Quality System to increase support from project personnel and senior managers. December 31, 2021 12 Develop and implement a means to track Quality System Assessments. Develop and implement a means to track Quality System Assessments. December 31, 2021 13 Complete Quality System Assessments for organizations that are outside of the required three- year assessment time frame. Complete Quality System Assessments for organizations that are outside of the required three-year assessment time frame on a rolling basis driven by the submission of new Quality Management Plans (QMPs) from each RPIO. June 30, 2025 Disagreement The chart below notes the report recommendations that the agency does not agreement along with detailed explanations. No. Recommendation Agency Response Proposed Alternative 14 Complete development and rollout of the agency wide tracking system that includes finalized core metrics. In response to an April 17, 2020, EQMD and Quality Community report to the CIO, he decided that QA annual reporting requirements need to be determined and approved by management in an updated Agency Quality Policy and Procedure. Also, the CIO determined that a single enterprise approach does not meet the unique tracking and programmatic needs of the N/A 20-P-0200 23 ------- No. Recommendation Agency Response Proposed Alternative Regions, National Programs and the Office of Research and Development. 15 Coordinate with program and regional offices to eliminate redundant or duplicative tracking systems. EPA already has a robust process to address redundant and duplicative tracking systems at the enterprise level. The CIO's October 1,2019 policy, "Improving the Management of Small and Non-Investments", establishes Mission Investments and uses financial IT data and data from EPA's READ system to streamline systems. EPA will not create a redundant policy on reducing and eliminating redundant applications just for tracking systems. The agency will continue managing this process through the enterprise approach. N/A CONTACT INFORMATION If you have any questions regarding this response, please contact Mitchell Hauser, OMS's audit follow-up coordinator on (202) 564-7636. cc: Erin Collard Dan Coogan Jan Jablonski Monisha Harris Marilyn Armstrong Mitchell Hauser Jeffrey Wells Katherine Chalfant Erin Barnes-Weaver Alicia Buchanan Jenny Drzewiecki Fred Light Annette Morant 20-P-0200 24 ------- Appendix B Distribution The Administrator Assistant Deputy Administrator Associate Deputy Administrator Chief of Staff Deputy Chief of Staff/Operations Agency Follow-Up Official (the CFO) Agency Follow-Up Coordinator General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Public Affairs Assistant Administrator for Mission Support Principal Deputy Assistant Administrator for Mission Support Associate Deputy Assistant Administrator for Mission Support Deputy Assistant Administrator for Environmental Information and Chief Information Officer Director, Office of Continuous Improvement, Office of the Administrator Director, Office of Enterprise Information Programs, Office of Mission Support Audit Follow-Up Coordinator, Office of the Administrator Audit Follow-Up Coordinator, Office of Mission Support Audit Follow-Up Coordinator, Office of Enterprise Information Program, Office of Mission Support 20-P-0200 25 ------- |