p«crt^c Office of Inspector General Final Memorandum Report Clean Air Design Evaluation Results Report No. 2002-M-000013 April 23, 2002 ------- Inspector General Resource Centers Conducting the Review: Mid-Atlantic Audit/Evaluation Resource Center Philadelphia, PA Southern Audit/Evaluation Resource Center Research Triangle Park Regional Office Research Triangle Park, NC Program Offices Involved: Office of Air and Radiation Office of Research and Development Office of General Counsel Contributors: Rick Beusse Jim Hatfield Tiffine Johnson Geoff Pierce Stephen Nelson Alice Chow (Region 3 Air Protection Division) Joe Wood (Office of Air Quality Planning and Standards) ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, DC 20460 April 23, 2002 MEMORANDUM OFFICE OF INSPECTOR GENERAL SUBJECT: FROM: Clean Air Design Evaluation Results Report No. 2002-M-000013 TO: Elizabeth Craig Deputy Assistant Administrator for Management Office of Air and Radiation This memorandum transmits the results of our pilot evaluation of the Environmental Protection Agency's (EPA's) Clean Air Program and formally closes the pilot phase of our work. We appreciate the cooperation your staff extended to us during this project and look forward to working with the Office of Air and Radiation (OAR) in the future as we evaluate different aspects of the Clean Air Program. Purpose As discussed with you and your staff on March 15, 2001, and as outlined in the Assistant Inspector General for Program Evaluation's April 19, 2001 memorandum, the purpose of this evaluation was to: Identify and document the design of the Clean Air Program to achieve its Government Performance and Results Act (GPRA) goals; Identify any opportunities for improving the design of the program; and Recommend specific evaluations and audits to be conducted over a period of time to evaluate EPA's success in meeting Clean Air goals. This memorandum includes our observations regarding opportunities to improve the design of the program, particularly with regard to EPA's GPRA structure. We plan to provide you with logic models depicting pertinent program designs for each of the 12 Clean Air subobjectives. Attachment A provides a complete list of these models, along with the web address where each of these models may be obtained. Our identification of key programs and specific evaluations and audits that we may perform, which we discussed with you previously, are not included in this report. We plan ------- to develop these potential evaluations with further input from you and your staff, as well as external stakeholders. Background In 1993, Congress enacted GPRA to shift Federal planning, management, and decision-making away from a traditional focus on resources and activities to a focus on results and outcomes. To make the government more outcome-oriented, the Act required agencies to: Specify the outcomes they are trying to achieve; Identify specific goals and measures; and Evaluate performance against those goals and measures. In response to GPRA, EPA established 10 major goals, with "Clean Air" as Goal 1. The Clean Air goal includes 3 objectives, which are further divided into 12 subobjectives. For Fiscal Year 2002, EPA established 15 annual performance goals (APGs) for these 12 subobjectives. Nine of the 15 APGs pertain to programs implemented by OAR, while the remaining 6 relate to research activities conducted by EPA's Office of Research and Development in support of the Clean Air program. In addition to annual goals, EPA has established a long-term strategic goal for Clean Air, as well as for each of the objectives and subobjectives. Attachment B provides the Fiscal Year 2002 GPRA structure for Clean Air and the associated goals for each clean air objective and subobjective. Logic Models As a Program Evaluation Tool In order to document the design of EPA's Clean Air program, and identify potential opportunities for improving the design of the program, we constructed logic models depicting the program design for each of the Agency's 12 Clean Air subobjectives. Logic models are an evaluation tool to help programs establish clear linkages between inputs (resources), activities, outputs, and outcomes. They help managers explain the elements of a program and describe the logic of how the program works. Such models also articulate the inherent underlying assumptions, expectations, and intentions of a program. These models often provide implicit program theory in writing for the first time.1 As such, logic models can be useful to program managers, stakeholders, and decisionmakers for a number of reasons, including2: Providing a common understanding of the program and the expectations for resources, customers reached, and results; "Evaluation: A Systematic Approach, "Rossi, Peter H.; Freeman, Howard E.; and Lipsey, Mark W.; Sixth Edition (1999), 110- 117. "Logic Models: A Tool for Telling Your Program 's Performance Story, "McLaughlin, J. A.; and Evaluation and Program Planning 22 (1999) 65-72, Jordan, G.B. 2 ------- Helping in the design or improvement of programs by identifying projects that are critical to goal attainment, as well as identifying inconsistent linkages among key program elements; Improving data collection, analysis, and usefulness by pointing to key performance measurements; and • Communicating the placement of the program in the organizational or long-term problem hierarchy. Senior OAR officials indicated their intent to employ the logic models as a tool for managing their programs. Scope and Methodology We conducted this review from March 15, 2001, through December 19, 2001, at OAR offices in Washington, DC; Research Triangle Park, NC; and Ann Arbor, MI. We also conducted work at the Region 3 Air Protection Division in Philadelphia, PA; EPA's Office of General Counsel in Washington, DC; and EPA's Office of Research and Development facilities in Washington, DC, and Research Triangle Park, NC. We conducted an extensive literature search of both EPA and non-EPA generated documents to identify the objectives, strategies, legislative requirements, programs and activities, and overall design of the Clean Air Program3. These documents included, but were not limited to: • Clean Air Act and implementing regulations; Web-based Agency program documents; • EPA Annual Plans and Congressional Justifications; • Memorandums of Agreement between OAR and EPA Regions; • Section 105 Grant Workplan commitments; EPA budget documents for Fiscal Years 2001 and 2002; • General Accounting Office and EPA Office of Inspector General reports; Agency and external group reports on program performance; and 3 OAR also conducts air-related activities under EPA Goal 4 (Preventing Pollution and Reducing Risk in Communities, Homes, Workplaces, and Ecosystem) and Goal 6 (Reduction of Global and Cross- Border Environmental Risks). Our review did not include those activities. 3 ------- National Academy of Sciences/National Research Council reports. We also discussed the design of the air program with agency officials familiar with the design of the air program. We used the information gathered to construct logic models depicting the intended design of the program for each of the 12 GPRA subobjectives within the Clean Air goal. Logic models depict the elements of program design and show the critical linkages between an agency's resources and activities and eventual long-term outcomes. We provided these logic models to over 30 EPA officials for review, and made revisions when appropriate. Attachment C of this report provides an illustration of a generic logic model for an environmental program. With the information we gathered and the logic models we developed, we made observations about the design of the program. Our review did not include interviews with Clean Air external stakeholders, such as Congressional Oversight Committee members or staff, industry groups, environmental groups, or state officials. Also, these external stakeholders have not yet had the opportunity to review and comment on the logic models we prepared. As we proceed with our program evaluation efforts, we will be contacting external stakeholders to obtain their input regarding the Clean Air Program's design, activities, and results. We did not perform work to test the extent to which the intended program design, as depicted in our models, represents the actual implemented program. Future air program evaluations will address program implementation and its impacts. Results of Evaluation Using the logic models developed, we analyzed the likelihood of whether EPA's "blueprint" of resources and program activities would enable EPA to reach its projected outcomes. The key outcomes for the Clean Air goals, depicted in the chart below, represent performance measurement points that follow the assumed sequence in the progression towards cleaner air and improved health. The logic models we created highlighted instances where these key measurement points are not included in the GPRA performance measurement system. In some instances, this is because research data is lacking to provide the information needed to measure performance in these outcome areas. As such, we identified instances where we believe EPA will be challenged to demonstrate progress toward and achievement of its long-term goals. The majority — eight of nine — of OAR's non-research4 Clean Air APGs for FY 2002 are intermediate outcome-oriented goals, such as reductions in emissions or improvement in ambient conditions. Establishing viable intermediate and long-term outcome goals and reliable measures can be a formidable undertaking, sometimes requiring years of research. Nonetheless, to further improve the results orientation of EPA's Clean Air GPRA structure, the agency needs to continue its efforts to 4 Of the 15 APGs for Clean Air, 6 pertain to research conducted by the Office of Research and Development in support of the Clean Air program. These six APGs are output-oriented goals for developing tools and information such as improved air quality models, health and exposure data for particulate matter, and methods for measuring human exposure and effects from air toxics. 4 ------- overcome the challenges of developing more long-term health-related goals and measures. For example, it may take years for agency activities and programs to have a noticeable impact on human health and the environment. Although challenges remain, opportunities exist for EPA to use higher-level intermediate outcome data (e.g., ambient data over emissions data) in the Clean Air performance measurement and reporting process, and to more prominently incorporate air- related enforcement activities into the GPRA and strategic planning process for Clean Air. Summaries of our four observations follow: 1. EPA May Not Be Able to Demonstrate Achievement of Long-Term Strategic Goal Under Current GPRA Structure The assumed chain of events or desired outcomes necessary to achieve the long-term strategic goal for Clean Air is as follows: Process For Achieving Long-Term Goals Program activities will result in reductions in air pollutant emissions... ... which will result in improvements in ambient air conditions... .. .which should ultimately result in the desired long- term goal or outcome of improved human and ecological health. ... which will result in reduced human and ecological exposure to harmful pollutants. . . Although the desired outcomes or impacts for Clean Air are health-related outcomes, the long- term goals for two of the three Clean Air objectives (Attain the National Ambient Air Quality Standards [NAAQS] and Acid Rain) do not directly address human and ecological health5. Instead, they are stated in terms of intermediate environmental outcomes - either reductions in pollutant emissions or improvements in ambient air quality. It is important to note that the long- term goal of human and ecological health is difficult and costly to measure directly, and hard to explicitly link to agency activities. However, EPA must continue to strive toward developing true human and ecological health-related goals and measures. While the long-term goal for the Air Toxics objective is a health-related outcome (i.e., reduced health risk), there is concern about the adequacy of the measures. For example, as stated in EPA's Fiscal Year 2002 Annual Plan and Congressional Justification, "There is a tremendous amount of uncertainty associated with the potential health risk, both cancer and non-cancer, and the levels of exposure that result from air 5 The long-term goal for Objective 1 indirectly addresses human health in that the NAAQS are health- based standards. Through the criteria-setting process, EPA determines at what ambient concentrations a particular pollutant is harmful to human health and sets the ambient standard accordingly. 5 ------- toxic emissions." Consequently, under the current GPRA structure, it will be difficult for EPA to demonstrate whether it has met its stated long-term outcomes for Clean Air. 2. Outcome Information Available But Not Utilized Within the GPRA Framework for the Acid Rain Goal The long-term GPRA goal for the Acid Rain objective is to reduce deposition of nitrates, total nitrogen, sulfates, and total sulfur by certain levels from the 1990 baseline. Annual performance goals and measures reported to Congress track emission reductions; however, they do not indicate whether these reductions are resulting in less acidic deposition to the environment. Deposition information is available through two monitoring networks, and is used internally by the Agency to assess the program's progress in reducing acid deposition. However, performance goals for reducing acid deposition have not been formally included in the GPRA structure for the air program, but are needed to demonstrate whether EPA will meet its stated long-term goal for this objective. 3. EPA's Annual Performance Report Could Be More Focused on Environmental Outcomes As required by GPRA, EPA reports its performance for all goals in an Annual Report to Congress. The Clean Air section of EPA's Fiscal Year 2000 Annual Report6 focused primarily on activities and outputs, with some information on environmental outcomes. Our June 2001 report, "EPA's Progress in Using Government Performance and Results Act to Manage for Results (2001-B-000001)," noted that EPA's performance reporting needed to be more focused on outcomes, and cited one cause to be a lack of outcome-oriented data. Moreover, the environmental outcomes associated with any given year's activities may take years to materialize. Recognizing these obstacles, opportunities still exist for the Clean Air Program to include more outcome-oriented performance data in its report. For example, the discussion on the Acid Rain objective only included performance data related to emission reductions. However, data was available to show trends in both wet and dry deposition, as well as changes in lake acidity for certain areas of the country. Similarly, for Objective 1, the performance report focused primarily on reductions in emissions and activities designed to reduce emissions. EPA's Air Quality Trends Report included data on trends in ambient air quality that could be used in the performance report to better illustrate progress in cleaning the air. e The Fiscal Year 2000 Annual Performance Report was the most recent document available at the time of our evaluation. 6 ------- 4. Role of Enforcement Not Linked to Clean Air Program Industry compliance with environmental rules and regulations represents an initial outcome in the chain of events necessary to achieve clean air. As such, compliance monitoring activities are critical to the success of the Clean Air Program. Although the compliance monitoring and enforcement function represents a critical program needed to achieve EPA's goal for Clean Air, the Agency decided to organize and manage this program under a separate EPA GPRA goal.7 Under this arrangement, a key Clean Air related outcome — industry compliance rates — is not included as a performance goal or measure under Clean Air. Further, strategic planning documents, which outline how EPA will achieve its Clean Air goal, include minimal references to enforcement's role in this endeavor. Because enforcement is managed under a separate EPA GPRA Goal (Goal 9), it was not represented in our logic models. However, since enforcement activities are critical to achieving the long-term goal of clean air, enforcement should be fully represented in EPA's clean air strategic planning and program design. Ongoing EPA Activities Prior General Accounting Office and EPA Office of Inspector General reviews noted areas where EPA could improve its GPRA processes, including strategic planning. Most recently, our June 2001 report, "EPA's Progress in Using Government Performance and Results Act to Manage for Results," cited that EPA needs to place greater emphasis on the results and outcomes of its activities, rather than the actions performed. The report also stated that EPA needs to invest in developing information that is more outcome oriented. That review focused on three major EPA goals, including Clean Air. The Agency, and in particular OAR, agreed to take actions to address the observations and recommendations in that report. Recognizing that OAR is currently working with EPA's Office of the Chief Financial Officer to improve its GPRA measures and other related issues, we are not making recommendations in this memorandum. We are presenting our observations to be considered as part of OAR's efforts in improving the GPRA design and planning for the Clean Air Program. If you have any questions regarding this report, please contact me at (215) 814-2326 or Rick Beusse at (919) 541-5747. 7 While air program activities and outcomes are identified and measured under Goal 1 (Clean Air), enforcement activities and outcomes are identified and measured under Goal 9 (A Credible Deterrent to Pollution and Greater Compliance with the Law). 7 ------- ------- Attachment A List of Logic Models Depicting the Design of EPA's Clean Air Program Under GPRA for Fiscal Year 2002 GPRA Subobjective Title of Logic Model Subobjective 1.1.1 Attain Ozone NAAQS Subobjective 1.1.2 Attain Particulate Matter NAAQS Subobjective 1.1.3 Improve Visibility Subobjective 1.1.4 Attain Carbon Monoxide NAAQS Subobjective 1.1.5 Attain Sulfur Dioxide NAAQS Subobjective 1.1.6 Maintain Nitrogen Dioxide NAAQS Subobjective 1.1.7 Attain Lead NAAQS Subobjective 1.1.8 Conduct NAAQS-Related Research Subobjective 1.2.1 Conduct Air Toxics Research Subobjective 1.2.2 Characterize Air Toxics Subobjective 1.2.3 Reduce Cancer and Non-Cancer Risks From Air Toxics Subobjective 1.3.1 Reduce Acid Rain Pre-Cursors The above logic models are available on the Office of Inspector General internet site: httD://www.eDa.aov/oiaearth/eroom.htm ------- ------- Attachment B Clean Air GPRA Structure - Fiscal Year 2002 Goal 1: Clean Air The Air In Every American Community Will Be Safe And Healthy To Breathe Objective 1: Attain NAAQS Meet national clean air standards by 2005 (CO, S02, N02, and Lead); by 2012 (Ozone); and by 2018 (PM). Objective 2: Air Toxics Risk By 2020, eliminate risks of cancer and other significant health problems from air toxic emissions for at least 95 percent of the population. Objective 3: Acid Rain By 2005, reduce nitrates and nitrogen deposition to 1990 levels. By 2010, reduce sulfates and sulfur deposition by up to 30 percent from 1990 levels. r 1.1.1. NAAQS for Ozone ^ By 2012, air throughout the country meets the national standards for Ozone. v By 2018, air throughout the country meets the national standards for PM. 1.1.2. Attain PM NAAQS By 2018, Visibility will be improving in 80 percent of our 156 national parks and wilderness areas from 1999 levels. 1.1.3. Visibility By 2005, air throughout the country meets the national standards for Carbon Monoxide. 1.1.4. NAAQS for CO Through 2018, provide methods, models, data and assessment criteria on the health risks associated with NAAQS. 1.1.8. NAAQS-Related Research 1.1.7. NAAQS for Lead By 2005, air throughout the country meets the national standards for Lead. 1.1.6. NAAQS for N02 Through 2005, air throughout the country continues to meet the national standards for Nitrogen Dioxide. I T 1.1.5. NAAQS for S02 By 2005, air throughout the country continues to meet the national standards for Sulfur Dioxide. v J 1.2.1. Air Toxics Research Through 2020, develop and improve: (1) models and tools to identify the sources of hazardous air pollutants; (2) cost-effective pollution prevention and other control options; and (3) scientific information and tools for quantitative assessment of Air Toxic risks. I ^ 1.2.2. Characterize Air Toxics Through 2020, continue to use and improve air toxics information and tools to support the evaluation, characterization, and tracking of risk- based indicators. Develop the tools, strategies, and programs to reduce Air Toxic exposure risks. ^ * 1.2.3. Reduce Cancer and Non-Cancer Risks from Air Toxics By 2020, reduce cancer incidence from stationary sources by 75 percent (from 1990 levels) and from mobile source emissions by 65 percent. Reduce non-cancer risk from all sources and address disproportionate impacts on populations and areas that are highly exposed to Air Toxics. I 1.3.1. Acid Rain: By 2005, reduce NOx from electric power generation sources by 2 million tons from projected levels. By 2010, reduce S02 from electric power generation sources by 8.5 million tons below 1980 levels. ¦* ------- ------- Attachment C Generic Logic Model for an Environmental Program Resources Human and financial resources, and other inputs needed to support the program. Activities The specific actions and tasks needed to produce the program's outputs. Outputs Customers Reached Short-Term Outcomes Products, Users of the Changes in goods, and program's customer services outputs. knowledge, provided to the program's attitudes, skills, and aspirations customers. — (KASA)... ...followed by changes in customer actions intermediate Outcomes Environmental changes resulting from customer actions... ...reduced pollutant emissions, improved ambierjt conditions, and reduced exposure to harmful pollutants. Long-Term Outcomes Desired program impacts... ...such as improved human and ecosystem health. Externalities These are factors outside of the program's control that may influence (help or hinder) the success of the program and the accomplishment of i ------- |