Office of Inspector General
Final Memorandum Report
Clean Air
Design Evaluation
Report No. 2002-M-000013
April 23, 2002

Inspector General Resource
Centers Conducting the Review:
Mid-Atlantic Audit/Evaluation Resource Center
Philadelphia, PA
Southern Audit/Evaluation Resource Center
Research Triangle Park Regional Office
Research Triangle Park, NC
Program Offices Involved:	Office of Air and Radiation
Office of Research and Development
Office of General Counsel
Contributors:	Rick Beusse
Jim Hatfield
Tiffine Johnson
Geoff Pierce
Stephen Nelson
Alice Chow (Region 3 Air Protection Division)
Joe Wood (Office of Air Quality Planning
and Standards)

April 23, 2002
Clean Air Design Evaluation Results
Report No. 2002-M-000013
Elizabeth Craig
Deputy Assistant Administrator for Management
Office of Air and Radiation
This memorandum transmits the results of our pilot evaluation of the Environmental Protection
Agency's (EPA's) Clean Air Program and formally closes the pilot phase of our work. We
appreciate the cooperation your staff extended to us during this project and look forward to
working with the Office of Air and Radiation (OAR) in the future as we evaluate different
aspects of the Clean Air Program.
As discussed with you and your staff on March 15, 2001, and as outlined in the Assistant Inspector
General for Program Evaluation's April 19, 2001 memorandum, the purpose of this evaluation was to:
Identify and document the design of the Clean Air Program to achieve its Government
Performance and Results Act (GPRA) goals;
Identify any opportunities for improving the design of the program; and
Recommend specific evaluations and audits to be conducted over a period of time to
evaluate EPA's success in meeting Clean Air goals.
This memorandum includes our observations regarding opportunities to improve the design of
the program, particularly with regard to EPA's GPRA structure. We plan to provide you with
logic models depicting pertinent program designs for each of the 12 Clean Air subobjectives.
Attachment A provides a complete list of these models, along with the web address where each of
these models may be obtained. Our identification of key programs and specific evaluations and audits
that we may perform, which we discussed with you previously, are not included in this report. We plan

to develop these potential evaluations with further input from you and your staff, as well as external
In 1993, Congress enacted GPRA to shift Federal planning, management, and decision-making away
from a traditional focus on resources and activities to a focus on results and outcomes. To make the
government more outcome-oriented, the Act required agencies to:
Specify the outcomes they are trying to achieve;
Identify specific goals and measures; and
Evaluate performance against those goals and measures.
In response to GPRA, EPA established 10 major goals, with "Clean Air" as Goal 1. The
Clean Air goal includes 3 objectives, which are further divided into 12 subobjectives. For
Fiscal Year 2002, EPA established 15 annual performance goals (APGs) for these 12
subobjectives. Nine of the 15 APGs pertain to programs implemented by OAR, while the
remaining 6 relate to research activities conducted by EPA's Office of Research and
Development in support of the Clean Air program. In addition to annual goals, EPA has
established a long-term strategic goal for Clean Air, as well as for each of the objectives and
subobjectives. Attachment B provides the Fiscal Year 2002 GPRA structure for Clean Air and
the associated goals for each clean air objective and subobjective.
Logic Models As a Program Evaluation Tool
In order to document the design of EPA's Clean Air program, and identify potential
opportunities for improving the design of the program, we constructed logic models depicting
the program design for each of the Agency's 12 Clean Air subobjectives. Logic models are an
evaluation tool to help programs establish clear linkages between inputs (resources), activities, outputs,
and outcomes. They help managers explain the elements of a program and describe the logic of how
the program works. Such models also articulate the inherent underlying assumptions,
expectations, and intentions of a program. These models often provide implicit program theory
in writing for the first time.1 As such, logic models can be useful to program managers,
stakeholders, and decisionmakers for a number of reasons, including2:
Providing a common understanding of the program and the expectations for resources,
customers reached, and results;
"Evaluation: A Systematic Approach, "Rossi, Peter H.; Freeman, Howard E.; and Lipsey, Mark W.;
Sixth Edition (1999), 110- 117.
"Logic Models: A Tool for Telling Your Program 's Performance Story, "McLaughlin, J. A.; and
Evaluation and Program Planning 22 (1999) 65-72, Jordan, G.B.

Helping in the design or improvement of programs by identifying projects that are
critical to goal attainment, as well as identifying inconsistent linkages among key
program elements;
Improving data collection, analysis, and usefulness by pointing to key performance
measurements; and
•	Communicating the placement of the program in the organizational or long-term
problem hierarchy.
Senior OAR officials indicated their intent to employ the logic models as a tool for managing
their programs.
Scope and Methodology
We conducted this review from March 15, 2001, through December 19, 2001, at OAR offices
in Washington, DC; Research Triangle Park, NC; and Ann Arbor, MI. We also conducted
work at the Region 3 Air Protection Division in Philadelphia, PA; EPA's Office of General
Counsel in Washington, DC; and EPA's Office of Research and Development facilities in
Washington, DC, and Research Triangle Park, NC.
We conducted an extensive literature search of both EPA and non-EPA generated documents to
identify the objectives, strategies, legislative requirements, programs and activities, and overall
design of the Clean Air Program3. These documents included, but were not limited to:
•	Clean Air Act and implementing regulations;
Web-based Agency program documents;
•	EPA Annual Plans and Congressional Justifications;
•	Memorandums of Agreement between OAR and EPA Regions;
•	Section 105 Grant Workplan commitments;
EPA budget documents for Fiscal Years 2001 and 2002;
•	General Accounting Office and EPA Office of Inspector General reports;
Agency and external group reports on program performance; and
3 OAR also conducts air-related activities under EPA Goal 4 (Preventing Pollution and Reducing Risk
in Communities, Homes, Workplaces, and Ecosystem) and Goal 6 (Reduction of Global and Cross-
Border Environmental Risks). Our review did not include those activities.

National Academy of Sciences/National Research Council reports.
We also discussed the design of the air program with agency officials familiar with the design
of the air program.
We used the information gathered to construct logic models depicting the intended design of the
program for each of the 12 GPRA subobjectives within the Clean Air goal. Logic models depict the
elements of program design and show the critical linkages between an agency's resources and
activities and eventual long-term outcomes. We provided these logic models to over 30 EPA
officials for review, and made revisions when appropriate. Attachment C of this report provides
an illustration of a generic logic model for an environmental program. With the information we gathered
and the logic models we developed, we made observations about the design of the program.
Our review did not include interviews with Clean Air external stakeholders, such as Congressional
Oversight Committee members or staff, industry groups, environmental groups, or state officials. Also,
these external stakeholders have not yet had the opportunity to review and comment on the logic
models we prepared. As we proceed with our program evaluation efforts, we will be contacting
external stakeholders to obtain their input regarding the Clean Air Program's design, activities, and
results. We did not perform work to test the extent to which the intended program design, as depicted
in our models, represents the actual implemented program. Future air program evaluations will address
program implementation and its impacts.
Results of Evaluation
Using the logic models developed, we analyzed the likelihood of whether EPA's "blueprint" of
resources and program activities would enable EPA to reach its projected outcomes. The key
outcomes for the Clean Air goals, depicted in the chart below, represent performance measurement
points that follow the assumed sequence in the progression towards cleaner air and improved health.
The logic models we created highlighted instances where these key measurement points are not
included in the GPRA performance measurement system. In some instances, this is because research
data is lacking to provide the information needed to measure performance in these outcome areas. As
such, we identified instances where we believe EPA will be challenged to demonstrate progress toward
and achievement of its long-term goals.
The majority — eight of nine — of OAR's non-research4 Clean Air APGs for FY 2002 are intermediate
outcome-oriented goals, such as reductions in emissions or improvement in ambient conditions.
Establishing viable intermediate and long-term outcome goals and reliable measures can be a formidable
undertaking, sometimes requiring years of research. Nonetheless, to further improve the results
orientation of EPA's Clean Air GPRA structure, the agency needs to continue its efforts to
4 Of the 15 APGs for Clean Air, 6 pertain to research conducted by the Office of Research and
Development in support of the Clean Air program. These six APGs are output-oriented goals for
developing tools and information such as improved air quality models, health and exposure data for
particulate matter, and methods for measuring human exposure and effects from air toxics.

overcome the challenges of developing more long-term health-related goals and measures. For
example, it may take years for agency activities and programs to have a noticeable impact on
human health and the environment. Although challenges remain, opportunities exist for EPA to
use higher-level intermediate outcome data (e.g., ambient data over emissions data) in the Clean
Air performance measurement and reporting process, and to more prominently incorporate air-
related enforcement activities into the GPRA and strategic planning process for Clean Air.
Summaries of our four observations follow:
1. EPA May Not Be Able to Demonstrate Achievement of
Long-Term Strategic Goal Under Current GPRA Structure
The assumed chain of events or desired outcomes necessary to achieve the long-term strategic
goal for Clean Air is as follows:
Process For Achieving Long-Term Goals
Program activities
will result in
reductions in air
... which will result
in improvements in
ambient air
.. .which should
ultimately result in
the desired long-
term goal or
outcome of
improved human
and ecological
... which will result
in reduced human
and ecological
exposure to
pollutants. . .
Although the desired outcomes or impacts for Clean Air are health-related outcomes, the long-
term goals for two of the three Clean Air objectives (Attain the National Ambient Air Quality
Standards [NAAQS] and Acid Rain) do not directly address human and ecological health5.
Instead, they are stated in terms of intermediate environmental outcomes - either reductions in
pollutant emissions or improvements in ambient air quality. It is important to note that the long-
term goal of human and ecological health is difficult and costly to measure directly, and hard to
explicitly link to agency activities. However, EPA must continue to strive toward developing
true human and ecological health-related goals and measures.
While the long-term goal for the Air Toxics objective is a health-related outcome (i.e.,
reduced health risk), there is concern about the adequacy of the measures. For
example, as stated in EPA's Fiscal Year 2002 Annual Plan and Congressional
Justification, "There is a tremendous amount of uncertainty associated with the potential
health risk, both cancer and non-cancer, and the levels of exposure that result from air
5 The long-term goal for Objective 1 indirectly addresses human health in that the NAAQS are health-
based standards. Through the criteria-setting process, EPA determines at what ambient
concentrations a particular pollutant is harmful to human health and sets the ambient standard

toxic emissions." Consequently, under the current GPRA structure, it will be difficult
for EPA to demonstrate whether it has met its stated long-term outcomes for Clean Air.
2.	Outcome Information Available But Not Utilized
Within the GPRA Framework for the Acid Rain Goal
The long-term GPRA goal for the Acid Rain objective is to reduce deposition of nitrates, total
nitrogen, sulfates, and total sulfur by certain levels from the 1990 baseline. Annual performance
goals and measures reported to Congress track emission reductions; however, they do not
indicate whether these reductions are resulting in less acidic deposition to the environment.
Deposition information is available through two monitoring networks, and is used internally by
the Agency to assess the program's progress in reducing acid deposition. However,
performance goals for reducing acid deposition have not been formally included in the GPRA
structure for the air program, but are needed to demonstrate whether EPA will meet its stated
long-term goal for this objective.
3.	EPA's Annual Performance Report Could Be
More Focused on Environmental Outcomes
As required by GPRA, EPA reports its performance for all goals in an Annual Report to
Congress. The Clean Air section of EPA's Fiscal Year 2000 Annual Report6 focused primarily
on activities and outputs, with some information on environmental outcomes. Our June 2001
report, "EPA's Progress in Using Government Performance and Results Act to Manage for
Results (2001-B-000001)," noted that EPA's performance reporting needed to be more
focused on outcomes, and cited one cause to be a lack of outcome-oriented data. Moreover,
the environmental outcomes associated with any given year's activities may take years to
Recognizing these obstacles, opportunities still exist for the Clean Air Program to include more
outcome-oriented performance data in its report. For example, the discussion on the Acid Rain
objective only included performance data related to emission reductions. However, data was
available to show trends in both wet and dry deposition, as well as changes in lake acidity for
certain areas of the country. Similarly, for Objective 1, the performance report focused
primarily on reductions in emissions and activities designed to reduce emissions. EPA's Air
Quality Trends Report included data on trends in ambient air quality that could be used in the
performance report to better illustrate progress in cleaning the air.
e The Fiscal Year 2000 Annual Performance Report was the most recent document available at the
time of our evaluation.

Role of Enforcement Not Linked to Clean Air Program
Industry compliance with environmental rules and regulations represents an initial outcome in the
chain of events necessary to achieve clean air. As such, compliance monitoring activities are
critical to the success of the Clean Air Program. Although the compliance monitoring and
enforcement function represents a critical program needed to achieve EPA's goal for Clean Air,
the Agency decided to organize and manage this program under a separate EPA GPRA goal.7
Under this arrangement, a key Clean Air related outcome — industry compliance rates — is not
included as a performance goal or measure under Clean Air. Further, strategic planning
documents, which outline how EPA will achieve its Clean Air goal, include minimal references
to enforcement's role in this endeavor.
Because enforcement is managed under a separate EPA GPRA Goal (Goal 9), it was not
represented in our logic models. However, since enforcement activities are critical to achieving
the long-term goal of clean air, enforcement should be fully represented in EPA's clean air
strategic planning and program design.
Ongoing EPA Activities
Prior General Accounting Office and EPA Office of Inspector General reviews noted areas where EPA
could improve its GPRA processes, including strategic planning. Most recently, our June 2001 report,
"EPA's Progress in Using Government Performance and Results Act to Manage for Results," cited that
EPA needs to place greater emphasis on the results and outcomes of its activities, rather than the
actions performed. The report also stated that EPA needs to invest in developing information that is
more outcome oriented. That review focused on three major EPA goals, including Clean Air. The
Agency, and in particular OAR, agreed to take actions to address the observations and
recommendations in that report.
Recognizing that OAR is currently working with EPA's Office of the Chief Financial Officer to improve
its GPRA measures and other related issues, we are not making recommendations in this memorandum.
We are presenting our observations to be considered as part of OAR's efforts in improving the GPRA
design and planning for the Clean Air Program.
If you have any questions regarding this report, please contact me at (215) 814-2326 or Rick Beusse
at (919) 541-5747.
7 While air program activities and outcomes are identified and measured under Goal 1 (Clean Air),
enforcement activities and outcomes are identified and measured under Goal 9 (A Credible Deterrent
to Pollution and Greater Compliance with the Law).


Attachment A
List of Logic Models Depicting
the Design of EPA's Clean Air Program
Under GPRA for Fiscal Year 2002
Title of Logic Model
Subobjective 1.1.1
Attain Ozone NAAQS
Subobjective 1.1.2
Attain Particulate Matter NAAQS
Subobjective 1.1.3
Improve Visibility
Subobjective 1.1.4
Attain Carbon Monoxide NAAQS
Subobjective 1.1.5
Attain Sulfur Dioxide NAAQS
Subobjective 1.1.6
Maintain Nitrogen Dioxide NAAQS
Subobjective 1.1.7
Attain Lead NAAQS
Subobjective 1.1.8
Conduct NAAQS-Related Research
Subobjective 1.2.1
Conduct Air Toxics Research
Subobjective 1.2.2
Characterize Air Toxics
Subobjective 1.2.3
Reduce Cancer and Non-Cancer Risks From Air Toxics
Subobjective 1.3.1
Reduce Acid Rain Pre-Cursors
The above logic models are available on the Office of Inspector General internet site:


Attachment B
Clean Air GPRA Structure - Fiscal Year 2002
Goal 1: Clean Air
The Air In Every American Community Will Be Safe And Healthy To Breathe
Objective 1: Attain NAAQS
Meet national clean air standards by 2005
(CO, S02, N02, and Lead); by 2012 (Ozone);
and by 2018 (PM).
Objective 2: Air Toxics Risk
By 2020, eliminate risks of cancer
and other significant health
problems from air toxic emissions
for at least 95 percent of the
Objective 3: Acid Rain
By 2005, reduce nitrates
and nitrogen deposition to
1990 levels.
By 2010, reduce sulfates
and sulfur deposition by up
to 30 percent from 1990
r 1.1.1. NAAQS for Ozone ^
By 2012, air throughout the
country meets the national
standards for Ozone.
By 2018, air throughout the
country meets the national
standards for PM.
1.1.2. Attain PM NAAQS
By 2018, Visibility will be
improving in 80 percent of our
156 national parks and
wilderness areas from 1999
1.1.3. Visibility
By 2005, air throughout the
country meets the national
standards for Carbon
1.1.4. NAAQS for CO
Through 2018, provide methods,
models, data and assessment
criteria on the health risks
associated with NAAQS.
1.1.8. NAAQS-Related
1.1.7. NAAQS for Lead
By 2005, air throughout the
country meets the national
standards for Lead.
1.1.6. NAAQS for N02
Through 2005, air throughout the
country continues to meet the
national standards for Nitrogen
T 1.1.5. NAAQS for S02
By 2005, air throughout the
country continues to meet the
national standards for Sulfur
v	J
1.2.1. Air Toxics Research
Through 2020, develop and improve: (1) models
and tools to identify the sources of hazardous air
pollutants; (2) cost-effective pollution prevention
and other control options; and (3) scientific
information and tools for quantitative assessment
of Air Toxic risks.
I	^
1.2.2. Characterize Air Toxics
Through 2020, continue to use and improve air
toxics information and tools to support the
evaluation, characterization, and tracking of risk-
based indicators. Develop the tools, strategies,
and programs to reduce Air Toxic exposure risks.
^ *
1.2.3. Reduce Cancer and
Non-Cancer Risks from Air Toxics
By 2020, reduce cancer incidence from stationary
sources by 75 percent (from 1990 levels) and from
mobile source emissions by 65 percent.
Reduce non-cancer risk from all sources and
address disproportionate impacts on populations
and areas that are highly exposed to Air Toxics.
1.3.1. Acid Rain:
By 2005, reduce NOx from electric power
generation sources by 2 million tons from
projected levels. By 2010, reduce S02
from electric power generation sources by
8.5 million tons below 1980 levels.


Attachment C
Generic Logic Model for an Environmental Program
Human and
resources, and
other inputs
needed to
support the
The specific
actions and
tasks needed to
produce the



Users of the

Changes in
goods, and




provided to the

attitudes, skills,

and aspirations

...followed by
changes in
customer actions
changes resulting
from customer
improved ambierjt
conditions, and
reduced exposure
to harmful
Desired program
...such as
improved human
and ecosystem

These are factors outside of the program's control that may influence (help or hinder) the success of the program and the accomplishment of i