SERfV
United States
Environmental Protection
Agency
Office of Policy,
Economics and Innovation
(1807T)
June 2003
EPA-100-F-03-006
Improving EPA's Performance
with Program Evaluation
An Evaluation of OAR's Tribal Air Program
Series No. 2
By continuously evaluating its programs, EPA is able to capitalize on lessons learned and incorporate
that experience into other programs. This enables the Agency to streamline and modernize its operations
while promoting continuous improvement and supporting innovation. This series of short sheets on
program evaluation is intended to share both the results and benefits of evaluations conducted
across the Agency, and share lessons learned about evaluation methodologies in this evolving discipline.
For more information contact EPA's Evaluation Support Division at www.epa.gov/evaluate.
At a Glance
Evaluation Purpose
To assess the use of resources to build Tribal
program capacity, address air quality problems
on Tribal lands, and provide tools needed to
achieve these goals.
Evaluation Type
Outcome Evaluation
Duration
June 2001-June 2002
Partners
Office of Air and Radiation
Contacts
Darrel Harmon, OAR Tribal Program
(202) 564-7416
Advisory Support
Brian Swett, OPEI (202) 566-2202;
Kane McMickle, OCFO (202) 564-2901
Web Links
http://www.epa.gov/oar/tribal/announce/
2002.html
Background: Why was an evaluation
performed?
The 1990 Clean Air Act (CAA) Amendments authorized Indian
Tribes to implement CAA programs on Tribal reservations. In
support of this charge/mandate, EPA's Office of Air and Radiation
(OAR) further developed its Tribal Air Program to provide Tribes
with technical and financial assistance to address CAA
requirements in their jurisdictions. Since the program's
implementation, OAR has provided grants to more than 100 Tribes
to support the development of Tribal air programs. Amidst limited
funds and staff resources, Tribes have made significant progress
in developing capacity to implement CAA programs through small-
and large-scale Tribal programs. Small air programs address issues
related to emissions inventories or initiating monitoring programs,
while larger programs address problems associated with hazardous
air pollutants, indoor air quality, acid rain, mobile sources, radiation,
and other issues of local concern. It is at this juncture that OAR
determined to evaluate how effectively the program is using its
resources to: (1) build Tribal program capacity; (2) address
significant air quality problems on Tribal lands; and (3) provide
the tools necessary to achieve items one and two.
NCEI
NATIONAL CENTER FOR
ENVIRONMENTAL INNOVATION
-------
Basic Evaluation Approach: How
did they do it?
The evaluation was conducted by a team comprising
staff from the Office of Policy, Economics and
Innovation (OPEI), the Office of the Chief Financial
Officer (OCFO), and contractors led by Darrel
Harmon, Senior Indian Program Manager for the Office
of Air and Radiation Tribal Program. The evaluation
was completed using the six steps outlined below.
Step I: Developing an activities flow chart to
identify Tribal Air Program stakeholders,
and their responsibilities and activities.
Step II: Developing a list of 56 interviewees (25
air professionals from Tribes, 25 EPA
Regional and Headquarters staff, and six
representatives from non-governmental
organizations that work with Tribes).
Step III: Preparing a one-page interview guide
addressing the key issues presented in the
activities flow chart.
Step IV: Conducting interviews, primarily by phone,
individually or in small group settings with
project participants.
Step V: Collecting and analyzing data on resource
expenditures from the Regions and EPA
Headquarters.
Step VI: Analyzing data and information from the
interviews to prepare report findings and
recommendations.
Evaluation Results: What was
learned?
In 2002, OAR allocated approximately $9.9 million in
State and Tribal Assistance Grants to fund Tribal Air
Program activities. The evaluation helped OAR
determine that "overall," the Tribal Air Program has
helped Tribes make significant progress toward
developing capacity to carry out the CAA on Tribal
lands through the development of small- and large-
scale Tribal air programs. By providing funding to
organizations like the Institute for Tribal Environmental
Professionals and Tribal Air Monitoring Support
Centers, EPA has provided useful technical assistance.
Areas needing improvement still exist, however. For
example, funds for grants are distributed from OAR to
the Regions, which then distribute the grants to eligible
Tribes. The evaluation found that the criteria used to
distribute the grants to Tribes differs considerably in
each Region, suggesting the need to develop a national
approach to distributing funds that can be tailored to
meet the specific needs of a region. Additionally,
improved tracking of information on how Tribes use
their resources is needed to determine where resources
are being used effectively or ineffectively. The
evaluation indicates that EPA should work closely with
Tribes to develop performance goals and measures that
will help EPA systematically review the efficacy of
Tribal air programs. Furthermore, Tribes indicate they
require more comprehensive training in areas of
technical assistance. They recommend that existing
guidance materials and tools that are developed by
Tribal organizations, EPA Headquarters, Regional
offices, or Tribal organizations be centralized in a
database or Web site to facilitate improved
communication and information exchange. Tribes also
requested that they be involved in the preliminary
planning stages and the decision-making process in
developing rules or guidance documents that affect
them.
Approach for this Evaluation
Step I
Develop Activities Flow Chart
Step II
Develop List of Interviewees
Step III
Develop an Interview Guide
Step IV
Conduct Interviews
Step V
Collect and Analyze Data on Resource Expenditure
Step VI
Develop Findings and Recommendations
-------
Evaluation Outcomes: What
happened as a result?
The evaluation, which brought visibility to OAR's Tribal
Air Program both internally (through management
review of the report) and externally, found that the
Tribal Air Program helped Tribes make significant
progress toward developing capacity to carry out the
CAA on Tribal lands, given limited resources. The
evaluation also identified inconsistencies in the
allocation of Tribal grant program funds across the
Regions, and the need to develop standard criteria for
the disbursement of funds. Further, the evaluation
highlighted the need for better data collection
requirements to measure performance and to fully
evaluate the efficiency and effectiveness of the grant
programs and how Tribes use grant funding for their
air programs. The evaluation identified approximately
27 recommendations in the areas of training,
communication, grants management, performance
measures and management, regulatory and guidance
development, technical assistance, and program
implementation. To date, half of the recommendations
included in the report have been implemented and
OAR staff are determining which of the remaining
recommendations should be implemented. OAR staff
continue to monitor the program to identify areas of
improvement and update the report as new information
becomes available.
-------
------- |