United States
                    Environmental Protection
                           Office of Policy,
                           Economics and Innovation
     June 2003
Improving  EPA's Performance
with  Program  Evaluation
An Evaluation of OAR's Tribal Air  Program
                                                                                Series No. 2
By continuously evaluating its programs, EPA is able to capitalize on lessons learned and incorporate
that experience into other programs. This enables the Agency to streamline and modernize its operations
while  promoting continuous improvement and supporting  innovation.  This series of short sheets on
program evaluation is  intended to  share both the results  and  benefits of evaluations conducted
across the Agency, and share lessons learned about evaluation methodologies in this evolving discipline.
For more information contact EPA's Evaluation Support Division  at
           At a Glance
Evaluation Purpose
To assess the use of resources to build Tribal
program capacity, address air quality problems
on Tribal lands, and provide tools needed to
achieve these goals.
Evaluation Type
Outcome Evaluation
June 2001-June 2002
Office of Air and Radiation
Darrel Harmon, OAR Tribal Program
(202) 564-7416
Advisory  Support
Brian Swett, OPEI (202) 566-2202;
Kane McMickle, OCFO (202) 564-2901
Web Links
                  Background:  Why was an evaluation

                  The 1990 Clean Air Act (CAA) Amendments authorized Indian
                  Tribes to implement CAA programs on Tribal reservations.  In
                  support of this charge/mandate, EPA's Office of Air and Radiation
                  (OAR) further developed its Tribal Air Program to provide Tribes
                  with technical and financial  assistance to address CAA
                  requirements in their jurisdictions.  Since the program's
                  implementation, OAR has provided grants to more than 100 Tribes
                  to support the development of Tribal air programs. Amidst limited
                  funds and staff  resources, Tribes have made significant progress
                  in developing capacity to implement CAA programs through small-
                  and large-scale Tribal programs. Small air programs address issues
                  related to emissions inventories or initiating monitoring programs,
                  while larger programs address problems associated with hazardous
                  air pollutants, indoor air quality, acid rain, mobile sources, radiation,
                  and other issues of local concern. It is at this juncture that OAR
                  determined to evaluate  how effectively the program is using its
                  resources to:  (1) build Tribal program capacity; (2)  address
                  significant air quality problems on Tribal lands; and (3) provide
                  the tools necessary to achieve items one and two.

                                                          NATIONAL  CENTER FOR
                                                          ENVIRONMENTAL  INNOVATION

Basic Evaluation Approach:  How
did they do  it?
The evaluation was conducted by a team comprising
staff  from the Office of Policy, Economics  and
Innovation (OPEI), the Office of the Chief Financial
Officer (OCFO), and contractors led by Darrel
Harmon, Senior Indian Program Manager for the Office
of Air and Radiation  Tribal Program. The evaluation
was completed using  the six steps outlined below.
Step I:    Developing an activities flow chart to
          identify Tribal Air Program stakeholders,
          and their  responsibilities  and activities.
Step II:   Developing a list of 56 interviewees (25
          air professionals from Tribes, 25 EPA
          Regional and Headquarters staff, and six
          representatives from non-governmental
          organizations that work with Tribes).
Step III:  Preparing a  one-page  interview guide
          addressing the key issues presented in the
          activities flow chart.
Step IV:  Conducting interviews, primarily by phone,
          individually or in small group settings with
          project participants.
Step V:  Collecting and analyzing data on resource
          expenditures  from the  Regions and  EPA
Step VI:  Analyzing data  and information from the
          interviews to prepare report findings  and

Evaluation Results:  What was
In 2002, OAR allocated approximately $9.9 million in
State and Tribal Assistance Grants to fund Tribal Air
Program  activities.  The evaluation  helped OAR
determine that "overall," the Tribal Air Program has
helped Tribes  make significant progress toward
developing capacity to carry out the CAA on Tribal
lands  through the development of small- and large-
scale Tribal air  programs.  By providing funding to
organizations like the Institute for Tribal Environmental
Professionals and Tribal  Air Monitoring Support
Centers, EPA has  provided useful technical assistance.
Areas needing improvement still exist, however.  For
example, funds for grants are distributed from OAR to
the Regions, which then distribute the grants to eligible
Tribes. The evaluation found that the criteria used to
distribute the grants to  Tribes differs considerably in
each Region, suggesting the need to develop a national
approach to distributing funds that can be tailored to
meet the specific needs of a region.  Additionally,
improved tracking of information on how Tribes use
their resources is needed to determine where resources
are being used effectively  or ineffectively.  The
evaluation indicates that EPA should work closely with
Tribes to develop performance goals and measures that
will help EPA systematically review the efficacy of
Tribal air programs. Furthermore, Tribes indicate they
require more comprehensive training in areas of
technical assistance.  They recommend that existing
guidance materials and tools that are developed by
Tribal organizations, EPA Headquarters, Regional
offices, or  Tribal organizations be  centralized in a
database  or  Web  site  to  facilitate  improved
communication and information exchange.  Tribes also
requested that they be involved in the preliminary
planning stages and the decision-making  process in
developing rules or guidance documents  that affect
      Approach  for this Evaluation
Step  I
Develop Activities Flow Chart
Step  II
Develop List of Interviewees
Step  III
Develop an Interview Guide
Step  IV
Conduct Interviews
Step  V
Collect and Analyze Data on Resource Expenditure
Step  VI
Develop Findings and Recommendations

Evaluation Outcomes:  What
happened as a result?

The evaluation, which brought visibility to OAR's Tribal
Air Program both internally  (through management
review of  the report) and externally, found that the
Tribal Air Program helped Tribes make significant
progress toward developing capacity to carry out the
CAA  on Tribal lands, given limited resources.  The
evaluation also identified inconsistencies  in  the
allocation of Tribal grant program funds across the
Regions, and the need to develop standard criteria for
the disbursement of funds.  Further, the evaluation
highlighted the need for better data collection
requirements to measure performance and to  fully
evaluate the efficiency and effectiveness of the grant
programs and how Tribes use grant funding for their
air programs. The evaluation identified approximately
27 recommendations  in  the areas  of training,
communication, grants management,  performance
measures  and management, regulatory  and guidance
development, technical assistance, and program
implementation. To date, half of the recommendations
included in the report have been implemented and
OAR  staff are determining which of the  remaining
recommendations should  be implemented. OAR staff
continue to monitor the program to identify areas of
improvement and update the report as new information
becomes available.