United States
Environmental
Protection Agency
Office of Policy
(1807T)
March 2011
EPA-100-R-11-006
Evaluation of the Drinking
Water and Clean Water
Infrastructure Tribal Set-
Aside Grant Programs
Final Report
Promoting Environmental Results
< >
Through Evaluation
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
ACKNOWLEDGEMENTS
Ross & Associates Environmental Consulting, Ltd. (Ross & Associates) and Industrial Economics, Incorporated (lEc)
conducted this evaluation under Contract EP-W-07-028 between lEc and EPA's Office of Policy (OP). The
evaluation team consisted of Anna Williams, Amy Wheeless, and Darcy Peth of Ross & Associates and Angela
Helman and Lindsay Ludwig of lEc. Planetary Engineers, staffed by Crispin Kinney, a former IMS manager who
assisted with the development of the IMS STARS database, also provided assistance in this evaluation.
The Evaluation Advisory Team that supported this evaluation included David Harvey and Kyle Carey of EPA's Office
Ground Water and Drinking Water (OGWDW), Matthew Richardson of EPA's Office of Wastewater Management
(OWM), and Michelle Mandolia of EPA's OP's Evaluation Support Division. An Evaluation Review Group held two
calls to provide feedback on evaluation products: this group included the Evaluation Advisory Group members,
Michael Mason (OWM), Gerald McKenna (Region 2), Brian Smith (Region 4), Stephen Poloncsik (Region 5), Minnie
Adams (Region 8), Erskine Benjamin (Region 9), Linda Reeves (Region 9), Loretta Vanegas (Region 9), and Dennis
Wagner (Region 10).
EPA's Program Evaluation Competition, sponsored by OP, provided support for this evaluation. To access copies of
this or other EPA program evaluations, please visit EPA's Evaluation Support Division's website at
http://www.epa.gov/evaluate.
Evaluation Report-March 2011
-------
EVALUATION OF Til IE AND CLEAN INFRASTRUCTURE TRIBAL SET-ASIDE
Table of Contents
Executive Summary ES-1
Conclusions ES-2
Recommendations ES-3
Introduction 1
Evaluation Purpose and Audience 1
DWIG-TSAand CWISA Program Background 2
History and Funding 2
EPA Infrastructure Grant Program Goals and Measures 3
Projects Eligible for DWIG-TSA and CWISA Funding 5
Role of IMS in EPA Infrastructure Grant Programs 6
Methods 8
Evaluation Approach 8
Information Collection Approach 8
Findings: Program Implementation 12
EPA Headquarters Implementation of Tribal Infrastructure Programs 12
EPA Regional Implementation: DWIG-TSA Program 16
EPA Regional Implementation: CWISA Program 19
Regional Communication with IMS, Tribes, and Other Parties 23
Role of IMS in Program Implementation 25
ARRA Implementation 27
Findings: Program Goals and Measures 30
The Compliance Strategic Measure: Progress and Factors Influencing Progress 30
The Access Strategic Measures: Progress and Factors Influencing Progress 36
Conclusions and Recommendations 43
Conclusions 43
Recommendations 45
Additional Considerations 54
Appendices (provided separately)
Appendix A. DWIG-TSA and CWISA Program Evaluation Logic Model
Appendix B Evaluation Interview Guides
Appendix C. DWIG-TSA Regional Implementation Table
Appendix D. Results of Correlation Analyses
Appendix E. Detailed Data Tables
Appendix F. Information Sources
Evaluation Report-March 2011
-------
EVALUATION OF THE DRINKING AND CLEAN INFRASTRUCTURE TRIBAL SET-ASIDE
ACRONYMS
ACS Annual Commitment System
AI/AN American Indian and Alaska Native
ARRA American Recovery and Reinvestment Act of 2009
CDP Community Deficiency Profile (a component of STARS)
CWA Clean Water Act
CWISA Clean Water Infrastructure Set-Aside
CWS Community Water System
CY Calendar Year
DG Direct Grant
DL Deficiency Level (a STARS designation)
DWIG-TSA Drinking Water Infrastructure Grant Tribal Set-Aside
DWINS Drinking Water Infrastructure Needs Survey
EPA United States Environmental Protection Agency
FDL Final Deficiency Level
FTE Full Time Equivalent
FY Fiscal Year
HUD United States Department of Housing and Urban Development
IA Interagency Agreement
IDL Initial Deficiency Level
IHS United States Department of Health and Human Services, Indian Health Service
NPDWR National Primary Drinking Water Regulation
NPM National Program Manager
O&M Operations and Maintenance
OGWDW EPA Office of Ground Water & Drinking Water (part of the EPA Office of Water)
OP EPA Office of Policy
OW EPA Office of Water
OWM EPA Office of Wastewater Management (part of the EPA Office of Water)
PDS Project Data System (a component of STARS)
PWS Public Water System
PWSID Public Water System Identification Number
SDS Sanitary Deficiency System (a component of STARS)
SDWA Safe Drinking Water Act of 1974, amended in 1986 and 1996
SDWIS Safe Drinking Water Information System
SFC IHS Division of Sanitation Facilities Construction
STARS Sanitation Tracking and Reporting System
USDA United States Department of Agriculture
Evaluation Report-March 2011
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
EXECUTIVE SUMMARY
The Environmental Protection Agency (EPA) Clean Water Indian Set-Aside (CWISA) Grant Program and the Drinking
Water Infrastructure Grant Tribal Set-Aside (DWIG-TSA) Program provide funding for wastewater infrastructure
and drinking water infrastructure to American Indian tribes and Alaska Native (AI/AN) Villages. EPA coordinates
with Indian Health Service (IMS) on these programs.
Through this program evaluation, EPA sought to determine the extent to which the combined efforts of the EPA
DWIG-TSA program and the EPA CWISA program have resulted in increased access to safe drinking water and
sanitation facilities and in increased compliance with the Safe Drinking Water Act (SDWA) in Indian country. EPA
also sought to better understand and explain how these programs are implemented, how implementation
influences program performance toward their goals, as tracked by each program's strategic measures, and
whether these measures are accurate indicators of each program's progress. In addition, because IMS often
oversees and directly implements the infrastructure projects, EPA hoped to gain a deeper understanding of its own
scope of influence and if and how EPA could better demonstrate program results. The evaluation looked at DWIG-
TSA and CWISA program activities and influence since 2003. The evaluation team did not focus on IMS
implementation of the programs, except as it relates to EPA program implementation and accomplishments.
Ross & Associates Environmental Consulting, Ltd. (Ross & Associates) and Industrial Economics, Incorporated (lEc)
conducted this evaluation with support through the EPA Program Evaluation Competition, sponsored by the EPA
Office of Policy (OP).
The evaluation focused on responding to the following questions:
I. How does EPA implement the DWIG-TSA and CWISA programs at headquarters and in the regions?
II. To what extent are the DWIG-TSA and CWISA programs achieving their stated goals?
What factors influence whether the programs achieve their stated goals?
Are the current DWIG-TSA and CWISA program performance measures accurate indicators of EPA's
progress?
V. What implementation improvements or innovations could be made by EPA to make the DWIG-TSA and
CWISA programs more effective in meeting the water and wastewater infrastructure needs of tribes?
To answer these questions, the evaluators collected and reviewed documents, conducted interviews with 39
people, and analyzed data on over 650 DWIG-TSA and CWISA projects that EPA contributed funding to between
2003 and 2009.
EPA tracks progress made by the DWIG-TSA and CWISA programs toward the EPA Strategic Plan Goal 2—
protecting America's waters—according to performance on three Strategic Plan measures.
Compliance Strategic Measure: By 2015, 88 percent of the population in Indian country served by
community water systems will receive drinking water that meets all applicable health-based drinking water
standards. (2005 baseline: 86 percent. Status as of FY 2009: 81 percent.)1
1 In the FY2011-2015 EPA Strategic Plan, this measure is under Goal 2: Protecting America's Waters, Objective 2.1: Protect Human Health
Evaluation Report - March 2011 Page ES-1
-------
EVALUATION OF Til IE AND CLEAN INFRASTRUCTURE TRIBAL SET-ASIDE
Drinking Water Access Strategic Measure: By 2015, in coordination with other federal agencies, provide
access to safe drinking water for 136,100 American Indian and Alaska Native homes. (FY 2009 baseline:
80,900 homes. Universe: 360,000 homes.)2
Sanitation Access Strategic Measure: By 2015, in coordination with other federal agencies, provide access to
basic sanitation for 67,900 American Indian and Alaska Native homes. (FY 2009 baseline: 43,600 homes.
Universe: 360,000 homes.)3
The access strategic measures relate to a US commitment made in 2002 at the United Nations World Summit on
Sustainable Development, held in Johannesburg, South Africa.4 Specifically, the US committed to reducing by
half the proportion of people that did not have access to safe drinking water or basic sanitation.5
Conclusions
The goals and priorities for these programs could be clearer and more focused. The goals and priorities for
both programs lack precision and clarity: EPA staff have different ideas of what the term "goal" means for
these programs and how the drinking water program priorities of compliance and access relate to each other.
EPA has the opportunity to more clearly articulate and focus the goals of both programs, and then to focus
program design and implementation accordingly in a set of "cascading" decisions and actions that would
follow clearer goals and priorities.
Funding allocation for both programs could be improved to be more in line with EPA's priorities. The
method used to allocate funds for each program has an understandable basis and history; however, neither
allocation process is an ideal match for the current program priorities and strategic measures.
For both programs, project selection could be more clearly and consistently tied to EPA's priorities, while
still maintaining regional discretion and flexibility. Project selection under the DWIG-TSA program differs
from region to region, and for both programs the relationship between project selection and EPA's priorities
and scope of influence is not as clear or consistent as it could be. For the CWISA program, using the IMS
Sanitary Deficiency System (SDS) list is not ideal for CWISA's priorities and ability to fund projects.
Despite improvement in recent years, communication within EPA and between EPA and IMS is inconsistent
and not optimal for strategic program management or strategic learning and improvement.
IMS is a vital partner for EPA to implement these programs. IMS has the in-house expertise, field presence,
and relationships with tribes necessary to do much of the work of these grant programs. In addition, the
STARS database is also generally considered to be the best source of information on tribal water infrastructure
information and is thus an invaluable tool for these EPA programs. The STARS database is an inventory of
wastewater and water infrastructure need and ongoing projects in Indian country, as tracked by IMS.
There are opportunities for improved communication with tribes about these programs. EPA could
strengthen its direct communication with the tribes to enhance knowledge of tribal contexts and needs, more
consistently provide tribes with information about funding options, understand program accomplishments,
and identify areas for program improvements.
2 Ibid.
3 In the FY2011-2015 EPA Strategic Plan, this measure is under Goal 2: Protecting America's Waters, Objective 2.1: Protect and Restore
Watersheds and Aquatic Ecosystems.
4 United Nations. Plan of Implementation of the World Summit on Sustainable Development. Section II. September 4, 2002. Available:
http://un-documents.net/iburgpln.htmtfll.
5 Specific commitment language was as follows: "The provision of clean drinking water and adequate sanitation is necessary to protect human
health and the environment. In this respect, we agree to halve, by the year 2015, the proportion of people who are unable to reach or to afford
safe drinking water (as outlined in the Millennium Declaration) and the proportion of people who do not have access to basic sanitation..."
Evaluation Report - March 2011 Page ES-2
-------
EVALUATION OF THE DRINKING AND CLEAN INFRASTRUCTURE TRIBAL SET-ASIDE
EPA has limited ability to make progress on the current strategic measures focused on providing access to
drinking water and clean water to tribal homes. EPA's limitations include:
Lower program funding relative to the funding needed to provide for all identified access needs;
A history of funding a substantial proportion of projects that do not provide access (as defined by
moving from a higher deficiency level (DL) to a lower one) due to project selection processes;
Restrictions on EPA's authority to fund some projects that would provide access, such as funding
individual drinking water wells or providing sanitation infrastructure services to homes that are not
on reservation land;
Reliance on a metric which assumes changes in DLs (from a DL4-5 to a lower DL) that are immediate
and permanent, when in reality, DLs may not change in this manner as a result of funding (e.g.,
because a DL-4 project only partially funds the infrastructure needed to change the associated homes
to a DL-3 or lower) and there is sometimes "backsliding" of DL designations due to compliance
violations, lack of infrastructure maintenance, or other reasons;
Substantial reliance on other federal agencies, particularly IMS, to contribute toward progress on the
access measure; and
Disconnects between reported progress on providing access (reported to be immediate upon project
funding) and actual progress on the ground, which typically occurs years later, after construction is
completed and the new or upgraded infrastructure is operating.
EPA appears to be quite limited in its ability to make progress toward the current compliance strategic
measure, though the extent of the limitations is not clear due to a lack of measurable data. The drivers of
some if not most compliance problems may related to operations and maintenance (O&M) at tribal systems or
the introduction of new drinking water rules, rather than to infrastructure problems that the DWIG-TSA
program can address. In addition, the allocation of DWIG-TSA funding has not been directly tied to
compliance needs or opportunities, nor has project selection clearly and consistently been linked to
compliance support on a national scale.
EPA's meaningful contributions to meeting tribal infrastructure needs are not well reflected by the reported
program performance. Despite the identified challenges surrounding making progress toward the current
strategic measures, both programs have made significant contributions over the years to Indian country
through these programs by providing funding for vital water infrastructure needs. These contributions—and
the public and environmental health benefits they have provided—have not been clearly recognized in a
public or formal way. As one EPA regional coordinator said, "I don't think we hear enough about what the
achievements are—we know what the goals are, but I don't know what the achievements have been."
Recommendations
In response to the evaluation question about possible program improvements (evaluation question V), we offer
eight recommendations and three additional items for consideration. We consider the first four
recommendations to be policy recommendations as they would require approval from senior EPA management
and, in the case of at least recommendations 1-3, approval from the US Office of Management and Budget as well.
Recommendations 5-8 are more program-oriented and could be considered and implemented at the program
level. Additional discussion on each recommendation is provided in the main body of the report (beginning on
page 45).
Evaluation Report - March 2011 Page ES-3
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
1. Clarify the goals and priorities of both programs.
To reduce confusion and increase program focus and management, we recommend that EPA identify clear,
specific, and measurable goals and priorities for both the DWIG-TSA and CWISA programs. A set of cascading
changes to both programs' processes (e.g., allocation of funds, project selection) and performance measures could
then logically follow as described in the following recommendations.
2. Consider changing the access strategic measures by which EPA tracks its performance to measures that
more directly reflect EPA's mandate, authority, and scope of influence; take into consideration data
availability and quality; and reflect changes or clarifications to the programmatic goals made in response to
recommendation 1.
In the Recommendations section of the full report, we offer several options for EPA's consideration regarding the
current access strategic measures, which we do not think serve EPA or reflect progress toward meeting tribal
needs as well as other options would.
3. Consider changing the DWIG-TSA compliance strategic measure to a measure that more directly reflects the
drivers of compliance problems and EPA's scope of influence over these problems.
A revised compliance measure targeted specifically to infrastructure-solvable compliance problems that EPA has
the ability to address would more directly and clearly reflect the influence of the DWIG-TSA program on
compliance. In the Recommendations section of the full report, we offer three options for such a measure.
4. Reassess the national annual budget allocation for both programs to be more clearly tied to EPA's goals and
priorities.
Once EPA has determined whether and how to update each program's goals, priorities, and measures, deciding
whether and how to update the allocation formulas for one or both programs will be easier. However, we
recommend updating the allocation formulas to more directly and clearly support EPA's current strategic measures
even if EPA makes no changes in response to recommendations 1-3.
5. For both programs, update and clarify expectations for project selection to be more clearly in line with the
program priorities, allow for regional flexibility and discretion, and promote increased consistency and
transparency.
The project selection processes for both programs have their bases and merits, but neither process is ideal. For
the DWIG-TSA program, consideration of both compliance needs and provision of access varies from region to
region; regions often choose projects based on other considerations, making it difficult for EPA to make progress
on its strategic measures. For the CWISA program, exclusive use of the SDS list is limiting and does not recognize
EPA's distinct mission, authority to fund, or priorities, including provision of access. Updating the project selection
guidances to focus on a simple set of core program elements while allowing for regional flexibility and discretion
would provide more coherence across the regions, particularly for the DWIG-TSA program, and more direct linkage
to program priorities for both programs.
6. For both programs, routinely collect and analyze data to enhance transparency and strategic coordination
and improve EPA's ability to report on, advocate, and improve both programs through learning and
adaptation.
Evaluation Report - March 2011 Page ES-4
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Improved information availability will enable EPA to better understand its contributions through these programs
and support learning over time to improve program effectiveness. Given there is currently very little information
collected by EPA nationally other than basic funding statistics, even a modest increase in data collection would
provide a markedly improved ability to understand, explain, and advocate for these infrastructure programs at the
regional and national levels. The type of information that could be beneficial would not be complicated or
extensive.
7. Update the national guidelines for both programs.
Following the consideration of the above recommendations and any resulting changes, we recommend that EPA
headquarters work with EPA regions to update the guidelines for both programs. In addition, EPA headquarters
should work with the EPA regions to update any communication materials about these programs.
8. Continue to enhance and improve communication within EPA, between EPA and IMS, and between EPA and
the tribes.
Communication between EPA headquarters and EPA regions, between EPA and IMS, and between EPA and the
tribes is important to effectively implement and measure the impact that these programs have on tribal
communities. Although the evaluators heard from interviewees many positive reflections on communication, we
also heard that communication improvements in several areas would be useful.
Additional Considerations
We offer three additional considerations which reflect our support for efforts that are already underway or, in one
case, previous work done to reflect on many of the same issues that this evaluation has covered:
1. Continue to focus on the critical issues surrounding infrastructure sustainability, including tribal O&M capacity
issues;
2. Continue to communicate about interagency agreement (IA) requirements, expectations, and inter-agency
roles between EPA and IMS; and
3. Reflect on other reports and recommendations prepared by other workgroups that pertain to these programs.
Evaluation Report - March 2011 Page ES-5
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
INTRODUCTION
The Environmental Protection Agency (EPA) Clean Water Indian Set-Aside (CWISA) Grant Program and the Drinking
Water Infrastructure Grant Tribal Set-Aside (DWIG-TSA) Program contributes funding for wastewater infrastructure
and drinking water infrastructure to American Indian tribes and Alaska Native (AI/AN) Villages. EPA coordinates
with Indian Health Service (IMS) on these programs. This evaluation is intended to assess these tribal set-aside
programs, including their implementation, program measures and outcomes, and identify opportunities for
program improvements.
Evaluation Purpose and Audience
Through this program evaluation, EPA sought to determine the extent to which the combined efforts of the DWIG-
TSA and CWISA programs have resulted in the provision of access to safe drinking water and sanitary facilities in
Indian country, and in increased compliance with the Safe Drinking Water Act (SDWA) in Indian country. EPA also
sought to better understand and explain how these programs are implemented, how implementation influences
program performance toward their goals, as tracked by each program's strategic measures, and whether these
measures are accurate indicators of each program's progress. In addition, because IMS often oversees and directly
implements the infrastructure projects, EPA hoped to better understand its own scope of influence and, in so
doing, better understand if and how EPA could better demonstrate program results. The evaluation looked at
DWIG-TSA and CWISA program activities and influence since 2003. The evaluation team did not focus on IMS
implementation of the programs, except as it relates to EPA program implementation and accomplishments.
The primary audiences for this evaluation are the EPA managers and staff who oversee and implement these
programs in the Office of Ground Water and Drinking Water (OGWDW), Office of Wastewater Management
(OWM), and EPA regions. IMS managers and staff may also be interested in the results of this evaluation, as may
AI/AN representatives who are eligible for funding from the CWISA or DWIG-TSA programs.
Evaluation Report - March 2011 Page 1
-------
OF THE AND
DWIG-TSA AND CWISA PROGRAM BACKGROUND
Historically, as compared non-tribal populations, those living in Indian country and in Alaska Native Villages have
had disproportionately lower access to safe drinking water and sanitary facilities, which indicates that there is a
lack of a safe water supply and/or a system for sewage disposal. For example, in 2009, 43,437 or 12.1 percent of
the tribal homes tracked by IMS lacked access to safe drinking water in 2009, compared to the 0.6 percent of non-
tribal homes that lacked such infrastructure (as measured by the US Census Bureau in 2005).6
The CWISA and DWIG-TSA programs are intended to provide funding to infrastructure projects that will help
overcome these AI/AN access challenges. The EPA regions, often through federal interagency agreements (lAs)
with the IMS Division of Sanitation Facilities Construction (SFC)7, administer the programs. Grants are available to
any federally recognized Indian Tribe, as well as to Alaska Native Villages working with the State of Alaska. Funding
recipients can receive funds either through a direct grant from EPA or through IMS, which administers an IA with
EPA to receive and disperse the funds.
History and Funding
The DWIG-TSA Program was established under section 1452(i) of the 1996 amendments to the Safe Drinking Water
Act (SDWA) and is administered under the EPA Office of Ground Water and Drinking Water (OGWDW). Through
the 2009 federal fiscal year (FY2009), this program was funded from a 1.5 percent set-aside from the annual
Drinking Water State Revolving Fund (DWSRF); as of FY2010, this set-aside amount increased to 2.0 percent.
DWIG-TSA funds are allocated to the EPA regions based on a combination of: (1) drinking water needs identified in
the IMS Sanitary Deficiency Survey (SDS) list, an annual inventory of water infrastructure need in Indian country,
and (2) the tribal portion of the EPA Drinking Water Infrastructure Needs Survey (DWINS), a survey of drinking
water infrastructure need undertaken approximately every four years. Annual funding since the program's
inception until FY2010 has ranged from $10.9 million to $27.7 million. Funding fluctuations have been a function
of variances in DWSRF funding amounts and, in FY2010, the increased funding percentage from the DWSRF.
The CWISA Program was established in 1987 under section 518(c) of the Clean Water Act (CWA) and is
administered under the EPA Office of Wastewater Management (OWM). From FY2001 to FY2009, this program
was funded from a 1.5 percent set-aside from the annual Clean Water State Revolving Fund (CWSRF); as of FY2010,
this set-aside amount increased to 2.0 percent.8 These funds are allocated to the EPA regions based on the level of
wastewater needs indicated in the IMS area SDS lists. Annual funding since the program's inception until FY2010
has ranged from $4.7 million to $42 million, due to different set-aside percent allocations and CWSRF funding
amounts.
The American Recovery and Reinvestment Act (ARRA) of 2009 provided the CWISA and DWIG-TSA programs with
significant additional funding ($60 million for CWISA and $30 million for DWIG-TSA, or two percent of the
6 US EPA. FY2011 National Water Program Guidance (April 2010). Available:
http://www.epa.gov/ocfo/npmguidance/2011/owater/nwp program guidanceSOS 050510.pdf.
7 IMS is a Federal agency underthe US Department of Health and Human Services.
8 Between FY1987 and FY1990, CWISA was funded by a 0.5 percent set-aside from the Construction Grants Program. In FY1991, no funding was
appropriated. In FY1992, Congress gave EPA the authority to take a 0.5 percent set-aside from the CW State Revolving Fund. In FY2001, the
set-aside was increased to 1.5 percent.
Evaluation Report - March 2011 Page 2
-------
OF THE AND
respective State Revolving Fund). Guidance associated with the ARRA funding directed that funds be allocated to
projects in a similar manner as the annual allocations, with a few management changes; these are described in the
section of this report that discusses ARRA implementation.
EPA Infrastructure Grant Program Goals and Measures
The DWIG-TSA and CWISA programs are guided by the EPA Strategic Plan. In September 2010, EPA issued its
FY2011-2015 EPA Strategic Plan, which includes strategic Goal 2: Protecting America's Waters. The Strategic
Plan defines Goal 2 as, "Protect and restore our waters to ensure that drinking water is safe, and that aquatic
ecosystems sustain fish, plants and wildlife, and economic, recreational, and subsistence activities."9
The DWIG-TSA and CWISA programs provide funding for water infrastructure in Indian country and are aimed at
realizing Goal 2 specifically through the Strategic Plan's Objective 2.1: Protect Human Health and Objective 2.2:
Protect and Restore Aquatic Watersheds and Ecosystems.10 Infrastructure funded by the DWIG-TSA program
protects public health through exposure reduction by helping systems in tribal communities provide water in
compliance with the SDWA National Primary Drinking Water Regulations (NPDWRs). Infrastructure funded by the
CWISA program protects aquatic watershed and ecosystems by helping tribal communities ensure that wastewater
introduced to aquatic environments has been properly treated and in compliance with CWA regulations. With
these and other programs and regulations within the Office of Water, EPA works with other federal agencies and
state, local, and tribal governments to make progress on this high-level strategic goal.
EPA tracks progress made by the DWIG-TSA and CWISA programs toward the Strategic Plan's Goal 2 and related
Objectives according to performance on three Strategic Plan "measures." These measures are reiterated as
strategic annual commitment measures in the Office of Water National Program Manager (NPM) Guidances.
We list these strategic measures below with the EPA annualized measure designation from the Annual
Commitment System (ACS).
> By 2015, 88 percent of the population in Indian country served by community water systems will receive
drinking water that meets all applicable health-based drinking water standards. (2005 baseline: 86 percent.
Status as of FY 2009: 81 percent.) (ACS Measure SP-3)11
> By 2015, in coordination with other federal agencies, provide access to safe drinking water for 136,100
American Indian and Alaska Native homes. (FY 2009 baseline: 80,900 homes. Universe: 360,000 homes.)
(ACS Measure SOW-18)12
> By 2015, in coordination with other federal agencies, provide access to basic sanitation for 67,900 American
Indian and Alaska Native homes. (FY 2009 baseline: 43,600 homes. Universe: 360,000 homes.) (ACS Measure
WQ-24)13
These FY2011-2015 EPA strategic measures are updates from the FY2006-2011 EPA Strategic Plan, which listed
three similar "strategic targets" related to the DWIG-TSA and CWISA programs. '
9 US EPA. FY2011-2015 EPA Strategic Plan. Page 46. Available: http://www.epa.gov/planandbudget/strategicplan.html.
10 Ibid.
11 In the FY2011-2015 EPA Strategic Plan, this measure is under Goal 2: Protecting America's Waters, Objective 2.1: Protect Human Health
12 Ibid.
13 In the FY2011-2015 EPA Strategic Plan, this measure is under Goal 2: Protecting America's Waters, Objective 2.1: Protect and Restore
Watersheds and Aquatic Ecosystems
14 US EPA. FY2006-2011 EPA Strategic Plan. Available: http://www.epa.gov/planandbudget/archive.html.
Evaluation Report - March 2011 Page 3
-------
EVALUATION OF THE DRINKING AND CLEAN INFRASTRUCTURE TRIBAL SET-ASIDE
By 2011, 86 percent of the population in Indian country served by community water systems will receive
drinking water that meets all applicable health-based drinking water standards. (2005 Baseline: 86 percent)
(ACS Measure: SP-3)16
By 2015, in coordination with other federal agencies, reduce by 50 percent the number of homes on tribal
lands lacking access to safe drinking water. (2003 baseline: Indian Health Service data indicate that 12
percent of homes on tribal lands lack access to safe drinking water (38,637 homes lack access).) (ACS
Measure: SP-5)17
By 2015, in coordination with other federal partners, reduce by 50 percent the number of homes on tribal
lands lacking access to basic sanitation (cumulative). (2003 baseline: Indian Health Service data indicate that
8.4 percent of homes on tribal lands lack access to basic sanitation (26,777 homes of an estimated 319,070
homes).) (ACS Measure: SP-15)18
This evaluation has focused on the progress made toward the three FY2011-2015 EPA Strategic Plan DWIG-
TSA and CWISA strategic "measures" listed above. Where we refer to the higher-level Strategic Plan Goal 2,
"Protecting America's Waters," we note so explicitly in the text.
Throughout the course of the evaluation, when discussing the DWIG-TSA and CWISA programs, EPA and IHS staff
also discussed four related "goals":
1. A long-term drinking water program compliance "goal" of having all public water systems in compliance
with the Safe Drinking Water Act. Although the current Strategic Plan does not specifically reference
this goal, EPA program managers routinely consider this programmatic goal when assessing progress on
its programs. In this report, we refer to this as the "long-term drinking water compliance goal."
2. The CWISA program has a long-term access "goal" of all serviceable tribal homes having access to basic
sanitation. In this report, we refer to this as the "long-term clean water access goal."
3. The US government commitment to the Plan of Implementation of the 2002 United Nations World
Summit on Sustainable Development, held in Johannesburg, South Africa.19 Specifically, the US
committed to reducing by half the proportion of people that did not have access to safe drinking water
or basic sanitation.20 In 2007, a group of federal agencies assembled an Infrastructure Task Force to
help achieve this commitment as it relates to tribal homes. This Infrastructure Task Force is comprised
of EPA, IHS, the US Department of Agriculture (USDA), and the US Department of Housing and Urban
SP-5 and SP-15, the former access strategic targets, were kept as "indicators" in the FY2011 Office of Water NPM Guidance, meaning that
EPA will continue to report on the number of tribal homes lacking access to safe drinking water and basic sanitation, but not against an annual
or long-term target.
16 In the FY2006-2011 EPA Strategic Plan, this target is under Goal 2: Clean and Safe Water, Objective 2.1: Protect Human Health, Sub-objective
2.1.1: Water Safe to Drink.
"ibid.
18 In the FY2006-2011 EPA Strategic Plan, this target is under Goal 2: Clean and Safe Water, Objective 2.2: Protect Water Quality, Sub-objective
2.2.1: Improve Water Quality on a Watershed Basis.
19 United Nations. Plan of Implementation of the World Summit on Sustainable Development. Section II. September 4, 2002. Available:
http://un-documents.net/iburgpln.htmtfll.
20 Specific commitment language was as follows: "The provision of clean drinking water and adequate sanitation is necessary to protect human
health and the environment. In this respect, we agree to halve, by the year 2015, the proportion of people who are unable to reach or to afford
safe drinking water (as outlined in the Millennium Declaration) and the proportion of people who do not have access to basic sanitation..."
Evaluation Report - March 2011 Page 4
-------
OF THE AND
Development (HUD).21 Stakeholders often refer to this cross-agency federal goal as the "access goal."
In this report, we refer to this commitment as the "US access goal."
4. The logic model developed at the beginning of the evaluation (Appendix A) identified a general "goal"
for both the DWIG-TSA and CWISA programs: To improve access to drinking water and basic sanitation
in Indian country and to improve compliance of community water systems with applicable health-based
standards. In this report, unless noted otherwise, we are not referring to this more generalized goal,
which supports and reiterates all of the aforementioned goals and measures.
Projects Eligible for DWIG-TSA and CWISA Funding
The DWIG-TSA and CWISA programs fund infrastructure projects in tribal communities. For the DWIG-TSA
program, most projects that improve a tribal drinking water system are eligible for funding. For example, eligible
projects could be those that rehabilitate or develop sources of drinking water; install treatment or storage
facilities; and replace aging water system infrastructure. CWISA funding can be used for the planning, design, and
construction of wastewater collection and treatment systems. For example, on-site septic systems and major
sewer rehabilitation are projects that are eligible for funding. However, there are some limitations in what each
program is able to fund, based the statutes that established the programs and interpretations of their respective
guidances.
The DWIG-TSA program cannot fund the following kinds of projects (per the DWIG-TSA national guidance, except
where footnoted):
> Projects that are not primarily or exclusively serving a tribal population;
> Individual home plumbing except when retrofitting for water efficiency;22
> New systems, except when an existing water source for a population is contaminated or a public health
concern, and a new system is a cost effective solution for addressing the public health risk;
> Individual home wells;
> Projects for systems that are for-profit non-community water systems; and
> Other ineligible projects listed in DWIG-TSA national guidance (e.g., reservoirs, dams, new growth, O&M).
The CWISA program cannot fund the following kinds of projects (per the CWISA national guidance, except where
footnoted):
> Indoor plumbing in individual homes;23
> Connection of individual homes to laterals (i.e., the connection from the home to the property line where a
lateral connection exists);24
21 For more information, see http://www.epa.gov/indian/trprograms/infra-water.htm.
22 Mehan, G.Tracy. US EPA. Memorandum: "Use of Drinking Water State Revolving Fund (DWSRF) Program Funds for Water Efficiency
Measures." July 25, 2003. Available: http://www.epa.gov/ogwdw/dwsrf/pdfs/memo dwsrf policy 2003-07-25.pdf.
23 US EPA Office of Water. Clean Water Infrastructure Set-Aside Grant Program: Answers to Frequently Asked Questions. March 2007.
Available: http://water.epa.gov/type/watersheds/wastewater/upload/CWISA-tribal-faq-highres.pdf.
24 Frace, Sheila, US EPA. Memorandum: Eligibility of Lateral Connections under the Clean Water Indian Set-Aside Grant Program. September
18, 2007.
Evaluation Report - March 2011 Page 5
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Projects not located within the jurisdiction of the tribal recipient (i.e., tribal reservation land) with the
exception of Alaska Native Villages and tribes in Oklahoma on former reservations;
Land purchases other than land used for the integral part of the wastewater treatment processes or used
for the ultimate disposal of treatment residues; and
Other ineligible projects listed in CWISA national guidance (e.g., new growth, economic development, O&M,
industrial wastewater treatment systems, asset management).
Role of IMS in EPA Infrastructure Grant Programs
To implement these infrastructure grant programs, EPA works in conjunction with IHS's Division of Sanitation
Facilities Construction (SFC). The mission of the IMS is to work with AI/AN peoples to raise their health status to
the highest possible level. To support IHS's mission, IMS SFC works with tribal governments to eliminate sanitation
facilities deficiencies. Through SFC, IMS provides the following services related to the EPA tribal infrastructure
programs.
Sanitation Deficiency System (SDS) List. IMS develops and maintains an inventory of sanitation deficiencies
in tribal communities, also called the SDS List, which is a part of the Sanitation Tracking and Reporting
System (STARS). EPA uses this list to make funding allocation decisions for both the DWIG-TSA and CWISA
programs. This list also guides project selection for the CWISA program and is used by some EPA regions for
DWIG-TSA project selection. Exhibit 2 provides definitions for deficiency level designations that IMS uses.
Interagency Agreements (lAs). In cases where a tribe chooses to have funds awarded through an
interagency agreement (IA), EPA works with IMS to set up the IA, and then IMS establishes a Memoranda of
Agreement (MOA) with the tribal grant recipients. Under the ARRA funding of 2009, all projects were
funded through two national-level lAs with IMS: one for the DWIG-TSA program, and one for the CWISA
program.
Coordinated Project Funding. Often, IMS provides a proportion of funding for projects that EPA chooses to
fund.
Project Implementation. Whether or not IMS administers the project funds through an IA, the tribes can
use IMS to implement projects. IMS has staff with technical expertise for sanitation and drinking water
systems, and can provide assistance with feasibility studies, design, and construction of infrastructure
projects.
Program Reporting. IMS annually provides data from the STARS system to EPA on progress made for EPA to
report on its strategic measures. IMS also responds to ad hoc requests for additional information from
STARS to answer questions on performance and implementation that arise.
Evaluation Report - March 2011 Page 6
-------
OF THE AND
Exhibit 2. IMS Deficiency Level Designations
A home that lacks access to safe drinking water or basic sanitation is one that is defined as having a
deficiency level (DL) ranking of 4 or 5 (on a 1-5 scale) by IMS in the STARS system. Deficiency levels have
the following general definitions:
> DL-5: An Indian tribe or community that lacks a safe water supply and a sewage disposal systems.
s DL-4: An Indian tribe or community with a sanitation system that lacks either a safe water supply or a
sewage disposal system.
s DL-3: An Indian tribe or community with a sanitation system which has an inadequate or partial water
supply and sewage disposal facility that does not comply with all applicable water supply and
pollution control laws, or has no solid waste disposal facility.
> DL-2: An Indian tribe or community with a sanitation system which complies with all applicable water
supply and pollution control laws, and in which the deficiencies relate to capital improvements that
are necessary to improve the facilities in order to meet the needs of such tribe or community for
domestic sanitation facilities.
s DL-1: An Indian tribe or community with a sanitation system which complies with all applicable water
supply and pollution control laws, and in which the deficiencies relate to routine replacement, repair,
or maintenance needs. A minimum level of technical assistance is required from the IMS.
s DL-0: No deficiencies to correct.
Evaluation Report - March 2011 Page 7
-------
OF THE AND
METHODS
Evaluation Approach
As a first step in the evaluation, the evaluation team refined the initial evaluation questions that had been
submitted as a part of OP's evaluation competition, identified the information needed to answer the refined
questions, and updated the draft program logic model (Appendix A) that had been previously developed by EPA.
Identification of Evaluation Questions
The evaluation team, with input from a small EPA evaluation advisory group, decided to focus on the following five
core evaluation questions:
Exhibit 1. Core Evaluation Questions
I. How does EPA implement the DWIG-TSA and CWISA programs at headquarters and in the regions?
II. To what extent are the DWIG-TSA and CWISA programs achieving their stated goals?
III. What factors influence whether the programs achieve their stated goals?
IV. Are the current DWIG-TSA and CWISA program performance measures accurate indicators of EPA's
progress?
V. What implementation improvements or innovations could be made by EPA to make the DWIG-TSA and
CWISA programs more effective in meeting the water and wastewater infrastructure needs of tribes?
The core questions focused on how the EPA headquarters and regions implement the programs, whether the
programs were and are meeting their stated goals as articulated through the program strategic measures (and if
not, why not), and whether EPA could make program adjustments to maximize program effectiveness and success.
These core evaluation questions informed data collection and analysis approaches.
Information Collection Approach
The evaluation team employed a mixed-method approach to information collection, including document review,
interviews with stakeholders, and new data collection and synthesis.
Document Review
The evaluation team reviewed relevant EPA strategic reports and guidances, EPA OW NPM guidances, EPA regional
guidances and grant requests for proposals, IMS annual reports and other documentation, and Infrastructure Task
Force reports. A full list of information sources reviewed is included as Appendix F.
Interviews
Interviews were a crucial component of the qualitative data collection for the evaluation, providing information,
insights, and perspectives that complemented and reinforced the other information. The evaluation team first
conducted an initial set of scoping calls with 13 EPA regional contacts during the development of the evaluation
methodology to inform the evaluation approach. Over the course of the evaluation, the evaluation team gathered
Evaluation Report - March 2011 Page 8
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
information from 39 people during 28 separate interviews; these interviewees were from EPA headquarters and
regions, IMS headquarters and area offices, and tribal governments. The evaluation team interviewed almost all of
the EPA regional coordinators across the nine regions that receive funding for these programs25 and a few other
regional staff members who were also familiar with these programs or had a related role in their regions. EPA
headquarters staff interviewees consisted of DWIG-TSA and CWISA program managers and individuals with
oversight roles or knowledge of program measures and reported performance. IMS interviewees consisted of two
managers at IMS headquarters and three IMS area office staff members (IMS headquarters staff suggested who to
interview from the IMS area offices). Tribal interviewees were tribal government staff familiar with these programs
(EPA regional coordinators suggested the tribal interviewees).
The roles of the individuals interviewed break down as follows:
EPA regional staff: 20
EPA headquarters staff: 10
IMS staff: 5
Tribal government staff: 4
Please see Appendix B for the interview guides used to structure the interviews.
Quantitative Data Synthesis and Analyses
The evaluation team collected data from a variety of sources:
Historic DWIG-TSA and CWISA data. EPA provided historic program data in Microsoft Excel spreadsheets
that contained data, where available, on EPA funding amounts, funding years, IA numbers, and in some
cases, IMS Sanitation Tracking and Reporting System (STARS) database identifiers such as Sanitation
Deficiency System (SDS) and Project Data System (PDS) numbers.
Safe Drinking Water Information System (SDWIS) compliance data. This data included SDWA compliance
violation information for tribal public water systems (PWS) for violation years (calendar years (CY)) 2004 to
2008, in Excel spreadsheets. The evaluation team also received summary SDWIS information in Excel pivot
tables that is used for EPA reporting on its strategic measures related to compliance. These pivot tables
include data as far back as 1998, but do not include specific violation information at the system level.
IMS STARS data. IMS provided STARS data (in Microsoft Access format) that covered data from three areas
of STARS:
Community deficiency profile (CDP) 'snapshots' for 2004-2009: Each community served by IMS is
evaluated annually by IMS engineers. Each profile for a community includes groups of homes
designated with a deficiency level for solid waste, drinking water, and sewer. This home and
deficiency level data was used to report on EPA's access strategic targets in the FY2006-2011 EPA
Strategic Plan.
SDS List 'snapshots' for 2004-2009: Each year's 'snapshot' shows the SDS projects listed as needs
within a given IMS area, its area priority within the list in that year, the estimated initial deficiency
EPA Region 3 has no federally recognized tribes within its boundaries and therefore does not receive funding or otherwise implement these
programs.
Evaluation Report - March 2011 Page 9
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
level (IDL) of the project, the estimated final deficiency level (FDL) of the project, and the scores in
eight areas that contributed to the project's priority level within the area.
PDS information: This information includes data on all projects where funding is directed through IMS,
including IDL and FDL of the project, number of homes served, funding amount, and milestone
information (e.g., date when construction started). This is likely to be the data source for reporting on
EPA's access strategic measures in the FY2011-2015 EPA Strategic Plan. This data was a 'snapshot'
from 2009.
To best use this information, the evaluation team worked with EPA headquarters and EPA regional coordinators to
review the EPA project data and fill in gaps (e.g., to add PWS identification numbers (PWSIDs) and PDS numbers),
where possible. After the evaluation team completed data cleanup, lEc developed an Access database ("project
database") that linked, as possible, EPA project data from both programs with IMS STARS information. The project
database also linked EPA DWIG-TSA project information to the detailed SDWIS violation data, by PWSID, as
possible. Once connections between the datasets were made, the evaluation team was able to conduct data
analyses to help answer certain evaluation questions. The evaluation team is providing this project database to
EPA at the end of this evaluation, along with a memorandum describing the database and how EPA could update
the database with new project data, STARS data, and/or SDWIS data.
The evaluation team conducted a series of iterative data analyses, including statistical correlations (Appendix D)
and analyses of project performance information (e.g., project duration, IDLs and FDLs of projects funded)
(Appendix E). In the evaluation methodology, the evaluation team had considered a regression analysis, with the
dependent variable being the STARS CDP deficiency level in a set of community homes, and the independent
variables including EPA region, IMS area, project duration, project funding, available compliance information, and
other factors that may influence a community's access deficiency designation. However, the correlation analyses
conducted indicated a lack of evidence to suggest that a more sophisticated regression analysis would yield
meaningful results. This conclusion was further supported by the evaluation team's increased understanding of
the lack of available data on many of the independent variables of interest.
Data analysis spanned projects from both programs that were funded between calendar years 2003 and 2009. In
this evaluation, we refer to "funding years", which we define as the calendar year when a project was funded by
EPA, and except where noted, this unit is the timescale for analyses. This designation is different from when
regions received funding from headquarters, which occurs on a federal fiscal year basis. It should be noted that
projects funded under the American Recovery and Reinvestment Act of 2009 (ARRA) are analyzed separately from
the "routine" program implementation captured in the 2003-2009 time scope.
Data Analysis Limitations
The data available for this evaluation had several gaps, and, as is typical of most quantitative analyses, the
evaluation team made assumptions in order to proceed with particular data analyses. The evaluation team does
not think that either the gaps in the data or and the assumptions made have undermined or weakened the findings
and recommendations for this evaluation, though we do provide caveats and other explanations where needed. A
few notes about data gaps and assumptions are provided below.
Evaluation Report - March 2011 Page 10
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Lack of Linkages to STARS orSDWIS Data
Even though the evaluation team, EPA headquarters staff, and EPA regional coordinators reviewed the EPA
DWIG-TSA and CWISA project data, some gaps still persisted in the EPA project data, such as missing
PWSIDs, IA numbers, and PDS or SDS numbers that would allow linkages between the EPA and IMS STARS
information or SDWIS information. In particular, 51 of the 330 DWIG-TSA projects (15 percent) provided
with funding between 2003 and 2009 (including ARRA projects) either lacked a PWSID or the PWSID could
not be linked to provided SDWIS data.
For CWISA projects, information on when funds were awarded to a tribe was not provided. Therefore the
evaluation team, in consultation with EPA headquarters staff, assumed that the fiscal year in which funds
were appropriated was the same as the year in which a project was funded.
Under both programs, projects funded by direct grant (13.5 percent of DWIG-TSA projects and 4.9 percent
of CWISA projects, between 2003 and 2009) rather than through an IA with IMS often did not include PDS
information available from IMS (e.g., IDL, FDL, number of homes served, milestone information), unless IMS
was involved in implementation of the tribe's grant or otherwise tracking the project. The data analyses
that reflect progress toward the access measures therefore did not cover all projects funded through the
programs.
Limited SDWIS Data
The SDWIS dataset provided to the evaluators contained only violations that occurred for the first time
between 2004 and 2008. It did not contain ongoing violations that occurred prior to 2004, so the data may
be missing some information relevant to systems provided funding under the DWIG-TSA program. In
addition, as only 2004 to 2008 violation data were included, this evaluation did not have information on pre-
2004 compliance history for EPA-funded systems or more recent compliance information.
For analyses related to the SDWIS data, if a project was funded the same year as when a violation occurred,
the analyses assumed that violation occurred before funding.
Evaluation Report - March 2011 Page 11
-------
OF THE AND
FINDINGS: PROGRAM IMPLEMENTATION
Core Evaluation Question I: How does EPA implement the DWIG-TSA and CWISA programs
at headquarters and in the regions?
This section explores core evaluation question I, including how the programs allocate funds, how communication is
working within EPA and with tribes and IMS, and project selection under both programs. In addition, this section
highlights differences in implementation under the 2009 one-time ARRA funding.
EPA Headquarters Implementation of Tribal Infrastructure Programs
EPA headquarters administers these tribal infrastructure programs in two offices within EPA Office of Water:
OGWDW for the DWIG-TSA program and OWM for the CWISA program. In addition to creating guidance for
implementation, EPA headquarters is responsible for program oversight and ensuring that the programs are
implemented effectively. EPA headquarters manages the allocation of annual funding to the regions,
communicates with EPA regions regarding program implementation, monitors program implementation in the
context of the programs' strategic measures, and coordinates with IMS headquarters as needed on program
coordination and oversight. For ARRA-funded projects, EPA headquarters staff are also the project officers for the
lAs with IMS, whereas for the annual appropriations, EPA regions staff are the project officers.
DWIG-TSA Allocation of Funds
Each of the nine regions with federally recognized tribes (all EPA regions except Region 3) receives a two percent
base allocation from the annual DWSRF allotment, regardless of the extent of identified infrastructure needs in
each region. EPA headquarters allots the remaining 82 percent of the funds to the regions based on the identified
needs in the IMS area SDS lists and the EPA DWINS. The national guidance for the DWIG-TSA program describes
why this allocation method was developed.
> The two percent base allocation was provided to, "allow for at least one viable project in each Region, each
year, and thus allow every Tribe to at least have the opportunity to receive a grant from the DWIG TSA
funds."26
> For using both the SDS list and DWINS, the guidelines acknowledge that neither tool is perfect, but that both
are used to balance out each tool's weaknesses.
Overall, the DWIG-TSA program stakeholders interviewed indicated that they understood the reasons for the
design of the allocation formula, but many shared some concerns with the formula. In particular, interviewees
noted that EPA has not updated the DWINS for tribal systems since 1999 and thus the allocation is not based on
current need in Indian country. EPA is currently updating the tribal drinking water needs information through the
2011 DWINS, but the survey results will not be available until approximately 2013.2S In addition, a few
26 US EPA. Drinking Water Infrastructure Grants, Tribal Set-Aside Program: Final Guidelines (October 1998). Available:
http://www.epa.gov/safewater/dwsrf/allotments/tribes/pdf/guidelines dwsrf tribal.pdf.
"ibid.
28 Dougherty, Cynthia, US EPA. Letter to Tribal leaders on 2011 Drinking Water Infrastructure Needs Survey, September 15, 2010.
Evaluation Report - March 2011 Page 12
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
interviewees noted that the two percent base allocation to each region means that a region may receive funds but
not have any significant access needs or compliance issues to be addressed.
Interviewees also noted that the IMS updates the SDS list annually, but that it may not be a complete list of all
drinking water infrastructure needs. In particular, interviewees noted that IMS does not systematically track SDWA
compliance issues on the SDS list, and thus compliance issues that may contribute to public health impacts may
not be captured in the SDS list. In addition, a few interviewees said that some tribes do not work with IMS to get
their needs evaluated and listed, or that IMS may not be aware of all the specific needs in some tribal communities.
The IMS SDS list also contains drinking water needs that are not eligible for DWIG-TSA funding, such as individual
drinking water wells. However, projects that are not eligible for DWIG-TSA funding in the SDS list still contribute to
the overall drinking water need in tribal communities and thus the allocation of resources to the EPA regions.
Further, the SDS list includes projects that address needs at all deficiency levels. Though EPA regions can fund
projects that address lower deficiency needs (i.e., DLs 1-3), the DWIG-TSA program progress is partly measured by
the number of homes provided access to safe drinking water, as defined from moving from a DL-4 or 5 designation
to a lower level. If an EPA region had many projects on the SDS list addressing lower deficiency needs that resulted
in a high dollar amount of need, EPA would allocate proportionally more funding to this region, though these
needs do not align with meeting the DWIG-TSA strategic measure.
CWISA Allocation of Funds
EPA allocates the annual CWISA funding based on the percent of needs of individual IMS areas as they are
identified in the SDS list. Each year, IMS provides the area SDS lists to EPA with all of the sewer need infrastructure
costs summed in each IMS area; EPA headquarters allocates the funds based on the proportion of funding need
across all IMS areas. Allocation by IMS area need means that some EPA regions may not fund any clean water
projects in a given year, depending on where a region's projects fall on that year's IMS area SDS list. As some IMS
areas cover multiple EPA regions, one EPA region's clean water needs might all rank higher than another region's
needs within the given funding allocation, and thus all the funding allocated to an IMS area would go to projects in
the one EPA region.
For the CWISA allocation formula, some regional interviewees said that the allocation formula should be by EPA
region rather than IMS area so that at least one clean water project could be funded in an EPA region every year.
However, other regional and EPA headquarters staff indicated that, as long as the funding is addressing the highest
priority needs throughout Indian country, the allocation formula does not need to change.
Like the DWIG-TSA program, the SDS list includes projects that are not eligible for CWISA funding, such as projects
for individual home plumbing or projects that are not on tribal reservation land. However, projects that are not
eligible for CWISA funding in the SDS list still contribute to the overall wastewater infrastructure need and thus the
allocation of funds from this program. Due to the inclusion of these non-EPA eligible projects, funding to a
particular IHS area would not align well with the CWISA program if many projects in the year's list were not eligible
for CWISA funding.
In addition, the SDS list includes projects that address needs at all deficiency levels. Though EPA regions can fund
projects that address lower deficiency needs (i.e., DLs 1-3), the CWISA program is measured by the number of
homes provided access, as defined by moving from a DL-4 or 5 designation to a lower level. If an IHS area had
many projects addressing lower deficiency needs that resulted in a high dollar amount of need, EPA would allocate
Evaluation Report - March 2011 Page 13
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
proportionally more funding to this area, though these needs do not align with meeting the CWISA strategic
measure on providing access to basic sanitation.
Allocation Between Programs
A few regional coordinators from both infrastructure programs and a tribal government interviewee indicated
interest in being able to shift funds between the CWISA and DWIG-TSA programs to address the overall higher
priority infrastructure needs that may exist in a region within a given year. For example, a region may have many
drinking water projects that have pressing public health concerns that could be addressed by infrastructure
funding, but not enough DWIG-TSA funding to address the issues in that year.
Language that would allow this transfer of funds is in EPA's proposed FY2011 budget, but as of this reporting
writing, EPA does not yet know whether the US Congress will pass this language. If the US Congress approves the
language, EPA headquarters will work with the regions to establish how this transfer could be conducted.29
EPA Headquarters Communication and Coordination with EPA Regions
EPA headquarters has oversight responsibilities for the implementation of the infrastructure programs which
includes summary and analysis of fund utilization and annual impact on strategic measures, and the
communication of these findings to interested senior leaders within EPA and other federal managers (e.g., the
White House's Office of Management and Budget (OMB)). To fulfill these duties, EPA headquarters must
coordinate and communicate with EPA regional staff on project selection and implementation.
For both grant programs, interviews with EPA staff indicate that, with the exception of ARRA funding, EPA regional
coordinators and EPA headquarters staff communication about project selection and progress is ad hoc and
project tracking is inconsistent. As a headquarters staff member said, "communication [about project selection] is
pretty superficial—we may hear about particular projects retrospectively on a case-by-case basis." A few regional
coordinators said that the staff turnover at EPA headquarters had been one contributing factor to the lack of
historic communication between headquarters and the regions. Some of these coordinators and others
mentioned communications improvements in recent years due to more engaged and consistent headquarters
staff.
Previously the DWIG-TSA Grant Tracking Tool was used for national project tracking. However, the DWIG-TSA
regional coordinators said that the tool was difficult to use, requested information available in other EPA
databases, and that headquarters did not seem to indicate to the regions that its use was a priority. One regional
coordinator said, "Headquarters always talked about wanting updates on the projects and they developed this
clunky grant tracking tool. However, they never seemed serious about wanting the information." (Headquarters
staff did request that regional coordinators use the tool in annual funding memoranda to the regions.) Both
CWISA and DWIG-TSA regional staff said that they provide information on which projects are selected to
headquarters, but do not provide other information, unless requested.
Regional coordinators and headquarters staff said that communication and coordination around these programs
has increased and improved as a result of the ARRA processes (we discuss ARRA implementation in more depth
later in this report). However, with this increased communication, some regional coordinators expressed concern
that EPA headquarters may want to review regions' selected projects and potentially take away the regional
D. Harvey & M. Richardson, US EPA Office of Water. Personal communication. December 14, 2010.
Evaluation Report - March 2011 Page 14
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
autonomy in project selection. One regional coordinator said, "We think headquarters is going toward the idea
that they want to decide what projects we fund and under what circumstances." Headquarters staff, for their part,
said that they want to know more about which projects EPA regions fund to better understand how EPA is meeting
or could better meet its strategic measures, but that they view the programs' regional coordinators as being in the
best position to understand tribal needs and determine project selection.
EPA headquarters staff that work directly on these programs said that they are often asked for reports on program
progress from the OMB, the EPA Office of the Chief Financial Officer (OCFO), the US Congress, and EPA OW
management. The lack of consistent and routine tracking and communication about these programs and
associated projects makes it difficult for EPA headquarters staff to respond in a timely and useful way to these
ongoing requests. This dynamic also stymies the ability of headquarters staff to effectively advocate for the DWIG-
TSA and CWISA programs. A headquarters manager noted, "We hear a lot of anecdotal information about what's
going on... we don't have a lot of quantitative information to respond to specific questions about how we can best
protect that investment."
In addition to the issues described by interviewees, during the course of the evaluation, the evaluators came to
recognize that there is a basic lack of information flowing between the EPA regions and headquarters, as well as
between the regions. For example, headquarters DWIG-TSA staff did not have a strong understanding of how the
regions select projects, and the staff with both programs did not have a clear understanding of how many staff
resources the regions devote to these programs. The regions also had an inconsistent understanding of each
others' work, including where they could learn from each other's solutions to particular challenges that have arisen
in multiple regions. Moreover, for both programs, there is insufficient information for EPA at both regional and
headquarters levels to learn about what is working and not working well program-wide, and to provide a basis for
adapting the programs accordingly.
EPA Headquarters Communication and Coordination with IMS
As already described, IMS administers a large percentage of DWIG-TSA and CWISA projects through lAs with EPA.
Further, EPA relies on the IMS STARS database to report to OMB, the US Congress, and the public on EPA's own
progress toward its strategic measures. Due to these implementation relationships and reporting dependencies,
as well as EPA's program oversight responsibilities, EPA headquarters staff periodically communicate with IMS
headquarters staff to, for example, request updated performance information from STARS and to track dispersed
IA funds. Finally, IMS and EPA are both part of the Interagency Task Force related to the US access goal for water
infrastructure and work jointly, along with HUD and USDA, to help meet this commitment.
From discussions with EPA headquarters and IMS headquarters staff, it appears that communication and
cooperation between the two agencies is working well overall. The ARRA process has led to more coordination,
such as filling in funding data gaps and developing a standard template for lAs. Three IMS staff members
mentioned, however, that multiple and duplicative requests for information from EPA can be frustrating and time
consuming, but that this issue seems to be improving due to reduced staff turnover at EPA headquarters.
From interviews with IMS headquarters and area staff members, there appears to be ongoing questions about the
expectations and requirements around who is the "lead" federal agency on lAs and which agency's rules and
guidelines apply under lAs. One tribal government interviewee who works closely with these programs said, "It
would be good if EPA would accept IMS as the lead agency. One federal entity needs to follow their [own] rules-
otherwise there are two sets of federal rules [to be followed]." As mentioned, a standard template with terms and
Evaluation Report - March 2011 Page 15
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
conditions for all EPA-IHS DWIG-TSA and CWISA lAs was developed as a result of the ARRA funding; the
standardized IA may address these questions in the future, but it is too soon for this evaluation to say what effect
it may have both functionally on the implementation of these programs. One IMS staff person also mentioned that
having specific funds from EPA for IMS to administer projects through lAs would be useful. EPA had previously
provided administrative funds to IMS through the CWISA program, but there are currently no funds for IA
administration for either infrastructure program. These IA questions may be one area for continued discussion
and coordination between EPA and IMS headquarters.
Nationally, EPA does not consistently provide data on tribal community water systems' compliance history to IMS
for inclusion into the formula that IMS uses to rank projects on the SDS list. As discussed later in this report on
DWIG-TSA regional implementation, at a December 2010 meeting of EPA headquarters and regional staff, the
DWIG-TSA staff decided that, going forward, regions would more consistently provide compliance information to
IMS area offices to better inform the SDS list.
EPA Regional Implementation: DWIG-TSA Program
The evaluators examined DWIG-TSA regional project selection from three perspectives: the processes used by each
region to select projects, the role of SDWA compliance history, and provision of access to drinking water through
DWIG-TSA funding. We address our findings on each of these perspectives below, following a brief review of
projects funded and staffing levels in each region.
Table 1 provides the total number of DWIG-TSA projects funded by EPA between 2003 and 2009, excluding ARRA
projects. Of these 267 projects funded between 2003 and 2009, EPA administered 36 projects (13.5 percent of all
projects, representing 13.7 percent of funding) as direct grants to the tribes. EPA has funded the remaining
projects through lAs with IMS. Additional summary information about these projects, including total and average
funding amounts and number of lAs versus direct grants, is available in Appendix E (Tables 1-5).
Table 1. Number of DWIG-TSA Projects, by EPA Region and Funding Year
1
2
4
5
6
7
8
9
10
Total
0
0
1
4
2
2
2
19
16
46
0
1
1
2
3
4
4
16
14
45
4
1
1
4
2
1
6
11
10
40
0
0
1
8
2
3
4
13
7
38
4
2
1
4
3
3
8
6
8
39
3
0
1
3
0
3
7
7
4
28
0
1
1
4
2
2
4
9
8
31
11
5
7
29
14
18
35
81
67
267
Note: This table includes all projects funded by EPA from 2003-2009, excluding ARRA projects, but including direct grants and lAs with IMS.
Some projects are funded multiple times over multiple years under the same IMS STARS PDS number; we consider these separate projects for
this table to be consistent with EPA records. There are 257 unique projects within this timeframe.
As the data above shows, regions with larger tribal populations (e.g., Regions 9 and 10) have funded a higher
number of projects.
Evaluation Report-March 2011
Page 16
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
Table 2. Estimated Level of
DWIG-TSA Regional Staffing
Region Staffing (FTE)
1
2
4
5
6
7
8
9
10
0.15
0.25
0.30
0.50
0.15
0.25
1.0
1.0
0.50
In terms of administering programs, we asked regional coordinators to
estimate the level of staffing (full-time equivalent, or FTE) dedicated to
coordinating the DWIG-TSA program in their regions. Table 2 provides these
estimates of staffing. Overall, regions with smaller tribal populations devote
fewer staff to the program. In addition regional coordinators have other
responsibilities beyond the DWIG-TSA program, though no regional
coordinators mentioned that having multiple responsibilities hinders their
ability to implement the DWIG-TSA program. In Region 10, two regional
coordinators split the DWIG-TSA responsibilities between Alaska and the rest
of the Region 10 states; the one in Alaska works with the State of Alaska on
compliance issues, as systems at Alaska Native Villages are state-regulated,
rather than EPA-regulated.
DWIG-TSA Regional Project Selection Processes
The national DWIG-TSA guidelines provide regions with flexibility in most aspects of project solicitation, ranking,
and selection. However, the national DWIG-TSA guidelines do require the following of EPA regions in project
solicitation and selection:30
> A method for identifying water system projects and prioritizing those projects, and providing this method to
tribes and other potentially interested parties in the region for review and comment;
> A quantifiable method of ranking all of the identified projects, with priority given to projects that address
the most serious risk to human health; are necessary to ensure compliance with the requirements of the
SDWA; and assist systems most in need on a per household basis;
> Coordination with local IMS area office(s) on projects selected, even if the area SDS list is not used for
project ranking and selection; and,
> Determination of adequate capacity at a system (beginning in FY2011, any system to be assisted with DWIG-
TSA funds must have a certified operator).
In practice, project solicitation and selection varies widely by region: some regions rely completely on the IMS SDS
list for project selection while others use a request for proposals (RFP) process for soliciting projects from the
tribes and then rank those submissions. The table in Appendix C provides a summary of the different processes
used in each of the regions. Most but not all regional processes are in accordance with the requirements listed
above from the national guidelines. From the information the evaluation team received, it appears that three
regions are not explicit in how they quantify or consider compliance information or public health impacts in project
selection.31 However, the extent to which the regions believe they have discretion in developing processes that fit
individual regional conditions and circumstances, beyond what is outlined in the national guidance, is unclear.
30 US EPA. Drinking Water Infrastructure Grants, Tribal Set-Aside Program: Final Guidelines (October 1998). Available:
http://www.epa.gov/safewater/dwsrf/allotments/tribes/pdf/guidelines dwsrf tribal.pdf.
31 Regions 2, 4, and 7 did not provide any quantifiable guidance to the evaluators on how projects are selected and thus it is not transparent to
the evaluation how compliance or public health impacts are considered in project selection. Both Region 2 and Region 7 indicated that there
has not been a need to choose between projects in a given year during the program history. Region 4 uses a rotating method of awarding
projects to its six eligible tribes, working with the tribes to determine which project(s) to fund in a given year.
Evaluation Report-March 2011
Page 17
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
DWIG-TSA Project Selection: Compliance History
Several EPA staff expressed a keen interest in improving compliance with the Safe Drinking Water Act (SDWA) and
achieving the associated safe drinking water access performance measure through improved system compliance
with health-based standards. However, the role of compliance history and prevention of new compliance
problems in project selection differs from region to region. From the regional guidances reviewed, it appears that
all regions consider compliance and/or public health impacts when selecting projects to some extent, though, as
mentioned above, how regions without quantifiable or written guidance weight these considerations is not clear.
In addition, compliance history or public health impact is not the only factor regions consider in selecting projects;
other factors, such as economic feasibility and tribal engagement (e.g., interest by the tribe in the project,
contributing funds) and capacity affect project selection. The table in Appendix C provides an overview of each
region's project selection process, including how compliance history or public health impacts of the project are
considered.
EPA does not provide data on tribal community water systems' compliance history to IMS for inclusion into the
annual SDS process on a nation-wide basis, though some EPA regions may share this information with IMS area
offices. Further, compliance history, even when it is considered in the generation of the SDS list, would not
necessarily result in a high rank for associated projects. Other factors such economic feasibility and funding from
the tribe affect the ranking of projects and can outweigh any compliance issues. Those regions that select projects
based on their rank order on the SDS list may therefore not be selecting projects with the most pressing
compliance needs. Moreover, information collected through this evaluation, described in more detail in the
section on Program Goals and Measures, suggests that many compliance problems are not driven by infrastructure
deficiencies that can be addressed with DWIG-TSA funding.
At a December 2010 meeting of EPA headquarters staff and regional coordinators, the DWIG-TSA staff decided
that, going forward, regions would more consistently provide compliance information to IMS area offices to better
inform the SDS list. They also agreed to more routinely and thoroughly analyze their own regional compliance data
to understand the issues underlying tribal SDWA compliance violations and, based on this analysis, consider
funding eligible projects that would address these issues. We support these actions for increased analysis and
information sharing to help ensure that EPA and IMS are aware of pressing compliance issues that would be eligible
for DWIG-TSA funding.
As described above, compliance issues at tribal water systems may not necessarily be addressed by DWIG-TSA
funding, due to other weighting factors in regions' project selection. To better understand the extent to which
DWIG-TSA projects addressed compliance issues, the evaluators examined funded systems' compliance versus the
universe of non-funded tribal community water systems (CWSs). From an analysis of available SDWIS information,
it appears that water systems associated with DWIG-TSA projects have had more compliance issues than non-
funded tribal CWSs, suggesting that existing compliance issues may contribute to EPA regional project selection.
As Table 3 shows, a higher percentage of EPA-funded systems had at least one health-based violation between
2004 and 2008 when compared against non-funded systems. In addition, a higher percentage of EPA-funded
systems had repeat health-based violations of the same type and same rule between 2004 and 2008, when
compared to non-funded systems. Note that with the data available, the evaluators cannot determine what how
many of these violations are infrastructure-related. Table 12 in Appendix E provides more information about the
common types of violations at EPA-funded systems.
Evaluation Report - March 2011 Page 18
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
Table 3. Compliance Information for Systems in Evaluation Universe (SDWIS: 2004-2008)32
Universe
All EPA Tribal Systems (CWSs)
EPA-funded Systems (CWSs and 12 non-CWSs)
Non-Funded Systems (CWSs)
,of systems
733
249
496
# with at least 1 HB
314 (42.9%)
116 (46.6%)
198 (39.9%)
# with repeat HB
violations
195 (26.6%)
83 (33.3%)
112 (22.6%)
CWSs = Community Water Systems; HB= Health-Based
A requirement for certified operators at public water systems that receive EPA DWIG-TSA funding went into effect
for the FY2011 DWIG-TSA funding cycle. This requirement may have consequences for which water systems are
eligible for funding, but it is not yet clear what the impacts may be on project selection and to what extent it will
affect non-compliant systems from being able to receive EPA infrastructure funding.
DWIG-TSA Project Selection: Access to Safe Drinking Water
Across the regions, analysis shows that the ability of a project to provide access to safe drinking water (as defined
as a change from a DL-4 or -5 to DL-3 (or lower) is one factor in determining if a project is selected, but not the
primary factor. The evaluators analyzed available information from STARS to determine the extent to which
DWIG-TSA funding has addressed these deficiencies to safe drinking water. From PDS information, DWIG-TSA
projects addressed the needs of DL-2 and DL-3 homes more than those of DL-4 and DL-5 homes: only 36 percent of
projects addressed DL-4 or DL-5 needs, representing approximately 26 percent of homes served by the programs.
However, these data do not explain why EPA regions selected certain projects. For example, an EPA region may
not have any DL-4 or DL-5 projects to choose from because access is not a problem in the region or the drinking
water access needs that do exist are not feasible to fund. For reference, Appendix E (Tables 6-10) provides
detailed data where it was available on the SDS priority level of DWIG-TSA projects funded, deficiency level
information from the SDS list, and the IDLand FDL information of funded projects from PDS.33
EPA Regional Implementation: CWISA Program
The evaluators examined CWISA regional project selection from three perspectives: the processes used by each
region to select projects, the specific role of the SDS list in project selection, and whether use of the SDS list to
select projects has maximized EPA's funding for projects that would provide access to basic water sanitation for
tribal homes that currently do not have access. We share our findings on each of these perspectives below
following a brief review of CWISA projects funded by and staffing levels in each region.
Table 4 provides the total number of CWISA projects funded by EPA between 2003 and 2009, excluding ARRA
projects. Of these 429 projects funded between 2003 and 2009, EPA administered 21 projects (4.9 percent of all
projects; reflecting 2.9 percent of CWISA funding in this time period) as direct grants to the tribes. EPA funded the
32 Non-funded systems analyses only looked at community water systems (CWSs) to be consistent with the EPA-reported numbers on
compliance. However, EPA DWIG-TSA projects have funded three non-community water systems and nine non-transient non-community water
systems between 2003 and 2009 and analyses on EPA DWIG-TSA-funded systems includes these systems. Unless otherwise specified, when
referring to "systems", we are referring to only CWSs for the non-funded universe and to all funded PWSs for the funded universe. These
analyses do not include the 51 DWIG-TSA projects that did not have PWSIDdata or had PWSIDs that could not be linked to provided SDWIS
data.
33 These analyses examine projects that could be linked to IHS STARS information. For PDS, this information includes 204 of the 257 unique EPA
projects (79 percent). For SDS, this information includes 72 of the 214 projects funded between 2004 and 2009 (34 percent). SDS 'snapshots'
were only available to this evaluation for 2004 to 2009.
Evaluation Report-March 2011
Page 19
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
remaining projects as lAs with IMS. Additional summary information about these projects, including total and
average funding amounts and number of lAs versus direct grants, is available in Appendix E (Tables 14-19).
Table 4. Number of CWISA Projects by EPA Region and Funding Year
Region 2003 2004 2005 2006 2007 2008 2009 Total
1
2
4
5
6
7
8
9
10
Total
2
0
9
1
5
2
7
52
17
95
2
1
3
5
3
1
7
23
12
57
1
1
0
7
3
0
6
34
9
61
0
1
4
2
3
1
4
20
10
45
1
1
1
1
5
1
4
42
9
65
0
0
1
3
2
1
3
26
11
47
1
1
3
5
3
0
6
35
5
59
7
5
14
24
24
4
36
224
65
429
Note: This table includes all projects funded by EPA from 2003-2009, excluding ARRA projects, but including direct grants and lAs with IMS.
Some projects are funded multiple times over multiple years under the same PDS number; we consider these as separate projects for this table
to be consistent with EPA records. There are 402 unique projects in thistimeframe.
By far, EPA Region 9 has funded the most CWISA projects, followed by EPA Region 10. Given that Region 9
includes four IMS area offices and that funding is allocated by needs highlighted in IMS areas, this result is not
surprising.
Table 5. Estimated Level of
DA/ISA regional Staffing
In terms of administering programs, we asked regional coordinators during
interviews to estimate the level of FTE dedicated to coordinating the
CWISA program in their regions. Table 5 provides these estimated staffing
levels. Overall, regions with smaller tribal populations have fewer FTE and
staff generally have responsibilities beyond the CWISA program. Further,
for some regions, fewer FTEs work on the CWISA program than on the
DWIG-TSA program, as the staff do not need to work with tribes to select
projects or there were no or very few projects funded from a region.
CWISA Regional Processes for Project Selection
When compared to the DWIG-TSA program, the national guidelines for the
CWISA program provide less flexibility to regions in selecting projects. Per
the guidelines, regional coordinators are expected to select projects that
are on the SDS list, and fund them in priority order down the list. Regions may skip projects that have costs that
exceed available funding, have funding from another source such as IMS, or are ineligible for CWISA funding (e.g.,
drinking water projects, solid waste projects, or projects not on tribal land). Regions may also skip over projects
for other compelling reasons, but must provide an explanation to EPA headquarters for approval; according to EPA
headquarters, there does not appear to have been such a request within period of time covered in the
evaluation.34
Region Staffing (FTE)
1
2
4
5
6
7
8
9
10
Negligible
Unknown (did
not interview)
0.15
0.5
0.15
0.10
1.0
1.0
0.50
Richardson, M., US EPA Office of Water. Personal Communication. February 10, 2010.
Evaluation Report-March 2011
Page 20
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
According to interviews with regional coordinators, CWISA project selection at the regions is conducted in
accordance with the national guidelines, with the exception of one region. The coordinator in this region
described how the weighting factors that determine ranking on the SDS list mean that projects can rank highly
without having access problems, and that the region considers the EPA strategic measure on access as a
compelling reason to skip projects on the SDS list. The coordinator said the his region looks for planned,
economically feasible projects that will address EPA's strategic measure for access to basic sanitation
infrastructure, and that, if EPA wanted the regions to rank straight down the SDS list without using judgment, then
EPA should give the money directly to IMS for it to use.35
Given that, with the exceptions noted above, the CWISA program uses the SDS list in rank order to select projects
for funding, this evaluation looked at the DLs of funded projects to determine the extent to which funded projects
would provide access as it is defined as an immediate change in DLs (from a DL-4 or DL-5 to a lower deficiency
level) upon project funding. The analysis, summarized in Table 6, shows that the projects funded by CWISA range
widely across IMS areas' SDS priority listing. For example, between 2004 and 2009, the CWISA program has funded
projects in the Nashville Area offices with priority ranking between 1 and 13, which represents projects falling
within the top 7 percent of needs in the area. In the Navajo Area office, the CWISA program has funded projects
ranking between 1 and 546, with the latter falling in the bottom quartile of SDS projects ranked in that area in that
year. The percentiles (listed in parentheses after the minimum and maximum ranking) are more illustrative than
are the absolute ranking of the projects selected because some are SDS lists are substantially longer than others.
(Note that the EPA regional rankings are listed with each related IMS area office because each IMS area has its own
SDS list from which the EPA regions choose from. In some cases, EPA regions are choosing from multiple IMS area
SDS lists because the regional office-area office boundaries overlap.)
This analysis shows that, on average, most of the EPA regions are selecting CWISA projects from within the top
quarter of the area SDS priority lists. In some cases, a few regions chose projects that were not ranked highly on
the SDS lists; however, the data do not explain why the regions chose or skipped specific projects in any given IMS
area. For example, an IMS area may already be funding highly ranked projects with IMS funds. Area office lists may
also have a large number of projects that are drinking water-related instead of sewer-related, are too expensive to
be funded with available EPA funds, or are otherwise ineligible for funding, forcing the EPA region to skip many
projects on the list to find an eligible one. Appendix E (Tables 21-22) provides additional data on information from
the SDS lists.36
Poloncsik, S., US EPA Region 5. Interview and personal communication. August 26, 2010 and January 11, 2011.
36 These analyses examine projects that could be linked to the IMS SDS list, which is approximately 82 percent of the EPA, non-ARRA, projects
funded between 2004 and 2009 (255 out of 310). SDS 'snapshots' were only available to this evaluation for 2004 to 2009.
Evaluation Report - March 2011 Page 21
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
Table 6. IMS Area Priority Ranking of CWISA Projects for Funding Years 2004-2009, excluding ARRA projects, by
IMS Area and EPA Region
IMS Area/ EPA Region # of Projects Minimum Maximum Average SDS Median SDS
Funded Rank/Percentile Rank/Percentile Rank Rank
Aberdeen
Region 7
Region 8
Alaska (RIO)
Albuquerque
Region 6
Region 8
Bemidji (R5)
Billings (R8)
California (R9)
Nashville
Region 1
Region 2
Region 4
Region 6
Navajo (R9)
Oklahoma
Region 6
Region 7
Phoenix
Region 8
Region 9
Portland (RIO)
Tucson (R9)
Overall
9
0
9
37
8
7
1
19
12
28
18
3
4
9
2
107
9
6
3
14
0
14
6
10
277
2 (1.0%)
n/a
2 (1.0%)
1 (0.2%)
2 (0.9%)
2 (0.9%)
8 (4.1%)
11 (6.1%)
1 (0.7%)
1 (0.4%)
1 (0.5%)
2 (1.3%)
1 (0.6%)
1 (0.5%)
1 (0.5%)
1 (0.1%)
4(1.2%)
4(1.2%)
15 (4.6%)
4(1.6%)
n/a
4(1.6%)
5 (2.5%)
1 (0.7%)
1
12 (5.7%)
n/a
12 (5.7%)
152 (27.2%)
21 (9.8%)
21 (9.8%)
8 (4.1%)
152 (66.4%)
34 (24.0%)
120 (40.4%)
13 (6.5%)
6 (3.0%)
9 (5.1%)
13 (6.5%)
9 (4.6%)
536 (76.4%)
36 (11.0%)
27 (8.3%)
36 (11.0%)
109 (42.7%)
n/a
109 (44.0%)
92 (46.9%)
40 (26.0%)
536
6.89
n/a
6.89
20.00
8.75
8.86
8.00
79.89
6.92
19.36
5.33
4.00
5.00
6.00
5.00
67.35
18.44
14.83
25.67
25.64
n/a
25.64
22.33
10.70
40.06
7
n/a
7
15
8
8
8
79
4
11
4.5
4
6
5
5
20
16
14
26
11.5
n/a
11.5
8
2
14
Note: The above is the minimum, maximum, average, and median priority rankings, by area, for all EPA projects funded between 2004 and
2009, excluding ARRA. Forthe minimum and maximum, the percentile ranks within that year's area SDS list is in parentheses. This information
is presented by area, as SDS lists are by IMS area, rather than in EPA regions. Where multiple EPA regions work with an IMS area, the region's
specific information is presented. Percentiles are not provided for median and average numbers, as these are a result of multiple years of data
and area SDS lists vary in length each year.
In short, choosing projects in rank order from the area office SDS lists is not a straight-forward exercise for EPA
regions; and based on this analysis alone, it is not clear whether this process supports EPA's access strategic
measure for basic sanitation.
CWISA Project Selection: Access to Basic Sanitation
The existing deficiency level (called "initial deficiency level" or IDL in STARS) is one of eight criteria that determines
a project's SDS ranking. Given the EPA strategic measure related to providing access to basic sanitation (moving
from a IDL-4 or IDL-5 to a lower final deficiency level (FDL)), the evaluators examined the IDLs and FDLs of selected
projects in the SDS list to understand if access to basic sanitation as EPA currently defines it was supported in
practice. Analysis of SDS data showed that a majority of EPA CWISA projects funded (60.0 percent) were "access"
projects, with IDL-4 and DL-5 designations and FDL-3 or lower designations; however, the other 40.0 percent of
projects addressed IDL-2 and IDL-3 needs or would not be expected to provide access (i.e., IDL and FDL of 4). No
Evaluation Report-March 2011
Page 22
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
DL-1 projects were funded.37 A further analysis showed that this breakdown of IDLs is similar to the overall
deficiency designations of the top ranked projects across the IMS area SDS lists, which indicates that top-ranked
projects within an SDS list are not necessarily addressing DL-4 or DL-5 needs. Appendix E provides tables related to
these analyses (Tables 21-22).
The Project Data System (PDS) is a part of STARS that contains information on funded and implemented projects,
and when looking at EPA-funded projects in PDS, we were able to find slightly more and different information than
when looking at the SDS list information. In PDS, of all homes in communities that received CWISA funding
between 2003 and 2009, only 16 percent were DL-4 or DL-5 homes, even though a majority of projects (56
percent) were addressing DL-4 and DL-5 needs. This finding suggests that many CWISA projects address the needs
of a small number of homes with high access needs. These analyses and related tables from PDS information are
available in Appendix E (Tables 23-24).
We note that, until FY2008, the CWISA annual funding memos to the EPA regions directed the regions to consider
homes lacking access to basic sanitation as those with DLs of 3, 4, or 5, rather than just DLs of 4 and 5. EPA
headquarters changed this direction in FY2008 to be consistent with the FY2006-2011 EPA strategic measure on
access to basic sanitation. Though regions are expected to fund in order down the SDS lists for the CWISA
program, rather than by DL designation, it is possible that this change in how homes are defined as lacking access
has changed project selection in terms of the deficiency levels of projects selected. However, with only two years
of data since EPA changed the definition of "access" to exclude DL-3s, this evaluation cannot definitively say that
there has been a change in project selection related to access due to this change.
Regional Communication with IMS, Tribes, and Other Parties
EPA regional program staff coordinates with IMS and tribes in several respects, to promote awareness of available
funding, to manage project selection, and to track project implementation and progress. The evaluators explored
how communication and coordination are working, and whether improvements in these areas would be helpful to
better meet the needs of the tribes.
Communication with IMS Area Offices
Based on the interviews, communication between the EPA regions and IMS area offices on the whole appears to be
functioning fairly well, with only a modest number of issues raised by interviewees in both agencies. None of the
EPA regional coordinators from either infrastructure program mentioned major issues associated with working
with their local IMS area offices when selecting projects or getting updates on projects funded. However, during
both the interviews and initial evaluation scoping calls, some regional coordinators indicated that IMS is
understaffed given the level of work underway and ongoing need in Indian country. Regional coordinators
indicated that this understaffing can contribute to some delays, including lag time in closing out projects. During a
few interviews, staff in both agencies indicated that their working relationships were better with some EPA
regions/IMS area offices than others, as is typical of most working relationships between different organizations
and their respective offices. For the DWIG-TSA program, EPA had not been consistently sharing data on SDWA
compliance with IMS for inclusion in the SDS list, but as discussed earlier, EPA DWIG-TSA regional coordinators
made a commitment in December 2010 to communicate this information going forward.
These analyses examine projects that could be linked to the IMS PDS list, which is approximately 85% of the unique EPA, non-ARRA, projects
funded between 2003 and 2009 (343 out of 402).
Evaluation Report - March 2011 Page 23
-------
EVALUATION OF Til IE AND CLEAN INFRASTRUCTURE TRIBAL SET-ASIDE
Communication with Tribes
For both programs, the level and type of communication with tribes varies across the regions, but is generally
more limited for the CWISA program.
For the DWIG-TSA program, interviews indicate that some regions with smaller tribal populations, such as Region
2, often rely on IMS to communicate more of the information about the program, while other regions often work
directly with tribes on a regular basis to communicate about this program. Region 4, for example, has a rotating
process with the tribes, which necessitates regular communication with its six eligible tribes. Some regions, such
as Region 6, communicate with the tribes through notifications of funding or RFPs. In some cases, regions also
communicate information about the program and available funding through other tribal work or meetings within
the regions (e.g., regional meetings or announcements about all EPA grant funding); regional coordinators often
have other communication avenues with tribes beyond these infrastructure grant programs.
Beyond the DWIG-TSA program, EPA's Public Water System Supervision (PWSS) grant program has funded "circuit
rider" positions where staff work for IMS and provide direct drinking water technical assistance to tribal systems.
As of FY2010, EPA had funded ten FTEs at IMS area offices, across six EPA regions to provide this support. One
regional coordinator and an IMS area staff member noted that these positions are helpful for understanding tribal
drinking water needs and compliance issues, and for providing technical support to tribal communities that need it
the most.
For the CWISA program, communication with the tribes is generally limited to discussions on the two available
funding mechanisms, typically after projects have been selected. This difference may be due to the nature of the
project selection; as CWISA projects are selected directly from the SDS list, tribes must work with IMS to get needs
identified and listed on the SDS list, rather than discussing project ideas with EPA regions as they might through
the DWIG-TSA program.
Interviews from across all the interview groups for this evaluation indicated that direct, in-person engagement
between tribes and EPA regions (as well as EPA headquarters) is important for understanding tribal context and
needs, as well as progress on projects underway and the benefits of completed projects. One tribal interviewee
noted that a recent field visit by the EPA Administrator and EPA staff to a remote area showed a positive level of
engagement that they had not seen in other federal agencies. An IMS area staff member said that EPA staff should
be able to see the projects that they are funding to better understand progress and needs. In regions with smaller
tribal populations or where the regional coordinator works on other tribal programs, the coordinators may be able
to regularly visit project sites. However, with limited funding available for travel in most years, regions with larger
tribal populations or those with remote project sites may not have this opportunity.
Communication with Other Parties
Overall, the regional staff have limited or no interaction with agencies or parties other than IMS and the tribes
when supporting these programs. In some cases, USDA or HUD may also be contributing funding to a project, but
any needed coordination happens with IMS, rather than with EPA. As discussed, the Tribal Water infrastructure
Task Force is composed of four federal partners (EPA, IMS, USDA, and HUD) and some strategic coordination takes
place within this group.38 We do not have sufficient information upon which to base any particular findings on this
topic.
! For more information, see: http://www.epa.gov/indian/trprograms/infra-water.htm.
Evaluation Report - March 2011 Page 24
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Role of IMS in Program Implementation
EPA relies heavily on IMS for program implementation. EPA uses STARS data to select CWISA projects and some
DWIG-TSA projects and to report on EPA's strategic access measures. In addition, much of the programs' funds
have been provided to IMS through lAs to administer projects on behalf of the tribes. Implementation of both
infrastructure programs is largely in IHS's realm, given that over 86 percent of DWIG-TSA projects and 95 percent
of CWISA projects funded between 2003 and 2009 were implemented under lAs with IMS. In this section, we
explore IHS's role and how it influences EPA's own program implementation and influence.
IMS STARS Database
As already noted, EPA uses the IMS STARS database to select CWISA projects, select DWIG-TSA projects in some
EPA regions, track performance on the EPA strategic measures related to access, and track progress for projects
funded through lAs. All of these functions are vital for the implementation of these programs.
STARS is considered by the Infrastructure Task Force and other stakeholders to be the best source of information
on tribal needs related to access to drinking water and sanitation.39 However, interviewees who participated in
this evaluation and the Infrastructure Task Force have noted issues with STARS, summarized below, in terms of its
use for EPA implementation of the DWIG-TSA and CWISA programs.
STARS does not include all tribal homes or needs. STARS does not include all tribal water infrastructure
needs in the country; it includes needs that have been identified by tribal or IMS staff and entered into
STARS. Some tribes choose not to work with IMS to have their infrastructure needs on the list or they may
not know how or have the capacity to do so; the extent to which this situation occurs is not clear to the
evaluators.
STARS does not track SDWA compliance history.
STARS internal data linkages are limited. There is no automatic link in STARS between information on a
community's deficiency profile (CDP) and projects proposed and/or funded. Instead, IMS engineers are
expected to keep the community deficiency profile up-to-date based on a holistic community-level view of
identified needs and funded projects. While there may be advantages and disadvantages for this approach
to data tracking in STARS, the lack of an automatic link may mean that information about a community's
deficiency designation is neither current nor a good indicator of project accomplishments. For EPA, the
reported progress on its previous FY2006-2011 EPA Strategic Plan access "strategic targets" was based on
data from the CDP system, rather than from the Project Data System (PDS). This evaluation has shown that
progress made through EPA-funded projects was not strongly related to changes in the community-level
deficiencies (discussed further in the Program Goals and Measures section, and Appendix D, Correlation 6).
IMS has different priorities than EPA. STARS was developed for IHS's activities, mission, and scope.
Therefore, some projects and homes within STARS are not within EPA's purview to fund or do not reflect
EPA's priorities (e.g., individual wells, sanitation plumbing). This difference can skew the allocation of funds
under both programs, as these ineligible projects are included when calculating total need nationally and in
the IMS areas and EPA regions.
Federal Infrastructure Task Force Access Subgroup. Meeting the Access Goal: Strategies for Increasing Access to Safe Drinking Water and
Wastewater Treatment to American Indian and Alaska Native Homes. (March 2008)
Evaluation Report - March 2011 Page 25
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
STARS does not uniquely identify homes. Individual homes are not uniquely identified in the system for
projects, which can contribute to double counting for reporting on measures when homes are associated
with multiple projects.
Interviews with IMS staff indicate that IMS is in the process of updating STARS to address some of the above issues.
In particular, the system will uniquely identify tribal homes, which will aim to address double counting and linkages
between a community's deficiency profile and projects in the same community.
IMS Project Implementation
For the projects implemented through lAs with EPA, IMS often contributes critical design, planning, and
construction management services to tribes, and in doing so, helps tribes to receive needed infrastructure in a
timely manner. These and other implementation responsibilities require a substantial amount of IMS staff time;
yet staff resources at IMS appear to be limited for the work underway. In 2006, IMS estimated that the current
funding for program staff is less than 40 percent of what is needed to meet the US access goal (i.e., at least $5.7
million annually for staffing).40 (Funding increased in FY2010 and under ARRA for EPA's programs, but not enough
to bridge this gap over time to meet all access needs or to meet all IMS staffing needs.) Interviewees from EPA
regions, EPA headquarters, IMS, and tribal governments all noted that IMS is understaffed for the current amount
of work underway. With the amount of work that there is yet to do in Indian country, this capacity issue at IMS is
not likely to improve without other changes such as additional staffing and funding for the current as well as
increasing responsibilities.
For EPA-funded projects, some regional coordinators indicated that understaffing at IMS can manifest in longer lag
times for closing out projects. An analysis of the universe of EPA projects (i.e., including projects funded before
2003 for which we had data) found that, according to the project milestones in STARS, the final project report had
been completed in only one-quarter of CWISA construction-completed projects and just over half of DWIG-TSA
construction-completed projects (see Table 7). However, IMS provides a one-year warranty from the time
construction is complete for projects to which IMS provides engineering services. Given that warranty expenses
can affect financial information for this project, the IMS cannot complete the final close-out report for these
projects at least until after the warranty expires. Not having the final report for a project prevents the associated
IA between IMS and EPA from closing, but does not affect the provision of infrastructure to the tribal recipient.
Table 7 also provides project duration information for all EPA projects with available project milestone information
from IMS, excluding ARRA projects. This analysis includes information on projects funded by EPA before 2003, as
well as during the evaluation time scope, to reduce any skewing of the results due to more-recently funded
projects that were implemented quickly. Within the IMS areas and the EPA regions, there is a wide variation of
average duration times for projects in the two different programs. Level of IMS staffing, different water
infrastructure needs, type of facility construction, and length of construction season are all significant factors that
could account for these differences. For example, in Alaska, the construction season is very short in comparison to
other areas, which may force some projects to stretch over more construction seasons. However, a correlation
analyses for this evaluation found that, generally, for both programs, there is no correlation between the duration
of projects being constructed and either IMS areas or EPA regions. More detailed information, including these
correlations, and average, median, minimum, and maximum durations by EPA region and IMS area, is provided in
Appendix D (Correlations 1-4) and Appendix E (Tables 25-30).
J Ibid.
Evaluation Report - March 2011 Page 26
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
Table 7. Project Duration Information, All EPA Project Information, excluding ARRA Projects
Duration Information DWIG-TSA CWISA
Number of Projects
Projects with Construction Completed
Projects with Final Report Completed (Percentage of Completed Projects)
Average Time to Complete Construction41
Median Time to Complete Construction
Average Time to Complete Final Report after Construction Complete42
Median Time to Complete Final Report after Construction Complete
403
210(52.1%)
109 (51.9%)
3. 51 years
3. 29 years
1.69 years
1.53 years
457
188 (41.1%)
48 (25.5%)
3. 22 years
2. 86 years
2. 14 years
1.69 years
Note: this table includes project duration information for all unique EPA projects where data was available and that had milestone information
in STARS: 403 of 462 DWIG-TSA projects (87 percent) and 457 of 554 CWISA projects (83 percent). Project data was available as far back as
1993 for some CWISA projects and 1997 for DWIG-TSA projects.
Project duration is an important factor of project implementation and progress for both EPA and IMS. Having
projects completed in a timely way ensures that homes receive needed water infrastructure as soon as possible. In
addition, any possible reductions in the length of time a project takes to complete yields cost savings in both
construction costs and project staffing resources. One of IHS's strategic measures is to have projects constructed
in an average of four years or less and IMS has programmatic strategies to help achieve this target.43'44 Based on
the data available for this evaluation, EPA projects are, on average, within this IMS target.
ARRA Implementation
The American Recovery and Reinvestment Act of 2009 (ARRA) provided the CWISA and DWIG-TSA Programs with
significant additional one-time funding ($60 million for CWISA and $30 million for DWIG-TSA). Guidance
associated with ARRA funding directed that funds be managed in a manner similar to routine program
implementation, with three major changes.
> Rather than enter into separate lAs with IMS for each project, EPA administered one national IA with IMS for
CWISA and one IA for DWIG-TSA. In addition, the lAs were administered by EPA headquarters instead of the
regions. (Tribes still had the choice of receiving a direct grant instead of funds through IMS, but EPA regions
were to be the administrator of the direct grants if tribes selected that option; no tribes selected this
option.)
> To the extent that there were eligible projects, 20 percent of each allocation was to be provided to projects
that, "address green infrastructure, water or energy efficiency improvements or other environmentally
innovative activities."
> Regions were asked to work with tribes and IMS to identify projects that could be under contract or
construction within twelve months.
45, 46
41 From when an MOA is signed with the tribe to when construction is complete.
42 From when construction is complete to when the final report is complete.
43 Indian Health Services. Strategic Plan 2006-2011. Available:
http://www.ihs.gov/PlanningEvaluation/documents/IHSStrategicPlan20062011.pdf.
44 Indian Health Services. FY2011 Online Performance Appendix. Available:
http:/
jix.pdf.
Bergman, Ron, US EPA. Memorandum to Regional Drinking Water Branch Chiefs. "Drinking Water Infrastructure Grants-Tribal Set Aside
(DWIG-TSA) Program Guidance for projects funded using the American Recovery and Reinvestment Act of 2009."
46 Frace, Sheila, US EPA. Memorandum to Regional Water Division Directors. "Clean Water Indian Set-Aside (CWISA) Program Guidance for
projects funded using the American Recovery and Reinvestment Act of 2009." (July 9, 2009).
Evaluation Report-March 2011
Page 27
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
For ARRA projects oversight, in EPA headquarters and the regions have held monthly calls and two in-person
meetings. As a part of this process, EPA and IMS also worked together to develop standard IA terms and conditions
that all regions must use going forward for routine annual implementation.
As of December 2010, all ARRA funds had been assigned to projects, with CWISA funding 96 projects and DWIG-
TSA funding 64.47
Table 8 provides a breakdown of number of ARRA projects, level of funding, and homes served by EPA regions.
Table 8. ARRA Projects Summary
10
Total
$668,800
$698,000
$1,100,000
$1,692,700
$2,865,600
$765,700
$3,139,600
$8,448,500
$10,620,900
$30,000,000
13
16
64
2,051
1,311
2,163
387
1,480
858
2,916
5,604
5,588
22,358
$346,670
$1,343,330
$1,700,000
$1,590,010
$4,430,760
$7,160
$6,417,660
$22,300,460
$21,863,950
$60,000,000
46
23
96
50
228
1,170
59
500
125
1,583
6,178
2,159
12,052
Note: The "homes served" value likely double counts some homes as sometribes received multiple grants from one or both programs.
Appendix E (Tables 31-37) provides funding and initial deficiency level information of ARRA projects and compares
the information to other projects within the evaluation's universe (2003-2009). On average, projects funded under
ARRA had significantly higher average funding levels than projects funded under routine program implementation.
In terms of the DLs of ARRA-funded projects, DWIG-TSA projects funded, on average, slightly higher DL needs with
ARRA projects than under routine implementation; for CWISA projects, the ARRA-funded DLs were not significantly
different than under routine implementation.
Interviewee Perspectives on ARRA Implementation
Interviewees identified the following ways in which the differences in ARRA implementation have changed or
informed more routine implementation for both programs.
Coordination within EPA: Overall, interviewees were positive about the added communication and coordination
within EPA, and between EPA and IMS, as a result of ARRA. They anticipated that this improved communication
will continue with future routine program implementation. One regional coordinator said, "I think the added
interest that headquarters has taken [as a result of ARRA] will hopefully lead to better policies and processes with
the various EPA regions and IMS area offices." In December 2010, the EPA headquarters and regional coordinators
and IMS staff met to discuss these programs, and feedback from participants in this meeting were generally
positive, with a number of comments asking for the in-person meetings to continue after ARRA projects are
During the course of the evaluation, an additional DWIG-TSA project was funded with ARRA funding in Region 6. This project is included in
Table 8, but is not included in analysesforthis evaluation or in thetables in Appendix E.
Evaluation Report-March 2011
Page 28
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
completed. However, in phone interviews, when asked about communication under the ARRA processes, two
regional coordinators cautioned against having meetings for the sake of meetings and to encourage discussion
rather than simply updates.
National IA: Interviewees held a variety of perspectives on whether a national IA, such as was done for each
program under ARRA, would be appropriate for routine program implementation. All who discussed this idea did
note that the different timing of project selections within the regions could be a problem for a national IA, as
funding decisions would need to be more coordinated and on the same timeline across all the regions.
Coordination with IMS: As discussed earlier in this report, the ARRA process has also spurred more coordination
between EPA's and IHS's headquarters staff, such as filling funding data gaps and developing the standard terms
and conditions for lAs. Many interviewees, from both IMS and EPA, indicated that the development of standard
terms and conditions for IMS and EPA lAs was useful and that EPA regions will be using these in routine program
implementation. These standard terms and conditions may help alleviate some questions about IA expectations
noted earlier in the report.
Coordination with Tribes: Under ARRA, more travel money was available to headquarters and the regions to
conduct site visits. Many interviewees from across the interview groups indicated that travel by EPA to visit
project sites was useful to understand needs and track project progress, and that having these funds under routine
program implementation would be useful.
Green Projects: No interviews noted the "green" project requirement under ARRA, so it is not clear to the
evaluators whether this requirement caused any differences in project selection.
Evaluation Report - March 2011 Page 29
-------
OF THE AND
FINDINGS: PROGRAM GOALS AND MEASURES
Core Evaluation Questions II-IV:
To what extent are the DWIG-TSA and CWISA programs achieving their stated goals?
What factors influence whether the programs achieve their stated goals?
Are the current DWIG-TSA and CWISA program performance measures accurate
indicators of EPA's progress?
The evaluators examined available information to understand how well the two programs are meeting their overall
program goals as articulated through the strategic measures, what factors could be influencing progress toward
the strategic measures, and whether the strategic measures are good indicators of progress for these programs.
The strategic measures for these programs support the higher-level EPA strategic goal, "Protecting America's
Waters", in the FY2011-2015 EPA Strategic Plan.
The Compliance Strategic Measure: Progress and Factors Influencing Progress
EPA uses the compliance strategic measure (SP-3) to communicate the impact of the DWIG-TSA program, as well
as the general SDWA program implementation in Indian country. The compliance strategic measure is one
measure that gauges progress toward EPA's long-term drinking water compliance goal of 100 percent compliance
with SDWA as well as EPA's Strategic Goal 2, "Protecting America's Waters." This section discusses reported
performance on the compliance strategic measure; factors influencing progress on the compliance strategic
measure (and thus on the long-term drinking water compliance goal and EPA's strategic Goal 2); and whether this
performance measure is an accurate indicator of program progress.
Reported Performance on Compliance Measure
EPA's current target for its compliance strategic measure is to have 88 percent of the population served by tribal
community water systems in compliance with all health-based rules by 2015. (In the EPA FY2006-2011 Strategic
Plan, the strategic target was for 86 percent of the tribal population in compliance by 2011.) Since FY2003,
reported progress on this strategic measure has been mixed, with the most recently reported (FY2009) data
showing the lowest percentage of population in compliance over the previous seven years (Table 9). Note that this
measure includes water systems for which EPA has SDWA direct implementation responsibility. The measure,
however, excludes Alaska native systems, which are administered by the State of Alaska. It also excludes non-
community water systems.
Evaluation Report - March 2011 Page 30
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
Table 9. EPA Reported Performance on Tribal CWS Compliance
SP-3
, „ Total Systems out of , _ . . Population Served by _ ,
Fiscal Year ,. Total Population . ,. Reported
Systems Compliance Systems out of Compliance
2003
2004
2005
2006
2007
2008
2009
750
739
752
747
736
731
733
81
76
84
68
96
111
117
684,649
792,481
737,149
755,432
761,623
834,970
861,695
99,658
77,459
101,321
101,296
95,713
142,721
162,133
85.4%
90.2%
86.3%
86.6%
87.4%
82.9%
81.2%
The evaluation looked specifically at water systems that had EPA-funded DWIG-TSA projects to see if there is a
difference in performance between the nationally reported data for all tribal CWSs and the systems that were the
recipients of EPA DWIG-TSA funding.48
Table 10 shows the SP-3 results for EPA-funded systems for calendar year (CY) 2008, the last year for which this
evaluation had detailed violation data. The evaluators compared the SP-3 results for EPA-funded systems against
the overall SP-3 results for FY2009, which, given the reporting window for this strategic measure, is the most
accurate comparison for the SDWA data the evaluators have. The analysis showed that compliance at systems
with EPA funding was lower than the overall universe of tribal CWSs, which is not surprising, given earlier
information that more systems funded by EPA had health-based violations than non-funded CWSs. In addition, the
compliance rate for systems after EPA-funded projects have been completed is not markedly improved over the
average compliance rate for all CWSs, suggesting a marginal or minimal influence on compliance as a result of the
new infrastructure investments. When Alaska systems are removed from the results, as it is for EPA reporting
purposes on SP-3, the compliance rate for systems with completed projects is slightly lower than the overall
universe of tribal CWSs. This information is, however, a 'snapshot' in time that does not show trends in
compliance, and, importantly, does not explain the drivers of compliance problems, which may not be related to
infrastructure or be solvable with funding from these programs.
Table 10. SP-3 Results for EPA-Funded Systems
EPA Funded EPA Funded SP-3 Result for EPA-
EPA Funded Systems Systems: Systems: Population funded systems
All Systems
All Systems, excluding Alaska
Systems with Completed
Projects49
Systems with Completed
Projects, excluding Alaska
214,686
196,899
44,566
40,002
164,093
152,297
34,701
30,683
76.4%
77.3%
77.9%
76.7%
SP-3 (2009) Result
for Tribal CWSs
81.2%
The nationally reported results for SP-3 for 2009 are calculated on four quarters of data, spanning July 2008 to June 2009. To more accurately
compare to this information with available detailed SDWIS data, this evaluation looked at the compliance information across the four quarters
of calendar year 2008.
EPA funding has been provided primarily to CWSs, but funding has been provided to 12 non-profit non-CWSs (e.g., systems that serve schools
or other non-permanent populations).
49 Completed as of the end of CY2008.
Evaluation Report- March 2011
Page 31
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
This evaluation also examined whether there was any impact on compliance after a project was completed, rather
than just for systems that had received funding but had not necessarily had projects completed. From the SDWIS
information available for this evaluation, it appears that, on the whole, EPA-funded systems had fewer violations
after projects were completed when compared to their pre-project compliance.50
34 of 92 completed drinking water projects (36.6 percent) were at systems that had health-based violations
prior to when construction was completed. On average, these systems had 0.11 health-based violations per
year and 1.08 monitoring and reporting violations per year prior to when construction was completed.
15 of 92 completed drinking water projects (16.1 percent) were at systems that had health-based violations
since construction has been completed. On average, these systems had 0.03 health-based violations per
year and 0.47 monitoring and reporting violations per year after construction was completed.
By comparison, 16.1 percent of non-funded systems had health-based violations in 2008. These systems had an
average of 0.39 health-based violations for that year and 0.16 monitoring and reporting violations for that year.
(The evaluation used a single year to evaluate the number of health-based violations for the non-funded systems,
as there is not a similar "before" and "after" universe for these systems to use as a comparison group with funded
systems.)
However, given the data available and the likely influence of other factors on system compliance (see discussion
below), this evaluation cannot conclude definitively that DWIG-TSA infrastructure projects have directly
contributed to improved compliance at funded systems. Even with the modest pre- and post-project compliance
rates described above, different scenarios could be at play. For example, it is possible that compliance rates are
typically better just after a new project is completed because new staff have just been trained and are more
actively engaged in ensuring that compliance is supported.
Factors Influencing Progress on Achieving Compliance
Four factors suggested by interviewees and available data may influence the extent to which EPA has been able to
improve in tribal system compliance with health-based standards through the DWIG-TSA program: operations and
maintenance (O&M) at tribal systems, compliance with new drinking water rules, the selection of DWIG-TSA
projects that may address existing compliance issues at a system, and the duration of projects to have an impact
on compliance. Other factors may also influence progress on the compliance strategic measure, but not enough
information was available to this evaluation to identify them.
O&M at Tribal Systems
Adequate O&M of tribal systems is essential to both ensure that the infrastructure funded by EPA is maintained
over its projected design life, and to maintain SDWA compliance. Typically, O&M at tribal water systems has been
poor, resulting in failures in infrastructure and public health protection. As part of an effort to address this issue,
the Infrastructure Task Force Access Subgroup noted that, in areas with low population densities or remote
geography, the technical, managerial, and financial capacity necessary for adequate system O&M can be difficult
to develop and maintain. However, the Subgroup did also note that there is no data available to quantitatively
understand the relationship between the quality of management at a tribal water system and the impact on
As discussed previously, the SDWIS data that provides detailed violation data is only available for 2004 to 2008 and does not include ongoing
violations that began before 2004. In addition, the data does not tell us whether violations were infrastructure-related or O&M related.
Evaluation Report - March 2011 Page 32
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
infrastructure resources.51 The Subgroup recommended a number of actions for the involved federal agencies,
such as better coordinating technical assistance programs to build workforce capacity. EPA and other federal
agencies have done some work to begin addressing these recommendations through the formation of a Tribal
Technical Assistance Workgroup.
Interviews with most regional coordinators, as well with a tribal government representative, echoed that adequate
capacity is essential for maintaining water system infrastructure, and that there is a general lack of tribal and
federal funding available to address capacity and O&M issues at tribal water systems. Statutorily, EPA is prohibited
from directly funding O&M through the DWIG-TSA program. A few regional coordinators noted that many tribal
compliance problems are less related to infrastructure than to inadequate O&M at tribal systems and thus
infrastructure funding would not itself improve compliance.
The regional coordinator in Alaska examined the significant non-complier (SNC) list (systems considered by EPA to
be significantly out of compliance) in that state to determine the extent to which compliance issues could be
addressed with DWIG-TSA funding. As of May 2010, in Alaska, 60 percent of the systems (43 out of 72) at Alaska
Native Villages on this SNC list had violations solely related to operations. Of the remaining systems on the SNC
list, seven had violations related to unfunded hardware needs, five did not have adequate capacity to qualify for
DWIG-TSA funding, and only the remaining two systems would qualify for DWIG-TSA funding. Therefore, only
three percent of the Alaska Native Village systems on the SNC list had infrastructure-related compliance problems
and were eligible for DWIG-TSA funding.53 This analysis covered only one state in one EPA region, and it
represented a 'snapshot' in time, but its results are potentially significant, calling into question the assumption
that DWIG-TSA funding could solve many SDWA compliance problems at tribal water systems. Similar analyses in
other EPA regions could provide invaluable information about where DWIG-TSA funding could support compliance
and where compliance of other systems could be better supported through other types of assistance, such as
training or enforcement actions.
No source of sustained federal funding currently exists for the O&M of water infrastructure on tribal lands. Given
this constraint, interviews suggested the following strategies to help address tribal O&M issues for EPA to consider
or to continue to support.
Appropriately-scaled infrastructure: Some interviewees discussed the importance of building simple water
infrastructure for smaller tribal systems or tribes with a history of compliance problems to enable easier
maintenance and sustained compliance. A few expressed concern that DWIG-TSA projects (and also CWISA
projects) have on occasion been "over built" (i.e., too technically complicated or large for the needs and
capacity of the tribes that will have to maintain them) and that this alone could lead to future compliance
violations.
Technical assistance: EPA and other federal agencies have technical assistance programs that include
training for tribal system operators. For example, as of FY2010, EPA has funded ten positions in IMS that
work directly with tribal systems in six EPA regions through the Public Water System Supervision (PWSS)
program. Interviewees familiar with these positions thought that this program is a very useful way to
Federal Infrastructure Task Force Access Subgroup. Meeting the Access Goal: Strategies for Increasing Access to Safe Drinking Water and
Wastewater Treatment to American Indian and Alaska Native Homes (March 2008).
52 Tribal Technical Assistance Workgroup, Federal Infrastructure Task Force on Tribal Access to Safe Drinking Water and Basic Sanitation.
Strategies for Improving Technical Assistance Delivery in American Indian and Alaska Native Communities (March 2010).
53 D. Wagner, US EPA Region 10. Personal Communication. December 23, 2010. Note: this analysis was performed to inform the region, but
was provided to this evaluation as additional information.
Evaluation Report - March 2011 Page 33
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
support tribal system compliance, and also to keep EPA and IMS in close communication about compliance
needs and issues. EPA also has a large number of on-line training programs, which interviewees noted were
useful for helping enhance tribal understanding of compliance issues before violations occur.
Usage fees: Five interviewees identified usage fees as a barrier to proper O&M: tribal systems were not
charging usage fees to customers, were not charging enough, or were not enforcing their collection. As a
result, even systems with adequate capacity were hampered from addressing O&M needs to due to lack of
funds. One regional coordinator said, "I don't necessarily advocate federal funding [for tribal O&M],
because the problem is so big, but maybe the tribes need to increase their user fees so that they have the
appropriate amount of funds. Right now, tribes are charging fees that are 10 years old."
Asset-Management Plans: One EPA regional coordinator recommended that DWIG-TSA projects include
funding for asset management plans at tribal systems to ensure that there is a strategy in place to manage
infrastructure adequately. Asset management can help reduce operating costs by enabling water systems
to anticipate and plan for both the operational and capital costs of its infrastructure. Expanded EPA support
for asset management planning could potentially be another form of technical assistance.
Compliance with New Drinking Water Rules
The evaluators sought to understand whether the advent of additional or revised drinking water rules result in
influence system compliance, including those systems that received infrastructure funded through the DWIG-TSA
program in the past. During the 2004-2008 timeframe for which we had data on tribal compliance with SDWA,
only one drinking water rule came into effect, the 2006 Arsenic Rule.54 EPA adopted a new standard for arsenic in
drinking water of ten parts per billion (ppb), replacing the old standard of 50 ppb. Systems were required to
comply with this new rule by January 2006. Two other rules were promulgated during the timeframe, but have not
been monitored yet by EPA for compliance.
Interviews with EPA regional coordinators, an IMS area staff member, and a tribal government staff member all
suggest that the 2006 Arsenic Rule increased tribal system non-compliance. The tribal staff member said that,
under the new Arsenic Rule, "All of the sudden, lots of systems were no longer compliant... Now, the dollar
amount of needed improvements is huge."
Data analysis supported the same conclusion. Fifteen of 113 (13.3 percent) systems that had EPA-funded projects
prior to the introduction of the Arsenic Rule in 2006 had a total of 68 arsenic maximum contaminant level (MCL)
violations between CY2006 and CY2008 (all of the systems with violations were CWSs). Of these systems, none had
health-based violations associated with arsenic in CY2004 or CY2005. Looking at the broader universe of tribal
CWSs, there were 24 arsenic MCL violations in FY2007, 75 in FY2008, and 92 in FY2009—there were no arsenic
MCL violations from FY1998 to FY2006 for any tribal CWSs. Though this is a small universe of violations for both
the EPA-funded systems and the overall universe of tribal CWSs, it does seem that the 2006 Arsenic Rule had an
impact on tribal system compliance, thereby affecting reported performance on the compliance measure.
54 Detailed SDWIS data was only available for 2004-2008 for this evaluation, as compared to the evaluation timeframe of 2003-2009.
55 (1) The Stage 2 Disinfectants and Disinfection By-products Rule: This rule addresses risks from microbial pathogens and
disinfectants/disinfection by-products, specifically total trihalomethanes (TTHM) and five haloacetic acids (HAAS). Compliance monitoring for
this rule begins between 2012 and 2016, so this rule likely did not have any impact on compliance rates during the evaluation time scope. (2)
The Long-Term 2 Enhanced Surface Water Treatment Rule: This rule addresses Cryptosporidium and other disease-causing microorganisms.
Monitoring related to this rule began to be required between 2006 and 2008, but systems had three years to comply with implementing
treatment techniques. As such, this rule likely did not have much impact on compliance rates during the evaluation time scope.
Evaluation Report - March 2011 Page 34
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Project Selection
As already discussed, the DWIG-TSA compliance strategic measure assumes that DWIG-TSA projects can and will
address and correct health-based compliance issues caused by infrastructure deficiencies. However, it appears
that some health-based compliance issues are related to O&M capacity at tribal systems and the introduction of
new drinking water rules, rather than infrastructure. In this section, we explore whether project selection and
project duration may also influence SDWA compliance.
Our review of regional project selection guidances and processes indicates that the EPA regions use tribal
compliance history or anticipated public health impacts of projects as one factor in determining whether to fund
projects. However, other factors such as economic feasibility of the project and the availability of funding may in
some cases play a more prominent role in determining project selection. These other factors limit the influence
selection has on addressing system non-compliance.
Project Duration
Given the typical length of time needed for project completion, the effect on compliance at a system is likely to
take years to be realized. DWIG-TSA projects have taken an average of 3.51 years to be completed, and there have
been some projects that have taken eight or nine years. Therefore, the yearly compliance data as measured by SP-
3 is not likely to reflect recent infrastructure investments. As discussed in the section on reported progress on the
compliance measure, systems with completed projects have fewer compliance violations after a construction is
completed than before the project is complete. This result supports the idea that we would not expect to see
changes in compliance at a system until after a project is completed if the violations are infrastructure-related.
Compliance Measure as Indicator of Progress
Given the findings described above, it appears as if factors outside the scope or control of the DWIG-TSA program
funding and implementation, such as system O&M, the introduction of new drinking water rules, project selection
and subsequent duration of the project, influence compliance at systems and thus the reported performance
indicated by the SP-3 compliance strategic measure.
Interviews and observations of the data additionally indicate that the SP-3 strategic measure is imperfect as an
indicator of DWIG-TSA progress as the measured universe is different than the universe of potential DWIG-TSA
influence. SP-3 measures the compliance of EPA-regulated tribal CWSs, rather than the universe served by EPA
projects, which has included non-CWSs and state-regulated systems.
Notably, systems in Alaska serving Alaska Native Villages eligible under the DWIG-TSA program received 25 percent
of DWIG-TSA funding between 2003 and 2009. SP-3 does not capture any of the compliance improvements that
may have been made as a result of these projects because the State of Alaska implements the SDWA Alaska Native
Village water systems. (Note that in this report we provide the data with and without Alaska for comparison
purposes.)
In addition, between 2003 and 2009, DWIG-TSA projects have provided funding to tribal systems that account for
approximately one quarter of the population served by CWSs.56 How*
compliance at systems where DWIG-TSA funding has not been provided.
approximately one quarter of the population served by CWSs.56 However, the SP-3 strategic measure includes
In 2008, 214,686 were served by EPA-funded PWSs out of an overall population of 834,970 (28.7 percent). Removing Alaska-funded PWSsfor
a more direct comparison to reported numbers, 196,899 were served (27.2 percent).
Evaluation Report - March 2011 Page 35
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
The Access Strategic Measures: Progress and Factors Influencing Progress
Reported Performance on the Previous Access Strategic Targets
As discussed in the Background section, in the EPA FY2006-2011 Strategic Plan, EPA listed two "strategic targets"
to indicate progress on providing access to water infrastructure in Indian country:
> By 2015, in coordination with other federal agencies, reduce by 50 percent the number of homes on tribal
lands lacking access to safe drinking water. (This target used a 2003 baseline of 38,637 homes lacking
access to safe drinking water out of an estimated 319,070 tribal homes.
,57
> By 2015, in coordination with other federal agencies, reduce by 50 percent the number of homes on tribal
lands lacking access to basic sanitation. (This target used a 2003 baseline of 26,777 homes lacking access to
basic sanitation out of an estimated 319,070 homes.)
58
According to reported numbers (Chart 1), the overall progress toward these targets was stagnant or trending
downward. In 2003, 12.1 percent of all identified tribal homes lacked access to safe drinking water, and in 2009,
12.1 percent of all homes still lacked access, indicating that no progress has been made on this measure. In terms
of clean water, the reported trend was the reverse of intended progress: In 2003, 7.9 percent of tribal homes
lacked access to basic sanitation and in 2009, 10.1 percent lacked access. These trends reflected the number and
percent of homes lacking access, rather than the number of homes provided access.
Chart 1. Number of Tribal Homes that Lack Access to Safe Drinking Water and Wastewater Services
50,000
45,000
40,000
35,000
30,000
o
25,000
ra
.a
20,000
15,000
10,000
5,000
The orange and red lines show the intended
targets for these measures, while the blue
and green show the actual results of homes
lacking access.
•CWTargetSP-15
•CWActualSP-15
•DW Target SP-5
•DW Actual SP-5
Source: EPA Office of Water. "Tribal Drinking Water and Clean Water Access Measure Modification" (January 2010). Based on Community
Deficiency Profile (CDP) data from IMS STARS.
The Annual Commitment System code for this target was SP-5.
8 The Annual Commitment System code for this target was SP-15.
Evaluation Report-March 2011
Page 36
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
The current (FY2011-2015) and former (FY2006-2011) access strategic measures59 that the DWIG-TSA and CWISA
programs work toward result from EPA's commitment to reduce the number of tribal homes lacking access to safe
drinking water and basic sanitation as part of the broader US access goal made at the 2002 World Summit on
Sustainable Development. Providing access, as defined by these measures, is moving a home from a DL-4 or -5
designation to a lower DL These measures also provide an indication of progress toward EPA's strategic Goal 2,
"Protecting America's waters." This section discusses reported performance on these strategic measures; factors
influencing progress toward achieving these measures (and thus the higher-level goals); and whether these
performance measures are accurate indicators of DWIG-TSA and CWISA program progress.
Over the same time period, if instead the data analyzed is the cumulative number of tribal homes provided access,
the number of homes that that received access to drinking water and wastewater facilities exceeded the stated
target of 50 percent of the 2003 baseline: as of 2009, 80,941 tribal homes had been provided access to drinking
water, as compared to the target of 19,319 homes, and 43,562 tribal homes had been provided access to basic
wastewater sanitation, as compared to the target of 13,289 homes (Chart 2).60 According to a paper prepared by
EPA OW staff on these strategic measures, these trends affecting reported numbers could be due to:
The addition of tribal homes to STARS (from 2003 to 2009, the number of homes in STARS increased from
319,070 to 360,000, causing the relative percentage to appear smaller due to an increasing denominator);
"Backsliding" of DL designations (e.g., from a DL-3 to a DL-4) due to aging infrastructure or due to water
systems falling out of compliance with drinking water rules; and
Renewed interest by tribes in federal funding and thus tribes working with IMS to get project needs listed in
STARS.61
As previously discussed, the FY2006-2011 access strategic targets were replaced during the course of this
evaluation by the strategic measures in the FY2011-2015 EPA Strategic Plan. EPA explained that the reason for this
replacement was, "the changing [number of homes] baseline distorted OW's reported performance in increasing
access."62 EPA OW kept the previous access strategic targets as "indicators" in the EPA OW FY2011 NPM
Guidance, meaning that EPA OW will continue to report on the number of tribal homes lacking access to safe
drinking water and basic sanitation in annual reports in future, but not against an annual or long-term target.
In addition to the issue of the increasing number of identified homes relative to the 2003 baseline, the past access
strategic measures relied on data reported in the STARS Community Deficiency Profile (CDP), which, as previously
noted, is not automatically linked to project-specific data in the STARS system. Interviews with some program
stakeholders indicate that the CDP is not necessarily current with all known needs and funded projects in a
community.
In the FY2006-2011 EPA Strategic Plan, these commitments were referred to as strategic targets.
60 US EPA Office of Water. "Tribal Drinking Water and Clean Water Access Measure Modification" (January 2010).
61 Ibid.
62 US EPA. FY2011 National Water Program Guidance, Appendix D: Explanation of Key Changes Summary (April 2010). Available:
http://www.epa.gov/ocfo/npmguidance/2011/owater/nwp program guidance appendix d 508.pdf.
Evaluation Report - March 2011 Page 37
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
Chart 2. Cumulative Number of Tribal Homes Provided Access to Safe Drinking Water and Wastewater
Infrastructure, 2003-2009
10,000
DWTarget: 19,319
CW Target: 13,389
Source: EPA Office of Water. SP-5 and SP-15 Tracking Spreadsheet (Excel) (January 2010). Based on Project Data System (PDS) data from IMS
STARS.
To understand whether EPA projects would have influenced the community-1 eve I DLs, and therefore changes in
access as it was previously calculated at the community level, the evaluators conducted a correlation analysis of
the relationship between changes in DLs in homes provided a service through EPA projects (at the PDS level) and
the changes in DLs at homes (at the CDP level). The analysis results showed no significant correlation, meaning
that a provision of access (in PDS) for a home served by an EPA project would not be expected to result in a change
in access designation for a home at the community level (in CDP). This result could either be because the effect on
the community is small, or because the update to the community designation of homes is not immediate.63 (See
Correlation 6 in Appendix D for more details.) In other words, EPA was reporting based on number of homes from
community/CDP data, but EPA-funded projects were not influencing the changes in the CDP data.
Future upgrades to STARS will likely help address this CDP-project level linkage issue, though there will still be
limits in the impacts of individual projects, as homes may be associated with multiple infrastructure needs. In
2010, EPA decided to use project level (PDS) information as the basis for tracking performance on the access
strategic measure. This improvement will alleviate the problem of lack of connectivity to the community-level
influences, but it does introduce a double counting issue, and also, as we describe in more detail below, the
assumptions behind the term "access" as it is used by EPA for performance measurement purposes are
problematic.
Forthis analysis, we looked only at EPA projects; the results of the correlation analysis may be different when looking at all projects and all
communities tracked by IMS in STARS.
Evaluation Report-March 2011
Page 38
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Reported Performance on the Current Access Measures
In the FY2011-2015 EPA Strategic Plan, EPA replaced the former access strategic targets with two new strategic
measures.
By 2015, in coordination with other federal agencies, provide safe drinking water access to 136,100
American Indian and Alaska Native homes.64
By 2015, in coordination with other federal agencies, provide basic sanitation access to 67,900 American
Indian and Alaska Native homes.65
The new strategic measures rely on a count of the number of homes provided access, rather than the percent and
number of homes lacking access. These measures are cumulative (i.e., in 2009, 80,900 homes have been provided
access to safe drinking water; 55,200 more should be provided access by 2015). These new strategic measures
address the issue of a "changing baseline," but some interviewees pointed out that the there is no longer context
(e.g., the proportion of the entire population) from which to gauge progress and know its relative impact.
According to the latest STARS data available for this evaluation (2009), the number of tribal homes lacking access
to safe drinking water was 43,437 homes, which was fewer than the remaining number of homes (55,200) that EPA
has set for itself and other federal agencies to provide safe drinking water access to by 2015. In other words, EPA's
2015 measure identified more homes to provide access to then there were homes that needed access as of 2009.
EPA may, however, be anticipating that more homes with high drinking water needs will be added to STARS (e.g.,
new SDWA rules causing systems to fall out of compliance, new tribal homes being added to STARS).
According to STARS data, there were 28,052 homes lacking access to basic sanitation in 2009; this number is
slightly higher than the cumulative target (19,990 homes) that EPA has set in the new strategic measure.
EPA has not yet reported performance on these measures, but, from discussions with EPA, we expect that EPA will
use data from the Project Data System (PDS) from STARS to report on this measure alleviating the problem with
the previously reported progress based on data from CDP in STARS.
Factors Influencing Achievement of Progress Toward Access
Information collected through the interviews and data analysis suggest that six factors influence the extent to
which EPA has been able to make progress on the access strategic measure: O&M at tribal systems, compliance
with new rules, available funding, EPA's scope of influence, data availability and quality, and project selection.
Other factors may also influence progress on the access strategic measures, but information with which to analyze
potential additional factors was not available.
O&M at Tribal Systems
As discussed in the section on factors influencing progress on the compliance measure, adequate O&M capacity is
vital for ensuring the infrastructure funded by EPA is functional and maintained over its projected life. The
Infrastructure Task Force Access Subgroup has noted that sufficient tribal capacity for O&M is crucial for ensuring
access to water and sanitation infrastructure, and that sub-optimal capacity is a factor in infrastructure becoming
unusable before its design life is over. The Subgroup also noted that, in areas with low population densities or
The Annual Commitment System code for this measure is SDW-18.
5 The Annual Commitment System code for this measure is WQ-24.
Evaluation Report - March 2011 Page 39
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
remote geography, sufficient capacity for O&M can be difficult to attract and sustain, and as discussed earlier in
this report,
addressing.6
this report, made recommendations related to O&M capacity that the Infrastructure Task Force has begun
66
Interviews with most EPA regional coordinators, as well with a tribal government representative, echoed that
adequate tribal capacity is necessary to maintain infrastructure, and that there is a general lack of tribal and
federal funding and capacity available to address O&M issues at tribal systems. If existing infrastructure degrades
due to inadequate O&M, homes previously provided access to drinking water or basic sanitation could no longer
have access, given the way access is currently defined.
The lack of O&M capacity is a daunting issue, and no source of sustained federal funding currently exists for the
operation and maintenance of water infrastructure on tribal lands. The earlier report section on factors
influencing the progress on the compliance strategic measure discusses a few possible strategies suggested by
interviewees for addressing tribal O&M issues, which are applicable to both drinking water and clean water
infrastructure maintenance.
Compliance with New Drinking Water and Clean Water Rules
Compliance with new drinking water rules influences performance on the drinking water access measure. Critical
health-based violations resulting from new rules can cause IMS staff to re-designate homes from a lower DL to a
DL-4, assuming IMS is aware of the compliance problem. Compliance with new rules associated with the Clean
Water Act (CWA) may affect whether tribal homes lack access to basic sanitation; however, the evaluation
received no data or information about this issue, and we therefore cannot say whether CWA compliance would
affect progress on access to basic sanitation.
Available Funding
The Infrastructure Task Force Access Subgroup report referenced previously stated that sufficient federal funds are
not available to meet the US access goal by 2015. This paper estimated that a 40 to 50 percent increase over
current project funding is needed to meet the US access goal by 2015, or approximately $50 million to $63 million
more federal dollars over current funding levels. Interviewees with EPA headquarters staff and some regional
coordinators emphasized that a more funding than the current level is needed to address all infrastructure needs
in Indian country. As one EPA headquarters staff person said, "If I look at the data from IMS and the needs out
there... it looks like we would need a significant boost in funding to address the total need." In addition, this need
for funding for access to water infrastructure does not account for the additional funding needed to maintain that
infrastructure, to upgrade infrastructure if EPA promulgates new rules that would require that, or to expand to
account for new population growth.
Scope of Influence
EPA's access strategic measures are explicit that achieving these measures will be done in coordination with other
federal partners. The DWIG-TSA and CWISA programs are limited in the amount of impact that they can have on
the reported progress of the programs' strategic measures. As discussed in the program background section of
this report, the DWIG-TSA and CWISA programs have limitations on which projects are eligible for funding. Despite
the limits on what these EPA infrastructure programs can fund, ineligible projects and associated tribal homes are
Federal Infrastructure Task Force Access Subgroup. Meeting the Access Goal: Strategies for Increasing Access to Safe Drinking Water and
Wastewater Treatment to American Indian and Alaska Native Homes (March 2008).
Evaluation Report - March 2011 Page 40
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
still included in the IMS STARS database and thus are included in reported numbers of homes provided or lacking
access. As EPA funds cannot be used for these projects, EPA cannot directly impact these homes.
In addition to not having the influence to fund all projects that would affect water infrastructure access at tribal
homes, EPA is not the direct implementer of these projects; tribes often work with IMS to at least administer the
funds for these grants and IMS staff often serve as the engineers. Thus, EPA cannot directly influence the time
projects take to complete and thus the provision of access to homes. As an IMS area staff person noted, "EPA is
put in a difficult situation being held accountable for work that IMS ends up doing."
Data Availability and Quality
For the current access strategic measures, EPA has addressed the changing baseline of homes issue, as well as the
issue of connecting project data to performance gauged at the community level.67 However, there are several
data-related issues associated with these strategic measures that will impact EPA's ability to fully affect both
reported and actual on-the-ground performance.
Double-counting of homes: Due to the way the PDS system is designed and how the access numbers are
calculated, double counting of homes provided access will likely occur if EPA or IMS fund multiple projects in
a particular community. Homes may also be double counted if funded through an infrastructure project and
provided "access" (by moving from a DL-4 or DL-5 to something lower), only later to return to a DL-4 or DL-5
and be funded again. As an IMS staff person said, "Looking at access as changes in DLs may be skewing
things, because from year to year, those numbers can change a lot and we don't necessarily know if it's new
homes or it's the same homes." As discussed, IMS is working on improving the ability to uniquely identify
individual homes, which will hopefully enable prevention of double counting in the future.
Deficiency level "backsliding": Related to the previous point, DLs can change from one year to the next.
EPA relies on the provision of access (as defined as a change in DLs) in reported numbers on the measures,
but these DL designations can shift following project completion.
Timing of Project Impact: According to the STARS manual, when a project is funded, the homes receiving
the service should immediately have their DL changed to the final DL (FDL) associated with the project.68
Given that most projects take years to complete, the reliance on a metric that requires immediate progress
in reducing deficiencies is misleading, yet EPA also relies on the same immediate provision of "access" when
reporting performance.
Direct Grant Inclusion: Between both programs, approximately eight percent of all EPA-funded projects
(and eight percent of EPA funding) between 2003 and 2009 were provided as direct grants to tribes.
Although this is a relatively small amount overall, some regions, including those with large populations, have
a significant portion of their funding directed to direct grants. For example, 60 percent of the DWIG-TSA
projects between 2003 and 2009 in Region 8 were direct grants to tribes. Direct grant projects have not
(and will not) contribute to the reported number of homes provided access unless the homes associated
with these projects are already being tracked by IMS due to their involvement in implementing the projects
or active communication by EPA or the tribe.
This statement assumes that the DLs of homes will be tracked using only PDS data, not CDP data. Our understanding from talking with EPA
staff is that only PDS data will be used in reporting on the new access measures.
68 Indian Health Services. STARS User Manual (September 2008).
Evaluation Report - March 2011 Page 41
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Project Selection
As discussed in the section on implementation, many of the projects selected by the CWISA and DWIG-TSA
programs are not addressing DL-4 and DL-5 needs or DL-4 and DL-5 homes. For the CWISA program, EPA regions
generally fund projects in order down the SDS list, where deficiency level is only one factor of eight of a project's
ranking score on the list. Similarly, for DWIG-TSA projects, regional coordinators consider several factors—not only
providing access—when deciding which projects to fund. However, for both programs, we also note that DL-2 and
DL-3 needs are often necessary to address for tribes to have quality water system infrastructure and outputs.69
Access Strategic Measures as Indicator of Program Progress
In addition to the factors discussed above that may influence progress that are outside the control or scope of
these programs' funding and implementation, the current access strategic measures are not specific to EPA's
activities. They rely on EPA's "coordination with other federal partners" and thus the actions and influence of
other federal agencies. This evaluation did not analyze the extent to which the other federal agencies involved
with the Infrastructure Task Force could affect progress on these measures, but a significant part of IHS's activities
are to work with tribes on developing drinking water and basic sanitation infrastructure. In addition, there are
HUD and USDA funds available to tribes that would affect infrastructure projects. To put it simply, the lack of an
EPA-specific strategic measure means that it is difficult to assess the impact of the EPA programs on tribal access to
safe drinking water and basic sanitation.
For comparison, IHS has four strategic measures related to both serving and providing access to tribal homes, with
three of these related directly to IHS-specific activities, rather than more general indicators of progress across
multiple federal agencies:70
Number of new or like-new tribal homes and existing homes provided with sanitation facilities;
Percentage of existing homes served by the program at DL-4 or above;
Average project duration; and
Percentage of tribal homes with sanitation facilities.
For example, IHS designates systems that do not meet national secondary drinking water standards as having DL-2 water needs, and lagoons
in need of repair as having DL-2 sewer needs. Source: Indian Health Services. Sanitation Deficiency System (SDS) Guide for Reporting Sanitation
Deficiencies for Indian Homes and Communities. May 2003. Available: http://www.ihs.gov/dsfc/documents/SDSWorkingDraft2003.pdf.
70 Indian Health Services. Strategic Plan 2006-2011. Available:
http://www.ihs.gov/PlanningEvaluation/documents/IHSStrategicPlan20062011.pdf.
Evaluation Report - March 2011 Page 42
-------
OF THE AND
CONCLUSIONS AND RECOMMENDATIONS
Conclusions
Based on the findings presented throughout this report, the evaluators have drawn the following summary
conclusions about the implementation of the DWIG-TSA and CWISA programs, the progress on achieving goals as
tracked by the programs' strategic measures, and the use of these performance measures as indicators of program
progress.
> The goals and priorities for these programs could be clearer and more focused. The goals and priorities for
both programs lack precision and clarity: EPA staff have different ideas of what the term "goal" means for
these programs (e.g., is it the US access goal? Or EPA Strategic Plan Goal 2? Is the drinking water program
more focused on a compliance goal or an access goal, or both?) and how the drinking water program priorities
of compliance and access relate (e.g., does access automatically support compliance and vice versa?). EPA has
the opportunity to more clearly articulate and focus the goals of both programs, and then to focus program
design and implementation accordingly in a set of "cascading" decisions and actions that would follow clearer
goals and priorities.
> Funding allocation for both programs could be improved to be more in line with EPA's priorities. The
method used to allocate funds for each program has an understandable basis and history; however, neither
allocation process is an ideal match for the current program priorities and strategic measures.
The DWIG-TSA program allocation of funds is not ideal for supporting the programs' current program
strategic measures and maximizing progress toward these measures. For example, if compliance with
SDWA is the highest priority for the drinking water program, funding could more directly be allocated
based on compliance needs. In addition, the use of the SDS list as a way to calculate overall drinking
water needs in a region means that ineligible projects and projects addressing lower deficiency needs
are contributing to the DWIG-TSA allocation. In addition, the two percent base allocation under
DWIG-TSA for each region is necessarily a strategic process for making progress toward the program's
strategic measures.
For the CWISA program allocation of funds, the SDS list is a valuable tool for EPA, but it does not align
with the CWISA program's ability to fund projects or with the strategic measure related to basic
sanitation.
We support the work that is already underway to allow the fund transfer between programs to more
effectively address the most pressing water infrastructure problems in a given year.
> For both programs, project selection could be more clearly and consistently tied to EPA's priorities, while
still maintaining regional discretion and flexibility.
Project selection under the DWIG-TSA program differs from region to region, and these processes
have developed due to differing regional circumstances. Though regional coordinators have the best
understanding of regional conditions, these varied project selection processes may mean differential
treatment of tribes across the EPA regions and that some regions are less consistent with helping
meet EPA headquarters program priorities than are others.
Evaluation Report - March 2011 Page 43
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
For the CWISA program, the IMS SDS list is an invaluable tool for the CWISA program in terms of
guiding project selection, but using the SDS list to fund in rank order is not ideal for CWISA's priorities
and ability to fund projects.
Communication and coordination between EPA headquarters and EPA regions had been inconsistent until
the recent ARRA funding. Based on interviews and our own observations, it appears that communication
between EPA regions and EPA headquarters is generally less than optimal for facilitating mutual
understanding of national and regional priorities and progress, program implementation, a unified EPA
approach to these programs, and strategic learning and improvement. The management processes used to
implement ARRA funding led to improved communication and coordination within EPA and it appears that
some of these processes will persist into routine program implementation. From interviews, it seems that a
single IA for all DWIG-TSA projects and one for all CWISA projects, as was done under ARRA, could have
advantages for coordination However, trying to coordinate simultaneous funding and closure for all projects
would likely be challenging if EPA pursued this funding mechanism for routine program implementation.
Despite improvement in recent years, communication between EPA and IMS is inconsistent and not optimal
for strategic program management or strategic learning and improvement. Though interviewees reported
that communication on the whole between the two agencies is currently working well, it has been
inconsistent in the past. The two agencies could enhance communication at the regional/area office and
headquarters levels to better understand each others' priorities and roles and the details of the new
standardized IA. For the DWIG-TSA program, increased communication around SDWA compliance problems
would be worthwhile, and it appears from the December 2010 meeting of DWIG-TSA staff that EPA is already
committed to doing this.
IMS is a vital partner for EPA to implement these programs. EPA relies on IMS to implement these programs
and help achieve improvements in access at tribal homes. EPA could not realistically take over IHS's
responsibilities: EPA does not have the in-house expertise, field presence, or relationships with tribes that
would be needed to do so. In addition, the IMS STARS database is an important tool for the grant program
implementation. As noted by the Infrastructure Task Force and interviewees for this evaluation, there are
issues with using the STARS database as a reporting mechanism and project selection tool for these EPA
infrastructure grant programs. However, this database is also generally considered by program staff and other
stakeholders to be the best source of information on tribal water infrastructure information and is thus an
invaluable tool for these EPA programs.
There are opportunities for improved communication with tribes about these programs. Based on the
interviews with tribal government staff, EPA, and IMS, the evaluators find that tribal communication about
these programs is inconsistent and "light" on the whole. EPA could strengthen its direct communication with
the tribes to enhance knowledge of tribal contexts and needs and more consistently provide tribes with
information about funding options (lAs versus direct grants). Increased communication could help EPA
understand program accomplishments and identify areas for program improvements.
EPA has limited ability to make progress on the current strategic measures focused on providing access to
drinking water and clean water to tribal homes. EPA's limitations include:
Lower program funding relative to the funding needed to provide for all identified access needs;
A history of funding a substantial proportion of projects that do not provide access (as defined by
moving from a higher deficiency level (DL) to a lower one) due to project selection processes;
Evaluation Report - March 2011 Page 44
-------
OF THE AND
Restrictions on EPA's authority to fund some projects that would provide access, such as by funding
individual drinking water wells or providing sanitation infrastructure services to homes that are not
on reservation land;
Reliance on a metric which assumes changes in DLs (from a DL-4 or -5 to a lower DL) that are
immediate and permanent, when in reality, DLs may not change in this manner as a result of funding
(e.g., because a DL-4 project only partially funds the infrastructure needed to change the associated
homes to a DL-3 or lower) and there is sometimes "backsliding" of DL designations due to compliance
violations, lack of infrastructure maintenance, or other reasons;
Substantial reliance on other federal agencies, particularly IMS, to contribute toward progress on the
access measure; and
Disconnects between reported progress on providing access (reported to be immediate upon project
funding) and actual progress on the ground, which typically occurs years later, after construction is
completed and the new or upgraded infrastructure is operating.
i EPA appears to be quite limited in its ability to make progress toward the current compliance strategic
measure, though the extent of the limitations is not clear due to a lack of measurable data. According to the
evaluation interviews, an analysis of compliance data from Region 10 (Alaska area), and the Infrastructure
Task Force Access Subgroup report, the drivers of some if not most compliance problems may be related to a
lack of sufficient O&M capacity at tribal systems or the introduction of new drinking water rules, rather than
to infrastructure problems that the DWIG-TSA program can address. In addition, the allocation of DWIG-TSA
funding has not been directly tied to compliance needs or opportunities, nor has project selection clearly and
consistently been linked to compliance support on a national scale. EPA has the opportunity to conduct
additional data analysis that will further explore the relationship between SDWA violations and the DWIG-TSA
program and then to focus project selection on projects that that will be more likely to support compliance
improvements over time.
i EPA's meaningful contributions to meeting tribal infrastructure needs are not well reflected by the reported
program performance. Despite the identified challenges surrounding making progress toward the current
strategic measures, both programs have made significant contributions over the years to Indian country
through these programs by providing funding for vital water infrastructure needs. These contributions—and
the public and environmental health benefits they have provided—have not been clearly recognized in a
public or formal way. As one EPA regional coordinator said, "I don't think we hear enough about what the
achievements are—we know what the goals are, but I don't know what the achievements have been."
Recommendations
Core Question ¥: What implementation improvements or innovations could be made by
EPA to make the DWIG-TSA and CWISA programs more effective in meeting the water and
wastewater infrastructure needs of tribes?
Based on the evaluation's findings and conclusions, we offer eight recommendations and three additional items for
consideration. We consider the first four recommendations to be policy recommendations as they would require
approval from senior EPA management and, in the case of at least recommendations 1-3, approval from the Office
of Management and Budget (OMB). Recommendations 5-8 are more program-oriented and therefore could be
considered and implemented by program staff. We present the recommendations in this order because we feel
that they are iterative considerations for the programs.
Evaluation Report - March 2011 Page 45
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
1. Clarify the goals and priorities of both programs.
To reduce confusion and increase program focus and management, we recommend that EPA identify clear,
specific, and measurable goals and priorities for both the DWIG-TSA and CWISA programs. We encourage EPA to
clarify, for example, whether providing "access" (or perhaps simply "serving homes that currently lack access"
which might be preferable—see Recommendation 2) is the highest level priority for both programs, and, if so, to
identify clear program-specific goals that correspond to this priority. A set of cascading changes to both programs'
processes (e.g., allocation of funds, project selection) and performance measures could then logically follow as
described in the following recommendations. For DWIG-TSA, if compliance with SDWA is the highest priority, then
making this clear would be helpful for program planning, implementation, and reporting. We are aware that EPA
has not consistently shared compliance information with IMS, so it is likely that many compliance-related issues
that would result in a lack of "access" as defined by IMS are not reflected in reported numbers. Ideally, clearer
program-specific priorities and goals would involve specific targets and performance measures that EPA can readily
track and report on.
2. Consider changing the access strategic measures by which EPA tracks its performance to measures that
more directly reflect EPA's mandate, authority, and scope of influence; take into consideration data
availability and quality; and reflect changes or clarifications to the programmatic goals made in response to
recommendation 1.
We offer several options for EPA's consideration regarding the current access performance measures, which we do
not think serve EPA or reflect progress toward meeting tribal needs as well as other options would. Of the options
we list below, most are variations on the current access strategic measures, but they address some or all of the
challenges described in this report. (One option is to simply leave the current measures unchanged.)
Note that, in this recommendation, we use the term "measure" (to be consistent with the usage of this term in the
FY2011-2015 EPA Strategic Plan) for what would generally be considered a 5-year "goal" or "target" against which
performance would be "measured" from year to year. Also, if EPA's highest priority for the DWIG-TSA program is
to support SDWA compliance, EPA may want to consider dropping the drinking water access strategic measure
altogether in support of a measure that is more clearly related to compliance for the program (see also
recommendation 3, below).
Table 11 below provides a list of options for the access-related strategic measures, including pros and cons for
each option. This list and associated pros/cons are not necessarily exhaustive; they are intended to illustrate
tradeoffs between alternatives.
Options 1-5 assume that the number of homes will be counted using the PDS data of the IMS STARS database; the
STARS database, while imperfect for EPA's uses, is currently the best available national data on the number of
tribal homes and water infrastructure needs. Options 6-7 assume use of the EPA Safe Drinking Water Information
System (SDWIS) by connecting EPA project data on the water systems served with tribal population data available
inSDWIS.
Evaluation Report - March 2011 Page 46
-------
EVALUATION OF THE DRINKING WATER AND CLEAN WATER INFRASTRUCTURE TRIBAL SET-ASIDE GRANT PROGRAMS
Table 11. Options Pertaining to Access Strategic Measures
Options -Both Programs Pros
1. By [YEAR] provide infrastructure services71 to [X]
tribal homes
2. By [YEAR] provide infrastructure services to [X%]
of tribal homes
3. By [YEAR] fund [X%] of projects [or could be a %
of funding need] with initial DL-4 or DL-573
4. By [YEAR] provide infrastructure services to [X%]
of homes with DL-4 or DL-5 designations
5. By 2015, in coordination with other federal
agencies, provide access75 to safe drinking water
for 136,100 AI/AN homes and basic sanitation to
67,900 AI/AN homes (Current measures)
Options - DWIG-TSA only
6. By [YEAR] provide infrastructure services to [X%]
of all tribal community water systems
7. By [YEAR] provide infrastructure services to [X%]
of the tribal population served by community
water systems
A, B, C, G
Note: One IMS measure is very
similar72
A, C, G
D, G
D, G
Note: One IMS measure is very
similar74
B, D, E, F
^^ifl^^^^^H
I,J, 0
I,J, 0
1, (M -depending on %
chosen), 0
I,K L, M, 0
1, J, K, L, M, 0
I^S
More concrete, comprehensive
measure of EPA's influence,
would provide a numerical
context (total tribal population
served by PWSs), would not rely
on external data source
Would require additional
data analysis by EPA,
would not reflect AI/AN
population not served by
public water systems
Pros
A. Relatively clean, clear measure of homes provided some type of infrastructure service each year with EPA
funding; recognizes and "gives credit" to all work done by EPA
B. Does not require a percent or other relational number that would be tied to a potentially-fluctuating
number of homes in the homes universe (STARS homes inventory), which, given its fluctuations, can make
EPA's performance look poor even if it is serving more than its original target number and percentage
(relative to the original universe) of homes
C. Does not assume that EPA is or should be exclusively funding DL-4/5 needs, or that all IDL-4/5 projects will
result in "access" as defined by moving to DL-3 or lower
D. Supports the US access goal
E. May be able to reduce burden on EPA for performance and accountability given that the measure is "in
conjunction with other federal agencies"
F. Has been approved by senior EPA managers and OMB
G. Is not dependent on other agencies' funding, project selection, or implementation (in some cases other
agencies may co-fund projects with EPA, but the measure could be tracked only in relation to EPA-funded
work)
H. Provides context through a percentage or other relational number intended to represent the larger
population
Cons
I. Homes will sometimes be double counted (extent unknown)
71 Infrastructure services (or a similar term) would coverall projects funded, rather than just projects serving homes with high deficiency levels,
and would include both significant upgrades to systems, as well as partial upgrades.
72 IMS measure (FY2006-2011 IMS strategic plan): Number of new or like-new AI/AN homes and existing homes provided with sanitation facilities
(Targets: FY2010: 21,811; FY2011: 21,500).
73 Some projects with IDLs of 4 or 5 also have FDLs of 4 or 5 due to other existing issues.
IMS measure: Percentage of existing homes served by the program at DL-4 or above (Targets: FY2010: 37%; FY2011: 37%)
75 "Access" is defined as a shift in deficiency levels (in the IMS STARS data system) from a 4 or 5 to a 1-3.
Evaluation Report-March 2011
Page 47
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
J. Would not have a numerical context (such as the total universe of homes) from which to gauge
relative/proportional performance
K. Assumes immediate "access" results even though projects take years to complete and therefore for the
access or benefit to be realized on the ground immediately after funding
L Does not account for factors, such as compliance violations that result overnight from the introduction of
new EPA rules or lack of adequate maintenance that could cause home to move from a DL-1-3 to a DL-4-5
and in so doing overlook the improvements EPA has invested in and achieved toward the measure
M. Does not reflect or give "credit" to the good work being done through projects that are serving DL-1-3
homes (a sizable proportion of projects have IDL designations of DL-2 or -3)
N. Puts pressure on EPA to fund a lot (or essentially all, depending on the numbers chosen) of DL-4/5 homes
regardless of whether EPA has the authority to (e.g., individual home wells; sanitation projects not on
reservation land), cost feasibility, funding levels, availability of access needs, etc.
O. Reporting on the measure from STARS does not necessarily include direct grants, and therefore does not
reflect all EPA work or influence.
3. Consider changing the DWIG-TSA compliance strategic measure to one that more directly reflects the
drivers of compliance problems and EPA's scope of influence over these problems.
A revised compliance measure targeted specifically to infrastructure-solvable compliance problems that EPA has
the ability to address would more directly and clearly reflect the influence of the DWIG-TSA program on
compliance. We offer three options for such a measure. All three options assume that DWIG-TSA funds (or some
portion of these funds) would be targeted to those public water systems (community water systems and non-profit
non-community water systems) that, based on additional data analysis conducted by EPA regions, would
reasonably be expected to see improved compliance as a result of receiving new infrastructure through this
program.
Option 1: By [X YEAR], achieve [x] reductions in health-based compliance violations at systems funded by the
DWIG-TSA program.
This option would likely require tracking over some years the compliance improvements that would be expected to
result after new infrastructure is completed and up and running. Therefore, it could be difficult to track direct
compliance improvements from a particular project in the year it was funded.
Option 2: By [X YEAR], target [X] percent of DWIG-TSA funds to those systems that have had compliance violations
that can be solved with new infrastructure funded through DWIG-TSA.
This relatively straight-forward option would aim to ensure that funding is focused (to an agreed upon extent) on
compliance improvements. One advantage of this option would be that it could be done annually—without
waiting for years for the compliance benefit to be realized. A disadvantage would be that it is more focused on the
activities of the DWIG-TSA program, rather than the expected or desired outcomes.
Option 3: By [X YEAR], [X] percent of the population in Indian country served by community water systems that
have received DWIG-TSA funding will receive drinking water that meets all applicable health-based drinking water
standards.
Option 3 is a variation on the current compliance strategic measure. It would limit EPA's commitment to DWIG-
TSA funded community water systems (rather than all tribal community water systems) to more directly
understand and track influence of the program. Like the other options above, this would also ideally involve
Evaluation Report - March 2011 Page 48
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
increased analysis of compliance drivers and funding targeted specifically to systems that have problems that
DWIG-TSA funding could help to solve or could help to prevent. (Without these additional data analysis and
targeted funding steps, EPA would continue to have a measure for compliance that risks being outside of the
DWIG-TSA program's sphere of influence.) Like option 1 above and the current compliance measure, compliance
improvements would take time (usually years) to be realized, creating a time lag between each year's DWIG-TSA
funded work and compliance outcomes. EPA would need to consider how and if to include compliance
information for non-profit non-community water systems that receive funding, as well as state-delegated water
systems that receive funding (e.g., systems serving Alaska Native Village).
4. Reassess the national annual budget allocation for both programs to be more clearly tied to EPA's goals and
priorities.
Once EPA has determined whether and how to update each program's goals, priorities, and measures, deciding
whether and how to update the allocation formulas for one or both programs will be easier; however, we
recommend updating the allocation formulas even if EPA makes no changes in response to recommendations 1-3.
CWISA Allocation
For the CWISA program, EPA current allocates resources based on sewer infrastructure needs identified in each IMS
area's SDS priority list. Per the CWISA national guidelines, EPA regions select projects in rank order down their
respective IMS area office SDS list(s), unless there are reasons to skip projects as described earlier in this report.
Allocating funding based on the SDS list does not align directly with DL designation (i.e., "access") or with EPA's
authority to fund (e.g., projects listed that are not within tribal jurisdiction; projects that would require plumbing
or water connections from a home). The CWISA allocation formula is thus allocating based on needs in the IMS
areas that do not necessarily support progress on the access strategic measure for basic sanitation.
A more direct way to support access (assuming access continues to be a priority and strategic measure) would be
to use the DL designation alone and divide funding proportionately according to those EPA regions or IMS areas
that have the highest proportion of DL-4/5 projects from the SDS list. If, however, EPA were to focus more on
homes served, rather than homes provided access, the current allocation method may be the best method, given
that it prioritizes the highest nation-wide sanitation needs. There may also be ways to factor in EPA priorities such
as decentralized infrastructure, though we are not clear from the work done for this evaluation how EPA could do
this.
DWIG-TSA Allocation
For the DWIG-TSA program, allocation is based on a combination of information from DWINS and the SDS list, after
each region is first allocated two percent of the annual funding available.
The current two percent base allocation means that all regions with federally recognized tribes receive funding,
even if there are no projects that would have significant public health impacts, otherwise improve compliance with
SDWA, or provide access to safe drinking water in a region. We recommend that EPA consider removing this base
allocation in favor of a formula that would be more needs- and results-focused based on (clarified) program goals
and priorities.
The DWINS results, once completed in the next few years, will be a helpful resource from which to better
understand tribal infrastructure needs and consider allocation.
Evaluation Report - March 2011 Page 49
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
In the meantime, if EPA considers provision of access to be the highest program priority, a more direct way to
support access would be to use the DL designations alone and divide proportionately according to those EPA
regions that have the highest proportion of DL-4/5 projects from the SDS list.
The current allocation method does not consider SDWA compliance history or issues. Assuming compliance with
SDWA is a priority, if not the highest priority, for the DWIG-TSA program, EPA could allocate funding to the regions
in proportion to health-based compliance violations in a given year. If this method were used, it would be
extremely important for EPA to have a deeper understanding of the drivers of the compliance problems, isolating
infrastructure-related violations that would be appropriately solved through DWIG-TSA funding versus through
other technical assistance or enforcement options. As EPA staff are also aware, focusing funding on systems with
infrastructure-related compliance problems may create a disincentive for systems to comply in the first place. EPA
could also choose to allocate based on compliance vulnerabilities (versus violation histories); however, this too
would require deeper analysis by EPA headquarters and regions to understand systems that are vulnerable to
infrastructure-related compliance issues. We recommend that, to the extent that EPA considers SDWA compliance
as a priority, EPA shift allocation to more directly support compliance.
5. For both programs, update and clarify expectations for project selection to be more clearly in line with the
program priorities, allow for regional flexibility and discretion, and promote increased consistency and
transparency.
The project selection processes for both programs have their bases and merits, but neither process is ideal: For
the DWIG-TSA program, consideration of both compliance needs and provision of access varies from region to
region; regions often choose projects based on other considerations, making it difficult for EPA to make progress
on its strategic measures. For CWISA, exclusive use of the SDS list is limiting and does not recognize EPA's distinct
mission, authority to fund, or priorities, including provision of access.
Updating the project selection guidances to focus on a simple set of core program elements while allowing for
regional flexibility and discretion would provide more coherency across the regions, particularly for the DWIG-TSA
program, and more direct linkage to program priorities for both programs. In the guidances, EPA could provide
guidelines on use of the SDS76 and other data sources.
EPA headquarters and regional staff could create a simple form that lists each program's priorities for funding.
Regions could use this form to show how the projects they have selected match with the identified priorities or
core program elements. (We are suggesting that the regions list only those projects they select for funding, not all
potential projects that are not funded.) This form could include basic questions or checklists such as the following:
The IMS SDS list is an important resource with no clear substitute, but given its limitations for EPA's purposes, SDS is not the ideal information
source from which to pick projects without considering other information sources. For example, the SDS list does not directly represent EPA
priorities such as SDWA compliance, infrastructure security, and decentralized wastewater infrastructure, and it includes tribal homes and
projects that are not eligible for EPA funding. Further, choosing projects in rank order from the SDS list is not likely to support strong
performance toward the access measure.
Evaluation Report - March 2011 Page 50
-------
OF THE AND
1. Which of EPA's priorities are addressed through this project (check all that apply)?
a. Priority 1
b. Priority 2
c. Priority 3
d. Etc.
(Questions 2-3 may not be needed if EPA headquarters staff can quickly access this information directly
from STARS)
2. Was this project on the IMS SDS list? (Y/N) Optional comments:
a. If yes, which IMS area?
b. If yes, what is the SDS ranking?
c. If yes, what are the project's estimated IDL and FDL?
3. In your assessment (or based on an IMS or other assessment), does the tribal community receiving
funding have adequate capacity to manage the results of the project? (Y/N) Please describe.
4. Has the direct grant option been communicated to the funding recipient?
5. Will this project be funded through a direct grant or an interagency agreement with IMS?
6. What are the estimated construction start and construction completion dates?
For DWIG-TSA:
7. Was past compliance history (where relevant) evaluated?
8. Are past compliance violations able to be addressed through funding for infrastructure? (Y/N)
Optional comments:
9. Have other compliance improvement options been considered and pursued?
Regions could submit the populated form to a central information source. Headquarters staff would not be
selecting or rejecting projects for funding, but could ask questions to clarify why certain projects were selected.
This type of transparent, priority-driven, yet flexible approach would likely make EPA less vulnerable to scrutiny
from national decision makers or auditors who could otherwise cast doubt on how selected projects support EPA's
priorities.
To further support the selection of DWIG-TSA projects that address compliance issues, EPA regional staff could also
continue working with regional compliance programs to support a more unified EPA approach, whether that be
through considering how to promote prevention of compliance problems, considering enforcement options in lieu
of DWIG-TSA funding, or other options. During the meeting of EPA regional coordinators in December 2010, the
DWIG-TSA coordinators decided to more proactively share compliance information with IHS area offices to ensure
that priority compliance needs are reflected on the SDS list. This increased information sharing between the
agencies could help IHS to be better aware of the compliance issues and improve the reflection of compliance
issues in the SDS ranking.
6. For both programs, routinely collect and analyze data to enhance transparency and strategic coordination
and improve EPA's ability to report on, advocate, and improve both programs through learning and
adaptation.
Evaluation Report - March 2011 Page 51
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Improved information availability will enable EPA to better understand its contributions through these programs
and support learning over time to improve program effectiveness. Given there is currently very little information
collected by EPA nationally other than basic funding statistics, even a modest increase in data collection would
provide a markedly improved ability to understand, explain, and advocate for these infrastructure programs at the
regional and national levels. The type of information that could be beneficial would not be complicated or
extensive. It would likely include the following.
Summary information about projects funded similar to what is currently collected and supplemented with
information from the project selection form described in recommendation 5.
The current status of the project on an annual or semi-annual basis. This information could be taken from the
STARS milestone information or from grant reports if projects are administered through a direct grant with no
IMS involvement.
For the DWIG-TSA program, an annual summary of the current compliance trends at tribal water systems in
the regions and a "diagnosis" about which of these problems are potential candidates for DWIG-TSA-funded
solutions. This type of information would help the DWIG-TSA program better target funding toward
compliance issues that can be addressed with infrastructure funding, as well as understand when there are
issues that cannot be addressed with this funding but could be supported by other EPA programs, such as
technical assistance.
An on-line tool that is professionally developed using current technology would allow for easy data entry, data
access, and reporting. With an on-line reporting tool, the reporting burden on the regions would be modest and
infrequent, and the benefits to the program would be worth the effort (including some decreased burden on the
regions from headquarters' staff making as many ad hoc inquiries).
EPA headquarters made a previous attempt to use an on-line database to track information for the DWIG-TSA
program. We understand from interviews that users found the database to be difficult to use and update and
therefore it was not consistently populated and was ultimately abandoned. In addition, some regional
interviewees indicated that the EPA Integrated Grant Management System (IGMS) already has some of the kind of
data that we have suggested collecting, and that any additional reporting beyond what is already required would
be duplicative. It is our understanding that IGMS does have some basic data on EPA grants, but that the data on
lAs is very limited, and on the whole, that IGMS does not have most of the information we would suggest
collecting. We would, however, encourage EPA headquarters staff to directly address and clarify this question of
dual reporting from IGMS, and wherever possible (whether from IGMS, STARS, or another source), gather data
from existing information first before asking the regions to report data.
A retrospective on the reporting done under ARRA would further help to identify when ARRA-like reporting may
helpful for routine program implementation.
7. Update the national guidelines for both programs.
Following the consideration of the above recommendations and any resulting changes, we recommend that EPA
headquarters work with EPA regions to update the guidelines for both programs to:
Reflect any changes or clarifications in goals, priorities, and measures;
Reflect any changes or clarifications to expected project selection processes;
Evaluation Report - March 2011 Page 52
-------
EVALUATION OF THE DRINKING AND INFRASTRUCTURE TRIBAL SET-ASIDE GRANT
Clarify (or reiterate) what projects are eligible for program funding, based on experiences with
implementation over the course of the program;
Clarify reporting expectations;
Clarify the expected communication and roles regarding lAs (including interactions with IMS) and direct grants;
Make the guidelines as concise and clear as possible; and
For the DWIG-TSA program, also clarify expectations around:
Identifying and communicating about compliance violations to IMS area staff;
Tracking the cause of compliance violations (i.e., whether infrastructure related or not) and whether
there is a possible infrastructure fix to the problem; and
Working with other regional compliance staff to pursue available options for improving compliance.
In addition, EPA headquarters should work with the EPA regions to update any communication materials about
these programs. For example, the CWISA program has a Frequently Asked Questions document developed in 2007
that should be updated with new information about eligible projects.
8. Continue to enhance and improve communication within EPA, between EPA and IMS, and between EPA and
the tribes.
As discussed in the findings section, communication between EPA headquarters and EPA regions, between EPA
and IMS, and between EPA and the tribes is important to effectively implement and measure the impact that these
programs have on tribal communities. Although the evaluators heard from interviewees many positive reflections
on communication, we also heard that communication improvements in the following areas would be useful:
Improving the "we are in this together as one agency" collaboration between EPA headquarters and EPA
regions;
Ensuring that calls and meetings on these programs have clear objectives and substantive exchanges, rather
than holding them just for the sake of holding them;
Increasing the frequency and quality of communication between the EPA regional staff and IMS area office
staff where communication is currently sparse. This communication is particularly important around sharing
of compliance history for inclusion into the IMS area SDS lists, but communication would also be valuable
around each agency's (and region/area's) priorities, project selection considerations, and knowledge of tribal
specifics;
Encouraging EPA regional staff to have some direct contact with the tribes where they do not already. At a
minimum, this contact could be a letter sent to tribes reminding them of EPA's role in funding, including the
process for project selection and the direct grant/IA options. There may also be the opportunity or need for
more direct communication with tribes around compliance issues, though in some regions, this could likely be
handled through other regional compliance program channels; and
Considering ways to highlight the accomplishments of the program that may not be reflected through
numbers, both within EPA and to external stakeholders, such as through the issuance of an annual program
report.
Evaluation Report - March 2011 Page 53
-------
EVALUATION OF Til IE AND INFRASTRUCTURE TRIBAL SET-ASIDE
Additional Considerations
Below we offer three additional considerations which reflect our support for efforts that are already underway or,
in one case, previous work done to reflect on many of the same issues that this evaluation has covered.
1. Continue to focus on the critical issues surrounding infrastructure sustainability, including tribal O&M
capacity issues.
It is clear that maintaining water infrastructure for the duration of its expected lifespan is critical for ensuring the
best use of taxpayer dollars and for best supporting tribal needs in the long run. In 2010, EPA issued its Clean
Water and Drinking Water Infrastructure Sustainability Policy, which encourages all communities to develop
sustainable systems that employ effective utility management practices to build and maintain the level of
technical, financial, and managerial capacity necessary to ensure long-term sustainability.77 As this evaluation
found, tribal systems have particular challenges associated with adequate O&M capacity to sustain and maintain
infrastructure.
Given the number of underlying factors that contribute to infrastructure longevity, including ensuring appropriate
design and the role of ongoing maintenance and operation, the evaluators recognize that, to a large extent, the
long-term effectiveness of these infrastructure programs will be determined by tackling these bigger-picture
challenges. At the same time, we recognize EPA's statutory limitations (e.g., these programs cannot fund
operation and maintenance) and resource constraints. Therefore we encourage EPA to work with other federal
agencies as well as with tribes to identify and address these complex, but critically important, sustainability issues,
including identifying any needed changes to federal statute(s). Reports from the Infrastructure Task Force on
meeting the access goal and on improving technical assistance to AI/AN communities have recommendations for
all the involved federal agencies to consider to best support infrastructure longevity, including addressing funding
for O&M and coordinating current technical assistance programs to maximize their reach.
2. Continue to communicate about IA requirements, expectations, and inter-agency roles.
We heard from IMS and one of the tribal interviewees that clarifying IA requirements, expectations, and roles
would be useful. Some, if not all of these individuals were already familiar with the new standardized IA terms and
conditions, yet still expressed concerns about different interpretations of the requirements and expectations, and
at least two IMS staff noted continued concerns that lAs are perceived incorrectly as grants by EPA. Additional or
reiterated clarification of requirements, expectations, and roles may be helpful.
3. Reflect on other reports and recommendations that pertain to these programs.
Other efforts have or are examining several of the same issues that this evaluation has covered, including statutory
limitations of EPA to provide funding for certain needs (e.g., operations and maintenance, individual wells), the gap
between the funding needed to meet access needs and actual funding, and the challenges of tracking progress
toward the United States' access goal using the available data. The Infrastructure Task Force Access Subgroup
U.S. EPA. Clean Water and Drinking water Infrastructure Sustainability Policy. Available:
http://water.epa.gov/aboutow/upload/Sustainabilitv-Policv.pdf.
78 Federal Infrastructure Task Force Access Subgroup. Meeting the Access Goal: Strategies for Increasing Access to Safe Drinking Water and
Wastewater Treatment to American Indian and Alaska Native Homes. (March 2008).
Tribal Technical Assistance Workgroup, Federal Infrastructure Task Force on Tribal Access to Safe Drinking Water and Basic Sanitation.
Strategies for Improving Technical Assistance Delivery in American Indian and Alaska Native Communities (March 2010).
Evaluation Report - March 2011 Page 54
-------
EVALUATION OF THE AND INFRASTRUCTURE TRIBAL SET-ASIDE
developed a report in 2008, "Meeting the Access Goal: Strategies for Increasing Access to Safe Drinking Water and
Wastewater Treatment to American Indian and Alaska Native Homes." This paper made a number of
recommendations for the involved federal agencies to consider in the areas of infrastructure funding, operational
and maintenance cost and capacity, and programmatic coordination. The Infrastructure Access Task Force Tribal
Technical Assistance Workgroup also developed a report on strategies for improving technical assistance delivery
in tribal communities. Several recommendations in both of these reports are still valid and consistent with the
findings and recommendations in this evaluation, and we encourage EPA to reflect upon this work and other
efforts, as well as the recommendations in this report to determine best steps forward for these programs and
their interaction with other federal work underway.
Evaluation Report - March 2011 Page 55
------- |