§
<
3J
\
r"
*L P R
T>
2
LU
O
T
J
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Evaluation Report
Partnership Programs May
Expand EPA's Influence
Report No. 2007-P-00003
November 14, 2006
ENERGYSTAR
CLIMATE v
U.S. Environmental Protection Agency
"\WA
w
<«r*
WASTE
ISE
r
m?
M u.s e™
National
Environmental
PerformanceTrack
Enwworsnental Prg^fctior* Agency
^JPar
v.
National
Partnership
for
Environmental
Priorities
-------
Report Contributors: Laurie Adams
Natasha Besch-Turner
Daniel Carroll
Jeffrey Harris
Jeffrey S. Hart
Olga Stein
Thane Thompson
Abbreviations
EPA U.S. Environmental Protection Agency
FTE Full Time Equivalent
HFC Hydrofluorocarbon
IAC Innovation Action Council
ICR Information Collection Request
OA Office of the Administrator
OAR Office of Air and Radiation
OARM Office of Administration and Resources Management
OIG Office of Inspector General
OPEI Office of Policy, Economics, and Innovation
OPPTS Office of Prevention, Pesticides, and Toxic Substances
ORD Office of Research and Development
OSWER Office of Solid Waste and Emergency Response
OW Offi ce of Water
PRA Paperwork Reduction Act
Cover images: Logos of six of the U.S. Environmental Protection Agency's partnership
programs.
-------
£
<
33
V
^60S%
V PRO^4-0
o
LU
a
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
2007-P-00003
November 14, 2006
Catalyst for Improving the Environment
why we Did This Review Partnership Programs May Expand EPA's Influence
Prior to our review, the U.S.
Environmental Protection
Agency (EPA) did not have
comprehensive information
describing each of its
partnership programs. We
conducted this evaluation to
collect current and consistent
information about EPA's
headquarters partnership
programs.
Background
Over the last few years, EPA
has worked to develop new
types of environmental
solutions. The Agency now
relies more heavily on
partnership programs to help
protect the environment.
EPA's 54 headquarters
partnership programs are
diverse, providing a variety of
benefits to several different
customer groups, including
some nongovernmental
organizations and the public.
For further information,
contact our Office of
Congressional and Public
Liaison at (202) 566-2391.
To view the full report,
click on the following link:
www.epa.qov/oiq/reports/2007/
20061114-2007-P-00003.pdf
What We Found
Partnership programs may expand EPA's environmental influence. These programs
may achieve this effect by broadening EPA's potential participant base and
addressing environmental problems not governed by regulations. Partnership
programs are diverse in staff size and budget levels. The number of EPA's
partnership programs has grown in the last few years.
We found that partnership programs reported contributing to EPA's overall
environmental mission. The majority of the programs also reported having annual
goals and program outputs or outcomes. All 54 headquarters partnership program
managers we spoke with stated that their program contributed to at least one EPA
strategic goal. Thirty-eight managers reported that their programs contribute to
more than one goal. In a few cases, managers said they work together to solve a
large environmental problem. Partnership programs build on the Agency's
traditional regulatory efforts, but are not intended as substitutes for regulations.
Instead, some partnership programs work together with regulatory efforts to help
participants go beyond compliance with existing regulations. Other programs work
to address environmental concerns that are not governed by Federal regulations,
such as recycling and climate change.
Many partnership program managers said they collect complete and reliable data
that they then use to make changes to their programs. However, barriers to data
collection, including data collection costs, exist. Partners and participants
contribute to some partnership programs by sharing program tasks. In a few cases,
this includes participants undertaking program management roles. However, this
type of collaboration may make it difficult to determine the outcomes of individual
programs.
Next Steps
The Agency has had difficulty in defining, identifying, and characterizing these
programs. Managers claim that their programs help to achieve EPA strategic goals,
but we have not yet gathered sufficient measurement and outcome data to verify
that partnership programs are achieving these claims. Since as many as half of the
programs receive marketing assistance through collaboration, as well as other in-
kind services and occasional participation fees, we will need to conduct further
evaluation work to determine:
1. How partnership programs manage their collaborative relationships.
2. How those collaborative relationships support program outcomes.
-------
^£0SX
i -jQt-7 5 UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
| X\\/V ? WASHINGTON, D.C. 20460
V
OFFICE OF
INSPECTOR GENERAL
November 14, 2006
MEMORANDUM
SUBJECT: Partnership Programs May Expand EPA's Influence
Report No. 2007-P-00003
TO: Luis Luna, Assistant Administrator
Office of Administration and Resources Management
Bill Wehrum, Acting Assistant Administrator
Office of Air and Radiation
James Gulliford, Assistant Administrator
Office of Prevention, Pesticides, and Toxic Substances
George Gray, Assistant Administrator
Office of Research and Development
Susan Bodine, Assistant Administrator
Office of Solid Waste and Emergency Response
Benjamin Grumbles, Assistant Administrator
Office of Water
Brian Mannix, Associate Administrator
Office of Policy, Economics and Innovation
This is our report on the subject of U.S. Environmental Protection Agency (EPA) partnership
programs conducted by the EPA Office of Inspector General (OIG). This report represents the
opinion of the OIG and does not necessarily represent the final EPA position.
The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost billing rates in effect at the time - is $293,349.21.
-------
We have no objections to the further release of this report to the public. This report will be
available at http://www.epa.gov/oig. If you or your staffs have any questions regarding this
report, please contact me at 202-566-0847, Jeff Hart, Assignment Manager, at 303-312-6169, or
Thane Thompson, Project Manager, at 513-487-2361.
Sincerely,
-"BtTTXTRoderi ck
Acting Inspector General
cc: Marcus Peacock, Deputy Administrator
-------
Partnership Programs May Expand EPA's Influence
Table of Contents
Chapters
1 Introduction 1
Purpose 1
Background 1
Scope and Methodology 2
Partnership Programs in Brief 3
2 Partnership Programs May Help EPA Expand its Potential Audience 7
Partnership Programs May Help Achieve Several Strategic Outcomes 7
Program Managers Report Serving a Broad Participant Group 8
Program Managers Report That Regulatory
Relationships Vary Significantly 9
Conclusion 10
3 Data Availability and Partner Collaboration Impact Program Management . .. 12
Managers Report That Trusted Data Inform Program
Decisions, but Barriers Remain 12
Collaboration Can Provide Benefits, but May
Pose Management Challenges 13
Conclusion 16
Status of Recommendations and Potential Monetary Benefits 17
Appendices
A EPA Partnership Programs Reviewed 18
B Summary of Survey Responses 19
C Glossary of Terms 27
D Agency Responses to the Draft Report 30
E OIG's Comments on Agency's Responses 35
F Distribution 37
-------
Chapter 1
Introduction
Purpose
Over the last few years, EPA has worked to develop new types of environmental
solutions. Recently, the Agency has begun relying on partnership programs to
complement traditional regulatory approaches to protect the environment. To
address whether partnership programs are meeting their environmental goals, we
searched for existing Agency information about these programs. However, no
current or central set of information described what each of these programs did,
how they contributed to EPA's overall mission, or how EPA managed these
programs. Therefore, we identified a group of headquarters partnership program
managers and surveyed them on the following topics.1
1. What are the characteristics of partnership programs?
2. How do partnership programs contribute to EPA's overall mission?
3. How are individual partnership programs managed?
Background
EPA has implemented partnership programs for more than 20 years. These
programs address a variety of environmental and human health problems,
including loss of wetlands and pesticide exposure. They also address water and
energy use, recycling, or the environmental actions of individuals. Some
partnership programs report very small budgets and only a fraction of a staff
member's time devoted to operating them. Other programs reported they have
dozens of staff with budgets in the tens of millions of dollars. Partnership
programs reported wide diversity in their environmental focus, program benefits,
and participant groups. Nearly all program managers said their programs
encourage participants to voluntarily engage in environmentally beneficial
activities that are not covered by regulations. Managers said their programs offer
opportunities to go beyond compliance, or work to solve non-regulated
environmental or human health problems.
EPA has recognized the importance of better coordination across all partnership
programs. This includes enhanced accountability for sound program design and
improved strategic management of programs. In June 2004, EPA's Innovation
Action Council2 was charged with partnership program oversight by then Acting
Deputy Administrator Stephen Johnson. The Partnership Program Coordination
1 Detailed results are located in Appendix B.
2 Established in 1996, the Innovation Action Council (IAC) is composed of EPA's top career executives, and has
overall responsibility for formulating and advancing the Agency's innovation agenda. The mission of the IAC is to
develop and promote innovative approaches to addressing increasingly complex enviromnental challenges.
1
-------
Team was created to develop general program guidelines and improve support for
partnership program managers. Since its inception, the coordination team has
issued several guidance documents. The team has also provided training and one-
on-one coaching sessions, and worked to stay in regular contact with many of the
partnership programs. However, the coordination team has limited authority to
require specific program management processes or consistent outcome reporting.
EPA has had difficulty defining, identifying, and characterizing its partnership
program population. These programs have been grouped into numerous
overlapping categories, including "Voluntary Programs," "Partnership Programs,"
and "Stewardship Programs." In December 2003, EPA reported that it had
approximately 75 "Voluntary Programs." Just 2 years later, in 2005, EPA
reported approximately 133 EPA "Partnership Programs." According to the
Office of Policy, Economics, and Innovation (OPEI), this change is due to both an
increase in the overall program population and in the way these types of programs
are defined. OPEI informed us that the 2005 report used a broader definition of
"partnership program" than the 2003 report.
Scope and Methodology
EPA's definition of partnership programs has continued to evolve during our
review. For the purpose of this evaluation, we define a partnership program as "A
program currently operated by a Headquarters program office that is national in
scope, and that works to encourage participants to voluntarily engage in
environmentally beneficial activities." In consultation with the OIG, each EPA
program office provided us with a list of partnership programs that met our
criteria. This list resulted in a universe of 54 partnership programs operated by
seven different program offices. We have included a list of the programs we
reviewed in Appendix A.
We developed a survey to collect partnership program information and
administered the survey through a telephone interview with each partnership
program. We sent completed interviews to each partnership program for data
verification. We did not independently verify the responses for accuracy. For
more information on the survey questions and program responses used in this
report, please see Appendix B.
This evaluation builds on a previous report, "Ongoing Management
Improvements and Further Evaluation Vital to EPA Stewardship and Voluntary
Programs" (2005-P-00007), issued in February 2005. In that report, we
recommended that EPA should address management issues, including
strategically planning, coordinating, and managing its voluntary programs.
This evaluation was performed in accordance with Government Auditing
Standards, issued by the Comptroller General of the United States. We reviewed
management controls of the Agency-wide guidance regarding the design and
measurement of partnership programs.
2
-------
Partnership Programs in Brief
Seven different program offices operate EPA's headquarters partnership
programs. The number of programs operated by each office varies from 23 in
OAR to 1 each in OARM and ORD. See Table 1-1.
Table 1-1. Partnership Program Distribution
Number of
Programs
EPA Program Office
23
Office of Air and Radiation (OAR)
12
Office of Prevention, Pesticides, and Toxic Substances (OPPTS)
8
Office of Water (OW)
6
Office of Solid Waste and Emergency Response (OSWER)
3
Office of the Administrator (OA)
1
Office of Acquisition and Resource Management (OARM)
1
Office of Research and Development (ORD)
The oldest partnership program still in operation began in 1985 (Radon), though
the median program age is 7 years. Figure 1-1 captures programs that were
operating on December 31, 2005. Any programs that were being developed and
have subsequently been launched, or are no longer operated by EPA, are not
included in this chart. See Figure 1-1.
Figure 1-1. Growth of Partnership Programs
$
O
C ^
.2 co
ro csj
o> TI
Q. c
o o
c C
— o
E 2
as
2 °
Q_ —
_q nj
5=
*= >,
60
50
40-
30
20
10
0
a
aR
1985 1988 1991 1994 1997 2000 2003
Since partnership programs encourage their participants to voluntarily engage in
environmentally beneficial activities, they often give participants program
benefits as an incentive to change their behavior. Program managers reported
offering a variety of benefits to participants. As Figure 1-2 shows, most programs
3
-------
provide technical
assistance and information;
relatively few provide
regulatory flexibility. Only
one program provided
fewer than three of the
program benefits listed in
this figure. Most programs
provided between five or
six benefits; two programs
provide eight of these
benefits to their
participants. Each program
benefit is defined in the
glossary, located in
Appendix C.
Partnership programs often require participants to meet program-specific
commitments. We identified three types of partnership program commitment
approaches, including
formal requirements,
informal
requirements, and no
commitment
requirements.
Programs with formal
requirements may ask
participants to sign a
memorandum of
understanding, and
may even remove
participants from the
program if they do
not meet
commitments.
Programs using
informal
requirements may ask participants to submit data or participate in periodic
meetings. Programs that have no commitment requirements may simply provide
information or ask participants to achieve environmental outcomes in good faith.
See Figure 1-3.
Figure 1-2. Number of Programs Offering Each Benefit
LO
O
¦ 16
19
¦ 39
46
46
52
52
0 20 40 60
Types of Program Benefits Offered
~ Regulatory Flexibility
~ Netw orking
¦ Environmentally Preferable Product Designation
¦ Other unique benefits
¦ Competitive Business Solutions
~ Public Recognition
¦ EPA/State Partnership Opportunities
~ Information
~ Technical Assistance
Figure 1-3. Program Commitment Approaches
No
Comrritment
Requirement
30%
Informal
Requirerrent
Formal
Requirement
31%
4
-------
Figure 1-4. Partnership Program Budgets
251
20-
15-
10-
5-
0-P U
Less than $100k to $1 IVbre than $1
$100k million million
Partnership program managers said their annual budgets total an estimated $352
million.3 One of these programs (National Non-Point Source Pollution Program)
reported a budget of more than $209 million. Of the remaining programs, 36
reported budgets of $1 million or less, and two program managers said they had
no budget at all. The median program budget is $492,500. Program budgets may
not, however, take into account the total costs of implementing these programs,
such as in-kind participant contributions. See Figure 1-4.
Most partnership programs
operate with relatively few
staff (hereinafter full time
equivalent, or FTE).
Program managers reported
that a total of 365 FTEs run
these programs, though the
median number of program
staff is 2.65. As shown in
Figure 1-5, 72 percent (39 of
54) of partnership program
managers said they have
fewer than 5 FTEs.
~ less than 1 FTE
¦ 1 to 2 FTEs
~ more than 2, to 5 FTEs
~ more than 5 FTEs
3 This total was derived from the question. "What is your annual budget in dollars?" However, some programs
provided estimates or noncurrent year totals.
5
w
CC
L_
O)
o
CD
_Q
Figure 1-5. Distribution of Partnership
Program Staff
-------
Partnership programs
provide their benefits
to many different
types of participants,
such as individuals,
schools, or industry.
Although 7 of the 54
programs only
served 1 type of
participant, most
programs served 4 or
more of the
participant types
listed in Figure 1-6.
Figure 1-6. Partnership Program Participants
F ed eral
Agencies
Individuals
S tate/Local
G overn m en t
O 10 20 30 40 50 60
Number of Programs Serving Each Participant Type
6
-------
Chapter 2
Partnership Programs May Help EPA
Expand its Potential Audience
Partnership programs may have expanded the potential audience that can help
EPA accomplish its overall environmental and human health mission. Program
managers said their programs, as a group, impact all five strategic goals. Many
managers reported that their programs provide a diverse set of benefits to a broad
group of participants. Some of these participants include nongovernmental
organizations and individuals. Some program managers said they work in
conjunction with the Agency's traditional regulatory activities. Other program
managers said that they help participants voluntarily protect the environment
where there are no Federal regulations. In some cases, programs may attempt to
provide solutions in lieu of environmental laws or regulations. While partnership
programs may make contributions to EPA's mission, it is necessary to conduct
future evaluation work to determine if program outcomes are being achieved.
Partnership Programs May Help Achieve Several Strategic Outcomes
Partnership programs may help achieve several strategic outcomes. In some
cases, several programs work on a single environmental problem. Or, a single
program can be spread wide enough to encompass several related environmental
issues. Most program managers said their programs develop annual goals, and all
assert that their programs contribute to one of EPA's strategic goals.
All 54 program managers identified one EPA strategic goal as their program's
area of primary impact. We also asked each manager to identify other strategic
goals that their program impacts. Because of the large number of OAR programs,
the largest number of program managers listed Goal 1: Clean Air and Global
Climate Change as their primary impact. However, 38 of the 54 program
managers claimed their program supports at least two strategic goals. Also, 14
asserted that their programs support all five strategic goals. An example of these
Table 2-1. Partnership Programs Reported Support for EPA's 2003-2008 Strategic Goals
Strategic Goal
Primary
Impact
Secondary
Impact
Totals
GoaM: Clean Air and Global Climate
24
17
41
Change
Goal 2: Clean and Safe Water
7
21
28
Goal 3: Land Preservation and Restoration
4
13
17
Goal 4: Healthy Communities and
7
27
34
Ecosystems
Goal 5: Compliance and Environmental
12
21
33
Stewardship
7
-------
reported multi-goal outcomes includes the AgStar program, which focuses
primarily on methane emission reduction activities (Goal 1). As a result, the
program claimed that organic loading into waterbodies has been reduced (Goal 2).
In another example, the Smart Way Transport program focuses on reducing
greenhouse gas emissions (Goal 1) from diesel engines used in the trucking
industry. According to the manager, the program also claims to have significantly
reduced particulate matter. This pollutant has been linked to asthma attacks in
sensitive populations (Goal 4). These outcomes show that partnership programs
may have an expanded influence on environmental problems beyond their
primary goal or audience.
Several programs reported that they work to solve a single overarching
environmental problem. These "umbrella" efforts can be undertaken by one large
and diverse program, or can be the result of several individual programs. For
example, the Indoor Air Quality program manager outlined several subprograms,
each focusing on a separate issue. These programs include indoor air quality in
elementary and secondary schools, asthma, radon, and secondhand smoke. Some
groups of programs focus on the same problem, but with different participant
groups. For example, six program managers said their programs work to reduce
methane emissions from sources as widespread as animal operations, coal beds,
landfills, and power production facilities. In contrast to umbrella efforts, almost
half of our respondents claimed that their program was unique and did not overlap
with any other EPA partnership program. These programs address a range of
niche environmental issues, including diverting electronic waste from landfills, or
helping magnesium manufacturers reduce greenhouse gas emissions.
The strategic outcomes of partnership programs may be difficult to determine
because a shared definition of effectiveness across partnership programs does not
exist. Some managers defined effectiveness as program participation rates.
Others defined it as the environmental outcomes the program achieved. In some
cases, managers defined effectiveness in terms of their costs to achieve program
outputs or environmental outcomes. The extent that these definitions vary
complicates EPA's ability to determine the overall effectiveness of partnership
programs. Further, EPA does not have guidance that defines program
effectiveness.
Program Managers Report Serving a Broad Participant Group
Partnership programs may help EPA reach a broad group of participants. Types
of program participant groups can range from narrow user groups defined by their
use of a single chemical, to entire industry populations. For example, the HFC
(Hydrofluorocarbon) 23 Emission Reduction Program manager said the program
works to reduce the use and emission of one specific ozone-depleting refrigerant.
While the manager said only three manufacturing facilities in the United States
use this chemical, all three facilities participate in the program. Those
participants also receive ongoing technical assistance to reduce their HFC 23
8
-------
emissions. In contrast, the Environmental Technology Verification Program
manager said the program has tested over 350 technologies and has more than 800
stakeholders.
Programs managers reported that they offer several program benefits. As shown
in Table 2-2 below, almost all partnership programs offer technical assistance and
information as benefits to their participants. In addition to the benefits listed here,
35 program managers said they provided other types of benefits to participants.
These benefits include, for example, grants, increased public awareness of the
environmental problem, or specific program outcomes. Managers reported 19
additional benefits that were unique and did not fit into any specific category.
These benefits
included "improved
quality of life,"
"improved corporate
environmental
image," and
"helping individuals
become engaged as
stewards of their
watershed." These
types of benefits further demonstrate the diversity of partnership programs. They
further serve as examples of the efforts that these managers reported making to
tailor their services to the needs of their participants.
Some partnership program managers also reported working with several different
types of participants to solve a particular environmental problem. For example,
the managers for both the Pesticide Environmental Stewardship Program and the
Energy Star Energy Management Program told us that they have private sector
participants who actively engage in voluntary activities. However, these
managers also said they work with nonprofit organizations and community
participants that are actively involved in helping keep the program running.
Several programs sometimes included these program partners in an overall
program participant count because while they are an integral part of the program,
they are not the primary target participant. We discuss these types of
collaborative efforts, as well as the benefits and challenges that partner
collaboration provides, in the next chapter.
Program Managers Report That Regulatory Relationships Vary
Significantly
EPA has traditionally worked to protect human health and the environment by
serving in the role of regulator. EPA enforces environmental statutes, typically
through using penalties for regulatory noncompliance. In contrast, partnership
programs encourage their participants to voluntarily engage in environmental
Table 2-2. Number of Programs Offering Each Benefit
# of Programs
Type of Benefit
52
Technical Assistance
52
Information
46
EPA/State Partnership Opportunities
46
Public Recognition
39
Competitive Business Solutions
16
Environmentally Preferable Product Designation
15
Networking
10
Regulatory Flexibility
9
-------
activities by offering various benefits. EPA has begun relying on partnership
programs as another tool for protecting the environment. Because of this reliance,
we assessed the relationships these programs have with traditional regulatory
programs. We found that there is not an either/or relationship between
partnership programs and regulatory activities. Rather, we found that partnership
programs can have several types of regulatory relationships that, together with
traditional regulation, may reach a broader audience and achieve additional
environmental outcomes.
Half of the 54 partnership program managers said their programs work in
conjunction with environmental regulations. Some reported offering regulatory
flexibility or providing opportunities for participants to go beyond compliance
with existing regulations. These programs work with a variety of laws. The
majority of program managers identified a relationship with one of the following
statutes: the Clean Air Act, the Clean Water Act, the Resource Conservation and
Recovery Act, or the Toxic Substances Control Act. These programs encourage
full compliance with environmental laws, and urge their participants to
voluntarily go beyond these standards to achieve greater environmental results.
Most of the other managers we interviewed said their programs focus on
environmental problems not governed by regulations. These programs sometimes
recruit community groups or individual citizens. Neither of these groups typically
falls within the traditional regulated community. Many of the programs in this
category also recruit businesses, States, and other participants who usually are the
focus of traditional regulations. Regardless of their focus, these programs work to
encourage participants to solve environmental problems not covered by
regulations. For example, 18 program managers said their programs were
designed to target global climate change. These efforts support both the United
States Climate Change Action Plan and the 2002 Global Climate Change
Initiative. This example shows that in some cases, partnership programs may be
designed to address environmental problems where specific regulations do not
exist.
Finally, we found that some programs can have several different regulatory
relationships at the same time. For example, one manager reported that their
program can offer regulatory flexibility, help participants go beyond compliance
with existing Federal regulations, and encourage participants to voluntarily
protect the environment where there are no Federal regulations. In two cases,
program managers reported that the results of their programs are used to either
develop or meet regulations.
Conclusion
Partnership programs may make contributions to the Agency's overall mission.
Managers claim that their programs help to achieve EPA strategic goals, expand
the Agency's participant groups, and work to expand the agency's relationships
10
-------
beyond the traditional regulated community. However, we have not yet gathered
sufficient measurement and outcome data to verify that partnership programs are
achieving these claims. Because of this, future evaluation work needs to be
conducted to assess the extent to which each program is achieving its intended
outcomes.
11
-------
Chapter 3
Data Availability and Partner Collaboration
Impact Program Management
Many partnership program managers said they believe they collect complete and
reliable data. The managers said they use the data to make changes to their
programs, but barriers to data collection, including data collection costs, exist.
Program managers said that their partners and participants contribute to EPA's
partnership programs by sharing program tasks. In a few cases, they said
participants can even undertake program management roles. This collaboration,
however, may introduce additional challenges to determining the outcomes of
individual programs. Because collaboration can expand the overall scope of
programs, we will need to conduct further assessment to determine how programs
manage their collaborative relationships, and how those relationships support
program outcomes.
Managers Report That Trusted Data Inform Program Decisions, but
Barriers Remain
Managers expressed a high level of
confidence in the reliability of their
program data. We asked managers to
rate the reliability of their data on a
scale of 1 to 4. As Figure 3-1
demonstrates, about twice as many
program managers rated their data as
the most reliable, or "4," compared
with those rating their data as a "3."
Confidence in data reliability was
associated with having sufficient data.
Those program managers who rated
their data as a "3" were about twice as
likely to say data they were not
currently collecting would improve
program management as those
managers who responded with a "4." Overall, fewer than half of the managers
responded that data they were not collecting might improve program
management.
Many partnership program managers said they use their data to change their
programs. For example, they changed the program services offered to
participants, changed the target participants, changed internal program operations,
Figure 3-1. Reported Reliability of
Program Data
is 15
12 3 4
1 is least reliable, 4 is most reliable
12
-------
or even changed program goals. One program manager offered that "If emission
reductions aren't meeting program goals, then we evaluate what additional efforts
are necessary to achieve those goals." The program manager for the Indoor Air
Quality program said that their "Asthma Survey revealed some asthma triggers
where focus was lacking. This information was used to focus more on these
triggers."
Another type of data some program managers can use is the results of cost-benefit
analyses. About half of the program managers responded that they had performed
a cost-benefit analysis for their partnership program. Some program managers
then characterized success based on an ability to implement the program cost
effectively. Specifically, 11 program managers defined success for their
programs in terms of dollars spent per output or dollars spent per outcome; 6 of
the 11 said they had conducted a cost-benefit analysis.
Across all partnership programs, the most frequently cited barriers to getting
environmental outcome data were that a partnership program cannot require data
submission, and that collected data may not be completely accurate. One issue
raised by some program managers was the cost of data collection. In fact, 10
program managers asserted that data that might improve program management
were not collected because the cost was too expensive. Capturing appropriate and
adequate data is central to tracking program outcomes and making improvements
to partnership programs. Without needed data, these programs may be hindered
in their ability to demonstrate program success or adapt to changing partner and
participant needs. Four program managers cited the difficulty of linking data to
environmental outcomes as a general barrier. Other reported barriers included the
nature of some environmental data as proprietary and therefore sensitive, and
barriers posed by the Office of Management and Budget's information collection
request process.4 Four other program managers reported that they do not collect
any data at all. Of these, three managers mentioned access to data as a top
challenge to fully achieving their expected program outcomes.
Collaboration Can Provide Benefits, but May Pose Management
Challenges
Partnership programs can benefit from collaborative relationships with their
partners and participants. Program managers reported that they regularly
collaborate with other government organizations. This collaboration includes
sharing program tasks with their partners. In some cases, partners and
participants can even undertake some program management roles. However,
collaboration may introduce additional program management challenges, such as
restricted EPA decisionmaking roles and joint EPA-partner goal-setting. Further,
collaboration may complicate determining individual program outcomes.
4 According to the Paperwork Reduction Act (PRA) of 1995, Federal agencies must seek OMB approval prior to
collecting data from 10 or more members of the public.
13
-------
Table 3-1. Number of Programs That Report
Collaborating with Government Agencies
Number of
Government Agency or Organization
Programs
EPA Regions
45
States
44
Other Federal Agencies
44
Other EPA Partnership Programs
42
EPA Regulatory Programs
30
Program managers said
they often collaborate with
a number of government
agencies or organizations to
complete program tasks
(see Table 3-1). Program
managers said this
collaboration fulfills
multiple purposes. One
purpose is to help programs
with similar outcome goals properly account for the measured or observed
environmental outcomes of each program. For example, of the 18 programs that
address climate change, 6 programs reported that they have a process to attribute
the appropriate amount of emission reductions that each program can claim. In
this way, partnership programs share the task of allocating measured outcomes.
Another purpose for collaboration is to track if participants in one program are
also members of other partnership programs, as participants may enroll in more
than one program. Managers for nine partnership programs said they collaborate
with each other to track participant enrollment. A third purpose given for
collaboration is obtaining data for tracking environmental program outcomes.
Some programs use data collected by other Federal agencies. For example, 10
EPA program managers said they get the information they need to implement and
manage their programs from other Federal agencies, such as from the U.S. Census
Bureau.
Program managers also reported collaborating with their participants. Many
managers said they ask their participants to engage in marketing and recruitment
efforts. Nineteen managers even described their partners and participants as a
central component in their marketing strategy. Further, three program managers
said they require participants to market or promote the program as part of their
commitment.5 This work-sharing may allow program managers to focus on other
aspects of program management.
Managers said that another way that participants assist with program
implementation is by supplying funds. At least three managers said that their
participants share the costs of
the program by paying a
participation fee. However, 11
program managers reported
concerns that the costs of
participation may limit their
program from fully achieving its
expected outcomes. For
5 One program manager responded both that partners and participants are central to their marketing strategy, and
their programs require participants to market or promote the program as part of their commitment.
Table 3-2. Tasks Accomplished by
Collaboration
¦S Allocate outcomes among other programs
¦S Track participant enrollment
¦S Get data used to track outcomes
¦S Share program marketing responsibilities
¦S Share program costs
14
-------
example, several program managers cited the high cost of technology or
equipment. This may demonstrate that some partnership program managers
recognize the need to be sensitive to the expenses of their participants. Managers
reported several tasks where participants contribute to program operations. These
tasks are listed in Table 3-2.
Managing programs jointly with other agencies or organizations is not
uncommon. A few program managers said they have developed different types of
co-management relationships with their partners and participants. However, each
is accompanied by its own limitations. For example, one manager said their
program shares application review responsibilities with a nonprofit partner
organization. This process determines which participants are allowed in the
program. There may be the potential for perceived bias when programs engage in
this type of collaboration. Another manager said their program distributes
technical standards to participants through a partner Federal agency. Though
EPA develops these standards, the manager depends on their partner agency to
communicate these standards to its participants. If this information is not
communicated correctly to participants, it is possible that program outcomes will
suffer.
Another type of collaboration process can restrict EPA's decisionmaking role in a
partnership program. The annual and overall outcome goals of the Carpet
America Recovery Effort program are reportedly established jointly by industry,
nongovernmental associations, and government agencies at the Federal, State, and
local levels. EPA is only one of many stakeholders making decisions about this
program. In fact, four program managers said they are not the sole actor in
determining their program's participation goals.
Program collaboration may
complicate EPA's ability to
determine program outcomes and
total implementation costs. As
listed in Table 3-3, the complete
cost of implementing a
partnership program can include
other aspects in addition to a basic program budget. Our analysis shows that most
- if not all - aspects of assessing costs are complicated by program collaboration.
For example, 11 program managers said their budget was only partially funded by
their own program office. One program manager responded that the program was
funded entirely by another program office within EPA. Another said their budget
was set by a congressional earmark and has not been established as an annually-
funded activity. To establish comparable costs for some partnership programs, it
may be necessary to include implementation costs from other EPA program
offices or even other Federal agencies.
Table 3-3. Some Factors Included in
Calculating Total Implementation
Cost
•
Basic program budget
•
In-kind contributions
•
Fees paid by participants
15
-------
Many program managers said that collaboration provides their program with in-
kind services from partners or participants. As mentioned earlier these services
can include marketing activities or participant screening and registration. All of
these services can be considered part of a program's total implementation costs,
but may be difficult to value. Further, as mentioned earlier, several programs
report difficulty getting proprietary data from private entities. It is therefore
important to determine the extent that programs rely on collaboration to collect
data used to measure outcomes.
Conclusion
Partnership program managers consistently claim to use data to make changes to
their programs. Further, collaboration plays a significant role in the management
of most partnership programs, as nearly half of the programs claim they receive
marketing assistance through collaboration, as well as other in-kind services and
occasional participation fees. As a result, we will need to conduct further
evaluation work to determine how programs manage their collaborative
relationships, and how those relationships support program outcomes.
16
-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Action Official
Planned
Completion
Date
No Recommendations
Claimed
Amount
Agreed To
Amount
1 0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is undecided with resolution efforts in progress
17
-------
Appendix A
EPA Partnership Programs Reviewed
The following 54 partnership programs (listed alphabetically) met our evaluation criteria and
were verified by EPA program offices as being in operation on December 31, 2005.
Partnership Program Name
Program Office
Adopt Your Watershed
Office of Water
AgStar
Office of Air and Radiation
Best Workplaces for Commuters
Office of Air and Radiation
Carpet America Recovery Effort
Office of Solid Waste and Emergency Response
Clean School Bus USA
Office of Air and Radiation
Clean Water Act Recognition Awards
Office of Water
Climate Leaders
Office of Air and Radiation
Coal Combustion Products Partnership
Office of Solid Waste and Emergency Response
Coalbed Methane Outreach Program
Office of Air and Radiation
Combined Heat and Power Partnership
Office of Air and Radiation
Decentralized Wastewater Treatment Systems Program
Office of Water
Design for the Environment
Office of Prevention, Pesticides, and Toxic Substances
Diesel Retrofit Program
Office of Air and Radiation
Energy Star Energy Management
Office of Air and Radiation
Energy Star Product Certification
Office of Air and Radiation
Environmental Technology Verification Program
Office of Research and Development
Environmentally Preferable Purchasing
Office of Prevention, Pesticides, and Toxic Substances
EPA's Volunteer Monitoring Program
Office of Water
Federal Electronics Challenge
Office of Prevention, Pesticides, and Toxic Substances
Five-Star Restoration Program
Office of Water
Green Chemistry
Office of Prevention, Pesticides, and Toxic Substances
Green Engineering
Office of Prevention, Pesticides, and Toxic Substances
Green Power Partnership
Office of Air and Radiation
Green Suppliers Network
Office of Prevention, Pesticides, and Toxic Substances
GreenScapes
Office of Solid Waste and Emergency Response
HFC 23 Emission Reduction Program
Office of Air and Radiation
High Production Volume Challenge
Office of Prevention, Pesticides, and Toxic Substances
Hospitals for a Healthy Environment
Office of Prevention, Pesticides, and Toxic Substances
Indoor Air Quality
Office of Air and Radiation
Labs 21
Office of Administration and Resources Management
Landfill Methane Outreach Program
Office of Air and Radiation
Methane to Markets Partnerships
Office of Air and Radiation
Mobile Air Conditioning Climate Protection
Office of Air and Radiation
National Award for Smart Growth Achievement
Office of the Administrator
National Fish and Wildlife Contamination Program
Office of Water
National Non-Point Source Management Program
Office of Water
National Partnership for Environmental Priorities
Office of Solid Waste and Emergency Response
Natural Gas Star
Office of Air and Radiation
Partnership for Safe Water
Office of Water
Performance Track
Office of the Administrator
Pesticide Environmental Stewardship Program
Office of Prevention, Pesticides, and Toxic Substances
PFC Emission Reduction Partnerships for Semi-Conductor Industry
Office of Air and Radiation
Plug-In to eCycling
Office of Solid Waste and Emergency Response
Radon
Office of Air and Radiation
Reduced Risk for Conventional Pesticides
Office of Prevention, Pesticides, and Toxic Substances
Sector Strategies
Office of the Administrator
SF-6 Emission Reduction Partnership for Electric Power Systems
Office of Air and Radiation
SF-6 Emission Reduction Partnership for the Magnesium Industry
Office of Air and Radiation
Smart Way Transport
Office of Air and Radiation
Sustainable Futures
Office of Prevention, Pesticides, and Toxic Substances
The SunWse School Program
Office of Air and Radiation
Voluntary Aluminum Industrial Partnership
Office of Air and Radiation
Voluntary Children's Chemical Evaluation Program
Office of Prevention, Pesticides, and Toxic Substances
WasteWise
Office of Solid Waste and Emergency Response
18
-------
Appendix B
Summary of Survey Responses
As discussed in the Scope and Methodology section of Chapter 1, we supported the statements
made in this report with evidence obtained through structured interviews with partnership
program managers. We have included those survey questions and responses below. The
information below relates how the interviewees responded to each question; these responses are
provided from the program's perspective. In some cases, the number of responses provided for a
question will total to more than our population of 54 programs, as some programs gave more
than one answer to an open-ended question. For example, when responding to "How is this
program marketed to potential participants?" many program managers provided more than one
answer. We have not independently verified these responses through documentary evidence
What EPA office and division is this program in?
Frequency
Program Office
Office/Division
23
OAR
17
Office of Atmospheric Programs
4
Office of Transportation and Air Quality
2
Office of Radiation and Indoor Air
1
OARM
1
Facilities Management and Services Division
3
OA
1
Office of Development, Community and Environment
Division
2
Office of Policy, Economics and Innovation
12
OPPTS
3
Economics, Exposure and Technology Division
2
Office of Pesticide Program
4
Pollution Prevention Division
3
Risk Assessment Division
1
ORD
1
National Risk Management Research Lab
6
OSWER
5
Municipal and Industrial Solid Waste Division
1
Waste and Hazardous Waste Minimization & Mgmt
Division.
8
OW
1
Office of Groundwater & Drinking Water
1
Office of Science and Technology
1
Office of Wastewater Management
5
Office of Wetlands, Oceans and Watersheds
19
-------
What year did this program begin?
Frequency
Year
Frequency
Year
Frequency
Year
Frequency
Year
2
1985
1
1990
5
1995
4
2000
0
1986
3
1991
1
1996
6
2001
2
1987
2
1992
1
1997
2
2002
1
1988
2
1993
5
1998
5
2003
0
1989
5
1994
4
1999
3
2004
I am going to read EPA's 5 Strategic Goals, as listed in EPA's 2003-2008 Strategic Plan. Under
which of the following goals is this Voluntary Program structured?
Goal
# of Programs
1. Clean Air and Global Climate Change
24
2. Clean and Safe Water
7
3. Land Preservation and Restoration
4
4. Healthy Communities and Ecosystems
7
5. Compliance and Environmental Stewardship
12
Does this voluntary program impact any other Strategic Goal? As I read them all again, please
indicate whether or not this program impacts the Strategic Goal:
Primary
No
Don't
No
Impact
Impact
impact
Know
answer
Clean Air and Global Climate Change?
24
17
12
1
0
Clean and Safe Water?
7
21
25
1
0
Land Preservation and Restoration?
4
13
34
2
1
Healthy Communities and Ecosystems?
7
27
18
2
0
Compliance and Environmental Stewardship?
12
21
20
1
0
Does this program support a strategic objective within EPA's Strategic Plan?
53 Yes
1 No
Does the program have annual goals?
40
Yes
9
No
5
Other
20
-------
What environmental problem does the program address?
11
Human health
9
Air (other/none)
7
Water
4
Land
5
Ecosystem / community
16
Multi-media
18
Air (greenhouse gas reduction / climate change)
4
Other
Does this program help participants go beyond compliance with an existing Federal regulation?
27 Yes
24 No
3 Other
How do you define effectiveness for this voluntary program?
9 Dollars spent per environmental outcome
31 Environmental outcomes (no costs mentioned)
2 Dollars spent per program output
3 Outputs - GPRA / PART mentioned
24 Outputs - Participation rates
13 Other
Do you use data to track the environmental outcomes for this program?
50 Yes
4 No data at all
Where does the existing data (data collected by someone else) come from?
6
Participants
10
Other Federal agencies
12
Trade Associations / Industry sector
4
Other government organizations (States, regions, etc)
5
Existing EPA databases
10
Other
25
Not applicable
21
-------
On a scale of 1 to 4, where 1 is the least and 4 is the most, how reliable would you say the data you
use is?
0 1 the LEAST reliable
1 2
16 3
33 A the MOST reliable
4 Not applicable
Do you use this data to make changes to the program?
41 Yes
6 No
3 Other
4 Not applicable
How?
4
Changed data collection processes
8
Changed program goals
10
Changed target participants
16
Changed program services (to participants)
10
Not used to make changes to the program
8
Changed internal program operations
5
Other
4
Not applicable
What barriers exist to getting environmental outcome data?
14
Data submission can't be required
12
Cost of data collection / data analysis
4
Nature of environmental data / linking outcomes to programs
8
ICR issues / PRA
10
Sensitivity of data / proprietary
13
Getting accurate data
16
Other
4
None
I am going to read a list of 4 benefits that voluntary program participants can receive. We took this
list from OPEI's guidelines for designing voluntary programs. Please tell me whether your
program provides any of the following program benefits:
Yes
No
No
answer
Environmentally Preferable Product Designation
16
38
0
EPA/State Partnership opportunity
46
8
0
Public Recognition
46
7
1
Regulatory Flexibility
10
44
0
22
-------
We identified a few additional program benefits that participants can receive, not covered by
OPEI's guidelines. Please indicate whether this voluntary program offers the following three
benefits:
Yes
No
Don't Know
Technical Assistance
52
2
0
Information
52
2
0
Competitive Business Solutions, defined as helping
companies realize greater profits while achieving
environmental results
39
15
0
Does this program provide any other benefits to
participants?
What are these? (describe)
5 Grants
15 Networking
3 Increased public awareness of problem.
3 Intended program activity outcomes
19 Other
19 Not applicable
35
18
1
I am going to read a list of six categories and ask you to indicate whether the participants in this
program fall into any of the categories:
?...
Yes
No
No
answer
Would you say they are:
Industry or Business
48
6
0
Would you say they are:
Non-profit
41
13
0
Would you say they are:
State/Local Government
44
10
0
Would you say they are:
Individuals
20
34
0
Would you say they are:
Schools or Universities
36
18
0
Would you say they are:
Anything else
29
24
1
Who?
J
17 Other Federal agencies
5 Tribes
4 Trade Associations
4 International community
1 NGOs
2 Other EPA offices or regions
9 Other
25 Not applicable
23
-------
How is this program marketed to potential participants?
35 Website
40 Conferences: presentations, trainings, networking
19 Use partners to market or recruit
25 Advertising: print, radio, TV, brochures, literature, publications
6 Formal avenues: RFP, Federal Register, etc.
20 Direct / targeted outreach (including mailings and list-serves)
17 Word of mouth
9 Other
Do you set program participation goals?
30 Yes
22 No Why not?
1 Other
1 Not applicable
Why not?
7 Outcome focused program (outcome goal is not participation)
4 Program is full / at capacity with participants
4 EPA is not the sole actor in determining participation goals.
5 Participation is lower priority than other program management activities
3 Other
32 Not applicable
What type of commitment is required of participants?
21
Data
8
Self-defined plan / program
26
In line with program goals: reduction / outcome specific commitments
14
Mention of formal MOU-type agreement
3
Market / promote the program
9
Other
3
Not applicable
7
None
Participant results could potentially be counted twice if a participant is in more than one EPA
voluntary program. Do you track if your program participants are in other voluntary programs?
22 Yes
24 No
2 Other
6 Not applicable
24
-------
How do you determine if participant results are a product of your program and not the other(s)?
7
We don't know
27
No overlap with any other program
2
We have a process to do so - other/none
11
We have a process to do so - coordination
7
We have a process to do so - outcome attribution model
7
Other
6
Not applicable
How many FTEs are assigned to this program?
Total FTEs
364.70
0 n < 1 FTEs
12
Mean
6.75
1 n<2
13
Median
2.65
2 n < 5
14
>5 FTEs
15
This voluntary program may coordinate with other offices, programs or agencies. I'm going to
read a list of possible opportunities for coordination, and please indicate for each one if you do
coordinate with them, if you could coordinate with them but don't at this time, or if you don't
coordinate with them at all.
Could
Don't
Other
Don't
Do
but don't
know
Other EPA voluntary programs?
42
1
Other EPA regulatory programs?
30
1
23
0
0
Other Federal agencies?
44
5
3
2
0
Regions?
45
2
7
0
0
States?
44 I
I
0
Local agencies?
36
1
17
0
0
What is your annual budget in dollars?
$ 351,529,000 Total
$ 6,509,796 Mean
$ 492,500 Median
Does this funding come solely from your office?
40
Yes
11
No
3
Other
25
-------
Have you performed a cost-benefit analysis of this voluntary program?
24 Yes
27 No
2 Other
1 Not applicable
26
-------
Appendix C
Glossary of Terms
Basic Program Budget
The level of funds appropriated or allocated to a program for operation and
implementation.
Competitive Business Solutions
This is a partnership program benefit that helps participating companies realize greater
profits while achieving environmental results.
Environmentally Preferable Product Designation
This is a partnership program benefit. As defined in Section 201 of Executive Order
13101, this designation means products or services that "have a lesser or reduced effect
on human health and the environment when compared with competing products or
services that serve the same purpose."
EPA/State Partnership Opportunities
This is a partnership program benefit that provides the opportunity for a regulated or non-
regulated entity to work as a cooperative partner with the EPA or a State to solve an
environmental problem.
Fees Paid by Participants
Those fees paid by partnership program participants.
Formal Commitment Requirements
Partnership programs with formal commitment requirements may require participants to
sign a memorandum of understanding, and may even remove participants from the
partnership program if they fail to meet their agreed upon commitments.
Informal Commitment Requirements
Partnership programs with informal commitment requirements may ask participants to
submit data, participate in regular meetings, or meet environmental goals in non-specific
or non-binding ways.
27
-------
Information
This is a partnership program benefit, and includes the various publications, newsletters,
and website content provided by a partnership programs to make potential participants
aware of the environmental problem the program targets, target audience, program
benefits provided, and program participation requirements.
In-kind Contributions
This is a contribution of equipment, supplies, or other property in lieu of a dollar
contribution. Some organizations may also donate space or staff time as an in-kind
contribution.
Networking
This is a partnership program benefit that helps program participants establish a
relationship with other partnership program participants, EPA or State agency staff, or
other like-minded businesses or individuals who are willing to share best practices for the
mutual benefit of all those participating.
No Commitment Requirements
Partnership programs with no commitment requirements may simply ask participants to
achieve environmental outcomes in good faith, or may not formally enroll participants at
all.
Partnership Program
For the purpose of this evaluation, partnership programs are defined as, "A program
currently operated by a Headquarters program office that is national in scope, and that
works to encourage participants to voluntarily engage in environmentally beneficial
activities."
Program Benefits Provided to Participants
These are the various services and benefits provided to partnership program participants.
These include the various partnership program benefits defined in this glossary, as well as
many more unique benefits that have been developed and tailored for partnership
program participants.
Program Partners
Those organizations, individual citizens, or partnership program participants who take an
active role in helping to implement partnership programs by sharing program tasks, and
in some cases, by undertaking program management roles. These partners are sometimes
28
-------
included in the overall program participant count because they are described as an
integral part of the program, though they are not the primary target participant.
Public Recognition
This is a partnership program benefit that is provided to a program participant to publicly
recognize them for their efforts in a partnership program.
Regulatory Flexibility
This is a partnership program benefit that is provided to participants of some partnership
programs. This benefit allows for the development, review, and approval of an
alternative regulatory environmental standard that is at least as protective of human
health and the environment as the existing standard.
Technical Assistance
This is a partnership program benefit that provides best practices, technical
recommendations, and assistance in meeting regulatory requirements, exceeding
regulatory requirements, or in implementing activities that reduce the impacts of
environmental issues not covered by regulations.
Unique Program Benefits
These are types of partnership program benefits that do not fit into any specific category,
and which are unique to the program that offers them. The programs we interviewed
provided 19 different examples of these types of benefits, including such things as:
"improved quality of life;" "improved corporate environmental image;" and, "helping
individuals become engaged as stewards of their watershed."
29
-------
Appendix D
Agency Responses to the Draft Report
Comments from the Office of Policy. Economics, and Innovation
October 12, 2006
MEMORANDUM
Subject: NCEI Comment on Draft Evaluation Report
Diverse Partnership Programs May Expand EPA's Environmental Influence
Assignment Number: 2006-0093
From: Jay Benforado, Director National Center for Environmental Innovation
To: Jeffrey Harris, Director for Program Evaluation, Cross-Media Issues
We appreciate the opportunity to comment on the Draft report- EPA's Partnership Programs
May Expand Its Influence. Overall we feel that the content of the report provides a good
understanding of the Agency's Partnership Programs, and we look forward to using the
information to help us in our efforts to coordinate the Agency's Partnership Programs.
Comments on Next Steps
At a broad level, we endorse further evaluation as outlined in "next steps," but would appreciate
your consideration of the following:
1. EPA Partnership Program Strategic Management/Coordination
Work Completed and Under Way. As you assess the existing capacity
for conducting performance measurement (or the extent to which
programs actually measure performance), the analysis should also look
at the work that is under way to enhance that capacity over time. This
includes the recently released Guidelines for Measuring the
Performance of EPA Partnership Programs which can be found on the EPA intranet at
www.intranet.epa.gov/partners. and the training courses, Fundamentals of EPA
Partnership Program Design Training and Social Marketing Training that have been
well received. It is important to recognize that the snapshot at a particular moment may
not be an accurate picture of the longer term situation.
See OIG
Comments
in Appendix
E, Note 1
2. The important indirect impacts of EPA Partnership Programs. The
report focuses heavily on measures of performance, and the cost of
implementation. We think both of these are important to evaluate.
However, it is critical to recognize that the effects of partnership
programs are complex and sometimes indirect. It is therefore important
See OIG
Comments
in Appendix
E, Note 2
30
-------
not to look too narrowly at easily quantifiable benefits, or at benefits that are the
immediate result of program participation. To use just one example, the Performance
Track program reports "results" based on what its members report in terms of reduced
pollution or resource use. The long run value of the program to EPA, however, may lie
in less tangible benefits. For example, by highlighting and publicizing the achievements
of leaders, the program may influence the behavior of laggards (who can no longer claim
that they are doing everything possible when pressed by communities or stockholders).
Furthermore, by joining Performance Track the firm is making a public commitment that
would be more difficult to back away from in later years than would be the case in the
absence of the program. Lastly, many experts believe that capacity-building is an
essential pre-requisite for companies and other organizations to move "beyond
compliance". Performance Track plays a major role in organizational capacity building. It
is highly important to recognize that the mechanisms by which programs have their effect
vary greatly; again, this makes it important to fully appreciate the complexity of assessing
the "results" of those programs.
General Concerns with Draft Report
The inclusion of the grant-based National Non-point Source
Management Program skews data and analysis. Of all the programs you
analyze in your evaluation, this is the only one to our knowledge that is
primarily a grants-based program. By grants-based we mean that most of the
work and resources of the program is devoted to managing grants to other
organizations. We believe that grants based programs should not be included in this analysis
because:
a. While the program may involve "partnerships" it does not qualify as one of the
"EPA Partnership Programs" as defined by the IAC Partnership Programs
Workgroup. The definition used by this IAC Workgroup did not want to include
programs for which EPA does not have ultimate authority over (EPA supports but
does not fully control or have oversight over programs run by grantee
organizations. Therefore they should not be allowed to claim they are "EPA
Partnership Programs"). For this reason we do not plan to include it in our EPA
Partnership Program Accomplishments Report.
b. Because it is primarily a grants-based program, it represents an inherently
different policy approach than all the other EPA Partnership Programs surveyed
in the evaluation.
c. If this program must be included, we strongly recommend for the sake of
consistency and ensuring the integrity of your analysis, that all grants-based
programs involving partnerships be included. This would mean adding dozens if
not hundreds of other EPA grant-based programs that involve {
d. To include only this grants-based program in your evaluation
skews the otherwise highly valuable budget analysis in your
report substantially, since this program is nearly an order of
magnitude larger than most EPA Partnership Programs. Please
See OIG
Comments
in Appendix
E, Note 3
partnerships.
See OIG
Comments
in Appendix
E, Note 4
31
-------
see the attached list of grants-based stewardship programs that may need to be
included to ensure consistency if the National Non-point Source Management
Program is included in this report.
We believe the financial incentives used by EPA Partnership Programs need to be
mentioned in the report. While we believe grants programs should be excluded from
this report, we believe many of the most successful EPA Partnership Programs have
found creative and powerful ways to leverage financial incentives and relationships. We
are concerned this major incentive used by these programs was not conveyed in the
report. The terms "savings," "financial" or "monetary" do not even appear in the report.
Had the survey been written a bit differently (e.g., with a focus on the term "incentive" to
the target audience instead of the term "benefit"), your research may have found:
a. Many EPA Partnership Programs have successfully leveraged
the incentive of cost savings (AgStar, Combined Heat and
Power, Energy Star Product Labeling, Energy Star Homes,
GreenScapes, Labs 21, SmartWay Transport, WasteWise and
many others).
b. Some programs achieved substantial environmental results by convincing the
federal government and other major buyers to "spec" certain products or services,
yielding a significant financial incentive (Energy Star Office Equipment family of
programs, many EPA Partnership Programs linked to the USGBC LEED green
building standards set by many local and regional governments).
c. Some EPA Partnership Programs are leveraging financial value of selling a waste
product as an input to another company (Methane to Markets, WasteWise).
d. At least one EPA Partnership Programs is leveraging the financial incentive Wall
Street investors and analysts have on the firms they select and recommend firms
that own and operate "high performance buildings" that demonstrably yield
higher worker productivity (Energy Star Buildings/Commercial and Industrial).
e. Some EPA Partnership Programs are leveraging the incentive by shareholders and
company executives have to avoid financial risk (Climate Leaders, Hospitals for a
Healthy Environment).
f. At least one EPA Partnership Program is leveraging the financial influence that
major manufacturers have on their supplier networks (Green Suppliers Network).
Line Edits
Some specific comments on the text are provided below:
1. Title Page: The title language is a little confusing the way it currently
reads it is not clear whose influence is being expanded.
2. At a Glance, First Sentence: "Partnership Programs may expand
EPA's... " should be changed to "Partnership Programs have expand
EPA's...". There is evidence that these programs have allowed us to
See OIG
Comments
in Appendix
E, Note 5
See OIG
Comments
in Appendix
E, Note 6
See OIG
Comments
in Appendix
E, Note 7
32
-------
make environmental improvements in areas where we do not have statutory authority or
where a regulation would be impractical.
3. At a Glance, Middle of the Second Paragraph: "Partnership Programs
build on the Agency's traditional regulatory efforts, but are not intended
as alternatives to regulatory services." We suggest changing this
sentence to read "Partnership Programs... .efforts, but are not intended as
substitutes for regulations."
See OIG
Comments
in Appendix
E, Note 8
4. Page 2: It is unclear from the definition listed how grant programs fit
into the scope of this evaluation. The Partnership Programs Team does
not include grants programs in our analysis.
See OIG
Comments
in Appendix
E, Note 9
5. Page 3: Radon is listed as the oldest partnership program, this is a
program that we have not historically included in our analysis, we
question whether it truly belongs on this list.
See OIG
Comments
in Appendix
E, Note 10
6. Page 4: We would also suggest adding the fact that the ten largest
programs account for the majority of Agency investment in Partnership
Programs. Our data shows 7 programs account for two-thirds of the
spending on national partnership programs.
See OIG
Comments
in Appendix
E, Note 11
7. Page 6, Second Sentence: There appears to be some language
confusion, "Program Managers said (assert?) their programs...". Should
this not read "Program Managers said their programs.
See OIG
Comments
in Appendix
E, Note 12
Thank you for considering these comments and we look forward to an on-going collaboration.
33
-------
Comments from the Office of Solid Waste and Emergency Response
October 20, 2006
MEMORANDUM
SUBJECT: OSWER Response to OIG Draft Evaluation Report "Divers Partnership Programs
May Expand EPA's Environmental Influence"
FROM: Susan Parker Bodine
Assistant Administrator
TO: Bill A. Roderick
Acting Inspector General
Office of Inspector General
Thank you for the opportunity to comment on the subject draft
evaluation report. We offer the following comment: On page 7, you mention
niche issues, "including diverting electronic waste from landfills". Please note,
that the focus of Plug-in to eCy cling is to conserve resources, not divert wastes.
If you have any questions, please contact Thea McManus at (703) 308-:
See OIG
Comments
in Appendix
E, Note 13
34
-------
Appendix E
OIG's Comments on Agency's Response
1. As we continue to evaluate EPA's partnership programs, we will review all
appropriate Agency guidance and policy documents.
2. We agree that there are many possible outcomes that EPA can achieve through
implementing partnership programs. While some of those outcomes may be indirect,
we believe that it is important to assess and evaluate quantifiable partnership program
outcomes as they are able to be identified.
3. As noted in the Scope and Methodology section on Page 2,
"EPA's definition of partnership programs has continued to evolve during our
review. For the purpose of this evaluation, we define a partnership program as 'A
program currently operated by a Headquarters program office that is national in
scope, and that works to encourage participants to voluntarily engage in
environmentally beneficial activities.' In consultation with the OIG, each EPA
program office provided us with a list of partnership programs that met our
criteria."
The Office of Water's partnership program list included the National Non-Point
Source Program, so it was included in our census population.
4. In their comments, OPEI expressed concern that including the National Non-Point
Source Program "skews the otherwise highly valuable budget analysis in your report
substantially, since this program is nearly an order of magnitude larger than most
EPA Partnership Programs." However, if we were to remove the program from our
financial analysis, there would still 36 programs with budgets of $1 million or less,
two programs that still report no budget at all, and the median program budget with
this program removed would be $485,000, a difference of only $7,500.
5. Because the scope of our survey did not include collaboration - in particular financial
collaboration - with external partners, we did not collect the necessary information to
discuss such incentives in this report. However, we agree that this is an important
partnership program issue, and we plan to address it in future work.
6. We have changed the title of the report to, "Partnership Programs May Expand EPA's
Influence."
7. Because we did not independently verify the survey responses for accuracy (including
program outcomes), we cannot validate the results that each program claimed in their
survey responses.
35
-------
8. We have made this change to the At a Glance section.
9. As noted in Comment 3, each program office compiled their list of partnership
programs. While we recognize that OPEI considers the Non-Point Source Program to
be a grant program, it was included in the Office of Water's partnership program list
based on our criteria and was therefore included in our census population.
10. As noted in Comment 3, each program office compiled their list of partnership
programs. The Office of Air and Radiation's partnership program list included the
Radon program, so it was included in our census population.
11. We agree that a small number of programs receive the majority of partnership
program spending. However, the results of our survey do not support this specific
assertion.
12. We have made this change to Chapter 2.
13. We have changed this sentence to read as follows: "These programs address a range
of niche environmental issues, including diverting electronic waste from landfills, or
helping magnesium manufacturers reduce greenhouse gas emissions." This is based
on the program's verified response to the survey question, "What environmental
problem does the program address?" as follows:
• Resource Conservation and Land Preservation.
• Divert waste from landfills, including heavy metals. Reduce potential for
releases and exposures to hazardous substances.
• Use resources more efficiently through increased reuse and refurbishing of
reusable products. Increase recovery of materials and reuse of them in new
products.
We recognize that the program may address several environmental problems, and
determined that this particular portion of the response was a niche issue.
36
-------
Appendix F
Distribution
Office of the Administrator
Assistant Administrator, Office of Air and Radiation
Assistant Administrator, Office of Administration and Resources Management
Associate Administrator, Office of Policy, Economics and Innovation
Assistant Administrator, Office of Office of Prevention, Pesticides, and Toxic Substances
Assistant Administrator, Office of Research and Development
Assistant Administrator, Office of Solid Waste and Emergency Response
Assistant Administrator, Office of Water
Audit Follow-up Coordinator, Office of Air and Radiation
Audit Follow-up Coordinator, Office of Administration and Resources Management
Audit Follow-up Coordinator, Office of Prevention, Pesticides, and Toxic Substances
Audit Follow-up Coordinator, Office of Research and Development
Audit Follow-up Coordinator, Office of Solid Waste and Emergency Response
Audit Follow-up Coordinator, Office of Water
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Office of General Counsel
Acting Inspector General
37
------- |