Evaluating the Effectiveness of EPA's
Pollution Prevention Information
Network (PPIN) Grant Program
Fact Sheet
http://www.epa.gov/evaluate
For more information on
completed evaluations at
EPA or the Evaluation
Support Division, visit the
above link.
September 2008
Introduction
This evaluation focuses on the Pollution Prevention Information Network (PPIN), a grant
program started in 1997 to promote sharing of pollution prevention (P2) information, training,
and technical assistance expertise among states. The eight regional centers receiving these
grants are collectively known as the Pollution Prevention Resource Exchange (P2Rx), although
the centers also receive funding from other sources. P2Rx is managed by EPA's Office of
Pollution Prevention and Toxics, Pollution Prevention Division.
P2Rx is a national network of regional pollution prevention information centers (referred to as
"the centers") dedicated to increasing the adoption of pollution prevention by improving the
dissemination of relevant information. The centers provide pollution prevention information,
networking opportunities, and other services to states, local governments, technical assistance
providers, and businesses.
The purpose of the evaluation is to:
1) Assess the effectiveness of the PPIN Program in promoting the adoption of P2 opportunities.
2) Identify opportunities for creating a more effective interface with customers (both
intermediate customers (i.e., TAPs) and final customers (i.e., businesses and other polluting
entities).
3) Identify current efficiencies in information collection and dissemination as well as future
opportunities for improving efficiency.
4) Determine how P2Rx activities (outputs) can be linked to long-term environmental
outcomes (listed under the goals of the Government Performance and Results Act [GPRA]
and the PPIN's environmental outcomes).
Evaluation Questions
1. What model(s) do the P2Rx centers use to encourage exploration and adoption of P2
practices?
a. How are centers currently operating and what model(s) are they using to disseminate
information?
b. Which of the centers' model(s) is most efficient in information sharing and/or delivery?
c. When comparing among centers, what opportunities for centers' information delivery to
become more efficient are observed?
d. What opportunities are there for centers to become more efficient in information
delivery by sharing tasks?
-------
2. What groups/types of customers access P2Rx information?
a. How do the customers vary according to method of outreach by the centers?
• Via the Web site
• Via TAPs
• Via direct contact (e.g., telephone, correspondence, or other communication)
• Via P2Rx workshops, trainings, and regional meetings
3. What products, services, and content do businesses need to make a decision/consider the
adoption of P2 practices?
a. Are the centers providing the right products, services, and content to businesses to promote
the adoption of P2 practices?
4. What products, services, and content do TAPs need to promote businesses' adoption of P2 practices?
a. Are the centers providing the right products, services, and content to TAPs to promote the
adoption of P2 practices by businesses?
5. To what extent do businesses that use P2Rx centers change their practices or adopt P2 practices?
a. Do businesses use P2Rx information as a resource to solve pollution problems or to help
them adopt P2Rx practices?
b. What causes/influences businesses to seek P2Rx information?
6. Are the centers effective in reaching businesses through TAPs?
a. What do centers do to reach businesses through TAPs?
b. Can any outcomes (short-term, intermediate, or long-term) be attributed to information
received from P2Rx centers?
7. Are the centers effective in reaching businesses directly?
a. What do centers do to reach businesses directly?
b. Can any outcomes (short-term, intermediate, or long-term) be attributed to information
received from P2Rx centers?
8. What performance measures do P2Rx centers need to collect in order to demonstrate/ document
client behavior and other outcomes?
a. How would the centers that currently do not have the capacity to make/gather the desired
measurements build their capacity?
b. How can the information best be collected based on available technology, funds, and survey
approval requirements?
Evaluation Methods
Qualitative data were used to determine the structure and operation of individual P2Rx centers
and the types of P2Rx activities and outputs that are most important and useful to stakeholders
for enabling change. Qualitative data were also used to make observations about the perceived
effectiveness of the P2Rx centers in meeting their stated goals.
o Interviewed Center Directors (8), TAPs (7), and business representatives from all regions
(8)
o Conducted one focus group with TAPs(lO) and one focus group with business
representatives (5) from Regions 9 and 10;
o Conducted reviews of:
National P2Rx and center documents
• Grant reports (2005, 2006, 2007)
Needs assessments
Case studies
Page 2
-------
Quantitative data were used to examine which P2Rx resources and tools were used most
frequently, and with what audiences the P2Rx centers are communicating.
o National-level surveys, conducted online and in person in 2004 and 2007/2008; and
o Web Measures and Activity data reported twice annually by the centers.
Key Findings and Recommendations
Findings
Center Operation Models (Evaluation Question 1)
• The centers use a wide variety of models to fulfill the mission of P2Rx.
• The evaluation is inconclusive as to which single model may be most efficient but indicates that an
eight-center network is an effective method to reach the network's objectives.
• Considerable efficiency has been gained over the years in centralizing certain functions in one or two
centers.
• Examples include: the case study database, centralized TV support and Web support, P2
Programs Database, and the library.
P2Rx Information Users (Evaluation Question 2)
• The evaluation characterized the groups of customers that access P2Rx information, but was not able
to determine how the groups or types of customers vary according to the method of outreach by the
centers.
• Information that would characterize the users of the Web sites, TAPs, telephone or e-mail
correspondence, or face-to-face sessions, either does not exist or has not been collated.
Usefulness of P2Rx Products (Evaluation Questions 3 and 4)
• Businesses expressed three main priorities in their information needs:
• Up to date technical content, easy to find on the Web using search engines and organized by
process, topic or technology rather than by sector
• Information on cost of alternatives, viable substitutes, product-specific vendor information,
military prohibitions, and case studies; and
Services to help them with the cost and technical aspects of implementing P2, ways to address or
preclude regulatory requirements, and networking opportunities.
• TAPs expressed three main priorities in their information needs:
• Up-to-date information with working links;
• Help with marketing and "selling" P2 to their customers; and
• Assistance with measurement of P2 effectiveness and cost savings.
• The centers appear to be providing some of the right products, services and content to businesses to
promote P2, but the information is anecdotal.
Adoption of P2 Practices by Businesses (Evaluation Question 5)
• The evaluation provided an indication that businesses use P2Rx to change their awareness and
practices, leading to environmental outcomes.
• The causes for business to seek P2Rx information are the same as those that cause them to seek any
P2 information, i.e., cost savings, need for safer substitutes, responding to customer demand,
regulatory compliance, etc.
• The design of this evaluation was not able to isolate the affect of P2Rx on the target populations
from other factors that may influence the adoption of pollution prevention practices.
Outreach to Businesses Directly and Via TAPS (Evaluation Questions 6 & 7)
• The centers are targeting businesses to varying degrees, and for many centers the TAP/center
interaction has worked well to leverage resources and expertise to serve end users.
• The evaluation concludes that more effectively serve intermediate customers than end users for
several reasons:
• Not all centers have the resources to provide direct assistance to end users.
• Serving end users would appear to duplicate the work of the TAPs.
Page 3
-------
• The relationship between the centers and TAPs appears to be highly effective.
• Businesses see benefits in using P2Rx services, but that does not necessarily indicate that centers
should actively direct their efforts toward end users.
Performance Measurement (Evaluation Question 8)
• The centers' capabilities to measure performance vary, and additional capacity is needed to establish
a robust network-wide system of measurement.
• There exist many ways to improve the usefulness of existing measures, as well as additional
measures that can be implemented to better assess the centers' performance.
Recommendations
Center Operation Models (Evaluation Question 1)
• Continue to work together as a network of distinct programs, and share responsibilities and centralize
tasks where appropriate.
P2Rx Information Users (Evaluation Question 2)
• Use a consistent method of characterizing customers for all of P2Rx. Identification according to the
two groups used in the 2007/8 Behavior Survey is recommended, as it addresses how the customer
uses P2RX and not just who they are. If an additional level of detail is desired, affiliation ca be used
as subcategories.
• Characterize and analyze the identity of conference attendees. While the number of attendees is
captured for the P2Rx Activity Measures, the identity of these participants is not examined in a
comprehensive way.
• Require or offer registration for use of the site. This would provide the centers with information they
could use to identify their users and follow up with them. Consideration can be given to have
registration for discrete parts of the site rather than the entire site.
Usefulness of P2Rx Products (Evaluation Questions 3 and 4)
• Make better use of steering committees, advisory boards, boards of directors, and/or state or
regional roundtables or other stakeholder groups to solicit information about needs and whether
needs are being met by P2Rx. This should have two key components: (1) an attempt should be made
to do this more consistently across centers, possibly having a core set of questions that are asked
and an agreed-upon periodicity; and (2) questions should include those regarding effectiveness, not
just priorities and needs. This will primarily provide information from intermediate users.
Adoption of P2 Practices by Businesses (Evaluation Question 5)
• Continue to collect case studies and consider compiling and tabulating them across all centers.
• Perform follow up with the target audience to determine if approaches are effective at making
change happen.
• Collect outcome information as intrinsic part of the activity.
• Dedicate discrete resources to do follow up. The evaluation provided an indication that businesses
use P2Rx to change their awareness and practices, leading to environmental outcomes.
Outreach to Businesses Directly and Via TAPS (Evaluation Questions 6 & 7)
• Continue to focus on intermediate users rather than on end users. Work with end users as a
secondary audience as appropriate for each individual center.
• If Centers are interested in pursuing the end user, promote and market center resources beyond the
usual venues. For example, advertise in business journals or on business Websites, attend business
meetings and workshops.
Page 4
-------
Performance Measurement (Evaluation Question 8)
• Standardize use of pre- and post-test surveys at conferences and workshops for all centers
• Ask respondents to identify themselves by two groups, similar to what was used in the 2007/2008
Behavior Survey
=> Defines Customers
• In addition to asking questions about the effectiveness of the specific event (for center use), ask
general and standard questions about needs and recommendations for P2Rx as a whole
=> Short-Term Outcomes, Needs Assessment
• Have questionnaires be collected by EPA and analyzed.
=> Defines Customers, Short-Term Outcomes, Needs Assessment Network-wide
• Conduct follow up with attendees to record changes
=> Intermediate and Long-Term Outcomes
• Conduct a random assignment survey of a specified target population
• Note: May be used to provide a probability sample and would allow more statistically valid
inferences to be drawn from the survey. Would likely require an ICR.
=> Short-Term, Intermediate, Long-Term Outcomes
• Provide a resource to an established group of people (e.g., partners) that has agreed to provide
results information. For example, a particular information packet can be distributed to a partner
audience (e.g., schools in a state or district, businesses in a consortium) whose identities are
known. Develop boilerplate language for agreements.
• Care should be taken in this case to address potential bias, and ideally a control group would be
used of similar individuals who did not receive the resource.
• Survey the audience pre- and post-assistance
=> Short-Term, Intermediate, Long-Term Outcomes
• Require a commitment to measure as a precondition of any provision of services, as some centers
are already doing or considering
=> Long-Term Outcomes
• Perform follow up with all customers for whom contact information is known to gather outcome
information (e.g., Rapid Response and other telephone or e-mail based on-demand services).
Attempt to make this an intrinsic part of the activity for maximum resource efficiency.
=> Short-Term, Intermediate, Long-Term Outcomes
Contact(s)
• Beth Anderson, Office of Pollution Prevention and Toxic Substances, anderson.beth@epa.gov
• Yvonne M. Watson, Office of Policy, Economics and Innovation, Evaluation Support Division,
watson .yvonne@epa.gov
Report Link: http://www.epa.gov/evaluate/
Date Completed: May 2008
Page 5
------- |