United States
Environmental Protection
i Agency
2011 Evaluation of the
Oregon Paint
Stewardship
Program
Promoting Environmental
Results Through
Evaluation
-------
Acknowledgements
This evaluation was performed by Eastern Research Group, Inc. (ERG)
under subcontract to Industrial Economics, Incorporated (lEc) for the U.S.
Environmental Protection Agency's (EPA's) Office of Planning, Analysis,
and Accountability under Contract EP-W-07-028 between EPA and lEc.
The ERG evaluation team included Lou Nadeau, Ph.D., Amy Stillings, and
Lauren Jankovic. Matt Keene of EPA's Office of Policy, Evaluation Support
Division acted as the technical advisor.
The Evaluation Team is grateful to the continuous and stalwart
support of the members of the Paint Product Stewardship Initiative
(PPSI) Evaluation Committee. We are especially thankful to the invaluable
assistance provided by Abby Boudouris, Alison Keane, Scott Cassel,
Barry Elman, Chris Metzner, Hedrick Strickland, and Brielle Kissell.
Without them, this work would not have been possible.
This report was developed under the Program Evaluation Competition,
sponsored by EPA's Office of Policy. To access copies of this or other EPA
program evaluations, please go to www.epa.gov/evaluate.
Evaluation of the Oregon Paint Stewardship Program
-------
TableofContents
Executive Summary i - vi
1.0 Introduction 1
1.1 Paint Product Stewardship Initiative
Oregon Pilot Program 2
1.2 Evaluation of Pilot Program 4
2.0 Overview of the Evaluation Method
and Key Data Sources 6
2.1 Evaluation Approach 6
2.2 Communicating Information 8
2.3 Evaluation Design 9
2.4 Key Data Sources Used in the Evaluation .. 14
3.0 Collaboration 17
3.1 Data Sources, Methods and Analysis... 17
3.2 Findings 18
3.3 Learning 23
4.0 Paint Stewardship Organization 25
4.1 Data Sources, Methods and Analysis... 25
4.2 Findings 26
4.3 Learning 31
5.0 Education and Outreach 32
5.1 Data Sources, Methods and Analysis... 32
5.2 Findings 33
5.3 Learning 44
6.0 Consumer Purchasing Decisions 46
6.1 Data Sources, Methods and Analysis... 46
6.2 Findings 46
6.3 Learning 48
7.0 Collection of Post-Consumer Paint 50
7.1 Data Sources, Methods and Analysis... 50
7.2 Findings 50
7.3 Learning 56
8.0 Paint Reprocessing, Recycling, and Energy
Recovery 58
8.1 Data Sources, Methods and Analysis... 58
8.2 Findings 59
8.3 Learning 61
9.0 Household Hazardous Waste Programs ... 62
9.1 Data Sources, Methods and Analysis... 62
9.2 Findings 63
9.3 Learning 64
10.0 Cost Effectiveness 66
10.1 Data Sources, Methods and Analysis... 66
10.2 Findings 66
10.3 Learning 67
11.0 Waste Hierarchy
(to be completed by January 2012) 69
11.1 Data Sources, Methods and Analysis... 69
11.2 Findings and Learning
(From Other Questions) 70
12.0 Market for Post-Consumer Paint 71
12.1 Data Sources, Methods and Analysis... 71
12.2 Findings 71
12.3 Learning 74
13.0 Transferability 76
13.1 Data Sources, Methods and Analysis... 76
13.2 Findings and Learning 76
14.0 Unexpected Results 82
14.1 Data Sources, Methods and Analysis... 82
14.2 Findings and Learning 82
15.0 Reference Materials and Interviews 84
Promoting Environmental Results Through Evaluation
-------
Executive Summary
OVERVIEW
In 2002, a coalition of paint manufacturers,
represented by American Coatings Association (ACA);
local, state, and federal environmental agencies;
retailers; and consumer and environmental agencies
formed the Paint Product Stewardship Initiative (PPSI)
and began negotiations facilitated by the Product
Stewardship Institute (PSI) to create an industry-
managed post-consumer paint management system.
After seven years of negotiations, two Memoranda
of Understanding (MOUs), and comprehensive
PPSI-sponsored research of the paint industry and
consumers, the state of Oregon became the first to
enact a law establishing product stewardship as the
preferred method for reducing the environmental
impacts and costs associated with leftover paint. This
report summarizes the results of the evaluation of
the Oregon paint recycling program performed by the
PPSI Evaluation Committee, which includes members
representing the diversity of participants in the PPSI.
The Oregon program, depicted in Figure ES-1, is
comprised of a diversity of interconnected systems,
actors, and processes. The major components of
the Oregon program include the Paint Stewardship
Organization (PSO; PaintCare), the oversight of the
Oregon Department of Environmental Quality (DEQ),
the paint market (producers, retailers, and consumers),
and the leftover paint management system (collection
sites, processing, etc.). To fund the program, Oregon
added an assessment fee onto the price of paint
sold in the state. Full documentation of the details
of the program appears in PaintCare's Oregon Paint
Stewardship Pilot Program Plan (PaintCare, 2010).
The evaluation addressed twelve questions and
the information collected and presented is extensive
and accessible in multiple formats to increase the
evaluation's utility for a diversity of audiences. As
a companion to this report, the PPSI Evaluation
Committee developed a website to organize and
present the results from this evaluation - www.
paintstewardshipprogram.com. The web site presents
Figure ES-1: The Oregon Paint Recycling Program
Paint Management
System
Evaluation of the Oregon Paint Stewardship Program
-------
the key findings and learning from this evaluation in a
dynamic, visual, and interactive manner that includes
links to information and data sources used in this
report. This executive summary presents some basic
findings and learning from the evaluation for each of
the 12 evaluation questions (see text box).
The evaluation was conducted between October
2009 and September 2011 by the PPSI Evaluation
Team, a subgroup of the Evaluation Committee, which
consisted of the U.S. Environmental Protection Agency
(EPA) and its subcontractor Eastern Research Group,
Inc. (ERG)1 in partnership with academic researchers
from Duke University, Georgia State University, and
Tufts University. While integrating measurement and
evaluation (M&E) into the design and implementation
of the Oregon pilot program, the Evaluation Committee
took a participatory approach to its work, emphasizing
transparency and communications while incorporating
aspects of developmental and participatory evaluation
into the evaluation design to account for program
complexity.
The Evaluation Team relied on three primary
sources of data and information:
Two surveys of consumers, one conducted in
August 2010 just after the program began and
another conducted in July 2011, provide key
information for Evaluation Questions 3 and 4
and data for several other questions.
In-person interviews and other personal
communications with program stakeholders:
the Evaluation Team conducted 21 interviews
during the course of the evaluation; including
seven interviews with HHW representatives,
three interviews with retailers, and 11
interviews with key program stakeholders.
Documents developed by PaintCare: the
Evaluation Team drew heavily on the 2010
Oregon Paint Stewardship Pilot Program Plan
(PaintCare, 2010) and the 2011 Annual Report
(PaintCare, 2011b) to provide information on
program design, implementation, and results.
1 ERG's work was performed under subcontract to Industrial Economics,
Inc. (IEC) under contract EP-W-07-028 between EPA and lEc.
1. Collaboration: To what degree was the pilot program,
from planning to implementation, a collaborative
process?
2. Paint Stewardship Organization: Describe the PSO,
including its funding mechanism and infrastructure.
3. Education and Outreach: How did education
materials and strategies affect consumer awareness
and behavior? Which messages were most effective
with which target audiences? What materials/strategies
were developed and what were the goals and target
audience of those materials/strategies? Did other factors
besides the program influence consumer behavior and
awareness? What are the lessons learned?
4. Consumer Purchasing Decisions: How has the
program affected consumers' purchasing decisions and
management of paint prior to drop-off at paint recycling
facilities? How did the fee assessment affect consumer
behavior?
5. Collection of Post-Consumer Paint: How has the
program affected the collection of post-consumer paint
in terms of volume, cost, environment, convenience,
and infrastructure? What other factors have affected the
amount of leftover paint?
6. Paint Reprocessing, Recycling, and Energy
Recovery: How has the program affected used paint
reprocessing, paint recycling, and paint-related energy
recovery in terms of volume, infrastructure, and cost?
7. Household Hazardous Waste Programs: What was
the impact of the program on the HHW facilities in terms
of the types of paint collected, costs, and the way in
which the facilities operate?
8. Cost Effectiveness: How cost effective is the program?
9. Waste Hierarchy: How was the program designed
and implemented to move consumers up the waste
hierarchy? With respect to moving customers up the
waste hierarchy, what were the program's obstacles,
opportunities, and decisions?
10. Market for Post-Consumer Paint: How has the
market for post-consumer paint been affected by the
program? What aspects of the program have had
an impact on the market and how? What market and
products represent potential opportunities for post-
consumer paint products?
11. Transferability: Based on the Oregon experience,
what implementation and outcome-related information
is required for other states to develop and implement
leftover paint management systems? To what extent are
the performance measurement and evaluation systems
transferable to other states? What are the best ways to
communicate the results of the evaluation?
12. Unexpected Results: During the program and for
each of its primary components, what were the primary
external, unexpected and/or unintended influences and
consequences?
Promoting Environmental Results Through Evaluation
-------
We draw on information derived from these three
sources, as well as a number of other sources tailored
to specific evaluation questions.2 Data sources, data
collection instruments, the evaluation methodology
and other relevant documentation can be accessed
atwww.paintstewardshipprogram.com.
COLLABORATION
A primary goal of the PPSI was that the pilot
program, from planning to implementation, was
to be a collaborative process. Most stakeholders
(84 percent) involved in the PPSI agreed that the
program development process was collaborative
(Braunz et al., 2010). Stakeholders also pointed to
break downs in collaboration when the program
transitioned from designing and planning the PPSI
pilot program to drafting Oregon legislation and
planning and implementing the Oregon program.
The level of importance and function of collaboration
changed through the different stages of the program's
development. To account for changes, process
facilitators can better maintain appropriate levels
of collaboration throughout the process by, at the
beginning, coming to agreement on and documenting
clear expectations for collaboration's role in each stage
of the process.
*
PAINT STEWARDSHIP ORGANIZATION
A goal of the PPSI was for a pilot program to create
a Paint Stewardship Organization (PSO) that would
operate under the direction of the paint industry
and this was achieved in the Oregon legislation that
created the program. The PSO for the Oregon program,
PaintCare, is a coalition of paint producers and is
operated by the American Coatings Association (ACA),
although membership in ACA is not required for a
paint producer to be part of PaintCare. PaintCare is
responsible for implementing and running the program
in Oregon, a responsibility which they contracted out
to Product Care Inc., which has run other stewardship
programs. The PSO built the infrastructure (e.g., paint
collection sites, logistics, and transportation) using
the existing infrastructure in the Portland metro area.
Having this existing infrastructure offered significant
advantages in implementing this program.
Though generally perceived as clearly denned
and complete, the lack of detailed, accessible cost
information reduced the transparency of the funding
mechanism. Few Oregon residents (11 percent) that
purchased paint in 2011 were aware that a fee was
added to the cost of paint to pay for the program
(PaintCare, 2 01 la).
jg±. EDUCATION AND OUTREACH
The Oregon program included an extensive
education and outreach program which was another
PPSI goal. The campaign consisted of a website, point
of sale materials, radio and television advertising,
press releases, direct mailings to contractors and
retailers, an 800 number, and participation in trade
shows. The education and outreach campaign used
a broad approach to spreading its messages and did
not strategically aim its messages at specific target
audiences (consumers with the most paint, contractors,
retailers, etc.) with specific means of communication
(TV, website, social media, radio, etc.). Consumers
of paint can be segmented into various groups
(new versus long-time homeowners, homeowners
versus contractors, age and other demographics).
Opportunities to improve the effectiveness of education
include identifying the desired changes in behavior
that will accomplish program goals, designing specific
messages to achieve those changes, (e.g., aligning
the messages with the program's priorities on the
waste hierarchy), prioritizing efforts allocated to
specific messages, and more explicitly targeting those
messages to the diversity of intended audiences (e.g.,
age groups, location, consumer vs. contractor). Overall,
there was little evidence that education and outreach
influenced consumer behavior. More useful evidence
of the effectiveness of education initiatives requires
more complete and transparent information such as
data related to consumer exposure to messaging and
consumer intentions and practices related to paint
management (e.g. purchasing, reuse, recycling, disposal).
2 For example, Evaluation Question 1 draws on a survey conducted by Duke University graduate students and Evaluation Question 2 draws on a set of
interviews conducted by Georgia State University graduate students.
Evaluation of the Oregon Paint Stewardship Program
-------
Collections by type of paint
Latex
352,136
75%
residents (92 percent) felt this type of program was
"somewhat important" or "very important." Oregon
residents' acceptance of the fee may be linked to the
inherent value they place on the goals and services
provided by recycling programs. The degree to
which consumers value such programs can be used
as an indicator of their willingness to pay a certain
fee. This information may be used in refining fee
structures within and across states. Documenting and
communicating consumer attitudes may also be useful
in navigating the legislative process necessary to create
programs similar to Oregon's.
469,665 total gallons collected
^ CONSUMER PURCHASING DECISIONS
A goal of the program was to encourage consumers
to purchase the correct amount of paint thereby
reducing the amount of leftover paint that must be
managed. Consumers indicated that the program
had little impact on their purchasing decisions. Most
Oregon residents (93 percent) that purchased paint
under the program indicated that the program had no
effect on the amount of paint they purchased. PaintCare
developed a "paint calculator" to help consumers
determine the amount of paint that was needed for
painting projects, but few consumers were aware
of the tool and very few actually used it. Overall, the
program focused less on the reduction of leftover paint
and more on recycling leftover paint. To focus planning
and implementation on reducing the generation of
post-consumer architectural paint, paint management
programs should more explicitly document the relative
emphasis they place on aligning the paint management
system with the existing waste hierarchy (reduce, reuse,
recycle). A clear articulation of program priorities will
provide a framework to organize and deploy outreach
strategies, messages, materials, and effort.
In terms of the fee charged, 73 percent of Oregon
residents felt that the fee was reasonable. Most
COLLECTION OF POST-CONSUMER
PAINT
The largest component of the program involved
collecting and processing leftover paint in Oregon.
In the first year of the program, PaintCare collected
469,665 gallons of paint. Of this total, 352,136 gallons
(75 percent) were latex paint and 117,529 gallons (25
percent) were oil-based paint (PaintCare, 2011b). As
of September 1,2011, the program had 98 collection
sites with 10 of these sites offering paint exchange. Sites
are open to the public, on average, six days a week for
a total of 58 hours per week (PaintCare, 2011b; Em2).
Most Oregon residents (92 percent) live within a 15
mile drive of a paint drop off site (Strickland, 2011).
PAINT REPROCESSING, RECYCLING,
AND ENERGY RECOVERY
In its first year, potential end-points for paint
collected by the program included: recycling into
another paint product, recycling as a non-paint product,
energy recovery (oil-based paint), appropriate disposal
and direct reuse by consumers. Table ES-1 summarizes
the volume of paint diverted to each end-point.
^ HOUSEHOLD HAZARDOUS WASTE
(HHW) PROGRAMS
HHWs are a key stakeholder in the paint
management system. Prior to the program, HHWs
would take in paint from consumers and process it
for disposal. Under the program, HHWs collected
and stored paint for eventual pick up by PaintCare's
transportation contractor. At the start of the program,
Promoting Environmental Results Through Evaluation
-------
Table ES-1: Summary of Paint Endpoints
Endpoint
Latex (Percent of Total)
Recycling to paint and paint reuse
211,281 (60%)
Oil-Based (Percent of Total)
3,526 (3%)
Reprocessing into non-paint product
28,171 (8%)
Energy recovery
14,085(4%)
114,003(97%)
Disposal
98,598 (28%)
Totals
352,136
117,529
the amount of paint that HHWs collected increased
and then settled back to pre-program levels. The
initial bump in collections is attributed to consumers
bringing in older paint stored in their homes. During
the program, the proportion of latex paint collected
by HHWs increased relative to oil-based paint. Some
HHWs reported cost-savings resulting from avoided
labor, disposal, and transportation costs associated with
handling oil-based and unusable latex paints.
COST EFFECTIVENESS
The program processed paint at $7.03 per gallon
(PaintCare, 2 Ollb), although that amount excludes
costs incurred by HHWs to take in and handle the paint
at their facilities. The $7.03 per gallon value was lower
than other estimates from other sources, although
the other estimates were not purely comparable. The
use of "processing cost per gallon" as a measure of
cost-effectiveness is limiting. A more comprehensive
measure would translate the gallons into environmental
benefits and then also include other management
options (e.g., reuse) as well as the cost-effectiveness of
reducing leftover paint by increasing the proportion of
consumers that "buy the right amount."
WASTE HIERARCHY
The Evaluation Committee is still assessing how
the program was designed and implemented to move
consumers up the waste hierarchy. The Committee has
partnered with a graduate student at Tufts University
to investigate this evaluation question and results will
be reported January 2012. The method for answering
this question will involve relating the components of
the program to the categories of the waste hierarchy
(reduce, reuse, and recycling) and then compiling
information on the obstacles, opportunities, decisions,
and relative emphasis related to each category of the
waste hierarchy.
MARKET FOR POST-CONSUMER PAINT
To improve the management of leftover paint, the
PPSI determined that the pilot program should explore
means to expand the market for post-consumer paint
products. If more consumers used post-consumer
paint, then they would generate less waste because less
new paint would be purchased (and ultimately need
to be disposed). About half of the total volume of paint
collected (217,157 gallons) under the program made
it into the post-consumer paint market. For latex paint
collected by the program, 60 percent was available to
the post-consumer paint market. Very little oil-based
paint (3 percent) was available to the post-consumer
paint market.
*
TRANSFERABILITY
The Evaluation Committee identified several types
of information that other states will need to collect
in order to implement paint stewardship programs,
including: volume data on paint, information on current
infrastructure, a system map, cost information, and
information on consumer awareness of infrastructure
and consumer behavior and attitudes. The Committee
also identified the aspects of the performance
measurement system and evaluation that are
Evaluation of the Oregon Paint Stewardship Program
-------
transferable to other states, which included: the pilot
program's evaluation questions (appropriately adapted
to other states), matrix of performance measures and
a web-based program model for communications
with diverse audiences (see Figure ES-1). Oregon and
other states can more systematically and consistently
learn, improve and communicate about paint
recycling programs and other product stewardship
initiatives by: 1) using (and adapting) the program
evaluation framework and methodology designed
and implemented by the PPSI Evaluation Committee,
and 2) collaborating to adopt common fundamental
frameworks for ongoing performance management
such as those developed by the Conservation Measures
Partnership's (CMP's) Open Standards for the Practice
of Conservation.3
UNEXPECTED RESULTS
The Evaluation Committee identified some
key unintended effects and unexpected scenarios
that significantly influenced program planning and
implementation. Highlights include: (1) collaboration
took more time, effort and resources than expected and
levels of collaboration fluctuated significantly at different
stages in the process; (2) paint legislation was vetoed
twice in Minnesota which delayed implementation of
a program by more than a year-and-a-half; eventually
leading to implementation in Oregon; (3) though the
program was originally intended to be voluntary,
legislation was necessary to implement the program;4
(4) the PPSI had limited representation from retailers
leading up to the program, but retailers became a
critical component of Oregon paint collection; (5) the
misalignment of the program's goals and the waste
hierarchy model (reduce, reuse, recycle); and (6) that
retailers serving as collection locations had a negative
impact on the program's ability to divert high quality
leftover paint for reuse.
3 www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_Version_2.0.pdf.
4 Legislation was needed to exempt the paint producers from anti-trust and collusion concerns. At the start, the idea of the program was to avoid having
states pass legislation or rules to tell the private sector what to do - the program was supposed to be voluntary.
Promoting Environmental Results Through Evaluation
-------
i.o Introduction
Over the last several decades, paint manufacturers
have dramatically reduced the environmental impacts
of leftover paint, including the reduction of toxic
ingredients found in leftover paint. Paint manufacturers
have also improved the performance of latex paints
to the point where they now constitute 80 percent of
paint sales, replacing a large portion of the market
for oil-based paints, which have more significant
environmental impacts (PaintCare, 2010). Additionally,
the paint industry works with retailers to educate
consumers on how to more accurately estimate the
amount of paint consumers need, thereby reducing the
quantities of leftover paint.
Despite these improvements, environmental
impacts from leftover paint remain significant.
Households generate some 75 million gallons of leftover
paint each year, which amounts to about 10 percent of
the amount of paint they purchase (Abt, 2007). When
disposed in landfills in liquid form, leftover latex paints
can contaminate groundwater, thereby harming fish and
other aquatic life. Furthermore, leftover oil-based paint
is considered hazardous waste due to its combustibility
and high solvent content. Managing leftover paint is
also costly to local governments; paint is the largest
volume of waste collected by household hazardous
waste (HHW) programs.5 The estimated cost to manage
leftover paint is more than $8 per gallon (SCS and
Cascadia, 2007).
While leftover paint, as currently managed, is
associated with negative environmental impacts, it
also represents an opportunity for reducing the life
cycle environmental impacts of paint production.
When manufacturers use leftover paint in their paint
production processes, the environmental impacts
associated with making paintincluding materials
extraction, processing, and end-of-life management
are reduced. By recycling paint containers,
manufacturers can further reduce the life-cycle impacts
of paint products (PSI, 2004a).
The movement towards product stewardship for
post-consumer paint began in 2002, when, due to the
high cost of managing leftover paint, a number of states
began to consider mandating paint management. In
2002, a coalition of paint manufacturers, represented
by American Coatings Association (ACA);6 local, state,
and federal environmental agencies; retailers; and
consumer and environmental agencies formed the
Paint Product Stewardship Initiative (PPSI) and began
negotiations facilitated by the Product Stewardship
Institute (PSI) to create an industry-managed post-
consumer paint management system. After seven years
of negotiations, two Memoranda of Understanding
(MOUs), and comprehensive PPSI-sponsored research
of the paint industry and consumers, Oregon became
5 Oregon Department of Environmental Quality, Paint Product Stewardship: www.deq.state.or.us/lq/sw/prodstewardship/paint.htm.
6 Formerly National Paint and Coatings Association (NPCA).
Evaluation of the Oregon Paint Stewardship Program
-------
the first state to enact a law establishing product
stewardship as the preferred method for reducing
the environmental impacts and costs associated with
leftover paint.
An early focus of the PPSI was in identifying a state
where a pilot project would be implemented. PPSI
identified Minnesota as a candidate for a pilot and it
became the focus of PPSI efforts. The PPSI developed
a work plan for implementing a pilot project in
Minnesota, focusing the work plan around the following
six goals (PPSI, 2008):7
Goal 1: The pilot project, from planning
to implementation, is a collaborative and
cooperative process.
Goal 2: Establish a paint stewardship
organization (PSO), which operates under the
direction of the paint industry.
Goal 3: Consumers (including painting
contractors) generate no or less waste paint
and containers.
Goal 4: The statewide post-consumer paint
management system should be designed to
ensure that it is environmentally beneficial,
economical, and convenient. With these
considerations, the system should strive to
use methods highest on the following waste
management hierarchy: reuse, recycling (into
paint or other products), energy recovery
(generally applicable to oil-based paint), and
proper disposal.
Goal 5: Identify cost-effective alternatives
for using post-consumer paint products and
explore means to expand the market for
products containing post-consumer paint.
Goal 6: Measure and evaluate the performance
of the pilot project, and ensure the results and
learning that the evaluation generates are
transferable and relevant to the rollout of a
national post-consumer paint management system.
PPSI formed an Evaluation Committee (EC) to
be the lead on Goal 6 and to evaluate the extent to
which all six goals were achieved. The Minnesota pilot
project, however, was never implemented, as legislation
required to implement the program was vetoed by
the Minnesota governor at that time. Nevertheless,
much of the work performed in designing a program
in Minnesota was carried into the Oregon program,
including the six goals described above. These six goals
formed the basis of the evaluation questions developed
under this project and for the design of an evaluation of
the Oregon program.
1.1 Paint Product
Stewardship Initiative
Oregon Pilot Program
As part of the broader dialogue, PPSI began
designing a pilot program to demonstrate the
effectiveness of an industry-led paint stewardship
program and sought states considering paint
stewardship program legislation to implement the
program. In July 2009, Oregon became the first state
to enact a paint product stewardship law which
directs manufacturers of paint sold in the state to
set-up and run a statewide system for the collection of
post-consumer latex and oil-based paint. ACA formed
PaintCare, Inc. (PaintCare), as the PSO responsible for
implementing the pilot program in Oregon. The Oregon
law required PaintCare to submit an implementation
plan to the Oregon Department of Environmental
Quality (DEQ) to be approved before the start of the
pilot program. The Oregon Paint Stewardship Pilot
Program Plan (PaintCare, 2010) was released for public
comment in June 2010. DEQ approved the plan at the end
of June, allowing the program to launch on July 1, 2010.
The Oregon program is comprised of a diversity
of interconnected systems, actors, and processes. The
major components of the Oregon program include the
PSO (PaintCare), the oversight of Oregon DEQ, the paint
market, and the leftover paint management system.
Figure 1 illustrates the pilot program and its primary
components and is discussed in more detail in Section
2.2. Full documentation of the details of the program
appear in PaintCare's Oregon Paint Stewardship Pilot
Program Plan (PaintCare, 2010).
Components of the Oregon pilot program include:
PPSI: Conceptualization of the pilot
program began with PPSI. Its goals included
collaboration amongst diverse participants to
7 Available at www.deq.state.or.us/lq/pubs/docs/sw/PaintProdStewardshipPilotPlan2010June.pdf.
Promoting Environmental Results Through Evaluation
-------
plan, implement, and evaluate the performance
of a statewide pilot to inform rollout of similar
programs in other states nationwide.
Program Administration: PaintCare and
the Oregon DEQ are responsible for program
administration. PaintCare is the industry non-
profit association formed as the PSO by ACA.
Oregon DEQ is the state agency charged with
approving the PSO program plan, including
the recovery fee, and responsible for program
oversight and enforcement. PaintCare receives
its funding from the paint market and uses
funding for administration, supporting DEQ,
running the paint management system,
and providing education and outreach to
consumers and Oregon residents.
Paint Market: The paint market consists of
three main participants:
Producers of paint sold in Oregon are
obligated under the law to participate
in a paint stewardship program.
Program participants are required
to register and provide a list of paint
brands manufactured. Remittance
of recovery fee payments on all
architectural paint sold in Oregon is a
responsibility of the producer.
Figure 1 - Graphic model of the Oregon Paint Stewardship Pilot Program
Retailers of paint sold in Oregon are
required to collect the recovery fee
from consumers at the time of sale.
Retailers pay the producer the fee
as part of the invoice for the paint
offered for sale (i.e., retailers pay the
producers). Additionally retailers can
volunteer to be collection sites for
leftover paint. They also collect the fee
from consumers and then pass that fee
back to producers.
Consumers of paint eligible (i.e.,
leftover architectural paint in 5
gallon containers or less) for the
paint management system can be
residential purchasers, trade painters,
institutions, and businesses.
Paint Management System: PaintCare funds
the system and has coordinated with retailers,
HHWs, and other groups to collect paint and
set up vendors for the collection, exchange,
transportation, consolidation, and reprocessing
of leftover paint.
Evaluation of the Oregon Paint Stewardship Program
-------
1.2 Evaluation of the
Pilot Program
A key element of the PPSI MOU was the inclusion
of an evaluation of the pilot program to inform the
development of programs in other states and improve
the Oregon program. The work plan developed for the
Minnesota demonstration project included six goals
for pilot program implementation (see text box on
page 5). The Oregon law required the PSO to report
certain performance measures to DEQ (e.g., volume
of paint collected; program costs) by September 1,
2011. In turn, Oregon DEQ was required to report to
the Oregon Legislature results from the program and
recommendations for improvement by October 1,2011.
The PPSI formed an Oregon pilot program
Evaluation Committee composed of industry
representatives, Oregon stakeholders, the U.S.
Environmental Protection Agency (EPA), and rollout
states (i.e., states working with PPSI to also enact
paint product stewardship legislation) in September
2009. As the evaluation results provide valuable
information for implementing post-consumer
management systems for various consumer products,
EPA took the lead and provided contractor support.
EPA facilitated the Evaluation Committee's work by
managing an "Evaluation Team" that included EPA and
its contractors, PSI, and academic partners.
The role of the Evaluation Committee was similar
to that of a technical working group; the members
of the Evaluation Committee contributed technical
expertise and guidance by participating on conference
calls and through review of the work performed by
the Evaluation Team. In late 2009 and into 2010, the
Evaluation Committee focused on providing input on
the Evaluation Team's development of the evaluation
methodology.8 During the fall of 2010 through the
summer of 2011, the Evaluation Team implemented
the evaluation methodology. This report documents the
results of the evaluation and should be considered a
companion document to PaintCare's 2011 Oregon Pilot
Program Annual Report (PaintCare, 2011b), which was
published to meet its reporting obligations.9 This report
is intended to achieve PPSI Goal 6 and provide input to
Oregon DEQ, as well as present independent evaluation
findings and learning for interested parties.
This report focuses on providing the results of
12 evaluation questions. Chapter 2 provides a brief
overview of the evaluation method, while subsequent
sections focus on individual evaluation questions. In
addition to this report, the Evaluation Committee also
created a web-based model of the Oregon program
to present the Evaluation Committee's work, which is
found at www.paintstewardshipprogram.com.
8 Available at http://paintstewardshipprogram.com/images/pe_evaluation_methodology_10_14_10.DOC.
9 Available at http://paintstewardshipprogram.com/images/2011-09-01 _Oregon PaintStewardship Pilot Program Annual Report.pdf.
Promoting Environmental Results Through Evaluation
-------
Pilot Program Goals (PPSI, 2008)
Goal 1: The pilot project, from planning to implementation, is a collaborative and cooperative process.
Goal 2: Establish a paint stewardship organization, which operates under the direction of the paint industry.
Goal 3: Consumers (including painting contractors) generate no or less waste paint and containers.
Goal 4: The statewide post-consumer paint management system should be designed to ensure that it is
environmentally beneficial, economical, and convenient. With these considerations, the system
should strive to use methods highest on the following waste management hierarchy: reuse, recycling
(into paint or other products), energy recovery (generally applicable to oil-based paint), and proper
disposal.
Goal 5: Identify cost-effective alternatives for using post-consumer paint products and explore means to
expand the market for products containing post-consumer paint.
Goal 6: Measure and evaluate the performance of the pilot project, and ensure the results and learning that
the evaluation generates are transferable and relevant to the rollout of a national post-consumer
paint management system.
Evaluation Committee Members
Abby Boudouris
Oregon Department of Environmental Quality
Tom Metzner
Connecticut Department of Environmental Protection
Scott Cassel
Product Stewardship Institute
Lou Nadeau
Eastern Research Group, Inc.
Cynthia Dunn
California Department of Resources Recycling and
Recovery
Jim Quinn
Metro Parks and Environmental Services Department
Barry Elman
Environmental Protection Agency
Steve Sides
American Coatings Association
Jen Holliday
Chittenden County, Vermont Solid Waste District
Amy Stillings
Eastern Research Group, Inc.
Lauren Jankovic
Eastern Research Group, Inc.
Theresa Stiner
Iowa Department of Natural Resources
Alison Keane
American Coatings Association
Emily Wang
California Department of Resources Recycling
and Recovery
Matt Keene
Environmental Protection Agency
Robert Wendoll
Dunn-Edwards Paint
Scott Klag
Metro Solid Waste and Recycling
Jennifer Nash
Harvard University
Evaluation of the Oregon Paint Stewardship Program
-------
2.o Overview of the Evaluation
Method and Key Data Sources
To account for the new and rapidly evolving
Oregon pilot, this evaluation incorporated aspects
of developmental evaluation. This emerging field of
evaluation involves designing and implementing an
evaluation in which the program changes over time to
accommodate a complex environment. The evaluation
is "developmental" in the sense that it matures over
time to meet the changing needs of the program
(Patton, 2010). Like the PPSI, the Evaluation Committee
took a participatory approach to its work, emphasizing
transparency and communications and integrating
measurement and evaluation (M&E) into the design and
implementation of the Oregon pilot.
2.1 Evaluation Approach
Program evaluators have long advocated the
consideration of measurement and evaluation in the
design of programs. By considering the questions
that various audiences will ask in the future about
the program, participants are better aware of the
baseline data that will be necessary to assess program
performance later. In theory, this leads to more efficient
organizational learning and better management of
program performance as programs evolve and mature.
The Oregon pilot program integrated measurement
and evaluation into program design through a series of
iterative and overlapping steps (see Figure 2):10
Described the program. First the Evaluation
Team engaged the Evaluation Committee
to clarify the program's mission, goals, and
objectives. The Evaluation Team used that
information to develop a preliminary logic
model, an illustration of how the program
is envisioned to work, that links program
activities and outputs to key audiences and
outcomes. For many evaluations, the logic
model is sufficient to communicate the theory
of how the program is expected to work; in this
case, the traditional logic model did not fully
capture the interaction between the program,
evaluation, and stakeholders. The Evaluation
Team, with the help of a graphic designer,
developed a more flexible, web-based version of
the logic model (see Section 2.2 for more details).
Developed evaluation questions. Working
with the Evaluation Committee, the Evaluation
Team drafted questions based on the pilot
program's goals and objectives. The questions
were designed to account for program context,
key audiences, communication of results,
and use of the evaluation findings. Part of the
process included the Evaluation Committee's
prioritization of evaluation questions.
Developed measures. The Evaluation
Team identified a set of measures for each
evaluation question. For each measure, the
team documented potential data sources,
developed collection methods and strategies,
identified the tools of analysis, and suggested
the strategies for data collection and data
management (see Section 2.3). This approach
resulted in a set of measures that are highly
specific to one or two of the evaluation
questions in design, but may produce
information that is relevant to several
questions. For example, the measures of
collaboration between program participants
during implementation (Evaluation Question 1)
provides information on the transferability
10 For a more detailed discussion of this topic, see Integrating Evaluation into Program Design (Keene, 2008). Available at: www.productstewardship.us/
displaycommon.cfm?an=1&subarticlenbr=754.
Promoting Environmental Results Through Evaluation
-------
Figure 2 - Iterative Steps of Integrating Evaluation into Program Design
1. Evaluation Policy
2. Evaluation
Methodology
1. Team
2. Mission
3. Goals &
Objectives
4. Logic Model
Questions
1. Context
2. Audience
3. Communication
4. Use
1. Data Sources
2. Collection Methods
& Strategy
3. Analysis Tools
4. Data Collection
5. Data Management
of the program to other states (Evaluation
Question 11) and to the unexpected results of
collaborative work (Evaluation Question 12).
Documented the process and results.
The methodology documents the decisions
with respect to evaluation questions,
measures, and communication strategies.
Note that the process is iterative; the final
stage of documentation feeds into program
implementation and the program management
cycle. This report documents the Evaluation
Committee's methods, data sources, findings,
and key learning points with respect to each
evaluation question.
Integrating evaluation into the design of the
Oregon pilot program also makes the evaluation more
amenable to a participatory approach. A participatory
evaluation can be defined as "a partnership between
trained evaluation personnel and practice-based
decision makers, organization members with program
responsibility or people with a vital interest in the
program" (Cousins and Earl, 1992). By working with
program designers, personnel, and stakeholders to
incorporate evaluation into the program design, the
program includes these individuals in the evaluation
process from the start.
This approach ensures that the evaluation results
are "accessible and important to users and responsive
to their needs while maintaining sufficient technical
quality" (Cousins and Earl, 1992). The evaluation
stakeholders are involved in the creation of evaluation
questions, data sources, and analysis, which in turn
ensures that the findings of the evaluation are open
to stakeholders and the public. In other words,
stakeholders and the public have access to data that are
collected and analyzed by the Evaluation Committee, as
Evaluation of the Oregon Paint Stewardship Program
-------
well as the learning derived from the analysis, and are
free to make their own conclusions about the findings
of the analysis.
The goal of a participatory approach, when
compared to traditional evaluation methods, is to
encourage participant ownership of the work and the
results of the evaluation. This ownership of the results
by the program participants fuels organizational
learning, decision making, and improvement (Cousins
and Whitmore, 1998). A participatory approach
allowed the Evaluation Team to distribute the costs of
the evaluation among multiple organizations and tap
into specialized skills and other resources to improve
the quality of the evaluation findings.
For example, the Evaluation Committee invited
several academic researchers to become "partners" in
the evaluation:
Amy Braunz, Julie Colvin, Whitney Knapp,
and Hedrick Strickland, graduate students
at Duke University's Nicholas School of
the Environment completing their masters
degrees in environmental management,
implemented a survey to address Evaluation
Question 1 (collaboration). These students
were participating in Dr. Randy Kramer's
course in survey methods, in which students
are required to develop and implement
a survey. The students coordinated with
Indiana University faculty member Anne
Marie Thompson, a leading expert in the
measurement of collaboration.
Wes Bledsoe, Eric Graves, and Andrei Roman,
graduate students at Georgia State University
(GSU) participating in Dr. Cynthia Searcy's
applied capstone course in evaluation
performed work under Evaluation Question 2.
Hedrick Strickland, one of the Duke University
graduate students mentioned above, based
her master's thesis on the application of
geographic information systems (CIS) analysis
to the Oregon project, including descriptive
information, visualization, and analysis
relevant to the evaluation. Ms. Strickland
addressed questions of the convenience and
efficiency of the infrastructure of the program
for consumers and transportation.
Oregon Pilot Program Stakeholders
American Coatings Association*
U.S. Environmental Protection Agency*
Household Hazardous Waste Facility
Operators*
Oregon Department of Environmental
Quality*
Oregon Legislature
Paint Producers*
Paint Retailers
Product Stewardship Community*
Roll-out States*
* Represented on the Evaluation Committee
Brielle Kissell, a graduate student at Tufts
University, based her masters thesis on the
exploration of Evaluation Question 9, which
relates to the waste hierarchy.
There are limitations to using a participatory
approach to evaluation. Inviting multiple participants
and stakeholders into an evaluation increases logistical
challenges. In a traditional approach, an external,
impartial evaluator controls the timeline and method of
the evaluation as well as the presentation of the results.
In a participatory approach, the evaluator coordinates
with a diversity of stakeholders and other participants
in the evaluation process, as well as other parties with
a direct interest in the evaluation results. Coordination
of multiple parties and reconciliation of competing
viewpoints and priorities can be time-consuming.
2.2 Communicating
Information
The complexity of the program and the
participatory approach to the evaluation demanded
an innovative method to communicate about the
evaluation to multiple, diverse audiences throughout
program design and implementation. Logic models can
be an important tool for describing a program's theory
Promoting Environmental Results Through Evaluation
-------
and assumptions, designing evaluation questions and
methods, and convening and communicating with
stakeholders about program and policy effectiveness.
The development and use of logic models, however,
is often limited to the initial stages of the evaluation
process and to clearly defined programs. Generally, logic
models can be described as linear, bounded, relatively
static, and rarely designed for communications
with the general public. Trends in performance
management in the public sector are pointing toward
greater transparency and accountability, systems
thinking, participation and empowerment of diverse
stakeholders, adaptive management, evidence-based
practice, and better collaboration and communications
with a wider range of audiences.
To complement the participatory nature of the PPSI
and the evaluation, the Evaluation Team developed a
web-based "fuzzy" logic model (see Figure 1) with the
intent of expanding the accessibility and use of the
evaluation (and program) and the evaluation process
to a greater diversity of stakeholders over a longer
period of time. A fuzzy logic model embraces fluid
and approximate reasoning and varied context and
assumptions with the aim of improving the capacity
of models of program theory to navigate non-linearity,
feedback loops, adaptive agents and other agents and
other traits common in the lifecycle of environmental
programs and policies. Integrating Web 2.0, graphic
design, systems theory, and data visualization with
traditional logic models gives the evaluator the capacity
to embed an unlimited type and quantity of content into
a Web-based model of the program.
The model of the PPSI's evaluation of the Oregon
Paint Stewardship Pilot Program, found at www.
paintstewardshipprogram.com (referred to as the
"evaluation website" in this report), is designed to be
used in tandem with this report. It describes both the
key elements of the pilot program (e.g., infrastructure,
funding) as well as the evaluation. For example,
evaluation questions are illustrated as numbered
paint "splatters" on the part of program that they are
evaluating. Clicking on an evaluation question brings
up a pop-up box that provides additional information
about that evaluation question, such as the question
and purpose, measures, data sources and methods
used to answer the question, key findings, and Web
links to supporting materials and resources (e.g.,
reports and data).
2.3 Evaluation Design
The Evaluation Team's evaluation methodology
provides a detailed discussion of the process used to
develop, prioritize, and refine the set of evaluation
questions, as well as a description of the development
of the measures and data sources for each question.
Table 1 lists the evaluation design matrix developed
for the Oregon pilot program evaluation. The table
presents each evaluation question, the measures used
to answer the question, and the data sources and
analytical methods used to develop the measures. Each
question is provided with its own section (beginning
with Section 3.0) in this report. Each question-specific
section begins with an overview of the data sources,
methods, and analysis used to answer the question and
then lists findings and learning points. In the context
of each question and given the findings, the learning
sections aim to answer the following three questions:
How do the findings affect the program?
What do the findings tell us?
How do the findings affect the program's
capacity to achieve its goals? What is the impact
on the program being able to achieve its goals?
What are the opportunities to improve
program performance for both the Oregon
program and other programs in the future?
Each the evaluation questions covers a diverse
and broad set of concepts. Thus, when presenting the
results in the sections that follow, we use the above
considerations as a guide, but we also adapt our
presentation according to the information in hand.
Evaluation of the Oregon Paint Stewardship Program
-------
Table 1 - Evaluation Design Matrix
Evaluation
1 . To what degree was
the pilot program,
from planning to
implementation,
a collaborative
process?
2. Describe the PSO,
including its funding
mechanism and
infrastructure.
Key
Audiences
Measures
EPA
PSI
Roll-out states
Manufacturers
and Retailers
PPSI
Oregon
Legislature
Oregon DEO
PPSI
Roll-out states
PSI
Retailers
Product
stewardship
community
Governance scale
based on the questions
provided in Table 1 of
Thompson, Perry, and
Miller (2007)
Administration scale
based on the questions
provided in Table 1 of
Thompson, Perry, and
Miller (2007)
Autonomy scale based
on the questions
provided in Table 1 of
Thompson, Perry, and
Miller (2007)
Mutuality scale based
on the questions
provided in Table 1 of
Thompson, Perry, and
Miller (2007)
Norms scale based on
the questions provided
in Table 1 of Thompson,
Perry, and Miller (2007)
Connectedness
measures
Narrative of PSO
development and
operation
Factors affecting
infrastructure choices
GIS representation
of infrastructure in
relation to demographic
information
Clarity
Transparency
Completeness
Data F
Interview
or survey
of those
involved in
process
Survey of
network
participants
(program
organizations)
PSO
documents;
Interviews
with Oregon
DEO staff,
PaintCare
staff
One-time
analysis
Baseline,
follow-up at 1 2
to 1 8 months
Within 6 months
of the program
start
Promoting Environmental Results Through Evaluation 10
-------
Evaluation Key »*,«,«...««. Data c^«. .«..
Question Audiences Measures Sources Frequency
3. How did education
materials and
strategies affect
consumer
awareness and
behavior?
4. How has the
program affected
consumers'
purchasing
decisions and
management of
paint prior to drop-
off at paint recycling
facilities?
Oregon DEQ
Oregon
Legislature
HHWand
other paint
collectors
PPSI
Roll-out states
Product
stewardship
community
Oregon DEQ
Retailers
Manufacturers
PPSI
Roll-out states
Product
stewardship
community
Program awareness
Program scope
awareness
Site location awareness
Improper disposal/
handling environmental
impact awareness
Consumer use of best
practices for purchasing
paint (percentage)
Consumer-reported
leftover paint amount
Consumer use of
leftover paint
Disposal practices
Storage practices
Recycling practices
Awareness of pilot
program education and
outreach messages.
Consumer use of best
practices for purchasing
paint
Consumer-reported
leftover paint amount
Disposal practices
Storage practices
Recycling practices
Fee awareness
Behavioral reaction to fee
Consumer
surveys,
focus
groups, or
interviews
Consumer
surveys,
focus
groups, or
interviews
Baseline,
follow-up at 12
to 1 8 months
Baseline,
follow-up at 12
to 1 8 months
Evaluation of the Oregon Paint Stewardship Program
-------
Evaluation Key
Question Audiences
5. How has the
program affected
the collection of
post-consumer
paint in terms
of volume,
cost, quality,
environment,
convenience, and
infrastructure?
6. How has the
program affected
used paint
reprocessing,
paint recycling,
and paint-related
energy recovery in
terms of volume,
infrastructure, and
cost?
PPSI
Oregon DEQ
Oregon
Legislature
Roll-out states
Retailers
Product
stewardship
community
PPSI
Oregon DEQ
Oregon
Legislature
Roll-out states
Retailers
Product
stewardship
community
Measures
Gallons collected
Containers collected
Cost per gallon
Reductions in amounts
of paint that could
impact the environment
Average distance
traveled to drop-off
point
Total number of hours
open
Average distance to
drop-off locations
Number of permanent
collection sites
Number of periodic
collection opportunities
Number of sites offering
paint exchange
Changes in retailers'
practices
Gallons per trip
Miles per trip
Transportation cost per
gallon transported
Volume of latex paint
exchanged
Volume of latex paint
recycled into paint
products
Volume of latex paint
recycled into non-paint
products.
Sources Frequency
Data
collected and
tracked by
the program
Surveys or
anecdotal
evidence
at drop-off
locations to
ask about
distance
traveled
Data
collected and
tracked by
the program
Data
collected and
tracked by
the program
and Oregon
DEQ prior to
program
Data
collected and
tracked by
the program
Interviews
with retailers
Data
collected and
tracked by
the program
Data
collected and
tracked by
the program
Baseline,
monthly
Annually or
more frequently
if possible
Based on
the volume
measures being
collected
Baseline,
follow-up
Annually or
more frequently
if possible
Baseline.
Annually or
more frequently
if possible.
Promoting Environmental Results Through Evaluation 12
-------
Evaluation Key »*,«,«...««. Data c^«. .«..
Question Audiences Measures Sources Frequency
6. How has the
program affected
used paint
reprocessing,
paint recycling,
and paint-related
energy recovery in
terms of volume,
infrastructure, and
cost?
7. What was the
impact of the
program on the
HHW facilities in
terms of the types
of paint collected,
costs, and the
way in which the
facilities operate?
8. How cost effective
is the program?
9. How was the
program designed
and implemented to
move consumers up
the waste hierarchy?
PPSI
Oregon DEQ
Oregon
Legislature
Roll-out states
Retailers
Product
stewardship
community
PPSI
Roll-out states
Oregon local
governments
PPSI
Oregon DEQ
Roll-out states
Product
stewardship
community
PPSI
Roll-out states
EPA
Retailers
Product
stewardship
community
Volume of latex paint
appropriately disposed
in landfill
Volume of oil paint
exchanged
Volume of oil paint
recycled into paint
products
Volume of oil paint
appropriately disposed
Number of facilities
capable of performing
each type of
management method
Capacity of facilities
performing each type of
management method
Processing cost
per gallon for each
management method
Gallons of different
types of paint collected
at HHW facilities
Changes in the amounts
of or capacities for other
products by HHW
Cost for HHW facilities
to take in and process
paint
Cost of managing other
products
Cost per gallon
collected at drop-off
facilities
Cost per gallon
exchanged, recycled, or
used for energy
Total program cost per
gallon collected
Evaluation Committee
assessment of hierarchy
attainment
Amounts of paint
processed in the program
that fall into each
category
Data
collected and
tracked by
the program
Data
collected and
tracked by
the program
Interviews
with HHW
facilities
Data
collected
and tracked
by the
program and
participants
Program
materials and
interviews
with program
staff
Data
collected and
tracked by the
program and
participants
Baseline.
Annually or
more frequently
if possible.
Annually or
more frequently
if possible
Annually or
more frequently
if possible
Collected after
the program has
run for some
time
Collected after
the program has
run for some
time
13 Evaluation of the Oregon Paint Stewardship Program
-------
Evaluation Key
Question Audiences
1 0. How has the
market for post-
consumer paint
been affected by
the program?
1 1 . Based on
the Oregon
experience, what
implementation
and outcome-
related information
is required for
other states to
develop and
implement leftover
paint management
systems?
12. During the
program and for
each of its primary
components,
what were the
primary external,
unexpected and/
or unintended
influences and
consequences?
PPSI
Retailers
Manufacturers
PPSI
Roll-out states
EPA
PSI
Product
stewardship
community
PSI
Roll-out states
Product
stewardship
community
Measures
Different products
offered
Number of facilities
(outlets) offering post-
consumer paint.
Total sales (dollars) of
post-consumer paint
products.
Total amount of paint
distributed for re-use
from HHW sites
Description of
implementation and
outcome-related
information that are
required for states
Transferability to other
states
Best ways to
communicate results of
the evaluation
Lists of external
influences and
outcomes
Sources Frequency
Interviews
with Oregon
DEO,
PaintCare,
HHW
programs
Interviews
with program
staff.
Qualitative
assessment
by Evaluation
Team
Interviews,
all available
data
Annual
Within 12
months of
program
implementation
Collected after
the program has
run for some
time
2.4 Key Data Sources Used
in the Evaluation
The evaluation team relied on many data sources
provided by our evaluation partners, the PPSI, and PSI,
with the primary sources including:
Two surveys of consumers
In-person interviews and other personal
communications with program stakeholders
Documents developed by PaintCare (the PSO)
11 For example, Evaluation Question 1 draws on a survey conducted by Duke University graduate students and Evaluation Question 2 draws on a set of interviews
conducted by Georgia State University graduate students. Both of those sources of information are discussed with their respective evaluation questions.
Throughout our findings on the evaluation
questions, we draw on information derived from
these three sources. Where other data or information
is used in answering an evaluation question,11 we
describe that information in our discussion of the
evaluation question.
Promoting Environmental Results Through Evaluation 14
-------
2.4.1 Consumer Surveys
Two surveys of consumers, one conducted in
2010 just after program rollout (Bradshaw, 2010) and
another conducted nearly one year later (PaintCare,
2011b), provide key information for Evaluation
Questions 3 and 4 and supplementary data for other
questions. Bradshaw Advertising conducted the first
survey, a telephone survey of 409 Oregon residents
from August 4 to 10, 2010, for PaintCare.12 We refer
to this survey as the "August 2010 survey" throughout
this report. The survey was conducted using random
sampling that resulted in a margin of error of 4.8
percent at the 95 percent confidence level. The survey
targeted having at least 80 percent of respondents in
the 25 to 64 age range with no more than 10 percent in
either the 18 to 24 or 65 or older age ranges. Overall,
the survey focused on consumer awareness of the
newly implemented program; specifically, the survey
asked consumers about:
How and where they would likely dispose of
unneeded paint.
The extent to which they are aware of the
program.
How they learned about the program.
The importance of this type of program.
The reasonableness of the fee.
How likely they are to recycle paint.
The second survey was a web-based survey of
1,064 Oregon residents over 18 years old, conducted
from July 7 to 12, 2011 by PaintCare through the ACA
paint consumers panel. We refer to this as the "July
2011 survey" throughout this report. The July 2011
survey was conducted using random sampling with
a margin of error of 3 percent. This survey targeted
individuals who had purchased paint between July 1,
2010 and June 30, 2011, as well as individuals who
paint as an occupation. This survey included questions
to consumers about:
Whether or not they recall hearing or seeing
advertisements for the program.
Where they saw or heard the advertisements.
Whether or not they are aware of the fee.
If and where they have purchased paint in the
past year.
The impact of the fee on their paint purchase.
The types of products covered by the program.
Awareness of leftover paint drop-off locations
and the distance to the location from their
home.
Perceived convenience of the drop-off locations.
Whether or not they had recalled seeing and
using a paint calculator to purchase paint.
The influence of program information on the
amount of paint purchased.
How they managed their leftover paint from
their most recent project.
2.4.2 Interviews and Other
Personal Communications
The evaluation team conducted 21 interviews
during the course of the evaluation; including seven
interviews with HHW representatives, three interviews
with retailers, and 11 interviews with key program
stakeholders. Additionally, the team followed up
with many of the interviewees for clarification of key
points and data and received clarifying responses in a
number of email correspondences. Each interview had
a different focus:
The HHW interviews focused on the impact
of the program on the management of HHW
programs, facilities, and collection events in
terms of paint volume collected, costs to collect
paint, and ability to collect other HHW products.
12 Full survey results are available at www.productstewardship.us/displaycommon.cfm?an=1&subarticlenbr=754.
15 Evaluation of the Oregon Paint Stewardship Program
-------
The interviews with retailers explored the
impact of the program on retailers with a focus
on the process of implementing the program,
the decision to act as a collection site or not,
and key challenges faced.
Interviews with program stakeholders did
not follow a set pattern and were often
customized to the interviewee based on
the timing of the interview. For example,
the evaluation team asked representatives
of the PSO about challenges faced during
program implementation, lessons learned, and
availability of data.
A full list of the interviews and personal
communications that were used in this report appear
in Section 15.0. When citing a specific interview in the
text we used an "S" followed by the interview number.
Similarly, we use an "Em" followed by a number to cite
an email correspondence.13
2.4.3 Paint Stewardship
Organization Documents
Two PSO documents provided integral information
about the program: the 2010 Oregon Paint Stewardship
Pilot Program Plan (PaintCare, 2010) and the 2011
Annual Report (PaintCare, 2011b). The Program Plan
provided the following information about the program:
Oregon paint stewardship legislation.
Stewardship organization and participation
of producers.
Program products.
Registration of producers and brands.
Product sold and available for collection.
Program budget and funding.
Outreach and education.
Collection system.
Environmental regulatory requirements.
Transportation.
Sound management methods.
Program reporting and audit.
The Annual Report provided the following types of
data:
The methods used to collect, transport, recycle
and process post-consumer architectural paint
in the state.
The volume and type of post-consumer
architectural paint collected in all regions of
the state.
The volume of post-consumer architectural
paint collected in the state by method of
disposition, including reuse.
An independent financial audit of the program.
A description of program costs.
An evaluation of the operation of the program's
funding mechanism.
Samples of education materials, evaluation of
the methods used to disseminate materials,
and assessment of the effectiveness of the
materials.
An analysis of the environmental costs and
benefits of collecting and recycling latex paint.
13 See Section 15.0.
Promoting Environmental Results Through Evaluation 16
-------
3.o Collaboration
EVALUATION QUESTION:
To what degree was the pilot program, from planning
to implementation, a collaborative process?
How was the collaborative process viewed by different groups involved in
the process?
What tools and strategies (including communication) were used to foster
collaboration and how effective were those tools?
PPSI's first goal for the pilot program was for
the program to be collaborative and cooperative,
to potentially serve as a model of governance that
EPA, states, and product stewardship initiatives
can use to achieve environmental and human
health policy objectives. A key point to understand
about collaboration under this program is that the
collaborative effort started well before the Oregon
program was developed. PSI was involved in developing
a national dialogue about paint stewardship starting
in 2002. In fact, PSI intended the entire process to be
collaborative from start to finish (S19).
3.1 Data Sources, Methods
and Analysis
Collaboration may be defined as "a process in
which autonomous or semi-autonomous actors
interact through formal and informal negotiation,
jointly creating rules and structures governing their
relationships and ways to act or decide on the issues
that brought them together; it is a process involving
shared norms and mutually beneficial interactions"
(Thompson et al., 2007). Thompson et al. (2007)
developed a set of five dimensions of collaboration,
which framed the Evaluation Committee's approach to
data collection, analysis and interpretation:
Governance: The degree to which there is
joint decision-making by the collaborative
parties about rules to govern the relationship.
Administration: The extent to which a
structure that moves the collaboration from
governance to action exists.
Organizational autonomy: The extent to
which each party to the collaboration retains
autonomy in the relationship.
Mutuality: The extent to which parties to the
collaboration experience mutually beneficial
interdependencies.
Norms: The extent to which there is reciprocity
and trust between the collaborating parties.
Thompson etal. (2007) developed these scales
through a systematic review of the literature,
synthesis of theoretical definitions of collaboration,
a series of interviews with organizational directors
on collaboration, and detailed case study research.
They identified a set of 10 to 12 indicators for each
dimension and constructed a survey question to
measure each indicator; each indicator was phrased as
17 Evaluation of the Oregon Paint Stewardship Program
-------
a statement and the survey respondents were asked to
rate the extent to which their organizations engage in
each behavior. The rating was designed as a scale from
1 ("not at all") to 7 ("to a great extent").
The Evaluation Team partnered with a team of
graduate students at Duke University's Nicholas School
of the Environment as part of their capstone course in
survey methods to help answer Evaluation Question 1.
Their primary mode of data collection was a Web-based
survey of PPSI participants. The survey results were
summarized in a report (Braunz et al., 2010).
Braunz et al. (2010) used the Thomson et al. (2007)
framework to develop a survey instrument to ask about
collaboration under the PPSI. The web survey targeted
current and former PPSI participants, including those
involved before the start of the Oregon program. The
survey's 23 questions focused on:
Demographics, including organization type,
role, and length of time involved.
Communication tools used and communication
tools preferred.
Level of participation.
Opinions about the collaboration
(i.e., the dimensions of collaboration).
Before implementing the full-scale survey, the
Duke students conducted an informal focus group
and performed ten pre-tests to gather feedback and
clarification on the survey instrument.
The questionnaire was distributed to 419 PPSI
participants in November 2010 and was completed by
125 participants (30 percent response rate) (Braunz
et al, 2010). In order to determine stakeholder
perceptions, survey respondents were grouped as
(see Figure 3): 1) local government (33 respondents);
2) state and federal government (23 respondents); 3)
private sector, which combines retailers, manufacturers
and other private enterprises (23 respondents); and
4) nongovernmental organization (NGO)/Non-profit/
University/Trade Associations (8 respondents).
After the Duke University graduate students
completed the survey, the Evaluation Team conducted a
series of interviews with key stakeholders to assess the
Figure 3 - Distribution of Respondents by Category to
Duke University Survey of PPSI Stakeholders
Local Government
State Government
Private Sector
NGOs / Non-Profits
/ University/Trade
Associations
Other
collaborative process. The Evaluation Team conducted
interviews with representatives of:
PSI
ACA and PaintCare
Oregon DEQ
HHWs14
The purpose of the interviews was to provide
a history of the collaborative effort and to get the
opinions and feedback from key stakeholders that were
involved in the process from start to finish.
3.2 Findings
In presenting the findings for this question, we
will first "tell the story" of collaboration under the
program, and then turn to providing an assessment of
the collaborative effort.
14 The interviews that were conducted with HHWs were primarily focused on providing information for Evaluation Question 7; however, the information
provided by HHW representatives during those interviews was applicable to this (and other) questions.
Promoting Environmental Results Through Evaluation 18
-------
3.2.1 What tools and strategies
(including communication) were
used to foster collaboration & how
effective were those tools?
Tools and Strategies Used
Prior to the PPSI, PSI, state, and local governments
identified leftover paint as one of their top five waste
management issues. As a result, a number of states
began to consider implementing mandatory leftover
paint management requirements. PSI interviewed
stakeholders during 2002 to 2004 to define the
problem and identify basic facts in order to start a
national dialogue. Its efforts resulted in the publication
of A Background Report for the National Dialogue on
Paint Product Stewardship (March 2004) to provide a
technical foundation for requirements (PSI, 2004a). At
the same time, PSI also released Product Stewardship
Action Plan for Leftover Paint (March 2004), which
outlined the key issues and potential solutions related
to leftover paint management (PSI, 2004b). Funding
for this initial work came from a variety of sources,
including most state and local governments involved in
the PPSI. "Seed money" from Massachusetts Office of
Energy and Environmental Affairs, an EPA grant for one
meeting, and a foundation grant to partially support
another meeting (S19).
In December 2003, PSI facilitated the first of four
in-person dialogue meetings to establish relationships
and build trust among stakeholders. An outcome
of these meetings was to establish workgroups to
focus on specific issues (e.g., markets for recycled
paint, education efforts). Industry and government
stakeholders volunteered to lead some of these work
groups and provide funding to support the group's
efforts (S19).
Throughout the process, PSI acted as a facilitating
organization, supported by contributions from
stakeholders after the initial grants were expended.
PSI's philosophy for facilitation was based on
collaboration and consensus and hence was interwoven
through all aspects of the Initiative (S19). PSI's strategy
to foster collaboration, implicitly adopted by PPSI early
in the process, involved:
Establishing the PPSI goals through a
consensus process (S19).
Developing a detailed work plan (PPSI, 2008)
during the first year to 1) determine major
tasks and timelines, 2) identify potential pilot
opportunities, 3) identify staffing and funding
requirements, 4) identify challenges and
barriers, and 5) draft agreement language.
Crafting MOUs to define purposes and
scopes of projects; goals and objectives; and
commitments by stakeholders (PPSI, 2007).
Establishing smaller groups including15
1) a Steering Committee, 2) an Education
Workgroup, 3) an Infrastructure Workgroup,
4) Market Workgroup, 5) Life Cycle Benefits
and Costs Committee, 6) Demonstration Project
Committee, and 7) Evaluation Committee, to
work on specific projects.
Providing agenda and background materials
prior to meetings and calls and providing
summaries of meeting and calls, allowing PPSI
participants the opportunity to comment (S19).
In implementing this approach, PSI relied primarily
on the following three tools (S19):
Monthly PPSI briefing conference calls.
PPSI conferences (once or twice a year).
A website featuring status of projects, reports,
and posting of agenda, participant lists,
meeting/call presentations, and meeting/call
minutes (S19).
The PPSI did not formally document the strategy
and tools the PPSI would use to foster collaboration, but
the intent to support collaboration is set out in several
key documents. The second MOU, signed October 24,
2007, established the agreement with participants
that appears in the text box on the next page. Although
15 These groups were not given decision-making authority (S19).
19 Evaluation of the Oregon Paint Stewardship Program
-------
some of the items listed were not completed, the MOU
did establish a framework for continued participation
in collaborative efforts. The framework was continued
through the development of a draft work plan for the
Minnesota Demonstration project. The work plan
described the consensus among participants on the
mission, method, goals and objectives of the project while
establishing the roles of those participants (PPSI, 2008).
PSI's role in the process was not to serve as an
independent facilitator, but rather to act as an objective
facilitator and, at times, act as an advocate for paint
stewardship. Program stakeholders agreed that PSI
provided a vital role in the sharing of information and
identifying state contacts (S16, S17, S18, S19, and
S20). Additionally, one stakeholder felt that PSI's lack
of impartiality was a benefit to the process since PSI
worked actively to push the process along (S18).
Effectiveness of Tools/Strategies
The survey of PPSI participants (Braunz et al,
2010) asked respondents to rate the effectiveness of
the periodic briefing calls and conferences. The results
from the survey, summarized in Figure 4, indicate that
PPSI participants viewed both the briefing calls and the
conferences as being effective.
"The ...PPSI participants agree to work
collaboratively over the next three years
to do the following:
Continue the dialogue through regular meeting
and workgroup conference calls.
Design, implement, and evaluate a
demonstration project.
Pursue a voluntary multi-stakeholder approach
for the demonstration project while addressing
mechanisms to avoid anti-trust implications.
Support and implement strategies to effectively
change consumer behavior.
Pursue completion of a life cycle assessment
and cost benefit analysis (LCA/CBA) project.
Collect baseline data on aerosol paints."
PPSI, 2007
Figure 4 - PPSI Participants' Perceived Effectiveness of Briefing Calls and Conferences
Briefing
Calls
Conferences
10 20 30 40 50 60 70
Effective Neither Effective nor Ineffective Ineffective
80
Promoting Environmental Results Through Evaluation 20
-------
3.2.2 To what degree was the
pilot program, from planning to
implementation, a collaborative
process? How was the
collaborative process viewed by
different groups involved in the
process?
Overall Impressions of the Process
To a large extent, PPSI participants agreed that
the process was collaborative (Braunz et al, 2010) (see
Figure 5) with 84 percent of respondents to the survey
agreeing that the process was collaborative (19 percent
strongly agreeing) and only 6 percent disagreeing that
the process was collaborative.
The survey implemented by Duke asked the PPSI
participants to rate five dimensions of collaboration
identified by Thomsen et al (2007) (see page 17):
governance, administration, mutuality, norms, and
autonomy. Respondents were presented with a series
of questions related to each dimension and were
asked their level of agreement with a statement. The
full details of this process can be found in Braunz et al
(2010). Respondents were provided with seven options
on a seven-point scale with the endpoints anchored
at "strongly agree" and "strongly disagree" and the
middle (4th option) being anchored at "neutral."16
Survey responses were translated to a numeric value by
assigning a +1 to "strongly disagree" and increasing that
value by one until a +7 was assigned to "strongly agree"
(see graphic below).
+1 +2
I I
I I
Strongly
Disagree
+3
I
I
+4
I
I
Neutral
+5
I
I
+6 +7
I I
I I
Strongly
Agree
Figure 6 provides a summary of the data collected
through the survey of PPSI participants for all
respondents, broken out by respondent categories
for the five dimensions of collaboration. The results
for governance, administration, mutuality, and
norms dimensions tend to be in the 4.5 and 5.5 range
indicating overall agreement that the process had
been consistent with those dimensions. Additionally,
within those four dimensions, there is little variation
between the four groups (local government, state
government, private, and non-profit/NGOs). One
consistency that can be found is that responses from
individuals in private companies tended to be lower
than those in local or state government. The dimension
of autonomy (balancing competing demands of one's
own organization with demands of collaborative
effort), however, was rated relatively low in the survey
by all respondents and by each group separately. In
other words, most respondents disagreed that the
process allowed them to balance the needs of their own
organization and that of the collaborative effort.
The survey results in Figure 6 indicate that, overall,
collaboration was viewed favorably by most respondents
in each dimension of collection, except for the autonomy
dimension. In response to open ended questions,
(Braunz et al., 2010), PPSI participants indicated that
the process for achieving collaboration was costly and
should have been outlined from the start so participants
were informed of how they were to be engaged.
Viewpoints on Collaboration over the Lifecycle
of the Process
When viewed from the different stages of the
process, some stakeholders saw a break down in
collaboration when the process moved from planning a
program in general to developing the legislation for the
Oregon program and in planning and implementing the
Oregon program.
ACA indicated that the legislation was a compromise
of all the parties (S17). PSI's intention was that the
entire process, including development of legislative
language, would be collaborative (S19). In the survey
of PPSI participants (Braunz etal., 2010), however,
10 respondents indicated that they did not see the
legislation as collaborative. Six of the 42 open-ended
comments on collaboration rated collaboration as lower
when the Initiative moved into the implementation stage.
HHW programs echoed the breakdown in
collaboration. Some HHWs indicated they wished
they could have contributed so the program tied in
16 The second, third, fifth, and sixth points on the scale were not provided a text anchor.
Evaluation of the Oregon Paint Stewardship Program
-------
Figure 5 - Agreement among PPSI Participants on the Extent to which the PPSI Process was Collaborative
Neutral
Disagreed
Agreed at some level
process was collaborative
Strongly agreed
Agreed
Somewhat agreed
Figure 6 - Likert Scale Values for Five Dimensions of Collaboration, by Respondent Category
Governance
Autonomy
Administration
Mutuality
Norms
0.00 1.00 2.00 3.00
All I Local Govt I State/Federal Govt
4.00 5.00 6.00 7.00
Private Non-Profit/NGO, etc.
Promoting Environmental Results Through Evaluation 22
-------
the existing solid waste infrastructure (S13, S14, and
S15). One HHW representative specifically commented
that they would have liked to have an opportunity to
provide meaningful input into the organization of the
program; they had attempted to participate early in the
implementation process but felt as though they were
getting in the way (S13).
From PSI's perspective, the vetoes in Minnesota
influenced the level of collaboration (S19). After
Minnesota governor vetoed legislation twice, the group
stopped work for 1.5 years and lost momentum. During
that time, funding for PSI was limited, which resulted in
less communication which did not change with passage
of Oregon legislation. Oregon was also less involved in
the PPSI's work leading up to the Oregon program (S19).
Nevertheless, the Evaluation Team observed
that PPSI products and artifacts of collaboration
(e.g. relationships, research, formal agreements, and
committees) created between 2002 and 2008 had a
prominent role in influencing Oregon legislation and the
resulting program plan (Oregon, 2009; PaintCare, 2010).
3.3 Learning
The findings discussed in this section have led to the
identification of a number of key learning points. These
points include:
Collaboration can provide clear benefits. At
its inception in 2003, the PPSI embraced a
"From the start, collaboration should be
more purposeful with clearer goals for
collaboration that define and drive the
type, level, and timing of collaborative
effort necessary from various groups and
individuals...If collaboration were more
directed at achieving particular goals,
resources (time, money, staff) could be more
efficiently distributed across the overall
effort to make the PPSI more effective."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
collaborative approach to achieving its goals.
Much of the work, developed collaboratively,
under the Minnesota program carried over
to Oregon and much of that work found its
way into the Oregon legislation and plan. The
collaborative effort resulted in a pilot program
that reflected the needs of diverse stakeholders.
This evaluation and much of the data collection
and analysis are a product of the collaborative
process. Furthermore, the products and artifacts
of the PPSI process (e.g. relationships, research,
formal agreements, goals, committees) prior
to conception of the Oregon pilot provided the
background and baselines for legislation and
planning in Oregon.
Open communication is key to good
collaboration. Without open communication
about fundamental changes in the collaborative
process as the PPSI transitioned to the
implementation stage, PPSI participants lost
confidence in the process and their ability
to contribute to the process. To maintain
participants' confidence and engagement in
the process, the anticipated level of desired
and feasible collaboration should, at all stages
of the process (implementation and drafting
legislation) be agreed upon and documented.
Collaboration has its place but that may not
be at every stage of the process. Collaboration
decreased during the legislative process
and during program design as legislative
requirements set strict timelines; during this
time, ACA and Oregon DEQ worked independently
of the PPSI.
"...the time and resources committed to
getting to this point were great and not likely
replicable for other specific product areas.
This...in my mind doesn't provide a viable
model...for other product areas."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
23 Evaluation of the Oregon Paint Stewardship Program
-------
Sustained commitment is important. Effective
collaboration requires sustained commitments
to a facilitator, funding, and communication.
Declining collaboration coincided with
decreased PPSI funding, loss of the facilitator
to lead the process, and overall reduced
communications amongst PPSI participants.
The facilitator may need to also be an advocate.
As noted, one stakeholder felt that PSI's lack of
impartiality was a benefit to the process since
PSI acted as an advocate for paint management
(S18). Thus, use of an impartial facilitator (S18)
(i.e., one without an interest in the outcome)
may be less effective at pushing the issue
forward and attaining results.
Duke
NICHOLAS SCHOOL OF THE
ENVIRONMENT
The Duke University graduate student team, advised by Dr. Randy
Kramer of Duke University's Nicholas School of the Environment,
presented an overview and a full report of their work on Evaluation
Question 1 regarding collaboration which can be found as items #21 and
#22 under Additional Materialsatwww.paintstewardshipprogram.com.
Strategy and roles should be defined upfront
and documented. Collaboration requires
the commitments of increasingly scarce
resources (money, time, etc.) for local and state
governments. If collaboration is determined
to be a core component of achieving paint
stewardship goals, an explicit strategy set to
achieve clear goals (e.g., equity, cost efficiency,
diversity) will help to manage expectations
throughout the process. Groups need to
decide early in the process if collaboration is
important and for which aspects of program it
serves a greater priority. During the process,
open discussions led by facilitators or those
requesting changes, should accompany any
significant unanticipated adjustments to the
level and/or type of collaboration.
"Some organizations tend to work at
cross purposes. There was also a point
where ACA and Oregon were sealed off in
negotiations and the rest of the group was
sealed out."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
"...there was not collaboration by the
group on the actual text of legislation."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
"In Minnesota, the collaboration was
extremely effective until the issue reached
the required legislation to implement it."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
"It was a collaborative effort from a
planning standpoint - but has been
much less collaborative in its actual
implementation."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
"I don't think implementation has been
a collaborative effort with the PPSI nor
the legislative process. The dialogue and
keeping the PPSI informed as well as
the roll-out I strongly agree has been a
collaborative process."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
Promoting Environmental Results Through Evaluation 24
-------
4.0 Paint Stewardship Organization
^ A *- . EVALUATION QUESTION:
b Describe the Paint Stewardship Organization (PSO)
(Pa intC a re), including its funding mechanism and
infrastructure.
What factors contributed to its infrastructure choices?
Was the funding mechanism clearly defined, transparent, and complete?
What are the lessons learned?
4.1 Data Sources, Methods
and Analysis
The Evaluation Team answered this evaluation
question primarily through a qualitative approach of
reviewing PSO documents and interviews with key
program stakeholders.
In order to describe the infrastructure funding
mechanism of the PSO as it relates to the Oregon Pilot
Program, the Evaluation Team partnered with graduate
students at Georgia State University (GSU) as part of
their capstone project in evaluation. The GSU team
created an interview questionnaire and conducted
telephone interviews with eight participants who were
involved in the implementation, current operation,
and oversight of the Oregon Pilot Program. In addition
to conducting interviews, the GSU students gathered
and analyzed information from the Oregon legislation
and from the Oregon Pilot Program documentation
made available by the PSI, PaintCare Inc., the Oregon
Department of Environmental Quality, EPA, and ERG
(Bledsoe et al, 2011). Following the final report by GSU,
the Evaluation Team also conducted interviews with
The GSU graduate student team, advised
by Dr. Cynthia Searcy developed a report for
Evaluation Question 2, which can be found
under Additional Materials as item #25 at
www.paintstewardshipprogram.com.
key stakeholders to clarify information and to provide
additional details for the findings.
To determine the factors that contributed to PSO
infrastructure choices, the Evaluation Team analyzed
four components of infrastructure: 1) education
and outreach, 2) collections, 3) transportation, and
4) processing. To assess PSO funding mechanisms,
the Evaluation Team gathered data from interview
questions with regard to the meaning of the terms
clarity, transparency, and completeness in the context of
funding mechanisms. The team adopted the following
working definitions:
Clarity: the extent to which the funding source
for the PSO is clearly defined in implementing
legislation, rules, and program documentation.
Transparency: the extent to which a member
of the public could track how the funding is
being used by the PSO.
Completeness: the extent to which PSO funding
covers the expenses for operating the PSO.
25 Evaluation of the Oregon Paint Stewardship Program
-------
Clarity: The extent to which the funding
source for the PSO is clearly defined in
implementing legislation, rules, and program
documentation.
Transparency: The extent to which a
member of the public could track how the funding
is being used by the PSO.
Completeness: The extent to which PSO
funding covers the expenses for operating the PSO.
4.2 Findings
4.2.1 Describe the Paint
Stewardship Organization
(PaintCare), including its funding
mechanism and infrastructure.
Infrastructure: Process Actors
Figure 7 provides an overview of the structure of
the program (PaintCare, 2010; Bledsoe et al., 2011).
ACA created PaintCare as a 501 (c) (3) non-profit PSO
to manage the reuse, recycling, and proper disposal of
unused paint as mandated by the Oregon legislation.
The Board of PaintCare consists of nine non-paid
representatives of architectural paint manufacturers
(PaintCare, 2010; Bledsoe etal., 2011).
PaintCare contracted with Product Care Inc., which
has experience implementing similar stewardship
programs, to implement and manage the Oregon
program (Bledsoe etal, 2011). PaintCare selected PSC
and Metro as transportation and processing vendors;
these organizations had been part of the pre-program
infrastructure (Bledsoe etal., 2011).
Metro, PSC, and Amazon Environmental are the
service providers for the processing of latex paint
(PaintCare, 2010; Bledsoe et al, 2011). Their roles are
as follows:
Metro collects recyclable paint from its
locations and accepts recyclable latex collected
from the remainder of the state from PSC's
consolidation center. This paint is made into
new recycled content paint, which is then sold
(PaintCare 2010,2011b).
Non-recyclable latex paint collected by Metro
is disposed of by biodegradation at Columbia
Ridge Landfill in Arlington, Oregon. Through
a research permit, the paint is mixed with
wastewater and pumped into the landfill to
improve degradation of wastes and recovery of
landfill gas (PaintCare, 2011b).
Non-recyclable latex paint collected and
consolidated by PSC is processed into recycled
content paint, a biomass fuel product, and
processed latex pigment (PLP) (an alternative
raw material for cement manufacture) by
Amazon Environmental (PaintCare, 2011b).
Oil-based paint is collected and processed for fuel
blending by PSC (PaintCare 2010, 2011b).
Infrastructure: Collection Sites
PaintCare's strategy to set up the collection
infrastructure included the following components:
(PaintCare, 2011b; S17):
Incorporate pre-program collection
infrastructure (i.e., HHW facilities).
Reach out to paint and independent hardware
retailers to fill gaps in underserved areas.
Send a letter to retailers of paint and decorator
trade association list informing them of the
program and giving them an opportunity to be
a collection site.
Conduct Web searches and review yellow pages to
reach out to additional paint retailers in gap areas.
The collection program began with 45 collection
sites in July 2010 and, as of August 2011; there were
a total of 98 sites collecting paint (see Figure 8)
(PaintCare, 2011b). As of August 2011, all HHW
facilities that collected latex paint prior to the program
were collection locations in the PaintCare program
Promoting Environmental Results Through Evaluation 26
-------
Figure 7 - Organization of the Oregon Paint Stewardship Program
Product Care
Association
Admin
Services
American Coatings
Association
PaintCare, Inc.
(program operator)
Management Services
Contract
Producers
Program
Coordinator
PCA Paint
Stewardship, Inc.
(program manager)
Service Providers
Figure 8 - Numbers of Collections Sites in July 2010
and August 2011
Numbers and Types
of Collection Sites
Retailers
Restores
HHWs
July 2010
August 2011
(S17). None of PaintCare's 98 collection locations are
paid for collection services (S17).
The program provided each collection site with the
following (PaintCare, 2011b):
A Collection Site Procedures Manual (see text
box on page 28).
Program training on collection site operation,
customer service, environmental risk
reduction, visual inspection of post-consumer
paint containers, and placement of collected
paint in program containers.
Compliance visits to confirm the site is
following program requirements and to
investigate complaints.17
Finally, the program transportation service
provider (PSC) places collection containers at each
collection site; collection sites schedule a pickup by
calling the transportation service provider when the
collection containers are approximately 50 percent full
(PaintCare, 2011b).
17 PaintCare conducted 76 compliance visits during the first year of the program.
27 Evaluation of the Oregon Paint Stewardship Program
-------
Contents of the Collection Site Procedures Manual
Collection site standards
Screening procedures for conditionally exempt
small quantity generators
Accepted and non-accepted program products
Management of the Paint Exchange program,
including required waiver forms
Reporting requirements
Management requirements and operational
procedures
Infrastructure: Other Collection Services and
Other Services
During the first year of the program, collection
events were held to collect paint in rural locations
(PaintCare, 2011b). PaintCare anticipates completing
arrangements with permanent locations in order to
reduce costs (S17). By September 1, 2011, the program
had participated in 57 HHW collection events and two
PaintCare-sponsored events.
PaintCare provided additional collection services
through large volume direct pickups and collection
events (PaintCare, 2011b). Between July 1,2010 and
June 30, 2011 the program provided direct pickup
service to 19 institutional, commercial, and industrial
entities, such as trade painters, apartment complexes,
housing authorities and other private businesses
that meet the criteria for conditionally exempt small
quantity generators.
PaintCare offers a reuse program though Metro and
several Restores where good, usable paint in containers
that are at least half full are offered to the public for free
or at a discounted price (PaintCare, 2011b).
Finally, the program identified opportunities for
recycling plastic pails and steel paint cans from the
Metro program; during the first year, the program
recycled 47.1 tons of plastic and 64.8 tons of metal
paint cans (PaintCare, 2011b).
Funding Mechanism
PaintCare collects a recovery fee from paint
producers on all architectural paint sales to fund the
program. This fee was authorized in the legislation and
the fee was approved by the Director of the Oregon
Department of Environmental Quality (PaintCare,
2010). The assessment rate by container size is
summarized in Figure 9.
PaintCare's budget and fee were developed based
Figure 9 - Summary of Assessment Fee
$2.00
$1.50
$1.00
$0.50
$0.00
Note: Program does not accept containers
greater than 5 gallons.
1/2 pint or less 1/2 pint - quart
Quart - 1 gallon
1 gallon - 5 gallons
Promoting Environmental Results Through Evaluation 28
-------
on the estimated sales of architectural paint in Oregon,
estimated proportions of leftover paint available for
collection, and volume and cost data from the Metro
program and Oregon DEQ (non-metro areas). According
to the 2010 Program Plan, 2011 Annual Report, and
additional explanation by a PaintCare representative
(Em8):
Revenue was estimated by pro-rating the
national architectural paint volume sold by
population. This volume was compared with
ProductCare sales data from British Columbia
and will continue to be refined as PaintCare
receives actual sales data from program
participants. The final number multiplied by an
average cost per container produced the initial
revenue estimate.
Program costs included:
Projected collection volumes estimated
from rates of leftover paint derived
from research; Metro and non-
Metro paint collection volumes; and
assumptions about the amount of paint
available for collection (10 percent), the
recovery rate of program products sold
(7.1 percent), capture rate (71 percent),
and anticipated growth in collection
volumes per year (6 percent).
Administrative fees.
Vendor pricing to provide services
based on projected collection volumes.
The values were estimated for a 4 year budget,
then used as inputs into a spreadsheet model
which was used to model collection fees
under several scenarios (i.e., per container fee,
per gallon fee, flat fee, and graduated fee) to
identify the scenario and the fee amounts that
would best cover budgeted costs.
PaintCare developed a graduated fee (i.e., less costly
per gallon as container volume increases; see Figure 9)
based on the assumption that less waste paint is typically
generated from those purchasing a 5 gallon container
and that a higher volume of oil-based paint, which is
more expensive to manage as waste paint, is sold in
smaller containers (S17).
Figure 10 provides an overview of how the fee is
transmitted between different actors in the program
(PaintCare, 2010; Bledsoe et al, 2011). Producers
report monthly sales and pay the assessment directly
to PaintCare through a secure online filing system,
electronic fund transfer, or check The assessment is
passed through to a distributer or retailer who must
add the fee to the final sales price of their products
(PaintCare, 2011b). Retailers are responsible for
collecting the fee from consumers and transferring to
Figure 10 - Flow of Assessment Fee
within the Program System
Stewardship fee paid per
container sold in Oregon
Paint
Manufacturer
Paint Stewardship
Organization
Invoice & payment
includes fee
t»
Funding to oversee program
$
Paint Retailer
(Purchase price of paint
includes fee
Consumers
t»
$
Program Delivery:
Administration
Collection
Transportation
Processing
Communication
Flow of Money
Flow of Paint
29 Evaluation of the Oregon Paint Stewardship Program
-------
paint producers, who are then responsible for paying
the fee to PaintCare (PaintCare, 2011b). The legislation
provides retailers the option to show or not show the
fee on the consumer's sales receipt - this decision was a
compromise among PPSI stakeholders (S17, S20).
4.2.2 What factors contributed to
its infrastructure choices?
Processing and Transportation
In terms of processing and transportation, the
primary factor that contributed to the infrastructure
choices was the pre-existing infrastructure put in
place prior to the program. Specifically, Metro had an
established, mature paint collection and processing
infrastructure in the Portland metro area. Metro
continued to operate collection sites and process
recyclable latex paint into MetroPaint, a recycled
paint product sold by Metro. The program leveraged
this infrastructure in developing the current Oregon
program in areas outside of the Portland metro
(Bledsoe et al, 2011). Recyclable latex collected outside
of the Portland metro area was shipped from PSC
consolidation facilities to existing Metro facilities for
recycling. The program also relied on the pre-program
transportation provider (PSC) to transport the paint.18
Collection Sites
As shown in Figure 8 (see page 27), most sites
were retailers (70 sites; 71 percent in August 2Oil)
and HHWs (18 sites; 18 percent in August 2011).
PaintCare's coverage goal was to provide a permanent
collection site within 15 miles for more than 70 percent
of Oregon's population, which served as a primary
driver in determining the collection site locations
(PaintCare, 2011b; S17). Initially, PaintCare did not add
additional locations in an area (i.e., estimated as the
area within a 15 mile radius of a population center)
where a collection point was already established.
However, at the request of one local retailer, a state
representative encouraged PaintCare to reassess this
approach. PaintCare's current policy is to add additional
collection locations near another location only if no
significant costs are incurred (S17). PaintCare reported,
however, that it took longer than anticipated to
negotiate contracts with many collection sites (S17).
18 PSC, however, was selected through a Request for Proposals process.
4.2.3 Was the funding mechanism
clearly defined, transparent, and
complete?
Clearly Defined
Stakeholders were asked to assess the clarity of the
funding mechanism in a series of interviews (Bledsoe et
al, 2011). Stakeholders indicated that:
Legislation clearly details funding mechanisms
in terms of manufacturer responsibility, and
indicates that consumers will pay a fee at the
time of a paint purchase.
Program brochures and the PaintCare website
communicate the details of the funding
mechanism.
Implementing a graduated fee based on
container size as opposed to flat fees was
viewed positively by the public and paint
contractors.
As detailed in the Annual Report the program
estimates that most retailers show the fee on receipts
(PaintCare, 2011b). However, for some retailers,
reprogramming their computer system acted as a
disincentive to show the fee as a separate line item
on the receipt (S7, S8, and S9). The July 2011 survey
of consumers found that 11 percent of respondents
were aware of the fee (PaintCare, 2011a). Interviews
conducted by the Evaluation Team with retailers and
HHW programs support the July 2011 survey finding
of low consumer awareness; two HHW interviewees
reported some consumers confusing the fee with a
bottle deposit (i.e., by returning the empty container
the fee would be refunded) (S12, S15).
Transparent
The PaintCare report contained a detailed financial
audit of the program which provides some level of
transparency. Stakeholders' views on the degree of
transparency were dependent on the interviewee's
point of view. Those involved with PaintCare felt that
a publically available budget meant the program was
transparent, while other interviewees felt the lack of
broken out costs made it less transparent.
Promoting Environmental Results Through Evaluation 30
-------
Complete
In the 2011 Annual Report, PaintCare states that
despite the first year deficit projected in the program
plan and lower than estimated sales, the project
actually produced a surplus of $259,911 due to lower
than estimated expenses. This surplus, however, was
not sufficient to reduce assessment costs and will be
carried over to year two and used for program costs
(PaintCare, 2 Ollb).
4.3 Learning
4.3.1 Infrastructure
Existing infrastructure influenced program
design. The Oregon program was built
upon existing institutional knowledge and
experience, relationships, and infrastructure,
resulting in reduced need for training and
upfront costs. Other states and municipalities
will have varying levels and types of
infrastructure to accomplish similar goals.
The efficiency and feasibility of program
design and implementation will benefit from
initial thorough assessments of existing
infrastructure, including transportation,
reprocessing capacity, and related knowledge
and experience.
A more explicit and collaborative strategy
would improve collection site selection.
A collaborative and/or strategic process
for selecting collection sites may improve
efficiency of planning and implementation.
Opportunities include clear statements of
process steps, criteria for site selection,
and designated opportunities for specific
stakeholders (such as retailers and HHW
operators) to provide input in the process.
4.3.2 Assessment Fee
Strategic and targeted outreach is needed
to improve fee awareness. One goal of the
Oregon paint legislation is that consumers
become aware of the fee. In July 2011, few (11
percent) recent purchasers of paint were aware
of the fee (PaintCare, 2011b). Fee awareness
may be increased through a more strategic and
targeted education and outreach campaign (see
Learning section under Evaluation Question 3).
Fee awareness should be reconsidered
as a program goal. Future iterations of paint
legislation may consider whether consumer fee
awareness is a necessary policy goal.
In August 2010, most Oregon residents
(73 percent) indicated that the fee was
reasonable with few (23 percent) indicating
it was not reasonable (Bradshaw Advertising,
2010). If awareness of the fee is set as
a program goal, then a target level of
performance (e.g., a targeted percentage
becoming aware of the fee among Oregon
residents) should be set.
Oregon fee structure could act as a model
for other states. The process used to set/
design the fee structure is a baseline model for
other states; it covered costs in the first year
of the program. Notably, lower than expected
collection of oil-based program products
contributed to the one-year surplus.
Evaluation of the Oregon Paint Stewardship Program
-------
5.o Education and Outreach
EVALUATION QUESTION:
How did education materials and strategies affect
consumer awareness and behavior?
Which messages were most effective with which target audiences?
What materials/strategies were developed and what were the goals and
target audience of those materials/strategies?
Did other factors besides the program influence consumer behavior and
awareness?
What are the lessons learned?
One of the key goals of the Oregon Pilot project
is to minimize waste paint and containers generated
by consumers. Toward that end, one of the first steps
PaintCare took during the project was to launch an
outreach and education campaign; the campaign is
described in the June 29, 2010 approved pilot program
plan (PaintCare, 2010). Evaluation Question 3 focuses
on assessing the impact of this outreach and education
program on consumer awareness and behavior.
5.1 Data Sources, Methods
and Analysis
To develop the measures of program awareness,
the Evaluation Team drew on the results of consumer
surveys that asked the target audience to provide self-
ratings of degrees of awareness on the program. The
primary data sources for the measures include:19
Initial consumer survey: A telephone survey
of 409 Oregon homeowners, conducted by
Bradshaw for PaintCare, performed from
August 4 to 10, 2010. Full survey results are
available in Bradshaw Advertising's report to
PaintCare (Bradshaw Advertising, 2010), as
well as in the appendix to PaintCare's annual
report (PaintCare, 2011b). We refer to this as
the August 2010 survey.
Follow-up survey: An online survey of 1,064
Oregon residents over 18 years old, conducted
from July 7 to 12, 2011. This survey targeted
individuals who have purchased paint between
July 1,2010 and June 30,2011 as well as
individuals who paint as an occupation. Full
survey results are available in PaintCare's
report on the survey and in the appendix to
its annual report (PaintCare, 2011a,b). In this
report, we refer to this as the July 2011 survey.
19 These data sources are described in more detail in Section 2.4.
Promoting Environmental Results Through Evaluation 32
-------
The two surveys are not identical, constraining the
before and after comparisons that can be made. First,
the two surveys cover different target populations. The
August 2010 survey covers Oregon homeowners and
the July 2011 survey covered Oregon residents that
had recently purchased paint or that paint for a living.
Second, the two surveys used different sets of questions
with little overlap in the questions asked between the
two surveys.
The data collected through the surveys was
supplemented by a qualitative approach; including a
review of the materials produced by PaintCare and
interviews with key program stakeholders.
5.2 Findings
5.2.1 What materials/strategies
were developed and what were the
goals and target audience of those
materials/strategies?
The communication firm chosen by ACA developed
an outreach and education strategy with messages
consistent with the themes of ACA's "Be Paintwise"
program and targeted to all Oregon consumers. The
educational materials used brief taglines such as "From
Storage to Spectacular," "From Garage to Glorious," and
"From Basement to Beautiful" to catch the attention of
consumers (PaintCare, 2010,2011b). These messages
appeared with supporting text that highlighted the
following themes:
The name of the program and its website.
The advantages (cost savings, environment,
storage space) of recycling leftover paint.
The availability of information on purchasing
the correct amount of paint and managing and
disposing of paint properly on the program
website.
The recovery fee.
The program website (www.paintcare.org) also
featured educational information while providing
additional detail on each of the messages and on the
program itself (e.g., collection locations).
Figure 11 provides an overview of the education
campaign under the program. Samples of the materials
that were used by the program can be found in
appendices of the program plan (PaintCare, 2010) and
the annual report (PaintCare, 2011b). The appendix
to the annual report (PaintCare, 2011b) also contains
details on the where (e.g., radio stations) and when
advertising spots were purchased.
Despite its wide reach, the outreach and education
campaign did not identify or target specific audiences
(PaintCare, 2010). The messages, materials, and
delivery mechanisms were designed to have broad
appeal and reach as many Oregon residents as possible,
particularly those interested in purchasing paint.
Objectives of the education and
outreach campaign:
Build consumer awareness.
Identify program products.
Identify collection site locations.
Emphasize the environmental impact of
poorly managed leftover paint.
Emphasize the importance of purchasing the
correct amount of paint.
Promote paint reuse.
Promote recycling and proper disposal of
leftover paint.
33 Evaluation of the Oregon Paint Stewardship Program
-------
Figure 11 - Overview of the Education and Outreach Campaign
Point of sale materials
54,748 counter cards and
320 posters to 190 locations
Direct mailings to contractors
and retailers
1 2,414 factsheets to trade
painters
1 1,400 factsheets to retailers in
3 mailings (12/09, 4/10 and 6/10)
Website (www.PaintCare.org)
531,038 hits between March 4,
2010 and August 26, 2011
Education
and Outreach
Campaign
Press releases and information sharing
with the news media
Prior to start date: 13 TV and radio stories,
56 print articles, and 36 Internet articles
Earth day tie-in with 2 magazine articles
and 8 Internet articles
At the 1 -year anniversary: 6 articles and
a new radio interview
1-800-CLEANUP
(Numbers of calls received
not available at this time)
Advertising through local
solid waste partners
Trade shows
8 trade shows in
2010 and 7 in 2011
Purchased advertising
Radio advertisements on 17 radio stations in 2010
and 24 stations in 2011
TV advertisements on 2 stations in the Portland
Metropolitan market
Newspaper advertisements in 33 newspapers
across 29 markets in 2011
5.2.2 How did education materials
and strategies affect consumer
awareness and behavior?
Program-Related Awareness
Figure 12 summarizes program awareness in
both surveys. In the August 2010 survey, 22 percent of
respondents report being "aware" or "very aware" of
the program (Bradshaw Advertising, 2010). In the July
2011 survey 27 percent of respondents recall seeing
or hearing advertisements for the program (PaintCare,
2011a). However, it is difficult to determine the change,
if any, in awareness because results are based on
different target audiences.
Figure 13 summarizes survey data on the ability of
respondents to correctly identify in-scope products. A
reported 44 percent of respondents to the August 2010
survey incorrectly indicated that oil-based paint was not
covered by the program. Furthermore, one in five respondents
incorrectly indicated that latex paint is not covered.
The consumer outreach materials provided little
information on the types of products that would be
covered or not covered under the program. The rack
card20 contained a question and answer section that
indicated the types of products covered, while the
program website covers types of products.
Several of the materials produced by PaintCare
provided information on collection sites or how to
find collection sites. A January 2008 survey of Oregon
residents found that 67 percent of respondents have
taken leftover or unwanted household products to a
collection event or facility in the past (Oregon DEQ,
2008). However, since this includes any leftover
household products it may be an overestimate of
baseline awareness of sites that accept leftover paint.
The August 2010 survey did not ask consumers
about their awareness of site locations (Bradshaw
Advertising, 2010). The July 2011 survey found that
31 percent of respondents who purchased paint were
aware of a collection site (PaintCare, 2011a).
20 The card "Why Everyone Should Get with the Program" that consumers would have found at paint or checkout counters in retail locations contained
summary information on the program. A sample of the card can be found in the Appendix to the PaintCare annual report (PaintCare, 2011b).
Promoting Environmental Results Through Evaluation 34
-------
Figure 12 - Program Awareness in August 2010 & July 2011
August 2010
July 2011
Awareness of Environmental Impacts of Paint
The program and accompanying education
materials do not provide consumers with information
about the environmental impacts of leftover paint.
Two exceptions include: the rack card entitled "Why
Everyone Should Get with the Program," which
indicates one benefit of recycling paint as "(e)liminating
storage hazards where you live," and a page on the
website (www.paintcare.org/) indicates that leftover
paint may find its way into landfills and frames this
as an environmental issue. The July 2011 survey did
not address this aspect of awareness, so there is no
indication of whether awareness improved (PaintCare,
2011a). However, the August 2010 survey found that 61
percent of respondents indicated it is "very important"
and 31 percent indicated it is "somewhat important" to
have a program that accepts unneeded, leftover paint
for reuse, recycling and proper disposal (see Figure 14)
(Bradshaw Advertising, 2010).
Consumer Behavior
The program promoted few messages related
to reducing the amount of paint purchased. These
messages encourage consumers to buy the right
amount to:21
Save money: buying the correct amount
reduces the amount of money spent on paint
Save the environment: reducing leftover
paint reduces the amount of paint that might
find its way into landfills
Save storage space: less leftover paint
means more garage and basement storage
space for consumers
Reduce the risk of exposure to hazardous
materials: storing paint in home can pose
exposure and fire risks.
Figure 13-Ability of Respondents to July 2011 Survey to Correctly Identify In-Scope Products
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Oil-based is covered
Latex is covered
Correct
Paint thinner is
not covered
Containers with
>5 gals not covered
Incorrect
21 www.paintcare.org/index.php.
35 Evaluation of the Oregon Paint Stewardship Program
-------
The paint calculator served as the program's primary
tool of the program for assisting consumers in buying
the right amount of paint. In the July 2011 survey, 15 out
of 84 (18 percent) respondents reported being aware of
the paint calculator, and 3 of those respondents indicated
using the paint calculator to determine how much
paint to purchase (see Figure 15) (PaintCare, 2011a).
However, neither survey directly addressed whether
consumers understood the calculator.
The August 2010 survey asked respondents about
what they would likely do with leftover or unneeded
paint; the results of this survey question are summarized
in Figure 16 (Bradshaw Advertising, 2010).
In the August 2010 survey, respondents were also
asked whether the existence of the program would
affect the likelihood that they would recycle (see Figure
17) (Bradshaw Advertising, 2010). These data points
should be interpreted in the context of the January
2008 survey that found that 67 percent of residents had
taken materials to a collection event or facility in the
past (Oregon DEQ, 2008). That is, the 40 percent who
responded that the existence of a program makes "no
difference" whether they recycle may still be willing
to recycle paint and the 56 percent may include some
of that one-third of respondents in 2008 who had not
taken materials to an event or facility in the past.
The data from the July 2011 survey show that no
respondents recycled or donated leftover paint and 72
percent of respondents would store the paint for later
projects or touch-ups (PaintCare, 2 01 la).
ACA suggested that, based on anecdotal evidence, it
appeared that the education and outreach program had
little effect on consumers' practices (SI 7).
Additional Findings
The surveys conducted as part of the program did not collect information on amounts of leftover paint. However,
previous studies of leftover paint have found that 2.5 to 16 percent of paint sold may remain as leftover paint,
and the average amount of leftover paint per household ranges from 0.15 to 0.6 gallons (PaintCare, 2010).
The evaluation was unable to collect data on how consumers store leftover paint. The program did not include
questions on storage practices.
Figure 14 - Consumer Opinions on Importance of a Paint Recycling Program
Not important at all
Somewhat important
Very important
0%
10%
20%
30%
40%
50%
60%
70%
Promoting Environmental Results Through Evaluation 36
-------
Figure 15 - Awareness and Use of Paint Calculator in July 2011 Survey
Not aware
Aware
Did not use it
Used it
Figure 16 - Intended Consumer Management of Leftover
Paint (August 2010)
35%
30%
25%
20%
15%
10%
5%
0%
Figure 17 - Consumer Likelihood of Recycling Paint Due
to Existence of Paint Recycling Program
Call Go to
garbage/ govt
recycle/ recycler
disposal
org
Store for Put it
future in the
use garbage
More likely to recycle No difference
No answer
37 Evaluation of the Oregon Paint Stewardship Program
-------
5.2.3 Which messages were
most effective with which target
audiences?
Data collected during the first year of the program
did not directly address the effectiveness of specific
messages. To address this question would require
linking consumers' viewing specific messages to the
actions taken by consumers. The data for this evaluation
did not include which messages were viewed by survey
respondents. Nevertheless, the two surveys can provide
some insight into differences across groups. Table 2
provides a summary of the information available in each
survey on respondents' awareness and opinions. Table
3 provides a summary of the respondent characteristics
that can be cross-tabulated with the awareness and
opinion data. We provide a series of cross-tabulations
by the different respondent characteristics in Figure 18
through Figure 21 and we used those cross-tabulations
to investigate any potential differences across
respondents, which can be summarized as follows:
Importance of this type of program: All
demographic categories (age, region, gender)
indicated very strong support for a program
that manages leftover paint (see August 2010
survey in Figure 18, Figure 19, and Figure 20).
Awareness of program: With respect to
program awareness, there are a few notable
differences across respondents in the July 2011
survey:
Table 2 - Available Information on Respondents' Awareness or Opinions in Each Survey
Respondents' Awareness or Opinions
Importance of this type of program
August 2010 Survey July 2011 Survey
Awareness of the program
Awareness of the fee being charged
Feel the fee is reasonable
Awareness of drop-off locations
Program makes it more like they will recycle
Table 3 - Respondent Categories Available in Each Survey
Respondent Categories
August 2010 Survey July 2011 Survey
Age
Region
Gender
Education
Income
Promoting Environmental Results Through Evaluation 38
-------
Respondents who were older tended
to be more aware of the program (see
July 2011 survey in Figure 18).
Respondents living in the south and
central regions of the state tended to
be less aware of the program in both
surveys and those in the Portland
metro area tended to be more aware
(see July 2011 survey in Figure 19).
Respondents who were more educated
and had higher incomes also tended
to be more aware of the program (see
Figure 21).
Awareness of fee: For the most part, there is
little difference in terms of awareness of the
fee. One exception is respondents making more
than $100,000 may have higher awareness
relative to other income groups (see Figure 21).
Reasonableness of fee: Overall, respondents
to the August 2010 survey indicated that the
fee was reasonable with the lowest levels of
agreement being among those 60 and older
(68 percent; see Figure 18) and among men
(67 percent; see Figure 20).
Awareness of drop-off locations: There are
some differences in awareness of drop-off
locations for respondents to the July 2011
survey:
Respondents younger than 44 tended
to be less aware of drop-off locations
than those who were 45 and older (see
July 2011 survey in Figure 18).
Respondents who lived in the Portland
metro and central regions of the state
tended to be more aware of drop-off
locations (see July 2011 survey in
Figure 19).
Men tended to be more aware of drop-
off locations compared to women
(see July 2011 survey in Figure 20).
Respondents with higher levels of
income and education also tended to
be more aware of drop-off locations
(see income portion of July 2011
survey in Figure 21).
Likelihood of recycling: Older and younger
Oregon residents (see August 2010 survey
in Figure 18) as well as women respondents
(see August 2010 survey in Figure 20)
indicated they would be more likely to
recycle paint because of the existence of
the program.
39 Evaluation of the Oregon Paint Stewardship Program
-------
Figure 18 - Differences by Age Group: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of
Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey)
August 2010 Survey
Aware of program
18-34
Feels it is important
to have this program
35-44 I 45-54
Feels fee is
reasonable
55-64
Program makes it
more likely they
will recycle
I 65+
July 2011 Survey
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Aware of program
Aware of fee
Aware of drop-off
locations
18-34
35-44
45-54
55-64
65+
Promoting Environmental Results Through Evaluation 40
-------
Figure 19 - Differences by Region: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of
Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey)
August 2010 Survey
Aware of program
Metro
Feels it is important
to have this program
Feels fee is
reasonable
Program makes it
more likely they
will recycle
Willamette Valley
Central/East
Southern
Coast
July 2011 Survey
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Aware of program
Aware of fee
Aware of drop-off
locations
Metro
Willamette Valley
Central
East
Southern
Coast
Evaluation of the Oregon Paint Stewardship Program
-------
Figure 20 - Differences by Gender: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of
Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey)
August 2010 Survey
Aware of program
Feels it is important
to have this program
Feels fee is
reasonable
Male
Female
July 2011 Survey
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Aware of program
Aware of fee
Aware of drop-off
locations
Male
Female
Promoting Environmental Results Through Evaluation 42
-------
Figure 21 - Differences by Education and Income: Program Awareness, Fee Awareness, and Drop-off Site Awareness
(July 2011 Survey)
July 2011 Survey
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Aware of program
Aware of fee
Aware of drop-off
locations
HS or less
Some college - college grad
Post grad degree
July 2011 Survey
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Aware of program
Aware of fee
Aware of drop-off
locations
<$25K
$25K - $50K
$50K-$100K
>100K
No answer
43 Evaluation of the Oregon Paint Stewardship Program
-------
5.2.4 Did other factors besides
the program influence consumer
behavior and awareness?
The economy may have influenced markets for paint
products and thus reduced Metro collection volumes.
ACA tabulated quarterly Census Bureau data on gallons
sold which indicated a decline from 153.2 million
gallons in the first quarter of 2010 to 146.8 million
gallons in the first quarter of 2011 (PaintCare, 2011b).
Another potential influence to consumer behavior
and awareness lies in the value Oregon residents place
on recycling programs in general. Prior to the Oregon
program, Metro had a well-established and organized
recycling program in the Portland metro area, which
serves as home to 48 percent of Oregon residents.
Although 25 percent of consumers indicated that they
were aware of the program in the August 2010 survey,
56 percent of respondents indicated that the existence
of the program made it more likely that they will recycle
leftover paint and 92 percent said it is "very important"
or "somewhat important" to have this type of program
(Bradshaw Advertising, 2010).
5.3 Learning
Existing culture and infrastructure is
important. The culture and infrastructure
related to recycling in a state or region can
inform the design, implementation, and
emphasis placed on the program's education
and outreach campaign. In Oregon, a strong
culture of recycling coupled with existing
infrastructure was helpful to the program.
Furthermore, a well organized and established
program that is centralized with respect to the
state population, such as Metro, increases the
efficiency of program roll out and the transition
of a large segment of state population toward
recycling paint.
Targeting messages may be more effective.
The education and outreach campaign used a
broad approach to spreading its messages and
did not strategically aim its messages at specific
target audiences (consumers with the most paint,
contractors, retailers, etc.) with specific means
of communication (TV, website, social media,
radio, etc.). Consumers of paint can be segmented
into various groups (new versus long-time
homeowners, homeowners versus contractors,
age and other demographics). The most effective
ways of reaching each group may differ. For future
outreach efforts, consider determining which
groups are the largest purchasers of paint or who
has the most leftover paint and direct messages
using appropriate communications at those
groups. For example, the best ways of reaching
consumers who are aged 60+ may differ from the
best ways of reaching consumers under 30 (via
newspaper, retailer, Facebook, Twitter, etc.).
Awareness does not necessarily translate
to recycling. In the August 2010 survey of
homeowners, 10 percent indicated they would
put the leftover paint in the garbage (Bradshaw
Advertising, 2010), but in the July 2011 survey of
those that had painted recently, no respondents
indicated they had disposed of the leftover paint
in the garbage (PaintCare, 2011a). Though it is
possible that the program contributed to this
outcome, it is not possible to attribute this change
to the program because, for instance, there may
be a lag time between recent painting projects
and time of disposal not accounted for in the
survey. A consumer's lack of site awareness does
not automatically translate to not recycling paint.
Consumers with leftover paint may attempt to
locate a site to drop off paint when the need
arises. Although awareness of a drop-off location
in the July 2011 survey was 31 percent, a January
2008 survey found that 67 percent of Oregon
residents had brought items to a drop-off location
or event (Oregon DEQ, 2008). This suggests that
consumers who intend to recycle their paint
may know that drop-off locations are available
and they need easy access to information
about specific locations when they are ready to
recycle; in other words, it may not be important
that a consumer is aware of the program or
is able to name collection sites on demand as
long as she intends to recycle and is able to
access information about where and how to
Promoting Environmental Results Through Evaluation 44
-------
recycle. Going forward, the Oregon program and
similar programs may use education campaigns
to prioritize behavior change (e.g. recycle
paint rather than store it) and accessibility of
information over messages of general awareness.
Outreach products need to be linked to
a prioritized set of goals. Paint recycling
programs should prioritize the goals of outreach
products. For instance, the program should
determine whether the goal of a particular
message or product is to increase consumer
awareness of the program or increase the number
of consumers returning paint. These two goals
may be related or mutually exclusive. That is,
a consumer may find out about the program
and then decide to recycle paint or they may
find a collection location and return paint and
be unaware of a "PaintCare Program," related
legislation, or a collection fee. Focusing on
getting consumers to drop-off sites maybe more
important than focusing on program awareness.
However, using education campaigns to ensure
consumers purchase the right amount of paint
(possibly eliminating leftover paint) may be
more effective at reducing waste paint than, after
purchasing surplus paint, helping consumers
identify drop-off locations. Setting priorities
and focusing outreach on those priorities will
present opportunities to improve effectiveness
and efficiency of marketing campaigns and the
program as a whole.
Outreach should balance the different,
sometimes competing, program goals. A
balanced approach to setting and prioritizing
education and outreach is key and requires
ongoing measurement and evaluation of the
efficacy of the chosen approach. Putting too much
focus on getting consumers to reduce the amount
of paint purchased may lead to consumers with
leftover paint who do not know what to do with
it, while focusing too heavily on collections may
cause consumers to not worry about the amount
purchased since the drop-off locations provide
an outlet for their unused paint (e.g., "whatever
I don't use, I'll just drop off"). Measuring the
effectiveness of education and outreach materials
and strategies on consumer behaviors in the
context of the paint management system can
provide the information necessary to maintain
balanced progress toward program goals.
45 Evaluation of the Oregon Paint Stewardship Program
-------
6.0 Consumer Purchasing Decisions
^ A *- > EVALUATION QUESTION:
m How has the program affected consumers' purchasing
decisions and management of paint prior to drop-off at
paint recycling facilities?
How did the fee assessment affect consumer behavior?
One goal of the program is to help consumers
purchase only the amount of paint they need and
consequently reduce the amount of leftover paint
that must be managed. The pilot program attempts to
improve the ways in which consumers manage their
leftover paint; the pilot program provides consumers
with information on proper storage practices (e.g.,
not allowing leftover paint to freeze), encourages
consumers to bring leftover paint to recycling facilities
(rather than storing it), and educates consumers on
the environmental consequences of improper disposal
of leftover paint. Another key aspect of the program
is the application of a fee to the consumer purchase
price of paint in order to fund the program. Evaluation
Question 4 addresses the program's effect on consumer
behavior related to purchasing and managing paint, and
the extent to which the fee influenced consumer paint
purchasing behavior.
6.1 Data Sources, Methods
and Analysis
Evaluation Question 4 relies primarily on the same
two surveys as Evaluation Question 3 (see Section 5);
the August 2010 survey of Oregon homeowners and the
July 2011 survey of recent paint purchasers in Oregon.
Whereas the previous evaluation question focused on
the impact of the program on consumer awareness, this
question focuses on the impact on consumer behavior
(e.g., paint purchasing).
Additionally, the Evaluation Team drew on the
2008 Oregon Department of Environmental Quality
Household Hazardous Waste Survey (Oregon DEQ,
2008) for baseline contextual information on consumer
behavior related to paint management. The 2008
Oregon DEQ HHW survey was conducted in January
2008 by the Portland State University Survey Research
laboratory; a total of 615 Oregon residents completed
the random statewide telephone survey, answering
questions about use and perception of the dangers of
common household products (e.g., oil-based and latex
paints, pesticides, drain openers).
6.2 Findings
6.2.1 How has the program
affected consumers' purchasing
decisions and management of paint
prior to drop-off at paint recycling
facilities?
Purchasing decisions
Data from the July 2011 survey indicate that the
program had little impact on the amount of paint
purchased, as 93 percent of respondents indicated that
information about the program had no influence on the
Promoting Environmental Results Through Evaluation 46
-------
amount of paint purchased (see Figure 22) (PaintCare,
2011a).
Fifteen respondents (out of 84 total respondents) in
the July 2011 survey recalled seeing the paint calculator
and among those 15 respondents, three used the
calculator to decide how much paint to purchase (See
Figure 2 3).
Management of paint prior to drop-off
In the August 2010 survey, respondents were also
asked whether the existence of the program would
affect the likelihood that they would recycle (Bradshaw
Advertising, 2010). Results indicated that:
56 percent of respondents reported that they
are more likely to recycle leftover paint
40 percent reported that the program makes
"no difference."
Neither survey nor any of our other data sources
addressed how or where the consumers stored the
paint and how long they stored it prior to drop-off. The
surveys did, however, address what consumers planned
to do or did with paint (Figure 16 on page 38).
6.2.2 How did the fee assessment
affect consumer behavior?
When asked about the fee assessment in the August
2010 survey (see Figure 24), 33 percent of respondents
indicated the fee assessment was "very reasonable," 40
percent found the fee "somewhat reasonable," and 23
percent indicated that the fee was "not reasonable at
all" (Bradshaw Advertising, 2 010).22
In the July 2011 survey of recent painters
(PaintCare, 2011a), 11 percent of respondents indicated
that they are aware of the fee. Additionally, when asked
whether the fee affected the amount of paint purchased,
93 percent of respondents said the fee had no impact
(see Figure 25). This survey, however, covered those
that had recently purchased paint. There may be a set
of consumers that chose not to buy paint at the fee-
inflated price, but who would have purchased paint at
the price of paint without the fee added.
Figure 22 - Consumer-Reported Impact of Program on Amount of Paint Purchased (July 2011 Survey)
No effect; 64
Purchased fewer
containers; 2
Purchased smaller
containers; 1
Other; 2
10 20 30 40 50
Number of Respondents
60
70
22 The remaining respondents were in the refused/don't know categories.
47 Evaluation of the Oregon Paint Stewardship Program
-------
Figure 23 - Awareness and Use of Paint Calculator
Aware
Saw paint
calculator
Did not use it
Used it
6.3 Learning
Purchasing Decisions and Paint Management
Program had little impact on consumer
purchasing. There is no indication that the
Oregon program has influenced the consumers'
paint purchasing decisions. If consumers
continue to purchase the same amount of
surplus paint, there will be no change in the
volume per capita eventually destined for the
paint management system, thus requiring
expenditure of more resources than if consumers
purchased amounts of paint that more closely
matched their needs. Reducing surplus (waste/
leftover) paint purchased by consumers would
reduce subsequent resources required for a
statewide paint management system.
Program focused on recycling, not
reduction. The outreach materials focused on
the recycling aspects of the program (i.e., where
to bring leftover paint) and not on reusing
leftover paint or reducing the amount of paint
purchased. The purpose of the paint calculator
provided at paint retailers is to improve the
accuracy of the amount of paint purchased but
consumers were generally unaware of it and
did not use it, so it was not effective at reducing
volumes of post-consumer paint. If improving
the consumers' ability to purchase the correct
amount of paint is to be considered a viable
option to achieving the program and policy
goal of reducing leftover paint, effort should be
focused on understanding and improving the
effectiveness of the tools being used to achieve
this goal (e.g., paint calculator, retailer training,
consumer assistance).
Waste hierarchy emphasis should be
clearly defined. To achieve goals of reducing
the generation of post-consumer architectural
paint, paint management programs and
policies may explicitly document the relative
emphasis they want to place on aligning the
paint management system with the existing
waste hierarchy. For instance, prioritize options
such as reuse of leftover paint, reducing
leftover paint through improving accuracy of
paint purchases (less leftover) and increasing
volumes of managed post-consumer paint.
Explicit attention to these options will provide
a structure to prioritize outreach strategies,
messages, materials, and effort.
Promoting Environmental Results Through Evaluation 48
-------
Figure 24 - Consumer Perceptions of the Reasonableness
of the Fee (August 2010 Survey)
Figure 25 - Consumer-Reported Impact of Fee
on Amount of Paint Bought
Very reasonable
Somewhat
reasonable
Not reasonable at all
Refused/don't know
100
90
80
70
60
50
40
30
20
10
0
Purchased
same
amount;
93%
Was only asked
of recent paint
purchasers - no data
on how many did
not purchase paint
because of the fee.
Purchased
less; 7%
Impact of fee on consumer purchasing
Oregon residents value this program.
Ninety-two percent of respondents indicated
the need for this type of program was
"somewhat important" or "very important."
Oregon residents' acceptance of the fee may be
linked to the inherent value they place on the
goals and services provided by such programs.
The degree to which consumers value such
programs may indicate their willingness to
pay a certain fee. This information may be
used in refining fee structures within and
across states. Understanding and leveraging
consumer attitudes may also prove useful in
navigating the legislative process, where, as the
PPSI learned in Minnesota, without evidence
of public opinion and providing evidence of
the public's willingness to pay a fee, legislation
based on a fee mechanism may be rejected by
executive or legislative leadership even with
broad bipartisan legislative support.
More details are needed on fee impacts.
A full understanding of the impact of the fee
would assess whether some consumers were
deterred from buying paint due to the higher
cost associated with the added fee. The fact
that consumers are deterred at the margin may
or may not be a good thing. On the one hand,
a higher price may reduce the total amount of
paint demanded by consumers and affect paint
retailers' revenues. On the other hand, reducing
the total amount of paint purchased reduces the
total amount of leftover paint that would have
to be managed by the program in the future.
49 Evaluation of the Oregon Paint Stewardship Program
-------
7.0 Collection of Post-Consumer
Paint
- -f ^ EVALUATION QUESTION:
How has the program affected the collection of post-
consumer paint in terms of volume, cost, environment,
convenience, and infrastructure?
What other factors have affected the amount of leftover paint?
7.1 Data Sources, Methods
and Analysis
Program data collected and tracked by PaintCare
serves as the primary source of data for the analysis
of volume, cost, environment, convenience, and
infrastructure. PaintCare reported this information in
its annual report (PaintCare, 2011b). The program data
are supplemented with data and analysis provided by
Hedrick Strickland, who performed a CIS analysis of
collection site locations to assess convenience, as well
as baseline data from the Oregon DEQ, and a set of
interviews with key program stakeholders and retailers.
7.2 Findings
7.2.1 How has the program
affected the collection of post-
consumer paint?
Volume
In the first year of the program, PaintCare collected
469,665 gallons of paint. Of this total, 352,136 gallons
(75 percent) were latex paint and 117,529 gallons (25
percent) were oil-based paint (see Figure 26). Metro
collected 50.4 percent of the total gallons (PaintCare,
2011b).
Overall, the program collected approximately 3
percent more pounds of paint than was collected by
Oregon DEQ in 2008 (assuming a volume to weight
conversion of 10 pounds per gallon for paint collected
under the program) (PaintCare, 2011b). However,
PaintCare notes in its report that there may be issues in
comparing the two estimates due to conversion factors.
In addition, PaintCare collected approximately 24,000
gallons of paint in areas where collection was not
previously available.
PaintCare collected less paint than anticipated in
the projected budget (Figure 27), collecting 31,969
fewer gallons of latex (8 percent) and 47,088 fewer
gallons of oil-based (29 percent) than projected
(PaintCare, 2011b). PaintCare indicated that the
discrepancy in oil-based collection is a result of a
miscalculation in how much oil-based paint was being
collected prior to the program start (S17). Additionally,
PaintCare moved to a weight-based system to more
accurately reflect collected volumes.
For Metro, the number of households dropping
off paint decreased under the program by two
percent from the previous year and the number of
non-households (e.g., contractors) dropping off paint
increased significantly. Metro believes the increase in
Promoting Environmental Results Through Evaluation 50
-------
Figure 26 - Volume by Paint Type and Service Area
Collections by type of paint
Volume by service provider/area
- Reuse
3.2%
Oil
117,529
25%
Latex
352,136
75%
Outside
Metro
46.4%
Metro
50.4%
469,665 total gallons collected
469,665 total gallons collected
Figure 27 - Projected vs. Collected Amounts of Latex and Oil
450,000
400,000
350,000
300,000
250,000
200,000
150,000
100,000
50,000
0
164,617
117,529
Latex
Oil
Projected
Collected
Evaluation of the Oregon Paint Stewardship Program
-------
non-household consumers reflects the fact that disposal
of paint is now free for these organizations (Eml),
whereas prior to the program non-households had to
pay to drop-off paint under the program. Metro also
reported that the volume of latex paint collected was
down 9 percent over the previous year. There was an
initial spike in volumes for Metro during the first few
months, but overall the total volumes were down (Eml,
S16). Metro also indicated that they collected a large
number of unlabeled containers of paint, which are
classified as non-program materials and as such were
the responsibility of Metro to handle (Eml, S16).
Costs
PaintCare reported that the cost of managing
paint collected under the program is $7.03 per gallon
(PaintCare Annual Report, 2011b). This metric does
not include collection costs, such as labor and storage
space, incurred by collection sites. In an Oregon paint
recycling pilot program conducted in 1997 to 1998,
Oregon DEQ-sponsored collection events were reported
to cost $13.86 per gallon (inflated to 2011 dollars) and
the pilot program cost $11.32 per gallon, during which
collection was done at retailers' sites (Cascadia, 1998).
Without more detailed analysis, these estimates are not
appropriate for use as baseline, but may indicate some
cost savings relative to the 1997 to 1998 program design.
Environment
The PPSI Lifecycle Workgroup (2005 to 2009)
consisted of representatives from government and
industry. ACA funded contractors to conduct a lifecycle
assessment (LCA) of six options for managing leftover
paint (e.g., dry and dispose at home to centrally
collecting and processing into recycled paint) (PSI,
2009). LCA is a tool for the systematic evaluation of
the environmental impacts of a product through all
stages of its life cycle, which include extraction of
raw materials, manufacturing, transport and use of
products, and end-of-life management (e.g., reuse,
recycling and/or disposal). The PPSI LCA scenarios
were developed prior to the enactment of any
legislation and therefore not based on the Oregon pilot
(PaintCare, 2011b).
Initial LCA modeling results were found to
depend crucially on the amount of virgin paint that
consumers replaced with recycled paint (PSI, 2009). If
consumers were assumed to replace all virgin paint with
recycled paint, then recycling proved to be the more
environmentally beneficial outcome. However, if no
virgin paint was assumed to be replaced, then producing
recycled paint was found to have a larger environmental
impact than just having consumers dry out and dispose
of the latex paint in their household trash. Agreement
between industry and government representatives on
what percentage of virgin paint is avoided could not
be reached and the workgroup disbanded. Without
continued funding, the planned sensitivity analysis to
identify other assumptions and parameters that had a
major impact on the modeled results was not completed.
ACA's conclusion as reported in the Annual
Report (PaintCare, 2011b) was "[t]he [LCA] results
demonstrate that unless and until recycled content
paint is marketable at the same or similar rates as
virgin paints...the environmentaL.cost [and] benefits or
recycling paint do not outweigh drying and disposing
of such." The draft LCA is available as Appendix J of
PaintCare's report. Some members of the workgroup
had anticipated that the Oregon pilot could provide real
world data to refine the LCA model to complete PPSI's
goal of better understanding the societal impacts of
latex paint management approaches (PSI, 2 009). The
modeling results could help with the planning and
selection of appropriate paint management methods in
light of environmental benefits weighed against costs.
Convenience and Infrastructure
As of September 1,2011, the program had 98
collection sites with 10 of these sites offering paint
exchange. Sites are open to the public, on average,
six days a week for a total of 58 hours per week
(PaintCare, 2011b; Em2). Figures 28 and 29 provide
maps of the pre-program and June 30,2011 drop-off
sites, respectively (PaintCare, 2011b).23 The increased
number of sites can be seen in comparing the two
figures. Table 4 highlights the increased amounts of
the Oregon population living within a 15 mile radius
or within 15-mile drive of a drop-off location. The
increased number of drop-off sites increased the
number of Oregon residents living within a 15 mile
radius of a drop-off location by 38.5 percent (PaintCare,
2011b). Additionally, based on a CIS analysis using
June 2011 sites, 92 percent of Oregon residents live
23 Both Figure 28 and Figure 29 were taken from the PaintCare's annual report (PaintCare, 2011 b).
Promoting Environmental Results Through Evaluation 52
-------
Table 4 - Proportions of Oregon Population Living within 15 Mile Radius and 15 Mile Drive of Collection Sites
Time Period
Percent of Oregon
residents living in Percent of all Oregon
incorporated cities, residents living within
towns, and CDPs who are 15 mile radius
within a 15 mile radius of collection site
of a collection site [a] (PaintCare, 2011)
Percent of all Oregon
residents living
within a 15 mile drive
of a collection site
(Strickland, 2011a,b)
(Pa
Pre-program
(date unknown)
January 2011
June 2011 [b]
Change between pre-
program and June 201 1
intnarp 2011}
69.2%
--
95.9%
38.5%
51.2%
--
70.9%
38.5%
91%
92%
[a] GDP = Census Designated Places
[b] The analysis performed for the PaintCare report on the percent within the 15-mile radius and the analysis performed by Strickland are seemingly
contradictory. Specifically, the percentage within a 15 mile radius should exceed the percentage within a 15-mile drive. The Evaluation Team provided
Strickland with the set of sites listed as being the June 30, 2011 sites. There may be some analytical details that account for some of the discrepancy. For
example, Strickland's analysis uses the percentage of roads in a Census block group that are within the 15-mile drive as a proxy for the percentage of the
population within that Census block group. Also, the most recent block level census data at the time of Strickland's analysis was from the year 2000, whereas
the most recent data at the time of PaintCare's analysis was from 2010. This could also account for the discrepancy.
Figure 28 - Pre-Program Drop-Off Sites
53 Evaluation of the Oregon Paint Stewardship Program
-------
Figure 29 - Drop-Off Sites on June 30, 2011.
Ugtnd
\ SW.HMW lacing
| ReUUora
O Loe*l B«Vt rvtmt ,
Figure 30 - Percent of Residents Within a 15 Mile Drive by County.
IMDU OP«?O- Sunndi Umtxn Fin M
Promoting Environmental Results Through Evaluation 54
-------
Figure 31 - Change in Collection Service Availability
Pre-program
June 2011
HHWs offering
collection
Counties served
by events only
Counties with
no service
Retailers serving
as collection sites
within a 15-mile drive of a drop-off location (Strickland,
2011). Figure 30 presents the mapped results of the CIS
analysis showing the percentage of residents within a
15 mile drive of drop-off locations for each county in
Oregon.
Figure 31 compares the pre-program network of
collection sites to the June 2011 network based on data
presented in the PaintCare annual report (PaintCare,
2011b). The data in the figure show a substantial
increase in the availability of collection services.
Additionally, PaintCare provided 19 large volume
direct pickups to institutional, commercial, and
industrial entities between July 1,2010 and June 30,
2011, collecting approximately 4,050 gallons of paint
(PaintCare, 2011b). Finally, 63 percent of consumers
in July 2011 who purchased paint recently think their
nearest collection site is convenient (PaintCare, 2011a).
7.2.2 What other factors have
affected the amount of leftover
paint?
Factoring into baseline considerations is the fact
that the quantities of latex paint collected at DEQ events
fell significantly after consumers were discouraged
from bringing latex paint to collection events starting
in 2008 (Oregon DEQ, 2011). To identify the factors
that have affected the amounts of leftover paint,
information on consumer handling of leftover paint
is needed. Although some information is available on
what consumers intend to do with the leftover paint,
information on non-program factors was not assessed
under this project.
7.2.3 How has the program
affected transportation of paint
from collection sites to other
facilities in terms of volume,
environment, and cost?
The transportation contractor (PSC) is minimizing
transportation costs by designing "milk-run"
transportation routes with software designed by Red
Prairie (Em3). The trucks service PaintCare locations as
well as other clients that fall along the transportation
route. PSC reported that transportation costs are
similar pre- and post-program since the primary cost
driver is diesel fuel price (Em3). However, without
collection of Oregon pilot program data (before and
after the program start) the Evaluation Team was not
able to make an assessment on changes of transported
volumes or the environmental impacts.
55 Evaluation of the Oregon Paint Stewardship Program
-------
7.2.4 How has the program
affected retailers' behavior?
Retailer interviewees note that the program had
little effect on their practices in terms of marketing or
sales (S7, S8, and S9). However, key challenges included:
Programming the fee assessment into pricing
and invoicing computer systems (S8).
Preparing staff for program rollout (S7, S8).
The challenge for one larger retailer was to
make sure staff are aware of and understand
the program, putting materials out, and are
able to answer customer questions. Time
for internal outreach and education and the
magnitude of time/resources needed are
related to the number of employees.
7.3 Learning
Reporting on amounts of paint should
be standardized. PaintCare's preliminary
reporting used paint container fullness
assumptions for calculating volume of
paint collected. This assumption proved to
underestimate gallons collected, and PaintCare
found that a weight-based measurement was
more accurate and more common for HHW
reporting to Oregon DEQ (S17).
Contractors are a stakeholder in the
program. The amount of waste paint
generated by contractors needs clarification.
One interviewee stated that contractors should
not be subject to the recovery fee because
they don't generate waste paint (Bledsoe et al.,
2011). Meanwhile, Metro saw an increase in
the number of non-households dropping off
paint because under the program no fees were
collected for drop-off (Eml). Contractors were
not a stakeholder group participating in PPSI
and the amount of leftover paint they generate
was not quantified in EPA's report Quantifying
the Disposal of Post-Consumer Architectural
Paint (Abt, 2007). PPSI could further explore
if contractors are a significant contributor of
paint to the management system and therefore
need different and more audience-specific
messaging about reducing waste paint.
Unlabeled paint containers are an
issue. The Evaluation Team is uncertain of
the number of unlabeled paint containers
that consumers are attempting to turn in.
Consumers know the unlabeled container has
paint, but collection sites are instructed to treat
unlabeled paint containers as a non-program
product and to not accept them. If a significant
volume of leftover paint is being excluded
from the program, then that paint is essentially
being managed as it was prior to the program -
as a waste product; rather than being recycled
or reused.
More definitive information is needed
to provide an accurate assessment of
the cost per gallon. The cost of $7.03 per
gallon can be used as a starting point for
other states. However, the reported PaintCare
system costs do not reflect the true costs of a
leftover paint management system. PPSI's Paint
Product Stewardship Initiative Infrastructure
Project (the "infrastructure report"), which
modeled a comprehensive system to manage
leftover paint, stated that the collection step
in the process costs $1.98 per gallon (SCS
and Cascadia, 2007)however, this is not
an apples to apples comparison (e.g., the
infrastructure report included transportation
to aggregate facility as part of the collection
step). Efforts are underway in Washington
State to estimate local government costs
associated with collection of waste paint, which
includes a methodology and development
of a tool for local governments to collect
data (Em5). Furthermore, as noted under
Evaluation Question 4, the Oregon program
had a relatively well-established paint recycling
infrastructure in the Portland metro area prior
to this pilot program. Thus, program costs per
gallon for other states may depend on the level
of existing infrastructure.
Promoting Environmental Results Through Evaluation 56
-------
Further discussion of reimbursement
of collection costs is needed. Non-
reimbursement of collection costs continues
to be a controversial issue among stakeholders
(S18, S20). This non-reimbursement was
a topic that PSI had anticipated further
collaborative discussion on as part of detailed
work plan for the implementation stage. (S19)
Additional LCA work is needed. An
uncompleted task of the lifecycle workgroup
was to conduct a sensitivity analysis for
the LCA which could have identified other
assumptions with large environmental effects.
For example, the initial results indicated that
the virgin offset made the largest impact,
but transportation could be another factor.
For example, Metro's less desirable recycled
paint colors are being sent to Asia (for a fee,
paid by Metro (Em4)). Does that change the
environmental benefits of a recycled paint
management system if paint is shipped across
the globe? With the key assumptions and
parameter data identified, future evaluations
and ongoing examination of program
performance information could collect
groundtruthing data (e.g., virgin paint offset;
miles traveled) to improve the ability to model
a paint management system. The LCA is a tool
to estimate a wide range of environmental and
social impacts, and interpreters of its results
will have to determine which impacts (e.g.,
greenhouse gases, water usage, air toxics)
are the most important when selecting a
management approach. LCA can be integral
to future program design and planning, and
requires funding for complete development
and interpretation of the model.
'ak
57 Evaluation of the Oregon Paint Stewardship Program
-------
8.0 Paint Reprocessing, Recycling
and Energy Recovery
- - *- ^ EVALUATION QUESTION:
How has the program affected used paint reprocessing,
paint recycling, and paint-related energy recovery in
terms of volume, infrastructure, and cost?
An important aspect of assessing product
stewardship outcomes is understanding what happens
to the materials that are collected under the program.
The program plan sets forth the following management
options for collected leftover paint:
Paint exchange: In order to increase the
quantity of leftover paint that is reused,
program collection sites and other locations
can voluntarily offer a "paint exchange" to their
customers. Under this system, better quality
containers of paint are placed on display
shelving and available at no cost to another
consumer. As discussed in Evaluation Question
9, the exchange management option is an
efficient way to achieve reuse as the paint does
not require transportation and reprocessing.
Recycling as paint: Good quality leftover
latex paint may be bulked with similar colors
then filtered and placed in smaller containers
for resale to consumers.
Recycling as another product: In the
Oregon program, some unusable latex paint
was sent to Amazon Environmental Inc. to be
converted to PLP, a cement additive. Amazon
also used some "bad" latex to bind biomass
used to fuel its facility (referred to as Amazon
biomass in this report).
Energy recovery: Oil-based paint is blended
with other oil-based solvents to be burned as
fuel for some operations, such as incinerators.
Appropriate disposal: Metro disposes of
latex paint that is not of suitable quality for
recycling via landfill biodegradation. Metro's
non-saleable paint and washwater from
the recycling process is transported to the
Columbia Ridge Landfill in Arlington, Oregon.
Under a special Research, Development and
Demonstration permit at the landfill, liquids
including the paint-washwater mixtureare
intentionally circulated throughout the solid
waste to expedite degradation.
Table 5 summarizes these methods in order of
preference for latex paint, oil-based paint, and paint
containers. The purpose of Evaluation Question 6 is to
track the volume, infrastructure, and cost associated
with each management method.
8.1 Data Sources, Methods
and Analysis
The primary data source for answering Evaluation
Question 6 is the data collected by the program and
summarized in the PaintCare Annual Report (PaintCare,
2011b). The Evaluation Team supplemented these data
with key stakeholder interviews to provide details on
the numbers being reported.
Promoting Environmental Results Through Evaluation 58
-------
Table 5 - Hierarchy of Management Options for Post-Consumer Paint Products
Paint exchange/reuse
Recycling as paint
Recycling as another product
Appropriate disposal
Oil-Based Paint
Paint reuse
Energy recovery
Paint Containers
Recycling
Appropriate landfill
8.2 Findings
Table 6 shows the disposition of latex and oil-based paint collected under the program. These data are also
summarized in Figure 32 (latex) and Figure 33 (oil-based). In what follows, we describe the data presented in Table i
Table 6 - Disposition of Paint by Management Method
Paint and
Catego
ercent of Total Paint
Collected [:
Latex Paint
Reprocessing
Amazon PLP
Recycling/Reuse
Metro recycling
Amazon recycling
Consumer reuse
Energy Recovery
Amazon biomass
Disposal
Metro biodegradation
>il-Based Paint
Recycling/Reuse
Consumer reuse
Energy Recovery
Amazon fuel blending
8%
53%
4%
3%
4%
28%
3%
97%
28,171
186,632
14,085
10,564
14,085
98,598
3,526
114,003
[a] For latex this is the percentage of total latex paint collected (352,136 gallons) and for oil-based is the percentage of oil-based paint collected (117,529 gallons).
59 Evaluation of the Oregon Paint Stewardship Program
-------
Reprocessing
Disposal
Eight percent of latex paint was processed into
Amazon PLP, a cement additive (PaintCare, 2011b).
However, information on costs for reprocessing was not
available (PaintCare, 2011b).
Recycling/Reuse
Fifty-seven percent of latex paint was recycled as
paint by Metro and Amazon (PaintCare, 2011b).
Metro's peak volume of paint processed
into recycled product increased after program
implementation from approximately 280,000 to
325,000 gallons (S4). Metro has additional processing
capacity available (S4).
Three percent of latex paint and 5 percent of oil-
based paint was reused by consumers (PaintCare, 2011b).
PaintCare provided collection locations an
incentive of $0.2 5 per "reuse" gallon to reflect avoided
transportation and processing costs (PaintCare, 2011b).
An organizational representative notes that no sites
took advantage of this incentive; sites that were already
offering reuse had systems in place and considered it an
important service (S17). However, less than 10 percent
of collection locations have shelves for paint exchange;
PaintCare did not recruit retail sites for paint reuse due
to concerns about liability (S17).24
Finally, approximately 47 tons of plastic paint pails
and 65 tons of metal cans were recycled (PaintCare
2011b). Assuming that each plastic pail weighed about
1.6 pounds, and each metal can weighed about 1.4
pounds, then this tonnage translates to approximately
93 thousand metal paint cans and 59 thousand plastic
pails. Data on the total number of containers collected
is not available so there is no available estimate on the
percent of collected containers that were recycled.
Energy Recovery
Almost all (97 percent) of the oil-based paint
collected under the program was bulked for energy
recovery (fuel blending) (PaintCare, 2011b). Four
percent of latex paint went to Amazon Biomass; the
paint is used to bind materials with high British thermal
unit (BTU) value (e.g., sawdust) for energy recovery
(PaintCare, 2011b)
Twenty-eight percent of latex paint was processed
for biodegradation; the paint is mixed with wastewater
and injected into a landfill in order to improve
degradation of landfilled wastes and increase gas
recovery (PaintCare, 2011b).
Figure 32 - Destination of Collected Latex Paint
Metro recycling
Metro biodegradation
Paint reuse at
Restores
Figure 33 - Destination of
Collected Oil-Based Paint
Fuel
blending
Paint
reuse
Amazon recycling
Amazon PLP
Amazon biomass
(fuel blending)
3%
24 Liability concerns can take several different formsenvironmental, health and safety, and product. For example, spilled paint from a consumer wanting to
open a can to confirm color or the old paint product doesn't perform as expected. These liabilities can be mitigated. See PPSI's Guidance Manual for Paint
Reuse Programs available at www.paint.org/component/docman/cat_view/47-education-a-safety.html.
Promoting Environmental Results Through Evaluation 60
-------
8.3 Learning
Existing infrastructure was important for
determining the destination of collected
paint. The presence of existing latex paint
recyclers with available processing capacity
allowed recycling of paint to be the dominant
method of processing leftover latex paint.
There are barriers to increasing reuse.
Recycling is lower on the waste hierarchy than
reuse, and recycling results in higher costs
(management, environment) compared to
reuse. More information is needed to better
understand the barriers (e.g., liability and
space) to retailer facilitation of paint reuse.
One option to increasing reuse, suggested by
Oregon DEQ (S20), is for PaintCare to
coordinate with the Restores and local
government to redistribute reusable paint
collected from the collection sites. This option,
however, also creates costs and environmental
effects (e.g., additional transportation and
labor).
Retailer liability may limit opportunities for
reuse. Legislation that directly addresses reuse
and liability is another option for clarifying
program priorities and management. PaintCare
incurred $2.4 million in transportation and
processing costs to collect 469,665 gallons of
paint, or $5.13 per gallona value 20 times
more than the reuse incentive offered ($0.25
per gallon)indicating a potential cost savings
in shifting more paint to reuse and room to
increase the reuse incentive.
Evaluation of the Oregon Paint Stewardship Program
-------
9.0 Household Hazardous Waste
Programs
- -f ^ EVALUATION QUESTION:
What was the impact of the program on the HHW
facilities in terms of the types of paint collected,
costs, and the way in which the facilities operate?
HHW facilities are a key stakeholder in the
evaluation and an important part of the infrastructure
of the statewide post-consumer paint management
system. The Oregon DEQ solid waste program
website lists event and facility information for 17
HHW programs in the state. These programs accept
common household hazardous wastes (e.g., oil-based
paint, pesticides, and mercury thermometers) from
households, as well as conditionally exempt small
quantity generators (often by appointment or special
event) and then sort the various products for recycling,
reuse, energy recovery, or proper disposal. HHW
programs use a variety of mechanisms in order to
provide services to residents, such as:
Rural county local government solid waste
or public health programs may offer periodic
collection events.
The Tri-County Hazardous Waste and Recycling
program coordinates collection facilities and
events for Hood River, Sherman, and Wasco
counties.
Several counties hire companies that specialize
in hauling, disposal, and recycling to provide
HHW collection opportunities. For example,
Jackson and Josephine counties offer biannual
events through Rogue Disposal.
The number of customers served by a program
varies. One HHW interviewee noted that they serve
85 - 140 cars at each monthly event (S12). Another
interviewee reported that, in 2009 and 2010, their
permanent facility served approximately 4,000
customers (S10).
9.1 Data Sources, Methods
and Analysis
To answer Evaluation Question 7, the Evaluation
Team conducted phone interviews with seven
representatives from county HHW programs in Oregon
in June 2011 and data from the PaintCare annual report.
The Evaluation Team supplemented the interview data
with a review of materials related to the impact of the
program on HHW such as PowerPoint presentations
from HHW stakeholder meetings and other documents
provided by interviewees. HHW programs have limited
detailed quantitative data on paint volumes and costs.
There are several reasons why there is a limited
amount of quantitative data on HHWs. First, HHW
programs operate under lean budgets and do not
necessarily have the resources nor the expertise to track
detailed paint volumes or cost information. Second,
Federal agencies and their contractors are subject to the
Office of Management and Budget (OMB) Paperwork
Reduction Act requirements for information collection
and as a result EPA was limited to interviewing nine or
fewer individuals on this topic (i.e., a detailed survey of
HHWs was not feasible).
Promoting Environmental Results Through Evaluation 62
-------
9.2 Findings
9.2.1 Types of Paint Collected
at Household Hazardous Waste
Facilities
Table 7 shows a data summary of the PaintCare
annual report on the gallons of paint collected at
HHWs under the program. Interviewees noted that
they had not observed a significant change in the total
gallons of latex or oil-based paint collected since the
PaintCare program began; however, two interviewees
reported that the first event or month following the
program start collected very high volumes that then
tapered off through subsequent events (S4, S10).
In most cases, the interviewees reported that they
were able to accommodate the higher volumes with a
few adjustments. However, one HHW faced delays in
finalizing the contract with PaintCare and needed to
store collected paint until the contract was in place and
the paint could be picked up (S14). One interviewee
noted a shift towards collecting a higher proportion of
latex than oil-based paint (S14). Another interviewee
observed increased traffic from paint contractors
because the county charged a fee for large volumes
prior to the program (S10). After the program began,
one HHW site ceased collecting paint entirely (S15)
while another program began collecting latex paint
again after a long hiatus (S12).
9.2.2 Changes in the amounts of or
capacities for other products
Interviewees reported no significant changes in the
amounts of other HHW products and moderate shifts in
available capacity for processing other HHW products.
Three HHWs saw an increased capacity to handle
other products because they were able to reassign staff
from crushing paint cans and bulking the material for
disposal to other tasks (S4,S10, and Sll); one HHW site
stopped collecting paint (S15).
9.2.3 Costs for household
hazardous waste facilities to take
in and process paint
Interviewees noted that HHW programs do not
have sophisticated systems in place for detailed cost
tracking; staff may simply compare monthly internal
costs (e.g., labor, overhead) to external costs from
vendor invoices (e.g., hazardous waste hauler) (S12).
Additionally, these costs are not likely to be tracked by
task or material type, such as paint processing versus
other hazardous materials. As a result, the interviewees
were not able to provide detailed cost data, but many
were able to provide estimates of the changes in costs
following implementation of the program.
Five of seven of the interviewees reported
significant cost savings resulting from the program even
when accounting for increases in customer traffic and
material volume. The cost savings are reported to result
from the avoided labor, disposal, and transportation
Table 7 - Gallons of Paint Collected at Different HHW Facilities
between July 2010 and June 2011
Facility
Gallons
Collected
Columbia County
4,140
Beaver Hill Disposal Site
675
Crook County Solid Waste
188
Deschutes County
20,375
Tri-County Hood River
3,150
Marion County
2,970
Tri-County - The Dalles
1,890
Lane County
12,529
Lincoln Cty SW Transfer
Station
720
Capital Plant
585
Source: PaintCare, 2011 b, Appendix D.
Note: Metro's value was not broken out by HHWs versus other facilities.
Additionally, the values do not include collection events. HHWs not listed had
no value listed (i.e., a "-") in Appendix D of the PaintCare report.
63 Evaluation of the Oregon Paint Stewardship Program
-------
costs associated with handling oil-based and unusable
latex paints. Some examples of avoided costs from
interviewees included:
Saving $1,500 per month of a $100,000 annual
budget due to avoided paint disposal costs
(S12).
Cumulative savings of $45,000 to $50,000
resulting from less paint handling by staff and
avoided disposal, landfill, and transportation
costs (S10).
A cost decrease of 40 percent from pre-
program costs due to avoided disposal costs
associated with oil-based paints (S15).
Avoided costs of $75,000 to $100,000 per year
of an annual budget of approximately $300,000
due to avoided costs of disposal and transport
of paint (Sll).
A separate cost reported by three interviewees
involved the need for HHW staff to conduct outreach
and education before and during paint collection
(S13, S14, and S15). These interviewees felt that the
efforts by the program did not achieve sufficient reach
within their county. One interviewee supplemented
the program efforts with additional, locally targeted
promotion around the beginning of the program
(S15). Of particular interest to these interviewees
was providing additional information to customers
on the location of the nearest collection site, the types
of materials that would be accepted, and the purpose
of the fee. Two interviewees reported that some
customers mistook the fee for a bottle deposit and
returned empty cans in the hopes of getting the fee
returned (S12, S15). Two other interviewees reported
that customers were bringing in materials that did not
fit the criteria for program products; for example, some
customers were bringing in materials, such as asphalt
sealant, that are applied with a brush (S13, S12).
9.2.4 Cost of managing other
products
Data on the costs of managing other HHW products
is limited for the reasons described at the beginning of
Section 9.2.3. Two of the interviewees commented that
the costs of handling non-program materials returned
by customers could be significant. When asked about
the challenges associated with rollout of the program,
two interviewees noted that correctly identifying
program materials and non-program materials was
challenging at first, and that the volume of unacceptable
items, such as unlabeled or rusty cans, could be
significant (S4, S13).
9.3 Learning
Analysis of prior opportunities and
conditions can help predict how HHW
volumes may be affected by a paint
recycling program. Change in volume and
types of paint collected at HHW sites will vary
depending on the convenience and availability
of prior opportunities for consumers to return
paint. Some HHW programs stopped collecting
latex paint prior to program rollout due to
resource constraints and so experienced a
spike in paint volume when the program began.
Paintlatex in particularhas historically
been a significant percentage of the volume of
material collected by HHW programs. Shifting
processing of the paint to another entity frees
up resources for the HHW to handle other
materials. However, if the HHW is located in
a state with limited infrastructure for paint
collection, the HHW might experience the
opposite effect: capacity to process other
products may decrease significantly as
customers bring in a relatively high volume
Promoting Environmental Results Through Evaluation 64
-------
of legacy paint (especially if dropped off
with other materials at the same time). A CIS
analysis could assist HHWs to anticipate the
increased volume of paint under a recycling
program. An effective version of this type of
analysis would include paint purchasing data
related to demographics. For example, CIS can
be used to map and analyze the demographics
of an area (age, homeownership, etc.) in
the context of paint purchasing behavior.
Combining CIS with data on demographics
with paint sales in the area could identify the
potential quantity of paint that an HHW could
expect.
Labor and oil-based paint disposal costs
are the key cost elements for HHW. States
with limited pre-existing infrastructure for
paint collection may experience initial spikes
in labor needs at HHWs as customers bring
in relatively high volumes of legacy paint;
however, the program should mitigate the costs
of sorting, bulking, and disposal. Areas with
well-established infrastructure and steady
paint volume collected could see significant
cost savings.
Cost information is lacking for HHWs.
HHWs represent a significant set of program
stakeholders. Without knowing the costs
incurred to process paint at HHWs and the
cost avoided from paint recycled, the full cost
effectiveness of the program cannot be judged.
Education and outreach for HHWs to use
with their customer base may be needed.
Development of education and outreach
campaigns for future roll-out states should
include HHWs to ensure the inclusion of
complete and relevant information for HHWs.
Consideration should be given to non-
program materials and how to deal with
them. Collecting non-program materials
imposes costs on HHWs and quantities may
be significant. HHWs may need additional
support from the program to ensure that these
materials are correctly identified and sorted
separately from program materials. States with
limited prior infrastructure should anticipate
collecting older, deteriorated (e.g., "rusty"), and
unlabeled cans of paint and be prepared to sort
them from program paint.
65 Evaluation of the Oregon Paint Stewardship Program
-------
io.o Cost Effectiveness
EVALUATION QUESTION:
How cost effective is the program?
10.1 Data Sources, Methods
and Analysis
Cost-effectiveness can be assessed by comparing an
amount reflecting an outcome (e.g., paint collected) to
the costs incurred to achieve that outcome. To answer
this question, the Evaluation Team intended to analyze
cost data collected to answer Evaluation Questions 5, 6,
and 7 at a level that would have allowed for comparison
with the PPSI's estimated cost of $7.46 per gallon
collected, as found in Paint Product Stewardship Initiative
Infrastructure Project of a comprehensive system to
manage leftover paint (SCS and Cascadia, 2007).
The PaintCare Annual Report (PaintCare, 2011b)
provided cost information, amounts of collected paint
and cost per gallon collected information. However,
since PaintCare's budget does not reimburse for
collection costs, the program's budget does not
include the full system costs. Additionally, much of
the initial year's costs of PaintCare included higher
level of administrative costs due to efforts to educate
consumers and retailers (both about the fee and those
collecting) and other set-up costs associated with a
new organization and setting up an infrastructure.
Further, legacy paint entering the system will
represent significant volume in the program's early
years. Evaluation Question 8 may be more thoroughly
answered with several years of operating expenses
available.
10.2 Findings
Actual PaintCare program costs for the first year
were $7.03 per collected gallon, excluding collection
costs (PaintCare, 2011b). PaintCare's initial projection
of program costswhich excluded collection costs at
permanent facilitieswas estimated to be $8.26 per
collected gallon in the first year with a decreasing cost
of $7.30/gallon in year two, $7.27/gallon in year three,
and $7.18/gallon in year four (PaintCare, 2010). The
primary reasons for the lower costs compared to pre-
program include that a higher percentage of latex paint
was collected as compared to oil-based (latex paint
is more expensive to handle). In addition, less non-
program materials were collected, so overall disposal
costs for these items were less than expected (Em2).
Despite the overall lower than expected costs, some
initial year expenses were higher than expected. For
example, start-up costs (e.g., legal and banking fees)
and cost for education and outreach efforts were higher
than expected (S17).
Figure 34 presents the cost per gallon from the
Oregon program alongside other estimated costs per
gallon to process leftover paint from other programs
and analyses. Table 8 provides details on the different
cost per gallon estimates. Although the $7.03 per gallon
estimate is lower than other estimates, the PaintCare
estimate excludes some aspects (labor for collection)
that are included in other, higher estimates.
PPSI's life cycle workgroup was scoped to have
a full cost-benefit analysis conducted on the same
management scenarios as were modeled in the LCA
(PSI, 2009). Such an effort would have examined cost
effectiveness from a social perspective, accounting
Promoting Environmental Results Through Evaluation 66
-------
for additional costs such as consumer time to drop off
paint and externalities associated with environmental
impacts (e.g. air emissions, greenhouse gases, and
water use).
10.3 Learning
The exclusion of HHW collection costs
from the total program costs is significant.
Even though it appears that costs decreased
for HHWs, gauging total program cost
effectiveness is not possible without knowing
collection costs for HHWs. Calculating a value
for collection costs is complex. Some variables
such as estimating the cost of labor are
relatively straightforward, while other items
such as valuing the loss in building space that
is used for storing waste paint and the percent
of overhead expenses (e.g., insurance and
administrative) that should be attributed to the
collection are more challenging to estimate.
Expand cost-effectiveness to include
environmental benefits. Product stewardship
initiatives can broaden the scope of cost-
effectiveness by factoring in environmental
benefits. The cost-effectiveness measured
here is the cost per gallon collected. A more
comprehensive measure would translate the
gallons into environmental benefits and also
include other management options (e.g., reuse)
as well as the cost-effectiveness of reducing
leftover paint by convincing consumers to "buy
the right amount." Completion of a full life cycle
cost-benefit analysis could supply some of this
information. The key is to develop a measure
of the environmental benefit associated with
different paint management options.
Expand cost-effectiveness to include
program development. Beyond expanding
cost effectiveness considerations to
environmental benefits, product stewardship
initiatives can consider the cost-effectiveness
associated with program development,
including the costs of collaboration.
Collaboration for this program involved
significant time and effort on the part of several
stakeholders. However, what did collaboration
achieve relative to a less collaborative
approach? Addressing this question would
allow for a more complete assessment of this
type of program and for developing programs
through a collaborative approach.
Figure 34 - Costs per Gallon Collected from Oregon Program and Other Programs or Studies
$15.00
$12.00
$9.00
$6.00
$3.00
$0.00
1997-1998 Pilot,
collection events
($2011)
1997-1998 Pilot,
retailers ($2011)
Paint
Infrastructure
project estimate
Projected by
program
Actual
67 Evaluation of the Oregon Paint Stewardship Program
-------
Table 8 - Details on Different Cost per Gallon Estimates
Data Source
Amount[a] Cost Elements Included
Cost Elements
Excluded
PaintCare - actual
program costs
(2010 to 2011)
$7.03
($2011)
Education and outreach
Materials for collection (e.g., bins)
Transportation from collection site
Processing, recycling, and
disposal
Labor at collection events
Labor at permanent
collection sites
Indirect costs
(e.g., storage space)
Social and environmental
costs (e.g., air pollution)
Paint Smart Pilot
Program
(Cascadia, 1998)
Special
collection
events: $13.86
($2011)
Materials for collection (e.g., bins)
Transportation from collection site
Processing, recycling, and
disposal
Labor at collection events
Education and outreach
Labor at permanent
collection sites is not
applicable.
Indirect costs
(e.g., storage space)
Societal costs
(e.g., air pollution)
Paint Smart Pilot
Program
(Cascadia, 1998)
Retailer Materials for collection (e.g., bins)
collection: Transportation from collection site
$12.24 Processing, recycling, and
($2011) disposal
Education and outreach
Collection labor
Indirect costs
(e.g., storage space)
Societal costs
(e.g., air pollution)
PPSI Infrastructure
Report (SCS and
Cascadia, 2007)
Model of
national
infrastructure:
$8.15 ($2011)
Education and outreach
Materials for collection (e.g., bins)
Transportation from collection site
Processing, recycling, and
disposal
Labor at collection events &
permanent collection sites
Indirect costs
(e.g., storage space)
Societal costs
(e.g., air pollution)
[a] Cost estimates inflated to 2011 dollars using U.S. Bureau of Labor Statistics' Inflation Calculator. Available at www.bls.gov/data/inflation_calculator.htm.
Promoting Environmental Results Through Evaluation 68
-------
11.0 Waste Hei
(to be completed Jan 201 2)
^ . L EVALUATION QUESTION:
*
How was the program designed and implemented to
move consumers up the waste hierarchy?
i
With respect to moving customers up the waste hierarchy, what were the
program's obstacles, opportunities, and decisions?
A "waste management hierarchy" classifies waste
management options according to their desirability.
PPSI considers reducing leftover paint to be the
preferred management option, followed by reuse,
recycling (including energy recovery), and disposal. A
key goal of the program is to encourage greater reliance
on the most preferred strategies in the hierarchy and
less reliance on least preferred strategies (i.e., "moving
up" the hierarchy). This question assesses how well the
program has achieved that goal.
Many obstacles stand in the way of waste reduction.
Thus, Evaluation Question 9 also considers the
obstacles, opportunities, and decision-making related
to moving up the hierarchy. Oregon and other states
can use this information to design paint stewardship
programs that more efficiently encourage reliance on
the most preferred management options.
The Evaluation Committee has partnered with
Brielle Kissell, a graduate student at Tufts University, to
investigate this question as part of her master's thesis.
Ms. Kissell is currently working on developing the
analysis for Evaluation Question 9. At this point, we can
present the approach proposed in the method, as well
as some initial findings related to the waste hierarchy
from other questions.
11.1 Data Sources, Methods
and Analysis
The method for answering this question will
involve relating the components of the program to
the categories of the waste hierarchy (reduce, reuse,
and recycling) and then compiling information on
the obstacles, opportunities, decisions, and relative
emphasis related to each category of the waste
hierarchy. The matrix may look something like the one
at the bottom of the page.
The Evaluation Committee will explore the
degree to which the paint stewardship program
emphasizes waste reduction over others as reflected in
educational materials, convenience, infrastructure, and
performance. The Evaluation Committee will compare
the effectiveness of waste reduction messages to others,
resources devoted to waste reduction compared to
other management methods, and the number of paint
exchanges and volumes exchanged before and after
paint stewardship.
Waste hierarchy
category
Program
component
Decision made
Obstacles
Opportunities
Relative emphasis
69 Evaluation of the Oregon Paint Stewardship Program
-------
11.2 Findings and Learning (From Other Questions)
As noted above, the detailed answer to Evaluation Question 9 is currently in process. Table 9 provides a
summary of findings and learning from other evaluation questions related to waste hierarchy.
Table 9 - Findings and Learning Related to Evaluation Question 9 from Other Evaluation Questions
Other Evaluation
Question
Learning in Question 4
Findings in Question 6
Learning in Question 6
TiTSrTiTn
To achieve goals of reducing the generation of post-consumer
architectural paint, paint management programs and policies may
explicitly document the relative emphasis they want to place on aligning
the paint management system with the existing waste hierarchy. For
instance, prioritize options such as reuse of leftover paint, reducing
leftover paint through improving accuracy of paint purchases (less
leftover) and increasing volumes of managed post-consumer paint.
Organizing these priorities will provide a structure to prioritize outreach
strategies, messages, materials and effort.
57 percent of latex paint (approximately 201,000 gallons) was recycled as
paint by Metro and Amazon (PaintCare, 2011 b).
3 percent of latex paint (approximately 10,000 gallons) and 5 percent of
oil-based paint (approximately 3,500 gallons) was reused by consumers
(PaintCare, 2011 b).
PaintCare provided collection locations an incentive of $0.25 per "reuse"
gallon to reflect avoided transportation and processing costs. An
organizational representative notes that no sites took advantage of this
incentive; sites that were already offering reuse had systems in place and
considered it an important service (S17).
Less than 10 percent of collection locations have shelves for paint
exchange; PaintCare did not recruit retail sites for paint reuse due to
concerns about liability (S17).
Recycling is lower on the waste hierarchy than reuse, but there are
barriers to increasing paint reuse. Furthermore, recycling results in higher
costs (management, environment) compared to reuse. More information
is needed to better understand the barriers (e.g., liability, space) to retailer
facilitation of paint reuse
The private-public partnership and the waste hierarchy model (reduce,
reuse, recycle) were at odds in this program. The public preference
in the waste hierarchy model is for reduction and reuse. There were
opportunities to stress both reduction and reuse under this program,
however, the program as-designed tended to focus more on collection
and recycling. Other state programs can more clearly articulate program
priorities to ensure alignment with existing program and policy goals and
reduce the potential for confusion amongst consumers.
Unintended Consequence
in Question 12
Promoting Environmental Results Through Evaluation 70
-------
i2.o Market for Post-Consumer Paint
- - ' EVALUATION QUESTION:
*
How has the market for post-consumer paint been
* affected by the program?
i
What aspects of the program have had an impact on the market and how?
What markets and products represent potential opportunities for
post-consumer paint products?
Expanding the market for post-consumer paint
products is an important element of the pilot program
and could support improved management of leftover
paint. Ideally, if more consumers used post-consumer
paint, then they would generate less waste because
less new paint would be purchased (and ultimately
need to be disposed). Additionally, leftover paint can be
reprocessed into alternative non-paint products such
as cement additive. An increase in businesses using
non-paint products produced from post-consumer paint
could also reduce the amount of paint flowing into the
waste stream. This question assesses the extent to which
the program has helped develop a market for post-
consumer paint.
12.1 Data Sources, Methods
and Analysis
Evaluation Question 10 was answered by performing
a qualitative assessment of the market by reviewing
available information on the post-consumer market
for paint in Oregon. The measures consist of a listing of
the post-consumer paint products offered, the facilities
that offer them, and the total sales of these products.
The data sources for these measures were interviews
with staff from PaintCare, recycled paint manufacturers,
and HHW facilities, as well as data and information
maintained by the program on these products. Based
on the information collected from the program and
Oregon DEQ, the Evaluation Team performed additional
research into the post-consumer paint products market
in Oregon to verify information from the interviews and
to collect additional information on these products and
markets. This additional research consists of both Web
searches and interviews with non-program paint experts
in Oregon.
12.2 Findings
12.2.1 How has the market for
post-consumer paint been affected
by the program?
As part of the first PPSI MOU, stakeholders
identified the need to develop post-consumer paint
markets as a key issue (PPSI, 2004). The following three
projects were suggested:
Market Development Strategy: Research
potential markets for recycled paint and
develop model procurement policies. Only
initial funding was provided.
71 Evaluation of the Oregon Paint Stewardship Program
-------
Recycled Paint Marketing Guide for
Distributors: Develop a marketing guide for
both public and private distributors, targeting
a variety of market segments. This project
was placed on hold until the Recycled Paint
Certification System was completed and
additional funds were raised.
Recycled Paint Standard and Certification
System: Working with Green Seal, PPSI developed
guidelines and performance specifications for
recycled paint, along with a system for certifying
specific recycled paint products. This project was
completed in August 2006.
The Oregon pilot program's primary focus,
however, has been on the collection and management
of leftover paint and has not focused on developing a
market for post-consumer paint (S17). The education
materials developed under the program focus on the
collection aspects of the program and not on consumer
reuse or purchasing.
In Oregon, there are three types of paint that fall
into the category of post-consumer recycled paint and
that can be considered part of the post-consumer/
recycled paint market for Oregon:25
Leftover paint (reuse): paint that is dropped
off and could be used as-is by another
consumer. This is the type of paint offered for
paint exchange by PaintCare through Restores
in Oregon.
MetroPaint: Metro sells recycled paint under
the name MetroPaint, which contains 2 percent
or less of total volume of virgin paint and
additives (Metro, 2011).
Amazon Select: Amazon Environmental Inc.
sells Amazon Select, a recycled paint with 95
percent recycled content (S12).
The amount of paint collected under the program
and made available for the market, distribution
channels, and amounts sold for each product are
summarized in Table 10. In terms of reuse, 3 percent
of latex (10,564 gallons) collected under the program
and 3 percent of the oil-based paint (3,526 gallons)
collected under the program was available (supply) as
leftover paint (PaintCare, 2011b).
According to the PaintCare report, 53 percent of
the latex collected under the program (186,632 gallons)
was sent to Metro to become recycled paint (PaintCare,
Table 10 - Summary of Post-Consumer Paint Products in Oregon: Amounts Available, Distribution Channels, and Amounts Sold
Post-consumer
paint product
Amount of
paint available
for product
Distribution
channels
Leftover paint - paint that is dropped
off and could be used as-is by another
consumer
10,564 gallons
of latex
3,526 gallons
of oil
Restores in
Oregon
Amount
of program-
related
product sold
Unknown
Recycled paint (Metro) - small amount
of virgin paint and additives (2 percent
or less of total volume) are combined
with leftover paint
186,632 gallons
of latex
45 retailers in
Oregon and
Washington
115,343 gallons
of MetroPaint
Recycled paint (Amazon Environmental)
- small amount of virgin paint and
additives (5 percent or less of total volume)
are combined with leftover paint
14,085 gallons
of latex
Sold as
Amazon Select
Not available*
25 The categories below do not represent official definitions, but reflect the Evaluation Team's view of the types of products on the market.
Promoting Environmental Results Through Evaluation 72
-------
2011b). Metro sold 122,705 gallons of paint between
July 1,2010 and June 30, 2011 (S16). During that time,
94 percent of the paint Metro took in came from the
Oregon program. Thus, approximately 115,343 gallons
of paint (2 5 percent of all paint collected by PaintCare)
taken in by the program between July 1, 2010 and
June 30, 2011 were sold by Metro as post-consumer
paint. Metro has been selling post-consumer paint
(MetroPaint) since 1992 and consumers can purchase
MetroPaint at more than 45 retailers throughout Oregon
and Washington (Metro, 2011). Metro indicated that the
program had no effect in their sales of post-consumer
paint. For the past several years, independent of the
Oregon pilot program, Metro increased sales of post-
consumer paint by expanding the number of outlets
offering its paint (S3). Metro reported that demand for
light color recycled paint exceeds the program's supply.
The opposite is true of darker colors (S16).
According to the PaintCare report, 4 percent of latex
collected under the program (14,085 gallons) was sent
to Amazon to become recycled paint (PaintCare, 2011b).
Amazon, Inc. considers its sales figures as proprietary
and did not provide these figures to the Evaluation
Committee. However, Amazon indicated that only a
small percentage of its paint sales stem from paint
collected through the Oregon program (S21).
Table 11 summarizes the total amounts of latex
and oil-based paint collected under the program and
the amounts that were made available in the post-
consumer paint markets. In total, 217,157 gallons of
the paint collected (46 percent of the total amount
collected) under the program made it into the post-
consumer paint market. For latex paint, 60 percent of
the amount collected under the program was made
available to the post-consumer paint market. For oil-
based paint, 3 percent of the paint collected made it
into the post-consumer paint market.
Nationally, the annual potential market for
post-consumer latex paint products is $1.2 billion
and includes environmentally conscious adults
($540 million) and contractors ($230 million), and
governments ($420 million) (Hult, 2011). According
to the 2010 Census, Oregon represented 7.6 percent
of the U.S. population. Using this percentage to adjust
the national numbers, the annual potential market in
Oregon would be approximately $90 million.
Table 11 - Amounts of Latex and Oil-Based Paint Collected Under the Program and Made Available in Post-Consumer Paint Markets
Paint Type
Latex
Total Gallons Collected
Under the Program
352,136
Total Gallons Collected Under
the Program Made Available in
Post-Consumer Paint Markets
(Percent of total)
211,281 (60%)
Oil-Based
117,529
3,526 (3%)
Latex and Oil
469,665
217,157(46%)
73 Evaluation of the Oregon Paint Stewardship Program
-------
12.2.2 What aspects of the
program have had an impact on the
market and how?
The Oregon program has an indirect impact on
developing post-consumer paint markets by promoting
recycling over other management options (e.g.
disposal) and by increasing the supply of leftover paint.
By encouraging consumers to return the paint, the
program is developing the supply side of the market.
The centralized collection of paint by the program is
funneling more than 200,000 gallons of leftover paint to
be recycled into paint for resale to consumers.
12.2.3 What markets and products
represent potential opportunities
for post-consumer paint products?
There are two post-consumer non-paint products
being produced from Oregon leftover paint: Amazon
Environmental's cement additive (PLP) and a binding
material for biomass that Amazon Environmental uses to
fuel their recycled paint facility (PaintCare, 2011b). The
sale of recycled latex paint is challenged by the technical
limitations of recycled paint colors and consumer
perceptions on quality (Hult, 2011).
12.3 Learning
Tracking data on the post-consumer
paint market is an important program
component. One component of determining
the effect of the program on the market for
post-consumer paint is tracking data on post-
consumer paint sales, including both price and
volume.
The existence of a market for post-
consumer paint was important. A paint
recycling program can help to develop a market
for post-consumer paint. In Oregon, a market
existed already to some degree, as Metro has
been selling MetroPaint since 1992. Recycling
programs can help to develop markets in
states without existing markets by identifying
retail outlets and assessing potential paint
reprocessing or recycling facilities and capacity.
Standard market indicators are not
adequate for the post-consumer paint
market. The market outcomes for paint should
be viewed in a full cost accounting framework.
Market price, quantity sold, and profitability
are limited measures of the success of a market
for post-consumer paint. A broader view
would take into account the avoided costs of
managing the leftover paint and the reduced
environmental impacts of recycling the paint.
Additional information is needed to
determine impacts. The paint market in
Oregon encompasses consumers, retailers, and
producers, but also PaintCare and Oregon DEQ.
In building the market for post-consumer paint,
the program will have an impact on each set of
actors. However, we have little information on
how the Oregon program has impacted each
group. Some of the impacts and key questions
to consider include:
Consumers: The post-consumer
paint market will offer a lower-
priced alternative to virgin paint.
Will the availability of a lower-priced
alternative impact the price consumers
pay for virgin paint? Will consumers
accept post-consumer paint as a viable
alternative to virgin paint?
Promoting Environmental Results Through Evaluation 74
-------
Retailers: Will retailers begin to
stock post-consumer paint? Will
the availability of a lower-priced
alternative reduce sales of virgin paint?
How will profitability of retail paint
sales be affected?
Producers: Will the availability of
post-consumer paint affect producers'
sales of virgin paint? Will producers
consider expanding their product lines
into the post-consumer market and
begin offering post-consumer paint?
Providing answers to these questions will require
extensive and more detailed research into the impacts
of post-consumer paint on paint markets in general.
Addressing consumer concerns about
leftover and recycled paint is important.
Information is needed on consumer acceptance
of leftover and post-consumer/recycled paints
as a viable product for them to use. This basic
level of market research would provide a
baseline for the potential size of the market
and also help to identify specific concerns that
consumers may have. These concerns could
then be addressed as part of an education and
outreach campaign to promote the use of post-
consumer paint as a viable alternative to virgin
paint. Furthermore, an education and outreach
campaign that targets messages to different
audiences may be more effective at increasing
acceptance by consumers. Although the PPSI
Recycled Paint Standard and Certification
System project was completed, stakeholders
have not funded the Recycled Paint Marketing
Guide for Distributors project. The intended
purpose of that guide is to educate buyers
about the nature of recycled content paint,
including its quality and performance.
Pricing of leftover/recycled paint may
be complex. Some consideration should be
given to balanced pricing for leftover paint.
A low price for leftover paint may encourage
consumers to purchase too much leftover paint,
leading to the paint being leftover again. This
needs to be balanced against the cost of new
paint. In short, leftover paint should be priced
at a level that both represents a significant
discount compared to new paint but not too
low that it encourages consumers to purchase
"extra" paint they may not need. Currently,
little information on paint markets exists to
determine that tipping point and research
into consumer preferences may be needed to
determine a balance.
75 Evaluation of the Oregon Paint Stewardship Program
-------
i3.o Transferability
EVALUATION QUESTION:
Based on the Oregon experience, what implementation
and outcome-related information is required for
other states to develop and implement leftover paint
management systems?
To what extent are the performance measurement and evaluation systems
transferable to other states?
What are the best ways to communicate the results of the evaluation?
13.1 Data Sources, Methods 13.2 Findings and
and Analysis Learnings
Evaluation Questions 11 and 12 are designed as
pure learning questions. The Evaluation Team developed
answers to these questions from data and information
collected while answering other evaluation questions
and supplemented with interviews conducted with a
variety of stakeholders.
13.2.1 What implementation and
outcome-related information is
required for other states to develop
and implement leftover paint
management systems?
States developing paint management systems
should start with reliable baseline data. Baseline data
would include:
Volume data on paint: Baseline data
on paint sales, amounts of paint that are
managed through various methods in the state
(recycling, conversion to energy, etc.), and
amounts disposed.
Current infrastructure: Document the
extent of current infrastructure (recycling
facilities, locations, processing facilities, events,
transportation etc.), including:
Promoting Environmental Results Through Evaluation 76
-------
Information on the capacity (amount
of paint that could be processed or
collected).
The knowledge and experience of
those who would be working with the
leftover paint.
Attitudes toward recycling and waste
reduction among consumers and
those who would be working with the
leftover paint.
A link between infrastructure data
and volume data to determine the
amounts being collected, processed, or
transported within different parts of
the infrastructure.
A system map: Developing a map of how
paint is currently processed in the state
provides a useful starting point to developing
a new system. The map developed by the PSO
(Figure 7) is a starting point. However, a more
detailed process flow diagram would be better
suited for tracking the current movement of
paint in a state.
Cost information: Information on the costs
processors of paint incur, including costs for
HHWs to process paint.
Awareness: Information on consumer
awareness of disposal and paint recycling
options, paint-related environmental issues
and hazards, and information on where
consumers get their information.
Consumer, behavior, attitudes and
practices: Information on how consumers
decide on how much to buy, what they
currently do with leftover paint, and how
much leftover paint they currently have stored
in their homes. Additionally, information on
attitudes toward recycling and the environment
would provide useful information that can be
used to develop outreach and education (e.g., a
consumer base that has strong preferences for
recycling and preserving the environment may
not need convincing that recycling paint for
environmental reasons is a good idea).
As California and other states with robust
infrastructures roll out leftover paint management
systems, these states should first ensure that plans
incorporate pre-existing infrastructure into the
program. Oregon DEQ allocated a quarter-time
employee for its program responsibilities. OR DEQ
indicated that a quarter-time employee would have
been sufficient in fulfilling program responsibilities,
except for the extra demands generated by being
the first state (e.g., participating in the Evaluation
Committee; requests from other states for information,
etc.) (S20).
The Oregon program stakeholders identified the
following important gaps in the implementation and
outcome information currently available:
The cost to local governments (e.g.,
permanent HHW facilities and events) to act
as collection sites, including cost of labor and
the opportunity cost of storage space. Costs
reported by PaintCare may not capture the full
costs of the program, as local governments may
be "subsidizing" some of the costs by providing
labor for paint collection (S20).
The proportion of program costs
attributable to overhead and PSO
Structure. In the Annual Report, PaintCare
notes that revealing some vendor costs would
result in a breach of contract; there is interest
in the cost-effectiveness of employing 'layers"
of contractors in order to implement the
program (S20).
The best approach to recruit retail
collection sites. PaintCare used targeted
mailings to retailers to recruit them as potential
collection sites; as a result, the initial list of
retail collection sites was primarily based on
retailer self-selection into the program rather
than site location. Upon inspection some of the
initial sites were not suitable to act as collection
sites and that in the future, the organization
should visit each site before designating it as
a collection site (S17). Another stakeholder
interviewee expressed interest in a more
strategic approach to recruiting sites (S20).
77 Evaluation of the Oregon Paint Stewardship Program
-------
The best approach to measure
convenience. The Annual Report focuses
on the percent of Oregon residents that live
within a certain radius of the nearest collection
site as a measure of convenience. A more
comprehensive measure would include the
considerations addressed by Strickland (2011)
that included drive time and driving distance
to collection sites. Although the CIS analysis
to develop the drive-time and driving distance
measures is more complex, Strickland has
provided an approach that could be replicated
in other states with CIS capacity.
The majority of inquiries that PSI received about
the program relate to the fee (S19). Other states
developing these programs should consider providing
outreach to residents in general and paint consumers
more specifically related to the fee structure.
13.2.2 To what extent are the
performance measurement and
evaluation systems transferable to
other states?
The PPSI Evaluation Committee developed
the framework for the evaluation (i.e., Evaluation
Committee, evaluation questions, methods, and
measures) of the pilot program initially planned for
Minnesota. When the Minnesota legislation was not
signed into law, PPSI shifted its focus to Oregon. The
Minnesota framework was used to develop the full
methodology for evaluating the Oregon program and
could be transferred to other states as well. In short, the
model for this framework involves:
Working collaboratively with all
stakeholders to define the goals of
the program using the PPSI goals as a
starting point, but adapting those goals to
match state-specific context.
Drafting evaluation questions that are
relevant for the program based on those
goals. The evaluation questions that were
developed for Oregon are a starting point, but
other states may have other issues to explore.
The Evaluation Committee began with 27
questions and narrowed that number to 12
through a participatory prioritization process
(Appendix A of Evaluation Methodology) while
maintaining the core topics included in the
27 original questions. Because the Evaluation
Committee was focused on the opportunity to
maximize learning from the Oregon pilot, and
provide thorough information and a starting
point for performance management of future
state-wide paint management programs,
it was ambitiously comprehensive in its
approach to evaluating this program. Other
states may benefit from narrowing the scope
of questions and focusing effort and resources
on developing comprehensive information
about the issues most critical to the success of
a similar program in that particular state.
Determining the measures and data
necessary to answer the evaluation
questions and design collection methods
for obtaining those data. The Evaluation
Committee was ambitious in its design of
measures and some measures were not
activated or only partially activated because:
The program was in the design
phase and the level of information
that would be available throughout
implementation was an unknown.
By articulating what would be
measured, the Evaluation Committee
intended to influence what would be
done and what information would be
provided.
The measures designed are, the
evaluation committee believes, the
measures necessary to answer the
questions, so even if the measures
were not activated for the Oregon
pilot, they might be a starting point for
future similar programs.
Other states may want to consider identifying
the key measures for each evaluation question and
Promoting Environmental Results Through Evaluation 78
-------
work to ensure those data are collected. This requires
collaboration and explicit agreements between the
stakeholders to ensure that the data is collected,
credible, accessible, and timely. The evaluation method
document is designed as a flexible starting point;
interested states should identify key areas of interest
and prioritize matching evaluation questions and
measures, as well as plan data collection accordingly.
Some key caveats to the transferability of the evaluation
framework include that evaluation priorities must be
customized to the interests of each state's stakeholders
and full development of the methodology requires
commitment of resources (time and funding).
Additionally, fully integrating evaluation into program
design requires close coordination and integration of
the Evaluation Committee with the design, planning and
implementation teams and efforts.
This evaluation committee created a Web-based
model (referred to as a "fuzzy logic model" by the
Evaluation Team; see Figure 1) of the Oregon pilot
program to depict the program and related systems,
organizations, processes and products as well as
the role of measurement and evaluation in the
program. Other states may consider this approach
to depicting the program, evaluation design, and
ongoing performance management information (e.g
performance dashboards). For one, this type of model
allows the evaluation team to develop a depiction
of the program that can convey to a wider range of
audiences, including the public, a sense of what the
program is intended to accomplish and how it is
intended to operate. Second, a Web-based model allows
the program to maintain communication with program
participants about performance, results and outcomes.
Performance measurement and evaluation can be
costly. Building data collection into program operations
and creating partnerships offer opportunities to reduce
costs and increase the efficiency of data collection and
performance management. For example, evaluation
questions related to the impact of the education and
outreach campaign required baseline and follow-
up surveys of aspects of consumer awareness and
behavior. The Evaluation Committee expected to receive
the necessary data to determine impact of education
and outreach from surveys implemented by PaintCare,
but the baseline survey performed in August 2010
did not contain some data elements that were needed
to assess impact. Additional resources for evaluation
would have allowed for baseline and follow-up surveys
to have been implemented and conducted based on
the evaluation's needs. In the interest of efficiency and
reduced costs, the Committee partnered with Duke
University, GSU, and two graduate students to perform
analyses and data collection. This approach worked
well and allowed for data collection and analysis
despite funding constraints.
Oregon and other states considering implementing
product stewardship programs (paint or otherwise)
may consider a model similar to the Conservation
Measures Partnership's (CMP's) Open Standards for
the Practice of Conservation.26 The purpose of the Open
Standards is to address the following two key questions:
What actions are needed to achieve
a program's goals?
Are these actions effective in achieving
the goals?
To provide practitioners with the tools to
answer these two questions, the Open Standards
"bring together common concepts, approaches,
and terminology in conservation project design,
management, and monitoring in order to help
practitioners improve the practice of conservation.
In particular, these standards are meant to provide
the steps and guidance necessary for the successful
implementation of conservation projects" (CMP, 2007).
Figure 35 depicts the approach used by the Evaluation
Committee to incorporate evaluation into program
design of the Minnesota and Oregon pilot projects on
the top (in blue). On the bottom (in green) is the Open
Standards model, which depicts the process of ongoing
program adaptive management. In Oregon, this process
is already well underway, and this report is a part of the
program efforts to capture and share learning from the
Oregon pilot to encourage more effective practices in
the future.
26 www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_Version_2.0.pdf.
79 Evaluation of the Oregon Paint Stewardship Program
-------
Figure 35 - Shifting from Incorporating Evaluation into Program Design to CMP Open Standards
1. Evaluation Policy
2. Evaluation
Methodology
Integrating
valuation
into"
Program
Design
1. Team
2. Mission
3. Goals &
Objectives
4. Logic Model
Data Sources
2. Collection Methods
& Strategy
Analysis Tools
1. Context
2. Audience
3. Communication
4. Use
1. Conceptualize
Define initial team
Define scope, vision, targets
Identify critical threats
Complete situation analysis
2. Plan Actions & Monitoring
5. Capture & Share Learning
Develop goals, strategies,
assumptions, and objectives
Develop monitoring plan
Develop operational plan
Document learning
Share learning
Create learning environment
OPEN
STANDARDS
4. Analyze, Use, Adapt
3. Implement Actions & Monitoring
Prepare data for analysis
Analyze results
Adapt strategic plan
Develop work plan and timeline
Develop and refine budget
Implement plans
Promoting Environmental Results Through Evaluation 80
-------
13.2.3 What are the best ways to
communicate the results of the
evaluation?
The Evaluation Team spent time upfront during this
project to identify the information needs of different
stakeholders. This upfront communication helped
clarify what was needed for effective communication
with PSI, Oregon DEQ, PaintCare, rollout states and other
stakeholders (e.g., HHWs). However, the Evaluation
Team was also operating under tight resources and tight
deadlines. To effectively communicate to stakeholders,
the Evaluation Team developed a Web-based
model of the Oregon program to communicate with
stakeholders about the program design, planning, and
implementation and findings from the evaluation (www.
paintstewardshipprogram.com).
Based on the findings under Evaluation Question
1, PPSI participants prefer to use conference calls and
email to communicate about the program. Participants
also expressed a preference for face-to-face meetings
when possible. The best modes for communicating
evaluation results likely include the following:
Conference calls briefing stakeholders on the
results.
Emails summarizing results in the format of a
newsletter or fact sheet.
Presentation of results during a conference or
other in-person meetings.
The findings under Evaluation Question 1 also
suggest that the use of a website is an underutilized
mode for information communication. During the PPSI,
the website provided fairly limited information for
use (i.e. minutes from the most recent meeting) and
no opportunities for providing feedback or otherwise
interacting with the information. This mode could
become a useful venue for sharing evaluation results
with a few changes, as follows:
Provide more information on a regular basis.
Allow readers the opportunity to interact with
and respond to the information.
Update the information strategically. Add
new resources and materials to the website
around a focal point, such as a conference call
or reporting date, when the material is most
relevant to readers.
The evaluation website (www.
paintstewardshipprogram.com) is a visualization of the
program incorporating functional features that provide
deep and diverse information about the program. It
illustrates program implementation as well as the
performance measurement and evaluation system
and could be an effective tool for sharing information
on an ongoing basis. A website may be accessible to a
wider audience than other communication modes (e.g.,
conference calls) and would allow diverse audiences
to focus on different components of the system
(e.g., education and outreach materials, costs, and
environmental impacts).
States should also consider using a dashboard
reporting system to provide ongoing updates of
program performance. A dashboard system could be
incorporated into the web-based program model and
would involve reporting a few key results on a frequent
basis (e.g., volume of paint collected, number of collection
sites, and number of leftover paint gallons sold).
Evaluation of the Oregon Paint Stewardship Program
-------
i4.o Unexpected Results
EVALUATION QUESTION:
During the program and for each of its primary
components, what were the primary external,
unexpected and/or unintended influences and
consequences?
Not all of the changes in paint management were
driven by the program itself. A number of factors
could also have influenced paint usage and handling
by consumers including economic activity and
characteristics of the paint market. Some consideration
must be given to external influences on consumer
behavior. Additionally, programs often have unintended
or unforeseen consequences. This section documents
some of these results and their implications for Oregon
and other states.
14.1 Data Sources, Methods
and Analysis
The Evaluation Team answered Evaluation Question
12 by reviewing the lessons learned from the program
and through interviewing those involved in program
implementation. The interviews covered the different
groups involved in planning and implementation and
extracted information related to different stages in the
program's development.
14.2 Findings and Learning
External
Minnesota Governor Tim Pawlenty twice vetoed
the bill to create an industry-managed statewide paint
stewardship and recycling program after the Minnesota
legislature had passed legislation twice by near-
unanimous margins. Governor Pawlenty's rationale was
that the bill would put a double burden on the public,
since it authorized a new fee to fund the program,
and local governments currently spend public funds
to dispose of and recycle paint (Pawlenty, 2009). The
PPSI organized collective support from industry, local
governments, EPA, and environmental organizations
for the Minnesota legislation; the politics in blocking
the program was unanticipated and delayed the
implementation of a pilot program for 1.5 years (S19).
Unexpected
Effective collaboration was more time
consuming and costly, involving more meetings,
calls and committees than expected (See
Evaluation Question 1). Few parties involved
had any notion of how time-consuming the
program would be and did not know what to
budget for collaborative work (e.g., meetings,
reviewing products, etc.). Additionally, seven
years passed from the beginning discussions
to implementation of a pilot program - a longer
time-frame than was anticipated by most
participants.
Some retailers declined to show the fee on the
receipt. The PPSI anticipated that retailers
would show the fee to indicate the extra amount
was not part of their price but the PPSI was not
aware that reprogramming store systems was
such a burden to store managers/owners
Promoting Environmental Results Through Evaluation 82
-------
The process became less collaborative when
industry was required to submit a plan,
negotiate with Oregon DEQ, and implement
the approved program plan. This breakdown
in collaboration was caused partly by lack of
funding for a facilitator/champion (S19, S20).
There were a large number of unlabeled
and/or rusty cans of paint that cannot be
accepted by the program. The collection sites
are instructed to not open the cans of paint
and those cans are treated as non-program
materials which must be managed by Oregon.
(See Evaluation Question 5)
PPSI had limited representation from retailers
and so the fact that retailers play a large role
in the Oregon paint collection infrastructure
was surprising (S18). Additionally, large retail
chains (e.g., Lowe's, Home Depot, and Wal-
Mart) declined to participate.
That legislation was needed to implement
a program was unexpected. The need for
legislation arose when issues of anti-trust and
collusion arose since paint producers would
be working together. Thus, legislation was
needed to exempt the paint producers from
anti-trust and collusion concerns. At the start,
the idea of the program was to avoid having
states pass legislation to tell the private sector
what to do the program was intended to be
voluntary.
The lack of a champion/funder to collect real
world data under the Oregon pilot to inform
appropriate levels of virgin paint offset to
model in an LCA meant that no additional
progress was made on estimating the
environmental impacts of paint management
approaches.
The evaluation committee was unable to get
cost data broken out by category to perform a
more thorough analysis.
Unintended
The private-public partnership and the waste
hierarchy model (reduce, reuse, recycle) were at odds
in this program. The public preference in the waste
hierarchy model is for reduction and reuse. There
were opportunities to stress both reduction and reuse
under this program. However, the program as-designed
tended to focus more on collection and recycling. Other
state programs should consider a clear articulation
of the program priorities to ensure alignment with
existing program and policy goals and reduce the
potential for confusion amongst consumers.
The reliance on retailers as collection locations
had a negative impact on the program's ability to divert
high quality leftover paint for reuse, which is ranked as
a preferred use. Further exploration into this topic
what are the obstacles (e.g., liability and competition
for the sale of virgin paint) and solutions (e.g.,
redistribution to HHW collection locations or Reuse
type stores)is needed.
83 Evaluation of the Oregon Paint Stewardship Program
-------
i5.o Reference Materials and
Abt Associates (Abt). 2007. Quantifying the Disposal of Post-Consumer Architectural Paint, U.S.
Environmental Protection Agency - Sector Strategies Program.
Amazon. 2010. Amazon Environmental, Inc. Latex Paint Recycling. Available at: www.amazonpaint.com/
other-products.html.
Bledsoe, Wes, Eric Graves, and Andrei Roman. 2011. Description of the Oregon Pilot Program Paint
Stewardship Organization. Prepared for PPSI Evaluation Committee and PMAP 8900 Public Service
Capstone course in the Andrew Young School of Policy Studies, Georgia State University.
Bradshaw Advertising. 2010. PaintCare Awareness Survey of Oregon Homeowners. Report prepared for
PaintCare.
Braunz, Amy, Julie Calvin, Whitney Knapp, Hedrick Strickland. 2010. Assessment of Collaboration of
the Paint Product Stewardship Initiative (PPSI) and Pilot Program. Prepared for PPSI Evaluation
Committee and ENV 280 Social Science Survey Design course at Duke University.
Cascadia Consulting Group (Cascadia). 1998. Paint Smart Pilot Program Evaluation and Final Report.
Prepared for Oregon Department of Environmental Quality. Available at: www.deq.state.or.us/lq/sw/
prodstewardship/painthtm.
Conservation Measures Partnership (CMP). 2007. Open Standards for the Practice of Conservation.
Available at: www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_
Version_2.0.pdf.
Cousins, J. Bradley and Lorna M. Earl. 1992. "The case for participatory evaluation," Educational
Evaluation and Policy Analysis, 14, 397-418. p.399
Cousins, J.Bradley and Elizabeth Whitmore. 1998. "Framing Participatory Evaluation," New Directions for
Evaluation, No. 80, Winter 1998, 5-23.
Eastern Research Group, Inc. 2010. Method for Evaluating the Paint Product Stewardship Initiative's
Oregon Pilot Program. Prepared for U.S. EPA Evaluation Support Division.
Promoting Environmental Results Through Evaluation 84
-------
Hult International Business School Project Team 2. 2011. Paint Recycling: An Environmentally and
Economically Sustainable Business Model. Boston, MA.
Keene, Matt, Lou Nadeau, Amy Stillings and Jennifer Nash. 2010. Memo entitled "Leftover Paint Oregon
Pilot Project Evaluation: Prioritizing Evaluation Questions," Oregon Paint Stewardship Pilot Program
Environmental Committee. Available at: www.paintstewardshipprogram.com/images/pe_leftover_
paint_oregon_pilot_project_evaluation_prioritizing_evaluation_questions.DOC
Keene, Matt. 2008. Integrating Evaluation into Program Design. Available at: www.productstewardship.
us/displaycommon.cfm?an=l&subarticlenbr=754. Available at: www.paintstewardshipprogram.
com/images/pe_integrating_evaluation_into_program_design_06_01_08.DOC and www.
paintstewardshippro gram.com/images/pe_inte grating_evaluation_into_oregon_pilot_program.DOC
Metro. 2011. MetroPaint: Quality Recycled Latex Paint, www.oregonmetro.gov/index.cfm/go/by.web/
id=521.
National Paint and Coatings Association. February 2008. The Paint Industry Works Toward a Nationally
Coordinate System for Post-Consumer Paint Management, Issue Backgrounder. Available at: www.paint.
org/pubs/background.cfm.
Oregon Legislative Assembly. 2009. Committee of Sustainability and Economic Development, 75th
Assembly, 2009 Regular Season. HR 3037-A. Available at www.leg.state.or.us/09reg/measures/
hb3000.dir/hb3037.en.html.
Oregon Department of Environmental Quality (DEQ). 2011. Oregon Household Hazardous Waste Report,
2008-2009.
OR DEQ. 2008. Oregon Department of Environmental Quality Household Hazardous Waste Survey: Results
Report. Portland State University Survey Research Lab: Portland, OR.
PaintCare. 2010. Oregon Paint Stewardship Plot Program Plan. Washington, DC.
PaintCare. 2011a. PaintCare Awareness Survey of Oregon Residents. Washington, DC.
PaintCare. 2011b. Oregon Paint Stewardship Pilot Program Annual Report. Washington, DC.
Patton, Michael Quinn. 2010. Developmental Evaluation: Applying Complexity Concepts to Enhance
Innovation and Use. The Guilford Press.
Governor Tim Pawlenty. 2009. "Letter to President Metzen," State of Minnesota (May 21, 2009).
Available at: www.calpsc.org/products/docs/2009/5-21%20Metzen%20VETO%20CH%20121%20
SF%2 0477.pdf
85 Evaluation of the Oregon Paint Stewardship Program
-------
Paint Product Stewardship Initiative (PPSI). 2004. Paint Product Stewardship Initiative
Memorandum of Understanding, Available at: www.productstewardship.us/displaycommon.
cfm?an=l&subarticlenbr=118.
PPSI. 2007. Final Paint Product Stewardship Initiative 2nd Memorandum of Understanding. Available at:
wwwproductstewardship.us/displaycommon.cfm?an=l&subarticlenbr=118
PPSI. 2008. Paint Product Stewardship Initiative paint Demonstration Project Committee Draft Work Plan.
Product Stewardship Institute (PSI). 2004a. Paint Product Stewardship: A Background Report for the
National Dialogue on Paint Product Stewardship. Boston, MA.
PSI. March 2004b. Product Stewardship Action Plan for Leftover Paint.
PSI. 2009. "Paint Dialogue Stakeholder Meeting: Final Meeting Notes," Portland, Oregon. December 9-10,
2009. Available at: www.productstewardship.us/associations/6596/files/FINAL_PPSI_Meeting_
Summary_Portland_OR_Dec9-10_2009.doc
PSI. 2010. Extended Producer Responsibility Legislation as of August 2010. Available at: http://
products tewardship.us/displaycommon.cfm?an=l&subarticlenbr=615.
SCS Engineers and Cascadia Consulting Group (SCS and Cascadia). 2007. Paint Product Stewardship
Initiative Infrastructure Project, Washington Department of Ecology. Available at: www.
products tewardship.us/displaycommon.cfm?an=l&subarticlenbr=12 8.
Strickland, Hedrick. 2011. Convenience analysis of the Oregon Paint Management Pilot Program. Duke
University, Masters Thesis, advised by Dr. Lynn Maguire and Matt Keene, EPA.
Thompson, Ann Marie, James L. Perry, and Theodore K. Miler. 2007. "Conceptualizing and Measuring
Collaboration," Journal of Public Administration Research and Theory, 19: 23-56.
Torres, Preskill and Piontek. 2005. Evaluation Strategies for Communicating and Reporting: Enhancing
Learning in Organizations. 2nd Ed. Sage Publishing.
NOTE: Stock photography purchased from www.123rf.com. Special thanks to Duke University for permission
to use the logo and image of Nicholas School of the Environment on page 24.
Promoting Environmental Results Through Evaluation 86
-------
Interviews and Personal Communications
K*Q
'
itact Number Contact and Organization
^^
Stakeholder Interview Number [S]
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
S19
S20
S21
Abby Boudouris, Oregon DEQ
Steve Sides, ACA
Jim Quinn, Metro
Jim Quinn, Metro [HHW]
Abby Boudouris, Oregon DEQ
Alison Keane, PaintCare
Tom Wroblewski, Home Depot [Retailer]
Rick Barnard, Rodda Paint Company [Retailer]
Jack Wickham, Ace Hardware [Retailer]
Larry Gibbs, Lane County [HHW]
Timm Schimke, Deschutes County [HHW]
Roy Weedman, Columbia County [HHW]
Jennifer Purcell, Tillamook County [HHW]
Tina French, Lincoln County [HHW]
Garry Penning, Rogue Disposal [HHW]
Jim Quinn, Metro
Alison Keane, PaintCare
Barry Elman, EPA
Scott Casell, PSI
Abby Boudouris, Oregon DEQ
Lorraine Segala, Amazon Environmental
09-23-2010
12-2010
12-2010
06-27-201 1
03-02-201 1
03-14-2011
05-03-201 1
05-09-201 1
05-13-2011
06-21-2011
06-21-2011
06-21-2011
06-28-201 1
06-29-201 1
07-01-2011
08-26-201 1
08-26-201 1
09-02-201 1
09-06-201 1
09-06-201 1
09-14-2011
Person Communications With ERG Via Email [Em]
Em1
Em2
Em3
Em4
Em5
Em6
Em 7
Em 8
Jim Quinn, Metro
Alison Keane, PaintCare
Mike O'Donnell, PaintCare
Jim Quinn, Metro
Albert Salvi, Washington State Department of Ecology
Alison Keane, PaintCare
Alison Keane, PaintCare
Alison Keane, PaintCare
08-24-201 1
08-26-201 1
09-14-2011
09-15-2011
09-08-201 1
09-08-201 1
09-09-201 1
09-12-2011
87
Evaluation of the Oregon Paint Stewardship Program
-------
Recycled/Recyclable Printed with vegetable oil based inks on 100% postconsumer, process chlorine free recycled paper.
------- |