United States
Environmental Protection
i Agency
             2011 Evaluation of the
             Oregon Paint
             Stewardship
             Program
             Promoting Environmental
             Results Through
             Evaluation

-------
Acknowledgements
    This evaluation was performed by Eastern Research Group, Inc. (ERG)
under subcontract to Industrial Economics, Incorporated (lEc) for the U.S.
Environmental Protection Agency's (EPA's) Office of Planning, Analysis,
and Accountability under Contract EP-W-07-028 between EPA and lEc.
The ERG evaluation team included Lou Nadeau, Ph.D., Amy Stillings, and
Lauren Jankovic. Matt Keene of EPA's Office of Policy, Evaluation Support
Division acted as the technical advisor.
    The Evaluation Team is grateful to the continuous and stalwart
support of the members of the Paint Product Stewardship Initiative
(PPSI) Evaluation Committee. We are especially thankful to the invaluable
assistance provided by Abby Boudouris, Alison Keane, Scott Cassel,
Barry Elman, Chris Metzner, Hedrick Strickland, and Brielle Kissell.
Without them, this work would not have been possible.
    This report was developed under the Program Evaluation Competition,
sponsored by EPA's Office of Policy. To access copies of this or other EPA
program evaluations, please go to www.epa.gov/evaluate.
       Evaluation of the Oregon Paint Stewardship Program

-------
TableofContents
Executive Summary	i - vi

1.0  Introduction	1
     1.1   Paint Product Stewardship Initiative
           Oregon Pilot Program	2
     1.2   Evaluation of Pilot Program	4

2.0  Overview of the Evaluation Method
     and Key Data Sources	6
     2.1   Evaluation Approach	6
     2.2   Communicating Information	8
     2.3   Evaluation Design	9
     2.4   Key Data Sources Used in the Evaluation .. 14

3.0  Collaboration	 17
     3.1   Data Sources, Methods and Analysis... 17
     3.2   Findings	 18
     3.3   Learning	23

4.0  Paint Stewardship Organization	25
     4.1   Data Sources, Methods and Analysis... 25
     4.2   Findings	26
     4.3   Learning	 31

5.0  Education and Outreach	 32
     5.1   Data Sources, Methods and Analysis... 32
     5.2   Findings	 33
     5.3   Learning	44

6.0  Consumer Purchasing Decisions	46
     6.1   Data Sources, Methods and Analysis... 46
     6.2   Findings	 46
     6.3   Learning	 48

7.0  Collection of Post-Consumer Paint	50
     7.1   Data Sources, Methods and Analysis... 50
     7.2   Findings	 50
     7.3   Learning	 56
                                                  8.0  Paint Reprocessing, Recycling, and Energy
                                                       Recovery	 58
                                                       8.1  Data Sources, Methods and Analysis... 58
                                                       8.2  Findings	 59
                                                       8.3  Learning	 61

                                                  9.0  Household Hazardous Waste Programs ... 62
                                                       9.1  Data Sources, Methods and Analysis... 62
                                                       9.2  Findings	 63
                                                       9.3  Learning	 64

                                                  10.0 Cost Effectiveness	 66
                                                       10.1 Data Sources, Methods and Analysis... 66
                                                       10.2 Findings	 66
                                                       10.3 Learning	 67

                                                  11.0 Waste Hierarchy
                                                       (to be completed by January 2012)	69
                                                       11.1 Data Sources, Methods and Analysis... 69
                                                       11.2 Findings and Learning
                                                            (From Other Questions)	70

                                                  12.0 Market for Post-Consumer Paint	71
                                                       12.1 Data Sources, Methods and Analysis... 71
                                                       12.2 Findings	 71
                                                       12.3 Learning	 74

                                                  13.0 Transferability	 76
                                                       13.1 Data Sources, Methods and Analysis... 76
                                                       13.2 Findings and Learning	76

                                                  14.0 Unexpected Results	82
                                                       14.1 Data Sources, Methods and Analysis... 82
                                                       14.2 Findings and Learning	82

                                                  15.0 Reference Materials and Interviews	84
                                             Promoting Environmental Results Through Evaluation

-------
Executive Summary
OVERVIEW
    In 2002, a coalition of paint manufacturers,
represented by American Coatings Association (ACA);
local, state, and federal environmental agencies;
retailers; and consumer and environmental agencies
formed the Paint Product Stewardship Initiative (PPSI)
and began negotiations facilitated by the Product
Stewardship Institute (PSI) to create an industry-
managed post-consumer paint management system.
After seven years of negotiations, two Memoranda
of Understanding (MOUs), and comprehensive
PPSI-sponsored research of the paint industry and
consumers, the state of Oregon became the first to
enact a law establishing product stewardship as the
preferred method for reducing the environmental
impacts and costs associated with leftover paint. This
report summarizes the results of the evaluation of
the Oregon paint recycling program performed by the
PPSI Evaluation Committee, which includes members
representing the diversity of participants in the PPSI.
   The Oregon program, depicted in Figure ES-1, is
comprised of a diversity of interconnected systems,
actors, and processes. The major components of
the Oregon program include the Paint Stewardship
Organization (PSO; PaintCare), the oversight of the
Oregon Department of Environmental Quality (DEQ),
the paint market (producers, retailers, and consumers),
and the leftover paint management system (collection
sites, processing, etc.). To fund the program, Oregon
added an assessment fee onto the price of paint
sold in the state. Full documentation of the details
of the program appears in PaintCare's Oregon Paint
Stewardship Pilot Program Plan (PaintCare, 2010).
   The evaluation addressed twelve questions and
the information collected and presented is extensive
and accessible in multiple formats to increase the
evaluation's utility for a diversity of audiences. As
a companion to this report, the PPSI Evaluation
Committee developed a website to organize and
present the results from this evaluation - www.
paintstewardshipprogram.com. The web site presents
Figure ES-1: The Oregon Paint Recycling Program
                                          Paint Management
                                              System
       Evaluation of the Oregon Paint Stewardship Program

-------
the key findings and learning from this evaluation in a
dynamic, visual, and interactive manner that includes
links to information and data sources used in this
report. This executive summary presents some basic
findings and learning from the evaluation for each of
the 12 evaluation questions (see text box).
    The evaluation was conducted between October
2009 and September 2011 by the PPSI Evaluation
Team, a subgroup of the Evaluation Committee, which
consisted of the U.S. Environmental Protection Agency
(EPA) and its subcontractor Eastern Research Group,
Inc. (ERG)1 in partnership with academic researchers
from Duke University, Georgia State University, and
Tufts University. While integrating measurement and
evaluation (M&E) into the design and implementation
of the Oregon pilot program, the Evaluation Committee
took a participatory approach to its work, emphasizing
transparency and communications while incorporating
aspects of developmental and participatory evaluation
into the evaluation design to account for program
complexity.
    The Evaluation Team relied on three primary
sources of data and information:

    •   Two surveys of consumers, one conducted in
        August 2010 just after the program began and
        another conducted in July 2011, provide key
        information for Evaluation Questions 3 and 4
        and data for several other questions.

    •   In-person interviews and other personal
        communications with program stakeholders:
        the Evaluation Team conducted 21 interviews
        during the course of the evaluation; including
        seven interviews with HHW representatives,
        three interviews with retailers, and 11
        interviews with key program stakeholders.

    •   Documents developed by PaintCare: the
        Evaluation Team drew heavily on the 2010
        Oregon Paint Stewardship Pilot Program Plan
        (PaintCare, 2010) and the 2011 Annual Report
        (PaintCare, 2011b) to provide information on
        program design, implementation, and results.
1 ERG's work was performed under subcontract to Industrial Economics,
 Inc. (IEC) under contract EP-W-07-028 between EPA and lEc.
1.   Collaboration: To what degree was the pilot program,
    from planning to implementation, a collaborative
    process?
2.   Paint Stewardship Organization: Describe the PSO,
    including its funding mechanism and infrastructure.
3.   Education and Outreach: How did education
    materials and strategies affect consumer awareness
    and behavior? Which messages were most effective
    with which target audiences? What materials/strategies
    were developed and what were the goals and target
    audience of those materials/strategies? Did other factors
    besides the program influence consumer behavior and
    awareness? What are the lessons learned?
4.   Consumer Purchasing Decisions: How has the
    program affected consumers' purchasing decisions and
    management of paint prior to drop-off at paint recycling
    facilities? How did the fee assessment affect consumer
    behavior?
5.   Collection of Post-Consumer Paint: How has the
    program affected the collection of post-consumer paint
    in terms of volume, cost, environment, convenience,
    and infrastructure? What other factors have affected the
    amount of leftover paint?
6.   Paint Reprocessing, Recycling, and Energy
    Recovery: How has the program affected used paint
    reprocessing, paint recycling, and paint-related energy
    recovery in  terms of volume, infrastructure, and cost?
7.   Household Hazardous Waste Programs: What was
    the impact of the program on the HHW facilities in terms
    of the types of paint collected, costs, and the way in
    which the facilities operate?
8.   Cost Effectiveness: How cost effective is the program?
9.   Waste Hierarchy:  How was the program designed
    and implemented to move consumers up the waste
    hierarchy? With respect to moving customers up the
    waste hierarchy, what were  the program's obstacles,
    opportunities, and decisions?
10.  Market for Post-Consumer Paint: How has the
    market for post-consumer paint been affected by the
    program? What aspects of the program have had
    an impact on the market and how?  What market and
    products represent potential opportunities for post-
    consumer paint products?
11.  Transferability: Based on the Oregon experience,
    what implementation and outcome-related information
    is required for other states to develop and implement
    leftover paint management systems? To what extent are
    the performance measurement and evaluation systems
    transferable to other states? What are the best ways to
    communicate the results of the evaluation?
12.  Unexpected Results: During the program and for
    each of its primary components, what were the primary
    external, unexpected and/or unintended influences and
    consequences?
                                                    Promoting Environmental Results Through Evaluation

-------
    We draw on information derived from these three
sources, as well as a number of other sources tailored
to specific evaluation questions.2 Data sources, data
collection instruments, the evaluation methodology
and other relevant documentation can be accessed
atwww.paintstewardshipprogram.com.

      COLLABORATION
    A primary goal of the PPSI was that the pilot
program, from planning to implementation, was
to be a collaborative process. Most stakeholders
(84 percent) involved in the PPSI agreed that the
program development process was collaborative
(Braunz et al., 2010). Stakeholders also pointed to
break downs in collaboration when the program
transitioned from designing and planning the PPSI
pilot program to drafting Oregon legislation and
planning and implementing the Oregon program.
The level of importance and function of collaboration
changed through the different stages of the program's
development. To account for changes, process
facilitators can better maintain appropriate levels
of collaboration throughout the process by, at the
beginning, coming to agreement on and documenting
clear expectations for collaboration's role in each stage
of the process.
 *
PAINT STEWARDSHIP ORGANIZATION
    A goal of the PPSI was for a pilot program to create
a Paint Stewardship Organization (PSO) that would
operate under the direction of the paint industry
and this was achieved in the Oregon legislation that
created the program. The PSO for the Oregon program,
PaintCare, is a coalition of paint producers and is
operated by the American Coatings Association (ACA),
although membership in ACA is not required for a
paint producer to be part of PaintCare. PaintCare is
responsible for implementing and running the program
in Oregon, a responsibility which they contracted out
to Product Care Inc., which has run other stewardship
programs. The PSO built the infrastructure (e.g., paint
collection sites, logistics, and transportation) using
the existing infrastructure in the Portland metro area.
Having this existing infrastructure offered significant
advantages in implementing this program.
    Though generally perceived as clearly denned
and complete, the lack of detailed, accessible cost
information reduced the transparency of the funding
mechanism. Few Oregon residents (11 percent) that
purchased paint in 2011 were aware that a fee was
added to the cost of paint to pay for the program
(PaintCare, 2 01 la).

jg±. EDUCATION AND OUTREACH
    The Oregon program included an extensive
education and outreach program which was another
PPSI goal. The campaign consisted of a website, point
of sale materials, radio and television advertising,
press releases, direct mailings to contractors and
retailers, an 800 number, and participation in trade
shows. The education and outreach campaign used
a broad approach to spreading its messages and did
not strategically aim its messages at specific target
audiences (consumers with the most paint, contractors,
retailers, etc.) with specific means of communication
(TV, website, social media, radio, etc.).  Consumers
of paint can be segmented into various groups
(new versus long-time homeowners, homeowners
versus contractors, age and other demographics).
Opportunities to improve the effectiveness of education
include identifying the desired changes in behavior
that will accomplish program goals, designing specific
messages to achieve those changes, (e.g., aligning
the messages with the program's priorities on the
waste hierarchy), prioritizing efforts allocated to
specific messages, and more explicitly targeting those
messages to the diversity of intended audiences (e.g.,
age groups, location, consumer vs. contractor). Overall,
there was little evidence that education and outreach
influenced consumer behavior. More useful evidence
of the effectiveness of education initiatives requires
more complete and transparent information such as
data related to consumer exposure to messaging and
consumer intentions and practices related to paint
management (e.g. purchasing, reuse, recycling, disposal).
2 For example, Evaluation Question 1 draws on a survey conducted by Duke University graduate students and Evaluation Question 2 draws on a set of
 interviews conducted by Georgia State University graduate students.
        Evaluation of the Oregon Paint Stewardship Program

-------
          Collections by type of paint
               Latex
              352,136
               75%
residents (92 percent) felt this type of program was
"somewhat important" or "very important." Oregon
residents' acceptance of the fee may be linked to the
inherent value they place on the goals and services
provided by recycling programs. The degree to
which consumers value such programs can be used
as an indicator of their willingness to pay a certain
fee. This information may be used in refining fee
structures within and across states. Documenting and
communicating consumer attitudes may also be useful
in navigating the legislative process necessary to create
programs similar to Oregon's.
            469,665 total gallons collected
 ^  CONSUMER PURCHASING DECISIONS
    A goal of the program was to encourage consumers
to purchase the correct amount of paint thereby
reducing the amount of leftover paint that must be
managed. Consumers indicated that the program
had little impact on their purchasing decisions. Most
Oregon residents (93 percent) that purchased paint
under the program indicated that the program had no
effect on the amount of paint they purchased. PaintCare
developed a "paint calculator" to help consumers
determine the amount of paint that was needed for
painting projects, but few consumers were aware
of the tool and very few actually used it. Overall, the
program focused less on the reduction of leftover paint
and more on recycling leftover paint. To focus planning
and implementation on reducing the generation of
post-consumer architectural paint, paint management
programs should more explicitly document the relative
emphasis they place on aligning the paint management
system with the existing waste hierarchy (reduce, reuse,
recycle). A clear articulation of program priorities will
provide a framework to organize and deploy outreach
strategies, messages, materials, and effort.
    In terms of the fee charged, 73 percent of Oregon
residents felt that the fee was reasonable. Most
      COLLECTION OF POST-CONSUMER
      PAINT
    The largest component of the program involved
collecting and processing leftover paint in Oregon.
In the first year of the program, PaintCare collected
469,665 gallons of paint. Of this total, 352,136 gallons
(75 percent) were latex paint and 117,529 gallons (25
percent) were oil-based paint (PaintCare, 2011b). As
of September 1,2011, the program had 98 collection
sites with 10 of these sites offering paint exchange. Sites
are open to the public, on average, six days a week for
a total of 58 hours per week (PaintCare, 2011b; Em2).
Most Oregon residents (92  percent) live within a 15
mile drive of a paint drop off site  (Strickland, 2011).

      PAINT REPROCESSING, RECYCLING,
      AND ENERGY RECOVERY
    In its first year, potential end-points for paint
collected by the program included: recycling into
another paint product, recycling as a non-paint product,
energy recovery (oil-based paint), appropriate disposal
and direct reuse by consumers. Table ES-1 summarizes
the volume of paint diverted to each end-point.

 ^  HOUSEHOLD HAZARDOUS WASTE
      (HHW) PROGRAMS
    HHWs are a key stakeholder in the paint
management system. Prior to the program, HHWs
would take in paint from consumers and process it
for disposal. Under the program, HHWs collected
and stored paint for eventual pick up by PaintCare's
transportation contractor. At the start of the program,
                                              Promoting Environmental Results Through Evaluation

-------
Table ES-1: Summary of Paint Endpoints
 Endpoint
Latex (Percent of Total)
 Recycling to paint and paint reuse
211,281 (60%)
                                                                           Oil-Based (Percent of Total)
                                                                           3,526 (3%)
 Reprocessing into non-paint product
28,171 (8%)
 Energy recovery
14,085(4%)
                                                                           114,003(97%)
 Disposal
98,598 (28%)
 Totals
352,136
                                                                           117,529
the amount of paint that HHWs collected increased
and then settled back to pre-program levels. The
initial bump in collections is attributed to consumers
bringing in older paint stored in their homes. During
the program, the proportion of latex paint collected
by HHWs increased relative to oil-based paint. Some
HHWs reported cost-savings resulting from avoided
labor, disposal, and transportation costs associated with
handling oil-based and unusable latex paints.

      COST EFFECTIVENESS
    The program processed paint at $7.03 per gallon
(PaintCare, 2 Ollb), although that amount excludes
costs incurred by HHWs to take in and handle the paint
at their facilities. The $7.03 per gallon value was lower
than other estimates from other sources, although
the other estimates were not purely comparable. The
use of "processing cost per gallon" as a measure of
cost-effectiveness is limiting.  A more comprehensive
measure would translate the gallons into environmental
benefits and then also include other management
options (e.g., reuse) as well as the cost-effectiveness of
reducing leftover paint by increasing the proportion of
consumers that "buy the right amount."

      WASTE HIERARCHY
    The Evaluation Committee is still assessing how
the program was designed and implemented to move
consumers up the waste hierarchy. The Committee has
partnered with a graduate student at Tufts University
to investigate this evaluation  question and results will
be reported January 2012. The method for answering
                                                    this question will involve relating the components of
                                                    the program to the categories of the waste hierarchy
                                                    (reduce, reuse, and recycling) and then compiling
                                                    information on the obstacles, opportunities, decisions,
                                                    and relative emphasis related to each category of the
                                                    waste hierarchy.

                                                          MARKET FOR POST-CONSUMER PAINT
                                                        To improve the management of leftover paint, the
                                                    PPSI determined that the pilot program should explore
                                                    means to expand the market for post-consumer paint
                                                    products. If more consumers used post-consumer
                                                    paint, then they would generate less waste because less
                                                    new paint would be purchased (and ultimately need
                                                    to be disposed). About half of the total volume of paint
                                                    collected (217,157 gallons) under the program made
                                                    it into the post-consumer paint market. For latex paint
                                                    collected by the program, 60 percent was available to
                                                    the post-consumer paint market. Very little oil-based
                                                    paint (3 percent) was available to the post-consumer
                                                    paint market.
                                                     *
               TRANSFERABILITY
                                                        The Evaluation Committee identified several types
                                                    of information that other states will need to collect
                                                    in order to implement paint stewardship programs,
                                                    including: volume data on paint, information on current
                                                    infrastructure, a system map, cost information, and
                                                    information on consumer awareness of infrastructure
                                                    and consumer behavior and attitudes. The Committee
                                                    also identified the aspects of the performance
                                                    measurement system and evaluation that are
       Evaluation of the Oregon Paint Stewardship Program

-------
transferable to other states, which included: the pilot
program's evaluation questions (appropriately adapted
to other states), matrix of performance measures and
a web-based program model for communications
with diverse audiences (see Figure ES-1). Oregon and
other states can more systematically and consistently
learn, improve and communicate about paint
recycling programs and other product stewardship
initiatives by: 1) using (and adapting) the program
evaluation framework and methodology designed
and implemented by the PPSI Evaluation Committee,
and 2) collaborating to adopt common fundamental
frameworks for ongoing performance management
such as those developed by the Conservation Measures
Partnership's (CMP's) Open Standards for the Practice
of Conservation.3
      UNEXPECTED RESULTS
    The Evaluation Committee identified some
key unintended effects and unexpected scenarios
that significantly influenced program planning and
implementation. Highlights include: (1) collaboration
took more time, effort and resources than expected and
levels of collaboration fluctuated significantly at different
stages in the process; (2) paint legislation was vetoed
twice in Minnesota which delayed implementation of
a program by more than a year-and-a-half; eventually
leading to implementation in Oregon; (3) though the
program was originally intended to be voluntary,
legislation was necessary to implement the program;4
(4) the PPSI had limited representation from retailers
leading up to the program, but retailers became a
critical component of Oregon paint collection; (5) the
misalignment of the program's goals and the waste
hierarchy model (reduce, reuse, recycle); and (6) that
retailers serving as collection locations had a negative
impact on the program's ability to divert high quality
leftover paint for reuse.
3 www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_Version_2.0.pdf.
4 Legislation was needed to exempt the paint producers from anti-trust and collusion concerns. At the start, the idea of the program was to avoid having
 states pass legislation or rules to tell the private sector what to do - the program was supposed to be voluntary.
                                                  Promoting Environmental Results Through Evaluation

-------
i.o    Introduction
    Over the last several decades, paint manufacturers
have dramatically reduced the environmental impacts
of leftover paint, including the reduction of toxic
ingredients found in leftover paint. Paint manufacturers
have also improved the performance of latex paints
to the point where they now constitute 80 percent of
paint sales, replacing a large portion of the market
for oil-based paints, which have more significant
environmental impacts (PaintCare, 2010). Additionally,
the paint industry works with retailers to educate
consumers on how to more accurately estimate the
amount of paint consumers need, thereby reducing the
quantities of leftover paint.
    Despite these improvements, environmental
impacts from leftover paint remain significant.
Households generate some 75 million gallons of leftover
paint each year, which amounts to about 10 percent of
the amount of paint they purchase (Abt, 2007). When
disposed in landfills in liquid form, leftover latex paints
can contaminate groundwater, thereby harming fish and
other aquatic life. Furthermore, leftover oil-based paint
is considered hazardous waste due to its combustibility
and high solvent content. Managing leftover  paint is
also costly to local governments; paint is the largest
volume of waste collected by household hazardous
waste (HHW) programs.5 The estimated cost to manage
leftover paint is more than $8 per gallon (SCS and
Cascadia, 2007).
   While leftover paint, as currently managed, is
associated with negative environmental impacts, it
also represents an  opportunity for reducing the life
cycle environmental impacts of paint production.
When manufacturers use leftover paint in their paint
production processes, the environmental impacts
associated with making paint—including materials
extraction, processing, and end-of-life management—
are reduced.  By recycling paint containers,
manufacturers can further reduce the life-cycle impacts
of paint products (PSI, 2004a).
    The movement towards product stewardship for
post-consumer paint began in 2002, when, due to the
high cost of managing leftover paint, a number of states
began to consider mandating paint management. In
2002, a coalition of paint manufacturers, represented
by American Coatings Association (ACA);6 local, state,
and federal environmental agencies; retailers; and
consumer and environmental agencies formed the
Paint Product Stewardship Initiative (PPSI) and began
negotiations facilitated by the Product Stewardship
Institute (PSI) to create an industry-managed post-
consumer paint management system. After seven years
of negotiations, two Memoranda of Understanding
(MOUs), and comprehensive PPSI-sponsored research
of the paint industry and consumers, Oregon became
5 Oregon Department of Environmental Quality, Paint Product Stewardship: www.deq.state.or.us/lq/sw/prodstewardship/paint.htm.
6 Formerly National Paint and Coatings Association (NPCA).
       Evaluation of the Oregon Paint Stewardship Program

-------
the first state to enact a law establishing product
stewardship as the preferred method for reducing
the environmental impacts and costs associated with
leftover paint.
   An early focus of the PPSI was in identifying a state
where a pilot project would be implemented. PPSI
identified Minnesota as a candidate for a pilot and it
became the focus of PPSI efforts. The PPSI developed
a work plan for implementing a pilot project in
Minnesota, focusing the work plan around the following
six goals (PPSI, 2008):7

   •   Goal 1: The pilot project, from planning
       to implementation, is a collaborative and
       cooperative process.

   •   Goal 2: Establish a paint stewardship
       organization (PSO), which operates under the
       direction of the paint industry.

   •   Goal 3: Consumers (including painting
       contractors) generate no or less waste paint
       and containers.

   •   Goal 4: The statewide post-consumer paint
       management system should be designed to
       ensure that it is environmentally beneficial,
       economical, and convenient. With these
       considerations, the system should strive to
       use methods highest on the following waste
       management hierarchy: reuse, recycling (into
       paint or other products), energy recovery
       (generally applicable to oil-based paint), and
       proper disposal.

   •   Goal 5: Identify cost-effective alternatives
       for using post-consumer paint products and
       explore means to expand the market for
       products containing post-consumer paint.

   •   Goal 6: Measure and evaluate the performance
       of the pilot project, and ensure the results and
       learning that the evaluation generates are
       transferable and relevant to the rollout of a
       national post-consumer paint management system.

   PPSI formed an Evaluation Committee (EC) to
be the lead on Goal 6 and to evaluate the extent to
which all six goals were achieved. The Minnesota pilot
project, however, was never implemented, as legislation
required to implement the program was vetoed by
the Minnesota governor at that time. Nevertheless,
much of the work performed in designing a program
in Minnesota was carried into the Oregon program,
including the six goals described above. These six goals
formed the basis of the evaluation questions developed
under this project and for the design of an evaluation of
the Oregon program.


1.1  Paint Product

Stewardship Initiative

Oregon Pilot Program

   As part of the broader dialogue, PPSI began
designing a pilot program to demonstrate the
effectiveness of an industry-led paint stewardship
program and sought states considering paint
stewardship program legislation to implement the
program.  In July 2009, Oregon became the first state
to enact a paint product stewardship law which
directs manufacturers of paint sold in the state to
set-up and run a statewide system for the collection of
post-consumer latex and oil-based paint. ACA formed
PaintCare, Inc. (PaintCare), as the PSO responsible for
implementing the pilot program in Oregon. The Oregon
law required PaintCare to submit an implementation
plan to the Oregon Department of Environmental
Quality (DEQ) to be approved before the start of the
pilot program. The Oregon Paint Stewardship Pilot
Program Plan (PaintCare, 2010) was released for public
comment in June 2010.  DEQ approved the plan at the end
of June, allowing the program to launch on July 1, 2010.
   The Oregon program is comprised of a diversity
of interconnected systems, actors, and processes. The
major components of the Oregon program include the
PSO (PaintCare), the oversight of Oregon DEQ, the paint
market, and the leftover paint management system.
Figure 1 illustrates the pilot program and its primary
components and is discussed in more detail in Section
2.2. Full documentation of the details of the program
appear in PaintCare's Oregon Paint Stewardship Pilot
Program Plan (PaintCare, 2010).
   Components of the Oregon pilot program include:

   •   PPSI: Conceptualization of the pilot
       program began with PPSI. Its goals included
       collaboration amongst diverse participants to
7 Available at www.deq.state.or.us/lq/pubs/docs/sw/PaintProdStewardshipPilotPlan2010June.pdf.
                                               Promoting Environmental Results Through Evaluation

-------
        plan, implement, and evaluate the performance
        of a statewide pilot to inform rollout of similar
        programs in other states nationwide.

    •    Program Administration: PaintCare and
        the Oregon DEQ are responsible for program
        administration. PaintCare is the industry non-
        profit association formed as the PSO by ACA.
        Oregon DEQ is the state agency charged with
        approving the PSO program plan, including
        the recovery fee, and responsible for program
        oversight and enforcement. PaintCare receives
        its funding from the paint market and uses
        funding for administration, supporting DEQ,
        running the paint management system,
        and providing education and outreach to
        consumers and Oregon residents.

    •    Paint Market: The paint market consists of
        three main participants:
                                                          •
               Producers of paint sold in Oregon are
               obligated under the law to participate
               in a paint stewardship program.
               Program participants are required
               to register and provide a list of paint
               brands manufactured. Remittance
               of recovery fee payments on all
               architectural paint sold in Oregon is a
               responsibility of the producer.

Figure 1 - Graphic model of the Oregon Paint Stewardship Pilot Program
        Retailers of paint sold in Oregon are
        required to collect the recovery fee
        from consumers at the time of sale.
        Retailers pay the producer the fee
        as part of the invoice for the paint
        offered for sale (i.e., retailers pay the
        producers). Additionally retailers can
        volunteer to be collection sites for
        leftover paint. They also collect the fee
        from consumers and then pass that fee
        back to producers.

        Consumers of paint eligible (i.e.,
        leftover architectural paint in 5
        gallon containers or less) for the
        paint management system can be
        residential purchasers, trade painters,
        institutions, and businesses.

Paint Management System: PaintCare  funds
the system and has coordinated with retailers,
HHWs, and other groups to collect paint and
set up vendors for the collection, exchange,
transportation, consolidation, and reprocessing
of leftover paint.
        Evaluation of the Oregon Paint Stewardship Program

-------
1.2   Evaluation of the

Pilot Program

    A key element of the PPSI MOU was the inclusion
of an evaluation of the pilot program to inform the
development of programs in other states and improve
the Oregon program. The work plan developed for the
Minnesota demonstration project included six goals
for pilot program implementation (see text box on
page 5). The Oregon law required the PSO to report
certain performance measures to DEQ (e.g., volume
of paint collected; program costs) by September 1,
2011. In turn, Oregon DEQ was required to report to
the Oregon Legislature results from the program and
recommendations for improvement by October 1,2011.
    The PPSI formed an Oregon pilot program
Evaluation Committee composed of industry
representatives, Oregon stakeholders, the U.S.
Environmental Protection Agency (EPA), and rollout
states (i.e., states  working with PPSI to also  enact
paint product stewardship legislation) in September
2009. As the evaluation results provide valuable
information for implementing post-consumer
management systems for various consumer products,
EPA took the lead and provided contractor support.
EPA facilitated the Evaluation Committee's work by
managing an "Evaluation Team" that included EPA and
its contractors, PSI, and academic partners.
    The role of the Evaluation Committee was similar
to that of a technical working group; the members
of the Evaluation  Committee contributed technical
expertise and guidance by participating on conference
calls and through review of the work performed by
the Evaluation Team. In late 2009 and into 2010, the
Evaluation Committee focused on providing input on
the Evaluation Team's development of the evaluation
methodology.8 During the fall of 2010 through the
summer of 2011,  the Evaluation Team implemented
the evaluation methodology. This report documents the
results  of the evaluation and should be considered a
companion document to PaintCare's 2011 Oregon Pilot
Program Annual Report (PaintCare, 2011b), which was
published to meet its reporting obligations.9 This report
is intended to achieve PPSI Goal 6 and provide input to
Oregon DEQ, as well as present independent evaluation
findings and learning for interested parties.
    This report focuses on providing the results of
12 evaluation questions. Chapter 2 provides a brief
overview of the evaluation method, while subsequent
sections focus on individual evaluation questions. In
addition to this report, the Evaluation Committee also
created a web-based model of the Oregon program
to present the Evaluation Committee's work, which is
found at www.paintstewardshipprogram.com.
8 Available at http://paintstewardshipprogram.com/images/pe_evaluation_methodology_10_14_10.DOC.
9 Available at http://paintstewardshipprogram.com/images/2011-09-01 _Oregon PaintStewardship Pilot Program Annual Report.pdf.
                                                Promoting Environmental  Results Through Evaluation

-------
 Pilot Program Goals (PPSI, 2008)
 Goal 1: The pilot project, from planning to implementation, is a collaborative and cooperative process.

 Goal 2: Establish a paint stewardship organization, which operates under the direction of the paint industry.

 Goal 3: Consumers (including painting contractors) generate no or less waste paint and containers.

 Goal 4: The statewide post-consumer paint management system should be designed to ensure that it is
        environmentally beneficial, economical, and convenient. With these considerations, the system
        should strive to use methods highest on the following waste management hierarchy: reuse, recycling
        (into paint or other products), energy recovery (generally applicable to oil-based paint), and proper
	disposal.	
 Goal 5: Identify cost-effective alternatives for using post-consumer paint products and explore means to
        expand the market for products containing post-consumer paint.
 Goal 6: Measure and evaluate the performance of the pilot project, and ensure the results and learning that
        the evaluation  generates are transferable and relevant to the rollout of a national post-consumer
        paint management system.
 Evaluation Committee Members
 Abby Boudouris
 Oregon Department of Environmental Quality
Tom Metzner
Connecticut Department of Environmental Protection
 Scott Cassel
 Product Stewardship Institute
Lou Nadeau
Eastern Research Group, Inc.
 Cynthia Dunn
 California Department of Resources Recycling and
 Recovery
Jim Quinn
Metro Parks and Environmental Services Department
 Barry Elman
 Environmental Protection Agency
Steve Sides
American Coatings Association
 Jen Holliday
 Chittenden County, Vermont Solid Waste District
Amy Stillings
Eastern Research Group, Inc.
 Lauren Jankovic
 Eastern Research Group, Inc.
Theresa Stiner
Iowa Department of Natural Resources
 Alison Keane
 American Coatings Association
Emily Wang
California Department of Resources Recycling
and Recovery
 Matt Keene
 Environmental Protection Agency
Robert Wendoll
Dunn-Edwards Paint
 Scott Klag
 Metro Solid Waste and Recycling
Jennifer Nash
Harvard University
       Evaluation of the Oregon Paint Stewardship Program

-------
2.o   Overview of  the Evaluation
Method  and  Key Data Sources
   To account for the new and rapidly evolving
Oregon pilot, this evaluation incorporated aspects
of developmental evaluation. This emerging field of
evaluation involves designing and implementing an
evaluation in which the program changes over time to
accommodate a complex environment. The evaluation
is "developmental" in the sense that it matures over
time to meet the changing needs of the program
(Patton, 2010). Like the PPSI, the Evaluation Committee
took a participatory approach to its work, emphasizing
transparency and communications and integrating
measurement and evaluation (M&E) into the design and
implementation of the Oregon pilot.


2.1   Evaluation Approach

   Program evaluators have long advocated the
consideration of measurement and evaluation in the
design of programs. By considering the questions
that various audiences will ask in the future about
the program, participants are better aware of the
baseline data that will be necessary to assess program
performance later. In theory, this leads to more efficient
organizational learning and better management of
program performance as programs evolve and mature.
   The Oregon pilot program integrated measurement
and evaluation into program design through a series of
iterative and overlapping steps (see Figure 2):10

   •   Described the program. First the Evaluation
       Team engaged the Evaluation Committee
       to clarify the program's mission, goals, and
       objectives. The Evaluation Team used that
       information to develop a preliminary logic
       model, an illustration of how the program
       is envisioned to work, that links program
activities and outputs to key audiences and
outcomes. For many evaluations, the logic
model is sufficient to communicate the theory
of how the program is expected to work; in this
case, the traditional logic model did not fully
capture the interaction between the program,
evaluation, and stakeholders. The Evaluation
Team, with the help of a graphic designer,
developed a more flexible, web-based version of
the logic model (see Section 2.2 for more details).

Developed evaluation questions. Working
with the Evaluation Committee, the Evaluation
Team drafted questions based on the pilot
program's goals and objectives. The questions
were designed to account for program context,
key audiences, communication of results,
and use of the evaluation findings. Part of the
process included the Evaluation Committee's
prioritization of evaluation questions.

Developed measures. The Evaluation
Team identified a set of measures for each
evaluation question. For each measure, the
team documented potential data sources,
developed collection methods and strategies,
identified the tools of analysis, and suggested
the strategies for data collection and data
management (see Section 2.3). This approach
resulted in a set of measures that are highly
specific to one or two of the evaluation
questions in design, but may produce
information that is relevant to several
questions. For example, the measures of
collaboration between program participants
during implementation (Evaluation Question 1)
provides information on the transferability
10 For a more detailed discussion of this topic, see Integrating Evaluation into Program Design (Keene, 2008). Available at: www.productstewardship.us/
  displaycommon.cfm?an=1&subarticlenbr=754.
                                            Promoting Environmental Results Through Evaluation

-------
Figure 2 - Iterative Steps of Integrating Evaluation into Program Design
                1. Evaluation Policy
                2. Evaluation
                  Methodology
                                                     1. Team
                                                     2. Mission
                                                     3. Goals &
                                                       Objectives
                                                     4. Logic Model
                                                                       Questions
                                                                     1. Context
                                                                     2. Audience
                                                                     3. Communication
                                                                     4. Use
1. Data Sources
2. Collection Methods
  & Strategy
3. Analysis Tools
4. Data Collection
5. Data Management
        of the program to other states (Evaluation
        Question 11) and to the unexpected results of
        collaborative work (Evaluation Question 12).

    •    Documented the process and results.
        The methodology documents the decisions
        with respect to evaluation questions,
        measures, and communication strategies.
        Note that the process is iterative; the final
        stage of documentation feeds into program
        implementation and the program management
        cycle. This report documents the Evaluation
        Committee's methods, data sources, findings,
        and key learning points with respect to each
        evaluation question.

    Integrating evaluation into the design of the
Oregon pilot program also makes the evaluation more
amenable to a participatory approach. A participatory
                                      evaluation can be defined as "a partnership between
                                      trained evaluation personnel and practice-based
                                      decision makers, organization members with program
                                      responsibility or people with a vital interest in the
                                      program" (Cousins and Earl, 1992). By working with
                                      program designers, personnel, and stakeholders to
                                      incorporate evaluation into the program design, the
                                      program includes these individuals in the evaluation
                                      process from the start.
                                          This  approach  ensures that the evaluation results
                                      are "accessible and important to users and responsive
                                      to their needs while maintaining sufficient technical
                                      quality" (Cousins and Earl, 1992). The evaluation
                                      stakeholders are involved in the creation of evaluation
                                      questions, data sources, and analysis, which in turn
                                      ensures that the findings of the evaluation are open
                                      to stakeholders and the public. In other words,
                                      stakeholders and the public have access to data that are
                                      collected and analyzed by the Evaluation Committee, as
        Evaluation of the Oregon Paint Stewardship Program

-------
well as the learning derived from the analysis, and are
free to make their own conclusions about the findings
of the analysis.
   The goal of a participatory approach, when
compared to traditional evaluation methods, is to
encourage participant ownership of the work and the
results of the evaluation. This ownership of the results
by the program participants fuels organizational
learning, decision making, and improvement (Cousins
and Whitmore, 1998). A participatory approach
allowed the Evaluation Team to distribute the costs of
the evaluation among multiple organizations and tap
into specialized skills and other resources to improve
the quality of the evaluation findings.
   For example, the Evaluation Committee invited
several academic researchers to become "partners" in
the evaluation:

   •   Amy Braunz, Julie Colvin, Whitney Knapp,
       and Hedrick Strickland, graduate students
       at Duke University's Nicholas School of
       the Environment completing their masters
       degrees in environmental management,
       implemented a survey to address Evaluation
       Question 1 (collaboration). These students
       were participating in Dr. Randy Kramer's
       course in survey methods, in which students
       are required to develop and implement
       a survey. The students coordinated with
       Indiana University faculty member Anne
       Marie Thompson, a leading expert in the
       measurement of collaboration.

   •   Wes Bledsoe, Eric Graves, and Andrei Roman,
       graduate students at Georgia State University
       (GSU) participating in Dr. Cynthia Searcy's
       applied capstone course in evaluation
       performed work under Evaluation Question 2.

   •   Hedrick Strickland, one of the Duke University
       graduate students mentioned above, based
       her master's thesis on the application of
       geographic information systems (CIS) analysis
       to the Oregon project, including descriptive
       information, visualization, and analysis
       relevant to the evaluation.  Ms. Strickland
       addressed questions of the convenience and
       efficiency of the infrastructure of the program
       for consumers and transportation.
  Oregon Pilot Program Stakeholders
        American Coatings Association*
     U.S. Environmental Protection Agency*
      Household Hazardous Waste Facility
                   Operators*
      Oregon Department of Environmental
                    Quality*
               Oregon Legislature
                Paint Producers*
                 Paint Retailers
       Product Stewardship Community*
                Roll-out States*
        * Represented on the Evaluation Committee
    •   Brielle Kissell, a graduate student at Tufts
       University, based her masters thesis on the
       exploration of Evaluation Question 9, which
       relates to the waste hierarchy.
    There are limitations to using a participatory
approach to evaluation. Inviting multiple participants
and stakeholders into an evaluation increases logistical
challenges. In a traditional approach, an external,
impartial evaluator controls the timeline and method of
the evaluation as well as the presentation of the results.
In a participatory approach, the evaluator coordinates
with a diversity of stakeholders and other participants
in the evaluation process, as well as other parties with
a direct interest in the evaluation results. Coordination
of multiple parties and reconciliation of competing
viewpoints and priorities can be time-consuming.


2.2  Communicating

Information

    The complexity of the program and the
participatory approach to the evaluation demanded
an innovative method to communicate about the
evaluation to multiple, diverse audiences throughout
program design and implementation. Logic models can
be an important tool for describing a program's theory
                                               Promoting Environmental Results Through Evaluation

-------
and assumptions, designing evaluation questions and
methods, and convening and communicating with
stakeholders about program and policy effectiveness.
The development and use of logic models, however,
is often limited to the initial stages of the evaluation
process and to clearly defined programs. Generally, logic
models can be described as linear, bounded, relatively
static, and rarely designed for communications
with the general public. Trends in performance
management in the public sector are pointing toward
greater transparency and accountability, systems
thinking, participation and empowerment of diverse
stakeholders, adaptive management, evidence-based
practice, and better collaboration and communications
with a wider range of audiences.
    To complement the participatory nature of the PPSI
and the evaluation, the Evaluation Team developed a
web-based "fuzzy" logic model (see Figure 1) with the
intent of expanding the accessibility and use of the
evaluation (and program) and the evaluation process
to a greater diversity of stakeholders over a longer
period of time. A fuzzy logic model embraces fluid
and approximate reasoning and varied context and
assumptions with  the aim of improving the capacity
of models of program theory to navigate non-linearity,
feedback loops, adaptive agents and other agents and
other  traits common in the lifecycle of environmental
programs and policies. Integrating Web 2.0, graphic
design, systems theory, and data visualization with
traditional logic models gives the evaluator the capacity
to embed an unlimited type and quantity of content into
a Web-based model of the program.
    The model of the PPSI's evaluation of the Oregon
Paint Stewardship Pilot Program, found at www.
paintstewardshipprogram.com (referred to as the
"evaluation website" in this report), is designed to be
used in tandem with this report. It describes both the
key elements of the pilot program (e.g., infrastructure,
funding) as well as the evaluation. For example,
evaluation questions are illustrated as numbered
paint "splatters" on the part of program that they are
evaluating. Clicking on an evaluation question brings
up a pop-up box that provides additional information
about that evaluation question, such as the question
and purpose, measures, data sources and methods
used to answer the question, key findings, and Web
links to supporting materials and resources (e.g.,
reports and data).


2.3   Evaluation  Design

    The Evaluation Team's evaluation methodology
provides a detailed discussion of the process used to
develop, prioritize, and refine the set of evaluation
questions, as well as a description of the development
of the measures and data sources for each question.
Table 1 lists the evaluation design matrix developed
for the Oregon pilot program evaluation. The table
presents each evaluation question, the measures used
to answer the question, and the data sources and
analytical methods used to develop the measures.  Each
question is provided with its own section (beginning
with Section 3.0) in this report. Each question-specific
section begins with an overview of the data sources,
methods, and analysis used to answer the question and
then lists findings and learning points. In the context
of each question and given the findings, the learning
sections aim to answer the following three questions:

    •   How do the findings affect the program?
        What do the findings tell us?

    •   How do the findings affect the program's
        capacity to achieve its goals? What is the impact
        on the program being able to achieve its goals?

    •   What are the opportunities to improve
        program performance for both the Oregon
        program and other programs in the future?

    Each the evaluation questions covers a diverse
and broad set of concepts. Thus, when presenting the
results in the sections that follow, we use the above
considerations as a guide, but we also adapt our
presentation according to the information in hand.
        Evaluation of the Oregon Paint Stewardship Program

-------
Table 1 - Evaluation Design Matrix
Evaluation
1 . To what degree was
the pilot program,
from planning to
implementation,
a collaborative
process?
2. Describe the PSO,
including its funding
mechanism and
infrastructure.
Key
Audiences
Measures


• EPA
• PSI
• Roll-out states
• Manufacturers
and Retailers
• PPSI
• Oregon
Legislature
• Oregon DEO
• PPSI
• Roll-out states
• PSI
• Retailers
• Product
stewardship
community
• Governance scale
based on the questions
provided in Table 1 of
Thompson, Perry, and
Miller (2007)
• Administration scale
based on the questions
provided in Table 1 of
Thompson, Perry, and
Miller (2007)
• Autonomy scale based
on the questions
provided in Table 1 of
Thompson, Perry, and
Miller (2007)
• Mutuality scale based
on the questions
provided in Table 1 of
Thompson, Perry, and
Miller (2007)
• Norms scale based on
the questions provided
in Table 1 of Thompson,
Perry, and Miller (2007)
• Connectedness
measures
• Narrative of PSO
development and
operation
• Factors affecting
infrastructure choices
• GIS representation
of infrastructure in
relation to demographic
information
• Clarity
• Transparency
• Completeness
Data F
• Interview
or survey
of those
involved in
process
• Survey of
network
participants
(program
organizations)
• PSO
documents;
Interviews
with Oregon
DEO staff,
PaintCare
staff
• One-time
analysis
• Baseline,
follow-up at 1 2
to 1 8 months
• Within 6 months
of the program
start
                                                    Promoting Environmental  Results Through Evaluation     10

-------
Evaluation Key »*,«,«...««. Data c^«. .«„„..
Question Audiences Measures Sources Frequency
3. How did education
materials and
strategies affect
consumer
awareness and
behavior?
4. How has the
program affected
consumers'
purchasing
decisions and
management of
paint prior to drop-
off at paint recycling
facilities?
• Oregon DEQ
• Oregon
Legislature
• HHWand
other paint
collectors
• PPSI
• Roll-out states
• Product
stewardship
community
• Oregon DEQ
• Retailers
• Manufacturers
• PPSI
• Roll-out states
• Product
stewardship
community
• Program awareness
• Program scope
awareness
• Site location awareness
• Improper disposal/
handling environmental
impact awareness
• Consumer use of best
practices for purchasing
paint (percentage)
• Consumer-reported
leftover paint— amount
• Consumer use of
leftover paint
• Disposal practices
• Storage practices
• Recycling practices
• Awareness of pilot
program education and
outreach messages.
• Consumer use of best
practices for purchasing
paint
• Consumer-reported
leftover paint— amount
• Disposal practices
• Storage practices
• Recycling practices
• Fee awareness
• Behavioral reaction to fee
• Consumer
surveys,
focus
groups, or
interviews
• Consumer
surveys,
focus
groups, or
interviews
• Baseline,
follow-up at 12
to 1 8 months
• Baseline,
follow-up at 12
to 1 8 months
Evaluation of the Oregon Paint Stewardship Program

-------
Evaluation Key
Question Audiences
5. How has the
program affected
the collection of
post-consumer
paint in terms
of volume,
cost, quality,
environment,
convenience, and
infrastructure?
6. How has the
program affected
used paint
reprocessing,
paint recycling,
and paint-related
energy recovery in
terms of volume,
infrastructure, and
cost?
• PPSI
• Oregon DEQ
• Oregon
Legislature
• Roll-out states
• Retailers
• Product
stewardship
community
• PPSI
• Oregon DEQ
• Oregon
Legislature
• Roll-out states
• Retailers
• Product
stewardship
community

Measures
• Gallons collected
• Containers collected
• Cost per gallon
• Reductions in amounts
of paint that could
impact the environment
• Average distance
traveled to drop-off
point
• Total number of hours
open
• Average distance to
drop-off locations
• Number of permanent
collection sites
• Number of periodic
collection opportunities
• Number of sites offering
paint exchange
• Changes in retailers'
practices
• Gallons per trip
• Miles per trip
• Transportation cost per
gallon transported
• Volume of latex paint
exchanged
• Volume of latex paint
recycled into paint
products
• Volume of latex paint
recycled into non-paint
products.
Sources Frequency
• Data
collected and
tracked by
the program
• Surveys or
anecdotal
evidence
at drop-off
locations to
ask about
distance
traveled
• Data
collected and
tracked by
the program
• Data
collected and
tracked by
the program
and Oregon
DEQ prior to
program
• Data
collected and
tracked by
the program
• Interviews
with retailers
• Data
collected and
tracked by
the program
• Data
collected and
tracked by
the program
• Baseline,
monthly
• Annually or
more frequently
if possible
• Based on
the volume
measures being
collected
• Baseline,
follow-up
• Annually or
more frequently
if possible
• Baseline.
Annually or
more frequently
if possible.
Promoting Environmental Results Through Evaluation     12

-------
Evaluation Key »*,«,«...««. Data c^«. .«„„..
Question Audiences Measures Sources Frequency
6. How has the
program affected
used paint
reprocessing,
paint recycling,
and paint-related
energy recovery in
terms of volume,
infrastructure, and
cost?
7. What was the
impact of the
program on the
HHW facilities in
terms of the types
of paint collected,
costs, and the
way in which the
facilities operate?
8. How cost effective
is the program?
9. How was the
program designed
and implemented to
move consumers up
the waste hierarchy?
• PPSI
• Oregon DEQ
• Oregon
Legislature
• Roll-out states
• Retailers
• Product
stewardship
community
• PPSI
• Roll-out states
• Oregon local
governments
• PPSI
• Oregon DEQ
• Roll-out states
• Product
stewardship
community
PPSI
Roll-out states
EPA
Retailers
Product
stewardship
community
• Volume of latex paint
appropriately disposed
in landfill
• Volume of oil paint
exchanged
• Volume of oil paint
recycled into paint
products
• Volume of oil paint
appropriately disposed
• Number of facilities
capable of performing
each type of
management method
• Capacity of facilities
performing each type of
management method
• Processing cost
per gallon for each
management method
• Gallons of different
types of paint collected
at HHW facilities
• Changes in the amounts
of or capacities for other
products by HHW
• Cost for HHW facilities
to take in and process
paint
• Cost of managing other
products
• Cost per gallon
collected at drop-off
facilities
• Cost per gallon
exchanged, recycled, or
used for energy
• Total program cost per
gallon collected
• Evaluation Committee
assessment of hierarchy
attainment
• Amounts of paint
processed in the program
that fall into each
category
• Data
collected and
tracked by
the program
• Data
collected and
tracked by
the program
• Interviews
with HHW
facilities
• Data
collected
and tracked
by the
program and
participants
• Program
materials and
interviews
with program
staff
• Data
collected and
tracked by the
program and
participants
• Baseline.
Annually or
more frequently
if possible.
• Annually or
more frequently
if possible
• Annually or
more frequently
if possible
• Collected after
the program has
run for some
time
• Collected after
the program has
run for some
time
13     Evaluation of the Oregon Paint Stewardship Program

-------
Evaluation Key
Question Audiences
1 0. How has the
market for post-
consumer paint
been affected by
the program?
1 1 . Based on
the Oregon
experience, what
implementation
and outcome-
related information
is required for
other states to
develop and
implement leftover
paint management
systems?
12. During the
program and for
each of its primary
components,
what were the
primary external,
unexpected and/
or unintended
influences and
consequences?
• PPSI
• Retailers
• Manufacturers
• PPSI
• Roll-out states
• EPA
• PSI
• Product
stewardship
community
• PSI
• Roll-out states
• Product
stewardship
community

Measures
• Different products
offered
• Number of facilities
(outlets) offering post-
consumer paint.
• Total sales (dollars) of
post-consumer paint
products.
• Total amount of paint
distributed for re-use
from HHW sites
• Description of
implementation and
outcome-related
information that are
required for states
• Transferability to other
states
• Best ways to
communicate results of
the evaluation
• Lists of external
influences and
outcomes
Sources Frequency
• Interviews
with Oregon
DEO,
PaintCare,
HHW
programs
• Interviews
with program
staff.
Qualitative
assessment
by Evaluation
Team
• Interviews,
all available
data
• Annual
• Within 12
months of
program
implementation
• Collected after
the program has
run for some
time
2.4   Key Data Sources Used
in the Evaluation

    The evaluation team relied on many data sources
provided by our evaluation partners, the PPSI, and PSI,
with the primary sources including:
    •   Two surveys of consumers
    •   In-person interviews and other personal
       communications with program stakeholders
    •   Documents developed by PaintCare (the PSO)
11 For example, Evaluation Question 1 draws on a survey conducted by Duke University graduate students and Evaluation Question 2 draws on a set of interviews
  conducted by Georgia State University graduate students. Both of those sources of information are discussed with their respective evaluation questions.
    Throughout our findings on the evaluation
questions, we draw on information derived from
these three sources. Where other data or information
is used in answering an evaluation question,11 we
describe that information in our discussion of the
evaluation question.
                                                Promoting Environmental Results Through  Evaluation      14

-------
2.4.1  Consumer Surveys

    Two surveys of consumers, one conducted in
2010 just after program rollout (Bradshaw, 2010) and
another conducted nearly one year later (PaintCare,
2011b), provide key information for Evaluation
Questions 3 and 4 and supplementary data for other
questions. Bradshaw Advertising conducted the first
survey, a telephone survey of 409 Oregon residents
from August 4 to 10, 2010, for PaintCare.12 We refer
to this survey as the "August 2010 survey" throughout
this report. The survey was conducted using random
sampling that resulted in a margin of error of 4.8
percent at the 95 percent confidence level. The survey
targeted having at least 80 percent of respondents in
the 25 to 64 age range with no more than 10 percent in
either the 18 to 24 or 65 or older age ranges. Overall,
the survey focused on consumer awareness of the
newly implemented program; specifically, the survey
asked consumers about:

    •   How and where they would likely dispose of
       unneeded paint.

    •   The extent to which they are aware of the
       program.

    •   How they learned about the program.

    •   The importance of this  type of program.

    •   The reasonableness of the fee.

    •   How likely they are to recycle paint.

    The second survey was a web-based survey of
1,064 Oregon residents over 18 years old, conducted
from July 7 to 12, 2011 by PaintCare through the ACA
paint consumers panel. We refer to this as the "July
2011 survey" throughout this report. The July 2011
survey was conducted using random sampling with
a margin of error of 3 percent. This survey targeted
individuals who had purchased paint between July 1,
2010 and June 30, 2011, as well as individuals who
paint as an occupation. This survey included questions
to consumers about:
    •   Whether or not they recall hearing or seeing
       advertisements for the program.
    •   Where they saw or heard the advertisements.

    •   Whether or not they are aware of the fee.

    •   If and where they have purchased paint in the
       past year.

    •   The impact of the fee on their paint purchase.
    •   The types of products covered by the program.

    •   Awareness of leftover paint drop-off locations
       and the distance to the location from their
       home.
    •   Perceived convenience of the drop-off locations.

    •   Whether or not they had recalled seeing and
       using a paint calculator to purchase paint.

    •   The influence of program information on the
       amount of paint purchased.

    •   How they managed their leftover paint from
       their most recent project.


2.4.2  Interviews and Other
Personal  Communications

The evaluation team conducted 21 interviews
during the course of the evaluation; including seven
interviews with HHW representatives, three interviews
with retailers, and 11 interviews with key program
stakeholders. Additionally, the team followed up
with many of the interviewees for clarification of key
points and data and received clarifying responses in a
number of email correspondences. Each interview had
a different focus:

    •   The HHW interviews focused on the impact
       of the program on the management of HHW
       programs, facilities, and collection events in
       terms of paint volume collected, costs to collect
       paint, and ability to collect other HHW products.
12 Full survey results are available at www.productstewardship.us/displaycommon.cfm?an=1&subarticlenbr=754.
15    Evaluation of the Oregon Paint Stewardship Program

-------
    •   The interviews with retailers explored the
       impact of the program on retailers with a focus
       on the process of implementing the program,
       the decision to act as a collection site or not,
       and key challenges faced.

    •   Interviews with program stakeholders did
       not follow a set pattern and were often
       customized to the interviewee based on
       the timing of the interview. For example,
       the evaluation team asked representatives
       of the PSO about challenges faced during
       program implementation, lessons learned, and
       availability of data.

    A full list of the interviews and personal
communications that were used in this report appear
in Section 15.0. When citing a specific interview in the
text we used an "S" followed by the interview number.
Similarly, we use an "Em" followed by a number to cite
an email correspondence.13


2.4.3  Paint Stewardship
Organization  Documents

    Two PSO documents provided integral information
about the program: the 2010 Oregon Paint Stewardship
Pilot Program Plan (PaintCare, 2010) and the 2011
Annual Report (PaintCare, 2011b). The Program Plan
provided the following information about the program:

    •   Oregon paint stewardship legislation.

    •   Stewardship organization and participation
       of producers.

    •   Program products.

    •   Registration of producers and brands.

    •   Product sold and available for collection.

    •   Program budget and funding.
    •   Outreach and education.

    •   Collection system.

    •   Environmental regulatory requirements.

    •   Transportation.

    •   Sound management methods.

    •   Program reporting and audit.

    The Annual Report provided the following types of
data:
       The methods used to collect, transport, recycle
       and process post-consumer architectural paint
       in the state.

       The volume and type of post-consumer
       architectural paint collected in all regions of
       the state.

       The volume of post-consumer architectural
       paint collected in the state by method of
       disposition, including reuse.

       An independent financial audit of the program.

       A description of program costs.

       An evaluation of the operation of the program's
       funding mechanism.

       Samples of education materials, evaluation of
       the methods used to disseminate materials,
       and assessment of the effectiveness of the
       materials.

       An analysis of the environmental costs and
       benefits of collecting and recycling latex paint.
13 See Section 15.0.
                                               Promoting Environmental Results Through Evaluation     16

-------
3.o   Collaboration
                EVALUATION QUESTION:
                To what degree was the pilot program, from planning
                to implementation, a collaborative process?
   How was the collaborative process viewed by different groups involved in
   the process?
   What tools and strategies (including communication) were used to foster
   collaboration and how effective were those tools?
   PPSI's first goal for the pilot program was for
the program to be collaborative and cooperative,
to potentially serve as a model of governance that
EPA, states, and product stewardship initiatives
can use to achieve environmental and human
health policy objectives. A key point to understand
about collaboration under this program is that the
collaborative effort started well before the Oregon
program was developed. PSI was involved in developing
a national dialogue about paint stewardship starting
in 2002. In fact, PSI intended the entire process to be
collaborative from start to finish (S19).


3.1   Data Sources, Methods

and Analysis

   Collaboration may be defined as "a process in
which autonomous or semi-autonomous actors
interact through formal and informal negotiation,
jointly creating rules and structures governing their
relationships and ways to act or decide on the issues
that brought them together; it is a process involving
shared norms and mutually beneficial interactions"
(Thompson et al., 2007). Thompson et al. (2007)
developed a set of five dimensions of collaboration,
which framed the Evaluation Committee's approach to
data collection, analysis and interpretation:

   •   Governance: The degree to which there is
       joint decision-making by the collaborative
       parties about rules to govern the relationship.

   •   Administration: The extent to which a
       structure that moves the collaboration from
       governance to action exists.

   •   Organizational autonomy: The extent to
       which each party to the collaboration retains
       autonomy in the relationship.

   •   Mutuality: The extent to which parties to the
       collaboration experience mutually beneficial
       interdependencies.

   •   Norms: The extent to which there is reciprocity
       and trust between the collaborating parties.

   Thompson etal. (2007) developed these scales
through a systematic review of the literature,
synthesis of theoretical definitions of collaboration,
a series of interviews with organizational directors
on collaboration, and detailed case study research.
They identified a set of 10 to 12 indicators for each
dimension and constructed a survey question to
measure each indicator; each indicator was phrased as
17     Evaluation of the Oregon Paint Stewardship Program

-------
a statement and the survey respondents were asked to
rate the extent to which their organizations engage in
each behavior. The rating was designed as a scale from
1 ("not at all") to 7 ("to a great extent").
    The Evaluation Team partnered with a team of
graduate students at Duke University's Nicholas School
of the Environment as part of their capstone course in
survey methods to help answer Evaluation Question 1.
Their primary mode of data collection was a Web-based
survey of PPSI participants. The survey results were
summarized in a report (Braunz et al., 2010).
    Braunz et al. (2010) used the Thomson et al. (2007)
framework to develop a survey instrument to ask about
collaboration under the PPSI. The web survey targeted
current and former PPSI participants, including those
involved before the start of the Oregon program. The
survey's 23 questions focused on:

    •   Demographics, including organization type,
       role, and length of time involved.

    •  Communication tools used and communication
       tools preferred.

    •   Level of participation.

    •  Opinions about the collaboration
        (i.e., the dimensions of collaboration).

    Before implementing the full-scale survey, the
Duke students conducted an informal focus group
and performed ten pre-tests to gather feedback and
clarification on the survey instrument.
    The questionnaire was distributed to 419 PPSI
participants in November 2010 and was completed by
125 participants (30 percent response rate) (Braunz
et al, 2010). In order to determine stakeholder
perceptions, survey respondents were grouped as
(see Figure 3): 1) local government (33 respondents);
2) state and federal government (23 respondents); 3)
private sector, which combines retailers, manufacturers
and other private enterprises (23 respondents); and
4) nongovernmental organization (NGO)/Non-profit/
University/Trade Associations (8 respondents).
    After the Duke University graduate students
completed the survey, the Evaluation Team conducted a
series of interviews with key stakeholders to assess the
  Figure 3 - Distribution of Respondents by Category to
  Duke University Survey of PPSI Stakeholders
        Local Government
        State Government
        Private Sector
NGOs / Non-Profits
/ University/Trade
Associations
Other
collaborative process. The Evaluation Team conducted
interviews with representatives of:

    •   PSI

    •   ACA and PaintCare

    •   Oregon DEQ

    •   HHWs14

    The purpose of the interviews was to provide
a history of the collaborative effort and to get the
opinions and feedback from key stakeholders that were
involved in the process from start to finish.
3.2   Findings
    In presenting the findings for this question, we
will first "tell the story" of collaboration under the
program, and then turn to providing an assessment of
the collaborative effort.
14 The interviews that were conducted with HHWs were primarily focused on providing information for Evaluation Question 7; however, the information
  provided by HHW representatives during those interviews was applicable to this (and other) questions.
                                                 Promoting Environmental  Results Through Evaluation     18

-------
3.2.1   What tools and strategies
(including communication) were
used to foster collaboration  & how
effective were those tools?


Tools and Strategies Used
    Prior to the PPSI, PSI, state, and local governments
identified leftover paint as one of their top five waste
management issues. As a result, a number of states
began to consider implementing mandatory leftover
paint management requirements. PSI interviewed
stakeholders during 2002 to 2004 to define the
problem and identify basic facts in order to start a
national dialogue. Its efforts resulted in the publication
of A Background Report for the National Dialogue on
Paint Product Stewardship (March 2004) to provide a
technical foundation for requirements (PSI, 2004a). At
the same time, PSI also released Product Stewardship
Action Plan for Leftover Paint (March 2004), which
outlined the key issues and potential solutions related
to leftover paint management (PSI, 2004b). Funding
for this initial work came from a variety of sources,
including most state and local governments involved in
the PPSI. "Seed money" from Massachusetts Office of
Energy and Environmental Affairs, an EPA grant for one
meeting, and a foundation grant to partially support
another meeting (S19).
    In December 2003, PSI facilitated the first of four
in-person dialogue meetings to establish relationships
and build trust among stakeholders. An outcome
of these meetings was to establish workgroups to
focus on specific issues (e.g., markets for recycled
paint, education efforts). Industry and government
stakeholders volunteered to lead some of these work
groups and provide funding to support the group's
efforts (S19).
    Throughout the process, PSI acted as a facilitating
organization, supported by contributions from
stakeholders after the initial grants were expended.
PSI's philosophy for facilitation was based on
collaboration and consensus and hence was interwoven
through all aspects of the Initiative (S19). PSI's strategy
to foster collaboration, implicitly adopted by PPSI early
in the process, involved:

    •   Establishing the PPSI goals through a
       consensus process (S19).

       Developing a detailed work plan (PPSI, 2008)
       during the first year to 1) determine major
       tasks and timelines, 2) identify potential pilot
       opportunities, 3) identify staffing and funding
       requirements, 4) identify challenges and
       barriers, and 5) draft agreement language.

    •   Crafting MOUs to define purposes and
       scopes of projects; goals and objectives; and
       commitments by stakeholders (PPSI, 2007).

    •   Establishing smaller groups including15
       1) a Steering Committee, 2) an Education
       Workgroup, 3) an Infrastructure Workgroup,
       4) Market Workgroup, 5) Life Cycle Benefits
       and Costs Committee, 6) Demonstration Project
       Committee, and 7) Evaluation Committee, to
       work on specific projects.

    •   Providing agenda and background materials
       prior to meetings and calls and providing
       summaries of meeting and calls, allowing PPSI
       participants the opportunity to comment (S19).

    In implementing this  approach, PSI relied primarily
on the following three tools (S19):

    •   Monthly PPSI briefing conference calls.

    •   PPSI conferences (once or twice a year).

    •   A website featuring status of projects, reports,
       and posting of agenda, participant lists,
       meeting/call presentations, and meeting/call
       minutes (S19).

    The PPSI did not formally document the strategy
and tools the PPSI would use to foster collaboration, but
the intent to support collaboration is set out in several
key documents. The second MOU, signed October 24,
2007, established the agreement with participants
that appears in the text box on the next page. Although
15 These groups were not given decision-making authority (S19).
19     Evaluation of the Oregon Paint Stewardship Program

-------
some of the items listed were not completed, the MOU
did establish a framework for continued participation
in collaborative efforts. The framework was continued
through the development of a draft work plan for the
Minnesota Demonstration project. The work plan
described the consensus among participants on the
mission, method, goals and objectives of the project while
establishing the roles of those participants (PPSI, 2008).
    PSI's role in the process was not to serve as an
independent facilitator, but rather to act as an objective
facilitator and, at times, act as an advocate for paint
stewardship. Program stakeholders agreed that PSI
provided a vital role in the sharing of information and
identifying state contacts (S16, S17, S18, S19, and
S20). Additionally, one stakeholder felt that PSI's lack
of impartiality was a benefit to the process since PSI
worked actively to push the process along (S18).

Effectiveness of Tools/Strategies
    The survey of PPSI participants (Braunz et al,
2010) asked respondents to rate the effectiveness of
the periodic briefing calls and conferences. The results
from the survey, summarized in Figure 4, indicate that
PPSI participants viewed both the briefing calls and  the
conferences as being effective.
"The ...PPSI participants agree to work
collaboratively over the next three years
to do the following:
•   Continue the dialogue through regular meeting
    and workgroup conference calls.
•   Design, implement, and evaluate a
    demonstration project.
•   Pursue a voluntary multi-stakeholder approach
    for the demonstration project while  addressing
    mechanisms to avoid anti-trust implications.
•   Support and implement strategies to effectively
    change consumer behavior.
•   Pursue completion of a  life cycle assessment
    and cost benefit analysis (LCA/CBA) project.
•   Collect baseline data on aerosol paints."
    PPSI, 2007
  Figure 4 - PPSI Participants' Perceived Effectiveness of Briefing Calls and Conferences
           Briefing
             Calls
      Conferences
                            10        20        30        40       50        60        70
                         Effective       •  Neither Effective nor Ineffective        •  Ineffective
                                       80
                                                 Promoting Environmental Results Through  Evaluation     20

-------
3.2.2 To what degree was the
pilot program, from planning to
implementation, a collaborative
process? How was the
collaborative process viewed by
different groups  involved in the
process?

Overall Impressions of the Process
    To a large extent, PPSI participants agreed that
the process was collaborative (Braunz et al, 2010) (see
Figure 5) with 84 percent of respondents to the survey
agreeing that the process was collaborative (19 percent
strongly agreeing) and only 6 percent disagreeing that
the process was collaborative.
    The survey implemented by Duke asked the PPSI
participants to rate five dimensions of collaboration
identified by Thomsen et al (2007) (see page 17):
governance, administration, mutuality, norms, and
autonomy. Respondents were presented with a series
of questions related to each dimension and were
asked their level of agreement with a statement. The
full details of this process can be found in Braunz et al
(2010). Respondents were provided with seven options
on a seven-point scale with the endpoints anchored
at "strongly agree" and "strongly disagree" and the
middle (4th option) being anchored at "neutral."16
Survey responses were translated to a numeric value by
assigning a +1 to "strongly disagree" and increasing that
value by one until a +7 was assigned to "strongly agree"
(see graphic below).
+1 +2
I I
I I
Strongly
Disagree
+3
I
I
+4
I
I
Neutral
+5
I
I
+6 +7
I I
I I
Strongly
Agree
    Figure 6 provides a summary of the data collected
through the survey of PPSI participants for all
respondents, broken out by respondent categories
for the five dimensions of collaboration. The results
for governance, administration, mutuality, and
norms dimensions tend to be in the 4.5 and 5.5 range
indicating overall agreement that the process had
been consistent with those dimensions. Additionally,
within those four dimensions, there is little variation
between the four groups (local government, state
government, private, and non-profit/NGOs). One
consistency that can be found is that responses from
individuals in private companies tended to be lower
than those in local or state government. The dimension
of autonomy (balancing competing demands of one's
own organization with demands of collaborative
effort), however, was rated relatively low in the survey
by all respondents and by each group separately. In
other words, most respondents disagreed that the
process allowed them to balance the needs of their own
organization and that of the collaborative effort.
    The survey results in Figure 6 indicate that, overall,
collaboration was viewed favorably by most respondents
in each dimension of collection, except for the autonomy
dimension. In response to open ended questions,
(Braunz et al., 2010), PPSI participants indicated that
the process  for achieving collaboration was costly and
should have been outlined from the start so participants
were informed of how they were to be engaged.

Viewpoints on Collaboration over the Lifecycle
of the Process
    When viewed from the different stages of the
process, some stakeholders saw a break down in
collaboration when the process moved from planning a
program in general to developing the legislation for the
Oregon program and in planning and implementing the
Oregon program.
    ACA indicated that the legislation was a compromise
of all the parties (S17). PSI's intention was that the
entire process, including development of legislative
language, would be collaborative (S19). In the survey
of PPSI participants (Braunz etal., 2010), however,
10 respondents indicated that they did not see the
legislation as collaborative. Six of the 42 open-ended
comments on collaboration rated collaboration as lower
when the  Initiative moved into the implementation stage.
    HHW programs echoed the breakdown in
collaboration. Some HHWs indicated they wished
they could have contributed so the program tied in
16 The second, third, fifth, and sixth points on the scale were not provided a text anchor.
       Evaluation of the Oregon Paint Stewardship Program

-------
 Figure 5 - Agreement among PPSI Participants on the Extent to which the PPSI Process was Collaborative
                                                                                 Neutral
                                                                                 Disagreed
                                                                                 Agreed at some level
                                                                                 process was collaborative

                                                                                 Strongly agreed
                                                                                 Agreed
                                                                                 Somewhat agreed
 Figure 6 - Likert Scale Values for Five Dimensions of Collaboration, by Respondent Category
Governance
 Autonomy
Administration
 Mutuality
  Norms
         0.00          1.00          2.00          3.00
        All        I  Local Govt         I  State/Federal Govt
4.00          5.00         6.00          7.00
     Private       •  Non-Profit/NGO, etc.
                                                   Promoting Environmental Results Through Evaluation     22

-------
the existing solid waste infrastructure (S13, S14, and
S15). One HHW representative specifically commented
that they would have liked to have an opportunity to
provide meaningful input into the organization of the
program; they had attempted to participate early in the
implementation process but felt as though they were
getting in the way (S13).
    From PSI's perspective, the vetoes in Minnesota
influenced the level of collaboration (S19). After
Minnesota governor vetoed legislation twice, the group
stopped work for 1.5 years and lost momentum. During
that time, funding for PSI was limited, which resulted in
less communication which did not change with passage
of Oregon legislation. Oregon was also less involved in
the PPSI's work leading up to the Oregon program (S19).
    Nevertheless, the Evaluation Team observed
that PPSI products and artifacts of collaboration
(e.g. relationships, research, formal agreements, and
committees) created between 2002 and 2008 had a
prominent role in influencing Oregon legislation and the
resulting program plan (Oregon, 2009; PaintCare, 2010).
3.3   Learning
    The findings discussed in this section have led to the
identification of a number of key learning points. These
points include:
    •    Collaboration can provide clear benefits. At
        its inception in 2003, the PPSI embraced a

   "From the start, collaboration should be
   more purposeful with clearer goals for
   collaboration that define and drive the
   type, level, and timing of collaborative
   effort necessary from various groups and
   individuals...If collaboration were more
   directed at achieving particular goals,
   resources (time, money, staff) could be more
   efficiently distributed across the overall
   effort to make the PPSI more effective."
   (PPSI survey respondent as quoted in Braunz
   et all, 2010)
     collaborative approach to achieving its goals.
     Much of the work, developed collaboratively,
     under the Minnesota program carried over
     to Oregon and much of that work found its
     way into the Oregon legislation and plan. The
     collaborative effort resulted in a pilot program
     that reflected the needs of diverse stakeholders.
     This evaluation and much of the data collection
     and analysis are a product of the collaborative
     process. Furthermore, the products and artifacts
     of the PPSI process (e.g. relationships, research,
     formal agreements, goals, committees) prior
     to conception of the Oregon pilot provided the
     background and baselines for legislation and
     planning in Oregon.

     Open communication is key to good
     collaboration. Without open communication
     about fundamental changes in the collaborative
     process as the PPSI transitioned to the
     implementation stage, PPSI participants lost
     confidence in the process and their ability
     to contribute to the process. To maintain
     participants' confidence and engagement in
     the process, the anticipated level of desired
     and feasible collaboration should, at all stages
     of the process (implementation and drafting
     legislation) be agreed upon and documented.
     Collaboration has its place but that may not
     be at every stage of the process. Collaboration
     decreased during the legislative process
     and during program design as legislative
     requirements set strict timelines; during this
     time, ACA and Oregon DEQ worked independently
     of the PPSI.
"...the time and resources committed to
getting to this point were great and not likely
replicable for other specific product areas.
This...in my mind doesn't provide a viable
model...for other product areas."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
23     Evaluation of the Oregon Paint Stewardship Program

-------
     Sustained commitment is important. Effective
     collaboration requires sustained commitments
     to a facilitator, funding, and communication.
     Declining collaboration coincided with
     decreased PPSI funding, loss of the facilitator
     to lead the process, and overall reduced
     communications amongst PPSI participants.

     The facilitator may need to also be an advocate.
     As noted, one stakeholder felt that PSI's lack of
     impartiality was a benefit to the process since
     PSI acted as an advocate for paint management
     (S18). Thus, use of an impartial facilitator (S18)
     (i.e., one without an interest in the outcome)
     may be less effective at pushing the issue
     forward and attaining results.
Duke
NICHOLAS SCHOOL OF THE
ENVIRONMENT
The Duke University graduate student team, advised by Dr. Randy
Kramer of Duke University's Nicholas School of the Environment,
presented an overview and a full report of their work on Evaluation
Question 1 regarding collaboration which can be found as items #21 and
#22 under Additional Materialsatwww.paintstewardshipprogram.com.
                                          Strategy and roles should be defined upfront
                                          and documented. Collaboration requires
                                          the commitments of increasingly scarce
                                          resources (money, time, etc.) for local and state
                                          governments. If collaboration is determined
                                          to be a core component of achieving paint
                                          stewardship goals, an explicit strategy set to
                                          achieve clear goals (e.g., equity, cost efficiency,
                                          diversity) will help to manage expectations
                                          throughout the process. Groups need to
                                          decide early in the process if collaboration is
                                          important and for which aspects of program it
                                          serves a greater priority. During the process,
                                          open discussions led by facilitators or those
                                          requesting changes, should accompany any
                                          significant unanticipated adjustments to the
                                          level and/or type of collaboration.
"Some organizations tend to work at
cross purposes. There was also a point
where ACA and Oregon were sealed off in
negotiations and the rest of the group was
sealed out."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
"...there was not collaboration by the
group on the actual text of legislation."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
"In Minnesota, the collaboration was
extremely effective until the issue reached
the required legislation to implement it."
(PPSI survey respondent as quoted in Braunz
et all, 2010)
                                   "It was a collaborative effort from a
                                   planning standpoint - but has been
                                   much less collaborative in its actual
                                   implementation."
                                   (PPSI survey respondent as quoted in Braunz
                                   et all, 2010)
                                   "I don't think implementation has been
                                   a collaborative effort with the PPSI nor
                                   the legislative process. The dialogue and
                                   keeping the PPSI informed as well as
                                   the roll-out I strongly agree has been a
                                   collaborative process."
                                   (PPSI survey respondent as quoted in Braunz
                                   et all, 2010)
                                            Promoting Environmental Results Through Evaluation     24

-------
4.0    Paint  Stewardship Organization
  ^  A *- .   „  EVALUATION QUESTION:
             b  Describe the Paint Stewardship Organization (PSO)
               (Pa intC a re), including its  funding mechanism and
               infrastructure.
   What factors contributed to its infrastructure choices?
   Was the funding mechanism clearly defined, transparent, and complete?
   What are the lessons learned?
4.1  Data Sources, Methods

and Analysis

   The Evaluation Team answered this evaluation
question primarily through a qualitative approach of
reviewing PSO documents and interviews with key
program stakeholders.
   In order to describe the infrastructure funding
mechanism of the PSO as it relates to the Oregon Pilot
Program, the Evaluation Team partnered with graduate
students at Georgia State University (GSU) as part of
their capstone project in evaluation. The GSU team
created an interview questionnaire and conducted
telephone interviews with eight participants who were
involved in the implementation, current operation,
and oversight of the Oregon Pilot Program. In addition
to conducting interviews, the GSU students gathered
and analyzed information from the Oregon legislation
and from the Oregon Pilot Program documentation
made available by the PSI, PaintCare Inc., the Oregon
Department of Environmental Quality, EPA, and ERG
(Bledsoe et al, 2011). Following the final report by GSU,
the Evaluation Team also conducted interviews with
                                               The GSU graduate student team, advised
                                               by Dr. Cynthia Searcy developed a report for
                                               Evaluation Question 2, which can be found
                                               under Additional Materials as item #25 at
                                               www.paintstewardshipprogram.com.
key stakeholders to clarify information and to provide
additional details for the findings.
   To determine the factors that contributed to PSO
infrastructure choices, the Evaluation Team analyzed
four components of infrastructure: 1) education
and outreach, 2) collections, 3) transportation, and
4) processing. To assess PSO funding mechanisms,
the Evaluation Team gathered data from interview
questions with regard to the meaning of the terms
clarity, transparency, and completeness in the context of
funding mechanisms. The team adopted the following
working definitions:

   •   Clarity: the extent to which the funding source
      for the PSO is clearly defined in implementing
      legislation, rules, and program documentation.

   •   Transparency: the extent to which a member
      of the public could track how the funding is
      being used by the PSO.

   •   Completeness: the extent to which PSO funding
      covers the expenses for operating the PSO.
25    Evaluation of the Oregon Paint Stewardship Program

-------
   Clarity:  The extent to which the funding
   source for the PSO is clearly defined in
   implementing legislation, rules, and program
   documentation.
   Transparency: The extent to which a
   member of the public could track how the funding
   is being used by the PSO.

   Completeness: The extent to which PSO
   funding covers the expenses for operating the PSO.
4.2   Findings
4.2.1   Describe the Paint
Stewardship Organization
(PaintCare), including its funding
mechanism and infrastructure.

Infrastructure: Process Actors
    Figure 7 provides an overview of the structure of
the program (PaintCare, 2010; Bledsoe et al., 2011).
ACA created PaintCare as a 501 (c) (3) non-profit PSO
to manage the reuse, recycling, and proper disposal of
unused paint as mandated by the Oregon legislation.
The Board of PaintCare consists of nine non-paid
representatives of architectural paint manufacturers
(PaintCare, 2010; Bledsoe etal., 2011).
    PaintCare contracted with Product Care Inc., which
has experience implementing similar stewardship
programs, to implement and manage the Oregon
program (Bledsoe etal, 2011). PaintCare selected PSC
and Metro as transportation and processing vendors;
these organizations had been part of the pre-program
infrastructure (Bledsoe etal., 2011).
    Metro, PSC, and Amazon Environmental are the
service providers for the processing of latex paint
(PaintCare, 2010; Bledsoe et al, 2011). Their roles are
as follows:
    •   Metro collects recyclable paint from its
       locations and accepts recyclable latex collected
       from the remainder of the state from PSC's
       consolidation center. This paint is made into
       new recycled content paint, which is then sold
       (PaintCare 2010,2011b).

    •   Non-recyclable latex paint collected by Metro
       is disposed of by biodegradation at Columbia
       Ridge Landfill in Arlington, Oregon. Through
       a research permit, the paint is mixed with
       wastewater and pumped into the landfill to
       improve degradation of wastes and recovery of
       landfill gas (PaintCare, 2011b).

    •   Non-recyclable latex paint collected and
       consolidated by PSC is processed into recycled
       content paint, a biomass fuel product, and
       processed latex pigment (PLP) (an alternative
       raw material for cement manufacture) by
       Amazon Environmental (PaintCare, 2011b).

    Oil-based paint is collected and processed for fuel
blending by PSC (PaintCare 2010, 2011b).

Infrastructure: Collection Sites
    PaintCare's strategy to set up the collection
infrastructure included the following components:
(PaintCare, 2011b; S17):

    •   Incorporate pre-program collection
       infrastructure (i.e., HHW facilities).

    •   Reach out to paint and independent hardware
       retailers to fill gaps  in underserved areas.

    •   Send a letter to retailers of paint and decorator
       trade association list informing them of the
       program and giving them an opportunity to be
       a collection site.

    •   Conduct Web searches and review yellow pages to
       reach out to additional paint retailers in gap areas.

    The collection program began with 45 collection
sites in July 2010 and, as of August 2011; there were
a total of 98 sites collecting paint (see Figure 8)
(PaintCare, 2011b). As of August 2011, all HHW
facilities that collected latex  paint prior to the program
were collection locations in the PaintCare program
                                               Promoting Environmental Results Through Evaluation    26

-------
  Figure 7 - Organization of the Oregon Paint Stewardship Program
         Product Care
          Association
                        Admin
                       Services
                                        American Coatings
                                            Association
                                          PaintCare, Inc.
                                          (program operator)
                                          Management Services
                                               Contract
                                                  Producers
            Program
          Coordinator
                PCA Paint
            Stewardship, Inc.
             (program manager)
                                            Service Providers
  Figure 8 - Numbers of Collections Sites in July 2010
  and August 2011
  Numbers and Types
   of Collection Sites

       •  Retailers
       •  Restores
       •  HHWs
           July 2010
August 2011
(S17). None of PaintCare's 98 collection locations are
paid for collection services (S17).
    The program provided each collection site with the
following (PaintCare, 2011b):

    •   A Collection Site Procedures Manual (see text
       box on page 28).

    •   Program training on collection site operation,
       customer service, environmental risk
       reduction, visual inspection of post-consumer
       paint containers, and placement of collected
       paint in program containers.

    •   Compliance visits to confirm the site is
       following program requirements and to
       investigate complaints.17

    Finally, the program transportation service
provider (PSC) places collection containers at each
collection site; collection sites schedule a pickup by
calling the transportation service provider when the
collection containers are approximately 50 percent full
(PaintCare, 2011b).
17 PaintCare conducted 76 compliance visits during the first year of the program.
27    Evaluation of the Oregon Paint Stewardship Program

-------
  Contents of the Collection Site Procedures Manual
  •   Collection site standards
  •   Screening procedures for conditionally exempt
     small quantity generators
  •   Accepted and non-accepted program products
  •   Management of the Paint Exchange program,
     including required waiver forms
  •   Reporting requirements
  •   Management requirements and operational
     procedures
Infrastructure: Other Collection Services and
Other Services
    During the first year of the program, collection
events were held to collect paint in rural locations
(PaintCare, 2011b). PaintCare anticipates completing
arrangements with permanent locations in order to
reduce costs (S17). By September 1, 2011, the program
had participated in 57 HHW collection events and two
PaintCare-sponsored events.
    PaintCare provided additional collection services
through large volume direct pickups and collection
events (PaintCare, 2011b). Between July 1,2010 and
June 30, 2011 the program provided direct pickup
        service to 19 institutional, commercial, and industrial
        entities, such as trade painters, apartment complexes,
        housing authorities and other private businesses
        that meet the criteria for conditionally exempt small
        quantity generators.
            PaintCare offers a reuse program though Metro and
        several Restores where good, usable paint in containers
        that are at least half full are offered to the public for free
        or at a discounted price (PaintCare, 2011b).
            Finally, the program identified opportunities for
        recycling plastic pails and steel paint cans from the
        Metro program; during the first year, the program
        recycled 47.1 tons of plastic and 64.8 tons of metal
        paint cans (PaintCare, 2011b).

        Funding Mechanism
            PaintCare collects a recovery fee from paint
        producers on all architectural paint sales to fund the
        program. This fee was authorized in the legislation and
        the fee was approved by the Director of the Oregon
        Department of Environmental  Quality (PaintCare,
        2010). The assessment rate by container size is
        summarized in Figure 9.
            PaintCare's budget and fee were developed based
  Figure 9 - Summary of Assessment Fee
     $2.00
     $1.50
     $1.00
     $0.50
     $0.00
                                                                   Note: Program does not accept containers
                                                                         greater than 5 gallons.
             1/2 pint or less   1/2 pint - quart
Quart - 1 gallon
1 gallon - 5 gallons
                                                 Promoting Environmental Results Through Evaluation     28

-------
on the estimated sales of architectural paint in Oregon,
estimated proportions of leftover paint available for
collection, and volume and cost data from the Metro
program and Oregon DEQ (non-metro areas). According
to the 2010 Program Plan, 2011 Annual Report, and
additional explanation by a PaintCare representative
(Em8):

    •    Revenue was estimated by pro-rating the
        national architectural paint volume sold by
        population. This volume was compared with
        ProductCare sales data from British Columbia
        and will continue to be refined as PaintCare
        receives actual sales data from program
        participants. The final number multiplied by an
        average cost per container produced the initial
        revenue estimate.

    •    Program costs included:

               Projected collection volumes estimated
               from rates of leftover paint derived
               from research; Metro and non-
               Metro paint collection volumes; and
               assumptions about the amount of paint
               available for collection (10 percent), the
               recovery rate of program products sold
               (7.1 percent), capture rate (71 percent),
               and anticipated growth in collection
               volumes per year (6 percent).
                                             Administrative fees.

                                             Vendor pricing to provide services
                                             based on projected collection volumes.

                                  •   The values were estimated for a 4 year budget,
                                     then used as inputs into a spreadsheet model
                                     which was used to model collection fees
                                     under several scenarios (i.e., per container fee,
                                     per gallon fee, flat fee, and graduated fee) to
                                     identify the scenario and the fee amounts that
                                     would best cover budgeted costs.

                                  PaintCare developed a graduated fee (i.e., less costly
                              per gallon as container volume increases; see Figure 9)
                              based on the assumption that less waste paint is typically
                              generated from those purchasing a 5 gallon container
                              and that a higher volume of oil-based paint, which is
                              more expensive to manage as waste paint, is sold in
                              smaller containers (S17).
                                  Figure 10 provides an overview of how the fee is
                              transmitted between different actors in the program
                              (PaintCare, 2010; Bledsoe et al, 2011). Producers
                              report monthly sales and pay the assessment directly
                              to PaintCare through a secure online filing system,
                              electronic fund transfer, or check The assessment is
                              passed through to a distributer or retailer who must
                              add the fee to the final sales price of their products
                              (PaintCare, 2011b). Retailers are responsible for
                              collecting the fee from consumers and transferring to
  Figure 10 - Flow of Assessment Fee
  within the Program System
               Stewardship fee paid per
               container sold in Oregon
            Paint
       Manufacturer
                 Paint Stewardship
                    Organization
        Invoice & payment
           includes fee
t»
                                              Funding to oversee program
                                                                   $
       Paint Retailer
       (Purchase price of paint
           includes fee
        Consumers
t»
                                     $
Program Delivery:
• Administration
• Collection
• Transportation
• Processing
• Communication
                                                                  Flow of Money
                                                                  Flow of Paint
29     Evaluation of the Oregon Paint Stewardship Program

-------
paint producers, who are then responsible for paying
the fee to PaintCare (PaintCare, 2011b). The legislation
provides retailers the option to show or not show the
fee on the consumer's sales receipt - this decision was a
compromise among PPSI stakeholders (S17, S20).


4.2.2  What factors contributed to

its  infrastructure choices?


Processing and Transportation
    In terms of processing and transportation, the
primary factor that contributed to the infrastructure
choices was the pre-existing infrastructure put in
place prior to the program. Specifically, Metro had an
established, mature paint collection and processing
infrastructure in the Portland metro area. Metro
continued to operate collection sites and process
recyclable latex paint into MetroPaint, a recycled
paint product sold by Metro.  The program leveraged
this infrastructure in developing the current Oregon
program in areas outside of the Portland metro
(Bledsoe et al, 2011). Recyclable latex collected outside
of the Portland metro area was shipped from PSC
consolidation facilities to existing Metro facilities for
recycling. The program also relied on the pre-program
transportation provider (PSC) to transport the paint.18


Collection Sites
    As shown in Figure 8 (see page 27), most sites
were retailers (70 sites; 71 percent in August 2Oil)
and HHWs (18 sites; 18 percent in August 2011).
PaintCare's coverage goal was to provide a permanent
collection site within 15 miles for more than 70 percent
of Oregon's population, which served as a primary
driver in determining the collection site locations
(PaintCare, 2011b; S17). Initially, PaintCare did not add
additional locations in an area (i.e., estimated as the
area within a 15 mile radius of a population center)
where a collection point was already established.
However, at the request of one local retailer, a state
representative encouraged PaintCare to reassess this
approach. PaintCare's current policy is to  add additional
collection locations near another location only if no
significant costs are incurred (S17). PaintCare reported,
however, that it took longer than anticipated to
negotiate contracts with many collection sites (S17).

18 PSC, however, was selected through a Request for Proposals process.
4.2.3  Was the funding mechanism
clearly defined, transparent, and
complete?


Clearly Defined
    Stakeholders were asked to assess the clarity of the
funding mechanism in a series of interviews (Bledsoe et
al, 2011). Stakeholders indicated that:

    •   Legislation clearly details funding mechanisms
       in terms of manufacturer responsibility, and
       indicates that consumers will pay a fee at the
       time of a paint purchase.

    •   Program brochures and the PaintCare website
       communicate the details of the funding
       mechanism.

    •   Implementing a graduated fee based on
       container size as opposed to flat fees was
       viewed positively by the public and paint
       contractors.

    As detailed in the Annual Report the program
estimates that most retailers show the fee on receipts
(PaintCare, 2011b). However, for some retailers,
reprogramming their computer system acted as a
disincentive to show the fee as a separate line item
on the receipt (S7, S8, and S9). The July 2011 survey
of consumers found that 11 percent of respondents
were aware of the fee (PaintCare, 2011a). Interviews
conducted by the Evaluation Team with retailers and
HHW programs support the July 2011 survey finding
of low consumer awareness; two HHW interviewees
reported some consumers confusing the fee with a
bottle deposit (i.e., by returning the empty container
the  fee would be refunded) (S12, S15).

Transparent
    The PaintCare report contained a detailed financial
audit of the program which provides  some level of
transparency. Stakeholders' views on the degree of
transparency were dependent on the interviewee's
point of view. Those involved with PaintCare felt that
a publically available budget meant the program was
transparent, while other interviewees felt the lack of
broken out costs made it less transparent.
                                               Promoting Environmental Results Through Evaluation     30

-------
Complete
    In the 2011 Annual Report, PaintCare states that
despite the first year deficit projected in the program
plan and lower than estimated sales, the project
actually produced a surplus of $259,911 due to lower
than estimated expenses. This surplus, however, was
not sufficient to reduce assessment costs and will be
carried over to year two  and used for program costs
(PaintCare, 2 Ollb).
4.3   Learning
4.3.1   Infrastructure

    •   Existing infrastructure influenced program
       design. The Oregon program was built
       upon existing institutional knowledge and
       experience, relationships, and infrastructure,
       resulting in reduced need for training and
       upfront costs. Other states and municipalities
       will have varying levels and types of
       infrastructure to accomplish similar goals.
       The efficiency and feasibility of program
       design and implementation will benefit from
       initial thorough assessments of existing
       infrastructure, including transportation,
       reprocessing capacity, and related knowledge
       and experience.

    •   A more explicit and collaborative strategy
       would improve collection site selection.
       A collaborative and/or strategic process
       for selecting collection sites may improve
       efficiency of planning and implementation.
       Opportunities include clear statements of
       process steps, criteria for site  selection,
       and designated opportunities for specific
       stakeholders (such as retailers and HHW
       operators) to provide input in the process.
4.3.2 Assessment Fee

    •   Strategic and targeted outreach is needed
       to improve fee awareness. One goal of the
       Oregon paint legislation is that consumers
       become aware of the fee. In July 2011, few (11
       percent) recent purchasers of paint were aware
       of the fee (PaintCare, 2011b). Fee awareness
       may be increased through a more strategic and
       targeted education and outreach campaign (see
       Learning section under Evaluation Question 3).

    •   Fee awareness should be reconsidered
       as a program goal. Future iterations of paint
       legislation may consider whether consumer fee
       awareness is a necessary policy goal.
       In August 2010, most Oregon residents
       (73 percent) indicated that the fee was
       reasonable with few (23 percent) indicating
       it was not reasonable (Bradshaw Advertising,
       2010). If awareness of the fee is set as
       a program goal, then a target level of
       performance (e.g., a targeted percentage
       becoming aware of the fee among Oregon
       residents) should be set.

    •   Oregon fee structure could act as a model
       for other states.  The process used to set/
       design the fee structure is a baseline model for
       other states; it covered costs in the first year
       of the program. Notably, lower than expected
       collection of oil-based program products
       contributed to the one-year surplus.
       Evaluation of the Oregon Paint Stewardship Program

-------
5.o   Education  and Outreach
               EVALUATION QUESTION:
               How did education materials and strategies affect
               consumer awareness and behavior?
  Which messages were most effective with which target audiences?

  What materials/strategies were developed and what were the goals and
  target audience of those materials/strategies?
   Did other factors besides the program influence consumer behavior and
   awareness?
  What are the lessons learned?
   One of the key goals of the Oregon Pilot project
is to minimize waste paint and containers generated
by consumers. Toward that end, one of the first steps
PaintCare took during the project was to launch an
outreach and education campaign; the campaign is
described in the June 29, 2010 approved pilot program
plan (PaintCare, 2010). Evaluation Question 3 focuses
on assessing the impact of this outreach and education
program on consumer awareness and behavior.


5.1  Data Sources, Methods
and Analysis

   To develop the measures of program awareness,
the Evaluation Team drew on the results of consumer
surveys that asked the target audience to provide self-
ratings of degrees of awareness on the program. The
primary data sources for the measures include:19
Initial consumer survey: A telephone survey
of 409 Oregon homeowners, conducted by
Bradshaw for PaintCare, performed from
August 4 to 10, 2010. Full survey results are
available in Bradshaw Advertising's report to
PaintCare (Bradshaw Advertising, 2010), as
well as in the appendix to PaintCare's annual
report (PaintCare, 2011b). We refer to this as
the August 2010 survey.

Follow-up survey: An online survey of 1,064
Oregon residents over 18 years old, conducted
from July 7 to 12, 2011. This survey targeted
individuals who have purchased paint between
July 1,2010 and June 30,2011 as well as
individuals who paint as an occupation. Full
survey results are available in PaintCare's
report on the survey and in the appendix to
its annual report (PaintCare, 2011a,b). In this
report, we refer to this as the July 2011 survey.
19 These data sources are described in more detail in Section 2.4.
                                       Promoting Environmental Results Through Evaluation    32

-------
    The two surveys are not identical, constraining the
before and after comparisons that can be made. First,
the two surveys cover different target populations. The
August 2010 survey covers Oregon homeowners and
the July 2011 survey covered Oregon residents that
had recently purchased paint or that paint for a living.
Second, the two surveys used different sets of questions
with little overlap in the questions asked between the
two surveys.
    The data collected through the surveys was
supplemented by a qualitative approach; including a
review of the materials produced by PaintCare and
interviews with key program stakeholders.
5.2  Findings
5.2.1  What materials/strategies
were developed and what were the
goals and target audience of those
materials/strategies?

    The communication firm chosen by ACA developed
an outreach and education strategy with messages
consistent with the themes of ACA's "Be Paintwise"
program and targeted to all Oregon consumers. The
educational materials used brief taglines such as "From
Storage to Spectacular," "From Garage to Glorious," and
"From Basement to Beautiful" to catch the attention of
consumers (PaintCare, 2010,2011b). These messages
appeared with supporting text that highlighted the
following themes:
    •   The name of the program and its website.
    •   The advantages (cost savings, environment,
       storage space) of recycling leftover paint.
    •   The availability of information on purchasing
       the correct amount of paint and managing and
       disposing of paint properly on the program
       website.
    •   The recovery fee.
    The program website (www.paintcare.org) also
featured educational information while providing
additional detail on each of the messages and on the
program itself (e.g., collection locations).
    Figure 11 provides an overview of the education
campaign under the program. Samples of the materials
that were used by the program can be found in
appendices of the program plan (PaintCare, 2010) and
the annual report (PaintCare, 2011b). The appendix
to the annual report (PaintCare, 2011b) also contains
details on the where (e.g., radio stations) and when
advertising spots were purchased.
    Despite its wide reach, the outreach and education
campaign did not identify or target specific audiences
(PaintCare, 2010). The messages, materials, and
delivery mechanisms were designed to have broad
appeal and reach as many Oregon residents as possible,
particularly those interested in purchasing paint.
   Objectives of the education and
   outreach campaign:

   •  Build consumer awareness.

   •  Identify program products.

   •  Identify collection site locations.

   •  Emphasize the environmental impact of
     poorly managed leftover paint.
     Emphasize the importance of purchasing the
     correct amount of paint.
     Promote paint reuse.
     Promote recycling and proper disposal of
     leftover paint.
33    Evaluation of the Oregon Paint Stewardship Program

-------
  Figure 11 - Overview of the Education and Outreach Campaign
           Point of sale materials
           • 54,748 counter cards and
            320 posters to 190 locations
     Direct mailings to contractors
     and retailers
     1 2,414 factsheets to trade
      painters
     1 1,400 factsheets to retailers in
      3 mailings (12/09, 4/10 and 6/10)
Website (www.PaintCare.org)
• 531,038 hits between March 4,
 2010 and August 26, 2011
        Education
      and Outreach
        Campaign
         Press releases and information sharing
         with the news media
         • Prior to start date: 13 TV and radio stories,
           56 print articles, and 36 Internet articles
         • Earth day tie-in with 2 magazine articles
           and  8 Internet articles
         • At the 1 -year anniversary: 6 articles and
           a new radio interview
1-800-CLEANUP
(Numbers of calls received
not available at this time)
     Advertising through local
     solid waste partners
                                        Trade shows
                                        • 8 trade shows in
                                          2010 and 7 in 2011
               Purchased advertising
               • Radio advertisements on 17 radio stations in 2010
                and 24 stations in 2011
               • TV advertisements on 2 stations in the Portland
                Metropolitan market
               • Newspaper advertisements in 33 newspapers
                across 29 markets in 2011
5.2.2  How did education  materials
and strategies affect consumer
awareness and behavior?

Program-Related Awareness
    Figure 12 summarizes program awareness in
both surveys. In the August 2010 survey, 22 percent of
respondents report being "aware" or "very aware" of
the program (Bradshaw Advertising, 2010). In the July
2011 survey 27 percent of respondents recall seeing
or hearing advertisements for the program (PaintCare,
2011a). However, it is difficult to determine the change,
if any, in awareness because results are based on
different target audiences.
    Figure 13 summarizes survey data on the ability of
respondents to correctly identify in-scope products. A
reported 44 percent of respondents to the August 2010
survey incorrectly indicated that oil-based paint was not
covered by the program. Furthermore, one in five respondents
incorrectly indicated that latex paint is not covered.
                   The consumer outreach materials provided little
                information on the types of products that would be
                covered or not covered under the program. The rack
                card20 contained a question and answer section that
                indicated the types of products covered, while the
                program website covers types of products.
                   Several of the materials produced by PaintCare
                provided information on collection sites or how to
                find collection sites. A January 2008 survey of Oregon
                residents found that 67 percent of respondents have
                taken leftover or unwanted household products to a
                collection event or facility in the past (Oregon DEQ,
                2008). However, since this includes any leftover
                household products it may be an overestimate of
                baseline awareness of sites that accept leftover paint.
                The August 2010 survey did not ask consumers
                about their awareness of site locations (Bradshaw
                Advertising, 2010). The July 2011 survey found that
                31 percent of respondents who purchased paint were
                aware of a collection site (PaintCare, 2011a).
20 The card "Why Everyone Should Get with the Program" that consumers would have found at paint or checkout counters in retail locations contained
  summary information on the program. A sample of the card can be found in the Appendix to the PaintCare annual report (PaintCare, 2011b).
                                                Promoting Environmental Results Through Evaluation     34

-------
  Figure 12 - Program Awareness in August 2010 & July 2011
               August 2010
July 2011
Awareness of Environmental Impacts of Paint
    The program and accompanying education
materials do not provide consumers with information
about the environmental impacts of leftover paint.
Two exceptions include: the rack card entitled "Why
Everyone Should Get with the Program," which
indicates one benefit of recycling paint as "(e)liminating
storage hazards where you live," and a page on the
website (www.paintcare.org/) indicates that leftover
paint may find its way into landfills and frames this
as an environmental issue. The July 2011 survey did
not address this aspect of awareness, so there is no
indication of whether awareness improved (PaintCare,
2011a). However, the August 2010 survey found that 61
percent of respondents indicated it is "very important"
and 31 percent indicated it is "somewhat important" to
have a program that accepts unneeded, leftover paint
for reuse, recycling and proper disposal (see Figure 14)
(Bradshaw Advertising, 2010).

Consumer Behavior
    The program promoted few messages related
to reducing the amount of paint purchased. These
messages encourage consumers to buy the right
amount to:21

    •  Save money: buying the correct amount
       reduces the amount of money spent on paint

    •  Save the environment: reducing leftover
       paint reduces the amount of paint that might
       find its way into landfills

    •  Save storage space: less leftover paint
       means more garage and basement storage
       space for consumers

    •  Reduce  the risk of exposure to hazardous
       materials: storing paint in home can pose
       exposure and fire risks.
  Figure 13-Ability of Respondents to July 2011 Survey to Correctly Identify In-Scope Products

    100%
     90%
     80%
     70%

     60%

     50%
     40%
     30%

     20%
     10%
      0%
                Oil-based is covered
                                       Latex is covered
                                         Correct
                                                             Paint thinner is
                                                              not covered
                                                   Containers with
                                                 >5 gals not covered
                          Incorrect
21 www.paintcare.org/index.php.
35     Evaluation of the Oregon Paint Stewardship Program

-------
    The paint calculator served as the program's primary
tool of the program for assisting consumers in buying
the right amount of paint. In the July 2011 survey, 15 out
of 84 (18 percent) respondents reported being aware of
the paint calculator, and 3 of those respondents indicated
using the paint calculator to determine how much
paint to purchase (see Figure 15) (PaintCare, 2011a).
However, neither survey directly addressed whether
consumers understood the calculator.
    The August 2010 survey asked respondents about
what they would likely do with leftover or unneeded
paint; the results of this survey question are summarized
in Figure 16  (Bradshaw Advertising, 2010).
    In the August 2010  survey, respondents were also
asked whether the existence of the program would
affect the likelihood that they would recycle (see Figure
17) (Bradshaw Advertising,  2010). These data points
should be interpreted in the context of the January
2008 survey that found that 67 percent of residents had
taken materials to a collection event or facility in the
past (Oregon DEQ, 2008). That is, the 40 percent who
responded that the existence of a program makes "no
difference" whether they recycle may still be willing
to recycle paint and the 56 percent may include some
of that one-third of respondents in 2008 who had not
taken materials to an event or facility in the past.
    The data from the July 2011 survey show that no
respondents recycled or donated leftover paint and 72
percent of respondents would store the paint for later
projects or touch-ups (PaintCare, 2 01 la).
    ACA suggested that, based on anecdotal evidence, it
appeared that the education and outreach program had
little effect on consumers' practices (SI 7).
   Additional Findings
       The surveys conducted as part of the program did not collect information on amounts of leftover paint. However,
       previous studies of leftover paint have found that 2.5 to 16 percent of paint sold may remain as leftover paint,
       and the average amount of leftover paint per household ranges from 0.15 to 0.6 gallons (PaintCare, 2010).
       The evaluation was unable to collect data on how consumers store leftover paint. The program did not include
       questions on storage practices.
  Figure 14 - Consumer Opinions on Importance of a Paint Recycling Program
                                                                          Not important at all
                                                                          Somewhat important
                                                                          Very important
             0%
                       10%
                                  20%
                                             30%
                                                        40%
                                                                   50%
                                                                             60%
                                                                                        70%
                                                 Promoting Environmental Results Through Evaluation     36

-------
  Figure 15 - Awareness and Use of Paint Calculator in July 2011 Survey
                                                                                               Not aware


                                                                                               Aware



                                                                                               Did not use it


                                                                                               Used it
  Figure 16 - Intended Consumer Management of Leftover
  Paint (August 2010)
    35%


    30%


    25%


    20%


    15%


    10%


     5%


     0%
                            Figure 17 - Consumer Likelihood of Recycling Paint Due
                            to Existence of Paint Recycling Program
               Call       Go to
             garbage/    govt
             recycle/    recycler
             disposal
               org
Store for     Put it
 future      in the
  use      garbage
                                    More likely to recycle  •  No difference
                                              •  No answer
37     Evaluation of the Oregon Paint Stewardship Program

-------
5.2.3  Which messages were
most effective with which target
audiences?
    Data collected during the first year of the program
did not directly address the effectiveness of specific
messages. To address this question would require
linking consumers' viewing specific messages to the
actions taken by consumers. The data for this evaluation
did not include which messages were viewed by survey
respondents. Nevertheless, the two surveys can provide
some insight into differences across groups. Table 2
provides a summary of the information available in each
survey on respondents' awareness and opinions. Table
3 provides a summary of the respondent characteristics
that can be cross-tabulated with the awareness and
opinion data. We provide a series of cross-tabulations
by the different respondent characteristics in Figure 18
through Figure 21 and we used those cross-tabulations
to investigate any potential differences across
respondents, which can be summarized as follows:

    •   Importance of this type of program: All
       demographic categories (age, region, gender)
       indicated very strong support for a program
       that manages leftover paint (see August 2010
       survey in Figure 18, Figure 19, and Figure 20).

    •   Awareness of program: With respect to
       program awareness, there are a few notable
       differences across respondents in the July 2011
       survey:
Table 2 - Available Information on Respondents' Awareness or Opinions in Each Survey
 Respondents' Awareness or Opinions
 Importance of this type of program
 August 2010 Survey   July 2011 Survey
 Awareness of the program
 Awareness of the fee being charged
 Feel the fee is reasonable
 Awareness of drop-off locations
 Program makes it more like they will recycle
Table 3 - Respondent Categories Available in Each Survey
 Respondent Categories
 August 2010 Survey   July 2011 Survey
 Age
 Region
 Gender
 Education
 Income
                                             Promoting Environmental Results Through Evaluation     38

-------
        Respondents who were older tended
        to be more aware of the program (see
        July 2011 survey in Figure 18).

        Respondents living in the south and
        central regions of the state tended to
        be less aware of the program in both
        surveys and those in the Portland
        metro area tended to be more aware
        (see July 2011 survey in Figure 19).

        Respondents who were more educated
        and had higher incomes also tended
        to be more aware of the program (see
        Figure 21).

Awareness of fee: For the most part, there is
little difference in terms of awareness of the
fee. One exception is respondents making more
than $100,000 may have higher awareness
relative to other income groups (see Figure 21).

Reasonableness of fee: Overall, respondents
to the August 2010 survey indicated that the
fee was reasonable with the lowest levels of
agreement being among those 60 and older
(68 percent; see Figure 18) and among men
(67 percent; see Figure 20).
                                                            Awareness of drop-off locations: There are
                                                            some differences in awareness of drop-off
                                                            locations for respondents to the July 2011
                                                            survey:

                                                                    Respondents younger than 44 tended
                                                                    to be less aware of drop-off locations
                                                                    than those who were 45 and older (see
                                                                    July 2011 survey in Figure 18).

                                                                    Respondents who lived in the Portland
                                                                    metro and central regions of the state
                                                                    tended to be more aware of drop-off
                                                                    locations (see July 2011 survey in
                                                                    Figure 19).

                                                                    Men tended to be more aware of drop-
                                                                    off locations compared to women
                                                                    (see July 2011 survey in Figure 20).

                                                                    Respondents with higher levels of
                                                                    income and education also tended to
                                                                    be more aware of drop-off locations
                                                                    (see income portion of July 2011
                                                                    survey in Figure 21).

                                                            Likelihood of recycling: Older and younger
                                                            Oregon residents (see August 2010 survey
                                                            in Figure 18) as well as women  respondents
                                                            (see August 2010 survey in Figure 20)
                                                            indicated they would be more likely to
                                                            recycle paint because of the existence of
                                                            the program.
39    Evaluation of the Oregon Paint Stewardship Program

-------
Figure 18 - Differences by Age Group: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of
Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey)
                                            August 2010 Survey
                 Aware of program
                    18-34
 Feels it is important
to have this program
35-44          I  45-54
        Feels fee is
        reasonable
                 55-64
         Program makes it
          more likely they
            will recycle

               I  65+
                                              July 2011 Survey
         100%

          90%

          80%

          70%

          60%

          50%

          40%

          30%

          20%

          10%

           0%
                 Aware of program
                                                 Aware of fee
                                       Aware of drop-off
                                          locations
                    18-34
35-44
45-54
55-64
65+
                                                  Promoting Environmental Results Through Evaluation     40

-------
Figure 19 - Differences by Region: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of
Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey)
                                            August 2010 Survey
                 Aware of program
             Metro
             Feels it is important
            to have this program
             Feels fee is
             reasonable
                Program makes it
                more likely they
                  will recycle
    Willamette Valley
     Central/East
          Southern
               Coast
                                              July 2011 Survey
         100%

          90%

          80%

          70%

          60%

          50%

          40%

          30%

          20%

          10%

           0%
                 Aware of program
                                                 Aware of fee
                                                   Aware of drop-off
                                                       locations
          Metro
Willamette Valley
Central
East
Southern
Coast
      Evaluation of the Oregon Paint Stewardship Program

-------
Figure 20 - Differences by Gender: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of
Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey)
                                            August 2010 Survey
                 Aware of program
 Feels it is important
to have this program
Feels fee is
reasonable
                                            Male
                       Female
                                              July 2011 Survey
         100%

          90%

          80%

          70%

          60%

          50%

          40%

          30%

          20%

          10%

           0%
                 Aware of program
                                                Aware of fee
                                      Aware of drop-off
                                          locations
                                            Male
                       Female
                                                  Promoting Environmental Results Through Evaluation     42

-------
  Figure 21 - Differences by Education and Income: Program Awareness, Fee Awareness, and Drop-off Site Awareness
  (July 2011 Survey)
                                              July 2011 Survey
           100%

            90%

            80%

            70%

            60%

            50%

            40%

            30%

            20%

            10%

             0%
                      Aware of program
                                                   Aware of fee
                                              Aware of drop-off
                                                 locations
                    HS or less
          Some college - college grad
                          Post grad degree
                                              July 2011 Survey
           100%

            90%

            80%

            70%

            60%

            50%

            40%

            30%

            20%

            10%

             0%
                   Aware of program
                                                 Aware of fee
                                             Aware of drop-off
                                                 locations
               <$25K
$25K - $50K
$50K-$100K
>100K
No answer
43     Evaluation of the Oregon Paint Stewardship Program

-------
5.2.4  Did other factors besides
the program influence consumer
behavior and awareness?

    The economy may have influenced markets for paint
products and thus reduced Metro collection volumes.
ACA tabulated quarterly Census Bureau data on gallons
sold which indicated a decline from 153.2 million
gallons in the first quarter of 2010 to 146.8 million
gallons in the first quarter of 2011 (PaintCare, 2011b).
    Another potential influence to consumer behavior
and awareness lies in the value Oregon residents place
on recycling programs in general. Prior to the Oregon
program, Metro had a well-established and organized
recycling program in the Portland metro area, which
serves as home to 48 percent of Oregon residents.
Although 25 percent of consumers indicated that they
were aware of the program in the August 2010  survey,
56 percent of respondents indicated that the existence
of the program made it more likely that they will recycle
leftover paint and 92 percent said it is "very important"
or "somewhat important" to have this type of program
(Bradshaw Advertising, 2010).
5.3   Learning
        Existing culture and infrastructure is
        important. The culture and infrastructure
        related to recycling in a state or region can
        inform the design, implementation, and
        emphasis placed on the program's education
        and outreach campaign. In Oregon, a strong
        culture of recycling coupled with existing
        infrastructure was helpful to the program.
        Furthermore, a well organized and established
        program that is centralized with respect to the
        state population, such as Metro, increases the
        efficiency of program roll out and the transition
        of a large segment of state population toward
        recycling paint.

        Targeting messages may be more effective.
        The education and outreach campaign used a
        broad approach to spreading its messages and
        did not strategically aim its messages at specific
target audiences (consumers with the most paint,
contractors, retailers, etc.) with specific means
of communication (TV, website, social media,
radio, etc.). Consumers of paint can be segmented
into various groups (new versus long-time
homeowners, homeowners versus contractors,
age and other demographics). The most effective
ways of reaching each group may differ. For future
outreach efforts, consider determining which
groups are the largest purchasers of paint or who
has the most leftover paint and direct messages
using appropriate communications at those
groups. For example, the best ways of reaching
consumers who are aged 60+ may differ from the
best ways of reaching consumers under 30 (via
newspaper, retailer, Facebook, Twitter, etc.).

Awareness does not necessarily translate
to recycling. In the August 2010 survey of
homeowners, 10 percent indicated they would
put the leftover paint in the garbage (Bradshaw
Advertising, 2010), but in the July 2011 survey of
those that had painted recently, no respondents
indicated they had disposed of the leftover paint
in the garbage (PaintCare, 2011a). Though it is
possible that the program contributed to this
outcome, it is not possible to attribute this change
to the program because, for instance, there may
be a lag time between recent painting projects
and time of disposal not accounted for in the
survey. A consumer's lack of site awareness does
not automatically translate to not recycling paint.
Consumers with leftover paint may attempt to
locate a site to drop off paint when the need
arises. Although awareness of a drop-off location
in the July 2011 survey was 31 percent, a January
2008 survey found that 67 percent of Oregon
residents had brought items to a drop-off location
or event (Oregon DEQ, 2008). This suggests that
consumers who intend to recycle their paint
may know that drop-off locations are available
and they need easy access to information
about specific locations when they are ready to
recycle; in other words, it may not be important
that a consumer is aware of the program or
is able to name collection sites on demand as
long as she intends to recycle and is able to
access information about where and how to
                                                Promoting Environmental Results Through Evaluation     44

-------
        recycle. Going forward, the Oregon program and
        similar programs may use education campaigns
        to prioritize behavior change (e.g. recycle
        paint rather than store it) and accessibility of
        information over messages of general awareness.

        Outreach products need to be linked to
        a prioritized set of goals. Paint recycling
        programs should prioritize the goals of outreach
        products. For instance, the program should
        determine whether the goal of a particular
        message or product is to increase consumer
        awareness of the program or increase the number
        of consumers returning paint. These two goals
        may be related or mutually exclusive. That is,
        a consumer may find out about the program
        and then decide to  recycle paint or they may
        find a collection location and return paint and
        be unaware of a "PaintCare Program," related
        legislation, or a collection fee. Focusing on
        getting consumers  to drop-off sites maybe more
        important than focusing on program awareness.
        However, using education campaigns to ensure
        consumers purchase the right amount of paint
        (possibly eliminating leftover paint) may be
        more effective at reducing waste paint than, after
purchasing surplus paint, helping consumers
identify drop-off locations. Setting priorities
and focusing outreach on those priorities will
present opportunities to improve effectiveness
and efficiency of marketing campaigns and the
program as a whole.

Outreach should balance the different,
sometimes competing, program goals.  A
balanced approach to setting and prioritizing
education and outreach is key and requires
ongoing measurement and evaluation of the
efficacy of the chosen approach. Putting too much
focus on getting consumers to reduce the amount
of paint purchased may lead to consumers with
leftover paint who do not know what to do with
it, while focusing too heavily on collections may
cause consumers to not worry about the amount
purchased since the drop-off locations provide
an outlet for their unused paint (e.g., "whatever
I don't use, I'll just drop off"). Measuring the
effectiveness of education and outreach materials
and strategies on consumer behaviors in the
context of the paint management system can
provide the information necessary to maintain
balanced progress toward program goals.
45     Evaluation of the Oregon Paint Stewardship Program

-------
6.0   Consumer Purchasing  Decisions
  ^  A *-   >  EVALUATION QUESTION:
            m  How has the program affected consumers' purchasing
               decisions and management of paint prior to drop-off at
               paint recycling facilities?
   How did the fee assessment affect consumer behavior?
   One goal of the program is to help consumers
purchase only the amount of paint they need and
consequently reduce the amount of leftover paint
that must be managed. The pilot program attempts to
improve the ways in which consumers manage their
leftover paint; the pilot program provides consumers
with information on proper storage practices (e.g.,
not allowing leftover paint to freeze), encourages
consumers to bring leftover paint to recycling facilities
(rather than storing it), and educates consumers on
the environmental consequences of improper disposal
of leftover paint. Another key aspect of the program
is the application of a fee to the consumer purchase
price of paint in order to fund the program. Evaluation
Question 4 addresses the program's effect on consumer
behavior related to purchasing and managing paint, and
the extent to which the fee influenced consumer paint
purchasing behavior.


6.1  Data  Sources, Methods

and Analysis

   Evaluation Question 4 relies primarily on the same
two surveys as Evaluation Question 3 (see Section 5);
the August 2010 survey of Oregon homeowners and the
July 2011 survey of recent paint purchasers in Oregon.
Whereas the previous evaluation question focused on
the impact of the program on consumer awareness, this
question focuses on the impact on consumer behavior
(e.g., paint purchasing).
   Additionally, the Evaluation Team drew on the
2008 Oregon Department of Environmental Quality
Household Hazardous Waste Survey (Oregon DEQ,
2008) for baseline contextual information on consumer
behavior related to paint management. The 2008
Oregon DEQ HHW survey was conducted in January
2008 by the Portland State University Survey Research
laboratory; a total of 615 Oregon residents completed
the random statewide telephone survey, answering
questions about use and perception of the dangers of
common household products (e.g., oil-based and latex
paints, pesticides, drain openers).
6.2  Findings
6.2.1  How has the program
affected consumers' purchasing
decisions and management of paint
prior to drop-off at paint recycling
facilities?

Purchasing decisions
   Data from the July 2011 survey indicate that the
program had little impact on the amount of paint
purchased, as 93 percent of respondents indicated that
information about the program had no influence on the
                                       Promoting Environmental Results Through Evaluation    46

-------
amount of paint purchased (see Figure 22) (PaintCare,
2011a).
    Fifteen respondents (out of 84 total respondents) in
the July 2011 survey recalled seeing the paint calculator
and among those 15 respondents, three used the
calculator to decide how much paint to purchase (See
Figure 2 3).


Management of paint prior to drop-off
    In the August 2010 survey, respondents were also
asked whether the existence of the program would
affect the likelihood that they would recycle  (Bradshaw
Advertising, 2010). Results indicated that:

    •  56 percent of respondents reported that they
       are more likely to recycle leftover paint

    •  40 percent reported that the program makes
       "no difference."

    Neither survey nor any of our other data sources
addressed how or where the consumers stored the
paint and how long they stored it prior to drop-off. The
surveys did, however, address what consumers planned
to do or did with paint (Figure 16 on page 38).
6.2.2  How did the fee assessment
affect consumer behavior?

    When asked about the fee assessment in the August
2010 survey (see Figure 24), 33 percent of respondents
indicated the fee assessment was "very reasonable," 40
percent found the fee "somewhat reasonable," and 23
percent indicated that the fee was "not reasonable at
all" (Bradshaw Advertising, 2 010).22
    In the July 2011 survey of recent painters
(PaintCare, 2011a), 11 percent of respondents indicated
that they are aware of the fee. Additionally, when asked
whether the fee affected the amount of paint purchased,
93 percent of respondents said the fee had no impact
(see Figure 25). This survey, however, covered those
that had recently purchased paint. There may be a set
of consumers that chose not to buy paint at the fee-
inflated price, but who would have purchased paint at
the price of paint without the fee added.
  Figure 22 - Consumer-Reported Impact of Program on Amount of Paint Purchased (July 2011 Survey)
       No effect; 64
     Purchased fewer
       containers; 2

    Purchased smaller
       containers; 1
           Other; 2
                           10        20       30       40       50

                                        Number of Respondents
                                                                        60
                                                                                  70
22 The remaining respondents were in the refused/don't know categories.
47     Evaluation of the Oregon Paint Stewardship Program

-------
  Figure 23 - Awareness and Use of Paint Calculator
                                                                                      Aware
                                                                                      Saw paint
                                                                                      calculator
                                                                                      Did not use it

                                                                                      Used it
6.3   Learning
Purchasing Decisions and Paint Management
    •   Program had little impact on consumer
       purchasing. There is no indication that the
       Oregon program has influenced the consumers'
       paint purchasing decisions. If consumers
       continue to purchase the same amount of
       surplus paint, there will be no change in the
       volume per capita eventually destined for the
       paint management system, thus requiring
       expenditure of more resources than if consumers
       purchased amounts of paint that more closely
       matched their needs. Reducing surplus (waste/
       leftover) paint purchased by consumers would
       reduce subsequent resources required for a
       statewide paint management system.

    •   Program focused on recycling, not
       reduction. The outreach materials focused on
       the recycling aspects of the program (i.e., where
       to bring leftover paint) and not on reusing
       leftover paint or reducing the amount of paint
       purchased. The purpose of the paint calculator
       provided at paint retailers is to improve the
accuracy of the amount of paint purchased but
consumers were generally unaware of it and
did not use it, so it was not effective at reducing
volumes of post-consumer paint. If improving
the consumers' ability to purchase the correct
amount of paint is to be considered a viable
option to achieving the program and policy
goal of reducing leftover paint, effort should be
focused on understanding and improving the
effectiveness of the tools being used to achieve
this goal (e.g., paint calculator, retailer training,
consumer assistance).

Waste hierarchy emphasis should be
clearly defined. To achieve goals of reducing
the generation of post-consumer architectural
paint, paint management programs and
policies may explicitly document the relative
emphasis they want to place on aligning the
paint management system with the existing
waste hierarchy. For instance, prioritize options
such as reuse of leftover paint, reducing
leftover paint through improving accuracy of
paint purchases (less leftover) and increasing
volumes of managed post-consumer paint.
Explicit attention to these options will provide
a structure to prioritize outreach strategies,
messages, materials, and effort.
                                               Promoting Environmental Results Through Evaluation     48

-------
  Figure 24 - Consumer Perceptions of the Reasonableness
  of the Fee (August 2010 Survey)
                          Figure 25 - Consumer-Reported Impact of Fee
                          on Amount of Paint Bought
         Very reasonable
         Somewhat
         reasonable
Not reasonable at all
Refused/don't know
100

 90

 80

 70

 60

 50

 40

 30

 20

 10

  0
                                                                   Purchased
                                                                     same
                                                                    amount;
                                                                     93%
                                                                                    Was only asked
                                                                                     of recent paint
                                                                                   purchasers - no data
                                                                                    on how many did
                                                                                   not purchase paint
                                                                                   because of the fee.
                                                                                   Purchased
                                                                                   less; 7%
Impact of fee on consumer purchasing
    •    Oregon residents value this program.
        Ninety-two percent of respondents indicated
        the need for this type of program was
        "somewhat important" or "very important."
        Oregon residents' acceptance of the fee may be
        linked to the inherent value they place on the
        goals and services provided by such programs.
        The degree to which consumers value such
        programs may indicate their willingness to
        pay a certain fee. This information may be
        used in refining fee structures within and
        across states. Understanding and leveraging
        consumer attitudes may also prove useful in
        navigating the legislative process, where, as the
        PPSI learned in Minnesota, without evidence
        of public opinion and providing evidence of
        the public's willingness to pay a fee, legislation
        based on a fee mechanism may be rejected by
        executive or legislative leadership even with
        broad bipartisan legislative support.
                                More details are needed on fee impacts.
                                A full understanding of the impact of the fee
                                would assess whether some consumers were
                                deterred from buying paint due to the higher
                                cost associated with the added fee. The fact
                                that consumers are deterred at the margin may
                                or may not be a good thing. On the one hand,
                                a higher price may reduce the total amount of
                                paint demanded by consumers and affect paint
                                retailers' revenues. On the other hand, reducing
                                the total amount of paint purchased reduces the
                                total amount of leftover paint that would have
                                to be managed by the program in the future.
49     Evaluation of the Oregon Paint Stewardship Program

-------
7.0   Collection of  Post-Consumer
       Paint
  -  -f    ^  EVALUATION QUESTION:
               How has the program affected the collection of post-
               consumer paint in terms of volume, cost, environment,
               convenience,  and infrastructure?
• What other factors have affected the amount of leftover paint?
7.1  Data Sources, Methods
and Analysis

   Program data collected and tracked by PaintCare
serves as the primary source of data for the analysis
of volume, cost, environment, convenience, and
infrastructure. PaintCare reported this information in
its annual report (PaintCare, 2011b). The program data
are supplemented with data and analysis provided by
Hedrick Strickland, who performed a CIS analysis of
collection site locations to assess convenience, as well
as baseline data from the Oregon DEQ, and a set of
interviews with key program stakeholders and retailers.
7.2  Findings
7.2.1  How has the program
affected the collection of post-
consumer paint?

Volume
   In the first year of the program, PaintCare collected
469,665 gallons of paint. Of this total, 352,136 gallons
(75 percent) were latex paint and 117,529 gallons (25
percent) were oil-based paint (see Figure 26). Metro
collected 50.4 percent of the total gallons (PaintCare,
2011b).
   Overall, the program collected approximately 3
percent more pounds of paint than was collected by
Oregon DEQ in 2008 (assuming a volume to weight
conversion of 10 pounds per gallon for paint collected
under the program) (PaintCare, 2011b). However,
PaintCare notes in its report that there may be issues in
comparing the two estimates due to conversion factors.
In addition, PaintCare collected approximately 24,000
gallons of paint in areas where collection was not
previously available.
   PaintCare collected less paint than anticipated in
the projected budget (Figure 27), collecting 31,969
fewer gallons of latex (8 percent) and 47,088 fewer
gallons of oil-based (29 percent) than projected
(PaintCare, 2011b). PaintCare indicated that the
discrepancy in oil-based collection is a result of a
miscalculation in how much oil-based paint was being
collected prior to the program start (S17). Additionally,
PaintCare moved to a weight-based system to more
accurately reflect collected volumes.
   For Metro, the number of households dropping
off paint decreased under the program by two
percent from the previous year and the number of
non-households  (e.g., contractors) dropping off paint
increased significantly. Metro believes the increase in
                                        Promoting Environmental Results Through Evaluation    50

-------
Figure 26 - Volume by Paint Type and Service Area
             Collections by type of paint
    Volume by service provider/area
                                                                                          - Reuse
                                                                                            3.2%
                                Oil
                              117,529
                                25%
               Latex
             352,136
                75%
     Outside
      Metro
      46.4%
                                                                                     Metro
                                                                                     50.4%
              469,665 total gallons collected
       469,665 total gallons collected
Figure 27 - Projected vs. Collected Amounts of Latex and Oil
  450,000

  400,000

  350,000

  300,000

  250,000

  200,000

  150,000

  100,000

   50,000

        0
      164,617
117,529
                                  Latex
                                                                          Oil
                                        Projected
Collected
      Evaluation of the Oregon Paint Stewardship Program

-------
non-household consumers reflects the fact that disposal
of paint is now free for these organizations (Eml),
whereas prior to the program non-households had to
pay to drop-off paint under the program. Metro also
reported that the volume of latex paint collected was
down 9 percent over the previous year. There was an
initial spike in volumes for Metro during the first few
months, but overall the total volumes were down (Eml,
S16). Metro also indicated that they collected a large
number of unlabeled containers of paint, which are
classified as non-program materials and as such were
the responsibility of Metro to handle (Eml, S16).

Costs
    PaintCare reported that the cost of managing
paint collected under the program is $7.03 per gallon
(PaintCare Annual Report, 2011b). This metric does
not include collection costs, such as labor and storage
space, incurred by collection sites. In an Oregon paint
recycling pilot program conducted in 1997 to 1998,
Oregon DEQ-sponsored collection events were reported
to cost $13.86 per gallon (inflated to 2011 dollars) and
the pilot program cost $11.32 per gallon, during which
collection was done at retailers' sites (Cascadia, 1998).
Without more detailed analysis, these estimates are not
appropriate for use as baseline, but may indicate some
cost savings relative to the 1997 to 1998 program design.

Environment
    The PPSI Lifecycle Workgroup (2005 to 2009)
consisted of representatives from government and
industry. ACA funded contractors to conduct a lifecycle
assessment (LCA) of six options for managing leftover
paint (e.g., dry and dispose at home to centrally
collecting and processing into recycled paint) (PSI,
2009).  LCA is a tool for the systematic evaluation of
the environmental impacts of a product through all
stages of its life cycle, which include extraction of
raw materials, manufacturing,  transport and use of
products, and end-of-life management (e.g., reuse,
recycling and/or disposal). The PPSI LCA scenarios
were developed prior to the enactment of any
legislation and therefore not based on the Oregon pilot
(PaintCare, 2011b).
    Initial LCA modeling results were found to
depend crucially on the amount of virgin paint that
consumers replaced with recycled paint (PSI, 2009). If
consumers were assumed to replace all virgin paint with
recycled paint, then recycling proved to be the more
environmentally beneficial outcome. However, if no
virgin paint was assumed to be replaced, then producing
recycled paint was found to have a larger environmental
impact than just having consumers dry out and dispose
of the latex paint in their household trash. Agreement
between industry and government representatives on
what percentage of virgin paint is avoided could not
be reached and the workgroup disbanded. Without
continued funding, the planned sensitivity analysis to
identify other assumptions and parameters that had a
major impact on the modeled results was not completed.
    ACA's conclusion as reported in the Annual
Report (PaintCare, 2011b) was "[t]he [LCA] results
demonstrate that unless and until recycled content
paint is marketable at the same or similar rates as
virgin paints...the environmentaL.cost [and] benefits or
recycling paint do not outweigh drying and disposing
of such." The draft LCA is available as Appendix J of
PaintCare's report. Some members of the workgroup
had anticipated that the Oregon pilot could provide real
world data to refine the LCA model to complete PPSI's
goal of better understanding the societal impacts of
latex paint management approaches (PSI, 2 009). The
modeling results could help with the planning and
selection of appropriate paint management methods in
light of environmental benefits weighed against costs.

Convenience and Infrastructure
    As of September 1,2011, the program had 98
collection sites with 10 of these sites offering paint
exchange. Sites are open to the public, on average,
six days a week for a total of 58 hours per week
(PaintCare, 2011b; Em2). Figures 28 and 29 provide
maps of the pre-program and June 30,2011 drop-off
sites, respectively (PaintCare, 2011b).23 The increased
number of sites can be seen in  comparing the two
figures. Table 4 highlights the increased amounts of
the Oregon population living within  a 15 mile radius
or within 15-mile drive of a drop-off location. The
increased number of drop-off sites increased the
number of Oregon residents living within a 15 mile
radius of a drop-off location by 38.5  percent (PaintCare,
2011b). Additionally, based on a CIS analysis using
June 2011 sites, 92 percent of Oregon residents live
23 Both Figure 28 and Figure 29 were taken from the PaintCare's annual report (PaintCare, 2011 b).
                                                Promoting Environmental Results Through Evaluation     52

-------
Table 4 - Proportions of Oregon Population Living within 15 Mile Radius and 15 Mile Drive of Collection Sites
 Time Period
Percent of Oregon
residents living in            Percent of all Oregon
incorporated cities,          residents living within
towns, and CDPs who are   15 mile radius
within a 15 mile radius      of collection site
of a collection site  [a]        (PaintCare, 2011)
Percent of all Oregon
residents living
within a 15 mile drive
of a collection site
(Strickland, 2011a,b)
(Pa
Pre-program
(date unknown)
January 2011
June 2011 [b]
Change between pre-
program and June 201 1
intnarp 2011}

69.2%
--
95.9%
38.5%


51.2%
--
70.9%
38.5%


91%
92%

[a] GDP = Census Designated Places
[b] The analysis performed for the PaintCare report on the percent within the 15-mile radius and the analysis performed by Strickland are seemingly
contradictory. Specifically, the percentage within a 15 mile radius should exceed the percentage within a 15-mile drive. The Evaluation Team provided
Strickland with the set of sites listed as being the June 30, 2011 sites. There may be some analytical details that account for some of the discrepancy. For
example, Strickland's analysis uses the percentage of roads in a Census block group that are within the 15-mile drive as a proxy for the percentage of the
population within that Census block group. Also, the most recent block level census data at the time of Strickland's analysis was from the year 2000, whereas
the most recent data at the time of PaintCare's analysis was from 2010. This could also account for the discrepancy.
  Figure 28 - Pre-Program Drop-Off Sites
53      Evaluation of the Oregon Paint Stewardship Program

-------
Figure 29 - Drop-Off Sites on June 30, 2011.
           Ugtnd
              \ SW.HMW lacing
              | ReUUora
             O Loe*l B«Vt rvtmt  ,


Figure 30 - Percent of Residents Within a 15 Mile Drive by County.
                                                                    IMDU OP«?O- Sunndi Umtxn Fin M
                                                     Promoting Environmental  Results Through Evaluation      54

-------
  Figure 31 - Change in Collection Service Availability
                                     Pre-program
                     June 2011
                   HHWs offering
                     collection
Counties served
 by events only
Counties with
 no service
 Retailers serving
as collection sites
within a 15-mile drive of a drop-off location (Strickland,
2011). Figure 30 presents the mapped results of the CIS
analysis showing the percentage of residents within a
15 mile drive of drop-off locations for each county in
Oregon.
    Figure 31 compares the pre-program network of
collection sites to the June 2011 network based on data
presented in the PaintCare annual report (PaintCare,
2011b).  The data in the figure show a substantial
increase in the availability of collection services.
    Additionally, PaintCare provided 19 large volume
direct pickups to institutional, commercial, and
industrial entities between July 1,2010 and June 30,
2011, collecting approximately 4,050 gallons of paint
(PaintCare, 2011b). Finally, 63 percent of consumers
in July 2011 who purchased paint recently think their
nearest collection site is convenient (PaintCare, 2011a).


7.2.2  What other factors have
affected the amount of leftover
paint?

    Factoring into baseline considerations is the fact
that the quantities of latex paint collected at DEQ events
fell significantly after consumers were discouraged
from bringing latex paint to collection events starting
in 2008 (Oregon DEQ, 2011). To identify the factors
               that have affected the amounts of leftover paint,
               information on consumer handling of leftover paint
               is needed. Although some information is available on
               what consumers intend to do with the leftover paint,
               information on non-program factors was not assessed
               under this project.


               7.2.3  How has the program
               affected transportation of paint
               from collection sites  to other
               facilities in terms of volume,
               environment, and cost?

                   The transportation contractor (PSC) is minimizing
               transportation costs by designing "milk-run"
               transportation routes with software designed by Red
               Prairie (Em3). The trucks service PaintCare locations as
               well as other clients that fall along the transportation
               route. PSC reported that transportation costs are
               similar pre- and post-program since the primary cost
               driver is diesel fuel price (Em3).  However, without
               collection of Oregon pilot program data (before and
               after the program start) the Evaluation Team was not
               able to make an assessment on changes of transported
               volumes or the environmental impacts.
55    Evaluation of the Oregon Paint Stewardship Program

-------
7.2.4  How has the program
affected retailers' behavior?

    Retailer interviewees note that the program had
little effect on their practices in terms of marketing or
sales (S7, S8, and S9). However, key challenges included:
    •   Programming the fee assessment into pricing
       and invoicing computer systems (S8).
    •   Preparing staff for program rollout (S7, S8).
       The challenge for one larger retailer was to
       make sure staff are aware of and understand
       the program, putting materials out, and are
       able to answer customer  questions. Time
       for internal outreach and education and the
       magnitude of time/resources needed are
       related to the number of employees.
7.3   Learning
       Reporting on amounts of paint should
       be standardized. PaintCare's preliminary
       reporting used paint container fullness
       assumptions for calculating volume of
       paint collected. This assumption proved to
       underestimate gallons collected, and PaintCare
       found that a weight-based measurement was
       more accurate and more common for HHW
       reporting to Oregon DEQ (S17).

       Contractors are a stakeholder in the
       program. The amount of waste paint
       generated by contractors needs clarification.
       One interviewee stated that contractors should
       not be subject to the recovery fee because
       they don't generate waste paint (Bledsoe et al.,
       2011). Meanwhile, Metro saw an increase in
       the number of non-households dropping off
       paint because under the program no fees were
       collected for drop-off (Eml). Contractors were
       not a stakeholder group participating in PPSI
       and the amount of leftover paint they generate
       was not quantified in EPA's report Quantifying
       the Disposal of Post-Consumer Architectural
       Paint (Abt, 2007). PPSI could further explore
if contractors are a significant contributor of
paint to the management system and therefore
need different and more audience-specific
messaging about reducing waste paint.

Unlabeled paint containers are an
issue. The Evaluation Team is uncertain of
the number of unlabeled paint containers
that consumers are attempting to turn in.
Consumers know the unlabeled container has
paint, but collection sites are instructed to treat
unlabeled paint containers as a non-program
product and to not accept them. If a significant
volume of leftover paint is being excluded
from the program, then that paint is essentially
being managed as it was prior to the program -
as a waste product; rather than being recycled
or reused.

More definitive information is needed
to provide an accurate assessment of
the cost per gallon. The cost of $7.03 per
gallon can be used as a starting point for
other states. However, the reported PaintCare
system costs do not reflect the true costs of a
leftover paint management system. PPSI's Paint
Product Stewardship Initiative Infrastructure
Project (the "infrastructure report"), which
modeled a comprehensive system to manage
leftover paint, stated that the collection step
in the process costs $1.98 per gallon (SCS
and Cascadia, 2007)—however, this is not
an apples to apples comparison (e.g., the
infrastructure report included transportation
to aggregate facility as part of the collection
step). Efforts are underway in Washington
State to estimate local government costs
associated with collection of waste paint, which
includes a methodology and development
of a tool for local governments to collect
data (Em5). Furthermore, as noted under
Evaluation Question 4, the Oregon program
had a relatively well-established paint recycling
infrastructure in the Portland metro area prior
to this pilot program. Thus, program costs per
gallon for other states may depend on the level
of existing infrastructure.
                                               Promoting Environmental Results Through Evaluation     56

-------
        Further discussion of reimbursement
        of collection costs is needed. Non-
        reimbursement of collection costs continues
        to be a controversial issue among stakeholders
        (S18, S20). This non-reimbursement was
        a topic that PSI had anticipated further
        collaborative discussion on as part of detailed
        work plan for the implementation stage. (S19)

        Additional LCA work is needed. An
        uncompleted task of the lifecycle workgroup
        was to conduct a sensitivity analysis for
        the LCA which could have identified other
        assumptions with large environmental effects.
        For example, the initial results indicated that
        the virgin offset made the largest impact,
        but transportation could be another factor.
        For example, Metro's less desirable recycled
        paint colors are being sent to Asia (for a fee,
paid by Metro (Em4)). Does that change the
environmental benefits of a recycled paint
management system if paint is shipped across
the globe? With the key assumptions and
parameter data identified, future evaluations
and ongoing examination of program
performance information could collect
groundtruthing data (e.g., virgin paint offset;
miles traveled) to improve the ability to model
a paint management system. The LCA is a tool
to estimate a wide range of environmental and
social impacts, and interpreters of its results
will have to determine which impacts (e.g.,
greenhouse gases, water usage, air toxics)
are the most important when  selecting a
management approach. LCA can be integral
to future program design and  planning, and
requires funding for complete development
and interpretation of the model.
                  'ak
57     Evaluation of the Oregon Paint Stewardship Program

-------
8.0   Paint Reprocessing,  Recycling
       and Energy Recovery
  -  - *-   ^  EVALUATION QUESTION:
               How has the program affected used paint reprocessing,
               paint recycling, and paint-related energy recovery in
               terms of volume, infrastructure, and cost?
   An important aspect of assessing product
stewardship outcomes is understanding what happens
to the materials that are collected under the program.
The program plan sets forth the following management
options for collected leftover paint:

   •  Paint exchange: In order to increase the
      quantity of leftover paint that is reused,
      program collection sites and other locations
      can voluntarily offer a "paint exchange" to their
      customers. Under this system, better quality
      containers of paint are placed on display
      shelving and available at no cost to another
      consumer. As discussed in Evaluation Question
      9, the exchange management option is an
      efficient way to achieve reuse as the paint does
      not require transportation and reprocessing.

   •  Recycling as paint: Good quality leftover
      latex paint may be bulked with similar colors
      then filtered and placed in smaller containers
      for resale to consumers.

   •  Recycling as another product: In the
      Oregon program, some unusable latex paint
      was sent to Amazon Environmental Inc. to be
      converted to PLP, a cement additive. Amazon
      also used some "bad" latex to bind biomass
      used to fuel its facility (referred to as Amazon
      biomass in this report).

   •  Energy recovery: Oil-based paint is blended
      with other oil-based solvents to be burned as
      fuel for some operations, such as incinerators.
   •  Appropriate disposal: Metro disposes of
      latex paint that is not of suitable quality for
      recycling via landfill biodegradation. Metro's
      non-saleable paint and washwater from
      the recycling process is transported to the
      Columbia Ridge Landfill in Arlington, Oregon.
      Under a special Research, Development and
      Demonstration permit at the landfill, liquids—
      including the paint-washwater mixture—are
      intentionally circulated throughout the solid
      waste to expedite degradation.
   Table 5 summarizes these methods in order of
preference for latex paint, oil-based paint, and paint
containers. The purpose of Evaluation Question 6 is to
track the volume, infrastructure, and cost associated
with each management method.


8.1  Data Sources, Methods

and Analysis

   The primary data source for answering Evaluation
Question 6 is the data collected by the program and
summarized in the PaintCare Annual Report (PaintCare,
2011b). The Evaluation Team supplemented these data
with key stakeholder interviews to provide details on
the numbers being reported.
                                       Promoting Environmental Results Through Evaluation    58

-------
Table 5 - Hierarchy of Management Options for Post-Consumer Paint Products
 • Paint exchange/reuse
 • Recycling as paint
 • Recycling as another product
 • Appropriate disposal
                                     Oil-Based Paint
• Paint reuse
• Energy recovery
                              Paint Containers
• Recycling
• Appropriate landfill
8.2  Findings
    Table 6 shows the disposition of latex and oil-based paint collected under the program. These data are also
summarized in Figure 32 (latex) and Figure 33 (oil-based). In what follows, we describe the data presented in Table i
            Table 6 - Disposition of Paint by Management Method
             Paint and
             Catego
      ercent of Total Paint
           Collected [:
            Latex Paint
                Reprocessing
                   Amazon PLP
                Recycling/Reuse
                   Metro recycling
                   Amazon recycling
                   Consumer reuse
                Energy Recovery
                   Amazon biomass
                Disposal
                   Metro biodegradation
             >il-Based Paint
                Recycling/Reuse
                   Consumer reuse
                Energy Recovery
                   Amazon fuel blending
               8%
              53%
               4%
               3%
               4%
              28%
               3%
              97%
       28,171
      186,632
       14,085
       10,564
       14,085
       98,598
       3,526
      114,003
[a] For latex this is the percentage of total latex paint collected (352,136 gallons) and for oil-based is the percentage of oil-based paint collected (117,529 gallons).
59    Evaluation of the Oregon Paint Stewardship Program

-------
Reprocessing
Disposal
    Eight percent of latex paint was processed into
Amazon PLP, a cement additive (PaintCare, 2011b).
However, information on costs for reprocessing was not
available (PaintCare, 2011b).

Recycling/Reuse
    Fifty-seven percent of latex paint was recycled as
paint by Metro and Amazon (PaintCare, 2011b).
    Metro's peak volume of paint processed
into recycled product increased after program
implementation from approximately 280,000 to
325,000 gallons (S4). Metro has additional processing
capacity available (S4).
    Three percent of latex paint and 5 percent of oil-
based paint was reused by consumers (PaintCare, 2011b).
    PaintCare provided collection locations an
incentive of $0.2 5 per "reuse" gallon to reflect avoided
transportation and processing costs (PaintCare, 2011b).
An organizational representative notes that no sites
took advantage of this incentive; sites that were already
offering reuse had systems in place and considered it an
important service (S17). However, less than 10 percent
of collection locations have shelves for paint exchange;
PaintCare did not recruit retail sites for paint reuse due
to concerns about liability (S17).24
    Finally, approximately 47 tons of plastic paint pails
and 65 tons of metal cans were recycled (PaintCare
2011b). Assuming that each plastic pail weighed about
1.6  pounds, and each metal can weighed about 1.4
pounds, then this tonnage translates to approximately
93 thousand metal paint cans and 59 thousand plastic
pails. Data on the total number of containers collected
is not available so there is no available estimate on the
percent of collected containers that were recycled.

Energy Recovery
    Almost all (97 percent) of the oil-based paint
collected under the program was bulked for energy
recovery (fuel blending) (PaintCare, 2011b). Four
percent of latex paint went to Amazon Biomass; the
paint is used to bind materials with high British thermal
unit (BTU) value (e.g., sawdust) for energy recovery
(PaintCare, 2011b)
    Twenty-eight percent of latex paint was processed
for biodegradation; the paint is mixed with wastewater
and injected into a landfill in order to improve
degradation of landfilled wastes and increase gas
recovery (PaintCare, 2011b).

  Figure 32 - Destination of Collected Latex Paint
       Metro recycling
       Metro biodegradation
       Paint reuse at
       Restores
  Figure 33 - Destination of
  Collected Oil-Based Paint
       Fuel
       blending

       Paint
       reuse
Amazon recycling
Amazon PLP
Amazon biomass
(fuel blending)
             3%
24 Liability concerns can take several different forms—environmental, health and safety, and product. For example, spilled paint from a consumer wanting to
  open a can to confirm color or the old paint product doesn't perform as expected. These liabilities can be mitigated. See PPSI's Guidance Manual for Paint
  Reuse Programs available at www.paint.org/component/docman/cat_view/47-education-a-safety.html.
                                                  Promoting Environmental Results Through Evaluation     60

-------
8.3   Learning
        Existing infrastructure was important for
        determining the destination of collected
        paint. The presence of existing latex paint
        recyclers with available processing capacity
        allowed recycling of paint to be the dominant
        method of processing leftover latex paint.

        There are barriers to increasing reuse.
        Recycling is lower on the waste hierarchy than
        reuse, and recycling results in higher costs
        (management, environment) compared to
        reuse. More information is needed to better
        understand the barriers (e.g., liability and
        space) to retailer facilitation of paint reuse.
        One option to increasing reuse, suggested by
        Oregon DEQ (S20), is for PaintCare to
        coordinate with the Restores and local
government to redistribute reusable paint
collected from the collection sites. This option,
however, also creates costs and environmental
effects (e.g., additional transportation and
labor).

Retailer liability may limit opportunities for
reuse. Legislation that directly addresses reuse
and liability is another option for clarifying
program priorities and management. PaintCare
incurred $2.4 million in transportation and
processing costs to collect 469,665 gallons of
paint, or $5.13 per gallon—a value 20 times
more than the reuse incentive offered ($0.25
per gallon)—indicating a potential cost savings
in shifting more paint to reuse and room to
increase the reuse incentive.
        Evaluation of the Oregon Paint Stewardship Program

-------
9.0   Household  Hazardous Waste
        Programs
  -  -f     ^   EVALUATION QUESTION:
                What was the impact of the program on the HHW
                facilities in terms of the types of paint collected,
                costs, and the way in which the facilities operate?
   HHW facilities are a key stakeholder in the
evaluation and an important part of the infrastructure
of the statewide post-consumer paint management
system. The Oregon DEQ solid waste program
website lists event and facility information for 17
HHW programs in the state. These programs accept
common household hazardous wastes (e.g., oil-based
paint, pesticides, and mercury thermometers) from
households, as well as conditionally exempt small
quantity generators (often by appointment or special
event) and then sort the various products for recycling,
reuse, energy recovery, or proper disposal. HHW
programs use a variety of mechanisms in order to
provide services to residents, such as:

   •  Rural county local government solid waste
      or public health programs may offer periodic
      collection events.

   •  The Tri-County Hazardous Waste and Recycling
      program coordinates collection facilities and
      events for Hood River,  Sherman, and Wasco
      counties.

   •  Several counties hire companies that specialize
      in hauling, disposal, and recycling to provide
      HHW collection opportunities. For example,
      Jackson and Josephine counties offer biannual
      events through Rogue  Disposal.

   The number of customers served by a program
varies. One HHW interviewee noted that they serve
85 - 140 cars at each monthly event (S12). Another
interviewee reported that, in 2009 and 2010, their
permanent facility served approximately 4,000
customers (S10).


9.1   Data Sources,  Methods

and Analysis

   To answer Evaluation Question 7, the Evaluation
Team conducted phone interviews with seven
representatives from county HHW programs in Oregon
in June 2011 and data from the PaintCare annual report.
The Evaluation Team supplemented the interview data
with a review of materials related to the impact of the
program on HHW such as PowerPoint presentations
from HHW stakeholder meetings and other documents
provided by interviewees. HHW programs have limited
detailed quantitative data on paint volumes and costs.
   There are several reasons why there is a limited
amount of quantitative data on HHWs. First, HHW
programs operate under lean budgets and do not
necessarily have the resources nor the expertise to track
detailed paint volumes or cost information. Second,
Federal agencies and their contractors are subject to the
Office of Management and Budget (OMB) Paperwork
Reduction Act requirements for information collection
and as a result EPA was limited to interviewing nine or
fewer individuals on this topic (i.e., a detailed survey of
HHWs was not feasible).
                                         Promoting Environmental Results Through Evaluation    62

-------
9.2  Findings
9.2.1  Types of Paint Collected
at Household Hazardous Waste
Facilities

   Table 7 shows a data summary of the PaintCare
annual report on the gallons of paint collected at
HHWs under the program. Interviewees noted that
they had not observed a significant change in the total
gallons of latex or oil-based paint collected since the
PaintCare program began; however, two interviewees
reported that the first event or month following the
program start collected very high volumes that then
tapered off through subsequent events (S4, S10).
In most cases, the interviewees reported that they
were able to accommodate the higher volumes with a
few adjustments. However, one HHW faced delays in
finalizing the contract with PaintCare and needed to
store collected paint until the contract was in place and
the paint could be picked up (S14). One interviewee
noted a shift towards collecting a higher proportion of
latex than oil-based paint (S14). Another interviewee
observed increased traffic from paint contractors
because the county charged a fee for large volumes
prior to the program (S10). After the program began,
one HHW site ceased collecting paint entirely (S15)
while another program began collecting latex paint
again after a long hiatus (S12).


9.2.2  Changes in the  amounts of or
capacities for other products

   Interviewees reported no significant changes in the
amounts of other HHW products and moderate shifts in
available capacity for processing other HHW products.
Three HHWs saw an increased capacity to handle
other products because they were able to reassign staff
from crushing paint cans and bulking the material for
disposal to other tasks (S4,S10, and Sll); one HHW site
stopped collecting paint (S15).
9.2.3 Costs for household
hazardous waste facilities to take
in and process paint

    Interviewees noted that HHW programs do not
have sophisticated systems in place for detailed cost
tracking; staff may simply compare monthly internal
costs (e.g., labor, overhead) to external costs from
vendor invoices (e.g., hazardous waste hauler) (S12).
Additionally, these costs are not likely to be tracked by
task or material type, such as paint processing versus
other hazardous materials. As a result, the interviewees
were not able to provide detailed cost data, but many
were able to provide estimates of the changes in costs
following implementation of the program.
    Five of seven of the interviewees reported
significant cost savings resulting from the program even
when accounting for increases in customer traffic and
material volume. The cost savings are reported to result
from the avoided labor, disposal, and transportation
Table 7 - Gallons of Paint Collected at Different HHW Facilities
between July 2010 and June 2011
 Facility
Gallons
Collected
 Columbia County
4,140
 Beaver Hill Disposal Site
675
 Crook County Solid Waste
188
 Deschutes County
20,375
 Tri-County Hood River
3,150
 Marion County
2,970
 Tri-County - The Dalles
1,890
 Lane County
12,529
 Lincoln Cty SW Transfer
 Station
720
 Capital Plant
585
Source: PaintCare, 2011 b, Appendix D.
Note: Metro's value was not broken out by HHWs versus other facilities.
Additionally, the values do not include collection events. HHWs not listed had
no value listed (i.e., a "-") in Appendix D of the PaintCare report.
63    Evaluation of the Oregon Paint Stewardship Program

-------
costs associated with handling oil-based and unusable
latex paints. Some examples of avoided costs from
interviewees included:

    •   Saving $1,500 per month of a $100,000 annual
       budget due to avoided paint disposal costs
       (S12).

    •   Cumulative savings of $45,000 to $50,000
       resulting from less paint handling by staff and
       avoided disposal, landfill, and transportation
       costs (S10).

    •   A cost decrease of 40 percent from pre-
       program costs due to avoided disposal costs
       associated with oil-based paints (S15).

    •   Avoided costs of $75,000 to $100,000 per year
       of an annual budget of approximately $300,000
       due to  avoided costs of disposal and transport
       of paint (Sll).

    A separate  cost reported by three interviewees
involved the need for HHW staff to conduct outreach
and education before and during paint collection
(S13, S14, and S15). These interviewees felt that the
efforts by the program did not achieve sufficient reach
within their county. One interviewee supplemented
the program efforts with additional, locally targeted
promotion around the beginning of the program
(S15). Of particular interest to these interviewees
was providing additional information to customers
on the location  of the nearest collection site, the types
of materials that would be accepted, and the purpose
of the fee. Two interviewees reported that some
customers mistook the fee for a bottle deposit and
returned empty cans in the hopes of getting the fee
returned (S12, S15). Two other interviewees reported
that customers were bringing in materials that did not
fit the criteria for program products; for example, some
customers were bringing in materials, such as asphalt
sealant, that are applied with a brush (S13, S12).
9.2.4  Cost of managing other
products

    Data on the costs of managing other HHW products
is limited for the reasons described at the beginning of
Section 9.2.3. Two of the interviewees commented that
the costs of handling non-program materials returned
by customers could be significant. When asked about
the challenges associated with rollout of the program,
two interviewees noted that correctly identifying
program materials and non-program materials was
challenging at first, and that the volume of unacceptable
items, such as unlabeled or rusty cans, could be
significant (S4, S13).
9.3   Learning
       Analysis of prior opportunities and
       conditions can help predict how HHW
       volumes may be affected by a paint
       recycling program. Change in volume and
       types of paint collected at HHW sites will vary
       depending on the convenience and availability
       of prior opportunities for consumers to return
       paint. Some HHW programs stopped collecting
       latex paint prior to program rollout due to
       resource constraints and so experienced a
       spike in paint volume when the program began.
       Paint—latex in particular—has historically
       been a significant percentage of the volume of
       material collected by HHW programs. Shifting
       processing of the paint to another entity frees
       up resources for the HHW to handle other
       materials. However, if the HHW is located in
       a state with limited infrastructure for paint
       collection, the HHW might experience the
       opposite effect:  capacity to  process other
       products may decrease significantly as
       customers bring in a relatively high volume
                                               Promoting Environmental Results Through Evaluation     64

-------
       of legacy paint (especially if dropped off
       with other materials at the same time). A CIS
       analysis could assist HHWs to anticipate the
       increased volume of paint under a recycling
       program. An effective version of this type of
       analysis would include paint purchasing data
       related to demographics. For example, CIS can
       be used to map and analyze the demographics
       of an area (age, homeownership, etc.) in
       the context of paint purchasing behavior.
       Combining CIS with data on demographics
       with paint sales in the area could identify the
       potential quantity of paint that an HHW could
       expect.

       Labor and oil-based paint disposal costs
       are the key cost elements for HHW. States
       with limited pre-existing infrastructure for
       paint collection may experience initial spikes
       in labor needs at HHWs as customers bring
       in relatively high volumes of legacy paint;
       however, the program should mitigate the costs
       of sorting, bulking, and disposal. Areas with
       well-established infrastructure and steady
       paint volume collected could see significant
       cost savings.
Cost information is lacking for HHWs.
HHWs represent a significant set of program
stakeholders. Without knowing the costs
incurred to process paint at HHWs and the
cost avoided from paint recycled, the full cost
effectiveness of the program cannot be judged.

Education and outreach for HHWs to use
with their customer base may be needed.
Development of education and outreach
campaigns for future roll-out states should
include HHWs to  ensure the inclusion of
complete and relevant information for HHWs.

Consideration should be given to non-
program materials and how to deal with
them. Collecting non-program materials
imposes costs on HHWs and quantities may
be significant. HHWs may need additional
support from the program to ensure that these
materials are correctly identified and sorted
separately from program materials. States with
limited prior infrastructure should anticipate
collecting older, deteriorated (e.g., "rusty"), and
unlabeled cans of paint and be prepared to sort
them from program paint.
65    Evaluation of the Oregon Paint Stewardship Program

-------
io.o   Cost Effectiveness
                 EVALUATION QUESTION:
                 How cost effective is the program?
10.1   Data Sources, Methods

and Analysis

    Cost-effectiveness can be assessed by comparing an
amount reflecting an outcome (e.g., paint collected) to
the costs incurred to achieve that outcome. To answer
this question, the Evaluation Team intended to analyze
cost data collected to answer Evaluation Questions 5, 6,
and 7 at a level that would have allowed for comparison
with the  PPSI's estimated cost of $7.46 per gallon
collected, as found in Paint Product Stewardship Initiative
Infrastructure Project of a comprehensive system to
manage leftover paint (SCS and Cascadia, 2007).
    The PaintCare Annual Report (PaintCare, 2011b)
provided cost information, amounts of collected paint
and cost per gallon collected information. However,
since PaintCare's budget does not reimburse for
collection costs, the program's budget does not
include the full system costs. Additionally, much of
the initial year's costs of PaintCare included higher
level of administrative costs due to efforts to educate
consumers and retailers (both about the fee and those
collecting) and other set-up costs associated with a
new organization and setting up an infrastructure.
Further, legacy paint entering the system will
represent significant volume in the program's early
years. Evaluation Question 8 may be more thoroughly
answered with several years of operating expenses
available.
10.2  Findings
   Actual PaintCare program costs for the first year
were $7.03 per collected gallon, excluding collection
costs (PaintCare, 2011b). PaintCare's initial projection
of program costs—which excluded collection costs at
permanent facilities—was estimated to be $8.26 per
collected gallon in the first year with a decreasing cost
of $7.30/gallon in year two, $7.27/gallon in year three,
and $7.18/gallon in year four (PaintCare, 2010). The
primary reasons for the lower costs compared to pre-
program include that a higher percentage of latex paint
was collected as compared to oil-based (latex paint
is more expensive to handle). In addition, less non-
program materials were collected, so overall disposal
costs for these items were less than expected (Em2).
Despite the overall lower than expected costs, some
initial year expenses were higher than expected. For
example, start-up costs (e.g., legal and banking fees)
and cost for education and outreach efforts were higher
than expected (S17).
   Figure 34 presents the cost per gallon from the
Oregon program alongside other estimated costs per
gallon to process leftover paint from other programs
and analyses. Table 8 provides details on the different
cost per gallon estimates. Although the $7.03 per gallon
estimate is lower than other estimates, the PaintCare
estimate excludes some aspects (labor for collection)
that are included in other, higher estimates.
   PPSI's life cycle workgroup was scoped to have
a full cost-benefit analysis conducted on the same
management scenarios as were modeled in the LCA
(PSI, 2009). Such an effort would have examined cost
effectiveness from a social perspective, accounting
                                             Promoting Environmental Results Through Evaluation    66

-------
for additional costs such as consumer time to drop off
paint and externalities associated with environmental
impacts (e.g. air emissions, greenhouse gases, and
water use).
10.3   Learning
        The exclusion of HHW collection costs
        from the total program costs is significant.
        Even though it appears that costs decreased
        for HHWs, gauging total program cost
        effectiveness is not possible without knowing
        collection costs for HHWs. Calculating a value
        for collection costs is complex. Some variables
        such as estimating the cost of labor are
        relatively straightforward, while other items
        such as valuing the loss in building space that
        is used for storing waste paint and the percent
        of overhead expenses (e.g., insurance and
        administrative) that should be attributed to the
        collection are more challenging to estimate.

        Expand cost-effectiveness to include
        environmental benefits. Product stewardship
        initiatives can broaden the scope of cost-
        effectiveness by factoring in environmental
benefits. The cost-effectiveness measured
here is the cost per gallon collected. A more
comprehensive measure would translate the
gallons into environmental benefits and also
include other management options (e.g., reuse)
as well as the cost-effectiveness of reducing
leftover paint by convincing consumers to "buy
the right amount." Completion of a full life cycle
cost-benefit analysis could supply some of this
information. The key is to develop a measure
of the environmental benefit associated with
different paint management options.

Expand cost-effectiveness to include
program development. Beyond expanding
cost effectiveness considerations to
environmental benefits, product stewardship
initiatives can consider the cost-effectiveness
associated with program development,
including the costs of collaboration.
Collaboration for this program involved
significant time and effort on the part of several
stakeholders. However, what did collaboration
achieve relative to a less collaborative
approach? Addressing this question would
allow for a more complete assessment of this
type of program and for developing programs
through a collaborative approach.
  Figure 34 - Costs per Gallon Collected from Oregon Program and Other Programs or Studies

      $15.00
      $12.00
       $9.00
       $6.00
       $3.00
       $0.00
                         1997-1998 Pilot,
                         collection events
                         ($2011)

                         1997-1998 Pilot,
                         retailers ($2011)

                         Paint
                         Infrastructure
                         project estimate

                         Projected by
                         program

                         Actual
67     Evaluation of the Oregon Paint Stewardship Program

-------
Table 8 - Details on Different Cost per Gallon Estimates
 Data Source
Amount[a]      Cost Elements Included
                                    Cost Elements
                                    Excluded
 PaintCare - actual
 program costs
 (2010 to 2011)
$7.03
($2011)
• Education and outreach
• Materials for collection (e.g., bins)
• Transportation from collection site
• Processing, recycling, and
  disposal
• Labor at collection events
• Labor at permanent
  collection sites
• Indirect costs
  (e.g., storage space)
• Social and environmental
  costs (e.g., air pollution)
 Paint Smart Pilot
 Program
 (Cascadia, 1998)
Special
collection
events: $13.86
($2011)
• Materials for collection (e.g., bins)
• Transportation from collection site
• Processing, recycling, and
  disposal
• Labor at collection events
• Education and outreach
• Labor at permanent
  collection sites is not
  applicable.
• Indirect costs
  (e.g., storage space)
• Societal costs
  (e.g., air pollution)
 Paint Smart Pilot
 Program
 (Cascadia, 1998)
Retailer          • Materials for collection (e.g., bins)
collection:        • Transportation from collection site
$12.24           • Processing, recycling, and
($2011)           disposal
                                    • Education and outreach
                                    • Collection labor
                                    • Indirect costs
                                      (e.g., storage space)
                                    • Societal costs
                                      (e.g., air pollution)
 PPSI Infrastructure
 Report (SCS and
 Cascadia, 2007)
Model of
national
infrastructure:
$8.15 ($2011)
• Education and outreach
• Materials for collection (e.g., bins)
• Transportation from collection site
• Processing, recycling, and
  disposal
• Labor at collection events &
  permanent collection sites
• Indirect costs
  (e.g., storage space)
• Societal costs
  (e.g., air pollution)
[a] Cost estimates inflated to 2011 dollars using U.S. Bureau of Labor Statistics' Inflation Calculator. Available at www.bls.gov/data/inflation_calculator.htm.
                                                Promoting Environmental Results Through Evaluation     68

-------
11.0  Waste  Hei
                                                   (to be completed Jan 201 2)
  ^ . L      EVALUATION QUESTION:
             *
                How was the program designed and implemented to
                move consumers up the waste hierarchy?
           i
   With  respect to moving customers up the waste hierarchy, what were the
   program's obstacles, opportunities, and decisions?
   A "waste management hierarchy" classifies waste
management options according to their desirability.
PPSI considers reducing leftover paint to be the
preferred management option, followed by reuse,
recycling (including energy recovery), and disposal. A
key goal of the program is to encourage greater reliance
on the most preferred strategies in the hierarchy and
less reliance on least preferred strategies (i.e., "moving
up" the hierarchy). This question assesses how well the
program has achieved that goal.
   Many obstacles stand in the way of waste reduction.
Thus, Evaluation Question 9 also considers the
obstacles, opportunities, and decision-making related
to moving up the hierarchy. Oregon and other states
can use this information to design paint stewardship
programs that more efficiently encourage reliance on
the most preferred management options.
   The Evaluation Committee has partnered with
Brielle Kissell, a graduate student at Tufts University, to
investigate this question as part of her master's thesis.
Ms. Kissell is currently working on developing the
analysis for Evaluation Question 9. At this point, we can
present the approach proposed in the method, as well
as some initial findings related to the waste hierarchy
from other questions.
                                               11.1   Data Sources, Methods

                                               and Analysis

                                                   The method for answering this question will
                                               involve relating the components of the program to
                                               the categories of the waste hierarchy (reduce, reuse,
                                               and recycling) and then compiling information on
                                               the obstacles, opportunities, decisions, and relative
                                               emphasis related to each category of the waste
                                               hierarchy. The matrix may look something like the one
                                               at the bottom of the page.
                                                   The Evaluation Committee will explore the
                                               degree to which the paint stewardship program
                                               emphasizes waste reduction over others as reflected in
                                               educational materials, convenience, infrastructure, and
                                               performance. The Evaluation Committee will compare
                                               the effectiveness of waste reduction messages to others,
                                               resources devoted to waste reduction compared to
                                               other management methods, and the number of paint
                                               exchanges and volumes exchanged before and after
                                               paint stewardship.
Waste hierarchy
category


Program
component


Decision made


Obstacles


Opportunities


Relative emphasis


69     Evaluation of the Oregon Paint Stewardship Program

-------
11.2  Findings and Learning  (From Other Questions)

   As noted above, the detailed answer to Evaluation Question 9 is currently in process. Table 9 provides a
summary of findings and learning from other evaluation questions related to waste hierarchy.
Table 9 - Findings and Learning Related to Evaluation Question 9 from Other Evaluation Questions
 Other Evaluation
 Question
 Learning in Question 4
 Findings in Question 6
 Learning in Question 6
                               TiTSrTiTn
                             • To achieve goals of reducing the generation of post-consumer
                               architectural paint, paint management programs and policies may
                               explicitly document the relative emphasis they want to place on aligning
                               the paint management system with the existing waste hierarchy. For
                               instance, prioritize options such as reuse of leftover paint, reducing
                               leftover paint through improving accuracy of paint purchases (less
                               leftover) and increasing volumes of managed post-consumer paint.
                               Organizing these priorities will provide a  structure to prioritize outreach
                               strategies, messages, materials and effort.
                             • 57 percent of latex paint (approximately 201,000 gallons) was recycled as
                               paint by Metro and Amazon (PaintCare, 2011 b).
                             • 3 percent of latex paint (approximately 10,000 gallons) and 5 percent of
                               oil-based paint (approximately 3,500 gallons) was reused by consumers
                               (PaintCare, 2011 b).
                             • PaintCare provided collection locations an incentive of $0.25 per "reuse"
                               gallon to reflect avoided transportation and processing costs. An
                               organizational representative notes that no sites took advantage of this
                               incentive; sites that were already offering reuse had systems in place and
                               considered it an important service (S17).
                             • Less than 10 percent of collection locations have shelves for paint
                               exchange; PaintCare did not recruit retail sites for paint reuse due to
                               concerns about liability (S17).
                                Recycling is lower on the waste hierarchy than reuse, but there are
                                barriers to increasing paint reuse. Furthermore, recycling results in higher
                                costs (management, environment) compared to reuse. More information
                                is needed to better understand the barriers (e.g., liability, space) to retailer
                                facilitation of paint reuse
                                The private-public partnership and the waste hierarchy model (reduce,
                                reuse, recycle) were at odds in this program. The public preference
                                in the waste hierarchy model is for reduction and reuse. There were
                                opportunities to stress both reduction and reuse under this program,
                                however, the program as-designed tended to focus more on collection
                                and recycling. Other state programs can more clearly articulate program
                                priorities to ensure alignment with existing program and policy goals and
                                reduce the potential for confusion amongst consumers.
Unintended Consequence
in Question 12
                                              Promoting Environmental Results Through Evaluation     70

-------
i2.o   Market for Post-Consumer Paint

  -  -  '       EVALUATION QUESTION:
            *
               How has the market for post-consumer paint been
            *  affected by the program?
          i
• What aspects of the program have had an impact on the market and how?
  What markets and products represent potential opportunities for
  post-consumer paint products?
   Expanding the market for post-consumer paint
products is an important element of the pilot program
and could support improved management of leftover
paint. Ideally, if more consumers used post-consumer
paint, then they would generate less waste because
less new paint would be purchased (and ultimately
need to be disposed). Additionally, leftover paint can be
reprocessed into alternative non-paint products such
as cement additive. An increase in businesses using
non-paint products produced from post-consumer paint
could also reduce the amount of paint flowing into the
waste stream. This question assesses the extent to which
the program has helped develop a market for post-
consumer paint.


12.1  Data Sources, Methods

and Analysis

   Evaluation Question 10 was answered by performing
a qualitative assessment of the market by reviewing
available information on the post-consumer market
for paint in Oregon. The measures consist of a listing of
the post-consumer paint products offered, the facilities
that offer them, and the total sales of these products.
The data sources for these measures were interviews
with staff from PaintCare, recycled paint manufacturers,
and HHW facilities, as well as data and information
maintained by the program on these products. Based
on the information collected from the program and
Oregon DEQ, the Evaluation Team performed additional
research into the post-consumer paint products market
in Oregon to verify information from the interviews and
to collect additional information on these products and
markets. This additional research consists of both Web
searches and interviews with non-program paint experts
in Oregon.
12.2  Findings
12.2.1  How has the market for
post-consumer paint been affected
by the program?

   As part of the first PPSI MOU, stakeholders
identified the need to develop post-consumer paint
markets as a key issue (PPSI, 2004). The following three
projects were suggested:
   •    Market Development Strategy: Research
       potential markets for recycled paint and
       develop model procurement policies. Only
       initial funding was provided.
71    Evaluation of the Oregon Paint Stewardship Program

-------
    •   Recycled Paint Marketing Guide for
       Distributors: Develop a marketing guide for
       both public and private distributors, targeting
       a variety of market segments. This project
       was placed on hold until the Recycled Paint
       Certification System was completed and
       additional funds were raised.

    •   Recycled Paint Standard and Certification
       System: Working with Green Seal, PPSI developed
       guidelines and performance specifications for
       recycled paint, along with a system for certifying
       specific recycled paint products. This project was
       completed in August 2006.

    The Oregon pilot program's primary focus,
however, has been on the collection and management
of leftover paint and has not focused on developing a
market for post-consumer paint (S17). The education
materials developed under the program focus on the
collection aspects of the program and not on consumer
reuse or purchasing.
    In Oregon, there are three types of paint that fall
into the category of post-consumer recycled paint and
that can be considered part of the post-consumer/
recycled paint market for Oregon:25
               •   Leftover paint (reuse): paint that is dropped
                  off and could be used as-is by another
                  consumer. This is the type of paint offered for
                  paint exchange by PaintCare through Restores
                  in Oregon.

               •   MetroPaint: Metro sells recycled paint under
                  the name MetroPaint, which contains 2 percent
                  or less of total volume of virgin paint and
                  additives (Metro, 2011).

               •   Amazon Select: Amazon Environmental Inc.
                  sells Amazon Select, a recycled paint with 95
                  percent recycled content (S12).

               The amount of paint collected under the program
           and made available for the market, distribution
           channels, and amounts sold for each product are
           summarized in Table 10. In terms of reuse, 3 percent
           of latex (10,564 gallons) collected under the program
           and 3 percent of the oil-based paint (3,526 gallons)
           collected under the program was available (supply) as
           leftover paint (PaintCare, 2011b).
               According to the PaintCare report, 53 percent of
           the latex collected under the program (186,632 gallons)
           was sent to Metro to become recycled paint (PaintCare,
Table 10 - Summary of Post-Consumer Paint Products in Oregon: Amounts Available, Distribution Channels, and Amounts Sold
 Post-consumer
 paint product
Amount of
paint available
for product
Distribution
channels
 Leftover paint - paint that is dropped
 off and could be used as-is by another
 consumer
• 10,564 gallons
  of latex
• 3,526 gallons
  of oil
• Restores in
  Oregon
Amount
of program-
related
product sold
• Unknown
 Recycled paint (Metro) - small amount
 of virgin paint and additives (2 percent
 or less of total volume) are combined
 with leftover paint
• 186,632 gallons
  of latex
• 45 retailers in
  Oregon and
  Washington
• 115,343 gallons
  of MetroPaint
 Recycled paint (Amazon Environmental)
 - small amount of virgin paint and
 additives (5 percent or less of total volume)
 are combined with leftover paint
  14,085 gallons
  of latex
• Sold as
  Amazon Select
• Not available*
25 The categories below do not represent official definitions, but reflect the Evaluation Team's view of the types of products on the market.
                                               Promoting Environmental Results Through Evaluation     72

-------
2011b). Metro sold 122,705 gallons of paint between
July 1,2010 and June 30, 2011 (S16). During that time,
94 percent of the paint Metro took in came from the
Oregon program. Thus, approximately 115,343 gallons
of paint (2 5 percent of all paint collected by PaintCare)
taken in by the program between July 1, 2010 and
June 30, 2011 were sold by Metro as post-consumer
paint. Metro has been selling post-consumer paint
(MetroPaint) since 1992  and consumers can purchase
MetroPaint at more than 45 retailers throughout Oregon
and Washington (Metro, 2011). Metro indicated that the
program had no effect in their sales of post-consumer
paint. For the past several years, independent of the
Oregon pilot program, Metro increased sales of post-
consumer paint by expanding the number of outlets
offering its paint (S3). Metro reported that demand for
light color recycled paint exceeds the program's supply.
The opposite is true of darker colors (S16).
    According to the PaintCare report, 4 percent of latex
collected under the program (14,085 gallons) was sent
to Amazon to become recycled paint (PaintCare, 2011b).
Amazon, Inc. considers its sales figures as proprietary
and did not provide these figures to the Evaluation
Committee. However, Amazon indicated that only a
small percentage of its paint sales stem from paint
collected through the Oregon program (S21).
                              Table 11 summarizes the total amounts of latex
                          and oil-based paint collected under the program and
                          the amounts that were made available in the post-
                          consumer paint markets. In total, 217,157 gallons of
                          the paint collected (46 percent of the total amount
                          collected) under the program made it into the post-
                          consumer paint market. For latex paint, 60 percent of
                          the amount collected under the program was made
                          available to the post-consumer paint market. For oil-
                          based paint, 3 percent of the paint collected made it
                          into the post-consumer paint market.
                              Nationally, the annual potential market for
                          post-consumer latex paint products is $1.2 billion
                          and includes environmentally conscious adults
                          ($540 million) and contractors ($230 million), and
                          governments ($420 million) (Hult, 2011). According
                          to the 2010 Census, Oregon represented 7.6 percent
                          of the U.S. population. Using this percentage to adjust
                          the national numbers, the annual potential market in
                          Oregon would be approximately $90 million.
Table 11 - Amounts of Latex and Oil-Based Paint Collected Under the Program and Made Available in Post-Consumer Paint Markets
 Paint Type
 Latex
Total Gallons Collected
Under the Program
352,136
Total Gallons Collected Under
the Program Made Available in
Post-Consumer Paint Markets
(Percent of total)
211,281 (60%)
 Oil-Based
117,529
3,526 (3%)
 Latex and Oil
469,665
217,157(46%)
73    Evaluation of the Oregon Paint Stewardship Program

-------
12.2.2  What aspects of the
program have had an impact on the
market and how?

    The Oregon program has an indirect impact on
developing post-consumer paint markets by promoting
recycling over other management options (e.g.
disposal) and by increasing the supply of leftover paint.
By encouraging consumers to return the paint, the
program is developing the supply side of the market.
The centralized collection of paint by the program is
funneling more than 200,000 gallons of leftover paint to
be recycled into paint for resale to consumers.

12.2.3  What markets and products
represent potential  opportunities
for post-consumer paint products?

    There are two post-consumer non-paint products
being produced from Oregon leftover paint: Amazon
Environmental's cement additive (PLP) and a binding
material for biomass that Amazon Environmental uses to
fuel their recycled paint facility (PaintCare, 2011b). The
sale of recycled latex paint is challenged by the technical
limitations of recycled paint colors and consumer
perceptions on quality (Hult, 2011).
12.3  Learning
       Tracking data on the post-consumer
       paint market is an important program
       component. One component of determining
       the effect of the program on the market for
       post-consumer paint is tracking data on post-
       consumer paint sales, including both price and
       volume.
The existence of a market for post-
consumer paint was important. A paint
recycling program can help to develop a market
for post-consumer paint. In Oregon, a market
existed already to some degree, as Metro has
been selling MetroPaint since 1992. Recycling
programs can help to develop markets in
states without existing markets by identifying
retail outlets and assessing potential paint
reprocessing or recycling facilities and capacity.

Standard market indicators are not
adequate for the post-consumer paint
market. The market outcomes for paint should
be viewed in a full cost accounting framework.
Market price, quantity sold, and profitability
are limited measures of the success of a market
for post-consumer paint. A broader view
would take into account the avoided costs of
managing the leftover paint and the reduced
environmental impacts of recycling the paint.

Additional information is needed  to
determine impacts. The paint market in
Oregon encompasses consumers, retailers, and
producers, but also PaintCare and Oregon DEQ.
In building the market for post-consumer paint,
the program will have an impact on each set of
actors. However, we have little information on
how the Oregon program has impacted each
group. Some of the impacts and key questions
to consider include:

       Consumers: The post-consumer
       paint market will offer a lower-
       priced alternative to virgin paint.
       Will the availability of a lower-priced
       alternative impact the price consumers
       pay for virgin paint? Will consumers
       accept post-consumer paint as a viable
       alternative to virgin paint?
                                             Promoting Environmental Results Through Evaluation     74

-------
               Retailers: Will retailers begin to
               stock post-consumer paint? Will
               the availability of a lower-priced
               alternative reduce sales of virgin paint?
               How will profitability of retail paint
               sales be affected?

               Producers: Will the availability of
               post-consumer paint affect producers'
               sales of virgin paint? Will producers
               consider expanding their product lines
               into the post-consumer market and
               begin offering post-consumer paint?

    Providing answers to these questions will require
extensive and more detailed research into the impacts
of post-consumer paint on paint markets in general.

    •   Addressing consumer concerns about
       leftover and recycled paint is important.
       Information is needed on consumer acceptance
       of leftover and post-consumer/recycled paints
       as a viable product for them to use. This basic
       level of market research would provide a
       baseline for the potential size of the market
       and also help to identify specific concerns that
       consumers may have. These concerns could
       then be addressed as part of an education and
       outreach campaign to promote the use of post-
consumer paint as a viable alternative to virgin
paint. Furthermore, an education and outreach
campaign that targets messages to different
audiences may be more effective at increasing
acceptance by consumers. Although the PPSI
Recycled Paint Standard and Certification
System project was completed, stakeholders
have not funded the Recycled Paint Marketing
Guide for Distributors project. The intended
purpose of that guide is to educate buyers
about the nature of recycled content paint,
including its quality and performance.

Pricing of leftover/recycled paint may
be complex. Some consideration should be
given to balanced pricing for leftover paint.
A low price for leftover paint may encourage
consumers to purchase too much leftover paint,
leading to the paint being leftover again. This
needs to be balanced against the cost of new
paint. In short, leftover paint should be priced
at a level that both represents a significant
discount compared to new paint but not too
low that it encourages consumers to purchase
"extra" paint they may not need. Currently,
little information on paint markets exists to
determine that tipping point and research
into consumer preferences may be needed to
determine a balance.
75     Evaluation of the Oregon Paint Stewardship Program

-------
i3.o   Transferability
             EVALUATION QUESTION:
             Based on the Oregon experience, what implementation
             and outcome-related information is required for
             other states to develop and implement leftover paint
             management systems?
  To what extent are the performance measurement and evaluation systems
  transferable to other states?
  What are the best ways to communicate the results of the evaluation?
13.1  Data Sources, Methods  13.2 Findings and
and Analysis                      Learnings
   Evaluation Questions 11 and 12 are designed as
pure learning questions. The Evaluation Team developed
answers to these questions from data and information
collected while answering other evaluation questions
and supplemented with interviews conducted with a
variety of stakeholders.
13.2.1 What implementation and
outcome-related information is
required for other states to develop
and implement leftover paint
management systems?

   States developing paint management systems
should start with reliable baseline data. Baseline data
would include:

   •  Volume data on paint: Baseline data
     on paint sales, amounts of paint that are
     managed through various methods in the state
     (recycling, conversion to energy, etc.), and
     amounts disposed.

   •  Current infrastructure: Document the
     extent of current infrastructure (recycling
     facilities, locations, processing facilities, events,
     transportation etc.), including:
                                   Promoting Environmental Results Through Evaluation   76

-------
               Information on the capacity (amount
               of paint that could be processed or
               collected).

               The knowledge and experience of
               those who would be working with the
               leftover paint.

               Attitudes toward recycling and waste
               reduction among consumers and
               those who would be working with the
               leftover paint.

               A link between infrastructure data
               and volume data to determine the
               amounts being collected, processed, or
               transported within different parts of
               the infrastructure.

       A system map: Developing a map of how
       paint is currently processed in the state
       provides a useful starting point to developing
       a new system. The map developed by the PSO
       (Figure 7) is a starting point. However, a more
       detailed process flow diagram would be better
       suited for tracking the current movement of
       paint in a state.

       Cost information: Information on the costs
       processors of paint incur, including costs for
       HHWs to process paint.

       Awareness: Information on consumer
       awareness of disposal and paint recycling
       options, paint-related environmental issues
       and hazards, and information on where
       consumers get their information.

       Consumer, behavior, attitudes and
       practices: Information on how consumers
       decide on how much to buy, what they
       currently do with leftover paint, and how
       much leftover paint they currently have stored
       in their homes. Additionally, information on
       attitudes toward recycling and the environment
       would provide useful information that can be
       used to develop outreach and education (e.g., a
       consumer base that has strong preferences for
       recycling and preserving the environment may
       not need convincing that recycling paint for
       environmental reasons is a good idea).
    As California and other states with robust
infrastructures roll out leftover paint management
systems, these states should first ensure that plans
incorporate pre-existing infrastructure into the
program. Oregon DEQ allocated a quarter-time
employee for its program responsibilities. OR DEQ
indicated that a quarter-time employee would have
been sufficient in fulfilling program responsibilities,
except for the extra demands generated by being
the first state (e.g., participating in the Evaluation
Committee; requests from other states for information,
etc.) (S20).
    The Oregon program stakeholders identified the
following important gaps in the implementation and
outcome information currently available:

    •   The cost to local governments (e.g.,
       permanent HHW facilities and events) to act
       as collection sites, including cost of labor and
       the opportunity cost of storage space. Costs
       reported by PaintCare may not capture the full
       costs of the program, as local governments may
       be "subsidizing" some of the costs by providing
       labor for paint collection (S20).

    •   The proportion of program costs
       attributable to overhead and PSO
       Structure. In the Annual Report, PaintCare
       notes that revealing some vendor costs would
       result in a breach of contract; there is interest
       in the cost-effectiveness of employing 'layers"
       of contractors in order to implement the
       program (S20).

    •   The best approach to recruit retail
       collection sites. PaintCare used targeted
       mailings to retailers to recruit them as potential
       collection sites; as a result, the initial list of
       retail collection sites was primarily based on
       retailer self-selection into the program rather
       than site location. Upon inspection some of the
       initial sites were not suitable to act as  collection
       sites and that in the future, the organization
       should visit each site before designating it as
       a collection site (S17). Another stakeholder
       interviewee expressed interest in a more
       strategic approach to recruiting sites (S20).
77     Evaluation of the Oregon Paint Stewardship Program

-------
    •   The best approach to measure
       convenience. The Annual Report focuses
       on the percent of Oregon residents that live
       within a certain radius of the nearest collection
       site as a measure of convenience. A more
       comprehensive measure would include the
       considerations addressed by Strickland (2011)
       that included drive time and driving distance
       to collection sites. Although the CIS analysis
       to develop the drive-time and driving distance
       measures is more complex, Strickland has
       provided an approach that could be replicated
       in other states with CIS capacity.
    The majority of inquiries that PSI received about
the program relate to the fee (S19). Other states
developing these programs should consider providing
outreach to residents in general and paint consumers
more specifically related to the fee structure.

13.2.2  To  what extent are the
performance measurement and
evaluation systems transferable to
other states?

    The PPSI Evaluation Committee developed
the framework for the evaluation (i.e., Evaluation
Committee, evaluation questions, methods, and
measures) of the pilot program initially planned for
Minnesota. When the Minnesota legislation was not
signed into law, PPSI shifted its focus to Oregon. The
Minnesota framework was used to develop the full
methodology for evaluating the Oregon program and
could be transferred to other states as well. In short, the
model for this framework involves:

    •   Working collaboratively with all
       stakeholders to define the goals of
       the program using the PPSI goals as a
       starting point, but adapting those goals to
       match state-specific context.
    •   Drafting evaluation questions that are
       relevant for the program based on those
       goals.  The evaluation questions that were
       developed for Oregon are a starting point, but
       other states may have other issues to explore.
       The Evaluation Committee began with 27
       questions and narrowed that number to 12
       through a participatory prioritization process
       (Appendix A of Evaluation Methodology) while
       maintaining the core topics included in the
       27 original questions. Because the Evaluation
       Committee was focused on the opportunity to
       maximize learning from the Oregon pilot, and
       provide thorough information and a starting
       point for performance management of future
       state-wide paint management programs,
       it was ambitiously comprehensive in its
       approach to evaluating this program. Other
       states may benefit from narrowing the scope
       of questions and focusing effort and resources
       on developing comprehensive information
       about the issues most critical to the success of
       a similar program in that particular state.

    •   Determining the measures and data
       necessary to answer the evaluation
       questions and design collection methods
       for obtaining those data. The Evaluation
       Committee was ambitious in its design of
       measures and some measures were not
       activated or only partially activated because:

               The program was in the design
               phase and the level of information
               that would be available throughout
               implementation was an unknown.

               By articulating what would be
               measured, the Evaluation Committee
               intended to influence what would be
               done and what information would be
               provided.

               The measures designed are, the
               evaluation committee believes, the
               measures necessary to answer the
               questions, so even if the measures
               were not activated for the Oregon
               pilot, they might be a starting point for
               future similar programs.

    Other states may want to consider identifying
the key measures for each evaluation question and
                                              Promoting Environmental Results Through Evaluation     78

-------
work to ensure those data are collected. This requires
collaboration and explicit agreements between the
stakeholders to ensure that the data is collected,
credible, accessible, and timely. The evaluation method
document is designed as a flexible starting point;
interested states should identify key areas of interest
and prioritize matching evaluation questions and
measures, as well as plan data collection accordingly.
Some key caveats to the transferability of the evaluation
framework include that evaluation priorities must be
customized to the interests of each state's stakeholders
and full development of the methodology requires
commitment of resources (time and funding).
Additionally, fully integrating evaluation into program
design requires close coordination and integration of
the Evaluation Committee with the design, planning and
implementation teams and efforts.
    This evaluation committee created a Web-based
model  (referred to as a "fuzzy logic model" by the
Evaluation Team; see Figure 1) of the Oregon pilot
program to depict the program and related systems,
organizations, processes and products as well  as
the role of measurement and evaluation in the
program. Other states may consider this approach
to depicting the program, evaluation design, and
ongoing performance management information (e.g
performance dashboards). For one, this type of model
allows  the evaluation team to develop a depiction
of the program that can convey to a wider range of
audiences, including the public, a sense of what the
program is intended to accomplish and how it is
intended to operate. Second, a Web-based model allows
the program to maintain communication with  program
participants  about performance, results and outcomes.
    Performance measurement and evaluation can be
costly. Building data collection into program operations
and creating partnerships offer opportunities to reduce
costs and increase the efficiency of data collection and
performance management. For example, evaluation
questions related to the impact of the education and
outreach campaign required baseline and follow-
up surveys of aspects of consumer awareness and
behavior. The Evaluation Committee expected  to receive
the necessary data to determine impact of education
and outreach from surveys implemented by PaintCare,
but the baseline survey performed in August 2010
did not contain some data elements that were needed
to assess impact. Additional resources for evaluation
would have allowed for baseline and follow-up surveys
to have been implemented and conducted based on
the evaluation's needs. In the interest of efficiency and
reduced costs, the Committee partnered with Duke
University, GSU, and two graduate students to perform
analyses and data collection. This approach worked
well and allowed for data collection and analysis
despite funding constraints.
    Oregon and other states considering implementing
product stewardship programs (paint or otherwise)
may consider a model similar to the Conservation
Measures Partnership's (CMP's) Open Standards for
the Practice of Conservation.26 The purpose of the Open
Standards is to address the following two key questions:

    •   What actions are needed to achieve
       a program's goals?

    •   Are these actions  effective in achieving
       the goals?

   To provide practitioners with the tools to
answer these two questions, the Open Standards
"bring together common concepts, approaches,
and terminology in conservation project design,
management, and monitoring in order to help
practitioners improve the  practice of conservation.
In particular, these standards are meant to provide
the steps and guidance necessary for the successful
implementation of conservation projects" (CMP, 2007).
Figure 35 depicts the approach used by the Evaluation
Committee to incorporate evaluation into program
design of the Minnesota and Oregon pilot projects on
the top (in blue). On the bottom (in green) is the Open
Standards model, which depicts the process of ongoing
program adaptive management. In Oregon, this process
is already well underway, and this report is a part of the
program efforts to capture and share learning from the
Oregon pilot to encourage more effective practices in
the future.
26 www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_Version_2.0.pdf.
79     Evaluation of the Oregon Paint Stewardship Program

-------
Figure 35 - Shifting from Incorporating Evaluation into Program Design to CMP Open Standards
                        1. Evaluation Policy
                        2. Evaluation
                          Methodology
                                             Integrating
                                               valuation
                                                   into"
                                               Program
                                                Design
                                                        1. Team
                                                        2. Mission
                                                        3. Goals &
                                                          Objectives
                                                        4. Logic Model
               Data Sources
             2. Collection Methods
               & Strategy
               Analysis Tools
1. Context
2. Audience
3. Communication
4. Use
                                              1. Conceptualize
                                            Define initial team
                                            Define scope, vision, targets
                                            Identify critical threats
                                            Complete situation analysis
                                                                       2. Plan Actions & Monitoring
           5. Capture & Share Learning
                                                                        Develop goals, strategies,
                                                                        assumptions, and objectives
                                                                        Develop monitoring plan
                                                                        Develop operational plan
Document learning
Share learning
Create learning environment
                                      OPEN
                                 STANDARDS
                     4. Analyze, Use, Adapt
                                                      3. Implement Actions & Monitoring
                   • Prepare data for analysis
                   • Analyze results
                   • Adapt strategic plan
                                                      Develop work plan and timeline
                                                      Develop and refine budget
                                                      Implement plans
                                                  Promoting Environmental  Results Through Evaluation     80

-------
13.2.3  What are the best ways to
communicate the  results of the
evaluation?

    The Evaluation Team spent time upfront during this
project to identify the information needs of different
stakeholders. This upfront communication helped
clarify what was needed for effective communication
with PSI, Oregon DEQ, PaintCare, rollout states and other
stakeholders (e.g., HHWs). However, the Evaluation
Team was also operating under tight resources and tight
deadlines. To effectively communicate to stakeholders,
the Evaluation Team developed a Web-based
model of the Oregon program to communicate with
stakeholders about the program design, planning, and
implementation and findings from the evaluation (www.
paintstewardshipprogram.com).
    Based on the findings under Evaluation Question
1, PPSI participants prefer to use conference calls and
email to communicate about the program. Participants
also expressed a preference for face-to-face meetings
when possible. The best modes for communicating
evaluation results likely include the following:

    •    Conference calls briefing stakeholders on the
        results.

    •    Emails summarizing results in the format of a
        newsletter or fact sheet.

    •    Presentation of results during a conference or
        other in-person meetings.

    The findings under Evaluation Question 1 also
suggest that the use of a website is an underutilized
mode for information communication. During the PPSI,
the website provided fairly limited information for
use (i.e. minutes from the most recent meeting) and
no opportunities for providing feedback or otherwise
interacting with the information. This mode could
become a useful venue  for sharing evaluation results
with a few changes, as follows:
    •    Provide more information on a regular basis.

    •    Allow readers the opportunity to interact with
        and respond to the information.

    •    Update the information strategically. Add
        new resources and materials to the website
        around a focal point, such as a conference call
        or reporting date, when the material is most
        relevant to readers.

    The evaluation website (www.
paintstewardshipprogram.com) is a visualization of the
program incorporating functional features that provide
deep and diverse information about the program. It
illustrates program implementation as well  as the
performance measurement and evaluation system
and could be an effective tool for sharing information
on an ongoing basis. A website may be accessible to a
wider audience than other communication modes (e.g.,
conference calls) and would allow diverse audiences
to focus on different components of the system
(e.g., education and outreach materials, costs, and
environmental impacts).
    States should also consider using a dashboard
reporting system to provide ongoing updates of
program performance. A dashboard system  could be
incorporated into the web-based program model and
would involve reporting a few key results on a frequent
basis (e.g., volume of paint collected, number of collection
sites, and number of leftover paint gallons sold).
       Evaluation of the Oregon Paint Stewardship Program

-------
i4.o    Unexpected  Results
                EVALUATION QUESTION:
                During the program and for each of its primary
                components, what were the primary external,
                unexpected and/or unintended influences and
                consequences?
   Not all of the changes in paint management were
driven by the program itself. A number of factors
could also have influenced paint usage and handling
by consumers including economic activity and
characteristics of the paint market. Some consideration
must be given to external influences on consumer
behavior. Additionally, programs often have unintended
or unforeseen consequences. This section documents
some of these results and their implications for Oregon
and other states.


14.1   Data Sources, Methods

and Analysis

   The Evaluation Team answered Evaluation Question
12 by reviewing the lessons learned from the program
and through interviewing those involved in program
implementation. The interviews covered the different
groups involved in planning and implementation and
extracted information related to different stages in the
program's development.


14.2   Findings  and Learning

External
   Minnesota Governor Tim Pawlenty twice vetoed
the bill to create an industry-managed statewide paint
stewardship and recycling program after the Minnesota
legislature had passed legislation twice by near-
unanimous margins. Governor Pawlenty's rationale was
that the bill would put a double burden on the public,
since it authorized a new fee to fund the program,
and local governments currently spend public funds
to dispose of and recycle paint (Pawlenty, 2009). The
PPSI organized collective support from industry, local
governments, EPA, and environmental organizations
for the Minnesota legislation; the politics in blocking
the program was unanticipated and delayed the
implementation of a pilot program for 1.5 years (S19).

Unexpected
   •  Effective collaboration was more time
      consuming and costly, involving more meetings,
      calls and committees than expected (See
      Evaluation Question 1). Few parties involved
      had any notion of how time-consuming the
      program would be and did not know what to
      budget for collaborative work (e.g., meetings,
      reviewing products, etc.). Additionally, seven
      years passed from the beginning discussions
      to implementation of a pilot program - a longer
      time-frame than was anticipated by most
      participants.

   •  Some retailers declined to show the fee on the
      receipt. The PPSI anticipated that retailers
      would show the fee to indicate the extra amount
      was not part of their price but the PPSI was not
      aware that reprogramming store systems was
      such a burden to store managers/owners
                                         Promoting Environmental Results Through Evaluation    82

-------
        The process became less collaborative when
        industry was required to submit a plan,
        negotiate with Oregon DEQ, and implement
        the approved program plan. This breakdown
        in collaboration was caused partly by lack of
        funding for a facilitator/champion (S19, S20).

        There were a large number of unlabeled
        and/or rusty cans of paint that cannot be
        accepted by the program. The collection sites
        are instructed to not open the cans of paint
        and those cans  are treated as non-program
        materials which must be managed by Oregon.
        (See Evaluation Question 5)

        PPSI had limited representation from retailers
        and so the fact that retailers play a large role
        in the Oregon paint collection infrastructure
        was surprising  (S18). Additionally, large retail
        chains (e.g.,  Lowe's, Home Depot, and Wal-
        Mart) declined  to participate.

        That legislation was needed to implement
        a program was  unexpected. The need for
        legislation arose when issues of anti-trust and
        collusion arose since paint producers would
        be working together. Thus, legislation was
        needed to exempt the paint producers from
        anti-trust and collusion concerns. At the start,
        the idea of the program was to avoid having
        states pass legislation to tell the private sector
        what to do— the program was intended to be
        voluntary.

        The lack of a champion/funder to collect real
        world data under the Oregon pilot to inform
        appropriate levels of virgin paint offset to
        model in an LCA meant that no additional
        progress was made on estimating the
        environmental  impacts of paint management
        approaches.

        The evaluation  committee was unable to get
        cost data broken out by category to perform a
        more thorough analysis.
Unintended
    The private-public partnership and the waste
hierarchy model (reduce, reuse, recycle) were at odds
in this program. The public preference in the waste
hierarchy model is for reduction and reuse. There
were opportunities to stress both reduction and reuse
under this program. However, the program as-designed
tended to focus more on collection and recycling. Other
state programs should consider a clear articulation
of the program priorities to ensure alignment with
existing program and policy goals and reduce the
potential for confusion amongst consumers.
    The reliance on retailers as collection locations
had a negative impact on the program's ability to divert
high quality leftover paint for reuse, which is ranked as
a preferred use. Further exploration into this topic—
what are the obstacles (e.g., liability and competition
for the sale of virgin paint) and solutions (e.g.,
redistribution to HHW collection locations or Reuse
type stores)—is needed.
83     Evaluation of the Oregon Paint Stewardship Program

-------
i5.o    Reference  Materials  and
Abt Associates (Abt). 2007. Quantifying the Disposal of Post-Consumer Architectural Paint, U.S.
   Environmental Protection Agency - Sector Strategies Program.

Amazon. 2010. Amazon Environmental, Inc. Latex Paint Recycling. Available at: www.amazonpaint.com/
   other-products.html.

Bledsoe, Wes, Eric Graves, and Andrei Roman. 2011. Description of the Oregon Pilot Program Paint
   Stewardship Organization. Prepared for PPSI Evaluation Committee and PMAP 8900 Public Service
   Capstone course in the Andrew Young School of Policy Studies, Georgia State University.

Bradshaw Advertising. 2010. PaintCare Awareness Survey of Oregon Homeowners. Report prepared for
   PaintCare.

Braunz, Amy, Julie Calvin, Whitney Knapp, Hedrick Strickland. 2010. Assessment of Collaboration of
   the Paint Product Stewardship Initiative (PPSI) and Pilot Program. Prepared for PPSI Evaluation
   Committee and ENV 280 Social Science Survey Design course at Duke University.

Cascadia Consulting Group (Cascadia). 1998. Paint Smart Pilot Program Evaluation and Final Report.
   Prepared for Oregon Department of Environmental Quality. Available at: www.deq.state.or.us/lq/sw/
   prodstewardship/painthtm.

Conservation Measures Partnership (CMP). 2007. Open Standards for the Practice of Conservation.
   Available at: www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_
   Version_2.0.pdf.

Cousins, J. Bradley and Lorna M. Earl. 1992. "The case for participatory evaluation," Educational
   Evaluation and Policy Analysis, 14, 397-418. p.399

Cousins, J.Bradley and Elizabeth Whitmore. 1998. "Framing Participatory Evaluation," New Directions for
   Evaluation, No. 80, Winter 1998, 5-23.

Eastern Research Group, Inc. 2010. Method for Evaluating the Paint Product Stewardship Initiative's
   Oregon Pilot Program. Prepared for U.S. EPA Evaluation Support Division.
                                         Promoting Environmental Results Through Evaluation    84

-------
Hult International Business School Project Team 2. 2011. Paint Recycling: An Environmentally and
   Economically Sustainable Business Model. Boston, MA.

Keene, Matt, Lou Nadeau, Amy Stillings and Jennifer Nash. 2010. Memo entitled "Leftover Paint Oregon
   Pilot Project Evaluation: Prioritizing Evaluation Questions," Oregon Paint Stewardship Pilot Program
   Environmental Committee. Available at: www.paintstewardshipprogram.com/images/pe_leftover_
   paint_oregon_pilot_project_evaluation_prioritizing_evaluation_questions.DOC

Keene, Matt. 2008. Integrating Evaluation into Program Design. Available at: www.productstewardship.
   us/displaycommon.cfm?an=l&subarticlenbr=754. Available at: www.paintstewardshipprogram.
   com/images/pe_integrating_evaluation_into_program_design_06_01_08.DOC and www.
   paintstewardshippro gram.com/images/pe_inte grating_evaluation_into_oregon_pilot_program.DOC

Metro. 2011. MetroPaint: Quality Recycled Latex Paint, www.oregonmetro.gov/index.cfm/go/by.web/
   id=521.

National Paint and Coatings Association. February 2008. The Paint Industry Works Toward a Nationally
   Coordinate System for Post-Consumer Paint Management, Issue Backgrounder. Available at: www.paint.
   org/pubs/background.cfm.

Oregon Legislative Assembly. 2009. Committee of Sustainability and Economic Development, 75th
   Assembly, 2009 Regular Season. HR 3037-A. Available at www.leg.state.or.us/09reg/measures/
   hb3000.dir/hb3037.en.html.

Oregon Department of Environmental Quality (DEQ).  2011. Oregon Household Hazardous Waste Report,
   2008-2009.

OR DEQ. 2008. Oregon Department of Environmental Quality Household Hazardous Waste Survey: Results
   Report. Portland State University Survey Research Lab: Portland, OR.

PaintCare. 2010. Oregon Paint Stewardship Plot Program Plan. Washington, DC.

PaintCare. 2011a. PaintCare Awareness Survey of Oregon Residents. Washington, DC.

PaintCare. 2011b. Oregon Paint Stewardship Pilot Program Annual Report. Washington, DC.

Patton, Michael Quinn. 2010. Developmental Evaluation: Applying Complexity Concepts to Enhance
   Innovation and Use. The Guilford Press.

Governor Tim Pawlenty. 2009. "Letter to President Metzen," State of Minnesota (May 21, 2009).
   Available at: www.calpsc.org/products/docs/2009/5-21%20Metzen%20VETO%20CH%20121%20
   SF%2 0477.pdf
85     Evaluation of the Oregon Paint Stewardship Program

-------
Paint Product Stewardship Initiative (PPSI). 2004. Paint Product Stewardship Initiative
   Memorandum of Understanding, Available at: www.productstewardship.us/displaycommon.
   cfm?an=l&subarticlenbr=118.

PPSI. 2007. Final Paint Product Stewardship Initiative 2nd Memorandum of Understanding. Available at:
   wwwproductstewardship.us/displaycommon.cfm?an=l&subarticlenbr=118

PPSI. 2008. Paint Product Stewardship Initiative paint Demonstration Project Committee Draft Work Plan.

Product Stewardship Institute (PSI). 2004a. Paint Product Stewardship: A Background Report for the
   National Dialogue on Paint Product Stewardship. Boston, MA.

PSI. March 2004b. Product Stewardship Action Plan for Leftover Paint.

PSI. 2009. "Paint Dialogue Stakeholder Meeting: Final Meeting Notes," Portland, Oregon. December 9-10,
   2009. Available at: www.productstewardship.us/associations/6596/files/FINAL_PPSI_Meeting_
   Summary_Portland_OR_Dec9-10_2009.doc

PSI. 2010. Extended Producer Responsibility Legislation as of August 2010. Available at: http://
   products tewardship.us/displaycommon.cfm?an=l&subarticlenbr=615.

SCS Engineers and Cascadia Consulting Group (SCS and Cascadia). 2007. Paint Product Stewardship
   Initiative Infrastructure Project, Washington Department of Ecology. Available at: www.
   products tewardship.us/displaycommon.cfm?an=l&subarticlenbr=12 8.

Strickland, Hedrick. 2011. Convenience analysis of the Oregon Paint Management Pilot Program. Duke
   University, Masters Thesis, advised by Dr. Lynn Maguire and Matt Keene, EPA.

Thompson, Ann Marie, James L. Perry, and Theodore K. Miler. 2007. "Conceptualizing and Measuring
   Collaboration," Journal of Public Administration Research and Theory, 19: 23-56.

Torres, Preskill and Piontek. 2005. Evaluation Strategies for Communicating and Reporting: Enhancing
   Learning in Organizations. 2nd Ed. Sage Publishing.
NOTE: Stock photography purchased from www.123rf.com. Special thanks to Duke University for permission
   to use the logo and image of Nicholas School of the Environment on page 24.
                                            Promoting Environmental Results Through Evaluation     86

-------
Interviews and Personal Communications

K*Q
	 '
itact Number Contact and Organization
^^
Stakeholder Interview Number [S]
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
S19
S20
S21
Abby Boudouris, Oregon DEQ
Steve Sides, ACA
Jim Quinn, Metro
Jim Quinn, Metro [HHW]
Abby Boudouris, Oregon DEQ
Alison Keane, PaintCare
Tom Wroblewski, Home Depot [Retailer]
Rick Barnard, Rodda Paint Company [Retailer]
Jack Wickham, Ace Hardware [Retailer]
Larry Gibbs, Lane County [HHW]
Timm Schimke, Deschutes County [HHW]
Roy Weedman, Columbia County [HHW]
Jennifer Purcell, Tillamook County [HHW]
Tina French, Lincoln County [HHW]
Garry Penning, Rogue Disposal [HHW]
Jim Quinn, Metro
Alison Keane, PaintCare
Barry Elman, EPA
Scott Casell, PSI
Abby Boudouris, Oregon DEQ
Lorraine Segala, Amazon Environmental
09-23-2010
12-2010
12-2010
06-27-201 1
03-02-201 1
03-14-2011
05-03-201 1
05-09-201 1
05-13-2011
06-21-2011
06-21-2011
06-21-2011
06-28-201 1
06-29-201 1
07-01-2011
08-26-201 1
08-26-201 1
09-02-201 1
09-06-201 1
09-06-201 1
09-14-2011
Person Communications With ERG Via Email [Em]
Em1
Em2
Em3
Em4
Em5
Em6
Em 7
Em 8
Jim Quinn, Metro
Alison Keane, PaintCare
Mike O'Donnell, PaintCare
Jim Quinn, Metro
Albert Salvi, Washington State Department of Ecology
Alison Keane, PaintCare
Alison Keane, PaintCare
Alison Keane, PaintCare
08-24-201 1
08-26-201 1
09-14-2011
09-15-2011
09-08-201 1
09-08-201 1
09-09-201 1
09-12-2011

87
Evaluation of the Oregon Paint Stewardship Program


-------
Recycled/Recyclable • Printed with vegetable oil based inks on 100% postconsumer, process chlorine free recycled paper.

-------