United States Environmental Protection i Agency 2011 Evaluation of the Oregon Paint Stewardship Program Promoting Environmental Results Through Evaluation ------- Acknowledgements This evaluation was performed by Eastern Research Group, Inc. (ERG) under subcontract to Industrial Economics, Incorporated (lEc) for the U.S. Environmental Protection Agency's (EPA's) Office of Planning, Analysis, and Accountability under Contract EP-W-07-028 between EPA and lEc. The ERG evaluation team included Lou Nadeau, Ph.D., Amy Stillings, and Lauren Jankovic. Matt Keene of EPA's Office of Policy, Evaluation Support Division acted as the technical advisor. The Evaluation Team is grateful to the continuous and stalwart support of the members of the Paint Product Stewardship Initiative (PPSI) Evaluation Committee. We are especially thankful to the invaluable assistance provided by Abby Boudouris, Alison Keane, Scott Cassel, Barry Elman, Chris Metzner, Hedrick Strickland, and Brielle Kissell. Without them, this work would not have been possible. This report was developed under the Program Evaluation Competition, sponsored by EPA's Office of Policy. To access copies of this or other EPA program evaluations, please go to www.epa.gov/evaluate. Evaluation of the Oregon Paint Stewardship Program ------- TableofContents Executive Summary i - vi 1.0 Introduction 1 1.1 Paint Product Stewardship Initiative Oregon Pilot Program 2 1.2 Evaluation of Pilot Program 4 2.0 Overview of the Evaluation Method and Key Data Sources 6 2.1 Evaluation Approach 6 2.2 Communicating Information 8 2.3 Evaluation Design 9 2.4 Key Data Sources Used in the Evaluation .. 14 3.0 Collaboration 17 3.1 Data Sources, Methods and Analysis... 17 3.2 Findings 18 3.3 Learning 23 4.0 Paint Stewardship Organization 25 4.1 Data Sources, Methods and Analysis... 25 4.2 Findings 26 4.3 Learning 31 5.0 Education and Outreach 32 5.1 Data Sources, Methods and Analysis... 32 5.2 Findings 33 5.3 Learning 44 6.0 Consumer Purchasing Decisions 46 6.1 Data Sources, Methods and Analysis... 46 6.2 Findings 46 6.3 Learning 48 7.0 Collection of Post-Consumer Paint 50 7.1 Data Sources, Methods and Analysis... 50 7.2 Findings 50 7.3 Learning 56 8.0 Paint Reprocessing, Recycling, and Energy Recovery 58 8.1 Data Sources, Methods and Analysis... 58 8.2 Findings 59 8.3 Learning 61 9.0 Household Hazardous Waste Programs ... 62 9.1 Data Sources, Methods and Analysis... 62 9.2 Findings 63 9.3 Learning 64 10.0 Cost Effectiveness 66 10.1 Data Sources, Methods and Analysis... 66 10.2 Findings 66 10.3 Learning 67 11.0 Waste Hierarchy (to be completed by January 2012) 69 11.1 Data Sources, Methods and Analysis... 69 11.2 Findings and Learning (From Other Questions) 70 12.0 Market for Post-Consumer Paint 71 12.1 Data Sources, Methods and Analysis... 71 12.2 Findings 71 12.3 Learning 74 13.0 Transferability 76 13.1 Data Sources, Methods and Analysis... 76 13.2 Findings and Learning 76 14.0 Unexpected Results 82 14.1 Data Sources, Methods and Analysis... 82 14.2 Findings and Learning 82 15.0 Reference Materials and Interviews 84 Promoting Environmental Results Through Evaluation ------- Executive Summary OVERVIEW In 2002, a coalition of paint manufacturers, represented by American Coatings Association (ACA); local, state, and federal environmental agencies; retailers; and consumer and environmental agencies formed the Paint Product Stewardship Initiative (PPSI) and began negotiations facilitated by the Product Stewardship Institute (PSI) to create an industry- managed post-consumer paint management system. After seven years of negotiations, two Memoranda of Understanding (MOUs), and comprehensive PPSI-sponsored research of the paint industry and consumers, the state of Oregon became the first to enact a law establishing product stewardship as the preferred method for reducing the environmental impacts and costs associated with leftover paint. This report summarizes the results of the evaluation of the Oregon paint recycling program performed by the PPSI Evaluation Committee, which includes members representing the diversity of participants in the PPSI. The Oregon program, depicted in Figure ES-1, is comprised of a diversity of interconnected systems, actors, and processes. The major components of the Oregon program include the Paint Stewardship Organization (PSO; PaintCare), the oversight of the Oregon Department of Environmental Quality (DEQ), the paint market (producers, retailers, and consumers), and the leftover paint management system (collection sites, processing, etc.). To fund the program, Oregon added an assessment fee onto the price of paint sold in the state. Full documentation of the details of the program appears in PaintCare's Oregon Paint Stewardship Pilot Program Plan (PaintCare, 2010). The evaluation addressed twelve questions and the information collected and presented is extensive and accessible in multiple formats to increase the evaluation's utility for a diversity of audiences. As a companion to this report, the PPSI Evaluation Committee developed a website to organize and present the results from this evaluation - www. paintstewardshipprogram.com. The web site presents Figure ES-1: The Oregon Paint Recycling Program Paint Management System Evaluation of the Oregon Paint Stewardship Program ------- the key findings and learning from this evaluation in a dynamic, visual, and interactive manner that includes links to information and data sources used in this report. This executive summary presents some basic findings and learning from the evaluation for each of the 12 evaluation questions (see text box). The evaluation was conducted between October 2009 and September 2011 by the PPSI Evaluation Team, a subgroup of the Evaluation Committee, which consisted of the U.S. Environmental Protection Agency (EPA) and its subcontractor Eastern Research Group, Inc. (ERG)1 in partnership with academic researchers from Duke University, Georgia State University, and Tufts University. While integrating measurement and evaluation (M&E) into the design and implementation of the Oregon pilot program, the Evaluation Committee took a participatory approach to its work, emphasizing transparency and communications while incorporating aspects of developmental and participatory evaluation into the evaluation design to account for program complexity. The Evaluation Team relied on three primary sources of data and information: Two surveys of consumers, one conducted in August 2010 just after the program began and another conducted in July 2011, provide key information for Evaluation Questions 3 and 4 and data for several other questions. In-person interviews and other personal communications with program stakeholders: the Evaluation Team conducted 21 interviews during the course of the evaluation; including seven interviews with HHW representatives, three interviews with retailers, and 11 interviews with key program stakeholders. Documents developed by PaintCare: the Evaluation Team drew heavily on the 2010 Oregon Paint Stewardship Pilot Program Plan (PaintCare, 2010) and the 2011 Annual Report (PaintCare, 2011b) to provide information on program design, implementation, and results. 1 ERG's work was performed under subcontract to Industrial Economics, Inc. (IEC) under contract EP-W-07-028 between EPA and lEc. 1. Collaboration: To what degree was the pilot program, from planning to implementation, a collaborative process? 2. Paint Stewardship Organization: Describe the PSO, including its funding mechanism and infrastructure. 3. Education and Outreach: How did education materials and strategies affect consumer awareness and behavior? Which messages were most effective with which target audiences? What materials/strategies were developed and what were the goals and target audience of those materials/strategies? Did other factors besides the program influence consumer behavior and awareness? What are the lessons learned? 4. Consumer Purchasing Decisions: How has the program affected consumers' purchasing decisions and management of paint prior to drop-off at paint recycling facilities? How did the fee assessment affect consumer behavior? 5. Collection of Post-Consumer Paint: How has the program affected the collection of post-consumer paint in terms of volume, cost, environment, convenience, and infrastructure? What other factors have affected the amount of leftover paint? 6. Paint Reprocessing, Recycling, and Energy Recovery: How has the program affected used paint reprocessing, paint recycling, and paint-related energy recovery in terms of volume, infrastructure, and cost? 7. Household Hazardous Waste Programs: What was the impact of the program on the HHW facilities in terms of the types of paint collected, costs, and the way in which the facilities operate? 8. Cost Effectiveness: How cost effective is the program? 9. Waste Hierarchy: How was the program designed and implemented to move consumers up the waste hierarchy? With respect to moving customers up the waste hierarchy, what were the program's obstacles, opportunities, and decisions? 10. Market for Post-Consumer Paint: How has the market for post-consumer paint been affected by the program? What aspects of the program have had an impact on the market and how? What market and products represent potential opportunities for post- consumer paint products? 11. Transferability: Based on the Oregon experience, what implementation and outcome-related information is required for other states to develop and implement leftover paint management systems? To what extent are the performance measurement and evaluation systems transferable to other states? What are the best ways to communicate the results of the evaluation? 12. Unexpected Results: During the program and for each of its primary components, what were the primary external, unexpected and/or unintended influences and consequences? Promoting Environmental Results Through Evaluation ------- We draw on information derived from these three sources, as well as a number of other sources tailored to specific evaluation questions.2 Data sources, data collection instruments, the evaluation methodology and other relevant documentation can be accessed atwww.paintstewardshipprogram.com. COLLABORATION A primary goal of the PPSI was that the pilot program, from planning to implementation, was to be a collaborative process. Most stakeholders (84 percent) involved in the PPSI agreed that the program development process was collaborative (Braunz et al., 2010). Stakeholders also pointed to break downs in collaboration when the program transitioned from designing and planning the PPSI pilot program to drafting Oregon legislation and planning and implementing the Oregon program. The level of importance and function of collaboration changed through the different stages of the program's development. To account for changes, process facilitators can better maintain appropriate levels of collaboration throughout the process by, at the beginning, coming to agreement on and documenting clear expectations for collaboration's role in each stage of the process. * PAINT STEWARDSHIP ORGANIZATION A goal of the PPSI was for a pilot program to create a Paint Stewardship Organization (PSO) that would operate under the direction of the paint industry and this was achieved in the Oregon legislation that created the program. The PSO for the Oregon program, PaintCare, is a coalition of paint producers and is operated by the American Coatings Association (ACA), although membership in ACA is not required for a paint producer to be part of PaintCare. PaintCare is responsible for implementing and running the program in Oregon, a responsibility which they contracted out to Product Care Inc., which has run other stewardship programs. The PSO built the infrastructure (e.g., paint collection sites, logistics, and transportation) using the existing infrastructure in the Portland metro area. Having this existing infrastructure offered significant advantages in implementing this program. Though generally perceived as clearly denned and complete, the lack of detailed, accessible cost information reduced the transparency of the funding mechanism. Few Oregon residents (11 percent) that purchased paint in 2011 were aware that a fee was added to the cost of paint to pay for the program (PaintCare, 2 01 la). jg±. EDUCATION AND OUTREACH The Oregon program included an extensive education and outreach program which was another PPSI goal. The campaign consisted of a website, point of sale materials, radio and television advertising, press releases, direct mailings to contractors and retailers, an 800 number, and participation in trade shows. The education and outreach campaign used a broad approach to spreading its messages and did not strategically aim its messages at specific target audiences (consumers with the most paint, contractors, retailers, etc.) with specific means of communication (TV, website, social media, radio, etc.). Consumers of paint can be segmented into various groups (new versus long-time homeowners, homeowners versus contractors, age and other demographics). Opportunities to improve the effectiveness of education include identifying the desired changes in behavior that will accomplish program goals, designing specific messages to achieve those changes, (e.g., aligning the messages with the program's priorities on the waste hierarchy), prioritizing efforts allocated to specific messages, and more explicitly targeting those messages to the diversity of intended audiences (e.g., age groups, location, consumer vs. contractor). Overall, there was little evidence that education and outreach influenced consumer behavior. More useful evidence of the effectiveness of education initiatives requires more complete and transparent information such as data related to consumer exposure to messaging and consumer intentions and practices related to paint management (e.g. purchasing, reuse, recycling, disposal). 2 For example, Evaluation Question 1 draws on a survey conducted by Duke University graduate students and Evaluation Question 2 draws on a set of interviews conducted by Georgia State University graduate students. Evaluation of the Oregon Paint Stewardship Program ------- Collections by type of paint Latex 352,136 75% residents (92 percent) felt this type of program was "somewhat important" or "very important." Oregon residents' acceptance of the fee may be linked to the inherent value they place on the goals and services provided by recycling programs. The degree to which consumers value such programs can be used as an indicator of their willingness to pay a certain fee. This information may be used in refining fee structures within and across states. Documenting and communicating consumer attitudes may also be useful in navigating the legislative process necessary to create programs similar to Oregon's. 469,665 total gallons collected ^ CONSUMER PURCHASING DECISIONS A goal of the program was to encourage consumers to purchase the correct amount of paint thereby reducing the amount of leftover paint that must be managed. Consumers indicated that the program had little impact on their purchasing decisions. Most Oregon residents (93 percent) that purchased paint under the program indicated that the program had no effect on the amount of paint they purchased. PaintCare developed a "paint calculator" to help consumers determine the amount of paint that was needed for painting projects, but few consumers were aware of the tool and very few actually used it. Overall, the program focused less on the reduction of leftover paint and more on recycling leftover paint. To focus planning and implementation on reducing the generation of post-consumer architectural paint, paint management programs should more explicitly document the relative emphasis they place on aligning the paint management system with the existing waste hierarchy (reduce, reuse, recycle). A clear articulation of program priorities will provide a framework to organize and deploy outreach strategies, messages, materials, and effort. In terms of the fee charged, 73 percent of Oregon residents felt that the fee was reasonable. Most COLLECTION OF POST-CONSUMER PAINT The largest component of the program involved collecting and processing leftover paint in Oregon. In the first year of the program, PaintCare collected 469,665 gallons of paint. Of this total, 352,136 gallons (75 percent) were latex paint and 117,529 gallons (25 percent) were oil-based paint (PaintCare, 2011b). As of September 1,2011, the program had 98 collection sites with 10 of these sites offering paint exchange. Sites are open to the public, on average, six days a week for a total of 58 hours per week (PaintCare, 2011b; Em2). Most Oregon residents (92 percent) live within a 15 mile drive of a paint drop off site (Strickland, 2011). PAINT REPROCESSING, RECYCLING, AND ENERGY RECOVERY In its first year, potential end-points for paint collected by the program included: recycling into another paint product, recycling as a non-paint product, energy recovery (oil-based paint), appropriate disposal and direct reuse by consumers. Table ES-1 summarizes the volume of paint diverted to each end-point. ^ HOUSEHOLD HAZARDOUS WASTE (HHW) PROGRAMS HHWs are a key stakeholder in the paint management system. Prior to the program, HHWs would take in paint from consumers and process it for disposal. Under the program, HHWs collected and stored paint for eventual pick up by PaintCare's transportation contractor. At the start of the program, Promoting Environmental Results Through Evaluation ------- Table ES-1: Summary of Paint Endpoints Endpoint Latex (Percent of Total) Recycling to paint and paint reuse 211,281 (60%) Oil-Based (Percent of Total) 3,526 (3%) Reprocessing into non-paint product 28,171 (8%) Energy recovery 14,085(4%) 114,003(97%) Disposal 98,598 (28%) Totals 352,136 117,529 the amount of paint that HHWs collected increased and then settled back to pre-program levels. The initial bump in collections is attributed to consumers bringing in older paint stored in their homes. During the program, the proportion of latex paint collected by HHWs increased relative to oil-based paint. Some HHWs reported cost-savings resulting from avoided labor, disposal, and transportation costs associated with handling oil-based and unusable latex paints. COST EFFECTIVENESS The program processed paint at $7.03 per gallon (PaintCare, 2 Ollb), although that amount excludes costs incurred by HHWs to take in and handle the paint at their facilities. The $7.03 per gallon value was lower than other estimates from other sources, although the other estimates were not purely comparable. The use of "processing cost per gallon" as a measure of cost-effectiveness is limiting. A more comprehensive measure would translate the gallons into environmental benefits and then also include other management options (e.g., reuse) as well as the cost-effectiveness of reducing leftover paint by increasing the proportion of consumers that "buy the right amount." WASTE HIERARCHY The Evaluation Committee is still assessing how the program was designed and implemented to move consumers up the waste hierarchy. The Committee has partnered with a graduate student at Tufts University to investigate this evaluation question and results will be reported January 2012. The method for answering this question will involve relating the components of the program to the categories of the waste hierarchy (reduce, reuse, and recycling) and then compiling information on the obstacles, opportunities, decisions, and relative emphasis related to each category of the waste hierarchy. MARKET FOR POST-CONSUMER PAINT To improve the management of leftover paint, the PPSI determined that the pilot program should explore means to expand the market for post-consumer paint products. If more consumers used post-consumer paint, then they would generate less waste because less new paint would be purchased (and ultimately need to be disposed). About half of the total volume of paint collected (217,157 gallons) under the program made it into the post-consumer paint market. For latex paint collected by the program, 60 percent was available to the post-consumer paint market. Very little oil-based paint (3 percent) was available to the post-consumer paint market. * TRANSFERABILITY The Evaluation Committee identified several types of information that other states will need to collect in order to implement paint stewardship programs, including: volume data on paint, information on current infrastructure, a system map, cost information, and information on consumer awareness of infrastructure and consumer behavior and attitudes. The Committee also identified the aspects of the performance measurement system and evaluation that are Evaluation of the Oregon Paint Stewardship Program ------- transferable to other states, which included: the pilot program's evaluation questions (appropriately adapted to other states), matrix of performance measures and a web-based program model for communications with diverse audiences (see Figure ES-1). Oregon and other states can more systematically and consistently learn, improve and communicate about paint recycling programs and other product stewardship initiatives by: 1) using (and adapting) the program evaluation framework and methodology designed and implemented by the PPSI Evaluation Committee, and 2) collaborating to adopt common fundamental frameworks for ongoing performance management such as those developed by the Conservation Measures Partnership's (CMP's) Open Standards for the Practice of Conservation.3 UNEXPECTED RESULTS The Evaluation Committee identified some key unintended effects and unexpected scenarios that significantly influenced program planning and implementation. Highlights include: (1) collaboration took more time, effort and resources than expected and levels of collaboration fluctuated significantly at different stages in the process; (2) paint legislation was vetoed twice in Minnesota which delayed implementation of a program by more than a year-and-a-half; eventually leading to implementation in Oregon; (3) though the program was originally intended to be voluntary, legislation was necessary to implement the program;4 (4) the PPSI had limited representation from retailers leading up to the program, but retailers became a critical component of Oregon paint collection; (5) the misalignment of the program's goals and the waste hierarchy model (reduce, reuse, recycle); and (6) that retailers serving as collection locations had a negative impact on the program's ability to divert high quality leftover paint for reuse. 3 www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_Version_2.0.pdf. 4 Legislation was needed to exempt the paint producers from anti-trust and collusion concerns. At the start, the idea of the program was to avoid having states pass legislation or rules to tell the private sector what to do - the program was supposed to be voluntary. Promoting Environmental Results Through Evaluation ------- i.o Introduction Over the last several decades, paint manufacturers have dramatically reduced the environmental impacts of leftover paint, including the reduction of toxic ingredients found in leftover paint. Paint manufacturers have also improved the performance of latex paints to the point where they now constitute 80 percent of paint sales, replacing a large portion of the market for oil-based paints, which have more significant environmental impacts (PaintCare, 2010). Additionally, the paint industry works with retailers to educate consumers on how to more accurately estimate the amount of paint consumers need, thereby reducing the quantities of leftover paint. Despite these improvements, environmental impacts from leftover paint remain significant. Households generate some 75 million gallons of leftover paint each year, which amounts to about 10 percent of the amount of paint they purchase (Abt, 2007). When disposed in landfills in liquid form, leftover latex paints can contaminate groundwater, thereby harming fish and other aquatic life. Furthermore, leftover oil-based paint is considered hazardous waste due to its combustibility and high solvent content. Managing leftover paint is also costly to local governments; paint is the largest volume of waste collected by household hazardous waste (HHW) programs.5 The estimated cost to manage leftover paint is more than $8 per gallon (SCS and Cascadia, 2007). While leftover paint, as currently managed, is associated with negative environmental impacts, it also represents an opportunity for reducing the life cycle environmental impacts of paint production. When manufacturers use leftover paint in their paint production processes, the environmental impacts associated with making paintincluding materials extraction, processing, and end-of-life management are reduced. By recycling paint containers, manufacturers can further reduce the life-cycle impacts of paint products (PSI, 2004a). The movement towards product stewardship for post-consumer paint began in 2002, when, due to the high cost of managing leftover paint, a number of states began to consider mandating paint management. In 2002, a coalition of paint manufacturers, represented by American Coatings Association (ACA);6 local, state, and federal environmental agencies; retailers; and consumer and environmental agencies formed the Paint Product Stewardship Initiative (PPSI) and began negotiations facilitated by the Product Stewardship Institute (PSI) to create an industry-managed post- consumer paint management system. After seven years of negotiations, two Memoranda of Understanding (MOUs), and comprehensive PPSI-sponsored research of the paint industry and consumers, Oregon became 5 Oregon Department of Environmental Quality, Paint Product Stewardship: www.deq.state.or.us/lq/sw/prodstewardship/paint.htm. 6 Formerly National Paint and Coatings Association (NPCA). Evaluation of the Oregon Paint Stewardship Program ------- the first state to enact a law establishing product stewardship as the preferred method for reducing the environmental impacts and costs associated with leftover paint. An early focus of the PPSI was in identifying a state where a pilot project would be implemented. PPSI identified Minnesota as a candidate for a pilot and it became the focus of PPSI efforts. The PPSI developed a work plan for implementing a pilot project in Minnesota, focusing the work plan around the following six goals (PPSI, 2008):7 Goal 1: The pilot project, from planning to implementation, is a collaborative and cooperative process. Goal 2: Establish a paint stewardship organization (PSO), which operates under the direction of the paint industry. Goal 3: Consumers (including painting contractors) generate no or less waste paint and containers. Goal 4: The statewide post-consumer paint management system should be designed to ensure that it is environmentally beneficial, economical, and convenient. With these considerations, the system should strive to use methods highest on the following waste management hierarchy: reuse, recycling (into paint or other products), energy recovery (generally applicable to oil-based paint), and proper disposal. Goal 5: Identify cost-effective alternatives for using post-consumer paint products and explore means to expand the market for products containing post-consumer paint. Goal 6: Measure and evaluate the performance of the pilot project, and ensure the results and learning that the evaluation generates are transferable and relevant to the rollout of a national post-consumer paint management system. PPSI formed an Evaluation Committee (EC) to be the lead on Goal 6 and to evaluate the extent to which all six goals were achieved. The Minnesota pilot project, however, was never implemented, as legislation required to implement the program was vetoed by the Minnesota governor at that time. Nevertheless, much of the work performed in designing a program in Minnesota was carried into the Oregon program, including the six goals described above. These six goals formed the basis of the evaluation questions developed under this project and for the design of an evaluation of the Oregon program. 1.1 Paint Product Stewardship Initiative Oregon Pilot Program As part of the broader dialogue, PPSI began designing a pilot program to demonstrate the effectiveness of an industry-led paint stewardship program and sought states considering paint stewardship program legislation to implement the program. In July 2009, Oregon became the first state to enact a paint product stewardship law which directs manufacturers of paint sold in the state to set-up and run a statewide system for the collection of post-consumer latex and oil-based paint. ACA formed PaintCare, Inc. (PaintCare), as the PSO responsible for implementing the pilot program in Oregon. The Oregon law required PaintCare to submit an implementation plan to the Oregon Department of Environmental Quality (DEQ) to be approved before the start of the pilot program. The Oregon Paint Stewardship Pilot Program Plan (PaintCare, 2010) was released for public comment in June 2010. DEQ approved the plan at the end of June, allowing the program to launch on July 1, 2010. The Oregon program is comprised of a diversity of interconnected systems, actors, and processes. The major components of the Oregon program include the PSO (PaintCare), the oversight of Oregon DEQ, the paint market, and the leftover paint management system. Figure 1 illustrates the pilot program and its primary components and is discussed in more detail in Section 2.2. Full documentation of the details of the program appear in PaintCare's Oregon Paint Stewardship Pilot Program Plan (PaintCare, 2010). Components of the Oregon pilot program include: PPSI: Conceptualization of the pilot program began with PPSI. Its goals included collaboration amongst diverse participants to 7 Available at www.deq.state.or.us/lq/pubs/docs/sw/PaintProdStewardshipPilotPlan2010June.pdf. Promoting Environmental Results Through Evaluation ------- plan, implement, and evaluate the performance of a statewide pilot to inform rollout of similar programs in other states nationwide. Program Administration: PaintCare and the Oregon DEQ are responsible for program administration. PaintCare is the industry non- profit association formed as the PSO by ACA. Oregon DEQ is the state agency charged with approving the PSO program plan, including the recovery fee, and responsible for program oversight and enforcement. PaintCare receives its funding from the paint market and uses funding for administration, supporting DEQ, running the paint management system, and providing education and outreach to consumers and Oregon residents. Paint Market: The paint market consists of three main participants: Producers of paint sold in Oregon are obligated under the law to participate in a paint stewardship program. Program participants are required to register and provide a list of paint brands manufactured. Remittance of recovery fee payments on all architectural paint sold in Oregon is a responsibility of the producer. Figure 1 - Graphic model of the Oregon Paint Stewardship Pilot Program Retailers of paint sold in Oregon are required to collect the recovery fee from consumers at the time of sale. Retailers pay the producer the fee as part of the invoice for the paint offered for sale (i.e., retailers pay the producers). Additionally retailers can volunteer to be collection sites for leftover paint. They also collect the fee from consumers and then pass that fee back to producers. Consumers of paint eligible (i.e., leftover architectural paint in 5 gallon containers or less) for the paint management system can be residential purchasers, trade painters, institutions, and businesses. Paint Management System: PaintCare funds the system and has coordinated with retailers, HHWs, and other groups to collect paint and set up vendors for the collection, exchange, transportation, consolidation, and reprocessing of leftover paint. Evaluation of the Oregon Paint Stewardship Program ------- 1.2 Evaluation of the Pilot Program A key element of the PPSI MOU was the inclusion of an evaluation of the pilot program to inform the development of programs in other states and improve the Oregon program. The work plan developed for the Minnesota demonstration project included six goals for pilot program implementation (see text box on page 5). The Oregon law required the PSO to report certain performance measures to DEQ (e.g., volume of paint collected; program costs) by September 1, 2011. In turn, Oregon DEQ was required to report to the Oregon Legislature results from the program and recommendations for improvement by October 1,2011. The PPSI formed an Oregon pilot program Evaluation Committee composed of industry representatives, Oregon stakeholders, the U.S. Environmental Protection Agency (EPA), and rollout states (i.e., states working with PPSI to also enact paint product stewardship legislation) in September 2009. As the evaluation results provide valuable information for implementing post-consumer management systems for various consumer products, EPA took the lead and provided contractor support. EPA facilitated the Evaluation Committee's work by managing an "Evaluation Team" that included EPA and its contractors, PSI, and academic partners. The role of the Evaluation Committee was similar to that of a technical working group; the members of the Evaluation Committee contributed technical expertise and guidance by participating on conference calls and through review of the work performed by the Evaluation Team. In late 2009 and into 2010, the Evaluation Committee focused on providing input on the Evaluation Team's development of the evaluation methodology.8 During the fall of 2010 through the summer of 2011, the Evaluation Team implemented the evaluation methodology. This report documents the results of the evaluation and should be considered a companion document to PaintCare's 2011 Oregon Pilot Program Annual Report (PaintCare, 2011b), which was published to meet its reporting obligations.9 This report is intended to achieve PPSI Goal 6 and provide input to Oregon DEQ, as well as present independent evaluation findings and learning for interested parties. This report focuses on providing the results of 12 evaluation questions. Chapter 2 provides a brief overview of the evaluation method, while subsequent sections focus on individual evaluation questions. In addition to this report, the Evaluation Committee also created a web-based model of the Oregon program to present the Evaluation Committee's work, which is found at www.paintstewardshipprogram.com. 8 Available at http://paintstewardshipprogram.com/images/pe_evaluation_methodology_10_14_10.DOC. 9 Available at http://paintstewardshipprogram.com/images/2011-09-01 _Oregon PaintStewardship Pilot Program Annual Report.pdf. Promoting Environmental Results Through Evaluation ------- Pilot Program Goals (PPSI, 2008) Goal 1: The pilot project, from planning to implementation, is a collaborative and cooperative process. Goal 2: Establish a paint stewardship organization, which operates under the direction of the paint industry. Goal 3: Consumers (including painting contractors) generate no or less waste paint and containers. Goal 4: The statewide post-consumer paint management system should be designed to ensure that it is environmentally beneficial, economical, and convenient. With these considerations, the system should strive to use methods highest on the following waste management hierarchy: reuse, recycling (into paint or other products), energy recovery (generally applicable to oil-based paint), and proper disposal. Goal 5: Identify cost-effective alternatives for using post-consumer paint products and explore means to expand the market for products containing post-consumer paint. Goal 6: Measure and evaluate the performance of the pilot project, and ensure the results and learning that the evaluation generates are transferable and relevant to the rollout of a national post-consumer paint management system. Evaluation Committee Members Abby Boudouris Oregon Department of Environmental Quality Tom Metzner Connecticut Department of Environmental Protection Scott Cassel Product Stewardship Institute Lou Nadeau Eastern Research Group, Inc. Cynthia Dunn California Department of Resources Recycling and Recovery Jim Quinn Metro Parks and Environmental Services Department Barry Elman Environmental Protection Agency Steve Sides American Coatings Association Jen Holliday Chittenden County, Vermont Solid Waste District Amy Stillings Eastern Research Group, Inc. Lauren Jankovic Eastern Research Group, Inc. Theresa Stiner Iowa Department of Natural Resources Alison Keane American Coatings Association Emily Wang California Department of Resources Recycling and Recovery Matt Keene Environmental Protection Agency Robert Wendoll Dunn-Edwards Paint Scott Klag Metro Solid Waste and Recycling Jennifer Nash Harvard University Evaluation of the Oregon Paint Stewardship Program ------- 2.o Overview of the Evaluation Method and Key Data Sources To account for the new and rapidly evolving Oregon pilot, this evaluation incorporated aspects of developmental evaluation. This emerging field of evaluation involves designing and implementing an evaluation in which the program changes over time to accommodate a complex environment. The evaluation is "developmental" in the sense that it matures over time to meet the changing needs of the program (Patton, 2010). Like the PPSI, the Evaluation Committee took a participatory approach to its work, emphasizing transparency and communications and integrating measurement and evaluation (M&E) into the design and implementation of the Oregon pilot. 2.1 Evaluation Approach Program evaluators have long advocated the consideration of measurement and evaluation in the design of programs. By considering the questions that various audiences will ask in the future about the program, participants are better aware of the baseline data that will be necessary to assess program performance later. In theory, this leads to more efficient organizational learning and better management of program performance as programs evolve and mature. The Oregon pilot program integrated measurement and evaluation into program design through a series of iterative and overlapping steps (see Figure 2):10 Described the program. First the Evaluation Team engaged the Evaluation Committee to clarify the program's mission, goals, and objectives. The Evaluation Team used that information to develop a preliminary logic model, an illustration of how the program is envisioned to work, that links program activities and outputs to key audiences and outcomes. For many evaluations, the logic model is sufficient to communicate the theory of how the program is expected to work; in this case, the traditional logic model did not fully capture the interaction between the program, evaluation, and stakeholders. The Evaluation Team, with the help of a graphic designer, developed a more flexible, web-based version of the logic model (see Section 2.2 for more details). Developed evaluation questions. Working with the Evaluation Committee, the Evaluation Team drafted questions based on the pilot program's goals and objectives. The questions were designed to account for program context, key audiences, communication of results, and use of the evaluation findings. Part of the process included the Evaluation Committee's prioritization of evaluation questions. Developed measures. The Evaluation Team identified a set of measures for each evaluation question. For each measure, the team documented potential data sources, developed collection methods and strategies, identified the tools of analysis, and suggested the strategies for data collection and data management (see Section 2.3). This approach resulted in a set of measures that are highly specific to one or two of the evaluation questions in design, but may produce information that is relevant to several questions. For example, the measures of collaboration between program participants during implementation (Evaluation Question 1) provides information on the transferability 10 For a more detailed discussion of this topic, see Integrating Evaluation into Program Design (Keene, 2008). Available at: www.productstewardship.us/ displaycommon.cfm?an=1&subarticlenbr=754. Promoting Environmental Results Through Evaluation ------- Figure 2 - Iterative Steps of Integrating Evaluation into Program Design 1. Evaluation Policy 2. Evaluation Methodology 1. Team 2. Mission 3. Goals & Objectives 4. Logic Model Questions 1. Context 2. Audience 3. Communication 4. Use 1. Data Sources 2. Collection Methods & Strategy 3. Analysis Tools 4. Data Collection 5. Data Management of the program to other states (Evaluation Question 11) and to the unexpected results of collaborative work (Evaluation Question 12). Documented the process and results. The methodology documents the decisions with respect to evaluation questions, measures, and communication strategies. Note that the process is iterative; the final stage of documentation feeds into program implementation and the program management cycle. This report documents the Evaluation Committee's methods, data sources, findings, and key learning points with respect to each evaluation question. Integrating evaluation into the design of the Oregon pilot program also makes the evaluation more amenable to a participatory approach. A participatory evaluation can be defined as "a partnership between trained evaluation personnel and practice-based decision makers, organization members with program responsibility or people with a vital interest in the program" (Cousins and Earl, 1992). By working with program designers, personnel, and stakeholders to incorporate evaluation into the program design, the program includes these individuals in the evaluation process from the start. This approach ensures that the evaluation results are "accessible and important to users and responsive to their needs while maintaining sufficient technical quality" (Cousins and Earl, 1992). The evaluation stakeholders are involved in the creation of evaluation questions, data sources, and analysis, which in turn ensures that the findings of the evaluation are open to stakeholders and the public. In other words, stakeholders and the public have access to data that are collected and analyzed by the Evaluation Committee, as Evaluation of the Oregon Paint Stewardship Program ------- well as the learning derived from the analysis, and are free to make their own conclusions about the findings of the analysis. The goal of a participatory approach, when compared to traditional evaluation methods, is to encourage participant ownership of the work and the results of the evaluation. This ownership of the results by the program participants fuels organizational learning, decision making, and improvement (Cousins and Whitmore, 1998). A participatory approach allowed the Evaluation Team to distribute the costs of the evaluation among multiple organizations and tap into specialized skills and other resources to improve the quality of the evaluation findings. For example, the Evaluation Committee invited several academic researchers to become "partners" in the evaluation: Amy Braunz, Julie Colvin, Whitney Knapp, and Hedrick Strickland, graduate students at Duke University's Nicholas School of the Environment completing their masters degrees in environmental management, implemented a survey to address Evaluation Question 1 (collaboration). These students were participating in Dr. Randy Kramer's course in survey methods, in which students are required to develop and implement a survey. The students coordinated with Indiana University faculty member Anne Marie Thompson, a leading expert in the measurement of collaboration. Wes Bledsoe, Eric Graves, and Andrei Roman, graduate students at Georgia State University (GSU) participating in Dr. Cynthia Searcy's applied capstone course in evaluation performed work under Evaluation Question 2. Hedrick Strickland, one of the Duke University graduate students mentioned above, based her master's thesis on the application of geographic information systems (CIS) analysis to the Oregon project, including descriptive information, visualization, and analysis relevant to the evaluation. Ms. Strickland addressed questions of the convenience and efficiency of the infrastructure of the program for consumers and transportation. Oregon Pilot Program Stakeholders American Coatings Association* U.S. Environmental Protection Agency* Household Hazardous Waste Facility Operators* Oregon Department of Environmental Quality* Oregon Legislature Paint Producers* Paint Retailers Product Stewardship Community* Roll-out States* * Represented on the Evaluation Committee Brielle Kissell, a graduate student at Tufts University, based her masters thesis on the exploration of Evaluation Question 9, which relates to the waste hierarchy. There are limitations to using a participatory approach to evaluation. Inviting multiple participants and stakeholders into an evaluation increases logistical challenges. In a traditional approach, an external, impartial evaluator controls the timeline and method of the evaluation as well as the presentation of the results. In a participatory approach, the evaluator coordinates with a diversity of stakeholders and other participants in the evaluation process, as well as other parties with a direct interest in the evaluation results. Coordination of multiple parties and reconciliation of competing viewpoints and priorities can be time-consuming. 2.2 Communicating Information The complexity of the program and the participatory approach to the evaluation demanded an innovative method to communicate about the evaluation to multiple, diverse audiences throughout program design and implementation. Logic models can be an important tool for describing a program's theory Promoting Environmental Results Through Evaluation ------- and assumptions, designing evaluation questions and methods, and convening and communicating with stakeholders about program and policy effectiveness. The development and use of logic models, however, is often limited to the initial stages of the evaluation process and to clearly defined programs. Generally, logic models can be described as linear, bounded, relatively static, and rarely designed for communications with the general public. Trends in performance management in the public sector are pointing toward greater transparency and accountability, systems thinking, participation and empowerment of diverse stakeholders, adaptive management, evidence-based practice, and better collaboration and communications with a wider range of audiences. To complement the participatory nature of the PPSI and the evaluation, the Evaluation Team developed a web-based "fuzzy" logic model (see Figure 1) with the intent of expanding the accessibility and use of the evaluation (and program) and the evaluation process to a greater diversity of stakeholders over a longer period of time. A fuzzy logic model embraces fluid and approximate reasoning and varied context and assumptions with the aim of improving the capacity of models of program theory to navigate non-linearity, feedback loops, adaptive agents and other agents and other traits common in the lifecycle of environmental programs and policies. Integrating Web 2.0, graphic design, systems theory, and data visualization with traditional logic models gives the evaluator the capacity to embed an unlimited type and quantity of content into a Web-based model of the program. The model of the PPSI's evaluation of the Oregon Paint Stewardship Pilot Program, found at www. paintstewardshipprogram.com (referred to as the "evaluation website" in this report), is designed to be used in tandem with this report. It describes both the key elements of the pilot program (e.g., infrastructure, funding) as well as the evaluation. For example, evaluation questions are illustrated as numbered paint "splatters" on the part of program that they are evaluating. Clicking on an evaluation question brings up a pop-up box that provides additional information about that evaluation question, such as the question and purpose, measures, data sources and methods used to answer the question, key findings, and Web links to supporting materials and resources (e.g., reports and data). 2.3 Evaluation Design The Evaluation Team's evaluation methodology provides a detailed discussion of the process used to develop, prioritize, and refine the set of evaluation questions, as well as a description of the development of the measures and data sources for each question. Table 1 lists the evaluation design matrix developed for the Oregon pilot program evaluation. The table presents each evaluation question, the measures used to answer the question, and the data sources and analytical methods used to develop the measures. Each question is provided with its own section (beginning with Section 3.0) in this report. Each question-specific section begins with an overview of the data sources, methods, and analysis used to answer the question and then lists findings and learning points. In the context of each question and given the findings, the learning sections aim to answer the following three questions: How do the findings affect the program? What do the findings tell us? How do the findings affect the program's capacity to achieve its goals? What is the impact on the program being able to achieve its goals? What are the opportunities to improve program performance for both the Oregon program and other programs in the future? Each the evaluation questions covers a diverse and broad set of concepts. Thus, when presenting the results in the sections that follow, we use the above considerations as a guide, but we also adapt our presentation according to the information in hand. Evaluation of the Oregon Paint Stewardship Program ------- Table 1 - Evaluation Design Matrix Evaluation 1 . To what degree was the pilot program, from planning to implementation, a collaborative process? 2. Describe the PSO, including its funding mechanism and infrastructure. Key Audiences Measures EPA PSI Roll-out states Manufacturers and Retailers PPSI Oregon Legislature Oregon DEO PPSI Roll-out states PSI Retailers Product stewardship community Governance scale based on the questions provided in Table 1 of Thompson, Perry, and Miller (2007) Administration scale based on the questions provided in Table 1 of Thompson, Perry, and Miller (2007) Autonomy scale based on the questions provided in Table 1 of Thompson, Perry, and Miller (2007) Mutuality scale based on the questions provided in Table 1 of Thompson, Perry, and Miller (2007) Norms scale based on the questions provided in Table 1 of Thompson, Perry, and Miller (2007) Connectedness measures Narrative of PSO development and operation Factors affecting infrastructure choices GIS representation of infrastructure in relation to demographic information Clarity Transparency Completeness Data F Interview or survey of those involved in process Survey of network participants (program organizations) PSO documents; Interviews with Oregon DEO staff, PaintCare staff One-time analysis Baseline, follow-up at 1 2 to 1 8 months Within 6 months of the program start Promoting Environmental Results Through Evaluation 10 ------- Evaluation Key »*,«,«...««. Data c^«. .«.. Question Audiences Measures Sources Frequency 3. How did education materials and strategies affect consumer awareness and behavior? 4. How has the program affected consumers' purchasing decisions and management of paint prior to drop- off at paint recycling facilities? Oregon DEQ Oregon Legislature HHWand other paint collectors PPSI Roll-out states Product stewardship community Oregon DEQ Retailers Manufacturers PPSI Roll-out states Product stewardship community Program awareness Program scope awareness Site location awareness Improper disposal/ handling environmental impact awareness Consumer use of best practices for purchasing paint (percentage) Consumer-reported leftover paint amount Consumer use of leftover paint Disposal practices Storage practices Recycling practices Awareness of pilot program education and outreach messages. Consumer use of best practices for purchasing paint Consumer-reported leftover paint amount Disposal practices Storage practices Recycling practices Fee awareness Behavioral reaction to fee Consumer surveys, focus groups, or interviews Consumer surveys, focus groups, or interviews Baseline, follow-up at 12 to 1 8 months Baseline, follow-up at 12 to 1 8 months Evaluation of the Oregon Paint Stewardship Program ------- Evaluation Key Question Audiences 5. How has the program affected the collection of post-consumer paint in terms of volume, cost, quality, environment, convenience, and infrastructure? 6. How has the program affected used paint reprocessing, paint recycling, and paint-related energy recovery in terms of volume, infrastructure, and cost? PPSI Oregon DEQ Oregon Legislature Roll-out states Retailers Product stewardship community PPSI Oregon DEQ Oregon Legislature Roll-out states Retailers Product stewardship community Measures Gallons collected Containers collected Cost per gallon Reductions in amounts of paint that could impact the environment Average distance traveled to drop-off point Total number of hours open Average distance to drop-off locations Number of permanent collection sites Number of periodic collection opportunities Number of sites offering paint exchange Changes in retailers' practices Gallons per trip Miles per trip Transportation cost per gallon transported Volume of latex paint exchanged Volume of latex paint recycled into paint products Volume of latex paint recycled into non-paint products. Sources Frequency Data collected and tracked by the program Surveys or anecdotal evidence at drop-off locations to ask about distance traveled Data collected and tracked by the program Data collected and tracked by the program and Oregon DEQ prior to program Data collected and tracked by the program Interviews with retailers Data collected and tracked by the program Data collected and tracked by the program Baseline, monthly Annually or more frequently if possible Based on the volume measures being collected Baseline, follow-up Annually or more frequently if possible Baseline. Annually or more frequently if possible. Promoting Environmental Results Through Evaluation 12 ------- Evaluation Key »*,«,«...««. Data c^«. .«.. Question Audiences Measures Sources Frequency 6. How has the program affected used paint reprocessing, paint recycling, and paint-related energy recovery in terms of volume, infrastructure, and cost? 7. What was the impact of the program on the HHW facilities in terms of the types of paint collected, costs, and the way in which the facilities operate? 8. How cost effective is the program? 9. How was the program designed and implemented to move consumers up the waste hierarchy? PPSI Oregon DEQ Oregon Legislature Roll-out states Retailers Product stewardship community PPSI Roll-out states Oregon local governments PPSI Oregon DEQ Roll-out states Product stewardship community PPSI Roll-out states EPA Retailers Product stewardship community Volume of latex paint appropriately disposed in landfill Volume of oil paint exchanged Volume of oil paint recycled into paint products Volume of oil paint appropriately disposed Number of facilities capable of performing each type of management method Capacity of facilities performing each type of management method Processing cost per gallon for each management method Gallons of different types of paint collected at HHW facilities Changes in the amounts of or capacities for other products by HHW Cost for HHW facilities to take in and process paint Cost of managing other products Cost per gallon collected at drop-off facilities Cost per gallon exchanged, recycled, or used for energy Total program cost per gallon collected Evaluation Committee assessment of hierarchy attainment Amounts of paint processed in the program that fall into each category Data collected and tracked by the program Data collected and tracked by the program Interviews with HHW facilities Data collected and tracked by the program and participants Program materials and interviews with program staff Data collected and tracked by the program and participants Baseline. Annually or more frequently if possible. Annually or more frequently if possible Annually or more frequently if possible Collected after the program has run for some time Collected after the program has run for some time 13 Evaluation of the Oregon Paint Stewardship Program ------- Evaluation Key Question Audiences 1 0. How has the market for post- consumer paint been affected by the program? 1 1 . Based on the Oregon experience, what implementation and outcome- related information is required for other states to develop and implement leftover paint management systems? 12. During the program and for each of its primary components, what were the primary external, unexpected and/ or unintended influences and consequences? PPSI Retailers Manufacturers PPSI Roll-out states EPA PSI Product stewardship community PSI Roll-out states Product stewardship community Measures Different products offered Number of facilities (outlets) offering post- consumer paint. Total sales (dollars) of post-consumer paint products. Total amount of paint distributed for re-use from HHW sites Description of implementation and outcome-related information that are required for states Transferability to other states Best ways to communicate results of the evaluation Lists of external influences and outcomes Sources Frequency Interviews with Oregon DEO, PaintCare, HHW programs Interviews with program staff. Qualitative assessment by Evaluation Team Interviews, all available data Annual Within 12 months of program implementation Collected after the program has run for some time 2.4 Key Data Sources Used in the Evaluation The evaluation team relied on many data sources provided by our evaluation partners, the PPSI, and PSI, with the primary sources including: Two surveys of consumers In-person interviews and other personal communications with program stakeholders Documents developed by PaintCare (the PSO) 11 For example, Evaluation Question 1 draws on a survey conducted by Duke University graduate students and Evaluation Question 2 draws on a set of interviews conducted by Georgia State University graduate students. Both of those sources of information are discussed with their respective evaluation questions. Throughout our findings on the evaluation questions, we draw on information derived from these three sources. Where other data or information is used in answering an evaluation question,11 we describe that information in our discussion of the evaluation question. Promoting Environmental Results Through Evaluation 14 ------- 2.4.1 Consumer Surveys Two surveys of consumers, one conducted in 2010 just after program rollout (Bradshaw, 2010) and another conducted nearly one year later (PaintCare, 2011b), provide key information for Evaluation Questions 3 and 4 and supplementary data for other questions. Bradshaw Advertising conducted the first survey, a telephone survey of 409 Oregon residents from August 4 to 10, 2010, for PaintCare.12 We refer to this survey as the "August 2010 survey" throughout this report. The survey was conducted using random sampling that resulted in a margin of error of 4.8 percent at the 95 percent confidence level. The survey targeted having at least 80 percent of respondents in the 25 to 64 age range with no more than 10 percent in either the 18 to 24 or 65 or older age ranges. Overall, the survey focused on consumer awareness of the newly implemented program; specifically, the survey asked consumers about: How and where they would likely dispose of unneeded paint. The extent to which they are aware of the program. How they learned about the program. The importance of this type of program. The reasonableness of the fee. How likely they are to recycle paint. The second survey was a web-based survey of 1,064 Oregon residents over 18 years old, conducted from July 7 to 12, 2011 by PaintCare through the ACA paint consumers panel. We refer to this as the "July 2011 survey" throughout this report. The July 2011 survey was conducted using random sampling with a margin of error of 3 percent. This survey targeted individuals who had purchased paint between July 1, 2010 and June 30, 2011, as well as individuals who paint as an occupation. This survey included questions to consumers about: Whether or not they recall hearing or seeing advertisements for the program. Where they saw or heard the advertisements. Whether or not they are aware of the fee. If and where they have purchased paint in the past year. The impact of the fee on their paint purchase. The types of products covered by the program. Awareness of leftover paint drop-off locations and the distance to the location from their home. Perceived convenience of the drop-off locations. Whether or not they had recalled seeing and using a paint calculator to purchase paint. The influence of program information on the amount of paint purchased. How they managed their leftover paint from their most recent project. 2.4.2 Interviews and Other Personal Communications The evaluation team conducted 21 interviews during the course of the evaluation; including seven interviews with HHW representatives, three interviews with retailers, and 11 interviews with key program stakeholders. Additionally, the team followed up with many of the interviewees for clarification of key points and data and received clarifying responses in a number of email correspondences. Each interview had a different focus: The HHW interviews focused on the impact of the program on the management of HHW programs, facilities, and collection events in terms of paint volume collected, costs to collect paint, and ability to collect other HHW products. 12 Full survey results are available at www.productstewardship.us/displaycommon.cfm?an=1&subarticlenbr=754. 15 Evaluation of the Oregon Paint Stewardship Program ------- The interviews with retailers explored the impact of the program on retailers with a focus on the process of implementing the program, the decision to act as a collection site or not, and key challenges faced. Interviews with program stakeholders did not follow a set pattern and were often customized to the interviewee based on the timing of the interview. For example, the evaluation team asked representatives of the PSO about challenges faced during program implementation, lessons learned, and availability of data. A full list of the interviews and personal communications that were used in this report appear in Section 15.0. When citing a specific interview in the text we used an "S" followed by the interview number. Similarly, we use an "Em" followed by a number to cite an email correspondence.13 2.4.3 Paint Stewardship Organization Documents Two PSO documents provided integral information about the program: the 2010 Oregon Paint Stewardship Pilot Program Plan (PaintCare, 2010) and the 2011 Annual Report (PaintCare, 2011b). The Program Plan provided the following information about the program: Oregon paint stewardship legislation. Stewardship organization and participation of producers. Program products. Registration of producers and brands. Product sold and available for collection. Program budget and funding. Outreach and education. Collection system. Environmental regulatory requirements. Transportation. Sound management methods. Program reporting and audit. The Annual Report provided the following types of data: The methods used to collect, transport, recycle and process post-consumer architectural paint in the state. The volume and type of post-consumer architectural paint collected in all regions of the state. The volume of post-consumer architectural paint collected in the state by method of disposition, including reuse. An independent financial audit of the program. A description of program costs. An evaluation of the operation of the program's funding mechanism. Samples of education materials, evaluation of the methods used to disseminate materials, and assessment of the effectiveness of the materials. An analysis of the environmental costs and benefits of collecting and recycling latex paint. 13 See Section 15.0. Promoting Environmental Results Through Evaluation 16 ------- 3.o Collaboration EVALUATION QUESTION: To what degree was the pilot program, from planning to implementation, a collaborative process? How was the collaborative process viewed by different groups involved in the process? What tools and strategies (including communication) were used to foster collaboration and how effective were those tools? PPSI's first goal for the pilot program was for the program to be collaborative and cooperative, to potentially serve as a model of governance that EPA, states, and product stewardship initiatives can use to achieve environmental and human health policy objectives. A key point to understand about collaboration under this program is that the collaborative effort started well before the Oregon program was developed. PSI was involved in developing a national dialogue about paint stewardship starting in 2002. In fact, PSI intended the entire process to be collaborative from start to finish (S19). 3.1 Data Sources, Methods and Analysis Collaboration may be defined as "a process in which autonomous or semi-autonomous actors interact through formal and informal negotiation, jointly creating rules and structures governing their relationships and ways to act or decide on the issues that brought them together; it is a process involving shared norms and mutually beneficial interactions" (Thompson et al., 2007). Thompson et al. (2007) developed a set of five dimensions of collaboration, which framed the Evaluation Committee's approach to data collection, analysis and interpretation: Governance: The degree to which there is joint decision-making by the collaborative parties about rules to govern the relationship. Administration: The extent to which a structure that moves the collaboration from governance to action exists. Organizational autonomy: The extent to which each party to the collaboration retains autonomy in the relationship. Mutuality: The extent to which parties to the collaboration experience mutually beneficial interdependencies. Norms: The extent to which there is reciprocity and trust between the collaborating parties. Thompson etal. (2007) developed these scales through a systematic review of the literature, synthesis of theoretical definitions of collaboration, a series of interviews with organizational directors on collaboration, and detailed case study research. They identified a set of 10 to 12 indicators for each dimension and constructed a survey question to measure each indicator; each indicator was phrased as 17 Evaluation of the Oregon Paint Stewardship Program ------- a statement and the survey respondents were asked to rate the extent to which their organizations engage in each behavior. The rating was designed as a scale from 1 ("not at all") to 7 ("to a great extent"). The Evaluation Team partnered with a team of graduate students at Duke University's Nicholas School of the Environment as part of their capstone course in survey methods to help answer Evaluation Question 1. Their primary mode of data collection was a Web-based survey of PPSI participants. The survey results were summarized in a report (Braunz et al., 2010). Braunz et al. (2010) used the Thomson et al. (2007) framework to develop a survey instrument to ask about collaboration under the PPSI. The web survey targeted current and former PPSI participants, including those involved before the start of the Oregon program. The survey's 23 questions focused on: Demographics, including organization type, role, and length of time involved. Communication tools used and communication tools preferred. Level of participation. Opinions about the collaboration (i.e., the dimensions of collaboration). Before implementing the full-scale survey, the Duke students conducted an informal focus group and performed ten pre-tests to gather feedback and clarification on the survey instrument. The questionnaire was distributed to 419 PPSI participants in November 2010 and was completed by 125 participants (30 percent response rate) (Braunz et al, 2010). In order to determine stakeholder perceptions, survey respondents were grouped as (see Figure 3): 1) local government (33 respondents); 2) state and federal government (23 respondents); 3) private sector, which combines retailers, manufacturers and other private enterprises (23 respondents); and 4) nongovernmental organization (NGO)/Non-profit/ University/Trade Associations (8 respondents). After the Duke University graduate students completed the survey, the Evaluation Team conducted a series of interviews with key stakeholders to assess the Figure 3 - Distribution of Respondents by Category to Duke University Survey of PPSI Stakeholders Local Government State Government Private Sector NGOs / Non-Profits / University/Trade Associations Other collaborative process. The Evaluation Team conducted interviews with representatives of: PSI ACA and PaintCare Oregon DEQ HHWs14 The purpose of the interviews was to provide a history of the collaborative effort and to get the opinions and feedback from key stakeholders that were involved in the process from start to finish. 3.2 Findings In presenting the findings for this question, we will first "tell the story" of collaboration under the program, and then turn to providing an assessment of the collaborative effort. 14 The interviews that were conducted with HHWs were primarily focused on providing information for Evaluation Question 7; however, the information provided by HHW representatives during those interviews was applicable to this (and other) questions. Promoting Environmental Results Through Evaluation 18 ------- 3.2.1 What tools and strategies (including communication) were used to foster collaboration & how effective were those tools? Tools and Strategies Used Prior to the PPSI, PSI, state, and local governments identified leftover paint as one of their top five waste management issues. As a result, a number of states began to consider implementing mandatory leftover paint management requirements. PSI interviewed stakeholders during 2002 to 2004 to define the problem and identify basic facts in order to start a national dialogue. Its efforts resulted in the publication of A Background Report for the National Dialogue on Paint Product Stewardship (March 2004) to provide a technical foundation for requirements (PSI, 2004a). At the same time, PSI also released Product Stewardship Action Plan for Leftover Paint (March 2004), which outlined the key issues and potential solutions related to leftover paint management (PSI, 2004b). Funding for this initial work came from a variety of sources, including most state and local governments involved in the PPSI. "Seed money" from Massachusetts Office of Energy and Environmental Affairs, an EPA grant for one meeting, and a foundation grant to partially support another meeting (S19). In December 2003, PSI facilitated the first of four in-person dialogue meetings to establish relationships and build trust among stakeholders. An outcome of these meetings was to establish workgroups to focus on specific issues (e.g., markets for recycled paint, education efforts). Industry and government stakeholders volunteered to lead some of these work groups and provide funding to support the group's efforts (S19). Throughout the process, PSI acted as a facilitating organization, supported by contributions from stakeholders after the initial grants were expended. PSI's philosophy for facilitation was based on collaboration and consensus and hence was interwoven through all aspects of the Initiative (S19). PSI's strategy to foster collaboration, implicitly adopted by PPSI early in the process, involved: Establishing the PPSI goals through a consensus process (S19). Developing a detailed work plan (PPSI, 2008) during the first year to 1) determine major tasks and timelines, 2) identify potential pilot opportunities, 3) identify staffing and funding requirements, 4) identify challenges and barriers, and 5) draft agreement language. Crafting MOUs to define purposes and scopes of projects; goals and objectives; and commitments by stakeholders (PPSI, 2007). Establishing smaller groups including15 1) a Steering Committee, 2) an Education Workgroup, 3) an Infrastructure Workgroup, 4) Market Workgroup, 5) Life Cycle Benefits and Costs Committee, 6) Demonstration Project Committee, and 7) Evaluation Committee, to work on specific projects. Providing agenda and background materials prior to meetings and calls and providing summaries of meeting and calls, allowing PPSI participants the opportunity to comment (S19). In implementing this approach, PSI relied primarily on the following three tools (S19): Monthly PPSI briefing conference calls. PPSI conferences (once or twice a year). A website featuring status of projects, reports, and posting of agenda, participant lists, meeting/call presentations, and meeting/call minutes (S19). The PPSI did not formally document the strategy and tools the PPSI would use to foster collaboration, but the intent to support collaboration is set out in several key documents. The second MOU, signed October 24, 2007, established the agreement with participants that appears in the text box on the next page. Although 15 These groups were not given decision-making authority (S19). 19 Evaluation of the Oregon Paint Stewardship Program ------- some of the items listed were not completed, the MOU did establish a framework for continued participation in collaborative efforts. The framework was continued through the development of a draft work plan for the Minnesota Demonstration project. The work plan described the consensus among participants on the mission, method, goals and objectives of the project while establishing the roles of those participants (PPSI, 2008). PSI's role in the process was not to serve as an independent facilitator, but rather to act as an objective facilitator and, at times, act as an advocate for paint stewardship. Program stakeholders agreed that PSI provided a vital role in the sharing of information and identifying state contacts (S16, S17, S18, S19, and S20). Additionally, one stakeholder felt that PSI's lack of impartiality was a benefit to the process since PSI worked actively to push the process along (S18). Effectiveness of Tools/Strategies The survey of PPSI participants (Braunz et al, 2010) asked respondents to rate the effectiveness of the periodic briefing calls and conferences. The results from the survey, summarized in Figure 4, indicate that PPSI participants viewed both the briefing calls and the conferences as being effective. "The ...PPSI participants agree to work collaboratively over the next three years to do the following: Continue the dialogue through regular meeting and workgroup conference calls. Design, implement, and evaluate a demonstration project. Pursue a voluntary multi-stakeholder approach for the demonstration project while addressing mechanisms to avoid anti-trust implications. Support and implement strategies to effectively change consumer behavior. Pursue completion of a life cycle assessment and cost benefit analysis (LCA/CBA) project. Collect baseline data on aerosol paints." PPSI, 2007 Figure 4 - PPSI Participants' Perceived Effectiveness of Briefing Calls and Conferences Briefing Calls Conferences 10 20 30 40 50 60 70 Effective Neither Effective nor Ineffective Ineffective 80 Promoting Environmental Results Through Evaluation 20 ------- 3.2.2 To what degree was the pilot program, from planning to implementation, a collaborative process? How was the collaborative process viewed by different groups involved in the process? Overall Impressions of the Process To a large extent, PPSI participants agreed that the process was collaborative (Braunz et al, 2010) (see Figure 5) with 84 percent of respondents to the survey agreeing that the process was collaborative (19 percent strongly agreeing) and only 6 percent disagreeing that the process was collaborative. The survey implemented by Duke asked the PPSI participants to rate five dimensions of collaboration identified by Thomsen et al (2007) (see page 17): governance, administration, mutuality, norms, and autonomy. Respondents were presented with a series of questions related to each dimension and were asked their level of agreement with a statement. The full details of this process can be found in Braunz et al (2010). Respondents were provided with seven options on a seven-point scale with the endpoints anchored at "strongly agree" and "strongly disagree" and the middle (4th option) being anchored at "neutral."16 Survey responses were translated to a numeric value by assigning a +1 to "strongly disagree" and increasing that value by one until a +7 was assigned to "strongly agree" (see graphic below). +1 +2 I I I I Strongly Disagree +3 I I +4 I I Neutral +5 I I +6 +7 I I I I Strongly Agree Figure 6 provides a summary of the data collected through the survey of PPSI participants for all respondents, broken out by respondent categories for the five dimensions of collaboration. The results for governance, administration, mutuality, and norms dimensions tend to be in the 4.5 and 5.5 range indicating overall agreement that the process had been consistent with those dimensions. Additionally, within those four dimensions, there is little variation between the four groups (local government, state government, private, and non-profit/NGOs). One consistency that can be found is that responses from individuals in private companies tended to be lower than those in local or state government. The dimension of autonomy (balancing competing demands of one's own organization with demands of collaborative effort), however, was rated relatively low in the survey by all respondents and by each group separately. In other words, most respondents disagreed that the process allowed them to balance the needs of their own organization and that of the collaborative effort. The survey results in Figure 6 indicate that, overall, collaboration was viewed favorably by most respondents in each dimension of collection, except for the autonomy dimension. In response to open ended questions, (Braunz et al., 2010), PPSI participants indicated that the process for achieving collaboration was costly and should have been outlined from the start so participants were informed of how they were to be engaged. Viewpoints on Collaboration over the Lifecycle of the Process When viewed from the different stages of the process, some stakeholders saw a break down in collaboration when the process moved from planning a program in general to developing the legislation for the Oregon program and in planning and implementing the Oregon program. ACA indicated that the legislation was a compromise of all the parties (S17). PSI's intention was that the entire process, including development of legislative language, would be collaborative (S19). In the survey of PPSI participants (Braunz etal., 2010), however, 10 respondents indicated that they did not see the legislation as collaborative. Six of the 42 open-ended comments on collaboration rated collaboration as lower when the Initiative moved into the implementation stage. HHW programs echoed the breakdown in collaboration. Some HHWs indicated they wished they could have contributed so the program tied in 16 The second, third, fifth, and sixth points on the scale were not provided a text anchor. Evaluation of the Oregon Paint Stewardship Program ------- Figure 5 - Agreement among PPSI Participants on the Extent to which the PPSI Process was Collaborative Neutral Disagreed Agreed at some level process was collaborative Strongly agreed Agreed Somewhat agreed Figure 6 - Likert Scale Values for Five Dimensions of Collaboration, by Respondent Category Governance Autonomy Administration Mutuality Norms 0.00 1.00 2.00 3.00 All I Local Govt I State/Federal Govt 4.00 5.00 6.00 7.00 Private Non-Profit/NGO, etc. Promoting Environmental Results Through Evaluation 22 ------- the existing solid waste infrastructure (S13, S14, and S15). One HHW representative specifically commented that they would have liked to have an opportunity to provide meaningful input into the organization of the program; they had attempted to participate early in the implementation process but felt as though they were getting in the way (S13). From PSI's perspective, the vetoes in Minnesota influenced the level of collaboration (S19). After Minnesota governor vetoed legislation twice, the group stopped work for 1.5 years and lost momentum. During that time, funding for PSI was limited, which resulted in less communication which did not change with passage of Oregon legislation. Oregon was also less involved in the PPSI's work leading up to the Oregon program (S19). Nevertheless, the Evaluation Team observed that PPSI products and artifacts of collaboration (e.g. relationships, research, formal agreements, and committees) created between 2002 and 2008 had a prominent role in influencing Oregon legislation and the resulting program plan (Oregon, 2009; PaintCare, 2010). 3.3 Learning The findings discussed in this section have led to the identification of a number of key learning points. These points include: Collaboration can provide clear benefits. At its inception in 2003, the PPSI embraced a "From the start, collaboration should be more purposeful with clearer goals for collaboration that define and drive the type, level, and timing of collaborative effort necessary from various groups and individuals...If collaboration were more directed at achieving particular goals, resources (time, money, staff) could be more efficiently distributed across the overall effort to make the PPSI more effective." (PPSI survey respondent as quoted in Braunz et all, 2010) collaborative approach to achieving its goals. Much of the work, developed collaboratively, under the Minnesota program carried over to Oregon and much of that work found its way into the Oregon legislation and plan. The collaborative effort resulted in a pilot program that reflected the needs of diverse stakeholders. This evaluation and much of the data collection and analysis are a product of the collaborative process. Furthermore, the products and artifacts of the PPSI process (e.g. relationships, research, formal agreements, goals, committees) prior to conception of the Oregon pilot provided the background and baselines for legislation and planning in Oregon. Open communication is key to good collaboration. Without open communication about fundamental changes in the collaborative process as the PPSI transitioned to the implementation stage, PPSI participants lost confidence in the process and their ability to contribute to the process. To maintain participants' confidence and engagement in the process, the anticipated level of desired and feasible collaboration should, at all stages of the process (implementation and drafting legislation) be agreed upon and documented. Collaboration has its place but that may not be at every stage of the process. Collaboration decreased during the legislative process and during program design as legislative requirements set strict timelines; during this time, ACA and Oregon DEQ worked independently of the PPSI. "...the time and resources committed to getting to this point were great and not likely replicable for other specific product areas. This...in my mind doesn't provide a viable model...for other product areas." (PPSI survey respondent as quoted in Braunz et all, 2010) 23 Evaluation of the Oregon Paint Stewardship Program ------- Sustained commitment is important. Effective collaboration requires sustained commitments to a facilitator, funding, and communication. Declining collaboration coincided with decreased PPSI funding, loss of the facilitator to lead the process, and overall reduced communications amongst PPSI participants. The facilitator may need to also be an advocate. As noted, one stakeholder felt that PSI's lack of impartiality was a benefit to the process since PSI acted as an advocate for paint management (S18). Thus, use of an impartial facilitator (S18) (i.e., one without an interest in the outcome) may be less effective at pushing the issue forward and attaining results. Duke NICHOLAS SCHOOL OF THE ENVIRONMENT The Duke University graduate student team, advised by Dr. Randy Kramer of Duke University's Nicholas School of the Environment, presented an overview and a full report of their work on Evaluation Question 1 regarding collaboration which can be found as items #21 and #22 under Additional Materialsatwww.paintstewardshipprogram.com. Strategy and roles should be defined upfront and documented. Collaboration requires the commitments of increasingly scarce resources (money, time, etc.) for local and state governments. If collaboration is determined to be a core component of achieving paint stewardship goals, an explicit strategy set to achieve clear goals (e.g., equity, cost efficiency, diversity) will help to manage expectations throughout the process. Groups need to decide early in the process if collaboration is important and for which aspects of program it serves a greater priority. During the process, open discussions led by facilitators or those requesting changes, should accompany any significant unanticipated adjustments to the level and/or type of collaboration. "Some organizations tend to work at cross purposes. There was also a point where ACA and Oregon were sealed off in negotiations and the rest of the group was sealed out." (PPSI survey respondent as quoted in Braunz et all, 2010) "...there was not collaboration by the group on the actual text of legislation." (PPSI survey respondent as quoted in Braunz et all, 2010) "In Minnesota, the collaboration was extremely effective until the issue reached the required legislation to implement it." (PPSI survey respondent as quoted in Braunz et all, 2010) "It was a collaborative effort from a planning standpoint - but has been much less collaborative in its actual implementation." (PPSI survey respondent as quoted in Braunz et all, 2010) "I don't think implementation has been a collaborative effort with the PPSI nor the legislative process. The dialogue and keeping the PPSI informed as well as the roll-out I strongly agree has been a collaborative process." (PPSI survey respondent as quoted in Braunz et all, 2010) Promoting Environmental Results Through Evaluation 24 ------- 4.0 Paint Stewardship Organization ^ A *- . EVALUATION QUESTION: b Describe the Paint Stewardship Organization (PSO) (Pa intC a re), including its funding mechanism and infrastructure. What factors contributed to its infrastructure choices? Was the funding mechanism clearly defined, transparent, and complete? What are the lessons learned? 4.1 Data Sources, Methods and Analysis The Evaluation Team answered this evaluation question primarily through a qualitative approach of reviewing PSO documents and interviews with key program stakeholders. In order to describe the infrastructure funding mechanism of the PSO as it relates to the Oregon Pilot Program, the Evaluation Team partnered with graduate students at Georgia State University (GSU) as part of their capstone project in evaluation. The GSU team created an interview questionnaire and conducted telephone interviews with eight participants who were involved in the implementation, current operation, and oversight of the Oregon Pilot Program. In addition to conducting interviews, the GSU students gathered and analyzed information from the Oregon legislation and from the Oregon Pilot Program documentation made available by the PSI, PaintCare Inc., the Oregon Department of Environmental Quality, EPA, and ERG (Bledsoe et al, 2011). Following the final report by GSU, the Evaluation Team also conducted interviews with The GSU graduate student team, advised by Dr. Cynthia Searcy developed a report for Evaluation Question 2, which can be found under Additional Materials as item #25 at www.paintstewardshipprogram.com. key stakeholders to clarify information and to provide additional details for the findings. To determine the factors that contributed to PSO infrastructure choices, the Evaluation Team analyzed four components of infrastructure: 1) education and outreach, 2) collections, 3) transportation, and 4) processing. To assess PSO funding mechanisms, the Evaluation Team gathered data from interview questions with regard to the meaning of the terms clarity, transparency, and completeness in the context of funding mechanisms. The team adopted the following working definitions: Clarity: the extent to which the funding source for the PSO is clearly defined in implementing legislation, rules, and program documentation. Transparency: the extent to which a member of the public could track how the funding is being used by the PSO. Completeness: the extent to which PSO funding covers the expenses for operating the PSO. 25 Evaluation of the Oregon Paint Stewardship Program ------- Clarity: The extent to which the funding source for the PSO is clearly defined in implementing legislation, rules, and program documentation. Transparency: The extent to which a member of the public could track how the funding is being used by the PSO. Completeness: The extent to which PSO funding covers the expenses for operating the PSO. 4.2 Findings 4.2.1 Describe the Paint Stewardship Organization (PaintCare), including its funding mechanism and infrastructure. Infrastructure: Process Actors Figure 7 provides an overview of the structure of the program (PaintCare, 2010; Bledsoe et al., 2011). ACA created PaintCare as a 501 (c) (3) non-profit PSO to manage the reuse, recycling, and proper disposal of unused paint as mandated by the Oregon legislation. The Board of PaintCare consists of nine non-paid representatives of architectural paint manufacturers (PaintCare, 2010; Bledsoe etal., 2011). PaintCare contracted with Product Care Inc., which has experience implementing similar stewardship programs, to implement and manage the Oregon program (Bledsoe etal, 2011). PaintCare selected PSC and Metro as transportation and processing vendors; these organizations had been part of the pre-program infrastructure (Bledsoe etal., 2011). Metro, PSC, and Amazon Environmental are the service providers for the processing of latex paint (PaintCare, 2010; Bledsoe et al, 2011). Their roles are as follows: Metro collects recyclable paint from its locations and accepts recyclable latex collected from the remainder of the state from PSC's consolidation center. This paint is made into new recycled content paint, which is then sold (PaintCare 2010,2011b). Non-recyclable latex paint collected by Metro is disposed of by biodegradation at Columbia Ridge Landfill in Arlington, Oregon. Through a research permit, the paint is mixed with wastewater and pumped into the landfill to improve degradation of wastes and recovery of landfill gas (PaintCare, 2011b). Non-recyclable latex paint collected and consolidated by PSC is processed into recycled content paint, a biomass fuel product, and processed latex pigment (PLP) (an alternative raw material for cement manufacture) by Amazon Environmental (PaintCare, 2011b). Oil-based paint is collected and processed for fuel blending by PSC (PaintCare 2010, 2011b). Infrastructure: Collection Sites PaintCare's strategy to set up the collection infrastructure included the following components: (PaintCare, 2011b; S17): Incorporate pre-program collection infrastructure (i.e., HHW facilities). Reach out to paint and independent hardware retailers to fill gaps in underserved areas. Send a letter to retailers of paint and decorator trade association list informing them of the program and giving them an opportunity to be a collection site. Conduct Web searches and review yellow pages to reach out to additional paint retailers in gap areas. The collection program began with 45 collection sites in July 2010 and, as of August 2011; there were a total of 98 sites collecting paint (see Figure 8) (PaintCare, 2011b). As of August 2011, all HHW facilities that collected latex paint prior to the program were collection locations in the PaintCare program Promoting Environmental Results Through Evaluation 26 ------- Figure 7 - Organization of the Oregon Paint Stewardship Program Product Care Association Admin Services American Coatings Association PaintCare, Inc. (program operator) Management Services Contract Producers Program Coordinator PCA Paint Stewardship, Inc. (program manager) Service Providers Figure 8 - Numbers of Collections Sites in July 2010 and August 2011 Numbers and Types of Collection Sites Retailers Restores HHWs July 2010 August 2011 (S17). None of PaintCare's 98 collection locations are paid for collection services (S17). The program provided each collection site with the following (PaintCare, 2011b): A Collection Site Procedures Manual (see text box on page 28). Program training on collection site operation, customer service, environmental risk reduction, visual inspection of post-consumer paint containers, and placement of collected paint in program containers. Compliance visits to confirm the site is following program requirements and to investigate complaints.17 Finally, the program transportation service provider (PSC) places collection containers at each collection site; collection sites schedule a pickup by calling the transportation service provider when the collection containers are approximately 50 percent full (PaintCare, 2011b). 17 PaintCare conducted 76 compliance visits during the first year of the program. 27 Evaluation of the Oregon Paint Stewardship Program ------- Contents of the Collection Site Procedures Manual Collection site standards Screening procedures for conditionally exempt small quantity generators Accepted and non-accepted program products Management of the Paint Exchange program, including required waiver forms Reporting requirements Management requirements and operational procedures Infrastructure: Other Collection Services and Other Services During the first year of the program, collection events were held to collect paint in rural locations (PaintCare, 2011b). PaintCare anticipates completing arrangements with permanent locations in order to reduce costs (S17). By September 1, 2011, the program had participated in 57 HHW collection events and two PaintCare-sponsored events. PaintCare provided additional collection services through large volume direct pickups and collection events (PaintCare, 2011b). Between July 1,2010 and June 30, 2011 the program provided direct pickup service to 19 institutional, commercial, and industrial entities, such as trade painters, apartment complexes, housing authorities and other private businesses that meet the criteria for conditionally exempt small quantity generators. PaintCare offers a reuse program though Metro and several Restores where good, usable paint in containers that are at least half full are offered to the public for free or at a discounted price (PaintCare, 2011b). Finally, the program identified opportunities for recycling plastic pails and steel paint cans from the Metro program; during the first year, the program recycled 47.1 tons of plastic and 64.8 tons of metal paint cans (PaintCare, 2011b). Funding Mechanism PaintCare collects a recovery fee from paint producers on all architectural paint sales to fund the program. This fee was authorized in the legislation and the fee was approved by the Director of the Oregon Department of Environmental Quality (PaintCare, 2010). The assessment rate by container size is summarized in Figure 9. PaintCare's budget and fee were developed based Figure 9 - Summary of Assessment Fee $2.00 $1.50 $1.00 $0.50 $0.00 Note: Program does not accept containers greater than 5 gallons. 1/2 pint or less 1/2 pint - quart Quart - 1 gallon 1 gallon - 5 gallons Promoting Environmental Results Through Evaluation 28 ------- on the estimated sales of architectural paint in Oregon, estimated proportions of leftover paint available for collection, and volume and cost data from the Metro program and Oregon DEQ (non-metro areas). According to the 2010 Program Plan, 2011 Annual Report, and additional explanation by a PaintCare representative (Em8): Revenue was estimated by pro-rating the national architectural paint volume sold by population. This volume was compared with ProductCare sales data from British Columbia and will continue to be refined as PaintCare receives actual sales data from program participants. The final number multiplied by an average cost per container produced the initial revenue estimate. Program costs included: Projected collection volumes estimated from rates of leftover paint derived from research; Metro and non- Metro paint collection volumes; and assumptions about the amount of paint available for collection (10 percent), the recovery rate of program products sold (7.1 percent), capture rate (71 percent), and anticipated growth in collection volumes per year (6 percent). Administrative fees. Vendor pricing to provide services based on projected collection volumes. The values were estimated for a 4 year budget, then used as inputs into a spreadsheet model which was used to model collection fees under several scenarios (i.e., per container fee, per gallon fee, flat fee, and graduated fee) to identify the scenario and the fee amounts that would best cover budgeted costs. PaintCare developed a graduated fee (i.e., less costly per gallon as container volume increases; see Figure 9) based on the assumption that less waste paint is typically generated from those purchasing a 5 gallon container and that a higher volume of oil-based paint, which is more expensive to manage as waste paint, is sold in smaller containers (S17). Figure 10 provides an overview of how the fee is transmitted between different actors in the program (PaintCare, 2010; Bledsoe et al, 2011). Producers report monthly sales and pay the assessment directly to PaintCare through a secure online filing system, electronic fund transfer, or check The assessment is passed through to a distributer or retailer who must add the fee to the final sales price of their products (PaintCare, 2011b). Retailers are responsible for collecting the fee from consumers and transferring to Figure 10 - Flow of Assessment Fee within the Program System Stewardship fee paid per container sold in Oregon Paint Manufacturer Paint Stewardship Organization Invoice & payment includes fee t» Funding to oversee program $ Paint Retailer (Purchase price of paint includes fee Consumers t» $ Program Delivery: Administration Collection Transportation Processing Communication Flow of Money Flow of Paint 29 Evaluation of the Oregon Paint Stewardship Program ------- paint producers, who are then responsible for paying the fee to PaintCare (PaintCare, 2011b). The legislation provides retailers the option to show or not show the fee on the consumer's sales receipt - this decision was a compromise among PPSI stakeholders (S17, S20). 4.2.2 What factors contributed to its infrastructure choices? Processing and Transportation In terms of processing and transportation, the primary factor that contributed to the infrastructure choices was the pre-existing infrastructure put in place prior to the program. Specifically, Metro had an established, mature paint collection and processing infrastructure in the Portland metro area. Metro continued to operate collection sites and process recyclable latex paint into MetroPaint, a recycled paint product sold by Metro. The program leveraged this infrastructure in developing the current Oregon program in areas outside of the Portland metro (Bledsoe et al, 2011). Recyclable latex collected outside of the Portland metro area was shipped from PSC consolidation facilities to existing Metro facilities for recycling. The program also relied on the pre-program transportation provider (PSC) to transport the paint.18 Collection Sites As shown in Figure 8 (see page 27), most sites were retailers (70 sites; 71 percent in August 2Oil) and HHWs (18 sites; 18 percent in August 2011). PaintCare's coverage goal was to provide a permanent collection site within 15 miles for more than 70 percent of Oregon's population, which served as a primary driver in determining the collection site locations (PaintCare, 2011b; S17). Initially, PaintCare did not add additional locations in an area (i.e., estimated as the area within a 15 mile radius of a population center) where a collection point was already established. However, at the request of one local retailer, a state representative encouraged PaintCare to reassess this approach. PaintCare's current policy is to add additional collection locations near another location only if no significant costs are incurred (S17). PaintCare reported, however, that it took longer than anticipated to negotiate contracts with many collection sites (S17). 18 PSC, however, was selected through a Request for Proposals process. 4.2.3 Was the funding mechanism clearly defined, transparent, and complete? Clearly Defined Stakeholders were asked to assess the clarity of the funding mechanism in a series of interviews (Bledsoe et al, 2011). Stakeholders indicated that: Legislation clearly details funding mechanisms in terms of manufacturer responsibility, and indicates that consumers will pay a fee at the time of a paint purchase. Program brochures and the PaintCare website communicate the details of the funding mechanism. Implementing a graduated fee based on container size as opposed to flat fees was viewed positively by the public and paint contractors. As detailed in the Annual Report the program estimates that most retailers show the fee on receipts (PaintCare, 2011b). However, for some retailers, reprogramming their computer system acted as a disincentive to show the fee as a separate line item on the receipt (S7, S8, and S9). The July 2011 survey of consumers found that 11 percent of respondents were aware of the fee (PaintCare, 2011a). Interviews conducted by the Evaluation Team with retailers and HHW programs support the July 2011 survey finding of low consumer awareness; two HHW interviewees reported some consumers confusing the fee with a bottle deposit (i.e., by returning the empty container the fee would be refunded) (S12, S15). Transparent The PaintCare report contained a detailed financial audit of the program which provides some level of transparency. Stakeholders' views on the degree of transparency were dependent on the interviewee's point of view. Those involved with PaintCare felt that a publically available budget meant the program was transparent, while other interviewees felt the lack of broken out costs made it less transparent. Promoting Environmental Results Through Evaluation 30 ------- Complete In the 2011 Annual Report, PaintCare states that despite the first year deficit projected in the program plan and lower than estimated sales, the project actually produced a surplus of $259,911 due to lower than estimated expenses. This surplus, however, was not sufficient to reduce assessment costs and will be carried over to year two and used for program costs (PaintCare, 2 Ollb). 4.3 Learning 4.3.1 Infrastructure Existing infrastructure influenced program design. The Oregon program was built upon existing institutional knowledge and experience, relationships, and infrastructure, resulting in reduced need for training and upfront costs. Other states and municipalities will have varying levels and types of infrastructure to accomplish similar goals. The efficiency and feasibility of program design and implementation will benefit from initial thorough assessments of existing infrastructure, including transportation, reprocessing capacity, and related knowledge and experience. A more explicit and collaborative strategy would improve collection site selection. A collaborative and/or strategic process for selecting collection sites may improve efficiency of planning and implementation. Opportunities include clear statements of process steps, criteria for site selection, and designated opportunities for specific stakeholders (such as retailers and HHW operators) to provide input in the process. 4.3.2 Assessment Fee Strategic and targeted outreach is needed to improve fee awareness. One goal of the Oregon paint legislation is that consumers become aware of the fee. In July 2011, few (11 percent) recent purchasers of paint were aware of the fee (PaintCare, 2011b). Fee awareness may be increased through a more strategic and targeted education and outreach campaign (see Learning section under Evaluation Question 3). Fee awareness should be reconsidered as a program goal. Future iterations of paint legislation may consider whether consumer fee awareness is a necessary policy goal. In August 2010, most Oregon residents (73 percent) indicated that the fee was reasonable with few (23 percent) indicating it was not reasonable (Bradshaw Advertising, 2010). If awareness of the fee is set as a program goal, then a target level of performance (e.g., a targeted percentage becoming aware of the fee among Oregon residents) should be set. Oregon fee structure could act as a model for other states. The process used to set/ design the fee structure is a baseline model for other states; it covered costs in the first year of the program. Notably, lower than expected collection of oil-based program products contributed to the one-year surplus. Evaluation of the Oregon Paint Stewardship Program ------- 5.o Education and Outreach EVALUATION QUESTION: How did education materials and strategies affect consumer awareness and behavior? Which messages were most effective with which target audiences? What materials/strategies were developed and what were the goals and target audience of those materials/strategies? Did other factors besides the program influence consumer behavior and awareness? What are the lessons learned? One of the key goals of the Oregon Pilot project is to minimize waste paint and containers generated by consumers. Toward that end, one of the first steps PaintCare took during the project was to launch an outreach and education campaign; the campaign is described in the June 29, 2010 approved pilot program plan (PaintCare, 2010). Evaluation Question 3 focuses on assessing the impact of this outreach and education program on consumer awareness and behavior. 5.1 Data Sources, Methods and Analysis To develop the measures of program awareness, the Evaluation Team drew on the results of consumer surveys that asked the target audience to provide self- ratings of degrees of awareness on the program. The primary data sources for the measures include:19 Initial consumer survey: A telephone survey of 409 Oregon homeowners, conducted by Bradshaw for PaintCare, performed from August 4 to 10, 2010. Full survey results are available in Bradshaw Advertising's report to PaintCare (Bradshaw Advertising, 2010), as well as in the appendix to PaintCare's annual report (PaintCare, 2011b). We refer to this as the August 2010 survey. Follow-up survey: An online survey of 1,064 Oregon residents over 18 years old, conducted from July 7 to 12, 2011. This survey targeted individuals who have purchased paint between July 1,2010 and June 30,2011 as well as individuals who paint as an occupation. Full survey results are available in PaintCare's report on the survey and in the appendix to its annual report (PaintCare, 2011a,b). In this report, we refer to this as the July 2011 survey. 19 These data sources are described in more detail in Section 2.4. Promoting Environmental Results Through Evaluation 32 ------- The two surveys are not identical, constraining the before and after comparisons that can be made. First, the two surveys cover different target populations. The August 2010 survey covers Oregon homeowners and the July 2011 survey covered Oregon residents that had recently purchased paint or that paint for a living. Second, the two surveys used different sets of questions with little overlap in the questions asked between the two surveys. The data collected through the surveys was supplemented by a qualitative approach; including a review of the materials produced by PaintCare and interviews with key program stakeholders. 5.2 Findings 5.2.1 What materials/strategies were developed and what were the goals and target audience of those materials/strategies? The communication firm chosen by ACA developed an outreach and education strategy with messages consistent with the themes of ACA's "Be Paintwise" program and targeted to all Oregon consumers. The educational materials used brief taglines such as "From Storage to Spectacular," "From Garage to Glorious," and "From Basement to Beautiful" to catch the attention of consumers (PaintCare, 2010,2011b). These messages appeared with supporting text that highlighted the following themes: The name of the program and its website. The advantages (cost savings, environment, storage space) of recycling leftover paint. The availability of information on purchasing the correct amount of paint and managing and disposing of paint properly on the program website. The recovery fee. The program website (www.paintcare.org) also featured educational information while providing additional detail on each of the messages and on the program itself (e.g., collection locations). Figure 11 provides an overview of the education campaign under the program. Samples of the materials that were used by the program can be found in appendices of the program plan (PaintCare, 2010) and the annual report (PaintCare, 2011b). The appendix to the annual report (PaintCare, 2011b) also contains details on the where (e.g., radio stations) and when advertising spots were purchased. Despite its wide reach, the outreach and education campaign did not identify or target specific audiences (PaintCare, 2010). The messages, materials, and delivery mechanisms were designed to have broad appeal and reach as many Oregon residents as possible, particularly those interested in purchasing paint. Objectives of the education and outreach campaign: Build consumer awareness. Identify program products. Identify collection site locations. Emphasize the environmental impact of poorly managed leftover paint. Emphasize the importance of purchasing the correct amount of paint. Promote paint reuse. Promote recycling and proper disposal of leftover paint. 33 Evaluation of the Oregon Paint Stewardship Program ------- Figure 11 - Overview of the Education and Outreach Campaign Point of sale materials 54,748 counter cards and 320 posters to 190 locations Direct mailings to contractors and retailers 1 2,414 factsheets to trade painters 1 1,400 factsheets to retailers in 3 mailings (12/09, 4/10 and 6/10) Website (www.PaintCare.org) 531,038 hits between March 4, 2010 and August 26, 2011 Education and Outreach Campaign Press releases and information sharing with the news media Prior to start date: 13 TV and radio stories, 56 print articles, and 36 Internet articles Earth day tie-in with 2 magazine articles and 8 Internet articles At the 1 -year anniversary: 6 articles and a new radio interview 1-800-CLEANUP (Numbers of calls received not available at this time) Advertising through local solid waste partners Trade shows 8 trade shows in 2010 and 7 in 2011 Purchased advertising Radio advertisements on 17 radio stations in 2010 and 24 stations in 2011 TV advertisements on 2 stations in the Portland Metropolitan market Newspaper advertisements in 33 newspapers across 29 markets in 2011 5.2.2 How did education materials and strategies affect consumer awareness and behavior? Program-Related Awareness Figure 12 summarizes program awareness in both surveys. In the August 2010 survey, 22 percent of respondents report being "aware" or "very aware" of the program (Bradshaw Advertising, 2010). In the July 2011 survey 27 percent of respondents recall seeing or hearing advertisements for the program (PaintCare, 2011a). However, it is difficult to determine the change, if any, in awareness because results are based on different target audiences. Figure 13 summarizes survey data on the ability of respondents to correctly identify in-scope products. A reported 44 percent of respondents to the August 2010 survey incorrectly indicated that oil-based paint was not covered by the program. Furthermore, one in five respondents incorrectly indicated that latex paint is not covered. The consumer outreach materials provided little information on the types of products that would be covered or not covered under the program. The rack card20 contained a question and answer section that indicated the types of products covered, while the program website covers types of products. Several of the materials produced by PaintCare provided information on collection sites or how to find collection sites. A January 2008 survey of Oregon residents found that 67 percent of respondents have taken leftover or unwanted household products to a collection event or facility in the past (Oregon DEQ, 2008). However, since this includes any leftover household products it may be an overestimate of baseline awareness of sites that accept leftover paint. The August 2010 survey did not ask consumers about their awareness of site locations (Bradshaw Advertising, 2010). The July 2011 survey found that 31 percent of respondents who purchased paint were aware of a collection site (PaintCare, 2011a). 20 The card "Why Everyone Should Get with the Program" that consumers would have found at paint or checkout counters in retail locations contained summary information on the program. A sample of the card can be found in the Appendix to the PaintCare annual report (PaintCare, 2011b). Promoting Environmental Results Through Evaluation 34 ------- Figure 12 - Program Awareness in August 2010 & July 2011 August 2010 July 2011 Awareness of Environmental Impacts of Paint The program and accompanying education materials do not provide consumers with information about the environmental impacts of leftover paint. Two exceptions include: the rack card entitled "Why Everyone Should Get with the Program," which indicates one benefit of recycling paint as "(e)liminating storage hazards where you live," and a page on the website (www.paintcare.org/) indicates that leftover paint may find its way into landfills and frames this as an environmental issue. The July 2011 survey did not address this aspect of awareness, so there is no indication of whether awareness improved (PaintCare, 2011a). However, the August 2010 survey found that 61 percent of respondents indicated it is "very important" and 31 percent indicated it is "somewhat important" to have a program that accepts unneeded, leftover paint for reuse, recycling and proper disposal (see Figure 14) (Bradshaw Advertising, 2010). Consumer Behavior The program promoted few messages related to reducing the amount of paint purchased. These messages encourage consumers to buy the right amount to:21 Save money: buying the correct amount reduces the amount of money spent on paint Save the environment: reducing leftover paint reduces the amount of paint that might find its way into landfills Save storage space: less leftover paint means more garage and basement storage space for consumers Reduce the risk of exposure to hazardous materials: storing paint in home can pose exposure and fire risks. Figure 13-Ability of Respondents to July 2011 Survey to Correctly Identify In-Scope Products 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Oil-based is covered Latex is covered Correct Paint thinner is not covered Containers with >5 gals not covered Incorrect 21 www.paintcare.org/index.php. 35 Evaluation of the Oregon Paint Stewardship Program ------- The paint calculator served as the program's primary tool of the program for assisting consumers in buying the right amount of paint. In the July 2011 survey, 15 out of 84 (18 percent) respondents reported being aware of the paint calculator, and 3 of those respondents indicated using the paint calculator to determine how much paint to purchase (see Figure 15) (PaintCare, 2011a). However, neither survey directly addressed whether consumers understood the calculator. The August 2010 survey asked respondents about what they would likely do with leftover or unneeded paint; the results of this survey question are summarized in Figure 16 (Bradshaw Advertising, 2010). In the August 2010 survey, respondents were also asked whether the existence of the program would affect the likelihood that they would recycle (see Figure 17) (Bradshaw Advertising, 2010). These data points should be interpreted in the context of the January 2008 survey that found that 67 percent of residents had taken materials to a collection event or facility in the past (Oregon DEQ, 2008). That is, the 40 percent who responded that the existence of a program makes "no difference" whether they recycle may still be willing to recycle paint and the 56 percent may include some of that one-third of respondents in 2008 who had not taken materials to an event or facility in the past. The data from the July 2011 survey show that no respondents recycled or donated leftover paint and 72 percent of respondents would store the paint for later projects or touch-ups (PaintCare, 2 01 la). ACA suggested that, based on anecdotal evidence, it appeared that the education and outreach program had little effect on consumers' practices (SI 7). Additional Findings The surveys conducted as part of the program did not collect information on amounts of leftover paint. However, previous studies of leftover paint have found that 2.5 to 16 percent of paint sold may remain as leftover paint, and the average amount of leftover paint per household ranges from 0.15 to 0.6 gallons (PaintCare, 2010). The evaluation was unable to collect data on how consumers store leftover paint. The program did not include questions on storage practices. Figure 14 - Consumer Opinions on Importance of a Paint Recycling Program Not important at all Somewhat important Very important 0% 10% 20% 30% 40% 50% 60% 70% Promoting Environmental Results Through Evaluation 36 ------- Figure 15 - Awareness and Use of Paint Calculator in July 2011 Survey Not aware Aware Did not use it Used it Figure 16 - Intended Consumer Management of Leftover Paint (August 2010) 35% 30% 25% 20% 15% 10% 5% 0% Figure 17 - Consumer Likelihood of Recycling Paint Due to Existence of Paint Recycling Program Call Go to garbage/ govt recycle/ recycler disposal org Store for Put it future in the use garbage More likely to recycle No difference No answer 37 Evaluation of the Oregon Paint Stewardship Program ------- 5.2.3 Which messages were most effective with which target audiences? Data collected during the first year of the program did not directly address the effectiveness of specific messages. To address this question would require linking consumers' viewing specific messages to the actions taken by consumers. The data for this evaluation did not include which messages were viewed by survey respondents. Nevertheless, the two surveys can provide some insight into differences across groups. Table 2 provides a summary of the information available in each survey on respondents' awareness and opinions. Table 3 provides a summary of the respondent characteristics that can be cross-tabulated with the awareness and opinion data. We provide a series of cross-tabulations by the different respondent characteristics in Figure 18 through Figure 21 and we used those cross-tabulations to investigate any potential differences across respondents, which can be summarized as follows: Importance of this type of program: All demographic categories (age, region, gender) indicated very strong support for a program that manages leftover paint (see August 2010 survey in Figure 18, Figure 19, and Figure 20). Awareness of program: With respect to program awareness, there are a few notable differences across respondents in the July 2011 survey: Table 2 - Available Information on Respondents' Awareness or Opinions in Each Survey Respondents' Awareness or Opinions Importance of this type of program August 2010 Survey July 2011 Survey Awareness of the program Awareness of the fee being charged Feel the fee is reasonable Awareness of drop-off locations Program makes it more like they will recycle Table 3 - Respondent Categories Available in Each Survey Respondent Categories August 2010 Survey July 2011 Survey Age Region Gender Education Income Promoting Environmental Results Through Evaluation 38 ------- Respondents who were older tended to be more aware of the program (see July 2011 survey in Figure 18). Respondents living in the south and central regions of the state tended to be less aware of the program in both surveys and those in the Portland metro area tended to be more aware (see July 2011 survey in Figure 19). Respondents who were more educated and had higher incomes also tended to be more aware of the program (see Figure 21). Awareness of fee: For the most part, there is little difference in terms of awareness of the fee. One exception is respondents making more than $100,000 may have higher awareness relative to other income groups (see Figure 21). Reasonableness of fee: Overall, respondents to the August 2010 survey indicated that the fee was reasonable with the lowest levels of agreement being among those 60 and older (68 percent; see Figure 18) and among men (67 percent; see Figure 20). Awareness of drop-off locations: There are some differences in awareness of drop-off locations for respondents to the July 2011 survey: Respondents younger than 44 tended to be less aware of drop-off locations than those who were 45 and older (see July 2011 survey in Figure 18). Respondents who lived in the Portland metro and central regions of the state tended to be more aware of drop-off locations (see July 2011 survey in Figure 19). Men tended to be more aware of drop- off locations compared to women (see July 2011 survey in Figure 20). Respondents with higher levels of income and education also tended to be more aware of drop-off locations (see income portion of July 2011 survey in Figure 21). Likelihood of recycling: Older and younger Oregon residents (see August 2010 survey in Figure 18) as well as women respondents (see August 2010 survey in Figure 20) indicated they would be more likely to recycle paint because of the existence of the program. 39 Evaluation of the Oregon Paint Stewardship Program ------- Figure 18 - Differences by Age Group: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey) August 2010 Survey Aware of program 18-34 Feels it is important to have this program 35-44 I 45-54 Feels fee is reasonable 55-64 Program makes it more likely they will recycle I 65+ July 2011 Survey 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Aware of program Aware of fee Aware of drop-off locations 18-34 35-44 45-54 55-64 65+ Promoting Environmental Results Through Evaluation 40 ------- Figure 19 - Differences by Region: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey) August 2010 Survey Aware of program Metro Feels it is important to have this program Feels fee is reasonable Program makes it more likely they will recycle Willamette Valley Central/East Southern Coast July 2011 Survey 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Aware of program Aware of fee Aware of drop-off locations Metro Willamette Valley Central East Southern Coast Evaluation of the Oregon Paint Stewardship Program ------- Figure 20 - Differences by Gender: Program Awareness, Program Importance, Reasonableness of Fee, and Likelihood of Recycling (August 2010 Survey) and Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey) August 2010 Survey Aware of program Feels it is important to have this program Feels fee is reasonable Male Female July 2011 Survey 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Aware of program Aware of fee Aware of drop-off locations Male Female Promoting Environmental Results Through Evaluation 42 ------- Figure 21 - Differences by Education and Income: Program Awareness, Fee Awareness, and Drop-off Site Awareness (July 2011 Survey) July 2011 Survey 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Aware of program Aware of fee Aware of drop-off locations HS or less Some college - college grad Post grad degree July 2011 Survey 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Aware of program Aware of fee Aware of drop-off locations <$25K $25K - $50K $50K-$100K >100K No answer 43 Evaluation of the Oregon Paint Stewardship Program ------- 5.2.4 Did other factors besides the program influence consumer behavior and awareness? The economy may have influenced markets for paint products and thus reduced Metro collection volumes. ACA tabulated quarterly Census Bureau data on gallons sold which indicated a decline from 153.2 million gallons in the first quarter of 2010 to 146.8 million gallons in the first quarter of 2011 (PaintCare, 2011b). Another potential influence to consumer behavior and awareness lies in the value Oregon residents place on recycling programs in general. Prior to the Oregon program, Metro had a well-established and organized recycling program in the Portland metro area, which serves as home to 48 percent of Oregon residents. Although 25 percent of consumers indicated that they were aware of the program in the August 2010 survey, 56 percent of respondents indicated that the existence of the program made it more likely that they will recycle leftover paint and 92 percent said it is "very important" or "somewhat important" to have this type of program (Bradshaw Advertising, 2010). 5.3 Learning Existing culture and infrastructure is important. The culture and infrastructure related to recycling in a state or region can inform the design, implementation, and emphasis placed on the program's education and outreach campaign. In Oregon, a strong culture of recycling coupled with existing infrastructure was helpful to the program. Furthermore, a well organized and established program that is centralized with respect to the state population, such as Metro, increases the efficiency of program roll out and the transition of a large segment of state population toward recycling paint. Targeting messages may be more effective. The education and outreach campaign used a broad approach to spreading its messages and did not strategically aim its messages at specific target audiences (consumers with the most paint, contractors, retailers, etc.) with specific means of communication (TV, website, social media, radio, etc.). Consumers of paint can be segmented into various groups (new versus long-time homeowners, homeowners versus contractors, age and other demographics). The most effective ways of reaching each group may differ. For future outreach efforts, consider determining which groups are the largest purchasers of paint or who has the most leftover paint and direct messages using appropriate communications at those groups. For example, the best ways of reaching consumers who are aged 60+ may differ from the best ways of reaching consumers under 30 (via newspaper, retailer, Facebook, Twitter, etc.). Awareness does not necessarily translate to recycling. In the August 2010 survey of homeowners, 10 percent indicated they would put the leftover paint in the garbage (Bradshaw Advertising, 2010), but in the July 2011 survey of those that had painted recently, no respondents indicated they had disposed of the leftover paint in the garbage (PaintCare, 2011a). Though it is possible that the program contributed to this outcome, it is not possible to attribute this change to the program because, for instance, there may be a lag time between recent painting projects and time of disposal not accounted for in the survey. A consumer's lack of site awareness does not automatically translate to not recycling paint. Consumers with leftover paint may attempt to locate a site to drop off paint when the need arises. Although awareness of a drop-off location in the July 2011 survey was 31 percent, a January 2008 survey found that 67 percent of Oregon residents had brought items to a drop-off location or event (Oregon DEQ, 2008). This suggests that consumers who intend to recycle their paint may know that drop-off locations are available and they need easy access to information about specific locations when they are ready to recycle; in other words, it may not be important that a consumer is aware of the program or is able to name collection sites on demand as long as she intends to recycle and is able to access information about where and how to Promoting Environmental Results Through Evaluation 44 ------- recycle. Going forward, the Oregon program and similar programs may use education campaigns to prioritize behavior change (e.g. recycle paint rather than store it) and accessibility of information over messages of general awareness. Outreach products need to be linked to a prioritized set of goals. Paint recycling programs should prioritize the goals of outreach products. For instance, the program should determine whether the goal of a particular message or product is to increase consumer awareness of the program or increase the number of consumers returning paint. These two goals may be related or mutually exclusive. That is, a consumer may find out about the program and then decide to recycle paint or they may find a collection location and return paint and be unaware of a "PaintCare Program," related legislation, or a collection fee. Focusing on getting consumers to drop-off sites maybe more important than focusing on program awareness. However, using education campaigns to ensure consumers purchase the right amount of paint (possibly eliminating leftover paint) may be more effective at reducing waste paint than, after purchasing surplus paint, helping consumers identify drop-off locations. Setting priorities and focusing outreach on those priorities will present opportunities to improve effectiveness and efficiency of marketing campaigns and the program as a whole. Outreach should balance the different, sometimes competing, program goals. A balanced approach to setting and prioritizing education and outreach is key and requires ongoing measurement and evaluation of the efficacy of the chosen approach. Putting too much focus on getting consumers to reduce the amount of paint purchased may lead to consumers with leftover paint who do not know what to do with it, while focusing too heavily on collections may cause consumers to not worry about the amount purchased since the drop-off locations provide an outlet for their unused paint (e.g., "whatever I don't use, I'll just drop off"). Measuring the effectiveness of education and outreach materials and strategies on consumer behaviors in the context of the paint management system can provide the information necessary to maintain balanced progress toward program goals. 45 Evaluation of the Oregon Paint Stewardship Program ------- 6.0 Consumer Purchasing Decisions ^ A *- > EVALUATION QUESTION: m How has the program affected consumers' purchasing decisions and management of paint prior to drop-off at paint recycling facilities? How did the fee assessment affect consumer behavior? One goal of the program is to help consumers purchase only the amount of paint they need and consequently reduce the amount of leftover paint that must be managed. The pilot program attempts to improve the ways in which consumers manage their leftover paint; the pilot program provides consumers with information on proper storage practices (e.g., not allowing leftover paint to freeze), encourages consumers to bring leftover paint to recycling facilities (rather than storing it), and educates consumers on the environmental consequences of improper disposal of leftover paint. Another key aspect of the program is the application of a fee to the consumer purchase price of paint in order to fund the program. Evaluation Question 4 addresses the program's effect on consumer behavior related to purchasing and managing paint, and the extent to which the fee influenced consumer paint purchasing behavior. 6.1 Data Sources, Methods and Analysis Evaluation Question 4 relies primarily on the same two surveys as Evaluation Question 3 (see Section 5); the August 2010 survey of Oregon homeowners and the July 2011 survey of recent paint purchasers in Oregon. Whereas the previous evaluation question focused on the impact of the program on consumer awareness, this question focuses on the impact on consumer behavior (e.g., paint purchasing). Additionally, the Evaluation Team drew on the 2008 Oregon Department of Environmental Quality Household Hazardous Waste Survey (Oregon DEQ, 2008) for baseline contextual information on consumer behavior related to paint management. The 2008 Oregon DEQ HHW survey was conducted in January 2008 by the Portland State University Survey Research laboratory; a total of 615 Oregon residents completed the random statewide telephone survey, answering questions about use and perception of the dangers of common household products (e.g., oil-based and latex paints, pesticides, drain openers). 6.2 Findings 6.2.1 How has the program affected consumers' purchasing decisions and management of paint prior to drop-off at paint recycling facilities? Purchasing decisions Data from the July 2011 survey indicate that the program had little impact on the amount of paint purchased, as 93 percent of respondents indicated that information about the program had no influence on the Promoting Environmental Results Through Evaluation 46 ------- amount of paint purchased (see Figure 22) (PaintCare, 2011a). Fifteen respondents (out of 84 total respondents) in the July 2011 survey recalled seeing the paint calculator and among those 15 respondents, three used the calculator to decide how much paint to purchase (See Figure 2 3). Management of paint prior to drop-off In the August 2010 survey, respondents were also asked whether the existence of the program would affect the likelihood that they would recycle (Bradshaw Advertising, 2010). Results indicated that: 56 percent of respondents reported that they are more likely to recycle leftover paint 40 percent reported that the program makes "no difference." Neither survey nor any of our other data sources addressed how or where the consumers stored the paint and how long they stored it prior to drop-off. The surveys did, however, address what consumers planned to do or did with paint (Figure 16 on page 38). 6.2.2 How did the fee assessment affect consumer behavior? When asked about the fee assessment in the August 2010 survey (see Figure 24), 33 percent of respondents indicated the fee assessment was "very reasonable," 40 percent found the fee "somewhat reasonable," and 23 percent indicated that the fee was "not reasonable at all" (Bradshaw Advertising, 2 010).22 In the July 2011 survey of recent painters (PaintCare, 2011a), 11 percent of respondents indicated that they are aware of the fee. Additionally, when asked whether the fee affected the amount of paint purchased, 93 percent of respondents said the fee had no impact (see Figure 25). This survey, however, covered those that had recently purchased paint. There may be a set of consumers that chose not to buy paint at the fee- inflated price, but who would have purchased paint at the price of paint without the fee added. Figure 22 - Consumer-Reported Impact of Program on Amount of Paint Purchased (July 2011 Survey) No effect; 64 Purchased fewer containers; 2 Purchased smaller containers; 1 Other; 2 10 20 30 40 50 Number of Respondents 60 70 22 The remaining respondents were in the refused/don't know categories. 47 Evaluation of the Oregon Paint Stewardship Program ------- Figure 23 - Awareness and Use of Paint Calculator Aware Saw paint calculator Did not use it Used it 6.3 Learning Purchasing Decisions and Paint Management Program had little impact on consumer purchasing. There is no indication that the Oregon program has influenced the consumers' paint purchasing decisions. If consumers continue to purchase the same amount of surplus paint, there will be no change in the volume per capita eventually destined for the paint management system, thus requiring expenditure of more resources than if consumers purchased amounts of paint that more closely matched their needs. Reducing surplus (waste/ leftover) paint purchased by consumers would reduce subsequent resources required for a statewide paint management system. Program focused on recycling, not reduction. The outreach materials focused on the recycling aspects of the program (i.e., where to bring leftover paint) and not on reusing leftover paint or reducing the amount of paint purchased. The purpose of the paint calculator provided at paint retailers is to improve the accuracy of the amount of paint purchased but consumers were generally unaware of it and did not use it, so it was not effective at reducing volumes of post-consumer paint. If improving the consumers' ability to purchase the correct amount of paint is to be considered a viable option to achieving the program and policy goal of reducing leftover paint, effort should be focused on understanding and improving the effectiveness of the tools being used to achieve this goal (e.g., paint calculator, retailer training, consumer assistance). Waste hierarchy emphasis should be clearly defined. To achieve goals of reducing the generation of post-consumer architectural paint, paint management programs and policies may explicitly document the relative emphasis they want to place on aligning the paint management system with the existing waste hierarchy. For instance, prioritize options such as reuse of leftover paint, reducing leftover paint through improving accuracy of paint purchases (less leftover) and increasing volumes of managed post-consumer paint. Explicit attention to these options will provide a structure to prioritize outreach strategies, messages, materials, and effort. Promoting Environmental Results Through Evaluation 48 ------- Figure 24 - Consumer Perceptions of the Reasonableness of the Fee (August 2010 Survey) Figure 25 - Consumer-Reported Impact of Fee on Amount of Paint Bought Very reasonable Somewhat reasonable Not reasonable at all Refused/don't know 100 90 80 70 60 50 40 30 20 10 0 Purchased same amount; 93% Was only asked of recent paint purchasers - no data on how many did not purchase paint because of the fee. Purchased less; 7% Impact of fee on consumer purchasing Oregon residents value this program. Ninety-two percent of respondents indicated the need for this type of program was "somewhat important" or "very important." Oregon residents' acceptance of the fee may be linked to the inherent value they place on the goals and services provided by such programs. The degree to which consumers value such programs may indicate their willingness to pay a certain fee. This information may be used in refining fee structures within and across states. Understanding and leveraging consumer attitudes may also prove useful in navigating the legislative process, where, as the PPSI learned in Minnesota, without evidence of public opinion and providing evidence of the public's willingness to pay a fee, legislation based on a fee mechanism may be rejected by executive or legislative leadership even with broad bipartisan legislative support. More details are needed on fee impacts. A full understanding of the impact of the fee would assess whether some consumers were deterred from buying paint due to the higher cost associated with the added fee. The fact that consumers are deterred at the margin may or may not be a good thing. On the one hand, a higher price may reduce the total amount of paint demanded by consumers and affect paint retailers' revenues. On the other hand, reducing the total amount of paint purchased reduces the total amount of leftover paint that would have to be managed by the program in the future. 49 Evaluation of the Oregon Paint Stewardship Program ------- 7.0 Collection of Post-Consumer Paint - -f ^ EVALUATION QUESTION: How has the program affected the collection of post- consumer paint in terms of volume, cost, environment, convenience, and infrastructure? What other factors have affected the amount of leftover paint? 7.1 Data Sources, Methods and Analysis Program data collected and tracked by PaintCare serves as the primary source of data for the analysis of volume, cost, environment, convenience, and infrastructure. PaintCare reported this information in its annual report (PaintCare, 2011b). The program data are supplemented with data and analysis provided by Hedrick Strickland, who performed a CIS analysis of collection site locations to assess convenience, as well as baseline data from the Oregon DEQ, and a set of interviews with key program stakeholders and retailers. 7.2 Findings 7.2.1 How has the program affected the collection of post- consumer paint? Volume In the first year of the program, PaintCare collected 469,665 gallons of paint. Of this total, 352,136 gallons (75 percent) were latex paint and 117,529 gallons (25 percent) were oil-based paint (see Figure 26). Metro collected 50.4 percent of the total gallons (PaintCare, 2011b). Overall, the program collected approximately 3 percent more pounds of paint than was collected by Oregon DEQ in 2008 (assuming a volume to weight conversion of 10 pounds per gallon for paint collected under the program) (PaintCare, 2011b). However, PaintCare notes in its report that there may be issues in comparing the two estimates due to conversion factors. In addition, PaintCare collected approximately 24,000 gallons of paint in areas where collection was not previously available. PaintCare collected less paint than anticipated in the projected budget (Figure 27), collecting 31,969 fewer gallons of latex (8 percent) and 47,088 fewer gallons of oil-based (29 percent) than projected (PaintCare, 2011b). PaintCare indicated that the discrepancy in oil-based collection is a result of a miscalculation in how much oil-based paint was being collected prior to the program start (S17). Additionally, PaintCare moved to a weight-based system to more accurately reflect collected volumes. For Metro, the number of households dropping off paint decreased under the program by two percent from the previous year and the number of non-households (e.g., contractors) dropping off paint increased significantly. Metro believes the increase in Promoting Environmental Results Through Evaluation 50 ------- Figure 26 - Volume by Paint Type and Service Area Collections by type of paint Volume by service provider/area - Reuse 3.2% Oil 117,529 25% Latex 352,136 75% Outside Metro 46.4% Metro 50.4% 469,665 total gallons collected 469,665 total gallons collected Figure 27 - Projected vs. Collected Amounts of Latex and Oil 450,000 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000 0 164,617 117,529 Latex Oil Projected Collected Evaluation of the Oregon Paint Stewardship Program ------- non-household consumers reflects the fact that disposal of paint is now free for these organizations (Eml), whereas prior to the program non-households had to pay to drop-off paint under the program. Metro also reported that the volume of latex paint collected was down 9 percent over the previous year. There was an initial spike in volumes for Metro during the first few months, but overall the total volumes were down (Eml, S16). Metro also indicated that they collected a large number of unlabeled containers of paint, which are classified as non-program materials and as such were the responsibility of Metro to handle (Eml, S16). Costs PaintCare reported that the cost of managing paint collected under the program is $7.03 per gallon (PaintCare Annual Report, 2011b). This metric does not include collection costs, such as labor and storage space, incurred by collection sites. In an Oregon paint recycling pilot program conducted in 1997 to 1998, Oregon DEQ-sponsored collection events were reported to cost $13.86 per gallon (inflated to 2011 dollars) and the pilot program cost $11.32 per gallon, during which collection was done at retailers' sites (Cascadia, 1998). Without more detailed analysis, these estimates are not appropriate for use as baseline, but may indicate some cost savings relative to the 1997 to 1998 program design. Environment The PPSI Lifecycle Workgroup (2005 to 2009) consisted of representatives from government and industry. ACA funded contractors to conduct a lifecycle assessment (LCA) of six options for managing leftover paint (e.g., dry and dispose at home to centrally collecting and processing into recycled paint) (PSI, 2009). LCA is a tool for the systematic evaluation of the environmental impacts of a product through all stages of its life cycle, which include extraction of raw materials, manufacturing, transport and use of products, and end-of-life management (e.g., reuse, recycling and/or disposal). The PPSI LCA scenarios were developed prior to the enactment of any legislation and therefore not based on the Oregon pilot (PaintCare, 2011b). Initial LCA modeling results were found to depend crucially on the amount of virgin paint that consumers replaced with recycled paint (PSI, 2009). If consumers were assumed to replace all virgin paint with recycled paint, then recycling proved to be the more environmentally beneficial outcome. However, if no virgin paint was assumed to be replaced, then producing recycled paint was found to have a larger environmental impact than just having consumers dry out and dispose of the latex paint in their household trash. Agreement between industry and government representatives on what percentage of virgin paint is avoided could not be reached and the workgroup disbanded. Without continued funding, the planned sensitivity analysis to identify other assumptions and parameters that had a major impact on the modeled results was not completed. ACA's conclusion as reported in the Annual Report (PaintCare, 2011b) was "[t]he [LCA] results demonstrate that unless and until recycled content paint is marketable at the same or similar rates as virgin paints...the environmentaL.cost [and] benefits or recycling paint do not outweigh drying and disposing of such." The draft LCA is available as Appendix J of PaintCare's report. Some members of the workgroup had anticipated that the Oregon pilot could provide real world data to refine the LCA model to complete PPSI's goal of better understanding the societal impacts of latex paint management approaches (PSI, 2 009). The modeling results could help with the planning and selection of appropriate paint management methods in light of environmental benefits weighed against costs. Convenience and Infrastructure As of September 1,2011, the program had 98 collection sites with 10 of these sites offering paint exchange. Sites are open to the public, on average, six days a week for a total of 58 hours per week (PaintCare, 2011b; Em2). Figures 28 and 29 provide maps of the pre-program and June 30,2011 drop-off sites, respectively (PaintCare, 2011b).23 The increased number of sites can be seen in comparing the two figures. Table 4 highlights the increased amounts of the Oregon population living within a 15 mile radius or within 15-mile drive of a drop-off location. The increased number of drop-off sites increased the number of Oregon residents living within a 15 mile radius of a drop-off location by 38.5 percent (PaintCare, 2011b). Additionally, based on a CIS analysis using June 2011 sites, 92 percent of Oregon residents live 23 Both Figure 28 and Figure 29 were taken from the PaintCare's annual report (PaintCare, 2011 b). Promoting Environmental Results Through Evaluation 52 ------- Table 4 - Proportions of Oregon Population Living within 15 Mile Radius and 15 Mile Drive of Collection Sites Time Period Percent of Oregon residents living in Percent of all Oregon incorporated cities, residents living within towns, and CDPs who are 15 mile radius within a 15 mile radius of collection site of a collection site [a] (PaintCare, 2011) Percent of all Oregon residents living within a 15 mile drive of a collection site (Strickland, 2011a,b) (Pa Pre-program (date unknown) January 2011 June 2011 [b] Change between pre- program and June 201 1 intnarp 2011} 69.2% -- 95.9% 38.5% 51.2% -- 70.9% 38.5% 91% 92% [a] GDP = Census Designated Places [b] The analysis performed for the PaintCare report on the percent within the 15-mile radius and the analysis performed by Strickland are seemingly contradictory. Specifically, the percentage within a 15 mile radius should exceed the percentage within a 15-mile drive. The Evaluation Team provided Strickland with the set of sites listed as being the June 30, 2011 sites. There may be some analytical details that account for some of the discrepancy. For example, Strickland's analysis uses the percentage of roads in a Census block group that are within the 15-mile drive as a proxy for the percentage of the population within that Census block group. Also, the most recent block level census data at the time of Strickland's analysis was from the year 2000, whereas the most recent data at the time of PaintCare's analysis was from 2010. This could also account for the discrepancy. Figure 28 - Pre-Program Drop-Off Sites 53 Evaluation of the Oregon Paint Stewardship Program ------- Figure 29 - Drop-Off Sites on June 30, 2011. Ugtnd \ SW.HMW lacing | ReUUora O Loe*l B«Vt rvtmt , Figure 30 - Percent of Residents Within a 15 Mile Drive by County. IMDU OP«?O- Sunndi Umtxn Fin M Promoting Environmental Results Through Evaluation 54 ------- Figure 31 - Change in Collection Service Availability Pre-program June 2011 HHWs offering collection Counties served by events only Counties with no service Retailers serving as collection sites within a 15-mile drive of a drop-off location (Strickland, 2011). Figure 30 presents the mapped results of the CIS analysis showing the percentage of residents within a 15 mile drive of drop-off locations for each county in Oregon. Figure 31 compares the pre-program network of collection sites to the June 2011 network based on data presented in the PaintCare annual report (PaintCare, 2011b). The data in the figure show a substantial increase in the availability of collection services. Additionally, PaintCare provided 19 large volume direct pickups to institutional, commercial, and industrial entities between July 1,2010 and June 30, 2011, collecting approximately 4,050 gallons of paint (PaintCare, 2011b). Finally, 63 percent of consumers in July 2011 who purchased paint recently think their nearest collection site is convenient (PaintCare, 2011a). 7.2.2 What other factors have affected the amount of leftover paint? Factoring into baseline considerations is the fact that the quantities of latex paint collected at DEQ events fell significantly after consumers were discouraged from bringing latex paint to collection events starting in 2008 (Oregon DEQ, 2011). To identify the factors that have affected the amounts of leftover paint, information on consumer handling of leftover paint is needed. Although some information is available on what consumers intend to do with the leftover paint, information on non-program factors was not assessed under this project. 7.2.3 How has the program affected transportation of paint from collection sites to other facilities in terms of volume, environment, and cost? The transportation contractor (PSC) is minimizing transportation costs by designing "milk-run" transportation routes with software designed by Red Prairie (Em3). The trucks service PaintCare locations as well as other clients that fall along the transportation route. PSC reported that transportation costs are similar pre- and post-program since the primary cost driver is diesel fuel price (Em3). However, without collection of Oregon pilot program data (before and after the program start) the Evaluation Team was not able to make an assessment on changes of transported volumes or the environmental impacts. 55 Evaluation of the Oregon Paint Stewardship Program ------- 7.2.4 How has the program affected retailers' behavior? Retailer interviewees note that the program had little effect on their practices in terms of marketing or sales (S7, S8, and S9). However, key challenges included: Programming the fee assessment into pricing and invoicing computer systems (S8). Preparing staff for program rollout (S7, S8). The challenge for one larger retailer was to make sure staff are aware of and understand the program, putting materials out, and are able to answer customer questions. Time for internal outreach and education and the magnitude of time/resources needed are related to the number of employees. 7.3 Learning Reporting on amounts of paint should be standardized. PaintCare's preliminary reporting used paint container fullness assumptions for calculating volume of paint collected. This assumption proved to underestimate gallons collected, and PaintCare found that a weight-based measurement was more accurate and more common for HHW reporting to Oregon DEQ (S17). Contractors are a stakeholder in the program. The amount of waste paint generated by contractors needs clarification. One interviewee stated that contractors should not be subject to the recovery fee because they don't generate waste paint (Bledsoe et al., 2011). Meanwhile, Metro saw an increase in the number of non-households dropping off paint because under the program no fees were collected for drop-off (Eml). Contractors were not a stakeholder group participating in PPSI and the amount of leftover paint they generate was not quantified in EPA's report Quantifying the Disposal of Post-Consumer Architectural Paint (Abt, 2007). PPSI could further explore if contractors are a significant contributor of paint to the management system and therefore need different and more audience-specific messaging about reducing waste paint. Unlabeled paint containers are an issue. The Evaluation Team is uncertain of the number of unlabeled paint containers that consumers are attempting to turn in. Consumers know the unlabeled container has paint, but collection sites are instructed to treat unlabeled paint containers as a non-program product and to not accept them. If a significant volume of leftover paint is being excluded from the program, then that paint is essentially being managed as it was prior to the program - as a waste product; rather than being recycled or reused. More definitive information is needed to provide an accurate assessment of the cost per gallon. The cost of $7.03 per gallon can be used as a starting point for other states. However, the reported PaintCare system costs do not reflect the true costs of a leftover paint management system. PPSI's Paint Product Stewardship Initiative Infrastructure Project (the "infrastructure report"), which modeled a comprehensive system to manage leftover paint, stated that the collection step in the process costs $1.98 per gallon (SCS and Cascadia, 2007)however, this is not an apples to apples comparison (e.g., the infrastructure report included transportation to aggregate facility as part of the collection step). Efforts are underway in Washington State to estimate local government costs associated with collection of waste paint, which includes a methodology and development of a tool for local governments to collect data (Em5). Furthermore, as noted under Evaluation Question 4, the Oregon program had a relatively well-established paint recycling infrastructure in the Portland metro area prior to this pilot program. Thus, program costs per gallon for other states may depend on the level of existing infrastructure. Promoting Environmental Results Through Evaluation 56 ------- Further discussion of reimbursement of collection costs is needed. Non- reimbursement of collection costs continues to be a controversial issue among stakeholders (S18, S20). This non-reimbursement was a topic that PSI had anticipated further collaborative discussion on as part of detailed work plan for the implementation stage. (S19) Additional LCA work is needed. An uncompleted task of the lifecycle workgroup was to conduct a sensitivity analysis for the LCA which could have identified other assumptions with large environmental effects. For example, the initial results indicated that the virgin offset made the largest impact, but transportation could be another factor. For example, Metro's less desirable recycled paint colors are being sent to Asia (for a fee, paid by Metro (Em4)). Does that change the environmental benefits of a recycled paint management system if paint is shipped across the globe? With the key assumptions and parameter data identified, future evaluations and ongoing examination of program performance information could collect groundtruthing data (e.g., virgin paint offset; miles traveled) to improve the ability to model a paint management system. The LCA is a tool to estimate a wide range of environmental and social impacts, and interpreters of its results will have to determine which impacts (e.g., greenhouse gases, water usage, air toxics) are the most important when selecting a management approach. LCA can be integral to future program design and planning, and requires funding for complete development and interpretation of the model. 'ak 57 Evaluation of the Oregon Paint Stewardship Program ------- 8.0 Paint Reprocessing, Recycling and Energy Recovery - - *- ^ EVALUATION QUESTION: How has the program affected used paint reprocessing, paint recycling, and paint-related energy recovery in terms of volume, infrastructure, and cost? An important aspect of assessing product stewardship outcomes is understanding what happens to the materials that are collected under the program. The program plan sets forth the following management options for collected leftover paint: Paint exchange: In order to increase the quantity of leftover paint that is reused, program collection sites and other locations can voluntarily offer a "paint exchange" to their customers. Under this system, better quality containers of paint are placed on display shelving and available at no cost to another consumer. As discussed in Evaluation Question 9, the exchange management option is an efficient way to achieve reuse as the paint does not require transportation and reprocessing. Recycling as paint: Good quality leftover latex paint may be bulked with similar colors then filtered and placed in smaller containers for resale to consumers. Recycling as another product: In the Oregon program, some unusable latex paint was sent to Amazon Environmental Inc. to be converted to PLP, a cement additive. Amazon also used some "bad" latex to bind biomass used to fuel its facility (referred to as Amazon biomass in this report). Energy recovery: Oil-based paint is blended with other oil-based solvents to be burned as fuel for some operations, such as incinerators. Appropriate disposal: Metro disposes of latex paint that is not of suitable quality for recycling via landfill biodegradation. Metro's non-saleable paint and washwater from the recycling process is transported to the Columbia Ridge Landfill in Arlington, Oregon. Under a special Research, Development and Demonstration permit at the landfill, liquids including the paint-washwater mixtureare intentionally circulated throughout the solid waste to expedite degradation. Table 5 summarizes these methods in order of preference for latex paint, oil-based paint, and paint containers. The purpose of Evaluation Question 6 is to track the volume, infrastructure, and cost associated with each management method. 8.1 Data Sources, Methods and Analysis The primary data source for answering Evaluation Question 6 is the data collected by the program and summarized in the PaintCare Annual Report (PaintCare, 2011b). The Evaluation Team supplemented these data with key stakeholder interviews to provide details on the numbers being reported. Promoting Environmental Results Through Evaluation 58 ------- Table 5 - Hierarchy of Management Options for Post-Consumer Paint Products Paint exchange/reuse Recycling as paint Recycling as another product Appropriate disposal Oil-Based Paint Paint reuse Energy recovery Paint Containers Recycling Appropriate landfill 8.2 Findings Table 6 shows the disposition of latex and oil-based paint collected under the program. These data are also summarized in Figure 32 (latex) and Figure 33 (oil-based). In what follows, we describe the data presented in Table i Table 6 - Disposition of Paint by Management Method Paint and Catego ercent of Total Paint Collected [: Latex Paint Reprocessing Amazon PLP Recycling/Reuse Metro recycling Amazon recycling Consumer reuse Energy Recovery Amazon biomass Disposal Metro biodegradation >il-Based Paint Recycling/Reuse Consumer reuse Energy Recovery Amazon fuel blending 8% 53% 4% 3% 4% 28% 3% 97% 28,171 186,632 14,085 10,564 14,085 98,598 3,526 114,003 [a] For latex this is the percentage of total latex paint collected (352,136 gallons) and for oil-based is the percentage of oil-based paint collected (117,529 gallons). 59 Evaluation of the Oregon Paint Stewardship Program ------- Reprocessing Disposal Eight percent of latex paint was processed into Amazon PLP, a cement additive (PaintCare, 2011b). However, information on costs for reprocessing was not available (PaintCare, 2011b). Recycling/Reuse Fifty-seven percent of latex paint was recycled as paint by Metro and Amazon (PaintCare, 2011b). Metro's peak volume of paint processed into recycled product increased after program implementation from approximately 280,000 to 325,000 gallons (S4). Metro has additional processing capacity available (S4). Three percent of latex paint and 5 percent of oil- based paint was reused by consumers (PaintCare, 2011b). PaintCare provided collection locations an incentive of $0.2 5 per "reuse" gallon to reflect avoided transportation and processing costs (PaintCare, 2011b). An organizational representative notes that no sites took advantage of this incentive; sites that were already offering reuse had systems in place and considered it an important service (S17). However, less than 10 percent of collection locations have shelves for paint exchange; PaintCare did not recruit retail sites for paint reuse due to concerns about liability (S17).24 Finally, approximately 47 tons of plastic paint pails and 65 tons of metal cans were recycled (PaintCare 2011b). Assuming that each plastic pail weighed about 1.6 pounds, and each metal can weighed about 1.4 pounds, then this tonnage translates to approximately 93 thousand metal paint cans and 59 thousand plastic pails. Data on the total number of containers collected is not available so there is no available estimate on the percent of collected containers that were recycled. Energy Recovery Almost all (97 percent) of the oil-based paint collected under the program was bulked for energy recovery (fuel blending) (PaintCare, 2011b). Four percent of latex paint went to Amazon Biomass; the paint is used to bind materials with high British thermal unit (BTU) value (e.g., sawdust) for energy recovery (PaintCare, 2011b) Twenty-eight percent of latex paint was processed for biodegradation; the paint is mixed with wastewater and injected into a landfill in order to improve degradation of landfilled wastes and increase gas recovery (PaintCare, 2011b). Figure 32 - Destination of Collected Latex Paint Metro recycling Metro biodegradation Paint reuse at Restores Figure 33 - Destination of Collected Oil-Based Paint Fuel blending Paint reuse Amazon recycling Amazon PLP Amazon biomass (fuel blending) 3% 24 Liability concerns can take several different formsenvironmental, health and safety, and product. For example, spilled paint from a consumer wanting to open a can to confirm color or the old paint product doesn't perform as expected. These liabilities can be mitigated. See PPSI's Guidance Manual for Paint Reuse Programs available at www.paint.org/component/docman/cat_view/47-education-a-safety.html. Promoting Environmental Results Through Evaluation 60 ------- 8.3 Learning Existing infrastructure was important for determining the destination of collected paint. The presence of existing latex paint recyclers with available processing capacity allowed recycling of paint to be the dominant method of processing leftover latex paint. There are barriers to increasing reuse. Recycling is lower on the waste hierarchy than reuse, and recycling results in higher costs (management, environment) compared to reuse. More information is needed to better understand the barriers (e.g., liability and space) to retailer facilitation of paint reuse. One option to increasing reuse, suggested by Oregon DEQ (S20), is for PaintCare to coordinate with the Restores and local government to redistribute reusable paint collected from the collection sites. This option, however, also creates costs and environmental effects (e.g., additional transportation and labor). Retailer liability may limit opportunities for reuse. Legislation that directly addresses reuse and liability is another option for clarifying program priorities and management. PaintCare incurred $2.4 million in transportation and processing costs to collect 469,665 gallons of paint, or $5.13 per gallona value 20 times more than the reuse incentive offered ($0.25 per gallon)indicating a potential cost savings in shifting more paint to reuse and room to increase the reuse incentive. Evaluation of the Oregon Paint Stewardship Program ------- 9.0 Household Hazardous Waste Programs - -f ^ EVALUATION QUESTION: What was the impact of the program on the HHW facilities in terms of the types of paint collected, costs, and the way in which the facilities operate? HHW facilities are a key stakeholder in the evaluation and an important part of the infrastructure of the statewide post-consumer paint management system. The Oregon DEQ solid waste program website lists event and facility information for 17 HHW programs in the state. These programs accept common household hazardous wastes (e.g., oil-based paint, pesticides, and mercury thermometers) from households, as well as conditionally exempt small quantity generators (often by appointment or special event) and then sort the various products for recycling, reuse, energy recovery, or proper disposal. HHW programs use a variety of mechanisms in order to provide services to residents, such as: Rural county local government solid waste or public health programs may offer periodic collection events. The Tri-County Hazardous Waste and Recycling program coordinates collection facilities and events for Hood River, Sherman, and Wasco counties. Several counties hire companies that specialize in hauling, disposal, and recycling to provide HHW collection opportunities. For example, Jackson and Josephine counties offer biannual events through Rogue Disposal. The number of customers served by a program varies. One HHW interviewee noted that they serve 85 - 140 cars at each monthly event (S12). Another interviewee reported that, in 2009 and 2010, their permanent facility served approximately 4,000 customers (S10). 9.1 Data Sources, Methods and Analysis To answer Evaluation Question 7, the Evaluation Team conducted phone interviews with seven representatives from county HHW programs in Oregon in June 2011 and data from the PaintCare annual report. The Evaluation Team supplemented the interview data with a review of materials related to the impact of the program on HHW such as PowerPoint presentations from HHW stakeholder meetings and other documents provided by interviewees. HHW programs have limited detailed quantitative data on paint volumes and costs. There are several reasons why there is a limited amount of quantitative data on HHWs. First, HHW programs operate under lean budgets and do not necessarily have the resources nor the expertise to track detailed paint volumes or cost information. Second, Federal agencies and their contractors are subject to the Office of Management and Budget (OMB) Paperwork Reduction Act requirements for information collection and as a result EPA was limited to interviewing nine or fewer individuals on this topic (i.e., a detailed survey of HHWs was not feasible). Promoting Environmental Results Through Evaluation 62 ------- 9.2 Findings 9.2.1 Types of Paint Collected at Household Hazardous Waste Facilities Table 7 shows a data summary of the PaintCare annual report on the gallons of paint collected at HHWs under the program. Interviewees noted that they had not observed a significant change in the total gallons of latex or oil-based paint collected since the PaintCare program began; however, two interviewees reported that the first event or month following the program start collected very high volumes that then tapered off through subsequent events (S4, S10). In most cases, the interviewees reported that they were able to accommodate the higher volumes with a few adjustments. However, one HHW faced delays in finalizing the contract with PaintCare and needed to store collected paint until the contract was in place and the paint could be picked up (S14). One interviewee noted a shift towards collecting a higher proportion of latex than oil-based paint (S14). Another interviewee observed increased traffic from paint contractors because the county charged a fee for large volumes prior to the program (S10). After the program began, one HHW site ceased collecting paint entirely (S15) while another program began collecting latex paint again after a long hiatus (S12). 9.2.2 Changes in the amounts of or capacities for other products Interviewees reported no significant changes in the amounts of other HHW products and moderate shifts in available capacity for processing other HHW products. Three HHWs saw an increased capacity to handle other products because they were able to reassign staff from crushing paint cans and bulking the material for disposal to other tasks (S4,S10, and Sll); one HHW site stopped collecting paint (S15). 9.2.3 Costs for household hazardous waste facilities to take in and process paint Interviewees noted that HHW programs do not have sophisticated systems in place for detailed cost tracking; staff may simply compare monthly internal costs (e.g., labor, overhead) to external costs from vendor invoices (e.g., hazardous waste hauler) (S12). Additionally, these costs are not likely to be tracked by task or material type, such as paint processing versus other hazardous materials. As a result, the interviewees were not able to provide detailed cost data, but many were able to provide estimates of the changes in costs following implementation of the program. Five of seven of the interviewees reported significant cost savings resulting from the program even when accounting for increases in customer traffic and material volume. The cost savings are reported to result from the avoided labor, disposal, and transportation Table 7 - Gallons of Paint Collected at Different HHW Facilities between July 2010 and June 2011 Facility Gallons Collected Columbia County 4,140 Beaver Hill Disposal Site 675 Crook County Solid Waste 188 Deschutes County 20,375 Tri-County Hood River 3,150 Marion County 2,970 Tri-County - The Dalles 1,890 Lane County 12,529 Lincoln Cty SW Transfer Station 720 Capital Plant 585 Source: PaintCare, 2011 b, Appendix D. Note: Metro's value was not broken out by HHWs versus other facilities. Additionally, the values do not include collection events. HHWs not listed had no value listed (i.e., a "-") in Appendix D of the PaintCare report. 63 Evaluation of the Oregon Paint Stewardship Program ------- costs associated with handling oil-based and unusable latex paints. Some examples of avoided costs from interviewees included: Saving $1,500 per month of a $100,000 annual budget due to avoided paint disposal costs (S12). Cumulative savings of $45,000 to $50,000 resulting from less paint handling by staff and avoided disposal, landfill, and transportation costs (S10). A cost decrease of 40 percent from pre- program costs due to avoided disposal costs associated with oil-based paints (S15). Avoided costs of $75,000 to $100,000 per year of an annual budget of approximately $300,000 due to avoided costs of disposal and transport of paint (Sll). A separate cost reported by three interviewees involved the need for HHW staff to conduct outreach and education before and during paint collection (S13, S14, and S15). These interviewees felt that the efforts by the program did not achieve sufficient reach within their county. One interviewee supplemented the program efforts with additional, locally targeted promotion around the beginning of the program (S15). Of particular interest to these interviewees was providing additional information to customers on the location of the nearest collection site, the types of materials that would be accepted, and the purpose of the fee. Two interviewees reported that some customers mistook the fee for a bottle deposit and returned empty cans in the hopes of getting the fee returned (S12, S15). Two other interviewees reported that customers were bringing in materials that did not fit the criteria for program products; for example, some customers were bringing in materials, such as asphalt sealant, that are applied with a brush (S13, S12). 9.2.4 Cost of managing other products Data on the costs of managing other HHW products is limited for the reasons described at the beginning of Section 9.2.3. Two of the interviewees commented that the costs of handling non-program materials returned by customers could be significant. When asked about the challenges associated with rollout of the program, two interviewees noted that correctly identifying program materials and non-program materials was challenging at first, and that the volume of unacceptable items, such as unlabeled or rusty cans, could be significant (S4, S13). 9.3 Learning Analysis of prior opportunities and conditions can help predict how HHW volumes may be affected by a paint recycling program. Change in volume and types of paint collected at HHW sites will vary depending on the convenience and availability of prior opportunities for consumers to return paint. Some HHW programs stopped collecting latex paint prior to program rollout due to resource constraints and so experienced a spike in paint volume when the program began. Paintlatex in particularhas historically been a significant percentage of the volume of material collected by HHW programs. Shifting processing of the paint to another entity frees up resources for the HHW to handle other materials. However, if the HHW is located in a state with limited infrastructure for paint collection, the HHW might experience the opposite effect: capacity to process other products may decrease significantly as customers bring in a relatively high volume Promoting Environmental Results Through Evaluation 64 ------- of legacy paint (especially if dropped off with other materials at the same time). A CIS analysis could assist HHWs to anticipate the increased volume of paint under a recycling program. An effective version of this type of analysis would include paint purchasing data related to demographics. For example, CIS can be used to map and analyze the demographics of an area (age, homeownership, etc.) in the context of paint purchasing behavior. Combining CIS with data on demographics with paint sales in the area could identify the potential quantity of paint that an HHW could expect. Labor and oil-based paint disposal costs are the key cost elements for HHW. States with limited pre-existing infrastructure for paint collection may experience initial spikes in labor needs at HHWs as customers bring in relatively high volumes of legacy paint; however, the program should mitigate the costs of sorting, bulking, and disposal. Areas with well-established infrastructure and steady paint volume collected could see significant cost savings. Cost information is lacking for HHWs. HHWs represent a significant set of program stakeholders. Without knowing the costs incurred to process paint at HHWs and the cost avoided from paint recycled, the full cost effectiveness of the program cannot be judged. Education and outreach for HHWs to use with their customer base may be needed. Development of education and outreach campaigns for future roll-out states should include HHWs to ensure the inclusion of complete and relevant information for HHWs. Consideration should be given to non- program materials and how to deal with them. Collecting non-program materials imposes costs on HHWs and quantities may be significant. HHWs may need additional support from the program to ensure that these materials are correctly identified and sorted separately from program materials. States with limited prior infrastructure should anticipate collecting older, deteriorated (e.g., "rusty"), and unlabeled cans of paint and be prepared to sort them from program paint. 65 Evaluation of the Oregon Paint Stewardship Program ------- io.o Cost Effectiveness EVALUATION QUESTION: How cost effective is the program? 10.1 Data Sources, Methods and Analysis Cost-effectiveness can be assessed by comparing an amount reflecting an outcome (e.g., paint collected) to the costs incurred to achieve that outcome. To answer this question, the Evaluation Team intended to analyze cost data collected to answer Evaluation Questions 5, 6, and 7 at a level that would have allowed for comparison with the PPSI's estimated cost of $7.46 per gallon collected, as found in Paint Product Stewardship Initiative Infrastructure Project of a comprehensive system to manage leftover paint (SCS and Cascadia, 2007). The PaintCare Annual Report (PaintCare, 2011b) provided cost information, amounts of collected paint and cost per gallon collected information. However, since PaintCare's budget does not reimburse for collection costs, the program's budget does not include the full system costs. Additionally, much of the initial year's costs of PaintCare included higher level of administrative costs due to efforts to educate consumers and retailers (both about the fee and those collecting) and other set-up costs associated with a new organization and setting up an infrastructure. Further, legacy paint entering the system will represent significant volume in the program's early years. Evaluation Question 8 may be more thoroughly answered with several years of operating expenses available. 10.2 Findings Actual PaintCare program costs for the first year were $7.03 per collected gallon, excluding collection costs (PaintCare, 2011b). PaintCare's initial projection of program costswhich excluded collection costs at permanent facilitieswas estimated to be $8.26 per collected gallon in the first year with a decreasing cost of $7.30/gallon in year two, $7.27/gallon in year three, and $7.18/gallon in year four (PaintCare, 2010). The primary reasons for the lower costs compared to pre- program include that a higher percentage of latex paint was collected as compared to oil-based (latex paint is more expensive to handle). In addition, less non- program materials were collected, so overall disposal costs for these items were less than expected (Em2). Despite the overall lower than expected costs, some initial year expenses were higher than expected. For example, start-up costs (e.g., legal and banking fees) and cost for education and outreach efforts were higher than expected (S17). Figure 34 presents the cost per gallon from the Oregon program alongside other estimated costs per gallon to process leftover paint from other programs and analyses. Table 8 provides details on the different cost per gallon estimates. Although the $7.03 per gallon estimate is lower than other estimates, the PaintCare estimate excludes some aspects (labor for collection) that are included in other, higher estimates. PPSI's life cycle workgroup was scoped to have a full cost-benefit analysis conducted on the same management scenarios as were modeled in the LCA (PSI, 2009). Such an effort would have examined cost effectiveness from a social perspective, accounting Promoting Environmental Results Through Evaluation 66 ------- for additional costs such as consumer time to drop off paint and externalities associated with environmental impacts (e.g. air emissions, greenhouse gases, and water use). 10.3 Learning The exclusion of HHW collection costs from the total program costs is significant. Even though it appears that costs decreased for HHWs, gauging total program cost effectiveness is not possible without knowing collection costs for HHWs. Calculating a value for collection costs is complex. Some variables such as estimating the cost of labor are relatively straightforward, while other items such as valuing the loss in building space that is used for storing waste paint and the percent of overhead expenses (e.g., insurance and administrative) that should be attributed to the collection are more challenging to estimate. Expand cost-effectiveness to include environmental benefits. Product stewardship initiatives can broaden the scope of cost- effectiveness by factoring in environmental benefits. The cost-effectiveness measured here is the cost per gallon collected. A more comprehensive measure would translate the gallons into environmental benefits and also include other management options (e.g., reuse) as well as the cost-effectiveness of reducing leftover paint by convincing consumers to "buy the right amount." Completion of a full life cycle cost-benefit analysis could supply some of this information. The key is to develop a measure of the environmental benefit associated with different paint management options. Expand cost-effectiveness to include program development. Beyond expanding cost effectiveness considerations to environmental benefits, product stewardship initiatives can consider the cost-effectiveness associated with program development, including the costs of collaboration. Collaboration for this program involved significant time and effort on the part of several stakeholders. However, what did collaboration achieve relative to a less collaborative approach? Addressing this question would allow for a more complete assessment of this type of program and for developing programs through a collaborative approach. Figure 34 - Costs per Gallon Collected from Oregon Program and Other Programs or Studies $15.00 $12.00 $9.00 $6.00 $3.00 $0.00 1997-1998 Pilot, collection events ($2011) 1997-1998 Pilot, retailers ($2011) Paint Infrastructure project estimate Projected by program Actual 67 Evaluation of the Oregon Paint Stewardship Program ------- Table 8 - Details on Different Cost per Gallon Estimates Data Source Amount[a] Cost Elements Included Cost Elements Excluded PaintCare - actual program costs (2010 to 2011) $7.03 ($2011) Education and outreach Materials for collection (e.g., bins) Transportation from collection site Processing, recycling, and disposal Labor at collection events Labor at permanent collection sites Indirect costs (e.g., storage space) Social and environmental costs (e.g., air pollution) Paint Smart Pilot Program (Cascadia, 1998) Special collection events: $13.86 ($2011) Materials for collection (e.g., bins) Transportation from collection site Processing, recycling, and disposal Labor at collection events Education and outreach Labor at permanent collection sites is not applicable. Indirect costs (e.g., storage space) Societal costs (e.g., air pollution) Paint Smart Pilot Program (Cascadia, 1998) Retailer Materials for collection (e.g., bins) collection: Transportation from collection site $12.24 Processing, recycling, and ($2011) disposal Education and outreach Collection labor Indirect costs (e.g., storage space) Societal costs (e.g., air pollution) PPSI Infrastructure Report (SCS and Cascadia, 2007) Model of national infrastructure: $8.15 ($2011) Education and outreach Materials for collection (e.g., bins) Transportation from collection site Processing, recycling, and disposal Labor at collection events & permanent collection sites Indirect costs (e.g., storage space) Societal costs (e.g., air pollution) [a] Cost estimates inflated to 2011 dollars using U.S. Bureau of Labor Statistics' Inflation Calculator. Available at www.bls.gov/data/inflation_calculator.htm. Promoting Environmental Results Through Evaluation 68 ------- 11.0 Waste Hei (to be completed Jan 201 2) ^ . L EVALUATION QUESTION: * How was the program designed and implemented to move consumers up the waste hierarchy? i With respect to moving customers up the waste hierarchy, what were the program's obstacles, opportunities, and decisions? A "waste management hierarchy" classifies waste management options according to their desirability. PPSI considers reducing leftover paint to be the preferred management option, followed by reuse, recycling (including energy recovery), and disposal. A key goal of the program is to encourage greater reliance on the most preferred strategies in the hierarchy and less reliance on least preferred strategies (i.e., "moving up" the hierarchy). This question assesses how well the program has achieved that goal. Many obstacles stand in the way of waste reduction. Thus, Evaluation Question 9 also considers the obstacles, opportunities, and decision-making related to moving up the hierarchy. Oregon and other states can use this information to design paint stewardship programs that more efficiently encourage reliance on the most preferred management options. The Evaluation Committee has partnered with Brielle Kissell, a graduate student at Tufts University, to investigate this question as part of her master's thesis. Ms. Kissell is currently working on developing the analysis for Evaluation Question 9. At this point, we can present the approach proposed in the method, as well as some initial findings related to the waste hierarchy from other questions. 11.1 Data Sources, Methods and Analysis The method for answering this question will involve relating the components of the program to the categories of the waste hierarchy (reduce, reuse, and recycling) and then compiling information on the obstacles, opportunities, decisions, and relative emphasis related to each category of the waste hierarchy. The matrix may look something like the one at the bottom of the page. The Evaluation Committee will explore the degree to which the paint stewardship program emphasizes waste reduction over others as reflected in educational materials, convenience, infrastructure, and performance. The Evaluation Committee will compare the effectiveness of waste reduction messages to others, resources devoted to waste reduction compared to other management methods, and the number of paint exchanges and volumes exchanged before and after paint stewardship. Waste hierarchy category Program component Decision made Obstacles Opportunities Relative emphasis 69 Evaluation of the Oregon Paint Stewardship Program ------- 11.2 Findings and Learning (From Other Questions) As noted above, the detailed answer to Evaluation Question 9 is currently in process. Table 9 provides a summary of findings and learning from other evaluation questions related to waste hierarchy. Table 9 - Findings and Learning Related to Evaluation Question 9 from Other Evaluation Questions Other Evaluation Question Learning in Question 4 Findings in Question 6 Learning in Question 6 TiTSrTiTn To achieve goals of reducing the generation of post-consumer architectural paint, paint management programs and policies may explicitly document the relative emphasis they want to place on aligning the paint management system with the existing waste hierarchy. For instance, prioritize options such as reuse of leftover paint, reducing leftover paint through improving accuracy of paint purchases (less leftover) and increasing volumes of managed post-consumer paint. Organizing these priorities will provide a structure to prioritize outreach strategies, messages, materials and effort. 57 percent of latex paint (approximately 201,000 gallons) was recycled as paint by Metro and Amazon (PaintCare, 2011 b). 3 percent of latex paint (approximately 10,000 gallons) and 5 percent of oil-based paint (approximately 3,500 gallons) was reused by consumers (PaintCare, 2011 b). PaintCare provided collection locations an incentive of $0.25 per "reuse" gallon to reflect avoided transportation and processing costs. An organizational representative notes that no sites took advantage of this incentive; sites that were already offering reuse had systems in place and considered it an important service (S17). Less than 10 percent of collection locations have shelves for paint exchange; PaintCare did not recruit retail sites for paint reuse due to concerns about liability (S17). Recycling is lower on the waste hierarchy than reuse, but there are barriers to increasing paint reuse. Furthermore, recycling results in higher costs (management, environment) compared to reuse. More information is needed to better understand the barriers (e.g., liability, space) to retailer facilitation of paint reuse The private-public partnership and the waste hierarchy model (reduce, reuse, recycle) were at odds in this program. The public preference in the waste hierarchy model is for reduction and reuse. There were opportunities to stress both reduction and reuse under this program, however, the program as-designed tended to focus more on collection and recycling. Other state programs can more clearly articulate program priorities to ensure alignment with existing program and policy goals and reduce the potential for confusion amongst consumers. Unintended Consequence in Question 12 Promoting Environmental Results Through Evaluation 70 ------- i2.o Market for Post-Consumer Paint - - ' EVALUATION QUESTION: * How has the market for post-consumer paint been * affected by the program? i What aspects of the program have had an impact on the market and how? What markets and products represent potential opportunities for post-consumer paint products? Expanding the market for post-consumer paint products is an important element of the pilot program and could support improved management of leftover paint. Ideally, if more consumers used post-consumer paint, then they would generate less waste because less new paint would be purchased (and ultimately need to be disposed). Additionally, leftover paint can be reprocessed into alternative non-paint products such as cement additive. An increase in businesses using non-paint products produced from post-consumer paint could also reduce the amount of paint flowing into the waste stream. This question assesses the extent to which the program has helped develop a market for post- consumer paint. 12.1 Data Sources, Methods and Analysis Evaluation Question 10 was answered by performing a qualitative assessment of the market by reviewing available information on the post-consumer market for paint in Oregon. The measures consist of a listing of the post-consumer paint products offered, the facilities that offer them, and the total sales of these products. The data sources for these measures were interviews with staff from PaintCare, recycled paint manufacturers, and HHW facilities, as well as data and information maintained by the program on these products. Based on the information collected from the program and Oregon DEQ, the Evaluation Team performed additional research into the post-consumer paint products market in Oregon to verify information from the interviews and to collect additional information on these products and markets. This additional research consists of both Web searches and interviews with non-program paint experts in Oregon. 12.2 Findings 12.2.1 How has the market for post-consumer paint been affected by the program? As part of the first PPSI MOU, stakeholders identified the need to develop post-consumer paint markets as a key issue (PPSI, 2004). The following three projects were suggested: Market Development Strategy: Research potential markets for recycled paint and develop model procurement policies. Only initial funding was provided. 71 Evaluation of the Oregon Paint Stewardship Program ------- Recycled Paint Marketing Guide for Distributors: Develop a marketing guide for both public and private distributors, targeting a variety of market segments. This project was placed on hold until the Recycled Paint Certification System was completed and additional funds were raised. Recycled Paint Standard and Certification System: Working with Green Seal, PPSI developed guidelines and performance specifications for recycled paint, along with a system for certifying specific recycled paint products. This project was completed in August 2006. The Oregon pilot program's primary focus, however, has been on the collection and management of leftover paint and has not focused on developing a market for post-consumer paint (S17). The education materials developed under the program focus on the collection aspects of the program and not on consumer reuse or purchasing. In Oregon, there are three types of paint that fall into the category of post-consumer recycled paint and that can be considered part of the post-consumer/ recycled paint market for Oregon:25 Leftover paint (reuse): paint that is dropped off and could be used as-is by another consumer. This is the type of paint offered for paint exchange by PaintCare through Restores in Oregon. MetroPaint: Metro sells recycled paint under the name MetroPaint, which contains 2 percent or less of total volume of virgin paint and additives (Metro, 2011). Amazon Select: Amazon Environmental Inc. sells Amazon Select, a recycled paint with 95 percent recycled content (S12). The amount of paint collected under the program and made available for the market, distribution channels, and amounts sold for each product are summarized in Table 10. In terms of reuse, 3 percent of latex (10,564 gallons) collected under the program and 3 percent of the oil-based paint (3,526 gallons) collected under the program was available (supply) as leftover paint (PaintCare, 2011b). According to the PaintCare report, 53 percent of the latex collected under the program (186,632 gallons) was sent to Metro to become recycled paint (PaintCare, Table 10 - Summary of Post-Consumer Paint Products in Oregon: Amounts Available, Distribution Channels, and Amounts Sold Post-consumer paint product Amount of paint available for product Distribution channels Leftover paint - paint that is dropped off and could be used as-is by another consumer 10,564 gallons of latex 3,526 gallons of oil Restores in Oregon Amount of program- related product sold Unknown Recycled paint (Metro) - small amount of virgin paint and additives (2 percent or less of total volume) are combined with leftover paint 186,632 gallons of latex 45 retailers in Oregon and Washington 115,343 gallons of MetroPaint Recycled paint (Amazon Environmental) - small amount of virgin paint and additives (5 percent or less of total volume) are combined with leftover paint 14,085 gallons of latex Sold as Amazon Select Not available* 25 The categories below do not represent official definitions, but reflect the Evaluation Team's view of the types of products on the market. Promoting Environmental Results Through Evaluation 72 ------- 2011b). Metro sold 122,705 gallons of paint between July 1,2010 and June 30, 2011 (S16). During that time, 94 percent of the paint Metro took in came from the Oregon program. Thus, approximately 115,343 gallons of paint (2 5 percent of all paint collected by PaintCare) taken in by the program between July 1, 2010 and June 30, 2011 were sold by Metro as post-consumer paint. Metro has been selling post-consumer paint (MetroPaint) since 1992 and consumers can purchase MetroPaint at more than 45 retailers throughout Oregon and Washington (Metro, 2011). Metro indicated that the program had no effect in their sales of post-consumer paint. For the past several years, independent of the Oregon pilot program, Metro increased sales of post- consumer paint by expanding the number of outlets offering its paint (S3). Metro reported that demand for light color recycled paint exceeds the program's supply. The opposite is true of darker colors (S16). According to the PaintCare report, 4 percent of latex collected under the program (14,085 gallons) was sent to Amazon to become recycled paint (PaintCare, 2011b). Amazon, Inc. considers its sales figures as proprietary and did not provide these figures to the Evaluation Committee. However, Amazon indicated that only a small percentage of its paint sales stem from paint collected through the Oregon program (S21). Table 11 summarizes the total amounts of latex and oil-based paint collected under the program and the amounts that were made available in the post- consumer paint markets. In total, 217,157 gallons of the paint collected (46 percent of the total amount collected) under the program made it into the post- consumer paint market. For latex paint, 60 percent of the amount collected under the program was made available to the post-consumer paint market. For oil- based paint, 3 percent of the paint collected made it into the post-consumer paint market. Nationally, the annual potential market for post-consumer latex paint products is $1.2 billion and includes environmentally conscious adults ($540 million) and contractors ($230 million), and governments ($420 million) (Hult, 2011). According to the 2010 Census, Oregon represented 7.6 percent of the U.S. population. Using this percentage to adjust the national numbers, the annual potential market in Oregon would be approximately $90 million. Table 11 - Amounts of Latex and Oil-Based Paint Collected Under the Program and Made Available in Post-Consumer Paint Markets Paint Type Latex Total Gallons Collected Under the Program 352,136 Total Gallons Collected Under the Program Made Available in Post-Consumer Paint Markets (Percent of total) 211,281 (60%) Oil-Based 117,529 3,526 (3%) Latex and Oil 469,665 217,157(46%) 73 Evaluation of the Oregon Paint Stewardship Program ------- 12.2.2 What aspects of the program have had an impact on the market and how? The Oregon program has an indirect impact on developing post-consumer paint markets by promoting recycling over other management options (e.g. disposal) and by increasing the supply of leftover paint. By encouraging consumers to return the paint, the program is developing the supply side of the market. The centralized collection of paint by the program is funneling more than 200,000 gallons of leftover paint to be recycled into paint for resale to consumers. 12.2.3 What markets and products represent potential opportunities for post-consumer paint products? There are two post-consumer non-paint products being produced from Oregon leftover paint: Amazon Environmental's cement additive (PLP) and a binding material for biomass that Amazon Environmental uses to fuel their recycled paint facility (PaintCare, 2011b). The sale of recycled latex paint is challenged by the technical limitations of recycled paint colors and consumer perceptions on quality (Hult, 2011). 12.3 Learning Tracking data on the post-consumer paint market is an important program component. One component of determining the effect of the program on the market for post-consumer paint is tracking data on post- consumer paint sales, including both price and volume. The existence of a market for post- consumer paint was important. A paint recycling program can help to develop a market for post-consumer paint. In Oregon, a market existed already to some degree, as Metro has been selling MetroPaint since 1992. Recycling programs can help to develop markets in states without existing markets by identifying retail outlets and assessing potential paint reprocessing or recycling facilities and capacity. Standard market indicators are not adequate for the post-consumer paint market. The market outcomes for paint should be viewed in a full cost accounting framework. Market price, quantity sold, and profitability are limited measures of the success of a market for post-consumer paint. A broader view would take into account the avoided costs of managing the leftover paint and the reduced environmental impacts of recycling the paint. Additional information is needed to determine impacts. The paint market in Oregon encompasses consumers, retailers, and producers, but also PaintCare and Oregon DEQ. In building the market for post-consumer paint, the program will have an impact on each set of actors. However, we have little information on how the Oregon program has impacted each group. Some of the impacts and key questions to consider include: Consumers: The post-consumer paint market will offer a lower- priced alternative to virgin paint. Will the availability of a lower-priced alternative impact the price consumers pay for virgin paint? Will consumers accept post-consumer paint as a viable alternative to virgin paint? Promoting Environmental Results Through Evaluation 74 ------- Retailers: Will retailers begin to stock post-consumer paint? Will the availability of a lower-priced alternative reduce sales of virgin paint? How will profitability of retail paint sales be affected? Producers: Will the availability of post-consumer paint affect producers' sales of virgin paint? Will producers consider expanding their product lines into the post-consumer market and begin offering post-consumer paint? Providing answers to these questions will require extensive and more detailed research into the impacts of post-consumer paint on paint markets in general. Addressing consumer concerns about leftover and recycled paint is important. Information is needed on consumer acceptance of leftover and post-consumer/recycled paints as a viable product for them to use. This basic level of market research would provide a baseline for the potential size of the market and also help to identify specific concerns that consumers may have. These concerns could then be addressed as part of an education and outreach campaign to promote the use of post- consumer paint as a viable alternative to virgin paint. Furthermore, an education and outreach campaign that targets messages to different audiences may be more effective at increasing acceptance by consumers. Although the PPSI Recycled Paint Standard and Certification System project was completed, stakeholders have not funded the Recycled Paint Marketing Guide for Distributors project. The intended purpose of that guide is to educate buyers about the nature of recycled content paint, including its quality and performance. Pricing of leftover/recycled paint may be complex. Some consideration should be given to balanced pricing for leftover paint. A low price for leftover paint may encourage consumers to purchase too much leftover paint, leading to the paint being leftover again. This needs to be balanced against the cost of new paint. In short, leftover paint should be priced at a level that both represents a significant discount compared to new paint but not too low that it encourages consumers to purchase "extra" paint they may not need. Currently, little information on paint markets exists to determine that tipping point and research into consumer preferences may be needed to determine a balance. 75 Evaluation of the Oregon Paint Stewardship Program ------- i3.o Transferability EVALUATION QUESTION: Based on the Oregon experience, what implementation and outcome-related information is required for other states to develop and implement leftover paint management systems? To what extent are the performance measurement and evaluation systems transferable to other states? What are the best ways to communicate the results of the evaluation? 13.1 Data Sources, Methods 13.2 Findings and and Analysis Learnings Evaluation Questions 11 and 12 are designed as pure learning questions. The Evaluation Team developed answers to these questions from data and information collected while answering other evaluation questions and supplemented with interviews conducted with a variety of stakeholders. 13.2.1 What implementation and outcome-related information is required for other states to develop and implement leftover paint management systems? States developing paint management systems should start with reliable baseline data. Baseline data would include: Volume data on paint: Baseline data on paint sales, amounts of paint that are managed through various methods in the state (recycling, conversion to energy, etc.), and amounts disposed. Current infrastructure: Document the extent of current infrastructure (recycling facilities, locations, processing facilities, events, transportation etc.), including: Promoting Environmental Results Through Evaluation 76 ------- Information on the capacity (amount of paint that could be processed or collected). The knowledge and experience of those who would be working with the leftover paint. Attitudes toward recycling and waste reduction among consumers and those who would be working with the leftover paint. A link between infrastructure data and volume data to determine the amounts being collected, processed, or transported within different parts of the infrastructure. A system map: Developing a map of how paint is currently processed in the state provides a useful starting point to developing a new system. The map developed by the PSO (Figure 7) is a starting point. However, a more detailed process flow diagram would be better suited for tracking the current movement of paint in a state. Cost information: Information on the costs processors of paint incur, including costs for HHWs to process paint. Awareness: Information on consumer awareness of disposal and paint recycling options, paint-related environmental issues and hazards, and information on where consumers get their information. Consumer, behavior, attitudes and practices: Information on how consumers decide on how much to buy, what they currently do with leftover paint, and how much leftover paint they currently have stored in their homes. Additionally, information on attitudes toward recycling and the environment would provide useful information that can be used to develop outreach and education (e.g., a consumer base that has strong preferences for recycling and preserving the environment may not need convincing that recycling paint for environmental reasons is a good idea). As California and other states with robust infrastructures roll out leftover paint management systems, these states should first ensure that plans incorporate pre-existing infrastructure into the program. Oregon DEQ allocated a quarter-time employee for its program responsibilities. OR DEQ indicated that a quarter-time employee would have been sufficient in fulfilling program responsibilities, except for the extra demands generated by being the first state (e.g., participating in the Evaluation Committee; requests from other states for information, etc.) (S20). The Oregon program stakeholders identified the following important gaps in the implementation and outcome information currently available: The cost to local governments (e.g., permanent HHW facilities and events) to act as collection sites, including cost of labor and the opportunity cost of storage space. Costs reported by PaintCare may not capture the full costs of the program, as local governments may be "subsidizing" some of the costs by providing labor for paint collection (S20). The proportion of program costs attributable to overhead and PSO Structure. In the Annual Report, PaintCare notes that revealing some vendor costs would result in a breach of contract; there is interest in the cost-effectiveness of employing 'layers" of contractors in order to implement the program (S20). The best approach to recruit retail collection sites. PaintCare used targeted mailings to retailers to recruit them as potential collection sites; as a result, the initial list of retail collection sites was primarily based on retailer self-selection into the program rather than site location. Upon inspection some of the initial sites were not suitable to act as collection sites and that in the future, the organization should visit each site before designating it as a collection site (S17). Another stakeholder interviewee expressed interest in a more strategic approach to recruiting sites (S20). 77 Evaluation of the Oregon Paint Stewardship Program ------- The best approach to measure convenience. The Annual Report focuses on the percent of Oregon residents that live within a certain radius of the nearest collection site as a measure of convenience. A more comprehensive measure would include the considerations addressed by Strickland (2011) that included drive time and driving distance to collection sites. Although the CIS analysis to develop the drive-time and driving distance measures is more complex, Strickland has provided an approach that could be replicated in other states with CIS capacity. The majority of inquiries that PSI received about the program relate to the fee (S19). Other states developing these programs should consider providing outreach to residents in general and paint consumers more specifically related to the fee structure. 13.2.2 To what extent are the performance measurement and evaluation systems transferable to other states? The PPSI Evaluation Committee developed the framework for the evaluation (i.e., Evaluation Committee, evaluation questions, methods, and measures) of the pilot program initially planned for Minnesota. When the Minnesota legislation was not signed into law, PPSI shifted its focus to Oregon. The Minnesota framework was used to develop the full methodology for evaluating the Oregon program and could be transferred to other states as well. In short, the model for this framework involves: Working collaboratively with all stakeholders to define the goals of the program using the PPSI goals as a starting point, but adapting those goals to match state-specific context. Drafting evaluation questions that are relevant for the program based on those goals. The evaluation questions that were developed for Oregon are a starting point, but other states may have other issues to explore. The Evaluation Committee began with 27 questions and narrowed that number to 12 through a participatory prioritization process (Appendix A of Evaluation Methodology) while maintaining the core topics included in the 27 original questions. Because the Evaluation Committee was focused on the opportunity to maximize learning from the Oregon pilot, and provide thorough information and a starting point for performance management of future state-wide paint management programs, it was ambitiously comprehensive in its approach to evaluating this program. Other states may benefit from narrowing the scope of questions and focusing effort and resources on developing comprehensive information about the issues most critical to the success of a similar program in that particular state. Determining the measures and data necessary to answer the evaluation questions and design collection methods for obtaining those data. The Evaluation Committee was ambitious in its design of measures and some measures were not activated or only partially activated because: The program was in the design phase and the level of information that would be available throughout implementation was an unknown. By articulating what would be measured, the Evaluation Committee intended to influence what would be done and what information would be provided. The measures designed are, the evaluation committee believes, the measures necessary to answer the questions, so even if the measures were not activated for the Oregon pilot, they might be a starting point for future similar programs. Other states may want to consider identifying the key measures for each evaluation question and Promoting Environmental Results Through Evaluation 78 ------- work to ensure those data are collected. This requires collaboration and explicit agreements between the stakeholders to ensure that the data is collected, credible, accessible, and timely. The evaluation method document is designed as a flexible starting point; interested states should identify key areas of interest and prioritize matching evaluation questions and measures, as well as plan data collection accordingly. Some key caveats to the transferability of the evaluation framework include that evaluation priorities must be customized to the interests of each state's stakeholders and full development of the methodology requires commitment of resources (time and funding). Additionally, fully integrating evaluation into program design requires close coordination and integration of the Evaluation Committee with the design, planning and implementation teams and efforts. This evaluation committee created a Web-based model (referred to as a "fuzzy logic model" by the Evaluation Team; see Figure 1) of the Oregon pilot program to depict the program and related systems, organizations, processes and products as well as the role of measurement and evaluation in the program. Other states may consider this approach to depicting the program, evaluation design, and ongoing performance management information (e.g performance dashboards). For one, this type of model allows the evaluation team to develop a depiction of the program that can convey to a wider range of audiences, including the public, a sense of what the program is intended to accomplish and how it is intended to operate. Second, a Web-based model allows the program to maintain communication with program participants about performance, results and outcomes. Performance measurement and evaluation can be costly. Building data collection into program operations and creating partnerships offer opportunities to reduce costs and increase the efficiency of data collection and performance management. For example, evaluation questions related to the impact of the education and outreach campaign required baseline and follow- up surveys of aspects of consumer awareness and behavior. The Evaluation Committee expected to receive the necessary data to determine impact of education and outreach from surveys implemented by PaintCare, but the baseline survey performed in August 2010 did not contain some data elements that were needed to assess impact. Additional resources for evaluation would have allowed for baseline and follow-up surveys to have been implemented and conducted based on the evaluation's needs. In the interest of efficiency and reduced costs, the Committee partnered with Duke University, GSU, and two graduate students to perform analyses and data collection. This approach worked well and allowed for data collection and analysis despite funding constraints. Oregon and other states considering implementing product stewardship programs (paint or otherwise) may consider a model similar to the Conservation Measures Partnership's (CMP's) Open Standards for the Practice of Conservation.26 The purpose of the Open Standards is to address the following two key questions: What actions are needed to achieve a program's goals? Are these actions effective in achieving the goals? To provide practitioners with the tools to answer these two questions, the Open Standards "bring together common concepts, approaches, and terminology in conservation project design, management, and monitoring in order to help practitioners improve the practice of conservation. In particular, these standards are meant to provide the steps and guidance necessary for the successful implementation of conservation projects" (CMP, 2007). Figure 35 depicts the approach used by the Evaluation Committee to incorporate evaluation into program design of the Minnesota and Oregon pilot projects on the top (in blue). On the bottom (in green) is the Open Standards model, which depicts the process of ongoing program adaptive management. In Oregon, this process is already well underway, and this report is a part of the program efforts to capture and share learning from the Oregon pilot to encourage more effective practices in the future. 26 www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_Version_2.0.pdf. 79 Evaluation of the Oregon Paint Stewardship Program ------- Figure 35 - Shifting from Incorporating Evaluation into Program Design to CMP Open Standards 1. Evaluation Policy 2. Evaluation Methodology Integrating valuation into" Program Design 1. Team 2. Mission 3. Goals & Objectives 4. Logic Model Data Sources 2. Collection Methods & Strategy Analysis Tools 1. Context 2. Audience 3. Communication 4. Use 1. Conceptualize Define initial team Define scope, vision, targets Identify critical threats Complete situation analysis 2. Plan Actions & Monitoring 5. Capture & Share Learning Develop goals, strategies, assumptions, and objectives Develop monitoring plan Develop operational plan Document learning Share learning Create learning environment OPEN STANDARDS 4. Analyze, Use, Adapt 3. Implement Actions & Monitoring Prepare data for analysis Analyze results Adapt strategic plan Develop work plan and timeline Develop and refine budget Implement plans Promoting Environmental Results Through Evaluation 80 ------- 13.2.3 What are the best ways to communicate the results of the evaluation? The Evaluation Team spent time upfront during this project to identify the information needs of different stakeholders. This upfront communication helped clarify what was needed for effective communication with PSI, Oregon DEQ, PaintCare, rollout states and other stakeholders (e.g., HHWs). However, the Evaluation Team was also operating under tight resources and tight deadlines. To effectively communicate to stakeholders, the Evaluation Team developed a Web-based model of the Oregon program to communicate with stakeholders about the program design, planning, and implementation and findings from the evaluation (www. paintstewardshipprogram.com). Based on the findings under Evaluation Question 1, PPSI participants prefer to use conference calls and email to communicate about the program. Participants also expressed a preference for face-to-face meetings when possible. The best modes for communicating evaluation results likely include the following: Conference calls briefing stakeholders on the results. Emails summarizing results in the format of a newsletter or fact sheet. Presentation of results during a conference or other in-person meetings. The findings under Evaluation Question 1 also suggest that the use of a website is an underutilized mode for information communication. During the PPSI, the website provided fairly limited information for use (i.e. minutes from the most recent meeting) and no opportunities for providing feedback or otherwise interacting with the information. This mode could become a useful venue for sharing evaluation results with a few changes, as follows: Provide more information on a regular basis. Allow readers the opportunity to interact with and respond to the information. Update the information strategically. Add new resources and materials to the website around a focal point, such as a conference call or reporting date, when the material is most relevant to readers. The evaluation website (www. paintstewardshipprogram.com) is a visualization of the program incorporating functional features that provide deep and diverse information about the program. It illustrates program implementation as well as the performance measurement and evaluation system and could be an effective tool for sharing information on an ongoing basis. A website may be accessible to a wider audience than other communication modes (e.g., conference calls) and would allow diverse audiences to focus on different components of the system (e.g., education and outreach materials, costs, and environmental impacts). States should also consider using a dashboard reporting system to provide ongoing updates of program performance. A dashboard system could be incorporated into the web-based program model and would involve reporting a few key results on a frequent basis (e.g., volume of paint collected, number of collection sites, and number of leftover paint gallons sold). Evaluation of the Oregon Paint Stewardship Program ------- i4.o Unexpected Results EVALUATION QUESTION: During the program and for each of its primary components, what were the primary external, unexpected and/or unintended influences and consequences? Not all of the changes in paint management were driven by the program itself. A number of factors could also have influenced paint usage and handling by consumers including economic activity and characteristics of the paint market. Some consideration must be given to external influences on consumer behavior. Additionally, programs often have unintended or unforeseen consequences. This section documents some of these results and their implications for Oregon and other states. 14.1 Data Sources, Methods and Analysis The Evaluation Team answered Evaluation Question 12 by reviewing the lessons learned from the program and through interviewing those involved in program implementation. The interviews covered the different groups involved in planning and implementation and extracted information related to different stages in the program's development. 14.2 Findings and Learning External Minnesota Governor Tim Pawlenty twice vetoed the bill to create an industry-managed statewide paint stewardship and recycling program after the Minnesota legislature had passed legislation twice by near- unanimous margins. Governor Pawlenty's rationale was that the bill would put a double burden on the public, since it authorized a new fee to fund the program, and local governments currently spend public funds to dispose of and recycle paint (Pawlenty, 2009). The PPSI organized collective support from industry, local governments, EPA, and environmental organizations for the Minnesota legislation; the politics in blocking the program was unanticipated and delayed the implementation of a pilot program for 1.5 years (S19). Unexpected Effective collaboration was more time consuming and costly, involving more meetings, calls and committees than expected (See Evaluation Question 1). Few parties involved had any notion of how time-consuming the program would be and did not know what to budget for collaborative work (e.g., meetings, reviewing products, etc.). Additionally, seven years passed from the beginning discussions to implementation of a pilot program - a longer time-frame than was anticipated by most participants. Some retailers declined to show the fee on the receipt. The PPSI anticipated that retailers would show the fee to indicate the extra amount was not part of their price but the PPSI was not aware that reprogramming store systems was such a burden to store managers/owners Promoting Environmental Results Through Evaluation 82 ------- The process became less collaborative when industry was required to submit a plan, negotiate with Oregon DEQ, and implement the approved program plan. This breakdown in collaboration was caused partly by lack of funding for a facilitator/champion (S19, S20). There were a large number of unlabeled and/or rusty cans of paint that cannot be accepted by the program. The collection sites are instructed to not open the cans of paint and those cans are treated as non-program materials which must be managed by Oregon. (See Evaluation Question 5) PPSI had limited representation from retailers and so the fact that retailers play a large role in the Oregon paint collection infrastructure was surprising (S18). Additionally, large retail chains (e.g., Lowe's, Home Depot, and Wal- Mart) declined to participate. That legislation was needed to implement a program was unexpected. The need for legislation arose when issues of anti-trust and collusion arose since paint producers would be working together. Thus, legislation was needed to exempt the paint producers from anti-trust and collusion concerns. At the start, the idea of the program was to avoid having states pass legislation to tell the private sector what to do the program was intended to be voluntary. The lack of a champion/funder to collect real world data under the Oregon pilot to inform appropriate levels of virgin paint offset to model in an LCA meant that no additional progress was made on estimating the environmental impacts of paint management approaches. The evaluation committee was unable to get cost data broken out by category to perform a more thorough analysis. Unintended The private-public partnership and the waste hierarchy model (reduce, reuse, recycle) were at odds in this program. The public preference in the waste hierarchy model is for reduction and reuse. There were opportunities to stress both reduction and reuse under this program. However, the program as-designed tended to focus more on collection and recycling. Other state programs should consider a clear articulation of the program priorities to ensure alignment with existing program and policy goals and reduce the potential for confusion amongst consumers. The reliance on retailers as collection locations had a negative impact on the program's ability to divert high quality leftover paint for reuse, which is ranked as a preferred use. Further exploration into this topic what are the obstacles (e.g., liability and competition for the sale of virgin paint) and solutions (e.g., redistribution to HHW collection locations or Reuse type stores)is needed. 83 Evaluation of the Oregon Paint Stewardship Program ------- i5.o Reference Materials and Abt Associates (Abt). 2007. Quantifying the Disposal of Post-Consumer Architectural Paint, U.S. Environmental Protection Agency - Sector Strategies Program. Amazon. 2010. Amazon Environmental, Inc. Latex Paint Recycling. Available at: www.amazonpaint.com/ other-products.html. Bledsoe, Wes, Eric Graves, and Andrei Roman. 2011. Description of the Oregon Pilot Program Paint Stewardship Organization. Prepared for PPSI Evaluation Committee and PMAP 8900 Public Service Capstone course in the Andrew Young School of Policy Studies, Georgia State University. Bradshaw Advertising. 2010. PaintCare Awareness Survey of Oregon Homeowners. Report prepared for PaintCare. Braunz, Amy, Julie Calvin, Whitney Knapp, Hedrick Strickland. 2010. Assessment of Collaboration of the Paint Product Stewardship Initiative (PPSI) and Pilot Program. Prepared for PPSI Evaluation Committee and ENV 280 Social Science Survey Design course at Duke University. Cascadia Consulting Group (Cascadia). 1998. Paint Smart Pilot Program Evaluation and Final Report. Prepared for Oregon Department of Environmental Quality. Available at: www.deq.state.or.us/lq/sw/ prodstewardship/painthtm. Conservation Measures Partnership (CMP). 2007. Open Standards for the Practice of Conservation. Available at: www.conservationmeasures.org/wp-content/uploads/2010/04/CMP_Open_Standards_ Version_2.0.pdf. Cousins, J. Bradley and Lorna M. Earl. 1992. "The case for participatory evaluation," Educational Evaluation and Policy Analysis, 14, 397-418. p.399 Cousins, J.Bradley and Elizabeth Whitmore. 1998. "Framing Participatory Evaluation," New Directions for Evaluation, No. 80, Winter 1998, 5-23. Eastern Research Group, Inc. 2010. Method for Evaluating the Paint Product Stewardship Initiative's Oregon Pilot Program. Prepared for U.S. EPA Evaluation Support Division. Promoting Environmental Results Through Evaluation 84 ------- Hult International Business School Project Team 2. 2011. Paint Recycling: An Environmentally and Economically Sustainable Business Model. Boston, MA. Keene, Matt, Lou Nadeau, Amy Stillings and Jennifer Nash. 2010. Memo entitled "Leftover Paint Oregon Pilot Project Evaluation: Prioritizing Evaluation Questions," Oregon Paint Stewardship Pilot Program Environmental Committee. Available at: www.paintstewardshipprogram.com/images/pe_leftover_ paint_oregon_pilot_project_evaluation_prioritizing_evaluation_questions.DOC Keene, Matt. 2008. Integrating Evaluation into Program Design. Available at: www.productstewardship. us/displaycommon.cfm?an=l&subarticlenbr=754. Available at: www.paintstewardshipprogram. com/images/pe_integrating_evaluation_into_program_design_06_01_08.DOC and www. paintstewardshippro gram.com/images/pe_inte grating_evaluation_into_oregon_pilot_program.DOC Metro. 2011. MetroPaint: Quality Recycled Latex Paint, www.oregonmetro.gov/index.cfm/go/by.web/ id=521. National Paint and Coatings Association. February 2008. The Paint Industry Works Toward a Nationally Coordinate System for Post-Consumer Paint Management, Issue Backgrounder. Available at: www.paint. org/pubs/background.cfm. Oregon Legislative Assembly. 2009. Committee of Sustainability and Economic Development, 75th Assembly, 2009 Regular Season. HR 3037-A. Available at www.leg.state.or.us/09reg/measures/ hb3000.dir/hb3037.en.html. Oregon Department of Environmental Quality (DEQ). 2011. Oregon Household Hazardous Waste Report, 2008-2009. OR DEQ. 2008. Oregon Department of Environmental Quality Household Hazardous Waste Survey: Results Report. Portland State University Survey Research Lab: Portland, OR. PaintCare. 2010. Oregon Paint Stewardship Plot Program Plan. Washington, DC. PaintCare. 2011a. PaintCare Awareness Survey of Oregon Residents. Washington, DC. PaintCare. 2011b. Oregon Paint Stewardship Pilot Program Annual Report. Washington, DC. Patton, Michael Quinn. 2010. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. The Guilford Press. Governor Tim Pawlenty. 2009. "Letter to President Metzen," State of Minnesota (May 21, 2009). Available at: www.calpsc.org/products/docs/2009/5-21%20Metzen%20VETO%20CH%20121%20 SF%2 0477.pdf 85 Evaluation of the Oregon Paint Stewardship Program ------- Paint Product Stewardship Initiative (PPSI). 2004. Paint Product Stewardship Initiative Memorandum of Understanding, Available at: www.productstewardship.us/displaycommon. cfm?an=l&subarticlenbr=118. PPSI. 2007. Final Paint Product Stewardship Initiative 2nd Memorandum of Understanding. Available at: wwwproductstewardship.us/displaycommon.cfm?an=l&subarticlenbr=118 PPSI. 2008. Paint Product Stewardship Initiative paint Demonstration Project Committee Draft Work Plan. Product Stewardship Institute (PSI). 2004a. Paint Product Stewardship: A Background Report for the National Dialogue on Paint Product Stewardship. Boston, MA. PSI. March 2004b. Product Stewardship Action Plan for Leftover Paint. PSI. 2009. "Paint Dialogue Stakeholder Meeting: Final Meeting Notes," Portland, Oregon. December 9-10, 2009. Available at: www.productstewardship.us/associations/6596/files/FINAL_PPSI_Meeting_ Summary_Portland_OR_Dec9-10_2009.doc PSI. 2010. Extended Producer Responsibility Legislation as of August 2010. Available at: http:// products tewardship.us/displaycommon.cfm?an=l&subarticlenbr=615. SCS Engineers and Cascadia Consulting Group (SCS and Cascadia). 2007. Paint Product Stewardship Initiative Infrastructure Project, Washington Department of Ecology. Available at: www. products tewardship.us/displaycommon.cfm?an=l&subarticlenbr=12 8. Strickland, Hedrick. 2011. Convenience analysis of the Oregon Paint Management Pilot Program. Duke University, Masters Thesis, advised by Dr. Lynn Maguire and Matt Keene, EPA. Thompson, Ann Marie, James L. Perry, and Theodore K. Miler. 2007. "Conceptualizing and Measuring Collaboration," Journal of Public Administration Research and Theory, 19: 23-56. Torres, Preskill and Piontek. 2005. Evaluation Strategies for Communicating and Reporting: Enhancing Learning in Organizations. 2nd Ed. Sage Publishing. NOTE: Stock photography purchased from www.123rf.com. Special thanks to Duke University for permission to use the logo and image of Nicholas School of the Environment on page 24. Promoting Environmental Results Through Evaluation 86 ------- Interviews and Personal Communications K*Q ' itact Number Contact and Organization ^^ Stakeholder Interview Number [S] S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S20 S21 Abby Boudouris, Oregon DEQ Steve Sides, ACA Jim Quinn, Metro Jim Quinn, Metro [HHW] Abby Boudouris, Oregon DEQ Alison Keane, PaintCare Tom Wroblewski, Home Depot [Retailer] Rick Barnard, Rodda Paint Company [Retailer] Jack Wickham, Ace Hardware [Retailer] Larry Gibbs, Lane County [HHW] Timm Schimke, Deschutes County [HHW] Roy Weedman, Columbia County [HHW] Jennifer Purcell, Tillamook County [HHW] Tina French, Lincoln County [HHW] Garry Penning, Rogue Disposal [HHW] Jim Quinn, Metro Alison Keane, PaintCare Barry Elman, EPA Scott Casell, PSI Abby Boudouris, Oregon DEQ Lorraine Segala, Amazon Environmental 09-23-2010 12-2010 12-2010 06-27-201 1 03-02-201 1 03-14-2011 05-03-201 1 05-09-201 1 05-13-2011 06-21-2011 06-21-2011 06-21-2011 06-28-201 1 06-29-201 1 07-01-2011 08-26-201 1 08-26-201 1 09-02-201 1 09-06-201 1 09-06-201 1 09-14-2011 Person Communications With ERG Via Email [Em] Em1 Em2 Em3 Em4 Em5 Em6 Em 7 Em 8 Jim Quinn, Metro Alison Keane, PaintCare Mike O'Donnell, PaintCare Jim Quinn, Metro Albert Salvi, Washington State Department of Ecology Alison Keane, PaintCare Alison Keane, PaintCare Alison Keane, PaintCare 08-24-201 1 08-26-201 1 09-14-2011 09-15-2011 09-08-201 1 09-08-201 1 09-09-201 1 09-12-2011 87 Evaluation of the Oregon Paint Stewardship Program ------- Recycled/Recyclable Printed with vegetable oil based inks on 100% postconsumer, process chlorine free recycled paper. ------- |