SERA EPA RECYCLING EDUCATION AND OUTREACH GRANTS A Quick Reference Guide for Evaluating Progress and Reporting Achievements Over Time EPA Office of Resource Conservation and Recovery January 2023 ------- About This Guide EPA is thrilled about the interest in the new Recycling Education and Outreach grant program, funded by the 2021 Infrastructure Investment and Jobs Act! EPA developed this Quick Reference Guide to provide potential Recycling Education and Outreach grant program applicants with information about the measures of success they will need to report on over the course of a project, should they be awarded a grant under this program. This guide is intended to serve as an overview of the more detailed Grant Evaluation Guide that EPA will provide to grant awardees as a tool for measuring the effectiveness of their grant-funded projects. EPA also will provide grant awardees with additional resources and technical assistance to measure their project's success and share information about its impact with EPA, their community, and communities across the country. Thanks to all who are considering applying for this funding opportunity! ------- Table of Contents I. Background 1 II. What measures of success will awardees need to report on over the course of their REO Grant project? 1 III. How will awardees be asked to report on each measure of success? 2 Number of Outreach Campaigns or Projects Implemented 2 Number of Communities Reached 2 Number of Temporary or Permanent Jobs Created 3 Change in Volume of MSW Collected, Recycled, Composted, or Managed 3 Percentage Change in Participation Rate 4 Reduction of Contamination in the Recycling Stream 4 IV. When will grant awardees be required to report results to EPA? 5 V. How can logic models help define projects and showcase their results? 5 VI. What other resources are available to help grant applicants and awardees think through measures of success? 6 Appendix A. Using a logic model as a roadmap for success 6 Example of a Logic Model 6 ------- I. Background EPA's Office of Land and Emergency Management is implementing the Recycling Education and Outreach (REO) grant program under the Infrastructure Investment and Jobs Act, Public Law 117-58, Section 70402, commonly referred to as the Bipartisan Infrastructure Law or BIL. The REO grant program will fund projects that: Inform the public about new or existing residential or community recycling programs; Provide information about the recycled materials that are accepted as part of a new or existing residential or community recycling program that provides for the separate collection of residential solid waste from recycled material; and Increase collection rates and decrease contamination in new or existing residential and community recycling programs. The BIL also directs EPA to develop a Model Recycling Program toolkit that states, tribes, and units of local government can use in carrying out the grant program. One of the required resources in the toolkit is a guide to measure the effectiveness of a recycling education and outreach grant. This Quick Reference Guide is intended to provide an overview of reporting requirements for potential applications. II. What measures of success will awardees need to report on over the course of their REO Grant project? There are different measures of success that can help tell the story of progress for a project funded under this grant program. The measures of success that each REO grant awardee must report on include: ~ O Number of outreach campaigns or projects implemented. Number of communities reached through outreach activities. Number of temporary or permanent jobs created. Change in the volume (measured in tons) of municipal solid waste (MSW) collected, recycled, composted, or managed through other management pathways. Percentage change in participation rate of the recycling or composting program. Reduction of contamination in the recycling stream. Of these measures of success, some focus on activities or work products that are developed through a project. These are often referred to as project outputs. An example of a project output is the number of outreach campaigns implemented. Other measures of success focus on the results achieved because of a project activitythe results could be behavioral, environmental, or health-related. These are often referred to as project outcomes. An example of a project outcome is a change in a recycling program's participation rate. (For additional information on outputs and outcomes, please refer to Section I.L of the Request for Applications, Measuring and Reporting Environmental Results: Outputs and Outcomes.) ------- III. How will awardees be asked to report on each measure of success? The REO grants will have a three-year term for awardees to complete their projects. Over those three years, an awardee will be expected to share meaningful information about measures of success on a quarterly basis, as specified in the REO Request for Applications. Short-term reporting is typically focused on output measures, such as number of outreach campaigns or projects implemented. For other measures of success, awardees may need more time to collect information on the project's progress. These longer-term measures are typically outcome measures, such as percentage change in participation rate, or reduction of contamination in the recycling stream as a result of outreach activities funded by the grant. A brief overview of the ways awardees can estimate each measure of success is provided below. Number of Outreach Campaigns or Projects Implemented Why This Output Is Important: Helps to communicate the project's level of effort and audience reached. What To Measure: The number of implemented outreach campaigns and activities, such as the number of public service announcements delivered, the number of informational materials distributed, the number of education and outreach campaigns executed, the number of toolkits for municipal and commercial recycling programs produced, the number of labeling signs created, etc. Measuring Success Over Time: There are several activities that awardees could count to estimate this measure of success from the outset of the project. As soon as an activity is complete, awardees can begin tallying the results. Examples of activities awardees may consider reporting on include: - Number of hard-copy materials distributed. - Number of advertisements showed. - Number of views of advertisements. - Number of social media posts made. - Number of "likes" or "comments" on social media posts. - Number of feedback sessions held. - Number of feedback responses received. - Number of educational labels or signs distributed. - Number of individual households (or the population of those households) receiving labels. - Number of businesses receiving labels. - Daily population of business facilities using labels. Number of Communities Reached Why This Output Is Important: Helps to communicate the project's level of effort and audience reached. (It's an output, not an outcome, because it does not measure the overall impact of the program on solid waste management.) What To Measure: The number of communities and community members reached through outreach activities funded by this grant, including disadvantaged communities. Reports should share information about the communities reached, including land area, population, location, type of community (e.g., city, township, village, tribal), and any other descriptive factors, such as whether the communities are designated rural or disadvantaged communities (as defined in the Request for Applications). ------- Measuring Success Over Time: There are several approaches for estimating this measure of success as awardees complete specific project activities. Examples metrics that awardees may considering using are listed below. Awardees may also create their own metrics. - Number of communities in which activities were conducted. - Number of disadvantaged communities in which activities were conducted. - Number of community members reached through grant project activities. - Number of disadvantaged community members reached through grant project activities. Number of Temporary or Permanent Jobs Created Why This Output Is Important: Helps to communicate the project's contribution to and impact on the local economy through recycling-related activities. What To Measure: The number of temporary and/or permanent jobs created through grant project activities. Measuring Success Over Time: There are several approaches for estimating this measure of success over time. Awardees may consider using the options below. Awardees can also create another metric to better reflect a specific project. - Number of employees paid using grant funds (include hours per week for part-time employees). - Number of jobs created due to an increase in recycling volume during the grant period (include hours per week for part-time employees). Change in Volume of MSW Collected, Recycled, Composted, or Managed Why This Outcome Is Important: Helps to communicate the project's impact on changing solid waste management in the community by comparing conditions before and after the project. What To Measure: For this measure of success, awardees must calculate two metrics: - The change in the total tons of MSW collected, recycled, composted, or managed via other management pathways as a result of grant project activities. EPA's Measuring Recycling: A Guide for State and Local Governments provides detailed information on how to measure progress. Awardees can report progress as a change in volume (measured in tons) and as a percentage change from the baseline. - GHG emissions reduced (in metric tons of carbon dioxide equivalent) as a result of grant project activities. Awardees can use EPA's Waste Reduction Model (WARM) to produce high-level estimates of potential GHG emissions reductions, energy savings, and economic impacts from several different waste management practices. Measuring Success Over Time: There are several approaches for estimating this measure of success over time. Awardees should modify the approach based on the types of MSW and management pathways for a specific project. Awardees must report both metrics. - Total tonnage of MSW collected, recycled, composted, or managed via other pathways before grant project activities and after grant project activities. - Total GHG emissions (calculated using WARM) before grant project activities and after grant project activities (based on total tonnage). ------- Percentage Change in Participation Rate Why This Outcome Is Important: Helps to communicate the project's impact on changing recycling behavior in a community by comparing recycling behaviors before and after the grant project. What To Measure: The change in participation rate of the recycling program funded by the grant. This means defining participation for the grant project and then developing a method for measuring a change in participation resulting from the project. Measuring Success Over Time: There are several approaches for measuring success over time. Awardees will need a metric for defining participation and a corresponding metric for measuring the change in participation as a result of the grant project. Consider the options listed below. If needed, awardees can create a set of participation-related metrics based on a specific grant project. Possible Metrics for Participation Rate - Number of households that put out curbside recycling once per week or month. - Number of residents utilizing a drop-off recycling site once per month or year. Possible Metrics for Participation Rate Change - Number of households participating in the curbside pickup program divided by total number of households in the community, calculated both before and after the grant project. - Number of drop-offs per year/month at the recycling site divided by the number of households the site serves, calculated both before and after the grant project. Reduction of Contamination in the Recycling Stream Why This Outcome Is Important: Helps to communicate the project's impact on changing recycling behavior and the overall impact on solid waste management in the community by comparing recycling contamination before and after the grant project. What To Measure: The reduction of contamination in the recycling system as a result of the grant project activities. What constitutes recycling contamination varies from community to community, depending on the range of acceptable materials in a recycling program. Awardees will need to define contamination for the recycling system and then develop a method for measuring the reduction of that contamination as a result of the grant project activities. Measuring Success Over Time: There are several approaches for estimating this measure of success over time. After defining contamination, awardees will need a corresponding metric for measuring the reduction in contamination as a result of the grant project. Awardees can use the metric below or create a set of contamination related metrics for a specific grant project. - Monitor contamination of recycled, composted or anaerobically digested materials from facilities in the community to show changes before and after the grant project. EPA recognizes that there is no single best way to collect contamination data and that not all programs want or need to perform data collection in the same way. Different data collection strategies may be used, and awardees should choose the method that's right for activities associated with a specific grant project, based on the scale and focus of the effort. For example, if an awardee wants to measure inbound contamination: - Data collection at the Material Recovery Facility (MRF) level or community level can involve completing a recycling characterization/composition study or conducting a contamination audit of the MRF. - Route- or neighborhood-level data collection can involve auditing what is collected by trucks. ------- - Household- or building-level data collection can involve curbside sampling or container audits, lid flipping, and/or cart tagging and tracking (visual audits). Whatever method an awardee selects, it is important to gather baseline data as well as regular follow-up data. This will enable an awardee to compare the rates of recycling contamination before and affer the grant project. IV. When will grant awardees be required to report results to EPA? Awardees will need to share quarterly information on their measures of success over the three-year project period. After the grant period ends, awardees will have 90 days to share information on all their measures of success in a final technical report. Both the quarterly reports and the final report should describe progress made on the measures of success. The reports may include information on numeric targets set for a quarter or the year, a description of the methodology used to measure progress, and/or discussion of planned actions that are intended to achieve progress on the measures of success (for activities that haven't started yet). EPA will provide reporting templates to help awardees complete their quarterly and final reports. EPA recognizes that awardees will be able to report quarterly on some measures, while reporting for other longer-term measures might not occur until the end of the three-year project period. The example timeline below shows how the timing for collecting information and reporting on different measures might work over the course of a project. Hire staff Engage communities in targeted area to inform design of campaign r Gather baseline contamination data - Gather baseline data on recycling and/or composting rate - Gather baseline data on current participation rate Use baseline data gathered to design outreach campaign Produce outreach and/or campaign materials Implement outreach campaign Track number of Gather data on participation rate Gather data on contamination 1 Gather data on recycling and/or composting rate ¦ Gather data on current participation rate Report results YEAR 1 YEAR 2 YEAR 3 V. How can logic models help define projects and showcase their results? There's a lot that goes into a grant project success story: the resources, the audiences, the activities, and the measures of success. The relationship between these project components showcases the big picture of how a project works and how it will make a difference in a grant awardees community. One tool that can help define a project's goals and showcase its results is a logic model. A logic model is a visual way to illustrate the resources or inputs required to implement a program, the activities and outputs of the program, and the desired program outcomes (both short-term and long-term)1. A logic model can help with both project formulation and reporting results. For awardees who wish to develop a logic model, EPA has provided an example of a logic model as an Appendix to this Guide. 1 https://www.acf.hhs.aov/sites/default/files/documents/prep-loaic-model-ts O.pdf 5 ------- VI. What other resources are available to help grant applicants and awardees think through measures of success? EPA is here to help! Please visit the REO website for resources to help interested applicants prepare an application. EPA hosted a series of webinars to answer questions. Recorded version will be available on the REO website under Webinars. Email EPA with any grant questions to RecvclingEd@epa.gov. Appendix A. Using a logic model as a roadmap for success Logic models are useful tools for visually demonstrating the relationship between planned work and desired results. They show the outputs a project is designed to produce and the outcomes the awardee wants to achieve, to accomplish the goals and objectives of the project. They communicate the performance story of the project, focusing attention on the important connections between actions and results. A logic model can serve as a basic road map for the project, explaining the current and desired future state. For awardees who wish to develop a logic model to help showcase the effectiveness of their activities, the first step is to identify key aspects of the project: Resources: Funding, staff time, and other resources that are invested into a project to reach educational and environmental goals. Activities: Actions the awardees and their partners undertake to produce the desired outcomes and reach the educational and environmental goals of the project. Audience: The intended recipients of activities, efforts, and/or work products. Outputs: Numbers and types of activities, efforts, and/or work products that are produced or provided during the project period, as well as the audiences that participate in those activities. Outcomes: The results, effects, or consequences that will occur from carrying out a program or activity that is related to a programmatic goal or objective. Example of a Logic Model The following example of a logic model is based on a fictitious recycling education campaign. EPA's use of this example does not constitute an endorsement of or preference for any of the activities presented. A city has decided to launch a recycling education campaign to target contamination in curbside recycling. The city will run a 10-week campaign to tag recycling carts that contain unacceptable items and to reward residents whose carts exemplify best practices. The Department of Public Works (DPW) will be responsible for carrying out this effort, in coordination with the city's contracted hauler. The campaign seeks to eliminate plastic wrap, bags, and film from single- and multi-family household recycling carts. DPW has conducted focus groups and research to determine that direct cart messaging can be an effective education strategy. The city plans to inspect carts on 20 recycling routes and leave tags when nonrecyclable items are found (e.g., plastic wrap, clothing, yard waste, furniture, or scrap metal). The cart inspectors will also look for carts that show good recycling practices (e.g., empty and well-rinsed recyclables placed in a government-issued bin, with no unacceptable items). The inspectors will give gift bags to those residents. ------- Resources EPA grant funding Recycling route staff time Cart tags Swag bags Audience research Activities & Audience Print bin and cart tags and distribute to city residents who have contaminated recycling Notify residents with exemplary carts and distribute swag bags Conduct media outreach to inform public about DPW efforts Conduct social media outreach Conduct door-to-door education and outreach campaigns to reach community members not likely on social media Train staff working on recycling routes to identify contamination and tag noncompliant carts Outputs Number of bins tagged Number of swag bags distributed Number of recycling routes targeted for tagging Number of news articles and stories covering the campaign Number of social media posts and interactions about the campaign Number of households visited Number of staff trained to identify contaimination and apply tags Number of employees paid using grant funds Number of jobs created due to increase in recycling volume during grant period Outcomes 20% increase in volume of recycled material collected 10% increase in participation rate of the recycling program Decrease in contamination in recycling carts to 4% for targeted audiences 7 ------- |