4>EPA Measurement Tips & Resources for Community Projects Community Action for a Renewed Environment (CARE) www.epa.gov/CARE ------- LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES. Measurement Tips and Resources This document is an introductory resource for new and seasoned Community Action for a Renewed Environment (CARE) communities looking for ways to measure progress made by their community projects. To receive a cooperative agreement under the CARE program, all grantees propose ways of measuring what they expect to achieve through their funded activities. This document gives a few basic measurement principles and some tips. Throughout the document we share examples and suggest resources for additional information. Basic principles Establish measures of success for your work based on your workplan goals and strategies. Gather data for each measure and use the data to assess progress. Revise activities as needed based on your assessment. Regularly report progress to your community and to the funders. 1. Establish measures of success for your work based on your workplan goals and strategies. What will success look like? How will you know you are achieving it? These are questions that are critical to project planning and management. The use of measurement enables you to demonstrate to yourself, the community, and other outside parties that you have thoughtfully weighed what work needs to be done to evoke positive environmental changes, and that you are accomplishing your goals and documenting progress. Choose measures touching on each key work area and use them to inform your work. Key work areas for CARE projects include building and maintaining a strong collaborative partnership, achieving environmental and public health benefits, and developing community capabilities. During your grant project, what measures can you select that will keep the partnership focused on meeting their goals? Definitions... A performance measure is a unit of information tracked over time that helps gauge the degree of success a project has had in achieving its stated objectives, goals, and planned activities. Performance measures may address the type or level of activities conducted (process), the direct products and services delivered (outputs), or the results of those products and services (outcomes). Performance measurement is the ongoing monitoring and reporting of accomplishments, particularly progress toward pre-established goals. To be most effective, each measure should include a: Baseline: the current state of performance Target: the desired level of performance Timeline: the goal date for when the desired level of performance will be achieved www.epa.gov/care ------- LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES. Select a few behavior change measures for your work, so that you can monitor the effect you are having on the ground. What key behaviors do you aim to influence? Write these behaviors down and determine ways to measure these actions, monitor them throughout your grant, and report your successes. Focus on a few strong environmental/human health measures and back them up with a solid calculation methodology. 2. Gather data for each measure and use the data to assess progress. Revise activities as needed based on your assessment. Evaluation followed by revision or adjustment is a critical component of good project management and sustaining a long-term project. Participants in the partnership and project need to have a clear and open way to develop lessons learned, correct mistakes, and celebrate breakthroughs. Without this, the effort will eventually fade away or break down. Determine a method to evaluate how as a collaborative you plan to learn from your successes and failures and grow as a group. 3. Regularly report progress to your community and to the funders. You will be required to report to funders on a regular basis about the progress you are making. Make this a reflective and valuable time to assess your work and processes. Report your findings and readjust the plan, if necessary. Example project goals and measures... The Boston Safe Shops Project had the goal of addressing negative environmental and public health impacts that small automotive repair and body shops have on their employees and neighbors, while avoiding putting these shops out of business by regulatory enforcement. Example knowledge and behavior change project measures... Number of in-shop tailgate training sessions conducted for shop workers and owners on pollution prevention and worker protection techniques, Changes in knowledge and practices Target: 40+ training sessions Timeline: Within the 2 year grant period Baseline and measurement: Baseline knowledge and practice established by pre-training questionnaire and shop assessment with behavior and knowledge change measured by follow up assessments one or more months after the training Number of shops that make at least one major physical improvement to reduce pollution and worker exposure (e.g., new equipment, improved ventilation) Target: 10 shops Timeline: Within the 2 year grant period Baseline and measurement: Baseline conditions measured by initial shop environmental assessments with adoption of changes measured by follow-up assessments and self-report/observations during follow- up visits Example estimated environmental results of behavior changes... 1 shop switched to waterborne paints, reducing VOCs by about 1,200 Ibs/yr Lead wheel weights were eliminated from 125 fleet vehicles about a 260 Ib elimination in lead www.epa.gov/care ------- LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES. To learn more... Read an introductory guide on measurement: Measuring what Matters: The Challenge of Quantifying Social Change This 16-page document outlines one organization's thinking on how social purpose organizations can approach the challenge of measurement. Check out the "Seven Practical Tips for Measuring What Matters" beginning on page 12. http://www.metgroup.com/assets/654 measuringwhatmatters0819.pdf Read an introductory guide to evaluating your own program: Taking Stock: A Practical Guide to Evaluating Your Own Program This resource is a practical guide to program evaluation written for community-based organizations. It provides information that you can use to help improve your programs. For those just getting started it includes sections on defining goals and objectives, using quantitative and qualitative data, and strategies for data collection. http://www.horizon-research.com/reports/1997/stock.pdf Read step-by-step "how to" information on developing measures and collecting data: Measuring Progress: An Evaluation Guide for Ecosystem and Community-Based Projects Pages 49-96 cover the topics "How will you know you are making progress?" and "How will you get the information you need?" and are most relevant for partnerships that are developing or refining their performance measures. This guide was written for an audience working on community based conservation or ecosystem management, but may be used more broadly. http://www.snre.umich.edu/ecomgt/evaluation/documents/Measuring%20Progress.pdf Tips TIP 1: Talk to your EPA Project Officer. He or she is your main resource as well as your door to other resources within EPA. Your Project Officer may not be a measurement expert himself or herself, but by talking to your Project Officer you can tap into other resources within EPA, including measurement experts within the EPA CARE family. If you are struggling with measurement and have not spoken about it, share your concerns the next time you talk. TIP 2: Consider using a "logic model." A logic model is a visual model that shows the relationship between your work and your desired results. A logic model can serve as a basic road map for the project, explaining where you are and where you hope to end up. The Web sites listed on the next page provide information on how to develop a logic model and how to use a logic model as a tool to develop your project measures. On the last page of this document we include an example of how a logic model can be used to develop measures. In its simplest form, a logic model looks like: if then if then INPUTS OUTPUTS OUTCOMES What is invested What is done What results www.epa.gov/care ------- LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES. To learn more about logic models... Take an online course: A University of Wisconsin Extension Service online course on enhancing performance using logic models http://www.uwex.edu/ces/lmcourse/ Read or download a manual: The Kellogg Foundation guide to developing logic models http://www.wkkf.org/knowledge-center/Resources-Page.aspx (search on "logic model" to pull up the guide) Watch and listen to an online Web seminar: An online Logic Model Webinar Sponsored by EPA and CDC Available as an archived presentation and audio recording on the Communities in Action for Asthma Friendly Environments Network site Developing a Logic Model and Focusing Your Evaluation (December 17, 2008) http://www.asthmacommunitvnetwork.org/webinars/program evaluation basics.aspx Look at some examples: EPA Region 10 Web page with sample logic models gives definitions and shows sample logic models for grant programs http://yosemite.epa.gov/R10/ECOCOMM.NSF/webpage/measuring+environmental+results TIP 3: To estimate environmental outcomes, look for available "calculators." The CARE Program has compiled a number of available online calculators and existing calculation methodologies in a single document available to you in electronic copy by request (contact mandolia.michelle@epa.gov). Work with your partners and Project Officer to see what else might be available. If you are using a methodology or calculator that is not yet in the CARE compilation, we would like to know about it so that we may share it with other communities. Establishing the calculation methodology you will be using (including any existing calculator you plan to use) at the beginning of your project ensures that you will have the data to run your calculations when the time comes. Examples of just a few of the calculators include: a diesel emissions quantifier, an anti-idling calculator, and the National Recycling Coalition Environmental Benefits Calculator. TIP 4: Look to your partner organizations or new partners for expertise. Have you engaged your partnership in a conversation about work planning, measurement, and evaluation? Perhaps there is an untapped resource in your midst. If not, is there a new partner you could approach? Some of our CARE grantees have had success partnering with local university students on measurement and evaluation. Could there be someone with the expertise to advise or guide your organization through the process? www.epa.gov/care ------- LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES. TIP 5: Look to your partner organizations or new partners for data. Is someone else already gathering the data that could help you understand whether your actions are having their intended affect? If not, does it make sense for someone else to gather the data? Example of partnering for successful measurement... The Superior Watershed Partnership in Marquette, Ml sought to reduce mercury discharges from dental offices to the local wastewater treatment facility as a part of their Level II CARE grant. The Partnership proposed a low cost solution to members of the regional dental association, who voted unanimously to participate and 100% of whom then voluntarily installed mercury amalgam separators in their offices. How a partner helped with data... The Marquette wastewater treatment facility provided the Superior Watershed Partnership with data regarding mercury concentrations in the water. First, this data helped the Partnership pinpoint dental offices as one major source of mercury, leading to the particular reduction strategy they pursued. This data then, over time, also told the Partnership that their efforts were working. The Superior Watershed Partnership in cooperation with the Marquette Wastewater Treatment Plant documented mercury reductions as much as 19% in the effluent entering Lake Superior. Additional note on measurement: The Partnership also tracked which dental offices had installed the amalgam separators, enabling them to follow up with those who had not. The Superior Watershed Partnership is a non-profit and all of these results were achieved without the need for additional regulation or policies. TIP 6: Talk to your EPA Project Officer about whether the TASC contract might be right for you. Through the Technical Assistance Services for Communities (TASC) contract, EPA offers assistance to help a community better understand the environmental issues confronting it and to be well informed while participating in the decision-making process. Some funds are available to CARE communities through TASC for certain educational and technical assistance not budgeted for in their original workplans. These activities could include work that would serve your measurement and evaluation needs as you seek to manage your project and partnership effectively. If you would like to provide comments on, examples for, or suggestions for revisions to this document, please send them to Michelle Mandolia at: mandolia.michelle@epa.gov www.epa.gov/care ------- EXAMPLE: A (very simplified) Childhood Lead Prevention Program INPUTS Funds Trained staff Partnerships with organizations Legal authority LOCAL PARTNERSHIPS. HEALTHY COMMUNITIES. OUTPUTS Outreach Screening Identification Case management Family training Medical treatment OUTCOMES Lead sources identified Lead sources eliminated Families adopt in-home techniques Elevated blood lead level reduced INPUTS OUTPUTS OUTCOMES Measurement Question: What do you want to know about your program performance? To what extent is the program building the local community health worker capacity? Will we have resources to maintain or grow our work? To what extent are families getting medical case management? Are families pleased with the services we have to offer? To what extent are lead sources in the home being eliminated? To what extent are families adopting the in home techniques we are sharing? How will you measure it? % of staff hired from community (75% target) Staff retention goal of at least 2 years each (80% target) # of new sources of funding by the end of the grant period (Target of 2) # of families receiving medical case management/* of families referred % participation of families receiving in home visits in at least one other event (25% target) # of homes where lead containing items were replaced/* of homes where they were identified % of families with elevated blood lead level children that clean all window sills and floors with the designated cleaning solution each week What data do you need to collect? Where staff live Staff start dates Counts of new funding sources Counts of: families referred for care families receiving care Names of: families receiving home visits participants in other events Counts of: homes where items were identified homes where items were replaced Count of families implementing key best practice each week How will you collect the data? Applicant resumes or applications Staff hiring paperwork Letters of award or commitment Home visit logs Health clinic numbers Log of all families receiving home visits Sign in sheets/logs from other events Home visit logs www.epa.gov/care June 2010 EPA-530-F-10-003 ------- |