4>EPA
Measurement Tips & Resources
for Community Projects
Community Action for a Renewed Environment (CARE)
www.epa.gov/CARE
-------
LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES.
Measurement Tips and Resources
This document is an introductory resource for new and seasoned Community Action for a
Renewed Environment (CARE) communities looking for ways to measure progress made by
their community projects. To receive a cooperative agreement under the CARE program, all
grantees propose ways of measuring what they expect to achieve through their funded
activities. This document gives a few basic measurement principles and some tips. Throughout
the document we share examples and suggest resources for additional information.
Basic principles
Establish measures of success for your work based on your workplan goals and strategies.
Gather data for each measure and use the data to assess progress. Revise activities as
needed based on your assessment.
Regularly report progress to your community and to the funders.
1. Establish measures of success for your work based on your workplan goals and
strategies. What will success look like? How will you know you are achieving it? These are
questions that are critical to project planning and management. The use of measurement
enables you to demonstrate to yourself, the community, and other outside parties that you
have thoughtfully weighed what work needs to be done to evoke positive environmental
changes, and that you are accomplishing your goals and documenting progress.
Choose measures touching on each key work area and use them to inform your
work. Key work areas for CARE projects include building and maintaining a
strong collaborative partnership, achieving environmental and public health
benefits, and developing community capabilities. During your grant project, what
measures can you select that will keep the partnership focused on meeting their goals?
Definitions...
A performance measure is a unit of information tracked over time that helps gauge the
degree of success a project has had in achieving its stated objectives, goals, and planned
activities. Performance measures may address the type or level of activities conducted
(process), the direct products and services delivered (outputs), or the results of those
products and services (outcomes).
Performance measurement is the ongoing monitoring and reporting of accomplishments,
particularly progress toward pre-established goals.
To be most effective, each measure should include a:
Baseline: the current state of performance
Target: the desired level of performance
Timeline: the goal date for when the desired level of performance will be achieved
www.epa.gov/care
-------
LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES.
Select a few behavior change measures for your work, so that you can monitor the
effect you are having on the ground. What key behaviors do you aim to influence?
Write these behaviors down and determine ways to measure these actions, monitor
them throughout your grant, and report your successes.
Focus on a few strong environmental/human health measures and back them up
with a solid calculation methodology.
2. Gather data for each measure and use the data to assess progress. Revise activities
as needed based on your assessment. Evaluation followed by revision or adjustment is a
critical component of good project management and sustaining a long-term project.
Participants in the partnership and project need to have a clear and open way to develop
lessons learned, correct mistakes, and celebrate breakthroughs. Without this, the effort will
eventually fade away or break down. Determine a method to evaluate how as a
collaborative you plan to learn from your successes and failures and grow as a group.
3. Regularly report progress to your community and to the funders. You will be required
to report to funders on a regular basis about the progress you are making. Make this a
reflective and valuable time to assess your work and processes. Report your findings and
readjust the plan, if necessary.
Example project goals and measures...
The Boston Safe Shops Project had the goal of addressing negative environmental and public health
impacts that small automotive repair and body shops have on their employees and neighbors, while
avoiding putting these shops out of business by regulatory enforcement.
Example knowledge and behavior change project measures...
Number of in-shop tailgate training sessions conducted for shop workers and owners on pollution
prevention and worker protection techniques, Changes in knowledge and practices
Target: 40+ training sessions
Timeline: Within the 2 year grant period
Baseline and measurement: Baseline knowledge and practice established by pre-training questionnaire
and shop assessment with behavior and knowledge change measured by follow up assessments one or
more months after the training
Number of shops that make at least one major physical improvement to reduce pollution and worker
exposure (e.g., new equipment, improved ventilation)
Target: 10 shops
Timeline: Within the 2 year grant period
Baseline and measurement: Baseline conditions measured by initial shop environmental assessments
with adoption of changes measured by follow-up assessments and self-report/observations during follow-
up visits
Example estimated environmental results of behavior changes...
1 shop switched to waterborne paints, reducing VOCs by about 1,200 Ibs/yr
Lead wheel weights were eliminated from 125 fleet vehicles about a 260 Ib elimination in lead
www.epa.gov/care
-------
LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES.
To learn more...
Read an introductory guide on measurement:
Measuring what Matters: The Challenge of Quantifying Social Change
This 16-page document outlines one organization's thinking on how social purpose
organizations can approach the challenge of measurement. Check out the "Seven Practical
Tips for Measuring What Matters" beginning on page 12.
http://www.metgroup.com/assets/654 measuringwhatmatters0819.pdf
Read an introductory guide to evaluating your own program:
Taking Stock: A Practical Guide to Evaluating Your Own Program
This resource is a practical guide to program evaluation written for community-based
organizations. It provides information that you can use to help improve your programs. For
those just getting started it includes sections on defining goals and objectives, using quantitative
and qualitative data, and strategies for data collection.
http://www.horizon-research.com/reports/1997/stock.pdf
Read step-by-step "how to" information on developing measures and collecting data:
Measuring Progress: An Evaluation Guide for Ecosystem and Community-Based Projects
Pages 49-96 cover the topics "How will you know you are making progress?" and "How
will you get the information you need?" and are most relevant for partnerships that are
developing or refining their performance measures. This guide was written for an audience
working on community based conservation or ecosystem management, but may be used more
broadly. http://www.snre.umich.edu/ecomgt/evaluation/documents/Measuring%20Progress.pdf
Tips
TIP 1: Talk to your EPA Project Officer. He or she is your main resource as well as your
door to other resources within EPA. Your Project Officer may not be a measurement expert
himself or herself, but by talking to your Project Officer you can tap into other resources within
EPA, including measurement experts within the EPA CARE family. If you are struggling with
measurement and have not spoken about it, share your concerns the next time you talk.
TIP 2: Consider using a "logic model." A logic model is a visual model that shows the
relationship between your work and your desired results. A logic model can serve as a basic
road map for the project, explaining where you are and where you hope to end up. The Web
sites listed on the next page provide information on how to develop a logic model and how to
use a logic model as a tool to develop your project measures. On the last page of this
document we include an example of how a logic model can be used to develop measures.
In its simplest form, a logic model looks like:
if then if then
INPUTS
OUTPUTS
OUTCOMES
What is invested What is done What results
www.epa.gov/care
-------
LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES.
To learn more about logic models...
Take an online course:
A University of Wisconsin Extension Service online course on enhancing performance using
logic models http://www.uwex.edu/ces/lmcourse/
Read or download a manual: The Kellogg Foundation guide to developing logic models
http://www.wkkf.org/knowledge-center/Resources-Page.aspx (search on "logic model" to pull
up the guide)
Watch and listen to an online Web seminar:
An online Logic Model Webinar Sponsored by EPA and CDC
Available as an archived presentation and audio recording on the Communities in Action for
Asthma Friendly Environments Network site
Developing a Logic Model and Focusing Your Evaluation (December 17, 2008)
http://www.asthmacommunitvnetwork.org/webinars/program evaluation basics.aspx
Look at some examples:
EPA Region 10 Web page with sample logic models gives definitions and shows sample
logic models for grant programs
http://yosemite.epa.gov/R10/ECOCOMM.NSF/webpage/measuring+environmental+results
TIP 3: To estimate environmental outcomes, look for available "calculators." The CARE
Program has compiled a number of available online calculators and existing calculation
methodologies in a single document available to you in electronic copy by request (contact
mandolia.michelle@epa.gov). Work with your partners and Project Officer to see what else
might be available. If you are using a methodology or calculator that is not yet in the CARE
compilation, we would like to know about it so that we may share it with other communities.
Establishing the calculation methodology you will be using (including any existing calculator you
plan to use) at the beginning of your project ensures that you will have the data to run your
calculations when the time comes. Examples of just a few of the calculators include: a diesel
emissions quantifier, an anti-idling calculator, and the National Recycling Coalition
Environmental Benefits Calculator.
TIP 4: Look to your partner organizations or new partners for expertise. Have you
engaged your partnership in a conversation about work planning, measurement, and
evaluation? Perhaps there is an untapped resource in your midst. If not, is there a new partner
you could approach? Some of our CARE grantees have had success partnering with local
university students on measurement and evaluation. Could there be someone with the
expertise to advise or guide your organization through the process?
www.epa.gov/care
-------
LOCAL PARTNERSHIPS, HEALTHY COMMUNITIES.
TIP 5: Look to your partner organizations or new partners for data.
Is someone else already gathering the data that could help you understand whether your
actions are having their intended affect? If not, does it make sense for someone else to gather
the data?
Example of partnering for successful measurement...
The Superior Watershed Partnership in Marquette, Ml sought to reduce mercury
discharges from dental offices to the local wastewater treatment facility as a part of
their Level II CARE grant. The Partnership proposed a low cost solution to members
of the regional dental association, who voted unanimously to participate and 100% of
whom then voluntarily installed mercury amalgam separators in their offices.
How a partner helped with data...
The Marquette wastewater treatment facility provided the Superior Watershed
Partnership with data regarding mercury concentrations in the water. First, this data
helped the Partnership pinpoint dental offices as one major source of mercury,
leading to the particular reduction strategy they pursued. This data then, over time,
also told the Partnership that their efforts were working. The Superior Watershed
Partnership in cooperation with the Marquette Wastewater Treatment Plant
documented mercury reductions as much as 19% in the effluent entering Lake
Superior.
Additional note on measurement: The Partnership also tracked which dental
offices had installed the amalgam separators, enabling them to follow up with those
who had not.
The Superior Watershed Partnership is a non-profit and all of these results were
achieved without the need for additional regulation or policies.
TIP 6: Talk to your EPA Project Officer about whether the TASC contract might be right
for you. Through the Technical Assistance Services for Communities (TASC) contract, EPA
offers assistance to help a community better understand the environmental issues confronting it
and to be well informed while participating in the decision-making process. Some funds are
available to CARE communities through TASC for certain educational and technical assistance
not budgeted for in their original workplans. These activities could include work that would
serve your measurement and evaluation needs as you seek to manage your project and
partnership effectively.
If you would like to provide comments on, examples for, or suggestions for revisions to
this document, please send them to Michelle Mandolia at: mandolia.michelle@epa.gov
www.epa.gov/care
-------
EXAMPLE:
A (very simplified)
Childhood Lead
Prevention Program
INPUTS
Funds
Trained staff
Partnerships with
organizations
Legal authority
LOCAL PARTNERSHIPS. HEALTHY COMMUNITIES.
OUTPUTS
Outreach
Screening
Identification
Case management
Family training
Medical treatment
OUTCOMES
Lead sources identified
Lead sources eliminated
Families adopt in-home
techniques
Elevated blood lead level
reduced
INPUTS
OUTPUTS
OUTCOMES
Measurement Question: What do you want to know about your program performance?
To what extent is the program building the
local community health worker capacity?
Will we have resources to maintain or grow
our work?
To what extent are families getting medical
case management?
Are families pleased with the services we have
to offer?
To what extent are lead sources in the home
being eliminated?
To what extent are families adopting the in
home techniques we are sharing?
How will you measure it?
% of staff hired from community (75%
target)
Staff retention goal of at least 2 years
each (80% target)
# of new sources of funding by the end of
the grant period (Target of 2)
# of families receiving medical case
management/* of families referred
% participation of families receiving in
home visits in at least one other event (25%
target)
# of homes where lead containing items
were replaced/* of homes where they were
identified
% of families with elevated blood lead level
children that clean all window sills and
floors with the designated cleaning
solution each week
What data do you need to collect?
Where staff live
Staff start dates
Counts of new funding sources
Counts of:
families referred for care
families receiving care
Names of:
families receiving home visits
participants in other events
Counts of:
homes where items were identified
homes where items were replaced
Count of families implementing key best
practice each week
How will you collect the data?
Applicant resumes or applications
Staff hiring paperwork
Letters of award or commitment
Home visit logs
Health clinic numbers
Log of all families receiving home visits
Sign in sheets/logs from other events
Home visit logs
www.epa.gov/care
June 2010
EPA-530-F-10-003
------- |