More Information about the Policy
Copies of the Policy and the Framework for implementing it are
available at http://www.epa.aov/Dublicinvolvement/policy2003/
policy2003.pdf and http://www.epa.aov/publicinvolvement/policy2003/
framework.pdf
The Web site for the "Internet Dialogue on Public Involvement in
EPA Decisions" is http://www.network-democracy.org/epa-pip
EPA's Response to Comments on the Draft 2000 Public
Involvement Policy is available at http://www.epa.gov/public
involvement/policy2003/response.pdf
How to Evaluate
Public Involvement
"Involvement brings the pieces together" artwork is the creation
of Erica Ann Turner, who contributed the work through an
agreement between the Art Institute of Washington and EPA.
Involvement brings
the pieces together
United States Environmental Protection Agency
National Center for Environmental Innovation
Public Involvement Staff
1200 Pennsylvania Avenue, NW1807T
Washington, DC 20460
EPA233-F-Q3-011
77 West Jackson Boulevard, 12th
Chicago, IL 60604-3590
-------
Work Your Plan
Implementing the Evaluation
- Develop any unique survey instruments to measure how
well you are meeting stated goals.
- Clear your surveys through the Office of Management and
Budget if necessary (if you expect that nine or more
nonfederal people will respond).
- Ask participants to tell you what they think the
goals/objectives of the public involvement activity are.
- Respondents may surprise you with ideas unrelated to the
program's intent, with expressions of confusion or with
other information that shows your outreach missed the
group targeted, or just missed.
- Informally ask participants in involvement activities what
went well and how to improve what did not, then fix things
immediately (don't wait!), and ask again.
- Use the formal survey instruments—do the fieldwork.
- Compile and analyze responses after each evaluation.
- Improve survey questions and measures.
- Tell staff and managers what you learned.
- Tell respondents what you changed to show them how you
used their ideas.
- Improve activities or process.
- Keep asking for opinions and continue using the
information to improve.
Additional Resources:
Stakeholder Involvement & Public Participation at the U.S.
EPA: Lessons Learned, Barriers, and Innovative Approaches,
EPA Office of Policy, Economics and Innovation, January
2001. http://www.epa.gov/publicinvolvement/ pd./sipp.pdf
Beierle, Thomas C. and Cayford, Jerry. "Democracy in
Practice: Public Participation in Environmental Decisions"
Resources for the Future, Washington, DC, 2002.
http://www.rtf.org/books/descriptions/deniocracyinpractice.htm
"Program Evaluation System at the U.S. Institute for
Environmental Conflict Resolution" U.S. Institute for
Environmental Conflict Resolution, Tucson, AZ, January
2002. http://www.ecr.aov/pdf/proaeval.pdf
Other EPA Public Involvement Brochures
Introducing EPA's Public Involvement Policy
How to Plan and Budget for Public Involvement
How to Identify People to Involve
How to Provide Technical and Financial Assistance for
Public Involvement
How to Do Outreach for Public Involvement
How to Consult with and Involve the Public
How to Review and Use Public Input and Provide Feedback
How to Improve Public Meetings and Hearings
How to Improve Working with Tribes
How to Involve Environmental Justice Communities
How to Overcome Barriers to Public Involvement
-------
Plan Your Work
Defining the Evaluation
- Set clear, measurable outcomes for the overall evaluation
process and activities, share them with the participants,
and use their ideas.
- Review past processes, activities, and evaluations to see
what actions the Agency took as a result, noting lessons
learned.
- Work to gain everyone's support for evaluating your
process as it happens and once it is completed.
• Recognize whose support you must have to make
changes, and get their commitments to act on results.
• Plan evaluations to match available staff time and
budget.
• Review EPA's public involvement feedback templates
series, and if they meet your needs, use them and
the related database to simplify your work.
• Understand what activities you want to measure,
when and how often (e.g., the effectiveness of a
series of public meetings at three stages of the
process).
• Decide how you will gauge success throughout the
process and for activities [e.g., 75% of respondents
rated (a meeting) at 4.52 or higher on a six-point
scale].
• Decide how to share your findings and take action on
them along the way and following the process.
Deciding What to Measure
Will you/did you?
- Provide enough preliminary information.
- Engage traditionally under-served communities.
- Reach all the potentially affected people to give them an
opportunity to participate.
- Learn why people participated or decided not to.
- Understand participants' satisfaction with aspects of the
activity and process.
- Provide for equitable stakeholder participation.
"A well-supported evaluation plan will make it possible to (1)
better understand if EPA is taking the necessary steps to work
with the public; (2) better understand the quality of EPA's
involvement processes; (3) allow for EPA to systematically
and consistently learn and make improvement; and (4) be
more accountable to the public."
Eric Marsh, EPA
2001 Dialogue on Public Involvement in EPA's Decisions
- Identify and simplify public involvement techniques that
produce more cost-effective decisions.
- Provide effective overall outreach efforts.
- Enable EPA to incorporate public values in its decision.
- Provide enough opportunity for participants to identify their
issues and concerns.
- Build increased public capacity to take part in future
environmental decision making.
- Create or increase trust in the Agency.
- Preclude misinformation and obstacles to EPA's decision
making.
- Provide suitable time frames for participation.
- Help build new relationships and collaborative alliances
between stakeholders.
- Reduce legal actions taken against a final proposal.
- Develop a decision more acceptable to all stakeholders.
Which Measurement Tools Will You Use?
- Telephone interviews
- Face-to-face interviews with opinion leaders or randomly
chosen people
- Focus groups
- Mail surveys
- Comment cards
- E-mail or Internet-based surveys
- On-site surveys at events
- Informal discussion at the end of an activity
- Something unique to your process
-------
Step 7: Evaluate
Public Involvement
Activities
Goal:
- To evaluate the effectiveness of the Policy and of public
involvement processes
The U.S. Environmental Protection Agency (EPA) issued its
new Public Involvement Policy in June 2003. The Policy's
overall goal is for excellent public involvement to become an
integral part of EPA's culture, thus supporting more effective
Agency actions.
The Policy provides guidance to EPA managers and staff
on how you can better involve the public in the Agency's
decisions. The Policy outlines seven steps to effective
involvement. This brochure (one in a series) offers
suggestions to help EPA staff members to "get started"
evaluating public involvement activities and processes.
Why Evaluate Public Involvement?
Evaluations of public involvement help to define, measure,
and improve public involvement effectiveness. Getting
feedback from the public on how well a specific involvement
activity or overall involvement process (e.g., meetings, notice
of action, rule-making) worked, can help you change those
processes and activities to make them more effective for
EPA and participants.
Evaluation should lead naturally to action. If you evaluate
formally and informally throughout your process, you can
improve as you go along.
"...the key goal of evaluation should be to improve a program.
Also, unless it can be demonstrated with some sort of
evaluation that public involvement "works," agencies won't
fund it and managers won't do it."
Caron Chess, Rutgers University
2001 Dialogue on Public Involvement in EPA's Decisions
"EPA should not view evaluation as something that is done
at the end of a cycle, but rather as something that should
be strategically planned at the start of a cycle as participatory
objectives and benchmarks are being mutually developed in
consultation with stakeholder populations."
John Stone, Institute for Food and Agricultural
Standards, University of Michigan
2001 Dialogue on Public Involvement in EPA Decisions
What to Consider in Public Involvement Evaluations
When you think about evaluating public involvement efforts,
start by analyzing the process and its component parts.
Thinking Through
the Big Picture
• Outcomes
• Lessons learned
• Institutional
support
• Staff/budget
• Tools
• Measures
What outcomes should your
involvement process or activity
produce? How will you know if you
succeed in meeting your goals?
For example, if one goal is to
achieve broad, inclusive
involvement, how will you
measure how well you consulted
with the targeted and affected
groups? When building
awareness is part of your public
involvement process, how will you
know if people are better informed
about the issues?
Evaluation tools such as surveys are used to set a
performance baseline for measuring current and future
improvements. Build your survey instruments and informal
feedback opportunities so you will learn what works, what
does not, and what to do to improve. Gather both qualitative
and quantitative information.
Counting outputs such as how many brochures people take
away from a meeting, how many people attended, or how
many flyers you mailed is easy. Measuring outcomes is
harder, but you will gather useful information that moves you
to more effective processes and activities.
------- |