Selecting the Right Tool for Evaluations Guidance for Community Involvement Practitioners

SELECTING THE RIGHT TOOL FOR EVALUATIONS: GUIDANCE
FOR COMMUNITY INVOLVEMENT PRACTITIONERS

by Seth Tuler, Caron Chess, Susan Santos, Stentor Danielson and Thomas
Webler

Community involvement is a priority for EPA, particularly in efforts to remediate
contaminated sites. It is also emphasized in watershed management, environmental
impact assessments and a range of other programs. Practitioners benefit from a large
amount of guidance about how to plan and organize community involvement activities.

But, how often is community involvement evaluated? Not often enough (Chess 2000,
EPA 2001, NRC 1996, NRC 2001). Evaluation is one of the seven steps in EPA's Public
Involvement Policy 1 and experienced practitioners know evaluation is important. But,
when you are juggling a lot, and resources are limited, evaluation can be all too easy to
drop. Often, evaluation is done informally, which can limit the usefulness of the
feedback.

Evaluation of public participation is evolving and the subject of much discussion among
practitioners and academics. In this article we provide guidance about evaluation that is
informed by research (our own and others) as well as our EPA-sponsored study (see
Box 1) in which we are exploring the usefulness of three evaluation tools or
methods: surveys, focus groups and Q Method, a promising approach that some
academics have been using.

For an overview of the three tools, see Box 2 (Guidance documents for focus groups
and Q method were also produced as part of this project and can be downloaded at
www.seri-ys.org/pybs/FGGuidance.pdf & www.seri-us.ora/pubs/QMethodGuidance.pdf.
respectively). Oftentimes interviews are used to gather feedback as well; in fact, in our
project we used interviews to gather background information to inform and complement
the focus groups and Q method approaches. In this paper, however, we do not discuss
them further.

We describe some of their strengths and limitations and provide some suggestions
about when to use them. Of course, a lot has been written about how to conduct focus
groups and surveys (Morgan 1998, Patton 1987, Dillman 2000, Charnley and Engelbert
2005), and we don't cover a lot of those basics. Instead, after providing a brief rationale
for evaluation, we explore the strengths and limitations of these evaluation tools, so that
better decisions about what tool to use can be made. Our presentation is framed around
three key questions:

1 EPA's public involvement policy is available at

www.epa.aov/publicinvolvenient/pdf/policv2003.pdf. A brochure about when and how to evaluate
is available at http://www.epa.aov/publicinvolvenient/brochures/evaluate.pdf

-------
1. What do you want to know?

2. Who do you most want to understand?

3. How will you use the information?

WHY EVALUATE?

There are several reasons to evaluate community involvement efforts.

Share lessons learned. Evaluation can be conducted at the end of a project (such as a
site remediation effort) to determine whether the community involvement effort was
"successful" and whether other goals were achieved. This evaluation (termed
"summative evaluation") is potentially useful to the agency and other participants to
validate community involvement efforts and provide "lessons learned" for other
community involvement projects (EPA 1999, Industrial Economics 2004).

Box 1. Overview of our project.

EPA asked us to explore evaluation methods not only to benefit agencies but also to improve
"community empowerment."

We conducted a study that explores how feedback can improve the quality of community
involvement efforts and clean-up decisions at Superfund sites. The premise of our
research is that effective methods for providing feedback can improve the exchange of
information and interactions among different stakeholders including agencies,
responsible parties, and local citizens. Improvements to information exchange and
stakeholder interactions may ultimately lead to better clean-up decisions.

Our project has explored the strengths and weaknesses of three methods for getting feedback
from those who participate in the Superfund clean-up process. We applied apply and critically
examined each of these methods at two separate Superfund sites where there is an ongoing
community involvement process. The first case study was of the Ciba-Geigy site in Toms River,
New Jersey. The second case study was of the Waukegan Harbor Area of Concern in Illinois.

In each case we had representatives of the community, government agencies, and potentially
responsible parties give us feedback using the three methods. After the focus groups and Q
sorts people filled out a written evaluation. In addition, after we gathered and analyzed the
results, we presented the findings in each community to a "feedback group." The purpose of
the feedback groups was to gather additional information about how the usefulness of the
findings and how people felt about participating in focus groups, surveys, and Q method.
Participants in the feedback groups were highly involved community members and federal,
state, and local agency staff.

For further details about the project and to download publications and presentations visit www.sen-
us.org/proiects/superfund.html.

Improve community involvement during the project. Evaluation during a process,
known as "formative evaluation," gives feedback throughout your project so you can
improve it. The feedback can positively influence remediation efforts as well as
community involvement, when preferences of stakeholders are revealed (Bradbury et al.
2003)

-------
Box 2: Overview of the three methods

Focus Groups: A focus group is a carefully guided group discussion intended to generate a rich understanding of
participants' experiences and beliefs. Focus groups are a proven research technique appropriate for a project that
is exploratory and/or descriptive in nature (Morgan and Krueger 1998). They are particularly well suited to
evaluation research (Morgan 1998). They are essentially group in-depth interviews with 8 to 12 individuals who are
brought together at a location convenient and comfortable to them to discuss a particular topic under the direction
of a trained moderator. Focus groups are an important way to listen to people, to learn about their views and
concerns, or to explore topics of interest. The information generated in the focus group is the opinions expressed
by group members in their own words. Participants in a focus group may have limited information on the specific
topic or question to be explored or they may be quite familiar with the topic and issue. Focus groups can also be a
useful tool when the subject matter is sensitive - such as concerns over health or disagreements that might occur
in the cleanup of hazardous waste sites.

Additional resources:

• Morgan, David L. & Richard A. Kruger 1998. The Focus Group Kit (6 volumes). Sage.

• Patton, Michael Quinn 1987. How to Use Qualitative Methods in Evaluation. Sage.

• Santos, S., Danielson, S., and Chess, C. 2007. Guidance on the use of focus groups for evaluation of public
involvement programs at contaminated sites. Greenfield, MA: Social and and Environmental Research Institute.
Available online at: www.seri-us.ora/pubs/FGGuidance.pdf.

Mail surveys: Mail surveys are questionnaires sent out to a statistically representative sample of people in the
population of interest - in this context, usually residents of the area around the site whose community involvement
is being evaluated. Questions are usually quantitative, such as asking people to rate their agreement with
statements on a scale of 1 to 7, or checking boxes for yes/no answers. In the context of a site cleanup, a survey
usually aims at taking 15-20 minutes for the respondent to complete. Some respondents may have little to no
knowledge about the site, and one of the main aims of a survey is often to explore the prevalence of ignorance
about the site. The results can be analyzed with statistical tests like T-tests, correlations, ANOVA, or chi-squared.
These simple tests can be done in a spreadsheet like Microsoft Excel.

Additional resources:

• Dillman, D. A. 2000. Mail and internet surveys: the tailored design method. New York: John Wiley and Sons.

• Resources are available online at www.epa.qov/publicinvolvenierTt/feedback/index.htnil

Q Method: Q method is a technique for revealing shared viewpoints that exist on an issue or topic. A study using Q
method, often called a Q study, can be a useful way of evaluating a public involvement process because it clarifies the
different views of various stakeholders about the process. It is also a useful way of assessing the different views of
stakeholders about their preferences for particular outcomes or satisfaction with them. Q methodology, like the survey
method, is a technique to explore peoples' subjective beliefs and attitudes. However, unlike a survey it allows
participants far more flexibility to express their beliefs. Q method can help you go beyond the simple idea that some
people are happy with the way things are while others are opposed, or the assumption that all people in a certain
group think the same way. A Q method study begins by identifying all the things people are saying about the topic.
From this, a sample of Q statements is strategically selected. People with clearly different opinions are asked to
express opinions about the Q statements by sorting them, i.e. "doing a Q sort." Typically one or two dozen Q sorts are
collected. The Q sorts are analyzed using statistical techniques that group together similar viewpoints. The product of
that analysis is interpreted to define different viewpoints, or "social perspectives," among those in the group. We also
learn how the individuals who did the Q sorts agree or disagree with these perspectives.

Additional resources:

• Q Method website: http://qmethod.org. This website contains information about Q as well as links to join the Q
Method listserv (a useful place to ask questions about doing Q) and to download the PQMethod and
MQMethod programs.

• Brown, S. R. 1986. Q technique and method: principles and procedures. In Berry, W. D., and M. S. Lewis-
Beck (ed.), New tools for social scientists: advances and applications in research methods, pp. 57-76. Beverly
Hills, CA: Sage

-------
Gather feedback on process and outcomes. We will side step the questions of the
difference between community involvement process and outcomes (Webler and Tuler
2002, Chess and Purcell 1999). For example, some would see public education as part
of the process and some would see it as an outcome. Regardless of what you call it,
you can explore the effectiveness of facilitation, outreach mechanisms, forums,
materials, etc. Or, you might want feedback on who is involved and why. In addition,
you can gauge participants' satisfaction with remediation, trust in agencies, level of
understanding...and the reasons behind them.

Promote dialogue among stakeholders. Discussion about the results of an evaluation
effort can in itself be an important catalyst for of interaction between an agency, a
community, and other stakeholders or parties. The process of evaluation provides
opportunities for the agency and engaged participants to discuss issues - in a more
structured way. An example of this was in the context of Department of Energy
evaluations of Site-Specific Advisory Boards at nuclear weapons facilities (Bradbury et
al. 1999, Bradbury et al. 2003).

Agencies have received anecdotal information about these and related issues for years.
But evaluation can be more "scientific," formal, and systematic, with the attendant
strengths and limitations.2

WHAT DO YOU WANT TO KNOW?

You never have enough time, money, (and participants' patience) to find out everything
you would like to know. Often, you need to cut down your "want" list in half or even more
As researchers, we work backwards: what will be the most useful information for
improving a situation or process? If the most important issue is how worried people are,
we will spend less time focusing on the effectiveness of public meetings and more time
trying to assess concerns. This seems obvious. But every time we can't figure out how
to reduce the load, one of us remembers to ask: What do we really need to know? For

2 Much of the information agencies receive is anecdotal (e.g., "several people came up to me
after the meeting") or potentially biased ("A representative from the union or local Chamber of
Commerce told me..."). Such information is not inconsequential. However, many professionals,
who have spent their entire lives conducting evaluations, say it can be misleading and therefore
dangerous. As those who study community involvement (including evaluation) as opposed to
evaluators (who actually do the evaluations), we can see both sides of the issue. We have
several thoughts on the issue. The evaluation should be in keeping with the scale of the
community involvement effort (both in resources and geography) and the needs of the agencies
and other participants. If the agency is going to scrutinize the results of the project, more than
anecdotal evidence is called for. Potential for other forms of controversy, differences in cultures
and ethnicities, and the weight of other factors that make community involvement especially
difficult, may make formal evaluation very important. Regardless, don't get trapped into thinking
that anecdotal evidence constitutes answers. Or that evaluation can be tacked onto the end of
the community involvement effort (no more than community involvement can be effectively tacked
on to the end of the remediation effort.) Or that you need a PhD to do it. Or conversely, that it is
easy.

-------
examples of questions, see Boxes 3, 4, and 5.

Finding out what you need to know is not merely a matter of identifying key issues. We
compared the information yielded by the three tools. We asked: what is different about
the kinds of information we can find out from using focus groups, surveys, or Q method?
If we asked about similar topics, would we find out similar things? We discuss some of
our key findings below.

Box 3. Types of questions asked in a focus group.

A comprehensive and well-structured discussion guide is essential for facilitating an interactive session that nets
valuable information. The guide spells out the topics that will be covered and provides a logical flow of questions
that are initially broad and open-ended to reduce the likelihood of biasing answers. Reponses are then followed by
probes to elicit more detail. The guide must also organize the time in a fashion that makes the focus group
comfortable and interesting for participants. It is important to remember, however, that the guide is not a script. A
skilled moderator will use it as a guide, exploring or further probing the comments participants make and manage
the dynamics of a group as necessary.

The following are example questions from a discussion guide for officials (including local and state officials and
agency staff).

Moderator: As I said earlier, we want to talk about community involvement in the clean up process at the
Waukegan Harbor Area of Concern and Waukegan River Watershed. I'd first like to ask what are the different
environmental problems/issues that need to be addressed in the clean up of the Waukegan AOC and watershed.

• What can you tell me about the different things being done to address some of these concerns? [Probe for
comments on separate clean up initiatives, studies, etc]

• How have you heard/learned about these things [probe for sources, specific groups, individuals, agencies
vs. media, etc.]

Moderator: Before we talk more about some of the things you have mentioned, I'd like to talk a little more
specifically about some of the opportunities and ways that different stakeholder groups have been involved in the
clean up process [and decisions about re-use] for the AOC and watershed.

• What are some of the different stakeholder opinions or concerns about the different clean up initiatives
that are occurring? [probe for specific concerns differentiate clean up from re-use issues]. Is that a
concern you think is shared by other stakeholder groups? Which ones? Are you aware of those who
might have a different perspective?

• What types of opportunities/activities exist regarding the Waukegan Harbor Area of Concern and
Waukegan River Watershed for people to learn about the various environmental issues and different clean
up [and re-use?] initiatives]? [Moderator will list on flip chart]

• Do you think most people in the community are aware of these? Which ones would you say they are
least/most aware of? Can you give me an example?

Moderator: Before we talk more about some of the things you have mentioned, I'd like to talk a little more
specifically about some of the information or activities that have existed/exist for community involvement?

• What types of CI activities are people aware of? Has anyone participated in any activities (moderator
will probe for the following: Public meetings? Public Availability Sessions? Technical Review meetings,
site tours, comment on documents, etc). How did you find out about these activities/opportunities?

• Have your expectations or needs related to CI ever changed? How/why? Did that ever get communicated
to EPA/PRP/Other? If so, was there a corresponding change in the CI activities to meet them?

Focus Groups

-------
Focus groups can provide a rich and in-depth understanding of viewpoints. The
moderator can probe for additional information about priorities and preferences of focus
group participants. Group dynamics can lead to deeper and richer information being
discussed. New lines of inquiry can emerge. In fact, you can even ask about the
fundamental purpose of the community involvement effort.

For example, in Toms River, New Jersey, we found that the focus group of highly
engaged participants saw the primary purpose of community involvement as keeping the
officials and agencies "at the table" to reach agreement. Participants also saw this as a
way to educate the agencies to facilitate better decisions.

Focus groups can provide insights into participants' views about the specifics of a
community involvement process. For example, we learned in Toms River that
uninvolved residents thought a community advisory group would be helpful but those
who were highly engaged did not see it as useful at this stage of the remediation. We
also learned that the experience of a failed remediation at a nearby site led people in a
focus group of highly involved participants to be somewhat skeptical of the chosen
remedies at the Ciba Geigy site. On the other hand, while participants in a focus group
of "officials" acknowledged such concerns they did not see community outreach efforts
as being able to address them and they discounted comparisons between the two sites.

Box 4. Types of questions asked in a survey.

The following are some of the questions we asked in our survey for the Waukegan Harbor Area of Concern
case study. After each of the questions a list of options were provided and the respondent was asked to
rank the familiarity, satisfaction, etc. on a 6-point scale.

• Compared to all of the other issues facing Waukegan, how important do you think the harbor
cleanup is?

• The overall harbor cleanup is made up of several connected parts. How familiar are you with the
different components of the harbor cleanup?

• How satisfied are you with the progress of the cleanup of each of these parts of the harbor cleanup?
Check "D/K" if you don't know what progress is being made on a part of the cleanup.

• How have you have learned about the harbor cleanup? (Check all that apply)

• How would you prefer to receive site information? (Check the ONE you most prefer)

• How interested are you in obtaining information about the following topics? (Circle one answer for
each question)

• What is the best way to get your participation? (Check the ONE you most prefer)

Surveys

Surveys provide data about specific topics. For example, our surveys asked respondents
for their sources of information, as shown in Box 6. Based on these responses,
evaluators can assess whether their efforts are reaching people including whether there
are differences among specific subgroups in the population (e.g., Caucasian vs. Latino
residents). These results suggest that newspapers are a much more important source
of information about the harbor for Latinos than for Caucasians. However, the survey
does not tell you why. It also cannot tell you how people feel about a particular source of

-------
information - if they trust it or if they understand it. Nor can it tell you anything about the
quality of the information provided by the source. A focus group could tell you a lot
about what a few people feel about the news coverage of the harbor area.

Because surveys ask about specific topics, important concerns, problems, or ideas may
be missed. For example, we asked about people's preferences about how to obtain
information concerning remediation activities, including preferences for "presentations at
local clubs and organizations." We could not find out from the survey results that some
segments of the Latino population were most likely to go to the social clubs. Instead, the
focus group was the source of that insight. Surveys ask questions evaluators think are
important, but evaluators might miss important questions.

Box 5. Types of questions asked in Q Method.

The sorting instruction defines the context in which the Q participant's perspective is being sought.
For example, one sorting instruction might ask the person to sort the statements based on how well
each statement describes the actual situation, while another may ask the person to sort the
statements based on how well each describes how they would like things to be. If you are evaluating
an on-going process then your sorting instruction should specify whether people should be giving
their views on the things that have happened so far or on what should happen moving forward.
For our case studies at Ciba-Geigy and Waukegan Harbor we had each person conduct two Q sorts
- one about the public involvement process and one about the clean-up outcomes. The sorting
instructions we used at Ciba-Geigy were:

Process: When you think about where the process is now, what should happen next? Sort
the statements according to most like I think the process needs to be to least like I think the
process needs to be.

Outcomes: When you think about the remediation of the Toms River Ciba Geigy site, what
do you think about what has been done in the past and is being done currently? Sort the
statements according to most like I think to least like I think.

Q Method

Q method provides a holistic representation ofpeople's views on a topic and the relative
importance of those views. However, the evaluator must choose in advance to ask about
certain issues—which are the basis of the Q statements that respondents prioritize.

Thus, Q shares some of the same limitations of surveys. For example, like the survey
our Q statements did not include anything about the use of social clubs (or churches) to
reach the Latino community in Waukegan. Therefore, we could not find out anything
about the relative benefits of clubs versus churches. Even if we included a statement
about this issue, we might not learn much. For example, suppose we included the
following statement: "Social clubs should be used for outreach to the Latino community."
This statement may not have been ranked among the "most important" by enough
respondents for it to become a distinguishing statement in any of the perspectives that
emerged. It would be unlikely that its relevance and importance would be understood by
the evaluator. This means that evaluators may not learn about some important issues if
they do not know to ask. As one participant told us, he felt "captive of the statements."

-------
Q can more dramatically highlight differences among perspectives than either focus
groups or surveys. Q requires each person to rank the same set of statements. In
addition, people prioritize statements without the potential for the 'bandwagon effect' that
can be found in groups and which can hide differences. Also, people may reveal more
outside of the group setting. For example, despite the trusting relationships in Toms
River, we found two different perspectives about desired outcomes. People associated
with one perspective are more concerned about the current bioremediation and
groundwater treatment efforts. They also feel that the air monitoring system is adequate
and that the air monitors are state-of-the-art. The second perspective is most concerned
with drums in a nearby landfill that is not slated for remediation. Respondents
associated with this view also fear that the air monitoring systems at the current site are
not

adequate and that the community may not be appropriately warned in case of an
emergency.

Box 6. Number of 'yes' and 'no' responses regarding past sources of information (Waukegan
Harbor AOC survey).

All respondents

Latino

Caucasian

X7 for Latino vs.
Caucasian

Yes

a. Mailings from the
responsible agencies

0.066

b. Mailings from the
Community Advisory Group
(CAG)

c. Newspaper articles

0.013

d. Radio or TV news

e. Family or friends

f. The internet

g. Public meeting or
information session

104

h. Direct conversation with
someone from the responsible
agencies

105

i. Direct conversation with
someone from the CAG

110

j. Information about the
lakefront is "common
knowledge"

k. Know someone who worked
at the lakefront

1. Participation on one or more
citizen groups

109

m. Events at the school (either
directly or through your school-
age children)

103

0.008

-------
Q also can reveal conflicts about process. For example, in the Waukegan Harbor case
study we found a few points of strong disagreement between the two perspectives. Both
perspectives strongly endorsed having clear standards for remediation. But, one
perspective saw the importance of asking for public preferences, while it was rejected
fairly strongly by those who were members of the second perspective. Respondents
associated with the second perspective feel strongly that community involvement may
delay the process and cost agency personnel too much time that could be spent solving
problems.

WHO DO YOU MOST WANT TO UNDERSTAND?

There are many people who may be interested in or affected by a community
involvement process and site remediation decisions. Sometimes it is useful to gather
feedback about how the entire community feels. In other cases, you may want to target
your efforts to gather feedback about specific groups within a community. There are a
number of possibilities, including the general public, highly involved community
members, environmental justice populations, local elected officials, staff of local
government agencies, staff of state and federal agencies, etc. There may also be great
diversity within some of these categories. For example, the general public can include
people who live within a certain radius of a site, who have attended prior events (e.g.,
public meetings), or whose native language is not English.

Our project explored what we could learn about various groups using the three different
tools. While in theory any of the tools could have been used for the same groups - some
tools are better suited to learning about particular people. We asked: what groups are
the tools most useful for gathering feedback from? We discuss some of our key findings
below.

Focus Groups

Focus groups gather feedback from a wide range of stakeholders. But some people
may not feel comfortable sharing their views in a group. Participation in a group of
"similar folks" can make some more comfortable about sharing their views. However,
others are more reluctant to speak publicly. This is not merely shyness. We found, for
example, that elected officials were less likely to attend a focus group and speak openly
in what is essentially a public venue. Similarly, discussion in a focus group may be
hampered by the presence of staff from the agency sponsoring the community
involvement effort or from the responsible party. Some people may not be used to
anyone asking them what they think or may not have experience with similar forums.

This may or may not be an impediment, as the following quote from a Latino leader in
Waukegan illustrates:

They [Latinos] didn't know at the beginning what is a focus group. They thought
the facilitator would answer their questions. It took time for them to figure out
that facilitator wanted to know what they think. They are not used to that, it was
a positive experience for them. They learned a lot. They got in touch with their
feelings, what else they want to learn. They got to hear what neighbors think. [It
was] very positive.

Focus groups can be challenging if there are significant conflicts among participants in
the same group. For example, according to participants in our Tom's River case study,

-------
when the remediation began, focus groups would have been problematic because of the
level of conflict among some people. However, after spending years dealing with the
site, opponents developed better relationships and could discuss issues openly in a
focus group. There is also a risk that members may sidestep conflicts and underlying
problems may never surface. On the other hand, they also felt a skilled moderator might
be able to create a safe enough environment for discussion. Another option is to
organize the groups carefully so that people with conflicts are not present in the same
focus group; the key is not to have too many extremes in a single group.

Focus groups may not attract the uninvolved or uninterested. They may not want to
invest their time. However, civic organizations have been known to recruit members for
focus groups in exchange for a contribution to their organization. In this case, their
sense of community overcomes their apathy about the topic. Another strategy to
overcome this obstacle is to frame the focus group discussion in a broader context. For
example, rather than say "come to a group to discuss community relations at the site,"
you might invite people to come talk about issues of importance to their community.

Surveys

Surveys gather information from a broad sample of people, but obtaining a
representative sample can be challenging. . Decisions must be made in advance, for
example, if race, gender, ethnicity, proximity to the site, or other characteristics are
important to consider. If so, the sample must be designed so that adequate responses
are obtained from each sub-group of interest.

In Waukegan we needed to make sure that we adequately sampled the Latino
population, which used the waterfront but was relatively uninvolved in making decisions
about the remediation. For example, when comparing Caucasian respondents with
Latinos, Latinos were more likely to have heard about the site through school meetings.
The differences were large and statistically significant. This suggests that outreach via
schools is a good way to reach members of the Latino community. Caucasians were
statistically more likely to have learned about the remediation activities from newspaper
articles than Latinos.

Surveys can be effective with the uninvolved or uninterested. In our two cases, surveys
worked well to gather information from people who had otherwise not participated in site
remediation activities (including community involvement activities; also see Charnley and
Engelbert 2005). For example, in the Waukegan Harbor case we found that many of our
respondents were not engaged with the remediation or community involvement
activities, yet they returned completed surveys (see Box 7).

Surveys require an appropriate response rate. While researchers may argue about what
constitutes a reasonable response rate, they will all agree that a survey with a low
response rate is likely to be biased or otherwise invalid (Dillman 2000). If the response
rate is low, generalizing the results to the full population is inappropriate. You are
unlikely to know what kind of people failed to respond to your survey. Was it the harried
moms? Or, people who live further from the site? The usefulness of survey results is
very dependent on who responds to the survey. If despite best efforts responses are not
obtained from a certain group, say families with young children, the results will say very
little about how that group thinks or about how that group's thinking differs from others in
the population. For example, our survey is Waukegan was appropriately criticized for

-------
missing "the black community - they are a relatively large minority within Waukegan."

Box 7. Surveys may be a good way to reach the uninvolved.

The responses to our survey in Waukegan indicate that people generally feel themselves to be
uninformed about issues that might affect their concerns about risks. We asked "The cleanup effort
addresses contamination and environmental health risks from several different sites through a variety of
activities. How familiar are you with each of these activities?" A relatively large number of surveys were
returned without a response to this question or with a response of "Don't know." Latino respondents were
much more likely to not response or to indicate "Don't know." Thus, lack of knowledge did not preclude
their completing the survey.

Question 4

All

respondents

Latino

Caucasian

a. Dredging of the harbor

16%

b. Cleanup ofthe OMC Superfund site

11%

22%

c. Cleanup ofthe Johns Manville Superfund site

13%

29%

d. Cleanup ofthe Yeoman Creek Superfund site

12%

24%

e. Planning for the Waukegan River watershed

14%

31%

f. Revitalization of downtown Waukegan

10%

22%

g. Redevelopment of the Waukegan harbor area

11%

22%

h. Development of recreational facilities

11%

24%

i. Cleanup of beach areas

10%

20%

j. Delisting of Waukegan Harbor

15%

31%

The results also indicate that people do not appear to be familiar with government agencies or other
groups involved in the remediation effort, as shown in the Table below. It is not just that they are
unfamiliar with these groups - they are also not familiar with what is being done to remediate the sites and
the lakefront area.

Table 3. Question 6: Familiarity with different government agencies and groups involved in the
clean-up effort in the Waukegan lakefront area.

(scale: 1 = very unfamiliar, 6 = very familiar).

All

respondents

Latino

Caucasian

a. US EPA

2.9

2.7

2.8

b. Illinois EPA

2.9

2.6

2.8

c. Lake County Health Department

3.0

2.6

2.8

d. City of Waukegan

3.2

2.8

3.1

e. Waukegan Harbor Citizens Advisory
Group

2.8

2.6

2.5

f. Waukegan Main Street

3.0

2.9

g. US Congressman Mark Kirk

3.1

2.6

3.1

While data in the two tables reveal that people were unfamiliar with clean-up activities or groups involved
in the remediation effort, they still responded to the survey. It would have been a challenge (but not
impossible) to invite such people to participate in an evaluation based on Q method or focus groups.

Because we asked respondents to indicate their race, we know our response rate for
this group was not representative of their actual numbers in the population.

-------
Q Method

Q method is useful for gathering information from people with a broad range of
perspectives. But, as with focus groups, if you don't know the community, you can fail to
identify important perspectives. When using Q method (and focus groups) the evaluators
must develop a sense about the variation in views among the population of interest -
and why these differences are likely to exist. For example, institutional affiliation and
group membership are often used as an indicator for different points of view - but they
may not be an accurate predictor of meaningful differences about preferences for
community involvement. Neither may race, gender, etc. The differences may arise from
fundamental values about, for example,

democratic participation or the role of expertise in decision-making. Unlike surveys, Q
method allows you to involve additional participants that you have previously overlooked
so that you can adapt as you learn more.

Q method does not effectively elicit opinions of people who are uninvolved or
uninterested in the process or site remediation. If people are uninvolved or not
interested, they are unlikely to have strong opinions about the statements, so it will be
hard for them to express clear preferences. They are also less likely to enjoy the
process. Sorting Q statements can be challenging and some people reported not liking
to have to make choices about how to rank statements. Others enjoyed the process
tremendously. We heard from some of our participants it would be hard to get other
people in the community to spend time doing a Q sort and that "you need something,
some knowledge first to do Q. You cannot just do it."

However, we have found that in exchange for a contribution to their organization some
people can be motivated to spend their time doing a Q sort.

HOW WILL THE INFORMATION BE USED?

The goal of an evaluation should be to gather feedback that can make a difference. But,
there can be many purposes for an evaluation. Tools may be better suited for some
purposes than others. Focus group data cannot be easily condensed to a graph, as can
surveys. Conversely, graphs don't speak to everyone, while information from focus
groups is relatively easy to understand. Similarly, if an agency wants to compare data
across sites, quantitative data can be easier to represent. Q method by virtue of its
uniqueness can attract attention when others do not.

In this section we discuss differences in the kind of information that is produced by each
tool, and how that affects how the information can be used. In addition, we found that
the tools elicited different ideas about how they can be used to engage and empower
community residents and support outreach efforts.

Focus Groups

Focus groups provide qualitative information that can be easily interpreted by a wide
range of audiences. They have a certain "face validity". Focus group findings are usually
reported in the own words of the participants. When we presented the results of our
focus groups, participants found it easy to understand the results. Use of narrative
quotes made the findings more salient.

-------
On the other hand, focus group results can be difficult to compare systematically. Data
derived from different groups and within groups may be a challenge to compare
systematically. Comparability depends to a great extent on the moderator, who can
direct the participants in different groups to discuss the same topics. But it is often the
case that different groups will discuss different content, even if the same general topics
are raised by the moderator. For example, one group's discussion about access to data
might focus on fairness and trust, another on limitations of the data. This can happen
because a participant may be responding to the specifics of what another person said,
and the same issues will not necessarily be discussed in each focus group or by all the
people within the same focus group. Lack of comparability may also result because of a
purposeful decision by the moderator - to explore, for example, an issue that was not
included initially in the discussion guide but was raised by participants in one of the
groups. Thus, information obtained may not be consistent across groups.

Focus groups provide an opportunity for people to meet and learn, which can be
particularly important for people that have been uninvolved. Participants in both of our
case studies noted this benefit. In Toms River one person told us that "focus groups
promote a lot of give and take among participants - generate ideas and new thoughts.
But there is a danger of opening up animosities." In Waukegan we were told that
"people left the focus group asking" 'what is next'? We had their interest, awareness,
they wanted to know what they can do now." Participants in the Spanish language focus
group felt more strongly that the focus group stimulated their thinking about remediation
options than did participants in the other groups. In Toms River participants found the
focus groups stimulated their thinking and provided a sense of how others think.

Surveys

Surveys provide quantitative measures of responses that facilitate comparison and they
can provide information that is generalizable to the larger population of interest. In a
survey people respond to the same set of questions and response options. When
enough responses are obtained it is possible to make claims about the general
population of interest as long as the sample is not biased, as we discussed above.

Surveys provide an opportunity for outreach. This was a point made in our feedback
groups, and by written comments on some surveys. For example, participants in our
feedback groups told that the questions were a good way to spark learning and that the
"process of doing survey may perk interest among those that have yet to be very
involved."

Q Method

While Q method facilitates both quantitative and qualitative comparisons of perspectives
about community involvement and about specific process features, the method does not
allow claims to be made about how many people hold those perspectives. In Q method
everyone who participates responds to the same questions and must express their
preferences in the same way. This allows comparisons to be systematic. However, Q
method is not geared toward gathering data from representative samples of respondents
(like focus groups). Instead, the approach is used to find differences among people with
different perspectives; it works best when data are gathered from people representing all
the important perspectives in the population of interest. But, because of the small

-------
number of people providing data, the prevalence of the perspectives cannot be
determined.

Q method can help individuals think about what is important to them because the
process forces them to make choices about their preferences. For example, people often
report that the effort stimulates their thinking. However, not everyone thinks so. When
asked about this issue in the questionnaire completed after doing a Q sort, we found that
Q sort participants in both Toms River and Waukegan were divided as to whether the
group Q sorts stimulated their thinking about the community involvement process or the
remediation options.

Conclusion

While ad hoc, informal evaluation is often done, more systematic efforts are usually
called for. They are more likely to produce "useable knowledge" and build a basis for
making improvements.

There can be a bias toward gathering as much information as possible. Our cautionary
note, however, is to be selective. Focus on what will help make specific kinds of
improvements - what is useful for your particular purpose.

Many tools, or methods, can be used to gather such information. They form a kind of
tool-kit. Practitioners should be clear about their choices, considering the trade-offs of
each. In this article we have presented information about three effective tools that can
be used to gather feedback about community involvement efforts and people's
preferences for outcomes: focus groups, surveys, and Q method. Other evaluation
tools, such as one-on-one interviews, should also be considered and can be readily
incorporated into the other methods. Being systematic in whatever approach is selected
is important to ensure results are robust and credible.

Usually, you will not be facing an either-or choice. In fact, adopting a multi-method
approach can be very useful. There are three reasons.

First, information gathered using one method may inform further evaluation using a
second method. You can use focus groups (or interviews) to generate statements for Q
method or identify important questions to ask in a survey. In our project we did
background interviews in each case to identify statements for the Q study.

Second, one approach may be better suited for a particular moment in time. For
example, interviews may be useful at the beginning of a process, because they will help
you build relationships with key people. Focus groups can also be used effectively mid-
stream to give you a broad feel for what is working, what's not, and enable you to make
changes. Q method is not going to be an effective tool unless people have gained
experiences they can reflect on. Later in the process it can be used to uncover
differences that might be critical to moving forward with remediation decisions or to
refine community involvement efforts to address possible roadblocks.

Third, the methods work well in gathering feedback from some groups, but not so well
with other groups. For example, focus groups and Q method are very effective for
gathering input from people that are really engaged in the process - people that have
strong opinions. Focus groups allow such people to give meaningful input. Q method

-------
works best with people who have rich experience to inform their preference about
statements. Of course, focus groups can also be a useful way of gathering input from
less involved people. Surveys are also effective in gathering feedback from the hard to
reach, disinterested, or uninvolved.

References

Bradbury, J. and Branch, K. 1999. An evaluation of the effectiveness of local site-specific
advisory boards for U.S. Department of Energy Environmental Restoration Programs. Report
PNNL-12139. Washington, DC: Pacific Northwest National Laboratory.

Bradbury, J., Branch, K., and Malone, E. 2003. An evaluation of DOE-EM public participation
programs. Report prepared for the DOE Environmental Management Program. Report
PNNL-14200. Washington, DC: Pacific Northwest National Laboratory.

Chess, C. 2000. Evaluating environmental public participation: Methodological questions, Journal

of Environmental Planning and Management 43 (6): 769-784.

Chess, C. and Purcell, K. 1999. Public Participation and the Environment: Do We Know What

Works?, Environmental Science and Technology Vol. 33 No. 16, pp. 2685-2692.

Dillman, D. A. 2000. Mail and internet surveys: the tailored design method. New York: John Wiley
and Sons.

EPA 1999. Lessons learned about Superfund community involvement. Available at

www.epa.qov/superfund/proqrams/reforms/docs/leslrncomplete.pdf.

EPA 2001. Lessons learned, barriers, and innovative approaches office of policy economics and
innovation. January 2001. Available on the web at: www.epa.gov/stakeholders/pdf/sipp.pdf
Industrial Economics 2004. Program evaluation: An internal review of procedures for community
involvement in Superfund risk assessments. Report Prepared For The Office Of Solid Waste
And Emergency Response, Environmental Protection Agency. Cambridge, Ma: Industrial
Economics, Inc.

Morgan, David L. & Richard A. Kruger 1998. The Focus Group Kit (6 volumes). Sage.

National Research Council 1996. Understanding risk: Informing decisions in a democratic

society. Washington, DC: National Academy Press.

National Research Council 2001. A risk management strategy for PCB-contaminated sediments.

Washington, DC: National Academy Press.

Patton, Michael Quinn 1987. How to Use Qualitative Methods in Evaluation. Sage.

Webler, T. And Tuler, S. 2002. Unlocking the Puzzle of Public Participation. Bulletin of
Science, Technology, & Society. 22(3):179-189.

Acknowledgements

This research was made possible by EPA Cooperative Agreement 831219-01-3. We
would like to thank the citizens of the case study areas and staff from EPA who shared
their time and knowledge.

About the Authors

Seth Tuler, a Senior Researcher at the Social and Environmental Research Institute in
Greenfield, MA, has published extensively on public participation and evaluation in a variety of
policy domains and seeks to apply the lessons of research to clean-up of sites contaminated by
the US nuclear weapons complex.

Caron Chess, an associate professor at Rutgers University's Department of Human Ecology,
has published widely on evaluation and whose experience in academia, government, and
environmental advocacy underpins her research in risk communication and public participation.

Susan Santos, an internationally recognized expert in risk communication, is founder and
principal of FOCUS GROUP in Medford, MA, a consultancy specializing in risk communication,
community relations, and health and environmental management.

Stentor Danielson recently completed his Ph.D. at Clark University's Department of
Geography on Discourses About Wldfire in New Jersey and New South Wales, using both
surveys and Q method.

-------
Thomas Webler is a Senior Researcher at the Social and Environmental Research Institute in
Greenfield, MA and co-editor of the book Fairness and competence in citizen participation:
Evaluating models for environmental discourse.

-------