United States
Environmental Protection
Agency
Office of Health
Research
Washington. DC 20460
Research and Development
EPA/600/S9-90/054 Aug. 1991
EPA Project Summary
Evaluation and Effective Risk
Communication Workshop
Proceedings
Ann Fisher, Maria Pavlova, and Vincent Covello
Many agencies and other organiza-
tions communicate with the public about
risk. How can these agencies and orga-
nizations learn whether they are com-
municating effectively? Are their
messages appropriate and clear to the
Intended audience? Are their messages
reaching that audience? Is the audience
understanding and internalizing the
message? To explore these questions,
the Workshop on Evaluation and Effec-
tive Risk Communication brought to-
gether experts from academia,
government agencies, and the private
sector under the auspices of the federal
Task Force on Environmental Cancer
and Heart and Lung Disease and its sub-
committee, the Interagency Group on
Public Education and Communication.
The workshop's objectives were to:
0 Improve understanding of evalua-
tion problems and tasks;
0 Survey principles and methods of
evaluation relevant to risk commu-
nication;
0 Illustrate the practice of evaluation
through examples;
° Provide guidance for organizations
engaged in planning and coordinat-
ing the evaluation of risk communi-
cation;
0 Derive recommendations for im-
proving risk communication; and
0 Identify future needs.
The workshop proceedings provide
an overview of the principles and meth-
ods of evaluation and of their application
to risk communication programs. That
volume includes four commissioned
papers, five presentations on how to
implement evaluation of risk communi-
cation, 14 discussions of various practi-
cal aspects of such evaluations, and
summaries of 1 6 case study evaluations.
This project summary hlghllghtsthe most
important Issues and conclusions.
This Project Summary was developed
by EPA '8 Risk Communication Program,
Washington, DC, to announce key find-
ings of the workshop that Is documented
in a full report of the same title (see
ordering Information at back).
Introduction
Risk communication can be defined as
any purposeful exchange of information
about health or environmental risks be-
tween interested parties. We define it as
exchanging information about:
- levels of health or environmental risks,
- the significance or meaning of health or
environmental risks,
- the data and methods used in deriving
estimates of risk, or
- decisions, actions, or policies aimed at
managing or controlling health or envi
ronmental risks.
Evaluation, in the context of risk commu-
nication, is any purposeful effort to deter-
mine the effectiveness of risk communication
programs. Evaluation encompasses a wide
range of activities, from diagnosing risk
communication problems to measuring and
analyzing program effects and outcomes.
Background
A fundamental question dominated initial
workshop discussions: Why is it important
to evaluate risk communication programs?
Participants agreed that evaluation is crrti-
r/y Printed on Recycled Paper
-------
ca! to effective risk communication; without
evaluation, there is no way to determine
whether risk communication activities are
achieving (or have achieved) their objec-
tives.
Evaluation should be an integral part of
the risk communication process. When car-
ried out at each stage of program develop-
ment, evaluation provides information that
is critical to program effectiveness. For ex-
ample, it provides essential planning infor-
mation, it provides program direction, and it
can help demonstrate program accomplish-
ments. Most important, evaluation can sig-
nal the need for timely modifications.
Evaluation has much to offer organiza-
tions that communicate about risk. During
the planning and pre-production phase,
evaluation can provide data critical to effec-
tive program design, including information
about health, environment, and lifestyle
needs and concerns, information about risk
management needs and concerns, and in-
formation about how to meet those needs
and concerns. Through surveys, question-
naires, focus groups, and other research
tools, evaluation can be used to identify
stakeholders and other relevant audiences,
to assess audience opinion or reaction, to
find out what people see as important prob-
lems, to find out what issues and events
people are aware of, and to find out how
people react to different sources of informa-
tion. Pretesting and pilot testing can be
used to forecast the effectiveness and fea-
sibility of alternative risk communication
activities, to determine the kinds of informa-
tion needed by target audiences to under-
stand messages about risk, to examine how
people process and interpret risk messages,
and to obtain feedback on draft materials.
Estimates of the effectiveness of alternative
risk communication activities can be com-
bined with information about their costs to
show which risk communication strategy
will be most cost-effective.
When the risk communication program is
operating, evaluation can address ques-
tions of accountability and performance.
For example, evaluation studies can deter-
mine whether the risk communication pro-
gram is reaching the intended audience,
provide feedback on the performance of
risk communicators, identify program
strengths, suggest ways these strengths
can be used to communicate more effec-
tively, and determine whether the program
is being implemented appropriately (for ex-
ample, what material was produced, how
much was produced, how long it took, what
it cost, and what audiences received the
material).
Once the risk communication program
has been implemented, evaluation can pro-
vide information on program impact and
outcome. For example, evaluation can de-
termine what members of the audience
actually received, what they learned, and
whether change occurred in the way they
feel, think, or behave. The results can be
used to answer the most important ques-
tion: Did the program achieve its goals?
Few organizations have the resources
needed to launch state-of-the-art risk com-
munication programs that address multiple
audiences through multiple channels. Thus,
one major reason for evaluating risk com-
munication activities is to help managers
choose messages and channels that use
their limited resources most effectively.
Discussion: Problems and
Difficulties
These advantages raise a second ques-
tion: If evaluation is so valuable, why are so
few risk communication activities formally
evaluated? The answer appears to lie in a
variety of problems and difficulties stem-
ming from conflicts and disagreements about
values, goals, resources, and usefulness.
Each is briefly discussed below.
Values. Many difficulties in evaluation
arise from its nature as a normative, value-
laden undertaking that carries important
policy, ethical, and practical implications.
Evaluation is value-laden partly because of
the many stakeholders interested in the
conduct and effectiveness of any given risk
communication activity or program. Gov-
ernment agencies, corporations and indus-
try groups, unions, the media, scientists,
professional organizations, public interest
groups, and individual citizens each have
varying and often conflicting needs, inter-
ests, and perspectives.
Evaluators often are asked to respond to
the interests of many of these constituen-
cies. But different audiences have different
goals, different audiences need different
types of information, and different risk com-
munication activities require different types
of evaluation studies. As a result, an initial
difficulty in any evaluation is determining
the perspective from which it will be con-
ducted. Choosing a perspective has sev-
eral reporting implications, including the
evaluator's responsibility to be explicit about
the chosen perspective and to acknowl-
edge the existence of other perspectives.
Several practical implications also follow,
including limits on the relevance and role
that evaluation can play in affecting risk
communication programs, and an increased
likelihood that evaluation results will be criti-
cized, even by the sponsors of the evalua-
tion.
Goals. A second problem affecting evalu-
ation is the difficulty in identifying goals for
risk communication. What goals are apprc
priate? Should the primary goal of risk com
munication be to help people become awan
of an issue, make more informed decisions
take action, seek information, seek help
protect themselves, change their behavior
or participate more effectively in the dec!
sbn making process? For some, the goal o
risk communication is narrowly defined a:
personal or organizational survival and dam
age control; for others, it is to overcome
opposition to decisions; for still others, it is
to achieve informed consent, enhanced pub-
lic participation, constructive dialogue, and
citizen empowerment.
Meaningful evaluation is possible only
when the program's goals, intended audi-
ence, and expected effects can be specified
clearly. Even the most basic risk communi-
cation activity, such as responding to a
telephone inquiry from a concerned citizen,
should have a specific goal. Without clear
communication goals-be they informational,
organizational, legally mandated, or pro-
cess goals-it is impossible to know if the
interaction and exchange has been suc-
cessful. Such specification is extremely dif-
ficult and sometimes impossible for many
risk communication programs. Evaluators
and those who commission the evaluation
often cannot agree on what the goals of the
risk communication program should be, let
alone which goals should be assessed or
what kinds of success measured (e.g.,
through measures of knowledge, attitudes,
and perceptions; measures of message
awareness, comprehension, and accep-
tance; measures of information demand; or
measures of behavioral intentions or actual
behavior).
Once risk communication goals have been
determined, they should occupy a key role
in the planning and implementation pro-
cess. At each stage of the program, activi-
ties should be evaluated in light of these
goals. If warranted, program goals should
be reviewed and changed as the program
develops.
Resources. Effective risk communication
requires a determined effort to ascertain
whetherthe program is working as intended.
Feedback is essential. If provided early
enough, it can identify places where mid-
course corrections could be effective.
Evaluation often is neglected in favor of
what managers perceive to be more urgent
tasks-especially if evaluation has not been
planned and budgeted in advance. Manag-
ers are reluctant to evaluate for several
reasons. Many of them believe that evalua-
tion is prohibitively expensive and that only
a few organizations have the resources and
skills to carry out evaluation. Program man-
agers tend to exhaust all available resources
-------
producing and distributing more risk com-
munication materials (in the hope of in-
creasing effectiveness by reaching more
people), rather than conducting evaluation
studies that ask whether the message has
reached the target audience and whether
the target audience has received and inter-
nalized the message. There also is an un-
derstandable reluctance on the part of many
program managers to support research that
has the potential for showing that the time,
resources, and effort they have invested in
a risk communication activity or program
have not produced the desired results.
Managers may not want to be told that their
programs have shortcomings, because this
may have implications for career advance-
ment, for intra-organizational decisions
about the allocation of resources, and for
program survival. Whenever an evaluation
is conducted, there is a chance that it will
reveal (serious) shortcomings. So not evalu-
ating avoids the potential for evidence of
failure. On the other hand, if a program
manager is convinced that evaluation can
demonstrate success, according to what he
judges to be appropriate measures, then
evaluation may be viewed very differently; it
becomes a tool to justify promotions, bo-
nuses, or increases in financial resources
and staff.
Another factor that may affect the deci-
sion to evaluate is the limited success of
previous risk communication programs
aimed at changing risk-related attitudes and
behaviors. Planned risk communication
activities make up only a small share of the
many factors that impinge on people's per-
ceptions and behavior. Most evaluation stud-
ies suggest that even when the message is
clearly communicated and appears to be in
the audience's best interest, the goals and
expectations for such programs should be
realistic. For example, a successful risk
communication program might change the
behavior of only a small percentage of the
population. Agencies that have a public
health mandate may view a small percent-
age change as insignificant even if the num-
ber of individuals affected is large. However,
from the perspective of competing for atten-
tion and recognizing the complexities of
behavioral change, risk communication
endeavors should be compared with mar-
keting efforts. For example, a marketing
effort that produced an increase of a few
percentage points in market share would be
judged a big success. Beyond this lack of
understanding of what level of impact should
be considered a success, program manag-
ers may prefer formative and process evalu-
ation over outcome and impact evaluation
because the former affords opportunities to
make changes in response to findings.
These factors suggest that increased at-
tention needs to be given to understanding
organizational and other barriers to evaluat-
ing risk communication activities. Equally
important is the need to develop strategies
to overcome these barriers. First among
these strategies is planning risk communi-
cation efforts early in the program planning
stage so that evaluation activities can be
integrated from the beginning. Evaluation is
less likely to be resisted when it is built into
each stage of the risk communication pro-
cess, when adequate resources are built
into the risk communication budget for evalu-
ation, and when changes suggested by
evaluation data can be made.
Second, greater attention needs to be
given to the use of informal, quick, and
simple evaluation methods, many of which
can produce extremely valuable planning
and program information. When more/igor-
ous, systematic evaluations are required,
these ideally should be carried out by par-
ties other than those who control and con-
duct the risk communication activity or
program.
Third, greater attention needs to be given
to developing incentives for program man-
agers to fund evaluations for the purpose of
better understanding which risk communi-
cation activities are most effective, not solely
for justifying what has been done.
Fourth, program managers should be
encouraged to develop well articulated
evaluation plans with clear goals and clear
explanations of what the evaluation is de-
signed to achieve.
Finally, program managers should be
encouraged to document and share risk
communication successes, including cases
in which community feedback was solicited
and used to enhance the risk communica-
tion activity or program.
Usefulness. A common criticism is that
evaluation results seldom are used. Implicit
in this criticism is the notion that use means
direct and immediate changes in risk com-
munication policies or programs. However,
not all types of use are immediately appar-
ent. For example, results may be used to
confirm that changes in the risk communi-
cation program are not needed. In some
cases, evaluation may indicate directions
for risk communication that are inappropri-
ate or infeasible. Evaluation results may
accumulate over time and be absorbed
slowly, eventually leading to changes in risk
communication concepts, perspectives, and
programs.
In assessing the usefulness of evaluation
research, an important consideration is that
the forces and events impinging on risk
communication programs are often more
powerfulthan the results derived from evalu-
ation studies. The environment in which risk
communication programs are developed
seldom permits swift and unilateral changes;
new information actually can slow down the
change process by making decisions more
complicated.
Recommendations
Several recommendations come from
these observations and from those in the
volume's papers.
Most of these recommendations are ori-
ented toward policy makers in public agen-
cies that have risk communication
responsibilities. However, they apply equally
well to risk communication efforts in private
organizations, such as public interest groups
and industrial corporations.
Short-Term Recommendations
1. Agencies and organizations should be
encouraged to use evaluation methods
that are appropriate to the scale and
importance of the risk communication
effort. Small-scale efforts may require
only quick and easy evaluation methods.
In contrast, more resource-intensive, sta-
tistically reliable methods are appropri-
ate for large-scale efforts where there
may be substantial negative impacts on
well-being if the risk communication ac-
tivities are not effective.
2. Agencies and organizations should be
encouraged to integrate evaluation strat-
egies and results into program planning
and decision making; evaluation should
become a routine part of risk communi-
cation practice.
3. Mechanisms are needed to permit agen-
cies to share evaluation methods and the
results of evaluation research.
4. Agencies and organizations should de-
velop guidelines to help managers
choose the most suitable evaluation
methods. Workshops or other training
mechanisms are needed to build the
skills required to design and implement
evaluation strategies.
5. Agencies and organizations should be
encouraged to evaluate risk communi-
cation programs so that mid-course cor-
rections can be made and program impact
can be assessed.
Long-Term Recommendations
1. Agencies and organizations should sup-
port research aimed at measuring the
effectiveness of risk communication ac-
tivities as well as the cost-effectiveness
of alternative approaches. Examples of
research questions that need to be an-
swered are: How can we evaluate the
role of risk communication in changing
behavior? Do people make better risk
management decisions as a result of
&U.S. GOVERNMENT PUNTING OFFICE: MM - S4MM/40O47
-------
more effective risk communication? Is it
more cost-effective to extend the time
period for existing risk communication
activities, to intensify their use in the
originally scheduled time period, or to
combine multiple risk communication
activities?
2. Agencies and organizations should spon-
sor forums for public and expert debate
on issues related to the appropriateness
of using different kinds of motivational
and persuasive messages within risk
communication programs. For example,
what guidelines are needed on ethical
issues related to using different types of
motivational and persuasive messages
to help foster a more informed public?
3. Agencies and organizations should sup-
port development of guidebooks and
manuals for practitioners on how to apply
evaluation techniques. These materials
should include information on how to
tailor an evaluation program to the scope
and importance of a risk communication
activity, as well as how to recognize the
limitationsof alternative methods. Guide-
books and manuals also should include
case studies demonstrating the value
and importance of evaluation research in
risk communication.
Final Thoughts
(from "Risk Communication: On the Road
to Maturity," by Milton Russell, page 9 in this
volume)
"At a time when known changes in indi-
vidual behavior could bring about the first
significant improvements in the quality and
length of life since antibiotics, the tools to
communicate this information are rudimen-
tary, the research is poorly supported, and
many of the front line troops lack training
and the support of those who send them into
the field. There is little reason to believe that
this situation will change as matters nov
stand...There will be no reason to take the
risk communication process seriously unti
evaluations are made about whether people
are truly empowered to make importan
choices about the way they lead their live:
or about the collective decisions that others
are making for them.
In summary, it is the risk communication
professionals who have the largest stake in
both facilitating and demanding evaluation
of their efforts. They have both the profes-
sional responsibility and the personal in-
centive to determine what has been
successful and what further efforts will be
required over time to fulfill the promise of
risk communication as a major element in
improving public health."
Ann Fisher is with The Pennsylvania State University, University Park, PA 17889; the EPA
author, Maria Pavlova, is with EPA Region 2, New York, NY 10278; and Vincent Covello
is with Columbia University, New York, NY 10032.
Thomas Miller is the EPA Project Officer (see below).
The complete report, entitled "Evaluation and Effective Risk Communication: Workshop
Proceedings," will be available from:
Center for Environmental Research Information
Cincinnati, OH 45268
(EPA&OQ&-90/054)
or from:
National Technical Information Service
5285 Port Royal Road
Springfield, VA 22161
Telephone: 703-487-4650
(Order No. PB-91211896/AS; Cost $31.00, subject to change)
The EPA Project Officer can be contacted at:
Office of Health Research
U.S. Environmental Protection Agency
Washington, DC 20460
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati, OH 45268
BULK RATE
POSTAGE & FEES PAID
EPA PERMIT NO. G-35
Official Business
Penalty for Private Use $300
EPA/600/S9-90/054
------- |