EPA QIC PROGRAM EVALUATION PILOT TEAM TRAINING
January 16-18, 2001
80817th StNW
Suite 400
National Center for Environmental Assessment
Washington DC 20006
AGENDA
Tuesday, January 16,1 00 PM - 5 00 PM
1 00 - 1 30 PM Welcome, Introductions, Agenda Rick
1 30 - 2 30 PM "What's Up With This'" Rick
2 30 - 2:45 PM
Break
2'45 - 4-45 PM The Basics of Program Evaluation
4 45 - 5 00 PM Wrap-up
Emmalou
Emmalou
Wednesday, January 17, 8 30 AM - 5:00 PM
83O -1015AM Logic Modeling Emmalou
1015-10 30 AM Break
10 30-11 45 AM The Pilot Process
11 45 - 1 00 PM Lunch
1 00 - 2 30 PM IT
2 30 - 2 45 PM
Break
2 45 - 5 OO PM The Pilot Process
Emmalou w/Connie, Dale, Art,
Rick
Ernie, Stephanie, Yvonne
Emmalou w/Connie, Dale, Art,
Rick
700PM
Dinner
-------
Thursday, January 18, 8 30 AM - 1 OO PM
8 30-9 30 AM The Pilot Process Emmalou, Rick, Consultants
9 30 - 12 00 Noon Team Meetings w/Facilitation Emmalou, Rick Consultants
12 00 - 1 00 PM Wrap-up Rick, Emmalou
1 OO PM Adjourn
-------
The Basics of Program
Evaluation
Emmalou Norland
>H
jmmii
what c|o you plan to gain...
I professionally from this training!*
I professionally from participating in the
pilot/
What Well Po This Afternoon,
• Share a little about yourselves and
your expectations
• Define program evaluation more fully
• Identify the links between program
development and evaluation
• Learn about the 'profession' of program
evaluation
• Distinguish program evaluation fro
other similar processes
What <\o you have to contributed
• Team skills
• Program knowledge
• Evaluation knowledge
• Special interests
• Special prior experiences
• Other important
contributions
-------
What is Program Evaluation?
I What is your definition?
I What is a definition from a reference
book/
I So... What does it involve/
land.-.Why do we do it/
iand-.For Whom is it done/
land. -What is the target of program
evaluation/
An4--Why 4o we evaluate?
• To improve the program -
FORMATIVE EVALUATION is done to'
help form or reform a program
• To prove the program - 5UMMATIVE
EVALUATION is done to sum up the
program's accomplishments
• BOTH are done for decision-making:
Formative - Change/
Summative - Keep or Kill/
So...What POES program evaluation
involve!*
• collaborating
• questioning
• planning
• information-gathering
• information-analysis
• communicating
• interpreting*
• judging*
• Decision-making*
An4 Why Else?
• Postponement
• Ducking Responsibility
• Window Dressing
• Public Relations
• Requirement
-------
Formative Evaluations
I Needs Assessments
I Design Evaluations and
Evaluability Assessments
[Implementation Evaluations
(Performance Audits)
lSome Outcome Evaluations
An4— For whom is program evaluation
Stakeholders of the program who hav<
information gaps about the activities
characteristics, and outcomes of it Ct
need information to make decisions)
For EPA OIG: Congress, the Agency
the Public, the Regulated Communi
other Agencies, States,...
Summative Evaluations
I Most Outcome Evaluations
I Impact Evaluations
I Cost-benefit and Cost-effectiveness
Analyses
iMeta-analysis
An4---What is the target of program
evaluation?
• a program (as opposed to policies,
personnel, or other 'object')
• "an organized set of activities that are
managed toward a particular set of goal
for which the program can be held
separately accountable" Kirchner, 2000
• "a general effort that marshals staff and
projects toward some (often poorly)
defined and funded goals" Scriven 19
-------
What are the levels of 'programs' we
can evaluate in EPA<*
I Objective
I Sub-objective
I Project fc
I other: ^P
Evaluation Questions, Phases of a Program,
and Corresponding Evaluation Types
I Should we have a
program/
I Is the program
designed to work/
I Is the program
implemented as
designed!*
I What outcomes
are being
achieved!*
I Before developed;
Needs Assessment
I'Before
conducted; Design
Evaluation
I During:
Implementation
Evaluation
lDuring:Outc
Evaluation
What actually comprises a program*)
• Inputs - resources used to accomplish
certain activities
•Activities - the activities which produce
products or services for customers
• Outputs -products or services for customers
• Outcomes - customer changes,
environmental changes,
environment/human health changes
• Externalities - contextual influences
Evaluation Questions, Phases of a Program,
and Corresponding Evaluation Types
•Are outcomes • During and after
caused by the the program:
program/ Impact Evaluation
• Were the benefits • After:
worth the cost/ Cost-benefit
Analysis
-------
Evaluation Questions, Phases of a Program,
and Corresponding Evaluation Types
• Could program
have been • After:
conducted more Cost-effectiveness
cost-effectively/ Analysis
• What are the big • After a series of
picture findings evaluations: Meta
and implications/ Analysis
Stan4ar4s of Practice
(Joint Committee Program Evaluation
Standards: Utility, Feasibility, Propriety,
Accuracy
I Yellow Book - GAO Auditing
I Turquoise Book CPCIE)
IAEA Guiding Principles
Evaluation is a Profession
• persons with specialized
knowledge and skills
• unique body of content
• preparation programs
• stablized career opportunities
• working on certification
• professiona I associations...
• ...which influence preparation
• standards
Evaluation an4 Similar Processes
• Evaluation is the broad category of
processes which gather and share
information with stakeholders to use in
decision-making (focus on users and use)
• Would also include auditing,
investigation, monitoring, assessment
• Research is different from evaluation in
that research questions are not nece
targeted by specific information user
use
-------
Are Auditing an4 Evaluation Different!*
I Activities
(Scope
I Types of Questions
I Resources Needed
[Purpose
(Stakeholders
I Utilization
(Results
No Yes
No Yes
No Yes
No Yes
No Yes
No Yes
No Yes
No Y
Can't Wait L/ntil Tomorrow!
(Wednesday, January 17)
• Starting Time: 8:3O AM
• Location: Here!
• Topics: Basic Steps in Program
Evaluation; Logic Modeling;
Pilot Process; Assistance for
the Teams
•Announcements:
What We DidThis Afternoon
• Shared a little about ourselves and
expectations
• Defined program evaluation more fully
• Identified the links between program
development and evaluation
• Learned about the 'profession' of
program evaluation
• Distinguished program evaluation fi
other similar processes
gm
mf
-------
EVALUATION IS.
1 My definition of evaluation is
2 A definition of evaluation from a reference book is
3 Here are some common definitions:
the systematic collection of information about the activities, characteristics, and
outcomes of programs to make judgments about the program, improve program
effectiveness, and/or inform decisions about future programming CPatton, 1997)
determining the extent to which a program has achieved its goals. (What about
implementation, program processes, unanticipated consequences, long-term
impacts')
determining the worth, merit, or value of the something - program, product
(Vsing what evidence; for what purposes')
the systematic assessment of the operation and/or the outcomes of a program or
policy, compared to a set of explicit or implicit standards, as a means of contributing
to the improvement of the program or policy (Weiss, 1998)
4 What is OPE's definition/
-------
Steps in Program
Evaluation
Fits Most Evaluation Need
1. Identify Key Stakeholders - Gather
their Questions
• Congress
• The Agency
• Other Agencies
• The Regulated Community
• The States
• The Public
• Results of Design Evaluation
r Steps
11. Identify Key Stakeholders and Questions
" 2. Assemble an Evaluation Team
i 3. Identify Information Needs and Plan Data
Collection
"4. Collect Data
• 5. Analyze Data
16. Develop Findings
• 7. Draw Conclusions and Make
Recommendations
(if there are questions for an
evaluation)...2. Assemble a Team to
Guide and Conduct the Evaluation
• Evaluation
• Subject Matter
• Agency
• IG Process
• Facilitator
-------
3. Identify Information Needs and
Plan the Data Collection
• What are the information needs/
• Are the data available Cgood, accessible...)/
• Do additional data have to be gathered!*
• I/sing what instruments and processes/
• I/sing what resources/
• Are there alternatives/
5. Analyze
Inductive process for Qualitative Data
(Interviews, Focus Groups, Some
Observational Process)
Deductive process for Quantitative Data
(questionnaires, tests, environmental
measures and monitoring data,
demographics, program descriptives)
4. Gather Data
• ifavailable.. gather
• if not, design and test instruments, gather
data
• OR contract out
6. Develop Findings
• if needs assessment...'The program fits in
these ways"
• if design evaluation... 'The program
appears like it can be implemented and •
can successfully reach these outcomes"
• if implementation evaluation .'The
program is being implemented in this
way"
-------
6. Develop Findings (continued)
• if outcome evaluation.. .'There are these
outcomes, anticipated and unanticipated"
• if impact evaluation...'The program is
causing these outcomes"
• if combination...
7. Draw Conclusions and Make
Recommendations (continued)
• Recommendations are based on findings
and conclusions of this evaluation, as well
as other evaluations and studies: "Based
on these findings and conclusions, as well
as other audit results, we recommend---"
7. Draw Conclusions and Make
Recommendations
• Conclusions are based on the findings and
some pre-identified criteria. 'The
program is being implemented in this way,
thus we conclude that..."
-------
TYPES OF EVALUATION
When, in the life of a program, 4oes evaluation take place (before, during, after), how are the
results of that evaluation used to make decisions about the program Cmid-course correction,
continuing, expanding, or institutionalizing the program, cutting, ending or abandoning the
program, testing a new program idea, choosing the best of several alternatives, deciding
continued funding.)
1 "Formative evaluations strengthen or improve the object being evaluated-they help form
it by examining the delivery of the program or technology, the quality of its
implementation, and the assessment of the organizational context, personnel, procedures,
inputs, and so on " Trochim, 2000
needs assessment
design evaluations.
evaluabihty assessment'
implementation evaluation.
2 "Summative evaluations, in contrast, examine the effects or outcomes of some
ob|ect-they summarize it by describing what happens subsequent to delivery of the program
or technology; assessing whether the object can be said to have caused the outcome,
determining the overall impact of the causal factor beyond only the immediate target
outcomes, and, estimating the relative costs associated with the object" Trochim, 200O
outcome evaluations
impact evaluation-
cost-benefit analysis and cost-effectiveness analysis
meta-analysis:
3 Other reasons for evaluation: postponement, ducking responsibility, window dressing,
public relations, fulfilling someone's requirements
-------
WHAT ARE WE EVALUATING/
1 What is a program'
"The term 'program' refers to an organized set of activities that are managed toward a
particular set of goals for which the program can be held separately accountable "
Kirch ner, 2000
"The general effort that marshals staff and projects toward some (often poorly)
defined and funded goals." Scriven, 1991
" .I will call a national program, like Head Start or Superfund environmental cleanup,
a program. The local operations of the program are each projects Thus, the Head
Start that is operated (locally! is a project An element of the [local] Head Start
project, like involving parents through weekly meetings, is a component. Evaluations
can be directed at any of these levels. We can evaluate national programs, local
projects, or sub-project components." Weiss, 1998
2 What are the various levels of'program' we might evaluate/
3 What comprises a program/
-------
EVALUATION AS A PROFESSION
Refer to articles in' Altschuld, James and Molly Engle, Eds (1994) New Directions for
Program Evaluation. No 62 San Francisco CA Jossey-Bass Publishers
Worthen, Blame "Is Evaluation a Mature Profession That Warrants the Preparation of
Evaluation Professionals/"
Mertens, Donna M "Training Evaluators l/nique Skills and Knowledge"
Kingsbury, Nancy and Terry E Hednck "Evaluator Training in a Government Setting"
What is a Profession/ Is Evaluation There Yet/
Q 1 It has persons with specialized knowledge and skills.
Q 2 There is a developed a body of content unique to its area of specialization.
Q 3. There are preparation programs designed to produce practitioners who are well qualified
in the unique knowledge and skills
Q 4- Stable career opportunities have'emerged for such well-qualified practitioners.
Q 5. The specialization has developed procedures for the certification or licensure ofthose
|udged qualified to practice it.
Q 6. There are associations devoted to furthering the professional development of its
practitioners
Q 7. There are criteria for determining membership in such associations
Q 8. The relevant professional associations influence the preparation programs.
Q 9. The specialization has developed standards to guide those who practice it
Who conducted evaluation before evaluators/
Accountants and auditors, management consultants, planning and systems analysts,
economists, research/product development, test marketing specialists, academics in social and
behavioral science.
Knowledge and Skills Associated with Evaluation:
Research Methodology, Project Management, Strategic Planning, Auditing, Program
Development, Communication Skills, People Skills, Negotiation Skills, Personal Skills
(credible, good judgment...), Cross-cultural Skills, Policy Analysis, Valuing, Economics,
Specific to discipline (education, psychology, health, business, government, environment)
-------
ihe An n Evaluation Association, the AEA's pasi. currem, and incoming
presidents—David Cordray, David Fetterman, and Karen Kirkhan, respec-
tively, Arnold Love, president or (he Canadian Evaluation Society (CES), the
regional CES presidents, Kaihy Jones of the CES. Gary Cox of the University
of Washington, the National Center for Science Teaching and Learning at The
Ohio Slate University, HealthEasi of Si Paul, Minnesota, and the University of
Alabama School of Medicine for assistance with preparation of the directory
The editors of this volume express their deep gratitude to those who partic-
ipated in the study Without them, it would not have been possible
James W Altschuid
Molly Engle
Editors
JAMES W ALTSCHULD is associate professor of educational research and evaluation
and evaluation coordinator for the National Center for Science Teaching and Learn-
ing at The Ohio State University His research interests include evaluation models
and methodology, needs assessment, and the development of evaluation training pro-
grams
MOLLY ENGLE is an assistant professor in the Behavioral Medicine Unit. Division of
Preventive Medicine, Department of Medicine tit the University of Alabama at Birm-
ingham School of Medicine She designs, implements, and conducts research and eval-
uations in behavioral medicine and community-based health services
Criteria for judging the matuiUy of any profession are applu .o
evaluation. Special attention is paid to the question of whether
programs for the preparation of evaluation specialists are
warranted
Is Evaluation a Mature Profession That
Warrants the Preparation of Evaluation
Professionals?
Blame R. Worihen
There is wide agreement that evaluation is an important piofessional special-
ization, but there is less certainly as to whether it has yet attained the status of
a distinct piofession To answer this question, I propose that a fully developed
profession has at least nine characteristics, and I will discuss these charauer-
istics in the context of the need for preparation Worthen and Sandtis (1991)
advanced six of these criteria in their discussion of trends in educational eval-
uation, and portions of this chapter draw on thai earlier work
First, a fully developed profession needs persons wiih specialized knowl-
edge and skills Second, it has developed a body ol content (knowledge and
skills) unique to us area of specialization Third, the profession has developed
preparation programs designed to produce practitioners who are well quali-
fied in ihe unique knowledge and skills Fourth, stable career oppoi mumcs
have emerged for such well-qualified practitioners Fifth, the specialization lias
developed procedures foi the certification or licensuie of those judged quali-
fied to practice it Sixth, the specialization has developed associations devoted
to furthering the professional development of us piaumoneis bevemh. the
specialization has developed criteria for determining membership in such asbo-
ciations Eighth, the relevant piofessional associations in flue nee the piepaiu-
tion programs Ninth, the specialization has developed sund.iulb 10 guide
those who practice it
A simple status check on each of the nine ciueriu propobed would he one
way of judging how far evaluation has moved towaid attaining ihe iluiauei-
isucs of a full-fledged profession However, the matuiaimn of evaluation
NLWDlIUIUNlHU PllH.IUM t-VAIlMIKJN llu OJ bummci I
-------
4 THE PREPARATION OF PROFESSIONAL EVALUATURS
toward the status of a profession can better be understood by considering the
forces that have shaped it across the past thirty years Although space will not
permit me to say much about the historical emergence and evolution of eval-
uation, I will sketch some portions of the histoncal backdrop when it helps me
to clarify the current status of evaluation on the nine catena proposed
Need for Evaluation Specialists
Although there were a few embryonic efforts to evaluate public programs pnor
to 1960 (Shadish, Cook, and Leviton, 1991, Worthen and Sanders, 1987).
most commentators believe that contemporary evaluation of educational and
social programs first emerged dunng the 1960s Early in that decade, the U S
Congress passed federal legislation that, in authorizing antipoverty, juvenile
delinquency prevention, and manpower development and training programs,
both required program evaluation and allocated funds for it (Wholey, 1986,
Weiss. 1987) Yet the emphasis on evaluation built into the Elementary and
Secondary Education Act (ESEA) of 1965 dwarfed previous efforts to mandate
the use of evaluation Broad in scope, the ESEA provided large-scale funding
for education that allowed tens of thousands of federal grants to be awarded
to local schools, state and regional education agencies, and universities Due
largely to the efforts of Robert F Kennedy, the ESEA required the recipients of
grants dealing either with compensatory education for disadvantaged youth or
with innovative educational projects (the great majority of grants) to file an
evaluation report showing what had resulted from the expenditure of public
funds
Overnight, thousands of educators were required to evaluate their own
efforts Few were up to the task Classroom teachers and building principals
were among those pressed into technical activities for which they had little
training The results were abysmal And when well-trained educational, psy-
chological, or sociological researchers were called in to help, the results were—
surprisingly—not much better Despite their technical prowess, these
researchers were not prepared for the complex tasks of identifying the influ-
ences that could be attributed to each of several components of a program or
even of separating the effects of the program from other activities going on in
the school Clearly, new evaluation approaches, methods, and strategies were
needed
Meanwhile, areas outside education were expenencing increased demands
for evaluation, although it was often called by other names By the late 1960s,
Congress had authorized monies for evaluation of social programs in areas as
diverse as the Job Corps, vocational rehabilitation, child health, and commu-
nity action Managers of the projects and programs funded under such social
legislation searched to find the individuals best equipped to fill the newly cre-
ated evaluation roles Faced with an absence of persons trained directly in eval-
uation, ihf ~«ployed people trained for roles that contained some evaluative
IS EVALUAI ION A MATURE PRO! ESSION? 5
functions—professional accountants and auditors, management consultants,
planning and systems analysts, economists, research, product development,
and lest marketing specialists from the private sector, and academics in areas
relevant to the collection and analysis of evaluative information (Shadish,
Cook, and Leviton, 1991).
The evaluations conducted by these persons were little better than those
conducted by the classroom teachers and educational psychologists who had
been pressed into service as evaluators on federally funded education projects
While most of those drafted or recruited into evaluation roles were very skill-
ful in some of the tasks required of evaluators. few were even aware of the
broad range of tasks that were essential for a complete and adequate evalua-
tion Fewer still possessed the skills that one must have in order to complete
those tasks. The need for persons with a new constellation of specialized skills
was evident to any insightful observer
Today the need for evaluation specialists is generally accepted, although
many policy makers and program managers who are naive about the knowl-
edge and skills that evalualors should possess still attribute evaluation exper-
tise to self-appointed or self-anointed "evaluators" who lack essential evaluation
skills and knowledge. Despite the frequent lapses when evalualors are selected,
that there is a need for evaluation specialists seems to be well established
Development of Unique Content
When demands for evaluation increased dramatically in the 1960s, the result-
ing evaluation studies revealed the conceptual and methodological impover-
ishment of evaluation as it then existed Theoretical and methodological work
related directly to evaluation did not exist, and evalualors were left to gather
what they could from theories in cognate disciplines and lo borrow whai they
could from the methodologies developed in such fields as experimental design,
psychometrics, survey research, and ethnography The results were disap-
pointing and underscored the need for the development of new conceptual-
izations and methods tailored to fit ihe needs of evalualors more precisely.
Scholars responded lo this need, and by 1970 important seminal wntmgs had
provided conceptual foundations and scaffolding for the young field of evalu-
ation (Cronbach. 1963. Scnven. 1967, Slake. 1967, Siufflebeam, 1968) Books
of readings on evaluation were published (Caro, 1971, Worthen and Sanders,
1973) Articles about evaluation appeared with increasing frequency in pro-
fessional journals. Together, these publications resulted in a proliferation ol
new evaluation models that collectively provided new ways of ihinkmg aboui
evaluation This emerging body of literature showed evaluation lo be a multi-
dimensional technical and political enterprise that required both new concep-
tualizations and new insights into the ways in which melhodologies borrowed
from other fields could be used appropriately
In recognizing the need for unique theories for evaluanon ' -lish. Cook.
-------
6 THE PREPARATION or PROI FSSIONAL EVALUAIORS
and Leviion (1991. p 31) noied that, "as evaluation matured, its theory look
on its own special character (hat resulted from the interplay among problems
uncovered by practitioners, the solutions they tried, and traditions of the aca-
demic discipline of each evaluator, winnowed by twenty years of experience "
Publications focusing exclusively on evaluation appeared in the 1970s
They included such journals and senes as Evaluation. Evaluation and Program
Planning, Evaluation Practice, Educational Evaluation and Policy Analysis, New
Directions for Program Evaluation, and the Evaluation Studies Review Annual The
number of books published expanded markedly m the second half of (he
1970s and throughout the 1980s Textbooks, reference books, and even com-
pendia and encyclopedias of evaluation all appeared Clearly, the necessary
conceptual underpinnings of a profession are accumulating in a body of eval-
uation literature that is arguably unique Thus, evaluation seems to qualify as
a profession on the second criterion There is a body of knowledge that out-
lines the content of the field and us unique (or adapted) theories, strategies,
and methods
Programs for the Preparation of Evaluators
Foreseeing that education had few persons trained in educational inquiry
skills, the U S Congress funded graduate training programs in educational
research and evaluation in 1965 These programs included fellowship stipends
for graduate study in these new specializations Several universities launched
full-fledged, federally funded graduate programs aimed at training educational
evaluators When federal funds disappeared, so did many of the graduate pro-
grams that they had supported In 1971. graduate programs for training eval-
uators existed at more than a hundred American universities (Worlhen and
Dyeis, 1971) Fifteen years later, only forty-four U S universities had such pro-
grams (May, Fleischer, Schener, and Cox, 1986) And many programs that had
had many courses m evaluation scaled back to a single elective course
The evaluation preparation programs that continued generally offered
training tailored to fit the reconceptualized views of evaluation that were
emerging Notions of how evalualors should be trained gradually expanded
beyond traditional training Courses in research design, statistics, and mea-
surement were often supplemented by a wide variety of applied methods and
techniques courses in such areas as naturalistic observation, interviewing tech-
niques, content analysis, peiformance assessment, and communication and
writing skills Evaluation internships, assistantships, and praclica became more
central in preparation programs as evaluation mentors realized that, in evalu-
ation as elsewhere, the best training is often apprenticeship (raining
In recent years, the training of evalualors has increasingly been relocated
to nonacaclemic settings In-service evaluation training for practitioners is often
offered in schools, slate agencies, and businesses On occasion, large corpora-
ls EVALUATION A MATURE PROFESSION' 7
(ions have established corporate (raining centers (such as Xerox Document
University) These centers, which resemble mmiuniversilies, provide training
in evaluation along with other techniques Some of these centers award cer-
tificates attesting that the recipient is qualified in the speuahzation in which
he or she has been trained And, of course, many persons follow serendipitous
career paths into evaluation roles where their preparation consists pnmanly of
on-ihe-job bootstrapping On balance, there are a sufficient number of evalu-
ation training programs m universities, government agencies, corporations,
and other settings to produce an ongoing supply of professional evalualors It
seems clear that evaluation meets this third criterion for having reached the
status of a profession.
Stable Career Opportunities for Evaluators
One sign that a specialization is a profession is a continuing need for the ser-
vices of personnel trained in that specialty No field that is only a fad that flour-
ishes briefly and then fades would qualify as a profession In judging evaluation
on this dimension, we must consider whether, despite uncertain social and
economic trends, evaluation provides the stable employment opportunities
that are typical of mature professions
At first, evaluation seemed to be just another boom-and-bust specialty
When the need for evaluators grew quickly between 1965 and 1975, it
seemed that evaluation training could provide stable career opportunities for
anyone who developed a reasonable degree of expeitise in evaluation That
view grew doubtful in the late 1970s, when a dip in the level of federal fund-
ing for evaluation appeared to signal a declining U S job market for evalua-
lors In the early 1980s, Ronald Reagan cast a darker shadow over the
evaluation scene Federal evaluation mandates were quietly shelved as the so-
called new federalism reduced federal funding for education and other social
programs and cut federal control over the ways m which states and local agen-
cies spent the federal funds that they received Much categoncal funding that
had required evaluation was replaced by block grants to states, which were
largely exempt from evaluation requirements Most analysis during the early
1980s were convinced that stale and local agencies, hard pressed for oper.i-
uonal funds, would use categoncal funding to buy supplies, repair equip-
ment, or add staff Evaluation was predicted to be one of the majoi casualties
of the Reagan administration Since federal mandates had spawned evaluation
in state and local agencies, it seemed reasonable to expect thai evaluation
would decline or even cease when federal evaluation requirements weie
relaxed or abolished
By 1982. these pessimistic prophecies seemed to have pioved accurate Gov-
emmenial monitoring of categorical funding programs was drastically reduced
Individual evalualors and evaluation agencies lhai depended on conirjcts wiih
-------
8 THE PREPARATION or PROFESSIONAL ^VALUATORS
federal programs found ihis source of income drying up For example,
Shadish. Cook, and Levuon (1991) noie thai ihe number of evaluation stud-
ies conducted by ihe U S Office of Planning. Budget, and Evaluation dropped
from 114 in 1980 to 11 in 1984 The declines in other evaluation activities that
depended on federal funds weie comparable
Gloom soon spread over the evaluation landscape, and evaluaiors' con-
ferences in the early and micl 1980s focused on such themes as the decline of
evaluation Evaluation trainers at many universities began to ask whether it
was ethical to tram neophytes in roles for which demand was thought to be
diminishing For a lime, it seemed that the evaluation bubble had burst Eval-
uation seemed destined for the graveyard of promising endeavors that had
failed to fulfill their potential
But the situation began to change For reasons that at first could not be
explained, some evaluation agencies seemed to be bucking the declining
trends Indeed, they found that the 1980s had brought a stronger surge of eval-
uation business than ever before, and soon they were eagerly seeking to add
well-qualified evaluaiors to their staffs Gradually, it became apparent that only
the evaluation agencies that depended primarily on federal funds had been
hard hit, while the agencies that served state and local agencies, corporations,
professional associations, and the like were finding that evaluation was still a
bustling, thriving enterpnse
Somehow—perhaps only the most sagacious historical analyst could
determine all the causes—decision makers in slate and local government, busi-
ness, and industry had begun to use evaluation for their own purposes—to
provide information that they needed to guide policy and program imple-
mentation Gradually, increasing numbers of agencies began to commission
evaluation studies not because they had been forced to but because they
believed that the resulting data would be helpful House (1990) noted this
trend in the emerging tendency for large bureaucracies to develop their own
evaluation offices, and Worthen and Seeley (1990) described how evaluation
had been institutionalized by a variety of enterprises across broad sectors of
contemporary society
For those who recognized that this widespread instilutionalizaiion of eval-
uation would lend stability to the evaluation job market, ihe pessimism thai
had prevailed at the beginning of the decade soon passed Openings for eval-
uniors were appearing in a wide variety of sellings, which included public and
private school districts, stale and regional education agencies, social service
agencies, univcrsmes and colleges, slate systems of higher education, test and
text publishers, and the military, business, and industry Evaluation academics
were at lirst amused and then amazed as ihcy saw their siudents recruited to
evaluate personnel training programs run by large, national aicounung fums,
insurance and brokerage houses, and fast-lood chains
The shortage of trained evaluatois is obvious today, as the traditional
employe- -f evaluaiors aic now forced to compete with suth firms as Aetna,
Is CVALUAI ION A MA luiir
Xerox, and Price Waterhouse Every year, the number of evaluation vacancies
outside academic sellings surpasses the number of qualified candidates And
the surge in federal program evaluation over the past few years has accentu-
ated the need for evaluators Ginsburg, Mclaughlin, and Takai (1992, p 24)
note lhai "spending on program evaluation by the U S Department of Educa-
tion exceeds $40 million per year, a tripling of the budget over the last five
years " So a career in evaluation seems again to be a very good possibility If
the probability of continued employment in a specialization is an imporiam
criterion for considenng u lo be a profession, evaluation may well be consid-
ered a viable profession
Procedures for the Certification or Licensure of Evaluaiors
Since an extensive discussion of this question is beyond the scope of this chap-
ter, I will only touch on U lightly For this chapter, the central question is
whether there are mechanisms for the certification or licensing of evaluators
similar to those thai mark teachers, psychologists, and certified public accoun-
tants as professionals The answer, of course, is no
Despite pleas that the Amencan Educational Research Association estab-
lish mechanisms to provide certification for qualified evaluators (Cagne, 1975,
Worthen, 1972), neither it nor any other association or agency has stepped for-
ward to assume responsibility for the licensing or ceilificalion of evaluation
practitioners As a result, there is currently no way of preventing incompetent
or unscrupulous operators from proclaiming themselves to be evaluaiors
Without some type of credenlialing process, it is difficult for ihosc who need
evaluation services lo determine in advance that those whom they select aie
indeed competent "Lei the buyer beware" is still the watchwoid for those who
must retain the services of an evaluation specialist In the absence of certifica-
tion or licensure, unprincipled hucksters can do much mischief and in the
process badly tamish ihe image of evaluation
Perhaps that cannot be helped 1 am much less sanguine today thai we can
set up credenlialing systems than 1 was two decades ago However desirable u
may be to have some way of ensuring that the unqualified cannot masquerade
as evaluaiors. the development of such a mechanism does noi seem feasible
for two reasons First, rooted as evaluation is in so many disciplines and with
today's evaluators trained in as many diverse specializations and through sui.li
diverse means as they are, it is hard to imagine how any broad agieemciu
about the essential elements of evaluation competencies could be luigeil I'm
bluntly, since there is so little agreement about the methods .uul ie(.tinu|iies
that evaluaiors should use, it seems almost certain thai a majoiny of |>iauit--
ing evaluaiors would reject an elfon 10 construct and use a lemplaie ul any son
to judge the qualifications of all evaluators Second, u seems unlikely that any
professional association or government agency will soon lie equipped to grap-
ple with the thorny and often litigious business of lucusinc •inlu.jiioii.
-------
10
1 IIL PREPARATION or PROIT-SSIONAI LVAI UAIOKS
especially in a field wheie those affected by the effort are more accustomed to
evaluating than to being evaluated
Nevertheless, until and unless we establish some feasible mechanism for
ensuring that those who practice evaluation are competent to do so, evalua-
tion cannot be considered a fully mature profession
Development of Professional Associations for Evaluators
Several professional associations m North America have emerged to provide
homes for evaluators (Similar trends are seen in other coumnes ) One of the
first North American efforts was not a full-blown association as such but rather
Division H of the American Educational Research Association, which provided
a home for school evaluators However, two professional associations for prac-
ticing evaluaiors were founded in 1976 The Evaluation Network (EN) con-
sisted laigely of educational evaluaiors, while most members of the Evaluation
Research Society (ERS) served in other professional fields
In 1985, the EN and the ERS merged to form the American Evaluation
Association (AEA), which, with about 2,200 members, is the largest profes-
sional association that exists solely to serve the needs of practicing evaluaiors
The Canadian Evaluation Society (CES) was launched 10 serve the needs of
Canadian evaluation practitioners who worked in settings ranging from provin-
cial ministries to private consulting groups Given the scope and stature of
these associations, it is clear that evaluators have viable professional organiza-
tions On this criterion, evaluation fares as well as any profession
Criteria for Determining Membership in Evaluation
Associations
Most professions have established criteria foi denying membership in profes-
sional associations to those who are patently unqualified in the business of ihe
profession This cannot be said of evaluation The criteria for membership in
all the professional evaluation associations just mentioned are lenient, and no
organization would effectively exclude those who were not qualified as evalu-
aiors from membership On this criterion, as on the criterion of cenificalion,
it appears (hat evaluation has not reached full maturity as a profession
Influence of Evaluation Associations on Preparation
Programs for Evaluaiors
In many professions, I he majoi professional associations play a powerful mle
in shaping university pieparanon progiams through accreditation or similar
mcilunisms Evaluation associations excit no such influence None of the pro-
fessional associations for evaluators mentioned earlier exercise any direct ton-
Is EVALUAIION A MAIURIE I'KOIKSSION'
II
trol or influence over any preservice program ihai purports to train evaluators
The evaluation associations do not accredit preservice training programs 01
control decisions about required course content, essential internship experi-
ences, or faculty qualifications On this criterion, too, evaluation is not fully a
profession
Development of Standards for Evaluation Practice
Most professions contain technical standards, ethical standards, or both that
are intended to ensure that professional practice is of high quality Evaluation
was without such standards during us early years Then in 1981, evaluation
took a giant step forward toward qualifying as a profession when several years
of work by the Joint Committee on Standards for Educational Evaluation, a
coalition of professional associations concerned with evaluation in education
and psychology, resulted in the publication of Standards for Evaluations of Edu-
cational Programs, Projects, and Materials (Joint Committee on Standards for
Educational Evaluation, 1981) These comprehensive standards were intended
to guide both those who conducted evaluations and those who made use of
evaluation reports
In 1982, the ERS published another set of standards for evaluation prac-
tice (Rossi, 1982) Six years later, the joint Committee on Standards for Edu-
cational Evaluation (1988) published the Personnel Evaluation Standards
Currently, the same organization is neanng the end of the process of revising
the standards first published in 1981 If a set of standards to guide professional
practice is a hallmark of a profession, then evaluation certainly qualifies, for us
standards are much better developed than those now used to guide practice in
several more venerable professions
Profession, Professional Specialization, or Field of
Professional Practice?
Up to this point, we have considered nine touchstones that seem useful in
ascertaining whether a field of endeavor has attained the status of a distinct
profession. Let us now consider these criteria together What do they tell us
about the progress of evaluation toward becoming a profession? Is evaluation
separate and distinct from the other professions and disciplines with which it
has been intertwined for decades7 In short, is evaluation a profession7
The answer depends on the rigor with which we apply the nine criiena
just examined If an area of specialization must meet all nine criteria before it
can be thought of as a profession, then evalujuon is not a piofcssion Tiguic
1 1, which summarizes the preceding discussion of evaluation and the char-
acteristics that most fully developed professions possess, shows that evaluation
falls short on three For evaluation to be consult ied a full-fledged profession.
-------
12 THH PREPARATION OF PROFESSIONAL EVALUAIORS
Figure 1.1. Criteria for Judging Whether Evaluation
Has Become a Profession
Does Evaluation Meet the Cntrnnn 0}
V«
N.i
1 A need for evaluation specialists7
2 Conicni (knowledge and skills) unique
lo evaluation7
3 Preparation progiams foi cvaluaiors7
4 Stable taieer opportunities in evalua-
tion7
5 Cenificaiion or licensure of evaluators7
6 Appropnaic professional associations
(or evaluators7
7 Exclusion of unqualified persons from
those associations7
8 Influence of evaluators' associations on
prcscrvice preparation programs for
e valuators7
9 Siandards lor the practice of evalua-
tion7
V
V
these three areas will need 10 be dealt wuh Nevertheless, some conditions may
be difficult ever to meei For example, we may never resolve ihe challenge of
certifying evaluators Does this mean that evaluation will never qualify as a pro-
fession7 Or can evaluation be considered a profession if 11 meets most of the
criteria7
Those who have commented on the status of evaluation as a profession are
not of one voice A decade ago, most writers seemed to hold the view that eval-
uation had not yet attained the status of a distinct profession For example,
Rossi and Freeman (1993, p 432) concluded that "evaluation is not a 'profes-
sion.' ai least in terms of the formal criteria (hat sociologists generally use to
characterize such groups. Rather, it can best be described as a 'near-group,' a
large aggregate of persons who are not formally organized, whose membership
changes rapidly, and who have little in common in terms of the range of (asks
undertaken, competencies, work sites, and shared outlooks " Merwm and
Werner (1985) have also concluded that evaluators cannot yet claim full pro-
fessional status
Several rcceni authors have reached a somewhat moie liberal conclusion
For example, Panon (1990) states unequivocally thai evaluation has become
a profession and thai it is a demanding and challenging one at ihai Shadish,
Cook, and Lcviion (1991, p 25) are slightly more cautious "hvaluation is a
piolcssion in the sense that n shares certain atiiibutcs wuh other professions
and differs from purely academic specialties, such as psychology or sociology
Although they may have academic loois and members, professions aie eco-
h EVALUATION A MAI LIRE PROFESSION?
13
nomically and socially structured to be devoted primarily 10 practical applica-
tion of knowledge in a circumscnbed domain wuh socially legitimated fund-
ing Professionals tend to develop standaids of practice, codes of ethics,
and other professional trappings Program evaluation is not fully professional-
ized, like medicine or the law, it has no licensure laws, for example but u
lends toward professionalization more than mosi disciplines"
To summarize, some now view evaluation as a profession because it pos-
sesses most of the touchstones that collectively define a profession Others
believe thai evaluation is no) now a full-blown piofession and that it may never
become one because u lacks licensure laws and some other characteristics ol
such professions as law and medicine Perhaps evaluation will forever be a
near-group that lends toward professionalization Perhaps we may best
describe u as a near-profession—an area of professional practice and special-
ization that has us own literature, as own preparation programs, us own stan-
dards of practice, and us own professional associations Or perhaps evaluation
is best viewed as a hybrid of profession and discipline thai possesses many
characteristics of both and lacks some essentials of each (Scriven, 1991,
Worthen and Van Dusen, in press) Or perhaps the label that we give lo the
practice of evaluation is of less consequence than the ways in which we struc-
ture programs aimed at preparing competent evaluation piaciuioncrs
Are Preparation Programs for Evaluation Practitioners
Warranted?
It would matter little whether we considered evaluation to be a profession if u
were not that our conceptions—and even our semantics—influence the ways
in which we prepare personnel for evaluation roles If we think of evaluation
as a discipline, then we will expect preservice programs for evaluators to be
patterned after those used to tram academics in other disciplines If we think
of it as a profession, the course work and internships in our evaluator prepa-
ration programs will tend to resemble the methods courses and practica used
to prepare practitioners for other professions If we think of evaluation as a
hybrid of discipline and profession, then our evaluation programs will com-
bine elements of programs aimed at training practitioners wuh elemcius ol pro-
grams used to prepare academics
However we think of evaluation, this much is cleai Evaluation has
matured rapidly dunng the pasi quarter century, and iheic is every uuli<_aiioii
that u will continue to develop and grow in the decades ahead Wuh lib own
journals, standards, and professional reference groups, evaluation has devel-
oped many of the important characteristics of a piofession And whcihci 01
not it can be considered a profession, u has emerged as .m impou.mi aic:i ol
specialization that demands uniquely prepaied personnel il u is to icadi us full
potential. Il has become institutionalized in many public and private sectors
-------
14 Till: fREPARATION OH PROITbSIONAL EVAIUAIURS
of oui society, and, if evaluaiois are prepared to meet the challenge, evaluation
can become one of the most useful and far-reaching areas of human endeavor
Against this backdrop, the present and potential importance of evaluation fully
warrants a careful consideration of the issues and strategics involved m prepar-
ing evaluation specialists
References
Caro. F G (ed ) Readings in Evaluation Research New York Sage, 1971
Cronbach, L J "Course Improvement Through Evaluation * Teachers College Record, 1963. 64,
672-683
Gagne, R M "Qualifications of Professionals in Educational R&D " Educational Researcher, 1975,
Gmsburg. A , Mclaughlin, M . and Takai, R "Remvigoraling Program Evaluation at ihe U S
Department of Education " Educational Researcher, 1992, 21 (3). 24-27
House. E R •Trends in Evaluation " Educational Researcher, 1990, 19(3). 24-28
|oini Committee on Standards for Educational Evaluation Standards /or Evaluations of Educational
Programs, Projects, and Materials New York McGraw-Hill, 1981
Joint Committee on Standards for Educational Evaluation The Personnel Evaluation Standards
Newbury Park, Calif Sage. 1988
May, R M , Fleischer, M , Schetrer, C J . and Cox, G B "Directory of Evaluation Training Pro-
grams " In B G Davis (ed ), leaching of Evaluation Across the Disciplines New Directions for
Program Evaluation, no 29 San Francisco jossey-Bass, 1986
Merwtn.J C , and Werner. P II "Evaluation A Profession'" Educational Evaluation and Policy
Analysts. 1985. 7(3). 253-259
Pallon.M Q " The Challenge of Being a Ptnfesston " Evaluation Practice, 1990. II (I). 45-51
Rossi, P M (ed ) Standards /or Evaluation Practice San Francisco Jossey-Bass, 1982
Rossi. P U , and Freeman, H E Evaluation A Systematic Approach (5th etl ) Newbury Park,
Calif Sage. 1993
Scnven.M "Ihe Methodology of Evaluation " In R C Slake (ed ). Cur niulum Evaluation Amer-
ican Educational Research Association Monograph Series on Evaluation, no I Chicago Rand
McNally. 1967
Scriven. M "Introduction The Nature of Evaluation " Evaluation Thesaurus (4ih ed ) Newbury
Park. Calif Sage. 1991
Shadish. W R, Jr. Cook, 1 D , and Levnon, L C Foundations o/ Program Evaluation Newbury
Park, Calif Sage, 1991
Slake, R E "The Countenance of Educational Evaluation " Teachers College Record, 1967, 68,
523-540
Stufflebeam, D L Evaluation as Enlightenment /or Decision Making Columbus Ohio Slate Uni-
versity Evaluation Center, 1968
Weiss. C H "Evaluating Social Programs What Have We Learned'" Society. 1987, 25(1). 40-^45
Wholey, J E "Using Evaluation lo Improve Government Performance " Evaluation Practice. 1986
7.5-13
Worthen. B R "Cemficanon for Educational Evaluaiors Problems and Potential " Paper pre-
sented ai the annual meeting of the American Educational Research Association, Chicago, Apr
15, 1972
Wculhen. B R , and Byers, M L "An Exploratory Study of Selected Varubles Related to the
Training and Caieers of Educational Research and Research-Related Personnel " Washington.
D C Ametican rducaiion.il Research Association. 1971
Wonhen. B R . and Sanders. | R Educational Evaluation Theory and Practice Belmont. Calif
Wadiwoitli, 1973
IS EVALUAI ION A MA I LIRE HKUrCSSION' I 5
Worlhen, B R , and Sanders, J R Educational Evaluation Alternative Approaches and Pradicul
Guidelines New York Longman, 1987
Wonhen, B R , and Sanders, J R "The Changing Face of Educational Evaluation * Theory into
Practice. 1991,30(1). 3-12
Wonhen, B R , and Seeley, C "Problems and Potential in Institutionalizing Evaluation in State
and Local Agencies " Paper presented al ihe annual meeting of the American Evaluation Asso
ciaiton, Washington, D C . Oci 19. 1990
Wonhen. B R . and Van Dusen. L M "The Nature of C valuation " In H Walben (ed ). Interna-
tional Encyclopedia oj Education (2nd ed ) Oxford. England Pergamon Press, in press
BLAINE R WORTHEN is piofessor and chan of the Research and Evaluation Method-
ology Program in the Department of Psychology at Utah State University and dira
tor of the Western Institute for Research and Evulualion in Logan, Utah
-------
The skills and knowledge that evalualors need include (dose bor-
rowed from other disciplines as well as (hose unique to the field of
evaluation Inclusion of multiple perspectives in evaluator (raining
can help to develop the field and improve the practice of evaluation
Training Evaluators: Unique Skills
and Knowledge
Donna M. Mertens
Evalualors work in complex environments, such as emichmem programs for
deaf, gifted adolescents, drug and alcohol abuse programs for (he homeless,
and managemenl programs for high-level radioactive waste The field of eval-
uation itself is evolving as u develops ihrough ihe leflccuvc practice of (he pro-
fessionals involved Evalualors have an ethical responsibility 10 continue their
education and keep up-to-date on developments in the field (Easimond,
1991) In consequence of this assertion, I have written this chapter for students
of evaluation, by whom 1 mean noi only those who are enrolled in formal
training programs bui also all practicing evalualors and teachers of evaluation
Perspectives and Assumptions
My answer (o (he question, What are (he unique skills and knowledge that
should be considered for the preparation of evalualors7 is based on four
assumptions First, we live in a multicultural society, and I assume thai evalu-
alors musi bnng a sensitivity to multicultural issues and perspectives to their
work Beaudry (1992, p 82) noies lhal program evaluators must seek in
include ihe multiple perspectives of eihmciiy. race, gender, social class, and
persons with disabilities "Program evaluation must lake IIOIILC ol the clungcs
I lhank the following people for their comments on my diafi Ir.imcwoik Jennifer Gicrne.
Jody Fuzpainck. Hallie Preskill. Nick Casimoncl. Jack McKillip. Di.iniu Nt-wiiian. and leiry
llednck
Ntw DlttLinm KM PniGUM FVAIIMIKW. no 61 Summer IVM c |IFMly BJU Publnhr» I 7
-------
18
Thi. i-Rf.PARAI ION Ol~ I'KOn SSIONAI. EVAI UAfORb
in our society and begin 10 respond to social issues represented by multicul-
tural education Hale crimes and ethnic strife are reported on the front pages
of newspapers and in the courts and the schools as well as all around the
world In education, much of what we know about negative lacial prejudice,
biases in testing, culluially biased instructional materials, and teacher effects
remains part of the hidden cumculum Multicultuial awareness and education
have equal relevance for health care, business, and industry as these sectors of
society cope with the shifting patterns of a culturally diverse work force "
Stanfield (1993, pp 6-7) addresses the same point "The dramatically
changing world in which we live demands lhai we cease 10 allow well-worn
dogma to keep us from designing research (read evaluation] projects that will
provide the data necessary for the formulation of adequate explanations for the
racial and ethnic dimensions of human life " Although the author just cued
speaks from the context of ethnicity and social science research, his comments
can be more bmadly interpreted as suggesting that evaluators must rethink tra-
ditional methods in order to be responsive to such alternative perspectives as
those of minorities, women, the poor, and persons with disabilities
Evaluation literature is only beginning to address the feminist (Farley and
Menens. 1993. Menens, 1992, Shapiro, 1987) and minority perspectives
(Madison. 1992) However, students of evaluation can borrow from the
research-based literature and cieate the applications and implications that are
necessary The peispeaive of persons with disabilities has not been addressed
as fully in the research literature (Mertens and McLaughhn, in press) as the
perspectives of other groups 1 include them here because a growing literature
suggests that they view themselves as an oppressed cultural group (Wilcox,
1989)
Second. I view evaluation as a unique discipline that has borrowed many
skills and much knowledge from social science research Evaluation is an
emerging profession with an expanding body of skills and knowledge that
require continual review Skills are things that evaluators need to be able to do
Knowledge is things that evaluators need to know The model that 1 propose
combines skills and knowledge, because evaluators need to be able to apply
what they know in order to conduct evaluations competently
Thud. 1 assume that a core set of skills and knowledge exists across disci-
plines for evaluators The particular emphasis of the skills and knowledge
requned in specific contexts depends on the discipline in which the training
occurs, the level of the training, the nature of the training (for example, degree
or nonacademic piogram, single course or program, new training or continu-
ing education), the aiea of application (foi example, education, economics,
psychology, criminal justice, public administration, business, health, sociol-
ogy, social work), the naiine of the organization that employs the evaluator,
and the level of the position that he or she holds Having asserted that there is
a coie set of skills and knowledge, I also want to recognize the dispute between
pioponents of the view that evaluation has content-specific knowledge and
advocates of the gencialist view Eisner (1991) describes a connoisseur as an
TRAINING EVALUATORS UNIQUI SKILLS AND KNOWIMH.I
19
individual who is highly perceptive in one domain and able to make fine dis-
criminations among complex and subtle qualities I believe that evaluatois
need either to be connoisseurs in the area of application (for example, drug
abuse, deafness) or to include a subject matter expert (connoisseur) in the
planning, conduct, and interpretation of an evaluation
Last, I assume also that evaluators must be capable of being responsive to
the needs of the client They must be capable of recommending the most
appropnate approach to an evaluation problem Some problems can best be
studied with quantitative data, while others call for qualitative data While
individual evaluators may not be expert in all quantitative and qualitative
research methods, they do need to be able to recommend the most appropn-
ate approach If necessary, they can work in a team with evaluators who have
greater expertise in other methods Sechrest (1992) argues that evaluators need
increased sophistication in quantitative methods I agree that there is room for
experts in either quantitative or qualitative methods, but there is also a need
for those who function comfortably m both domains Lincoln and Cuba
(1992) argue that a mixture of quantitative and qualitative methods can be
appropnale to any paradigm
Methodology
I used a number of different techniques to identify the skills and knowledge
unique to evaluation 1 reviewed existing literature, such as textbooks on eval-
uation (Bnnkerhoff, Brethower, Hluchyj. and Nowakowski, 1983. Popham.
1988; Shadish, Cook, and Leviton. 1991, Rossi and Freeman, 1993, Posavac
and Carey. 1992; Worthen and Sanders, 1987), presentations on training at the
annual meetings of the American Evaluation Association (Altschuld, 1992, Dar-
nngion. 1989; Covert, 1992, Eastmond. 1992, Mertens, 1992), literature iden-
tified through the use of ERIC and other data bases, training-related articles in
the journal Evaluation Practice, the U S General Accounting Office (1991) per-
formance appraisal system for evaluators, and Da vis's (1986) volume on eval-
uation training. I also consulted other evaluators and reflected on my own
expenence as an evaluator trainer for twenty-plus years I conducted a content
analysis of the skills and knowledge that I found in these various sources and
organized them into a conceptual framework I shared this conceptual frame-
work with evaluators and trainers in a variety of disciplines, including educa-
tion, psychology, business, administration, government, and interdisciplinary
programs.
Skills and Knowledge Needed
I have divided the skills and knowledge into four categories those unique to
evaluation, topics associated with typical training in the methodology of
research and inquiry, topics in such related areas as political science or anthro-
pology, and discipline-specific topics I chose this organizational framework
-------
20 THE PREPARATION or PRorhSbiONAL EVAUIAIORS
because it lends itself lo the overall design of an evaluation training program
A student who enrolls in an evaluation training program typically also receives
course work in research methodology, including research design, statistics, and
measurement This organizational framework suggests areas that need to be
included in evaluation courses because they are not often taught elsewhere (the
components of these other topic areas arising in evaluation are unique) It also
suggests other disciplines that can provide a more complete training experi-
ence Exhibit 2 1 displays the topics associated with research, related areas,
and specific disciplines Discussion of these topics is beyond the scope of this
chapter
I focus here on the unique skills and knowledge associated with evalua-
tion Standard evaluation textbooks typically cover some of these topics, and
I will therefore not elaborate on them I will discuss certain controversial and
emerging topics in evaluation for three reasons First, the standard textbooks
typically do not discuss them at length Second, students of evaluation should
be aware of the controversies m the field Third, I hope to push trainers to
think of including emerging topics—that is, topics that are still developing and
on which consensus has not yet developed—as valid for inclusion in the eval-
uation curriculum
The skills and knowledge listed in Exhibit 2 1 can all be taught with spe-
cial insights and examples from evaluation However, certain topics are not
generally covered in a course in research, related areas, or other disciplines
These topics are discussed in the sections lhat follow
Introductory Information About Evaluation. Evaluation textbooks typ-
ically include information about the definition of evaluation, the reasons why
evaluations are conducted, the various lypes of evaluations (for example,
implementation, process, outcome, impact, formative, summalive), trends
affecting evaluation, the roles thai evaluators can play (for example, external,
internal), and the history of evaluation
Philosophical Assumptions. Evaluation classes should teach the philo-
sophical assumptions underlying the positivisl and postpositive paradigmatic
orientations Although these assumptions should be taught in the research
methodology classes, the teacher of evaluation cannot safely assume lhat they
will be (Lopez and Mertens, 1993) Lather (1992) proposed an organizing
framework for paradigms that is relevant here positivists who seek to predict,
postposinvisis who seek to understand (this group includes those whom the
evaluation literature has labeled interpretive, natuialistic, and conslruclivisl), and
postposinvisis who seek 10 emancipate (this group includes feminists and race-
specific mqmrcis) For the reasons outlined in ihe section on perspectives and
assumptions, 1 would add pcisons wuh disabilities to ihc emancipatory cate-
gory Cuba and Lincoln (1989) have explained ihe assumptions underlying
positivists and post posit ivists (read consiruilwsis) in detail, and the trainer of
evaluation could follow up on emancipatory paradigms through such sources
as Lathe- ^1), Farley and Mertens (1993), Harding (1993), Shapiro (1987).
TRAINING EVALUATORS UNIQUE SKILLS AND KNOWLI IJCE 21
Exhibit 2.1. Knowledge and Skills Associated with Evaluation
I Knowledge and skills associated with research methodology
A Philosophical assumptions of alternative paradigms and perspectives, for exam-
ple, positives and postposinvisis (for example, construcnvisi, feminists, minoii-
lies, and persons with disabilities) (Lather, 1992)
B Methodological implications for alternative assumptions
C Planning and conducting research
1 Literature review strategies
2 Theoretical frameworks
3 Hypothesis/questions formulation
4 Research design (quantitative designs—for example, observational research,
surveys, experimental, quasi-expenmenial. correlational, causal comparative.
and single-subject—and internal and external validity qualitative designs—for
example, case studies and ethnography—and trustworthiness, and mixed
designs)
5 Data collection strategies sample selection, quantitative data collection (for
example, test construction, reliability, validity, application of tests, norm-anil
cnienon-referenced tests, selective measurement imniments, assessing mea-
surement instruments, measurement error and bias interpreting test results,
instrument construction), qualitative data collection (for example, observation,
interviewing, focus groups, document review, unobtrusive measures)
6 Data analysis and interpretation data preparation, construction of data bases.
handling missing data, computer usage for data analysis, statistical analysis,
qualitative data analysis strategies, display of data, presentation of well-sup-
ported findings, conclusions, and recommendations, communicative results
and follow-up
II Knowledge and skills needed for evaluation but borrowed from other areas
A Administration/business
1 Project management making effective use of resources, organizing and con-
ducting of meetings, developing and administering budgets, managing person-
nel, delegating work, supervising staff, reviewing work products, supervising
and evaluating staff, promoting teamwork, observing equal opportunity prin-
ciples
2 Strategic planning
3 Auditing and evaluation
4 Evaluation and program development
B Communication/psychology
1 Oral communication communicating with staff, external agencies, gener.il
public, and the press, obtaining needed information skillfully, avoiding mis-
understanding, projecting a positive image, using media appropnaiely in pie-
sentation. leading discussions, conducting productive meetings, handling
hostility and controversy, seeking and respecting others viewpoints
2 Written communication writing status reports, one-page faitual suinmants.
executive summaries, proposals, reports, briefing papers, mrmos. case studies.
interview notes, testimony, data collection, instrument pciloimaiiLC
appraisals, speeches, and professional articles, using umipmer software to
-------
22 THE PREPARATION OF PROFESSIONAL EVAI.UATORS
Exhibit 2 1. (continued)
produce appropriate lexi and graphics, establishing feedback loops 10 avoid
surprises and allow people to respond to drafts, providing constructive feed-
back on written products
3 People skills getting along with people, logically explaining expectations,
using sound judgment as to what should be said/written, counseling employ-
ees in need of remediation, resolving sensitive personnel problems, rewarding
good performance, providing timely feedback
4 Negotiation negotiating contract, evaluation questions, separating people
from ihe pioblem, dealing with issues and values, focusing on many interests
thai are represented, inventing options for mutual gam (Barnngion, 1989)
5 Personal qualities credible, good judgment, flexible, sense of humor, continu-
ally learning, self-reflexive, cunous about how things work, ability to show
respect for the efforts of others
C Philosophy
I Ethics
2 Valuing determining the value of an object, applying criteria to information
about an object to arrive at a defensible value statement
D Political science
I Policy analysis
2 Legislation and evaluation the place of evaluation in current legislation
E Anthropology cross-cultural skills
F Economics (T Hednck, personal communication, July 9, 1993)
1 Cost-benefit and cost-effectiveness analysis, supply/demand theory, discount-
ing, wage rate analysis
2 Controlling for economic factors, for example, changes in the unemployment
rale
III Knowledge and skills unique to specific disciplines
A Education educational objectives, instructional design, instructional product
evaluation, teacher evaluation, populations with special needs, accreditation,
alternative assessment strategies
B Psychology human development, social service programs, clinical models, goal
attainment scaling, outcome evaluation of psychotherapeutic interventions, psy-
chological measurement, work environment (motivation, job satisfaction, produc-
tivity) (Corday, Boruch. Howard, and Boozin, 1986)
C Health cpidemiological studies
D Business task analysis, job analysis, management, organizational change, market
research, organizational design and development, information systems, conflict
resolution (Perloff and Rich 1986)
E Government policies, procedures, regulations, and legislation that apply to the
work aica (U S General Accounting Office, 1991)
F Public administration distinctions between the public and private sectors (for
example, many "bosses" in the public sector, legislative, judicial, executive, pub-
lit., special micicsl groups), no clear bottom line as with piofil in the private sec-
tor, accountability 10 the public (J Fiizpatnck. personal communication. July 8.
1993)
TRAINING EVAI.UATORS UNiqut SKILLS AND KNOWLLUGE 23
and Nielsen (1990) for feminists, Madison (1992), Mann and Mann (1991),
and Slanfield and Dennis (1993) for minorities, and Menens and Mclaughlin
(in press) for persons with disabilities Evaluation courses should include these
diverse perspectives and should be integrated into the process of planning and
implementing an evaluation on the understanding that an inquirer's philo-
sophical assumptions and theoretical orientation influence every stage of the
design process
Theories and Models of Evaluation. Numerous methods for the orga-
nization of the many theories and models of evaluation have emerged
Shadish, Cook, and Levuon (1991) explore the knowledge base that has
emerged regarding evaluation theories Theories encompass the choice of eval-
uation method, philosophy of science, public policy, and value orientation
These authors have identified three stages of evaluation theones theories thai
use a rigorous, scientific method and emphasize the search for "truth", theo-
ries that emphasize the need for detailed knowledge about how organizations
in the public sector work to increase the political and social usefulness of
results; and theones that integrate alternatives generated in the first two
stages Cuba and Lincoln (1989) provide a contrasting framework for theo-
nes and models of evaluation that includes four "generations" measurement
(testing), descnplion (objectives), judgment, and the responsive, construc-
livisl theory of evaluation As 1 mentioned in the preceding sect ion, emerging
theones associated with the emancipatory paradigm provide ferule ground
for an exploration of the meaning of alternative perspectives and then
methodological implications
Planning and Conducting an Evaluation. Although the process of plan-
ning and conducting an evaluation vanes with the theoretical framework, the
student of evaluation should be knowledgeable about and able to apply the
following steps
1 Focusing the evaluation This step includes identifying ihe object of the eval-
uation, us purpose, its audiences, and the constraints and opportunities
Identification and involvement of stakeholders have been tied to the
increasing utilization of evaluation results, and they have also been a source
of controversy in the evaluation field Harding (1993) and Madison (1992)
assert that the stakeholders involved should be those with the least power
and that team-building and collaboration strategies should be devised to
include clients in a meaningful way T Hednck (personal communication.
July 9, 1993) believes that team building is not appropnate in such sellings
as federal oversight evaluations J Greene (personal communication. July
7, 1993) believes that what is distinctive about evaluation is the way in
which politics intertwines with public program and policy decisions and
the distinctive, contested audiences of an evaluation Students should
explore who should be involved in an evaluation, whose purposes an eval-
uation should serve, and how best they can be appropmidy involved
-------
24 Ti IE PRF.PARA riON OF PROPCSSIONAL I-VALUA i ORS
2 Designing ihe evaluation and formulating questions The choice of a theo-
retical framework and evaluation model discussed previously guides the
evaluator here
3 Planning data collection The evaluator needs to identify the information
needs, sources of information, instruments (including ways to describe the
program treatment and implementation), and ways to identify the theory
of the program being evaluated (that is. the context and presuppositions of
the organizations and groups involved)
4 Analyzing and interpreting daia The evaluator needs to identify appropri-
ate analytical approaches for the type of data collected Identifying them
will provide a mechanism for accurate and meaningful interpretation
5 Planning, reporting, and utilization The evaluator needs 10 facilitate effec-
tive communication and integrate utilization strategies throughout the eval-
uation process
6 Planning management The evaluaior needs to determine the resources
required and the time line of activities
7 Planning meta-evaluation The evaluaior needs lo know how to evaluate
the quality of the evaluation plan, process, and product
Students should be given the opportunity to implement their evaluation
plans through small evaluation projects completed as a class project or as a pan
of an internship Several authors have provided helpful hints concerning the
inclusion of practical experiences in training programs (Eastmond, Saunders,
and Merrell, 1989, Morris, 1989. Preskill. 1992)
Socialization into the Profession Students of evaluation should be
given the opportunity to become socialized into the profession by means of
involvement with professional organizations, networking with evaluaiors, and
interacting with the evaluation literature
Special Topics in Evaluation. The following topics are very important
in evaluation and should be included in the preparation of evaluators
1 Ethics professional behavior, use of information, confidentiality, sensitivity
to effect on others, pressure from client to distort facts, proper response to
discovering information that is morally or legally volatile, and so on (Mor-
ns and Cohn. 1992)
2 Standards for evaluation of programs and personnel (Joint Committee on
Standards for Educational Evaluation. 1981. 1988)
3 Politics of evaluation knowing the players, the policy environment, the
power of communication, how to get people to come to an agreement, and
how people are likely to use information (Barnnglon, 1989), knowing how
organizations work, how to understand an organization's goals and inter-
nal and external forces (that is, how to analyze us political context) (H
Preskill, personal communication, June 23, 1993)
TRAINING EvALUATORi UNIQUE SKII i.s ANO KNOWI EDC.I. 25
4 Specific methods and contexts, such as needs assessment (McKillip, 1987),
evaluabilily assessment, fulunng (the field of future studies) (Patlon. 1990).
and international evaluations
5 Evaluaior as trainer training evaluation clients and users
The section in Exhibit 2 1 on skills and knowledge borrowed from related
areas includes olher topics, such as policy analysis, communication skills, and
cost analysis, (hat are essential lo the preparation of evaluaiors
Summary
The training of evaluaiors should reflect ihe evolving, dynamic nature of the
field of evaluation. Many core topics identified here are reflected in evaluation
lextbooks. Evaluation also borrows skills and knowledge from olher disci-
plines, but a training program for evaluaiors should examine them specifically
through an evaluation lens. The inclusion of emerging topics in an evaluation
training program can sensitize students of evaluation 10 these issues and make
them better able to serve the people whom their evaluations affect The train-
ing of evaluators should prepare them to reflect on and engage in dialogue
about the besl ways of responding lo society's diverse demands The field of
evaluation needs to think in terms of multiple, not singular, pcispcciives when
il trains evaluaiors
References
Altschuld. J W "Structuring Programs to Prepare Professional Evaluaiors " Paper presented al
the annual meeting or the American Evaluation Association, Seattle. Wash . 1992
Bamngion, G V "Evaluator Skills Nobody Taught Me, or What's a Nice Girl Like You Doing in
a Place Uke This7" Paper presented at the annual meeting or the American Evaluation Associ-
ation, San Francisco, 1989
BeaudryJ S 'Synthesizing Research in Multicultural Teacher Education Findings and Issues
Tor Evaluation of Cultural Diversity " In A Madison (ed ), Minority Issues in Program Evalua-
tion New Directions for Program Evaluation, no 53 San Francisco Jossey-bass, 1992
Bnnkerhoff R O , Brethower, D M , Hluchyj, T , and Nowakowski. J R Program t'vuluulion
Boston Kluwer Academic Press. 1983
Cordray, D , Boruch, R , Howard. K , and Bootzm. R "Teaching of Evaluation in Psychology
Northwestern University " In B G Davis (ed ). The frothing oj Evaluation Across the Disciplines
New Directions for Program Evaluation, no 29 San Francisco Jossey-Bass. 1986
Covert, R W "Successful Competencies in Preparing Professional Evaluaiors " Paper presented
at the annual meeting of the American Evaluation Association, Seattle, Wash , 1992
Davis. B G "Overview of the Teaching of Evaluation Across the Disciplines "In B C Davis (ed).
The Teaching oj Evaluation Across ihe Disciplines New Directions for Program Evaluation, no
29 San Francisco Jossey-Bass. 1986
Eastmond. J N , Jr "Addressing Ethical Issues When Teaching Evaluation " Paper presented at
the annual meeting of the American Evaluation Association. Chicago, 1991
Eastmond, J N Jr "Structuring a Program lo Prepare Professional Evaluating " Paper presented
at the annual meeting of the American Evaluation Association. Seattle. Wash . 1992
-------
26
I III PRtPARATION OH PROI>SSIONAI. EVALUATOR-S
Easunond. J N , Jr . Saunders, W , and Merrell. D "leaching Evaluation Through Paid Con-
iraLlual Arrangements " Evaluation Practice. 1989, 10 (2). 58-62
Eisner. L W The Lnlightened Eye New York Macmillan. 1991
Farley, J . and Meriens. D M "The Feminist Voice in Evaluation Methodology " Paper presented
at the annual meeting of the American Evaluation Association, Dallas, 1993
Cuba, I: , .ind Lincoln. Y S Fouilh-Gfneralion Evaluation Ncwbury Park. Calif Sage, 1989
I larding. S "Rethinking Standpoint Epistemology 'What Is Strong Objectivity?1" In L Alcoffand
E Potter (cds ), /Vminist Episternology New York Koulledge, 1993
Joint Committee on Standards for Educational Evaluation Standards Jui Evaluationsof Educational
Programs. Projects, and Materials New York McGraw-Hill, 1981
joint Committee on Standards for Educational Evaluation The Peisonnel Evaluation Standards
Newbury Park. Calif Sage. 1988
Lather. P Celling Smart Feminist Research and Pedagogy with/in the Postmodern New York Rout-
ledge. 1991
Lather. P "Critical Frames in educational Research Feminist and Poststructural Perspectives "
Theory into Practice. 1992.31 (2). 1-12
Lincoln. Y S , and Cuba, E G "In Response to Lee Sechresi's 1991 AEA Presidential Address
•Roots Back to Our First Generations.' Feb 1991. 1-7 " Evaluation Practice. 1992, 13 (3).
165-170
Lopez. S D . and Menens. D M "Current Practices Integrating ihe Feminist Perspective in Edu-
cational Research Classes " Presentation at the annual meeting of the American Educational
Research Association. Atlanta, Ga , 1993
McKilhp.J Nerd Analysis Newbury Park. Calif Sage. 1987
Madison. A M (ed ) Minority Issues in Program Evaluation New Directions in Program Evalua-
tion, no 53 San Francisco Jossey-Bass, 1992
Mann, G . aYid Mann. B V Reseanh with Hispanic Populations Newbury Park, Calif Sage, 1991
Meriens, II M "Structuring a Program to Prepare Professional Evalualors Whal Aren l We Talk-
ing About (Ihai We Should Be)'" Paper presented al the annual meeting of the American Eval-
uation Association. Seattle. Wash . 1992
Menens. D M , and McLaughlm. J Reseanh Methods in Spenal Education Newbury Park. Calif
Sage,in press
Morns. M "Field Experiences in Evaluation Courses " In D M Meriens (ed ), Creative Ideas for
leaching Evaluation Norwell, Mass Kluwer, 1989
Morns. M , and Cohn, R "Program Evalualors and Ethical Challenges A National Survey" Paper
presented at the annual meeting of the Amencan Evaluation Association, Seattle. Wash . 1992
Nielsen. J M (ed ) Feminist Research Methods Boulder. Colo Westview Press, 1990
Ration. M Q "The Challenge of Being a Profession " Evaluation Piaclice. 1990. 11(1), 45-51
Perloff, R . and Rich, R F " The Teaching of Evaluation in Schools of Management" In B G Davis
(ed ), The Teaching of Evaluation Across the Disciplines New Directions for Program Evaluation.
no 29 San Francisco Jossey-Bass. 1986
Popham.WJ Educational Evaluation Englewood Cliffs, N J Prentice Hall. 1988
Posavac. E J . and Carey. R J Program Evaluation Method and Case Studies (4lh ed ) Englewood
Cliffs, NJ Prentice Hall. 1992
Preskill. II "Students, Client, and Teacher Observations from a Praclicum in Evaluation " Eval-
uation Pnulice. 1992. 13 (I). 39-46
Rossi. P . and Freeman, II E evaluation A Systematic Approach (5lh ed ) Newbury Park, Calif
Sage. 1993
Seihresi.l "Roots Back 10 our Firsi Generations " Fvuluulion Piaitue. 1992 1.1(1). 1-8
Shadish. W R.Jr. Cook. 1 1). and Leviton, L C Foundations of Program Evaluation fheones
ofPraititf Newbury Park, Calif Sage, 1991
Shapno.j I* "Cull.ibor.uivr Evaluation lowaidd rr.insform.mon of I valuation lor Feminist Pro-
grams and Projects " Paper presented al the annual meeting of the American Educational
Rcse:iich AiMKialion. Washington, 1} C , 1987
TRAINING EVALUATORS UNIQUE SKILLS AND KNOWI m.i 2 7
Stanfield.J H "Methodological Reflections " In J H Sianheld II and R M Dennis (eds ). Kace
and Ethnicity in Research Methods Newbury Park, Calif Sage. 1993
Sianfield. J H , II. and Dennis. R M (eds ) Race and Ethnicity in Reseanh Methods Newbury
Park. Calif Sage. 1993
U S General Accounting Office Performance Appraisal System o) Band I. II. and III Employees
Washington. DC US General Accounting Office. 1991
Wilcox. S "STUCK in School Meaning and Culture in a Deaf Education Classroom " In S
Wilcox (ed ), Amencan Deaf Culture Burtonsville. Md Linstock Press. 1989
Worlhen, B R , and Sanders. J R Educational Evaluation New York Longman. 1987
DONNA M MERTENS is professoi m the Depaitment of Educational I'ounilatumi. and
Research at Gallaudet University in Washington, D C
-------
In 1988, the US. General Accounting Office started an ongoing,
comprehensive evaluation (raining program for its staff. This
chapter sketches the program and describes the major substantive
areas of its curriculum
Evaluator Training in a Government
Setting
Nancy Kingsbury, Jerry E. Hednck
The U S General Accounting Office (GAO) is a nonpamsan agency in the leg-
islative branch of the federal governmem Its statutory mission, established by
the Budget and Accounting Act of 1921, is (among other things) to "investi-
gate, at the seat of government or elsewhere, all matters relating to the receipt,
disbursement, and application of public funds" (Budget and Accounting Act,
1921) Over the years, this responsibility has evolved from detailed audits of
individual agency purchases into economy and efficiency reviews of govern-
ment programs and more recently to a wide-ranging array of program evalua-
tions and policy analyses With very few exceptions, any program or activity
funded with federal tax dollars can be the subject of a GAO review And as
Congress grapples with the difficult decisions of the 1990s, the issues that the
GAO is asked to evaluate mirror the breadth and complexity of the day's head-
lines A recent sample of study requests includes these questions What strate-
gies have been most effective m reaching hard-la-serve recipients of welfare
programs7 What factors drive health care costs' How feasible arc various
approaches to the development of geothermal energy7 Are federal plans for the
elimination of tuberculosis from the United States achievable7 What are the
causes and effects of the European currency crisis7 What is the best strategy
for response by the federal government to natural disasters? Can nonnuclear
designs for aircraft carriers and submarines meet the Navy's need for future
missions7 What interventions are necessary to end the underrepiescnution o(
women and minorities in federal agencies7
In September 1993, the GAO had about 4.900 staff Three-quarters were
engaged in evaluation and auditing work Evaluaiors are organized into (hiriy-
six issue areas that correspond roughly to government progiar riil.li.lwit 61
-------
62 Tin.
PROI 1-bSIONAI. EVALUA10KS
such as healtli policy, employment and (raining, cnvironmenial issues, lax policy
and administration, administration of justice, management of defense and
space programs, international trade, and federal management issues
Because a substantial part of the GAO's evaluation work is earned out on
site where government programs operate, evaluation staff are located in Wash-
ington. D C , in fourteen regional offices around the U S continent, and in two
overseas offices (one in Hawaii, the other in Germany) Like many federal
agencies, the GAO expects to reduce us size over the next few years, although
(here is little likelihood that the work thai it will be asked lo do will decrease
Accordingly, ihe GAO is investing significantly in technology improvements
(computer networks, videoconferencing) that can improve its productivity and
in improving us work processes ihrough total quality management (TQM)
Evaluation and Evalualors at the GAO
The GAO defines evaluation broadly The term is used to describe a range of
activities Depending on the context, it can be synonymous with audit, review,
or policy analysis Methodologies and techniques from a variety of disciplines
are often brought to bear on an assignment, and, because of the agency's his-
tory in the accounting tradition, the results of the work must meet traditional
government auditing and accounting siandards as well as the standards of
other professional disciplines
Most staff responsible foi carrying oui reviews of governmem programs
are called cvalualois, and they arc expected to demonstrate strong genetic skills
in project planning, data gathering and analysis, written and oral communi-
cation, and interpersonal communications and management areas However,
(hey also need specialized expertise
The GAO recruits staff from an an ay of the professional disciplines found
in colleges and universities throughout the country In the 1930s and 1960s,
the GAO iccruited almost exclusively from the accounting profession In recent
years, it has greatly diversified its hiring practices, hiring master's degree and
doctoral level graduates in economics, ihe social sciences, public policy, pub-
lic administration, and business administration It continues to recruit accoun-
tants but generally at ihe bachelor's level Increasingly, GAO staff are
maintaining their professional identities (for example, as an economist) after
entcnng the agency Nevertheless, the requirements of the work necessitate thai
members of each discipline become familiar with the terminology and meth-
ods of othci disciplines
Although disciplinary diveisity is clearly an impoitant pan ol the GAO's
institutional capability, u is equally important that there be a common set of
coic values and woikmg procedures thai ovcilny the vjnety of disciplinary pci-
spet lives It is essential to have a common understanding of the GAO's mis-
sion, of what is meant by the expression "quality woik," of the GAO's
expectations for the w.iy in which woik will be carried out, of the way in
EVALUATOR TRAINING IN A GOVERNMENT SETIING 63
which ihe executive and legislauve branches operate and mteisect, and of the
way in which the GAO will ultimately meet the mformalion needs of Congress
and provide value to the taxpayer Meeting all these needs while developing spe-
cific technical and computer skills poses a large challenge for the GAO's tram-
mg programs Training serves as a vehicle for establishing values, leaching
agency procedures, and understanding the broad coniexi of evaluaiion work
Training at the GAO
In 1988, the GAO established us own Training Institute Training and educa-
tion responsibilities were consolidated and separated from career counseling.
personnel, and orgamzauonal developmeni suppon The intent was to high-
light the importance that the agency places on training and professional devel-
opment Investing in staff development was deemed critical to meeting ihe
mformalion needs of Congress.
In comparison with other federal agencies, ihe GAO makes a relatively
large investment in training opportunities for us siaff The physical plam
includes iwo major training ceniers in Washmgion. D C , lhai have a lotal of
seventeen classrooms Each regional office also has space and equipmeni for
training activities Completion of a nationwide videoconferencing capabiluy
this year will permu training lo be provided simuhaneously in headquarters
and regional offices
GAO evaluators averaged seventy-four hours of continuing education in
1992. Two-thirds of that irammg was delivered by the Training Institute The
institute has a rosier of 210 aciive courses and offers more lhan a ihousand
classes a year (Training Institute . ,1991) Evalualors have also been given
significant resources, boih centrally and wuhm organizational units, lo partic-
ipate in professional development activities outside ihe GAO The agency's on-
gm in accounung conmbuies directly lo ihis emphasis on irammg by imposing
a continuing education requirement on all evaluaiors To conunue lo be
deemed qualified to do ihe GAO's audit and evaluation work, every evaluator
musi oblam a minimum eighly hours of irammg every iwo years
Teaching Evaluation in a Work Setting
The environment of the GAO makes demands and imposes constramis on ihe
design of training programs that are quiie different from the forces thai opcr-
aie in academic sellings Our siudems are adults langmg from recent gradu-
ates of graduate-level programs to experienced evaluators who have bioad
evaluation and management experiences and mature auditors ncarmg rcine-
ment Almosi all ihese students work full-time and expect naming to be
directly relevam to what they will do on the job the very next week
Work schedules and geographic dispersion require ihai training l>e deliv-
ered in intensive segments A typical Training Institute course consists of iwo
-------
64 Tllfi I'RFPARATION OF PROFESSIONAL EVAIAIATORS
lo four successive eight-hour days of training This pattern permits a short
period of full-time training (about all that is manageable given the press of
ongoing work), and it gives regional participants reasonable (ravel time This
concentrated, full-time training schedule and the nature and expectations of
the Training Institute's students—evaluator staff—heavily influence training
methods Most courses are a mixture of lecture, case studies, and opportu-
nities for practical application of skills thiough role playing or demonstra-
tion, and most courses make extensive use of materials taken directly from
GAO work When possible, we give training participants oppoitunnies to
use material from their current assignments in class exercises (for example,
by using real data from an ongoing assignment when they practice writing
testimony)
In part as a means of focusing training on skills and activities directly
related to the work, instructors are heavily drawn from line staff and managers
Many of our senior executives regularly act as course instructors This pattern
of training delivery requires development of course frameworks and instruc-
tional materials that are easy for multiple instructors to use We also train our
instructors to leach effectively We are more likely to use external than inter-
nal instructors for courses in such things as statistics, writing, computer soft-
ware, and generic management topics
Major Areas of Emphasis
Overall, the GAO's formal training progiam for cvaluators has six areas of
emphasis agency mission and policies, assignment planning and execution,
communication skills and strategies, computers and information technology,
workplace relations and management, and issue area expertise With the
exception of issue area training, (he courses in each area have been designated
as required, core, or elective and determined to be appropnate for staff, senior
staff, management, and/or executive levels
All evaluators must lake the required courses, which contain information
that the agency believes is necessary for all peisons regardless of prior educa-
tion or work experience Core courses contain material with which all evalu-
alois should be familiar, but the agency recognizes that individuals may excuse
themselves from specific courses if they have previously mastered the mater-
ial Elective courses can be selected to fill specific needs, depending on the type
of work m which the individual is currently engaged Courses at the staff and
senior staff levels are concentraied in technical areas Courses at the manage-
ment and executive levels emphasize management The training provided for
the upper levels often relies on external opportunities for continuing educa-
tion, such as piofcssional conferences Cvaluators move thiough a structured
set of couiscs as they progress in their careers
The GAO's curriculum structure was developed in collaboration with an
advisory c-~ -nmee of managers diawn from the agency's divisions and offices
EVALUATOR TRAINING IN A GOVFKNMI-NI Sen INC, 65
Specific courses have been developed over the past four years, and only in the
past year can the GAO be said to have fully implemented us evaluatoi cur-
riculum The six sections that follow describe each of the major substantive
areas.
Mission, Policies, and Individual Responsibilities. All new staff mem-
bers are required to attend an initial orientation course that describes the GAO's
history, its mission, its role in supporting congressional decision making, ethics
guidelines, and the policies and procedures for the conduct of evaluation assign-
ments Subsequent required courses that evaluators take in the next few years
elaborate on standards for work, internal control issues concerning the quality
of acceptable evidence, and processes for ensuring accuracy in the agency's
reports
As staff move up the career ladder after each promotion, they are invited
to attend so-called promotion programs that lay out the organization's expec-
tations for their new roles The discussions in these programs are structured
both around people issues—interpersonal communication, supervision, per-
formance feedback—and around planning and reporting issues—for example,
what it means lo have responsibility for directing an evaluation or managing
the work of multiple teams of evaluaiors Ai mid levels, these programs can
include special topics, such as information on a manager's equal employment
opportunity responsibilities
Assignment Planning and Execution. Each audit or evaluation at the
GAO is referred to as an assignment, and the skills necessary lo design and
manage an assignment are a crucial part of the GAO's internal training pro-
gram for newly hired evaluaiors The goal here is twofold 10 create an aware-
ness of other professional disciplines and to build specific skills
The overall intent of ihis pan of the curriculum is lo fosier an awareness
of the wide range of work that the agency docs and of the need to apply appro-
pnate methodologies when the work is done All entering staff are required to
attend a workshop on the selection of an approach and methodology The
workshop provides guidance on how to lake an area of congressional concern
and develop focused questions lhat can be answered within the constraints
imposed by resources and lime Workshop participants then analyze these
questions lo determine how they can mosi appropriately be answered Stalf
then lake core methods courses on compliance auditing, economy and effi-
ciency reviews, program evaluation, and policy analysis Follow-on courses on
such lopics as procurement and contract processes, financial management.
budgeting processes, fraud awareness, and special issues in economics aie
available. The goals are for all individuals—staff and manageis—in become
comfortable with a variety of types of work and lo be able to work rfleuivcly
in a multidisciplmary environment
To meet ihe skill-buildmggoal. the institute offers couiscs on such topiis
as sampling, questionnaire design and structured interviewing, jpphed siatii-
lics (basic classes and elective classes on advanced topics, sir log-linear
-------
66 THE (-REPARATION or PROFESSIONAL EVALUATORS
modeling and lime senes analysis), and qualitative methods At entry, staff are
provided with self-paced training materials on organizing their documentation
(work papers) for the evidence and analysis thai support an audit or evalua-
tion
Communication. Although a GAO evaluation team may conduct an
excellent study, ihe value of the study will be weakened significantly if it is not
communicated effectively in published reports and oral bnefmgs For this rea-
son, the GAO's curriculum gives evaluators a series of courses reflecting the
latest research on written and oral communication and on cognitive psychol-
ogy For example, instructors may discuss readability principles and factors
that increase the retention of information read
Writing courses at the entry level clanfy the GAO's basic communications
policy and differentiate between academic writing and workplace writing The
focus is on producing an institutional—not an individual—product In class,
evalualors work on skills that they need in order to wnle GAO documents—
for example, analyzing the writing situation, wnting collaboratively, recogniz-
ing the difference between wnier-based and reader-based documents, assessing
the readability of their own documents, and using review comments to
improve documents
Class exercises for senior staff show how writing and thinking are inextri-
cably linked and how the structure of a written report can affect us interpre-
tation Training participants also practice constructing a succinct message out
of masses of data Using data from a case study, evalualors develop report
issues, prepare for a message conference (a meeting m which all evaluation
team members, advisers, and managers discuss the evaluation results and agree
on their interpretation), and conduct a simulated message conference Mes-
sage conferences are stressed because they improve the quality and timeliness
of the documents produced and reduce unnecessary rework A course for man-
agers called Managing Writing reviews the writing principles embodied in the
curnculum, suggests strategies for managing the writing process, and presents
ideas about the role of oral and written communication in public policy
processes
The wnting curnculum also includes specialized courses thai help staff to
write specific kinds of documents, such as an executive summary or written
testimony for oral presentation These two kinds of writing are emphasized
because both are highly visible statements of the GAO's work Both kinds of
wntten summaries receive close scrutiny, and many readers may never read the
full evaluation report During the testimony course, evaluators develop testi-
mony by following guidelines for effective congressional presentations They
practice testimony-wnimg skills and receive constructive feedback The course
also includes discussions with executives who excel in the delivery of written
testimony
Oral communication is equally important, because much of the agency's
work is conveyed through bnefmgs and testimony Training on oral presenta-
tion skills begins during the first three months after a new evaluator starts
EVALUATOR TRAINING IN A GOVI.RNMENT SETTING 67
work, and it seeks to improve his or her interviewing and briefing skills The
follow-on course is dedicated to honing presentation skills, it makes use of
videotaping and feedback Electives are available ai a more advanced level 10
improve presentation skills and learn how to conduct meetings effectively At
ihe most advanced levels, managers and executives can lake hands-on courses
involving practice in communicating effectively with ihe media and delivering
oral testimony to Congress Figure 6 1 shows how the conient of ihe commu-
nications courses vanes by position
Computer Use. As a large organization, ihe GAO uses several software
packages lhat must be supported with technical assistance and training Several
years of expenence have taughi us thai users prefer thai course maienal deliv-
ered in the classroom be very bnef. Maienal is often delivered in modules—for
example, WordPerfect sort features. WordPerfeci lexi columns, Loius 1-2-3 daia
lables. Users can enroll in ihe course most suiied lo iheir immediate needs
Forty courses are available (Training Institute , 1991) They relate to word
processing applications, spreadsheets, and database management systems Addi-
tional training is available on microcomputers, data analysis packages, such as
SAS and SPSS; computer communications; and support for local area networks
As ihe agency is moving to design and implement software applications for
organizing, sharing, and accessing working papers and databases, the institute
is designing training to support their use. Steps are also being taken lo revise
existing training in ways thai recognize how these apphcauons and the use of
local area networks can change the ways in which work gets done.
Figure 6.1. Communications Courses in the GAO's Evaluator Curricu-
Eiacunva
Producing
OganlMd
WMnoml
Wnuttop
Evwy ftiM
EiaCUM
Sufimwiy
Wortuliop
Tummy
rsa?
EiaoilM
Di.tvo.hg
lammony
Note Introductory Evaluaior 1 wining, the iwo-week orientation, includes module;, on wining and
oral bntfmg skills
-------
68
Tnr PREPARAIION or PROCESSIONAL EVAI.UAIORS
Workplace Relations and Management. Training in the area of work-
place relations and management contains material that traditionally has been
classified as both soft and hard skills As one might expect, the institute offers
classes on time management, relations with Congress, and management of
one's issue area, that is, a body of work in an area like transportation or energy
These kinds of courses focus on planning and coordination processes Super-
vision and performance management seminars are also available
Courses on interpersonal relations in the workplace, mediation, diversity
in the workplace, and advanced communications and negotiations are newer
additions to the GAO's curriculum. As the GAO has become increasingly
involved with total quality management (TQM), the emphasis on interpersonal
communication, teamwork, and management skills has increased Recent hires
have an opportunity to enroll in such courses as Workplace Relations and
Communications Mid-level managers take Managing Quality Improvement,
and top-level executives learn about the role and responsibilities of quality
councils Five-day courses prepare leaders of problem-solving teams to use
appropriate tools and techniques, leach fellow team members, and be aware
of how to foster positive group dynamics Additional training is expected to be
added in this area as the GAO advances in us implementation of TQM
This area also contains courses to build skills and heighten awareness and
knowledge about key supervisory and management responsibilities All supervi-
sory staff recently participated in workshops on ways of preventing sexual harass-
ment in the work environment, and a similar course is now available to
nonsupcrvisory staff All supervisors and managers have already received train-
ing on their equal employment opportunity (EEO) responsibilities To maintain
this awareness, the GAO auiomaucally enrolls newly promoted staff in the EEO
workshop And in recognition of the increasing diversity of its work force and of
the need to have a work environment that makes all staff feel welcome and val-
ued, the institute has started to provide workshops on the valuing of diversity
Issue Area Training. As noted earlier, the GAO has thirty-six issue areas
covenng work in areas as wide-ranging as national security policy and national
resource management Generally, the Training Institute has neither the exper-
tise nor the resources needed to develop issue area subject mailer training, so
in most cases issue area groups pursue their own strategies to develop and
maintain staff proficiency These strategies can include inviting subject matter
experts to give informal talks, using consultants on specific projects, and hold-
ing planning conferences with invited participants from government agencies
and academic and other relevant groups, including businesses, professional
associations, and think tanks
However, major training initiatives have been supported internally in two
key issue aieas In the financial management aiea, the institute worked closely
with the GAO's accounting and financial management expeits to develop a
financial auditing ami<_ulum And in the information management technol-
ogy aiea, •' sniute has offered courses on such topics as computer security,
EVALUATOR TRAINING IN A GOVCRNMEN r Sc RING 69
telecommunications, and systems development Several masier's-level courses
leading to a certificate in information systems from the George Washington
University have been offered at the GAO's training center on a regular basis at
the end of the workday
Self-Paced Training
Besides classroom courses, the institute provides a variety of self-paced courses
in interactive multimedia, audio, video, and print formats Many of these
course offerings are related to computer software packages, but others cover
such topics as management and supervision, human resource management,
writing, and administrative support activities In most cases, courses can be
mailed to the work site and used on the individual's own compuier When spe-
cial equipment is needed or licensing restrictions make widespread distribu-
tion impractical, individuals can sign up to take such courses in ihe msuiuie's
learning center Acceptance of this type of course delivery has grown Self-
paced hours increased by iwo-thirds in the past year More than 1,300 evalu-
ators enrolled in self-paced courses in fiscal year 1993. and there have been
about 800 course completions to date
Lessons Learned
Developing and delivering ihe GAO's evaluator curriculum has been a contin-
uous learning process for everyone involved Several valuable lessons have
been learned that may be useful to others who have a responsibility for nam-
ing functions in similar contexts
Firstfjob relevance isrrul^Drnnro. developers and msiructois need to
be able to assess training needs accurately, design effective learning experi-
ences, and demonstrate to training participants that the training material is
directly relevant to their work Training is not an end in uself. u mus,i be
designed to support the effectiveness of the individual and the organization
One technique for enhancing the relevance of training is to design courses that
use real case studies or have the students bring ongoing work to class and
apply the material to it This means that instructors have to be adaptable and
quite proficient in their area of expertise.
Second, th^tnvolvement of line managers and stottjncrrw* the credibil-
ity and quality of training The GAO's curriculum was developed under the
guidance of a management advisory committee, and all decisions regarding
mandated, core, and elective courses were made by this committee Housing
decision-making authority in the "line" makes ownership of the curriculum
greater than if the training department were to make all ihe decisions Execu-
tives, managers, and senior staff also often serve as mstruciors or preseniers on
panels or contribute course material, thereby endorsing the value of training
and building commitment
-------
70 THE PREPARATION OF PROI-ESSIONAL EVALUAIORS
Third, training needs lo dclivefcpnsistcnt messages at all levels^The
GAO's curriculum structure was intended to be an integrated one, with simi-
lar concepts and skills in courses at staff, senior staff, and management levels
We are still completing this process, and we hope by next year to have paral-
lel courses running at all levels, with the upper-level courses incorporating a
managerial perspective This consistency will ensure that managers and staff
are familiar with the same terminology, methodologies, and guidance We also
plan to increase the amounl of training that we deliver lo intact work groups
and reduce the number of open enrollment classes We believe that training
the members of work groups together makes it more likely thai the concepts
laughl in class will be reinforced and used on the job Work unit-based train-
ing is also expected lo have a direci effeci on improving teamwork and inira-
unu communication—important goals in an organization focused on quality
We plan to lesi ihese assumptions by conducting follow-up evaluations for
selected courses
Finally, in the spinl of quality^management, we strive l^fassess^he effec-
uveness of our training elTorts^oTuinuousTji and we revisit our delivery strate-
gies The challenges lhai ihe GAO laces Change consianily, the needs of our
work force change, and lechnological advances creaie new training demands
References
Budget and Accounting Act of 1921 (PL 67-13. 42 Stal 20, codified asamended al 31 U SC
712)
Training Institute. U S General Accounting Office Training and Education 1992-1993 Catalog
Washington. D C Training Institute. U S General Accounting Office. 1991
NANCY KINGSBURY is dneclor for federal human resource management issues in the
General Government Division of the V S General Accounting Office
TLRRY E HLDRICK is direcloi of the Training Institute al the U S General Account-
ing OJftic
The procedures used to collect information for the Directory of
Evaluation Training Programs are described, and the results of the
survefcflre discussed. Tables list the programs identified in the
United States, Canada, and Australia. '
The 1994 Directory of Evaluation
Training Progral
James W. Allschuld, Molly &wle, CarA Cullen,
Inyoung Kim, Barbara Rae Ma$ce /
The American Evalualion Association ^A) has periodically published a direc-
lory of evaluation training program? m fhe United Slates and Canada (May
Fleischer. Schreier. and Cox, 1986. ConnerMZIay. and Hill, 1980. Gephan and
Potter. 1976) May. Fleischer. Schreier. andVox (1986) listed fony-six pro-
grams in six different lypes of sellings PrograrXs locaied in education and psy-
chology predominated In late J992, ihe boarclof ihe American Evalualion
Associaiion commissioned a sliyly lo provide a cUrrem lisung and description
of evaluauon iraming progranjs This chapler describes ihe siudy meihodol-
ogy. reports some of ihe stud^s findings, and labulaifcs ihe informauon aboul
evaluation training prograr
Methodology
Sample. Since 198|($. numerous changes have occurred in ihe field of
evaluation thai could ajjfeci ihe nature of training programs \mong them are
changes m ihe meihods lhai evalualors use and improved understandings of
the relationship belween evaluation and policy development As\a result, new
programs have emerged, others have changed or altered theirtonieni and
structure, and still/others have ceased to exist
The initial and perhaps most challenging task for the study team was to
develop a comprehensive sampling frame of potential candidaics We used the
following process to develop the sampling frame First, we placed an
announcement in ihe call for papers for ihe AEA's 1992 annual conference
The announcemem asked AEA members to nominate candidates for the study
Ntw Di.bLiKiNS ioi PKH.MH EVAIIMIKW no 61 Summri IWM O |«K) ibu r.ibl»hr,, 7 |
-------
Meetings and Events http /Avww evaJ org/Conferences meetings html
American Evaluation Association Annual
Meetings
AEA holds its annual conference during the first week of November each year.
Topics of current interest are discussed in sessions proposed by members, as well
as in sessions presented by invited speakers. In addition, a computer-assisted Job
Bank is provided at the annual conference. Professional awards for outstanding
contributions to the field of evaluation are presented in the areas of: service to the
profession, evaluation theory, evaluation practice, and service to AEA. Preceding
the conference, over 25 training sessions are offered. Past training session topics
include: increasing evaluation use, cost-benefit analysis, reporting and debriefing,
applying professional standards, statistical analysis software, focus groups,
qualitative evaluation, and secondary analysis of available data.
Future AEA Annual Conferences
Evaluation 2001
Dates: November 7-10 in St Louis, MO
Hotel: Millennium
Call for Proposals in Mail: By January 7, 2001
Proposals Due: March 16, 2001
Notifications of Proposal Status: By July 1, 2001
Registration Materials Available: By July 1, 2001
Evaluation 2002
Dates: November 6-9 in Washington, DC
Hotel: Hyatt-Crystal City
Evaluation 2003
Dates: November 5-8 in Reno, NV
Hotel: Nugget
Other Evaluation-Related Meetings or
Conferences
• List of Events
Archives from Past AEA Conferences
,of2 1/4/01 1040AM
-------
Links
http.//www eval org. ListsLinks-EvaluationLmks/links htir.
Links of Interest to Evaluators
Here are some sites that you may find useful. If you know of other sites that might
be of interest to our members and others involved in evaluation, please send your
suggestions to AEA manager Susan Kistler at aea@kistcon.com.
AEA Topical Interest Groups
TIG on Alcohol. Drug Abuse, and Mental Health
TIG on Assessment in Higher Education
TIG on Business & Industry
TIG on Collaborative. Participatory & Empowerment Evaluation
TIG on Extension Evaluation Education
TIG for Graduate Student Association
TIG on International and Cross Cultural Evaluation
TIG on Minority Interests in Evaluation
TIG on Program Theory and Theory- Driven Evaluation
TIG on Research Technology and Development Evaluation
TIG on Teaching of Evaluation
AEA Local Affiliates
Arizona Evaluation Network fazENet)
Eastern Evaluation Research Society
Ohio Evaluator's Group
Oregon Program Evaluators Network
Southeast Evaluation Association
Washington Evaluators
Western Pennsylvania Evaluator's Network fWPEN)
Other National Evaluation Associations
1 of 3
1/4/01 1037 A^
-------
Links
http '/www eval orS'ListsLinks-EvaluationLinks/hnks him
A'. 5:.'?.'3S!-:- -. Evaluation Soae;:.'
Canaci:?n Evaluation Society
European Evaluation Socier-/
French Evaluation Society
German Evaluation Society (German language)
Ghana Evaluators Association
Italian Evaluation Society
Malavsian Evaluation Society
Monitoring & Evaluation in Latin America and the Carribean
Nigerian Network of Monitoring & Evaluation
Sri Lankan Evaluation
Swiss Evaluation Society
UK Evaluation Society
Walloon Evaluation Society
Other Associations of Potential Interest to Evaluators
• American Society for Public Administration
General Evaluation Sites
The American Educational Research Association
Applied Survey Research
Centre for Program Evaluation. The University of Melbourne
CRE5ST Home Page (Center for Research on Evaluation. Standards, and
Student Testing)
Educatiional Research Methods (follow link then click on textbook)
ERIC Clearinghouse on Assessment and Evaluation
Evaluation Associates Ltd.
The Evaluation Clearinghouse
The Evaluators' Institute
Harvard Family Research Proiect
The Joint Committee on Standards for Educational Evaluation
Lesley College Program Evaluation and Research Group
Literature on Programs
Monitoring and Evaluation News. A news service focusing on developments in monitoring and
evaluation methods relevant to development projects with social development objectives
(UK).
On-line Evaluation Resource Library. A resource of project evaluation tools (plans and
instruments) and reports used by the National Science Foundation's Directorate for Education
and Human Resources; topic areas focus on curriculum development, teacher education, and
faculty development, including minority group representation (US). URL: http //oeri.sn com/
Performance Assessment Links in Science An online resource of performance assessments
for students studying science in grades K-12 provides information on standards, tasks and
rubrics for evaluative purposes (US)
Program for Public Sector Evaluation. Roval Melbourne Institute of Technology This
interdisciplinary group focuses on public sector (e.g., program) evaluation; information is
provided on recent articles, coursework, and projects (AU)
2 of 3
1/4/01 10:37 AN
-------
Links http.//wwwevalorg'ListsLinks,Evaluai,,. -niks/'lmks.hnr
>--s on th^ Met A site with many resources, information is organized for
general audiences, students, professionals, researchers, with a "room" for chat and feedback
(CA)
• Gene Shackman's List of Free Evaluation pesourceq on the Web Resources on methods in
evaluation and social research (US)
• Student Evaluation Case Competition Open during the CES annual meeting to students of all
levels and disciplines, this is an opportunity for small teams to compete in the analysis of an
evaluation case file available in English and French. The cite includes archives with past
competition scenarios and winning entries. (CA)
• Bill Trochim s Center for Social Research Methods. Links for applied social research and
evaluation; look for the Knowledge Base (online textbook), statistical test selector, and the
simulation book (US)
• UK Evaluation Society Home page for a professional organization that promotes the use of
evaluation as a contribution to public knowledge (UK)
• UMASS Foundation Relations Responsible for managing the University of Massachusetts
relationships with private foundations, this office provides a wealth of information about
grants, philanthropy, and foundations (US)
• UNICEF Research and Evaluation A timely update of policy analyses, evaluations, and
research; links, statistical data, and newsletter archives can also be accessed (US)
• University of Wisconsin Program Development and Evaluation Full-text publications in PDF
format available for download; targeted to evaluators assessing extension programs, but
resources have general evaluation appeal, as well (US)
• Vanderbilt Center for Mental Health Policy Research focuses on child, adolescent, and family
mental health; follow the links to current projects, including the homeless families study (US)
• Virtual Library Evaluation
• Western Michigan Evaluation Site, the Evaluation Center/Evaluation Support Services.
Information on evaluation checklists and instruments, terminology; resources also include a
directory of evaluators and related links (US)
• The World Bank Institute The evaluation unit analyzes the learning activities for World Bank
Institute clients and staff; be sure to check out the newsletter, too (US)
• The World Bank. Operations Evaluation Department. This independent division evaluates the
lending operations of the World Bank; online publications in different languages are made
available as a result (US)
Federal Sites and Databases
CIESIN's US Demography
CYFERNet f Children Youth and Family Education and Research Network^ Gopher
List of WWW Servers (USA - Federal Government')
National Institutes of Health Home Page
Substance Abuse and Mental Health Services Administration
US Census Bureau Home Page
Statistics
• American Statistical Association
• One-Stop Federal Statistics Site
• Statistical country profiles and global maps of indicators of interest to UNICEF
• Statistics Canada
3 of 3 1/4/01 10-37 AN
-------
STANDARDS OF PRACTICE IN EVALUATION
Refer to
Articles in Fitzpatrick, Jody L and Michael Moms, eds (1999) New Directions for Program
Evaluation No 82 San Francisco CA Jossey-Bass Publishers.
Fitzpatrick, Jody L "Ethics in Disciplines and Professions Related to Evaluation"
Datta, Lois-ellin "The Ethics of Evaluation Neutrality and Advocacy"
"The Program Evaluation Standards" -Joint Committee on Evaluation Standards
"Cuidmg Principles for Evaluators" - American Evaluation Association
Your copy of "Government Auditing Standards" (the Yellow Book) - GAO
Your copy of "Quality Standards for Inspections (the Turquoise Book) - PCIE
1 Think about the differences between; ethics, standards, rules, principles, philosophy
Ethics
Standards
Rules'
Principles-
Philosophy
-------
1:1)1 it)KS' Nuirs
L FllZl'MMCK is dssoi late /);o/cssor D/ />uJ>lic ddmimstidfion d/ (lie University
o/ Colorado 5'it* mmniuins an rtciivf p«( ore in evaluation anil is interested in (/if
ethical nuances o/evaluaior-dient relations Slie serves tin the Board of the American
fl valuation Association and is working on a booh of IMC studies Joi the association
MlCHAFL MoKKfS is pro/essor o/ psyi luiltgy and director o/graduate jield training in
((immunity psychology at the University ajNew Haven He a/Us the column "lilln-
fcn ^jivcn /D (lie
slud^ o/e(lniul codes in evaluudun-rcluleJ (nnsul(mj> f)/o/cssiuns She
examines the ethical cades within evaluation and ielated disciplines
uriJ />i<|/essiuris and discusses implications foi content, dissemination,
and compliance
Ethics in Disciplines and Professions
Related to Evaluation
Jody L Filzpatnch
Donald Campbell (1969) bemoaned lite iracluional isulaiion, 01
of diffcieni disciplines Using (he analogy of a lish. he picsenicd a lish sea It-
mode I of omniscience in depicting the social sciences and iclaied discipline!)
Each discipline believes u knows the uiuh ((he whole fish) when, in lau. we tend
10 know only our own discipline (single scale) This chapter is designed to help
us avoid such ethnocenliisin in ethical malleis and instead, as Campbell aigued
we should, learn fiom related fields
F.lhicul Codes in Program Evaluation
To pi o vide a loundation fbi (Ins learning, I fust Imclly leview the liibimy ul
ethical codes in evaluation In the eaily 1980s two documents wcie
to guide evaluatois in then ethical consideiaiions the Standaid** /or (•
of Educational /'rogrums. I'lojecl^, and Malenal\ ()
-------
h l>ll Kl.INC I'llllC Al I".MAI I I Ml.IS IN LVAIIJAIUIN
maintained (hi- same four in.ijoi groups of standards—iinliiy, Icasihiliiy. pio-
pneiy, ami auuiacy—Inn wiilim these majoi categones some of die onginal
ilnily stiindaids weie combined .UK! oiheis wcie it-vised I uiilici, these newei
siandaids. Ihf /'M^KIIII /Ivdliidlioii Sliini/diiK, wcie intended lo addicss evalua-
luins beyond die educational aiena, though much of llic focus leinains on cdu-
(..iiion .Mid naming Siinil.uly, in 1995, (lie Amencan IZvalualion ASSOLIJIIOII
(AI:A) |)til)lisheil us new GiiK/mg Pniicijrii-s/oi F.valuiUon These guiding pnnci-
ples olfcied a set of values—foi example, honesiy. miegiiiy. anil Responsibility
loi public well.ue—.is guides loi evaluation practice (In 1986. the Lvaluaiion
kescaich SoLieiy had merged wnh I;valuation Neiwoik lo lomi die Amencan
livaluaiion Association, a piofessional associauon lo lepiesem the enure pio-
lession in ihe United Slates ALA those not 10 adopt (he old ERS standaids. but
i.11 hei 10 develop us own guiding prmuples ) Today, evaluaiois in the United
btales have these two documents lo advise their ethical practice Oilier coun-
tnes (Canada), oigamzations (Government Accounting Office), and groups of
tommies (Ausii jl.isian Evaluation Society) have developed then own standards
(See Won hen, Sandeis, and Fiizpainck 11997| foi a discussion of these )
What can we learn fiom these two documents and Irom the history ol eth-
ical codes for evaluation7 Compaimg the documents published in the eaily
1980s with those published in the mid-1990s teveals a major change a move lo
include a gieaiei locus on noii-inelhodological issues "I Ins change is most obvi-
ous in comparing the l:kS standaids and the ACA guiding principles Table 1 I
lists the niajoi headings loi both "Ihe I. KS siandaids generally minoi the stages
ol an evaluation In coniiast, the AI:A guiding pi maples are moie concerned
wiih qualities 01 pnnciples thai peime.ue the evaluation pioccss As I discuss
liuihei on. the natuie ol the guiding pnnciples is moic congruent wuh the eth-
ical codes ol other piolessional associations I hat is, the articulation of values.
as opposed 10 stages of tasks, is a moie lommon stialegy in olhei ethical codes
Peihaps moie mipoiiani lo the lusioiy ol evaluation, this change Illinois die
move in the education .met naming of evaluaiois fiom .1 veiy strong focus on
methodological issues (which ceitamly lemains die suit' <|i«i non of evaluation)
to a gieaiei examination ol the many political faciois anil peisonal judgments
entailed in conducting evaluations I his change has been positively noted in sev-
eial of the commcnianes on the AHA Guiding Pnnciples with special refeicnce
lahle I I I:US Standards veisus ATA Principles. Major Headings
f KS
AM I'MIII I/id's
I Ulllllll.il1011 .111(1 l
Man Inn- .ind
D.II.I iiillnlion .iiul |)ii|).ii.iiuiii
D.II.I .in.ilybib .uid iiiicipii'i.iiiDii
("(iniiiiiiiiii.iii' ilisi Insult-
Systematic nu|iniy
-Ll lor people
ki-spnii!>il>ililif<> loi genual
.ind pulilk will.ui-
I IIIKSIN DlSCII'IINI.S AND I'KOI I.SSIONS KlIAIII) IO I.VAII IA I ION 7
lo the pnnciple conceinmg lesponsibilities foi (he gencial and puhlu wellaie (!•)
(Covert, 1995. House, 1995)
Though the headings in "lahle I I ic-llecl changes in the lone and empha-
sis in evaluation ethics Irom (he 1980s to the 1990s, the dilleience should not
be overstated As one might expect, the oveilap between the enure body ol 1-kS
Siandaids and die AHA Guiding Principles is gieat, most ol the topics ami con-
tent covered in ihe first are lellected in (he second Ihe conveise is also line.
even in cnntioveisial aieas The HIS Siandaids thus recommended ideniifying
various gioups ol stakeholders and then "mloimillion needs and expei laiions"
(f.KS Standaids Committee, 1982, p 12) I hey even aigued thai "evaluaiois
should also help identify aieas of public inieiest m the piogram" (I:KS .Stan-
dards Coniminee, 1982. p 12) But the lone is diffeiem The AI:A Guiding
Principles emphasize the diveise groups we serve. 01 miglii seive. and our
obligation lo be inclusive in ensuring those gioups aie icpiesemecl I in.illy. die
language used in major headings is important One goal of piolessional codes
is to inspire ethical behavior among us members Lolly language can help m
thai regard As such, the major categories lor the AI:A Guiding Pniuiples. as
with the Joint Committee Standards, aie moie inspnalional than (he su-p-by-
siep emphasis ol the eailiei LiKS Standaids
The Social Sciences and Evaluation Codes
Ihe ethical codes discussed above have been stiongly influenced by ethics con-
cerning social science teseaich in specific disciplines 'I Ins influence can be
seen in (he initial impetus foi the codes and m'llien piocessol devclopim-m
The original Siandaids (1981) weie a spinoll fiom the levision ol the hiuiuiimJ*
for Educational and Psychological TCMS and Manual* by the Aineiuan liiliuational
Keseaich Associauon (AI:KA), the Amencan I'syihological Assoiianou (AI'A).
and the National Council on Measuiement m liducaiion I he twelve sponsoi -
mg organizations for ihe 1994 version continue 10 lepii'snil these areas, hut
llie validation panel for the newer Joint Comnmice btandaids also nu Killed
broader aiuliences lo n-piesenl adull naming m many aieas Ni-vtnlifles:>. die
locus remained on education and naming I he development ol the ALA tiiiul-
mg Principles was initialed by leviewmg the ethical codes in psychology (AI'A).
film anon (ALKA). and aiillnopology (Amencan Anlliiopology ASSOH.IIKUI.
AAA) I he committee also icviewed oihei codes dealing wuh iesi-aii.li, nii.liii.l-
mg the federal regulaiions on Pioftdnm i>[ llumtin .SuJ'/cus and tin lltltnuni
RffWH on biomedical and behavioial teseaich (bhadish. Newman. Si lu-nci. and
Wye. 1995)
"I hese di)cumenis. which locus pnmanly on iCM-auh. .m- icii.nnly |>i in-
neiii to the ethical pnnciples of piogiam evaluaims I hey piovidt impoiiaiu
gLiidelmesconceiningihe design ol leseaich and the c-thual UIIK fiiisciu.ulid
when one collects data horn people I lowevei. the almost t ivt liuuson
the social sciences fails 10 iiilonn usol the ethical coiilluis. us l.ncil by
piolt-ssioiis. suih as evaluation, thai woik ilneiily with i s As I h.ivt-
-------
. HiiiiCAi QIAIILNOI.SIN HVAI MA i ION
8
argued elsewhere, because the graduate naming of mosi evaluatois is in ihe
social sciences, we tend 10 use these disciplines as exemplars and neglect pio-
(essions that are similar in our own (Filzpatiick, 1994) As the AEA Guiding
Principles introduce ethical issues concerning relationships with clients and
the public, balancing of stakeholder needs, and the values involved in these
mieiaciions, the ethical codes of other professions that struggle with conflicts
among clients, other stakeholders, the public welfare, and the values of their
discipline become iinpoiiant learning tools In fact, the Joint Committee has
found some procedures from the accounting profession to be useful in devel-
oping their standards for evaluation (Sanders, 1999)
Ihe AEA Guiding Pi maples were initiated to stimulate dialogue among
program evaluators on how we deal with ethical dilemmas Yet, that dialogue
has not piogressed as much as many might have desired Some might argue
that the absence of dialogue is clue to the generality of the principles (Rossi.
1995) House wines that the "endorsement of general principles sometimes
seems platitudinous or irrelevant" (1995, p 27) However, he goes on to
encourage the dialogue, observing that, "Ethical concerns become interesting
only in conflicted cases, and it is often the balance of principles that is crucial
rather than the punciples themselves" (I louse. 1995, p 27) Examining the
codes, cases, and pioceduies of professions confronting similar conflicts can
be fiuitful m fuither stimulating this dialogue
Consulting versus Scholarly Professions
Bayles (1981), m witting about piolessional ethics as a broad subject, makes
an important distinction between consulting firu/fssioruils and scholaily piofes-
sinnfils Admitting the terms represent a continuum and the middle can become
murky, Bayles wines that consulnng ptofessionals differ from scholarly pro-
fessionals in two important ways they establish peisonal. working relation-
ships with their clients, and their method of reimbursement is typically
fee-for-service In contiast, scholaily piofessionals generally deal with clients
at a distance (students m a class, leadeib of a journal) and aie salaried Con-
sulting professionals woik us "eniiepieneuis" and. as such, "depend on attiact-
ing individual clients" (Bayles, 1981. p 9) Consulting piofessionals include
lawyers, physicians, aichitects. consulting engineers, accountants, and psy-
chologists Scholarly piolessionals include teacheis, professors, and scientific
researchers As program evaluators who use research methods, we may fall in
that murky middle, but I would argue that we are more akin to consulting
engmeeis 01 accountants in our relationships with clients than we are to oui
social science biethien
The diffeient economic and peisonal lelalionships with clients, Bayles
argues, "are ciucial in defining the kinds of ethical problems each confronts"
(1981, p 9) The peisonal lelationships that consulting professionals develop
with ilien clients and the expectations engendeied by clients' direct lining and
llIMICb IN DIM MM INI'S ANI> PKorLSblONS KUAIID IO IIVAIUAIION 9
Because the relationship ol the consulting professional is closer 10 ihe cliein
than to other stakeholder, the ptofessional must guaid against bias toward, or
ovendentification with, the clients' views or needs further, because the pio-
fessionals' ongoing livelihood depends on utiiacimg and retaining clients, it
can be against the piofessionals' self-interest (at least in the shoil-teim) 10 pui-
sue ethical norms that conflict with the clients' self-perceived needs I he schol-
arly professional is not so bulfeted by the pressures of individual clients'
expectations or the exigencies of maintaining a practice This distinction can
be uselul for evaluators m consideimg ethical codes Ceitamly, foi method-
ological issues, our ethical codes should build on those from the scholaily pro-
fessions But as evaluation ethics moves toward a focus on the values entailed
in dealing with diverse stakeholders and balancing the public interest, we can
also look to the consulting professions for guidance
The Content of Various Professional Codes
Codes of professional groups vary considerably in their comprehensiveness.
explicimess. and means of enforcement Table 1 2 piesents the principles of sev-
eral professional associations, both scholaily and consulnng ('I hese principles
are referred to as standards, canons, and principles by the difleiem gioups, but
they all represent the first level of values articulated m the code ) One first
notices the commonality across punciples m spue of (he variation in the fields
represented Several (accounting, engineering, public administration) begin with
a principle concerning the public service or public welfaie Psychology ends
Table 1.2. A Sample of Characteristics of Selected Professional Codes
I'ro/ession
I'HIM i/ilYj *
Accounting (AICI'A)
Inlinijl auditors (IAA)
Professional engineers
(NSIM2)
Psychology (APA)
Public admmislrJiioii
(ASPA)
Responsibilities as piolesstnnals. seiving tlie public inltiibi.
integrity, objectivity and mdt pendencc. exeuise due »aie.
apply piniLiples lo scope and nariiie of scivues
Honesty. ob|iilivtly. diltguiiL. luyjliy, lonlliil* ol mil KM. lies
or gills, confidentiality, due (..lie lo obt.tm biilln.ii in l.u iu.il i vi
denrr lo support tbe expulsion ol .in opinion
Safely, bcjllli. and wellaie ol tbe public, i output mi. H|>|LI|IVI
and liullifnl. failbful agent of employer 01 ilieiil. avoid dm p
live acts, conduct onestll bonorably and responsibly 10 lionm
I be profession
Coni|>eieiice, integrity, professional and bucniilic ii'>puu->ibiliiy.
ies|>ecl (or people's nglils and dignity, joniiin loi otbeis w«l
faie. social responsibility
Serve public interest, lespeil constitution and law. niirgiity.
pioinole ethical organisations, slnve loi piolessional exielluue
* Ilir plllicipk'b dlf llblnl in llirir Hitler (il pirMlilalinii in tin units. In i.HIV ilni milt i iiuy irllnl I lie
I'Mii'iin < ill llif ill*.' mint*1 ni
-------
10 |"MI:KI.IN<. 1-iinrAi CHAMKNC.ISIN CVAIIIAIION
wuh a pi maple concerning social lesponsibililics Tlic piommcnce of attention
lo I he public good in ni.niy ol these codes ni.iy lellecl the consulting piolcssions
desire to emphasize explicitly and prominently the importance of audiences
oilier (luin the duecl client Although the piofcssional engineer's code includes
a pimciple cunceining seiving as a faithful agent to a client or employer, their
code. too. emphasizes, lust and foi e most, the obligation to the safely, health, and
welfare of ihc public The complete discussion of their canons, uiles. and pio-
fessional obligations stresses this pnoiiiy The Amencan Institute of Cemlied
Public Accountants (AICI'A) expressly states, "In resolving those conflicts
(between dilfeieni audiences or stakeholdeisl, members should act with integiity,
guided by the precept that when membeis fulfill their responsibility to the pub-
lic, clients' and employers' interests are best served" (Albrecht, 1992, p 175)
They define the public interest as "the collective well-being of the community of
people and institutions the profession serves" (Albrecht, 1992. p 175)
Of couise, the lole of accountants generally differs from that of program eval-
uators But loi many public accountants, their work in assessing a piogram and
defining public interest might be quite similar to those of a program evaluator
(See Wisler 11996| for a discussion of the similarities and differences in the roles
of auditors and evaluators)
In contiast to these codes, which emphasize (he professional's obligation
to the gpneial public, the ALA Guiding Pi maples siiess the tJivmKy of partic-
ipants m the evaluation process and the need lo recognize these differences,
considei the interests of all gtoups, and piovide lesults in such a way that they
are accessible to all 'I he emphasis is on the heterogeneity of stakeholders, not
the homogeneity But the pimciples close with an exhortation to "encompass
the public interest and good," which, they acknowledge, "aie rarely the same
as the interests of any particular gioup " This lattet admonition more closely
Illinois the codes of othei associations The committee, however, noted strug-
gling with this issue, and as they acknowledge, furthei discussion and inter-
polation aie needed to apply (Ins pnnciplc effectively (Shadish and olheis,
1995) 'Ihese olliei codes might piovide some effective guidance
As a fiame foi analysis, Bayles (19HM) has identified six staiulaids ol a
good or iiusiwoithy piofessional honesty, candor, competence, diligence, loy-
ally, and discretion Most of these standaids can be seen in the piofessional
codes listed m 'lable I 2 and m the ALA Guiding Pimciples shown in Table
I I Honesty is addiessed in vaiymg ways It heads the list foi internal audi-
tois Accountants and engmeeis si i ess "objectivity", accountanls add "inde-
pendence" and evaluaiois add "mitgiity" Itaylcs sees "candoi" as going beyond
honesty to include lull disclosuie 1 he code ol piolessional engineers addresses
candoi, loi example, by aiticulaiinga pmlessional's obligations lo acknowledge
eiiuis to clients and to advise clients whrn a piojixi will not be successlul
Similaily. competence is addiessed in each of the piofessional codes, but
undei cK ->t woids Only the code of piolessional engineers, like the AHA
Ciindin, iples, directly uses ihe word "competence " Accounting and
iiiii-iiul iii.iiMisrmnh.isw "i|iii- i lie " wlll( ll lilt oilioi.ilrs In ill) dihtu'llic ,illd
I.IIIKSIN DlSUI'l INI S AND I'KOI I SSIONS KlIAIII) l() I.VAI IIAIIUN I I
competence Psychology and public admimsnrmon annulate pnuciplesioii-
ceining piolessional lesponsibilinesand piolessional excellence
Loyally is concerned laigely wuh the conllici belween obligations in
clients and ie.spousibililies lo olheis, including the piofession I he conllu.ii
faced by piolcssionals in this aiea are paitly addiessed by the pnnc iples con
ceinmg public and piolessional responsibilities Many piolessional codes deal
extensively with loyally Seveial of these codes addiess conflicts ol mien, si and
independence- of judgment 1 liese issues .lie subsumed uncki loyally liei ausi
the client has an expectation thai the piolessional they hue has u-ve.ilcd any
potential conflicts of interest that would (nuclei then completing ihc woik
fairly and will, in fact, be able to provide an independent judgment on tin
issue of concern Guiding Principle I: 4 stales the need to "maintain a balance-
between client needs and othei needs " 'I he evaluaioi is urged lo "meet legm
male client needs whenever u is feasible and appiopiiaic to do so," but tin-
principle notes that when client mieiests conllici with othei pimciples. "ev.il
uatois should explicitly identify and discuss the conflicts wuh the client and
relevant stakeholders" (American Lvaluation Association, 1995. p 25)
Discretion is addressed less duecily by most piolessional code:. Ihe AI:A
Guiding Principles, as with the code ol ethics loi psychology, add i esses the issue
of confidentiality under "lespeci for people " Bui. the implication is thai these
people aie pailicipams in the evaluation, not ihe agency 01 client What obliga-
tion does the evaluator have to the client in ic-gaid lo confidentiality and discic
(ion7 Guiding Pi maple II 3 advocates bioad dissemination of I Hidings Undei
what cncumsiances is such dissemination unethical7 An accounting ethics casi
asks readcis whether an accounting piofessor should use- inmeiuls (mm an oiu
side pioject in the classioom (Albieclil, 1992) II so. should the identity ol tin
firm be disclosed? Although laws on public iccoicls may cast an evaluation u |*>it
on a public program in a dilferem light, what ethical obligation^ loi disc tenon
does an evaluator have7 What loyalty does the evaluaioi have lo the diem7 I hes<
aie issues that should be discussed, luiihei building on the smiilanties and the
distinctions between oui piolession and those in lelatc-d lielcls
IZiiforccineiil of litlucal Codes
HiMoncally. most consulting piolessioiib have been se'll-iegulaimg As pmlts
sionsollen come undei sonic cnticism loi then lailuic- to legulair. |>iolrx,ioiul
associations have established mechanisms loi eiiloueiiinii ol die «»!> •, ih.u
the evaluation piolession cniienily lacks Ihe Amencan Psyi liologu.il Assou
anon, many ol whose membeis aie piacltcing psychologists, the Ann in an It.n
Association (ABA), ihe Amencan Medical Associauon (AMA). the Ann-man
Institute ol Ceinfied I'ublic Accountants (AIC PA), and the National Assucia
lion of Social Woikeis (NASW) all have enloufmeni bodies Hirst ionium
lees answei c|uestions. hear complaints, and issue disc i -iy decision:, m
sanctions as appiopnate I hen hearings and det ismns h .Id i ase law Im
ihe miei inflation ol the elhual codes
-------
12 IIMIRCINI. l.lllll'AI ClIAniNU-SIN I VAIUAIION
lii contiast. enforcement mechanisms are typically absent lioni ilie pro-
fessional associations ol scholaily piolessions Tor example, (he Amencan Edu-
cational Researcli Assou.ilion (Ai:RA). ilie American Anthropology Assouulion
(AAA). and even llic Aniencan Sotieiy foi Public Administration (ASPA)
develop and disseminate then ethical codes but have not developed official
mechanisms foi enloicement Plant (1998) discusses the reasons for the
absence of external enfoiceinent mechanisms for the ASPA code, drawing on
extensive writings in public administration concerning ways to create ethical
behavior Some aigue against even the codification of pmfessional ethics, main-
taining that practmoneis should be (heir own moral reasoncis (Rohr. 1978)
ASPA. however, argues that codes are necessary to socialize and educate the
practitioner about common standards, but thai enforcement al the individual
level is more appropriate than cenlial enforcement Organizations such as
ASPA appear to believe thai the development of "inner controls" will be more
successful at engendering ethical behavior among members than the "external
management of conduct" (Plant, 1998. p 165)
The AHA Guiding Pnnciples may not include enforcement mechanisms
because of a belief in the success of inner controls, however, the more likely
reason lor their absence may be the need to reach consensus on the meaning
and application of the principles, (he continuing tensions among the diverse
paradigms used in evaluation, and (he relative newness of the evaluation pro-
fession All of these factois ueate difficulties in developing and implementing
enforcement mechanisms As evaluation inatuies as a field and greater con-
sensus is achieved on the appropriate methods and actions of evaluators. devel-
opment of enfoicemi-ni meihanisms may be icconsidered They seem more
appropriate to the self-regulating role ol the consulting professions
If AEA continues to use internal mechanisms to motivate ethical behav-
ior among members, however, levisions ol the code may consider the use ol
language to bettei achieve that goal I he style of ASPAs Code of l-thics is con-
sistent with the puipose of instilling internal contiols The code is slioit, it
could lit on one legal-size page It amuilales five bioad principles wiih foui (o
eight bnef points explicating each 1 he woids and language used in this code-
are designed to inspire and are less legalistic than the codes of the piofessional
agencies that include enfoicement The ALA Guiding Pi maples make use of
this formal (principles widi brief points), but the (one of the language, as noted
by the authois, is moic legalistic than the ASPA codes
In the absence of foimal committees delegated with cnloiccment poweis,
oilier means ol educating memhcis and enloiung codes of ethics do exisi and
nuisi be used (o encomage uncle-islanding and compliance I he National Soci-
ety ol Piolession.il Lngmecis, which does not have an official enlorcement
body, uses us Boa id of lilhical Review to inteipiet ethical dilemmas submitied
by engineers, public officials, and membeis of (he public They publish these
cases on-line and in punt with an index, sponsoi an annual ethics contest in
which membeis lespond (o a case, and disseminate a senes of videotapes for
.in.I.-in- ii.,l iM, In You. descnbmg the code and
examining five case studies Congressional c niicism ol accountants assoc uiied
with the savings and loan fiasco in the 1980s stimulated these actions, the pio-
fession was aroused to attend to ethics and us public image (Mmtz. 1992)
The Future for Our Ethical Codes
Compared to the disciplines and professions leviewcd heie. progiam evalua-
tion is quite new The accounting piolession in the United Slates cclchiaied us
centennial a few years ago (Mintz. 1992) Physicians, engmeeis. and lawyeis
have been defining then piofessions and tinkeimg with then ethical codes foi
even longer It is therefoie not surprising that the ethical codes loi pmgiam
evaluators are less well loimcd, their slate lellects the stale ol oui held How
ever, we can leain from the codes levicwed heie A most immediaie issue.
which docs not icquiie consensus but instead action, is to continue to expand
the dialogue that the AI:A Guiding Pi ma pies weie iniended to aeale We need
more discussion of cases thiough oui publications and iimleicnces to aigue
and interpict the meaning ol various pnnciples I he publication ol a new senes
in 1 lie Amencan Journal of Evaluation on e(hica) challenges is a first step in thai
duection (Morns, 1998) EvalTalk and focus gioups at die annual conlcicnce
can be used to futlhei ailiculaie (he meaning ol vauous pnnciples I am cui-
rently working on a casebook for the Amencan Evaluation Association, loi
which I will diaw upon styles used in othci piolessions Ciiiiein discussions
of certification or licensure are peitmeni to oui ethical codes Q-iiilicaiion and
licensure. as with ace led nation, piovide a mechanism foi ensiinng thai pm-
fessionals are mfoimecl of and concerned with oui eihical codes I inally. con
sideiation might be given to linking inemheislnp in die Aim-man I valuation
Association with the AI:A Guiding Pnnciples The Joint Committee Siandauls
for Progiam Evaluation, AT As Guiding Pnnciples loi I valuaiois. and codes
from other disciplines have provided us wuh food foi thought Now we must
continue 10 discuss and amculaie what it means to be an ethical c valuaioi
References
Alhieelll. W b /.lined/ KMII •. in (lie- Tide (in- <>/ /Viiiiuifmi; ( nu iiin.ili '.milli Wi -,11 in
Ainencan Lvaln.iiinii Absoi I.IIIOM Ciiiidin^ l'iiiiii|>lfb loi I v.iliuiois In W |< Slmlisli.
D 1. Newiii.in, M A Sclic-im. .mil C Wye (i-d* ). dunlin); f'iiiui|>li >, /m / nilii.ilnis Ni w
Dirc'Llions Ini l'ruj;r.iin r.vjItinlKin, no 66 San 1 1. un ism |n->s< y I U-.-. I'W>
Bayles, M I) /'ni/t-sMoiiul kihit* hcliimni. C_.ilil Wdilbw.mli I'JHI
Campbell, D "rilinnteniiibin nl Disciplines .uul ilie fisli Sulc Miul< I nl i IINUIM u iu< In
M Shcuf and (1 Shcnf (ciU ), (nfi-i i/iu ifliiuiiy Kilii(inii\lii(Mii iJu \m,il \i,n, ,\ ( Im-i^n
Al.lini |«h«>
-------
14 I'MIRl-INC. I-IIIICAI ClIAIIINC.ISIN I'VAIUAIION
Covul K W "A Iwi'iiiy-Yeai ViliMan's Kcllct lions on ihe (.iiiiilmgl'nniipks for l.valuaiois"
lnW R Sliadish. D I Newman, M A Sclii'iiei, anil C. Wye (eds ). Glinting Pnnufiln JIN
(vuhuiiuis New Diieiiions loi Piogiam (.valuation, no 66 San francisco Jossey-Bass. 1995
I valuation Ki-icaii.il Sixieiy (I RS) Standaids Comniiiice "Evaluation Rcsean.h Souety
Siandauls loi 1'iogi.iin kvaluauon " In I' Rossi (eil ), 5liimii tvuliitilion PHIIIIIC
New Pireiiion* fin Program Evaluation, no 15 San riancisco jossey-Bass. 1982
I iiz|KUiu.k. J L "Aliemaiive Models lor ilie SiniLluringol 1'iofessiunal 1'ieparalion I'rogianis"
InJ W Aliscliuld and M (Ingle (oils ). Hit Pii/wiuliiiniiJPiiiJi-s\iiwKil Cvuliiufius ls«as. (Vi-
>|>fifm->. niiJ l>n>xi(iiiu New PIICUIUMS for Piogiam Evaluation, no 62 San 1'iani.ibLO
Josscy-llass. 199-4
House. L R "Principled Evaluation A Crinque of ihe AEA Guiding Principles " In W R
Sliadisli. D L Newman. M A Scheirer. and C Wye (eds ), Guulmx Piiiuif>/i-s/c>i Cvu/im-
(ui> New Diieuions loi Piogram I'valuaiion. no 66 San FranLisco Jossey-Bass. 1995
Joint Committee on Standards lor Educational Evaluation Standards Jui Cvuluiilimu of
Muuifioruil Curiums. I'Ki/uli, mill Miiii-iiuff New York M(.Graw-lldl. 1981
Joint Coniiiiiitee on Sundaids lor tducaiional Lvaluanon Tlie Pingum Cvu/uulinn S(lrs jai I'vuluahn* New Uirections for 1'iogiaiii
LvaliMlion, no 66 b.m I laniiMo |ossey-l).iss. 1995
Sanders. J IVisonal (.oininuiiKation. January. 1999
Shadish. W R . Newman. O I. . Siheiiei. M A . and Wye. C (1995) "Developing the
Guiding PiiiKiples" In W R bhadibli. D I Newman. M A Stlieiier. and C Wye (eds ).
CuiJiiigPiiiuifilir^dr LvdliuilDii New Direuions foi Program Evaluation, no 66 ban
liancisco Jossey-Uass, 1995
btulllcl)caiii. D L "A Next btep DIM ussion 10 Conmdei Unifying the I.RS and Joint Com-
mittee Siandaids " In I' II Rossi («l ). Sluiufcinl*/m /.v/ /»nl»li( m/ininiMiulKMi iu< IILC in fvuliiiHioH duel i.s inlcii'slrd i" llu1
rs.'i rviiliiiilni diriiJ nliiliniis Slir scivrsdii llir llndiilnf
Although empiiual /tit'im/i on evaluation fllncs i!> not /)lt-;i/i/ul,
strvctul impoitant findings have emeiged 7/iesf uu liulc on
Iticfe o/foriicnsiii wilhrn iht field concerning whal tonsli/ules on
ethical issue, the/ie^uent ouuncnit of elhmil /)/o/)/cnis ilunng the
latei stages of evaluation piojects, and the peiceivcd ethual
significance oflhe tendency foi evaluaiors to be nwic itspondve to
some stakeholder than others The authoi (/licuiu's the neea to
incoipoiate reseat di (jueslions (in ethics into nn^ytin^ eval
piojCLli, and to imciS systematically evaluating pen f/)d
Guiding Pnnci/)lfi
tion
Research on Evaluation E
What Have We Learned
Why\Is It Important?
cs:
Michael Morris
\
\
NtMily twenty yc^tis ago Slu'inle)d and l.oid (IS>KI) nuicd ili.n "i-inpiin.il Mud-
les of the ethical Concerns of /valuaiion leseaicheis aie lew" (p )HO) Wh.ii
was true then is only sligliily/fess nue tod.iy Indeed, at a leu-iu session devoted
to "What Should We\Be IVseai clung m l:valua(ion Films'" (Morns. I"A>7) .11
the American livjIiiaiW/Associaiion's (A I: As) annual iiieeting. I lie panelists
outnumbered the audience1 Whatever else ethical issues may he, they do noi
appear to have auiac^ed HIC attention of a large segment of the leseaich com-
munity in evaluulir
'I his is not lp say, of course, that theie has heen virtual silence on (he
subject beyond/he Guiding humpies jot l'.vahiatoi\ and the joini ( omiiiii
lee's Program Uvttluation Siandaids Analyses of ethical concerns, fiequeiuly
based on tlie/ju'isonal experiences ol the auihois. ate iclaiively easy to hiul
(see CnghsU; 1997. Gensheimer.Vers, and Roosa. 1991. Schwandi. 1997.
Slake and/Mabry. 1998) Far fewerNjiowever. are cases in which the auihois
have garnered primary data m a systematic fashion for the explicit puipose
of shedding light on ethical issues iiAvaluatton In a held that pi ides use 1 1
on bemg commuted to decision making informed by such data, ilns stall- ol
affairs is cause loi concein Accordmgly.Nhischapiei will locus on |>i 1111.11 y
studies and then value loi enhanctn)l om undeislanding of evaluation
ettcs
-------
76 EMLKGINC, rniu'Ai. CMAII LNGI.S IN EVAIIIAIION
Kawls, J A Theory ajjustut Cainbiulge. Mass Belkiiap 1'icss of Harvard Umveisiiy Press,
1971
Kyan. K L Using reiiiiin:.! Sii.negies loi Aildiessing Issues ot Social Justice Do I hey
Help7" Pa|>ei presenicil at the annual meeting of the Amentan Evaluation Assncianon.
November 1994
Sanders. J R . Newman. D I., Owen.J , and Woilhen. B "Making the Piogiani evaluation
Slandards Meaningful in (iiaduale IMucauon " Panel presented at the Annual Meeting ol
ihe Ameiu.au Lvalualion Association. Atlanta. November, 1996
Sk.hwandl.1 A 'Bctiei living I liniiigli Evaluation Images of Progirss Shaping Evaluation
Practice "{.'viiluimiiii fun (in-. 1992. (1(2). 135-H-t
,l)IANNA L NLWMAN is associate pioffssoi and dntiloi ojlhe Lvaluation Consoi-
duni (it (lit iiwif UniveiMty o/Nrw Yoili at Albany She is coauthoi oj Applied ElhiLs
in Pnigiam Evaluation, and "Guiding Pnnuplesfoi Hvaluatois" and has piesented
and published moneious papeis and unities on cdncs in mifiKiliou
on (i Jrvrli>f>iiu'iii«l inot/cl (i/lidiiiin^ cvalualon* that
issues sft'/n (o tiiuiisc the fxissum ^cm uiici/ by di\t IISSIOMS
oj whe
incieased Tlie federal governmeiil IMS incieaseil ilk- pciulncs .ly.tinsi ilic pei|x:-
liaiors and also lias laken ulhei nie,isuies An ev.ilu.uion ic.un g.uln.-ib 10 JbiCbS
I he effectiveness of these measuies One inetnbei ol the le.nn. like many oilier:,
in this nation, is strongly "pio-lifc," believing that under (he Niueinlxrig and othei
i tilings, any action is justified in pievenlmg what this pet son legaidb us the
slaughter of innocents 'I his team member sees the government as pioleiimg
"murderers," believing the genetal welfaic and public good demand doMiif ol the
(.limes Another team meml>ei, like many otheis in lliii, naiion. ib Miongly "pio-
Lhoice," believing that law and ethics give a woman coniiol ova hei own body
and regarding violence against the clinics and medical pcisnnnel asuiminal. not
heioic To this team member, ensuring the geneial welfaie anil public good
requires the pioieciion ofclmics olTeiing ahoinons lioih memlx:is call thcmselvt-.s
evaluaiois Should eithei of these ev.iluatois panicipaii- in 01 lead ihe evalnaiion'
For this chaptei. I have been asked to examine aigumenis loi and ag.nn.si ev.il
nation neuliahty and advocacy To exploic these issues I havr looked pimi.mly al
articles by [xist ptesidentsol the Ameiuan Lv.tlu.u ion Association and mini piunn
nent figures in out field 'I heie is an abundance ol pnoi wouls on this lopu . sonu
of which will be summaiized in the next sections Soiling ilnough tlitin, wh.u
struck me was not then dissimilanty but—with some exceptions—iln.it agKeineni
after one had woiked thiough die definitions given ol advoc.u y and neimaliiy
Nonetheless, some of the discourse on the ethics of advocacy in cvalnaiion si-ems
to take place as though the moial high giound had loom loi only om. luimu
Why the passion, given the common giouiuP (3nc U.IMMI may he die
|X)ieniial for common gionnd in theoiy to gel "balkamzal" in piauut A sicond
-------
7H llMIRC.INC MlIK Al ClIAl I I'NU.S IN HVAI.UAMON
leason may lie whether I lit' evaluaior pnmanly has in innul a nnuonal study or
one i lose to client service delivery The cxplanatoiy power of ihis distinction,
gianted. does not always hold Without deprecating the many ethical and moral
dilemmas confronting evaluatois, peihaps we could advance a hit funhei by
examining ( I ) specific evaluations earned out at ronifwicjiMr levels in light of (2)
the pi maples and theones put foith undei diffetent banners
Guiding Piincinles for Lvaluulors
Our suiting point is the "Guiding Principles for Evalualois" adopted hy the
American 1: valuation Association (A Li A) as "a set of principles that should
guide the ptofessional practice of evaluations, and thai should inform evalua-
tion clients and the geneial public about (he principles they can expect to be
upheld by prolessional evaluatois" (Amencan Evaluation Association, 1995,
p 21) There aie five broad principles systematic inquiry, competence,
miegiiiy/honesiy, icspect for people, and lesponsibilines for general and pub-
lic welfare These have been presented eailier in this volume
All ol the ALA Guiding Pi maples aie iclevant, to some degiee, to the ethics
of advocacy One cannot, howevei, sum the pi maples foi geneial guidance
regaidmg advocacy and know exactly what actions to take Fust, as intended,
the pi maples aie not siandaids They do not indicate, for example, matteis of
practice, such as what would constitute incompetent performance or what types
of education, abilities, skills, and expenence would be mappiopriaie for differ-
ent types of evaluation (asks for dilfeicnl evaluations Second, it is possible for
|>t i sons uiking dilleient positions on the ethics ot advocacy 01 neutiality in eval-
uation to cue one 01 anoihei principle as consistent with their views
In the next sections, these appaiently dissimilar positions are piescnted
togethei with the AIZA Guiding Pnnciples that seem 10 suppoit them, anil then
the positions aie teexammed to identify what may be common giound that
ledefmes the Pnnciples fust, loin definitions (Websteis. 1994)
Advocate One who ilcli-iuls, vindicates. ui espouses a cause hy argument,
upholder. defendei, one who pleads lur or in behalf ol another
Adveisaiy A person 01 gioup who opposes another , opponenl, foe, any enemy
who lights deteiminedly. lelcnilessly, loniinuoiisly
r.inis.in An ailheieni or supporter ol a |>erson, pany, or cause, hiased, partial
Nonpanisan Objective. not suppoiluigany of the established or tegular parlies
To IZvalualc Uequiics Credibility: No, livalualors
Should Not Uc Advocates
I heie is no laik ol woids and deeds lonccinmg what evaluation and the eval-
ualoi's lole a' u Some uuild be lead as indicating that the cvaluaiois wle
is .ihoui ne iiip.iiiis.in evaluations leg.ndless of how pauisan 01 non-
11 II IIS 111 I 111'
IHJi'll
I III Ll I IICS Ol II VALDAI ION Nl.UIKAl II Y ANU AllV()( A( >
For example, Chehmsky (1997) observes
In he listened in hy various stakeholders in even an ordinary political debate
requires a great deal of ellon by evaluaiois not only lo be competent and oh|ce-
live bul 10 appear so I here are a great many things we can do not just
technically, in the steps we lake lo make and explain our evaluative decisions,
bul also intellectually, in the effort we put forth 10 look at all sides and stake-
holders in an evaluation A second implication lor evalualors of a political
environment is the need lor courage Speaking out in situations that may
include numerous political adversanes, all with different viewpoints and axes to
grind, and also insisting on the right lo independence in speaking out, lakes a
strong stomach li takes courage 10 refuse sponsors the answers they want
to hear, lo resist becoming a "member of the team," to fight inapproptiate intru-
sion into the evaluation process bul when courage is lost, everything is lost
Ipp 57-60. see also Cook, I997|
This is Scnven( 1997)
Distancing can be thought of as a stale on which a minilx r of points aie nf par-
ticular inteiesl At one end ol the scale is complete distancing, as when a pio-
gratn (person, policy, or whatever) is evaluated on the basis ol extant data alone
At the other end is ownership or authorship of the progiain, usually conceded to
be a poor basis for objective evaluation of it Although u is belter in principle
to use extant data, u is often the case thai one needs moie. and the n&ks atten-
dant on personal involvement |bias| must be undeilaken So-called p.iiiiu
palory design, pan of the empowerment movement, is about as sloppy as one can
get. short of participatory authoring of the final repoit (unless thai report is
mainly done lor educational or therapeutic put poses) li is sometimes sug-
gested that the push for distance is tisell an attempt 10 lie superior, external, an
attempt to play God the Judge On the contrary, n is pan of the simple and sen-
sible human effort to gel things light, to uncover and report the truth— Deciding
when and to what extent lo withhold those findings horn those who paid loi
them is the "doing what's good for you, not what you asked me to do" step ovi i
the holder between expemse and censoiship/p.tiennng |pp -IH-I.
79
To Evaluate Is lo Advocate. Yes, F.valualors Should He-
Advocates
Other wotk could be lead as saying we aie about dealing painsan ev.tliuiions in
an inetnevably pauisan woild We should be advocates, weighing in on the side
of llie underdog, the oppressed, the maigmalizcd in the light foi sot i.il justice
Lincoln (1990) wines, "|To the posilivistsl, only il icseauli u-sulis wen-
free of human values, and. theieloie, liee fiom bi.ts, pie|udi- ' individual
slakes could social action be taken thai was neunal with it n> poltiK.il
n.illls.mslno I lie i OIISIIIK IIVIM ll.ii,ulii'ili Hi is ,i> us < i nli.il Im u-.| the
-------
80
LMI Kl.lNt. 1.1 Illl Al C IIAIII NC.IS IN I VAIIIAIUIN
picseniaiion ol multiple. holistic, competing .uul often conlliclual ic.ilnies of
multiple stakeholders JIK! icscaich participants the wntlcn report should
demonstrate the passion, (he commitment, and tlie involvement of the inquirer
with his or her eopaiiiapams in the mquiiy" (pp 70-71) She fuitliei corn-
menis, "We sliould abandon the lole ol dispassionate ohseivct in favor ol the
lole of passionate paiuupam" (p 86)
Greene (1995). expanding on this thought, uiges in hei classic, widely
cited aniclc. "Evaluation inlim-mly involves advocacy. so ilie inipoiiiint qucs-
tiun becomes advocacy ku whom The most defensible answer 10 this question
is that evaluation should advocate loi the interests of the paitiLipants" (p 1 ) In
a iclaied statement, rctieiman (1997) offers a nuanced argument, considering
both advocacy and data credibility, that evaluation is best seen as a foun of
empoweiment He observes, "Empowerment evaluation has an unambiguous
value orientation — it is designed to help people help themselves and improve
iheir programs, using a foim of self-evaluation and leflection Advocacy.
in this context, becomes a natuial by-pioduct of the self-evaluation process —
if the data mem it" (pp 1H2-384)
AndMeriens(l995)
I Ins prmuple (III D ">) coiiitinnig divcisny and inclusion has implications not
only ai ihc level of idemilying anil it-spelling I he viewpoints ol marginalized
groups, liul also for I lie li-ihiiK.il adequacy of whai evaluaiors do Evalua-
tors need it) leflnl on huw loaddiess validity and reliahilily honestly in acul-
liual context, so as not lo violate the human rights of I he culturally
uppie&sed 1 1 he ein.iiu ipauny iramewnikl is more appropriate to slop
oppression and bring about social justice Three characteristics |ol ilus fiame-
woik arc) (I) recognition ol silt-need voucs, eiisuiing dial groups liadinonally
marginalized in soueiy are e(|iially "lie-aid" tluimg the evalualion piotessaiul
foinianon of evalualion findings and n.-ionunendalions. (2) analysis of power
iiicc|iiiucs in lenns of suual iil.Miunships involved in I he planning, iniplcmeii-
laiion. and reporting of evahiaitous. (3) linking evaluation resold in polniial
|pp 41-92)
III the context of evaluation as advocacy, stakeholder involvement seems
to mean the- evaliuior should take up the cause ol the marginalized "I he e val-
uator should nuke 01 snppoii piOLcdiir.il, technical, and methodological deci-
sions l.ivoi ing the side ol the poisons directly leccivmg sei vices
Sonic Relevant Principles and "Ilicir Implications fur
Ami- and I'ro-Ailvocacy Stances
I he ALA Cjiiiding humpies do not tule out ciihei lite ami-advocacy 01 pio-
advncai y stances, anil vanous ones can lie cited to support eithci position
Against Advocacy bevcial ol the Al'A Guiding rnnciplcs can he cited to
emphasize the iiKompaiihihly ol evaluation with an atlvnc.u v posnum as nuli-
liir Knurs or MVAIUAIION NI.UIKAIIIY AND AIIVOI'AI r HI
cated in the quotes given These aie lound piiinanly under I'linciplc C
Integrity/Honesty In us subpaits. (his principle emphasizes that evaluatots
should assute the honesty and integrity ol the crime evaluation process
through practices such as being explicit about then own (and others') inleiesis
concerning the conduct and outcome of evaluations, disclosing any roles 01
relationships that might pose a significant conllict ol interest
As these words aie generally understood, they are inconsistent with an
advocacy position Accoidmg to Websieis (I99'l). fionrsf means "lloruualile
in principles, intentions, acuons, fan, genuine 01 unadulterated, iiiiihlul or
creditable, unadorned, just, incomipnhle. liustworthy, mnhlul. stiaiglit for-
ward, candid " In common understanding, as an evaluaioi one cannot he lair
to all stakeholders and at the same time lake a position of advocacy (or adver-
sary) for or against one stakeholder gioup or the other The principles tell us
to be scrupulous about identifying biases, values, pieconceptions favoimg one
outcome or another that may be held so strongly the evaluaini could find it
difficult to be fair, incorruptible, just, trustworthy "I hese ilneais to faiiness
specifically and explicitly included political stances 'I hat is. the principles
assume that evaluatois have biases, prejudices, values, opinions I hey require
us, however, to be ever mindful of how our values may alien oui londuci ol
the evaluation — and to disqualify ourselves from a paiticulai study il we can-
not be balanced, fair, just, incorruptible
Different organizations use slightly different leims loi (he same idea I he
U S General Accounting Oflice (1997) speaks of "impanmeius" in one's abil-
ity to be fair and just These impairments can come not only horn financial and
career interests, but also from values, attitudes, and political views Mowevei
phrased, and with appreciation for the nuances of phiasmg. the evaluatoi can-
not, this principle makes clear, take sides This is quite dilleient hum iepoii-
ing findings that may favoi the inleiesis of one pany or anoihei Kathei. it
means conducting the evaluation so that the findings aie not slanted begin-
ning, middle, and end by the e valuators own passions liy this punt iple, the
evaluatoi must forego balancing peiceived inequities with a thumb giving
greater weight to the scales of the oppressed
Considering this reading of Principle C. neithei the pro-life nor the pio-
choice evaluator should be on the evaluation UMMI llieii pnlitual p. .sir urns
seem so deeply held as to be considered an impaiiinenl to a Ian, just, nusi-
worthy evaluation
For Advocacy Another pimciple, howcvci. uuild l>c read as |viiiiiuing and
perhaps encour aging advocacy in evaluation I'liniiplc I! considers u^pDiisihiliiy
for the general and public welfare It explicitly stales "cvaluaiois have obligations
that encompass the public interest and general good ck-ar tineas to the pub-
lic good should never be ignored in any evaluation llecause the public mieicsi
and good are raiely the same as the inteiests of any paitu ular gioup ivalua-
tots will usually have to go beyond an analysis of puiticutai sukcliolilt-i micu-sis
when considering the welfare of society as a whole" (Anifiicau rv.iluaiioii A.->s«)-
ii.ition 199-5 pp 2V26)
-------
82 I.MiK(.iN(. limirAi CIIAIIINI.LS IN I VAIUAIION
A common language leading ul ilns pnnciple lequnes cvaluatois ID he
cvei conscious of the public good and gcneial welfare liui the guidelines do
noi indicate wh.ii view ol ilie gcnei.il welfaic and public good is considcicd
Wbai is slated in law' l)y unrenily elected olficials' By majority opinion? Ity
ilie views of whatever gmup seems mosi disenfiancbised by whatever indica-
tois' lly ibe evaluatoisown peiccplion ol social justice' As Rossi (1995). dis-
cussing ibis pimciple, points out." what is the public good is (lie bone of
contention .iniong political parties, political ideologies, and even world reli-
gions" (p 57) It seems as tliougb evaluatois can select any definition ol ilie
public good tbey choose
Wbai are the implications ol ibis position for the hypothetical abonion
clinics' evaluation7 Considenng this leading of Principle E, depending on your
poitu of view, eithei ihe pm-ltfe or die pro-choice evaluator should seive on
ilie learn bui noi boih Moreover, any evaluators who have not thought
through what ihe common good and general welfaie mean on this issue (thai
is. on abonion) should leach a position as pan of their responsibility
li seems noteworthy that the basis for Pniiciple E is not a belief (hai eval-
uators are irtemediably unable to l)e objective, but rather that we serve a htghei
social good beyond serving those in charge and ihe proximal and miermedi-
aie stakeholders, such as stalf and paiticipanis of a parttculai ptogram To do
only the bidding of those paying fui die evaluation is seen as making evalua-
tion hide moie than maikei icseaich Although icsponsible 10 our clienis.
wheihei internal or external, we are equally icsponsible. in light of ihis prin-
ciple, foi consideimg die geneial good and public welfaie
Hxactly what evaluatois have lo do beyond "considering" is left unsiaied
Presumably it includes inlusmg all aspects of the evaluation with the repie-
sciuaiion of the ultimate stakeholder—the public good as undeistood by die
evaluaioi—in die same way one would a more pioximal siakeholdei
Coiniiiun Ground
"Ibis bnef analysis illusti.iies what many oiliei evaluaiois have alieady noted
(see. lor example, Rossi, 1995) The pi maples appaiently can be cited in sup-
lion of neutialiiy or advocacy in evaluation It is theiefoie not to the ALA
(uncling hmciples as stated thai evaluaiois might look for standaids ol con-
duct in specific cases 01 foi a icconciliaiion. if (Ins is possible, between appai-
cntly ineconcilable views
I he ambiguity ol the AI~:A Guiding I'unciplcs is consistent with the inten-
tional dilleience between the geneial guidance of pi maples and the opeia-
nonal guidance ol siandaids Rossi (1995) commented The membership ol
AI:A is divided on a nurnbei ol cntical and substaniive lechnical issues A
stiongly woided set of standards might easily sundei the weak bonds that bind
us together :••- ' nullily the compromises that make AI:A possible" (p 5o)
Ihe |). es developed between 1992 and I99H weie intended as pail
ol loiiiinum^.iMiissionson ethical issues I hey seivrd us well ilint in ollei-
Im KIIIICSOI I'vAiiJAiioN NLDIKAMIY AND AIWIX AC Y H)
ing an ecumenical liamewoik lor lobust discouise on ethics Seven yeais laiei.
however, the pi maples may need lefieslung in oidei to leflect new appioaches.
such as emtiffnl italism (Henry, Julnes. and Mark, 1998), and to guide piac-
lice Indeed, some common ground may be piesent in the values shaied by vai-
ions perspectives on the ethics of evaluation advocacy and neutiality I bai is.
by examining possible common denominator in lecent commentaiies on these
issues, we may get back to a sense of how 10 balance appaiently competing
principles
Iwo sinking common denominatois aie the value placed on launess and
faithfulness to all stakeholders and on lespecimg deeply ihe dignity of all stake-
holders and their right to be heard A series of recent anicles by leaders in oui
field, such as Lincoln, House, and Greene, gives a window on contempoiaiy
definitions of advocacy in evaluation
This is Lincoln (Ryan, Greene, Lincoln. Matbison. and Meitens. 1998)
We opeiaic Iroin profound social commiiinenis whn.h honoi all siakeholdeis
groups' views and peispccnves, wheiher or not we liap|>en 10 agree wuli those
views We speak ol "jilvocacy" as if u meant we go inio an evaluaimn deiei
mined 10 take sides, and that would mean typically. "agamsi" ihr piog i man-
agers, administrators, flinders, or olliei uilical individuals When I lalk aliinii
advocacy, I don't mean taking sides in llial moie spv< iln sense Wliai I IIILJII
ralher icfeis 10 becoming an advocate for pluiahsm, lor many voii.es in Ictd mm
the evaluation Wliai I am ad vocal ing lor is less a paniciilar individual 01
group ilian a position whii.li insists dial all siakeliulders be idenulied and
solicited for their const muions of I lie siiengilis and weaknesses ol ihf |>uigiain
Ipp 102. IOH|
A similar idea was expressed vividly in hei discussion (1990) ol the need lo
"express multiple, socially constructed, and often conflicting lealities I he lai-
lei we termed fairness, and judgments weie made on the aihievemcnt of this.
criterion in much the same way that labor negoiiaiois and mediaiots delei-
mme fan ness in bargaining sessions" (p 72)
This is House and I lowe (1998)
We lliink llit liamewoik |ol a Uielnnsky study) must he something like tins
Include conllicung values and siakeholilci gioups in I lit- snuly Make smr all
major views aie sufficiently included and n-pirstnied Ui ing > outlining vuus
logeihei so llieie can lie lie-liberation and dialogue ahiuil (hem .iniong tin leli
vanl pailies Make sine I lie if is sulliiienl mom lor dialogue lo usolvi lonlliii
ing claims, but also 10 help ilie policy makeisand media u solve ilust- iLnms liy
soiling ihiough ihe good and bad information Is ilns advoi acy on die |i.ui
of the evaluaiors? We would say no. even though linn woik is heavily value
laden and incoipoiales judgment U is noi advoeaiy. sin h .is i.iki- •» :.idi 1.1
the- otlieil al ilie beginning ol (lie study and iliaiii|iioning 01 sidi 01
anolhei We suggest lliree iiilena loi evaliialions in In |iiopiil> kilinm!
-------
I MIK(.IN(. I: I Mil Al QlrtlllNC.IS IN I VAIIIAIION
FIISI (lie study sliiiulil IK- iiiLlusive so .is lo iipiest-iu .ill u-levam views, interests.
values anil stakeholder beioiul, I he it should In- sulliuent dialogue wiih the
relevant groups so lh.il I he views aie pmpeily anil authentically represented
llnrd. there should be biilhcienl deliberation ID anive ai proper findings |pp
'I Ins is Gicene (Kyan. ct .il ,
Ilicepl in unusual iiicumslanii-s. I ilo mil believe llial cvalualois should advo-
tale for (he piogiam being evaluated Sui.li advocacy LOiiiproiinscs (he peiceived
nedihilny and ihus ihe persuasiveness of (he evaluative claims whai evaln-
atuis should advocate for is ihcir own value commitment In panicipa-
loiy evalualion. tins value commitment is to democratic pluralism, lo
broadening the polity conversation to nuludc all legitimate perspeinvrs and
voices, and lo lull and fan stakeholder paiiiupaiiun in policy and program den
sion making the paiticipatory evaluaior needs in get in close to the pin-
gram Dm this closeness should not be misionsirued as piogiant
partisanship I hat is. pailiupalnry evalualors do advocate, not for a particular
progiam. but lather lor an open, inclusive, engaged conversation among stake-
holder about the mem and wonh ol dun progiam of fanly and fully rep-
lesennng all legiliiuaie inleiesis and concern!) in an evaluation |pp 109. Ill)
Refraining llic Discussion
Neulial A person or gioup not taking pan in a controversy, unahgned with
one side or another in a contioversy, ol no partuular kind or characteristic,
iiulrhnile
linpaitial Fair.jiist
Wiih ihese clcfiiiiiions (Webster's. 1994). a shared theme among die eval-
ujiors cited liete is iinparuality — (lie sense of fan ness and justice Neuiialily.
which might initially seem similai, is too passive, connoting a son of with-
drawal fiom the storms and complexities ol the wot Id Passivity does not seem
to me either characteristic of. 01 common giouncl for. our field I his review.
I hen, suggests
• Dive tse evaluaiois ayce thai ihe evaluatoi should not he an advocate (01
ptcsumahly, an advcisaiy) of a specific piogtam in the sense ol taking sides,
ol a pieconceived position of suppon (01 destiuction)
• *l here is agieement that the ev.iluatoi should he an advocate foi making sine
that as many lelevam cauls as possible gel laid on the table, face up, with
the values (wonh, mem) showing
• 'I heie is uycemcnt that the evaluaiot must be awaie of how less poweilul
voices or unpopulai views, positions, infotmaiion can get silenced and make
spa i.il rlloits lo etisuie ih.tl these voters (data, view minim. iimnl m-i heaid
III! htmisol I-VAIUAIION Nl.DIRAIIIY AND Al)V()( A( V H"i
'Iheiefore. it may be he I pi n I to lehame the iliscusstons in leims ol inipai-
lialily or fairness No evaluation appioach ol which I know would couiiicnam e
(I) delibeiately ignoring program ihcones leading to dilleienl expectations
about what should be studied 01 measuied, or what lesults to look lot, (2)
deliberately selecting measiues 01 quesiions to favoi one side ovei anothei, (5)
dehbetulely misquoting what an interviewee said, (4) delibeialely dealing data
out of the whole cloth to prove a point, (5) Jflilieialcly going fiom the teams
of raw data to com Instons hy a sneaky path siipponing one side ovei anoiliri,
(fi) delibcmtcly tailing to listen lo the views ol .ill panics with conlliiiing pi i
speclives, (7) dflibtialely suppressing mfoimaiion that did not suppon ihe
evalualors own values 01 ihe lesults ihe cvalualoi wished lo obtain, (H) iMtli-
mifffy using woids that cumulatively skew the repot I lo one side 01 (he oihct,
or (9) delibeiately presenting complex, nuancecl Imdings m a simplisiic way to
favor one position over anothci (Dana. 1997) Pet haps these points aie a Mail
lowaid expiessing standards in this atea with which many evaluaiois could
agiee
This is not to say that we may not inadvenenily m practice—through
methodological limitations, tgnoiance ol how out own views and language
cieate subtle biases, or failuie to use strategies lot achieving Ian and latihful
evaluations—do all of the above 01 mote Kathei, n is to say thai .is I tead
iccent elloits to atliculaie what we mean, I hnd that we seek balance and want
fairness, like a mighty rivet, lo pour down
QED? No, Dilemmas Will Remain in Application to
Practice
Principles aie not standards on how to be Ian and just m piacticc What pnn-
c i pies mean m practice is likely to requite continued leexaminaiion and tern
leipietalion as experience giows, evaluation iheones develop, and new
technologies aiisc l:or example, to some evaluaiois. such as (iiccnc and
Meitens, closeness and inclusion aie essential 1 he evaluatot models, by how
the evaluation itself is clone, ideals ol empowerment, demaigtnalizaiion ol the
disenfranchised and oppressed, and in so doing leaps many evaluation hem -
fits such as greater authenticity, bellei balance, gicaicr latiness, "natuial" i val-
uation utilization Since truth lies in the eye ol the beholdei. one logically gi is
as many beholdeis as possible
Toothers, such as Sciiven (1997), opinions and srll-iiiu-usi lu- m the tyt
of the involved stakeholders, albeit expeiietued by them as until ( lost tuss is
to be avoided, risking as H docs co-option and bias Also to be avoided is l>< mg
impartial on an issue dunng wot king lioins but an advcisaiy 01 advoc.nr on
the same issue when the metei isn't iiinmng Inclusion ol ielc vani, but unpop
ular or silenced views, lo such evaluaiois is as uncial lo evalualion as n is in
those encouraging closeness "I he techniques loi achieving stu h nu IIIMOII aie
not seen as requiring silting atomic! a table, as n wete, wiih tin evalnaioi as
moderator when decisions an-made about di-sii'n mfasiins analvsi s and
-------
H6 I'MI KfilNC. I. MIIC.AI ClIAl I LNC.Ib IN IJVAI IIAIION
reports. R:n her, (he techniques UK kale using cxlani cl,U;i and lelymg on pci-
lonnance cl.ua rathci ilian siall interviews, and wheic suc.li iniciviews aie
essential, being sine ihey involve piesiruciunng based on oilier daia and ate
conducted by well-named, wcll-supeivised inicivieweis Oihei appioachcs
include goal-free meihods, heavy inteiviewing with consumers and other
stakeholders, and in all of ilus. applying quality control procedures such as
audioiape backups Paieniheiically. a fine example ol an evaluation using such
approaches in a icsponsive evaluation frame wot k is now available (Slake.
Davis, and Guynii, 1997)
Chehmsky (1997) is pragmatic about methods for achieving inclusive-
ness Though considerably less daunted than Scriven about being captured,
subdued. 01 co-opted by silling down with stakeholders, she would be
highly on hei guaid against efloits to coeice evaluators or otherwise place
them in an advocacy or adversary position (For example, being set up as a
Congressional pit-bull chomping on a possibly effective but out-of-favoi pio-
gram such as WIG would be as threatening to the GAO's independence and
credibility as being cast as a slull for a possibly ineffective but popular pro-
gram such as chemical wai fare )
Are Greene and Meitens talking about diffeient types of progiams ihan
Senven and Chelnnsky. and thus ihe appaiem disagieements aie a case ol "It
depends"7 Lvaluaiois vary m ihe ease of public access to evaluations they have
completed. 01 in how closely ancboied then discussions ol advocacy and neu-
trality are in specilic woik li seems likely that positions lecommendmg close-
ness and mclusiveness are moic feasible with fauly small-scale evaluations.
peihaps on local 01 si.de levels Tot example, one could bung out all stake-
holders' voices lanly in a small piogiam. such as a local Hospice Center, an
individual school, or even a county-wide lecyclmg program
In coniiast. although u is easy to envision mclusiveness in a national eval-
uation, it is moie dillicuh to imagine one-on-one closeness As an example, the
fust issue of the new Head Stan journal aimed at promoting reseaichei-
evaluaior and piacimonei dialogue Ionises on stakeholder collaboration and
paiiicipation, but includes examples only fiom small studies, not the many
national evaluations Howevei. some ledeial agencies now are wining Requests
for Pioposals (RIH's) consistent with empowerment and participatory views
(such as the lluieau of Indian Affairs), so in the future, we may be able to see
more empincally ihe tianspoilability of the inclusive, close, paruupaioiy
appioaih
Peisonally, I would like 10 icad. in lull, an evaluation someone has com-
pleted (seveial. if possible) as a way ol seeing what diffeience the pimciples
make in piactite and wlieie, il any place, "it depends " We might be somewhat
lanhei along il such evaluations weie easily available as companion pieces to
die moic ihcoieiieal ailnles House (1995) wisely wrote, "It is difficult to wine
mielligently about ethics and values One icason is that ethical problems aie
mamlesied -n paiticulai loncieic cases and endoisement of general pim-
uplrs some _s sirms platitudinous 01 inelevam lillncal concerns become
III! I IIIIC SOI liVAIUAIION NllllltAIIIY AND AllVix New Dnrciioiib loi hiigiam I v.iliuiioii ,„,
66 San rianciscn Josscy-Hass. I«J95
Hell. S "Cullinga Non-l'.ims.in Lvjluaiiun in .1 r.inib.ui Woild Ihe Uih.m IIIMIIUU Niw
I'edtiahsm l:valu.ilion " I'.ipcr piesemed .11 (he Aim-man I v.iliuiinii ,\y>oi I.UI..M ( on
ference, San Diego, Niivemlier IW7
Ilicknun, 1. •Implications ol die Ion Biagg l.v.ilii.mon ' / I',I(II,KI,>II 1'i.nn., IWo //
( lieliinsky. l: "Ilie Pnliik.il I nviioniiieni of I v.ilu.iiion .mil Wli.n h MI.IM:,!,,! id, |i,i-,|
opineni ol ihe Field " In I. c.helimbky and W Sh.idi»li (eds I / v.iln.iiinii /,» id, JIM (. „
luiy A llanJIwIt lluuisand Oaks. C .ill) baj-e. I'W7
Cook. I D "I esiiins I.ejinetl in I vjlu.uion C)ver ihe I'.IM 2") \\ .lib ' In I < In limsky ,m,l
W K Shadish (eds ), llvttlualum jtn llif 2hl Ccnluiy A fdiiuflniiik Ihiui-nid ().iks ( .ilil
Sage. 1997
n.m.i. I TufliiiK Nnn-p S.IH l.v.ilu.nions in a I'.inis.in Woild ' I'.ij M MI«| .11 ilu
Aiiiiiu.inlv.il HI AssiiiMiHiiiCoiiliiiiiii 'u-|ii. L.I M..tll,il..i l'i-p/
-------
HH HMIRC.INC. nimCAi. CIIAIIINUSIN LVAIUAIION
I ciieiman. I) "I'nipowernicnl Lvaluaiion .mil Acneiliiaiion ID Ilighei Education " In II
Chelnnsky and W K Shadish (oils ) Evaluation JIH ihc 2lsl Century A llinullnHili "I liou-
sand Oaks, Calif Sage, 1997
(ill-cue. J C Tvjlu.ilion as Advocacy " l.vuliiiiliiin 1'iuilnc, 1995,/#. 25-36
lleiuy. li I . Julncs. G . and M.uk. M M (eds) Ktahsl Lvaluiilum An /"mfijjmj; lliatiy in
iii/i/>oil (i/ I'iMlKf NewDireciioiiblorbvaliidiion.no 78 San Francisco jossey-Uass.
1998
I IOUM;. II K . anil I lowe, K K "Hie Issue ol Advocacy in Evaluations " Antfiiuin Jiiuinul iif
Lvulutiiiiiii 1998. /'J. 233-236
House, L R Tinuiplcd I valuation A CnlU|iie oflhe ACA Guiding I'muiples" In W K
Shadish. D I. Newman. M A Sclii-ner. and C Wye (eils ), New UiiulKim/cii L'vuliiulinii.
66 San Franc isio |ossey-Bass, 1995
Lincoln. Y 'Hie Making ol a Consiruciivist A Remembrance of transformations I'asi " In
E Cuba. I he PaiaJiyn Unity I housaiul Oaks. Calif Sage. 1990
Meilens. I) "Identifying and Resigning Dtlfeiences Among Participants in Evaluation Stud-
ies " In W R Shadish. D Newman. M A Scheirer. and C Wye (eds ). Guiding Pnmiplers
foi r.vuliuilois New Dneciions lor Piogiam Evaluation, no 66 San Francisco Jossey-
Bass. 1995
Moms, M (Chair) "Wliai's an ["valuator 10 Do' Confronting Ethical Dilemmas in Practice "
Session piesented at the American ('valuation Association Confetence. San Diego,
November 1997
Nelkm. V S (Chair) T.ihiial Dilemmas in (.valuation " Session presented at tbe Ameiican
Evaluation Association Conlereiue. Vancouver, I) C , November 1995
Rossi, P II "Doing Good and Gelling It Right " In W K Shadtsh. D Newman, M A
Scheirer, and C Wye (eds ). Guiding I'umiplrsjoi Cvulnulnis New Directions for Prngiam
Lvaluaiion.no 66 San l:iancisco Jossey-Bass, 1995
Kyan, K . Gieene.) , Lincoln. Y , Mallnson, S . and Meilens, D M "Advantages and Disad-
vantages of Using Inclusive I.valuation Appioaches in Evaluation Piaclice " Amenutn Jinn-
nut o/ Emiiuuiiim. 1998, 19. 101-122
bcnven, M "Tiulh and Objectivity in Evaluation " In I: Chelunsky and W Shadish (eds ),
Evdluulion/or the 2Js( Ctiituiy A llanJIntuli Ihousand Oaks. Calif Sage. 1997
Slake. R E . Davis, R . and Guynn. S fc'vuluulion ofReadei FotuscJ VVniiiix/oi tl\e Veterans
Hfiif/ili Ailniiiiisiiuiion Champaign. Illinois CIRCE at the Univeistly of Illinois. 1997
United States General Accounlin|> Office Ain/Hiiig blumfuidi Washington. D C Author.
1997
Wtl»lei'i Encyilo|ifi/ii Unabiulyd Dulumttiy n/ the F.n^h Lunffiagf New York Gramercy
Hooks. 1994
s-i 11 IN DAI IA, /m-iiJfiil oj Dallu Aitdl^iis. is a />tiil pitsulent of the Evaluation
Rexanh Sttiicty ami ifci/nciil oj (lit- Alvd ami Gunnai MyiJal Awaidfoi livahm-
Mun in Goveinmcnl Set vu t ami oj ihe Ralwl Ingpl Awaidfot Exliaoidinary Sewtce
to (lie AmriiiuH Evaluation /\iSiiti(ilio;i Mir Juis been dneclor oj lestanli and eval-
uation /o/ Pmjecl Head Stait and ihc Childicn* Buieau. diiectHi fat («i<.hmg. leam-
in£. and dis^iwncn( at the National Inhume of llduialton. and Dnectoi }m Pioyam
'Ilieie ha* been little diuussion in evaluation litciatuie in (lie United
Stale* of ethical liiues in condui ting evaluation in mfcnuilioiKil
stKin^i Although many of the Mine ethical issues «mse wheievci the
evaluation is iondmted, two sets of ethical issiirs that ate i>aiticulaily
important in developing countne* umuin how Maheholden should be,
mvulvea\md to what extent the evuluator should leywct Imul custoi?^
and values
\
\
Ethical Issiks in Conducting
Evaluation ^International Bettings
Michael Bambeigci
This chapter reviews some ol llrtc eihicalXsucs iclctiiiliccl in evaliiiiitoiis in ilic
Unued Slaies and considers ihe Sjmilafiucs and diffeienees in application of
these issues in developing couninesvlnilso identities a nutnbei ol ethical issues
ansing in international evaluationsytat aie less common in U S domestic eval-
uations These concern the role/if uJlei national agencies in linaiiung. pro-
moting, and conducting evaluation ift developing commies and how the
political, economic, and cultural charactVistiLS of developing couiitties affect
evaluation practice We reftVTrequenily uAhe Joint Committees I'K^KIIII Eval-
uation Standutds and ihe Ainencan EvaluaiVin Association's (AHA) "Guiding
Principles for Evaluauon" as illustiationsXof how U S evaluaiois have-
approached issues reUtmg to professional eval^ialion standaids and lo show
how these approaches have been viewed fiom rhe pcispecitve of clevclo|)ing
countries
Ethical Issues in International Evaluation
Ihisihamer is concerned pnmaiily with evaluaitoiib iri developing l
that are/Conducted by. or sponsoied by. multilateial crvvi-lopmeiu agtiuu-s
(WoiUf Bank. UNICEF. InterAinencan Development UankX fot i-xampli). lnl.n-
etaUlevelopmeni agencies (USAID, CIDA). iiiieinaiiona\iiongovriniiiciu.il
o/gamzalions (NGOs) (OXPAM. CARL. Win Id Vision). an\J Noith Amciican
,<6r European-based iese.iK.li institutes (umveisities) rvaluait
-------
Program Evaluation Standards hrtp '.'www eval org. c .equationDocuments/proeeval html
The Program Evaluation Standards
Summary of the Standards
Utility Standards
The utility standards are intended to ensure that an evaluation will serve the
information needs of intended users.
Ul Stakeholder Identification—Persons involved in or affected by the evaluation
should be identified, so that their needs can be addressed.
U2 Evaluator Credibility—The persons conducting the evaluation should be both
trustworthy and competent to perform the evaluation, so that the evaluation
findings achieve maximum credibility and acceptance.
U3 Information Scope and Selection—Information collected should be broadly
selected to address pertinent questions about the program and be responsive to
the needs and interests of clients and other specified stakeholders.
U4 Values Identification—The perspectives, procedures, and rationale used to
interpret the findings should be carefully described, so that the bases for value
judgments are clear.
U5 Report Clarity—Evaluation reports should clearly describe the program being
evaluated, including its context, and the purposes, procedures, and findings of the
evaluation, so that essential information is provided and easily understood.
U6 Report Timeliness and Dissemination—Significant interim findings and
evaluation reports should be disseminated to intended users, so that they can be
used in a timely fashion.
U7 Evaluation Impact—Evaluations should be planned, conducted, and reported in
ways that encourage follow-through by stakeholders, so that the likelihood that
the evaluation will be used is increased.
Feasibility Standards
The feasibility standards are intended to ensure that an evaluation will be realistic,
prudent, diplomatic, and frugal.
Fl Practical Procedures—The evaluation procedures should be practical, to keep
disruption to a minimum while needed information is obtained.
F2 Political Viability—The evaluation should be planned and conducted with
anticipation of the different positions of various interest groups, so that their
I of 4 1/4/01 10 35 AiV
-------
Program Evaluation Standards http /Avww evaJ.ora- Evaluation Documents, progeval html
cooperation may be obtained, and so that possible attempts by any of these
groups to curtail evaluation operations or to bias or misapply the results can be
averted or counteracted.
F3 Cost Effectiveness—The evaluation should be efficient and produce information
of sufficient value, so that the resources expended can be justified.
Propriety Standards
The propriety standards are intended to ensure that an evaluation will be
conducted legally, ethically, and with due regard for the welfare of those involved
in the evaluation, as well as those affected by its results.
PI Service Orientation-Evaluations should be designed to assist organizations to
address and effectively serve the needs of the full range of targeted participants.
P2 Formal Agreements—Obligations of the formal parties to an evaluation (what is
to be done, how, by whom, when) should be agreed to in writing, so that these
parties are obligated to adhere to all conditions of the agreement or formally to
renegotiate it.
P3 Rights of Human Subjects—Evaluations should be designed and conducted to
respect and protect the rights and welfare of human subjects.
P4 Human Interactions—Evaluators should respect human dignity and worth in
their interactions with other persons associated with an evaluation, so that
participants are not threatened or harmed.
P5 Complete and Fair Assessment—The evaluation should be complete and fair in
its examination and recording of strengths and weaknesses of the program being
evaluated, so that strengths can be built upon and problem areas addressed.
P6 Disclosure of Findings—The formal parties to an evaluation should ensure that
the full set of evaluation findings along with pertinent limitations are made
accessible to the persons affected by the evaluation, and any others with
expressed legal rights to receive the results.
P7 Conflict of Interest-Conflict of interest should be dealt with openly and
honestly, so that it does not compromise the evaluation processes and results.
P8 Fiscal Responsibility-The evaluator's allocation and expenditure of resources
should reflect sound accountability procedures and otherwise be prudent and
ethically responsible, so that expenditu res are accounted for and appropriate.
Accuracy Standards
The accuracy standards are intended to ensure that an evaluation will reveal and
2 Of 4 1/4/01 1035AM
-------
Program Evaluation Standards hrtp '/www eval org'EvaluationDocuments.progeval html
convey technically adequate information about the features that determine worth
or merit of the program being evaluated.
Al Program Documentation—The program being evaluated should be described
and documented clearly and accurately, so that the program is clearly identified.
A2 Context Analysis—The context in which the program exists should be examined
in enough detail, so that its likely influences on the program can be identified.
A3 Described Purposes and Procedures—The purposes and procedures of the
evaluation should be monitored and described in enough detail, so that they can
be identified and assessed.
A4 Defensible Information Sources—The sources of information used in a program
evaluation should be described in enough detail, so that the adequacy of the
information can be assessed.
A5 Valid Information—The information gathering procedures should be chosen or
developed and then implemented so that they will assure that the interpretation
arrived at is valid for the intended use.
A6 Reliable Information—The information gathering procedures should be chosen
or developed and then implemented so that they will assure that the information
obtained is sufficiently reliable for the intended use.
A7 Systematic Information—The information collected, processed, and reported in
an evaluation should be systematically reviewed and any errors found should be
corrected.
A8 Analysis of Quantitative Information—Quantitative information in an evaluation
should be appropriately and systematically analyzed so that evaluation questions
are effectively answered.
A9 Analysis of Qualitative Information—Qualitative information in an evaluation
should be appropriately and systematically analyzed so that evaluation questions
are effectively answered.
A10 Justified Conclusions—The conclusions reached in an evaluation should be
explicitly justified, so that stakeholders can assess them.
All Impartial Reporting—Reporting procedures should guard against distortion
caused by personal feelings and biases of any party to the evaluation, so that
evaluation reports fairly reflect the evaluation findings.
A12 Metaevaluation-The evaluation itself should be formatively and summatively
evaluated agajnst these and other pertinent standards, so that its conduct is
appropriately guided and, on completion, stakeholders can closely examine its
3 of 4 1/4/01 1035AM
-------
Program Evaluation Standards
httpv/www.eval org/EvaluationDocuments,progeval htm
strengths and weaknesses.
Prepared by:
Mary E. Ramlow
The Evaluation Center
401B Ellsworth Hall
Western Michigan University
Kalamazoo, MI 49008-5178
Phone: 616-387-5895
Fax: 616-387-5923
Email: Marv.Ramlow(a)wmich.edu
4 of 4
1/4/01 10.35 A.\
-------
Guiding Principles for Evaluators htrp .'/www eval org/EvaluationDocuments-aeaprin6 hrmi
Guiding Principles for Evaluators
A Report from the AEA Task Force on
Guiding Principles for Evaluators
Members of the Task Force
Dianna Newman, University of Albany/SUNY
Mary Ann Scheirer, Private Practice
William Shadish, Memphis State University (Chair),
w.shadish@mail.Dsvc.memDhis.edu
Chris Wye, National Academy of Public Administration
I. Introduction
A. Background: In 1986, the Evaluation Network (ENet) and the
Evaluation Research Society (ERS) merged to create the American
Evaluation Association. ERS had previously adopted a set of standards
for program evaluation (published in New Directions for Program
Evaluation in 1982); and both organizations had lent support to work of
other organizations about evaluation guidelines. However, none of these
standards or guidelines were officially adopted by AEA, nor were any
other ethics, standards, or guiding principles put into place. Over the
ensuing years, the need for such guiding principles has been discussed
by both the AEA Board and the AEA membership. Under the presidency
of David Cordray in 1992, the AEA Board appointed a temporary
committee chaired by Peter Rossi to examine whether AEA should
address this matter in more detail. That committee issued a report to the
AEA Board on November 4, 1992, recommending that AEA should pursue
this matter further. The Board followed that recommendation, and on
that date created a Task Force to develop a draft of guiding principles for
evaluators. The AEA Board specifically instructed the Task Force to
develop general guiding principles rather than specific standards of
practice. This report summarizes the Task Force's response to the
charge.
B. Process: Task Force members reviewed relevant documents from
other professional societies, and then independently prepared and
circulated drafts of material for use in this report. Initial and subsequent
drafts (compiled by the Task Force chair) were discussed during
conference calls, with revisions occurring after each call. Progress
reports were presented at every AEA board meeting during 1993. In
addition, a draft of the guidelines was mailed to all AEA members in
September 1993 requesting feedback; and three symposia at the 1993
AEA annual conference were used to discuss and obtain further
feedback. The Task Force considered all this feedback in a December
1993 conference call, and prepared a final draft in January 1994. This
1 of 9 1/4/01 10-35 A.V
-------
Guiding Principles for Evaluators httpV/^ww eval org/EvaluationDocuments.aeapnn6 html
draft was presented and approved for membership vote at the January
1994 AEA board meeting.
C. Resulting Principles: Given the diversity of interests and employment
settings represented on the Task Force, it is noteworthy that Task Force
members reached substantial agreement about the following five
principles. The order of these principles does not imply priority among
them; priority will vary by situation and evaluator role.
1. Systematic Inquiry: Evaluators conduct systematic,
data-based inquiries about whatever is being evaluated.
2. Competence: Evaluators provide competent performance to
stakeholders.
3. Inteahtv/Honestv: Evaluators ensure the honesty and
integrity of the entire evaluation process.
4. Respect for People: Evaluators respect the security, dignity
and self-worth of the respondents, program participants,
clients, and other stakeholders with whom they interact.
5. Responsibilities for General and Public Welfare: Evaluators
articulate and take into account the diversity of interests and
values that may be related to the general and public welfare.
These five principles are elaborated in Section III of this
document.
D. Recommendation for Continued Work: The Task Force also
recommends that the AEA Board establish and support a mechanism for
the continued development and dissemination of these Guiding
Principles.
II. Preface: Assumptions Concerning Development of Principles
A. Evaluation is a profession composed of persons with varying interests,
potentially encompassing but not limited to the evaluation of programs,
products, personnel, policy, performance, proposals, technology,
research, theory, and even of evaluation itself. These principles are
broadly intended to cover all kinds of evaluation.
B. Based on differences in training, experience, and work settings, the
profession of evaluation encompasses diverse perceptions about the
primary purpose of evaluation. These include but are not limited to the
following: bettering products, personnel, programs, organizations,
governments, consumers and the public interest; contributing to
2 of 9 1/4/01 10-35 AN
-------
Guiding Principles for Evaluators hnp //www eval org/Evaluation Documents. aeaprm6 html
informed decision making and more enlightened change; precipitating
needed change; empowering,all stakeholders by collecting data from
them and engaging them in the evaluation process; and experiencing the
excitement of new insights. Despite that diversity, the common ground is
that evaluators aspire to construct and provide the best possible
information that might bear on the value of whatever is being evaluated.
The principles are intended to foster that primary aim.
C. The intention of the Task Force was to articulate a set of principles
that should guide the professional practice of evaluators, and that should
inform evaluation clients and the general public about the principles they
can expect to be upheld by professional evaluators. Of course, no
statement of principles can anticipate all situations that arise in the
practice of evaluation. However, principles are not just guidelines for
reaction when something goes wrong or when a dilemma is found.
Rather, principles should proactively guide the behaviors of professionals
in everyday practice.
D. The purpose of documenting guiding principles is to foster continuing
development of the profession of evaluation, and the socialization of its
members. The principles are meant to stimulate discussion and to
provide a language for dialogue about the proper practice and application
of evaluation among members of the profession, sponsors of evaluation,
and others interested in evaluation.
E. The five principles proposed in this document are not independent,
but overlap in many ways. Conversely, sometimes these principles will
conflict, so that evaluators will have to choose among them. At such
times evaluators must use their own values and knowledge of the setting
to determine the appropriate response. Whenever a course of action is
unclear, evaluators should solicit the advice of fellow evaluators about
how to resolve the problem before deciding how to proceed.
F. These principles are intended to replace any previous work on
standards, principles, or ethics adopted by ERS or ENet, the two
predecessor organizations to AEA. These principles are the official
position of AEA on these matters.
G. Each principle is illustrated by a number of statements to amplify the
meaning of the overarching principle, and to provide guidance for its
application. These statements are illustrations. They are not meant to
include all possible applications of that principle, nor to be viewed as
rules that provide the basis for sanctioning violators.
H. These principles are not intended to be or to replace standards
supported by evaluators or by the other disciplines in which evaluators
participate. Specifically, AEA supports the effort to develop standards for
3 of 9 1/4/01 I035AV
-------
Guiding Principles for Evaluators http-.'/www eval on>'EvaluationDocuments/aeapnn6.html
educational evaluation by the Joint Committee on Standards for
Educational Evaluation, of which AEA is a cosponsor.
I. These principles were developed in the context of Western cultures,
particularly the United States, and so may reflect the experiences of that
context. The relevance of these principles may vary across other
cultures, and across subcultures within the United States.
J. These principles are part of an evolving process of self-examination by
the profession, and should be revisited on a regular basis. Mechanisms
might include officially-sponsored reviews of principles at annual
meetings, and other forums for harvesting experience with the principles
and their application. On a regular basis, but at least every five years
from the date they intially take effect, these principles ought to be
examined for possible review and revision. In order to maintain
association-wide awareness and relevance, all AEA members are
encouraged to participate in this process.
III. The Principles
A. Systematic Inquiry: Evaluators conduct systematic, data-based
inquiries about whatever is being evaluated.
1. Evaluators should adhere to the highest appropriate
technical standards in conducting their work, whether that
work is quantitative or qualitative in nature, so as to increase
the accuracy and credibility of the evaluative information they
produce.
2. Evaluators should explore with the client the shortcomings
and strengths both of the various evaluation questions it might
be productive to ask, and the various approaches that might
be used for answering those questions.
3. When presenting their work, evaluators should communicate
their methods and approaches accurately and in sufficient
detail to allow others to understand, interpret and critique their
work. They should make clear the limitations of an evaluation
and its results. Evaluators should discuss in a contextually
appropriate way those values, assumptions, theories, methods,
results, and analyses that significantly affect the interpretation
of the evaluative findings. These statements apply to all
aspects of the evaluation, from its initial conceptualization to
the eventual use of findings.
B. Competence: Evaluators provide competent performance to
stakeholders.
4 of 9 1/4/01 10:35 AM
-------
Guiding Principles for Evaluators http '/www eval org'EvaluaiionDocumems
-------
Guiding Principles for Evaluators http "www eval org/EvaluanonDocuments.aeaprm6 hrnii
interests).
4. Evaluators should disclose any roles or relationships they
have concerning whatever is being evaluated that might pose a
significant conflict of interest with their role as an evaluator.
Any such conflict should be mentioned in reports of the
evaluation results.
5. Evaluators should not misrepresent their procedures, data
or findings. Within reasonable limits, they should attempt to
prevent or correct any substantial misuses of their work by
others.
6. If evaluators determine that certain procedures or activities
seem likely to produce misleading evaluative information or
conclusions, they have the responsibility to communicate their
concerns, and the reasons for them, to the client (the one who
funds or requests the evaluation). If discussions with the client
do not resolve these concerns, so that a misleading evaluation
is then implemented, the evaluator may legitimately decline to
conduct the evaluation if that is feasible and appropriate. If
not, the evaluator should consult colleagues or relevant
stakeholders about other proper ways to proceed (options
might include, but are not limited to, discussions at a higher
level, a dissenting cover letter or appendix, or refusal to sign
the final document).
7. Barring compelling reason to the contrary, evaluators should
disclose all sources of financial support for an evaluation, and
the source of the request for the evaluation.
D. Respect for People: Evaluators respect the security, dignity and
self-worth of the respondents, program participants, clients, and other
stakeholders with whom they interact.
1. Where applicable, evaluators must abide by current
professional ethics and standards regarding risks, harms, and
burdens that might be engendered to those participating in the
evaluation; regarding informed consent for participation in
evaluation; and regarding informing participants about the
scope and limits of confidentiality. Examples of such standards
include federal regulations about protection of human subjects,
or the ethical principles of such associations as the American
Anthropological Association, the American Educational
Research Association, or the American Psychological
Association. Although this principle is not intended to extend
the applicability of such ethics and standards beyond their
6 of 9 1/4/01 10-35 AN
-------
Guiding Principles for Evaluators httpVAvvvw eval org/EvaluationDocuments,aeaprm6 html
current scope, evaluators should abide by them where it is
feasible and desirable to do so.
2. Because justified negative or critical conclusions from an
evaluation must be explicitly stated, evaluations sometimes
produce results that harm client or stakeholder interests.
Under this circumstance, evaluators should seek to maximize
the benefits and reduce any unnecessary harms that might
occur, provided this will not compromise the integrity of the
evaluation findings. Evaluators should carefully judge when the
benefits from doing the evaluation or in performing certain
evaluation procedures should be foregone because of the risks
or harms. Where possible, these issues should be anticipated
during the negotiation of the evaluation.
3. Knowing that evaluations often will negatively affect the
interests of some stakeholders, evaluators should conduct the
evaluation and communicate its results in a way that clearly
respects the stakeholders' dignity and self-worth.
4. Where feasible, evaluators should attempt to foster the
social equity of the evaluation, so that those who give to the
evaluation can receive some benefits in return. For example,
evaluators should seek to ensure that those who bear the
burdens of contributing data and incurring any risks are doing
so willingly, and that they have full knowledge of, and
maximum feasible opportunity to obtain any benefits that may
be produced from the evaluation. When it would not endanger
the integrity of the evaluation, respondents or program
participants should be informed if and how they can receive
services to which they are otherwise entitled without
participating in the evaluation.
5. Evaluators have the responsibility to identify and respect
differences among participants, such as differences in their
culture, religion, gender, disability, age, sexual orientation and
ethnicity, and to be mindful of potential implications of these
differences when planning, conducting, analyzing, and
reporting their evaluations.
E. Responsibilities for General and Public Welfare: Evaluators articulate
and take into account the diversity of interests and values that may be
related to the general and public welfare.
1. When planning and reporting evaluations, evaluators should
consider including important perspectives and interests of the
full range of stakeholders in the object being evaluated.
7 of 9 1/4/01 10 35 AN
-------
Guiding Principles for Evaluators http /www eval.org/EvaluationDocuments/aeaprin6.html
Evaluators should carefully consider the justification when
omitting important value perspectives or the views of
important groups.
2. Evaluators should consider not only the immediate
operations and outcomes of whatever is being evaluated, but
also the broad assumptions, implications and potential side
effects of it.
3. Freedom of information is essential in a democracy. Hence,
barring compelling reason to the contrary, evaluators should
allow all relevant stakeholders to have access to evaluative
information, and should actively disseminate that information
to stakeholders if resources allow. If different evaluation
results are communicated in forms that are tailored to the
interests of different stakeholders, those communications
should ensure that each stakeholder group is aware of the
existence of the other communications. Communications that
are tailored to a given stakeholder should always include all
important results that may bear on interests of that
stakeholder. In all cases, evaluators should strive to present
results as clearly and simply as accuracy allows so that clients
and other stakeholders can easily understand the evaluation
process and results.
4. Evaluators should maintain a balance between client needs
and other needs. Evaluators necessarily have a special
relationship with the client who funds or requests the
evaluation. By virtue of that relationship, evaluators must
strive to meet legitimate client needs whenever it is feasible
and appropriate to do so. However, that relationship can also
place evaluators in difficult dilemmas when client interests
conflict with other interests, or when client interests conflict
with the obligation of evaluators for systematic inquiry,
competence, integrity, and respect for people. In these cases,
evaluators should explicitly identify and discuss the conflicts
with the client and relevant stakeholders, resolve them when
possible, determine whether continued work on the evaluation
is advisable if the conflicts cannot be resolved, and make clear
any significant limitations on the evaluation that might result if
the conflict is not resolved.
5. Evaluators have obligations that encompass the public
interest and good. These obligations are especially important
when evaluators are supported by publicly-generated funds;
but clear threats to the public good should never be ignored in
any evaluation. Because the public interest and good are rarely
8 of 9 1/4/01 10-35 AiV
-------
Guiding Principles for Evaluators http //www eval.org/EvaluationDocuments/aeaprm6 html
the same as the interests of any particular group (including
those of the client or funding agency), evaluators will usually
have to go beyond an analysis of particular stakeholder
interests when considering the welfare of society as a whole.
9 Of 9 1/4/01 1035 AN
-------
WHAT COMES TO MIND WHEN YOU HEAR THE WORD...
process
ma [or- Activities
purpose
worc|s to
Describe
Evaluation
Auditing
Investigation
Research
Monitoring
Assessment
-------
EVALUATION AND AUDITING
o 71
Refer to articles in- Carl Wisler, ecj. C1996) New Directions for Program Evaluation
San Francisco CA Jossey-Bass Publishers
Editor's Notes, Carl Wisler
Divorski, Stan "Differences in the Approaches of Auditors and Evaluators to the
Examination of Government Policies and Programs"
Chelimsky, Eleanor "Auditing and Evaluation Whither the Relationship/"
"Performance auditing is an objective and systematic examination of evidence for the
purpose of providing an independent assessment of the performance of a government
organization, program, activity, or function in order to provide information to improve
public accountability and facilitate decision-making by parties with responsibility to oversee
or initiate corrective action " Comptroller General of the L/S, 1994-
Evaluation
Auditing
New C1960's)
Offshoot of Social Sciences concerned with
theory and explanation
Less precise, more intellectually stimulating
Multiple audiences
Criteria selection flexible
Cooperative, interactive relationships with
evaluees
Examines a question of "why" (what will
produce desired/undesired effects')
Other
Older profession (stemming from
accounting, bookkeeping)
Verification of authoritative documents
Single client
Fixed criteria (comparing 'what is' to 'what
should be')
Examines a question of "whaf'Cdoes what
was done conform to standards/)
End result is an opinion- Is observed
performance consistent with accepted
norms/
Other.
-------
EDITOR'S NOTES
During the last several decades, the disciplines of evaluation and auditing
have each gone through substantial change The purpose of this volume is not
to explore all the reasons and consequences associated with such change, but
to focus on the extent to which audit and evaluation have converged on simi-
lar procedures and organizational structures and on the extent to which they
have remained different An effort has been made to understand and describe
the issues from both evaluation and auditing perspectives
Both evaluation and auditing claim to help decision makers by providing
them with systematic and credible information that can be useful in the cre-
ation, management, oversight, change, and occasionally abolishment of pro-
grams Yet despite considerable overlap in objectives, subject matter, and
clients, auditing and evaluation have until recently functioned largely in isola-
tion from one another. The literature of each discipline scarcely recognizes the
existence of the other Academic preparation of auditors and evaluators could
hardly be more different Organizationally, the two activities have traditionally
been separate The practitioners have difficulty communicating with one
another not only because of differences in vocabulary but also because of some
important differences in mind-set Will auditing and evaluation persist as dis-
tinctive services to decision makers or is there a possibility of a merge or blend
of the two activities?
Differences between auditing and evaluation are rooted in the older pro-
fessions from which they emerged Auditing evolved from financial account-
ing and so makes much use of concepts like verification, internal controls, and
good management practice. Evaluation emerged from the sciences, especially
social science, and so has tended to carry with it the trappings of measurement,
probability sampling, and experimentation. The authors represented in this
volume make frequent reference to the conceptual underpinnings of the two
disciplines when they point out how auditing and evaluation are different
One way to get a quick sense of some differences between auditing and
evaluation is to consider the kinds of questions for which the two fields have
tned to provide answers Three categories of questions are especially useful for
comparing auditing and evaluation descriptive, normative, and cause-and-
effect (Wisler. 1984a, 1984b) These categories are explicitly or implicitly
referred to any number of times in the following chapters
Program evaluation has always given much attention to cause-and-effect
questions, especially ones about the overall impact of a program The answer
to an impact question is usually formulated as the difference between an out-
come observed after a program has been in operation with the outcome that
would have been observed in the absence of the program Evaluation clients
also seek the answers to descriptive questions—ones that do not compare two
-------
2 IJHIOR'S NOM.S
conditions bin simply desuibc a stale ol ilic world Common examples include
questions about tbc societal needs, the selling in which a program operates, or
the way a progiam was implemented
Auditing has traditionally focused on normative questions for which the
answer compares "what is" with "what should be " Seldom do auditors seek
answers to descriptive questions nor do they often consider cause-and-effect
questions, at least m the sense that evaluators understand that term
One of the most interesting differences in disciplinary perspective is the
coniiasi between program audit, which usually starts with a normative ques-
tion, and impact evaluation, which focuses on cause and effect A brief
overview of these two forms of inquiry is illuminating because practitioners
from each discipline claim that their respective approaches address the issue
of program effectiveness However, the similarity seems to end there Method-
ologically, they are different because in fact they address two distinct questions
about program effectiveness Although the two methodological approaches can
be distinguished easily, it is probable that most clients, and at least some prac-
titioners as well, do not lully appreciate the important differences
As defined by the Comptroller General of the United States (1994, p 14),
peijoimance aud\l is "an objective and systematic examination of evidence
of the performance of a government organization, program, activity, or func-
tion in order to provide information to improve public accountability and facil-
itate decision-making " A piogiam audit is a subcatcgory of a performance audit
for which one objective (of three) is to determine "the extent to which the
desired results or benefits established by the legislature or other authorizing
body aie being achieved " This application of program audit provides an inter-
esting contrast with impact evaluation (The scope of auditing is broad, of
course, and the piactitioners may employ objectives and methodologies dif-
ferent from those reviewed here The notion of value-for-money audits, used
in many countries, is similar to the concept of performance auditing )
Schandl (1978, p 4) says "auditing is a human evaluation process to
establish the adherence to ccitam norms, resulting in an opinion (or judg-
ment) " I lerbeit (1979) describes developments at the U S General Account-
ing Office in the 1960s that led to a prescription for management and program
audits, those prescriptions are comparable to the general concept set forth by
Schandl regarding how to undertake management and program audits The
central idea in a program audit is to compare "what is" with "what should be"
(Compunllci General of the United Slates. 1979 |1974|) This notion seems
to How lioin earlier forms of auditing in which generally accepted accounting
puuiics 01, mote bioadly, generally accepted management practices played
key mlcs The methodology was to establish a standard generally accepted
management piaciicc, lor example, and to compete that standaicl with actual
piogiam piauiLC Any discovery of a scuous discrepancy would be regaided
as a deliLii— •• and lead to a negative audit icport
In n o broader issues of program effectiveness, auditors look along
the noun.. question To conduct a proginm audit, it was therefore nei.cs-
ni>noR's NOILS
sary to identify one or more specific program objectives to piny the role of
"what should be " Such objectives might come from legislation, regulations,
declarations of intent by program managers, and so on Actual program per-
formance, determined empirically, provided the "what is" component And
program effect was defined as the difference between the program objective
and actual performance (In auditing, an effect is sometimes defined differently
to mean the consequences of a discrepancy For example, if a program to
improve water quality showed a shortfall in the achievement of water pumy
standards, the effects might be greater incidence of disease, lost time at work
due to illness, and so on )
The audit approach to program effectiveness is in the spirit of the strat-
egy advocated in the problem-solving literature as exemplified by Kepner and
Tregoe (1976) It is also very much in the mode of objectives-oriented evalu-
ation approaches (see Chapter 5 of Worthen and Sanders, 1987) sometimes
used with educational programs — and especially of the version called dis-
crepancy evaluation (Provus, 1971) It does not correspond, however, to the
conventional notion of impact evaluation
Program impact evaluation stems from the experimental design used in a
number of sciences wherein compansons are made between outcomes associ-
ated with randomly assigned treatment and control groups In evaluation, the
methodology is generally understood not to require random assignment but 10
extend to other approaches that permit compansons between what happened
in the presence of the program and what would have happened in (he absence
of the program Such quasi-experimental designs are prominent in evaluation
as ways to answer impact questions (Cook and Campbell, 1979) (Auditors and
others new to the evaluation literature have to contend with the variety of terms
that may be used interchangeably with impact evaluation, most notably impact
assessment, impact analysis, and program effectiveness evaluation )
The two questions posed by auditors and evaluators about program effec-
tiveness can, and generally will, lead to quite different conclusions about the
performance of a program Both conclusions may be correct — they are just
answers to different questions Unfortunately, because of the language used,
the unwary may perceive the questions to be the same
The foregoing companson of program audit and impact evaluation illus-
trates the purpose of this volume, not to give a full-featured account of the
heartlands of evaluation and auditing, but more 10 focus on ihe territory where
they come together. The authors of the five chapters of this volume have all
been in positions to survey the terrain thai is roamed boih by bands of evalu-
ators and of auditors, and they offer their views on the jointly occupied terri-
tory along with occasional references to the heartlands Earlier vcisions of the
chapters were presented at a session of the International Evaluation Confer-
ence held in Vancouver, Bntish Columbia, November 1-5, 1995
Stan Divorski, from his vantage poini with the Oflicc of (he 'or Gen-
eral in Canada, sets forth five key dimensions that he finds d nsh the
mmrl rpic nf
i unr< fmiTi
Pivmr Rinnl/c
Im, iwisnri live 111 I hi'
-------
4 EDITOR'S NOTES
Minnesota Office of Legislative Auditor to consider the extent to which the
auditing and evaluation cultures have been blended, or at least have that
potential if blending is desired Christopher Pollitt and Hilkka Summa bring
their experiences with audit institutions m the United Kingdom and Finland,
respectively, to bear in comparing the ways m which auditors and evaluators
approach similar tasks Frans L Leeuw from the Netherlands Court of Audit
focuses on the contributions of evaluators and auditors to the improvement of
public sector performance and, in so doing, draws attention to a slice of the
international literature companng the two fields Finally, Eleanor Chelimsky,
formerly of the U S General Accounting Office, highlights the conclusions of
the other authors, based on their conference papers, and offers her own views
on the pros and cons of integrating auditing and evaluation or keeping them
as separate services to decision makers.
Collectively, the authors point out many similarities and differences between
auditing and evaluation Indeed, there is such vanety in the observations that it
is difficult to categorize them and gauge their importance If, as seems to be the
case, evaluation and auditing are moving closer together, what are the differences
that seem most likely to hold them apart? Readers may wish to consider the fol-
lowing three themes and conjectures as they read this volume
The inclination of auditors toward normative questions and of evaluators toward
descriptive and impact questions The difference is rooted in the history of the
disciplines and in the educational preparation of the practitioners Although
examples of crossover between the disciplines can be cited, a broad-scale mix
of the approaches seems likely to lake a long time, if it ever occurs
Independence versus collaboration with the subjects of audit and evaluation
Auditors have attached great importance to their independence from both
client and audilee, while evaluators have tended to work more closely with
their clients and to move toward yet greater collaboration with evaluees Even
if the tenets of fourth-generation evaluation are not adopted by most evalua-
tors, mainstream evaluators seem unlikely to return to the extremes of scien-
tific detachment that once prevailed The two disciplines therefore seem
destined to be some distance apart on the scale of independence
Di/ferences m the degree to which auditing and evaluation have become rou-
tinized government operations The role of auditing in government activities has
been generally acknowledged with legislative mandates, clearly identified
clients, and organizational permanence The shorter history of public program
evaluation reveals less widespread acceptance, more variability and multiplic-
ity of clients, and considerable fluctuation m fortune over the short term The
repercussions of these differences on achievements of the two disciplines (in
terms of program improvements, new legislation, and so on) may be hard to
sort out, but in the case of evaluation, one wonders how long the shakedown
cruise will last and how, or if, a greater stability will be achieved A continued
convergence of evaluation and auditing might be a course toward a higher-
quality, more steadfast service to decision makers
CDIIOR'S Nori-s 5
References
Comptroller General of the United Stales "Report Manual" As adapted in I. I lerben. Audit-
ing the Performance of Management Belmont. Calif Lifelime Learning. 1979 (Originally
published 1974) 6 '
Comptroller General of the United Stales Government Auditing Standards. 1994 Revision
Washington, DC US General Accounting Office. 1994
Cook, T D , and Campbell. D T Quasi-Expenmentation Skokie, III Rand McNally. 1979
Herbert. L Auditing the Performance ojManagement Belmom. Calif Lifetime Learning. 1979
Kepner.C H , and Tregoe, B B The Rational Manager (2nd ed ) Pnnceion NJ Kcpner-
Tregoe, 1976
Provus. M M Discrepancy Evaluation Berkeley. Calif McCulchan. 1971
Schandl.C W Theory of Auditing Houston. Tex Scholars. 1978
Wisler, C E "Topics in Evaluation " GAO Review. 1984a. 19-1
Wisler, C E "Topics in Evaluation " CAO Review. 1984b. 19-3
Wonhen, B R , and Sanders, J R Educational Evaluation New York Longman, 1987
CARL WISLER is an evaluation consultant with Wisln
Maryland
in Mitiliellvillf.
-------
Although audits and evaluations may have similar characteristics,
the perspectives of auditors and evaluators can be quite different
These differing perspectives are reflected in then respective treatments
of program impacts
Differences in the Approaches
of Auditors and Evaluators
to the Examination of Government
Policies and Programs
Stan Divorsfei
On any given dimension, the extent to which audit and evaluation differ is largely
a matter of degree Any characteristic thai can apply to audits will also apply to
some evaluations, somewhere, sometime Any distinguishing characteristic of
evaluations probably can be found to apply to some audit Overall, however,
audits and evaluations can be very different
This chapter attempts to describe the perspective, the mind-set, that audi-
tors bring to the examination of programs, so as to illustrate how different the
perspectives of auditors and evaluators can be Although this perspective has
us origins in the requirements of financial auditing, the focus here is on value-
for-money auditing, which includes the examination of program activities, as
well as of management systems and procedures for controlling these activities
Five key dimensions distinguish this mind-set from that of evaluators In
general, auditors.
• Make a judgment as to how adequate or inadequate are the mailers examined
• Make this judgment against a preestabhshed set of criteria
• Focus on management systems and procedures for coniiollmg program
activities rather than on the program activities themselves
The views expressed in this chapter are ihose of the author and should not br • onsiriicd a>>
representing I hose of the Auditor General of Canada
-------
8 EVAI.UAI ION AND AUDI PINO
• May avoid commenting on substantive government policies
• View information on program results (when considered at all) as evidence
of other mailers, rather than as an end in itself
Judging Against Pre-Established Criteria
The mandate for an auditors work requires a judgment about the adequacy of
the matters examined Such judgment is based on a comparison of the results
of the examination against a sei of criteria, or expectations, that the auditor
may draw from a variety of sources, including previous audits and professional
standards or guidelines
For example, in reporting an audu of The Control and Clean-up of Freshwa-
ter Pollution, ihe Auditor General of Canada slated that it "expected to find thai
the various federal components of action plans were coordinated and that the
means were m place to handle interdepartmental conflicts over policy, planning
and funding as action plans are implemented" (1993b, p 370) In addition to
general cniena of this nature, an auditor may establish more specific subcntena
The way in which a judgment is reached and expressed depends upon the
assignment the auditor receives The auditor may reach a judgment after exam-
ining directly the matters at hand or after examining the reliability of manage-
ment's assertions At limes the (wo approaches may be combined The choice
of approach may be imposed by ihe auditor, the client, or, in the case of a leg-
islated mandate, by law
An interesting example of ihe audit of penodic financial statements is that
recommended by ihe Canadian Comprehensive Auditing Foundation (CCAF)
for reporting on elfectiveness (1987) The CCAF recommends that management
make rcpresentaiions (that is, that they provide information about effectiveness
in their oigamzaiions), and that auditors provide opinions on the fairness of
ihose representations Rather than define effectiveness, the CCAF sets out twelve
attributes of effectiveness against which managers are expecied to report
1 Management direction (including clarity of objectives)
2 Continued relevance of a program
3 Appropriateness of program design
4 Achievement of intended results
5 Satisfaction of clients or stakeholders
6 Secondary impacts
7 Costs and produuiviiy
8 Responsiveness to changed circumstances
9 Financial results
10 The extent to which the organization provides an appropnate work envi-
ronment foi us employees
11 Safeguarding of assets
12 Monitoring and reporl ing of pci lormancc
DIFFERENCES IN THE APPROACHES OF AUDITORS AND EVALUATORS 9
The conditions on an audit engagement may also specify how ihe auditor is to
report the judgment In exception reporting, the auditor is required to report
any deficiencies, that is, situations that did not meet these criteria Canada's
Auditor General Act, Section 7(2), gives such a mandate
Each report of ihe Auditor General under subsection (1) shall call aiiemiun 10
anything thai he |sic| considers to be of significance and of a nature that should
be brought to the attention of the House of Commons, including any cases in
which he has observed thai
(a) accounts have not been faithfully and properly maintained or public money
has not been fully accounted for or paid, where so required by law, inio (he
consolidated revenue fund,
(b) essential records have not been maintained or the rules and proceduics
applied have been insufficient to safeguard and control public property, to
secure an effective check on the assessment, collection and proper alloca-
tion of the revenue and to ensure that expenditures have been made only
as authorized,
(c) money has been expended other than for purposes for which it was appro-
priated by Parliament,
(d) money has been expended without due regard to economy or efficiency,
(e) satisfactory procedures have not been established to measure and report ihe
effectiveness of programs, where such procedures could appropnaiely and
reasonably be implemented, or
(D money has been expended without due regard to ihe environmental effects
of those expenditures m the context of sustainable development
For example, in the audit of The Control and Clean-up of Freshwater Pollution
described earlier, ihe auditor reported instances of poor coordination and unre-
solved conflicts. "This difference in departmental objectives and program fund-
ing led to coordination problems Although the two departments agreed to
provide for a management structure to coordinate their respective programs.
llhe slructurel proved to be ineffective11 (1993b, p 374)
Alternatively, an auditor may be required to reporl on the level of comlort
or assurance (hat a third party can have regarding the management of ihe pro-
gram The CCAF model for effectiveness auditing is one example of uu assm-
ance engagement In this instance, the auditor provides assurance icgarding
management representations as to the matters at hand Another example of an
assurance mandate is provided m Part X of Canada's Financial AdminiMiuiion
Act (1991), bearing on the responsibilities of Crown Coiporanons Suhseiiion
138(1) of the act requires corporations to engage an auditor to undertake a
special examination, in which the auditor is required to "deicimine it the sys-
tems and practices referred to in paragraph I3l(l)(b) were, m the period
under examination, maintained in a manner that ptovidcd reasonable assur-
ance that they met the requirements of paragraphs 131 (2)(a) and (O "
-------
10
HVAIIIAIIUN AND AUDI I INC.
Subsection 131(2) clarifies ihm " I'hc hooks, records, systems and practices
referred lo in subsection (1) shall be kept and maintained in such manner as
will provide reasonable assurance that (c) the financial, human and physi-
cal resouiccs of the corporation and each subsidiary are managed economically
and efficiently and the operations of the corporation and each subsidiary are ear-
ned out effectively" In this instance, the inability to find an exception from (he
positive expectation would be considered a judgment in confirmation of it
In assuiance engagements, a positive judgment is reached and reported
with gieat care because (I) available methods may be incorrectly applied or
not fool pi oof, and (2) theic may be an undetected case that constitutes an
important variation fiom the expectations established
Focus on Management
The core business of an audit is the examination of management controls over
expendituie, including whether or not management has in place the systems
and procedures necessary to ensure that expenditures are made with due
regard to economy and efficiency and in compliance with existing regulations
or policies Evaluation laicly intrudes into these areas The distinction between
audit and evaluation has become blurred, however, through increasing atten-
tion by audit 10 management control over results, including program effects,
as an indication of good financial management and control
This change has been lemforced by trends in government to decrease the
emphasis on formal controls and increase the delegation of authority, accom-
panied by a gieatcr need for accountability for results on the part of managers
The mandate for audits of Crown Corporations under Canada's Financial
Administration Acl (1991) reflects this focus on managemenl, as it requires
attention to management systems and practices Of particular significance to a
comparison of auditors and evalualors, the act specifically requires the auditor
to consider managemenl control over the effectiveness of program operations
The approach recommended by the Canadian Comprehensive Auditing
Foundation (CCAP) for auditing effectiveness also reflects the auditor's focus
on improving managemenl The CCAF notes that "The decision to emphasize
management icpiescntations reflects the reporting obligations of managers, the
needs of governing bodies, and their mutual desire for better management"
(1991. p 9)
The focus on management issues rathei than outcomes per se is reflected
in the CCAF's twelve attributes of effectiveness, which include such matters as
managers' responsibility for costs and productivity, responsiveness lo changed
circumstances, piovision of an appropriate working environment, and so on
Restrictions on the Scope of Audit
Under tlu F approach, the auditor is potentially limited in the scope of
rhc iiivrsiiiMiioi) hv ihr information ihai ihr m.m.it'ri < linn^r--, in n-nori
DIFFERENCES IN HIE APPROACHES OF AUDITORS AND EVAI UAIURS 11
There may be other, more formal restrictions on the scope of audit work
This is especially likely to apply 10 ihe examination of major government poli-
cies For example, with regard lo the special examination of Crown Corpora-
tions, Section 145 of Canada's Financial Administration Act specifies. "Nothing
in this Pan or the regulations shall be construed as authorizing the examiner
of a Crown corporation to express any opinion on the merits of mailers of pol-
icy, including the merits of . (c) any business or policy decision of the cor-
poration or of the Governmenl of Canada "
Other examples are provided by the Swedish National Audii Office (1995,
p 8) and the Australian National Audit OITrce (1995, p 1 3). whose mandates
specifically exclude commenl on governmeni policy
When Auditors Examine Results
To this point, 1 have argued that auditors are more likely than cvaluators to
focus on management systems and procedures and to face restnciions on their
freedom to comment on policy I have also pointed oul lhai the mandate of
auditors frequently includes effectiveness li is in the area of examining effec-
tiveness that the distinction between audu and evaluation is least clear.
although reducing to iwo basic issues The first is the focus of effectiveness
work for each The second is the reasons why auditors and cvaluators tackle
issues of program effectiveness.
The model for results-based audit depicted m Fable 1 1 illustrates the pos-
sible foci for effectiveness work.
The model locales results measuremeni of government programs along
two dimensions the level of results measured and the level of program ana-
lyzed With regard to the program level, a distinction is made between ihe
results of management systems and procedures and ihe results of program
activities. Systems and procedures include such mailers as planning, manage-
menl information systems, and procedures for detecting, recording, and col-
lecting overpayments 10 program clientele Program evaluation itself is viewed
as a management control over program effectiveness The levels of results iden-
tified include economy and efficiency, the achievement of intermediate pro-
gram objectives, and ihe achievemeni of overall program objectives
Thus a results-based audit could potentially examine the efficiency of pro-
gram activities, the exteni lo which ihese activities further intermediate program
objectives, or ihe exteni lo which (hey further the achievement of overall pro-
gram objectives Audits may also examine ihe effect of management c ontrols on
efficiency or the achievemeni of intermediate or ultimate program objectives
A few examples will help illustrate the model In 1993, the Office ol the
Auditor General examined management controls over pension benefit pay-
ments. The audit concluded thai ihe sysiems and proceduies in place for the
recording, control, and collection of overpayments fell far short ol r irsniiril
-------
9
I
w*
O
i
u.
•5
•XJ
i
SB
Potential for end
foreign economic
development proj
anticited beca
-el'
.
C u D.
H .a
assured in
1993a Pa
ipat
not
ects
proj
i
e
5
Weakn
and
incre
whe
199
a «
-a >.
P «
la
u "" -• ft
6»- a B
£ 8«£
•g
rt
I
1
-3
c
-S
"o
!
K
DIFFERENCES IN THE APPROACHES OF AUDIIORS AND E VALUATORS 1 3
0 5 percent of total program payments, increasing ihe adminisirativc cost of
program delivery by more than 50 percent The observation does not bear on
the potential contribution that management controls may make to program
effectiveness, but rather on their effects on the costs of program delivery
By comparison, an audit of Search and Rescue examined the relative effi-
ciency of program activities, specifically the types of rescue vessels employed
The audit concluded that the largest class of vessels were the most costly and had
not been critical to saving any lives during the penod audited (1992b, p 225)
Management systems and controls may also have an impact on the
achievement of program objectives In 1992, the Auditor General of Canada
reported on a program of payments to government employees scheduled to be
laid off. The payments were expected to permit employees who so desired to
quit immediately if there was no work to be performed and to save costs ol
employee benefits, retraining, and finding a job (I992a, p 189) The audit
observed (hat the number of payments, in all years except one, had consis-
tently exceeded the reduction in person years (p 194) Moreover, the auditors
concluded that the situation was one indicator of problems in the administra-
tion of the policy, pointing to such matters as inadequate planning, no appro-
priate management framework, and the failure of senior management to
provide leadership direction and support (p 196)
Results as Evidence of Management Deficiencies
As noted earlier, the core business of audit is the examination of management
controls over expenditure management It is therefore the bottom row of Table
1 1 where audits will commonly be found Economy and efficiency of program
operations, the left-hand column of Table 1 I, is also a common matter for
attention by audit However, control over results as an indication of good
financial management and control has also become an audit concern, leading
to increased attention to the contribution of program operations to intermedi-
ate and overall program objectives The result is illustrated by the cases cued
in Table 1 1 No area of effectiveness is exempt from attention by audit
It is in attention to the contribution of program activities to intermediate
and ultimate program objectives where the functions of audit and evaluation
become most difficult to distinguish In principle, u is in the examination ol
activities in relationship to intermediate program objectives where the two may
be most likely to overlap, but the central focus is substantially diflcicnt l;m
an auditor, problems with the effectiveness of piogiam activities is ol interest
as an indication of the importance of deficiencies in program management I he
auditor will do sufficient work to assess whether piogiam clleciiveness is ai
risk, before turning attention to the management factors that should enable
managers to gain control over program results This may involve relying on
evaluations conducted by program management, synthesizing the findings of
evaluations of similar programs in other jurisdictions, and, as a last lesoii, con-
ducting measurement and analyses of program impaus
-------
1 4 F.VAI.UA1 ION AND AUDI I INC.
Tor an auditor, the aimhuiion of impacts to program activities is less
important than in the classic social scientific model of evaluation In the social
science-based model, the ngoious pursuit of the causal link between the pro-
giam and us outcomes is the main focus For an auditor, it is sufficient to
determine that results may be inadequate, whatever the cause If results appear
to be positive, there is no deficiency to report and no need to elaborate the
causal chain
If there aie pioblcms wnh results, the search is on for deficiencies in prac-
tices that may have impeded management from detecting and solving the prob-
lem or for related areas of program weakness However, this search does not
necessarily involve exploration of the causal chain, for the auditors job is not to
solve management's problems but to identify that there is a problem to be solved
and the areas that may be involved In fact, it may be sufficient for an auditor to
point out thai management has done little to measure program effectiveness
This lack of attention to a rigorous exploration of the causal chain may
puzzle evaluators The answer to the puzzle is that for an auditor, information
on results provides evidence of other matters and is not an end in itself
References
Auiliior General of Canada Taymems 10 Employees Under ihe Work Force Adjustment Pol-
icy " In Report i)/(he Auditor General of Canada Ouawa Minister of Public Works and Gov-
einincni Services Canada. 1992.1
Auditor General of Canada "Scaich and Rescue " In Report of the Auditoi General of Canada
Ottawa Minister of Public Works and Government Services Canada. I992b
Auditor General of Canada "CIDA—Bilateral Economic and Social Development Programs "
In Report o/ihe Auditor General of Canada Ottawa Minister of Public Works and Gov-
ernment Services Canada, 1993a
Auditor General of Canada "Department of the knvironment—1 he Control and Clean-Up
of Freshwater Pollution " In Rrpoit of the Auditor General of Canada Ottawa Minister of
Public Works and Government Services Canada, I993b
Auditor General of Canada "Department of Fisheries and Oceans—Northern Cod Adjust-
ment and Recovery Program " In Report of the Auditor Central of Canada Ottawa Minis-
ter of Public Works and Government Services Canada. 1993c
Auditor General of Canada "Department of National Health and Welfare—Programs for
Seniors " In Report of the Auditor General of Canada Ottawa Minister of Public Works and
Government Services Canada. 1993d
Austialian National Audi! Office Per/or rnunte Auditing Canberra Australian National Audit
Office. 1995
Canada Financial Admmisualum Act RS 1985. c F-ll.S I Ottawa Government of
Canada. 1991
C.iiiadi.in Compiehensivr Auditing foundation C//cc nvencss Re;x»fm£ ami Auditing in (he /lib-
lie Sfilni .StiriimMry Ki'/iori Ou.iwj Cuiailiaii Comprehensive Auditing round.uion, 1987
Swedish Naiion.il Audit Olficc l'eifi>inian
-------
60 EVAIUAIION AND AUDIT ING
Chchmsky. I: "Comparing ami Contrasting Auditing and [valuation Sonic Notes on Their
Relationship " Evaluation Revuw. I9S5. 9 (4). 483-503
Clielimsky, I: "Expanding GAO's Capabihlies in Program Evaluation " The GAOJournal.
Winter/Spring 1990,8. p 51
Chen, M Theory-Driven Evaluations 1 housand Oaks. Calif Sage, 1990
Davis. D I" "Do You Want a Peifoimance Audit or a Program Evaluation'" Public Adminis-
tration Review, 1990, 50. 35-41
Day, P . and Klein. D Accountabilities Five Public Services New York Tavistotk. 1987
Frey. B , and Serna, A "Erne polmsch-dkonomische Delraclilung des Rechnungsliofs "
nnanzarchiv. 1990.18. 244-270
Cuba. I:, and Lincoln, Y Fuuilh Generation Evaluation Thousand Oaks, Calif Sage. 1989
Leeuw, f L "Performance Auditing and Policy Evaluation Discussing Similarities and Dis-
similarities " Canadian Journal of Program Evaluation. 1992, 7. 53-68
Lceuw, F L "Performance Auditing, New Public Management, and Performance Improve-
ment Questions and Challenges " Accounting. Auditing and Accountability, forthcoming
Leeuw. F L , Rist. R C , and Sonmchsen, R C (eds ) Can Governments Learn? New
Brunswick. NJ Transaction, 1994
LeGrand, J , and Banleti, W (cds ) Quasi-Marfeels and Social Policy Old Tappan, N J
Macmillan. 1993
Mason, R , and Milroff, I Challenging Strategic Planning Assumptions New York Wiley. 1981
Moukheibir. C , and Barzelay. M "Performance Auditing Concept and Controversies "
Paper presented at the Public Management Group/Organization for Economic Coordi-
nation and Development Audit Symposium. Pans, June 6-7, 1995
Meyer, K . and O'Shaugnessy, K "Organizational Design and the Performance Paradox " In
R Swedberg (ed ). Explorations in Economic Sociology New York Russell Sage Founda-
tion. 1993
Osborne, D . and Gaebler, T Reinventing Government How the Entrepreneurial Spin! Is Trans-
/ormmg the Public Sector Reading, Mass Addison-Wesley, 1992
Pawson, R . and Tilley. N "Whither (European) Evaluation Methodology " The International
Journal o/Knowledge Transjer and Utilization. 1995. 8 (3). 20-34
Public Management Group/Organizaiion for Economic Coordination and Development
"Background paper to the OECD Conference on Auditing " Public Management Group/
Organization for Economic Coordination and Development Audit Symposium, Pans,
June 6-7, 1995
Rist, R C "Management Accountability The Signals Sent by Auditing and Evaluation "Jour-
nal of Public Policy. 1989, 9 (3). 355-369
Smith. P "On the Unintended Consequences of Publishing Perfoimance Data in the Pub-
lic Sector " International Journal o/ Public Administration. 1995, 18. 377-310
UdryJ (ed ) The Media and Family Planning Chapel Mill University of North Carolina
Press. 1974
Walker, W E "The Impact of General Account ing Office Program Evaluations on Govern-
ment " Evaluation and Program Planning, 1985, pp 359-366
Wilson J Bureaucracy New York Free Press, 1989
While there is wide consensus that evaluation ana* auditing are moving
closer together, there is disagreement on the width ojlhe remaining
gap Further integration has both advantages and disadvantages
Auditing and Evaluation:
Whither the Relationship?
Eleanor Chelimsky
In a paper wnuen more lhan ten years ago, I examined some of the similari-
ties and differences 1 perceived in the ways thai auditors and cvaluaiors,
respectively, assess program performance, I linked these (o the hisioncs, mind-
sets, training, functions, and methodological approaches of the two professions
and spoke to the important and promising relationships I was beginning to
glimpse between performance (then called program results) audits and program
evaluations (Chelimsky, 1985) Since that time, and based now not only on
Elmer Staats's pioneering introduction of program evaluation into the U S
General Accounting Office (GAO) in 1980, but also on a good many other
experiences of collaboration between auditors and evaluaiors worldwide, u
appears that major two-way influences are, at very least, changing ihe nature
of both professions, even if they have not as yet produced an actual "blending
of the two cultures," in Roger Brooks's phrase (1995)
Indeed, there is quite wide consensus today (and this is reflected by the
articles in this volume) that audit and evaluation have moved, and arc con-
tinuing to move, toward increasing closeness with regard to understanding
and to methodological approach Disagreement exists, however, about the
degree of closeness that has actually been achieved, along with the reasons foi
the change
I-RANS L LI:UIW is director of the Division of Policy Evaluation at the Netherlands
Court of Audit and pw/essor. Department of Sociology, University of Utrecht, the
Evidence from Five Observers
At one end of the spectrum, Leeuw (1995. pp 15-16) saw the two professions
as still quite different and spoke to the need "to improve the training of audi-
tors and open their minds to the social and behavioral mechanisms opciaung
-------
62 UVAI.UAI ION AND AUDI i ING
m (he public sector and in decision-making " Out he also indicated that cur-
lent efforts to combine "the strong points of evaluation (like the methodology
applied, the attention paid to theory, and theory-driven evaluations) with the
strong points of auditing (the orientation toward management, the focus on
'follow-the-money', and the attention paid to documentary evidence) may well
lead to a new interdiscipline in the 21st century"
At the other end of the spectrum, Pollitt and Summa (1995, pp 24-26)
noted thai "the methods and approaches of auditors and evaluators are com-
ing closer to each other," and found only a marginal difference in the tool kits
potentially available to performance auditors and evaluators They believe this
increasing closeness has occurred "as performance auditing becomes more
common," and they infer that many differences in methodological approach
may be more apparent than real, given that evaluators—who are typically
deprived of the statutory authority of auditors—have a vested interest in lay-
ing claim to "superior methodology and expertise "
Divorski, who spoke uniquely about the differences between auditing and
evaluation approaches, did not, given this topic, discuss the degree to which
auditors and evaluators are moving toward each other despite these differences
Like Leeuw—that is, at the opposite pole from Pollill and Summa—he viewed
evaluation and auditing as enterprises that are still very different, especially in
terms of focus and mind-set With respect to focus. Divorski invoked audil-
ing's need to judge whether management performance is adequate and audi-
tors' consequent targeting of management systems and controls, as opposed to
evaluation's targeting of management activities in the search to determine pro-
gram results. Differences m mind-set referred to the same issue that is.
Divorski saw auditors as interested m results only insofar as they reflected
management performance, whereas evaluators are interested in results for their
own sake These differences also give rise to other differences, for example, the
assumption by auditors that causes for problems found can be explained via
"auditors'judgment," whereas evaluators would see the cause-and-effecl ques-
tion as a matter for empirical inquiry As Divorski put it "If there are problems
with results, the search is on for deficiencies in practices that may have
impeded management from delecting and solving the problem " Bui evalua-
tors would say that deficiencies in management practices may not be the cause
of all program problems, and that improving managers' awareness may not do
much to solve many of them
Biooks (1995), somewheie m the middle of the spectrum, did not argue
that the methodological approaches of auditors and evaluators are now virtu-
ally the same, but ratliei, from his experience at the state level in the United
States, that auditors and evalualors today are "drawing upon the approach of
iradiuonal auditing as well as ihe approach of social science-based evaluation "
Like Lecuw, Brooks recognized that "the 'two cultures' of auditing and evalu-
ation sti1' 'St." and thai "when comparing approaches across states, one
observe tant differences " But even though he saw these differences as
real, he .. j with Polliil and Summa that they are declining However, he
WHITHER THE RI:LAiIONSIIIP' 63
believes thai the decline is "mostly because of a liberalization of traditional
auditing," rather than simply the increasing presence or the performance audit
Also, he views the "emergence of a blended approach to auditing and evalua-
tion" as a fan accompli, at least in some places
It is, of course, difficult to generalize persuasively across the widely dif-
ferent institutional arrangements within which evaluators and auditors work
together, and without empirical data, some of the points cited here may be
impressionistic Should we really assume thai because similar tool kits are
potentially available to evaluators and auditors alike, this means that the tools
are equally used? Given the existence of the "two cultures," even if (he method-
ological tool kits were identical, there would still likely be major differences
made by auditors and evaluators in the application, use, and pervasiveness of
the particular methods selected (For example, I would expect to see auditors
use many more surveys and case studies than, say, quasi-experimental designs,
and I would also expect to see the issues of reliability—in survey questions—
and of generalizability—in case study findings—handled very differently by
evaluators and auditors.)
Again, if there is more "togetherness" observed between auditors and eval-
uators, is this due to a liberalization in traditional auditing standards and pro-
cedures7 to the increasing prevalence of performance auditing7 to the changing
nature of policy makers' questions that force evaluators and auditors 10 bor-
row from each other? to the increased prestige of evaluation among auditors
today? to the belated recognition by evaluators that auditors are right to be
interested in costs7 We don't really know
Thus, although there may be little consensus as yet about how or why it
has happened that evaluators and auditors, despite some real differences, are
coming closer together, the important point is that many observers believe
they are And the question 1 would like to raise here is whether any of the
mingling and blending being discussed in this volume would have happened
without the physical juxtaposition of auditors and evaluators within an orga-
nization It may well be this kind of proximity that has allowed productive
comparisons of work methods and of ways to examine policy issues, to
resolve technical problems, to establish credibility, such as those we see in the
chapters in this volume.
It is much less frequent today than it was fifteen years ago, say. to hear
"the auditors judgment" being used as the sole basis for a finding, or to see
evaluators unable to explain what savings (or expenditures) are likely to result
from service changes they propose But if physical closeness is important, then
it is also important to examine what we know about how to manage auditors
and evaluators together in an organization That is, given the diffcicnces and
the similarities of the two fields, and given the benefits for public policy likely
to accrue from their increased collaboration, what is the oig.imzaiional device
that will allow us to reap the greatest rewards7 Should we integia* '~» func-
tions that is, house, and supervise auditors and evaluators togcthc hould
we keep them separate7
-------
64 LVALUAl ION AND AUDI TING
Separation or Integration-
Some Advantages and Disadvantages
Ai first glance, keeping the auditing and evaluation funuions separate has a
number of obvious disadvantages Increased costs are involved in maintaining
a separate group or department, and important potential diffusion-of-mfor-
malion benefits to the organization may be foregone unless an enlightened
management takes steps to break down walls (or pieveni them from rising in
the first place) In addition, differences in findings among auditing and evalu-
ation units working on the same issues but using different methods can
become a problem for the organization The normal competition between
groups for organizational hegemony can move from what is stimulating and
healthy to something profoundly unhealthy unless thai competition is quite
carefully managed
On the other hand, separation does allow evaluators the independence
they need for credibility (that very same independence for which auditors have
fought so hard over their long history) Separation also gives evaluators the
freedom to develop a critical mass of skills, to establish the legitimacy of their
work with policy makers who may not be familiar with evaluation, and to
respond to policy questions with strong studies that can demonstrate the worth
of evaluation not only to policy makers but to auditors as well Finally, one of
the most important advantages of separation is us feasibility start-up does not
require behavioial change in an organization, only funds, operational know-
how, and leadership
What about integration7 Most experience suggests that this can be quite
difficult to achieve, ai least immediately A first problem arises because audi-
tors and evaluators have been trained so differently, and tend to have dissimi-
lar mind-sets with regard to a study Auditors are taught, for example, that they
are wasting the taxpayers' money if an investigation does not uncover a major
pioblem "If nothing is wrong," they say, "then why should we be doing an
audit7" But evaluators are trained instead to ask whether some policy or pro-
gram has made a difference, any difference, good or bad That is. to evalualors,
positive findings are as important as negative ones in improving public policy
and need no special justification To find out what works is at least as useful,
evalualors think, as to find out what didn't work
These mind-sets have some ramifications for the work process Auditors
ofien do their besi 10 determine whether theie is (or is noi) a significant prob-
lem based, say, on a one-month informal investigation, and may then abandon
the project if no significant problem has turned up If evalualors expecl that a
study may genei.ite weak or strong findings in any direction, they will often
spend thiee months designing it (moie if the policy question asked is complex
or contiovcrsial), and they have difficulty in answering auditors' questions
about what their findings arc likely to be befoie they have collected their data
One result of this process is that evaluations arc often afflicted by findings that
aie anything but conclusive, and tins means that undei an unrelated organi-
WmiiiCRiiir. Ri.LArioNsmp? 63
zational arrangement, auditors may greel (he evalualors' findings wuh a bored
"So what?" and a large yawn, whereas evalualors will always be skeptical about
those exciting one-month findings thai are established before data on both
sides of the question could possibly have been collected
Emphasis (or the lack of it) on measurement is another training issue thai
causes tensions between auditors and evalualors The measurement questions
lhai preoccupy evalualors (like ihe reliability of items in a questionnaire, or
threats to ihe internal validity of siudy findings, or the real comparability of
before-after or cross-secuonal siudy daia) lend to take a bit of time to resolve,
are typically low on the auditors' pnonly list, and are usually nol well under-
stood by them Because of this, abseni training on boih sides, and especially
training of managers, auditors may reproach evalualors for their slowness and
evalualors may reproach auditors about the validity of their findings Unhap-
pily, these perceptions lend lo linger within an organization they engender us-
and-them mentalities, they cause morale problems, and they militate against
the continuing recruitment and retention of methodologically sirong evalua-
lors whose presence is critical to the success of cross-fertilization efforts
The point here is that when these tensions of mind-sei and measurement
bubble up, many of ihe advantages lhai integration was counted on lo supply
may not materialize For example, in a tense atmosphere, knowledge diffusion
and organizational learning may be even worse off than under separatum Dif-
ferences in findings based on methodological approach are still likely to surface
under integration, although al a lower level and thus more manageably—that
is, easier lo suppress lhan confront—from an organizational perspective And
even if evalualors may feel more protected institutionally than they do as pan
of a separate unit, the trade-off between thai protection and technical quality
may nol seem worth u 10 many evaluators
Organization for Production or for Cross-Fertilization?
Perhaps a useful way to think of separation versus integration is as a function
of ihe institutional goals to be pursued If the mam aim is lo increase the capa-
bilities of the organization to answer complex policy questions and to begin
doing thai as soon as possible, then separaie evaluation and audit units such
as ihose ai GAO and ai the Minnesota Program Evaluation Division have much
lo recommend them But if the aim is cross-fertilization in an organization,
then separation, especially if it is poorly managed, may bring two important
long-run costs communication wuh ihe larger institutional entity may not be
good enough, and evaluation staff may begin to feel excluded from important
policy decisions
A good example of this problem comes from the experience of the Office
of Management and Budget (once known as the Bureau of the Budget, or
BOB) In ils early days, BOB decided lo bring in people with strong technical
skills lo complemeni the work of their budget analysis, and they separated
these technical staff from the budget people by cicaimg technical centers in
-------
66 EVALUAI ION AND AUDI I INC.
which (he new skills would be deployed In the words of a former BOB Assis-
tant Director, William D Carey, what happened was this separation "built a
kind of concentrated quality in the technical centers and it successfully accu-
mulated a critical mass of top-flight specialists " But it also led to an organiza-
tion in which technical staff became alienated In Careys words
I he budget people sal al I he table during the Director's reviews, but the techni-
cal people had only backbench chairs and very limned possibility 10 participate
in the discussions When they did speak, their commenis were considered intru-
sive Promotions and supergrades went to line, not staff, personnel BOB Direc-
tors had little tune or interest in the technical work, and technical staff had little
or no access 10 them On their side, technical people tended 10 look down on
budget examiners as journeymen of very average capabilities |U S General
Accounting Office. 1990. p 23|
To remedy this developing nft within the organization, BOB decided to dis-
solve the technical centers and scatter their personnel across the budget divi-
sions In this way it was hoped that organizational cohesion and communication
could be improved and that the work of the budget divisions would be enriched
by the closer proximity of the technical staffs expertise Carey believes this move
to have been a mistake and its results unfortunate, for three reasons
First, the same problems reappeared, but at the lower, divisional level The scat-
tered technical staff continued 10 feel they were second-class citizens and now
the situation was worse in ihat they had no organizational voice Their sense was
thai they had 10 keep proving their worth (as technical people in a budget divi-
sion) and that they were no belter off in terms of having direct inputs into orga-
nizational decisions and products Second, the professional quality of the
technical staff weakened over lime because the technical centers which had
attracted some of the brightest people in their respective fields were no longer
there And finally, the dispersed technical personnel did not appear to have any
visible effect on the work of the budge) divisions |U S General Accounting
Office. 1990. p 24|
Ai the GAO and in Minnesota, where audit and evaluation units have been
separate, at least some of these evils have been avoided, and certainly, with
regaid lo visible effect, the influence of the evaluation work has been recog-
nized and highly regarded
In short, if integration has not been easy—and it has not—we may need
10 learn moic about why u has been so difficult before dismissing it as an
option Is u, for example, because evaluation and auditing have different
sources of credibility and legitimacy7 Is u because mind-sets and cultures
become cntieiKhed under the stresses of organizational competition7 Is u
bcL.ii- simply haven't yci developed both the management techniques and
the m .is needed lo do the job7
WmmcR i HE RLLAIIONSIIIP' 67
My own view is that — given effective managers who know and value both
auditing and evaluation, and given also some harmonization of training for
auditors and evaluators — it should be possible to integrate the evaluation func-
tion successfully in audit organizations But until that training has taken place
(and especially for audit offices beginning now to incorporate evaluation into
their work programs), keeping the functions separate, building multiple
bndges between them, and watching their interactions and managing them
carefully may be not only the most prudent but also the best organizational
course of action
A lot may be ndmg on this selection of the right organizational model To
be viable, both evaluation and audit functions need independence, skilled per-
sonnel. credibility, sponsors who understand the benefits to be drawn from
both audits and evaluations, and the capability to respond appropriately to the
policy questions of today's political environment Such capability requires the
use of both auditing and evaluation methods When we can bring these two
together and target them properly to policy makers' information needs, and
when findings from both types of studies can make their way unimpeded into
the policy process, then both evaluations and audits will have achieved their
real public purpose, to help make government services more effective, more
meaningful, more responsive, more accountable, and — last but not least — bet-
ter managed
References
Brooks. R "Blending Two Cultures State Legislative Auditing and Evaluation " Paper pre-
sented at (he International Evaluation Conference in Vancouver. D C , Nov 1995
Chelimsky, E "Comparing and Contrasting Auditing and Evaluation Some Notes on Their
Relationship." Evaluation Review. 1985. 9 (4). 483-503
Leeuw, F. L "Auditing and Evaluation Bndging a Gap, Worlds to Meet7" Paper presented
at the International Evaluation Conference in Vancouver. B C , Nov 1995
Pollill, C , and Summa, H "Performance Auditing Travellers' Tales " Paper presented at the
International Evaluation Conference in Vancouver. B C . Nov 1995
U S General Accounting Office Diversifying and Expanding Technical Skills at GAO GAO/
PEMD-90-18S. Vol 2 Washington. D C US General Accounting Office. Apr 1990
ELEANOR CHELIMSKY is an international consultant in evaluation
ri/wv nm\ nmf rmulrni nflhe Amenran FwiluriMnn
jncJ method-
-------
AUDITING
Art
Comprehensive
Objective
Prove
Quality
Present
Internal
Inputs
Separate
Simple
Inductive
Individual
What
Genera lizable
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
Science
Specific
Value-laden
Improve
Quantity
Future
External
Outcomes
Collaborative
Complex
Deductive
Team
Why
fnique
-------
EVALUATION
Art
Comprehensive
Objective
Prove
Quality
Present
Internal
Inputs
Separate
Simple
Inductive
Individual
What
Generalizable
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4-
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
Science
Specific
Value-laden
Improve
Quantity
Future
External
Outcomes
Collaborative
Complex
Deductive
Team
Why
L/nique
-------
Logic Mo4ehng
Refer to
Mclaughlin, John A and Gretchen B Jordan, (1999) "Logic Models a tool For telling your
program's performance story" Evaluation and Program Planning 22. Elsevier Science Ltd
pp 65-72
Articles in New Directions for Evaluation 87 Fall 2000, San Francisco CA Jossey-Bass
Publishers
Rogers, Patricia ) , Anthony Petrosmo, Tracy A Huebner, and Timothy A Hacsi
Editor's Notes and "Program Theory Evaluation Practice, Promise, and Problems"
pp1-13
Weiss, Carol Hirschon "Which Links in Which Theories Shall We Evaluate/"
pp 35 - 45
Rogers, Patricia J "Causal Models in Program Theory Evaluation" pp 47-55
1. What is logic modeling/
2 Who uses it/ Why/
3 What are the steps/
-------
Drawings of Logic Models
-------
Is Logic Modeling The 'Answer^
Before you proceed with your task, would
it be helpful to have...
...a visual diagram of the design of the
...a common description of a program to share with persons
internal and external to the program?
...a diagram to facilitate the development of hypotheses about
how program inputs are related to activities, activities are related
to outputs, outputs are related to outcomes?
...a digram to help identify what measures would be helpful in
testing your previous hypotheses?
...a digram to help determine whether the program is designed to
succeed?
if the answer is 'yes' to any of these, then let's model!
-------
The Worl4 of l_£
Modeling
Emmalou Norlan
1C
A LOGIC MODEL
Is a picture'Can4 corresponding text
description) of how a program is designed
to work.
Shows thgt a program has inputs, outputs,
-------
Who Uses Logic Mojels?
A A logic model serves 35 a common
communication tool among program
staff.
A Logic models help explain program logic
to flinders and other stakeholders.
Evaluators ALSO l/se Logic
Models
A To understand how the program works,
how it links to other programs, and how it
contributes to agency long-term goqls.
A To identify gaps in the logic of a
prog ram's design.
A To determine the critical links in the
program's logic so that appropriate
measures can be identified for use in
evaluations
A Program Logic Mocjel Explains
the ARROW
A How a Program
l/ses Resources to
Improve
Environmental and
Human Health
Congress
Provi4es
Money
Improved
Environment*!
4ixl Humjn
Health
A Program's Logic
Resources are used by the agency to
create a set of activities (the program).
The activities are designed to produce
outputs (products or services) for a set of
customers/clients Those clients react to
the outputs (and gain knowledge and
skills and attitude) such that they can and
will change their behavior in a desirable
way
-------
Logic Continue^..
When client behavior changes, there
environmental consequences (reduced
pollutants, changed ambient condition...)
resulting in improved environmental and
human health.
Activities
A Researching
A Permitting
A Regulating
A Developing
A Monitoring
A Training
A Communicating
A Enforcing
Resources
Can be budget, FTEs, equipment,
facilities, supplies, products from
other- programs, internal an4
external partners, key
information..
Outputs
Scientific Finding
A Permits
A Regulations
A New Technologies
A Pata Bases
A New Methods
A Cutting Edge Information
A Enforcement
-------
Customers/Clients of EPA
Programs
A Congress
A The Public
A States
A Regulated
Community
A Other agencies
Environmental Outcomes
Stressor Recjucecl
Ambient Condition Improves
Environment Better
Behavioral Outcomes
A Affective Reactions (satisfaction,
A
agreement, support...)
A Knowledge
A Skills
A Attitudes (beliefs,
perceptions, values]
A Behavior Change
Environmental
Human Health
IMPROVED!
-------
Developing a Logic Model
A Identify the program boundaries
A Gather written information about the
program
A Put a draft model together
A Review the model with program staffand
other stakeholders.
A I/sing that information, change model to be
the best representation of the program as
designed.
What Makes Up the PROGRAMS
V
\
Resources
/
/
/
Activities
/
)
/
Outputs
/
/
FTES. Budvjet.
F,
lilforrrMllon
>• \
Permit*.
Resfiirch. }
Enforcement /
y
Basic Components of an
tnvironrr)enfal~Prograro Logic"
Mocjel
Behavioral
Outcomes
al
es
/
«
Environment
Outcomes
•
/
X
Environment.)
and Ilimijn
Hejllli
Externalities
I if the resources support the 'right' activities, j
which produce the 'right' outputs, then the !
client has BEHAVIORAL OUTCOMES \
Starts. Sto(>> \
K» \
Pccre.l>o /
Heli.li 101 /
/
-------
if clients move through the behavioral chain to
ultimately change behavior, then
ENVIRONMENTAL OUTCOMES are achieved.
/
Stressor is
Reduced
/
/
/
Ambient
Condition
Improves
/
/
Kjs are
reduced
Improvement
What are Externalities'
People, events, an4 other entities which
couM have an effect upon what happens in
the boxes and/or on relationships between
the boxes
Weather, Political Climate, Competing
Programs, Other Agencies, Agenjas. Lacl.
of Information
Through m^ny programs achieving
BEHAVIORAL AND
ENVIRONMENTAL outcomes over a
long period of time, then the ultimate
goal can be reached: ENVIRONMENT
AND HUMAN HEALTH
Expanded Environmental Logic Model
-------
We can also use this logic in
reverse, when planning a program!
Then what Joe, science
tell us about how the
| environment]! condition:
need to change to reach
that goalf
if the ULTIMATE goal
is Environmental and
Human Health
What kinds of outputs would best
address the desired behaviors?
Information
based upon
souncl science
if we want those environmental conditions to
change, WHO/WHAT arc the pollutant sources/
(WHO - rnsfr>rr)t'is/rlirt)fs - r)cre\ tn rli^t)i_' WHAT
behaviors/)
Stop Polluting
Change
Household
Practices
Enact
Appropriate
Policy
Wh^t progi-gms 3nd Activities can EPA
undertake to produce those outputs/
Information
based on
sound
science
-------
AND, FINALLY, what resources are needed
to plan and conduct these programs
Cwhich, in turn, produce these outputs)/
Resource.
Resource.
Regulations
Information
based on
sound science
-------
Generic Logic Mocjel (with
Environmental Program Outcomes)
Program
Resources
Activities
\
Outputs
Customers
Problem
ShortTenn
Outcome
Intermediate
Outcome
Lull Term
Q il come
Externalities
Behavioral
Outcomes
Environmental
Outcomes
-------
Environmental Logic Mode
Rpgram
Resources
Ativities
Outputs
Customers
Externalities
PluUcm
Behavioral
Outcomes
Environmental
Outcomes
HLOHUI hk'idlh
Knowlcc^e,
Skills, Attitudes,
Behavior
Stntssor/Ambicnt
Concfition
-------
EVALUATION
and PROGRAM PLANNING
PERGAMON
Etalujuon and Progrjm Planning 12 (1999) 65-/:
Logic models: a tool for telling your program's performance story
John A. McLaughlmJ, Gretchen B. Jordan6*
'Independent Consultant 423 Hempsteati Roail Willtaniihurg \'A 2USS. L 5 ^
'Saiulia National Laboratories 950 L'Enfant Plaza SWSuite IItt V/a\hmqion DC .'OO.V. L 5 A
Accepted I August 1993
Abstract
Program managers across private and public sectors are being asked to describe and evaluate their programs in new ways People
want managers to present a logical argument for how and why the program is addressing a specific customer need and how
measurement and evaluation will assess and improve program effectiveness. Managers do not have clear and logically consistent
methods to help them with this task. This paper describes a Logic Model process, a tool used by program evaluators. in enough
detail that managers can use it to develop and tell the performance story for their program. The Logic Model describes the logical
linkages among program resources, activities, outputs, customers reached, and short, intermediate and longer term outcomes Once
this model of expected performance is produced, cntical measurement areas can be identified. © 1999 Elsevier Science Ltd All rights
reserved.
Ke\Hords Program theory. Program modeling. Performance measurement. Monitoring and evaluation. Government Performance and Results Act
1. The problem
"At its simplest, the Government Performance and
Results Act (GPRA) can be reduced to a single question:
What are we getting for the money we are spending?
To make GPRA more directly relevant for the thou-
sands of Federal officials who manage programs and
activities across the government, GPRA expands this
one question into three: What is your program or
organization trying to achieve0 How will its effec-
tiveness be determined? How is it actually doing7 One
measure of GPRA's success will be when any Federal
manager anywhere can respond knowiedgeably to all
three questions."
John A. Koskmen, 1997
Office of Management and Budget
Federal managers were being challenged by Mr Kos-
kmen (1997), Deputy Director of the OMB, to tell their
program's story in a way that communicates not only the
program's outcome goals, but also that these outcomes
are achievable For many public programs there is also
an implicit question. 'Are the results proposed by the
program the correct results?' That is, do the results
address problems appropriate for the program and
deemed by stakeholders to be important to the organ-
izational mission and national needs?
* Corresponding author E-mail gbjordaiii sandu gov
The emphasis on accountability and 'managing for
results' is found in state and local governments as well as
in public service organizations such as the United Way
of America and the American Red Cross It represents a
change in the way managers have to describe their pro-
grams and document program successes. Program man-
agers are not as familiar with describing and measuring
outcomes as they are with documenting inputs and pro-
cesses. Program design is not necessanly explicit, in part
because this allows flexibility should stakeholder pri-
orities change.
There is also an increasing interest among program
managers in continuous improvement and managing for
'quality'. Choosing what to measure and collecting and
analyzing the data necessary for improvement measure-
ment is new to many managers
The problem is that clear and logically consistent
methods have not been readily available to help program
managers make implicit understandings explicit. While
tools such as flow charts, risk analysis, systems analysis.
are used to plan and describe programs, there is a method
developed by program evaluators that more com-
prehensivelv addresses the increasing requirements for
both outcome measurement and improvement measure-
ment
Our purpose here is to describe a tool used by many m
the program evaluation community, the Logic Model
process, to help program managers better meet new
0149-7189 9') S - >ee front mailer f 1999 El»evier Science Ltd All rights reserved
PII S0149-7I89(9S)0004:-|
-------
66
J ^ A/c Launlilui C B Jordan Ci ulummn .nul /V«iv,«ii Plaiminv _V
requirements Documentation of the process by which a
manager or group would develop a Logic Model is not
readily available even within the evaluation cornmunu>.
thus the paper may also help evaluators serve their cus-
tomers better
2. The Program Logic Model
Evaluators have Found the Logic Model process useful
for at least twenty years. A Logic Model presents a plaus-
ible and sensible model of how the program will work
under certain conditions to solve identified problems
(Bickman, 1987). Thus the Logic Model is the basis for a
convincing story of the program's expected performance
The elements of the Logic Model are resources, activities.
outputs, customers reached, short, intermediate and
longer term outcomes, and the relevant external influ-
ences (Wholey, 1983, 1987)
Descriptions and examples of the use of Logic Models
can be found in Wholey (1983). Rush & Ogborne (1991),
Corbeil (1986), Jordan & Mortensen (1997), and Jordan,
Reed, & Mortensen (1997). Variations of the Logic
Model are called by different names, 'Chains of Reason-
ing' (Torvatn, 1998). Theory of Action, (Patton. 1997),
and 'Performance Framework' (Montague, 1997.
McDonald & Teather, 1997). The Logic Model and these
variations are all related to what evaluators call program
theory. According to Chen (1990), program theory
should be both prescriptive and descnptive. That is, a
manager has to both explain the elements of the program
and present the logic of how the program works. Patton
(1997) refers to a program description such as this as an
'espoused theory of action', that is. stakeholder per-
ceptions of how the program will work.
The benefits of using the Logic Model tool include.
• Builds a common understanding of the program and
expectations for resources, customers reached and
results, thus is good for shanng ideas, identifying
assumptions, team building, and communication,
• Helpful for program design or improvement, ident-
ifying projects that are critical to goal attainment,
redundant, or have inconsistent or implausible linkages
among program elements; and,
• Communicates the place of a program in the organ-
ization or problem hierarchy, particularly if there are
shared logic charts at various management levels:
• Points to a balanced set of key performance measure-
ment points and evaluation issues, thus improves data
collection and usefulness, and meets requirement of
GPRA.
A simple Logic Model is illustrated in Fig I.
Resources include human and financial resources as well
as other inputs required to support the program such
as partnerships Information on customer needs is an
e<,scnnal resource to the program Aiinnici include all
those action steps necessan. to produce program outputr
Output* arc the products, goods and services provided tc
the program's direct itmonier\ For example, conducting
research is an activity and the reports generated for other
researchers and technology developers could be thought
of as outputs of the activity
Customers had been dealt with implicitly in Logic
Models until Montague added the concept of Reach to
the performance framework. He speaks of the 3Rs of
performance: resources, people reached, and results
(Montague. 1997. 1994) The relationship between
resources and results cannot happen without people—
the customers served and the partners who work with the
program to enable actions to lead to results Placing
customers, the users of a product or service, explicitly in
the middle of the chain of logic helps program staff and
stakeholders better think through and explain what leads
to what and what population groups the program intends
to serve.
Outcomes are characterized as changes or benefits
resulting from activities and outputs. Programs typically
have multiple, sequential outcomes across the full pro-
gram performance story. First, there are short term out-
~comes. those changes or benefits that are most closely
associated with or 'caused' by the program's outputs
Second, there are intermediate outcomes, those change*
that result from an application of the short ter
outcomes. Long term outcomes or program impacts, foi
low from the benefits accrued though the intermediate
outcomes. For example, results from a laboratory proto-
type for an energy saving technology may be a short-term
outcome; the commercial scale prototype an intermediate
outcome, and a cleaner environment once the technology
is in use one of the desired longer term benefits or
outcomes.
A critical feature of the performance story is the identi-
fication and description of key contextual factors external
to the program and not under its control that could
influence its success either positively or negatively It is
important to examine the external conditions under
which a program is implemented and how those con-
ditions affect outcomes. This explanation helps clarify
the program 'niche' and the assumptions on which per- •
fortnance expectations are set. Doing this provides an
important contnbution to program improvement
(Weiss. 1997) Explaining the relationship of the problem
addressed through the program, the factors that cause
the problem, and external factors, enables the manager
to argue that the program is addressing an important
problem in a sensible way.
3. Building the Logic Model
As we provide detailed guidance on how to develop «.
Losic Model and use it to determine kev measurement
-------
J A MiLaiishlui C B Jordan! Eialualiim urul Prn^rani Plumtiiiy .'.' ,' 19991 6.!-~?
Resource* |
(inputs) p""
Activities !-*•
hfor
Customers — ••
Rejched
Snorl-lerm 1 ^
Outcomes 1 "~
Outcomes
(through
customers)
*
Outcomes
& Problem
Solution
External Influences and Related Program*
Fig I Elements or the Logic Model
and evaluation points, it will become more clear how the
Logic Model process helps program managers answer the
questions Mr Koskmen and others are asking of them
An example of a federal energy research and technology
development program will be used throughout. Program
managers in the U.S Department of Energy Office of
Energy Efficiency and Renewable Energy have been using
the Logic Model process since 1993 to help communicate
the progress and value of their programs to Congress.
partners, customers, and other stakeholders.
The Logic Model is constructed in five stages discussed
below. Stage 1 is collecting the relevant information.
Stage 2 is describing the problem the program will solve
and us context; Stage 3 is defining the elements of the
Logic Model in a table. Stage 4 is constructing the Logic
Model, and Stage 5 is verifying the Model.
3 1. Siage 1 collecting the relevant information
Whether designing a new program or describing an
existing program, it is essential that the manager or a
work group collect information relevant to the program
from multiple sources The information will come in the
form of program documentation, as well as interviews
with key stakeholders both internal and external to the
program While Strategic Plans, Annual Performance
Plans, previous program evaluations, pertinent legis-
lation and regulations and the results of targeted inter-
views should be available to the manager before the Logic
Model is constructed, as with any project, this will "be
an iterative process requiring the ongoing collection of
information. Conducting a literature review to gam
insights into what others have done to solve similar prob-
lems, and key contextual factors to consider in designing
and implementing the program, can present powerful
evidence that the program approach selected is correct.
Building the Logic Model for a program should be a
team effort in most cases. If the manager does it alone.
there is a great risk that parts viewed as essential by some
will be left out or incorrectly represented. In the following
steps to building the Logic Model we refer to the manager
as the key player However, we recommend that persons
knowledgeable of the program's planned performance.
including partners and customers, be involved in a work
group to develop the Model As the building process
begins it will become evident that there are multiple realit-
ies or views of program performance Developing a
shared vision of how the program is supposed to work
will be a product of persistent discovery and negotiation
between and among stakeholders.
In cases where a program is complex, poorly defined, or
communication and consensus is lacking, we recommend
that a small subgroup or perhaps an independent fac-
ilitator be asked to perform the initial analysis and syn-
thesis through document reviews and individual and
focus group interviews. The product of this effort can
then be presented to a larger work group as a catalyst for
the Logic Model process.
J 2. Stage 2: clearly defining the problem and its context
Clearly defining the need for the program is the basis
for all that follows in the development of the Logic
Model. The program should be grounded in an under-
standing of the problem that drives the need for the
program. This understanding includes understanding the
problems customers face and what factors 'cause' the
problems. It is these factors that the program will address
to achieve the longer term goal—working through cus-
tomers to solve the problem. For example.
There are economic and environmental challenges
related to the production, distribution, and end use of
energy. U.S. taxpayers face problems such as depen-
dence on foreign oil, air pollution, and threat of global
wanning from burning of fossil fuels Factors that
might be addressed to increase the efficiency of end use
of energy include the limited knowledge, risk aversion,
budget constraints of consumers, the lack of com-
petitively priced clean and efficient energy tech-
nologies, the externalities associated with public goods,
and restructuring of U.S electricity markets. To solve
the problem of economic and environmental challenges
related to the use of energy, the program chooses to
focus on factors related to developing clean and
efficient energy technologies and changing customer
values and knowledge. In this way, the program will
influence customer use of technologies that will lead to
decreased use of energy, particularly of fossil fuels
One of the greatest challenges faced by work groups
-------
6S
J A McLaugliltn CB Jordan, Eiuluaium ami Prtivrain Plumum; 22 (I999j 65-U
developing Logic Models is describing where their pro-
gram ends and others start. For the process of building
a specific program's Logic Model, the program's per-
formance ends with the problem it is designed to solve
with the resources it has acquired, including the external
forces that could influence its success in solving that prob-
lem Generally, the manager's concern is determining the
reasonable point of accountability for the program At
the point where the actions of customers, partners, or
other programs are as influential on the outcomes as
actions of the program, there is a shared responsibility for
the outcomes and the program's accountability for the
outcomes should be reduced. For example, the adoption
of energy efficient technologies is also influenced by fin-
anciers and manufacturers of those technologies.
3.3 Stage 3 defining the elements of the Logic Model
3.3.1. Starting with a table
Building a Logic Model usually begins with cat-
egorizing the information collected into 'bins', or col-
umns in a table. Using the categories discussed above the
manager goes through the information and tags it as a
resource', activity, output, short term outcome, inter-
mediate outcome, long term outcome or external factor
Since we are building a model of how the program works,
not every program detail has to be identified and catalo-
ged, just those chat are key to enhancing program staff
and stakeholder understanding of how the program
works.
Figure 2 is a table with some of the elements of the
Logic Model for a technology program.
3.3.2. Checking the logic
As the elements of the Logic Model are being gathered,
the manager and a work group should continually check
the accuracy and completeness of the information con-
tained in the table The checking process is best done b
involving representatives of key stakeholder groups u,
determine if they can understand the logical flow of che
program from resources to solving the longer term prob-
lem So the checking process goes beyond determining if
all the key elements identified, to confirming that reading
from left to right, there is an obvious sequence or bridge
from one column to the next.
One way to conduct the check is to start m any column
in the table and ask the question, 'How did we get here?'
For example, if we select a particular short term outcome,
is there an output statement that leads to this outcome0
Or, for the same outcome, we could ask, 'Why are we
aiming for that outcome?' The answer lies m a subsequent
outcome statement in the intermediate or long term out-
come columns. If the work group cannot answer either
the how or why question, then an element needs to be
added or clarified by adding more detail to the elements
in question.
3.4. Stage 4: drawing the Logic Model
The Logic Model captures the logical flow and linkages
that exist in any performance story. Using the program
elements in the table, the Logic Model organizes the
information, enabling the audience to understand and
evaluate the hypothesized linkages. Where the resource*
activities and outcomes are listed within their respective
columns in the story, they are specifically linked m the
Model, so that the audience can see exactly which activi-
ties lead to what intermediate outcomes and which inter-
mediate outcomes lead to what longer term outcomes or
impacts.
Although there are several ways to present the Logic
| Outcomes
Resources
-Budget SXXX
• #/ capacities
of staff
-ttfeataf
panners
•Cost ihareS
•Technology
Roadmap
- H of yean
experience
Activities
• Fund grants
(solidl, review,
etc.)
-Research
properties of
materials
- Provide
technical
aaiotance
- Set policy for
procurement
Outputs
-UrtypeoT
awards
•RAO
progress
reports
- Lab and
nmmeraal
prototypes
•Advice
provided
•S procure-
in nt i fleeted
Customer
Reached
• Federal and
private
researchers
• Industrial
firms impacted
•VUnufacturers
•Existing/future
consumers of
•dated product!
Short
Term
• Rejectees seek
venture capital
• R&D advances
made
• Lab prototype
started: results
documented
-Tech roadmap
revised
-Advice
considered
Intermediate
Term
-So me get
venture capital
- Lab prototype
com p wtco *
• Commercial
designed
-More efficient
processes
adopted
- Technology
purchased
Long
Term
Reduction in
energy use
£Oils.more
competitive,
emiuwni from
energy
are leu, thus
environment
cleaner
' 4
External Influences: Price of oil and other energy supply and distribution factors, economic growth.
perception of risk of globil dimale change, market assumptions, technology assumptions __
Fig :. A table with element* of ihe Logic Model for an energy technology program
-------
J 4 McLaughlm C B Jordan!Eialuuiiii'i ami Pruvruni Planning .V f19991 6i-~:
69
Model (Rush & Ogborne. 1991, Corbeil. 1986) the Lo«ic
Model is usually sec forth as a diagram with columns and
rows, \vith the abbreviated text put in a box and linkages
shown with connecting one-way arrows. We place inputs
or resources to the program m the first column at the left
of the Model and the longer term outcomes and problem
to be solved on the far right column. In the second
column, the major program activities are boxed In the
columns following activities, the intended outputs and
outcomes from each activity are shown, listing the
intended customer for each output or outcome. An exam-
ple of a Logic Model for an energy efficiency research
and development program is depicted in Fig. 3.
The rows are created according to activities or activity
groupings. If there is a rough sequential order to the
activities, as there often is, the rows will reflect that order
reading from top to bottom of the diagram. This is the
case if the accomplishments of the program come in
stages as demonstrated in our example of the if, then
statements. When the outcomes from one activity serve
as a resource for another activity chain, an arrow is drawn
from that outcome to the next activity chain. The last in
the sequence of activity chains could describe the efforts
of external partners, as m the example in Fig. 3. Rather
than a sequence, there could be a multi-faceted approach
with several concurrent strategies that tackle a problem.
For example, a program might do research in some areas
and technology development and deployment in others,
all working toward one goal such as reducing energy use
and emissions.
Although the example shows one-to-one relationships
among program elements, this is not always the case h
may be that one output leads to one or more different
outcomes, all of which are of interest to stakeholders and
are part of describing the value of the program
Activities can be described at many levels of detail
Since model* are simplifications, activities that lead to
the same outcome(s) may be grouped to capture the level
of detail necessary for a particular audience A rule of
thumb is that a Logic Model would have no more than
five activity groupings. Most programs are complex
enough that Logic Models at more than one level of
detail are helpful. A Logic Model more elaborate than
the simple one shown m Figure 1 can be used to portray
rr ore detail for all or any one of its elements. For example,
research activities may include literature reviews, con-
ducting experiments, collecting information from mul-
tiple sources, analyzing data, and writing reports. These
can be grouped and labeled research. However, it may be
necessary to formulate a more detailed and elaborate
description of research sub activities for those staff
responsible and if this area is of specific interest to a
stakeholder group. For example, funding agencies might
want to understand the particular approach to research
that will be employed to answer key research questions.
The final product may be viewed as a network dis-
playing the interconnections between the major elements
of the program's expected performance, from resources
to solving an important problem. External factors are
entered into the Model at the bottom, unless the program
has sufficient information to predict the point at which
they might occur.
Resources
Program I
l.SUff. Us*.
Management |
f
Program I
{.Staff. U*.
Management 1
f
Prugtmi 1
S.SUfT 1 e»
"*-— ~
!
Commercial 1
S. Staff 1
^\
Acnvnn*
PwTomi L^afc-
Research I"*1
Develop 1
Technology |~^
IOoptoy L.
Technology r^
Produce 1
technology 4 L-e*.
Educate Market |
^v
nces: Price of Ml and
and need ft
Outpun
Ideas for I
technology l_
dung, p~
Lap Prototype 1
"•port p"
Politic 1
InCMttVCSi ^^i*
Information |
Manufacture 1
In market ' |
S
uecMchy. economic g
for
Customer!
Reached
for Industry I
researchers 1 ~~
for 1
Users and l-s»
Manufacturers |
4(0, I
Users and 1— 01
Manufacturers |
f 1
technology |
^
tooth in Industry and i
Short-term
Outcomes
Leads to
appucattoMia _
energy
technologies
leads to 1
Commercial |>
iTmriialiiisi 1
riwwtjjrpW |
leads to
•ngflwAstfJeM.
less risk
toads to
Ttchnoiofly _•£
accepted.
purchased
f
n general. peicepOo
nt lechnoloov assu
Intermediate
Outcomes
Potential for
technology
"~ change
documented
Technology
Available for
CommercUmaftor
Early Adapters
to buy
1
Consequences
„ of use. lower
tfMify *>v*b
and emissions
/
n of nsk of global cl
1-
-I
Longer-Term
Outcomes-
P... la lane,
FMUUIfJHI
Solution
I
(snared
responsibility)
economy.
1 *• cleaner
••change 1
Fig 3 Logic chart for a research and technology development and deployment program.
-------
70
. L.III«II/III G B Jui dan E:ulnaliiHi tinti
3 5 Stage : (enj\ u>s ilic Logic Moik-l u irh stakeholders
As the Logic Model process unfolds, the *ork group
responsible for producing the Model should continuously
evaluate the Model u-ith respeci 10 its goal of representing
the program logic—how the program works under what
conditions to achieve its short, intermediate, and lona
term aims The verification process followed with the
table of program logic elements is continued wuh appro-
priate stakeholders engaged in the review process The
work group will use the Logic Model diagram(s) and the
supporting table and text During this time, the work
group also can address what critical information they
need about performance, setting the stage fora measure-
ment plan
In addition to the how-why and if-then questions, we
recommend four evaluation questions be addressed in the
final verification process1
(1) Is the level of detail sufficient to create under-
standings of the elements and their interrelationships?
(2) Is the program logic complete? That is, are all the
key elements accounted for1
(3) Is the program logic theoretically sound? Do all ihe
elements fit together logically0 Are there other plaus-
ible pathways to achieving the program outcomes''
(4) Have all the relevant external contextual factors been
identified and their potential influences described1
A good way to check the Logic Model is to describe
the program logic as hypotheses, a series of if, then state-
ments (United Way of America, 1996). Observations of
key contextual factors provide the conditions under
which the hypotheses will be successful The hypothesis or
proposition the work group is stating is. 'If assumptions
about contexiuai factors remain correct and the program
uses these resources with these activities, then it will pro-
duce these short-term outcomes for identified customers
who will use them, leading to longer term outcomes.'
This series of if-then statements is implicit in Fig. 1. If
resources, then program activities. If program activities.
then outputs for targeted customer groups. If outputs
change behavior, first short and then intermediate out-
comes occur If intermediate outcomes lead to the longer
term outcomes, this will lead to the problem being solved.
For example, given the problem of limited energy
resources, the hypothesis might go something like this
Under the conditions that the price of oil and elec-
tricit> increase as expected, i/ihe program performs
applied research, then it will produce ideas for tech-
nology change //"industry researchers take this infor-
mation and apply it to energy technologies, then the
potential for technology changes will be tested and
identified // this promising new knowledge is used
by technology developers, then prototypes of energy
efficient technologies can be developed IJ manu-
Pt.iC'i'"' Planning 22 SIWJ , rt.{-'.'
fjciurers u:>e the prototypes and perceive value and
lo«. risk, then commercially available energv savmc
technologic* wall result If there is sufficient marke:
education ;ind incentives and if the price is right, then
consumers will purchase the new technologies //ihe
targeted consumers use the newly purchased tech-
nologies, then there should be a net reduction in the
energy use. energy costs and emissions, thus making
the economy more competitive and the environment
cleaner
4. Measuring performance
Measurement activities take their lead from the Logic
Model produced by the work group. There are essentially
two purposes 10 measure program performance, account-
ability or communicating the value of the program to
others, and program improvement. When most managers
are faced with accountability requirements, they focus
on collecting information or evidence of their program's
accomplishments—ihe value added for their customers
and the degree to which targeted problems have been
solved Another way to be accountable is to be a good
manager. Good managers collect the kind of information
that enables them to understand how well their program
is working. In order to acquire such an understanding.
we believe that, in addition to collecting outcome infor-
mation, the program manager has to collect information
that provides a balanced picture of the health of the
program. When managers adopt the program improve-
ment orientation to measurement they will be able to
provide accountability information to stakeholders, as
well as make decisions regarding needed improvements
to improve the quality of ihe program
Measurement strategies should involve ongoing moni-
toring of what happened in the essential features of the
program performance story and evaluation to assess their
presumed causal linkage or relationships, including the
hypothesized influences of external factors Wiess (1997)
citing her earlier work, noted the importance of not only
capturing the program process but also collecting infor-
mation on the hypothesized linkages According to
Wiess. the measurement should 'track the steps of the
proeram'. In the Logic Model, the boxes are the steps
that can often be simply counted or monitored, and the
lines connecting the boxes are the hypothesized linkages
or causal relationships that require in-depth studv to
determine and explain what happened.
h is the measurement of the linkages, the arrows in the
loaic chart, which allows the manager to determine if the
program is working. Monitoring the degree to which
elements arc in place, even the intended and unintended
outcomes, will not explain the measurement or tell the
manager if the program is working. What is essential is
the testing of the program hypotheses. Even if the man-
-------
J -I MiLaiiglitin C B Jordan f. aluuimn jml friivniiii Pl.iiiiiniy .'.' .'IW9i 6i-"~
ager observes that intended outcomes were achieved, the
following question must be asked. 'What featurefs). if
any.'of the program contributed to the achievement of
intended and unintended outcomes?'
Thus adopting the program improvement orientation
to performance measurement requires going beyond
keeping score Earlier we referred to Ration's (1*997)
espoused theory of action The first step m improvement
measurement is determining whether what has been plan-
ned in the Logic Model actually occurred. Patton would
refer to this as determining theones-in-use. Scheirer
(1994) provides an excellent review of process evaluation,
including not only methods for conducting the evaluation
of how the program works, but also criteria to apply in
the evaluation
The Logic Model provides the hypothesis of how the
program is supposed to work to achieve intended results
If it is not implemented according to design, then there
may be problems reaching program goals. Furthermore,
information from the process evaluation serves as
explanatory information when the manager defends
accountability claims and attnbutes the outcomes to the
program.
Yin (1989) discusses the importance of pattern mat-
ching as a tool to study the delivery and impact of a
program. The use of the Logic Model process results in
a pattern that can be used in this way. As such it becomes
a tool to assess program implementation and program
impacts An iterative procedure may be applied that first
determines the theory-m-use. followed by either revisions
in the espoused theory or tightening of the implemen-
tation of the espoused theory. Next, the resulting tested
pattern can be used to address program impacts.
We should note that the verification and checking
activities descnbed earlier with respect to Steps 4 and 5
actually represent the first stages of performance
measurement. That is. this process ensures that the pro-
gram design is logically constructed, that it is complete,
and that it captures what program staff and stakeholders
believe to be an accurate picture of the program
Solving the measurement challenge often requires
stakeholder representatives be involved in the planmns
Stakeholders and the program should agree on the defi-
nition of program success and how it will be measured
And often the program has to rely on stakeholders to
generate measurement data Stakeholders have their own
needs for measurement data as well as constraints in
terms of resources and confidentiality of data
The measurement plan can be based on the logic
chart(s) developed for the program The manager or work
team should use Logic Models with a level of detail that
match the detail needed in the measurement Stake-
holders have different measurement needs. For example.
program staff have to think and measure at a more
detailed level than upper management.
The following are the performance measurement ques-
tions across the performance story which the manager
and work team will use to determine the performance
measurement plan
(I) li (was) each element proposed in the Logic Model
in place, at the level expected for the time period''
Are outputs and outcomes observed at expected per-
formance levels0 Are activities implemented as
designed? Are all resources, including partners, avail-
able and used ut projected levels9
(2) Did the causal relationships proposed m the Logic
Model occur as planned7 Is reasonable progress being
made along the logical path to outcomes0 Were there
unintended benefits or costs0
(3) Are there any plausible rival hypotheses that could
explain the outcome/result0
(4) Did the program reach the expected customers and
are the customers reached satisfied with the program
services and products?
A measurement plan will include a small set of critical
measures, balanced across the performance story, that
are indicators of performance. There may be strategic
measures at a high level of detail, and tactical measures
for implemented of the program. The plan will also
include the important performance measurement ques-
tions that must be addressed and suggest appropriate
timing for outcomes or impact evaluation This approach
to measurement will enable the program manager and
stakeholders to assess how well the program is working
to achieve its short term, intermediate, and long term
aims and to assess those features of the program and
external factors that may be influencing program success
5. Conclusion
This paper has set forth for program managers and
those who support them the Logic Model tool for telling
the program's performance story. Telling the story
involves answering the questions. 'What are you trying to
achieve and why is it important0', 'How will you measure
effectiveness0', and 'How are you actually doing0' The
final product of the Logic Model process will be a Logic
Model diagram(s) that reveals the essence of the program.
text that descnbes the Logic Model diagram, and a
measurement plan Armed with this information the
manaaer will be able to meet accountability requirements
and present a logical argument, or story, for the program.
Armed with this information, the manager will be able
to undertake both outcomes measurement and improve-
ment measurement. Because the story and the measure-
ment plan have been developed with the program
stakeholders, the story should be a shared vision with
clear and shared expectation of success.
The authors will continue to search for ways to facili-
tate the use of the Logic Model process and convince
-------
\ULaughlin C B Jordan, Evaluation ami frnvrmn Planning _V f/W9/ 65-72
nuinjgurs and stakeholders or the benefits of its use.
We welcome feedback from managers, stakeholders, and
facilitator:) who have tried this or similar tools to develop
and communicate a program's performance story.
Acknowledgments
In addition to the authors cited in the references, the
authors thank Joe Wholey, Jane Reismann. and other
reviewers for sharing their understanding of Logic
Models. The authors acknowledge the funding and sup-
port of Darrell Beschen and the program managers of
the U.S. Department of Energy Office of Energy
Efficiency and Renewable Energy, performed under con-
tract DE-AC04-94AL85000 with Sandia National Lab-
oratories. The opinions expressed and the examples used
are those of the authors, not the Department of Energy.
References
Bickman. L. (1987) The functions of program theory In L Bickman
(Ed ). Uwig program ilieon m naluanon New Directions for Pro-
gram Evaluation, no. 33 San Francisco Jossey-Bass.
Chen. H.T (1990). Theor\-dnci:n evaluations Newbury Park. CA Sage
Corbeil. R. (1936) Logic on logic models Eialuanon newsletter
Ottawa: Office of the Comptroller General of Canada. September
Jordan. G B . & Mortensen. J (1997) Measuring the performance of
research and technology programs' a balanced scorecard approach.
Journal of Tethnolog\ Transfer. 22. 2. Summer.
Jordan. C B.. Reed. J H.. & Mortensen. J C. (1997). Measuring and
managing the performance of energy programs: an in-depth case
Presented at the Eighth Annual National Energj Services
Conference. Washington. OC. June
KosXmcn. J A (1997) Office of Management and Budget Tesnmon>
Before the House Committee on Government Reform and Over-
sight Hearing February 12
McDonald. R . & leather. G (1997) Science and technology policy
evaluation practices in the Government of Canada Policy evaluation
in intimation and lechnologr lonards Best practices Proceedings of
the Organization For Economic Co-operation and Development.
Montague.S (1994) The three R's of performance-based management
Focus. December-January
Montague. S (1997) The three Rs of performance Ottawa. Canada
Performance Management Network. Inc. September
Palton. MQ (1997) Utiliiation-focused Evaluation the nen ceniurv
te.tt Thousand Oaks. Sage, pp 221-223
Rush. B . & Ogborne. A. (1991). Program logic models expanding their
role and structure for program planning and evaluation Canadian
Journal of Program Etaluaiion. 6. 2.
Scheirer. M A (1994) Designing and using process evaluation In Who-
ley et al. (Eds). Handbook of practical program evaluation, (pp 40-
66)
Torvatn. H. (1999). Using program theory models in evaluation of
industrial modernization programs: three case studies. Ecaluanon
and Program Planning. 22(\), 73-82.
United Way of America (1996). Measuring program outcomes' a prac-
tical approach. Arlington. VA: United Way of America.
Weiss. C (1997). Theory-based evaluation: past, present, and future.
In D Rog & D Foumer (Eds). Progress and future directions in
eialuanon- perspectires on theory, practice, and methods New Direc-
tions for Program Evaluation, no 76 San Francisco Jossey-Bass.
Wholey. J. S (1983) Ecaluanon and effectire public management Bos-
ton. Little. Brown.
Wholey. J S. (1987). Evaluabilty assessment: developing program
theory In L. Bickman (Ed). Using program theor\ tn eialuanon
New Directions for Program Evaluation, no 33. San Francisco
Jossey-Bass.
Yin, R K (1989) Case study research: design and methods Newbury
Park: Sage pp 109-113.
-------
7 Using Piogram "I hcoiy to Ucphcalc ^iiccitslul Piogiams 71
Tumidly A //(KM
Replicating successful piogiams is cxiicincly dillicull, progiam llicory
i .in help piovide inloniiiiiioii foi he tier icphc-.uion .ind adaptation ol
programs
8 rheory-liased Evaluation Gaming a Shared Understanding 79
Between School Stafl .uul Livalualors
I nicy A Miicbiifi
I licory-h.isccl cvnluaiion c.in enhance staff support Tor and under-
standing ol cvalii.inon, n can also entourage reflective practice
9 Developing and Using a Program Theory Matrix for 91
Piogram Evaluation and Performance Monitoring
Sue C I iiiiiicl/
An approach lo program llicory cvaluanon is dcsciihcd dial encom-
passes peilomiancc inoiiiioring and the broader contexts that affect
piogiams
10 Summing Up Piogram "I heory 103
(fiinniil liitlnntiH
1 lie issues i.useil in the volume aie suininarized. analyzed, and placed
in the context ol oihci developments in piogiam iheoiy and evaluaiion
INDLX 113
EDITORS' NOTES
li has been more than thirty years since evaluaiors lusi undnsinied ilu
advantages in making explicit and testing program tlu-oiy—ih.it is..UK
uiideilymg assumptions about how a program will woik to achieve intended
outcomes And it has been ten years since the last of two issues of Noi
Dilation* Jin Pnifttam Evaluation (the former name ol tin- sciics) w.i-.
devoted lo providing evaluators with some of the tools needed to i.uiy oni
piogram theory evaluations in practice and helped refine the many concep
ttia I and pi actual issues involved Since those volumes were published, tlu
liontiers ol cvaluanon have expanded lo meet new i hallenges-, such as pei
Ioj inanec JiKJSUiemcnl. organizational learning, cnllahoiaiivc- .uul paim i
patory tcscarch, and mcla-analysis
In I'ntgiain I hcoiy in Evaluation Challenge*, mid ()/)/IOIMIMIIICS, wi
examine (he real 01 potential role for program llit-oiy m these nrwei an-.is
Ihu some thorny issues remain foi cvaluatois implemrniiug such evalu.i
lions, pailiuilaily m legard lo causal mfncncc I ollowmg the liiiioduiiinii
Pail One niLludes lour ihapters that address some ol these tliallcngcs
Despite some persistent cjuestions, there arc opportuiiiiies loi piogiam ihr-
ory lo help evaluators in areas such as performance mcasuiemc-ni and mri.i-
analysts Part Two includes four chapters that discuss (his potential I In-
volume's summary chapter belongs lo Leonard Bickman
In Chapter One, the editors assess the current state ol piogiam ilu-oiy
drawing on then own review of the available htcratuie They use ilu- icvicu
as a backdrop to discuss program theory's histoiy, its complexiiy ol deliiu-
tions and variation m lexicon, us diversity in application, and us stiengili>.
and limitations in practice
Some of the challenges in implementing program ihcniy m rvaluaimii
(iiaciiLe aie addiessed in Part One A majoi issue lor c-valuaiois is causal
ink-rente in progiam theory evaluations In Chapter Two, Jane Davidson
aigucs I ha I randomization is but one of several ways (hat scientists .K know!
edge can establish causal inference and that evaluation chenis olu-n acctpi
lessei standards ol prool about causality to get the information ihey need
She offers that a program (heory evaluation could meet the lowei siand.uds
of prool required in settings that cvaluatois often Imil iheiusdvcs in In < mi
trnst, Thomas Cook argues m Chapter Three that piogiam tluoiy is not sul
IK lent to establish causal inference Raihei than "falsely choosing' lujw«-i n
landomization and progiam theory ihe cvaluat»)r tan iiKike_ilie"iip:imi'.'il
clu)ice_.ind combine both "
In addition to the complexities associated with causal minimi-, ihrii
aie challenges in deciding what types of progiam them les 01 models lo lesi
with an evaluaiion 1 heie are often a numbci ol plansilili iluoni's a!>
IIDVV a piogiam woiks and an abundance of links lo lonsidri In
-------
I'KOl.KVM
IN I VAI |l\IU>N
IIII cits' Ni III s
world, I he cvalualor would he .iMc ID siiuly llicm all lUil as Caiol Weiss
wines in Chaplci Font, (he cvalualor has lo make simplifying (.hours She
goes on lo provide guidance lor cvalualors who are pondering which links
in wlin.li ihcoiv ihey should test In Chapter 1'ive, P.uncia Rogers shows
how simple models — usually depleting causalion like a ch.un ol domi-
noes — may not accuialely rellecl how progiams work She provides exam-
ples ol how cvalualors have developed .more realistic causal motlfls and
makes (he point thai eomplexily is not the goal, hut rather useful "maps"
that inloim suhsequeni decisions " '
Pail Two exploit's the potential for program theory to make coninhu-
nons in evaluation's new frontiers. The advent of mcia-analysis has meant
lh.it inoic ev.iluatois not only aie asked to make sense ol multiple evalua-
tions hul also aic picssuicd lo impiove their own sludie.s foi siihsci|iiciii
ievie\\s In Clhaptei Six. Anthony Peliosmo deiiionsiiaies how t ven simple
piogiam llicoiy evaluations could be used in nicla-analysis to accumulate
Knowledge ticncializing liom a progiam ilieoiy cvahiaTltnTTii oiie"scmiig
lo"7iM"Tval\i.iiion in the next setting is Timothy Mat si's i OIK. cm in C'hapiei
Seven Allei leviewmg some hasu pioblcnis in piogiam diffusion, he shows
how the subtle inloimaiion geneialed by piogiam theory evaluation oilers
an alternative to cuncnl models loi replicating innovations
Another challenge laced by cvalualors is ""^iil* |>|i'-"i IIM llu'l'
In Chapter Eight, Tracy Huebncr provides several illustraiive case studies
to show the value-added of prouram theory in eduraiipn.il cvn|ii.inon Hueb-
ncr finds that the approach helped cvalualors coordinate their goals with those
of school sialf and reduced the normal lesisiancc that teachers and adminis-
iraiois olien have to "yet another study," pailicularly one thai iec|uncs them
lo collect the data.
Lvaluatois aie often being asked lo develop systems foi moniiormg per-
lormaiuc In Chapter Nine, Sue Funncll outlines the pioblcms encounieied
in developing such systems and draws on hei experience to show how pro-
giam theoiy can help cvalualors develop momionng tools that make sense
llcf matrix^ can be used lo draw out undei lying theories about why agencies
or programs should succeed, to identify indicators in which measurement
is needed, and lo ensure that important external factois outside the bound-
aries ol ihc oigamzation are also monitored
l-'mally, in Chapter Ten, Leonard Bickman summanzes these contribu-
tions and outlines a luture agenda for progiam theory evaluation As editor
of iwo seminal New Diinfions/iii Piogumi Lvahiatuin volumes on piogram
ihcoiy, liickman has ihe perfect perch Irom which lo critique (he volume
and oiler his observations and predictions
We believe this volume olfeis hope and piagmatism Piogiam ihcoiy
tan help evaluaiois ineei some ol (he new challenges I hey laic Ikil
implc' 'ing piogiam ihcoiy evaluation ceilamly ofleis Us own ih.il-
lenge |uaiul.uies Om hope is that llus Nrw Dim (ions issue noi only
will snmulale thinking about program ihcoiy evaluation bin also—am.
even mote important—will result in an incicasc in real-woild tests ant
applications
Anthony I'eliosino
Pal IK i.i | Itogris
1'iacy A Iliielmei
Timoihy A I lac si
I'dilois
ANMIU.VI P/-/ROS/VOIS icwaich fellow at theCentci /«» /•.vdfiKKioii,
/()/ Lluhln'n PKI^IHIM, Ant cue an Academy o/Arf.s anil Suriu i s, mid n si an It
msec KI/C at (In- llai vaid Graduate School of Education
/MINK IA J RtH.l K's is dinctoi of the Progicim/oi PII/'/IC Srdci / UI/IKIIIHII in
(hi1 ( / A/)/)/iC(/ Si initc, Royal Mclbouinc (nsliliifc o/ li < hnoln^y. AIIS
/\ /(( 'I li.vi K is cooidinaloi fin compiehcnsivi- s< hool n foi in al U'rs/J-J
//MOM/) A f/-u si is ii'M'dK/i fellow at r/ic lldivnid ( h 'is tnilmln't mid
Inn In s liislni y ill l/ic lltiivaid l:\lcn\ion .Sdnml
-------
1 IK /nsfoi it (if M>£> \.ni-
annns ol I11 r. in pi.\IIKC and inui_h to rccommciul n And elemenis ol
PIT, whetliei (he evalualors use (he terminology 01 not, aie liein^ used
in a wide lange of aieas of contern lo evalualors llased on this ii-view. in
ibis (.hapiei we dismss (he pracdie, promise, and pioblems of I'11
What Is Program Theory Evaluation?
Ik'iause llns volume is intended lo demonstrate the diveisily ol pia< lue,
we li.ivi usul a bio.id delimiion of program llieoiy evalualion We i onsidi i
it lo have iwo essential componenls, one conceptual and one empiiKal
P I'll consists ol an exphcil theory or model of how the piogiam < auscs ilic
intended 01 obseived outcomes and an evalualion that is al least p.uily
guiiled by ibis model llns definilion, I hough dehbei.iiely hioad, dots
exchule some veisions ol evaluation that have the woid iliroi v .111.u hi il lo
limn It does not covei all six types of theory-diiven evalualioii delined by
Chen (IWO) but only (he type he lefcrs lo as ind'ivcuni^ mc
-------
l'Klll.K\M III! i >K\ IN I \ M I • \ I U IN
piogiam hul lh.il ill) nol use I IK- ihcoiy lo guide ihc evaluation Nor docs
M iiuliulc evaluations in which I he piogiam ihroiy is a lisl ol acliMlics, like
a "to do" lisl. tailici than a model showing a sciics of inii-niutliau- out-
comes, ui mechanisms, hy whu h (he pi obtain activities aie uiiiU i stood lo
lead ID I lie dcsiied riuls
The idea ol basing piogiam cvaluaiion on a causal model ol llir pio-
giam is noi a new one Ai least as lai hack as I he Notts, Suihman .suggested
thai piogiam evalnaiion might address ihc aclueveineni of a "chain of ob|cc-
uves" ( I9f>7, p 55) and aiguccl lor ihc henefit ol doing this " I he evalua-
tion siuily lesis some hypothesis lhat activity A will aiiain objective I!
because u is ahle to inlluciur pioccss C which alfecis the occuiicncc of ilns
oh|i-uivc. An mule-islanding ol all three laclois — program, oh|ccnvc and
mici veiling process — is essenii.il 10 I he conduct of evaluative icscauh"
(I9d7.p 177)
Weiss ( 1072) wciii on 10 explain him an cvaluaiion could uleniily scv-
eial possible causal models ol :i leachci home-visiting piogiam and could
dcieiniine which was the hesi as suppoilcd by cvulcnic In the thicc
decades sinie, many dillcicnl lei ins b.i\e been used loi this t\pe ol cvalua-
iion, including iiiihomcs liicnin Jms (licnncii. 1975) and (Ino) y-oj-m fi<»n
(Si hon. 1997) Moie commonly, the leiius /JIO^KIIII »/ic«»iy (Milkman, 1987,
1990), fJirmv-luisuf i-viiliiiilinii (Weiss, 1995, 1997), and /)ic>g»K«II li. has now, in Us sixth edi-
tion, added a chaplei on tins appioach (Uossi, I iceman, and Upscy, I99«J)
Snnil.nl), / viiliiiifioii Models l-'vtmnKioii <>/ i.tliuiitiomil unil Stnml P>OJ>KIIIIS
(Maclaus. Si -earn, and Sciiven. I9H J) lias added a ihaplei on program
llieoiv exalt . in its second edition (Uogeis, loiiluonnng)
I'lMl.KVM till iM!N I \ M UAIIXN I'KA( I K I . I'KI IMIM . >\NI> l'l<( UII I Ms 7
Praclicc: Diverse Clioiccs lo Meet Diverse Needs
Piogiam iheoiy is know by many dilleieni names, ucalcd in many dillu-
cut ways, and used loi any number of puiposes lleie we piovulc a bnel
load map to the vainly ol ways people think about and employ piogiam
iheoiy
Locating Examples, lo ny 10 unileisiand ihe vanety ol ways m whu li
piogiam iheoiy evaluation is now being used, we began in eaily I1WH 10
comb thiough available bibliogiaphical databases, cilalion indexes, and eval-
uation K polls We also leviewed conference pioccedmgs, dissertations, and
ailicles bom a vaiiely of disciplines In addition, we received many helplul
examplis in lesponse lo an nu|Uiry lo the Amencan Evaluaiion Assonaiion's
Iniiinel discussion list. IIVAIJALK Oui cffoils turned up examples dalmg
horn I1)')/ to 2000 horn (he Umied Slates, Canada. Ausiiaha. New /calami.
and the United Kingdom We have not included every example that we
locaied m this volume bin instead have used examples lo uleniily and illus-
liale 11 nical challenges m using program iheoiy or ways ol addiessmg ihem
Oui ievie\v showed amazing diversity in theory and piaiiue auoss IM-O
mam aieas—how piogiam theories are developed and how they aie used lo
guide evaluations
Developing llic Program Theory—Who, When, and Wlial. In
some evaluiinons. ihe piogram theory has been developed laigely by ihe
evalualoi. based on a icview ol research hleialure on snnilai progiams 01
iclcviini i.uisal mechanisms, through discussions \\nli key mloimains,
ihiough a ieview ol piogiam documentation, or through obseivaiion ol ihe
piogiam ilsell (Lipsc) and Pollard, 1989) In oihci evnlu.itions, the pio-
giam iheoiy has been developed primarily by those associated wnh the
progiam. olten ibiough a gioup piocess Many praciilioneis advise using
a combination ol these appioaches (Pawson and "lilley, I9l)5, Pallon, llM)(i,
see also I unnell, C hapler Nine)
Ihe progiam iheoiy can he developed before the piogiam is imple-
mented or altci ihe piogiam is under way At limes, u is used lo change pio-
giam piactice as the evaluation is beginning Most piogiam iheones aie
summarized in a diagtam showing a causal chain Among ihe many van.i-
nons we Mill highlight |usi three for IIOM-, Rogeis discusses oiliei vanaiions
in Chaptct I iv c
At us simplest, a piogram theory shows a single iniermediaie outcome
by which the piogiam ai hieves its ullimaie outcome I 01 exampli. m .1 pio-
giam designed lo leiliui substanie abuse, we miglii lest whethei 01 not the
progiam succeeds m i hanging knowledge about possible dangeis .mil then
wheihei 01 nol this seems impoitant in achieving the desned hehavioi
change As Peiiosmo (Chapter Six) points out, loi some pmgiam aieas,
artiiulaling this mediating variable and incasuimg il would ' • signilu.ini
advaiue on iiineni piaclice
-------
K I'KiU.KAM IlllOin IN I VAIUAIION
Moic complex piogiam thcoiics show a scncs ol mlci mediate out-
comcs, sometimes in multiple strands thai combine to cause I lie ultimate
outcomes So loi a substance abuse picvcntion piogram, we might llieonze
lh;ii .111 cllcclivc piogi.nn will gcnciatc a positive reaction among partici-
pants, change both aldliules .nul knowledge, and develop panic ipants skills
in icsisling peei picssuic Although these more complex progiam llieones
may moie adequately repicsent the complexity ol programs, it is impossi-
ble to design an evaluation that adequately covers all the factors they iden-
tify Weiss (Chaptct Four) proposes some ways to select the particulai causal
links that any one evaluation might study
The ilind type ol program theory is represented by a series ol boxes
labeled m/mls, /mxrsM'i, outputs, and outcomes, with arrows connecting
them It is not specified which pioccsscs lead to which outputs Instead the
difleient components of .1 progiam theory are simply listed in each box
Although this type of progiam theoiy does not show the iclalionships
among dilleienl components, these lelalionships aie sometimes c.xploiccl in
the eiiipnit.il component ol the evaluation
Using I lie Program Theory to Guiilc the Evaluation. Piogiam thc-
01 y has been used in quite dilleient ways lo guide evaluation Lxamples
show diveisny in the purpose and audience ol the evaluation, the type ol
icscarch design, and the type of data collet led Within this diversity, it is
possible to identify two bioad clusters of practice
In some PTEs, the mam purpose of the evaluation is to lest the pio-
gram theory, to identify what it is about the program that causes the out-
comes This sort of PTIE is most commonly used in large, well-resouiced
evaluations locused on suth summative questions as, Does this piogtam
work7 and Should tins pilot be extended7 These theoiy-iesling PTEs wres-
tle with the issue of causal attribution—sometimes using experimental 01
quasi-experimental designs in conjunction with program theory and some-
times using progiam theory as an alternative to these designs Suth evalu-
ations tan be particularly helplul in distinguishing between theory failure
and implementation failure (Lipsey, 1993, Weiss, 1997) By identifying and
measuimg the intei mediate steps ol program implementation and the ini-
tial impacts, we can begin to answer these questions These intermediate
outcomes also provide some interim measure ol program success foi pio-
grams with long-term intended outcomes
An example of (his type of program theoiy evaluation can be found in
the I amily llmpowcimcnt Pro|ctl evaluation, in which Bickman and col-
leagues (1998) concluded an experimental test of the elfetts ol a progiam
that named patents to be strongci advocates foi clnldien in the mental
health system I hey articulated a model of how the program was assumed
lo woik l:nst. patent naming would increase the parent's knowledge, scll-
ellicacy, and advocacy skills Second, parents would then become more
.•ivolml m tin ii c hild's mental health care I inally. this collaboration would
l< nl IM ilii i Inlil's unmoved menial health ouU nines
I'KOi.KAM IIIIOKY I.VAIUAIION I'KAC lie I . I'KDMI .1 . AND I'KHIII I Ms 9
Ikil they did not slop wnh the ailiculalion ol a piogiam iheoiy I lit y
also constituted measuies, collected data, and analy .eel them lo test iluse
nuclei lying assumptions The program was able lo a* Ineve statistically sig-
nificant ellects on parental knowledge and scll-clhcat y, but no useliil mea-
suies loi testing advocacy skills could be found Unloiinnately, the
iniei\enlion had no apparent effect on caregiver involvement in iie.iimeni
or sci vice use and ultimately had no impact on the eventual menial health
status ol the children
(.valuations such as these seem to be ai least implu illy based on Weiss's
delimlion ol program theory, "|lt| refers lo the nietlmiusm* thai mediate
between the delivery (and receipt) of the program and the emeigence ol the
outcomes ol interest" (1998, p 57)
I he other type ol ptogram theory evaluation is often seen in small evalu-
ations done at (he pm|cct level by or on behalf ol pio|t el manageis and stall
In these cases, piogiam theory is more likely lo be used loi lonnalive evalua-
tion, ID guide then daily actions and decisions, than loi summalive evaluation
Such PI l.saie olten not coincided with causal attiibuiion Although some ol
these evaluations pay attention to the influence ol external l.iciois, theie is
taiely systematic lulmgoui ol nval explanations foi the outcomes Many ol
these evaluations have been developed in response lo ihe me leasing demands
lor piogiams and agencies lo report performance mloimaiion and to demon-
strate their use ol evaluation to improve their services In these c ire umstaiu es,
P IT. has often been highly regarded because of the bcnelits il piovides to pro-
gram manageis and stalf in teims of improved planning and management, in
addition lo us use as an evaluation tool
Stewart, Cotton, Ducked, and Mcleady (1990) pi ovule .m example ol
this type of PTE in then evaluation of a project that lecruiied and named
volunteers to provide emotional support for people with AIDS, ihen loveis,
families, and friends I'hc paper did not provide a diagram of the piogiam
theory model nor present any data Instead Stewart and colleagues lepoited
on the piocess ol developing the model, the types ol data dial weie gath-
ered, and how the data were used "Performance inchtaiois developed by
Ankah |ihc project| weie both foi the organisation's own (imposes and to
meet the requnements ol the funding body Both qualnaiive and quantita-
tive indicators were selected Ankah now uses the outcomes lueiaichy
dining orientation of volunteers and report|s| thai the piotess has assisted
with improved laigeiingol volunteers and referial agencies, modification to
the naming piogram, supei vision of clients and volunleeis. and devt lop-
mem ol pioposals loi expansion and enhancement ol the seivue' (1990, p
317)
This type ol piogiam llieoiy evaluation appeals to In- ilosei in ilia)
described by Wholey |li| identifies progiam lesouices, piogiam ac nvities,
and intended program outcomes, and specifies a chain ol causal assump-
tions linking progiam lesources, activities, intermediate oiiicoim s and iiln-
mate piogiam goals" (I9K7, p 78)
-------
10
Inioii'i IN I \ AI OAIIDN
Despite the apparent popularity ol piogram ihcoiy evaluation, we
louncl ih.il I he loi mat evaluation liictaiurc snll has comparatively lew exam-
ples I'oi instance, when we scaichcd the abstracts in six hibhogiaphical
databases lor the lime liamc 1995-1999, we lound program theory exphc-
iily mentioned m evaluations ol cliilihen's pmgiams only twice In aclclilion,
many ol the evaluations thai we louncl used theory in very limited and spc-
cilic ways, loi example, lo help plan an evaluation, hut very few used the-
ory as extensively as the most prominent proponents ol this approach
suggest But PI l:s conducted in small projects or local sites are rarely pub-
lished in lelereed journals or distributed widely, being more likely lo be
picscnicd as conlcicncc p.ipeis by piaclitioners or picsented m pcrfoi niancc
measuiemeni (oiuins And many of them lail to include what some would
considei an essential component ol a program ihcoiy evaluation—
systematic testing ol the causal model
In this volume, we include examples of both types ol I'll. Weiss
(( haptei Tour), ll.usi (C.haptei Seven), and IVirosmo (Chaplei Six) dis-
cuss issues associated with //icoiy-rc.sdn^ /'/i:s llucbner (Chaptei F.iglit)
discusses loin examples ol adion-guidiHg /'//is, and Pimnell (C haptei
Nine) discusses a technique lor assisting with this sort of I'TE
Promises and Problems
Progiam theory has been seen as an answci lo many different problems m
evaluation lleie we briefly discuss several areas wheie program iheoty has
been seen as promising
Understanding Why Programs Do or Do Not Work. Among the
promises made lot PTIi, the most tantalizing is thai it piovides some clues
to answci the question of why programs work or lail to wotk Considei the
usual pi act ice of Hying to mule-island why a ptogram succeeded 01 failed
rollowmg repotting ol results, cvalualois usually woik in a post hoc man-
ner lo suggest icasons lor observed lesulls (Pelrosmo, forthcoming, 2000)
But without data, such post hoc theories are never tested, and given the
poor slate of lephcation in the social sciences, they are likely ncvei to be
In contrast, by creating a model of the miciosieps or linkages m the
causal path from program lo ultimate outcome—and empirically testing it—
PTI. piovides something moie about why the program failed or succeeded
in reaching the distal goals it had hoped to achieve, as in Utckman and col-
leagues' evaluation of the lamily empowerment program (1998) Perhaps
the mieivention was not able lo impiove advocacy skills—lemembei, those
could not be measuied Oi maybe there was a critical mechanism missing
horn the model, which the piogiam was not activating 01 engaging We
learn something mote than the progiam's apparent lack ol impact on clul-
clien's ii> health
I v these issues t annoi be adequately addiessed in the oiigin.il
evaluation.., i> 11 can piovtcle an agenda loi the next piogiam and evaluation
I'KOl.H \M I III HICl I \AI IIA I It IN I'KAt IK I . I'KOMIsl , ANI> I'KUIII I Ms
II
I 01 example, a (.ritual link in the Uickman and colleagues study was not
tested (advocacy skills acquisition), given the paucity of measuiemeni devel-
opment in this aie.i Pointing out this deficiency suggests an agt nd.i to
develop an mstiument to measure this variable in the next similai study
Attributing Outcomes lo the Program. Another piomise sometimes
made loi P11: is beliei evidence for causal attribution—to answci the qucs
lion ol wbeihei the piogiam caused the observed outcomes Progi.im tin oi>
has been used by cvaluaiors to develop better evidence lot aiiiihuimg out
comes 10 a progiam in circumstances where random assignment is not pos
sible (loi example, lloinel, 1990, in an evaluation ol random hicaili listing
ol automobile chiveis) In the absence ol a countcrlactual, suppoii lot i.msal
aitiibuiion can come Irom evidence of achievement ol intermediate out-
comes, inxcsiigaiion ol altcinative explanations foi otilioincs, and patiein
in.HI lung Suppoii lot causal attribution can also come lioin piogiam stake-
holclei assessmems (loi example. Funnel! and Mogiaby. 1995, m then eval-
uation ol the mip.ici of piogiam evaluations in a ro.icl and liallu .iiilhoniy)
01 horn dal.i about .1 i.mge of indicaiors, including data on t xteinal l.u tins
likely lo i n Hue nee the ihconzed causal pathway (foi example. Waul, Maine,
McCaiihy, and Kamaia, 1994, in their evaluation of activities to icclucc
maternal moiiahiy in developing countries). It may be possible lo develop
testable hypotheses on (he basis of (he causal model (Pawson and lilley,
1995). especially il the model includes contingencies or clilleieiitiaiion—
expec'ted chllciences m outcomes depending on dillerences in coniext
Causal atlitlnition is also sometimes addressed by combining traditional
expenmeni.il or quasi-experimental designs with P'l I.
Many P'l l:s do not address attribution at all, simply leporimg implc -
mentation ol activities and achievement of intended outconus I his
apptoac h is parliculaily common where program ihcoiy is used in develop
ongoing monitoring and peifoimancc information systems ( ausal ainihii-
tion in P11 s is disc ussed in moie detail in the c hapleis in tins voluim by
( ook (C haptei Ihiee), Davidson (Chapter Iwo), and Mac si (C liaptei
Seven)
Improving the Program. Many of the claims lor the licneliis ol I'll
relei lo us capacity to improve progiams directly and indiiectly Ann ulai-
ing a piogiam ihcoiy can expose faulty thinking about why the piogiam
should woi k, wine h can be corrected before things aie up and iimnmg ai
full speed (Weiss, ll>95) Hie piocess of developing a piogiam ihcoii (.m
itself be a tewaicling expedience, as stalf develop common uiuK islanding ol
then wot k and identity the most important components Man) an minis ol
PI I- (such as Milne. 1993, and lluebner. Chapter I ighl) lepon that this lias
been the most positive benefit Irom conducting PI I: In this wa\. I'11 is
veiy similai lo Hie cailiei technique of evaluabihly assessment
Ihil PIT is supposed lo then use the program ihcoiy ' -ult tin eval-
uation, and n is lu n that some evaluations falli i When lahoi.itivcly
building a piogiam ihcoiy can be an energizing team .ulivi,,, exposing this
-------
12
I'KIM.KXM I Ml Din IN I VAIH.MION
to haibh cmpmcal tests (..in he less amactive h.iciic.il difficulties abound
as well When PII: is iniplcincnied at a small project, staff may not have the
lime or skills lo collect and analyze data in ways thai cither test the program
theoiy or provide useful information to guide decisions and action If pro-
gram theoiy is used 10 develop accounlahility .systems, there is a real n.sk ol
goal displacement, wherein stall seek to achieve targets and slated ohjcclivcs
at the cost of achieving the ultimate goal 01 sustainahility of the piogram
(Winston, 1991)
Conclusion
In ihischaptei, we have outlined the range of activity that can he consul-
cird piogt.ini iheoiy evaluation .mil have idcnlilicd major issues in Us ihe-
oiy and piaeliie These arc discussed in moie detail hy the other ch.ipieis
in this volume
References
Benncll.C- "Up I he I lu-rari hy " /oiumil <>/ l:\lrn\nm, 1475. IK2).7-12
UK kin Jii, I (cd ) l/Miij;''n'X'/ llir Aiisl»dldsirodili Alcxandiia, Va Unucil Wuy of Amenta, 1996
Home I, K "Random Brralh Testing in New South Wales The Evaluation of a Suuess-
ful Social Lxpcriiuenl " National Evaluation Confeicnie 1990, Piotmlmgs, vol I Aus-
ualasian Lvaluaiion Society, 1990
Lennc. B , and Clclaiul, II "Describing Program l-ogu " Pio^iam L'viiliuinun fiiillcdii
1987, no 2 Puhlic SCIVKC Uoard of New Soiuh Wales, 1987
I i|>si-y, M W " Iheoiy as Method Sin.ill 'I heoncs of I iiMlinenls " In I. Scihiisi .mil A
Si DI I (eils ), I'm/< i slum/nig Giusrs and CifiicMilicmj; Abtwt (firm New Duec HOIKS foi
Piogi.un I v.ilii.iiion. no 57 San Traiu-isco Jossey-ll.iss, 1993
I ipsey, M W , .mil Pollard. J "Driving loward I linny in Program Lvaluaiion More
Models 10 C liiio.se I loin " l.viiliiiiliDii unit I'lvgnnn Pliiiiiiiii)>. 1489, 12. M7-32S
Mad.ius, (• , Siiilllehc.ini, 0 , .mil Suivcn, M I vuliiiilinii Mm/rl\ I viidiiitinii <>/ (ihn uliiniiil
and SIKml Pio^'diiis Norwrll, M.iss Kluwcr, I9H1
Milne, C '(hiuomes Iliei.iulues .mil Piogi.im I ogu .isCoiueplu.il loots I IM ( .ist
Siudii-s ' Papei presented .11 the inieinalional conlcicnie ol the Ausli.il.i.sian I v.ilu.i
lion Sourly lliisli.me, 1993
P.iiion, M (.) Ulilizalnin-l IIIIIMI! / viiliiiiinui (3ulid) I lioiis.ind O.iks, C .ihl S.ig>.
|'I;H(.K\M Inunn IV.MIIAIIIIN PRAI IK i , PKHMIM , ANU Pitt mi i MS I }
Piliosino, A I 'Aiiswiiing I he Wliy Question in I v.ilii.iiion I lit I .ins.il Mmli I
Appio.u h " ( (iiuiilnm /OIIMKI! oj 1'iogiam l-.valuation. 2000, I >( I ), I -2-4
Rogiis P | "Piogram I In ory I. valuation Nol Whciliri Piogi mis Work Km Wh> In
d M.id.ms II Suirilelu .1111, anil T Kelleher (cils ), / vtiliKiiidii Mixfcls / iiiliniliiiii n/
( dm iifiiuuil iiiiif Sni iiil ('ioniums Norwell, Mass Kluwei, loiiluommj;
Rossi, P II . I itiin.in, II . mil I ipsey. M W Pinxniiii I vuliiniiiiii A Sysli'iiiiidi Afifiimnd
I hoiis.mil O.iks, ( ahl Sage. 1999
Si lion I) \ 'Ihroi) ol-<\> lion Lvaluaiion " Papei piesenlrd in ihe llarvaul I \.ilii.iiion
lasL I OKI-. Api 1997
Sle\\arl. K , ( ollon, K . Duckcll, M , and Mcleady, K " I lie New South VValis Piogiam
I ogu Modi I Ihe I \pinenie of the AIDS Bureau. New Soulh Wales Di p.uliiienl ol
lle.ilih ' Pniiiiiliii^s d/ (lit1 Aniiuiil Ciiii/ricnir oj (he AusOnfuiiciii f.viifnuniiii Smniy,
1990,^. II ^-322
Sin Inn, in I \ (viilinilMi Kcsiiixd I'liiifiplri anil 1'iailiii in 1'nhln SMVIII niu/ Sunnf
Ailh'ii l'iiiv;iiiiii\ Ni \\ Noik Itussell Sage I oundalion, I 'Hi 7
W.iul \' M M.inu. I) . Mi( aiihy, | .and K.imaia. A "A Snaltgy loi llu I v.ilii.iiion ol
At IIMIIIS in Kediiit Moil. ilny in Developing C.ounliies " I vdliiiilinn Id vn if. 199-1, IH.
Weiss. C II ( iiiliidduM KiM-iinli Mflli(»i/so/As5fSMM^ Pii^iiuii / //nlivriii \\ I nglcuood
Chlls N| Pienliiellall, 1972
Weiss C II Nolhmn As Prai Dial As Good Theory I xploring I heoiy llasid I valua-
tion loi C ompieliensivi Community Initiatives foi (.Inldii n and I ainilii s ' In | P
Council, A C Kuhisih I B Schorr, and C. M Weiss (eds ), Ni u< Af)fiiiiiiilii sin I ml
iidlinx( OIMIIIIIIII/V liiilidli\is ( Diucpls, Mel/ioils and C'dfilrxis Waslunginn. M ( Aspen
Instiiuli. 1995
Weiss ( || ' I low C. in Ilieory-Uased llvalualion Make Ciicalei llc.ulw.i) ' ' I vdliidlliiii
RI-VKH. 1947.21,501-524
Weiss. C II I \dliidliiin Mi (finds Jm !>liijyin£ 1'nigrdriis and l'iiln u \ (2nd id ) I ngle
wood C lills, N | Pu nine Mall. 1998
Whole).) S "I valuahiliiv Assessment Developing Piogiam lheory"lnl Bukni.iii
(ed ) IKinx I'mxidiii I In in \ in I v£ianijoi Public Sc< tin l:viiliuili«n in tin-
Faculty it) Applied i( iriirr, Rityal Melbourne Institute <»/ /rr/iiindi^v. Ai^tnilm
ANIMO.V) fi IKOSINO is icsri/ic/i fellow at the Cenlei Joi I valuation, linimini s
/oi C./ii/i/icn I'logitim. /\iiiciiC(iii A
-------
M
I III I'in IN I \AIII till >N
need 10 conduct iiadilional causal modeling analyses ol the pailcin ol influ-
ence Inini ihe mlei veiilinn in ilie various media I ing variables and llien Irom
these mediators to a disial outcome
Tew cvaluaiors will argue against the inoic licquenl and sophisticated
use ol substantive theoiy to detail mici veiling processes Piobably ilie sole
exceptions aic those who believe lhai I lie act of mcasunng piocess cieatcs
conditions diffeieni from those that would apply in the actual policy world
|-"ew evaluatois aiguc that n is not possible lo collect measuies of interven-
ing processes So it should be possible to construct and justify a iheory-based
lorm of evaluation that complements experiments and is in no way an alter-
native to them It would prompt experimenters to be more thoughtful about
how they conceptualize, ineasinv. and analyze miciveiling process It would
also lemind them ol the need lo lust probe whether an intervention leads to
change* in each ol the iliroicmally specified intervening processes and then
exploie wheihci these piocesscs could plausibly have caused changes in the
moic distal outcomes of policy inteiesi 1 want lo see theory-based methods
used within an cxpcimicnial li.imevvoik and not a.s an alleinative to it
References
AngiiM.) I) . linlm-ns. ("• W . and Kuliin. H I* "Ideniifiialioii of Causal I lUiis Using
liiMimm-iil.il Variables ' /inn mil »J l/ic Aiiirii«in MulislKiil /Usut union, 10.1)6, 'Jl.
4-H-402
Anson. A . :nul oilicis " NIC Comer School Development 1'iogi.iin A lhcorciu.il Analy-
sis "Joiinmlii/Cibun I.(/in ill in". I*)9I, 26, 5tt-82
Coiner.) I' .SVhiNil I'liwri New York Free Press. I"80
( ook. "I I) , and Campbell. !"» I QwiM-/.v/»niMiriiinlii>ii Pi MX» «". wiiiu-i 2000
Cilyiiioui. C . Vlu-mcs, K , Spines. I'. ami Kelly. K DIMUVCIIIIX («nn«il Slnidiiii Am/i-
iinl Inirllixi-iKf. I'liiloju/iliv o] .Sm-iicf. diul Snilisliidl Moitilmx Orlando, I l.i Au-
ilcmu 1'iess. 1987
Si men. M ••M.minizing I he Tower of Causal lnveslig;ilinii I IK- Modus OptT.nuli
Mriliod ' In Ci V lilass (cd ), t'viiliuKioii iliu/ic» Ktvifiv Aiiiimil 'IhoiisaiidOaks.
( .ilil Sjjir. 1476
(/ (Jli 1C is little i iillsriisil.s
-------
l'Klll,i;\M till OKI IN I \ II MMIHN
lahlc 4.1. 'I licory of a Job-Training Program
Piogiam publicizes a iob-iianinigprogi.ini
Youlh heai about llie piogram
Youili are nneresieil and nioliv.ilcil lo apply
Program eiiiolls eligible youlh
Youlh sign up
I'logiam pioviclts occiipalion.il naming in .in accessible location
'louili .iiiciul legulaily
"I Mining malt lies labor 111.11 kei IK eds
1 raining is earned oul well
Youlh leani skills
11.lining ii'.u lirs HIHH! woik h.ibn.s
Youlh internalize v.ilius ol legul.u emplii) mem and appiopn.ile beh.ivioi on (he
job
Piogiam 11 It is \ oulli 10 suilahli |i>lis
Youib appl\ loi jobs
Youlh beli.ive well in job mlcrvicws
I mployeis ollei jobs
Youlh .uirpl jobs
^oulli show up lor woik legiilail)
Piogiam assists \oiiih in m.ikmg liansiiion lo woik and In Ips \\ nh piolik ins
Youib acicpl anllioiily on llie job
Youlh do (hen woik well
Youlh beh.ne well wnli cowoikers
Youili slay on llie job
Suniii Atl.ipli'il liiini Wriss, IW8, |> 54
Till- is nn jncmpi U) sec how far (lie program succeeds in accomphsh-
iii}; all (lie intervening phases hciwcen enrollment in llie progiam and long-
tcrm job holding If uainccs do well all along (he rouie from participation
m (he training program to slaying on a job, there is at least plausible reason
10 believe ih.u the piogiam was icsponsihlc- (or the 11.uncos' woik success
(See Chapter I wo by Jane Davidson loi limber discussion ol establishing
causality)
13111 lei us lake a step baik Table 4 1 shows the expected steps in the
implementation ol the program It is what might be called the mi/>fciiienfl\ed \\ilh llie piogiam also expect them to help the youlh deal wnli
dillicult hie ciicumstaiKes, such as an abusive parent or involvement in gang
activities Some piogiam people also expect them to inleiiede loi tumbled
youth wuh social woikeis, police, or piobalion officers 01 to help ihc u-ens
sec me sei vices horn health clinics or other service agencies
Several ihcones of action might be operating Some people, maybe the
piogiam adminisliatois, think that the counselors arc role models loi the teens
Hecause ol common ethnic backgiounds and life circumstances, the teens tan
idcnlil) with them, will take then words of advice seriously, and will lollow a
moie positive suu.il path Anothei theory might be lhat the counselois uiuk'i
stand the perils and piessures lhat the teens face and will give advice that is
beiiei suited 10 the ie.il woild ol the inner city ihan would a mitlille-i lass
teaihei 01 counseloi they will know how to advise on laiiuly probli ins
because ol the commonality of their family backgrounds Anothei theoiy migln
be that the counsclois, uiiclcrsiaiidmg the local culture, can use ihreais .uul
penalties ellecdvely, something thai while middle-class counsclois would he
loath to do Yet anolhci theoiy is that the counselors will be well au|ii.iinird
with all the available seivices in the community and theieloie can iclei the
youlh loan appropriate souice ol help All of these assumptions glow Imm ilu
match ol counselors to the ethnic and socioeconomic status ol die leeiiagt is
A clillerent set ol assumptions would refer lo the specific steps .uul
actions that the counst-lois use in their relations wnli the inns, prih.ips
gioxMiig horn the p.ulicular training that (hey received in the comimimiv
college I hey may have icceived training in the use ol rcuauls loi sm.ill
steps lhai a youlh takes in a positive dneclion, such as oflt nug .1 movie
pass liu alli-iuling school live cl.iys in a low Oi ilu-y m.i) li.ivi IH-I n n.iiiuil
lo help with the developmeiil of peer suppoii gioups. win it- .1 gioup of
-------
18 I'KIU.IIXM IlllOin IN I \ \l IIAIION
youngsters help one another maintain good school attendance and piopei
completion ol school woik One might also imagine thai a counselor could
he ellective by lutonng young people in the subjects thai give them the
most double in school and help them oveicome cognitive deficits Theie
aie a plethoia ol iheoietic.il bases on which one might expect the piogi.un
to he successlul in cncouiaging young people lo lemain in school and do
good woik
II (he evalualor isembaikmgon a theory-based evaluation, which theoiy
does she1 hook the study to7 Hoes she follow ihe counselors encomagemcnt
ol school attendance7 His intervention into family disputes7 Mis relenals to
set vice agencies711 is establishment ol support gioups711 is coaching m math7
Oi what7 One study can raiely collect data on all possible activities and their
cascading consequences It would be buidensome lo follow each chain ol pos-
sible events, and the evaluation would become complex and pondeious
C hoiccs have to be made I he evaluate)) has lo decide which ol the seveial (he-
ones lo Hack thiougb the senes ol subsequent steps
Ovciall, theic aie two majoi sources ol theoiy—the social science hi-
eiatuie and the beliefs of program stakeholders "I he advantage ol social sci-
ence iheones is that they aie likely to be based on a body ol evidence that
has been systematically collected and analyzed I he mam disadvantage is
that available social science theoiy may not match the program under
leview, and even when u does, u may be at such a high degiee ol ahsii.u-
iion that n is difficult to opciationalize in the immediate context Ncvei-
theless, when social science piovides theoiy and concepts that giomul and
suppon local (cumulations, it can be ol gieai evaluative value (Chen and
Rossi, 1987) The evalualor should bring her knowledge of the social sci-
ence liteiaiuic lo beai on the evaluation at hand
A way 10 begin the task ol choosing a iheory to lollow is to ask (he pio-
giam designers, administrators, and practitioners how they believe ihc piogiam
will woik They may have clear-cm ideas about the chain ol actions and icac-
lions that they believe will lead to better school achievement of (he youth But
it is not unusual to find thai different people in (lie program hold difleienl
assumptions about the steps hy which inputs will translate into desned out-
comes What can the cvalualoi do7
Pnsi, she can convene a meeting of the stakeholders in (he piogiam,
peihaps including the youth who are the piogram's clients, and ask them to
discuss then assumptions about how the piogtam will reach the desned
icsulis They should discuss the mimstcps ol counselor action and youth
iespouse thai will lead lo success "I hrough such discussion, their onginally
hazy ideas may become deal, and they may icach consensus about what the
piogiam nuly aims to do and how u aims to do it
fiogiam stall will often hud a disc USSIOM of this type revealing and emi-
nently piaclir '"hey will leaf n what then colleagues assume should be
done (and wl y aie doing) Stall may all be peiloiming (he same Imu-
nons but dome iiicm will) different assummions about why lhe\ will be
Wnii n 1 INKS IN WIIK n'liiioKii s SHAI i Wi IV.MHAII' W
successlul Oi they may actually be doing difleienl things In discussion,
they can hnd out wheihei they arc working at uoss-pmposes 01 aie on the
same wavelength If they aic working in different directions, the progiam is
apt lo be liagmenled and ineffective Staff will often find the efloil to ic.u h
consensus.) stimulating and useful exercise It may help the piogiam .mam
cohciciicc and diiection
Including Several Theories
In some instances, some program staffs cannot reach consensus I hey
have m.ukedly dilleieni (he-ones about where they should put then nine
and what kind of actions they should take in ordei to engage piohlem
youth in school In sui h cases, it may be necessary to include seveial dil-
leunt iheones in (he evaluation design The evaluation can lollow the
chains ol assumption ol several theories lo see which of them is best sup-
ported by the data
When a numhci ol dillerenl assumptions are jostling loi piionty. a 1151:
is wise to include multiple theories If only one theory is tracked, and that ihc-
01 y is wiong 01 incomplete, the evalualor may miss impoilanl chains of
action I he final lesult may show that positive outcomes weie at hievecl but
not thiough the senes til steps posited by the iheory The cv.tluaioi will be
unable to explain how success was attained (sec Ring, Slccnhuis, Van Assrma.
and IV Vnes 1996, 1'uska, Nissmen, and Tuoniilehlo. IWi) Oi il the pio-
gram lias disappointing lesulls, and only one theory was irai ked, the cv.ilua-
toi may lace ieade)s who say, "But that's not how we thought good u suits
would come about anyway" When programs rest on fuzzy assumptions, it is
ohen useful loi TBL lo represent a range of theoretical expectations
Bui the more iheones that arc tracked, the more complex and expen-
sive the evaluation It is worthwhile lo try lo winnow down the numbei ol
possible iheones to a manageable number Three or four would seem to be
the maximum that an evalualor could explore in a single study I low can the
evalualor decide which of the several theories is worth including in the eval-
uation7
( Criteria for Select ing Theorifcs-^ ( /
Thc~fiisrcnTCMiorT iQhi-'lv-liils'iiiilip people associated with the piogi.nn.
pnmarily the dcsigneis and developers who planned the piogiam. the
admmisiiaiois who manage it, and the practitioners who i.iny n out on a
daily basis Also impoii.mt may he the beliefs of the sponsois whose money
kinds the piogiam and the t hems who receive the set vices ol the piogiam
Wl)4l di) lh"c" fl^imicigcitmi. are the pathways lo good outcomes7 SVhal
aie the mmisleps (hathavc lo he taken U (lie clients aie to ic -'lelienehis
that the piogiam piomises7 What (he people who aie deep! ed m the
piogiam believe is ciitual because iheir beliavioi largely u- .nnes how
-------
•\
I'KlH.K \M I III Hin IN I \ Al MM KIN
llu- piogiam inns When ilu-y hold chvcigcni assumptions about I he ionic
10 success, ihe seveial ihconcs dial ihey prollcr become candidates loi
inclusion
A second cnierion is pl.uisihiluy Can die program actually do die
I (lungs dim a iheoiy .issuincs, and will die clients he likely lo respond in (he
expected fashion7 'I he cv.iluator needs 10 see whal is really going on Onc_
'^ vv.iv is lo liillow (he money Wheie is die hudgel being:spent7 Where is the
piogr.im K-jlly pulling its chips7 Which resources are they pioviding lor
wlui kinds ol assisi.uue7 If die program makes available to each counseloi
a list ol accessible service agencies, llieir eligibility crilena, and hours ol
operation, (hen n is a leasonablc bc( ibai (hey think the referral mule is
impoitant II nobody gives the counselois any mfoi maiion .ibout available
icsouiccs, (hen this iheoiy is piobably not .in active candidate lot study II
piogiam designers and administrators talk a good deal about ethnic m.ikli
between counselor and client but end up lining pumarily white middle-
class counselois, ethnic match is not an opeialive theory in this piogiam
Similaily, il the counselois do not know enough about plane geometiy 01
nmciccmh-ccniiny Ameiican Insioiy 10 tutoi youth, then assumptions
about success ilnough luionngare noi apt to be the route lo follow (unless
the counsclois hnd olhei people to do the luloimg) I he evalualor needs 10
lake a haul look at the piogram in action, not just in its planning docu-
ments, in oidei to sec which theones aic at least plausible in this location
A third criiciion is lack of knowledge in die piugram field For exam-
ple, many piograms seem to assume thai providing information lo piogiam
panic ipanis will lead lo a change in (lien knowledge, and met eased knowl-
edge will lead to a positive change in behavioi This theory is (he basis loi
a wide lange of piograms, including those dial aim lo i educe the use ol
dings, prevent unwanted pregnancy, improve pain-ills' adheience to med-
ical icgimens, and so forth Program people assume dial il you lell panici-
pants about the evil effects of illegal drugs, (he chllicult long-term
consequences of unwed pregnancies, and the benefits of complying with
physician oideis, (hey will become more conscious of consequences, think
moie caiefully before embarking on dangeious couises ol action, and even-
tually behave m moie socially acceptable ways
I he theory seems commonsensical out social scientists—and many
piogiam people—know thai it is loo simplistic Much research and evalua-
tion has casi doubt on its umvcisal applicability Although some piogiams
dial convey knowledge in an elloit lo change behavioi have had good
lesults, many have been notoriously unsuccessful In an ellon lo add to die
stock ol knowledge m the progtam aiena, an evalualor may find it woiih-
wlnle lo puisne this theory in the context ol the particular program with
which she is woikmg bhc may want lo caielully Hack die conditions ol die
piogiam in oider to galhei moie information about when and where suih
i iheoiy is suppoiied 01 discontinued by die evidence (and whal elements
of i onicxi, inieinal oigamzaiion, and leinloicemeiii make a dillcmuc)
Wnu n I INKS IN WIIK n I'm OHM s SUM i wi |V,MUAII>
•II
So much elloil is expended in providing inloi maiion in an aiiempi
10 change behavior (through public service campaigns, material posted
to Web sues, distribution of printed materials, leciuies and speeches,
couises and discussion gioups, promotional messages disseminated
ilnough multiple media) that careful investigation of (Ins iheoiy is wai-
lanied Riithcrnioic so much uncertainly exists about the efficacy of pio-
vuling inloi maiion of different kinds (o different audiences dial piogiam
developeis need a belter sense of (he prospects I he evaluatoi who pui-
snes this iheoiy in a "I HIE may look lo social science iheoiy loi a sophis-
ticated undeisianding of when and where inlorniation is likely lo have
effects and under whal circumstances She can build tins knowledge into
(he evaluation When the results of (he evaluation are teady, she can ollei
piogiam developeis and stall a greater understanding ol the extent lo
wlnih inloi maiion i icales change wilhin the immediate piogram context
Mail) studies have shown thai information lan lead to change in knowl-
edge and ailiiudes but not often to change in behavioi The i uiiein eval-
uation can examine whether and where the sequence ol steps in the
iheoiy bieaks down and what forces undermine—01 leinloice—die
powei ol mloimaiion
A lin.il cnierion foi choosing which theories to examine m a iheoiy- f [.
based evaluation is die ccntrah(y of (he theory lo the pioKiam Some ilieo- V
lies aie so essential lo the operation of a program that no mallei what else
happens, the piogiams success lunges on the viability ol this paiiiculai the-
01 y I el us take die example of a comprehensive community piogiam I he
piogiam involves the provision ol funds (by government 01 a foundation) •'_
lo a gioup ol comnumily residents, who (hen decide wluc h enhancements I)
(he neighborhood needs in order lo improve die lot of us inhabit.mis I be^A-
lesidems can choose lo use (he funds (o add more seivues (mental health,
education, and so on), clean up the streets and parks, rehabilitate buildings,
hue puvate police, alliacl new business lo die neighhoihood m oidei lo cie-
ate jobs lor local people, begin a ear service lor elderly lesidenis, 01 what-
evei oihei sei vices they decide arc most likely lo impiove die loi.il quality
ol hie
An evaluation can study die services chosen and Imd om ilu conse-
quences ol adding police 01 rehabilitating buildings 01 whatevei olhei new
scmccs have been added Ikil a fundaniental premise ol this commiiiiiiy-
bascd approach is dial local lesiclents arc knowledgeable, lommitted, haul
woikmg, and altiuistic enough lo find out whal is most needed .mil to go
about gelling those sei vices into ihc community rurthei, tin y .in .issnmt d
lo lepiesent (he needs and wants of a wide swath ol the (ommumiy So .in
undeilying iheoiy has to do with die role ol citizen gioups in developing
and directing a compiehensive community initiative I he clleiiivt in ss ol a
gioup of lesidenis in lepresenting the interests ol dieir neigbboiliood .uul
seeming piionty sei vices is key to (he success of (he piogiam I bis .issump
lion becomes a pi line candidate lor die evaluation
-------
-12 I'KIM.KAM Iliroin IN I \AI.UAIK1N
Which Links in a Theory to Study
Many ihcoiu-;., il diawn oul in detail, consist ol a long series ol inteilinked
assumptions about how a piogram will achieve us effects Let us go back to the
job-ii.iimiig progiam in Table 4 1 J.I the evaluation does not have the resouic.cs
01 the lime to study all ol the siqis laid out in die iheory. which ol than should
The evaluation explore* Much ol ihe answer to this quesiion will depcnd~on~nie
7iraciicaTines of ihc'STtuaiion At whal jioml is ihe evaluaioi bi ought to the
scene' Is il alter ihe fusi several siejis have already been taken7 How much
money (Iocs the evaluation have to collect daia? Mow difficult is it to get some
kinds of data7 Tor example, what kind ol data will the cvalualor need in oicler
lo know whelhei die naming is earned oul well71 low will she liiul out whelhei
the iiameesado|ii and internalize the values ol icgulai employment •' II some
kinds ol daia aie dilhcull or expensive lo collect, that will sei practical limns
Second, pirgiam Mall m iy hqyc pailiculai coiiceins about soiiyc seg-
mcnisol ihe impleineniaiion iheoiy They may wain to know, foi example,
whelhei naineis aie giving jiinpci emphasis lo good work hahiis and other
"soli skills" 01 whelhei the youth in fact learn the oicujiaiion.il skills ihat
the nameis seek to convey They may want to know whether stall iclei them
to iclevant jobs and whelhei the youth coinpoil themselves apjiiopiiaicly
m job iniei views, so that it is deal why they do or do not get jobs
It may be even moie impoiianl to examine some links in the piogiam ihe-
ory about the psychosoc i.il piocesses thai underlie the piogram I leie is wheic
much ol the uin.eil.nniy in social programming lies Whal impels developing
counines to seek 10 aiuaci moie girls inio ihe school system7 Whal gels lac-
ully members in urban universities lo leach in mierclisciplmaiy courses in
oidci to iciam sludenls in school7 In our example, what are ihe icasons thai
tiameesjieisisi m the liammgcouiscand leain both job skills and woik icadi-
ness skills7 Is u the capacity of the nainers lo develop supportive communi-
ties among the youlh7 Is il the strength ol external rewinds and |iumslimenis7
An evaluation can concciinaic on understanding these kinds ol mech-
anisms and the extent to which they ojicratc within the piogram milieu The
evaluaioi can collect data on whelhei pcei groups develop dm ing the couisc
ol naming and the messages and supports thai these groups piovidc to their
members Do youlh affiliate in subgroujw7 Do members ol the various
gioups suppoii the aims ol the training piogram? (Or do they deingiaie the
ellort to Icain skills ihat will yield "chump change"?) Do the nainers
actively cncouiagc the foimanon of subgioups and provide leadership7
What messages emulate in the dillerenl subgroups about the \ aluc of work
and the willingness lo accept aiilhonty on the job7 Regarding the theory
about external threats, how important to pamcipanls in the naimng pro-
giam is the icduction in safely net suppoits7
Jjecausc e -lions lo date have lold theirjrcadcrs icjatively little about
ihe wliy ol pi success and lailuie. such iiujinrics may have gie.il ics-
oiiancc Studies mat exploic the jisye hosoc lal piocesses of jiingiam iheory
WlIK II I INkSIN WlllC.ll IlirOHIIxSllAII Wl IVAIUAII* II
will have much to tell piogiam designers, lessons (howevei tentative) ilia)
may be suggesiive lor a whole range of programs
Criteria for Selecting the Links to Study
The cruel 1.1 lor choosing which links to study arc smnlai to ihe ciiiena loi
choosing which theories to study Two are probably most important I he Insi
ci iierion is the link or links that are most critical lo the success ol the |im-
giam It seems wise to mvcsi resources m studying the pamculai assumpiion
on which the progiam most basically resis If the program is predicated on
the assumption that what keeps youth enrolled m the full naming piogiam
is the suppoit ol then peers, then that assumption warrants investigation
I he si-iond c ineiion is ilie degree of unceilainiv about the linkage II
nobody knows whelhei I lie assumpiion is likely lo be suppoilccl cmpnually,
01 il pi 101 studies have pioduccd conflicting lindmgs on the subject, thai
link max be woiihy ol systematic sludy Some linkages aie unsettled in ihe
social si lence and the evaluation literatures Some linkages seem lo be siiji-
poitcd in ihe social science literature (or m common sense), bin evaluations
ol i-ailiei piogiams show that they do not work in piaciicc An cxanijilc
would be the piemise of case management within a iiiulliscmcc progiam
A laige number of multiset vice programs have employed case maiiageis
who analyze the seivices thai a lamily needs, locate and looidinatc a i.ingc
ol SCIMCCS, and help the family members obtain appiopnair sei vices horn
rck'vant agencies I he idea ol a family coordinator, an ndvoi.ne and 1011-
suliani lo the lamily, sounds so utterly sensible that it is unsettling lo Inul
thai evaluations have usually not lound such piograms successful (loi
cxamjile. Iticknun and otheis, 1995, Si Pierre, Layzci, and Goodson. IW)
What aie the assunijiiions thai underlie case managemeni7 What is the i.ise
manager assumed lo do, with what immediate consequences, leading to
what ne.xt steps, with what laler consequences7 Including some ol these
kinds ol links in the evaluation would yield impoiianl mloimaiion
Conclusion
In selecting the theory 01 theories to use as scaffolding foi a I'll! , the eval-
uaioi should consider these ciiiena
• The assumptions ol the people associated with the piogiam What aie
then consnuclions ol the interlinked stejis by which piogiam IHJUHS aie
liansmuied into piogiam outcomes7
• I he plausibility ol the assumptions, given the mannei in whu h tin pio-
giam is allocating us lime and resources
• Uncertainty about the applicability of current assuiiijiiioiis tiiven the
olien m
-------
44 I'KIN.KAM I III DM IN I.VAI IIA IIUN
• 1 he ccnliahty oflhc assumptions u ihc piogiam If llic program is based
directly on a particular ihcory, it would be sensible lo make this theory
llie cemerpiece of ihc TBE
Once ihc cv.ilu.uor decides which theory 01 theories to use for structur-
ing the evaluation, she ought to spell out all the links in the theory iham—
what the progiam will do, how paiticipants will respond, whai the program
docs next, and so on Many evaluations will not realistically be able to follow
all (he links in each chain, and (he evaluaior needs lo choose ihc links on
which to locus Considerations for making thai choice include the practical-
ities of access, resomces, and methodological capability for studying given
links and ihc paniculai knowledge needs of piogram siafl, who wani lo know
which elements of the piogiam they need lo modify or shore up
In making both choices—which theories lo select and wlmh link* 10
study—the evaluauu needs 10 consider ihc underlying iiifi IKIIIIMM.N on
which the piogram icsis, wh.il I have called I lie fmignim (fnoiv m con-
nadistmi lion lo ihc im/>/riiirm I siiuiliins, I . V.m Assuiu. I* ,.IIK| De Vncs, II " I lie linp.iil of .1 ( oni|iiiu i l.tiloud
Nniiiiion liiu rvi-niion " 1'n irndvr Medicine. 1996, 25. 23O-242
(Inn II I ..nul Rossi, I' II "The I hcory-Driven Approach lo Validity " (•viifiuiiinii >in,l
I'liiXHiiii rfiiiiiiiiix. l°"7. ID. 95-10)
I'usk.i. I' NISSIIMII.A .ind I uomililimj "Ihe C'liniiniinily-llasnl Sli.iu-Ky l» l'i< vinl
( oioii.ii) IK .in DIMMM- I oiiilusions lor the Ten Years ol (lie Noilli K.iu-li.i l'io|(iis "
AiiiiiiiilKiinii ofl'iiMu (fnillfi. I'W>. fi. 147-19)
si I'UIH K d l.i)zii | I . (lOiiilsnii. li 15 , and llcriisicin. I S Niiliniuil Inifuui / in/
inilidii .i/(In (
K.Mfu. 1997, 21. 501-524
Weiss, C II I iiiliiiiiiiiii Methodsfoi SluilyingPiogiamsimJ Mmc-. (2il nl ) I ng
Chlfs N| I'rcniui-llall, 1998
(IlKSC (K.IN VV'I ISS is /JI()/CSS(ll (I/ CcJlKdflOII (l( (IlC MillVdlif ( .Mlifllilll
-------
I t UlJlKtflOIIN Kill be /MM'tl (III -
tivf (how the progiain actually works) The issues raised in this i haptei i.m
he applied lo eiihei normal we models or clescnpiive models
Null 'I'his ih.ipltT li.ts hciu'liiccl sMl)si.iiiti.illy from the liclplvil loiumrnts, i|\ifsiiiiii-..
.Hid SU^CSIIDIIS limn iiii-mlicis ul i In- ll.nv.iicl l.v.ikulioii 'I ask I out. p.inu ip.inls .11
ilu- ll)l)H Aiiii'iu.in I v.ilii.iinin ASMU i.iiinn nu-ciui);, the cdiinis anil si'iits uliim ol tins
r, .mil as ,il».iys my gi.nlii.iic r\.tliuii<>n bliiilfius linlial wink mi I Ins ili.i|i' .is
11) .1 It Ilim'ship limn (In S|n iucr I tniiul.Hioii
-------
48 I'KlK.KAM I III 010 IN I \ MIIAIION
What Do the Boxes and Arrows Represent?
Piogiam theory usually involves a diagram of boxes linked hyanows icp-
rcscnting causc-and-cffect rclaiionships It is perhaps templing 10 consider
these causal models lo lie like wn ing diagrams, in which, il we Hick .1 swiuh
at (he lust box in I he diagram, il will cause the lights in the other boxes to
illuminate And indeed, sometimes the descriptions of these models, using
a series ol if-ilicn statements, suggest this imagery (Owen, with Rogeis,
1999, IManiz, Grcenway, and Hendncks, 1997)
l:valualois who aie familiar wiih social science principles will not he
sin pi used that lew piogiam theoiy models aie based on simple i an sal ic-la-
tionships like this, even if diagrams do not explicitly show it llowevei,
some piogram theory models do explicitly attempt lo show the pioccsses
that aie "necessary and sufficient" to produce the desired results — lor exam-
ple, Cooley's causal model (1997) of a progiam designed lo mcicasc gills'
participation in high school in developing countries
Moje commonly, program theory models are based on a recognition
that other faciois may influence the achievement of intermediate and ulti-
mate outcomes For example, ihe United Way's generic causal model dins
not explicitly include other (actors, apart horn a list of consnaints on the
program Howeyej, in the instructions provided wyhjhis model, it is made
clear that the further away from aciual program outputs'one moves, the
weaker the program^. inCLuence becomes, and The likelihood ol outside
forces having an influence increases (Planiz, Grcenway, and HendncUs,
1997) They go on to give an example ol a program pioviding prenatal
counseling for pregnant teens, pointing out that the program can influence
what pregnant teens know about appropriate prenatal practices but cannot
influence what the teens' overall health was when they became pregnant
Nor can the programs affect whether teens were using drugs when they
became pregnant The authors recognize thai each of these issues, genei.il
health and involvement with drugs, can have as much long-lei m influence-
on the later health of babies as the program uself
II is inleresimg lo note that tins analysis focuses only on lixed chaiac-
icrisncs or events thai happen before ihe clieni begins in ihe program— ^soine-
nmes jelened to a> mudeiaiois Outcomes can also be influenced by factors
that occui ai ihe same time as the program and either help or hinder its work
I low can we represem ihese olher factors in our causal models7 Funnell's
piogram theory matrix (Chapter N income I udesolher factors explicitly .mjcxt
associ.aicd with each outcome 1 1 is also possible to sjiowjhcm on the program
tJlSUiy diagianuas I lalpernTf998, 1999) has done, as seen in j-igmc 5 1
We might even make a dramatic move compleicly__away horn our
PI'jiJi-nii-uMimc causal model and show ihe web of client lelationships
thai influence client outcomes, including the influence of family, friends,
schools, shops, economy, neighboihood, media, legal system, woik, econ-
i>. i .nicl pohiii.il system (Mullen. 1995)
C'AIISAI MOI)i:i.S IN I'KlX.KAM I III DHV I VAIII \ I II IN
•19
Figure 5.1. Representing Other Factors in a Logic Model Tor Reducing
Alcohol-Related Motor Vehicle Accident Injuries and Deaths
Si I \ K i
( Illkiillll
lli.ihli
I'liiiiiuiion
|S) I'lOl III lull
Dull isi Dunking
iV Dm nit; in
lifiui.il l'ii|iiil.iliiin
Addnnon
Irealmcnl
Semies
!
I'meigiiKy
. .. , Anilinl.ii
Mediul
Sl 1 Ml 1
Seivn.es
\/
1 limniiile Dunking
ta Driving in
Akoliol-UtjHiidciH
Chcnls
Miniinui Mmliidii)
iV Mi in. ilny
limn MVAs
Kignm.il
llfjhh
I Inu unii
KciliiLC AlLohol-
Kelaled MVA
fjialines
6j Injuries
I'niMiiiial
I It.ill li
lnl.ll
1'irin.iiurr
De.iih .mil
I'ifvi m.ihlc
(ii.iilu.iliil
I in using Itii
Noviir Dim is
Allnn.iiiv
SlKl.llA lllllll.ll
liilri.iim- lui
Dunking f-i
Dining
IWH,
-------
50 I'KOl.KAM IIIIOMV IN I V.\l H4I ION
Mtilliplc Strands in Causal Models
Many piogiam ihcoiy models potiray I he piogiam as a single chain ol uilci-
mcdiaic and ultimate outcomes, wlicic A leads to R and then n> C liui u
,nj£ |w- hdpluljo show nuihinlc strands, where A and C.boih lead in U—
culie'i "liTcoinbinaUoti or iis allcinatives Ideally, we would be able in distin-
guish between complemc-maiy causal paths and alternative lausal paihs in
a di.ini.nn. pcihaps by using line arrows for the complciiicni.ity paths and
block .mows lui die alicinative paihs
II a combination ol two causal pain-, is nc.ccis.iry in achieve llie intended
tesulis. u is mipni lain in make llus explicit m order to avoid maximizing only
one ol mem In many piogi.im!., stall must balance competing imperatives like
this When I winked with iu.iirin.il and child heallh muses lo develop a ians.il
model ol then piogi.im In guide the development til pciloini.iiKf mdii.ilois,
ihey weic paiinitially pleased iliat they could make visilile the balancing they
needed in inamiain heiwceii providing mformaiion lo paienisand sitppoiimg
P.HCIIIS tonhdcncc in llu-ir own anilines Part ol then program mode), which
used an adaptation ol Uciuicus Incraichy (Brnnell and Rockwell, 1999) in
ilesi nhc ilu-ii woik on infant leveling, .showed ilusclcaiU. as seen in 1'iguic 5 2
h was impoitant Ini the stall in nuUe visible to piogt.tm m.iii.igns the
compelmg demands on them and lo make sure ihai perfoimance mcasuies
lelenecl 10 both of these in otdei u> ensure llwi iheie weie not siiuiiui.il
l-igurc 5.2. A Partial Program Model Showing Competing Demand-,
(II I IMA 11
Ik ill I nun ilion mil
j;in\vlll III Kil'h-
III IIAVIOK
( IIANl.l
KNO\VII IX.I
III IIII s
I'DI s
C/U'SAI MODIISIN PRO(.KAM I I II I >K> I X Al I IA I U IN "j |
pu-ssu.es 10 IIMMIIIIZC cithei mlomiaiion giving 01 paiemal suppoii .» (he-
expense ol ihe otliei When programs are managed by managcis w.ilu.ui
cleiailetl knowledge ol jirogram processes or arc managed ilnougb lonnat-
iu.il aiungcmenis, it becomes more important to make cxpliut COIIIIMMIIIU
nnpeiatives sui h as these If performance measuics only iiuliklr one ol the
competing nnpo.ai.ves, then a program may seem to be perlormmg well ,,.
let MIS ol us iiucinuxluic outcomes because one of these 1S being nuxtiniznl
at the expense ol the other
Multiple sii.uuls m a causal model may instead represent ahein.uive
I.IHS.I! luilu, |-OI t-x.,,Mple, Weiss (1998) omlmcs four possible mc.lumsms
hy winch higher UMdici pay may be linked in increased siudem .« \»ew.
Minn II these aie seen as competing explanations for observed outcomes
then .t piogi.im ilK-oiy evaluation might focus on testing which of these besi
e\pl.ims the evident e (as Weiss discusses in ( haptcr Tour)
It is also possible lo see these ajlf rii.il ivi- ( .luJtajjJaih^^Jinm inu fot
i rrlam rnnrtiliaus. Drawing an .m.-ilogy
..- . .. ti-
|w»wdci, whuh will ,,nly f,,c In favorable conditions. I'awson and 'hlley
(1997) have suggested that program causal mechanisms only fnc wiihin
la\niablr tnniexis _Aney^ju.iiiori based on this lype of c.ins.d moJ.-l will iry
jojuidasLm^ljc^tixiiiinsiaticcs under which pa. in ul.ii nifVliams'iiis opcr-"
Jtc In then ic-analysis of n crime preveiHioifjimgiaTn iii public" housing
esiates. I'awson and Hlley (1997) demnnsirjird th,- tmp<,,i.m,«- ,,l ,,,.,lt-i-
SMiKlinj. llu- CO.IIC-M ..I ddfeiem sites, including mtc.au.oiis .miong v.nn.us
ma IUIIIMIIS (such as improved housing and increased lenani involvement
in esiate manage mem) and among oihei coexisting processes
ll'i_LliJ!iUi!u<> rc-Gresetit ihese morc^gmnplex iclai.o.isliiiis m ., ixvo-
dmwnsioii.il di^rajii Pawson and Tillcy (1997) insfraVl repies'em ihVn
causal model in a matrix of context-mcchaiiism-niiicoiiir conligui,uion.s
which describes m text the causal mechanism that produces the outcoiiu-
and the context in which the mechanism is opci alive
The characteristics of program clients— then motivations, .iiiimdes
previous knowledge, and skills— are an important p.iruihhe umtext within
which causal mechanisms work or fail to work An iterative senes ol data
collection and analysis activities can be used in identify impoiiam ways m
which clients vaiy and the implications of these foi program ellei tivrnrss
(McDonald and Uogers. 1999)
Tojully undeisiandjhe cunjcxnvijjiin which c.ntsal nircli.iiiisins «.pt-i-
aic.vve may need to develop program models that do moic than UK hide m.>-
giam ilicnis simply-.is passrvcrecTpienis of ircainienis that change thai lives
II llie IIC.IIII1CIH Involves swallowing a pill, we mighi expnl teiMn. |,hysi(;-
logical ellects, reg.udless ol the active involve mem ol the patient, hut even
m tins example, we know ih;il the patient's expectations about the iiealmeni
tan inlluence its irponed impacts It is even less realistic n nbe piogram
t hems as passive ,c< ip,ents when the program is nuk-.iv. hi mK alHiut
|k-iiii.iiifiii ch.mge ,„. lot example, students' school hrh., ,i .ommmii-
-------
'32 I'KlM.KAM llllCHU IN I V.MIIAIIHN
canon strategics ol llic healing nnp.iiial — changes lhal require progiam
i Ill-ins lo lc.ii ii, apply, and maintain new ways ol oni-i.il ing
Pawson and Tillcy (1997) have argued dial we need "lo shake off those
concrpiu.il hahns which allow us lo speak of a piogram 'producing oul-
comes1 and lo icplace ihem with an imagery which sees the piogram oflcr-
mg chances which may (or may not) be triggered into action via the
suhjccfs capacny to make choices Potential sublets will consider a pio-
gram (01 not), volunteer loi it (01 not), cooperate closely (01 not), slay the
couise (or not), learn lessons (or not), retain the lessons (01 not), apply the
lessons (01 not)" (p 38)
I Ins issue docs nol need to l>c assiu idled with a philosophical com-
mitmeiil lo saving progiam chenis and having their needs and perspectives
at the lorelroni of progiam planning and evaluation nor with a belief that
the prison.il dignity ol clients and stafl requires treating them as program
panneis railiri than as passive objects In lad, the same distinction holds
line loi piogiams such as buiglary pievenlion, which mlcnd It) change the
behavior ol potential burglars Programs tan be understood as changing the
options available to participants and their capacities to choose and enact
these choices Usually piograms seek to increase options and capacities,
some, such as burglary pieveniion, seek to icduce them
Causal Models from Systems Theory
Systems theory suggests other types ol causal models In ihis section, I dis-
cuss liner of these that appear lo he potentially useful for program evalua-
,K)n — viiiiious 01 vicious circles, symptomatic solutions, and feedback delays
Virtuous or Vicious Circles, bysjnnsjhmkingsuggesis that cause and
elU;u.mj.ghj ojiejihc aimiei.led.nol in a linear way bui in a circular way,
es of VJL/UHUS cncfrs (where an initial effect leads to us own
irrnloneiiirni and magnification) 01 VUIOHS rogiaiu.aieJjkely^ to dec ay
over lime or lo become stronger
Symptomatic Solutions. Symptomatic soln/ions ait- soluiions ihai
relieve the symptoms bin lhal actually make 11 harder to solve the pioblc-m
Ii would be like having ihe flu and taking tablets lo recline the sympioms
and then continuing to work excessively, rathci than convalescing, iheicby
making it haidci to actually recover
I Ins pmhlc-m has implications both lor evaluation and loi iiiomioimg
I 01 an evaluation, wheie we are trying lo understand how ellective a pai-
liculai piogiam h.ts been in solving a problem, we should design om eval-
uation so thai it can distinguish between lemporaiy i eduction of sympioms'
and sustainable solving ol the problem l:or monitoring, wheie we aie seek-
ing to simultaneously understand and influence progiam implementation,
we should set up systems lhal do nol encourage people to develop dys-
functional symptomatic solutions
Owen and Limbeit (1995) addressed this issue in then evaluation ol a pio-
gram in which all grade-live students used their individual notebook comput-
ers in all subjects One ol the unintended consequences ol this piogiam was an
incicasc in teacher stress, as teachers struggled lo develop then compuiei skills
and simultaneously acl.ipl their leaching material and piocesses him,illy, icai li-
ers lesponded to this increased stress by "gelling on with the |ob," avoiding
spending time in coordination meetings, or liaising with olhei leac hers, and the
administration sought lo support teachers by leaving them alone and nol mak-
ing addilional calls on their lime If the evaluation had measured icachri si i ess
at this point only, it would have found lhal in the shoil-teim teacliei stiess was
reduced thiough this coping mechanism Hut over time, this symptomatic solu-
tion led to hoaiding ol equipment, rivalry among groups, and pool attendance
at mloimation sessions—consequences thai made it harder to implement the
fundamental solution, which involved belter shanng and coordination ol
resoiuces and increased support and training for leacheis
Feedback Delays. We have probably all cxpcnenced ihe effects nl /tul-
badi-tkluy when Hying to adjust the water temperaluie in a shown If theie is
a delay in response, we lend to overcorreci—first loo hot, then too cold, until
eventually teaching (he desired cquilibnum 1 he Massachusetts liismin< nl
Irchnology "beer game' simulation (Senge, 1990) hasdemonsti.iied ihe ellt t is
ol feedback delay on a simple piogram—a system lor piodiicing ami disinh-
ulinga single biand ol beci Once there is a delay buill mio the sysiein, sn ili.u
the decision m.ikeisdo nol immediately see the nnpac I of the ih.mges ihey aie
making, the ordeis become moie and more excessive and unbalaiu cd
r]ie.tcasyHLliir_usiug rjcrjormancc mcasuies and mduaiois as pan ol
.oimmiiucy»ilu.i.lionjii_ihr public sector is lhal they can be used by iii.inagi-is
lo I.ike i DI iei live action in programs, muclilike some(ine innniliiiing and
-------
I'NIM.KAM
IN I VAItl.\lli>N
admstmg the shower icmpciaiure Unloitu.iately, lew if any ol these sysiems
addies. the pioblc.n ol feedback delay In lact, I have been unable 10 find a
single example iliai does
How Complex Should Program Theory Models Be?
Although ihis chapter has lot used mi more complex model* and i .ius.il ii-la-
uonshms. u is wonh remembering that si inplc models can olten be helplul,
pan.cuWly.iULUioiitamsjn.whjcl. .here have previoUsFylxrii lew explicit
™ic7niual ,uul emm. ical comieci.ons made between piogram activities and
ouicoiiS-s nmluTng ..'plausible model of bow ibe program is meant 10 wo.k
helps managm uleniily the mosi important moccsscs 01 mte.med.aie uui-
comes and lotus then mcasureincnl and attention there C.ivcn lhai many
niogiain evaluations still collect little data about progiain implementation
or miermediaic outcomes, there is often considerable value (as Lipscy. IWJ.
and Pctrosino Chapter Six, point out) in using even a simple iwo-siep pio-
Kraiu model that simply identifies and measures one mediating variable that
is undeisiiHid to be nccessaiy loi ihe achievement ol the ultimate outcomes
And having a common model ol how the piogiam is meant in woik can
help piogram staff work together and IOLUS on those activities that arc most
impoiiant loi piogiam success
In fact as Wcick (IW5) aigucs, ajiioueljnighi provide a useful heuriS;
nc foimirppseUiLaclioti without ncies'sanly being correct lie u-couiiis, ihe
sforfof ihe reconnaissance unii, lost in the snow in ihe Swiss AI|)S loi IhiCK
days in a blizzard, who eventually managed to find ibcir way safely back to
camp with the help of a map-a map. they laic, discovered. i>T tliePv.ien.ffis,
not ol the Alo_s.-n his incident raises the intriguing possibility thai when
•TOU" aiTuIsT any old map will do Once people begin to act, they
Kcneralc tangible outcomes in some context and this helps them dis-
cowt what is occui ring, what needs to be explained, and what
should be done next" (pp 54-55) Wcitk goes on to quote Suuhlle, I l.w-
/"ing an accurate environmental map may be less important than having some
( man that hi mg> order to the world and prompts action" (pp 56-57)
' V Tins analysis may well explain the positive lesponscs that program stall
/ohm have to progiam theory evaluation (see. for example, Huebner, Uiap-
/ ter fight) even when th.s is based on ve.y simple causal models Bui simple
causal models can be dysfunctional in high-stakes evaluations that are linked
tc, n h nn
()tiKniiirs'~^iiliivi sn/ rli\ui /Avwu
in.ipl nun .ui/A I linn)
tin-ill I I HIKHV OMIIII I viilii.iln.il I IIOIIS.IIH! Oaks. Calif S.igr, I WO
tonliv.l I'u M in. u inn in I In Waslimj>lon I. valuators' Confciemc. IW7
I iiinii-ll. S Piu^iain Lnj;ii An Ail.ijilaUlc "lool " Evaluation Nrws uiiiJ ( niiiiiiriii M>»7
(.( I ). •>- 1 7
ll,il|iiin. Kc.ilil)- Kvalualmg Innoviilivc t'l'igi.iins ill I'tihlu liisniu
nuns ' (iiiini'iidiHi Jiiiiimif (fir I'uhlic Seiloi Innovation Jnuuttil. Nov I WH. rovisul N.n
I*)*) |.i\.nl,ilili .u liiip//\v\vw iiinov.iiionic./rev_nilsi-x M I lii-oi\ as Mrlluul Small Iliconcsof Ircatmrnis ' In I SnliuM .mil A
Si on ( i-iU ), I'liili-iwcmifiiig ( tiu\r\ and Ceneiahzmx About I In in New Duniinns loi
riiiKMin I i-aliMiicui. mi 57 San I rrmcisco Jussey-Kass. IW»
I ipso M anil I'nllanl, | "Dnving I uwnnl "I heory in Progi.im I v:iliialinii Mini Mod
rls in C luuisi- I rom ' I Mihutlum tmtl Pn>)>mm Pluiimn)>, I981J. 12. \\7~MH
Mi( Iniiuik ( ' Ailminisiialois. is Applied Theorists " In I. llulun.ui (nl }.Ni\vHim
(KIMS /ni f uiliiiiiiiMi /Uvuiii rt in CIUJJIHIM llicoiy New D.rct lions l.u Piii^iaiii I valn-
.ilion no -i? San 1-r.mi.isi.o )osscy-Bass, 1990
MiDon.ilil. IS . and Knurrs, P "Markcl Segmentation as an An.il.ij>> foi liillt unu.iu.l
1'io^i.nii I In Hi) An 1 xamplc rroin I lie Oairy Industry " I'api-i prrsriilril a. iln .innii.il
ini'i'iiMn of ilu-AiiH-i 11.111 1 t'.ilu.-ition Association, Orlamlo, I la , I «W9
Owen. | M . with Rogers, I' | I'ni^nim h valuation f umii Hint A/i/noiii lies I linns.ind
Oaks. ( ahl Sjftc, IW9
llwin. | M .and l.imhtil, I C "Rules Tor Lvaluaitnn in frainin^ Digniiiz.ilions " f vul
iiiKiuii. IW5 1(2). 217-250
I'awsuii, K , and hlli-y, N KnilisiK I valuation "thousand Oaks. C.ahl S:ij;<-, l*)°7
riantz. M C , lliernwa), M I . and Mcndricks, M "Oulcome Measurcinciil S|HUMIIJ;
Kcsitlis in llu- Nonpiolit Sec lor " In K I- Newcomer (ed ), Using IViJuiiiiiiun Mm
MI 1 1 nil nl lit hni>it>\r I'lil'lu unii Niiiipnifit PiO£H>»is New Direilions for I \.ilu.iiioii. no
75 San I tain ISLII (ossry-lt.iss, 1997
Suigi- I' M lit, (ijili Disc i()linc Tin- Ail tiiuf I'ruilKrit/lhrf r
Weiss.. I II f iiiliiiKinii MilluxJs/oi 3(ii((ying/'i()gi(miMiii<(riiMiio (2n
-------
The Pilot Process Step by Step Outline
1 Step One- Create the Team and Team'
2. Step Two- Begin with the END IN MIND and PLAN YOfR WORK
3. Products
4. Product Contents
5. Tasks
6 For Each Task
7 Step Three: Plan and Conduct Initial Meeting with Agency Top Management
8. Step Four- Gather Documents
9. Step Five: Analyze Documents; Synthesize Information
10. Step Six: Develop Map of all Previous Work
11. Step Seven: Develop Draft Logic Models
12. Step Eight: Share Models with Agency
13. Step Nine: Change Models; Write Narrative
14. Step Ten: Assess Logic Models
15. Step Eleven: Identify Key Evaluation Questions
16. Step Twelve- Create Potential List of Jobs
17. Step Thirteen: Prioritize the List and Recommend Future Work
18. Step Fourteen: Assemble Final Products
19 Develop Pieces and Parts Along the Way
-------
The Pilot Process
Step by Step
Step Two: Begin with the END IN
MINP and PLAN YOUR WORK
• Identify the Products
• List the Tasks
• Create the Time Line
• Assign Responsibilities
• Identify Needed Resources
• Specify a Regular Process for
Communicating (phone, e-mail,
meetings)
Step One: Create the Team
Philosophy of Working in a Team
Mechanics of a Team
l/nique Aspects of the Virtual Te
Communication
Roles and Responsibilities
PRODUCTS
• Written Report
• Supporting Documentati
.-
• Oral Briefing & Materials
• AND, CONTAINED IN THESE
PRODUCTS...
-------
PRODUCT CONTENTS
• Logic Models
• Corresponding Narratives
• Observations
• Key Evaluation Questions
• Prioritization of Potential Jobs
• 'Lessons learned' about the Process
As 3 part of developing the logic
models, begin to 'overlay', on the
models, the following:
• GPRA annual performance measures
• Any metrics the agency uses
to measure any part of the
program (inputs — outcomes)
TASKS
• Plan & conduct initial meeting with
Agency Top Management
• Gather and analyze documents
• Synthesize information; develop tables
• Develop draft 'Logic Models' Ctheory of
program)
• Share draft models with Agency
TASKS CONTINUED...
• Change Logic Models per additional
information from Agency and other
sources
• Write corresponding narrative for models
• Assess the Logic Models
• Identify Evaluation Questions
• Create and prioritize list of Potential Jobs
• Complete Report, Briefing, and 'Lessons
Learned'
-------
For Each Task...
• determine if ft will be an effort that
involves individual team members, the
entire team, the Agency, the other pilo
team, the consultant, others!*
• identify (gather/request) resources
• seta deadline
• assign to a team member for follow-up
• keep notes, documents, references, a work
log
• •
Step Four-. Gather Documents
GPRA documents & other
information
Info on metrics gathered)
Published reports - internal
Published reports - external
Popular reports; articles...
Past evaluations'
Step Three: Plan and Conduct Initial
Meeting with Agency Top
Management
'Past Evaluations, Audits,
Investigations
• When synthesizing the information
collected, these documents will be very
valuable as 'pieces of the puzzle'
which may indicate
that there little need
for the same info againi
• Or can indicate a different
of info is needed
-------
Step Five: Analyze Pocuments;
Synthesize Information
• Read
• Sort
• Begin to develop
tables for each & every
'component'of
the 'program'
Step Seven: Develop Draft Logic
Models
• From tables, create models which indicate
sequencing, relationships, dependency
• fse 'Z' maps to indicate layered and
program to program relationships
• Models include: overall program,
clusters, individual programs •
Step Six: At the same time, begin to
develops Urge 'map'...
• ...of the complete prograr
• Include previous
audits,
investigations,
evaluations internjii^pjf'external to the
program
Step Eight: Share Models
with Agency
• Group meetings
• Individual meetings
• Ask: "Is this representative"; "What are we
missings'" "Where can we find that
information!*" "What other documents or
persons would you suggest we consult/"
• Purpose is to gather more information for
your models—not get agency approval
-------
Step Nine: Change Models; Write
Narrative
• Models are revised or sometimes scrapped
and begun again:(
• Narrative and footnotes are important;
models do not 'stand alone'
• Finished
products are siicTT
Assess the Logic Models..
(continued)
• Logic (In THEORY: Po the resoura
support the activities!' Can the activities
produce the outputs** Do the customers
receive the outputs** etc... Poes everything
make sensed)
• Information Gaps (Where are the gaps in
information/ Why/)
Step Ten: Assess the Logic
Models (You will have been doing
this step all along!)
it! • GPRA CPo the programs appear
to logically lead to the goals/)
Flow of goals, objectives
subobjectives...)
Metrics (Are there metrics/
Are they 'good'/ Can they
aggregated/)
Assess the Logic Models...
(continued)
• Completeness (Is the program logic
complete/)
• Externalities (Are there
unaccounted for externalities/)
• Mission (Poes the program fit
within the mission/ goals/
objectives/ subobjectives/)
-------
Step Eleven: Using Logic Model
Assessment, Identify Evaluation
Questions
• Is there a need for the program(*(calls for- a
NEEDS ASSESSMENT)
• Is the program logically designed!* Care all
parts of the logic model present-*) (calls
for further DESIGN EVALUATION)
• Can the program be implemented as
designed*' (calls for IMPLEMENTATION
EVALUATION)
Using Logic Model Assessment,
Identify Evaluation Questions
(continued)
Is this program worth
the money!* (calls
for a COST-BENEFIT
ANALYSIS)
Other Questions:
Using Logic Model Assessment,
Identify Evaluation Questions
Continued...
• if implemented as designed, can/will the
outcomes be attained!* What about
unintended outcomes!* (calls for
OUTCOME EVALUATION)
• Just how much is this program
contributing to the attainment of desired
outcomes!* (calls for an IMPACT
EVALUATION)
an
Step Twelve: Create a Potential List of
Jobs
• Group questions by type
• Translate evaluation questions into jobs
• Estimate amount of time, FTE's and
resources for each 'job
-------
wa
, ••
Step Thirteen: Prioritize the List and
Recommend Future Work
• Environmental risk and risk reduction
• Level of EPA investment £*
• Importance of knowledge gaps
• Level of stakeholder interest
• Importance for restoring or
preserving E PA's credibility
...DEVELOP 'PIECES ANP PARTS'OF
PRODUCTS ALONG THE WAY
• Logic Models
• Corresponding Narratives
• Observations
• Key Evaluation Questions
• Prioritization of Potential Jobs
• 'Lessons learned' about the Process
Step Fourteen: Assemble Fin^l
Products
• Written Report
• Supporting
Documentation
• Oral Briefing & Materials
• BUT, TRY TO...
From this.,
To this...
------- |