YOU TRUST YOUR ENVIRONMENTAL AUDITING PROGRAM?
ORGANIZATIONAL ISSUES IN ESTABLISHING
AND IMPROVING ENVIRONMENTAL AUDITING PROGRAMS
Alfred A. Marcus
Mark V. Nadel
Battelle Memorial Institute
Human Affairs Research Centers
4000 N.E. 41st Street
Seattle, WA 98105
November 1983
This report has been prepared under contact No. 68-02-3169(WA42) for the
Regulatory Reform Staff, Office of Policy and Resource Management, U.S.
Environmental Protection Agency. It reflects solely the findings and
conclusions of the contractor, not those of EPA or any other government
entity. It may, however, be useful to managers or officials in business,
state agencies, professional standards groups, or other organizations
interested in establishing or improving environmental auditing (EA)
programs.
-------
TABLE OF CONTENTS
1.0 INTRODUCTION . . . . . . . . . . 1
2.0 ORGANIZATIONAL CAPABILITY . . . . . . . 1
3.0 ORGANIZATIONAL INTEGRITY . . . . . . . . 1
4.0 ORGANIZATIONAL PLANNING . . . . . . . 3
5.0 STRATEGIC AND STRUCTURAL ISSUES . . . . . . 3
6.0 MODELS FOR AUDITING ORGANIZATIONS . . . . . . 4
7.0 METHODOLOGICAL ISSUES IN ASSESSING
ENVIRONMENTAL AUDITING PROGRAMS . . . . . . 7
REFERENCES . . . . . . . . . . .11
-------
1.0 INTRODUCTION
Many companies have established or are considering establishing
Environmental Auditing (EA) programs. These programs can result in
favorable publicity, better compliance, corporate protection from
liability suits, improved management of risk, reduced regulatory fines,
improved iarker health and job satisfaction, and decreased on-the-job
injuries.’ While much has been written about the evolution, role, and
benefits of auditing programs, a neglected topic has been the
organizational issues. Can corporate managers have faith in the
capability and integrity of their auditing programs?
2.0 ORGANIZATIONAL CAPABILITY
Whether or not EA programs are capable of achieving their objectives
depends on some of these factors: the program’s objectives, its tasks and
performance indicators, and the availability of human and material
resources.
Objectives-—The substantive direction of the EA program may involve
either or both: (1) the assessment of environmental hazards and risks,
control measures, and the effectiveness of mitigation systems; and (2) the
examination of a company’s adherence to environmental policies,
procedures, and practices.
Tasks and Performance tndicators———The tasks performed by an auditing
program need to support its goals. Management and EA personnel require
indicators for assessing program progress.
Staff Qualifications-—EA staff require appropriate training and
experience. Procedures for verifying staff competency as well as
providing training opportunities are necessary. The length of service of
staff and their time commitment (whether full—time, part-time, or on-call)
are important.
Budget——A significant manifestation of a program’s resources is its
budget. While size of budget is easily quantifiable, adequacy of budget
is another matter. Few program managers believe their budgets are
adequate. As with staff size, adequacy of budget depends on role,
mission, organizational place, and other functional factors, as well as on
such things as the degree of automation.
3.0 ORGANIZATIONAL INTEGRITY
Integrity refers to whether the EA program has the organizational
power and independence to achieve its goals. Indicators of integrity are
structure and placement within the organization, management level of
supervision, internal authority and autonomy, staff recruitment and exit
processes, and external professional relations.
1.
-------
Organizational Structure and Placement-—Organizational charts snow
reporting relations which are an indication of how much power and
independence the EA program has. They may also show whether the E
program has become a permanent part of the organization or whether it is
temporary and ad hoc. While there is no “best” method, a permanent
program is likely to provide a firmer basis for auditing because of
increased staff experience and organizational memory.
High—Level Reporting-—The level of EA reporting in the organization
is formal evidence of “clout.” While the EA program need not report
directly to the highest level in the organization, it should report to a
sufficiently high level to be taken seriously. Reporting level indicates
the organizational rank of the EA progam director and also the number of
layers between the director and senior corporate decision makers.
Ultimately, the official to whom the program director reports needs to
have the authority to bring about changes recommended by auditors,
especially if the plant managers do not or cannot do so.
Internal authority——Auditors need to have the internal authority to
carry out audits, recommend changes, and gain access to records and
personnel. To acquire such authority, they may have to rely on the
support of senior officials. So that the recommended changes of the EA
program are actually carried out, the program may need a formal system of
logging recommendations and keeping track 0 f the organization’s
compliance.
Autonomy——If the budget depends on the audited sections of the
organization, the EA program’s independence is likely to be
threatened.” Independence, however necessary, must be tempered with
attributes that prevent against isolation. Distance from line operations
and practical decision—making can render the EA program irrelevant.
Methods of preventing such isolation are: (1) close contact with
shop—floor operators including frequent interaction and observation;
(2) use of inuiediate exit interviews with facility managers on completion
of an audit; and (3) reliance on part—time management and technical
committees made up of individuals who have production responsibilities.
Recruitment and Exit Processes-—The career path for EA personnel is
important. If the audit staff is part of a fast-track advancement
process, then it is likely to attract better qualified personnel. On the
other hand, if auditing is a temporary assignment, auditors may be overly
concerned about the impact of their auditing activities on future
assignments. If auditing is perceived as a career detour or dead—end
assignment, it will be difficult to recuit or retain competent personnel.
External Relations--To increase independence, auditors should be
encouraged to communicate with their professional peers. Currently,
environmental auditing is less developed and institutionalized than
financial auditing. Internal financial auditors have their own
professional association, the Institute of Internal Auditors. A similar
Institution of internal environmental auditors could provide a reference
group that would bolster independent judgment, improve the state of the
2
-------
art, upgrade professional qualifications and standards for conipet nt
audits, and further generally the credibility of audit programs.
Increased peer communication between auditors can built these benefits,
whether or not such a professional organization emerges.
4.0 ORGANIZATIONAL PLANNING
Organizational planning can be a useful tool in creating EA programs
that have capability and integrity. A plan may cover (a) the formal
system of controlling and organizing the work; (b) relationships within
the program and between it and other organizational components;
(c) staffing; and (d) external relations.
In addition, the following information would be useful:
o organization charts showing how the audit program is structured
and fits into the overall organization;
o listings of the purposes, tasks, and responsibilities of EA
units including any departments, committees, groups, project
teams, and part-time and/or ad—hoc task forces;
o names of full—time, part—time, and on-call officials who are
involved; their qualifications and experience; and their
responsibilities (for audjting and/or production);
o provisions, if any, for routing and sign—off on items of
significance;
o rotational policies, career paths, and provisions for the
training and retraining of audit officials;
o performance indicators and other provisions for the analysis of
both environmental performance and environmental audit
performance; and
o other relevent policies, procedures, and practices that may
affect the authority or independence of auditing components, the
relationships among them, or the relationships between them and
the production organization.
5.0 STRATEGIC AND STRUCTURAL ISSUES
Even with the best plans, auditing may be difficult to integrate with
other organizational purposes. Auditing programs may not fit into a
firm’s structure and may lack clout and influence. The following types of
strategic and structural issues may affect auditing.
Competing Corporate Purposes
Corporate executives have to balance environmental compliance with
other corporate goals, the main one being profit. Even if not strict
3
-------
profit maximizers, bu iness leaders are likely to set “targets” of everall
financial well—being.” As a consequence, a common problem is that plant
supervisors tend to disengage their responsibility for environmental
quality from their production and supervisory duties. One method of
correcting this problem is to require supervisors to file regular reports
with auditors, as these reports compel consideration of auditor concerns
on a scheduled basis. However, good indicators of performance are not
easily found. They need to be reliable 1 available, valid, objective, and
conceptually distinct; and they must be adjusted statistically for company
and plant differences, i.e., plant size, age, and operating status.
Auditor/Organization Relations
How the organization responds to the concerns of auditor may be
strongly related to the auditors’ attitudes and capabilities. The
auditing staff invests a large amount of time with regulatory officials
and others outside the organization. They carry outside pressures into
the organization and often raise questions or anticipate problems that
other staff may prefer to ignore. Even product quality assurance units,
which are directly related to production, are sometimes at odds with the
organization’s production units. The potential for goal conflict between
EA personnel and other personnel in the organization is obviously greater.
Constructive corporate attitudes toward the EA program can be
manifested in formal and informal ways. On a formal level, official
company pronouhcements to staff about the EA unit need to stress the
importance of the program. Exhortations about the value of the program
may not solve all problems, but the absence of formal pronouncements sends
a negative message of its own. A company, if possible, should try to get
beyond formal platitudes. Efforts should be made to create a favorable
impression of the environmental audit program. In this regard, it may be
useful to conduct evaluations of company attitudes about the performance
of audit teams.
Additionally, capable personnel must be chosen to participate in the
auditing program. The power of auditing officials ultimately rests on
their persuasive abilities. They must be able to develop and present
cogent arguments that often require immediate action by plant managers and
top management officials. Top management commitment is a critical factor,
as well as line, staff, and worker involvement in implementation. To
improve the auditor/organization interface, a stable and committed
workforce, training, informalcommunications, and participation by
individuals from diverse levels in the organization are needed.
6.0 MODELS FOR AUDITING ORGANIZATIONS
Production organizations are characterizeØ by hierarchy, centralized
decision making, and control through planning.° However, auditing,
which can be equated with the quality of production, requires different
goals, performance measures, and organizational structures.’ In the
mechanistic model, the top gives the orders, and the effectiveness of the
4
-------
organization is assessed by the degree of compliance. 8 However, the
application of this model to auditing would tend to reduce search
procedures and the ability to detect and correct error, which are primary
auditing functions.
Different organizational arrangements may be used for auditing.
Roberts and Bluhm found the following environmental management
arrangements among the electric utility companies they examined: 9
o permanent integrating or planning units;
o semi-permanent project teams;
o temporary committees or task forces;
o formal or informal top management groups; and
o permanent i ntergroup teams.
Three organizational forms for auditing may be distinguished: (1) the
separate full-time department, (2) the permanent top echelon committee,
and (3) the ad hoc, temporary task force or team.
Studies have shown that separate, full-time regulatory response units
are becoming increasingly common. Holmes found in a survey of Fortune 500
firms that there was a movement toward greater formalization, away from
the task forces semi-permanent project teams and toward the permanent,
full—time units. ” Roberts and Bluhm conclude that no organizational
form is the best, but that strong managers tend to be those who encourage
“a certainjmount of questioning and dissent through organizational
diversity. ’
Organizational Diversity
A corporation often obtains information about environmental issues
from a diverse set of key individuals and groups with an interest in a
particular issue. The information received, however, often depends on
organizational position. Corporate executive officers, legal staff,
public relations specialists, lobbyists, technical people, and operating
managers or engineers all may have sharply different viewpoints depending
on their professional training and personal inclinations. Each is likely
to interact with different external groups in forums that differ in their
formality, specificity, and emotional intensity.
Nonetheless, in some situat pns the flow of information to top
officials may be very one_sided.s Lower level officials may (1) report
only those facts that support their position; (2) structure the
presentation of facts; and/or (3) distort the facts. Production oriented
managers may not encourage innovative ideas which threaten their control.
Janis explains a “groupthink” phenomenon in which there is an “illusion of
invulnerability, unanimity, a suppression of doubts, the functioning of
self appointed mindguards, and a docility fostered by suave
5
-------
leadership.° 13 A top manager can circumvent conscious or unconscious
efforts at selective misinformation by (1) seeking alternative sour:es of
information; (2) creating new channels of reliable information; and
(3) surrounding himself with divergent views.
As an alternate source of Information, auditing can improve the
quality of decision—making. Integrated programs may consist of
independent full—time units, part-time audit committees, and project teams
of on—call officials. Multiple auditing components increase the variety
of viewpoints and correct for bias and selective misinformation.
Advantages of the independent, full-time units include: more in—depth
analysis; less consideration of “bottom—line” questions of production and
efficiency; and sole focus on environmental quality considerations.
Disadvantages include: lack of power and organizational isolation;
difficulty maintaining staff quality and attracting an appropriate mix of
technical disciplines; and potential opposition from the production line
and technical/engineering units. Advantages of part-time committees
composed of management, technical, and operational officials include:
direct linkage between environmental issues and production realities;
significant authority for implementation; broad viewpoint across operating
units; and forum for discussion based on wide-ranging expertise.
Disadvantages include: competing responsibilities; insufficient time for
tasks and analysis; and primary goal of maintaining near-term production.
On-call officials and project teams can serve as important information
sources and implementation linkages and may be fruitfully used in
conjunction with either the full-time units and committees.
The “Best” Design
Obviously, the “best” organizational design for a part cular
company’s auditing program will depend on a variety of key factors
specific to the company including: 1) the number, type, and location of
the plants operated by the company; 2) the overall organizational
structure of the company; 3) the previous environmental record of the
company (e.g., more resources may be needed if there has been a history of
environmental problems); and 4) corporate culture. Nonetheless,
organizational diversity which may involve the use of part-time and
on—call officials in conjunction with full—time auditing units or groups
is likely to be a positive design feature. According to La Porte,
error—free organizational de .jgns rely on diversity to compensate for
problems and correct errors. In theory, systems are as reliable as
they are diverse; that is, diversity is meant to decrease the probability
of error and risk.
The existence of complementary organizational forms is like the dual
braking system in a car and like other redypdant features of technological
systems which are designed to reduce risk. ° Metlay has found th
diverse sources of information exist in almost all organizations.
However, often the organizational response is to ignore the information
provided, although sometimes policies may be altered. Administratively,
diverse units can increase the need for coordination. ’ 8 Greater
diversity may mean a greater likelihood that some individuals will
discover mistakes and propose corrective action, but a decreased
6
-------
likelihood that changes will be made. Thus, the value of additiona units
ultimately depends on their being linked to operational units that ise
their outputs. As Metlay emphasizes, the autonomy of d erse cc ents
has to be balanced with some degree of interdependence. ’ Independence
facilitates the detection of errors and may bring to light alternative
means for rectifying errors; interdependence is needed for the
organization to accept the identification of error as well as to take
ameliorative action.
7.0 METHODOLOGICAL ISSUES IN ASSESSING ENVIRONMENTAL AUDITING PROGRAMS
A useful technique for obtaining information about the functioning of
EA units is in-depth, open-ended interviews with audit officials and
production unit staff. Qualitative evaluations may generate the
information needed for improving audit programs. However, analysis of
open—ended interview material is diffic Jt because responses are not
necessarily systematic or standardized.’ ’ In—depth interviewing as a
method of assessing EA units assumes that:
a. Interviewees can and will express their subjective impressions
to interviewers . However, if the interviewers are working for
a management audit improvement committee and the interviewees
are officials of the audited unit, then the information provided
may be biased or distorted.
b. Interviewers can understand and interpret subjective
impressions. . However, if the interviewers are of one discipline
with one perspective and the interviewees are of another
discipline with a different perspective,’ comprehension may be
difficult.
c Interviewers can make comparisons and draw policy implications
from their interpretations . However, if the interpretations are
faulty because of lack of frankness on the part of the
interviewees and inability to understand on the part of the
interviewers, then the inferences drawn from such interviews may
be false.
Subjectivity
Organizational assessments based on interview information are largely
subjective; that is, they cannot be completely verified by scientific
methods. Qualitative analysis of interview information has these
subjective features:
o an emphasis on an inner perspective (what people know and have
experienced);
o problems of temporal sampling (decisions about the time-period
or periods of observation);
7
-------
o distortions introduced by the sequence, wording, and clarity of
questions, and by the probes, follow—ups, and note-taking
conducted during the interviews; and
o analysis and interpretation of the interview information (the
ability to separate the serious from the irrelevant).
Validation Techniques
Various methods of validation may be used to minimize subjectivity.
One method is to use standardized questionnaires rather than relying on
informal interviews. However, with standardized procedures, the
assessment is apt to miss important and relevant information such as the
subjects’ perceptions. Another method of validation is moving from
open—ended questions to formulation of hypotheses. As the inquiry reveals
patterns and major categories of interest, it may evolve to a more
deductive approach that focuses on verifying and elucidating what the
evaluators believe the initial information shows. Understanding based on
open-ended interviewing often involves moving between inductive
“encounters” to form “ iypotheses” to deductive attempts to obtain evidence
and “solidify” ideas. 2 ’ The danger is that initial interpretations may
bias additional analysis. Beyond focusing on confirmation, inquiry must
also be sensitive to different definitions of the problem, contradictory
explanations, and alternative insights that might invalidate or change
initial impressions.
Triangulation
“Triangulation” is the classic method of validating interview-based
evaluation information. It is the attempt to substantiate observations by
the use of diverse sources, analysts and perspectives, and methods.
Denzin has identified the following types of “triangulation”: (1) the-use
of a variety of data sources; (2) the use of several different j searchers
or evaluators; and (3) the use of multiple methods of analysis.’
Variety of data sources——Managers concerned with improving their EA
programs need to collect documents such as the EA program’s organizational
plan and examine records, statistical information, and program reports as
well as supplementing interview data with direct observation. Observation
can be very important. For example, the fact that an environmental
auditing group is located in a trailer in the company’s parking lot and
that members of the group must obtain visitor’s passes before passing
through plant gates can indicate that the group has little influence.
Multiple sources of information have to be brought together to
construct a comprehensive picture. This means that evaluators should
endeavor to compare and contrast different program information derived at
different times by different means. For example, in studying audit
groups, evaluators may need to understand not only the audit program under
investigation but the entire organization of the company. The boundaries
of the audit system may be hard to define. They may include not only the
audit group and the plant and production system, but also additional
8
-------
elements, such as the corporate organization and its quality assurance
departments, plus suppliers, trade associations, regulatory agencies, and
other external organizations.
Interviews may need to be carried out with some members of these
external organizations. At a mtnimum, leaders of the audit groups and
members have to be interviewed; to insure comprehensive coverage, separate
interview guides for other officials may be needed. Additionally, to
ensure accuracy and reliability, the same question should be asked many
times. However, deciding how many people need to interviewed and how many
times questions are asked requires judgment. Budgetary and time
constraints often play a major role in making these determinations.
Different evaluators-—Another method of verification, if the EA unit
and the evaluation group are both large, is to use different evaluators
with different disciplinary backgrounds and perspectives. More than one
person should do interviews, analyze information, and write assessment
reports. A single evaluator may not be able to maintain objectivity and
neutrality through the evaluation process. Therefore, teams of analysts
with different theoretical perspectives and attitudes may be needed. Some
may discover positive program findings, others negative findings, and a
more balanced picture may emerge.
Multiple methods of analysis——A final method of “triangulation” is to
use multiple methods of analysis. For example, it is possible to do both
qualitative and quantitative analysis or to convert detailed qualitative
descriptions into quantitative scales for the purposes of statistical
analysis. Qualitative and quantitative data may be gathered separately and
compared. With qualitative methods, issues may be studied in depth and
detail. Data collection and analysis are not constrained by
pre—determined categories. The advantage of using quantitative methods is
to measure the response of many subjects to a few pre-selected questions.
Using both methods can increase the validity of results.
Getting Reactions from Subjects
Managers can learn a great deal about the accurateness, fairness, and
validity Qf their findings by allowing the subjects of the study to
comment. 2 The fact that participants in the study are “unable to
relate” to t description and analysis indicates that a report lacks
credibility.’ However, when a management review committee is doing the
evaluation of the auditing unit and the subjects are members of the audit
unit, ft is unlikely that they will be able “to relate” neutrally to the
report, no matter how accurate or valid it may be.
Different methods of verification produce different results, and
decision makers need to be sensitive to the mode or modes they choose.
Assessment of EA units does not provide definitive answers to evaluation
questions. I does provide “credible, plausible, and probable”
perspectives.’ 5 Nevertheless, such assessment is neither completely
rhetorical nor entirely arbitrary. As House points out, “Persuasion is
9
-------
directly related to action . . . Once the burden of ç rtainty is lifted,
the possiblities for informed action are increased.” °
A Heuristic Action Approach
In making recommendations for EA establishment and improvement, a
perspective that allows decison-makers to watch recommendations evolve and
mature over time may be appropriate. The process of evaluating thus
becomes subject to “scientific discipline” or “rationality TM under a
“heuristic action research protocol,” the key elements of which are:
(a) an emphasis on decision latitude —- avoiding fixed or
irrevocable assessments to allow for subject initiative in
defining and solving problems;
(b) a shift from final assessments to in—process evaluations;
(c) systematic attempts to keep an open mind and allow for
corrections instead of relying on a priori assumptions; and
Cd) developing practical bases for transforming initially
qualitative assessments into empirically validated ones.
10
-------
REFERENCES
1. On environmental auditing, see Blumenfeld, K. and M. Haddad, “Beyond
the Battleground: A (Non) Regulatory Perspective on Environmental
Auditing, 1 ’ Draft, 2/15/83, prepared for U.S. Environmental Protection
Agency, Regulatory Reform Staff; Palmisano, J., “The Evolution of
Environmental Auditing: Implications for Firms and for Regulators,”
paper prepared for APCA Annual Meeting, Atlanta, GA, June 1g83;
Hedstrom, G. S., J. L. Greene, J. R. Ehrenfeld, et al.,
“Environmental Auditing Programs: Benefits to Industry,” prepared by
the Center for Environmental Assurance, Arthur D. Little, Inc., for
the U.S. Environmental Protection Agency, Regulatory Reform Staff; E.
H. Clark, II, “Environmental Auditing Programs: Benefits to the
Environment and Government,” prepared by the Sterling Hobe
Corporation f or the U.S. Environmental Protection Agency, Regulatory
Reform Staff. Also see Session 19 Environmental Auditing , papers
prepared for presentation at the 76th Annual Meeting of the Air
Pollution Control Association, Atlanta, Georgia, June 19—24, 1983.
2. On the capability and integrity of safety review programs, see
Battelle Pacific Northwest Laboratories, “Recommended Guidelines for
the Safety Review Function,” Seattle, WA, September 1983; Marcus, A.
A. and R. Osborn, “Safety Review at Nuclear Power Plants,” Battelle,
Seattle, WA, February 1983; Marcus, A. A. and S. McClay, “Safety
Review and the Business Corporation,” Battelle, Seattle, WA, April
1983.
3. Pfeffer, J. and G.R. Salancik. 1978. The External Control of
Organizations . New York: Harper and Row.
4. Joskow, P. 1974. Inflation and Environmental Control. Journal of
Law and Economics (October).
5. Roberts, M.J. and J.S. Bluhm. 1981. The Choices of Power .
Massachusetts: Harvard University Press.
6. Hull, F.M. and J. Hage. 1982. Organizing for Innovation: Beyond
Burns and Stalker’s Organic Type. Sociology 16(4):564-577.
7. Osborn, R.N., J. Olson, P.E. Sommers, S.D. McLaughlin, 11.5. Jackson,
M.V. Nadel, W.G. Scott, P.E. Conner, N. Kerwin, and J.K. Kennedy,
Jr. 1983. Organizational Analysis and Safety for Utilities with
Nuclear Power Plants: Vol. II. Perspectives for Organizational
Assessment . Draft Researrch Report. NUREG/CR - 3215.
8. Landau, 11. and R. Stout, Jr. 1979. To Manage is Not to Control: Or
the Folly of Type II Errors. Public Administration Review
39:148—156; Downs, A. 1967. Inside Bureaucracy . Boston: Little,
Brown and Co.
9. Roberts, 11.3. and J.S. Bluhm.
11
-------
10. Holmes, S.L. 1977. Corporate Social Performance: Past and Present
Areas of Commitment. The Academy of Management J . 20:433-438; and
Holmes, S.L. 1978. Adapting Corporate Structure for Social
Responsibility. California Management Review 21:47-54.
11. Roberts, M.J. and J.S. Bluhm.
12. Halperin, M. 1974. Shaping the Flow of Information. pp 102-115 in
F.E. Rouke (ed.), Bureaucratic Power in National Politics . Third
Edition. Boston: Little, Brown and Co.
13. Janis, I.L. 1972. Victims of Groupthink . Houghton Miff un Co.
14. Halperin, M.
15. La Porte, T.R. 1982. On the Design and Management of Nearly
Error—Free Organization control Systems. pp. 185-200 in D.L. Sills,
C.P. Wolf, and V.B. Shelanski (eds.), Accident at Three Mile Island:
The Human Dimensions . Boulder, Colorado: Westview Press.
16. Metlay, D.S. 1978. Error Correction in Bureaucracies . Ph.D.
Dissertation, Department of Political Science, University of
California at Berkeley.
17. Wilson, J.Q. 1967. Innovation in Organization: Notes Toward a
Theory. pp 193-218 in J. Thompson (ed.), Approaches to
Organizational Design . Pittsburgh: University of Pittsburgh Press.
18. Anthony, R. 1982. Accountability and Credibility in the Management
of Complex Hazardous Technology. Policy Studies Review 1(4):712; and
Pressman, J.S., and A. Wildavsky. 1973. Implementation . BerKeley:
University of California Press.
19. Metlay, D.S.
20. Patton, M.Q. 1980. Qualitative Evaluation Methods . Beverly Hills:
Sage.
21. Patton, M.Q.
22. Patton, M.Q.
23. Patton, M.Q.
24. Patton, M.Q.
25. Patton, M.Q.
26. House, E. R. 1977. The Logic of Evaluative Argument . Los Angeles:
Center for the Study of Evaluation, UCLA.
27. Osborn, R.N., and J. Sutherland. “An Elaboration of Linkages Between
Substantive and Diagnostic Components of a Proposed Management and
Organization Guidelines.” Unpublished manuscript.
12
------- |