I
ry
DRAFT
U.S. EPA Headquarters Library
Mail code 3201
1200 Pennsylvania Avenue NW
Washington DC 20460
1990 Strategy Paper:
Regional Oversight of State Programs
October 9, 1980
,,, '•-! Agency
**•*& • • - • . -i»-i FM-311-A
401 M Street. S.W.
l, DC
Bill Garetz
Joan LaRock
Stan Meiburg
Tom Kern
Martin Geitz
-------
-00/o
CONTENTS
Executive Summary
Introduction
Page
E-l
i
Part Is Assessing the Current State of the Art
Chapter 1: How Oversight is Now Conducted 1
Chapter 2: The Evolution of State Programs.... 10
— Phases I, II and III
Chapter 3: Analysis of Alternative 14
Oversight Procedures
for Phases I and II
Part II: Advancing the State of th_e_A.rt
Chapter 4: Shifting from Oversight 19
of Transactions and
Procedures to Oversight
of "Performance"
Chapter 5: Increasing the "Scope" of 26
Regional Oversight —
from Activity Management
to Project and Program
Management
Chapter 6: Monitoring "Performance" at 33
the Project and Program
Level
Chapter 7: Proposed Oversight Procedures 35
for Phases III, IV, V and VI
Part III: Tying It All Together
Chapter 8: The Proposed Role of the 39
Regions and of EPA
Headquarters
Chapter 9: Concise Summary of Recommended 44
Oversight Procedures for Phases
I through VI
Appendix A: Alternative Approaches for Improving A—1
Project-Level Review by the States
Appendix B: A Possible Experimental Approach for B-l
Assessing a State Program's Impact
on Water Quality
Background Paper: Current Regional Oversight Methods
(Region-by-Region Descriptions and
Summaries) — [not attached]
-------
1990 Strategy Paper:
Regional Oversight of State Programs
EXECUTIVE SUMMARY
EPA is now delegating substantial elements of the
construction grants program to State agencies, in accordance
with Congress' mandate in Section 205 (g) of the .Clean Water
Act. With the advent of delegation, Regions have had to
shift from directly operating the construction grants program
to overseeing how the States run it. This shift has resulted
in the use of a wide variety of oversight practices in diffe-
rent Regional offices. This paper attempts to: 1) catalogue
the different practices which Regional offices are now using
in construction grants; 2) analyze these practices and suggest
a pattern which seems to be evolving across Regions; and 3)
prescribe how Regional offices might proceed in the future of
full assumption by States of responsibility for the construction
grants program.
CURRENT OVERSIGHT PROCEDURES
At this time, Regions currently rely on both formal and
informal methods of oversight. The principal formal methods
now in use include:
o Review of Current Transactions; In most Regions,
this is a routine review of State documents requiring
Regional approval. Every Region receives a summary
or checklist of what is contained in the package
for EPA approval, detailing all of the significant
supporting documents in a form previously agreed
to. In addition, two Regions carry out a selective,
in-depth review of specific State actions {e.g.,
facility plan review) as they are received in the
Regional Office.
o Paperwork sampling: Quarterly, most Regional Offices
review, at the Regional Office, 10% of the paperwork
received for Regional approval during the previous
quarter. In general, the sample is not totally
random; Regions usually emphasize projects that
are large, difficult, environmentally sensitive, or
troubled. A few Regions instead conduct an intensive
semi-annual review at the State Office of 10% of
recent paperwork.
Despite the use of this formal oversight, informal over-
sight methods are the primary sources of information about
the State operations during the years immediately following
delegation. These methods include:
-------
E-2
o Day-to-Day Communication; Most Regional Staff are
in daily communication with staff in delegated States.
The Regions consider such contact to be the primary
means by which they become aware of problems within
the State program.
o Regularly Scheduled Meetings with_ Staff Program Managers
These meetings can be weekly, bi-weekly or monthly,
and aid in problem resolution, discussion of new
policy and information exchange. During the first
1-2 years after delegation, the Regions consider
these meetings to be the second most important
source of information about State problems with
delegation.
REGIONAL OVERSIGHT PROCEDURES ARE CHANGING
At present, most Regions use the formal and informal
oversight methods to review specific activities which States
perform (e.g., review of plans and specifications; promotion
of minority business enterprises; review of environmental
assessments). Some Regions have moved beyond this, however,
to reviews which look not at activities themselves but at the
procedures which States use to conduct these activities in
specific cases.
One Region, for example, conducts an in-depth review of
one or more projects in a particular State with an emphasis
on what the State could do to improve its procedures so that
".everything fits together" for better management of the
project. Another Region reviews 10% of all on-going projects
and compares expenditures and costs to the project budget.
Developments in these three Regions reflect a pattern
which we see occurring, over time, in other Regions and States
as well. We feel that three distinct phases of oversight can
be discerned in the first two years since the signing of the
first delegation agreement (Phases I, II and III). However,
most Regions are still operating in the first or second of
these phases, and are just beginning to look ahead to how
their activities should change as State capabilities improve.
If EPA can succeed in developing its oversight relation-
ship further, we may anticipate three additional phases,
based on current trends and the program's ultimate objectives
(Phases IV., V and VI). These phases of oversight should
evolve in conjunction with improvements in the State's ability
to carry out the constructions grants program. Table E-l
gives a brief description of the six phases, and the report
describes each in detail.
-------
E-3
WE RECOMMEND THAT REGIONS USE OVERSIGHT TO HELP
STATE PROGRAMS GROW
We believe that Regional offices should structure their
oversight methods in such a way that they encourage growth
in State capabilities. Based on our survey of the Regions,
we would hope for the following desirable changes in Regional
oversight methods:
o In Phases I and II, Regions should oversee how
States manage specific activities. That is, they
should focus on the management not of projects,
but of distinct "activities" and procedures for
carrying out these activities. This would include
making sure that States have adequate staff to do
a realistic job in performing a given activity.
By the end of Phase II, the State should have
achieved such high quality and consistency in
its management of activities that the Region.can
reduce its formal paperwork sample from 10% to 5%
of recent actions in each activity.
o In Phase III, Regional oversight should begin to
emphasize coordination among activities to help
ensure sound project-level management. Success
in Phase III would lead to Phase IV, in which
the State and Region would focus on improving the
management of the municipal wastewter program as
a whole. They would do so by emphasizing improved
priority setting among projects and better long-
term scheduling. By the end of Phase IV, the role.
of the Region should have to have shifted from that
of a "supervisor" to that of a "franchiser" or
indirect manager, concerned not with specific
details but with sound overall program management.
o In Phase V, the State and Region would strive for
full consolidation of the municipal wastewater
program into a broader program of integrated water
quality management. Regional oversight would
encourage States to coordinate the scheduling and
design of municipal wastewater facilities with other
point and non-point sources control efforts. In
Phase VI, the State should have achieved integrated
water quality management. Regional oversight, to
the extent that is is still required, would focus
on identifying new methods or procedures which
would help the State achieve greater efficiency in
its operations.
-------
E-4
These differing patterns of Regional oversight, based on
evolutionary changes in the quality of State programs, should
directly affect the kind of information EPA uses to oversee
State operations. While at the beginning of Phase I, the Region
oversees State operations solely through the review of current
transactions, (i.e., by reviewing documents for quality after
the State review is completed), this should not remain the
dominant pattern of Regional oversight for long.
By the end of Phase II, Regional oversight should have
initiated the review of State procedures and start to de-
emphasize the review of individual State transactions. For
example, with regard to MBE, the Region should focus more
on how the State goes about ensuring adequate MBE partici-
pation, and spend relatively less time reviewing individual
MBE documents. By the end of Phase III, the Region should
emphasize oversight through direct monitoring of State
performance. To return to the MBE example, a Region might
assess a State's MBE performance directly by determining
what portion of project dollars have been awarded to MBE
firms rather than by reviewing MBE documents or procedures.
In later phases, the Region would increase its use of direct
measures of performance in overseeing State performance.
Ultimately, they might make use of indicators of environmental
quality to provide feedback on a State's management performance.
Even in the most advanced stages of State delegation,
when the Regions are relying primarly on direct measures of
performance for overseeing State programs, we still believe
that Regions should continue a 2-5% sample review of indi-
vidual transactions (i.e., documents) for quality control
reasons, and should be prepared to review procedures when
performance (as measured directly) fails to meet expectations.
Nevertheless, we strongly urge Regions to move from oversight
based primarily on review of documents to oversight based
mainly on review of procedures, and finally to oversight
based primarily on direct measurement of performance, as
rapidly as the growth of a State's capability will allow.
This will encourage the Regional role to evolve from
"supervisor" to "franchiser" or indirect manager by the end
of Phase IV. By Phase VI, the Region's role will approach
that of a true environmental manager.
-------
^ —
^^B *T1
m 35
w >
cn
w
^
M
O
03 g|
Oi CO
ft
fo cn
Oi ft
^ ft
ft
(0 y
1 P
CO
03
C P
0) O
p- P'
ft CO
K <
(B
3 a
^^^. 0J
A 3 p-
^V P 3
CD CD
3 1
CO
3
ft
£
3
(0
P
CO
pj
0*
o
0
j^
•
r
V
p*
5*
cn
^
0
tQ ft fD i-3
C 0 3 =r
P 4! O CD
P-il C CO
ft QJ 1 ft
>•< co P P
03 ft
3 P* CD CD
9) 3 3
3 ft CD —
P fD 3 £
03 03 ft p-
fD H — ' ft
3 (D 3-
(D ft D*
3 CD ft
ft Q.03 3*
P- (D
P CO 53
ft fD
(D ftU3
1 O P-
0
£ 3
*1 CO
yif
cn ^3 co
p o P
3 <0 3
13 CO CD
CD Ml (1)
»l to
*-> o
H 3 P
(D O*
< Ui O
CD (D
O CD
0 X
MI p< n
CD
QJ totj
O dP ft
O
c ^rt ^T
3 P 3*
(0 13 fO
3 (D
CO ^ fD
*— ' 0 iQ
1 P-
yf o
3
CO
^d
IT]
J5
CO
M
H
<^
0
ft (D 13
0 3 31
£ O CD
(D 0
^ C CO
QJ 1 ft
co P (a
03 rt
^0 (D (D
W bj
O CD -^
03 3 £
M ft p-
P — ft
3 3*
1 tf
M CD ft
CD 03 31
< P" CD
CD 3
M CO p3
CD
3 ft 03
0) O P-
3 O
(0 1 CO
3 IT
fD
3
ft
O O 0
'•V tfP Mi S1 1 (Tel 3
fy 0 C^ * (U O
i"r3 ^^ j,,^ ^ ^ ^g u^ f^j
(DP'^lD^'t* C fD
H 3 O & * h &>
£ ^rt O W U^ (D QJ
Q U_* /ft ^p ^ 3 ^Q ^
^ [0 QJ Q
fQ O & CO M O
p- fl CO Mi Mi
CO C P- O 0
— 3 cn MI »i Q*
co 3 P-
O 3 O P- fl» 1
0 ft 3 3 3 ID
3 CO *O O O
rt n n CD ft
P- (DO
3 < <
C P- CD
(D CD QJ
*T3
CC
ft
CO
t'd
H
M
IH
0
2 ft CD H|
P O 3 31
D C n fD
P P O
03 i c cn
CO QJ ^1 ft
(B 03 ft
3 ft (D CO
ft 4 3
C (D —
CD 3 «
ft P-
*TJ '*-+ ft
hj y
0 0*
VJ. (O ft
CD 03 31
O P- CD
ft 3
1 CO Jd
M CD
CO rtiH
< 0 P-
CD O
1 CO
??*
000
p- QJ r1 o n o cn
O O CD Mi O Mi 0
<*> O co 3 3
C (Q *O ft *O (D
ft 3 M H- CD
O CD CD O 3 n C
3 3 0 C Mi cn
tn rt H3 fD CO 0 fD
<*P co 3* QJ Q. 11
. di c 30
13 CO n (D (H Ml
p p- CO 3 3
^J 50 to CO H3 O QJ
CD CO 3* CD P-
*1 O3 O P 1
SP- 3 CO CD
O P- O
•I 3 "1 CO ft
7f CD
03 < O 3
W 0 P- 3 CO
P CD (D P
3 co $ n co
13 CD C
H Ml O < 1
(D n Mi P- CD
o co co
3 *
^
a
CO
M
1-1
M
0
3 O
Q. ft
p-
o <
< p-
CO ft
*1 *^
CO 1
p-p-
03 CD
ft CO
§3
H P -
^ 3
P
03
CD
, CD
3
ft
O O
K. P- 0 O
CD (D Ml 1
< CO CD
P- CO U P
CO *1 ft
< CD 0 CD
Mi O 11
O Mi ^J
0 CD 31
O 03 CO P
CO ffl
3 CD P-
CD CO (6
3
ft P- O
CO 3 3
ft
O 11 .
CO
ft <
31 P-
CD CD
^
•t)
X
^*
CO
td
H
o
p >
3 0
QJ ft
P-
0 <
<; P-
CD ft
1 "^
CO I
p- p<
03 CO
3* <
ft fl>
'S«
H^ (jU
»< 3
P
03
(0
1
3
ft
O 0
ffff
3 <
CO P'
CD
1 C
< O
P- Ml
CD
O O
Mi C
3
^O (D
O ft
O CO
CD
flit?
C h
fl P-
2 3
11
P-
P1
DT3
CO 3*
^ P
CD CO
P* CD
O
13 0
U i.
CD
3
ft
P O
3 0
Qj*O
CD
O =
Ml
O
yd MI
CD
03 cn
p- ft
O P
3 ft
P CD
H
O P
< 3
CD P
1 O3
n (o
p- 3
03 CD
y 3
ft ft
1
ft Q ^
y CD p-
CO CD 3
Qj Pj
CO
ft Mi O
P O Ml
ft H
O 1 1
^ CD 3
O CD P
03 CD ft
n P- P-
P 3 O
3 ua 3
1
1
-------
Construction Grants Issue Paper:
Regional Oversight of State Programs
An earlier paper on the role of EPA in the construction
grants program identified three distinct Roles that EPA could
play in a program whose activities have been largely delegated
to the States. These roles can serve as models around which
the Agency's managers can more clearly define the goal of
full program delegation which we hope to achieve by 1990.
The purpose of this paper is to develop these roles in
more detail by outlining, in a preliminary way, how the
Regions might go about making the transition from the first
role, that of a "supervisor," to the second role, that of a
"franchiser" or indirect manager. It also shows how to begin
to make a further transition to the role of environmental
manager. (For a brief description of the three possible
roles, see Table 1.)
Specifically, the paper attempts to: 1) catalog the
different oversight practices which Regional offices are now
using in construction grants? 2) analyze these practices and
suggest a pattern which seems to be evolving across Regions;
and 3) prescribe how Regional offices might proceed in the
future to make oversight both more effective and more supportive
of full assumption by States of responsibility for the
construction grants program. The description and analysis of
current practices forms Part I of the paper; Part II suggests
directions for the future, and Part III recapitulates and
summarizes the conclusions and recommendations reached in the
first two parts.
PART I; ASSESSING THE CURRENT STATE OF THE ART
The paper begins by summarizing — in Chapter I — the
methods now used by the ten Regional Offices to oversee State
programs. This summary is based on a background paper
(distributed separately and not a part of this package) which
contains relatively complete though brief descriptions of how
oversight is now conducted in each of the Regional Offices,
and one- or two-page summaries of current oversight methods.*
* The information present in the Background Paper forms the
factual basis for the overview of current procedures in
Chapters 1 and 2 and for the analysis and recommendations in
Chapters 3-8.
-------
6
id
fc
Oi
-P
c
O
-P
o
3
fc
•P
co
C
0
u
•a
CD
-P
CD
CD
O
(0
C
fc
o
01
CD
rH
£
CD
CO
CO
o
04
CD
CD
fc
rf
&
W
MH
O ••
.p
to 3
0) 0
r-l
O *O
Pi *H
id
CD rH-
ff,
- fc
CD
T> 3
CD
H CO
4J CD
•rl rH
•P O
fc
c tn
id c
-H
^i 3
tn o
CD H
•P rH
id O
fc MH
jj
CO CD
rj
0 *J
en :
iH -
E
CD Id
4J tn
O
fc fc
O Cn
to
'O -P
CD C
fc id
id fc
ftU
CD
fc C
ft O
fc -p
CD O
ft 3
ft -P
to
-P C
MH O
id U
fc
•C CD
id -P
c c
r-C -H
O
fc
+J
c
o
o
Jj
c
CD
g
CD
tn
i
CD
-P
CO
id
CO
CD
-P
TO
^J
CO
fc
CD
0
.jj
U
CD
fc
•H
•O
E
0
44
to
CD
T3
CD
O
CD
ij
rfj
0,
pa
jg
o
•H
c*
3
C
•H
•*
fc
CD
tn
id
c
id
g
4J
O
CD
fc
•rH
TJ
n
•H
S_l
0
E
fc
CD
to
•H
r;
U
G
id
fc
En
"
0
4J UH
fc O
ft C
ft O
3 -H
CO +J
id
CD fc
> 4J
.rH CO
» -H
C C
(!) -rl
•P £
X -O
CD Id
£, *O
4J C
•H id
u
..
co tn
CD C
CD -H
4J g
C C
id id
fc H
•0 H
C id
id o
CO
CO -H
CD HH
-P
id -
JrJ JS,,
CO +J
•H
tQ rH
CD (d
•C 3
*H D"1
>
O rH
fc Id
ft o
-H
-P C
3 ,C
Xt 0
- +J
C
O CD
•H >
to O
•H fc
> a
fc E
CD -H
ft
3 0
CO 4->
t7i
C
•H
fc
0
•H
C
0
E
;
r-l
O
fc
4J
C
O
o
>^
4J
-H
f—i
id
3
JJ1
C
CD
^
C
O
•iH
-P
id
fc
-P
CO
•H
C
•H
g
-s
(d
E
id
fc
tn
0
fc
•
r-f
id
CD
-P
id
1 \
V3
CD
,c
4J
0
4.)
CO
CD
id
tn
CD
rH
CD
ft*O T3
-P
U
3
T3
c
o
o
0
CO
CD
3
C
•H
fi
O
u
<;
Cu
W
•
E
id
fc
tn
0
fc
ft
CD
^
c
to
CD
C
CD
fc
CD
-P
C
•H
T3
C
O
o
to
*H
"0
CD
fc
id
CO
CD
•H
O
C
CD
-H
O
•H
MH
CD
-P
C
0}
u
-H
4H
•H
C
tn
•rH
CO
f3?
cx
fr3
Xi
O
•rl
X
c
-H
*
*
• *
fc
CD
tn
td
c
id
s
rH
id
.p
c
CD
o
fc
•H
>
c
H
0
c
H
id
fc
CD
u
4J
/d
4J
C
o
•H
4J
,
4J
•H
fc
O
.P
3
id
CD
.C CD
&* -H
CD
• 0
•P C
CD Id
E E
CD 0
& MH
fc
CO CD
•P ft
CD
tn CD
fc -P
id ^
-P CD
•H rH
rH ,£
Id Id
3 4J
O1 ft
CD
rH O
Id O
4J fQ
c
CD tn
g c
C -H
0 fc
fc 3
•H tO
> C
C CD
CD
fc
fc 0
O , to
•P E
•H CO
O -H
Id C,
ft id
i
g fc
4-» id
id g
CD •<-!
fc fc
•P ft
CD
tn
fc •
id cu
rC H
0 3
CO ,
O id
CM -P
CO
-P
CO O
C -P
•H
(d tn
tn c
id -H
H
C -H
o id
•wl tlj
w^ ^*<
ij
id 0
4-> CO
c c
CD O
E -H
CD -P
O
-------
11
In Chapter 2 we describe how oversight tends to evolve
during the first two to three years after delegation. We
introduce the concept of phased development of State programs
and describe Phases I and II.
In Chapter 3 we then analyze the oversight procedures
currently in use and identify what we have found to be the
best procedures for use in Phases I and II, the period
immediately after delegation. In this period, EPA is still
clearly operating in the role of "supervisor." This analysis
completes our assessment of current oversight procedures.
PART II; ADVANCING THE STATE OF THE ART
Part II builds on the description and analysis in Part I
with specific recommendations for changes in Regional oversight
methods. Chapter 4 looks at the current efforts of the
Regions to move from oversight based on the review of individual
State transactions to oversight based on the review of State
procedures. We endorse these efforts and urge that the
Regions move a step further and begin to conduct direct
monitoring of State performance. We give meaning to the
concept "direct monitoring of performance" by citing specific
examples of how "performance" might be measured.
Chapter 5 continues the analysis of possible new direct-
ions in oversight by examining the potential "scope" of
Regional oversight. We begin by looking at current Regional
efforts to encourage States to move from activity management
to true project management. Again, we .endorse these efforts
and recommend that Regions continue to strive for a further
shift in the States from project management to program manage-
ment , that is, balanced management of the municipal wastewater
program as a whole. We also recommend that, where possible,
Regions and States move to incorporate the construction grants
program more fully into a broader program of integrated water-
quality management.
Chapter 6 ties together the distinct concepts and
objectives presented in Chapters 4 and 5 by suggesting how
the Regions might go about overseeing State performance
(rather than merely overseeing State transactions or pro-
cedures) once the State program is able to carry out true
project management. Appendix B to this paper contains a
possible experimental approach for measuring State perfor-
mance in Phase IV and beyond when a State has begun to
achieve full program management.
Chapter 7 follows this up by describing Phases III
through VI in some detail. This description includes a
projection of how a Region might go about overseeing a State
program during an extended period after delegation, as the
State shifts first to project management and then to full
program management. Chapter 7 also completes our analysis
-------
Ill
of how the Regions might go about shifting from the initial
role of direct "supervisor" of State activities to the second
role of "franchiser" or indirect manager, pointing out that
Regions should achieve the role of franchiser by the end of
Phase IV. When a State reaches Phase VI, the Regional role
should begin to approach that of an environmental manager.
PART III; TYING IT ALL TOGETHER
Part III summarizes the conclusions and recommendations
arrived at in the previous two parts of the paper. In Chapter
8, we review the responsibilities and objectives both of the
Regions and of EPA Headquarters under the blueprint for program
development and oversight developed in Part II. Chapter 9
concludes the paper by summarizing the recommended methods for
each of the six phases of oversight which we have identified
in the course of the paper. Table 3 contained in Chapter 9
presents a detailed but concise summary of these recommendations
Virtually all of the analysis contained in this paper is
based on - interviews with Regional staff. In five Regions
(Regions I, IV, VIII and IX), the study team conducted fairly
extensive fact-to-face interviews over several days with
numerous Regional and State officials, and with officials in
grantee communities and in the Corps of Engineers. In another
Region (Region III), one team member conducted limited face-
to-fact interviews. In all six of these Regions, there were
extensive follow-up telephone interviews with one or more
Regional staff members. In the remaining four Regions (Regions
II, VI, VII and X), we relied solely on telephone interviews,
typically with a single member of the Regional staff. We also
used Larry Walker's written analysis of Regions and States
where these analyses were available (Regions IV, V, VI and
VII).
Since this draft is based primarily on interviews with
Regional staff, it should be considered highly preliminary.
As we receive comments and additional information from the
Regions and from other participants in the program (i.e.,
headquarters staff, State staff, A/E firms, public interest
groups, environmentalists and others) we will make any
necessary revisions to ensure that the paper presents a fully
balanced analysis and reflects the insights and perspectives
of all participants in the construction grants program.
Comments should be forward to:
Bill Garetz,
Program Evaluation Division (PM-222),
401 M Street, S.W.,
Washington, DC 20460
(tel: (202) 755-3977;
(FTS) 755-3977).
Handwritten comments including "markups" of this paper are
welcome. Telephone comments are also welcome.
-------
PART I: ASSESSING THE CURRENT STATE OF THE ART
-------
CHAPTER 1: HOW OVERSIGHT IS NOW CONDUCTED
All Regions are now already overseeing State
construction grants programs. They are doing so under general
guidance from the Office of Water Program Operations (OWPO),
but each Region has developed its own specific oversight
methods.
This chapter briefly describes the specific approaches
used in each Region. For a more detailed survey, the reader
should refer to our separate background paper, "Current
Regional Oversight Methods."
THE REGIONS HOW USE A NUMBER OF APPROACHES
No two Regions use exactly the same methods of overseeing
construction grants programs in delegated States. This
apparent diversity, however, masks a number of common threads
which appear to run through most existing Regional oversight
programs.
The Basic OversightProceduresAre The Same
Each Region uses a slightly different combinations of a
limited set of basic approaches. Although specific implementation
varies somewhat from Region to Region, the similarities are
quite striking. Since the name applied to a given approach
varies from Region to Region, we have taken the liberty of
assigning a single name to each approach to minimize confusion.
We have grouped the Regions' oversight methods into two
categories — formal and informal methods. Table 2 contains
a list-of all these methods, which are described in more
detail below.
FORMAL OVERSIGHT METHODS
Formal oversight methods are those which the Regional
office and the State have agreed to as "standard, operating
procedures." Regions now carry out two such types of oversight
and focusing on a) specific activities, or b) individual
projects.
Activity—level Reviews
Most existing formal oversight methods now focus on the
State's management of specific "activities." Examples of
-------
-2-
Table 2: Summary of Oversight Procedures Now Used by ths
Regional Offices
FORMAL OVERSIGHT PROCEDURES
Activity - Level Reviews
1) Review of Current Transactions
o Cursory review of all incoming packages
o Selective in-depth review of incoming packages
2) Review of Recent Transactions: "Paperwork Sampling"
o Paperwork sampling at the Regional Office
o Paperwork sampling at the State Office
Project-level^ Reviews
1) Project-level review of transactions and procedures
2) Project-level performance review
o Cost review
o Timeliness review
Program-level Reviews
Review of the State's Internal Administration
1) Review of administrative procedures
2) Review of administrative performance — monthly
"report cards"
INFORMAL OVERSIGHT PROCEDURES
1) Day-to-day communications and feedback
2} Regular meetings with State program managers
-------
-3-
such activities include State reviews of MBE actions. Environ-
mental Assessments, and review of plans and specifications.
1) Review of current transactions; This consists of a
routine review of documents and document packages submitted
by the State for EPA approval. It can take one of two forms,
a cursory review or an in-depth review.
o Cursory review of all incoming packages. Virtually
every Region reviews a State checklist before accepting
the State's evaluation of material requiring EPA
approval. In at least one Region (Region I), this
"cursory review" is somewhat more elaborate, but is still
quite distinct from the "in-depth review" described
below. Region I examines not only the State checklist,
but also the State approval letter and and applicable
cost figures to ensure that they are reasonable.
o Selective in-depthreview of incomingpackages.
Some Regions(e.g.,RegionsI and II)conduct a
more detailed review, on a selective basis, of
specific activities whose results are contained
in State-submitted grant packages. In these
Regions, the 'frequency of in-depth reviews depends
on:
- the importance of the activity;
- how long it "has been delegated ;
the adequacy of State review (as based on the
quality of recent transactions); and
- the existence of any State requests for
assistance.
If a State's conduct of a specific activity is found to be
inadequate, one Region (Region I) responds with the following
corrective actions:
- preparing a sample review for State reference
(e.g., if the environmental assessment for a
project is inadequate, the Regional staff will
re-draft it properly so that State staff can use
it as a sample for future reference);
- going to the State office and conducting a review
side-by-side with the State staff member; or
- providing more formal training for the State
staff member.
-------
-4-
The Region will also monitor subsequent State actions in
this activity closely until quality improves, and stay in
contact with the State to help solve continuing problems.
2) Review of Recent Transactions — "PaperworkSampling";
"Paperwork sampling" refers to an in-depth review of
a sample of transactions received by the Regional
office in the previous 3 to 6 months. This paperwork
sample can take one of two distinct forms.
o Paperwork Sampling at the Regional Office
Most Regions conduct this paperwork review in the
Regional Office on a quarterly basis, relying
solely on material the State has submitted. The
Region selects a 10% sample of the State's actions
in each activity for a detailed check. In general,
the sample selected is not totally random. Most
Regions see to it that the sample a) includes some
small projects, but b) gives more than proportional
emphasis to projects which are large, environmentally
sensitive, unique or otherwise "difficult," or
believed to be "troubled."
The review is fairly mechanical and is concerned with:
- inspection of documents to make sure that the
State has filled them out correctly;
- review of forms and checklists to make sure the
State had updated them to incorporate recent
changes in regulations or procedures; and
- review of the document packages to ensure that
the State has submitted the proper support
doucments and checklists.
In some Regions, this review may take as much as two
months or more to complete.
o Paperwork Sampling at the State Office
Several Regions now sample paperwork at the State
office, for the most part on a semi-annual basis.
The activity is similar to sampling at the Regional
office, except that Regional staff go to the State
office to draw on the State files, and do not rely
on material sent to the region to conduct the review.
Two Regions (Regions I and X) often go to individual
project sites as well. Another difference is that
Regions normally complete reviews at the State Office
-------
within 2 to 5 days, rather that spreading them out
over weeks or months. Two Regions (Regions V and IX)
stagger their visits, looking at only a handful of
activities on each visit, but visiting the State
Office monthly so that all activities are reviewed
every six months.*
The paperwork sample closes with an exit interview with State
officials in which Regional staff discuss any problems they
have found. The Region then follows up with a formal report
to the State which focuses on major problems. One Region sends
this final report within two weeks of the review, while another
does not normally send it for two months.
Project-Level Reviews
Project-level reviews focus on the State's "project
management." "Project management" here refers to the State's
success in ensuring that each project as a whole is technically
adequate, environmentally sound, cost-effective and carried
out in a timely manner. There are two types of project-level
reviews: reviews of transactions and procedures,
and reviews of performance.
1) Project-level review of transactionsand procedures.
One Region(Region IV) hasjust begun to conduct such
reviews, and calls them "program management evaluations"
or "program oversight reviews." (We will use this term
later to cover a different and somewhat broader concept
of oversight.) In its first such review, conducted in
May 1980, Region IV selected a single project and had
specialized staff examine the State's handling of each
distinct activity in that project. The review took
one week, with Regional staff meeting each evening to
compare notes and identify coordination problems within
the State office. The Region then met on the last day
with key State staff to present the problems they had
discovered. The emphasis was on what the State could
do to improve its procedures so that "everything fits
together" on any given project.
* One Region(Region X)varies its level of review project-by-
project based on the Regional staff's judgment as to the
likelihood of problems.
Region X's levels of review are:
- checklist review - has the checklist been properly
followed and completed?
- content review - has the activity been performed properly?
- . in-depth review - EPA duplicates the State review to
determine whether it was conducted properly.
-------
-6-
2) Project-level performance review;
"Performance reviews" focus not on individual transactions
per se, but on whether individual grant projects as a whole
are proceeding satisfactorily. Only one Region now conducts
them,* though one other Region plans to adopt them in
the future. So far, two different types of "performance
reviews" have been conducted: a "cost review" and a
"progress review."
o Cost Review
One Region (Region VI) conducts an annual review
which focuses on identifying and correcting problems
by examining project costs. The Region selects 10%
of all on-going projects and compares costs to
the project budget. Unlike the paperwork sample,
this review covers a sample of all on-going, active
projects; the paperwork sample is limited to projects
on which State actions have been taken during the
previous 3 to 6 months. Another Region (Region IV)
now plans to conduct the same kind of review on an
annual basis.
o T ime1i ness Rev Jew
In conjunction with its planned cost review, one
Region (Region IV) also plans to conduct a time-
liness review (which it calls a "progress review")
to determine whether projects are proceeding on
schedule. Region IV will conduct both reviews on
the same sample of on-going projects.
Program-1eve1 Reviews
Several Regions (Regions IV, V, VII, and X) stated that
they wish to modify the nature of their semi-annual reviews
to emphasize "program management" rather than just "project
management." These Regions, however, are just beginning to
think through what a program management review might consist
of? they are not actually conducting such reviews at this
time.
*~"Ohe Region (Region IV) has taken an initial step in this
direction with the project-level review of transactions and
procedures described above.
-------
-7-
Reviewof the States InternalAdministration
Two Regions are now conducting such reviews: one Region
does so by examining the State's administrative procedures,
another does so by looking at the State's administrative
performance.
1. Review of administrative procedures
One Region (Region X) conducts such a review two weeks
prior to its paperwork sample at the State Office. This
earlier review (also conducted at the State Office)
examines the adequacy of the State's operations in certain
key administrative areas: staffing, training, grant
processing, and payments. The findings are reported along
with those from the Paperwork Sample.
Region II is proposing to conduct a similar review at
the State Office prior to a semi-annual Paperwork Sample.
Both Region X and Region II call the review of the
State's internal administration a "functional review."
2. Review of administrative performance —monthly "report card"
Several Regions have spoken of their desire to issue
monthly "report cards" on State performance. None,
however, has been specific about how it would go about
generating such reports. One Headquarters staff member
suggested that what the Regions probably have in mind
is a State report card similar to the current "Regional
report card", which evaluates several aspects of the
performance, including the Region's:
o progress to date in obligating funds
o time delays in reviewing current transactions, etc.
INFORMAL OVERSIGHT PROCEDURES
In addition to the formal procedures described above,
all Regions rely heavily on two highly informal methods for
overseeing State operations. Most Regions concede that these
informal procedures are their primary souce of information
about State operations and operational problems during the'
period (lasting two years or more) immediately after delegation.
Day-to-day communications and feedback
Most Regions are still in day-to-day telephone contact
with State staff to discuss the conduct of delegated activities.
These Regions consider such contact to be the primary means
by which they are alerted to problems in the State program.
In most cases, Regional staff attempt to correct such problems
by giving immediate feedback and guidance over the phone.
-------
-8-
Regular Meetings with StateProgram Managers
In many Regions, EPA's senior staff meet frequently
with the State Program Director and his key staff. The
purpose of these meetings is to bring the State program
managers up to date on new requirements and to provide a
forum for exchange of information. According to the Regions,
they are also an extremely important informal source of
information regarding difficulties the State is having in
implementing delegated activities.
Although the "EPA senior staff" who take part in these
meetings varies from Region to Region, it typically includes
one or more branch chiefs, the section chief responsible for
the State and the 205(g) coordinator. In many cases, these
meetings take place monthly. In other Regions, they take
place weekly or every two weeks.
In most cases face-to-face meetings are preferred,
although conference calls are often substituted in order to
conserve travel money. Meetings may take place either in the
State office or in the Regional office. Top Region "IX staff,
for example, meet monthly at the State office with the managers
of the California program. In Region V, the State program
managers meet monthly with the Regional staff at the Regional
office.
WHY ARE THERE DIFFERENCES AMONG REGIONS?
Although for the most part no single oversight procedure
is unique to any one Region,* there are significant differences
between Regions in the mix of procedures used and in the relative
emphasis given to those procedures. It has been suggested that
the differences might be due to three factors:
o the elapsed time since an activity was delegated;
o the adequacy of State staffing, and
o philosophical differences between Regions
Surprisingly, we found that philosophical differences
between Regions appeajr^to be a relatively minor factor in
determining how a Region approaches oversj.jght. Although such
differencesstrongly affect_how a Region describes its
intentions/when the^ jx:tual^conduct of oversight is examined,
the differences in practice turn out to be much less pronounced.
* The current exceptions are the use of "Project-level Reviews
of Transactions and Procedures" in Region IV, the use of
"Timeliness Reviews" in Region VI and the use of Review of
the State's Internal Administrative Procedures in Region X.
-------
-9-
Instead, the principal factor governing how a Region
conducts oversight appears to be the time elapsed since the
State began to accept delegated activities. Indeed, this
tendency is so pronounced that it is possible, based on the
testimony of Regional Staff, to discern a natural evolution
of oversight practices over time. In the first two years
after delegation, we have identified at least three distinct
"phases" of State development and Regional oversight. These
three phases will be described in considerable detail in the
next chapter.
-------
CHAPTER 2; THE EVOLUTION OF STATE PROGRAMS —
PHASES I, II, AND III
In examining the descriptions of State programs and of
oversight methods used by the Regions during the first two to
three years after delegation, we were able to identify three
distinct "phases" of development. Briefly, they are:
o Phase I — the initial phase — which normally
encompasses the first 12 to 24 months after delegation.
o Phase II — in which the focus is on improved
activity-level management by the State — and which
normally encompasses the next 12 to 24 months after the
completion of Phase I.
o Phase III — in which the State begins to undertake
and the Region begins to promote true project-level
management.
Later in Part II of the paper, we will discuss stages of
development beyond Phase III. No State or Regional office
has yet entered these later stages. We should note here, however,
that we do not regard Phase III as an endpoint for Regional/
State development, but as a milestone on the road to further
progress.
THE THREE INITIAL PHASES OF DEVELOPMENT OF A STATE PROGRAM
AFTER DELEGATION
We now present a composite picture of existing State-
Regional relations and of Regional oversight procedures during
the first three phases of State program development. This
composite picture is based primarily on the testimony of
selected Regional staff. The picture is most complete for
events in Phase I? Phases II and III are necessarily more
conjectural since fewer States and Regions have progressed to
this point.
Phase I — The Initial Phase — the First 12-24 Months After
Delegation
After EPA and a State conclude and sign a 205(g) delegation
agreement, Regional offices begin the process of disengaging
from the direct review of grant documents and begin to transfer
this responsibility to the State on an activity-by-activity
basis. States then begin to conduct these reviews themselves.
When the States completes its review of a particular document
-------
-11-
package, it submits the package to the Regional office for
approval. Accompanying the package is a "checklist" which
sets forth, in outline form, the requirements which the
documents had to meet and specifies whether or not these
requirements were in fact met. The packages themselves
normally include all required back-up documentation.
Immediately after the delegation of an activity, the
Region reviews each package carefully. Any problems result
in an immediate phone call to the State staff member who
prepared the package. During this period State staff also
frequently phone their Regional counterparts for advice and
guidance.
For those activities which a State consistently performs
well, the intensity of the Regional review gradually decreases.
After a few months with no problems, the responsible Regional
Staff member feels confident enough in the State staff member's
performance that a perfunctory review of the checklist is the
only review necessary. The intensity of review for -an activity,
increases, however, whenever there is a change in the State
staff performing that activity. In other words, in Phase I
the Regional staff doesn't trust the State's performance as
much as it trusts the performance of the individua1 member of
the staff.
No formal paperwork sampling is necessary at the be-
ginning of Phase I since effectively a 100% sample is subject
to in-depth review. Later in the year, or early in the
second year after delegation, the Region begins to conduct
formalized paperwork sampling in the Regional Office for
activities in which trust has developed. The paperwork
sample focuses on problem areas and problem projects as
revealed through daily telephone communications.
Typically, there are no formal reviews conducted by the
Regional Staff in the State Office before the end of this
first year. This is because the Regional and State staff
are in close communication and feel they are aware of each
other's problems and concerns.
By the end of the first year, calls from the State staff
for guidance have diminished considerably. Also, Regional
staff are relying solely on a checklist review of incoming
documents for most activities. State performance has reached
the point where the number of calls initiated by Regional
staff to inform State staff of problems has diminished markedly.
Most calls from the Region are either to inform the State of
changes in regulations or procedures, or to follow-up on
specific projects that are particularly troublesome. Never-
theless, the State is still playing "catch-up" throughout this
-------
I
-12-
phase. Most management attention in the State and in the
Region is devoted to the initial problems of implementing
delegation, rather than to ensuring smooth coordination of
non-crisis activities.
This phase continues until all delegable activities have
been assumed by the State and all are performed with "reasonable"
adequacy and consistency.
Phase II — Striving for Consistent, High Quality Activity
Management —the 12-24 Months After the Completion of Phase I
With the significant decrease in daily telephone contact
between State and Regional staff, formal review procedures take
on much greater importance. Formal paperwork samples are
conducted regularly; the Regional staff begins to conduct formal
reviews at the State office on a regular schedule (every six
months). Although telephone conversations with State staff
diminish, they are still the principal source of information
about problem areas.
At this time, the State begins to take charge of its
managerial responsibilities and takes steps to ensure consis-
tently high quality performance of each activity it reviews
(i.e., MBE activities, change orders, plans and specifications
preparation, etc.). By the end of Phase II, this objective
has been achieved: allState activity-level reviews are of
uniformly high quality,especially those of basic administrative
and engineering activities (see Table2 at the end of Chapter 5)•
areof uniformly high quality. This does not mean that they
are error-free, but that they have achieved a quality and
consistency more or less equivalent to that achieved prior
to delegation by EPA's Regional Offices.
At the end of Phase II, the quality of State activity
reviews should be so high that the Regional Office can review
a 5% sample of paperwork in place of the 10% sample originally
used, without jeopardizing adequate quality control of the
State's operations.
Phase III — Striving for Effective Project Management — the
18-24 Months after the Completion of Phase II."
Phase III begins when the Region's confidence in the State's
performance of all activities, especially basic administrative
and engineering activities, reaches the point that the Region can
rely on a 5% paperwork sample. The Region then begins to focus
on the State's ability to review and assess projects as a whole
-------
rather than just as a collection of activities. Meanwhile,
the State is also beginning to focus on improving coordination
among separate activities. The State thus starts to draw
independent, uncoordinated "activity reviews" into more
coordinated "project reviews." Yet, while the Region begins
to review the effectiveness of the State's overall project
management, it still continues, on a less intensive basis,
to review the quality of the State's conduct of specific
activities.
By the end of this period, the Regional staff has con-
siderable confidence in the State's ability to carry out
its delegated responsibilities at two levels: project review
as well as activity review. There is now trust in the
competence and effectiveness of the State's operation as a
whole as well as in that of specific individuals on the
State staff.
FURTHER PHASES OF DEVELOPMENT
Our descriptions of Phases I, II and III, complete those
phases of development which we have actually observed to date, in
however rudimentary a fashion. We believe, as noted above, that
the process of delegation is still in an early stage, and that
States and Regional offices should pass through three more phases
as the process continues. Part II of this paper (Chapters
4-7) describes what these further phases might look like.
We have discerned this general pattern of development of
Region/State relations despite the current differences in
oversight procedures within and between Regions. One factor,
however, can frequently prevent this normal pattern from
being followed. This factor is inadequate numbers of State
staff. Inadequate staffing can keep even a very strong,
capable state like New York from conforming to the rate of
development we would expect to see based on our observation
of the experiences of other States. One Region {Region VIII)
in particular expects to continue operating for an extended
period in Phases I and II in most of its States because of
the great likelihood of continued inadequate State staffing
levels. The issue of adequate State staffing clearly requires
resolution very early in the delegation process. It is the
sine qua non of further progress by both the State and the
Region; without it, neither States nor Regions can expect to
make much headway.
-------
-14-
CHAPTER 3: ANALYSIS OF ALTERNATIVE OVERSIGHT PROCEDURES FOR
PHASES I AND II
To be useful, any generalized oversight model must reflect
both the varying stages of development of Region/State relations
and the prospect of continued inadequate staffing levels in many
States. We therefore propose that EPA develop guidance which
formalizes the lessons learned in the last few years in dealing
with such conditions. In particular, we propose that oversight
procedures be tailored to the phases of development described
in the previous chapter.
In this chapter we will examine the situation of States
with less fully developed programs {i.e., States in Phase I,
Phase II or in the early part of Phase III) and propose
standard oversight procedures for use in such States. Since
most Regions now have had at least some experience in .over-
seeing the activities of States in Phases I and II, we have
drawn heavily on the experience and the insights of staff in
the Regional offices in proposing oversight procedures for
these two phases.
PHASE I -~ HELPING THE STATE GET UP TO SPEED
Regional staff paint a consistent picture of how State
staff react to delegation. While prior to delegation the State
staff had anticipated having more control over the program in
their State and had confidence in their ability to take over
the activities previously carried out by the Regional Office,
on the day of delegation some of the State's working staff may
suddenly be overwhelmed.
Some State staff may be overwhelmed at first by what they
perceive as the great personal responsibility they now bear
for managing their portion of a multi-billion dollar program.
Some may also be overwhelmed by the complexity of the projects
and the intricacy of the review requirements with which they
must deal. They may be overwhelmed as well by the variety of
unexpected problems and situations which accompany delegation.
Faced with this sudden burden, they know they need assistance
from "those who have been there before" and they readily call
on the Regional staff for advice. They normally find that
Regional staff are quite responsive to requests for advice
or assistance.
At the same time, delegation tends to be a wrenching
experience for Regional staff as well. Where they had
previously borne considerable personal responsibility for a
major Federal program and had extensive personal involvement
with the "nuts and bolts" of that program, now many feel as
their role changes though they have suddenly become mere
-------
-15-
"observers". As a result, key Regional staff often seek
other jobs. The sudden loss of these staff represents the
loss of valuable experience and insight that could be of.
great benefit to the State staff now responsible for carrying
out the program, especially during the first 12-24 months
after delegation.
Informal Contacts Are Important
Given these circumstances, it seems appropriate to soften
the impact of delegation both on the State staff and on the
Regional staff by encouraging free interaction between them.
Regional managers should especially encourage the natural tendency
toward steady — normally daily — interaction in the form of
frequent telephone calls in both directions. State staff should
want Regional staff to phone at once when the Regions becomes
aware of specific problems in the way the State is carrying out
a certain activity. Indeed, a common complaint of the States
is that they don't want the Region to wait for a formal quarterly
or semiannual review to tell them about deficiencies in their
operations. On the contrary, if there's a problem, the State
staff wants to know about it i mmed iate1y, so they can take
corrective action at once.
Similarly, close communication between the State and the
Region at the supervisory level is also desirable during the
first phase of delegation. Most Regions have stated that
they have found it useful for the Region's program managers
(at the branch and section chief level) to meet with the
State's program managers at least monthly throughout this
period, and preferably every two weeks during the first six
to twelve months. The purpose of these meetings is to discuss
and begin to resolve current problems, as well as to allow
the State program managers to draw on Regional experience in
dealing with unusual or especially difficult-projects. Since
travel funds are always limited, some Regions have substituted
conference calls for many of these meetings. Still, many
Regions have found that frequent face-to-face meetings are
highly desirable and should be held at least once every
month or two.
Formal Contacts Should Be Regular Too
The same principles hold for formal as well as informal
aspects of Region/State interaction.
Review of Current Transactions; Immediately after an
activity has been delegated, the Region staff should conduct
an in-depth review {essentially a duplicative review) of
State document packages related to that activity that are
submitted for Regional approval. These in-depth reviews can
serve as "models" for the State, and should continue until
the State staff is performing that activity properly. Any
problems discovered should be brought to the attention of the
responsible State staff member at once by telephone.
-------
-16-
Initial in-depth reviews and rapid telephone feedback
should result in early correction of most deficiencies in
State staff performance. Once initial problems are cleared
up, the intensity of Regional review should (and normally
does) ease off considerably. (This has apparently been the
case in Regions I and II.) As it becomes clear that the
State's performance of a certain activity is consistently of
good quality, the Region should reduce its review of current
transactions to inspection of the attached checklist only.
Paperwork Sampling; During the first quarter or two,
no formal paperwork sampling may be necessary since the
Regional staff is still carefully scrutinizing most State
packages coming to the Region. Later, as State performance
improves and the Regional staff begins to rely more on check-
list reviews, formal paperwork sampling should begin.
There are essentially two approaches now in use among
the Regions for conducting paperwork sampling:
o Quarterly reviews within the Regional Office of a
sample of packages received the previous quarter.
o Semi-annual reviews within the State Office of
State file copies of packages sent to the Region
during the previous six months.
Although each approach has its advantages, we recommend
that Regions conductthese reviews quarterly within their own
offices. We make this recommendation for the following reasons
o Because of limited travel funds, the cost of
conducting extended reviews in the State office for
all States in a Region, even on a semi-annual
basis, appears to be prohibitive for most Regions.
o Since this is simply a document review, there is no
need for access to State staff. It therefore seems
more reasonable to ship the documents to the Regional
staff than to "ship" the Regional staff to the
documents.
o Several Regions have described the paperwork sample
as a rather "mechanical" kind of review. If this
review occurs in the Regional Office, the semi-annual
visits to the State Office can be devoted to other
kinds of review that require access either to State
files or to State staff.
Semi-Annual Visits to the State Office; There has been
a tendency for Regions to dispense with formal semi-annual
reviews or visits to the State Office during the first year of
-------
-17-
delegation on the grounds that there is already sufficient
review of State activities and sufficient Region/State
communication. It does, however, seem desirable to establish
the precedent of formal, scheduled semi-annual "reviews"
which can increase in content over time. During the first
year of delegation, these visits could well be devoted simply
to a formalized general review of problems encountered over
the previous six months and of the adequacy of the State's
response. This is in contrast to the more informal bi-weekly
or monthly meetings which are, for the most part, devoted
only to the problems-of-the-moment.
PHASE II — TRANSITION TO LESS FREQUENT, MORE FORMALIZED OVERSIGHT
By the end of Phase I, the Region is relying almost
solely on a checklist review before signing off on incoming
documents. Considerable mutual trust and confidence should
have developed between State and Regional staff. The focus
of both the Region and the State is on helping to attain
consistently high quality State review of all delegated
activities, especially basic administrative and engineering
activities.
Formal Reviews Become More Important in Phase II
Phase II is a period of less frequent communications
between State and Regional staff. This calls for greater
Regional reliance on formalized oversight mechanisms. The
quarterly paperwork sample is initially the principal review
mechanism, but throughout Phase II, the semi-annual review at
the State office increases in importance until, by the end of
Phase II, it overshadows the Paperwork Sample in importance.
This shift occurs because the quality and consistency
of State "activity-level reviews" continues to improve
throughout Phase II. Early in the Phase, Regional paperwork
sampling is still discovering many deficiencies in State
reviews. By the end of Phase II, the Region should find so
few deficiencies that it is able to reduce its sample of
paperwork from 10% to 5%,
Regions Can Introduce "Performance Reviews" Here
At the end of Phase II, the Region might add to the
semi-annual visit a "timeliness review" of current projects
as described in Chapter 1. By quantifying one aspect of
State performance (timeliness of projects), the Region can
introduce the notion of "direct measurement of performance"
to its oversight methods. It will thereby begin to augment
its "oversight through review of transactions and through
review of procedures," with "direct measurement of performance."
-------
-18-
PHASE III — FOCUS ON IMPROVED PROJECT MANAGEMENT
Phase III formally begins when the Region acknowledges
that the quality of State performance is so high and so
consistent that the Region requires only a 5% Paperwork
Sample for basic quality control.
The Region and State both begin to focus on the State's
ability to go beyond compartmentalized activity-level reviews
to conduct coordinated project-level reviews. In a true
project-level review, each project is viewed not as a collection
of disjointed administrative and engineering activities, but
as a unique, multi-faceted whole.
Since only a few States have entered Phase III, we
cannot readily draw on State and Regional experience to date
in analyzing possible oversight procedures for this Phase.
Instead, we cover this task in Part II of this paper, which
attempts to determine the desirable future course of the
delegation process and to develop oversight procedures
appropriate for use in those circumstances. For convenience,
the recommended approaches for conducting oversight in Phase
I and Phase II, developed in this chapter, are summarized in
Chapter 9 along with the approaches we recommend for later
stages of development.
-------
PART II: ADVANCING THE STATE OF THE ART
-------
-19-
Chapter4; Shifting fromOversight of Transactions
and Procedures to Oversight ofPerformance
In this chapter we will focus on certain trends that
are apparent in the way Regions are modifying the manner in
which they oversee State operations.
This chapter (Chapter 4) will be devoted specifically
to an analysis of the k ind of information the Regions use
to conduct such oversight. In other words, do the Regions
evaluate State operations based on the review of individual
transactions, based on the review of the State's admini-
strative procedures or based on direct measures of the State's
performance?
In the next chapter (Chapter 5) we will look at the
"scope" of the State's operations and of the Region's over-
sight activities. By scope, we mean the level of perspective
of both the State and Region. Do they still focus only on
the quality of individual activities, or have they begun to
focus at a higher level, i.e., on each project as a whole,
on the municipal construction program as a whole or on the
larger water quality program as a whole in order to evaluate
the State's performance?
Then in Chapter 6, we examine whether direct measures of
performance will continue to be available as the management
scope moves from the activity level to project level and
beyond.
We will now examine the kinds of information that can
be used to oversee State operations that are still somewhat
narrow in scope, i.e., State operations that still focus for
the most part on "activity management." We will argue that
Regions should strive for oversight based on direct measure-
ment of performance in preference to oversight of transactions
or of State procedures.
OVERSEEING STATE OPERATIONS AT THE ACTIVITY LEVEL
In conducting such oversight, the Regions have until now
tended to engage in reviews of:
o State transactions. That is, the Regions have
examined the quality of the final documents gene-
rated after State review (e.g., How good is the
environmental assessment? Are the plans arid
specifications biddable and constructible?)
-------
-20-
o State procedures. That is, the Regions have attempted
to determine the adequacy of the approach the State
is taking in carrying out that activity (e.g., does it
follow the checklist?, are the necessary documents
filed?, is all the required "boilerplate" included?,
does the State follow the proper bidding procedures?).
Only to a very limited extent have- some Regions begun to
develop direct measures of:
o State performance. That is, the Regions have attempted
to develop and use quantitative measures of the effective-
ness of the State's operations (e.g., Are projects being
conducted within budget? Are projects moving forward
on a timely basis?).
The Current Shift is towards Review of Procedures and Direct
Measurement;of Performance'!
There is already a clear tendency for the Regions to
shift substantially from review of individual State, trans-
actions or documents to review of State procedures. This
trend was noted earlier in Chapters 1 and 2, At the begin-
ning of Phase I of delegation, the Region looks closely at
virtually every individual transaction requiring Regional
signoff. The Regional staff will then react to problems in
individual transactions. But throughout Phase I and Phase
II, the Region begins to focus more and more on the proce-
dures the State staff follows in carrying on each activity.
As the Regional staff become convinced of the soundness of
the State's procedures as reflected in the quality of
individual transactions, their attention shifts primarily
to the consistency with which the procedures are followed.
As a consequence, while the Regional staff conducted an
in-depth review of virtually 100% of current transactions at
the beginning of Phase I; by the beginning of Phase II, a
mere checklist review of each current transaction is con-
sidered adequate. At the beginning of Phase III, the Regional
staff, feel comfortable giving only 5% of these transactions
a closer look.
This tendency to shift substantially from review of
individual transactions to review of general procedures is
desirable since it requires less Regional staff time.
Furthermore, it does not increase the risk of a deterioration
in State performance because even a 5% paperwork sample of
recent transactions gives the Region enough quality control
information to detect any areas of deteriorating quality.
-------
-21-
At the same time that the Regions have been shifting
from reviews of transactions to reviews of procedures, a
second shift has begun to occur as well. One or two Regions
have begun to experiment on a very limited basis with the
use of direct measures of performance for overseeing State
operations. For example, the "cost review" carried out by
Region VI seeks to determine the adequacy of the State's
cost related activities not by looking at the apparent
adequacy of individual transactions or of general procedures»
but by looking at the "bottom line." Region VI looks to see
how the actual project cost compares with the original project
budget. If a project is going significantly over budget,
this is clear evidence that State procedures or handling of
individual transactions for cost-related activities are
inadequate. Indeed, such actual cost performance information
is the best and most meaningful evidence possible of how well
the State is carrying out its financial responsibilities.
While the quality of individual transactions or of general
procedures gives an indication of the probable quality of
the State's performance, direct measurement of that perfor-
mance tells the Region about the actual quality of that
performance.
DIRECT MEASURES OF PERFORMANCE AND HOW TO USE THEM
In many cases, reviews of State transactions (i.e., the
quality of the documents generated) or of State procedures
are virtually the only tools available for assessing State
performance. In other cases, however, more direct, often
quantitative measures of State performance are available.
Here are two examples:
Change Orders;
o The Region could detect most unwarranted use of
change orders by monitoring the percent increase
in project cost due to change orders.
Minority Business Enterprise:
o The Region could determine the ultimate effectiveness
of the State's MBE-related activities by monitoring
the fraction of project dollars awarded to minority
business enterprises.
For activities for which direct measures of performance
can be developed, it is possible to make use of a streamlined
approach to oversight. Briefly, the streamlined approach
consists of monitoring the State's conduct of that activity
by gathering the direct performance data (e.g., the actual
cost increases due to change orders). If that performance
data indicated that there is no problem, then no further
-------
-22.-
review is necessary (other than a review of 5% of recent
change orders in the Paperwork Sample to ensure adequate
quality control). If, however, the performance data indi-
cates that there is a problem, then there should be follow-up
assessment of the State's procedures for that activity (e.g.,
of the State's change order review procedures).
Since performance data is quantitative, the adequacy
or inadequacy of the State's performance must be assessed
initially against pre-specified expectations or "performance
targets." These performance targets should be agreed on
jointly by the State and the Regional Office and should
reflect realistic expectations as to what is "do-able" in
that State. The MBE performance targets, for example,
might be set much lower in New Hampshire, where there is a
very small minority population, than in California, where the
minority population is much larger.
Here, step-by-step, is how an oversight system based on
direct measurement of performance might work:
o Agree on Activities to be Subject to Direct Monitoring
of Performance
- First, the State and Region agree which State
activities will be subject to direct measures of
performance during the coming year. They also
agree on the specific measures of performance to
be used.
o Agree on the Appropriate Performance Targets
- The State and Region then negotiate performance
targets for each of these activities for the coming
year.
- Such targets can be revised annually based in part
on experience gained in previous years as to what
target levels are feasible and appropriate.
o Oversee these activities on the basis of measured
performance
- During the year, the Region samples projects
underway to determine actual performance.
If actual performance falls within the agreed-
upon norms, then the State's performance is
satisfactory and no further review is necessary.
-------
-23-
If actual performance falls outside the agreed-
upon norms, then the Region should examine the
State's review procedures to determine the probable
reasons for the deficiencies and should negotiate
changes in those procedures believed likely to
improve performance.
There isGreat Opportunity for Regionaland State Initiative
in Developing and Testing Additional Direct Measures Q.f
Performance
While it is clear how performance norms can be applied
to such factors as degree of MBE participation and extent of
cost overruns due to change orders, innovative Regions might
also seek ways to specify performance for other aspects of
the program. For example, some Regions and States may wish
to experiment with the use of additional performance measures
as such as:
o Reasonableness of Contract Cost.
For each class of facility to be built using
off-the-shelf technology, the State and Region
might agree on normal cost ranges and average
cost.
The Region would then periodically review contracts
awarded to determine:
- what percentage of each class of contract
falls outside the specified range,
the extent of deviation of the actual average
cost from the expected average cost.
If the performance measured in this way appears
satisfactory, there will be no further review.
If the actual performance seems to reflect un-
expectedly high costs, the Region would conduct
an in-depth review of the State's cost estimation,
bidding and bid selection procedures to determine
whether these procedures are contributing to the
the unexpectedly high costs.
Thus, the Region would again be applying the principle
that if the State's measured performance meets agreed upon
targets, then no detailed review of procedures for that
activity is necessary.
Many other applications of this approach are possible,
and the Regions and States should be encouraged to develop
and use appropriate performance measures in place of con-
ventional transactions or procedural reviews for as many
aspects of the program as possible.
-------
-24-
It should be emphasized that performance measures are
suitable not only for monitoring State performance at the
"activity-review" level but also at the "project-review"
level and potentially at the "program management" level as
well. Possible measures of performance at these levels are
presented in Chapter 6 and Appendix B.
THE POTENTIAL BENEFITS TO BE DERIVED FROM USE OF DIRECT
MEASURES OF PERFORMANCE
For the two specific activities cited earlier as lending
themselves to direct measures of performance, the following
impact can be seen.
MBE. Monitoring the percent of contract dollars awarded to
MBE firms would substitute for detailed reviews of the MBE
aspects of bid solicitation and contract awarding procedures.
Change orders. Monitoring cost overruns due to change orders
as a percentage of total project dollars would substitute for
detailed scrutiny of the State's change order review procedures
(procedure review) not to mention detailed Regional review of
each proposed change .order (transaction review.)
Such changes in the manner in which Regional oversight
is conducted would bring the following benefits:
o Reduction in RegionalReviewTime. Regional over-
sight based on direct, monitoring of performance
would in virtually every case be less time consuming
th2Ji Regional review of individual transactions and
would, in many cases, be less time consuming than
Regional review of State procedures.
o Increase in State Flexibility. Use of direct measures
of performance would potentially allow States to modify
their review procedures as they see fit as long as the
agreed-upon performance targets are met. Thus, it
would allow for the gradual evolution of the program
from one of "uniform" State programs that are required
to conform to pre-determined procedures to one of
."equivalent" State programs in which State procedures
are allowed to vary as long as the results meet reason-
able expectations.
o More Meaningful Indicators of Impact. States are now
required to perform certain specified procedures not
for the sake of the procedure itself or of procedural
uniformity, but because EPA believes that use of those
procedures will tend to produce a certain outcome, for
example:
-------
-25-
- MBE; a reasonable level of MBE participation in
current projects
- Change Orders; prevention of abuse of change
orders by contractors to inflate project costs or
to reduce project quality.
EPA can best ensure that these outcomes are being
achieved by measuring the outcomes themselves (i.e.,
amount of MBE participation, degree of overrun due
to change orders) rather than by verifying that
specified procedures have been followed. In many
cases, in fact, it may turn out that a State may
follow the specified MBE or change order proce-
dures and still have inadequate MBE participation
or excessive change-order related cost increases.
The monitoring of procedures entails monitoring
something one step away from EPA's real concern;
the monitoring of performance entails monitoring
the thing of concern itself.
As we have demonstrated, the impact on State/Regional
relations of adopting and using direct measures of performance
will be substantial. Indeed, the implications are so great
and so far-reaching that it can reasonably be said that a
Region that relies primarily on direct measures of performance
has shifted from the role of "supervisor" to "franchiser" or
"indirect manager."
In this chapter, we have endorsed the current shift in
the Regions from oversight based on review of individual
transactions to oversight based on review of State procedures.
We have also taken note of some initial efforts by the Regions
to develop and use direct measures of performance for over-
seeing State operations. We have given strong encouragement
to these efforts and have urged that additional direct measures
of performance be adopted as the primary tools for overseeing
State operations at the activity level, thereby facilitating
the shift of the Regional Office from the role of a "supervisor"
to that of a "franchiser."
-------
-26-
Chapter 5: Increasing the"Scope" of Regional Oversight _--
from Activity Management to Project, and
• Program Management
THE MANAGEMENT SCOPE IN A STATE PROGRAM
In the earlier descriptions, in Chapter 2, of the phases
through Which a State program develops, three distinct kinds
of changes over time can be discerned. The most obvious
changes are improvements in quality. At the end of Phase II,
for example, a State is simply doing a better job of reviewing
grant applications, plans and specifications, etc. than it
was at the beginning of that Phase.
A second change, highlighted in the preceding chapter, is
the change in the kind of information used to oversee State
programs. During the course of Phase I and II, there is a
pronounced shift from oversight based on the review of
individual transactions to oversight based on the review of
procedures. We are also beginning to see a second, even-more-
desirable shift to the use of direct measures of performance
as a major oversight tool.
At the same time a third and more subtle kind of change
is occurring. In Phases I and II, the State has its hands
full trying to ensure that it is adequately carrying out
specific reviews at the activity-level (e.g., reviews of
environmental assessments, reviews of MBE provisions, reviews
of plans and specifications). When a State enters Phase III,
however, it is relatively confident of its ability to conduct
reviews at the activity-level, and can begin to increase the
"scope" of its reviews by focusing on conducting adequately
coordinated reviews at the project-level. We expect to see
further shifts in time from project-level management to
program-level management and finally to integrated water
quality management.
Anticipated Shifts in Management Scope
When a State enters Phase III, it begins a pronounced
shift from mere activity management to true project manage-
ment. In Phases I and II, the State tended to view each
project as a disjointed set of activities to be reviewed.
In Phase III, it begins to look beyond those activities
and to see the project as a whole. While in Phase III the
State must necessarily continue to conduct activity-level
reviews, it now strives hard to focus on each project as a
single, though complex, entity. And in doing so, it begins
to ask broader questions:
-------
-27-
o is the project as a whole technically sound?
o is the project as a whole environmentally beneficial?
o is it being built in a timely manner?
o is it being built at a reasonable cost?
There are several ways a State might go about improving
its effectiveness in performing "project-level reviews."
These are discussed more fully in the next section and in
Appendix A.
Once a State has succeeded in achieving true project-
level review, it might be said to be leaving Phase III and
entering a new Phase — "Phase IV."
In Phase IV we might expect to see a similar change in
the level of review. By the beginning of Phase IV, a State
should be reasonably proficient in conducting effective
project-level reviews. It can therefore begin to focus on
better coordination among projects in order to improve its
overall management of the construction grants program.
Improved municipal wastewater program management might, for
example, include more thorough consideration of potential
water quality benefits when ranking municipal wastewater
projects on the priority list. It might also include
efforts to identify communities with significant water
quality problems where facilities are greatly needed but
that have not yet applied for grants. The State could then
work with such communities and assist them, if necessary, in
preparing grant applications. By "beating the bushes" in
this way, the State can help ensure that enough projects are
proposed that it can pick those with the most merit to go on
the priority list.* It is now often the case that a State,
in seeking to avoid re-allotment of its grant funds, will
include projects of less merit on its priority list simply
because they are the only additional ones available that
are "ready to go."
Similarly, we might expect some States, after having
achieved effective management of the construction grants
program as described above, to advance to a new "Phase V" in
which it begins to focus more on "integrated water quality
management." At this point, the State program would closely
coordinate its municipal construction efforts in specific
communities with other pollution control efforts there. At
* As noted below, this strategy is desirable even after a
"date certain" several years in the future has been set
for terminating the award of additional construction grants
-------
this point, strong efforts might, for instance, be made to
ensure that the scheduling of construction grants projects
in each locality is fully consistent with the scheduling of
industrial source and non-point source control efforts there.
Furthermore, the State might begin to take major steps towards
careful coordination with any groundwater related activities
planned for that area. At this point a State program will
have achieved the ultimate in meaningful, effective program
management.
Having now described briefly the four distinct levels at
which a State program can operate, we will now explore in more
detail what is involved in conducting each level of review.
We will begin our discussion with Phase III and project-level
reviews.
THE PHASE III PROBLEM — PROMOTING PRQJECT-LEVEL MANAGEMENT
In Phase III we are dealing with a State program that
has begun to demonstrate its proficiency in carrying out
the mechanics of performing each delegated activity.
Although the Region should continue to monitor the State
performance in reviewing each activity in order to ensure
that the quality remains consistently high, the Region does
not, at this point, really expect many serious problems to
be discovered.
The Region can therefore begin to focus on the quality
of the State program's operations as viewed from a broader
perspective — project-level management. Specifically, the
Region can begin to ask: how well is the State doing in
ensuring that each project as a whole is technically sound,
environmentally beneficial and is built in a timely manner
at reasonable cost?
One criticism of current activity-oriented reviews heard
often in discussions with Regional staff is that the States
are called upon to conduct too many separate reviews, each of
which is conducted independently without a clear understanding
of the role it plays in promoting the ultimate goal of sound,
efficient wastewater treatment. It was with this concern in
mind that Region IV developed the concept for the Project-Level
Management Review it recently conducted in Georgia.
Region IV 's procedure was to conduct an intensive review
of the whole range of State activities for a single project in
order to highlight how each activity contributed or failed to
contribute to the success of the project as a whole. The
review was designed also to uncover gaps or deficiencies in
-------
-29-
the State's review process that allowed problems to arise
that detracted from the project's quality or timeliness or
that added to its cost.
The intended result of such a review by the Region is
improved coordination between different aspects of a State's
activities and the elimination of gaps or deficiencies among
such activities so that the State will be better able
to assess a project as a whole.
Ideally, the State should propose specific changes in its
review procedures or internal coordination to minimize the
likelihood that such deficiencies will recur. In some cases,
the changes might alter the manner in which the State, the
grantees, or the consultants carry out their responsibilities.
For example, it might mean taking steps to ensure better
coordination between two or more separate staffs in the State
Office. In other cases, it might be determined that there
should be a consolidation of several staff groups to ensure
greater integration of closely related activities. In pro-
moting effective project-level management, the State and
Region should probably focus on the integration of certain
key activities in Step 1 that have a strong impact on basic
project design. These key activities are the "analyses of
alternatives" associated with the alternative/innovative
design review, the environmental assessment activities,
initial financial planning efforts conducted in the early
stages of the the facility planning process and any other
analysis of alternatives in the facility plan.*
Once the available options have been fully explored,
the State and the Region should reach agreement on the
corrective actions to be taken. This will often necessitate
changes in the corresponding checklists. In its next program
review (6 to 12 months later), the Region would then look to
see how effective these changes have been in eliminating the
deficiencies seen earlier. If necessary, further changes in
State procedures should be negotiated in this and other
subsequent reviews.
The kinds of corrective action that a State will normally
propose to improve coordination among closely related activities
are the following:
Indeed, there is a growing consensus both in the Regions
and at headquarters that such "up-front" planning activities
in Step 1 are the most crucial single element in ensuring
that each municipal project is both technically and
environmentally sound and is cost effective.
-------
-30-
o Require multiple, concurrent reviews of certain
documents "by different staffs with related functions.
o Institute "project review" meetings, with partici-
pation by each staff with distinct responsibilities.
o Create new "integrated review groups."
The relative advantages and disadvantages of these three
approaches for improving the degree of project-level review
in a State program are discussed at some length in Appendix A.
As noted above, there is a growing consensus that the
facility planning activities are the key to ensuring techni-
cally and environmentally sound, cost-effective projects.
Those activities leading to a sound ultimate decision as to
what kind of facility to build are therefore the "Tcey
activities" for ensuring good project-1eye1 management. The
key activities are, of course, the activities that help
identify alternative approaches for controlling municipal
wastewater and help define the advantages and disadvantages
of each alternative. The key activities for project-level
management and for each subsequent higher level or "scope"
of management are summarized in Table 3 at the end of this
Chapter and are discussed again briefly in Chapter 7.
THE PHASE IV PROBLEM — PROMOTING IMPROVED PROGRAM MANAGEMENT
Since in Phase III the Region and State will be empha-
sizing improved project-level review, by the end of Phase III,
the State will have developed a considerable degree of
proficiency in ensuring the soundness of each project as a
whole. Therefore, in Phase IV, the Region and State can then
begin to focus on improvement of the State's management of the
municipal wastewater program as a whole.
As was noted earlier, improved program management would
result in improved priority-setting and coordination among
projects. Specifically, the Region might, in Phase IV, begin
to give considerable attention to the State's performance
with regard to the relative priority assigned to municipal
wastewater projects in light of anticipated water quality
benefits.
The Region might also examine what steps the State is
taking to determine what additional communities are in need
of improved treatment facilities and to assist them in pre-
paring an appropriate grant application. As was noted earlier,
such action on the part of the State would increase the
number of potential projects with significant water quality
benefits. It would thereby reduce the need of the State to
-------
-31-
include projects of relatively less merit on its priority
list in a given year simply because they are "ready-to-go."*
Activities such as these would consitute the "key
activities" for ensuring adequate program management. By
focusing on the State's efforts to upgrade its performance
in such areas, the Region can help promote State program
management that is focused on maximizing the water quality
benefits derived from the State's construction grants program
as a whole.
PHASE V — PROMOTING INTEGRATED WATER QUALITY MANAGEMENT
Many Regions may be satisfied with a State program
that has met the Phase IV goal of having established a
balanced, water-quality-oriented construction grants program.
And they would have good reason to be satisfied since it is
likely that because of staffing limitations or for other
reasons, some States may never reach this goal.
Nevertheless, we should anticipate that some Regions and
States may wish to go one step further, to enter a '"Phase V"
in which the objective would be to integrate construction
grants activities fully into the State's broader program of
water quality management.' At this point, relative project
priorities would be set not just on the basis of the relative
merits of the projects themselves but on the basis of how
they fit into the larger context of water quality management
efforts as a whole. The State program's scheduling of
projects and guidance on facility planning would now fully
consider the nature and extent of other activities underway
or projected, including industrial and non-point source control
efforts and groundwater control needs. This would mean the
full incorporation of municipal facility planning and
construction into a broader framework of meaningful, ongoing
areawide planning and State-wide planning.** It would reflect
the full maturation of the construction grants program as
a key element in balanced environmental planning at the State
and National level.
* Even if a date certain is adopted for ending the awarding
of Federal construction grants, encouragement of additional
Step 1 grants where really needed is still desirable. This
will help ensure that the limited funds remaining to be
awarded can and will be spent where they will do the most
good in terms of improving water quality.
** Clearly the "key activities" for achieving water quality
management are areawide planning activities at the local
and State level. (See Table 3.)
-------
-32-
PHASE VI — A FULLY-DEVELOPED STATE PROGRAM
In the previous phase — Phase V — a State would begin
to strive for incorporation of its municipal wastewater
program into a integrated water quality management. Once that
goal is achieved, we can consider the State to have entered
the final phase of development. States that reach this
final phase — Phase VI — will have achieved the ultimate
in balanced, fully water quality oriented program management.
At this point, the State and Region should jointly work for
greater efficiency in State operations and in Regional over-
sight procedures as long this does not jeopardize the quality
or scope of the State's operations.
We believe that the four increasing levels of "management
scope" described in this chapter provide a useful framework
within which to assess State programs and help promote their
further development in order to ensure the most effective
possible management of the municipal wastewater program.
-------
Proposed Classification of State Activities
BASIC ENGINEERING AND ADMINISTRATIVE ACTIVITIES
o Review of Change Orders
o Review of MBE Activities
o Review of User Charge and Industrial Cost Recovery
o Review of Plans and Specifications
o etc. — includes all other State activities not
included in the list of "Key Activities" below
"KEY ACTIVITIES" (See Chapter 5 for discussion)
1 . Key activities for effective project.-level management
o Review the grantee's "analysis of alternatives"
- Identification and technical analysis of possible
alternative and innovative designs
Identification and technical analysis of other
design alternatives in the facility plan
Assessment of the environmental impacts of each
viable design alternative
- Identification of the costs and possible financing
arrangements for each viable alternative
Key activities for effective program-level management
o Needs - related and priority - setting activities:
- Complete Statewide assessment of municipal treatment
needs
- Incentives for communities with significant needs
to apply for Step 1 Grants
- Assistance to communities with significant needs
in preparing Step 1 applications
Priority list preparation with priorities based on
(true) relative need in terms of water quality
-------
Table 3: (cont'd)
3 . Key activities for effective integrated water quality
management
o Meaningful, comprehensive areawide water quality
planning:
at the local level
at the state level
o Joint scheduling of all water quality improvement
efforts in each locality including:
municipal control efforts
- industrial point-source control efforts
non-point source control efforts
groundwater protection efforts
-------
-33-
Chapter J5j_j4onitor ing "Performance" at the Project
and Program Level
In Chapter 4 we reached the following conclusion about the
type of information that should be used in overseeing State
operations at the activity-level: whenever it is possible
to do so, direct monitoring of performance is preferrable to
reviewing procedures or reviewing individual transactions.
Then, in Chapter 5, we concluded that the Regions and
States should continue to work for an ever broader management
scope in State operations. The States, we concluded, should
move from an activity-level focus to a project-level focus, and
then to a program-level focus.
The question we will now address briefly is whether it
is possible to continue making use of direct measures of
performance as we move from activity-level oversight to
project-level and program-level oversight.
PROJECT-LEVEL MEASURES OF PERFORMANCE
In Chapter 4, we presented two direct measures of perfor-
mance that operate at the activity-level. One measures the
State's performance in ensuring MBE participation in projects;
the other measures the State's performance in ensuring that
change orders are not used to inflate project costs.
But as a State enters Phase III and begins to focus on
"projects" rather than "activities," is oversight based on
the direct measurement of performance still possible? In
other words, is it possible to identify project-level measures
of performance? The answer, quite simply, is yes. In fact,
the performance measures already developed by Regions IV and
VI are really project-level measures. Here are slightly
refined variants of the performance measures developed by
these two Regions:
o Timeliness. Monitoring total time to completion
after approval of the facility plan for projects
of a conventional nature would give one indication
of the State's performance in expediting the
completion of current projects. (This is a slight
variation on the performance measure Region IV
intends to use in its "Progress Reviews.")
o Cost. Monitoring project cost (both cost at award
and cost upon final completion) against expected
cost would give an indication of the State's overall
effectiveness in applying bid and award requirements.
(This is a slight variation on the performance measure
Region VI uses in its "Cost Reviews.")
-------
-34-
Again individual Regions and States should be encouraged
to develop and test other project-level measures of perfor-
mance. Innovative measures of performance that are shown to
be both workable and useful can then be adopted by other
Regions as well.
PROGRAM-LEVEL MEASURES OF PERFORMANCE
It is clearly more difficult to develop direct measures
of performance that would work at a level higher than project-
level. Nevertheless, the prospects for developing program
level measures of performance are not hopeless. On the
contrary, performance measures can be envisioned that would
help monitor the effectiveness of the State's program manage-
ment as a whole.
The ultimate performance measures of this kind would
quantify the impact of the State program in improving water
quality. Such measures can be expected to be rather diffi-
cult to develop and apply. Furthermore, they are not suitable
for routine oversight of the State's program because- of the
extended time frames necessary for water quality impact to
be seen. Nevertheless, some Regions may wish to experiment
by attempting to use them for periodic assessment of the
State program's overall performance. Once successfully
demonstrated, they would then be available for use by other
States and Regions. One possible approach that some Regions
and States may be interested in exploring at some point in
the future is presented in Appendix B.
In this chapter and in Appendix B we have shown that
direct measures of performance are not only available for
oversight at the activity-level, they are also available for
project-level oversight and are potentially available for
program-level oversight as well.
-------
-35-
Chapter 7; Proposed Oversight Procedures for
~~" Phases III, IV, V and VI
In Chapter 3, we recommended specific oversight proce-
dures for use in Phases I and II based on an analysis of
procedures currently in use and Regional experience to date
in applying them.
In this chapter, we will recommend oversight procedures
for use in Phases III, IV, V and VI. Since there is little
Regional experience to date in overseeing State programs that
have reached this stage of development, we will have to draw
not on Regional experience but rather on the analysis in
Chapters 4, 5 and 6 in making our recommendations.
PHASE III — STRIVING FOR PROJECT-LEVEL MANAGEMENT
A State enters Phase III when, in the judgment of the
Regional Office, State operations in carrying out delegated
activities are of such high quality and consistency that a
10% sample of recent transactions is no longer necessary.
The Region now believes that a 5% sample will be adequate
to ensure good quality control.
The focus on both the Region and the State can therefore
shift to project-level management. The State strives to
ensure that activity-level reviews come together in such a
way that the State has a fully comprehensive project-level
review of the quality, environmental soundness and cost-
effectiveness of each project. To achieve this objective,
the State focuses on those activities that are essential to.
effective project-level management. These "key activities" are
the analyses of alternatives in Step 1, which should result
in the identification of all feasible design alternatives and
their analysis to determine the single most cost-effective
and environmentally sound alternative. These analyses include:
o the identification and technical analysis of possible
alternative and innovative designs,
o the identification and technical analysis of other
design alternatives in the facility plan,
o the environmental assessment — that is, the analysis
of the relative environmental effects of each viable
alternative
o the financial plan — which should contain both an
analysis of the cost of each viable alternative and
an analysis of the possible methods of financing for
each alternative.
-------
-36-
Once these analyses have been completed in a thorough
manner, the grantee is then in a position to select the
"best" of the available alternatives, subject to State
approval. Thus, if these analyses have been conducted
properly, the decision at the end of Step 1 will be the
selection of the best possible project design. All that
is then required in Steps 2 and 3 is to ensure that the
basic decisions made in Step 1 are faithfully carried
out and that they are carried out in a cost-effective and
timely manner.
In addition to encouraging the State to strive for true
project management by focusing on these key activities, the
Region should, in Phase III, also begin to introduce direct
measures of performance for use in overseeing the State's
operations. These would begin to substitute for oversight
based on review of individual State transactions and over-
sight based on review of State procedures. Nevertheless,
since the Region is now introducing direct measures of
performance for the first time, it will have to continue
to rely primarily on reviews of individual transactions
and of State procedures in overseeing the State program.
PHASE IV—STRIVING FOR PROGRAM-LEVEL MANAGEMENT
Once a State has achieved a reasonable level of proficiency
in project-level management, it can then begin to strive for
effective management of the municipal wastewater program as a
whole. This objective can be said to have been achieved when the
State achieves a balanced multi-year schedule with the highest
priority projects in terms of water quality needs receiving the
highest priority in terms of funding and scheduling. The key
activities for achieving these objectives relate primarily to
pre-Step 1 actions:
o Needs identification
o Providing incentives to communities to take needed
control measures
o Assisting communities in need of facilities to apply
for Step 1 grants
o Water-quality oriented priority setting by the State
when preparing the State's Priority List.
-------
-37-
Carefully carrying out these key activities will ensure a
balanced plan for construction of needed municipal facilities.*
In Phase IV, the Region should make increased use of
direct measures of performance. As it does so, it can de-
emphasize oversight based on the review of State procedures.
When a direct measure of performance is adopted, review of
procedures can be reserved as a back up approach for use only
when performance, as measured directly, is found to be deficient,
At the same time, the Region should continue to de-emphasize
oversight through review of individual transactions until,
by the end of Phase IV, the Region agrees that 2% sample of
transactions (rather than the 5% sample used in Phases III
and IV) will be adequate for quality control purposes.
A Region that works vigorously to substitute oversight
based on measures of performance for oversight based on
review of individual transactions on of State procedures
can reasonably be said to have made the transition from the
role of a "supervisor" to that of a "franchiser" or indirect
manager by the end of Phase IV .
PHASE V — STRIVING FOR INTEGRATED WATER QUALITY MANAGEMENT
Phase V begins when the State has achieved consistent,
well-coordinated program management of the municipal waste-
water program. The State and Region can then begin to
focus on the full integration of municipal wastewater control
efforts into a broader program of integrated water quality
management. This objective can be achieved by going beyond
careful priority setting and scheduling of municipal wastewater
projects to careful priority setting and joint scheduling of
all water quality management efforts. Thus, priorities would
now be set after looking at all pollution control needs in
each locality: industrial point source control needs, non-
point source control needs and groundwater protection needs
*Note that even if it is agreed to cut off Construction
grants at some date certain in the future (e.g., in 1990),
it is still important for the States to focus on all of these
key activities including encouraging new Step 1 applications.
Indeed, if a cut off date is set, these activities become even
more important since if only limited funds remain available,
it is crucial that they be spent on the most important projects
that are needed, and are not just spent on the most important
project that are "ready to go." In other words, there may be
some greatly needed projects that are not on the priority list.
It is the most needed projects that should be funded and the
States should therefore continue to promote new Step 1 appli-
cations for highly needed projects that have not yet entered
the funding pipeline.
-------
-38-
as well as municipal control needs. Furthermore, the
scheduling of municipal projects would also consider and be
consistent with the scheduling of control actions for the
other classes of water quality problems.
It is clear that the "key activities" for ensuring
effective integrated water quality mangement are comprehensive
water quality assessment and planning activities. In
Phase V, the State and Region should therefore upgrade and
more greatly emphasize areawide planning activities at
both the local and State level.
At the same time, the Region should continue to develop
and use additional direct measures of performance. In doing
so, it can continue to de-emphasize oversight through review
of procedures and oversight through review of individual
transactions.
PHASE VI — OPERATING IN A FULLY-DEVELOPED,
FULLY-DELEGATED MODE
Once the State achieves full integration of its municipal
wastewater program into its broader program of integrated
water quality management, it will have reached the ultimate
in "management scope" for construction grants activities. The
principal objective of the Region and the State should then
be to maintain this level of operation. A secondary objective
should be to achieve greater efficiencies in program operations
(at the State level) and in oversight methods (at the Regional
level) without jeopardizing the scope or quality of State
operations. Some increased efficiency in Regional operations
can continue to be made by introducing new direct measures of
performance wherever possible.
Once the State enters Phase VI, the Regional role has begun
to shift from that of "franchiser" or indirect manager to that
of "regulator" or environmental manager.
This completes our analysis and recommendations with
regard to the improvement over time in the scope of State
management and Regional oversight and in the kinds of over-
sight mechanisms used. These recommendations are summarized
briefly in the beginning of the next chapter (Chapter 8) and
are summarized Phase-by-Phase in more detail in Chapfer 9.
-------
PART III: TYING IT ALL TOGETHER
-------
-39-
Chapter 8; The Proposed Role for the Regions
~ and of EPA Headquarters
We will now briefly discuss the steps that the Regions
and EPA's Office of Water Program Operations at Headquarters
might take to speed both the development of State programs
and the transformation of State/EPA relations to those
appropriate for a fully delegated construction grants program.
THE REGIONS WILL CONTINUE TO PLAY THE KEY ROLE
Since we have already addressed in considerable detail
the actions that the Regions can take, we will merely summarize
them here:
The Regions should continue to ensure that each State is
carrying out its delegated functions with reasonable proficiency
and consistency. The amount of oversight necessary to achieve
this objective will decrease over time. The Regions should,
therefore, regularly reassess the State program's stage of
development to ensure that the oversight procedures in effect
for a given State are still consistent with current conditions.
As the performance of a State improves, the closeness of the
Region's scrutiny should be relaxed accordingly. In reviewing
State performance, the Regions should continue to use reasonable
standards. The benchmark should not be "zero errors" or "no
decisions that the Regional staff disagree with" since such a
standard can never be met. After all, mistakes are unavoidable,
and no two individuals will ever agree on every judgment call
that has to be made. Instead, the benchmark should be perfor-
mance of reasonable quality and consistency and decisions
based on a sound, .thoughtful, balanced analysis of the
available alternatives.
The Regions should continue to encourage each State to
enhance the "scope" at which the State is carrying out its
responsibilities. Once a State becomes reasonably proficient
at carrying out "activity-level reviews," the Region should
begin to focus on how it can restructure its oversight proce-
dures to encourage the State to begin conducting "integrated
project-level reviews." Once the State is reasonably proficient
at project-level reviews, the Region should adjust its over-
sight procedures to encourage the States to achieve full program
management of the municipal wastewater program, and finally,
integrated water quality managment.
The Regions should, wherever possible, replace oversight
procedures that rely on reviewing State procedures or on
reviewing the quality of individual transactions (i.e., of
documentssubmitted to the Region after State review) with
oversight based on the use of direct measures of performance.
-------
-40-
In conjunction with the use of quantitative measures of
performance, the Regions should begin to negotiate with
the State to arrive at performance targets against which
the State's actual performance can be measured. Whenever
performance targets are met, detailed scrutiny of the
corresponding State review procedures and of the related
project documents* can be dispensed with. In this way,
the Region can further reduce the extent of EPA's direct
"hands-on" involvement in State operations, without
sacrificing program quality.
The implications for State/Regional relation of con-
ducting oversight using direct measures of performance in
place of oversight based on review of individual transactions
or reviews of State procedures are far-reaching. Indeed,
if a Region moves vigorously to adopt direct measures of
performance during Phases III and IV, it can reasonably
claim to have shifted from the role of a "supervisor" to
that of a "franchiser" by the end of Phase IV. When the.
State reaches Phase VI, the Region will have begun to approach
the role of "regulator" or environmental manager.
.The three paragraphs above briefly but rather fully
characterize the steps each Region can take to promote the
full development of the construction grants program operating
in a delegated mode.
EPA HEADQUARTERS ALSO PLAYS A CRUCIAL ROLE
Until now, the Office of Water Program Operations (OWPO)
has chosen to play a fairly limited role in overseeing the
delegation of program responsibilities to the States and
overseeing the operation of the program in its delegated
mode. OWPO's primary actions to date have included issuing
some general advice to the Regions on how to oversee program
activities once delegated, and establishing a small staff to
monitor and coordinate delegation activities nationally.
There are several other key areas in which OWPO could
profitably play a strong role thereby expediting the successful
development of delegated State programs:
OWPO shouldprovide formal guidance to the Regions on
how to utilize specific oversight methods. For Phases I and
II, there is already sufficient Regional experience that
OWPO can now prepare detailed guidance on "how to do it."
For Phase III and beyond, the guidance must necessarily be
considerably less detailed; nevertheless, it is still possible
Other than a small sample review for quality control purposes.
-------
-41-
to prepare reasonably specific guidance. As some Regions gain
more experience operating in Phase III and later phases,
this guidance can be revised periodically to be more speci-
fic and, as necessary, modified to reflect experience gained
regarding what works and what doesn't. OWPO should neverthe-
less remain permissive in its attitude toward the guidance
that has been prepared. Regions should be allowed to depart
from the guidance in order to experiment with alternative
oversight methods (especially for the later Phases). The
results of such "experiments," when successful, can then be
reflected in subsequent revisions of the guidance.
OWPO should review Regional determinations as to each
State's stage of development. In this way, OWPO can help
ensure reasonable national consistency in assessing where
each State stands in terms of "phase" of development to
ensure that:
o a Region is not holding a State back by underrating
its degree of development,
o a Region is not establishing overambitious goals
for a State as a result of overrating its degree
of development.
OWPO should review direct measures of performance adopted
by States and Regionsand should carefully reviewthe nego-
tiated performance targets to ensure reasonable national
consistency where appropriate. Considerable latitude should
be allowed at first in the definition of performance measures
to allow needed experimentation and testing of alternative
approaches. But as certain specific performance measures
are demonstrated to be preferable to others, less latitude
should be allowed. Similarly, for performance targets, only
reasonable variations from State to State should be allowed
once there is enough experience to determine what levels are
reasonably achievable. For some aspects of performance,
considerable State-to-State variation in performance targets
may be desirable. For example, New Hampshire, with a very
small minority population, might appropriately have a much
lower MBE performance target than California, which has a
relatively large minority population.
OWPO should prepare detailed case studies of States
that have success f u 13.y^ ajchi e ved pr o j e ct-1 e ve 1 management,
program management or integrated water quality management.
This case study information would than be available
to provide guidance on how to increase the"scope"
of State operations and of Regional oversight methods
elsewhere.
-------
-42-
OWPOshould provide guidance to the Regions and States
on staffing patterns and staffing levels. More than one
Regional program manager has remarked that program operation
in a delegated mode calls for a new mix of expertise in both
the Regions and States. One in particular asserted that
whereas States tend to be presently staffed with 80% engi-
neers and 20% administrative personnel, the ideal mix for
full delegation should be more like 50% of each.
Other Regional managers have made similar remarks about
the need to change the skill mix in the Regional Office.
(Some seem to feel that few engineers are needed in a Regional
Office once operating in a fully delegated mode.) OWPO might
well analyze the situation and provide guidance on "target
staffing patterns." Such guidance might also indicate the
number of staff needed in the State Office to conduct various
delegable activities at various levels of effort. This is
merely recommended staffing; levels also could be proposed
based on expected workload measured as total number of- dollars
and total number of positions.
OWPO should also prepare material thatanswers "the
questions State staff most often ask." OWPO could begin by
polling the Regional staff to determine, for the program
as a whole and for each specific delegated activity, what
questions the State staff most often ask. OWPO could then
prepare standard responses and distribute this material in
question-and-answer form to all' the Regions and States.
This would reduce the need for Regional staff to have to
keep answering the same questions over and over as new
States assume delegation of activities or as new State
staff come on board.
OWPOshould prepare standard training packages or otherwise
arrange to make training available to State staff and Regional
staff on how they should function in a fully-delegated program.
The current situation appears to be that each Region is making
its own arrangements to train State staff, and there appears
to be no training of Regional staff at all on how to deal with
delegation. Such a situation is clearly inefficient in that
each Region must take the effort to set up its own training
procedures or programs, and the Regional staff who must do
this are generally inexperienced in designing and operating
training programs.
OWPO should arrange for new distribution mechanisms for
ensuring that State agencies and State staff responsible for
carrying out specific activities receive, directly, and on a
timely basis, notice of new regulations or other requirements
thataffect the performance of their responsibilities. Some
States have indicated that they are dissatisfied with the
-------
-43-
mechanisras now used to inform them of -new requirements and proce-
dures. They frequently become aware of new requirements only
through staff-level contacts with the Regional Office. In
other cases they discover that a new requirement has gone
into effect at a formal review point several months later.
The State is then told that everything they've done in that
period has to be redone since it failed to reflect the
new requirement.
To prevent such problems from recurring, OWPO should
develop a distribution mechanism that results in automatic
notification of State Program Managers of all new require-
ments or procedures that affect their operations. Such
notification might be timed to reach the States two to three
weeks after the Regional Offices are informed. Such timing
would help ensure that the Regions will be prepared to
respond to State inquiries regarding the new requirements
as soon as the State officials receive the formal notice,
and would allow the Regions reasonable time to prepare'
supplementary guidance when the Regions feel that this is
necessary.
L
-------
Chapter 9; Concise Summary of RecommendedOversight
Methods for each of Phases I throughVI
The specific oversight, mechanisms recommended for Phases
I and II in Chapter 3 and recommended for Phases III, IV, V
and VI in Chapter 7 are summarized in Table 4«
The remainder of this Chapter will present "checklists"
of oversight methods to be used in each of the six phases
of development of a State program.
Phase I Oversight--Helping the State Get up to Speed.
In Phase I:
1. The Region encourages inquiries from State staff.
Direct contacts between State staff and their
Regional counterparts are accepted as the norm.
2. Regional staff are encouraged to call their State
counterparts directly to inform them of any problems
in the way an activity is being conducted.
3. Informal communications are expected to serve as
the primary source of information about potential
problem areas.
4. Document packages from the States requiring Regional
signoff will be given in-depth (essentially duplicative)
review by Regional staff until they are satisfied
with the quality and consistency of State performance
for that activity,
5. The Regional staff gives an in-depth review to selected
State document packages based on the criteria
described earlier:
o the importance of the activity,
o how long it has been delegated,
o the adequacy and consistency of recent State
actions for that activity, and
o the extent of State requests for assistance
for that activity.
6. The Regional staff attempts to correct any problems
discovered by successively applying the following
corrective measures:
Provide telephone feedback to the State staff
member responsible, noting the exact nature
of the deficiency.
-------
CO
t!
(1)
14-1
1
i!
4J
f
2
I
•8
"8
I
is
> EH H E"1 ^ S
H M > rf O S
E-" > H gj H 0j
< £ S O § W
en
H < g CO
O »
to
H) -H
^ 4J
4-> O
.3*
S
O C
•H -H
ra 51
in
03 Ot -P
CO 0 M
^4 <1>
<4-t
<0
co
32
i «j -H
In
O c
•H -H
in tr.
{
S
0) fO
C H
*r -H QJ
CN +J OJ
co W
M E «} 0 0
SCO ra
(U
4J -H
'H
0) «O ^ (ti
U> £ CO 4J >i
^^aiS-^
I -H 4J -H 4-> H
I 4J C w tQ ro
' S i 01 '» &
fd
M
BU H
M
CO
-S
tion
staff
ega
nS
^"S
8*°
w
in
a O
0) -H
4J g
W|S
03
i
H
«
(0
i cc
5 5)
ffl -H
C0.4J
rOl-H
>i-H
i—' H
l
ct
a
fo
w
*
M-)
s
id
£
ffl W &
O 0) -H
•!l.2 b
•sa
*
4J O
a-
4J 4J
1
T->.
1
i-H
!
3
3$ *S
en
a
«
*
-------
•o
•**
cu
0 0
CN
OP
o
CD
ir,
•H
CN „ 2. .O 3.
fe°
itig .
S a-3Z
03 (U
SfUS
se!
CO
0) C
•H JJ O
+J C -H
•H Q) -U
^ g
CO W
a, c -H
f O J
4J -H
^
^^
s-d
E r^ O
iftj
K) (0 (X
O O
8
S
8
in
OrH O
& £
C S "5 S HJ S
O ••-» ® -H 01 -H
O. > C >
SM TP i
-p > -H •>-!
-------
t§ ^ 2 W
M CO _ 03 H tO
6-1 w O 53 s
H u !Z E-i S E
PiisSl
•O
Q)
&
l-l
2
CO
©
ggi
lie
4J ..
•H
CO 0)
s s
Cfi
if—
-*J it) 4.) w -j-1
-H 0 (0 Q O
i— I 0 4-> 0
O O 0
4J "
§ C 4-> CD
(Q H O n-t •-» co g —
-n a) <*p <8 *
•H > O
H G) {N
»—^
5
&
mm 0 m H CJ
y ^* r< — ' 1 "^
•H
U h-H
•H 0) fl)
•s
S
5) dc
-,-4 > O
r-l
> *
V-i fl) ro O
0) 40 <#P
(DO O C O
U f- (N O rH
I ^ § j ti j
W u) *U w (v Q/
O -H 0) -H W -H
m > Q > C >
01 i^ !h )-< >4 >-l
pt p1 g-i
U JJ
s 3 s o
(D
CO 0>
2 cl
'C C *0 S ®
C O 4-" 5) *C
-H <-( cn k-i 0}
CO C 6
d) 0 ft
4J > -S Ji CO
C -H 4-> DI n3
In CTl
C 1 fl)
4J rH
a s T5
-H J-l Ifl
O
Sl..
»w J-i (0
0
s
1
a
^
a
g
*
J-!
0)
•O
CO
-r4
|
-------
-45-
o Perform a sample {or model) review that
demonstrates to the State staff how the
activity should be carried out.
o Go to the State Office to conduct a sample
review jointly with the State staff member.
o Provide or arrange for additional training
for the State staff member.
Such actions are to be followed by:
o careful monitoring of subsequent State
actions of this kind to determine whether
the quality has improved, and
o remaining in continuous contact with the
State staff to provide feedback on perfor-
mance and to provide additional guidance
as needed.
7. Program Managers in the Region (at the Branch and
Section Chief level) meet regularly with State
Program Managers to discover and review current
problems and to inform them of any changes in
regulations or procedures. These meetings can be
conducted in person or by conference call, with
face-to-face meetings at least every month or
two.
8. Pending development of a system for direct noti-
fication by OWPO, the Region informs both the
State Director and appropriate State staff in
writing of changes in regulations or procedures
that affect their operations.
9. Later in Phase I, the Region begins to conduct
Paperwork Sampling at the Regional Office on a
quarterly basis for any activities that have
evolved to the point that the Region is conducting
only a checklist review. The sample consists of
10% of the actions in each activity during the
previous quarter (or more than 10% if the number
of actions is less than twenty). The sample
should include both large and small projects but
may give somewhat greater emphasis to:
o large projects
o environmentally sensitive projects
o unique or particularly troublesome projects
The Region prepares a formal report to the State
on the results of the Paperwork Sample.
-------
-46-
10. The Regional staff conduct formal "visits" to the
State Office at least twice per year:
o Early in Phase I, these visits should be
used to review the nature of the operational
problems discovered during the previous
six months and the adequacy of the response
by the State staff.
o Later in Phase I, these visits should also
incorporate a formal review of State proce-
dures for activities in which the State's
actions have been less than adequate.
Phase II Oversight — PromotingConsistent,
High Quality Activity-Level Management
The State enters Phase II when:
o all activities to be delegated have been
assumed by the State and
o all are being performed with reasonable
quality and consistency.
In Phase II;
1. The Region normally conducts only checklist reviews
of incoming documents.
2. The Region conducts formal Paperwork Samples at
the Regional Office as described above (Phase I,
Item 9).
3. The Regional staff continues to alert their State
staff counterparts of problems and inadequacies
on an informal basis as they are discovered.
4. The Regional managers continue to meet on a
monthly basis with State managers to discuss
current problems and changes in regulations and
procedures.
5. The Regional Staff conducts semi-annual reviews at
the State Offices. These reviews emphasize the
State procedures for carrying out specific activi-
ties, focusing on those activities shown in the
Paperwork Sample to be resulting in products that
are not consistently of high quality.
-------
_47-
Phase III — Promoting Project-Level Management and
Introducing Direct Measures of Performance
Phase III begins when:
o all delegable activities have been assumed by
the State and
o all activities are being performed with a con-
sistent high quality.* The Region inaugurates
Phase III when it finds the consistency and
quality of State activity-level operations to
be high enough that it is willing to go from
a 10% to a 5% sample in conducting the quarterly
Paperwork Samples.
In Phase III;
1. The State has attained*high quality in its activity-
level operations that the Region and State begin
to work towards project-level management and
oversight.
2. There is now substantially less need for informal
contact between the State and Regional staffs.
Most oversight information is now obtained through
formal ovrsight methods.
3. Paperwork samples are conducted quarterly in the
Regional Office using a 5% sample.
4. The Region begins to introduce some direct monitoring
of State performance as part of its oversight methods
The first performance measures introduced include:
o Activity-level measures of performance
- MBE performance
- Change order cost increases
o Project level measures of performance
- Actual project cost as compared with
project budget
- Project timeliness as compared with
"reasonable expectations"
Note, however, that performance measures are adopted
only after negotiation with the State on the specific
measures to be used. The Region and State also
negotiate "performance targets" for each aspect of
performance to be monitored directly during the
following year.
*This does not, however, mean that there will be a zero
error rate.
-------
-4*-
The Region conducts semi-annual reviews at the
State Office. The Region uses these visits
primarily to review State procedures. These
reviews of procedures are conducted for the
following reasons:
- to determine the degree and adequacy of
project-level management (i.e., the degree
of project-level coordination between
ativities). In doing so, the Regions
should focus on State procedures for the
"key activities" for effective project-
level management (i.e., activities related
to the analysis of alternatives).
to follow up on State procedures for acti-
vities that have been found to be deficient
through direct measures of performance.
- to check on State procedures for those
activities for which no direct measures of
performance were available.
Phase IV — Promoting Program-level Management and Increasing
the Use of Direct Measures of Performance
This phase begins when the Region determines that the
State has achieved consistent and effective project-level
management.
The oversight methods used in Phase IV are the same as
in Phase III with the following exceptions:
o Direct measures of performance are adopted for
additional aspects of State performance. (i.e.,
by agreement with the State, the Region continues
to substitute oversight through direct monitoring
of performance for oversight through review of
State procedures.)
o The Region and State focus on promoting program-
level management by the State. The Region therefore
focuses more on the "key activities" for effective
program management in its semi-reviews at the State
Office. These "key activities" include:
- improved assessment of needs
incentives for communities with strong needs
(based on water quality considerations) to
apply for Step 1 grants
-------
-49-
assistance to communities with strong needs
in preparing Step 1 grant applications
improved priority setting among projects based
on water quality consideration
Phase V -- Promoting Integrated Water Quality Management
and Increasing the Use of Direct Measures of
Performance
This phase begins when the Region determines that the
State has achieved consistent and effective program-level
management.
The oversight methods used in Phase V are the same as
those in Phase IV with the following exceptions:
o State operations are of such high quality that the
Region deems it appropriate to drop from a 5% sample
to a 2% sample in conducting its quarterly Paperwork
Sample.
o The Region, with the State's concurrence, continues
to substitute oversight based on direct measures
of State performance for oversight based on review
of State procedures.
o The Region and State focus on promoting integration
of the program into a broader program of integrated
water quality management. They do so, in part, by
placing more emphasis on the "key activities" for
achieving this objective:
- comprehensive areawide water quality planning
at the local level,
- comprehensive areawide water quality planning
at the State level, and
joint scheduling of all water quality improve-
ment efforts in each locality (including municipal
control efforts, industrial point source control
efforts, non-point source control efforts and
groundwater protection efforts).
Phase VI — Overseeing a, Fully-Developed, Fully-Delegated
Program
This phase begins when the Region determines that the
State has achieved consistent and effective management of
the program as an integral part of an integrated water quality
management program.
-------
-50-
The oversight procedures in Phase VI are the same as
those in Phase V with the following exceptions:
o The State, with Regional encouragement and assis-
tance, continues to seek ways to improve the
efficiency of State operations without sacrificing
the quality or "scope" of the State program.
o The Region continues to develop and adopt direct
measures of performance for overseeing the State
program. Use of these direct performance measures
replaces oversight based on review of procedures.
-------
A-l
Appendix A; Alternative ApproachesforImproving
Project-level Review by the States
In Phases I and II, the State staff tends to approach
each project as a disjointed set of activities to be re-
viewed. In Phase III, the State begins to review projects
as a whole. The Region attempts to promote this change in
management scope by assessing the State's project-level
performance. When deficiencies in the effectiveness of
project-level reviews are discovered, the Region and State
negotiate changes in the State's review procedures to
correct these deficiencies.
In negotiating with the States on possible improve-
ments in procedures to promote integrated project level
reviews, the Region should bear in mind the advantages and
disadvantages of the three distinct procedural changes
the State might propose. The three alternatives that the
State might propose are:
o Require multiple concurrent reviews of certain
documents by different Staffs with related
functions.
o Institute "project review meetings," with
participation by each staff with distinct
responsibilities
o Create new integrated review groups.
THE RELATIVE COST AND RELATIVE EFFECTIVENESS OF THE THREE
APPROACHES DIFFER GREATLY
Require Multiple^ Reviews. While requiring multiple
reviews of various documents by different staffs is the
solution most often proposed, these multiple reviews are
the least desirable means of achieving increased coordi-
nation. For example, it may be discovered that deficiencies
frequently arise in projets because of inadequate facility
plan review. A major reason for this is that the State
fails to discover fundamental flaws in the basic analysis
of alternatives because that analysis appears in four
separate documents. Typically, we might find that:
o A member of one staff group is responsible for
reviewing the alternatives analysis incorporated
in the environmental assessment (EA),
o A member of another group looks at the manner in
which the grantee and its consultant consider
alternative and innovative (A/I) technologies, and
-------
A-2
o A member of a third group looks at the remainder
of the alternative analysis contained in the
facility plan.
o A member of a fourth group reviews the financial
aspects of the project (the cost analysis of
possible alternatives, the proposed method of
financing the local share, and the user charge,
industrial cost recovery — UC/ICR — proposal*).
To ensure that its staff begins to consider the quality
of the analysis of alternatives for the project as a whole,
the State might simply propose that each of three groups
look at and comment on all four aspects of the alternative
analysis (i.e., the EA, the A/I analysis, and the remainding
analysis of alternatives contained in the facility plan, and
the financial plan). If any group raised objections to any
portion of a project's alternatives analysis, the project
would be "flagged" and a joint critique would be prepared.
Although such a solution is easy to implement — since
no change in organization or functional responsibilities is
required to implement it — it is usually not very satis-
factory because:
o the staff, who are already devoting full time to
reviewing their individual pieces of the alter-
natives analysis, resent having to "shuffle more
paper" than before. Furthermore, they can in fact
be overwhelmed by the total volume of material
they are now expected to review. In a real sense,
this approach increases total workload by requiring
multiple reviews of the same material.
o As a result, total review time increases, bringing
with it increased project delays.
o Also, the overworked staff does a less adequate job
in conducting each review.
The only way to make "increased coordination" work is to
increase the size of each of the affected staff groups so that
each staff member has fewer projects for which he or she is
responsible. But with current staffing limitations in the
States, such a course of action is not normally available.
Institute Project Review Meetings. An alternative
solution that avoids some of the negative aspects of simple
"increased coordination" is to have each staff group remain
responsible for reviewing only that portion of the project
for which it was originally responsible. Members of each
staff group then meet periodically as a project review team
The situation is complicated by the fact that the UC/ICR
work is not normally conducted until well after the completion
of Step 1 planning activities.
-------
A-3
to "compare notes." By having each representative discuss
the strengths and weaknesses of the material he or she has
reviewed, it is possible for the team as a whole to discover
deficiencies in the project as a whole that were not uncovered
in the individual reviews.
The advantages of such an approach are:
o the burden on each group does not increase except
to the extent that time is consumed in meetings of
the "project review team>"
, o each group can retain a specific expertise, yet
the project team (and therefore the project) can
benefit from that expertise when appropriate.
The approach has the disadvantage that:
o either considerable time is consumed in meetings
of the project review team, or
o the project review team meetings are so-hurried
and discussion is so limited that there is a
significant likelihood that major problems in
project consistency will go undetected.
Create Integrated Review Groups. A third possible
approach is to assign closely related review functions to a
single group. A single staff member, for example, would
then be responsible for reviewing all aspects of the analysis
of alternatives for a single project. By consolidating
the separate staffs that formerly conducted separate pieces
of the alternatives analysis (i.e., the EA, A/I and facility
plans and financial planning staffs*), enough staff will
be so available that the number of projects assigned to each
staff member can be significantly reduced. In this approach,
"coordination" among different staff members or staff groups
that perform related functions is no longer needed because
all of the functions to be coordinated have been "integrated"
and assigned to a single individual.
Such an approach has the following advantages:
o It does not increase the total workload.
o By assigning full responsibility for all aspects
of related reviews to a single individual, this
approach minimizes the need for coordination
Note that most States do not have distinct financial
planning staffs — indeed, little financial analysis is
now conducted at either the State or Regional level.
-------
A-4
between staff (and the attendant possibility of
failing to discover problems because of faulty
communication).
o It is likely, in the long run, to increase staff
satisfaction since each staff member is given a
larger, more significant aspect of each project
to be responsible for. (In other words, the
staff would tend to feel less like "workers on
an assembly line" and more like responsible
professionals.)
It has the following disadvantages:
o Normally, only one "mind" is brought to bear on
major aspects of each project.
o It requires a significant reorganization, including
a realignment of functions and the training of staff
to perform the new functions that are being added
to the ones they had before.
In general, the creation of integrated review groups
where feasible appears to be the best approach for promoting
"project level review."* Nevertheless, there is a limit to
the number of project activities that a single person can
be expected to be able to review adequately. Some coordi-
nation between staff and staff groups will therefore always
be necessary when reviewing individual projects. It is there-
fore inevitable that several individuals will be responsible
for reviewing any given project, and establishing "project
teams" appears to be the best mechanism for promoting adequate
coordination between these individuals. Such teams would
meet regularly to discuss ongoing projects for which they are
responsible and identify possible problems with the project
as a whole.
*While we have recommended the creation by States of integrated
review groups in order to improve the degree to which it
conducts effective project-level reviews, we will now
for illustrative purposes describe the effective use of
such groups use in a Regional Office.
One Region (Region IV) has already begun to make use of inte-
grated review groups in its own internal operations. To ensure
the fullest coordination of Facility Plans with Environmental
Assessment and with 208 Plans, the responsibility for reviewing
Facility Plans and preparing Environmental Assessments has
recently been assigned to the Region's Water Quality Management
Branch, which also has responsibility for 208 Plans.
-------
Appendix B:
A Possible Experimental Approach
for Assessing a State Program's
Impact on Water Quality
As stated in Chapter 6, the approach described here may
not be suitable for regular oversight of a State's performance.
It might, however, prove to be useful for periodic assessment
of the extent of the problem being addressed and the extent
of the program's beneficial impact. Application of this
approach presents sufficient technical problems that its use
should not now be required. At most we might expect that one or
two Regions may determine that its use would bring sufficient
benefits that they might, at some point in the future, attempt
to apply it on an experimental basis.
Assessing the Extent of the Problem
The basic purpose of the Municipal Wastewater Program is
to eliminate violations of water quality criteria due to dis-
charges of inadequately treated municipal wastewater. The
extent of this problem in a given State can be quantified
as follows using reasonable estimates based on available
monitoring data:
o number of stream miles violating water quality
criteria — solely or in great part — because
of inadequately treated municipal discharges
o amount of lake surface acreage violating criteria
because of municipal discharges
o amount of bay and coastal water acerage violating
criteria because of such discharges.
With a periodic reassessment of the extent of the problem
(e.g., every 1 to 3 years), it is possible to determine both
the extent to which water quality adversely affected by
municipal discharges is improving (or deteriorating) and to
quantify the extent of the benefits derived from projects
completed since the last assessment. As a result of such a
survey, it might be discovered, for example that "37 stream
miles that violated water quality criteria as a result of
inadequate treated municipal wastewater no longer do so
because of treatment facilities that have come on line in
the last two years. There are now only 123 stream miles in
this State that violate water quality criteria wholly or
substantially as a result of inadequately treated municipal
discharges."
This approach could also be applied to projecting the
water quality impact of projects now underway by using reason-
able estimates or projections. For example: "Treatment
projects initiated this year (i.e., new Step 3 awards) will
result in 55 additional stream miles that do not now meet
water quality criteria because of municipal discharges to
come into conformance with those criteria."
------- |