&EPA
68-01-6425
MIDCOURSE EVALUATION OF
AREA OF REVIEW:
EVALUATION OF ALTERNATIVE
STUDY METHODOLOGIES
SUBMITTED TO
DR. JENTAI YANG
OFFICE OF DRINKING WATER
U.S. ENVIRONMENTAL PROTECTION AGENCY
JANUARY 1983
BOOZ.ALLEN & HAMILTON, INC.
UNDER THE DIRECTION OF
GERAGHTY & MILLER, INC.
-------
SEPA
68-01-6425
MIDCOURSE EVALUATION OF
AREA OF REVIEW:
EVALUATION OF ALTERNATIVE
STUDY METHODOLOGIES
SUBMITTED TO
DR. JENTAI YANG
OFFICE OF DRINKING WATER
U.S. ENVIRONMENTAL PROTECTION AGENCY
JANUARY 1983
BOOZ.ALLEN & HAMILTON, INC.
UNDER THE DIRECTION OF
GERAGHTY & MILLER, INC.
-------
ACKNOWLEDGEMENT
This report was prepared by Booz, Allen & Hamilton, Inc., under
the direction of Geraghty & Miller, Inc. The Booz, Allen project
manager was Dr. Joanne Wyman. She was assisted by Mr. Walter Mardis
of Booz, Allen. The Geraghty & Miller project manager was Mr.
William Thompson, and Mr. Roger Anzzolin was the EPA task monitor.
-------
TABLE OF CONTENTS
Page
Number
1. OBJECTIVES OF AREA OF REVIEW 2
x
2. DESCRIPTION OF THE'ALTERNATIVES 6
3. EVALUATION OF THE THREE ALTERNATIVES 20
-------
LIST OF EXHIBITS
Page
Number
Definition and Measurement of AOR Program
Objectives
Overview of Options for Each Phase of
Study Alternatives I and II
Summary of Reporting Requirements
Applicable To Mid-Course Evaluation
11
-------
MID-COURSE EVALUATION OF AREA OF REVIEW:
EVALUATION OF ALTERNATIVE STUDY METHODOLOGIES
EPA has issued Underground Injection Control (UIC) and Consoli-
dated Permitting regulations that require operators of new oil and
gas wells to establish an Area of Review (AOR) defined as the
radial distance within which existing active, abandoned, or dry
wells penetrating the zone of injection must be checked. The ob-
jective of the review is to identify and correct wells which are
potential conduits for groundwater contaimination. Because the
final rules differ substantially from those proposed in 1976, the
Agency has agreed to undertake a "mid-course evaluation" in order
to evaluate the AOR program's costs and benefits.
The Office of Drinking Water (ODW) has asked Geraghty & Miller
(G&M) and Booz, Allen & Hamilton (BA&H) to assist it in designing
the mid-course evaluation study be describing and evaluating three
alternative study methodologies. The Agency has defined these al-
ternatives in terms of the data collection effort:
Alternative I (State-supplied data); Each state operated
program submits the data specified in 146.24 and 146.25
of the UIC regulations to the cognizant EPA regional
office.
Alternative II (EPA-conducted random survey);.EPA con-
ducts detailed reviews of selcted permit cases.
Alternative III (Combination of I and II); EPA supple-
ments its analysis of the state submitted data with an
analysis of EPA collected data.
Our approach to conduting this assingment consisted of the
following.three steps. First, we established a baseline against
which to evaluate the three alternatives. This involved defining
the substantive and management objectives of the AOR program, iden-
tifying effectiveness indicators, and determining the types of data
needed to apply the indicators. Second, we developed descriptions
of each alternative. Although our project team meetings generated
numerous variations on the principal steps in each alternative,
we discuss in this technical report only those variations which
appear to be compativle with current Federal and state environ-
mental program practices. In the next step we evaluated each
alternative with respect to its overall effectiveness and ease of
implementation in measuring area of review objectives.
-1-
-------
Based on our evaluation of the alternatives, we reached two
conclusions. First, the combination option provided EPA with
the most comprehensive and best quality evaluation. It enables
EPA to evaluate the quality of a state's permit decision process
by conduting on site interviews with permitting personnel, and
enables the Agency to evaluate that data within the context of
the aggregate and permit specific data the states must submit in
accordance with the UIC rules.
As we suggest in the final section of the memorandum, how-
ever, Alternative III is potentially quite costly . Recognizing
that the Agency mya not.want to make such a high resource commit-
ment, we have defined another alternative. First, EPA could con-
duct a preliminary evaluation based on Alternative I. Based on
the results of that evaluation, EPA could determined the need for
an independent survey.
1. OBJECTIVES OF AREA OR REVIEW
The purpose of the mid-course evaluation is to reassess the
need for and the benefits of the AOR program. Based on this re-
assessment, EPA could determined whether to broaden or reduce the
scope of the AOR requirements.
Our first step in assisting EPA in the selection of a mid-
course evaluation study methodology was to define the AOR program
objectives and develop methods and approaches suitable for measuring
state agency success in fulfilling those objectives. We then identi-
fied sets of achievement indicators appropriate for conducting these
measurements. Exhibit i presents an overview of the AOR program
objectives and the achievement indicators, together with types of
data required to measure those objectives.
(1) Definition of Objectives
We first defined primary and secondary objectives of the
AOR program. Primary, or substantive, objectives address the
direct intent of the SDWA: the prevention of groundwater
contamination. Secondary objectives such as the minimization
of permitting costs and processing delays also are important
because the quality of a state's permit-decision making process
is a reflection of the program's efficiency and effectiveness.
Additionally, the purpose of the revisions to the 1976 version
of the UIC rules was to reduce costs and delays. The measurement
of both sets of objectives provides a good indication of how
effectively a state implemented AOR requirements. The defini-
tions of the specific objectives presented in Exhibit 1
reflect consultations with EPA, a review of the SDWA and the
history of its regulatory development including the research
conducted in support of the rulemaking.
-2-
-------
EXHIBIT 1
Definition and Measurement of AOR Program Objectives
AOR PROGRAM OBJECTIVE
ACHIEVEMENT INDICATORS
DATA REQUIREMENTS
FOR MEASUREMENT
ASSESSMENT OF
•MEASUREMENT FEASIBILITY
Protection of current/
potential underground
sources of drinking
water
Direct measurement of prevention
or abatement of contamination is
virtually technically impossible.
State success in fulfilling the
other objectives listed serves as
an indirect indicator of effective-
ness with respect to this objective
See data requirements for measuring
other objectives.
See assessments of feasibility of
obtaining data required to measure
other objectives
I
Ul
I
Correction of wells
identified as posing a
potential threat to
groundwater
Direct indicators such as reduction
in contamination from poorly com-
pleted or abandoned wells are
difficult, if not impossible, to
measure
Indirect indicators include:
Identification of all potentially
threatening wells
Development of corrective action
plan for all threatening wells
Completion of all needed
corrective action
percentage of wells penetrating
injection zone selected for
corrective action
Percentage of state approved,
discontinued corrective actions
Rationale for corrective action
decisions
Number of confirmed, successful,
completed corrective actions
Collection of the statistical data is
easy; however, its evaluation, in terms
of the technical soundness of correc-
tive action decisions, requires an
independent, on site investigation.
Ultimate review of all
wells which pose a
potential threat to
groundwater
Extent of overlap of AOR of
first and second year permit
applicants
Overlays or other graphic displays
showing existing wells and AOR
for new well applicants
Easy to measure since states will
submit permit specific maps to EPA
which are amenable to aggregation.
Compliance by state
agencies with technical
standards and adminis-
trative procedures of
AOR regulations
Consistent, equitable imposi-
tion of corrective action
requirements
Detection of non-compliers
Enforcement against non-
compliers
Submittal of accurate, complete
records in mid-course report
Comparison of corrective action
plans for each permittee
Number of applicants for which
state approved discontinuation
of corrective action
Case specific flowcharts of
application of AOR requirements
Number of total and permittee
specific inspections
Number of violations detected
Number of each type of enforce-
ment action taken
Requires considerable on site file
examination and interviews with agency
staff; involves judgment on the part
of evaluation team, thereby necessi-
tating commitment of a team with
extensive technical and program
management expertise.
-------
EXHIBIT 1 (Continued)
AOR PROGRAM OBJECTIVES
ACHIEVEMENT INDICATORS
DATA REQUIREMENTS
FOR MEASUREMENT
ASSESSMENT OF
MEASUREMENT FEASIBILITY
Minimization of adverse
impacts on energy resource
development
Absence of increases in the
typical time taken to bring a
new well or conversion on-line
attributable to AOR
Absence of increases in project
costs associated with AOR
Absence of production losses
attributable to AOR
Time frames for permit applica-
tion preparation
Time frames for permit processing
Manhours and costs associated with
- Defining AOR and corrective
action
Implementing corrective action
Data on permittee manhours, cost, and
time frame for permit application is
difficult to collect. It requires
operator recordkeeping not consistent
with current practice. If EPA can
obtain the data, it can compare it
with estimates prepared in support of
the rulemaking.
Data on time frames for permit processing
requires laborious on site review of
state files to identify delays
attributable solely to AOR.
Maximization of management
and technical quality of
AOR decisions
Minimization of adverse
resource impacts on
state
Adequacy of manpower and budget
resources, including staff
stability
Adequacy of staff training
Timeliness of task completion
Consistency of requirements
imposed on permittees
Budget increases/decreases
Staff increases/decreases
Staff turnover rates
Hiring/promotion standards
Staff education and experience
Availability and use of continuing
training courses'
Availability and use by state and
permittee of technical guidance
manuals
Percentage of total UIC budget
allocated for AOR
Percentage of permit application
backlogs attributable to AOR
AOR decision review processes
Most states presently do not compile
this data in a systematic fashion. Its
collection requires on site file re-
views and its interpretation requires
the subjective judgment of individuals
thoroughly familiar with both UIC re-
quirements and state environmental
program operations.
-------
(2) Approaches To Measuring State Success in Achieving
Objectives
Based on our understanding of groundwater and oil and gas
technology and consultations with Dr. Donald Warner, Professor
of Geological Engineering, University of Missouri, we developed
an approach to measuring state success in achieving primary and
secondary objectives of the AOR program. In formulating this
approach we also took into account the problems EPA historically
has faced in attempting to measure the direct relationship
between its programs and environmental protection. To fully
consider these problems, we reviewed an EPA report, compiled in
1975, entitled "First Report of the Environmental Measures
Project" which deals with the problems of measuring program
success.
As a result, we have concluded that it is difficult, if
not impossible, to measure, except in a limited way at great
cost, any of the substantive objectives of the AOR program.
For example, it often is impossible to relate groundwater
pollution in a given area to an injection well failure or, con-
versely, to show that proper construction or abandonment of a
particular well prevents groundwater degradation. Consequently,
the mid-course evaluation cannot test the underlying assumption
that improperly constructed or abandoned wells may serve as
conduits for groundwater contamination. There also are state-
of-the-art constraints in determining the success of correc-
tive action. While a mechanical integrity test can indicate
the need for or success of corrective action performed on an
active well,* there is no direct way to test whether an in-
active well is abandoned or reabandoned successfully. Instead,
one must rely on visual inspection of the wells and review
of the abandonment records. Since state agencies, themselves,
have few mechanisms available to ascertain the adequacy of
corrective action, it will be difficult, if not impossible,
for EPA to make an independent assessment.
Although direct measurement of substantive objectives
is not possible, EPA will be able to measure administrative
objectives such as a high level of program efficiency, technical
EPA has some concerns regarding the effectiveness of mechanical integrity
testing practices and has asked G&M and BASH to develop an approach to
evaluating those practices.
-5-
-------
quality, and compliance with the requirements of the regula-
tions. In addition, direct measurement of the administrative
objectives can serve as an indicator or surrogate measure of
substantive objectives. For example, if state programs are
not characterized by high levels of efficiency and technical
quality, it indicates that the program's substantive objectives
are not being fulfilled. Conversely, if states are successful
in meeting the administrative objectives, it is a reasonable
indication* that the program's objectives are being met,
Based on this reasoning, we developed indicators, such as a
reduction in the number of leaking wells or an increase in
percentage of wells reviewed in a state during the first two
years of the program, with which to measure those objectives.
Exhibit 1 presents a preliminary scheme for EPA to con-
sider in setting program objectives which will be measured by
the AOR evaluation. There are a total of seven objectives;
for each objective we provide a series of potential achieve-
ment indicators and define the types of data needed to measure
them. A brief assessment of the feasibility of their use is
included.
2. DESCRIPTION OF THE ALTERNATIVES
Our second step was to outline the key elements of each of the
three study alternatives. First, we established three study phases:
Planning
Data collection
Data aggregation and analysis.
For each alternative, we developed several approaches to the conduct
of each study phase. Based on project team meetings and consultation
with the EPA Project Officer, we selected for further consideration
those approaches most compatible with current EPA and state agency
practice and procedure. These approaches are summarized in Exhibit 2.
We have assumed that the AOR and corrective action concepts are scientifi-
cally sound. As noted above, EPA will not be able to test these assumptions
during the mid-course evaluation because of the difficulty of directly
measuring cause effect relationships.
-6-
-------
EXHIBIT 2
Overview of Options for Each Phase of
Study Alternatives I and II*
Phase Element
Alternative I
Alternative II
in
x
z
o
I
Data Types
Permit specific as enumerated in
146.24 and 146.25
Detailed technical and administra-
tive data on a sample of permit
applications; number of cases and
amount of- detail varies with re-
source commitments
Format Options
Permit by permit tabulations
Above combined with aggregate
program statistics
Above supplemented by explana-
tory narrative
State sends data enumerated in
146.24 and 146.25 plus accom-
panying narrative for each case
selected
EPA develops protocol to guide
on site case reviews
Collection Frequency
Quarterly, semi-annually,
annually, bi-annually
Annually or bi-annually
Collection Procedure
State reports to Region
State reports to ODW
ODW selects technical team which
reviews random sample ODW selects
smaller team to review limited,
special cases
Aggregation Levels
State, region, nationwide
(statistical)
Same as Alternative I (narrative)
E
U
2
VI
I
Analytic Approaches
Manual analysis
Canned or tailored computer program
Quantitative
Manual, qualitative plus
statistical
Types of Analysis
State specific
Comparisons among states
Comparisons among regions
Nationwide evaluation
Same as Alternative II
Analysts
Regions (preliminary)
ODW (partial or all)
Contractor assistance (partial or all)
Scientific advisory panel (review)
Same as Alternative II: probably
requires outside assistance
0.
Detailed planning recommendations for either alternative cannot be developed until EPA makes the following
decisions:
Selection of objectives to be measured
Estimate of available resources
Selection of study alternative
Selection of approach to each phase element EPA finalizes
Commitment of estimated resource/needs
Once EPA finalizes these decisions, ODW must:
Develop a study plan including a study tracking system
Appoint ODW Task Officer/work group
Initiate procurement process, if necessary
Initiate advisory panel appointments, if necessary
Develop and disseminate guidance to regions and states
The planning process for each alternative is similar, containing the same elements but
varying in complexity.
Alternative III essentially is an approach which combines
Alternatives I and II
-7-
-------
(1) Alternative I
Alternative I is an evaluation methodology that relies
on data collected, aggregated, and submitted by the states to
EPA. Section 146.25 of the UIC regulations enumerates the data
to be submitted by the states specifically in connection with
the mid-course evaluation. The regulations also provide EPA
with the option of preparing detailed guidance on state report-
ing requirements. In addition, EPA may find the reports sub-
mitted in compliance with the consolidated permittina reaulations
useful because they contain data which demonstrate the diligence
of state agencies in fostering operator compliance with AOR and
other UIC requirements. However, usefulness of these reports
is limited because they provide data on only a small number
of permittees and some of the data is not entirely pertinent
to the AOR evaluation. Exhibit 3 summarizes UIC reporting
requirements in terms of reporting frequency and content. The
following sections describe the data collection system, including
the types of data collected and the collection procedures; the
data analysis process, including methods for data aggregation
and analysis; and agency planning requirements.
1. Data Collection
In accordance with Section 146.25 of the UIC regula-
tions, all state-directed UIC programs are required to
submit the following data to EPA:
Information sufficient to identify the permittee
A map showing the applicable area of review,
the location of all wells within the area of
review
A tabulation of data describing all the wells
within the area of review which penetrate the
proposed injection zone
Appropriate geological data on the injection
zone and confining zones
Specified characteristics of potentially
affected underground sources of drinking water
Engineering drawings of the surface and sub-
surface construction details of the system
-8-
-------
EXHIBIT 3
Summary of Reporting Requirements Applicable
To Mid-Course Evaluation
Report Name
Authority
Report Preparer
Frequency
Content/Format
Quarterly
Noncompliance
Report
40 CFR 122.18(a)
All permit issuing
authorities; state
UIC Program Directors
or EPA Regional
Administrators (who-
ever has primacy)
Quarterly
Narrative report on
noncomplying permit-
tees. Includes back-
ground information on
permittee, description
of noncompliance and
Director's action
Annual
Noncompliance
Report
40 CFR 122.18(c)
I
'vo
I
All permit issuing
authorities; state
UIC Program Directors
or EPA Regional
Administrators (who-
ever has primacy)
Annual
Statistical report
on permittees reviewed,
noncompliance, enforce-
ment, permit modifica-
tions, and extension
of compliance deadlines
Program Report
40 CFR 122.18(c) (4) (i)
State administered
UIC programs
Annual
Narrative report
describing program
implementation and
program changes. In-
cludes an inventory of
active underground
injection activity
AOR Report
40 CFR 122.18(c)(4)(ii)
State administered
UIC programs
One time
Due within 3 months
of completion of
second full year of
operation of state
program
Data on each new
Class II permit and
area of review as
described in 40CFR
146.25 of UIC
regulations
-------
Corrective action proposed for and performed
on new injection wells
A demonstration of mechanical integrity sub-
mitted to .the State Director prior to permit
approval
Results of all mechanical integrity tests con-
ducted on existing wells and new (conversion
only) wells during the first two years of
operation
A temperature log on noise for a sample of
wells for which operators submitted cementing
records to demonstrate mechanical integrity.
The regulations afford EPA with an opportunity to
provide the states with guidance on the preparation of
the mid-course evaluation report. In order to assist
the Agency in taking advantage of this opportunity to
promote reporting consistency, we developed several
reporting format variations for EPA's consideration.
Eac.h approach varies according to the relative manpower
burden on EPA and the states. The option which places the
most extensive burden on EPA involves state submitted '
tabular displays of permit by permit decisions and
activities. It requires EPA to aggregate the data prior
to analysis and interpretation. A second option, under
which the states supplement the permit specific tabula-
tions with aggregate program statistics increases the state
level of effort but reduces that of EPA. Neither of these
first two approaches, however, provides EPA with an under-
standing of the administrative and management context
within which these activity levels occurred. Accordingly,
a third option is for EPA to require the states to include
additional information on the organization of the AOR
program, flowcharts of the AOR process, estimates of man-
hours and budget devoted to AOR, and relationship of the
AOR procedures to the overall permitting process. If
EPA elects to require the states to submit reports more
frequently than at the end of two years of program im-
plementation, it can ask the states to submit the adminis-
trative descriptions' with the first report. As in the
air quality program, subsequent reports address only
substantive changes in program administration.
-10-
-------
EPA also can select from among several procedural
approaches to data collection. For example, the approach
most consistent with current air and water program
practices is for states to submit their reports to the
appropriate EPA regional office. This is the approach
EPA will follow with regard to the quarterly and annual
UIC noncompliance reporting. Nevertheless, because of
the particular purpose for which EPA is collecting this
data, ODW may want to receive the data directly.
Finally, EPA already is aware of the need to consider
various options regarding the timing and frequency of the
data collection. The frequency options are numerous. The
Agency can require the states to submit one comprehensive
report after two years of implementation. Other options
include annual, semiannual and quarterly reporting. ODW
already recognizes that in selecting an approach it needs
to address the fact that not all programs will commence
on the same date. Accordingly, in conjunction with its
decision on reporting frequency, ODW may want to specify
uniform reporting dates.
2. Data Analysis
Like the data collection system, the data analysis
system consists of several elements. Principal ones
include data aggregation procedures, analysis, and
interpretation of the '.analysis. For each, we present
several alternatives for ODW to consider. Further develop-
ment of either Alternative I or Alternative III is contin-
gent on EPA selection from among these options.
EPA has considerable flexibility in structuring the
aggregation procedures. One decision EPA must make con-
cerns the desired level of aggregation. Choices include
aggregation by state, region, and nationwide. EPA may find
it useful to prepare all three types of data aggregation,
depending on the objectives EPA selects for measurement
and the available manpower and budget resources. Another
decision ODW must make is that of who will have responsi-
bility for aggregation and analysis. With respect to
aggregation, the level of aggregation EPA decides to per-
form inevitably affects who will perform it. For example,
-11-
-------
it would be appropriate for regional offices to pre-
pare statewide and regional data aggregation and perhaps
even preliminary analysis and interpretation.* However,
no single region would have the resources or perspective
to conduct cross-regional comparisons or nationwide analysis.
Thus, it is most appropriate for ODW to perform 'the nation-
wide data aggregation as well as the analysis and
interpretation.**
Once aggregated, the data can be correlated and their
statistical significance ascertained. This can be done
manually, by use of a computer package like SPSS, or by
use of a computer program developed specifically for the
mid-course evaluation. Manual manipulation of the data may
prove to be unwieldy and will limit the sophistication of
the analysis that may be performed.
ODW should conduct the final evaluation of the data
and develop conclusions on the effectiveness of the AOR
program. ODW can handle this task internally or by
appointing an advisory panel. This panel would be composed
of appropriate agency personnel and technical expertise
from outside the agency as ODW deems appropriate.
3. Planning
Although the UIC regulations specify the data to be
collected, EPA must make several administrative decisions
in order to implement Alternative I. Many of these
decisions are of a general nature; others pertain only to
a particular study phase.
The Agency first must decide who in ODW will be
responsible for the study. Regardless of who conducts
the various study tasks, ODW will have oversight since it
has responsibility for UIC rulemaking. From the objectives
we define in Section 1, ODW then should determine what the
study should accomplish and select accordingly from among
the various options we describe for each study phase.
Based on those decisions, ODW should direct the development
of a study plan which, includes a study management system.
One constraint on this Regional role would be if EPA elects to use computer
analysis. In order to minimize error and computer costs, the data file prep-
aration and analysis should not be scattered among several offices.
ODW can perform the work in-house or arrange for outside .assistance from
consultants, the National Academy of Science, or other research organizations.
If ODW elects to obtain outside assistance, the external organization also
can relieve the various Regions of aggregation or analysis chores.
-12-
-------
The Agency has great discretion for the
management of various facets of the study. As
noted above, one such area concerns the timing
and frequency of the reporting procedure. Reports
could be submitted every six months (four reports),
annually (two reports), or after, two years (one
report). The advantage of quarterly, semiannual,
or annual reporting is that it gives EPA an oppor-
tunity to correct any ambiguities or misunderstand-
ings in the reporting procedure as well as give
states feedback so that future reporting can pro-
ceed more smoothly. However, the processing of
the additional reports will constitute an added
administrative burden for the Agency. In addition,
it will require a system for tracking the study's
progress and status. The Agency also must consider
whether the dates for submittal of each state'.s
report should be based on when each state initiated
the program (different dates) or on when the
entire program was initiated (on one date). The
latter would allow EPA the additional perspective
of reviewing programs at different stages of '.
implementation, while the former would allow EPA
to compare programs at the same stage of develop-
ment. EPA should select the approach most in line
with its goals.
The Agency also has the option of disseminating
guidance and training to the states. It can com-
pile a policy memorandum or guidance manual to aid
the states in report preparation and to promote
consistency in state reporting. The manual could
include the following types of information: sample
report formats, guidelines on permit application
requirements, (for example, requiring permittees
to submit an extra copy of those documents required
for the mid-course evaluation) and definitions of
each data category. As part of its planning
effort, EPA needs to distribute the guidance ...
material in a timely fashion. It may be distributed
at a training session at the appropriate regional
offices, a procedure which would allow EPA to
answer any questions pertinent to the reporting.
After ODW finalizes the study plan, it needs
to promote the commitment of required manpower and
budget resources. This may involve designation
of an ODW Task Officer and/or work group, initia-
tion of the procurement process if outside assistance
-13-
-------
is needed, designation of regional staffs, and
position advertising if ODW or the regions require
new hires. Finally, if an EPA appointed advisory
committee is to have an important role in preparing
the final evaluation and recommendations it should
be appointed.
(2) Alternative II
Alternative II involves independent EPA collecti.on
of data on a random sample of permittees from each state,
Unlike Alternative I, this alternative is not addressed
in the UIC regulations. Consequently, we had to con-
sider how to develop this alternative in somewhat
greater detail than Alternative I.
Alternative II is described in a manner similar to
Alternative I. Data collection, data analysis, and
planning are considered separately below. As in Alter-
native I, if EPA wants us to develop this approach
further, the Agency must select from among the varia-
tions presented for each study phase.
1. Data Collection
Since Alternative II data collection is not
addressed in the UIC regulations, we had to develop
a scheme including the sample size, the data types
to be collected, the collection procedure, and the
timing of the data collection. We have developed
variations for all these elements except sample
size; this element will be considered in detail
should EPA select Alternative II or the Combination
Alternative.
Alternative II addresses the need for EPA to
appraise the rationale behind the state AOR
decisions. Successful implementation of an area
of review program requires many state agency per-
sonnel at various levels to make judgmental
decisions. Personnel involved in the AOR decision
process may include the enhanced recovery engineers
responsible for processing the permit and the field
supervisors responsible for implementing and
enforcing corrective actions. Consistency and equity
of the decision process is desirable and requires
decisions to be based on a solid technical basis.
The data collection process set forth below is one
which should permit EPA to evaluate both the
-14-
-------
management and technical capability of the UIC
staff and by inference the success in achieving
UIC program objectives.
The types of data which will be available
to EPA will vary according to the data collection
procedure employed. We have developed two such
procedures. The first approach is based on the
"ideal" situation where the availability of funding
and qualified personnel to conduct the study is
not constrained significantly. Recognizing that
current economic conditions and the attendant
attempts to curb government spending may result
in resource constraints, we discuss a second
approach which involves a more modest commitment
of outside technical assistance and a lower level
of effort on the part of EPA.
The ideal approach involves sending a highly
skilled technical team to each state agency to
review a sample of UIC permit cases. In order to
ensure a comprehensive and high quality data
collection effort, team members ideally should
have expertise in:
The UIC program
Local geohydrology
Enhanced recovery
Reservoir engineering
Well design, construction, logging,
and well service.
This team meets at specified frequencies
with state personnel including the enhanced
recovery or area engineers responsible for permit
processing, field supervisors responsible for
implementation of corrective action, and any
interested industrial operators. Each state would
present a number of permit cases, explaining the
rationale and technical basis for the AOR and
corrective action decisions. The state presents
the cases based on an EPA developed selection
procedure. The procedure may be a random or
stratified random one or may emphasize particular
problem or success cases.
Under this approach, state personnel and the
review team engage in a round table discussion.
This enables the review team to pose case specific
-15-
-------
questions. Following completion of the case
reviews, the team can develop a narrative report
for each case which reflects the review team's
judgment on the quality of the AOR and corrective
action decisions. Depending on the sample size,
the team may also prepare a statistically representa-
tive profile of state activities.
The second approach to Alternative II involves
a two-step process. First, EPA requires each
state to submit permit case files to headquarters
or the cognizant regional offices. Included in
the submissions are narratives detailing how the
state made its AOR and corrective action decisions.
These cases would be selected by EPA headquarters
or the Region randomly, or on the basis of certain
criteria such as age of well field or depth of well.
Based upon its review of the sample, EPA elects
one of three possible courses of action for each
state. If EPA has questions on whether a state's
AOR and corrective action decisions are technically
sound, EPA sends a technical team such as that
described in the first approach to the state agency
to conduct a more thorough review. Second, where
EPA has questions of a non-technical nature, for
instance, on the quality of a state's permitting
procedure, it sends a non-technical team composed
of one or more individuals knowledgeable of UIC
permitting rules to meet with the state. Finally,
if EPA is satisfied with a state's files, it
develops an evaluation without additional investiga-
tion.
The data collected under this second approach
is similar to but not identical to that collected
under the first approach. The report is similar
in format and content. The principal difference
is that the second approach relies more on a review
of the case files than on meetings with state per-
mitting personnel. One limitation associated with
this approach is that the reports will tend to be
based more on the state's interpretation of their
decision process than on an independent assessment
of the decision process. Nonetheless EPA may
decide to follow this approach because of the
reduced costs associated with applying a pre-
screening a procedure. Regardless of which approach
EPA selects, ODW must coordinate the entire data
.collection. However, there are several
-16-
-------
different alternatives for how ODW can handle
its responsibility. One approach is for ODW to
ask the regions to develop case selection criteria
and collect data from each state in their region.
Alternatively, ODW can develop the criteria for
case selection and the regions have responsibility
only for the data collection. The latter procedure
would make the case selections more consistent and
uniform. Another alternative would be for ODW to
do the entire data collection. This approach
would involve a greater level of effort for head-
quarters than the other two approaches described.
Finally, ODW could oversee a contractor or other
independent group in the collection of data.
Because Alternative II requires the skills of
specialized personnel, hiring a consultant or
soliciting the necessary skills from sources such
as universities, NAS, or other government agencies
may be the best of the alternatives described.
2. Data Analysis
We considered various aspects of the data
analysis for Alternative II including various
options for how the data should be aggregated,
who should perform the data analysis, and how
this analysis should be performed. Analyses of
the data for the two data collection alternatives
described will follow similar procedures.
The review teams are responsible for the
preparation of narrative reports for each of the
permit cases. These reports may be compiled into
state-by-state profiles. State profiles can be
aggregated further into regional or a national
profile.
ODW should oversee the data analysis, decide
what levels of comparison it wishes to make, .arid
then determine the level of aggregation required
and the appropriate aggregation procedures. ODW
may want the regions to be responsible for develop-
ment of regional and state profiles; ODW, however,
must oversee preparation of the national profile.
'Alternatively, if ODW requests contractor assistance,
the contractor can develop all aggregation levels
and conduct all comparisons and evaluations ODW
requests. Even under this approach, however, ODW
has responsibility for preparation of the study
-17-
-------
findings. It has the option of appointing an
Advisory Committee to review the findings and
any proposed recommendations.
In contrast to''the quantitative nature of
the data analysis for Alternative I, the data
analysis for Alternative II is qualitative in
nature. The analysis procedure for each'-state
involves the systematic review of the case reports
and the development of a set of observations or
conclusions for each case. These case specific
conclusions will form an assessment of the quality
of the state's AOR program and corrective action
decisions. They are not likely to provide exten-
sive overview information on the coverage of a
state program.
3. Planning
Similar to Alternative I this alternative
requires extensive planning by EPA. First, EPA
must select from among the alternative approaches
for each study element. Then, it must develop
and implement a study management procedure.
\
ODW, in order to coordinate the study, must
determine who will be responsible for each phase. '
The study's requirement for highly trained per-
sonnel not available at EPA may affect these
decisions. ODW may have to take steps to re-
cruit these personnel (such as soliciting
local geological expertise from state universities)
on a temporary basis if it conducts the study
in-house. Alternatively, ODW may have to initiate
the procurement process in order to hire a con-
sulting firm with the necessary skills. Both
the Civil Service and procurement processes are
time consuming.
In order to determine the need for new
personnel and/or consultants, ODW must determine
how many review teams it will send to the field.
One possibility is to use a team comprised of a
core group; local experts can supplement this
team as necessary. ODW also has the option of
using more than one core group for the study.
The number of review teams not only affects
resource decisions but also the timing of the
-18-
-------
evaluation. For example, if there were a review
team for each state, all of the reviews could occur
simultaneously. If only one team is used, the
review would have to be conducted on a staggered
basis.
After EPA selects the study approach, it
should develop criteria for the selection of
sample cases; detailed options for this procedure
can be developed if EPA elects Alternative II or
III. In addition, EPA heeds to develop a procedure
for the presentation of the cases at the review
team meetings, so that the reviews can be con-
ducted as consistently and quickly as possible.
This information could be provided to the states
in the form of a guidance manual, which would be
distributed at meetings at the various regional '
offices.
The Agency also should develop criteria for
team members to.use in conducting case reviews.
This guidance would contain EPA's judgment as to
what constitutes good AOR and corrective action
decisions. It may contain a checklist of impor- .
tant factors that should have been covered in the
decisions and how to weight a state's considera-
tion of these factors. Similar criteria should
be developed for the preliminary evaluation of
case files if that approach is selected.
(3) Alternative 'III
Alternative III essentially is a combination of
Alternatives I and II. EPA evaluates AOR on the basis
of the required state submitted data sind an EPA in-
depth data collection and analysis of a sample of
permittees. Because we already have described alter-
natives I and II in detail in preceding sections, we
will develop Alternative III only briefly below.
EPA has two potential options for the conduct of
Alternative III. First, it can commit itself from the
outset to the independent collection of data on a
random sample of permittees. Alternatively, it can
make an initial commitment only to conduct a prelimi-
nary evaluation based on Alternative I. If that evalua-
tion suggests problems in one or more state programs,
EPA then can determine the need for supplementary
data collection and analysis. Under this second
-19-
-------
approach, the supplementary data can be from a random
sample of selected cases. If EPA elects the second
approach to Alternative III, we will assist EPA in
determining what types of preliminary findings would
suggest the need for further data collection and
analysis. i
3. EVALUATION OF THE THREE ALTERNATIVES
In evaluating the three alternatives we considered two
principal factors. The first, overall effectiveness, in-
cludes a comparison of the data collected with the data
required to measure the program objectives. The second,
ease of implementation, includes a consideration of the EPA
and state level of effort associated with each alternative.
Our evaluation indicates that Alternative III rates
highest in terms of overall effectiveness. The state mid-
course evaluation report provides EPA with a good narrative
and statistical background on nationwide AOR activities.
In addition, the independent EPA data collection allows the
assessment of the technical and management quality of
decisionmaking necessary to measure state success in ful-
filling administrative objectives.
In terms of ease of implementation, however, Alterna-
tive III rates very low, primarily because of its potential
cost. Given current economic conditions and intensive
efforts to curb government spending, the Agency may be
unable at this time to make the resource commitment necessary
for Alternative III. Accordingly, we developed the varia-
tion presented previously under which EPA undertakes a
preliminary evaluation based on Alternative I and then
determines the need for additional data collection.
(.1) Overall Effectiveness
In order to compare the overall effectiveness
of the three alternatives, we first evaluated each
alternative with respect to how well the collected :
data fulfilled the data requirements to measure the
success of the AOR program set forth in Section 1.
We then considered other factors including the quality
of the data received, the extent of EPA's role in promot-
ing good data quality, and the limitations of each
alternative.
The collection and analysis of data under Alter-
native I should fulfill many of the factual and statis-
tical data requirements specified in Section 1. Such
-20-
-------
data as the number of AOR's undertaken and the types
of corrective actions required and taken will be
ascertained under Alternative I. Because the data
furnished will be at the discretion of the
states, the quality and extent of the data received
will depend on the guidance EPA disseminates. If EPA
requires the states to submit a detailed report con-
taining overview data, as suggested in the discussion
of Alternative I, the Agency can develop many generali-
zations about a state's program. It should enable the
Agency to assess factors such as the coverage and extent
of each state's AOR program; the administrative burden
on each state; and the difficulties associated with the
program's implementation. The main shortcoming asso-
ciated with Alternative I is that it provides no
mechanism for ascertaining the rationale behind each
AOR and corrective action decision. As was illustrated
in Exhibit 1, information of this type is necessary
to make a total assessment of the AOR program.
Because Alternative II involves the interaction
of qualified technical personnel with state permitting
authorities, EPA will be able to ascertain the -rationale
and quality of state's AOR and corrective action
decisions for a number of permit cases. This alterna-
tive will provide data presented in Exhibit 1 such
as the well conditions leading to a corrective action
decision or why the state allowed exceptions to
standard procedure for a particular permit
applicant. How well this information is docu-
mented will depend on how EPA defines the reporting
requirement. Since Alternative II is limited to the
consideration of only a sample of cases, it will not
provide statistical overview information or data on
the coverage of the AOR program which were defined in
Section 1 as necessary to evaluate an AOR program.
It may be possible to extrapolate this data from the
data collected, but the alternative will not facilitate
an exact measurement.
The data from Alternatives.''! and II are in many
ways complementary. Together, tney furnish all of the
data specified in Exhibit 1. Consequently, a combina-
tion alternative such as Alternative III, will be the
most effective way of achieving the goals of the mid-
course evaluation.
-21-
-------
(2) Ease of Implementation
In addition to the overall effectiveness of each
alternative, we also considered how easily each alter-
native could be implemented in practice. The most
important factors in this consideration were the levels
of effort on the part of both the EPA and the states
required for data collection and analysis.
The level of effort for both EPA and the states
varies according to the alternative EPA selects.
However, EPA has the option of hiring a consultant
to conduct all or some part of its role in the mid-
course evaluation. Thus EPA's efforts can be limited
only to administrative planning of the evaluation if
a contractor performs the analysis or they can be
quite extensive should the Agency decide to conduct
the study in-house. Directing the regional offices
to conduct the data collection and preliminary data
analysis for whichever alternative is chosen will
relieve much of the administrative burden on EPA
headquarters.
Of the three alternatives, Alternative I is the
easiest to implement. It requires only a modest level
of effort and, therefore, resource commitment by EPA,
and the data collection approach is consistent with
other current reporting practices such .as those under
the air quality program.
Under Alternative I, the state's effort on data
collection is somewhat greater than that of EPA.
Because the regulations enumerate data categories,
the Agency's role on this phase primarily is to develop
and disseminate guidelines to promote reporting con-
sistency. State staffs, on the other hand, devote
considerable time to completion of the required mid-
course reports.
Data analysis for Alternative I will require more
effort on EPA's part than the data collection. How-
ever, the level of effort will vary according to the
level of aggregation EPA selects, and the quality and com-
pleteness of state submittals. Both manual and com-
puter evaluation will require considerable effort.
Regardless of whether EPA obtains outside assistance;
ODW still must devote a significant amount of atten-
tion to reviewing the preliminary findings, making
policy decisions, and preparing recommendations for
Agency consideration.
-22-
-------
Alternative II should be more difficult to imple-
ment primarily because it requires both EPA and the
states to devote considerably greater effort than on
Alternative I. Unlike Alternative I, it requires EPA
to take an active role in data collection. This role
includes selection of a sample, development of a
protocol for conducting case reviews, and participa-
tion in the reviews. The state role also is greater.
It still must comply with the UIC requirement for: a
mid-course report. In addition, the staff must
participate in time consuming case reviews and must be
available to assist the study team in conducting file
reviews.
At the same time, another factor affecting the
implementation of Alternative II may be the availability
of qualified personnel to perform the study. Recruiting
of individuals qualified to perform the study, especially
persons knowledgeable in local or regional geological
conditions, may prove to be a cumbersome task. Addition-
ally, the availability of such personnel may be limited.
Analysis of Alternative II data may be somewhat
less burdensome than analysis of data from Alternative
I. The collected data is not likely to be amenable
to statistical manipulation; hence, the data analysis
effort will involve expert judgment.
If Alternative III is adopted it will require
the combined levels of effort necessary to perform
Alternatives I and II. Consequently, Alternative III
is expected to represent a large administrative bur-
den for EPA. For this reason, EPA may want to consider
carefully the variation we developed to Alternative
III, where Alternative I data collection is conducted
first, based upon which a decision to conduct Alterna-
tive II or an abridged version of Alternative II is
made. This variation should be more effective than
Alternatives I or II alone in terms of measuring the
goals of the AOR program. It should require a lower
level of effort than a full-scale Alternative III.
-23-
------- |