&EPA
                  68-01-6425
       MIDCOURSE EVALUATION  OF
            AREA OF REVIEW:
      EVALUATION OF ALTERNATIVE
         STUDY  METHODOLOGIES
                  SUBMITTED TO
                 DR. JENTAI YANG
              OFFICE OF DRINKING WATER
          U.S. ENVIRONMENTAL PROTECTION AGENCY
                  JANUARY 1983
                         BOOZ.ALLEN & HAMILTON, INC.
                          UNDER THE DIRECTION OF
                          GERAGHTY & MILLER, INC.

-------
SEPA
                  68-01-6425
       MIDCOURSE  EVALUATION OF
            AREA OF REVIEW:
     EVALUATION OF ALTERNATIVE
        STUDY METHODOLOGIES
                 SUBMITTED TO
                 DR. JENTAI YANG
             OFFICE OF DRINKING WATER
          U.S. ENVIRONMENTAL PROTECTION AGENCY
                 JANUARY 1983
                         BOOZ.ALLEN & HAMILTON, INC.
                          UNDER THE DIRECTION OF
                          GERAGHTY & MILLER, INC.

-------
                          ACKNOWLEDGEMENT
     This report was prepared by Booz, Allen & Hamilton, Inc., under
the direction of Geraghty & Miller, Inc.  The Booz, Allen project
manager was Dr. Joanne Wyman.  She was assisted by Mr. Walter Mardis
of Booz, Allen.  The Geraghty & Miller project manager was Mr.
William Thompson, and Mr. Roger Anzzolin was the EPA task monitor.

-------
            TABLE   OF   CONTENTS
                                                    Page
                                                   Number
1.    OBJECTIVES OF AREA OF REVIEW                     2
                        x

2.    DESCRIPTION OF THE'ALTERNATIVES                  6


3.    EVALUATION OF THE THREE ALTERNATIVES            20

-------
        LIST   OF   EXHIBITS
                                               Page
                                              Number
Definition and Measurement of AOR Program
Objectives

Overview of Options for Each Phase of
Study Alternatives I and II

Summary of Reporting Requirements
Applicable To Mid-Course Evaluation
                       11

-------
              MID-COURSE EVALUATION OF AREA OF REVIEW:
            EVALUATION OF ALTERNATIVE STUDY METHODOLOGIES
     EPA has issued Underground Injection Control (UIC)  and Consoli-
dated Permitting regulations that require operators of new oil and
gas wells to establish an Area of Review (AOR) defined as the
radial distance within which existing active, abandoned, or dry
wells penetrating the zone of injection must be checked.  The ob-
jective of the review is to identify and correct wells which are
potential conduits for groundwater contaimination.  Because the
final rules differ substantially from those proposed in 1976, the
Agency has agreed to undertake a "mid-course evaluation" in order
to evaluate the AOR program's costs and benefits.

     The Office of Drinking Water (ODW)  has asked Geraghty & Miller
(G&M) and Booz, Allen & Hamilton (BA&H)  to assist it in designing
the mid-course evaluation study be describing and evaluating three
alternative study methodologies.  The Agency has defined these al-
ternatives in terms of the data collection effort:

           Alternative I (State-supplied data);  Each state operated
           program submits the data specified in 146.24 and 146.25
           of the UIC regulations to the cognizant EPA regional
           office.

           Alternative II (EPA-conducted random survey);.EPA con-
           ducts detailed reviews of selcted permit cases.

           Alternative III (Combination of I and II);   EPA supple-
           ments its analysis of the state submitted data with an
           analysis of EPA collected data.

     Our approach to conduting this assingment consisted of the
following.three steps.  First, we established a baseline against
which to evaluate the three alternatives.  This involved defining
the substantive and management objectives of the AOR program, iden-
tifying effectiveness indicators, and determining the types of data
needed to apply the indicators.  Second, we developed descriptions
of each alternative.  Although our project team meetings generated
numerous variations on the principal steps in each alternative,
we discuss in this technical report only those variations which
appear to be compativle with current Federal and state environ-
mental program practices.  In the next step we evaluated each
alternative with respect to its overall effectiveness and ease of
implementation in measuring area of review objectives.
                               -1-

-------
     Based on our evaluation of the alternatives, we reached two
conclusions.  First, the combination option provided EPA with
the most comprehensive and best quality evaluation.  It enables
EPA to evaluate the quality of a state's permit decision process
by conduting on site interviews with permitting personnel, and
enables the Agency to evaluate that data within the context of
the aggregate and permit specific data the states must submit in
accordance with the UIC rules.

     As we suggest in the final section of the memorandum, how-
ever, Alternative III is potentially quite costly .   Recognizing
that the Agency mya not.want to make such a high resource commit-
ment, we have defined another alternative.  First, EPA could con-
duct a preliminary evaluation based on Alternative I.  Based on
the results of that evaluation, EPA could determined the need for
an independent survey.

1.     OBJECTIVES OF AREA OR REVIEW

       The purpose of the mid-course evaluation is to reassess the
need for and the benefits of the AOR program.  Based on this re-
assessment, EPA could determined whether to broaden or reduce the
scope of the AOR requirements.

       Our first step in assisting EPA in the selection of a mid-
course evaluation study methodology was to define the AOR program
objectives and develop methods and approaches suitable for measuring
state agency success in fulfilling those objectives.  We then identi-
fied sets of achievement indicators appropriate for conducting these
measurements.  Exhibit i presents an overview of the AOR program
objectives and the achievement indicators, together with types of
data required to measure those objectives.

      (1)   Definition of Objectives

           We first defined primary and secondary objectives of the
      AOR program.   Primary,  or substantive,  objectives address the
      direct intent of the SDWA:   the prevention of groundwater
      contamination.   Secondary objectives such as the minimization
      of permitting costs and processing delays also are important
      because the quality of a state's permit-decision making process
      is a reflection of the program's efficiency and effectiveness.
      Additionally, the purpose of the revisions to the 1976 version
      of the UIC rules was to reduce costs and delays.   The measurement
      of both sets of objectives provides a good indication of how
      effectively a state implemented AOR requirements.  The defini-
      tions of the specific objectives presented in Exhibit 1
      reflect consultations with EPA, a review of the SDWA and the
      history of its regulatory development including the research
      conducted in support of the rulemaking.
                               -2-

-------
                                                                           EXHIBIT 1
                                        Definition and  Measurement of AOR  Program  Objectives
          AOR PROGRAM OBJECTIVE
                                           ACHIEVEMENT INDICATORS
                                                                                     DATA REQUIREMENTS
                                                                                      FOR MEASUREMENT
                                                                                         ASSESSMENT OF
                                                                                   •MEASUREMENT FEASIBILITY
        Protection of current/
        potential underground
        sources of drinking
        water
Direct measurement of prevention
or abatement of  contamination is
virtually technically impossible.
State success in fulfilling the
other objectives listed serves as
an indirect indicator of effective-
ness with respect to this objective
See data requirements  for measuring
other objectives.
                                    See assessments of feasibility of
                                    obtaining data required to measure
                                    other objectives
 I
Ul
 I
        Correction of wells
        identified as posing a
        potential threat to
        groundwater
Direct indicators  such as reduction
in contamination from poorly com-
pleted or abandoned  wells are
difficult,  if not  impossible, to
measure

Indirect indicators  include:

   Identification  of all potentially
   threatening wells

   Development of  corrective action
   plan for all threatening wells

   Completion of all needed
   corrective action
   percentage of  wells penetrating
   injection zone selected  for
   corrective action

   Percentage of  state approved,
   discontinued corrective  actions

   Rationale for  corrective action
   decisions

   Number of confirmed,  successful,
   completed corrective  actions
                                    Collection of the statistical data  is
                                    easy;  however,  its evaluation,  in terms
                                    of the technical soundness  of correc-
                                    tive action decisions,  requires an
                                    independent,  on site  investigation.
         Ultimate review of all
         wells which pose a
         potential threat to
         groundwater
Extent of overlap of AOR of
first and second year permit
applicants
Overlays or other graphic displays
showing existing wells  and AOR
for new well applicants
                                    Easy to measure since states  will
                                    submit permit specific maps to  EPA
                                    which are amenable to aggregation.
        Compliance by state
        agencies with technical
        standards and adminis-
        trative procedures of
        AOR regulations
   Consistent,  equitable  imposi-
   tion of corrective action
   requirements

   Detection of non-compliers

   Enforcement  against  non-
   compliers

   Submittal of accurate, complete
   records in mid-course  report
Comparison of corrective action
plans for each permittee

Number of applicants  for which
state approved discontinuation
of corrective action

Case specific flowcharts of
application of AOR requirements

Number of total and permittee
specific inspections

Number of violations  detected

Number of each type of  enforce-
ment action taken
                                       Requires considerable on site file
                                       examination and interviews with  agency
                                       staff; involves judgment on the  part
                                       of evaluation team,  thereby necessi-
                                       tating commitment of a team with
                                       extensive technical  and program
                                       management expertise.

-------
                                                         EXHIBIT  1  (Continued)
  AOR PROGRAM OBJECTIVES
                                    ACHIEVEMENT INDICATORS
                                                                             DATA REQUIREMENTS
                                                                              FOR MEASUREMENT
                                                                                       ASSESSMENT  OF
                                                                                  MEASUREMENT FEASIBILITY
Minimization of adverse
impacts on energy resource
development
Absence of increases in the
typical time taken to bring  a
new well or conversion on-line
attributable to AOR

Absence of increases in project
costs associated with AOR

Absence of production losses
attributable to AOR
Time frames for permit applica-
tion preparation

Time frames for permit processing

Manhours and costs associated with

-  Defining AOR and corrective
   action

   Implementing corrective  action
Data on permittee manhours, cost, and
time frame for permit application is
difficult to collect.  It requires
operator recordkeeping not consistent
with current practice.  If EPA can
obtain the data, it can compare it
with estimates prepared in support of
the rulemaking.

Data on time frames for permit processing
requires laborious on site review of
state files to identify delays
attributable solely to AOR.
Maximization of management
and technical quality of
AOR decisions

Minimization of adverse
resource impacts on
state
Adequacy of manpower and budget
resources, including staff
stability

Adequacy of staff training

Timeliness of task completion

Consistency of requirements
imposed on permittees
Budget increases/decreases

Staff increases/decreases

Staff turnover rates

Hiring/promotion standards

Staff education and experience

Availability and use of continuing
training courses'

Availability and use by state and
permittee of technical guidance
manuals

Percentage of total UIC budget
allocated for AOR

Percentage of permit application
backlogs attributable to AOR

AOR decision review processes
Most states presently do not compile
this data in a systematic fashion.  Its
collection requires on site file re-
views and its interpretation requires
the subjective judgment of individuals
thoroughly familiar with both UIC re-
quirements and state environmental
program operations.

-------
(2)   Approaches To Measuring State Success in Achieving
     Objectives

     Based on our understanding of groundwater and oil and  gas
technology and consultations with Dr. Donald Warner, Professor
of Geological Engineering, University of Missouri, we developed
an approach to measuring state success in achieving primary and
secondary objectives of the AOR program.  In formulating  this
approach we also took into account the problems EPA historically
has faced in attempting to measure the direct relationship
between its programs and environmental protection.  To  fully
consider these problems, we reviewed an EPA report, compiled in
1975, entitled "First Report of the Environmental Measures
Project" which deals with the problems of measuring program
success.

     As a result, we have concluded that it is difficult, if
not impossible, to measure, except in a limited way at  great
cost, any of the substantive objectives of the AOR program.
For example, it often is impossible to relate groundwater
pollution in a given area to an injection well failure or,  con-
versely, to show that proper construction or abandonment  of a
particular well prevents groundwater degradation.  Consequently,
the mid-course evaluation cannot test the underlying assumption
that improperly constructed or abandoned wells may serve  as
conduits for groundwater contamination.  There also are state-
of-the-art constraints in determining the success of correc-
tive action.  While a mechanical integrity test can indicate
the need for or success of corrective action performed on an
active well,* there is no direct way to test whether an in-
active well is abandoned or reabandoned successfully.   Instead,
one must rely on visual inspection of the wells and review
of the abandonment records.  Since state agencies, themselves,
have few mechanisms available to ascertain the adequacy of
corrective action, it will be difficult, if not impossible,
for EPA to make an independent assessment.

     Although direct measurement of substantive objectives
is not possible, EPA will be able to measure administrative
objectives such as a high level of program efficiency, technical
EPA has some concerns regarding the effectiveness of mechanical integrity
testing practices and has asked G&M and BASH to develop an approach to
evaluating those practices.
                            -5-

-------
      quality, and  compliance  with the requirements of the regula-
      tions.  In addition, direct  measurement of the administrative
      objectives can serve as  an indicator or surrogate measure of
      substantive objectives.  For example,  if state programs are
      not characterized by high levels of  efficiency and technical
      quality, it indicates  that the  program's substantive objectives
      are not being fulfilled.  Conversely,  if states are successful
      in meeting the administrative objectives,  it is a reasonable
      indication* that the program's  objectives are being met,
      Based on this reasoning, we  developed indicators, such as a
      reduction in  the number  of leaking wells or an increase in
      percentage of wells reviewed in a state during the first two
      years of the program,  with which to  measure those objectives.

           Exhibit  1 presents  a preliminary scheme for EPA to con-
      sider in setting program objectives  which will be measured by
      the AOR evaluation.  There are  a total of seven objectives;
      for each objective we  provide a series of potential achieve-
      ment indicators and define the  types of data needed to measure
      them.  A brief assessment of the feasibility of their use is
      included.

2.    DESCRIPTION OF THE ALTERNATIVES

      Our second step was to outline  the key elements of each of the
three study alternatives.  First,  we  established three study phases:

           Planning
           Data collection
           Data aggregation  and analysis.

For each alternative, we developed several approaches to the conduct
of each study phase.  Based  on project team meetings and consultation
with the EPA Project Officer,  we selected  for further consideration
those approaches most compatible with current EPA and state agency
practice and procedure.  These approaches  are summarized in Exhibit 2.
      We have assumed that the AOR and corrective action concepts are scientifi-
      cally sound.  As noted above, EPA will not be able to test these assumptions
      during the mid-course evaluation because of the difficulty of directly
      measuring cause effect relationships.
                                -6-

-------
                                                EXHIBIT  2
                          Overview of  Options  for  Each  Phase  of
                                 Study  Alternatives  I and  II*
          Phase Element
                                           Alternative I
                                                                                 Alternative II
in
x
z
o
I
    Data  Types
                             Permit specific  as enumerated in
                             146.24 and 146.25
                                       Detailed technical and administra-
                                       tive data on  a  sample of permit
                                       applications; number of cases and
                                       amount  of- detail varies with re-
                                       source  commitments
Format Options
Permit by permit  tabulations

Above combined with aggregate
program statistics

Above supplemented by explana-
tory narrative
State sends data enumerated in
146.24 and 146.25 plus accom-
panying narrative for each case
selected

EPA develops protocol to  guide
on site case reviews
Collection Frequency
Quarterly,  semi-annually,
annually,  bi-annually
Annually or bi-annually
    Collection  Procedure
                             State reports to  Region

                             State reports to  ODW
                                       ODW selects  technical team which
                                       reviews  random sample ODW selects
                                       smaller  team to review limited,
                                       special  cases
    Aggregation  Levels
                             State, region,  nationwide
                             (statistical)
                                       Same as Alternative  I  (narrative)
E
U
2
VI
I
Analytic Approaches
Manual analysis
Canned or tailored computer program
Quantitative
Manual, qualitative plus
statistical
    Types of Analysis
                             State specific
                             Comparisons among states
                             Comparisons among regions
                             Nationwide evaluation
                                                                        Same as  Alternative II
    Analysts
                             Regions (preliminary)
                             ODW (partial or all)
                             Contractor assistance  (partial or all)
                             Scientific advisory panel  (review)
                                       Same as Alternative II:  probably
                                       requires outside assistance
0.
Detailed planning  recommendations for either alternative cannot be developed until  EPA makes the following
decisions:

          Selection of objectives to be measured
          Estimate of available resources
          Selection of study alternative
          Selection of approach to each phase element EPA finalizes
          Commitment of estimated resource/needs

Once EPA finalizes these decisions, ODW must:

          Develop a  study  plan including a study tracking system
          Appoint ODW Task Officer/work group
          Initiate procurement process, if necessary
          Initiate advisory panel appointments, if necessary
          Develop and disseminate guidance to regions and states

The planning process for each alternative is similar, containing the same elements  but
varying in complexity.
         Alternative III essentially is  an approach which combines
         Alternatives I and II
                                                     -7-

-------
(1)  Alternative I

     Alternative I is an evaluation methodology that relies
on data collected, aggregated, and submitted by the states to
EPA.  Section 146.25 of the UIC regulations enumerates the data
to be submitted by the states specifically in connection with
the mid-course evaluation.  The regulations also provide EPA
with the option of preparing detailed guidance on state report-
ing requirements.  In addition, EPA may find the reports sub-
mitted in compliance with the consolidated permittina reaulations
useful because they contain data which demonstrate the diligence
of state agencies in fostering operator compliance with AOR and
other UIC requirements.  However, usefulness of these reports
is limited because they provide data on only a small number
of permittees and some of the data is not entirely pertinent
to the AOR evaluation.  Exhibit 3 summarizes UIC reporting
requirements in terms of reporting frequency and content.  The
following sections describe the data collection system, including
the types of data collected and the collection procedures; the
data analysis process, including methods for data aggregation
and analysis; and agency planning requirements.

     1.   Data Collection

          In accordance with Section 146.25 of the UIC regula-
     tions, all state-directed UIC programs are required to
     submit the following data to EPA:

               Information sufficient to identify the permittee

               A map showing the applicable area of review,
               the location of all wells within the area of
               review

               A tabulation of data describing all the wells
               within the area of review which penetrate the
               proposed injection zone

               Appropriate geological data on the injection
               zone and confining zones

               Specified characteristics of potentially
               affected underground sources of drinking water

               Engineering drawings of the surface and sub-
               surface construction details of the system
                           -8-

-------
                                                       EXHIBIT 3
                                 Summary of Reporting  Requirements Applicable
                                             To Mid-Course Evaluation
           Report Name
        Authority
   Report Preparer
      Frequency
   Content/Format
      Quarterly
      Noncompliance
      Report
40 CFR 122.18(a)
All permit issuing
authorities;  state
UIC Program Directors
or EPA Regional
Administrators (who-
ever has primacy)
Quarterly
Narrative report on
noncomplying permit-
tees.  Includes back-
ground information on
permittee, description
of noncompliance and
Director's action
      Annual
      Noncompliance
      Report
40 CFR 122.18(c)
 I
'vo
 I
All permit issuing
authorities;  state
UIC Program Directors
or EPA Regional
Administrators (who-
ever has primacy)
Annual
Statistical report
on permittees reviewed,
noncompliance, enforce-
ment, permit modifica-
tions, and extension
of compliance deadlines
      Program Report
40 CFR 122.18(c) (4) (i)
State administered
UIC programs
Annual
Narrative report
describing program
implementation and
program changes.  In-
cludes an inventory of
active underground
injection activity
      AOR Report
40 CFR 122.18(c)(4)(ii)
State administered
UIC programs
One time
Due within 3 months
of completion of
second full year of
operation of state
program
Data on each new
Class II permit and
area of review as
described in 40CFR
146.25 of UIC
regulations

-------
          Corrective action proposed for and performed
          on new injection wells

          A demonstration of mechanical integrity sub-
          mitted to .the State Director prior to permit
          approval

          Results of all mechanical integrity tests con-
          ducted on existing wells and new (conversion
          only) wells during the first two years of
          operation

          A temperature log on noise for a sample of
          wells for which operators submitted cementing
          records to demonstrate mechanical integrity.

     The regulations afford EPA with an opportunity to
provide the states with guidance on the preparation of
the mid-course evaluation report.  In order to assist
the Agency in taking advantage of this opportunity to
promote reporting consistency, we developed several
reporting format variations for EPA's consideration.
Eac.h approach varies according to the relative manpower
burden on EPA and the states.  The option which places the
most extensive burden on EPA involves state submitted  '
tabular displays of permit by permit decisions and
activities.  It requires EPA to aggregate the data prior
to analysis and interpretation.  A second option, under
which the states supplement the permit specific tabula-
tions with aggregate program statistics increases the state
level of effort but reduces that of EPA.  Neither of these
first two approaches, however, provides EPA with an under-
standing of the administrative and management context
within which these activity levels occurred.  Accordingly,
a third option is for EPA to require the states to include
additional information on the organization of the AOR
program, flowcharts of the AOR process, estimates of man-
hours and budget devoted to AOR, and relationship of the
AOR procedures to the overall permitting process.  If
EPA elects to require the states to submit reports more
frequently than at the end of two years of program im-
plementation, it can ask the states to submit the adminis-
trative descriptions' with the first report.  As in the
air quality program, subsequent reports address only
substantive changes in program administration.
                      -10-

-------
     EPA also can select from among several procedural
approaches to data collection.  For example, the approach
most consistent with current air and water program
practices is for states to submit their reports to the
appropriate EPA regional office.  This is the approach
EPA will follow with regard to the quarterly and annual
UIC noncompliance reporting.  Nevertheless, because of
the particular purpose for which EPA is collecting this
data, ODW may want to receive the data directly.

     Finally, EPA already is aware of the need to consider
various options regarding the timing and frequency of the
data collection.  The frequency options are numerous.  The
Agency can require the states to submit one comprehensive
report after two years of implementation.  Other options
include annual, semiannual and quarterly reporting.  ODW
already recognizes that in selecting an approach it needs
to address the fact that not all programs will commence
on the same date.  Accordingly, in conjunction with its
decision on reporting frequency, ODW may want to specify
uniform reporting dates.

2.   Data Analysis

     Like the data collection system, the data analysis
system consists of several elements.  Principal ones
include data aggregation procedures, analysis, and
interpretation of the '.analysis.  For each, we present
several alternatives for ODW to consider.  Further develop-
ment of either Alternative I or Alternative III is contin-
gent on EPA selection from among these options.

     EPA has considerable flexibility in structuring the
aggregation procedures.  One decision EPA must make con-
cerns the desired level of aggregation.  Choices include
aggregation by state, region, and nationwide.  EPA may find
it useful to prepare all three types of data aggregation,
depending on the objectives EPA selects for measurement
and the available manpower and budget resources.  Another
decision ODW must make is that of who will have responsi-
bility for aggregation and analysis.  With respect to
aggregation, the level of aggregation EPA decides to per-
form inevitably affects who will perform it.  For example,
                      -11-

-------
     it would be  appropriate for regional offices  to  pre-
     pare statewide  and regional data aggregation  and perhaps
     even preliminary analysis and interpretation.*   However,
     no single region would have the resources or  perspective
     to conduct cross-regional comparisons or nationwide analysis.
     Thus, it is  most appropriate for ODW to perform  'the nation-
     wide data aggregation as well as the analysis and
     interpretation.**

          Once aggregated, the data can be correlated and their
     statistical  significance ascertained.  This can  be done
     manually, by use of a computer package like SPSS,  or by
     use of a computer program developed specifically for the
     mid-course evaluation.  Manual manipulation of the data may
     prove to be  unwieldy and will limit the sophistication of
     the analysis that may be performed.

          ODW should conduct the final evaluation  of  the data
     and develop  conclusions on the effectiveness  of  the AOR
     program.  ODW can handle this task internally or by
     appointing an advisory panel.  This panel would  be composed
     of appropriate  agency personnel and technical expertise
     from outside the agency as ODW deems appropriate.

     3.   Planning

          Although the UIC regulations specify the data to be
     collected, EPA  must make several administrative  decisions
     in order to  implement Alternative I.  Many of these
     decisions are of a general nature; others pertain only to
     a particular study phase.

          The Agency first must decide who in ODW  will be
     responsible  for the study.  Regardless of who conducts
     the various  study tasks, ODW will have oversight since it
     has responsibility for UIC rulemaking.  From  the objectives
     we define in Section 1, ODW then should determine what the
     study should accomplish and select accordingly from among
     the various  options we describe for each study phase.
     Based on those  decisions, ODW should direct the  development
     of a study plan which, includes a study management system.
One constraint on this Regional  role would be if EPA elects to use computer
analysis.   In order to minimize  error and computer costs, the data file prep-
aration and analysis should not  be scattered among several offices.

ODW can perform the work in-house or arrange for outside .assistance from
consultants, the National Academy of Science, or other research organizations.
If ODW elects to obtain outside  assistance, the external organization also
can relieve the various Regions  of aggregation or analysis chores.


                                -12-

-------
     The Agency has great discretion for the
management of various facets of the study.  As
noted above, one such area concerns the timing
and frequency of the reporting procedure.  Reports
could be submitted every six months (four reports),
annually (two reports), or after, two years (one
report).  The advantage of quarterly,  semiannual,
or annual reporting is that it gives EPA an oppor-
tunity to correct any ambiguities or misunderstand-
ings in the reporting procedure as well as give
states feedback so that future reporting can pro-
ceed more smoothly.  However, the processing of
the additional reports will constitute an added
administrative burden for the Agency.   In addition,
it will require a system for tracking the study's
progress and status.  The Agency also must consider
whether the dates for submittal of each state'.s
report should be based on when each state initiated
the program (different dates) or on when the
entire program was initiated (on one date).   The
latter would allow EPA the additional perspective
of reviewing programs at different stages of '.
implementation, while the former would allow EPA
to compare programs at the same stage of develop-
ment.  EPA should select the approach most in line
with its goals.

     The Agency also has the option of disseminating
guidance and training to the states.  It can com-
pile a policy memorandum or guidance manual to aid
the states in report preparation and to promote
consistency in state reporting.  The manual could
include the following types of information:   sample
report formats, guidelines on permit application
requirements,   (for example, requiring permittees
to submit an extra copy of those documents required
for the mid-course evaluation) and definitions of
each data category.  As part of its planning
effort, EPA needs to distribute the guidance ...
material in a timely fashion.  It may be distributed
at a training session at the appropriate regional
offices, a procedure which would allow EPA to
answer any questions pertinent to the reporting.

     After ODW finalizes the study plan, it needs
to promote the commitment of required manpower and
budget resources.  This may involve designation
of an ODW Task Officer and/or work group, initia-
tion of the procurement process if outside assistance
                 -13-

-------
     is needed, designation of regional staffs, and
     position advertising if ODW or the regions require
     new hires.  Finally, if an EPA appointed advisory
     committee is to have an important role in preparing
     the final evaluation and recommendations it should
     be appointed.

(2)  Alternative II

     Alternative II involves independent EPA collecti.on
of data on a random sample of permittees from each state,
Unlike Alternative I, this alternative is not addressed
in the UIC regulations.  Consequently, we had to con-
sider how to develop this alternative in somewhat
greater detail than Alternative I.

     Alternative II is described in a manner similar to
Alternative I.  Data collection, data analysis, and
planning are considered separately below.  As in Alter-
native I, if EPA wants us to develop this approach
further, the Agency must select from among the varia-
tions presented for each study phase.

     1.   Data Collection

          Since Alternative II data collection is not
     addressed in the UIC regulations, we had to develop
     a scheme including the sample size, the data types
     to be collected, the collection procedure, and the
     timing of the data collection.  We have developed
     variations for all these elements except sample
     size;  this element will be considered in detail
     should EPA select Alternative II or the Combination
     Alternative.

          Alternative II addresses the need for EPA to
     appraise the rationale behind the state AOR
     decisions.  Successful implementation of an area
     of review program requires many state agency per-
     sonnel at various levels to make judgmental
     decisions.  Personnel involved in the AOR decision
     process may include the enhanced recovery engineers
     responsible for processing the permit and the field
     supervisors responsible for implementing and
     enforcing corrective actions. Consistency and equity
     of the decision process is desirable and requires
     decisions to be based on a solid technical basis.
     The data collection process set forth below is one
     which should permit EPA to evaluate both the
                      -14-

-------
management and technical capability of the UIC
staff and by inference the success in achieving
UIC program objectives.

     The types of data which will be available
to EPA will vary according to the data collection
procedure employed.  We have developed two such
procedures.  The first approach is based on the
"ideal" situation where the availability of funding
and qualified personnel to conduct the study is
not constrained significantly.  Recognizing that
current economic conditions and the attendant
attempts to curb government spending may result
in resource constraints, we discuss a second
approach which involves a more modest commitment
of outside technical assistance and a lower level
of effort on the part of EPA.

     The ideal approach involves sending a highly
skilled technical team to each state agency to
review a sample of UIC permit cases.  In order to
ensure a comprehensive and high quality data
collection effort, team members ideally should
have expertise in:

          The UIC program
          Local geohydrology
          Enhanced recovery
          Reservoir engineering
          Well design, construction, logging,
          and well service.

     This team meets at specified frequencies
with state personnel including the enhanced
recovery or area engineers responsible for permit
processing, field supervisors responsible for
implementation of corrective action, and any
interested industrial operators.  Each state would
present a number of permit cases, explaining the
rationale and technical basis for the AOR and
corrective action decisions.  The state presents
the cases based on an EPA developed selection
procedure.  The procedure may be a random or
stratified random one or may emphasize particular
problem or success cases.

     Under this approach, state personnel and the
review team engage in a round table discussion.
This enables the review team to pose case specific
                 -15-

-------
 questions.   Following  completion  of  the  case
 reviews,  the team  can  develop  a narrative  report
 for  each  case which  reflects the  review  team's
 judgment  on  the quality  of  the AOR and corrective
 action  decisions.  Depending on the  sample size,
 the  team  may also  prepare a statistically  representa-
 tive profile of state  activities.

      The  second approach to Alternative  II involves
 a  two-step process.  First, EPA requires each
 state to  submit permit case files to headquarters
 or the  cognizant regional offices.   Included in
 the  submissions are  narratives detailing how the
 state made its AOR and corrective action decisions.
 These cases  would  be selected  by  EPA headquarters
 or the  Region randomly, or on the  basis of  certain
 criteria  such as age of  well field or depth of well.
 Based upon its review  of the sample, EPA elects
 one  of  three possible  courses  of  action  for each
 state.  If EPA has questions on whether  a  state's
 AOR  and corrective action decisions  are  technically
 sound,  EPA sends a technical team such as  that
 described in the first approach to the state agency
 to conduct a more  thorough  review.   Second, where
 EPA  has questions  of a non-technical nature, for
 instance, on the quality of a  state's permitting
 procedure, it sends  a  non-technical  team composed
 of one  or more individuals  knowledgeable of UIC
 permitting rules to  meet with  the state.   Finally,
 if EPA  is satisfied  with a  state's files,  it
 develops  an  evaluation without additional  investiga-
 tion.

      The  data collected  under  this second  approach
 is similar to but  not  identical to that  collected
 under the first approach.   The report is similar
 in format and content.   The principal difference
 is that the  second approach relies more on a review
 of the  case  files  than on meetings with state per-
 mitting personnel.   One  limitation associated with
 this approach is that  the reports will tend to be
 based more on the  state's interpretation of their
 decision  process than  on an independent  assessment
 of the  decision process.  Nonetheless EPA  may
 decide  to follow this  approach because of  the
 reduced costs associated with  applying a pre-
 screening a  procedure.   Regardless of which approach
 EPA  selects,  ODW must  coordinate  the entire data
.collection.   However,  there are several
                  -16-

-------
different alternatives for how ODW can handle
its responsibility.  One approach is  for ODW to
ask the  regions  to develop case selection  criteria
and collect data from each state in their  region.
Alternatively, ODW can develop the criteria for
case  selection and the regions have responsibility
only  for the data collection.  The latter  procedure
would make the case selections more consistent and
uniform.  Another alternative would be for ODW to
do the entire data collection.  This  approach
would involve a  greater level of effort for head-
quarters than the other two approaches described.
Finally, ODW could oversee a contractor or other
independent group in the collection of data.
Because  Alternative II requires the skills of
specialized personnel, hiring a consultant or
soliciting the necessary skills from  sources such
as universities,  NAS, or other government  agencies
may be the best  of the alternatives described.

2.    Data Analysis

      We  considered various aspects of the  data
analysis for Alternative II including various
options  for how  the data should be aggregated,
who should perform the data analysis, and  how
this  analysis should be performed.  Analyses of
the data for the two data collection  alternatives
described will follow similar procedures.

      The review  teams are responsible for  the
preparation of narrative reports for  each  of the
permit cases.  These reports may be compiled into
state-by-state profiles.  State profiles can be
aggregated further into regional or a national
profile.

      ODW should  oversee the data analysis, decide
what  levels of comparison it wishes to make, .arid
then  determine the level of aggregation required
and the  appropriate aggregation procedures.  ODW
may want the regions to be responsible for develop-
ment  of  regional and state profiles;  ODW, however,
must  oversee preparation of the national profile.
'Alternatively, if ODW requests contractor  assistance,
the contractor can develop all aggregation levels
and conduct all  comparisons and evaluations ODW
requests.  Even  under this approach,  however, ODW
has responsibility for preparation of the  study
                  -17-

-------
findings.  It has the option of appointing an
Advisory Committee to review the findings and
any proposed recommendations.

     In contrast to''the quantitative nature of
the data analysis for Alternative I, the data
analysis for Alternative II is qualitative in
nature.  The analysis procedure for each'-state
involves the systematic review of the case reports
and the development of a set of observations or
conclusions for each case.  These case specific
conclusions will form an assessment of the quality
of the state's AOR program and corrective action
decisions.  They are not likely to provide exten-
sive overview information on the coverage of a
state program.

3.   Planning

     Similar to Alternative I this alternative
requires extensive planning by EPA.  First, EPA
must select from among the alternative approaches
for each study element.  Then, it must develop
and implement a study management procedure.
                                   \
     ODW, in order to coordinate the study, must
determine who will be responsible for each phase. '
The study's requirement for highly trained per-
sonnel not available at EPA may affect these
decisions.  ODW may have to take steps to re-
cruit these personnel  (such as soliciting
local geological expertise from state universities)
on a temporary basis if it conducts the study
in-house.  Alternatively, ODW may have to initiate
the procurement process in order to hire a con-
sulting firm with the necessary skills.  Both
the Civil Service and procurement processes are
time consuming.

     In order to determine the need for new
personnel and/or consultants, ODW must determine
how many review teams it will send to the field.
One possibility is to use a team comprised of a
core group; local experts can supplement this
team as necessary.  ODW also has the option of
using more than one core group for the study.

     The number of review teams not only affects
resource decisions but also the timing of the
                 -18-

-------
     evaluation.  For example, if there were a review
     team for each state, all of the reviews could occur
     simultaneously.  If only one team is used, the
     review would have to be conducted on a staggered
     basis.

          After EPA selects the study approach, it
     should develop criteria for the selection of
     sample cases; detailed options for this procedure
     can be developed if EPA elects Alternative II or
     III.  In addition, EPA heeds to develop a procedure
     for the presentation of the cases at the review
     team meetings, so that the reviews can be con-
     ducted as consistently and quickly as possible.
     This information could be provided to the states
     in the form of a guidance manual, which would be
     distributed at meetings at the various regional  '
     offices.

          The Agency also should develop criteria for
     team members to.use in conducting case reviews.
     This guidance would contain EPA's judgment as to
     what constitutes good AOR and corrective action
     decisions.  It may contain a checklist of impor-  .
     tant factors that should have been covered in the
     decisions and how to weight a state's considera-
     tion of these factors.  Similar criteria should
     be developed for the preliminary evaluation of
     case files if that approach is selected.

(3)  Alternative 'III

     Alternative III essentially is a combination of
Alternatives I and II.  EPA evaluates AOR on the basis
of the required state submitted data sind an EPA in-
depth data collection and analysis of a sample of
permittees.  Because we already have described alter-
natives I and II in detail in preceding sections, we
will develop Alternative III only briefly below.

     EPA has two potential options for the conduct of
Alternative III.  First, it can commit itself from the
outset to the independent collection of data on a
random sample of permittees.  Alternatively, it can
make an initial commitment only to conduct a prelimi-
nary evaluation based on Alternative I.  If that evalua-
tion suggests problems in one or more state programs,
EPA then can determine the need for supplementary
data collection and analysis.  Under this second
                      -19-

-------
     approach, the supplementary data can be from a random
     sample of selected cases.  If EPA elects the second
     approach to Alternative III, we will assist EPA in
     determining what types of preliminary findings would
     suggest the need for further data collection and
     analysis.                                      i

3.   EVALUATION OF THE THREE ALTERNATIVES

     In evaluating the three alternatives we considered two
principal factors.  The first, overall effectiveness, in-
cludes a comparison of the data collected with the data
required to measure the program objectives.  The second,
ease of implementation, includes a consideration of the EPA
and state level of effort associated with each alternative.

     Our evaluation indicates that Alternative III rates
highest in terms of overall effectiveness.  The state mid-
course evaluation report provides EPA with a good narrative
and statistical background on nationwide AOR activities.
In addition, the independent EPA data collection allows the
assessment of the technical and management quality of
decisionmaking necessary to measure state success in ful-
filling administrative objectives.

     In terms of ease of implementation, however, Alterna-
tive III rates very low, primarily because of its potential
cost.  Given current economic conditions and intensive
efforts to curb government spending, the Agency may be
unable at this time to make the resource commitment necessary
for Alternative III.  Accordingly, we developed the varia-
tion presented previously under which EPA undertakes a
preliminary evaluation based on Alternative I and then
determines the need for additional data collection.

     (.1)  Overall Effectiveness

          In order to compare the overall effectiveness
     of the three alternatives, we first evaluated each
     alternative with respect to how well the collected :
     data fulfilled the data requirements to measure the
     success of the AOR program set forth in Section 1.
     We then considered other factors including the quality
     of the data received, the extent of EPA's role in promot-
     ing good data quality, and the limitations of each
     alternative.

          The collection and analysis of data under Alter-
     native I should fulfill many of the factual and statis-
     tical data requirements specified in Section 1.  Such
                           -20-

-------
data as the number of AOR's undertaken and the types
of corrective actions required and taken will be
ascertained under Alternative I.  Because the data
furnished will be at the discretion of the
states, the quality and extent of the data received
will depend on the guidance EPA disseminates.  If EPA
requires the states to submit a detailed report con-
taining overview data, as suggested in the discussion
of Alternative I, the Agency can develop many generali-
zations about a state's program.  It should enable the
Agency to assess factors such as the coverage and extent
of each state's AOR program; the administrative burden
on each state; and the difficulties associated with the
program's implementation.  The main shortcoming asso-
ciated with Alternative I is that it provides no
mechanism for ascertaining the rationale behind each
AOR and corrective action decision.  As was illustrated
in Exhibit 1, information of this type is necessary
to make a total assessment of the AOR program.

     Because Alternative II involves the interaction
of qualified technical personnel with state permitting
authorities, EPA will be able to ascertain the -rationale
and quality of state's AOR and corrective action
decisions for a number of permit cases.  This alterna-
tive will provide data presented in Exhibit 1 such
as the well conditions leading to a corrective action
decision or why the state allowed exceptions to
standard procedure for a particular permit
applicant.   How well this information is docu-
mented will depend on how EPA defines the reporting
requirement.  Since Alternative II is limited to the
consideration of only a sample of cases, it will not
provide statistical overview information or data on
the coverage of the AOR program which were defined in
Section 1 as necessary to evaluate an AOR program.
It may be possible to extrapolate this data from the
data collected, but the alternative will not facilitate
an exact measurement.

     The data from Alternatives.''! and II are in many
ways complementary.  Together, tney furnish all of the
data specified in Exhibit 1.  Consequently, a combina-
tion alternative such as Alternative III, will be the
most effective way of achieving the goals of the mid-
course evaluation.
                      -21-

-------
(2)  Ease of Implementation

     In addition to the overall effectiveness of each
alternative, we also considered how easily each alter-
native could be implemented in practice.   The most
important factors in this consideration were the levels
of effort on the part of both the EPA and the states
required for data collection and analysis.

     The level of effort for both EPA and the states
varies according to the alternative EPA selects.
However, EPA has the option of hiring a consultant
to conduct all or some part of its role in the mid-
course evaluation.  Thus EPA's efforts can be limited
only to administrative planning of the evaluation if
a contractor performs the analysis or they can be
quite extensive should the Agency decide to conduct
the study in-house.  Directing the regional offices
to conduct the data collection and preliminary data
analysis for whichever alternative is chosen will
relieve much of the administrative burden on EPA
headquarters.

     Of the three alternatives, Alternative I is the
easiest to implement.  It requires only a modest level
of effort and, therefore, resource commitment by EPA,
and the data collection approach is consistent with
other current reporting practices such .as those under
the air quality program.

     Under Alternative I, the state's effort on data
collection is somewhat greater than that of EPA.
Because the regulations enumerate data categories,
the Agency's role on this phase primarily is to develop
and disseminate guidelines to promote reporting con-
sistency.  State staffs, on the other hand, devote
considerable time to completion of the required mid-
course reports.

     Data analysis for Alternative I will require more
effort on EPA's part than the data collection.  How-
ever, the level of effort will vary according to the
level of aggregation EPA selects, and the quality and com-
pleteness of state submittals.  Both manual and com-
puter evaluation will require considerable effort.
Regardless of whether EPA obtains outside assistance;
ODW still must devote a significant amount of atten-
tion to reviewing the preliminary findings, making
policy decisions, and preparing recommendations for
Agency consideration.
                      -22-

-------
     Alternative II should be more difficult to imple-
ment primarily because it requires both EPA and the
states to devote considerably greater effort than on
Alternative I.  Unlike Alternative I, it requires EPA
to take an active role in data collection.  This role
includes selection of a sample, development of a
protocol for conducting case reviews, and participa-
tion in the reviews.  The state role also is greater.
It still must comply with the UIC requirement for: a
mid-course report.  In addition, the staff must
participate in time consuming case reviews and must be
available to assist the study team in conducting file
reviews.

     At the same time, another factor affecting the
implementation of Alternative II may be the availability
of qualified personnel to perform the study.  Recruiting
of individuals qualified to perform the study, especially
persons knowledgeable in local or regional geological
conditions, may prove to be a cumbersome task.  Addition-
ally, the availability of such personnel may be limited.

     Analysis of Alternative II data may be somewhat
less burdensome than analysis of data from Alternative
I.  The collected data is not likely to be amenable
to statistical manipulation; hence, the data analysis
effort will involve expert judgment.

     If Alternative III is adopted it will require
the combined levels of effort necessary to perform
Alternatives I and II.  Consequently, Alternative III
is expected to represent a large administrative bur-
den for EPA.  For this reason, EPA may want to consider
carefully  the variation we developed to Alternative
III, where Alternative I data collection is conducted
first, based upon which a decision to conduct Alterna-
tive II or an abridged version of Alternative II is
made.  This variation should be more effective than
Alternatives I or II alone in terms of measuring the
goals of the AOR program.  It should require a lower
level of effort than a full-scale Alternative III.
                      -23-

-------