&EPA 68-01-6425 MIDCOURSE EVALUATION OF AREA OF REVIEW: EVALUATION OF ALTERNATIVE STUDY METHODOLOGIES SUBMITTED TO DR. JENTAI YANG OFFICE OF DRINKING WATER U.S. ENVIRONMENTAL PROTECTION AGENCY JANUARY 1983 BOOZ.ALLEN & HAMILTON, INC. UNDER THE DIRECTION OF GERAGHTY & MILLER, INC. ------- SEPA 68-01-6425 MIDCOURSE EVALUATION OF AREA OF REVIEW: EVALUATION OF ALTERNATIVE STUDY METHODOLOGIES SUBMITTED TO DR. JENTAI YANG OFFICE OF DRINKING WATER U.S. ENVIRONMENTAL PROTECTION AGENCY JANUARY 1983 BOOZ.ALLEN & HAMILTON, INC. UNDER THE DIRECTION OF GERAGHTY & MILLER, INC. ------- ACKNOWLEDGEMENT This report was prepared by Booz, Allen & Hamilton, Inc., under the direction of Geraghty & Miller, Inc. The Booz, Allen project manager was Dr. Joanne Wyman. She was assisted by Mr. Walter Mardis of Booz, Allen. The Geraghty & Miller project manager was Mr. William Thompson, and Mr. Roger Anzzolin was the EPA task monitor. ------- TABLE OF CONTENTS Page Number 1. OBJECTIVES OF AREA OF REVIEW 2 x 2. DESCRIPTION OF THE'ALTERNATIVES 6 3. EVALUATION OF THE THREE ALTERNATIVES 20 ------- LIST OF EXHIBITS Page Number Definition and Measurement of AOR Program Objectives Overview of Options for Each Phase of Study Alternatives I and II Summary of Reporting Requirements Applicable To Mid-Course Evaluation 11 ------- MID-COURSE EVALUATION OF AREA OF REVIEW: EVALUATION OF ALTERNATIVE STUDY METHODOLOGIES EPA has issued Underground Injection Control (UIC) and Consoli- dated Permitting regulations that require operators of new oil and gas wells to establish an Area of Review (AOR) defined as the radial distance within which existing active, abandoned, or dry wells penetrating the zone of injection must be checked. The ob- jective of the review is to identify and correct wells which are potential conduits for groundwater contaimination. Because the final rules differ substantially from those proposed in 1976, the Agency has agreed to undertake a "mid-course evaluation" in order to evaluate the AOR program's costs and benefits. The Office of Drinking Water (ODW) has asked Geraghty & Miller (G&M) and Booz, Allen & Hamilton (BA&H) to assist it in designing the mid-course evaluation study be describing and evaluating three alternative study methodologies. The Agency has defined these al- ternatives in terms of the data collection effort: Alternative I (State-supplied data); Each state operated program submits the data specified in 146.24 and 146.25 of the UIC regulations to the cognizant EPA regional office. Alternative II (EPA-conducted random survey);.EPA con- ducts detailed reviews of selcted permit cases. Alternative III (Combination of I and II); EPA supple- ments its analysis of the state submitted data with an analysis of EPA collected data. Our approach to conduting this assingment consisted of the following.three steps. First, we established a baseline against which to evaluate the three alternatives. This involved defining the substantive and management objectives of the AOR program, iden- tifying effectiveness indicators, and determining the types of data needed to apply the indicators. Second, we developed descriptions of each alternative. Although our project team meetings generated numerous variations on the principal steps in each alternative, we discuss in this technical report only those variations which appear to be compativle with current Federal and state environ- mental program practices. In the next step we evaluated each alternative with respect to its overall effectiveness and ease of implementation in measuring area of review objectives. -1- ------- Based on our evaluation of the alternatives, we reached two conclusions. First, the combination option provided EPA with the most comprehensive and best quality evaluation. It enables EPA to evaluate the quality of a state's permit decision process by conduting on site interviews with permitting personnel, and enables the Agency to evaluate that data within the context of the aggregate and permit specific data the states must submit in accordance with the UIC rules. As we suggest in the final section of the memorandum, how- ever, Alternative III is potentially quite costly . Recognizing that the Agency mya not.want to make such a high resource commit- ment, we have defined another alternative. First, EPA could con- duct a preliminary evaluation based on Alternative I. Based on the results of that evaluation, EPA could determined the need for an independent survey. 1. OBJECTIVES OF AREA OR REVIEW The purpose of the mid-course evaluation is to reassess the need for and the benefits of the AOR program. Based on this re- assessment, EPA could determined whether to broaden or reduce the scope of the AOR requirements. Our first step in assisting EPA in the selection of a mid- course evaluation study methodology was to define the AOR program objectives and develop methods and approaches suitable for measuring state agency success in fulfilling those objectives. We then identi- fied sets of achievement indicators appropriate for conducting these measurements. Exhibit i presents an overview of the AOR program objectives and the achievement indicators, together with types of data required to measure those objectives. (1) Definition of Objectives We first defined primary and secondary objectives of the AOR program. Primary, or substantive, objectives address the direct intent of the SDWA: the prevention of groundwater contamination. Secondary objectives such as the minimization of permitting costs and processing delays also are important because the quality of a state's permit-decision making process is a reflection of the program's efficiency and effectiveness. Additionally, the purpose of the revisions to the 1976 version of the UIC rules was to reduce costs and delays. The measurement of both sets of objectives provides a good indication of how effectively a state implemented AOR requirements. The defini- tions of the specific objectives presented in Exhibit 1 reflect consultations with EPA, a review of the SDWA and the history of its regulatory development including the research conducted in support of the rulemaking. -2- ------- EXHIBIT 1 Definition and Measurement of AOR Program Objectives AOR PROGRAM OBJECTIVE ACHIEVEMENT INDICATORS DATA REQUIREMENTS FOR MEASUREMENT ASSESSMENT OF •MEASUREMENT FEASIBILITY Protection of current/ potential underground sources of drinking water Direct measurement of prevention or abatement of contamination is virtually technically impossible. State success in fulfilling the other objectives listed serves as an indirect indicator of effective- ness with respect to this objective See data requirements for measuring other objectives. See assessments of feasibility of obtaining data required to measure other objectives I Ul I Correction of wells identified as posing a potential threat to groundwater Direct indicators such as reduction in contamination from poorly com- pleted or abandoned wells are difficult, if not impossible, to measure Indirect indicators include: Identification of all potentially threatening wells Development of corrective action plan for all threatening wells Completion of all needed corrective action percentage of wells penetrating injection zone selected for corrective action Percentage of state approved, discontinued corrective actions Rationale for corrective action decisions Number of confirmed, successful, completed corrective actions Collection of the statistical data is easy; however, its evaluation, in terms of the technical soundness of correc- tive action decisions, requires an independent, on site investigation. Ultimate review of all wells which pose a potential threat to groundwater Extent of overlap of AOR of first and second year permit applicants Overlays or other graphic displays showing existing wells and AOR for new well applicants Easy to measure since states will submit permit specific maps to EPA which are amenable to aggregation. Compliance by state agencies with technical standards and adminis- trative procedures of AOR regulations Consistent, equitable imposi- tion of corrective action requirements Detection of non-compliers Enforcement against non- compliers Submittal of accurate, complete records in mid-course report Comparison of corrective action plans for each permittee Number of applicants for which state approved discontinuation of corrective action Case specific flowcharts of application of AOR requirements Number of total and permittee specific inspections Number of violations detected Number of each type of enforce- ment action taken Requires considerable on site file examination and interviews with agency staff; involves judgment on the part of evaluation team, thereby necessi- tating commitment of a team with extensive technical and program management expertise. ------- EXHIBIT 1 (Continued) AOR PROGRAM OBJECTIVES ACHIEVEMENT INDICATORS DATA REQUIREMENTS FOR MEASUREMENT ASSESSMENT OF MEASUREMENT FEASIBILITY Minimization of adverse impacts on energy resource development Absence of increases in the typical time taken to bring a new well or conversion on-line attributable to AOR Absence of increases in project costs associated with AOR Absence of production losses attributable to AOR Time frames for permit applica- tion preparation Time frames for permit processing Manhours and costs associated with - Defining AOR and corrective action Implementing corrective action Data on permittee manhours, cost, and time frame for permit application is difficult to collect. It requires operator recordkeeping not consistent with current practice. If EPA can obtain the data, it can compare it with estimates prepared in support of the rulemaking. Data on time frames for permit processing requires laborious on site review of state files to identify delays attributable solely to AOR. Maximization of management and technical quality of AOR decisions Minimization of adverse resource impacts on state Adequacy of manpower and budget resources, including staff stability Adequacy of staff training Timeliness of task completion Consistency of requirements imposed on permittees Budget increases/decreases Staff increases/decreases Staff turnover rates Hiring/promotion standards Staff education and experience Availability and use of continuing training courses' Availability and use by state and permittee of technical guidance manuals Percentage of total UIC budget allocated for AOR Percentage of permit application backlogs attributable to AOR AOR decision review processes Most states presently do not compile this data in a systematic fashion. Its collection requires on site file re- views and its interpretation requires the subjective judgment of individuals thoroughly familiar with both UIC re- quirements and state environmental program operations. ------- (2) Approaches To Measuring State Success in Achieving Objectives Based on our understanding of groundwater and oil and gas technology and consultations with Dr. Donald Warner, Professor of Geological Engineering, University of Missouri, we developed an approach to measuring state success in achieving primary and secondary objectives of the AOR program. In formulating this approach we also took into account the problems EPA historically has faced in attempting to measure the direct relationship between its programs and environmental protection. To fully consider these problems, we reviewed an EPA report, compiled in 1975, entitled "First Report of the Environmental Measures Project" which deals with the problems of measuring program success. As a result, we have concluded that it is difficult, if not impossible, to measure, except in a limited way at great cost, any of the substantive objectives of the AOR program. For example, it often is impossible to relate groundwater pollution in a given area to an injection well failure or, con- versely, to show that proper construction or abandonment of a particular well prevents groundwater degradation. Consequently, the mid-course evaluation cannot test the underlying assumption that improperly constructed or abandoned wells may serve as conduits for groundwater contamination. There also are state- of-the-art constraints in determining the success of correc- tive action. While a mechanical integrity test can indicate the need for or success of corrective action performed on an active well,* there is no direct way to test whether an in- active well is abandoned or reabandoned successfully. Instead, one must rely on visual inspection of the wells and review of the abandonment records. Since state agencies, themselves, have few mechanisms available to ascertain the adequacy of corrective action, it will be difficult, if not impossible, for EPA to make an independent assessment. Although direct measurement of substantive objectives is not possible, EPA will be able to measure administrative objectives such as a high level of program efficiency, technical EPA has some concerns regarding the effectiveness of mechanical integrity testing practices and has asked G&M and BASH to develop an approach to evaluating those practices. -5- ------- quality, and compliance with the requirements of the regula- tions. In addition, direct measurement of the administrative objectives can serve as an indicator or surrogate measure of substantive objectives. For example, if state programs are not characterized by high levels of efficiency and technical quality, it indicates that the program's substantive objectives are not being fulfilled. Conversely, if states are successful in meeting the administrative objectives, it is a reasonable indication* that the program's objectives are being met, Based on this reasoning, we developed indicators, such as a reduction in the number of leaking wells or an increase in percentage of wells reviewed in a state during the first two years of the program, with which to measure those objectives. Exhibit 1 presents a preliminary scheme for EPA to con- sider in setting program objectives which will be measured by the AOR evaluation. There are a total of seven objectives; for each objective we provide a series of potential achieve- ment indicators and define the types of data needed to measure them. A brief assessment of the feasibility of their use is included. 2. DESCRIPTION OF THE ALTERNATIVES Our second step was to outline the key elements of each of the three study alternatives. First, we established three study phases: Planning Data collection Data aggregation and analysis. For each alternative, we developed several approaches to the conduct of each study phase. Based on project team meetings and consultation with the EPA Project Officer, we selected for further consideration those approaches most compatible with current EPA and state agency practice and procedure. These approaches are summarized in Exhibit 2. We have assumed that the AOR and corrective action concepts are scientifi- cally sound. As noted above, EPA will not be able to test these assumptions during the mid-course evaluation because of the difficulty of directly measuring cause effect relationships. -6- ------- EXHIBIT 2 Overview of Options for Each Phase of Study Alternatives I and II* Phase Element Alternative I Alternative II in x z o I Data Types Permit specific as enumerated in 146.24 and 146.25 Detailed technical and administra- tive data on a sample of permit applications; number of cases and amount of- detail varies with re- source commitments Format Options Permit by permit tabulations Above combined with aggregate program statistics Above supplemented by explana- tory narrative State sends data enumerated in 146.24 and 146.25 plus accom- panying narrative for each case selected EPA develops protocol to guide on site case reviews Collection Frequency Quarterly, semi-annually, annually, bi-annually Annually or bi-annually Collection Procedure State reports to Region State reports to ODW ODW selects technical team which reviews random sample ODW selects smaller team to review limited, special cases Aggregation Levels State, region, nationwide (statistical) Same as Alternative I (narrative) E U 2 VI I Analytic Approaches Manual analysis Canned or tailored computer program Quantitative Manual, qualitative plus statistical Types of Analysis State specific Comparisons among states Comparisons among regions Nationwide evaluation Same as Alternative II Analysts Regions (preliminary) ODW (partial or all) Contractor assistance (partial or all) Scientific advisory panel (review) Same as Alternative II: probably requires outside assistance 0. Detailed planning recommendations for either alternative cannot be developed until EPA makes the following decisions: Selection of objectives to be measured Estimate of available resources Selection of study alternative Selection of approach to each phase element EPA finalizes Commitment of estimated resource/needs Once EPA finalizes these decisions, ODW must: Develop a study plan including a study tracking system Appoint ODW Task Officer/work group Initiate procurement process, if necessary Initiate advisory panel appointments, if necessary Develop and disseminate guidance to regions and states The planning process for each alternative is similar, containing the same elements but varying in complexity. Alternative III essentially is an approach which combines Alternatives I and II -7- ------- (1) Alternative I Alternative I is an evaluation methodology that relies on data collected, aggregated, and submitted by the states to EPA. Section 146.25 of the UIC regulations enumerates the data to be submitted by the states specifically in connection with the mid-course evaluation. The regulations also provide EPA with the option of preparing detailed guidance on state report- ing requirements. In addition, EPA may find the reports sub- mitted in compliance with the consolidated permittina reaulations useful because they contain data which demonstrate the diligence of state agencies in fostering operator compliance with AOR and other UIC requirements. However, usefulness of these reports is limited because they provide data on only a small number of permittees and some of the data is not entirely pertinent to the AOR evaluation. Exhibit 3 summarizes UIC reporting requirements in terms of reporting frequency and content. The following sections describe the data collection system, including the types of data collected and the collection procedures; the data analysis process, including methods for data aggregation and analysis; and agency planning requirements. 1. Data Collection In accordance with Section 146.25 of the UIC regula- tions, all state-directed UIC programs are required to submit the following data to EPA: Information sufficient to identify the permittee A map showing the applicable area of review, the location of all wells within the area of review A tabulation of data describing all the wells within the area of review which penetrate the proposed injection zone Appropriate geological data on the injection zone and confining zones Specified characteristics of potentially affected underground sources of drinking water Engineering drawings of the surface and sub- surface construction details of the system -8- ------- EXHIBIT 3 Summary of Reporting Requirements Applicable To Mid-Course Evaluation Report Name Authority Report Preparer Frequency Content/Format Quarterly Noncompliance Report 40 CFR 122.18(a) All permit issuing authorities; state UIC Program Directors or EPA Regional Administrators (who- ever has primacy) Quarterly Narrative report on noncomplying permit- tees. Includes back- ground information on permittee, description of noncompliance and Director's action Annual Noncompliance Report 40 CFR 122.18(c) I 'vo I All permit issuing authorities; state UIC Program Directors or EPA Regional Administrators (who- ever has primacy) Annual Statistical report on permittees reviewed, noncompliance, enforce- ment, permit modifica- tions, and extension of compliance deadlines Program Report 40 CFR 122.18(c) (4) (i) State administered UIC programs Annual Narrative report describing program implementation and program changes. In- cludes an inventory of active underground injection activity AOR Report 40 CFR 122.18(c)(4)(ii) State administered UIC programs One time Due within 3 months of completion of second full year of operation of state program Data on each new Class II permit and area of review as described in 40CFR 146.25 of UIC regulations ------- Corrective action proposed for and performed on new injection wells A demonstration of mechanical integrity sub- mitted to .the State Director prior to permit approval Results of all mechanical integrity tests con- ducted on existing wells and new (conversion only) wells during the first two years of operation A temperature log on noise for a sample of wells for which operators submitted cementing records to demonstrate mechanical integrity. The regulations afford EPA with an opportunity to provide the states with guidance on the preparation of the mid-course evaluation report. In order to assist the Agency in taking advantage of this opportunity to promote reporting consistency, we developed several reporting format variations for EPA's consideration. Eac.h approach varies according to the relative manpower burden on EPA and the states. The option which places the most extensive burden on EPA involves state submitted ' tabular displays of permit by permit decisions and activities. It requires EPA to aggregate the data prior to analysis and interpretation. A second option, under which the states supplement the permit specific tabula- tions with aggregate program statistics increases the state level of effort but reduces that of EPA. Neither of these first two approaches, however, provides EPA with an under- standing of the administrative and management context within which these activity levels occurred. Accordingly, a third option is for EPA to require the states to include additional information on the organization of the AOR program, flowcharts of the AOR process, estimates of man- hours and budget devoted to AOR, and relationship of the AOR procedures to the overall permitting process. If EPA elects to require the states to submit reports more frequently than at the end of two years of program im- plementation, it can ask the states to submit the adminis- trative descriptions' with the first report. As in the air quality program, subsequent reports address only substantive changes in program administration. -10- ------- EPA also can select from among several procedural approaches to data collection. For example, the approach most consistent with current air and water program practices is for states to submit their reports to the appropriate EPA regional office. This is the approach EPA will follow with regard to the quarterly and annual UIC noncompliance reporting. Nevertheless, because of the particular purpose for which EPA is collecting this data, ODW may want to receive the data directly. Finally, EPA already is aware of the need to consider various options regarding the timing and frequency of the data collection. The frequency options are numerous. The Agency can require the states to submit one comprehensive report after two years of implementation. Other options include annual, semiannual and quarterly reporting. ODW already recognizes that in selecting an approach it needs to address the fact that not all programs will commence on the same date. Accordingly, in conjunction with its decision on reporting frequency, ODW may want to specify uniform reporting dates. 2. Data Analysis Like the data collection system, the data analysis system consists of several elements. Principal ones include data aggregation procedures, analysis, and interpretation of the '.analysis. For each, we present several alternatives for ODW to consider. Further develop- ment of either Alternative I or Alternative III is contin- gent on EPA selection from among these options. EPA has considerable flexibility in structuring the aggregation procedures. One decision EPA must make con- cerns the desired level of aggregation. Choices include aggregation by state, region, and nationwide. EPA may find it useful to prepare all three types of data aggregation, depending on the objectives EPA selects for measurement and the available manpower and budget resources. Another decision ODW must make is that of who will have responsi- bility for aggregation and analysis. With respect to aggregation, the level of aggregation EPA decides to per- form inevitably affects who will perform it. For example, -11- ------- it would be appropriate for regional offices to pre- pare statewide and regional data aggregation and perhaps even preliminary analysis and interpretation.* However, no single region would have the resources or perspective to conduct cross-regional comparisons or nationwide analysis. Thus, it is most appropriate for ODW to perform 'the nation- wide data aggregation as well as the analysis and interpretation.** Once aggregated, the data can be correlated and their statistical significance ascertained. This can be done manually, by use of a computer package like SPSS, or by use of a computer program developed specifically for the mid-course evaluation. Manual manipulation of the data may prove to be unwieldy and will limit the sophistication of the analysis that may be performed. ODW should conduct the final evaluation of the data and develop conclusions on the effectiveness of the AOR program. ODW can handle this task internally or by appointing an advisory panel. This panel would be composed of appropriate agency personnel and technical expertise from outside the agency as ODW deems appropriate. 3. Planning Although the UIC regulations specify the data to be collected, EPA must make several administrative decisions in order to implement Alternative I. Many of these decisions are of a general nature; others pertain only to a particular study phase. The Agency first must decide who in ODW will be responsible for the study. Regardless of who conducts the various study tasks, ODW will have oversight since it has responsibility for UIC rulemaking. From the objectives we define in Section 1, ODW then should determine what the study should accomplish and select accordingly from among the various options we describe for each study phase. Based on those decisions, ODW should direct the development of a study plan which, includes a study management system. One constraint on this Regional role would be if EPA elects to use computer analysis. In order to minimize error and computer costs, the data file prep- aration and analysis should not be scattered among several offices. ODW can perform the work in-house or arrange for outside .assistance from consultants, the National Academy of Science, or other research organizations. If ODW elects to obtain outside assistance, the external organization also can relieve the various Regions of aggregation or analysis chores. -12- ------- The Agency has great discretion for the management of various facets of the study. As noted above, one such area concerns the timing and frequency of the reporting procedure. Reports could be submitted every six months (four reports), annually (two reports), or after, two years (one report). The advantage of quarterly, semiannual, or annual reporting is that it gives EPA an oppor- tunity to correct any ambiguities or misunderstand- ings in the reporting procedure as well as give states feedback so that future reporting can pro- ceed more smoothly. However, the processing of the additional reports will constitute an added administrative burden for the Agency. In addition, it will require a system for tracking the study's progress and status. The Agency also must consider whether the dates for submittal of each state'.s report should be based on when each state initiated the program (different dates) or on when the entire program was initiated (on one date). The latter would allow EPA the additional perspective of reviewing programs at different stages of '. implementation, while the former would allow EPA to compare programs at the same stage of develop- ment. EPA should select the approach most in line with its goals. The Agency also has the option of disseminating guidance and training to the states. It can com- pile a policy memorandum or guidance manual to aid the states in report preparation and to promote consistency in state reporting. The manual could include the following types of information: sample report formats, guidelines on permit application requirements, (for example, requiring permittees to submit an extra copy of those documents required for the mid-course evaluation) and definitions of each data category. As part of its planning effort, EPA needs to distribute the guidance ... material in a timely fashion. It may be distributed at a training session at the appropriate regional offices, a procedure which would allow EPA to answer any questions pertinent to the reporting. After ODW finalizes the study plan, it needs to promote the commitment of required manpower and budget resources. This may involve designation of an ODW Task Officer and/or work group, initia- tion of the procurement process if outside assistance -13- ------- is needed, designation of regional staffs, and position advertising if ODW or the regions require new hires. Finally, if an EPA appointed advisory committee is to have an important role in preparing the final evaluation and recommendations it should be appointed. (2) Alternative II Alternative II involves independent EPA collecti.on of data on a random sample of permittees from each state, Unlike Alternative I, this alternative is not addressed in the UIC regulations. Consequently, we had to con- sider how to develop this alternative in somewhat greater detail than Alternative I. Alternative II is described in a manner similar to Alternative I. Data collection, data analysis, and planning are considered separately below. As in Alter- native I, if EPA wants us to develop this approach further, the Agency must select from among the varia- tions presented for each study phase. 1. Data Collection Since Alternative II data collection is not addressed in the UIC regulations, we had to develop a scheme including the sample size, the data types to be collected, the collection procedure, and the timing of the data collection. We have developed variations for all these elements except sample size; this element will be considered in detail should EPA select Alternative II or the Combination Alternative. Alternative II addresses the need for EPA to appraise the rationale behind the state AOR decisions. Successful implementation of an area of review program requires many state agency per- sonnel at various levels to make judgmental decisions. Personnel involved in the AOR decision process may include the enhanced recovery engineers responsible for processing the permit and the field supervisors responsible for implementing and enforcing corrective actions. Consistency and equity of the decision process is desirable and requires decisions to be based on a solid technical basis. The data collection process set forth below is one which should permit EPA to evaluate both the -14- ------- management and technical capability of the UIC staff and by inference the success in achieving UIC program objectives. The types of data which will be available to EPA will vary according to the data collection procedure employed. We have developed two such procedures. The first approach is based on the "ideal" situation where the availability of funding and qualified personnel to conduct the study is not constrained significantly. Recognizing that current economic conditions and the attendant attempts to curb government spending may result in resource constraints, we discuss a second approach which involves a more modest commitment of outside technical assistance and a lower level of effort on the part of EPA. The ideal approach involves sending a highly skilled technical team to each state agency to review a sample of UIC permit cases. In order to ensure a comprehensive and high quality data collection effort, team members ideally should have expertise in: The UIC program Local geohydrology Enhanced recovery Reservoir engineering Well design, construction, logging, and well service. This team meets at specified frequencies with state personnel including the enhanced recovery or area engineers responsible for permit processing, field supervisors responsible for implementation of corrective action, and any interested industrial operators. Each state would present a number of permit cases, explaining the rationale and technical basis for the AOR and corrective action decisions. The state presents the cases based on an EPA developed selection procedure. The procedure may be a random or stratified random one or may emphasize particular problem or success cases. Under this approach, state personnel and the review team engage in a round table discussion. This enables the review team to pose case specific -15- ------- questions. Following completion of the case reviews, the team can develop a narrative report for each case which reflects the review team's judgment on the quality of the AOR and corrective action decisions. Depending on the sample size, the team may also prepare a statistically representa- tive profile of state activities. The second approach to Alternative II involves a two-step process. First, EPA requires each state to submit permit case files to headquarters or the cognizant regional offices. Included in the submissions are narratives detailing how the state made its AOR and corrective action decisions. These cases would be selected by EPA headquarters or the Region randomly, or on the basis of certain criteria such as age of well field or depth of well. Based upon its review of the sample, EPA elects one of three possible courses of action for each state. If EPA has questions on whether a state's AOR and corrective action decisions are technically sound, EPA sends a technical team such as that described in the first approach to the state agency to conduct a more thorough review. Second, where EPA has questions of a non-technical nature, for instance, on the quality of a state's permitting procedure, it sends a non-technical team composed of one or more individuals knowledgeable of UIC permitting rules to meet with the state. Finally, if EPA is satisfied with a state's files, it develops an evaluation without additional investiga- tion. The data collected under this second approach is similar to but not identical to that collected under the first approach. The report is similar in format and content. The principal difference is that the second approach relies more on a review of the case files than on meetings with state per- mitting personnel. One limitation associated with this approach is that the reports will tend to be based more on the state's interpretation of their decision process than on an independent assessment of the decision process. Nonetheless EPA may decide to follow this approach because of the reduced costs associated with applying a pre- screening a procedure. Regardless of which approach EPA selects, ODW must coordinate the entire data .collection. However, there are several -16- ------- different alternatives for how ODW can handle its responsibility. One approach is for ODW to ask the regions to develop case selection criteria and collect data from each state in their region. Alternatively, ODW can develop the criteria for case selection and the regions have responsibility only for the data collection. The latter procedure would make the case selections more consistent and uniform. Another alternative would be for ODW to do the entire data collection. This approach would involve a greater level of effort for head- quarters than the other two approaches described. Finally, ODW could oversee a contractor or other independent group in the collection of data. Because Alternative II requires the skills of specialized personnel, hiring a consultant or soliciting the necessary skills from sources such as universities, NAS, or other government agencies may be the best of the alternatives described. 2. Data Analysis We considered various aspects of the data analysis for Alternative II including various options for how the data should be aggregated, who should perform the data analysis, and how this analysis should be performed. Analyses of the data for the two data collection alternatives described will follow similar procedures. The review teams are responsible for the preparation of narrative reports for each of the permit cases. These reports may be compiled into state-by-state profiles. State profiles can be aggregated further into regional or a national profile. ODW should oversee the data analysis, decide what levels of comparison it wishes to make, .arid then determine the level of aggregation required and the appropriate aggregation procedures. ODW may want the regions to be responsible for develop- ment of regional and state profiles; ODW, however, must oversee preparation of the national profile. 'Alternatively, if ODW requests contractor assistance, the contractor can develop all aggregation levels and conduct all comparisons and evaluations ODW requests. Even under this approach, however, ODW has responsibility for preparation of the study -17- ------- findings. It has the option of appointing an Advisory Committee to review the findings and any proposed recommendations. In contrast to''the quantitative nature of the data analysis for Alternative I, the data analysis for Alternative II is qualitative in nature. The analysis procedure for each'-state involves the systematic review of the case reports and the development of a set of observations or conclusions for each case. These case specific conclusions will form an assessment of the quality of the state's AOR program and corrective action decisions. They are not likely to provide exten- sive overview information on the coverage of a state program. 3. Planning Similar to Alternative I this alternative requires extensive planning by EPA. First, EPA must select from among the alternative approaches for each study element. Then, it must develop and implement a study management procedure. \ ODW, in order to coordinate the study, must determine who will be responsible for each phase. ' The study's requirement for highly trained per- sonnel not available at EPA may affect these decisions. ODW may have to take steps to re- cruit these personnel (such as soliciting local geological expertise from state universities) on a temporary basis if it conducts the study in-house. Alternatively, ODW may have to initiate the procurement process in order to hire a con- sulting firm with the necessary skills. Both the Civil Service and procurement processes are time consuming. In order to determine the need for new personnel and/or consultants, ODW must determine how many review teams it will send to the field. One possibility is to use a team comprised of a core group; local experts can supplement this team as necessary. ODW also has the option of using more than one core group for the study. The number of review teams not only affects resource decisions but also the timing of the -18- ------- evaluation. For example, if there were a review team for each state, all of the reviews could occur simultaneously. If only one team is used, the review would have to be conducted on a staggered basis. After EPA selects the study approach, it should develop criteria for the selection of sample cases; detailed options for this procedure can be developed if EPA elects Alternative II or III. In addition, EPA heeds to develop a procedure for the presentation of the cases at the review team meetings, so that the reviews can be con- ducted as consistently and quickly as possible. This information could be provided to the states in the form of a guidance manual, which would be distributed at meetings at the various regional ' offices. The Agency also should develop criteria for team members to.use in conducting case reviews. This guidance would contain EPA's judgment as to what constitutes good AOR and corrective action decisions. It may contain a checklist of impor- . tant factors that should have been covered in the decisions and how to weight a state's considera- tion of these factors. Similar criteria should be developed for the preliminary evaluation of case files if that approach is selected. (3) Alternative 'III Alternative III essentially is a combination of Alternatives I and II. EPA evaluates AOR on the basis of the required state submitted data sind an EPA in- depth data collection and analysis of a sample of permittees. Because we already have described alter- natives I and II in detail in preceding sections, we will develop Alternative III only briefly below. EPA has two potential options for the conduct of Alternative III. First, it can commit itself from the outset to the independent collection of data on a random sample of permittees. Alternatively, it can make an initial commitment only to conduct a prelimi- nary evaluation based on Alternative I. If that evalua- tion suggests problems in one or more state programs, EPA then can determine the need for supplementary data collection and analysis. Under this second -19- ------- approach, the supplementary data can be from a random sample of selected cases. If EPA elects the second approach to Alternative III, we will assist EPA in determining what types of preliminary findings would suggest the need for further data collection and analysis. i 3. EVALUATION OF THE THREE ALTERNATIVES In evaluating the three alternatives we considered two principal factors. The first, overall effectiveness, in- cludes a comparison of the data collected with the data required to measure the program objectives. The second, ease of implementation, includes a consideration of the EPA and state level of effort associated with each alternative. Our evaluation indicates that Alternative III rates highest in terms of overall effectiveness. The state mid- course evaluation report provides EPA with a good narrative and statistical background on nationwide AOR activities. In addition, the independent EPA data collection allows the assessment of the technical and management quality of decisionmaking necessary to measure state success in ful- filling administrative objectives. In terms of ease of implementation, however, Alterna- tive III rates very low, primarily because of its potential cost. Given current economic conditions and intensive efforts to curb government spending, the Agency may be unable at this time to make the resource commitment necessary for Alternative III. Accordingly, we developed the varia- tion presented previously under which EPA undertakes a preliminary evaluation based on Alternative I and then determines the need for additional data collection. (.1) Overall Effectiveness In order to compare the overall effectiveness of the three alternatives, we first evaluated each alternative with respect to how well the collected : data fulfilled the data requirements to measure the success of the AOR program set forth in Section 1. We then considered other factors including the quality of the data received, the extent of EPA's role in promot- ing good data quality, and the limitations of each alternative. The collection and analysis of data under Alter- native I should fulfill many of the factual and statis- tical data requirements specified in Section 1. Such -20- ------- data as the number of AOR's undertaken and the types of corrective actions required and taken will be ascertained under Alternative I. Because the data furnished will be at the discretion of the states, the quality and extent of the data received will depend on the guidance EPA disseminates. If EPA requires the states to submit a detailed report con- taining overview data, as suggested in the discussion of Alternative I, the Agency can develop many generali- zations about a state's program. It should enable the Agency to assess factors such as the coverage and extent of each state's AOR program; the administrative burden on each state; and the difficulties associated with the program's implementation. The main shortcoming asso- ciated with Alternative I is that it provides no mechanism for ascertaining the rationale behind each AOR and corrective action decision. As was illustrated in Exhibit 1, information of this type is necessary to make a total assessment of the AOR program. Because Alternative II involves the interaction of qualified technical personnel with state permitting authorities, EPA will be able to ascertain the -rationale and quality of state's AOR and corrective action decisions for a number of permit cases. This alterna- tive will provide data presented in Exhibit 1 such as the well conditions leading to a corrective action decision or why the state allowed exceptions to standard procedure for a particular permit applicant. How well this information is docu- mented will depend on how EPA defines the reporting requirement. Since Alternative II is limited to the consideration of only a sample of cases, it will not provide statistical overview information or data on the coverage of the AOR program which were defined in Section 1 as necessary to evaluate an AOR program. It may be possible to extrapolate this data from the data collected, but the alternative will not facilitate an exact measurement. The data from Alternatives.''! and II are in many ways complementary. Together, tney furnish all of the data specified in Exhibit 1. Consequently, a combina- tion alternative such as Alternative III, will be the most effective way of achieving the goals of the mid- course evaluation. -21- ------- (2) Ease of Implementation In addition to the overall effectiveness of each alternative, we also considered how easily each alter- native could be implemented in practice. The most important factors in this consideration were the levels of effort on the part of both the EPA and the states required for data collection and analysis. The level of effort for both EPA and the states varies according to the alternative EPA selects. However, EPA has the option of hiring a consultant to conduct all or some part of its role in the mid- course evaluation. Thus EPA's efforts can be limited only to administrative planning of the evaluation if a contractor performs the analysis or they can be quite extensive should the Agency decide to conduct the study in-house. Directing the regional offices to conduct the data collection and preliminary data analysis for whichever alternative is chosen will relieve much of the administrative burden on EPA headquarters. Of the three alternatives, Alternative I is the easiest to implement. It requires only a modest level of effort and, therefore, resource commitment by EPA, and the data collection approach is consistent with other current reporting practices such .as those under the air quality program. Under Alternative I, the state's effort on data collection is somewhat greater than that of EPA. Because the regulations enumerate data categories, the Agency's role on this phase primarily is to develop and disseminate guidelines to promote reporting con- sistency. State staffs, on the other hand, devote considerable time to completion of the required mid- course reports. Data analysis for Alternative I will require more effort on EPA's part than the data collection. How- ever, the level of effort will vary according to the level of aggregation EPA selects, and the quality and com- pleteness of state submittals. Both manual and com- puter evaluation will require considerable effort. Regardless of whether EPA obtains outside assistance; ODW still must devote a significant amount of atten- tion to reviewing the preliminary findings, making policy decisions, and preparing recommendations for Agency consideration. -22- ------- Alternative II should be more difficult to imple- ment primarily because it requires both EPA and the states to devote considerably greater effort than on Alternative I. Unlike Alternative I, it requires EPA to take an active role in data collection. This role includes selection of a sample, development of a protocol for conducting case reviews, and participa- tion in the reviews. The state role also is greater. It still must comply with the UIC requirement for: a mid-course report. In addition, the staff must participate in time consuming case reviews and must be available to assist the study team in conducting file reviews. At the same time, another factor affecting the implementation of Alternative II may be the availability of qualified personnel to perform the study. Recruiting of individuals qualified to perform the study, especially persons knowledgeable in local or regional geological conditions, may prove to be a cumbersome task. Addition- ally, the availability of such personnel may be limited. Analysis of Alternative II data may be somewhat less burdensome than analysis of data from Alternative I. The collected data is not likely to be amenable to statistical manipulation; hence, the data analysis effort will involve expert judgment. If Alternative III is adopted it will require the combined levels of effort necessary to perform Alternatives I and II. Consequently, Alternative III is expected to represent a large administrative bur- den for EPA. For this reason, EPA may want to consider carefully the variation we developed to Alternative III, where Alternative I data collection is conducted first, based upon which a decision to conduct Alterna- tive II or an abridged version of Alternative II is made. This variation should be more effective than Alternatives I or II alone in terms of measuring the goals of the AOR program. It should require a lower level of effort than a full-scale Alternative III. -23- ------- |