$ £\ I? Office of Inspector General i ¦ | I Special Review Consistency and Transparency in Determination of EPA's Anticipated Ozone Designations Report No. 2002-S-00016 August 15, 2002 ~SPHERE ------- Regions Covered: Regions 1-10 Program Office Involved: Inspector General Division Conducting the Audit: Office of Air Quality Planning and Standards Mid-Atlantic Audit and Evaluation Resource Center Philadelphia, PA ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL MID-ATLANTIC AUDIT AND EVALUATION RESOURCE CENTER 1650 Arch Street Philadelphia, Pennsylvania 19103-2029 (215)814-5800 August 15, 2002 MEMORANDUM SUBJECT: Special Review Report: Consistency and Transparency in Determination of EPA's Anticipated Ozone Designations Report No. 2002-S-00016 FROM: Lisa White /s/ Lisa White Project Manager Mid-Atlantic Audit and Evaluation Resource Center (3AI00) TO: Tom Curran, Acting Director, Office of Air Quality Planning and Standards (C40404) Attached is a copy of the final report on Consistency and Transparency in Determination of EPA's Anticipated Ozone Designations. This special review contains issues that describe conditions the Office of Inspector General (OIG) has identified and an OIG recommendation. This report represents the opinion of the OIG. Final determinations on matters in this report will be made by EPA managers in accordance with established EPA resolution procedures. Accordingly, the findings described in this special review do not necessarily represent the final EPA position and are not binding upon EPA in any enforcement proceeding brought by EPA or the Department of Justice. ACTION REQUIRED You are requested to provide a written response to this special review within 90 days of the date of this report. We have no objections to the further release of this report. If you or your staff have any questions regarding this report, please contact me or Lorraine Fleury at (215) 814-5800. For your convenience, this report will be available at httv://www.eva.sov/oieearth/eroom.htm. V O T ?* ------- ------- Executive Summary Purpose The 1990 Clean Air Act requires the Environmental Protection Agency (EPA) to designate counties as being in nonattainment, in attainment, or unclassifiable with respect to the national ambient air quality standards for ozone. In 1990, the acceptable ozone level was based on the 1-hour ozone standard. In 1997, EPA issued a new 8-hour ozone standard to provide increased protection to the public. This new standard was challenged in court. While EPA has prevailed on the basic issues regarding the challenges to its 8-hour ozone standard, pursuant to court decisions in that litigation, EPA is developing a new implementation plan for the 8-hour ozone standard. The objectives of this review were to determine (1) whether each of EPA's Regional offices used a specific process/method/approach for obtaining stakeholder input for the 1-hour ozone designations; (2) what process/method/approach was used for the preliminary analysis of the 8-hour ozone designations; and (3) the potential usefulness of the Multi- criteria Integrated Resource Assessment (MIRA) decision approach, in which stakeholder participation creates open, transparent decisions regarding the anticipated 8-hour ozone designations. Results in Brief EPA's guidance for the 1-hour ozone designations was not specific as to how stakeholder participation should be used in the ozone designations. The guidance for the preliminary 8-hour ozone designations is more comprehensive than the approach EPA followed in 1990, in that it acknowledges the importance of stakeholder participation. In addition, the guidance lists 11 criteria the states should consider if proposing larger or smaller metropolitan nonattainment boundaries. However, the preliminary 8-hour ozone guidance did not provide a methodical process for the Regions and states to use when considering the 11 criteria. Without a consistent Regional approach, the ozone designations may not be fair or equitable throughout the nation. Region 3 used the MIRA approach to address the preliminary 8-hour i Report No. 2002-S-00016 ------- ozone designations, and this or a similar multi-criteria approach could be useful for all EPA Regions. MIRA is a decision making methodology that documents stakeholders' interests and can assess the impacts of a given set of criteria simultaneously. Recommendation When determining the 8-hour ozone designations, we recommend that the Director of the Office of Air Quality Planning and Standards (OAQPS) instruct the EPA Regional offices to use a multi-criteria approach such as MIRA that offers transparency, consistency, and potential for consensus building. Agency Response and OIG Evaluation OAQPS does not agree with the recommendation in the draft report and maintains that MIRA cannot be used as the primary tool for designating areas under the Clean Air Act. They believe that the primary approach for assigning designations should be a case-by-case consideration and evaluation of each area's unique situation and circumstances. Contrary to OAQPS' response, our report does not recommend that MIRA be used as the primary tool for designating areas under the Clean Air Act; rather, it identifies those attributes of MIRA (consistency, documentation/transparency, and consensus building) that could help the agency improve its process for the 8-hour ozone designation. We request that OAQPS reconsider using a multi-criteria approach like MIRA that offers these attributes. ii Report No. 2002-S-00016 ------- Table of Contents Executive Summary i Introduction 1 Purpose 1 Background on Designating Ozone Areas 1 The 8-hour Ozone Standard 2 MIRA Background 3 Scope and Methodology 4 EPA Should Promote Consistency and Transparency Among Regional Offices 5 Varied Processes Used by the Regions for Ozone Designations 5 The 1-Hour Ozone Designation Process 6 Preliminary 8-Hour Ozone Designation Process 6 Benefits of MIRA Demonstrated 8 Recommendation 9 Agency Response 9 OIG Evaluation 10 Appendices 11 A: MIRA Structure 11 B: OIG Questionnaire 13 C: Agency Response and OIG Evaluation 15 D: Example of Sequential vs. Simultaneous Decision Approach 25 E: Distribution List 27 iii Report No. 2002-S-00016 ------- ------- Introduction Purpose The objectives of this review were to determine: ! Whether each of the Environmental Protection Agency's (EPA's) Regional offices used a specific process/method/approach for obtaining stakeholder input for the 1-hour ozone designations; ! The process/method/approach used for the preliminary analysis of the 8-hour ozone designations; and ! The potential usefulness of the Multi-criteria Integrated Resource Assessment (MIRA) decision approach, in which stakeholder participation creates open, transparent decisions regarding the anticipated 8-hour ozone designations. Background on Designating Ozone Areas The Clean Air Act requires EPA to designate areas as being in nonattainment, in attainment, or unclassifiable with respect to the national ambient air quality standards for ozone. The law requires that no later than 1 year after promulgation of a new or revised national ambient air quality standard, the states submit a list of all areas to the EPA Administrator with their designations. The Administrator has 2 years from the date of promulgation of the new or revised national ambient air quality standard to promulgate the designations of all areas. The Clean Air Act provides that the period may be extended for up to 1 year if the Administrator has insufficient information to promulgate the designation. EPA's Goal 1 ("Clean Air"), in the 2000 Strategic Plan, states the air in every American community will be safe and healthy to breathe. One of the Agency's objectives is that by 2012, air throughout the country meets the national standards for ozone. The Office of Inspector General's strategic area of emphasis, Contributing to improved environmental quality and human health, states we will analyze the use of scientific research in carrying out the Agency's programs, including the air program. 1 Report No. 2002-S-00016 ------- In 1990, the acceptable ozone level was .12 parts per million (ppm), based on the 1-hour ozone standard. In 1997, EPA revised its 1-hour primary ozone standard (health-based) with a new 8-hour ozone standard that decreases the ozone level to .08 ppm, thereby, increasing health benefits to the public. The benefits of achieving the 8-hour ozone standard include reduced risks of respiratory symptoms in children, lessened lung impairments that can lead to chronic respiratory illnesses (such as emphysema and chronic bronchitis later in life), and fewer hospital visits for respiratory problems for individuals with asthma. The new standard would require states to monitor over an 8-hour period, which is a more representative reading. The changes were made because many new health studies had shown that adverse health effects occur at exposure times longer than 1 hour and at levels lower than the previous standard. It is important to note that EPA and the states do not rely solely on air quality data when making the ozone designations. Because of the limited number of ozone monitors, data was not always available. EPA guidance states that any county with an ozone monitor showing a violation of the standard needs to be designated nonattainment. For those areas designated nonattainment, the guidance states that additional criteria be considered to determine the boundaries of nonattainment areas. These additional criteria include population density, traffic congestion, and pollution transport provided by various stakeholders. Because the ozone designations are based on the consideration of multiple criteria, EPA needs to be prepared to use a reliable, consistent method to apply this new standard. Region 3 developed a multi-criteria decision approach, called MIRA, that could be used to address the above criteria simultaneously when making the anticipated 8-hour ozone designations. The 8-hour Ozone Standard EPA is developing a new implementation strategy for the 8-hour standard, including how to classify areas designated as nonattainment for that standard. These classifications will determine the control measures each nonattainment area will be required to impose to help the area reach attainment. Once EPA's implementation strategy has been finalized through rule making, EPA plans to designate areas for the 8-hour ozone standard. 2 Report No. 2002-S-00016 ------- MIRA Background Initially, MIRA was developed by EPA Region 3's Air Protection Division in response to the Government Performance and Results Act. Among other things, this act was written to improve federal effectiveness and public accountability by promoting a focus on results, service quality, and customer satisfaction. Region 3 developed MIRA as an effort to link its decisions to environmental impacts and help meet the Air Protection Division's strategic goals. MIRA considers economic, social, and political impacts of alternatives before a decision is made and, ultimately, used to support a final decision. MIRA documents stakeholders' interests and can assess the impacts of a given set of preferences simultaneously. MIRA does this by providing a framework to make the data that is relevant to the decision transparent and to disclose stakeholders' interests when changing the weighting of the preferences to make a decision. MIRA helps articulate decisions using data. MIRA is not a substitute for the decision maker and is not set up to convince people that there is only one decision; rather, it facilitates discussion and the creation of additional options that stakeholders and decision makers may not have thought of but which meet their interests. MIRA allows for the comparison of the impacts between two or more options, and empowers decision makers and stakeholders to create and test options. Its output is not an optimal decision, but information that spurs discussion, debate, learning, and consensus building. Details on MIRA's structure are in Appendix A. This report discusses the "Decision Analysis" module within the MIRA structure as it relates to the ozone designations. Region 3 used MIRA to develop a response to the Office of Air Quality Planning and Standards' (OAQPS) request for a preliminary analysis of the 8-hour ozone designations; however, because MIRA is relatively new, it has not been utilized for a specific public policy decision. Decision Making ! All decisions are tradeoffs among relevant criteria. ! Decision making is not objective. ! Decision making requires using judgment (i.e., values). ! Science does not/cannot make decisions but can be used to drive decisions. Source - MIRA training course 3 Report No. 2002-S-00016 ------- Scope and Methodology Because the review included assistance by the Region 3 MIRA team, we considered this a special review and not an audit. However, we followed applicable Government Auditing Standards to the extent practical - particularly in regard to qualifications, due professional care, quality control, planning, supervision, evidence, and reporting. We conducted meetings with: the Region 3 Air Director; Region 3 MIRA team members; and OAQPS officials from EPA's Office of Air and Radiation. We held an entrance conference with OAQPS officials on September 6, 2001, and completed our fieldwork on December 28, 2001. We issued the draft special review on April 30, 2002. Agency comments and our evaluation are summarized after the recommendation, and a complete copy of their response is provided in Appendix C. We reviewed documentation at the Air and Radiation Information Center related to the 1-hour ozone designations. We read the draft journal article prepared by the Region 3 MIRA team, which included a comparison of MIRA to other multi-criteria decision approaches. We attended MIRA training. We reviewed the results of the Region 3 case study, which was able to closely duplicate the results of the 1991 1-hour ozone designations. We also reviewed EPA guidance for the preliminary 8-hour ozone standard and the status of the anticipated 8-hour ozone standard. We formulated a national questionnaire (see Appendix B) that we sent to the EPA Regional offices to identify the stakeholders and explain how their input was considered in the 1991 decision for the 1-hour standard, and methods used for the preliminary 8-hour standard. Next, we compared the similarities and differences among the Regional offices. For the purpose of this review, we looked at one decision approach - MIRA - and its ability to organize and analyze data and to facilitate decision making. Our review looked at the decision analysis module of MIRA and how it could be used with a new ozone standard. We relied on the technical expertise of the Region 3 MIRA team to demonstrate how MIRA can assist the Agency when making multi-criteria decisions. 4 Report No. 2002-S-00016 ------- EPA Should Promote Consistency and Transparency Among Regional Offices EPA's guidance for the 1-hour ozone designations was not specific as to how stakeholder participation should be used in the ozone designations. The guidance for the preliminary 8-hour ozone designations is more comprehensive, in that it acknowledges the importance of stakeholder participation. In addition, the guidance lists 11 criteria the states should consider if proposing larger or smaller metropolitan nonattainment boundaries. However, guidance for the preliminary 8- hour ozone designations did not provide a methodical process for the states to use when considering the 11 criteria. Without a consistent Regional approach, the ozone designations may not be fair or equitable throughout the nation. In January 2002, the Office of Management and Budget issued guidelines for ensuring and maximizing the quality, objectivity, utility, and integrity of information that is shared with the public by each Federal Agency. By October 1, 2002, Agencies will need to be more accountable for the information that supports the official position of the Agency. The more important the information, the higher the quality standards to which it should be held. We believe MIRA has the capability, by making its data and methods transparent, to provide the support EPA will need to meet this Office of Management and Budget requirement. Varied Processes Used by the Regions for Ozone Designations We asked all 10 EPA Regional offices to complete a questionnaire to enable us to evaluate their processes used to designate and classify areas for the 1-hour and preliminary 8-hour ozone standard. In particular, we asked the Regional offices to identify the stakeholders for the 1-hour ozone designations and explain how their input was considered. We believe that stakeholder input is essential when making the ozone designations because, ultimately, stakeholder cooperation is required to implement the Clean Air Act. For the prehminary 8-hour ozone designations, we wanted to know how many criteria were addressed, as stated in the March 28, 2000, OAQPS memorandum, "Boundary Guidance on Air Quality Designations for the 5 Report No. 2002-S-00016 ------- 8-Hour Ozone National Ambient Air Quality Standards," as well as the methodologies, if any, used to consider these criteria. The 1-Hour Ozone Designation Process In response to our inquiry regarding stakeholders for the 1-hour ozone designations, we determined that 8 of the 10 Regional offices obtained stakeholder input, but no consistent process, methodology, or approach for incorporating stakeholder input was applied. Many of the Regional offices indicated that they solicited stakeholder input through meetings or correspondence. Consequently, we could not evaluate the extent to which stakeholder participation influenced the final recommendations submitted by the states. For the two Regional offices that did not solicit stakeholder participation, one said stakeholder input was not applicable because their states' counties were all in attainment. Another said that their area was considered all in nonattainment prior to the 1990 designations; therefore, the counties would remain in nonattainment and stakeholder input was not needed. Preliminary 8-Hour Ozone Designation Process The promulgation of the 8-hour ozone standard in 1997 gave EPA the opportunity to improve on the ozone designation process. The March 28, 2000, OAQPS memorandum detailed the designation process for this standard. According to the memorandum, all counties must be designated in attainment/unclassifiable or in nonattainment. Attainment is defined when an area is meeting the ozone standard and unclassifiable is when, due to no data or insufficient data, EPA cannot make a determination. Nonattainment is defined when counties do not meet the standard or contribute to ambient air quality in a nearby area that does not meet the standard. OAQPS instructs the regions to treat the attainment and unclassifiable designations the same; therefore, areas without monitors are treated as if they were monitoring in attainment. Consequently, the designation of attainment/unclassifiable for these unmonitored areas is not a true representation of this designation because we cannot determine the number of counties measuring in attainment versus the number of counties without monitors. The memorandum also indicated that because it is best to consider controls on sources over a larger area due to the nature of ground-level ozone and transport of ozone, the Metropolitan Statistical Area or the 6 Report No. 2002-S-00016 ------- Consolidated Metropolitan Statistical Area should serve as the presumptive boundary for the 8-hour nonattainment area. These are statistical representations of counties and groups of counties which function as a unit economically as specified by the Office of Management and Budget. EPA believes this approach will best ensure public health protection from the adverse effects of ozone pollution caused by population density, traffic congestion, commercial development, and industrial development. If a state wanted to propose a larger or smaller nonattainment area other than the default Metropolitan Statistical Area or Consolidated Metropolitan Statistical Area boundaries, the guidance listed 11 criteria that could be considered. This guidance required the states to submit their recommendations to EPA for final designations. In our questionnaire, we asked the Regional offices how many criteria were considered, as well as what methodologies, if any, were used to consider the criteria listed below: 1. Emissions and air quality in adjacent areas 2. Population density and degree of urbanization, including commercial development 3. Monitoring data representing ozone concentrations in local areas and larger areas 4. Location of emission sources 5. Traffic and commuting patterns 6. Expected growth 7. Meteorology 8. Geography/topography 9. Jurisdictional boundaries 10. Level of control of emission sources 11. Regional emission reductions The Clean Air Act mandates that EPA should consider all areas when making ozone designations. Results of the questionnaire showed that, while most of the Regions said they considered the criteria, only three of the eight Regional offices used a multi-criteria approach. Additionally, based on discussions with the Regional offices, Region 3 is the only office, through MIRA, that considered all counties in their ozone designation. The reason given by the other Regions for not considering all counties was twofold: (1) it was too resource intensive; and (2) based on prior information, they did not believe every county needed to be considered in the analysis. 7 Report No. 2002-S-00016 ------- Once the air quality data has been entered in a database, the MIRA approach provides Region 3 the ability to consider all counties in the same timeframe as it would take to consider only those counties with monitors. This fact may be particularly appealing to the Regions that are concerned that considering all counties would entail using extensive resources. We believe the inconsistencies among the Regional offices resulted from the OAQPS memorandum being too general, thus not providing a methodical process on how to consider the 11 criteria for the nonattainment boundaries. Two Regional offices noted in their responses that they would like a more prescriptive way to rank the criteria, in order of importance, to properly designate boundaries for the 8-hour ozone standard. MIRA provides a framework for the Regional offices to consider how the criteria applies to their Region and then to apply those criteria consistently to each county. Also, in a January 19, 2001 memorandum, OAQPS recognizes there are differences among the Regions and states in their application of the earlier guidance for the preliminary 8-hour ozone designations. This memorandum emphasizes the importance of applying this guidance nationally in a fair and equitable manner; however, OAQPS did not provide specifics on how they would accomplish this task. Benefits of MIRA Demonstrated The Region 3 MIRA team demonstrated MIRA's capabilities to us in a case study related to the 1-hour ozone designations. First, the MIRA team obtained Region 3's 1-hour ozone designations made in 1991. Using MIRA's framework, the team was able to closely replicate the 1-hour ozone designations. For the 1-hour ozone designations, there was not always documentation to clearly support why a particular county was considered in attainment or nonattainment. MIRA documents how the criteria are weighted and the data the Agency relied on to make a decision that can be used for future reference. MIRA affords the decision maker an opportunity to learn about individual ranking of counties from most nonattainment to least nonattainment and their rankings relative to each other. Decision makers are then tasked with making a choice determined by the scientific data and social values to rank the counties in a manner that 8 Report No. 2002-S-00016 ------- can be articulated and reproduced. This is where discussion, debate, and learning occur to determine the appropriateness of the decision, thus providing stakeholders the opportunity to see how their input influenced the final designations. The 1-hour case study did not solicit stakeholder input because the replication was done for a decision already implemented (the 1991 ozone designations). Although MIRAhas attributes that make it a viable consideration for decision making, it also has specific limitations. MIRA is currently housed on several computers in Region 3 and it is not documented in manuals. While EPA needs to consider the time and money a Region would need to invest in obtaining the computer software and teaching MIRA to its employees, we believe it would be a beneficial investment. Recommendation When determining the 8-hour ozone designations, we recommend that the Director of OAQPS instruct the EPA Regional offices to use a multi- criteria approach such as MIRA that offers: ~ Documentation and transparency of EPA's decisions to meet the OMB requirement. ~ Consistency in considering any criteria identified. ~ Development of new options that reflect stakeholder and decision maker interests, thereby offering greater potential for consensus building. Agency Response OAQPS does not agree with the recommendation in the draft report and maintains that MIRA cannot be used as the primary tool for designating areas under the Clean Air Act. Their response contends that the designation process is prescribed by the Clean Air Act, and once EPA determines that an air quality monitor is recording a violation, they must designate the area nonattainment. They further state that MIRA, as implemented by Region 3, only recognizes the Clean Air Act and EPA regulatory requirement as an input factor of equal weight with all other considerations. However, the response also states that the primary approach for assigning designations should be a case-by-case consideration and evaluation of each area's unique situation and circumstances. OAQPS 9 Report No. 2002-S-00016 ------- indicated they considered the factors used to determine nonattainment boundaries sequentially and independently versus MIRA, which considers all the factors simultaneously. OIG Evaluation Contrary to OAQPS' response, our report does not recommend that MIRA be used as the primary tool for designating areas under the Clean Air Act; rather, it identifies those attributes of MIRA (consistency, documentation/transparency, and consensus building) that could help the agency improve its process for the 8-hour ozone designation. EPA and the states cannot rely solely on air quality data when making the ozone designations because, in 2001, only 965 of the 3,100 counties had ozone monitors. Therefore, they must rely on additional factors such as population density, traffic congestion, and pollution transport provided by various stakeholders. OAQPS' guidance did not provide a methodology for considering the factors, which in turn allows for the Regions and the states to apply them in whatever order they choose. MIRA is a tool that can be used to organize the factors listed in the OAQPS guidance that will ultimately provide options for the decision maker to use regarding the ozone designations. We disagree that MIRA only recognizes the Clean Air Act and EPA regulatory requirement as an input factor of equal weight with all other considerations. Given the fact that the ozone designations are an air quality decision, the decision makers, using MIRA, could weight air quality the heaviest, thereby ranking those areas with poor air quality the highest in nonattainment. We believe that considering the factors simultaneously is a more consistent approach (fair and equitable) than sequentially. When considering the factors sequentially, the factor that is evaluated first automatically becomes the most important (or most heavily weighted) factor. In contrast, the simultaneous evaluation of the factors, allows the decision maker to determine the relative importance of each factor to the other. Again, we believe this is a more consistent method than simply applying several factors to a given area in a random order. For an example see Appendix D. OAQPS provided an attachment to their response disagreeing with specific issues included in the draft report. We have inserted our comments directly in the attachment (Appendix C). 10 Report No. 2002-S-00016 ------- Appendix A MIRA Structure MIRA Structure Decision Anahsis Decision Criteria Fule & Transport Module GeosLaLislicul TtidieuLor Modulo (GiM) Data Collection manager (DOM) F,m is sin rift, AQ, Demographics, Fit. Decision Criteria - factors, determined by the decision makers, to be important to the decision. Data Collection Manager (DCM) - allows the analyst to store, sort, and retrieve data, such as source emissions, demographics, and air quality values. There is currently no such system that allows for storage and custom retrieval of data, which can also then be used directly in decision analysis. Geostatistical Indicators Module (GIM) - provides a means to reduce spatial fields of pollutant concentration, usually presented as maps, to a single quantitative index value. The creation of this area-weighted index allows the analyst the unique ability to compare the environmental impact among different areas, including non-contiguous areas. Both the data from the DCM and the indices from GIM can be used to populate the decision criteria in MIRA's decision analysis module. Fate & Transport Module - this air quality modeling system is a framework that demonstrates the ozone impact based on emissions input by the user. Decision Analysis - the decision maker preferences/judgments are obtained first in Expert Choice™ and then used in the decision analysis spreadsheet to produce decision alternatives. The modular construction of MIRA permits the analyst the flexibility to determine how the DCM and GIM will be used with the decision analysis module. 11 Report, No. 2002-S-00016 ------- Report No. 2002-S-00016 ------- Appendix B OIG Questionnaire As part of our OIG review entitled, "Review of the Agency's Decision Making Process, Using the 1 Hour Ozone Nonattainment Boundary Decision as an Illustrative Process Case Study," we need all the Regional offices to complete this questionnaire. Please complete the questions in their entirety and respond by electronic mail no later than Thursday, December 13. QUESTIONS 1. In response to the March 28, 2000, John Seitz memorandum, did the states within your Region submit a list recommending nonattainment/attainment designations for their counties by June 30, 2000? If yes, please indicate which states provided such a list. 2. a. Please identify who you believe were the stakeholders in the 1-hour ozone nonattainment designation decision (i.e., those parties or people that are affected by the designations decision). b. When your Region determined the designations, did you incorporate the stakeholders interests in the decision making? Explain the process/ method/approach that was used (i.e., how these stakeholder interests were considered). 3. Please list which of the 11 criteria, on page four of the memorandum, your states or you considered. a. Please indicate how the criteria that were used were considered. Give an example(s) to illustrate the process. b. Were the criteria ranked in terms of relative importance to each other and, if so, please explain the method/process? c. If there were criteria that were not considered, please indicate or explain why they were not or could not be considered. 4. Please comment on whether there are elements that you would have liked to consider in the Agency's decision making process (either specific to the 1 hour ozone nonattainment decision or generally for the Agency's decision making) but were not able to (or cannot) consider and why. 13 Report No. 2002-S-00016 ------- Report No. 2002-S-00016 ------- Appendix C Agency Response and OIG Evaluation June 26, 2002 MEMORANDUM SUBJECT: Response to Draft Report: Consistency and Transparency in Determination of EPA's Anticipated Ozone Designations, April 30, 2002 FROM: Thomas C. Curran, Acting Director (Signed by Tom Curran) Office of Air Quality Planning and Standards (C404-04) TO: Carl A. Jannetti Divisional Inspector General for Audit Mid-Atlantic Division (3AI00) Thank you for the opportunity to review and respond to the findings in the subject report. I appreciate the work your staff put into the review of the Multi- Criteria Integrated Resource Assessment (MIRA) approach. We believe that MIRA is a technique that can have potential benefits for certain types of decision making. For example, we have applied MIRA in developing our monitoring strategy to help determine the mix and location of various types of air quality monitors across the country. Also, as you may know, this office has provided funding to Region 3 to help support the model's development. However, we have major concerns about using this type of tool to fully address all of the legal, technical and practical constraints associated with the process of designating areas as attainment or nonattainment under the Clean Air Act. As a result, we do not agree with the recommendation in your draft report that MIRA be used as the primary tool for designating areas under the Clean Air Act. The draft report suggests that MIRA would provide greater consistency, transparency and equity in the designation process. We agree that those are important goals for the designation process and believe there may be a limited role for MIRA or MIRA-like tools to be used within the statutorily mandated process. However, OAQPS strongly believes that, while MIRA can help inform decisions on 15 Report No. 2002-S-00016 ------- designations, it would not be appropriate to use MIRA as the sole or primary basis for determining the boundaries of nonattainment areas. We believe that the primary approach for assigning designations should be a case-by-case consideration and evaluation of each area's unique situation and circumstances. The Clean Air Act defines a nonattainment area as any area that is violating the national ambient air quality standard (NAAQS) or is contributing to a violation of the NAAQS in a nearby area. Thus, once EPA determines that an air quality monitor is recording a violation of the 8-hour ozone NAAQS, EPA must designate the area in which the monitor is located and any nearby areas contributing to the violation as nonattainment. The MIRA, as implemented by Region 3, only recognizes this Clean Air Act and EPA regulatory requirement as an input factor of equal weight with all other considerations. Since application of MIRA in this way could result in the clearly unacceptable designation of an area that is violating the NAAQS as attainment/unclassifiable, the statutory and regulatory requirements should be applied before using MIRA and should not be overridden in determining how an area should be designated. For the purpose of designations, we do not believe that MIRA can be the primary basis for considering the unique and complex issues that arise within the designation process. These concerns are further detailed in the attachment. For these reasons, I cannot concur with the recommendation in this report to use a decision tool, such as MIRA, as the primary basis to determine designations. However, we do believe there may be certain circumstances where the use of a MIRA- type tool may have a role to play in helping inform the designation process. I would ask that the report specify that any use of MIRA, or a MIRA-type model, by an EPA Regional Office to help inform the designation process be coordinated with the Office of Air and Radiation to help us promote and ensure national consistency in decisions that are made and to help contribute to lessons learned from using such a tool. Thank you again for the opportunity to review the draft report and please direct the final report to the Administrator and my office. Attached are some additional comments on the draft report. Please let me know if you have any questions, or have your staff contact Sharon Reinders at (919) 541-5284 or me. Attachment EPA:OAR:OAQPS:AQSSD:OPSG:SREINDERS\LLassiter: N. Campus C539-02\l- 5526 File Name: I :\SEC\REINDERS\DraftResponseToOIGonDesigs6-14-02.wpd June 18, 2002 Coordinated w/ Lydia Wegman, John Silvasi and Jan Tierney, OGC. 16 Report No. 2002-S-00016 ------- Attachment MIRA's Ability to Address Unique and Complex Situations in the Designation Process As with MIRA, EPA considers multiple criteria to determine the area contributing to a nonattainment problem. However, unlike MIRA, EPA considers such factors sequentially and independently versus MIRA which considers such factors altogether with preferences and weights. For example, with regard to determining the area contributing to a violation, EPA could determine that an area should be designated nonattainment solely because of significant nitrogen oxide (NOx) emissions from a single stationary source, such as a power plant, in that area. Under MIRA, however, such an area may not be designated nonattainment due to a low model score based solely on a low weight for NOx emissions alone because the area does not also have other factors, such as significant mobile source emissions or a large population to raise the model score. Furthermore, MIRA might assign a value based on the total amount of NOx emissions within a set area, but it does not allow for consideration of unique circumstances such as the number or geographic location of such sources or the consequences of aggregating emissions across contiguous or adjacent areas. Nor does MIRA allow for appropriate consideration of the unique geographic features of an area or multiple areas, such as the height and location of mountains and hills in relation to the mix of sources. Consequently, MIRA's system of preferences and assigning predetermined weighting factors can easily result in important case-specific factors not being appropriately considered. Consistency in application and results is possible only if the criteria and factors are standardized and established nationally and not subject to change. Assuming that the desired consistency referred to in the report is national, changing the preferences and weights in MIRA according to subjective preferences will likely not preserve consistency from area to area, but promote conflicting results. The possibility of changing factors or assumptions complicates understanding and comparisons. Furthermore, the possibility of changing these based on stakeholder interests can easily undermine objectiveness in the designation process by allowing otherwise unsuitable or inappropriate considerations or weighting to influence the process. Alternatively, using the set of factors, weights and comparative values of MIRA as applied in Region 3 may not result in appropriate designation decision for other areas of the country. Again, the EPA believes that a case-by-case consideration of each area's facts and circumstances best achieves the goals of the designation process. It is important to note that a case-by-case consideration of factors can produce a consistent, transparent and equitable designation process. The goals of consistency, transparency and equity are more likely to be achieved through a centralized decision process for final designation determinations for all areas. This 17 Report No. 2002-S-00016 ------- is one reason the Administrator has not delegated this authority to designate an area under a new standard to the Regional Offices. To help ensure consistency and objectivity in the recommendations provided to the Administrator, the EPA has issued guidance enumerating a variety of factors that States should consider in making designation recommendations and that EPA would consider in reviewing those recommendations. A case-by-case review of each recommendation by the Regional Offices in conjunction with EPA headquarters offices ensures consistency in designations. In addition, EPA has made standardized data bases available to the States and tribes, which include among other things data on monitored air quality values and data related to the criteria used to determine boundaries. These data are available on the website www.epa.gov/ttn/naaqs/ozone/areas/. In the past, EPA has held internal national workshops with the Regional and headquarters offices to consider each Region's review of the State and tribe's recommendations in light of EPA guidance. Whereas, as specified in the Clean Air Act (CAA), our current process allows EPA to rectify inappropriate State and Regional recommendations, the formulaic structure of MIRA, which would assign values based on the State or tribal recommendations without examining the underlying considerations that the State and EPA examined, could carry any inconsistent recommendations forward into EPA's designation process. In terms of transparency, EPA notes that the designation process is clear and open with several opportunities for the public, States and tribes to review and be involved in the process. Initially, the process is opened at the State and tribal level, prior to the time their recommendations are made to EPA. Each State and tribe has its own process including public and stakeholder processes for responding to requirements under the CAA. The EPA does not seek to dictate their processes for responding to the CAA's requirements. Preliminary recommendations for the 8-hour standard and Regional Offices' initial responses were posted on EPA's website at www.epa.gov/ttn/naaqs/ozone/areas/recommend/. Another opportunity could arise at the time that EPA determines whether to modify a recommendation. The EPA's basis for a designation decision for the 8-hour standard will be documented in the record for that action. Furthermore, EPA has issued public guidance concerning the factors that EPA believes are relevant for determining boundaries of nonattainment areas. EPA also notes that the use of MIRA could actually delay the designation process. Since MIRA is a model involving algorithms and weights, if it were to be used as a sole or primary decision-making tool, it may need peer review and possibly national rulemaking similar to other mathematical models that we use. 18 Report No. 2002-S-00016 ------- OIG Evaluation We disagree with the example regarding the power plant not being considered nonattainment using the MIRA approach. The MIRAprocess may not necessarily consider the county in which the power plant is located to be in attainment if the weight for emissions was rated heavily. MIRA allows the decision maker to recognize that each area may have unique circumstances. Once the analysis is complete, if the decision maker wants to change a county that is currently ranked in attainment to nonattainment, they could change the weighting of the preferences and run the analysis again. Another option is to leave the analysis as is and selectively choose the county to be placed in the nonattainment ranking with a disclosure of why the designation changed. The MIRA process only informs the decision maker, it does not assign values, make decisions, or provide optimal answers. We disagree with OAQPS' response that transparency is the same as public participation. In the MIRA process, transparency is the ability to document how the criteria are weighted by the stakeholders and to document the data relied on by the Agency to make a decision that can be used for future reference. The following comments generally follow the order of the topics presented in the body of the report. However, our comments also apply if the topic was addressed in the Executive Summary. # Timing and Process of Designations The following information is intended to supplement this discussion in the report: The CAA provides for States to make designation recommendations no later than 1 year following promulgation of new or revised NAAQS and requires the Administrator of EPA to promulgate designations no later than 2 years after promulgation of a new or revised NAAQS. The EPA also provides the opportunity for recommendations to tribes. However, the CAA also provides that the period for the Administrator to promulgate designations may be extended by 1 year if the Administrator has insufficient information. The Transportation Equity Act for the 21st Century (TEA-21) modified these dates as they apply to the 8-hour ozone NAAQS promulgated in July 1997. Under TEA-21, States were provided with 2 years to recommend designations, and EPA was required to designate areas no later than 1 year following the time State recommendations were due. The date for EPA action was ultimately extended by HR-4635, which prohibited EPA from acting until either the Supreme Court ruled in the challenge to the NAAQS or until June 15, 2001. With the Supreme Court decision in February 2001, the prohibition was lifted. 19 Report No. 2002-S-00016 ------- The designation program for the air quality standards is not delegated to the Regional Administrators. Therefore, the Administrator promulgates all designations for new or revised standards. While the Regional Offices perform the initial review of designation recommendations, the Office of Air Quality Planning and Standards (OAQPS) evaluates the results of each review made by Regional Offices against both the recommendation and EPA guidance. This helps to ensure consistency in the designation process. It is not always necessary to evaluate all the criteria in EPA's guidance to determine that the area is in violation of the standard or is contributing to a violation. Since the criteria are to be considered independently, the area may be determined to be in violation of the standard or contributing to a violation based upon an evaluation of less than all of the criteria. For example, there is no need to look at all of the criteria as they would apply in an area monitoring attainment in the vicinity unless it is necessary to evaluate whether it is contributing to a violation in a nearby area. However, we would still evaluate the State or tribal recommendation. OIG Evaluation We added, the sentence regarding the Administrator being granted an additional year for insufficient data from the Clean Air Act in the background section. However, we did not add the wording about the Transportation Equity Act for the 21st Century or the HR-4635 because this section was background applicable to both the 1-hour ozone designation and the 8-hour ozone designation and the two acts are not subject to the 1-hour standard. # 8-hour Ozone Standard Our Office of General Counsel suggests the following language covering litigation on the standard to replace the sentence: "On March 26, 2002, EPA's 8-hour ozone standard was upheld by the District of Columbia Circuit Court of Appeals. As such key implementation decisions regarding the new standard are anticipated in the near future." In response to that challenge, the Supreme Court determined that the implementation approach EPA set forth at the time the standard was promulgated was unsupported by the statute. The Court remanded the issue to EPA; EPA expects to issue a proposed rule in response to that remand in the near future. Also, delete references to the "newly affirmed" 8-hour standard. The EPA did not seek Supreme Court review on one aspect on which we lost in the Court of Appeals. Thus, there is still an outstanding issue with respect to the standard and it is inaccurate to say that the standard has been "affirmed" or "upheld" by the court. 20 Report No. 2002-S-00016 ------- The language in the section entitled "8-hour Ozone Standard Affirmed" (also, delete "affirmed" from the heading) should be replaced with the following: The 8-hour ozone standard was challenged in 1997 by the American Trucking Association, other business groups and certain States. The Court of Appeals for the District of Columbia Circuit initially determined that EPA's interpretation of the statute conflicted with the constitutional non-delegation principle. In addition the court held that EPA had not considered the beneficial effects of UVB in establishing the NAAQS and that EPA's implementation policy conflicted with CAA requirements. The EPA sought review in the Supreme Court of the constitutional and implementation issues. The Supreme Court found that EPA's interpretation did not violate the non- delegation doctrine. However, the Court remanded EPA's implementation approach. The EPA did not seek review of the lower Court's decision that EPA must consider the beneficial effects of UVB. Because the lower Court had not considered additional issues at the time it made its ruling based on the non- delegation doctrine, the Supreme Court remanded the case back to the Court of Appeals to consider those issues. On remand, the Court of Appeals found that EPA's action in promulgating the 8-hour ozone NAAQS was not arbitrary and capricious. Pursuant to the remand from the Supreme Court, EPA is developing a new implementation strategy for the 8-hour standard, including how to classify areas designated as nonattainment for that standard. These classifications will determine the control measures each nonattainment area will be required to impose to help the area reach attainment,. Once EPA's implementation strategy has been finalized through rule making, EPA plans to designate areas for the 8-hour ozone standard. On the matter of the Court decision on the implementation plan for the 8-hour standard, it is not accurate to say that EPA has been instructed to develop an implementation plan that has the same structure that was written for the 1-hour ozone standard with regard to classifications. While the Supreme Court ruled that EPA could not ignore subpart 2 (the structure for the 1-hour ozone standard), it recognized a gap in coverage and left it up to EPA to resolve this issue. This implies that the 1-hour standard scheme may not be entirely appropriate for all areas that are designated nonattainment for the 8-hour standard. The draft report states that EPA is required to finalize the implementation plan prior to designations. It is not a statutory requirement to do so. Thus, it is EPA's current plan to finalize the implementation plan prior to designations because, in order to begin planning, it is important for the State to know whether and what implementation planning requirements the area will be subject to under subpart 2 of the statute. 21 Report No. 2002-S-00016 ------- The draft report indicates that EPA only relies on information supplied by States and stakeholders for designations. Once an area is determined to be nonattainment, EPA and the States use a variety of information to help determine whether a nearby area, e.g., a county or other geographical entity, contributes to the nonattainment problem and should be included in the designated nonattainment area. The EPA issued guidance for States and tribes on determining nonattainment area boundaries on March 2, 2000, and on July 18, 2002, specifically for tribes providing the criteria that are to be used to evaluate the appropriate boundaries for a nonattainment area. The EPA has posted on its web site -- www.epa.gov/ttn/rto/areas -- data in standardized format relating to the criteria in its guidance to assist States, tribes and Regional Offices in determining boundaries. These data include but are not limited to monitoring information for all monitors across the country, modeling for certain areas of the country, emissions sources and inventories, census data and area-specific information regarding vehicle miles traveled. The criteria in EPA guidance are used to help resolve the issue of whether there may be areas that have emissions that contribute to a violation of the standard in a nearby area. O/G Evaluation We revised this section of the final report. # Report Methodology/MIRA The report states that MIRA uses data to evaluate the environmental impacts of EPA's policies and action to designate areas but does not state how this is done or what the impacts might be. This needs clarification. O/G Evaluation We removed this sentence from the final report. # 1-Hour and Preliminary 8-hour Ozone Designation Process The report states that EPA's guidance for the 1-hour ozone designations did not address a process for stakeholder participation. However, stakeholder involvement can occur at two stages. The first stage is participation at the State and tribal level, prior to the time their recommendations are made to EPA. Each State and tribe has its own process for including stakeholders when developing responses to requirements under the CAA. The EPA does not seek to dictate the States' and 22 Report No. 2002-S-00016 ------- tribes' processes for responding to the CAA's requirements. The second step could arise at the time that EPA determines whether to modify a State recommendation. The statute does not contain a provision requiring EPA to perform a county- by-county assessment when designating areas; rather it provides that EPA must designate all areas. The States, tribes and EPA must use their discretion in determining how to apply the criteria in EPA's guidance for determining boundaries across the country. For example, in the West, there are large areas of the country where there are no monitored violations of the ozone standard, and EPA is aware that population and source density make it highly unlikely that there is an area violating the standard or contributing to a nearby area that is violating the standard. Thus, those areas of the country may deserve less scrutiny than areas along the northeastern seaboard where there have historically been many monitored exceedances of the ozone standard and population and source density are higher. The report states that it "could not determine the number of counties measuring in attainment versus the number of counties without monitors." It is, however, very straightforward to determine the number of counties measuring in attainment and the number of counties without monitors. What the report may have been trying to address is the adequacy of the States' ozone monitoring networks. The EPA has guidance and regulations addressing networks with which States comply and when coupled with guidance on boundaries is adequate to determine air quality and contributions to nonattainment problems. The report continues with a discussion of the treatment of areas designated attainment/unclassifiable and suggests that areas without monitors are treated as if they were monitoring in attainment,. This statement is very misleading. The EPA treats attainment and unclassifiable areas identically for purposes of designations primarily because there is no difference in the CAA requirements for attainment and the unclassifiable areas. The CAA also recognizes that larger urban areas are likely to have similar levels of ozone across the urbanized area and also have areas that contribute to those levels. Therefore, the CAA prescribed that even if portions of the C/MSA do not have monitors, the entire C/MSA be designated as nonattainment. Region 3 offers a reasonable solution to the problem of not having monitors everywhere - use of spatial interpolation of air quality data - for assessing whether an area is meeting the standard. The EPA is evaluating this approach for possible use in future designations. OIG Evaluation We do not dispute that stakeholder participation was part of the 1-hour ozone designation. However, our report states that we were unable to determine how stakeholder participation influenced the final recommendations made by the 2 3 Report No. 2002-S-00016 ------- states. It is important to note that there are many stakeholders, other than states and tribes, including industry, environmental groups, and citizens. It is not clear how the 1-hour ozone designations solicited or incorporated these stakeholders' concerns. We agree that the Clean Air Act requires areas be designated for the ozone standard rather than counties. However, in order to designate areas, a county- by-county assessment must first be performed. We disagree with OAQPS that it is very straightforward to determine the number of counties measuring in attainment and the number of counties without monitors. The response does not explain how the public could compare the air quality condition in counties measuring attainment versus the air quality condition in those that are considered attainment/unclassifiable by default. This default occurs when a county does not have a monitor to measure air quality and is not part of a Metropolitan Statistical Area or Consolidated Metropolitan Statistical Area. # Benefits of MIRA Demonstrated The report states that there was not always documentation to clearly support why a particular county was considered in attainment or nonattainment in EPA's designations promulgated in 1991. As stated above, the statute does not require EPA to consider and document its designations decisions on a county-by-county basis, moreover, EPA believes that the 1991 designations decisions were fully supported and adequately documented in the record for that action. It is important to note that, with the exception of one area in the entire country, those regulations were not challenged. OIG Evaluation In our limited review of the 1-hour ozone designation, we found instances where designation decisions were changed and there was no documentation in the docket to support the changes. However, it was not the intent of this review to criticize the 1-hour designation process but rather to help the Agency improve the 8-hour designation process. 24 Report No. 2002-S-00016 ------- Appendix D Example of Sequential vs. Simultaneous Decision Approach Prepared by: Region 3 MIRA Team In this example, there are four criteria (air quality, emissions, growth, and population), and three counties that need to be ranked with regard to nonattainment. The following chart, based on hypothetical data, ranks each county for all four criteria on a scale of 1 to 9, with 9 representing the most nonattainment: County A County B County C Air Quality 9 6 2 Emissions 2 4 7 Population 3 6 5 Growth 4 5 8 If a sequential model is used, for air quality, County A would be ranked the most nonattainment and County C the least. However, since nonattainment is not only about air quality status but contribution to downwind nonattainment, the other decision criteria must be considered. This can cause confusion. For example, while County C has the best air quality, it has the highest emissions of the three counties. Therefore, simply comparing rankings for the most important air quality criteria, or summing all four criteria for each county, is not appropriate. This presents a dilemma in terms of nonattainment ranking if the sequential model is used, unless decision makers set County A aside (as nonattainment) based solely on its air quality value and then evaluate the other counties based on the next single decision criterion selected by the decision makers, such as emissions, population, and so on. Proceeding in this manner effectively ignores all but the single criterion that is currently the subject of the evaluation. Therefore, if County A is set aside based on its poor air quality value (ignoring its low emissions), Counties B and C are now evaluated for the next most important criterion: emissions (or some other one if the decision maker chooses). County C's emissions are the highest of the three counties, and if we select it for nonattainment, we have effectively ignored its relatively good air quality. If this method of setting aside counties as nonattainment once they rank high under one criterion is used, decision makers will find that nearly all counties evaluated in this manner will be eventually considered nonattainment. In this example, all 25 Report No. 2002-S-00016 ------- counties will also eventually be considered nonattainment because although air quality is the most important criterion (to the exclusion of the others) in the initial evaluation, each criterion in turn will be evaluated as if it were the most important criterion for the remaining counties. In contrast, a simultaneous analysis would permit decision makers to determine which criteria should be considered more important than the others and give them more weight. For example, the most important "air quality" criteria can be considered 40 percent of the ranking, while the less important "population" would only be given a 10-percent ranking. This would enable decision makers to sum all of the criteria for each of the three counties and have composite scores that can be used for comparison purposes, as show below. County A County B County C Air Quality 9 x 40% 6 x 40% 2 x 40% Emissions 2 x 30% 4 x 30% 7 x 30% Population 3x10% 6x10% 5 x 10% Growth 4 x 20% 5 x 20% 8 x 20% TOTALS 5.3 5.2 5.0 The result may be what is expected, but it is important to note that these results are dependent on the preferences used. The results may be different if the preference value of air quality relative to emissions and the other criteria are changed, even if air quality is still ranked as the most important criterion. It should be noted that this technique does not require that the decision maker designate any particular county attainment or nonattainment - it simply informs the decision maker of the 'nonattainment-ness' of each county relative to the others based on the decision maker's preferences and the data. 26 Report No. 2002-S-00016 ------- Appendix E Distribution List Office of Inspector General Office of Inspector General (2410T) Headquarters Office Assistant Administrator for Air and Radiation (6101A) Director, Air Quality Strategies and Standards Division (C50401) Audit Followup Coordinator - Office of Air Quality Planning and Standards (C40402) Comptroller (2 731A) Agency Followup Official (2710A) Agency Audit Followup Coordinator (2724A) Associate Administrator for Congressional and Intergovernmental Relations (1301 A) Associate Administrator for Office of Public Affairs (1701 A) Director, Office of Regional Operations (1108A) Regional Offices Director, Air Protection Division - Region 3 (3AP00) Chief, Air Quality Planning and Information Services - Region 3 (3AP21) EPA Regional Administrators (Regions 1-10) 27 Report No. 2002-S-00016 ------- ------- |