OFFICE OF INSPECTOR (iI< N 1 
-------
 CAA                    Clean Air Act



 CWA                   Clean Water Act



 EPA                    Environmental Protection Agency



 EPCRA                 Emergency Planning and Community Right to Know Act



 FIFRA                  Federal Insecticide, Fungicide, and Rodenticide Act



 IQ Guidelines           Information Quality Guidelines



 NAAQS                 National Ambient Air Quality Standards
	**..	,..j	»».„	.....................P.......,...........*...,....,.....,,...	.


 NESHAP                National Emission Standard for Hazardous Air Pollutant



 NOx                    Nitrogen Oxides



 NPDWR                National Primary Drinking Water Regulations



 OAR                    Office of Air and Radiation



 OMB                    Office of Management and Budget



 OPEI                    Office of Policy,  Economics, and Innovation
	 t.**..,.....„„„ j..,.„...„„	••••••.•-«....•"».«»...........—...............«.........,«..,««...»..»,„.,...*.	


 OPPTS                 Office of Prevention, Pesticides, and Toxic Substances



 ORD                    Office of Research and Development



 OSWER                Office of Solid Waste and Emergency Response
...««».......	....A.......*...	,..,......,«»,,.«»..........„.......,.„..,,.,...»....»....,...........,....«..,.».,»,,,.».,	


 OW                     Office of Water
	..,.,,,,,.,,,,,..	4............	....•<..	,.,»..,,,..,.,..,.»,.,,.»..,..,...................,..„,.....*..„„..,	„.,,„„,,„„,,,,„.,.


 RAPIDS                Rule and Policy  Information Development System
....,,„.,,„..,..	*............	.••••• »*».,.„,,,,,.............. ..••......,„..,..,.................•......,...„.,„„.„„„	


 RCRA                  Resource Conservation and Recovery Act



 SDWA                  Safe Drinking Water Act
	*	

 TSCA                  Toxic Substances  Control Act

-------
                 UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                                  WASHINGTON, D.C. 20460
                                                                            OFFICE OF
                                                                        INSPECTOR GENERAL
                                 November 15, 2002

MEMORANDUM
SUBJECT:   Report 2002-P-0003
             Pilot Study: Science to Support Rulemaking
FROM:      Jeffrey Harris, Director, Cross-Media Issues
             Office of Inspector General

TO:          Thomas Gibson, Associate Administrator for
               Policy, Economics, and Innovation

             Paul Oilman, Science Advisor to the Agency

This report concludes the pilot study for an evaluation of science to support rulemaking
conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection
Agency (EPA).  It addresses the role played by science, the genesis of this science, and the extent
to which the science was peer reviewed. Although this was only a pilot study, we believe the
results are sufficient to offer suggestions to improve the transparency and consistency with which
science is applied to rulemaking in the EPA.  Some of the suggestions elaborate on
recommendations made in June 2001 to the Administrator by EPA's internal task force on
improving rulemaking. Our report also discusses the strengths and weaknesses of the
methodology we used in conducting the pilot study.

We appreciate the cooperation we received from the primary contacts for the rules covered by the
pilot study, and those others involved in these rulemakings who took the time to answer our
questions.

Since this report does not include recommendations to which you must respond., no action is
required.  In accordance with EPA Order 2750, we are closing the report in our tracking system
upon issuance.  We have no objections to the further release of this report to the public. For your
convenience, this report will be available at http://www.epa.gov/oigearth/eroom.htm.

If you have any questions about this report, please contact Christine Baughman at 202-566-2902.

-------

-------
             Rules, also known as regulations, are a critical cornerstone of the Environmental
             Protection Agency's (EPA's) mission.  By statute and executive order, they are to
             be based on the best reasonably obtainable scientific, technical, economic, and
             other information. EPA Administrator Whitman noted that the Agency's ability
             to "accomplish our mission and continue to have a meaningful impact on the
             quality of life for all Americans to a large extent is based on our ability to more
             fully integrate science into our programs, policies and decisions."

             By identifying the science that was critical to rules promulgated by EPA in the
             past, we hoped to determine whether better research planning, application of
             science to rules, and explanation of the role of science in rules could achieve
             improvements in the science behind future environmental regulations. By critical,
             we do not mean that the rules were promulgated because o/the science, but that
             without the science the rules would have been different, or even impossible to
             promulgate. We completed a pilot study consisting of 15 case studies involving
             16 of EPA's significant rulemakings to determine whether a full study would be
             useful and feasible.

Results In Brief

             The rules included in the pilot study were not a representative statistical  sample of
             EPA rules, and we did not identify all of the critical science inputs for every rule.
             However, we made observations that we believe transcend these limitations and
             will be useful to EPA rulemakers:

                    Role of Science: Science played an important role in the rules, but that
                    role was not always clear.  Even though the rules included in this pilot
                    study depended on hundreds of scientific documents, because the role of
                    science often was not presented in a manner consistent with the
                    conventions of communicating scientific information, it maybe unclear
                    what science was critical and why.

                    Sources of Science: Although critical science originated from a variety of
                    sources, research performed under contract to EPA and the regulated
                    community by private sector firms was the most common source. Grants
                    and cooperative agreements accounted for about 8 percent of the work.

                    Data: Some of the rules would be based on fewer assumptions if EPA
                    had more data and fewer scientific "blind-spots."
                                                                     Report 2003-P-00003

-------
                   Peer Review: The critical science supporting the rules often was not
                   independently peer reviewed. Consequently, the quality of some science
                   remains unknown.

             The pilot study identified significant challenges to identifying target populations
             of EPA's non-significant rules and identifying critical science consistently, and
             we do not intend to pursue a full study at this time.

Suggestions

             Based on our observations in the pilot study, we offered several suggestions:

             *     EPA should ensure that science in rulemaking is presented in a way that is
                   apparent and consistent with the conventions of science.
             •     Information technology could be better used to ensure that the
                   Administrator, Congress, and the public could determine that the science
                   behind rulemaking is adequate.
             *     The critical science behind EPA's rules should consistently be
                   independently peer reviewed.

Agency Comments and OIG Evaluation

             We received both formal and informal responses from EPA management.
             Generally, the comments from Agency officials were supportive of the
             suggestions, although they identified some concerns about the details of their
             implementation. EPA's Science Advisor has committed to review the Agency's
             progress in implementing its Peer Review policy during the coming year, and
             ensure that Agency decisions are based on sound science. We have incorporated
             many of the specific Agency comments directly into the report and its Addendum
             to improve clarity and factual accuracy. Because of the commitments made by
             Agency management in their comments, we believe the report's observations may
             serve as a baseline against which the Agency can chart progress.
                                                                   Report 2003-P-00003

-------
Executive Summary	    i

What We Did and Why 	    1

       We gained an understanding of rulemaking at EPA  	   2
       We identified significant rules promulgated since 1990 and
              selected a sample for 15 case studies	   3
       We sought to Identify the critical science behind each rule	   5
       We identified the sources of the critical science 	   8
       We asked the respondents about science gaps and science quality 	   8
      ' We identified the type of peer review undergone
              by the critical science	   9

What We Learned About the Science Behind the Rules	   10

       Science played an important role in the rules, but that role was not always clear  	   10
       Although critical science originated from a variety of sources,
              the private sector was the most common source 	   15
       More data and fewer "blind spots" could reduce assumptions	   17
       Critical science supporting the rules often was not independently peer reviewed  	  18
       Suggestions  	   20
       Agency Comments 	   21
       Response to Agency Comments  	  29

Pilot Lessons Learned	   33

Exhibits

        1:  Significant Rules Finalized -1990-2001  	  35
        2:  Numbers and Completeness of Critical Documents 	  39
        3:  Who Performed the Science Work	  40
        4:  Who Funded the Science Work	  41
        5:  Funding Mechanisms Used  	  42
        6:  Peer Review Actions Taken	  43
        7:  EPA Science Advisor Comments	.-	  44
        8:  Office of Water Comments	  48
        9:  Office of Prevention, Pesticides and Toxic Substances Comments  	  58
       10:  Office of Policy, Economics, and Innovation Comments	  69
       11:  Report Distribution 	  78
                                                                      Report 2003-P-00003

-------

-------
              Rules (also called regulations)1 are a cornerstone of the Environmental Protection
              Agency's (EPA's) mission. Rules are, first and foremost, legal documents written
              to meet legal goals. Nonetheless, they are to be based on the best reasonably
              obtainable scientific, technical, economic, and other information. In June 2001, a
              Regulatory Development Task Force established by the EPA Administrator to
              improve EPA regulations offered several recommendations to ensure that science
              has a more prominent role in EPA decision-making, and that there is timely and
              thorough analysis of issues.  The recommendations included improving existing
              processes to more effectively ensure broader Agency involvement and executive
              input on cross-cutting scientific, economic, or policy issues, as well as involving
              EPA scientists in determining  needed analyses and research, identifying
              alternatives, and selecting options. As noted in a May 2002 memorandum from
              the EPA Administrator, the ability of the Agency to "accomplish our mission and
              continue to have a meaningful impact on the quality of life for all Americans to a
              large extent is based on our ability to more fully integrate science into our
              programs, policies and decisions."

              By understanding what science was critical to the rules promulgated by EPA in
              the past, we hoped to determine whether better research planning, application of
              science to rules, and explanation of the role of science in rules could achieve
              improvements in the science behind future environmental regulations.  By critical,
              we do not mean that the rules were promulgated because of these documents;
              rather, without the documents, the rules would have been different, or even
              impossible to promulgate.

              We believed that such a study  should answer the following questions:

              >  What role does science play in supporting rules?
              *•  What science provided the most critical support for the rules and what was its
                 genesis?
              >  What were the most significant gaps in the science underpinning the rules, and
                 could they have been filled with better research planning and communication?
              »•  How was the quality of the science evaluated?
              *•  Do rules with better scientific underpinnings exhibit measurably better
                 outcomes?
1       "Regulations," or "rules," are agency statements of general applicability and future effect, which the
agency intends to have the force of and effect of law, and that are designed (1) to implement, interpret, or prescribe
law or policy, or (2) to describe the procedure or practice requirements of an agency. "Rulemaking" is synonymous
with "regulatory action."

                                            1                          Report 2003-P-00003

-------
             In August 2001, we started a pilot study to determine whether we could answer
             the above questions and, if so, the level of resources required. Early in the pilot
             we consulted two Agency groups. The Research Strategies Advisory Committee
             of the EPA Science Advisory Board is comprised of representatives of the various
             advisory boards that advise on science in rulemaking in EPA. EPA's Regulatory
             Steering Committee is the cross-agency group most closely involved with
             rulemaking in EPA.  With the advice of both groups, we designed this pilot study,
             which we completed June 2002.

             We began by identifying EPA's significant rules finalized after 1989, and then
             selected a small sample to pursue as case studies. For each rule in the sample, we
             identified primary contacts involved in the rulemaking and contacted each
             individual via e-mail for assistance in identifying the critical science documents
             underlying the rule. For each document, we established: who conducted the
             study, how the study was funded, and whether and how the document was peer-
             reviewed. Each step is discussed in more detail below, and the findings for each
             selected rule are summarized in the case studies located in an Addendum to this
             report available at http://www.epa.gov/oigearth/eroom.htm.

             Except for the limitations identified in this paragraph, the pilot  study was
             conducted in accordance with the Government Auditing Standards for
             performance audits.  Our research into the management controls over developing
             regulations was limited to a general understanding of the process, both for rules in
             general and the rules in the sample.  We did not test any controls (e.g., comparing
             the planning requirements to the actual plans for the rules in the pilot study).
             Also, because this was only a pilot study, not all of the attributes of a finding were
             pursued, such as those related to cause (e.g., why there was a lack of peer review).

We gained an  understanding of rulemaking at EPA

             EPA develops many types of regulations based on a variety of mandates, but
             certain processes are common to most rulemakings. Typically, once a rulemaking
             is triggered by a statute, court order, petition, or executive initiative, the
             appropriate program office(s) and the Office of Policy, Economics, and
             Innovation (OPEI) prioritize staff time and resources, determine the regulatory
             strategy to be pursued, identify the science and data needed and available, identify
             a set of regulatory options, select an option, and propose the rule in the Federal
             Register. The notice of proposed rulemaking in the Federal Register includes a
             preamble that describes the basis and purpose of the rule, the alternatives
             considered, and the underlying supporting information. To send a clear message,
             the preamble must be written in plain language.  The underlying supporting
             information is found in the "dockets" for the rulemaking (drawers of paper files
             available for public perusal and, for more recent rules, the equivalent electronic
             files available on EPA's website). The dockets also contain all public comments
             received, EPA's responses to public comments, and other information EPA

                                           2                        Report 2003-P-00003

-------
             considers relevant to its decisions. After allowing for public comment (typically
             60 days), EPA finalizes the rule by publishing it in the Federal Register, with a
             new preamble responding to important comments and identifying any changes in
             the rule since its proposal.

             EPA's OPEI estimates that EPA publishes Federal Register notices for between
             1,000 and 1,300 rules  each year. Approximately 20 of these rules each year are
             "significant" under Executive Order 12866 (see box). These significant rules
             must be reviewed by the Office of Management and Budget, unless they were
             specifically exempted. EPA proposes or finalizes approximately 200 rules each
             year that have a lower level of impact than the significant rules but are still
             generally national in scope. The remaining notices are for rules that primarily
             impact individual States, Tribes, sites, or manufacturers, or involve minor
             modifications and corrections to existing rules. Many proposed rules never
             become final, and proposed significant rules may become non-significant on
             finalization if their estimated costs decrease, they are determined to modify
             existing significant rules, or the Office of Management and Budget determines the
             rule is not significant.

We identified significant  rules  promulgated after 1989 and
selected a sample for 15 case studies

             To develop a sample of rules to study, we first needed to identify a target
             population of rules of interest. We decided to focus on significant rules (see box)
             finalized after 1989. OPEI maintains a database of planned and ongoing
             rulemakings - the Rule and Policy Information Development System (RAPIDS) —
             but at the time of the study, did not maintain a list of rules finalized over the
             previous 10 years and was not able to construct a complete list of the significant
             rules before 1998 using RAPIDS.

             We used the Federal
             Register and EPA's
                                           Under Executive Order 12866, "Significant" rules:
                                           *  Have an annual effect on the economy of $100 million
                                             or more or adversely affect in a material way the
                                             economy; a sector of the economy; productivity;
                                             competition; jobs; the environment; public health or
                                             safety; or State, local, or tribal governments.
                                           *  Create a serious inconsistency or otherwise interfere
                                             with an action taken or planned by another agency.
                                           «•  Materially alter the budgetary impact of entitlements,
                                             grants, user fees, or loan programs or the rights and
                                             obligations of recipients thereof.
                                           *  Raise novel legal or policy issues arising out of legal
                                             mandates, the President's priorities, or the principles
                                             set forth in the Executive Order.
website to identify the 89
significant rules finalized
in 1990 through 2001,
which are listed in Exhibit
1. The list of rules
promulgated before 1994
may be incomplete since
EPA's web-based materials
tended to be dated 1994
and later.  Focusing on the
75 rules finalized from
1994 on, we show in
Figure 1 that more than half (39) of these rales were issued under the Clean Air

                              3                         Report 2003-P-00003


-------
Act. Most of the rest involved either the Clean Water Act (16) or Safe Drinking
Water Act (6),  Of these 75 rules, 39 met the significant criteria because of their
expected economic impact.  (Note: because one of the rules involved two laws,
Figures 1  and 2 total 76 and 40, respectively.) As shown in Figure 2, a slightly
higher proportion of the economically significant rules involve the Clean Air Act,
but the economically significant rules are otherwise similar to the larger
population of significant rules.
             Figure 1
            1I«4-2«M Sianfffcam Rutei
                ByUw
                   Figure 2

                   1H4-2M1 EcenomfeiHy
CAA:   Clean Air Act
CWA:   Clean Water Act
EPCRA: Emergency Planning and
         Community Right to Know Act
FIFRA:  Federal Insecticide, Fungicide, and
        Rodenticide Act
RCRA:  Resource Conservation and Recovery Act
SDWA:  Safe Drinking Water Act
TSCA:  Toxic Substances Control Act
For the pilot study, we selected 16 significant rules finalized during 1991 through
2001.  Three of the rules concerned related land disposal restrictions for solid
wastes, so we combined them into one case study. The Integrated Pulp and Paper
rule involved two different laws, so we divided it into two case studies.  Thus,
there was a total of 15 case studies in the pilot. Each case study is identified in
Table  1 and detailed in the Addendum report, which is available at
http://www.epa.gov/oigearth/eroom.htm.
                                                           Report 2003-P-00003

-------
                                              Tablet:
                                           Case Studies
 1996
.1998,
 1994
,1995,
 1996
                J997.,
                1998
                .1998,
                1998
                -1998ii
                'l998
               JS85..
                2001
                         RCRA
                         CAA,.
                         CWA
CAA,,.,,
TSCA,
CAA
CWA(i>i
SDWA
TSCA,
"CM"
CAA,	
RFRA
                                      Acid Rain Permits
             Land Disposal Restrictions
.Grea^Lakes.WaterQual.itx,


Biotechnology
il>*lil*iiBtlt«illlllllll*ri4ll«Bil**Mt*Mtiiit*tittii
                       Polychlorinated Biphenyls
                       >iiMiMniiiii«tMtiii*MiiMMMMirMMMii*Hnt*iMii*iii»i
                       -Plaflt-lncorporMgAProtectants
                                                      ..§..
                                                      "e"
                                                               .1.
                                                                '$'.
                                                               "9"
                                                               ..19..
                                                                11
                                                               M12,
                                                                13
                                                               ..H.
                                                               "i's"
             This judgment sample was selected to represent as wide a range of statutes as
             possible, and to span the decade for rules under the Clean Air, Clean Water, and
             Safe Drinking Water Acts, but it is not statistically representative of EPA's
             significant rules during the period. For example, there was only one rule in the
             sample establishing a National Emission Standard for Hazardous Air Pollutant
             (NESHAP), whereas six NESHAPs were finalized between 1994 and 2001.
             Notably, there are no National Ambient Air Quality Standards (NAAQS) in the
             sample, even though three were finalized by EPA during this period. NAAQS are
             especially science-rich rules, but we excluded them because the particulate matter
             and ozone NAAQS were under remand to the District of Columbia Circuit Court
             during the study.

We sought to identify the critical science behind each rule

             For purposes of the pilot project, "science" included scientific and technical work
             products addressing, for example, pollutant emissions, environmental transport,
             cost impacts, exposure to humans and ecosystems, the effects of such exposure,
             risk assessments, monitoring methods, and databases, i.e., the kinds of documents
             that would be produced by scientists or engineers. We included economic
             analyses only when they had a critical impact on the rule. As noted earlier, by
             "critical," we do not mean that the rules were promulgated because of these
             documents; rather, without the documents, the rules would have been different, or
             perhaps would have even been impossible to promulgate.
                                                                    Report 2003-P-00003

-------
We had planned to identify the critical science documents supporting each rule by
relying on a variety of people involved in developing the regulation to identify the
documents. We asked the primary contact to identify one or more additional
contacts from the following groups: scientists from the program office who
worked on the rulemaking; EPA Office of Research and Development (ORD)
scientists who provided expertise to the rulemaking team; the Senior Executive
from the program office immediately responsible for the rule; independent peer
reviewers; environmental stakeholders; and representatives of the regulated
community. With this range of perspectives on the critical science, if there was
wide consensus, we could be reasonably sure we had identified all the critical
science. It also would require less reliance on an evaluation team to make
independent judgments about what science was critical.

We started by identifying primary contacts for each rule (the people from the EPA
program office who led the rulemaking). After 6 weeks, we had identified 83
contacts (including the primary contact for each case study). We sent e-mails to
the contacts, asking each of them to identify 5 to 10 science references (e.g.,
papers, reports) that they believed most critically influenced the rulemaking  and,
without which, the rules would have been substantively different. We asked that
they consider several categories of science that may be relevant to the rules.  We
also asked - if they had the discretion, funds, and "20/20" hindsight - what
science gaps would they have tried to fill that might have made the rule
substantially different.  Finally, we asked how they would rate the scientific
quality of the rule, on a scale of 1 (low) to 5 (high).

We received at least one detailed response for only 7 of the 15 case studies by late
November 2001, but  we received no helpful responses  from 58 of the contacts.
We received no useful e-mail responses from any of the EPA executives, peer
reviewers, environmental stakeholders, or representatives of the regulated
community. We then conducted interviews with primary contacts and other EPA
staff.  The interviews yielded specific references and more general explanations of
how the rulemaking proceeded, and advice to  look in the docket for reports on
particular studies. We were thus able to identify additional critical science
documents. Of the critical documents, we identified almost half from reading the
materials in the dockets. The pilot team identified all of the critical documents for
two of the rules (Cases 2 and 8). Because we were not able to interview peer
reviewers or stakeholders, and because we relied on only one or two Agency
contacts and the preambles and technical support documents developed  by EPA,
our results may reflect an EPA bias on what, science was critical.

Our process did not result in the identification of all of the critical science
documents for all of the rules. We believe we did identify all of the major
technical support that embodied  the final process of gathering together the science
and other information to support the rules for all of the case studies (see Exhibit
2). These documents had titles such as background information documents,


                              6                         Report 2003-P-00003

-------
technical support documents, regulatory impact analyses, and economic impact
analyses. They were often, but not always, cited in the preamble. If the preamble
was the only place where the science was brought together, we identified the
preamble as a critical document.

We encountered more difficulty as we identified the critical documents cited in
the major support documents. These documents included other documents
developed specifically to support the rule (e.g., databases, regulatory methods,
health criteria documents), as well as documents from the primary science
literature (e.g., toxicology data supporting a criterion, models used to do an
analysis, or even the data or mechanisms necessary to the model itself). For
example, two of the rules involved the development of National Primary Drinking
Water Regulations (Cases 2 and 11).  The standards under the regulations
(maximum concentrations of chemicals in the finished water) depend in part on
the risk to human health posed by the chemical. Health Criteria Documents
summarize and interpret the available toxicology and epidemiology available in
the literature to arrive at various "criteria" values that put the heaviest weight on
(reliable) studies that show effects at the lowest chemical concentrations.  These
underlying studies then become critical documents (there are usually more than
one per health criteria document because of the different types of effects, e.g.
cancer, reproductive effects, etc.). Thirty-four of these underlying criteria
documents were identified for Cases 2 and 11 alone. Because the number of
critical documents increases exponentially as one goes backward through the
citation chain, locating all of them becomes a very time-consuming process.

We believe we identified all of the critical documents cited in the major technical
support documents for six of the rules ("level 2" documents in Exhibit 2). In
three of the case studies, we went into the references for this second group of
documents, and identified the most critical references identified as "level 3"
documents in Exhibit 2 (e.g., the chemical mechanism upon which a component
of a model used to support a rule was based, or research that led to the
identification of a problem that led to the recognition of the problem that the rule
was promulgated to address).

Exhibit 2 shows a wide range in the numbers of critical documents per case. Two
of the cases for which we believe we have a complete list of level 2 documents
include only 10 and 12  critical documents, while two of the cases for which we
believe the lists are incomplete include as many as 25. Adding level three
documents also increases the number. We believe this wide range arises from a
combination of:  differences in the scientific complexity of the rules, which affects
the type of critical documents needed to support the rule; and failure to identify all
of the critical documents for each rule.

These  factors complicate the interpretation of the statistics for this or any full
study in the future. Because we did not identify all of the critical documents for


                              7                        Report 2003-P-00003

-------
             all the rules, comparative statistics on critical science documents (e.g., proportions
             of critical science documents funded under grants, or performed by other federal
             agencies) could be subject to bias, and interpretations based on such calculations
             could be misleading. Overcoming this problem would present a challenge for
             expanding the pilot study.

We identified the sources of the critical  science

             Once a critical science document was identified, we had to find a hard or
             electronic copy. We then examined it to determine who performed the science
             work, who funded it, the funding mechanism used, and whether and how it was
             peer reviewed. We categorized the resulting information for statistical analysis.

             Who performed the research often was identified on the title page of reports, the
             by-lines in journal articles, or in the acknowledgments (e.g.,  a report might state
             that valuable contributions had been made by a company or organization).
             Funding information (both who and what mechanism) often  could be  found on the
             title page of reports or the acknowledgments section of journal articles.  We found
             that some documents had an EPA cover, but inside indicated that the document
             had been prepared for EPA by a contractor. In those cases, we classified the
             report as funded by EPA, but prepared by a private sector firm under contract.  If
             instead, the report only acknowledged substantial support from a contractor, we
             classified the report as having been produced by EPA, with a contract as the
             funding mechanism. The criteria for the classifications are more fully explained
             in the Addendum.

             The document did not always address one or more of these characteristics. In
             some cases, we were able to conclude that a program office funded a technical
             support document (no other organization would have reason  to fund it), or that the
             only allowable or likely mechanism to support a scientific study was a contract.
             When all else failed, we asked those who had responded to our inquiries, either
             during the development of the case study or upon final review.

We asked the respondents about science gaps and science quality

             At the outset of the pilot study, we had hoped to identify indicators of the
             scientific quality of final rules that we could relate to the characteristics of the
             critical science inputs. For example, did more  rigorous peer review or the
             extensive use of science from academic labs lead to "better" rules? We were
             unable to identify any objective indicators, so at the suggestion of the  Research
             Strategies Advisory Committee, we asked respondents (some of whom were not
             on our initial list of contacts) to: (1) identify any additional science that would
             have been useful if it had been available ("gaps"); and (2) rate the  science quality
             of the rules on a scale of one to five, with five being the highest quality.
                                                                    Report 2003-P-00003

-------
We identified the type of peer review undergone
by the critical science

             Science used by EPA to support rules should be credible. Peer review is EPA's
             preferred method of ensuring credibility. Peer review is the documented, critical
             review of a work product, performed by experts who are independent of those
             who developed the product. In a 1994 policy, EPA specifically required peer
             review for scientifically and technically based work products intended to support
             EPA decisions. EPA's December 1998 Peer Review Handbook expanded this
             policy. Regulatory Management Guidance based on the 1998 Handbook was
             issued by EPA's Office of Policy in June 1999, requiring that all rules undergoing
             Final Agency Review must include either a statement that "no major scientific or
             technical documents were utilized to support the rulemaking," or a statement of
             compliance with peer review requirements. In a fact sheet related to this
             guidance, the criteria identified for such documents are very similar to those for
             significant rules in Executive Order 12866. The December 2000 Peer Review
             Handbook for the first time required that the final work product itself must
             describe the peer review to which it was subjected, or note that it was not peer
             reviewed.  A regulation itself is not subject to EPA's peer review policy, even
             though the major scientific work products that support it are.

             Peer review will become even more important as EPA implements recent
             guidance from the Office of Management and Budget (OMB) developed to
             comply with the Treasury and General  Government Appropriations Act of Fiscal
             Year 2001. The OMB  "Guidelines for Ensuring and Maximizing the Quality,
             Objectivity, Utility, and Integrity of Information Disseminated by Federal
             Agencies" state:

                    As a general matter, in the scientific and research context, we regard
                    technical information that has been subjected to formal, independent,
                    external peer review as presumptively objective.

             EPA's related guidelines, which became effective on October 1, 2002, are
             consistent with many existing practices and policies, including the above
             mentioned peer review policy, according to the Associate Administrator for
             Policy, Economics, and Innovation,

             Some documents indicated they had been peer reviewed, and by whom.  Critical
             documents in scientific journals were subject to the peer review policies of the
             journals. For the many documents that did not indicate their peer review method
             or status, we consulted EPA's Peer Review Database and asked respondents.
                                                                    Report 2003-P-00003

-------
                ,.fl£ • -^r^v^™™-^
                Wfiat We  LearnedFA&ou
           *.&**• ^
           the Science  Behind^tnelRuil
                                  '                           ' '
            The rules included in the pilot study were not a representative statistical sample of
            EPA rules, and we did not identify all of the critical science inputs for every rule.
            However, we made observations that we believe transcend these limitations and
            will be useful to EPA rulemakers:

            *•  Science played an important role in the rules, but that role was not always
               clear.

            *•  Although critical science originated from a variety of sources, the private
               sector was the most common  source.

            *•  More data and fewer scientific "blind-spots" could reduce assumptions.

            *•  The critical science supporting the rules often was not independently peer
               reviewed.

            At the end of this section, we provide some suggestions that, based on our
            observations, should help EPA improve its use of science in making rules.

Science played an important role in the rules,
but that role was not always clear

            We found that science played an important role in the rules, but that role was not
            always clear. Even though the rules included in this pilot study depended on
            hundreds of individual scientific documents, because the role of science was
            generally not presented in a manner consistent with the norms of science, it may be
            unclear to the public what science was critical or why.

            The role of science

            We identified 452 critical science  documents for the 16 rules in the pilot study.
            Each of these documents, had the  results been different, may have affected who
            was subject to the regulation, their cost of compliance, or risk to the public and the
            environment.
                                      10                      Report 2003-P-00003

-------
            The number of documents used per case study ranged from 8 to 852. Even though
            the 16 rules in the pilot are not a representative sample numerically, they are based
            on the same statutes as those for EPA's other significant rulemakings in the 1990s
            (Exhibit 1), so we believe the number of critical documents for these rules is not
            atypical of the importance of science in the formulation of EPA rules.

            Some examples of critical documents follow.

            »   Four of the rules in this pilot study set water quality standards for drinking
                water under the Safe Drinking Water Act (Cases 2 and 11) or for discharges to
                lakes and rivers under the Clean Water Act (Cases 6 and 10). Criteria
                documents summarized and interpreted the available toxicology and
                epidemiology available to arrive at "criteria" values - concentrations deemed
                "safe" for exposure to people and ecosystems. Other documents synthesized
                the likelihood of exposure of people and ecosystems to pollutants, as well as
                treatment costs.  These data were used to set the enforceable standards.  Had
                any of the 186 critical science documents been different, it is reasonable to
                expect that one of the enforceable standards could have been set at a different
                number.

            >   One rule involved setting emissions caps for nitrogen oxides (NOx) for 22
                eastern States by demonstrating that the caps would significantly reduce
                contributions to nonattainment of the NAAQS for ozone in downwind states
                (Case 13). There was no requirement to assess the risk of non-attainment of
                the NAAQS on health or welfare3.  Instead, most of the 42 critical science
                documents focused on establishing NOx emissions inventories and modeling
                the chemistry and downwind transport of ozone and its precursors.  This was to
                show which of the States would be required to reduce emissions, and that the
                proposed caps would significantly reduce nonattainment in the downwind
                states.  Without the inventories and modeling, there would be no scientific
                basis for showing which states were significant contributors, and some of the
                estimates would have been different.  That could have resulted in some States
                not being subject to the rule.  We also identified critical science documents
                without which it is reasonable to believe the section authorizing the rule might
                not have been included in the Clean Air Act.

            +   Several rules (Cases 1, 7, and 9) required that a particular technology must be
                used, or an emissions limit equivalent to using that technology achieved. These
                rules, involving a total of 50 critical science documents, required that EPA
                develop databases to characterize the universe of sources to be regulated
       Case 4, which had 85 critical documents, represents a cluster of 3 related significant rules on land disposal
restrictions that were finalized over a span of 5 years.

       The ozone NAAQS was the subject of a significant 1997 rulemaking.

                                          11                         Report 2003-P-00003

-------
   (municipal waste combustors, municipal landfills, and pulp and paper mills,
   respectively) and models to estimate emissions from these sources. Without
   the data and models, decisions about which sources would not be regulated,
   and the technology required for those that were regulated, would almost
   certainly have been different.  In the first two cases, EPA could have made an
   administrative decision whom to regulate, but the decision would likely have
   been different.  In the mills instance, law requires all sources to be regulated,
   but emissions limits for new mills must be no higher than those at the best
   existing mills, and standards for existing mills not at the "best" level also
   needed upgrades. Without an emissions model and database, EPA would have
   no legal basis to set the standards.

Role of science  not always clear

Even though science played a role in all of the rules in this pilot study, it may not
always be clear because of the way the role of science is presented in the
preambles. Science is communicated according to widely accepted professional
norms. For example:

*•  The question to be answered is introduced, along with any previous scientific
   results that the  current study builds on. Such results are explicitly referenced.

>  The methods used in the study are described in sufficient detail so that the
   study can be reproduced by others.

*•  The results are presented in the form of graphs and tables (not just the data that
   support the authors' conclusions), usually with estimates of statistical
   uncertainty. Authors discuss.how the data in each figure and table support and
   do not support  alternative answers and reach carefully bounded conclusions.

However, we saw little evidence of any of these conventions in communicating the
scientific underpinnings of the rules in the preambles, which form the legal basis
for the rule.  We found that the preambles did not consistently present the scientific
and technical questions to be answered in terms of exactly how the answers would
be used to support the rule.  The methods were not presented in sufficient detail to
reproduce the studies, or to understand how the studies were done. Preambles for
only 5 of the 16 rules presented any data tables, and only one presented any
indication of uncertainty estimates.

Two of the preambles provided examples of good practices in the presentation of
data:

*  The preamble of the regional  ozone rule (Case 13) describes the scientific
    approach taken to modeling, and tabulates the results of the model runs in
   terms of the quantitative contributions of upwind States to non-attainment of


                               12                          Report 2003-P-00003

-------
   the ozone NAAQS in each downwind State. The cost effectiveness of various
   ozone control options are compared in two other tables.

>  The preamble to the disinfectant and byproduct rule (Case 11) also presents
   data tables comparing compliance forecasts between the 1994 proposal and the
   final rule, different systems costs, populations potentially exposed, and cancer
   cases expected under different options. One figure shows the ranges of
   estimated benefits, the only example of uncertainty seen in any of the
   preambles, and others show how many households will incur different costs of
   treatment.

The preamble of a rule regulating emissions of air pollutants from landfills
(Case 7) instead illustrates an opportunity lost.

»•     The preamble explained that EPA decided to exclude 90 percent of the
       smallest landfills from the regulation, and to require that best demonstrated
       technology be applied to any of the larger landfills with estimated
       emissions of NMOC (non-methane organic hydrocarbons, which
       contributes to the formation of ozone) of more than 50 Mg/yr. The intent
       was to reduce these emissions from all landfills by 53 percent at a particular
       cost per ton removed. However, no data were presented showing the
       estimated emissions from landfills of different size classes, the estimated
       emissions reductions from each class  at different control levels, the
       corresponding costs, or any indication of the uncertainty in any of the
       estimates (emissions estimates were based on a mathematical model rather
       than actual measurements). Such a table  would have made it clearer why
       EPA had chosen the particular mix of landfill size and emissions caps in
       the final rule, based on a combination of science and economics.

Many of the technical support documents contained data supporting decisions, but
for many of the rules in the pilot  study, we observed no referencing or inconsistent
referencing of even the major critical science documents developed to support the
rule.  Although most of the preambles were meticulously referenced to previous
Federal Register notices and case law, only six preambles referenced the science
underpinnings.

»•  The municipal waste combustor rule (Case 1) cited seven documents at the
   beginning of the preamble, but did not tie the technical arguments in the
   preamble to the documents.

»•  The nonroad diesel rule (Case 14) cited specific science reports using footnotes
   and made some of the technical support documents available on the EPA web
   site.
                              13                         Report 2003-P-00003

-------
*•  The disinfectant and byproduct rule (Case 11) was referenced in the manner of
   a scientific paper, to a bibliography in a section in the preamble.

>•  The pulp and paper rule (Cases 9 and 10) referenced documents by the docket
   index number, but not by report title. Many of these documents were available
   on-line, but the docket index was not, so one could not go right from the
   citation to the on-line document.

>•  The biotechnology rule (Case 8) and the plant-incorporated protectant rule
   (Case 15) both cited references by number, which corresponded to a reference
   section in the preamble.

Finally, we found it difficult to determine the relative importance of science and
administrative discretion when preambles contained statements that began, "EPA
believes ..." or "EPA concludes...." In these cases, it was not always clear when
a decision was based on science or on administrative discretion and, if it was based
on administrative discretion, whether the science really mattered. Two quotes
from the regional ozone rule (Case 13), a rule with substantial scientific
underpinnings, are illustrative:

       ...these four jurisdictions rank among the six highest jurisdictions in the
       OTAG (Ozone Transport Assessment Group) region in terms ofNOx
       emissions density. EPA believes that this high density provides an
       appropriate basis for concluding that each of these four jurisdictions
       should be included as a significant contributor.

       Because no highly cost-effective controls are available to eliminate the
       remaining amounts ofNOx emissions, EPA concludes that those emissions
       do not contribute significantly to downwind nonattainment or maintenance
       problems.

The scientific basis for EPA's decision about which States to include as significant
contributors was air quality model runs that identified the degree to which upwind
States contributed to ozone formation episodes in downwind states. In these
quotations, it appears that EPA is arguing that emissions density, or even cost-
effectiveness of controls, are equally suitable criteria.

In summary, even though the rules included in this pilot study depended on
hundreds of individual  scientific documents, because the role of science was not
presented consistently in the preambles in accordance with the norms of science, it
may be unclear exactly what science was critical and why. We could find no
explicit guidance on the presentation of scientific findings necessary to support
EPA's rules on the "Process Guidance" section of EPA's on-line Regulatory
Reference Library, but  OPEI in its comments indicated that EPA's Risk
Characterization Guidelines may in part meet this need.


                              14                         Report 2003-P-00003

-------
Although critical science originated from a variety of sources,
the private sector was the most common source

             We determined (where possible) who performed the science, who funded it, and
             the funding mechanism used for the critical science documents identified for the
             15 case studies. For reasons explained earlier, we cannot generalize from the rules
             in the pilot to EPA rulemaking overall.  Nonetheless, we can still make some
             useful observations about the roles played by the various organizations conducting
             and funding critical science, and the funding mechanisms used for the rules in the
             pilot. Minimum numbers are cited in recognition that we did not identify all
             critical science documents in support of some of the rules.

             Who performed the  critical science?

             As summarized in Table 2 and detailed in Exhibit 3, critical research was
             performed by the private sector, EPA program offices, ORD, other Federal
             agencies, and other (non-Federal) government organizations (such as States).  We
             counted work as performed by an
             organization if the report was                       Table 2:
             under the organization's cover          Who Performed the Critical Science
             and did not indicate it was
             prepared by a contractor for the
             organization.  If more than one
             organization performed the work,
             we counted each so the total
             exceeds the number of critical
             documents (i.e., 452).

             Program offices or their
             contractors developed virtually all
             of the technical support
             documents that made the
             scientific case for the rule
             (Exhibit 3). Even for the two rules where State and Federal government agencies
             worked on research teams to develop technical support documents (the Great Lakes
             water quality guidance [Case 6] and regional ozone rule [Case 13]), the  EPA
             program offices developed the final technical support documents. EPA's ORD
             contributed 28 critical documents, including health criteria documents, monitoring
             methods, assessment protocols, engineering studies, and air quality models and
             field studies.

             Other Federal agencies  contributed 24 critical studies to the rules in the  pilot.
             Almost half of these supported the Great Lakes water quality guidance,  and
             involved the National Oceanic and Atmospheric Administration and the U.S. Fish
Private sector
EPA in-house - program office

Academia
EPA in-house - ORD
Other (non-Federal) government

Other Federal agency
Unknown
238
 95
 68
 28
 25
 24
                                          15
                      Report 2003-P-Q0003

-------
                     Table 3:
          Who Funded the Critical Science
and Wildlife Service, who have research laboratories in the Great Lakes region.
Other (non-Federal) government organizations contributed 25 critical studies, over
half of which involved the two rules in which interstate pollution issues were the
main focus (Great Lakes and regional ozone).

The large number (238) of critical science documents developed by the private
sector includes both reports contracted by EPA program offices and ORD, and
reports contracted by the regulated community (state and local governments, and
private industry). It also includes reports completed in-house by private industry.
Thus, the private sector was the most common source of the critical science behind
the rules.

Who funded the critical work?

As summarized in Table 3 and detailed in Exhibit 4, EPA program offices funded
the vast majority of the critical science documents. Other organizations (primarily
State governments and industry) funded  100 documents. Many industry
contributions involved data gathering to
support the various emissions rules (e.g.,
pulp and paper, reformulated gasoline,
and regional ozone). In some cases, the
regulated industry agreed with EPA
beforehand to a research or data
gathering strategy.  ORD funded at least
85 critical science documents, including
criteria documents, early development of
the model used to support the regional
ozone rule, development or evaluation
of monitoring methods, and research
grants and cooperative agreements that
produced findings that proved critical.
Other Federal organizations, including
the National Institutes of Health,
National Science Foundation, Department of Energy, Department of Agriculture,
and Department of Interior, funded the smallest number.

What mechanisms  were used to fund the critical work?

As summarized in Table 4 and detailed in Exhibit 5, the most common funding
mechanism was a contract. Contracts were used to support critical science by all
funding organizations.  Some reports or technical appendices were developed by
contractors and delivered as finished products (Table 2). In other instances
contractors gathered data, ran models, conducted analyses, provided specific
EPA program office
Other
EPA ORD
Other Federal agency
Unknown
260
100
85
31
10
         If more than one organization funded the
         work, we counted each so the total
         exceeds the number of critical documents
         (i.e., 452).
16
                                                        Report 2003-P-00003

-------
Contract
EPA in-house
Other
Unknown
Grant
Cooperative agreement
Interagency agreement
230
85
63
41
25
15
2
   If more than one funding mechanism was used,
   we counted each so the total exceeds the
   number of critical documents (i.e., 452).
             expertise, or otherwise                           Table 4:
             contributed substantial input        Funding Mechanisms for Critical Science
             to work products that were
             ultimately authored by EPA
             program office or ORD staff.
             Of the 230 critical documents
             provided through contracts,
             212 were funded by an EPA
             program office or ORD.

             Internal EPA funding was
             used by program offices to
             develop at least 85 of the
             technical support documents,
             including rule preambles.
             Since rulemaking is an
             inherently governmental
             function, program personnel
             should exercise control over
             the final application of science to the rule.  The "Other" category primarily
             included internal funding by other Federal and other governmental organizations,
             particularly those who do much of their own research.

             Grants and cooperative agreements by law cannot be used for the primary purpose
             of securing goods or services for the government, so it was not surprising that we
             identified them in only nine of the rules. More than half of these documents
             funded under assistance agreements were published eight years or more before the
             rule was finalized. Some of the documents were quite old. For example, an
             epidemiology study done in 1950, long before passage of the Clean Water Act,
             served as the basis for a Maximum Contaminant Level in the 1991 drinking water
             standards on synthetic chemicals. This suggests that much of the science funded
             by assistance agreements pays off many years in the future, and that it is not funded
             specifically to support a rule, but to address the larger environmental problem that
             was the target of the rulemaking.

More data and fewer "blind  spots" could reduce assumptions

             For 12 of the rules, respondents indicated additional science would have made their
             rules better. On a scale of 1 to 5, with 5 being the highest, the respondents gave
             the rules included in this pilot study quality scores ranging evenly between 3 and 5.
             Therefore, we concluded that, in the view of the EPA respondents who worked  on
17
                                                                    Report 2003-P-00003

-------
             the rules, these were good rules that could be even better if more science had been
             available4.  The most frequently expressed desires were for:

             >•  Data on emissions rates, characterization of regulated sources, and toxicity that
                could lead to less uncertainty (9 rules).

             >•  Science to fill "blind-spots" (6 rules).

             Based on the responses, we concluded that having more data would have resulted
             in more efficient rules, because they would have required fewer conservative
             assumptions. Scientific "blind spots" are areas where no body of scientific
             research was available at the time of the rulemaking to adequately assess some of
             the potential risks, or the particular risk was not anticipated at the time of
             rulemaking. In most cases, there was a sense that while the rulemakers believed
             EPA was doing a good job under the circumstances, the science and data were
             being generated under undue pressure.  The desire for additional scientific
             information in so many of the rules included in this pilot study suggests this desire
             may be common to many Agency rulemakings.

Critical science supporting the rules often was not
independently peer reviewed

             A regulation itself is not subject to EPA's peer review policy, even though the
             major scientific work products that support it are subject to peer review. Public
             comments are taken on almost all regulatory actions, but according to the Peer
             Review Handbook, public comment does not substitute for peer review. This is
             because public comment does not necessarily draw the kind of independent, expert
             information and in-depth analysis expected from a peer review.  Nonetheless, we
             were told by EPA staff members involved in 6 of the 15 case studies, that their
             documents did not require peer review because they had been subjected to public
             comment.  As noted in the Response to Comments, we acknowledge that the
             guidance on this issue has been evolving over the decade in which the rules in the
             case studies were finalized.
       These scores do not reflect input from peer reviewers, environmental organizations, or the regulated
 community, and they should not be assumed to hold true for EPA's rules in general,

                                           18                         Report 2003-P-00003

-------
A large number (276) of the critical documents supporting the rules either were not
peer reviewed (144) or their peer-review status was indeterminate (132). Details
on peer review actions
are summarized in
Table 5 and Exhibit 6.
Lack of peer review, or
of information about
peer review, may cast
doubt on the quality of
this science.
                      Table 5:
                 Peer Review Actions
EPA has a database -
the Peer Review
Product Tracking
System — to track peer
review of its scientific
and technical work
products. It is a single
repository for product-
specific peer review
reporting and tracking,
and uses a common
reporting form for all
entries. Work products
that were completed since
   No peer review was done
   Unknown whether a peer review was done
   Peer review was done by an external group
   in a non-public manner, such as a refereed
   journal or external experts hired by EPA
   Peer review was publicly done by a Federal
   advisory committee, such as the Science
   Advisory Board or National Research Council
   Peer review was done by some other public
   review process by external experts
   Independent internal EPA review done, such
   as through the risk assessment forum or by
   having ORD review a program office's
   document
144
132
                                                                     119
 29
 17
 11
1991 should be reported in one of four categories:
    List A: Work products for which peer review was completed.
    List B: Candidate work products for future peer review.
    List C: Work products not subject to peer review.
    List D: Scientific articles or reports by EPA staff that were peer reviewed
    outside EPA.

This database should be the primary means of tracking the present, past, and future
peer review status of the critical science documents identified in the pilot.
However, we could find very few of these documents in this database. We
searched the database using combinations of titles and keywords for documents
supporting the 10 cases included in this pilot study with rules finalized since 1994.
We were able to find only 4 of the 272 critical documents for these rules:

>   Two technical support documents for the pulp and paper National Emission
    Standard for Hazardous Air Pollutant (NESHAP) marked "peer review not
    needed."
                              19
                                 Report 2003-P-00003

-------
             *•  The primary modeling technical support document for the regional ozone rule,
                listed as Category "C" (a non-major scientific/technical product), marked "peer
                review not needed."

             »•  The BEIS2 model, a critical document in the regional ozone rule listed as
                category "C," marked "peer review not needed."

             The peer review status for these documents was signed off on as "complete" by the
             requirements reviewer from the ORD's Office of Science Policy. We also
             searched on "DBF," "disinfection," and "chlorinated," and found several entries
             related to the disinfectant and byproducts rule, including studies of by-products
             resulting from ozonation; a 1994 regulatory impact assessment; engineering and
             cost studies; and a study on cancer and chlorinated drinking water. However, these
             entries did not correspond by title or date to any of the 59 critical documents
             identified by the primary contacts and ORD scientists, so we were not sure whether
             these documents ultimately were superceded by publication in different form or
             later versions.

             We concluded that: there was little correspondence between the entries in the Peer
             Review Product Tracking System and the items in the docket; keyword searches
             were not effective in identifying the important science behind the rules included in
             this pilot study; and there was no consistency in the classification of items in the
             database, either with respect to their importance, intended use,  or need for peer
             review. Therefore, we determined that oversight of peer review of the critical
             science documents to support the rules included in this pilot study was limited  and
             ineffective.
Suggestions
             We offer the following suggestions to the Associate Administrator for Policy,
             Economics, and Innovation, and to EPA's Science Advisor, regarding science
             behind EPA's ralemaking:

             1.  Consider presenting the scientific findings that support a rule in specific
                 sections of the preambles. These findings should be organized according to the
                 norms of science, in summary form, and indicate:

                    >   Why the science is required to support the rule, and how the results will
                        be used.
                    *   The methods used.
                    »•   The important results (showing key data and their uncertainty).
                    >   Interpretation of the findings, and comparison with other studies that
                        appear to support or contradict the results.
                    *•   Scientific referencing of underlying scientific and technical documents.
                                           20                         Report 2003-P-00003

-------
                   >   A separate section of the preamble that would bring in issues of law,
                       policy, economics, and administrative discretion that do not depend on
                       the scientific findings.

             2.  Focus more attention in the development phase of regulations on collecting
                data and doing research to address "blind spots" to support rulemakings.

             3.  Take advantage of EPA's information technology capabilities to:

                   *   Hotlink references in preambles to documents in the docket.
                   »•   Link scientific and technical documents in the docket to the Science
                       Peer Review Database.
                   >•   Link RAPIDS to the Science Peer Review Database.
                   *•   Maintain through RAPIDS an inventory of all rules proposed and
                       finalized each year.

             4.  Reinforce EPA's current peer review policy, ensuring that all EPA-generated
                documents critical to significant and substantive rulemakings are independently
                peer reviewed, and that the  responses  to the significant comments appear in the
                documents.
Agency Comments
             OPEI, the EPA organization responsible for oversight of the Agency's regulatory
             activity, provided comments on the draft report, as did the Office of Water (OW);
             the Office of Prevention, Pesticides and Toxic Substances (OPPTS); and the EPA
             Science Advisor. The Office of Air and Radiation (OAR) and the Office of Solid
             Waste and Emergency Response (OSWER) provided informal comments.  Many
             of the comments led to improvements in the clarity and factual accuracy of the
             report.  In general, the comments supported the suggestions in the draft report, but
             identified both opportunities and concerns regarding details of their
             implementation. The comments are included in their entirety in Exhibits 7-10, but
             we summarize the main points in this section.

             On behalf of EPA, OPEI commented:

                The report does an excellent job of recognizing Agency institutional
                mechanisms which ensure that regulations are based on sound science. The
                role of peer review and the peer review process in the development of
                credible science is discussed in depth, and the Office of Policy, Economics
                and Innovation (OPEI) agrees with the heavy emphasis the report places on
                the utility  and importance of independent peer review. Not emphasized are
                two other  key "good science" processes: Analytic Blueprints and the Risk
                Characterization Policy.  The former was designed, in part, to ensure that
                critical science needs are identified early in the process and developed in

                                          21                        Report 2003-P-00003

-------
   time to inform regulatory decisions, and the latter requires that both the risk
   assessment process and risk the analyses are transparent, clear, reasonable
   and consistent. Taken together, these three existing mechanisms can assure
   that:

   •  Critical science is identified early, and developed in time to inform
      decisions (Analytic Blueprint),

   •  Critical science is of sufficient quality for regulatory decision making
      (Peer Review Process),

   •  The quality of the science and the associated uncertainty is clearly
      described (Risk Characterization Policy).

   Further, these three mechanisms appear to directly address three of the four
   findings of your report, i.e., that critical science supporting the rules often
   was not independently peer reviewed, that more data and fewer "blind
   spots" could reduce assumptions, and that the role of science (was> not
   made clear. Your report "determined that the oversight of peer review of
   the critical science documents to support the pilot rules was limited and
   ineffective." Applying the same logic suggests that shortfalls in identifying
   critical data needs, and the lack of transparency and clarity in science is due
   to inefficiencies or limitations in the two Agency processes intended to
   identify, develop and make critical science transparent.

The OPEI comments go on to address each of these issues. We have also
incorporated the significant comments from the other EPA Programs and the EPA
Science Advisor.

With regard to the presentation of science in the preambles, OPEI recommended
that we consider using EPA's risk characterization policy as a framework for
presenting the results and suggestions in the report. They said:

   Some of the science supporting rulemaking deals with health and
   environmental risks.  EPA adopted its policy on "Risk Characterization" in
   February 1992, via a memorandum from Henry Habicht, Deputy
   Administrator, and an accompanying document, prepared by a cross-office
   work group. The policy was reiterated and elaborated  in the mid 1990s.  At
   its core, the policy states that significant risk assessments should:

   •  Describe how the estimated risk is expected to vary across population
       groups, geographic areas, or other relevant break-outs,

   •  Describe the sources of uncertainties in the risk estimates, and quantify
       them, to the extent possible, and


                             22                         Report 2003-P-00003

-------
   •   Explicitly identify the impact of science and data, as opposed to
       policy choices, as the source of various elements of the risk assessment.

   We have found that this standard has been followed in an incomplete
   fashion in documents supporting regulations, as well as other EPA risk
   assessments.  The draft Office of the  Inspector General (OIG) report refers
   repeatedly to the second and third elements of EPA's Risk Characterization
   Policy, both in describing its findings and in its recommendations. We
   recommend that OIG examine this policy (in effect during most of the time
   period covered by the pilot study), and use it as a framework for presenting
   its results and suggestions.

The EPA Science Advisor strongly agreed:

   The key issue is that the preamble should present a clear summary of the
   science supporting the regulatory decision, including properly
   characterizing risks and the supporting science for risk management. The
   preamble should list the documents from which its science-based
   statements are made and the docket should contain the complete record.
   This would allow readers to refer to the source material, including the
   original primary science documents referenced in the critical documents
   (using "primary document" as traditionally used in the science  community).

OPEI also brought up EPA's new Information Quality Guidelines,  which were
recently implemented:

   This suggestion is consistent with the Agency's efforts related to the use of
   and dissemination of information covered by the new Information Quality
   Guidelines (IQ Guidelines)	[The Act] direct[s]  Federal agencies to:

   •   adopt a basic standard of quality  as a performance goal and take
       appropriate steps to incorporate information quality criteria into agency
       information dissemination practices; [and]

   •   issue guidance for ensuring and maximizing the quality, objectivity,
       utility, and integrity of information disseminated by the agency;
       establish administrative mechanisms allowing affected persons to
       obtain correction of information that does not comply with  the
       guidelines....

   OPEI believes that a full implementation of the IQ Guidelines will improve
   the Agency's performance  related to  its discussion regarding the use of
   science in rulemakings.  This is also  an area where OPEI and ORD together
   can develop more complete recommendations regarding the presentation of


                             23                        Report 2003-P-00003

-------
   scientific findings in preamble discussions. OPEI and ORD are both
   increasing their presence in Agency rulemakings as a result of last year's
   Task Force on Improving Regulation Development.  OPEI believes that this
   increased participation by ORD and OPEI analysts will improve the
   attention to and discussion of the results of the underlying analysis,
   including but not limited to science, used to support EPA regulations could
   be improved. This discussion would be consistent with the IQ Guidelines,
   existing policies such as the risk characterization policy, and some of the
   key findings of your report.

Finally,  OPEI suggested developing "Principles of Analytic Integrity":

   Recently, the Administrator reaffirmed the "Princip[le]s of Scientific
   Integrity" establishing clear and  ethical standards that should govern the
   conduct of scientific studies within the Agency. To date, there is no
   parallel document establishing standards for the use of research in a policy
   analytic setting. OIG may wish to recommend that such a document be
   developed expanding on its recommendations for clarity of presentation,
   etc. and drawing on other Agency guidelines such as The Guidelines for
   Preparing Economic Analyses,

There was considerable comment from the program offices on exactly how the
science in the preambles should be presented.

OW commented:

   We believe there is merit in the proposal to improve the consistency of the
   presentation of scientific data and conclusions in regulatory preambles.
   However, more work is needed to determine how to implement such a
   proposal given the wide variety of types of regulations the Agency
   develops.  We also need to consider the impact on the cost of developing
   rules and on the length of preambles.

OW also noted:

   Depending on statutory requirements for a given rule, the optimal preamble
   structure for communicating the role of science maybe quite different.
   Any recommendations to revise  the format for rule preambles across the
   Agency should be flexible and take this consideration into account. To
   achieve the same objectives of the report, we recommend modifying the
   recommendation to suggest that norms of science be applied consistently
   throughout current preamble formats where science is discussed, in order to
   improve the understanding of the scientific basis for rules.

OPPTS  commented:


                              24                          Report 2003-P-00003

-------
   The preamble to the rulemaking is not, nor has it ever been, considered the
   proper vehicle for communicating the science in the manner prescribed on
   page 11 [page  12]. The proper vehicle for communicating the science in
   that detail is in separate documents that are made available to the public as
   part of the rulemaking docket, with a general description provided in the
   preamble. The preamble must provide a layman's explanation of the basis
   for the rulemaking, including the science, economic and technical analyses
   and other considerations that informed the decisions represented in the rule-
   making.

   The suggested addition of these science discussions in the preamble is cost
   prohibitive and impractical	most stakeholders consulted in 1994, when
   we evaluated the level of detail, format, and function of the preamble as
   part of the government wide streamlining initiative, indicated that they
   prefer for the preamble to contain a succinct summary of the science,
   economic and  technical analyses and other considerations that went into the
   rulemaking. This allowed those responsible for or interested in  the
   different disciplines to obtain a general understanding of all of these
   considerations, as well as  the details of the one of most interest  to them.
   Since the primary audience for the rulemaking is not the scientists,
   including the detailed scientific information in the preamble would not
   serve as an effective way to communicate the scientific information to the
   primary audience.

The EPA Science  Advisor agreed:

   .... while Agency preambles should effectively communicate the scientific
   underpinnings of the rules, the description of the professional norms for
   such communication is not accurate. The norms as described accurately
   reflect how scientists communicate in their primary documents, but not
   how science is communicated in what is described as critical primary
   documents.

OAR, while in broad agreement with the other comments, provided the following
caveat, "However, noting that the preamble to a rule may be the only source of
background data that our stakeholders read, it would seem appropriate to ensure
that a complete (albeit brief) discussion of the critical science is included in future
rulemakings as suggested..."

With respect to focusing more attention in the development phase of
regulations  on collecting data and doing research to close "blind spots" to
support rulemakings, OPEI commented:

   The purpose of an analytic blueprint is to identify research needs and guide
   data collection and research studies during the development phase of


                             25                         Report 2003-P-00003

-------
   regulations. While a requirement for developing, updating, and following
   an "analytic blueprint" has been a formal part of EPA's rule-making
   process for more than a decade, it has been OPEI's experience that most
   analytic blueprints are treated as little more than formalities. As a result of
   last year's review and reassessment of EPA's rule-making process, OPEI
   and the program offices are taking steps to make the blueprints more central
   and relevant to actual rule-making decisions. We suggest that the OIG
   report consider referring to the analytic blueprints as one means to achieve
   the results desired in (this) suggestion.

None of the other comments spoke of the analytic blueprint process. However,
OW commented:

   Many EPA regulations are based on years of research and data gathering by
   EPA, other Federal agencies, academia, and industry.  For example, we
   have been working on the arsenic drinking water standard steadily since the
   1970s. OW and other programs have extensive processes of joint planning
   with ORD and outside stakeholders to anticipate information needs as
   much as possible. Yet, there are always data gaps and uncertainties which
   we must grapple with. This is in the nature of the rulemaking enterprise.
   While the Pilot Study cited respondents who said they would have liked to
   have  had more data, it did not identify any particular ways of obtaining it
   without increasing costs or slowing down action. Allocating resources to
   closing "blind spots" means something else will not be done, and delaying
   action means the status quo will continue.

OW also noted that:

   The 1996 SDWA Amendments require EPA to use "the best available,
   peer-reviewed science and supporting studies conducted in accordance with
   sound and objective scientific practices" when setting drinking water
   standards (sec.  1412 (b)(3)(A)). The US Court of Appeals for the District
   of Columbia Circuit determined that Congress's intent was best available
   evidence at the time of rulemaking.

They noted that otherwise, "it could also negatively impact the ability to meet
statutory deadlines."

The EPA Science Advisor commented that ORD is increasing its involvement in
the Agency's decision-making process.

With respect to the suggestion to take advantage of EPA's information
technology capabilities, OPEI commented about the characterization of the
RAPIDS data base and its capabilities:
                              26                         Report 2003-P-00003

-------
   RAPIDS tracks all substantive rulemakings appearing in the Semi-Annual
   Regulatory Agenda as well as a number of actions not in the Agenda, such
   as Reports to Congress, Policy Plans, etc.  RAPIDS does not track every
   non-substantive rulemaking (SIPs, SNURs, FIPs, State Approvals, etc.), but
   a sister database to RAPIDS (Federal Register Tracking Database - FR
   Dailies), also maintained by OPEI's Regulatory Management Staff (RMS),
   tracks every EPA action sent to and published in the Federal Register.
   These rules are not economically significant or normally reviewed by OMB
   and therefore are classified as "not significant."

   RAPIDS records go back a number of years (1996 forward) with some
   rulemaking records from earlier years available. RAPIDS also tracks
   NPRMs published in many of those same years. The Regulatory
   Management Staff (RMS) has built numerous views in RAPIDS and has a
   view (list) of rules finalized each year.

   The report seems to confuse or not  clearly differentiate between
   "significant" rulemakings (those OMB reviews) and "economically
   significant rulemakings" (economic impact of greater than $100 million per
   year). RAPIDS separates out those rules identified as "economically
   significant." This designation has only been in effect for rules in the Semi-
   annual Regulatory Agenda as Priority "A" (Economically Significant) since
   1995. Although for years before 1995, it is more difficult to clearly identify
   economically significant rules, RAPIDS identifies 50 final rules as
   economically significant for the years 1994-2001 and can produce lists of
   economically significant rules published final for the years 1990 to the
   present.

OPEI went on to say:

   OPEI is currently evaluating and enhancing RAPIDS in order to improve
   the management information that is available or potentially obtainable.  To
   date, RAPIDS has focused on tracking regulation development progress
   and facilitating EPA's submission of its portion of the Semi-Annual
   Regulatory Agenda to OMB. OPEI is interested in adding features that
   enhance management accountability and improved performance metrics.
   RAPIDS currently links to relevant guidance and policy documents.  OPEI
   will continue to improve RAPIDS and seek to take advantage of other
   information technology capabilities over the next year. Much of this work
   will be coordinated through the Regulatory Steering Committee or
   Regulatory Policy Council. We will follow up with you over the next
   several months to more fully understand these recommendations and
   identify what specific changes or opportunities we can adopt.
                             27                         Report 2003-P-00003

-------
OW commented, "We support the report's third recommendation to make better
use of the Agency's information technology capabilities. Consistent use of these
tools throughout the rulemaking process will improve communication and access
to the critical scientific support documents," and that, "We would support an effort
to identify and implement ways to improve the information the Agency makes
available on rulemaking."

OAR commented that:

    The value of the new Science Inventory Database is obvious. An up-to-
    date, searchable system (as is in place currently) is a valuable tool when
    researching the science behind rulemaking	[but that] the Science
    Inventory database was designed to be a "data-lean" system which provides
    enough information to direct the reader to the correct source for more
    details; it was not designed to be the repository of all information related to
    critical documents, especially those not issued by EPA. Whether or not this
    system should be linked to other databases should be the subject for the
    Science Inventory Work Group to consider.

OPEI commented on reinforcing EPA's current peer review policy ensuring that
all EPA-generated documents critical to significant and substantive rulemakings
are independently peer reviewed, and that the responses to the significant
comments appear in the documents:

    OPEI fully supports this recommendation on peer-review of critical
    documents and in fact has recently extended this peer-review policy to
    include economic analyses.  OPEI is working closely with the Agency's
    Program Offices to ensure that a full review of supporting economic
    analyses for all economically significant rules occurs prior to the rule's
    submission to OMB. In this way, the  application of sound and consistent
    economic practices is ensured and the Agency's position on the use of
    sound science strengthened.

With respect to the Agency's peer review policy, OW commented:

    We support the report's fourth suggestion to reinforce the Agency's peer
    review policy. Because the current policy does not explicitly require peer
    review, it may be appropriate to recommend updating the policy to require
    peer review in certain situations to ensure it is applied more consistently
    across the Agency.

OW further commented:

    EPA's Peer Review Policy was first issued in 1992, after some of the rules
    considered in the Pilot Study.  Full implementation has taken time and


                             28                        Report 2003-P-00003

-------
                continuous effort. Thus, it would not be surprising that compliance was
                limited in the earlier period, but we would hope that it had been improving
                as we approach the present time.  Unfortunately, the report does not present
                information on peer review performance over time, so we cannot tell
                whether this has happened.

             OSWER also commented on the changing peer review guidance over the course of
             the study, and urged that we appropriately caveat that fact in the summary of the
             report. In fact, they questioned whether the observations on peer review were
             meaningful, given that we did not compare the peer review status of the documents
             with the policy then in effect.

             The EPA Science Advisor commented, "I am concerned about the OIG's finding....
             that critical science supporting the rules often was not peer reviewed. I plan to
             review the Agency's progress in implementing its Peer Review Policy during the
             coming year."

             On a more general note, the EPA Science Advisor also stated:

                .... the Administrator named me to serve as the Agency's Science Advisor.
                I take this role very seriously and plan to make important strides in ensuring
                that Agency decisions are based on sound science, and that science is
                presented and characterized properly in our rules and other important
                documents.

Response to Agency Comments

             The OPEI, the program offices, and the EPA Science Advisor were in general
             agreement with our suggestions, but expressed some concerns about details
             regarding their implementation.  Even better, it appears that the mechanisms are in
             place, and some steps have been taken, to make substantial progress in
             implementing them.  We believe that the observations in the report may serve as a
             baseline against which progress can be charted.

             We have incorporated many of the specific Agency comments directly into the
             report and its Addendum to improve their clarity and factual accuracy. We made
             corrections in several of the case studies that led to small changes in the overall
             statistics on the critical documents, but they did not significantly alter the
             qualitative observations or the suggestions.  To simplify the report, we dropped the
             distinction between primary and secondary documents that appeared in the draft
             report. We also added an explanation of the different levels (one, two, and three)
             of documents we reviewed.

             Several questions were raised about the treatment of economics in the pilot study.
             We had  considered dealing with economics as thoroughly as the biological and

                                          29                         Report 2003-P-00003

-------
            physical sciences in the pilot study. Initial perusal of the primary economic studies
            (e.g., cost-benefit analyses and regulatory impact analyses) tended not to reveal
            many citations from the primary literature.  Consequently, we only included critical
            economics documents when they obviously had an impact on the rule (i.e., using
            the same criteria used for critical science documents) and did not cite any of the
            references contained therein. We agree with the OPEI comments that economic
            science is as critical as the physical, biological, and engineering sciences, and refer
            the reader to a recent report, Economic Analyses at EPA: Assessing Regulatory
            Impact5, that includes analyses of two of the rules (case studies 5 and 6) in the pilot
            study.

            Rather than making changes in the suggestions based on the comments, given the
            generally positive responses to them, we have chosen instead to view the Agency
            responses as a road map toward acting on them. In that spirit, we are responding
            more specifically to the comments on the four suggestions.

            We understand that preambles (as the embodiment of what is essentially a legal
            process) cannot take on the appearance of a science journal, or be extended by
            many pages to provide extensive graphs and data tabulations.  Well-organized,
            well-referenced, and peer reviewed technical support documents that carry the
            weight of the scientific underpinnings of the rule are suitable for this task. We do
            believe, however, that the critical scientific underpinnings of EPA's rules should
            be explained, in plain English, in terms of the methods used to gather data, the
            results obtained, and  the applicability and uncertainty associated with their
            application to the rule.  There are examples of good practices in communicating the
            scientific basis for regulations in many of the preambles of the rules in the pilot
            study, and we encourage OPEI to work with the programs and the Science Advisor
            to bring all preambles up to the highest standard possible. Referencing of this
            science (including economics) should be as careful as the legal referencing in the
            preambles.

            Also related to presentation of the role of science, we agree with the Agency
            comments that effectively implementing the Risk Characterization Guidelines
            should improve the explanation of the application of science to regulatory
            decisions. We would add that even though the Guidelines focus on risk
            assessments, the principles apply as well to science and technology applications
            that do not involve risk (e.g., establishment of the maximum achievable control
            technology for the Pulp and Paper NESHAP, or even the adoption of monitoring
            technology in the Acid Rain regulation). We agree that the new Information
            Quality Guidelines should have a positive influence on the application of science to
            regulations.  We urge OPEI and the Science Advisor to pursue the concept of
            developing a "Principles of Scientific Integrity" document, as suggested by OPEI.
5      Morganstem, R. [Ed.]. 1997. Resources for the Future: Washington, DC
                                          30                         Report 2003-P-00003

-------
We agree with OPEI that the regulatory blueprint represents an opportunity to
identity and close science and data gaps during the relatively short period between
the time a rulemaking is initiated and the final rule is proposed. Many of the rules
in the pilot study demonstrate how much can be accomplished during this period.
We urge OPEI and the Science Advisor to ensure that regulatory blueprints are
"more than formalities" in the future, and that they become central to identifying
the scientific data and analyses needed to support the regulation, and to plan and
ensure their independent peer review. The statistics on the critical science
documents funded under assistance agreements (grants and cooperative
agreements) suggest that Requests for Assistance must be planned 5-8 years in
advance of proposed rulemakings, which may be too late for regulatory blueprints.
ORD's multi-year planning process must take this time lag into account in
planning research that may be supportive of rulemaking in the future. We also note
that it was not just science gaps, but data gaps, that were highlighted by many of
the contacts in the pilot study.  Monitoring data were often as important as research
in supporting the rulemakings in the pilot study (e.g., Cases 1, 3,7,9 and 13).

We are encouraged that OPEI has made improvements to RAPIDS since we began
the pilot study, and that several of the programs agree that integrating and linking
EPA's databases on regulation, science, and peer review would be helpful. We
have one caveat, however.  OPEI commented that the FR Dailies database now
allows identification of all EPA rules, significant and otherwise, and that RAPIDS
now identified 50 economically significant rules (greater than $100 million/yr)
finalized since 1994.  We had identified only 37 in the draft report.  We checked
the list in the pilot against RAPIDS, and based on the information in the preambles
relating to Executive Order 12866, we determined that we failed to identify one of
the rules in Exhibit 1 as economically significant, and  one more was questionable
(the expected impact was $99 million the first year to the economy and $1 million
to EPA, and $50 Million/yr thereafter). We changed the listings in Exhibit 1 to
reflect these errors. However, we determined that 11 of the rules in RAPIDS were
not economically significant, and that two economically significant rules were
missing for this period. This reflects the human factor in information management
- information management systems are only as good as the quality of the data that
are input by the people who use them. We encourage the Agency teams
developing and integrating RAPIDS, the Science Inventory, and the Peer Review
Databases to not lose sight of this critical fact.

Finally, we are encouraged that OPEI, the programs, and the EPA Science Advisor
are in agreement about the need for more consistent independent peer review.
Three of the programs raised the point that EPA's peer review guidelines were in
flux during the ten years of rulemaking covered by the pilot study, and that we
should have taken that fact into account in interpreting the peer review statistics.
One program office even questioned whether the observations about peer review
had any validity, under those circumstances.  It was  beyond the scope of the pilot
study to compare Agency practice with then-current guidance, and we made no


                              31                         Report 2003-P-00003

-------
such observation.  Rather, the statistics should be seen as a baseline against which
progress may be measured. The EPA Science Advisor commented, "I am
concerned about the OIG's finding.,., that critical science supporting the rules often
was not peer reviewed. I plan to review the Agency's progress in implementing its
Peer Review Policy during the coming year." We encourage the Science Advisor
in his efforts regarding peer review and his commitment to ensure that Agency
decisions are based on sound science.
                             32                         Report 2003-P-00003

-------
                       ssonsLearned
We conducted the pilot study to determine whether a full study could provide
answers to the questions in the introduction to this report and, if so, the level of
resources required.  Proceeding with a full study would be resource-intensive. We
had intended for the pilot study to be completed in four months with less than half
a staff year of effort. Because we were not able to get timely responses to our e-
mail queries, and because it proved harder than we expected to determine funding
sources and peer review, the field phase of the pilot took 10 months and 1.5 staff
years, and we were  still unable to identify all of the critical science documents and
their corresponding data for the 14 rules. For those reasons, we do not intend to
pursue a full study at this time.

However, if such a  study were to be pursued at some future date, we believe:

*   Developing a list frame for a  target population of substantive or minor rules is
    not straightforward given the current capabilities of RAPIDS. We were not able
    to confirm that  it would be possible to easily identify using RAPIDS, a target
    population of rules, either current or past, on which to conduct any future
    studies. An alternative would be to draw sample rules out of RAPIDS (or the
    Federal Register for older rules).  This approach would have to be pilot tested.

*   There should be strict decision criteria for defining critical documents, and
    there should be periodic group review and agreement about which documents
    meet the criteria.

»•   A decision should be made about how far back in the decision process for a
    rule you can go before you can no longer determine with confidence that a
    science document was critical, but you should go back at least that far.

*   Interviews should be conducted with all parties involved with the rulemaking.
    Special effort should be made to interview peer-reviewers and stakeholders.
    E-mail is not an effective mechanism to elicit detailed responses.

>   There should be follow-up interviews with all respondents, asking them to
    confirm preliminary information about each document.

*   There should be at least one research scientist on the team to facilitate
    identifying critical documents.

»•   There should be an advisory committee of scientists who understand both
    research and rulemaking, to assist the review team.
                              33                        Report 2003-P-00003

-------
All the pertinent documents applicable to rulemaking during the period covered
should be reviewed, including the preamble to the proposed rule.  Critical
science identified in the proposed rule was not always cited in the preamble to
the final rule, or in the major technical support documents.

A different method would need to be devised to address the original questions
regarding research planning and rulemaking outcomes.
                           34                        Report 2003-P-00003

-------
                                                     Exhibit 1
 Significant Rules Finalized - 1990-2001
= Rule in pilot study
$ = Rule is significant because of its economic impact
1990
1991
1991
1991
1991
1992
1993
1993
1993
1993
1993
1993
1993
1994
1994
1994
1994
1994
CAA
SDWA
RCRA
CAA
ICAA
iCAA
ICAA
ICAA
!CM
CAA
[CAA
ICWA
IRCRA
iCAA
ICAA
JRCRA
ICAA
JCAA
[Volatility Regulations for Gasoline and Alcohol Blends Sold in Calendar
[Years 1 992 and Beyond
jNPDWR (National Primary Drinking Water Regulations): Synthetic
'[Organic Chemicals and Inorganic Chemicals; Monitoring for
'Unregulated Contaminants; NPDWR Implementation; NPDWR
'Regulations
jSoiid Waste Disposal Facility Criteria
[Standards of Performance for New Stationary Sources: Municipal
[Waste Combustors
'Tier 1 Light-Duty Tailpipe Standards and Useful Life Requirements
{Operating Permits Regulations Title V of the Clean Air Act
[Accelerated Phaseout of Class I & II Ozone Depleting Substances and
{Methyl Bromides
[Acid Rain Permits, Allowance System, Emissions Monitoring, Excess
[Emissions and Appeals Regulations Under Title IV of the Clean Air
'Amendments of 1990
jConformity of General Federal Actions to State Implementations Plans
[Control of Air Pollution from New Motor Vehicles and New Motor
[Vehicle Engines, Regulations Requiring on-Board Diagnostic Systems
ion 1994 and Later Model Year Light-Duty Vehicles
[Evaporative Emission Regulations for Gasoline-Fueled and Methanol-
[Fueled Light Duty Vehicles, Light-Duty Trucks, and Heavy Duty
iVehicles
[Oil and Gas Extraction Point Source Category, Offshore Subcategory,
[Effluent Limitations Guidelines and New Source Performance
(Standards
iCorrective Management Units (CAMU)
[Control of Air Pollution: Determination of Significance for Nonroad
[Sources and Emission Standards for New Nonroad Compression-
jlgnition Engines at or above 37 Kilowatts
[Fuel and Fuel Additives: Standards for Reformulated Gasoline
[Land Disposal Restrictions - Phase II - Universal Treatment Standards,
land Treatment Standards for Organic Toxicity Characteristics
[List of Substances and Threshold Quantities for Accidental Release
iPrevention
[NESHAP: Source Categories: Organic Hazardous Air Pollutants from
[the Synthetic Organic Chemical Manufacturing Industry (SOCMI) and
[Other Processes Subject to the Negotiated Regulation for Equipment
j Leaks
[55
=56
=56
[56
!56
!57
[58
|58
!58
[58
[58
[58
I58
[59
!59
[59
=59
[59
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
FR
23658
3526
50978
5488
25724
32250
65018
3590
63214
9468
16002
12454
8658
31306
7716
47982
4478
19402
[$
]$
!$l
=$
i$
;$
[$
j$
i$
j$
[$
l$

[
•!$
;$
j$
i$
                         35
                 Report 2003-P-00003

-------
1994  JCAA   iOn-Board Control of Refueling Emissions from Light Duty Vehicles and J59FR 16262
    ...1.       iysty P.y|y.
1994  iCAA   'Organic Air Emission Standards for Tanks, Surface Impoundments,    |59 FR 62896
      I       land Containers at Hazardous Waste Treatment, Storage, and Disposal i
      !       -Facilities and Hazardous Waste Generators                        i
 994
                                                                         J59.FR39258

                                                                         J59FR61432
                                                                         166FR4948
                           Release Reporting, Cornrnunity Right-
             [National Emission Standards for Chromium Emissions From Hard and
             Decprative '. Chromium Electro^latmg_and_Chromium_Anodizing Tanks
1995

1995
JCAA

ICAA
             jErnission Standards for Marine Tank Vessel Loading Operations
             [New Source Performance Standard (NSPS): Municipal Waste
             ICombustion - Phases II and I
                                                                          J60FR48388
                                                                          J60 FR 65381
1995

1995
JCAA

ICAA
             JOzone Transport Commission: Emission Vehicle Program for Northeast !60 FR 4712
             JOzone Transport                                              |
             iW.ateLQuMty.G.uldan99J9^...         ..    ]60.FR 15366
1995
1995
      iCWA
      !CAA
             NESHAP:£etroleurn .Refineries
             iWater Quality Standards for San Francisco Bay and Delta
             •!•	'•	'•	
                                                    .B^
                                                                     I60FR43244
                                                                     160 FR 4664
1995.
1996
iCWA
iCAA
                                                                     |61
                                                                     lei
                                                                             FR67112
1996  ICAA   iControl of Emissions of Air Pollution: Emission Standards for Gasoline
      |       {Spark-Ignition and Diesel Cornpression-lgnition Marine Engines
1996  ICAA   [Federal Test Procedure for Emissions From Motor Vehicles and Motor
    ...1.      iY?^'9'e. in9!ne,s.: ?.?,Y'e.w
                                                                             FR 52087
                                                                          161 FR 54851

                                                                          161 FR 15566
 1996 IRCRA Land Disposal Restrictions - Phase III: Decharacterized Wastewaters,
      |       |Carbarnate Wastes, and Spent Aluminum Potliners
             NSPS: Municipal Solid Waste Landfills Amendments
             Regulation of Fuel and Fuel Additives: Certification Requirements for
1996
1996
JCAA
iCAA
                                                                    |61
                                                                    lei'
                                                                             FR 9905
                                                                             FR 35310
1996 iTSCA

1996 ITSCA
             iLead: Requirements for Disclosure of Known Lead-Based Paint and/or J61 FR9064
             [Lead-Basec I Paint Hazards in Housing                            |
             iLead: Requirements for Lead-Based Paint Activities in Target Housing 161 FR 45788
             jand Chitd-Occupied Facilities                                    j
 1996 iCAA

 1996 ICWA
             iRisk Management Program for Chemical Accidental Release
             ! Prevention
                                                                          i61 FR 31668
             jOtl and Gas Extraction Point Source Category: Final Effluent
             [ymitattons Guidelmes and Stendards._for the Coastal^Subcateppry
                                                 P
                                                                          J61 FR 66085
 1996 JCAA
J9^.jCAA"
 1997 ICAA
                                                                           62FR38651
             ICompliance Assurance Monitoring Rule (Previously Enhanced
                                                                          !62 FR 549
 1997


 1997 JCAA
"i9&7 ITSCA

                                                                          !62 FR 17910
 1997..ICAA
 1997 fCAA
                                         Am.?M^£!!?;?!Q?x^^^^
             IControl of Air Pollution From New Motor Vehicles and New Motor
             iV.?.!l'.9'e. i.n9!.ne.s.: Y9!unAary ^*a.!?9!a^s. f9.C ySht-Duty Vehicles
                                                                           62FR31191
.1998.JQAA	jContrpI of.Ernissions of Air Pojlutj^
                                           36
                                                                      Report 2003-P-00003

-------
 1998 JCAA   [Finding of Significant Contribution and Rulemaking for Certain States in [63 FR 57355 l$
      I        -the Ozone Transport Assessment Group (OTAG) Region for Purposes  I             i

 1998 ICAA,   ^Integrated NESHAP and Effluent Guidelines: Pulp and Paper          I63FR 185041$
      ICWA   {	::	_	;	].......„...	I...
 1998 ICAA   lLocomo%e.E^163.FRJ8977 |$
 1998 JRCRA JLand Disposal Restrictions Phase IV: Final Rule Promulgating         J63.FR 28555 i
      •j        (Treatment Standards for Metal Wastes and Mineral Processing        i             I
      !        -Wastes; Mineral Processing Secondary Materials and Bevill Exclusion  i             I
      j        llssues; Treatment Standards for Hazardous Soils, and Exclusion of    i             i
      ]       [Recycled Wood .Preserving.Wastewaters _._    ^          .      ]       ..      |
..l?§.?ll§.9.wAlNPDWR: Stage!                                                16.3.FR,69389 ]$
 1998 JTSCA  jLead: Requirements for Hazard Education Before Renovation of Target I63 FR 29908 I
      |     ....jHousmg                                                         ]             }
...1.§?.?..ll§.c.A....ip°!X?.y9.d.n.?ted                                            .     ]§.3..?R.35384.]
 1998 ISDWA INPDWR: Interim Enhanced Surface Water Treatment                I63 FR 69477 !$
,....,..,,,.,.A.>.t...-<»-..--J...>.'.•.»•..•-*••*.•••.»•..»•-.•••««••-><••••»••••«•••*«••>.••••••••••• ••-•••-•••••••••••••••••--••••••••••••••-------•••••••••••••---•--••••^•••••••••""--••—•••*•*•* A.>r
 1998 [CWA  [Pharmaceutical Manufacturing Category Effluent Limitations           [63 FR 50387 l$
      I       [Guidelines, Pretreatment Standards, and New Source Performance    i             I
      1       iStendards                                                        1             1
 1998 ISDWA. JNPDRW^ Consumer Confidence Reports                            1§3.FR 44511J
 1999 ICAA   JFindings of Significant Contribution and Rulemaking on  Section 126    [64 FR 28250 i$
      !       •Petitions for Purposes of Reducing Interstate Ozone Transport        j             ]
 1999 [CWA  [National Pollution Discharge Elimination System (NPDES)            [64 FR 68722 |$
      1       i? 9.mpr.?!?e..n?.!y.e §t9r.n? yy.ater P.!??.?.6!! ?e.9y!?*'9n$                   1             i
 1?.?? 1?.AA   jAcid Rain Program: Continuous Emissions Monitoring Rule .Revisions  ]64 FR.,28564 |
 1999 JEPCRAITRI: Reporting Threshold Amendment for Certain Persistent and       I64 FR 58666 i$
      ]       i?]oaccurnulatiye[.Toxjc[.Chemicals j^PBTs)                            ]         .    }
 1999 ICWA  [Underground Injection Control Regulations for Class V Injection Wells, I64CR68545I
      j       iRevisiion                                                         ]             j
 1999 [CWA  [NPDES: Regulations for Revision of the Water Pollution Control        J64 CR 68721 [$
      |       ^Program Addressing Storm Water Discharges                       ]      _       ]
 1999 ICWA  IWater Quality Standards: Establishment of Numeric Criteria for Priority J64FR61181 l$
      i       iToxic Pollutants; States' Compliance-Revision of PCBs Criteria        I             I
.............. A .*«............ *............. •'••'...<.........;....-......«.............17..........,.,...................P..............'........................... ••.... A.-.---—--'---...........--^-'-
 1999 [CWA  INPDES Permit Application Requirements for Publicly Owned          |64 CR 42433 |$
      I       [Treatment Works and  Other Treatment Works Treating Domestic      I             I
      j       jS^age                                                         j             ]
 2000 JCAA   jControl of Emissions of Air Pollution from 2004 and Later Model Year  J65 FR 59895 |$
      I       [Heavy-Duty Highway Engines and Vehicles: Revision of Light-Duty    I             i
      ]     _u._jTruck Definition                                                   ]             |
 2000 [CAA   INonroad Spark-Ignition Engines At or Below 19 Kilowatts             [65 FR 24267 i$

 2000 ICAA   IProtection of Stratospheric Ozone: Incorporation of CAA for Reduction [65 FR 70795 j$
      1       .l'n. P.'a.55.'- .?r.9yp V 5.9n!.™!!.?d ?yp..?*?.n9.e.?                          i             i...
 2000 IRCRA [Hazardous Waste Management System - Identification &  Listing -      165 FR 67067 |
      I       iChlorinated Aliphatics Production Waster, Land Disposal  Restrictions,  I             I
      1       ICERCLA                                                        I             I
..............4...............,{. .....«...«..............«..........*.,...	.........^	.. + ..,
      .JQ^....JB.yte[nsK^
                                               37
Report 2003-P-00003

-------
SHwflisfMn
2000
2000
2000
2000
2000
2001
2001
2001
2001
2001
2001
2001
2001
2001
2001
2001
CAA [Tier II Light-Duty Vehicle and Light-Duty Truck Emission Standards and
iGasoline Sulfur Standards
SDWA INPDWR: Radionuclides
CWA iEffluent Limitations Guidelines, Pretreatment Standards, and New
jSource Performance Standards for the Centralized Waste Treatment
iPoint Source Category
CWA lEffluent Limitations Guidelines, Pretreatment Standards, and New
jSource Performance Standards for the Transportation Equipment
[Cleaning Point Source Category
CWA iRevisions to the Water Quality Planning and Management Regulation
land Revisions to the NPDES Program in Support of Revisions to the
[Water Quality Planning and Management Regulation
CAA iHeavy-Duty Engine Emission Standards & Diesel Fuel Sulfur Control
[Requirements
TSCA iLead: Identification of Dangerous Levels of Lead Pursuant to TSCA
'Section 403
FIFRA 'Plant-Incorporated Protectants
CAA JNESHAP: Chemical Recovery Combustion Sources at Kraft, Soda,
•Sulfite and Stand-Alone Semichemical Pulp Mills
CWA iFurther Revisions to the CWA Regulatory Definition of Discharge of
[Dredged Material
SDWA iNPDWR: Arsenic and Clarifications to Compliance and New Source
jContaminants Monitoring
CWA iEffluent Limitations Guidelines and New Source Performance
iStandards for the Oil and Gas Extraction Point Source Category; OMB
•Approval Under the Paperwork Reduction Act: Technical Amendment
SDWA iNPDWR: Filter Backwash Recycling Rule
CWA JNPDES: Regulations Addressing Cooling Water Intake Structures for
iNew Facilities
RCRA [Hazardous Waste Management System: Identification and Listing of
jHazardous Waste: Inorganic Chemical Manufacturing Wastes; Land
jDisposal Restrictions for Newly Identified Wastes; and CERCLA
[Hazardous Substance Designation and Reportable Quantities
RCRA {Amendments to the Corrective Action Management Unit Rule
65 FR 6698
65 FR 76707
65 FR 81 241
65 FR 49665
65 FR 43585
66 FR 5002
66 FR 1205
66 FR 37772
66 FR 31 79
66 CR 4549
66 FR 6975
66 CR 6849
66 FR 31085
66 FR 65256
66 FR 58257

$1




$
$

$

$





38
Report 2003-P-00003

-------
                                                       Exhibit 2
          Numbers and Completeness of
                 Critical Documents
Municipal Waste Combustors
Synthetic Chemicals Monitoring
Acid Rain Permits
Land Disposal Restrictions
Reformulated Gasoline
Great Lakes Water Quality
Municipal Solid Waste Landfills
Biotechnology
Pulp and Paper (Air)
Pulp and Paper (Water)
Disinfectants and Byproducts
Poiychlorinated Biphenyls
Regional Ozone
Nonroad Diesel Engines
Plant-Incorporated Protectants
TOTALS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

15
49
12
85
19
64
25
25
10
14
59
10
42
8
15
452
iSome level 2
IAII level 2 and some level 3
lAII level 2
iAII level 2 ;
iSome level 2
IAII level 2
[Some level 2
ISome level 2
iAII level 2
iSome level 2
iSome level 2 and some level 3
ISome level 2
'All level 2 and some level 3
iSome level 2
iSome level 2

Level 1 = Major support document (e.g., regulatory impact statement)
Level 2 = Document referenced in the major support document
Level 3 = Document referenced in a level 2 document
                             39
Report 2003-P-00003

-------
         Who Performed the Science Work
                                                            Exhibit 3

1991 iCAA IMunicipal Waste Combusters i 0 I 6
1991 ISDWA 'Synthetic Chemicals Monitoring
1993 ICAA lAcid Rain Permits
1994 IRCRA Land Disposal Restrictions
1994 ICAA iReformulated Gasoline
1995 !CW A iGreat Lakes Water Quality
1996 ICAA 'Municipal Solid Waste Landfills
1997 JTSCA 'Biotechnology
1998 iCAA iPu Ip and Paper (Air)
1998 ICWA iPulp and Paper (Water)
1998 IS DWA 'Disinfectants and Byproducts
1998 ITSCA IPolychlorinated Biphenyls
1998 JCAA jRegional Ozone
1998 ICAA iNonroad Diesel Engines
2001 IFIFRA iPlant-lncorporated Protectants
Total Number
4
0
0
3
21
1
12
0
0
16
0
6
1
4
68
7
2
4
5
14
3
7
3
9
7
4
10
4
10
95
1
5
1
0
0
5
0
0
0
0
11
1
4
0
0
28

0
3
0
0
0
12
0
2
0
0
4
0
3
0
0
24
1
2
0
0
1
5
1
0
0
0
6
0
7
1
1
25
9
27
10
81
10
11
21
5
7
5
19
5
20
5
3
238
0 I 17
3
0
0
0
0
0
0
0
0
2
0
0
0
0
5
51
13
85
19
68
26
26
10
14
65
10
50
11
18
hBWiygB
AC:    Academia
IP:    EPA in-house - Program Office
IO:    EPA in-house - ORD
OF:    Other Federal Agency
OG:    Other (non-Federal) government entity
PS:    Private sector
U:     Unknown
                               40
                    Report 2003-P-00003

-------
          Who Funded the Science Work
                                                         Exhibit 4
1991 ICAA 'Municipal Waste Combusters
1991 iSDWA iSynthetic Chemicals Monitoring
1993 iCAA lAcid Rain Permits
1994 IRCRA lLand Disposal Restrictions
1994 ICAA 'Reformulated Gasoline
1995 iCWA iGreat Lakes Water Quality
1996 iCAA iMunicipal Solid Waste Landfills
1997 ITSGA [Biotechnology
1998 iCAA iPulp and Paper (Air)
1998 ICWA IPulp and Paper (Water)
1998 ISDWA -'Disinfectants and Byproducts
1998 ITSCA iPolychlorinated Biphenyls
1998 iCAA 'Regional Ozone
1998 JCAA iNonroad Diesel Engines
2001 JFIFRA iPIant-lncorporated Protectants
Total Number
12
21
12
84
11
14
12
11
5
10
22
9
16
7
14
260
5
21
8
1
0
9
5
5
0
1
16
1
13
0
0
85
0
3
0
0
0
17
0
2
0
0
4
0
4
0
1
31
2
4
0
0
7
26
7
5
5
4
21
0
17
1
1
100
0
1
0
0
1
2
1
4
0
0
1
0
0
0
0
10
19
50
20
85
19
68
25
27
10
15
64
10
50
8
16

PO:   EPA Program Office
ORD:  EPA Office of Research and Development
OF;   Other Federal Agency
O:  Other
U:  Unknown
                             41
             Report 2003-P-00003

-------
               Funding  Mechanisms  Used
                                                                      Exhibit 5
l?.?Jj§P.w.AjSynthefc^
1993 ICAA   lAcid Rain Permits
 0
 0
                                                   0..
                                                   0
     33.
     10
                                                               0

                                                               0
                                                               0
                                                               5
                              2

                              0
                              0
                              1
                             T{
                              8
                             "5"
                              0
                              1
                              6
                              0
                             "2
                              0
0
0
                2
                4
                    12
                    85
1994 JRCRA jLand Disposal Restrictions
!?.?.4 ip.AA   jRefprmulated Gasoline
.l??5.icWAMJGreaJ_Lakes Wat^
 0
 1
.„..
                                                   0
                                                   0
     81
      6
1
ii
                5
               12
                    19
                    "Sis'
                                                   0.
                                                   0
     10
     11
I?.?.6. IP.AA   jMunicipaj Splid Waste Landfills
1?.?..?..1T.§.9.A. M'?.*?.^.1?.?!?^...
iggslcAA
2
2
                3
                7
                6
                                                               0
                                                              "T"
                                                               3
                                                               4
                                                               20
                                                               0
                                                               8
                                                              III
                                                              .......
                    25
                    ..„.„.
 4
"b"
                                                   1
                                                   0
      7
      7
0
0
                    10
                    14
1998JCWA
1?.?,?. .1§.P.W.A j.Q '?.in^9!:a.nj!,s..?.n.91..?y.p.r.9d.y!?.t.s..
1998*
 0
y
                                                   0
                                                   0
      8
     16
3
0
               12
                5
                    60
                    10
 0
..„..
                                                   0
                                                   1
      5
     24
3
1
                                                              4
                                                             "3
                    Diesel Engines
2001 jFIFRA iPlant-lncorporated Protectants
 0
..„..
 P...
......
 4
"6"

           Total Number
25
                                              15
     230
                    85
          63
G:     Grant
CA:    Cooperative Agreement
IAG:    Interagency Agreement
C:     Contract
  I:
  O:
  U:
                                                  EPA in-house
                                                  Other
                                                  Unknown
                                    42
                         Report 2003-P-00003

-------
                  Peer Review Actions Taken
                                                                       Exhibit 6

1991 !CAA -Municipal Waste Combusters
1991 ISDWA {Synthetic Chemicals Monitoring f
1993 iCAA lAcid Rain Permits
1994 iRCRA land Disposal Restrictions
1994 ICAA -Reformulated Gasoline
1995 JCW A jGreat Lakes Water Quality
1996 iCAA -Municipal Solid Waste Landfills
1997 ITSCA iBiotechnology
1998 ICAA iPulp and Paper (Air)
1998 iCWA IPulp and Paper (Water)
1998 ISDWA iDisinfectants and Byproducts
1998 ITSCA iPoIychlorinated Biphenyts
1998 ICAA iRegionai Ozone
1998 ICAA iNonroad Diesel Engines
2001 IFIFRA iPlant-lncorporated Protectants
Total Number

1
6
0
85
0
5
0
9
10
13
0
8
7
0
0
144

12
16
0
0
9
16
24

0
0
18
1
12
6
10
132

1
3
12
0
0
6
o
3
0
0
0
0
1
0
3
29

0
0
0
0
0
0
0
0
0
0
4
1
11
1
0
17

1
19
0
0
10
37
1
5
0
0
33
0
10
1
2
119
'UPI
0
5
0
0
0
0
0
0
0
1
4
0
1
0
0
11

15
49
12
85
19
64
25
25
10
14
59
10
42
8
15

N:     No peer review done              OEP:
U:     Unknown whether peer review done
FACA: Peer review done by              ENP:
        Federal advisory committee
Peer review done by some other public
 review process by external experts
Peer review done by external group
 in a non-public manner
Independent internal EPA review
                                      43
                      Report 2003-P-00003

-------
                                         Exhibit 7
EPA Science Advisor Comments
      The full text of the comments follows.
                 44                 Report 2003-P-00003

-------
                 UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                                   WASHINGTON, D.C. 20460

                                       August 19, 2002
                                                                          OFFICE OF
                                                                    RESEARCH AND DEVELOPMENT

MEMORANDUM

SUBJECT:   Response to OIG Draft Audit Report - Science to Support Rulemaking

FROM:      Paul Gilman        /s/Paul Oilman
             Science Advisor to the Agency (8101R)

TO:         Nikki Tinsley
             Inspector General (2410T)

       This memorandum transmits comments on the Office of Inspector General's (OIG) Draft
Report, Science to Support Rulemaking, dated July 19,2002.  Briefly, I concur with the
suggestions made to improve the transparency and consistency with which science is applied to
Agency rulemaking.

       In early spring of 2001, the Administrator recognized the need to improve the scientific
and economic basis of Agency decisions and commissioned a task force to develop
recommendations for improving the rulemaking process. The task force made many
recommendations, and the Agency is well along the way towards implementing them.

       One key outcome of the Administrator's Task Force is to increase the Office of Research
and Development (ORD) involvement in the Agency's decision-making process.  Another
outcome is that the Administrator named me to serve as the Agency's Science Advisor. I take this
role very seriously and plan to make important strides in ensuring that Agency decisions are based
on sound science, and that science is presented and characterized properly in our rules and other
important documents.

       I am concerned about the OIG's finding (discussed on pages 17-18 of the draft report) that
critical science supporting the rules often was not peer reviewed. I plan to review the Agency's
progress in implementing its Peer Review Policy during the coming year.

       In addition to implementing the recommendations to improve the scientific basis of our
decisions, the Agency is working to  finalize Information Quality Guidelines that will apply to all
information that it disseminates. These guidelines, which will be effective in October 2002,
present the Agency's procedures for ensuring the quality of information that we disseminate, and


                                          45                        Report 2003-P-00003

-------
provide an opportunity for the public to request correction of information that does not comply
with the guidelines.  These guidelines will help to improve the quality and transparency of our
decision-making.

       ORD would like to offer three related comments to sharpen the accuracy of the report.
First, the draft report uses the term "primary document" to refer to the documents considered to
have most critically influenced a regulatory decision. In the draft report, primary documents are
described as those that "embodied the final process of gathering together the science and other
information to support the rule," with examples being background support documents, regulatory
impact analyses, and economic impact analyses (page 6).  A different term should be used to refer
to these documents (perhaps "critical document"), because in the  scientific community a primary
document generally refers to original scientific research, rather than gathering, reviewing and
analyzing data collected by others.

       Second, the draft report indicates an effort was made to determine which organizations
performed and funded the science work embodied in the critical document (page 7).  While it
should be possible to determine who prepared and  funded the critical document, the value of
doing so  is not clear, because the scientific research embodied in what OIG refers to as the
primary documents was likely performed by many individuals and organizations whose work was
being summarized. For example, all of the critical scientific research could have been performed
by EPA scientists, but the critical document summarizing it was prepared by a contractor. This
might present an inaccurate picture about the contribution of EPA scientists. The ambiguity
inherent in this situation should be acknowledged.

       Finally, while Agency preambles should effectively communicate the scientific
underpinnings of the rules, the description of the professional norms for such communication
(pagel 1) is not accurate. The norms as described accurately reflect how scientists communicate
in their primary documents, but not how science is communicated in what is described as critical
primary documents.  The key issue is that the preamble should present a clear summary of the
science supporting the regulatory decision, including properly characterizing risks and the
supporting science for risk management. The preamble should list the documents from which its
science-based statements are made and the docket should contain the complete record. This
would allow readers to refer to the source material, including the  original primary science
documents referenced in the critical documents (using "primary document" as traditionally used in
the science community).

       I appreciate the opportunity to review and respond to the draft report.  Science must play a
more prominent role in Agency decision-making. As Science Advisor to the Agency, one of my
objectives is to ensure that the critical scientific information used in our decisions meets the
highest standards of quality and transparency.
                                           46                         Report 2003-P-00003

-------
cc: ORD Executive Council
   ORD Management Council
   ORD Science Council
   R.Dyer(8104R)
   C.Bosma(8104R)
   C.Varkalis(8102R)
                                      47                      Report 2003-P-00003

-------
                                        Exhibit 8
Office of Water Comments
   The full text of the comments follows.
               48                   Report 2003-P-00003

-------
5
                         UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                                          WASHINGTON, D.C. 20460
                                                                                   OFFICE OF
                                                                                    WATER
                                        September 10, 2002
MEMORANDUM


SUBJECT:  OIG Report:  Science to Support Rulemaking

FROM:    G. Tracy Mehan /s/
               Assistant Administrator for Water

TO:            Jeffrey Harris, Director for Program Evaluations
                   Cross-Media Issues
               Office of Inspector General

        Thank you for the opportunity to review the suggestions in the subject report for improving the use of
science in EPA rulemaking.  This is an issue of critical importance to the Agency's credibility.

        We believe there is merit in the proposal to improve the consistency of the presentation of scientific data
and conclusions in regulatory preambles. However, more work is needed to determine how to implement such a
proposal given the wide variety of types of regulations the Agency develops. We also need to consider the impact on
the cost of developing rules and on the length of preambles. OW will be happy to work with your staff, OPEI, and
other offices to develop these proposals further.

        The proposal to use information technology to hot-link various elements of the rulemaking package and
Agency databases is intriguing and should be developed further. We also support reinforcement of EPA's peer
review policy.

        We are less positive about the proposal to "focus more attention in the development phase of regulations on
collecting data and doing research to close "blind spots" to support ralemakings. For major rulemakings, there may
be many years of data gathering and research available, but gaps in knowledge always remain. A decision to defer
action while research is done is as much a risk-management decision as any other and implies the continuation of the
status-quo for an additional period of time.

        Comments by our staff are attached for your consideration.

Attachment

cc:     Jay Messer
        Thomas Gibson
        Al  McGartland
                                                                                Report 2003-P-00003

-------
                                      Comments

              OIG Draft Pilot Study: Science to Support Rulemaking
       The Pilot Study was undertaken to assess the use of science in EPA rulemaking.
Predictably, many methodological issues were encountered, exacerbated by the difficulty of
assessing efforts that occurred as much as a decade ago. The OIG is proposing not to undertake a
more comprehensive study at this time. We support this conclusion.

       The Pilot Study did lead, however, to some interesting suggestions. These comments
center on the suggestions, which were:

          1.  Consider presenting the scientific findings that support a rule in specific sections of
             the preambles. These findings should be organized according to the norms of
             science, in summary form, and indicate:

                    *•  Why the science is required to support the rule, and how the results will
                       be used.
                    *  The methods used.
                    *  The important results (showing key data and their uncertainty).
                    *  Interpretation of the findings, and comparison with other studies that
                       appear to support or contradict the results.
                    >  Scientific referencing of underlying scientific and technical documents.
                    *•  A separate section of the preamble that would bring in issues of law,
                       policy, economics, and administrative discretion that do not depend on
                       the scientific findings.

             2.  Focus more attention in the development phase of regulations on collecting
                 data and doing research to close "blind spots" to support rulemakings.

             3.  Take advantage of EPA's information technology capabilities to:

                    >  Hotlink references in preambles to documents in the docket.
                    >  Link scientific and technical documents in the docket to the Science
                       Peer Review Database.
                    >  Link RAPIDS to the Science Peer Review Database.
                    »•  Maintain through RAPIDS an inventory of all rules proposed and
                        finalized each year.

             4.  Reinforce EPA's current peer review policy, ensuring that all EPA-generated
                 documents critical to significant and substantive rulemakings are independently
                 peer reviewed, and that the responses to the significant comments appear in the
                 documents.
                                           50                         Report 2003-P-00003

-------
1. Consider presenting the scientific findings that support a rule in specific sections of the
preambles.

       The study team found an absence of consistency in reporting and using scientific findings
in the rules which made them difficult to compare. The goal of fostering consistency in this area
is a desirable one and will make things easier for our stakeholders.  At the same time, it must be
recognized that EPA develops a wide variety of types of regulations based on a variety of statutory
mandates: for example, some are technology based, some based on individual risk targets, some
on balancing costs and risks.  Some are directed at human health risks, others at ecological risks,
others at both.

       It is not clear how one would formulate a structure that would accommodate this variety.
The suggestions in the report, the so-called "norms of science", are really a model for reporting
research results from a particular investigation. What we have in rulemaking preambles and
supporting documents is generally a summary of knowledge in a particular area leading to a
conclusion that contributes to the decision. In terms of scientific literature, this is more like a
review article than a research report.

       Further, requirements in this area have the potential of increasing the costs of rulemaking
and increasing the length of preambles. These undesirable effects should be minimized as  we
develop ways of fostering greater consistency in the presentation of scientific findings.

       In summary, this proposal has merit but needs further development before it can be
adopted. OW will be happy to work with OIG, OPEI, and other offices on these issues.
2. Focus more attention in the development phase of regulations on collecting data and
doing research to close "blind spots" to support rulemakings.

       Many EPA regulations are based on years of research and data gathering by EPA, other
Federal agencies, academia, and industry. For example, we have been working on the arsenic
drinking water standard steadily since the 1970s. OW and other programs have extensive
processes of joint planning with ORD and outside stakeholders to anticipate information needs as
much as possible.

       Yet, there are always data gaps and uncertainties which we must grapple with. This is in
the nature of the rulemaking enterprise.  While the Pilot Study cited respondents who said they
would have liked to have had more data, it did not identify any particular ways of obtaining it
without increasing costs or slowing down action.  Allocating resources to closing "blind spots"
means something else will not be done, and delaying action means the status quo will continue.
                                           51                          Report 2003-P-00003

-------
3.    Take advantage of EPA's information technology capabilities.

      There are some intriguing possibilities here. We would support an effort to identify and
implement ways to improve the information the Agency makes available on rulemaking.
4.     Reinforce EPA's current peer review policy.

       EPA's Peer Review Policy was first issued in 1992, after some of the rales considered in
the Pilot Study.  Full implementation has taken time and continuous effort.  Thus, it would not be
surprising that compliance was limited in the earlier period, but we would hope that it had been
improving as we approach the present time. Unfortunately, the report does not present
information on peer review performance over time, so we cannot tell whether this has happened.
We also note that, in many cases, the investigators could not determine whether documents were
peer reviewed.

       It is not clear whether the investigators are proposing a change in the Agency's Peer
Review Policy.  The recommendation (quoted above) does not appear to be any different from the
current policy.  If a change is being suggested, this should be made clear.
                                          52                         Report 2003-P-00003

-------
                               OGWDW Comments

          OIG Draft Pilot Study: Science to Support Rulemaking
Factual Accuracy of Report

1) Exhibit 1 incorrectly lists the following rules as significant (by EO12866): Filter Backwash
Recycling, Consumer Confidence report. The Exhibit incorrectly lists the Radon Rule as finalized
in 1999; the Radon rule has not yet been finalized.

Exhibit 1  also incorrectly records the Arsenic rule as being withdrawn, which is not accurate.  The
original Arsenic rale promulgated on January 22, 2001.  EPA temporarily delayed the effective
date for this rule for 60 days, rom March 23, 2001 until May 22, 2001 (66 FR 16134), in
accordance with the memo from Andrew Card entitled "Regulatory Review Plan". The effective
date was again delayed to Februrary 22,2002 (66 FR 28342) to conduct reviews of the science
and cost analysis.

2) Different numbers for total critical documents are given for Exhibits 3 (471), 4 (469), 5 (443),
and 6 (436) and on page 18 of main text (2940). Also the number of critical documents listed for
individual rules are inconsistent across exhibits. It appears that these numbers should be the same
- check these numbers and correct or otherwise provide an explanation for differences.

3) Page 18 of the main report states that EPA's Peer Review Product Tracking System data base
should be the primary means for tracking present, past, and future peer review status of critical
science documents identified in the pilot. It further states that, since the tracking system was
developed, only 4 of 364 critical documents identified in the case studies, were found listed in this
data base. This appears to be an inappropriate suggestion as EPA's tracking system is designed for
EPA generated documents whereas only a fraction of the critical documents (116/471 -Exhibit 3)
are generated by EPA.
Report Content

1) The discussion of critical science source and funding did not discuss the significance of the
information or how it relates to the report's recommendations. The report does not discuss the
linkage between source of funding and peer review status, and these data are important. The
report should clarify how the two relate.

2) It is important to recognize, at least with regard to drinking water rules (largely due to the 1996
SDWA amendments), that the science discussion in preambles has evolved significantly in the last
10 years. Thus, the analysis of case study 2 is very outdated and does not reflect practices since
1996.
                                          53                         Report 2003-P-00003

-------
3) Please define the RAPIDS system and explain its purpose for the benefit of readers unfamiliar
with the database.

Report Recommendations

1) The report's first recommendation to separate the discussion of science versus non-science
influences into different preamble sections does not appear efficient as there would be significant
redundancy in the discussion of rule criteria in each section. If this recommendation is
implemented, it could result in doubling the preamble discussion, while not necessarily facilitating
a better understanding of the basis for the rule.

Depending on statutory requirements for a given rule, the optimal preamble structure for
communicating the role of science may be quite different. Any recommendations to revise the
format for rule preambles across the Agency should be flexible and take this consideration into
account. To achieve the same objectives of the report, we recommend modifying the
recommendation to suggest that norms of science be applied consistently throughout current
preamble formats where science is discussed, in order to improve the understanding of the
scientific basis for rules.

2) We support the report's second recommendation to reduce "blind spots." However, if more
greater data collection is mandated, this recommendation has major resource implications if
additional funding is not provided.  It could also negatively impact the ability to meet statutory
deadlines.

The 1996 SDWA Amendments require EPA to use "the best available, peer-reviewed science and
supporting studies conducted in accordance with sound and objective scientific practices" when
setting drinking water standards (sec. 1412 (b)(3)(A)). The US Court of Appeals for the District
of Columbia Circuit determined that Congress's intent was best available evidence at the time of
rulemaking. EPA agrees with this assessment.

3) We support the report's third recommendation to make better use of the Agency's information
technology capabilities. Consistent use of these tools throughout the rulemaking process will
improve communication and access to the critical scientific support documents.

4) We support the report's fourth suggestion to reinforce the Agency's peer review policy.
Because the current policy does not explicitly require peer review
(http://www.epa.gov/osp/spc/perevm em,htjn). it may be appropriate to recommend updating the
policy to require peer review in certain situations to ensure it is applied more consistently across
the Agency. A more consistent means of tracking peer reviewed documents will be very
beneficial, and should help clarify the fact that  many of the studies listed as having unknown peer
review status in the draft report were actually peer reviewed.
                                            54                         Report 2003-P-OOOQ3

-------
Factual Accuracy of Case Study Discussions

Synthetic Chemicals Monitoring - Case Study 2

(1) Page A-6: the  last paragraph beginning discussion of Category I, II and ffl pollutants.

This whole discussion needs some work - some statements are not quite accurate. We suggest the
following replacement paragraph:

"Category I contaminants are those which EPA has determined there is strong evidence of
carcinogenicity from drinking water ingestion and the MCLG is set at zero. Category II
contaminants are those which EPA has determined that there is limited evidence for
carcinogenicity from drinking water ingestion.  The MCLG for Category n contaminants is
calculated using the RfD/DWEL with an added margin of safety to account for cancer effects or
are based on a risk range of 10"4 to 10"6 when data are inadequate to derive an RfD. Category HI
contaminants are those which there is inadequate evidence of carcinogenicity by drinking water
ingestion. For Category in contaminants, the MCLG is established using the RfD. The science
issues with respect to the MCLGs thus involve health risk assessments that deal with all the above
aspects for each of the pollutants."

(2) Page A-8: third paragraph.

We suggest striking the first sentence that reads "Compliance with the MCL is determined by
analysis with approved analytical techniques."  While this is a true statement, this is not an
appropriate lead into the discussion on PQLs and analytical feasibility limitations in setting of an
MCL.  We suggest replacing it with the following sentence: "The feasibility of setting an MCL at
a precise level is also influenced by laboratory ability to reliably measure the contaminant in
drinking water."  Also, there is a typo toward the end of this paragraph ... instead of PCLs - this
should be PQLs.

(3) Page A-9: first full paragraph that begins with "EPA proposed monitoring requirements ...."

Unfortunately, this paragraph does not provide  complete information regarding the final decision
for unregulated contaminants and may be a little misleading. The report cites the January 30,
1991 Final NPDWR for SOCs, lOCs, and unregulated contaminants. This paragraph discusses
that EPA proposed monitoring requirements for ~ 110 unregulated contaminants and notes that
EPA adopted a scheme requiring all systems to monitor for the highest priority organics, unless a
vulnerability assessment determined that a system was not vulnerable to contamination, but it fails
to specifically state that the final rule settled on a one time monitoring requirement for 30
unregulated organic and inorganic contaminants.  The report does note this on the first page but
we think this should be restated here as well.

(4) Page A-9: second full paragraph on SMCLs.
                                           55                         Report 2003-P-00003

-------
Tthe next to the last sentence is missing a parentheses to close out "discoloration of water." Also,
we suggest breaking the second to the last sentence into two so that the following phrase for the
aluminum SMCL can be included:

"EPA dropped the proposed organics SMCLs but retained the existing odor SMCL of 3 Total
Odor Number (TON). The Agency finalized an SMCL range for aluminum (due to discoloration
of water) with the precise level for each system being determined by the State. Furthermore, the
Agency deleted an MCL for silver  and finalized an SMCL to protect against skin dicoloration or
argyria from a lifetime exposure."

(5) Page A-9: third full paragraph, five lines down - this should be 1,2-dichloropropane not 1,2-
dichloropropanol.

(6) A couple places - chromium is  capitalized (and it does not begin a sentence) - change to
small case.

(7) Page A-16 - Last paragraph ....  we suggest rewording the last two sentences as follows:

" Just as we were finishing the study, OGWDW announced its  preliminary decision not to revise
NPDWRs for 68 chemical contaminants. The Agency stated that the 68 chemical NPDWRs
should not be revised at this time for one of the following reasons:

>      36 NPDWRs were undergoing Agency health risk assessments. These assessments are not
       expected to be complete in time for EPA to make its final revise/not revise decisions.
*-      17 NPDWRs remained appropriate and any new information available to the Agency
       supports retaining the current regulatory requirements
*•      12 NPDWRs had new health, technological,  or other information that indicated a potential
       revision to MCLG and/or MCL; however, the Agency believed any potential revision
       would result in a minimal gain in the level of public health protection and/or provide
       negligible opportunity for significant cost-savings.
*•      3 NPDWRs had data gaps or research needs that needed to be addressed before EPA could
       make definitive regulatory  decisions. When  the data gaps have been resolved, EPA plans
       to consider the results  in the next review cycle.
Stage 1 DBPR - Case Study 11

(1) Page A - 80 : "Brief description of science input to the rule"

The last two sentences are incomplete in their intended coverage and should be revised. We
suggest replacing the last two sentences with the following: "In addition, EPA needed to assess
risks associated with DBF occurrence levels and to evaluate best available technologies for
reducing such risks to feasible levels (while not compromising microbial protection). Using
scientific and technological information gathered, EPA defined best available technologies,
criteria by which total organic carbon (naturally occurring organic precursors to DBP formation)


                                            56                          Report 2003-P-00003

-------
should be removed, and how various DBFs and disinfectants should be measured and
monitored.."

2) Many of the critical documents cited in the "Table of Critical Documents" as having unknown
peer review status are actually published in journals that require peer review (e.g., JAWWA,
Epidemiology). This may be true for other case studies. It will be worthwhile to reassess and ret-
tally these classifications with consideration of studies published in peer-reviewed journals-the
report's conclusion's may be influenced by such an exercise.

3) The following documents should be listed as primary (Ref #46,47, 48) to be consistent with
the text  describing "primary" documents in the main report.
                                           57                         Report 2003-P-00003

-------
                                         Exhibit 9
Office of Prevention, Pesticides and
   Toxic Substances Comments
       The full text of the comments follows.
                  58                 Report 2003-P-00003

-------
I" _ KM  \       UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
        ' |                     WASHINGTON D.C., 20460
                                                                      OFFICE OF
                                                                 PREVENTION, PESTICIDES AND
                                                                    TOXIC SUBSTANCES
                                      September 13, 2002

    MEMORANDUM

    SUBJECT:  Review of the Office of Inspector General's Pilot Study on Science to
               Support Rulemaking - OPPTS Comments

    FROM:  Angela F. Hofmann //s/ Angela F. Hof mann
            Director of Regulatory Coordination
            Office of the Assistant Administrator (7101M)

    TO:     Jeffrey Harris, Director for Program Evaluations on Cross-Media Issues
            Office of Inspector General (OIG) (2460T)

         Thank you for the opportunity to review the Office of Inspector General's (OIG's)
    draft report entitled: "Science to Support Rulemaking", a pilot study which evaluated the
    Agency's use of science to support EPA rulemaking.  We coordinated the review for
    OPPTS, and respectfully submit the attached comments and suggestions for your
    consideration.

         The pilot team sought to identify the role that science played in supporting 14
    EPA rules promulgated between 1994 and 2001, represented by 15 case studies. Of
    the 15 case studies, three were related to rulemakings promulgated by OPPTS, i.e.,
    case studies 8,12 and 15. We have specific comments on two of the case studies, i.e.,
    case studies 8 and 15, and do not have any comments on case study 12, which
    involved the PCB Disposal Amendments.

         We discussed our specific comments on Case Study 15, the Plant Incorporated
    Protectants Rule, with Chris Baughman, and she has addressed our comments with the
    changes that we identify in the attachment. We discussed our comments on Case
    Study 8, the TSCA Biotechnology Rule, with Jay Messer, and he has indicated that he
    will consider our specific suggestions contained in the attachment.  In addition, we have
    provided some general comments and suggestions that we believe will help improve
    the report.

          If you have any questions about our comments, please contact Sandy Evalenko
    on my staff at 564-0264.  Thank you.

    Attachment
                                         59                      Report 2003-P-00003

-------
cc:    1O: Steve Johnson; Susan Hazen; Sandy Evalenko
      OSCP: Joseph Merenda; Tom McClintock; Elizabeth Mi lews ki
      OIG:  Chris Baughman; Jay Messer
                                    60                     Report 2003-P-00003

-------
Attachment

               OPPTS Comments on the OIG Pilot Study
                    "Science to Support Rulemaking"

A.  Comments Specific to the OPPTS Rulemakings

1. Text Changes Requested on Case Study 15 - Plant Incorporated Protectants

      a.  On page 13 of the draft report, the statement identified is contradicted by the
      finding on page A-108.  The statement on page 13 should be revised as follows:

      "The biotechnology rule (Case 8) and the plant-incorporated protectant rule
      (Case 15) both cited  references by number, which corresponded to a reference
      section in the preamble, but many of these references were to dictionaries or
      general textbooks and did not support scientific statements."

      b.  On page A-107 under brief description of the rule, the following statement
      should be corrected as follows:

      "In addition, the rule establishes a new part in the Code of Federal Regulations
      (CFR) specifically for plant-incorporated protectants, i.e., 40 CFR 174.
      Procedures are also  set forth for Confidential Business Information (CBI); any
      claim of confidentiality must be substantiated when the claim is made.  The rule
      also requires, for exempted plant-incorporated protectants not registered, that
      any person who produces, for..."

      c.  On page A-108, under the brief description of science  input to the rule, the
      following statement should be corrected as follows:

      "The rule was a legal mechanism to confirm that plant-incorporated protectants
      were covered by FIFRA. The science aspects concerned the exception
      exemption for protectants derived through conventional breeding from sexually
      compatible plants. To comply with FIFRA, such protectants could  may not
      generally cause unreasonable adverse effects on the environment."

2. Text Changes Requested on Case  Study 8 - TSCA Biotechnology

      a.  On page A-60,  under the brief description of science input to the rule, the
      following statements should be corrected as indicated:

      "The intent of the rule was to establish EPA's regulatory program for
      microorganisms, with the goal of provide regulatory relief to those wishing to
      use certain products of microbial biotechnology, while ensuring that EPA could
      adequately identify and regulate ..."  '

      b.  On page A-61,  the characterization in  the first bullet appears inconsistent with
      the discussion  below it, which acknowledges that EPA's decision under the rule
      centered on whether the product is "new," i.e., not whether the 2 classes present


                                      61                     Report 2003-P-00003

-------
different levels or types of risks but whether one class is more likely to be "new."
The first bullet needs to be corrected as indicated below:

"The science inputs to the rule should involve three key issues:

*  Requirement for an MCAN - Is the intergeneric microorganism more likely
   to be "new" there significantly less risk associated with intrageneric transfers
   than with intergeneric transfers?  Is there a significant risk associated with
   transfer of non-coding DNA?

c. On page A-61, the following paragraph does not explain why these issues
"clouded" or adversely affected the science considered in this rulemaking.  An
explanation is need to support the statements in the paragraph below or it needs
to be revised and clarified. This rulemaking was more of a procedural rule, with
the science used to determine the process and informational requirements that
would be applied when these microorganisms were reviewed by EPA as part of
the new chemical premanufacture notification requirements under TSCA.

"Identifying the exact role of science in this rulemaking Is clouded by
several issues.  First, the the Biotechnology Science Coordinating Committee
(BSCC) of the Domestic  Policy Council Working Group on Biotechnology in the
1980s was developing a  coordinated policy for dealing with biotechnology across
the various agencies with a regulatory role (e.g., FDA, USDA, and EPA). Each
of the Agencies and Departments then had to adapt the BSCC guidance to the
particular statutory requirements under which the organization had regulatory
authority.  Under TSCA,  EPA had to regulate microorganisms as "chemicals,"
because Congress had not specifically anticipated that genetically engineered
microorganisms would themselves act as "products." This combination of
restrictions brought about by the desire of the Federal government to have an
integrated approach, and the need to stay within statutory boundaries that were
somewhat artificial, greatly constrained the way new science could be applied to
the rule."

d. On page A-62, the purpose of this paragraph needs to be better explained.
What is the basis for the conclusion (highlighted below) that the articles didn't
appear to be critical to support the final rule? This information was indeed
important in supporting the specific requirements and reviews established in the
rulemaking for these new microorganisms under the premanufacture notification
provisions of TSCA.

"ORD also had a substantial research program in biotechnology in the 1980s. In
a presentation to the BSAC in April, 1987, the AA for ORD indicated that ORD
had  a budget of $7 million for R&D in biotech, approximately 80% of which was
in external research grants, primarily directed at developing "widely accepted
methods in ... microbial ecology." ORD projects aimed at evaluating monitoring
strategies for planned field releases were presented to the BSAC at the JULY
 1987 and January 1988  meetings. ORD reported on several biotechnology
workshops at the January 1989 meeting, and at the December 1989 meeting,
ORD presented a progress report on 53 projects that had been  conducted under
the program, the "primary foci of these studies [were] on detection and

                                62                      Report 2003-P-00003

-------
      enumeration, survival and colonization, and genetic exchange." The following
      excerpt from one of the BSAC members is telling, however. After noting that
      although he did not totally agree that the program was a success, but was "one
      of the most important efforts in the area of environmental science," he noted that
      "much progress had been made in the considering genetic, ecologic, and
      evolutionary issues	the information was still insufficient to give a definitive
      answer on what merited review." Although several journal articles funded by
      ORD are included in the docket, none appears critical to support of the
      final rule (nor do papers funded by other organizations)."

      e. On page A-67, the following conclusion (the last sentence in the paragraph
      under methodology - see highlighted text) is not supported by this paragraph.
      This statement requires additional explanation or it needs to be revised.

      "OIG had no response from any of the respondents. The information was
      developed by reading the rule and preamble, the primary technical support
      documents, the ESA report (Tiedje 1989), the RIA, the response to comments
      report, and the reports and minutes from the BSAC meetings in the docket. The
      reference lists for the primary documents, as well as research papers cited in the
      docket table of contents were identified, and scanned for content, funding
      sources, etc. Research funded by ORD and identified by acquisition number
      were tracked back to the original decision memos in the GAD files (most turned
      out to be competitively awarded).  It became obvious during this exercise that
      the research cited, while broadly relevant to the survival of artificially
      introduced microorganisms in the field and mechanisms of gene transfer^
      did not specifically support (nor specifically not supportHhe positions to
      which they were referenced, and thus they are not included in the list of
      critical documents."
B.  General Comments

      The following are general comments, observations and suggestions for your
consideration.

1.  Scope of the Pilot Study.

      Please clarify early in the report whether the pilot team considered economic
analyses when they evaluated "science" for the purposes of the pilot study. At times it
appears that the team's consideration was limited to what is traditionally thought of as
"science," i.e., scientific research and analyses of risks and effects. For example, in the
first paragraph of the Executive Summary and in the detailed  discussion of
methodology. Since many consider economic analysis to be a scientific discipline, it
would be helpful to describe what the team included as "science" in the context of their
evaluation of science in support of rulemaking.

2.  Understanding Rulemaking at EPA.

      We would like to make a few comments and suggest several improvements to
the discussion on this topic that appears on page 2.

                                      63                      Report 2003-P-00003

-------
a. Rulemakings are not just triggered by a statute, court order or executive
initiative as stated in the first sentence of the first paragraph.  Rulemakings may
also be triggered by a citizen who petitions the Agency to take a specific
regulatory action or to issue a rulemaking to address a particular concern, i.e., in
addition to the Administrative Procedures Act, several statutes contain specific
provisions that require the Agency to consider these petitions (e.g., TSCA
section 21, EPCRA section 313, etc.). Licensing actions may also use
rulemakings as the mechanism for implementing the licensing action. To avoid
the potential for the reader to conclude that rulemakings are only triggered as
described, we suggest that you preference the statement by inserting "Typically,"
or "In general," at the beginning of the first sentence.

b. As you know, rulemaking dockets for the major program offices are
maintained in a specific facilities, which were recently consolidated to create the
new EPA Docket Center located in the basement of EPA West. The
parenthetical description should be revised to explain that these "drawers of
paper files" can easily be accessed by the public, and are not files that are only
maintained by the individual rule leads.  Although opened  to the public only  this
past April, it should be noted that the Agency now makes these files publicly
available in its new online electronic docket and comment system, EPA Dockets.
For future rulemakings, the public will have easier and online access to non-
copyrighted and non-confidential references that are used to support a
rulemaking.

c. The process summary that is provided does not include one of the most
significant steps required for any significant or economically significant
rulemaking, i.e., review by the Office or Management and  Budget (OMB) and
other interested federal agencies and offices pursuant to E.0.12866.  This
review may often play a critical role in shaping the final rule that publishes in the
Federal Register, and can impact on how the science is presented in the
preamble.  We suggest that you add a new sentence to recognize this critical
step in the rulemaking process for both the proposed  and  final rule stages.

d. Although the public comment period  may typically  be 60 days, the Agency
often provides for 90 days or longer for economically significant rulemakings,
and, on occasion, may also provide just 30 days for public comment. We
suggest that you reflect this be revising the following sentence as indicated:

"After allowing at least 60 days for public comment (typically 60 days), EPA
finalizes the rule by publishing it in the Federal Register, with a new preamble..."

e. In the second paragraph, please clarify whether the 20 rules were categorized
as "significant" or "economically significant" under E.O. 12866. Although the
criteria for "significant" rulemakings that appear  in section 3(f) of E.O. 12866 are
identified, there is no explanation of here for "economically significant," although
that phrase is not used until the next page. Although the EO itself does  not
define this term, OMB's implementing guidance for EO 12866 defines this term
as rulemakings that meet the criteria in section 3(f)(1) of the EO.  Please note
that the economic trigger here is not the only one.  It can also be cost savings or
                                 64                      Report 2003-P-00003

-------
      a non-cost related reason as indicated by the second part of the criteria in
      section 3(f)(1).

      In addition to clarifying these terms, we suggest that the report clarify which
      criteria were used in selecting the rules evaluated. It is also important to clarify
      what is meant by "substantive rules," because that was a specific term of art that
      was used under the previous E.O. (EO  12291), which was replaced by E.O.
      12866. Today rulemakers use this term to distinguish a non-substantive rule
      (e.g., something more technical in nature that does not impact the scope or
      requirements of the rule - like a rule that changes how a form should be sent to
      EPA) from a substantive rule that changes requirements or behavior, takes an
      action, implements a decision, etc..

      f. For OPPTS, the remaining rules do not "primarily impact individual States,
      Tribes or sites, or involve minor modification and corrections to significant or
      substantive rules."  The remaining rulemakings in OPPTS are substantive rules
      that OMB specifically exempted from E.O.  12866 that are categorized as exempt
      and not as "non-significant" (i.e., rulemakings that establish pesticide tolerances),
      or the are otherwise substantive rules that are categorized as non-significant
      under E.O.  12866.  Only a few of the remaining rules involve corrections or minor
      modifications, or are otherwise limited to individuals.

      g. The determination of whether a rule  that was categorized as significant at
      proposal can be categorized as non-significant at final is one that is based on the
      criteria in section 3(f), and OMB's implementing guidance for EO 12866. If, for
      example, the agency does not receive adverse comments and the final rule is
      substantively similar to the proposal,  OMB may determine that the final rule is
      not significant. The last sentence of the second paragraph implies that the only
      time the categorization for the rule might change at the final rule is if the
      estimated costs decrease or the rule  is  determined to modify existing significant
      rules.  This statement should be corrected.
3. Significant Rules Identified

      We would like to make a few comments and suggest several improvements to
the discussion on this topic that appears on page 3.

      a. RAPIDS is an internal agency tracking database that was first used by
      programs around 1995 primarily to help facilitate the development and review of
      the Regulatory Agenda.  Active use of the system by the program offices was
      phased in across the Agency, which meant that some offices were entering their
      information directly, while others had their information entered by OPEI staff. In
      addition to standard  reports that any user can access, a special report may be
      generated, as long as the information sought is maintained in the system.  We
      do not believe that the criticism of OPEI and RAPIDS in this discussion is
      accurate.  We suggest that you discuss these details with OPEI and revise this
      discussion accordingly.
                                       65                       Report 2003-P-00003

-------
      b. With regard to searches for rules promulgated before 1994, it is important to
      note that pilot team did not have easy access to the information on these rules
      because the electronic reference sources that were used by the team, i.e., the
      website, and the electronic Federal Register access systems, were under
      development and only contain limited information for rules issued in 1994, or
      earlier. For this reason, the team was uncertain that the information on these
      earlier rules was complete. Searches using commercial electronic referencing or
      indexing sources might have identified rules for these earlier periods, as a
      manual search of the Federal Register indexes would have. To avoid a reader
      interpreting this discussion incorrectly as an indication that there isn't a way to
      generate such  a list for this earlier time period, we believe that this discussion
      should be revised.

      c. Correct the  reference to why a rule might be "economically significant," as
      discussed above.

      d. Clarify whether the 14 rules  were taken from the economically significant
      group or both.  For example:

      On  page 2 of the draft Report, it indicates that OPEl estimated that the Agency
      publishes 1,000 to 3,000 rules each year, and that "approximately 20 of these
      rules are "significant" according to E.G. 12866."  Is that 20 a year, or 20 total for
      the  period of the evaluation?

      On  page 3, it continues with the team having "identified 88 "significant rules" that
      were finalized in 1990 through 2001" then later that the team settled on "74 rules
      from 1994 on." And eventually explains that the pilot study focused on 14 of
      these rules.

      The criteria the team used to select only 14 rules out of the  potentially over
      12,000 rules that EPA promulgated in that time are not clear. No criteria for
      selection are described in the study other than the attempt to ensure that the
      rules evaluated would provide a wide range of statutes, and that the rules were
      not intended to be a representative sample.

      The report needs to further explain the selection criteria to provide credibility for
      the pilot study  using only these 14 rules to serve as the basis for supporting the
      general conclusions regarding the Agency's use of science in rulemaking and the
      related recommendations.
4. Rulemaking Expertise

      Rules written by the EPA serve a number of purposes, not all of them strictly
scientific.  Factors affecting the form and content of a rule include statutory, scientific,
economic, political and enforcement/compliance considerations.  Rules are first and
foremost legal documents written to meet several legal goals. They must also
communicate information clearly to the lay public, particularly information on how
individuals may comply with the rules. Any team undertaking a study along the lines of
this pilot should include the perspective of these other disciplines.


                                       66                      Report 2003-P-00003

-------
      For example, although the report notes that the rulemaking process is governed
by specific requirements contained in various statutes and Executive Orders (these are
in addition to any in the environmental laws referenced), there isn't a discussion about
what those requirements are, or how they may impact the development of, the analysis
performed, or the information considered as part of the rulemaking. In addition, during
the period covered by the study, many of these requirements were either newly
imposed, or recently revised. For example, the only executive order related to these
requirements that is mentioned, EO 12866, was issued In October 1994 as a revision to
a previous EO. Since then, over 10 more executive orders or statues were issued that
directly impact not only when an Agency must consider certain factors in rulemaking,
but how the Agency must perform specific analyses.  To be complete, any study about
the use of science in rulemaking must also consider the rulemaking context, and all of
the factors that must be considered by the Agency in making a decision.

      Along these lines, we believe that any future studies should also consider how
new requirements, whether procedural or policy related, that are intended to increase or
make improvements, end up impacting the use of science in support of rulemakings.
For example, on October 1 the Information Quality Guidelines are supposed to take
effect It would be interesting to see, when and how those  requirements might impact
our current activities with regard to science and rulemaking. The new electronic docket,
which will substantially increase access  to critical documents that are used to support
rulemaking, will also impact this issue.
5. Method of Selecting "Critical" Documents

The methodology of selecting "contacts" is not clear.  Of most concern is that, of the 83
identified contacts, no helpful responses were received from 58 of these contacts.  A
response from so few identified contacts, with a response rate of 33%, can introduce a
strong bias into the study. To their credit, the authors recognize this potential for bias in
their report.  They nonetheless offer strong recommendations on how preambles of
rules should be written. The team also ignored for the purposes of their study
documents that they could not find in the rule dockets. Dockets do not necessarily
contain a hardcopy of all documents associated with a rulemaking. Documents need
not actually be physically in the docket for the Agency to have relied on them.
6. Rulemaking Preambles

      The preamble to the rulemaking is not, nor has it ever been, considered the
proper vehicle for communicating the science in the manner prescribed on page 11.
The proper vehicle for communicating the science in that detail is in separate
documents that are made available to the public as part of the rulemaking docket, with
a general description description provided in the preamble.  The preamble must provide
a layman's explanation of the basis for the rulemaking, including the science, economic
and technical analyses and other considerations that informed the decisions
represented in the rulemaking.

      The suggested addition of these science discussions in the preamble is cost
prohibitive and impractical. For example, the suggested inclusion of the scientific

                                      67                      Report 2003-P-00003

-------
charts, graphs, and tables in the preamble would not only significantly increase the
publication costs only, it would require additional resources and overly complicate the
Agency's ability to ensure that the Federal Register document complies with
accessibility provisions in section 508 of the American Disabilities Act because tables,
charts and graphs require special programing to be electronically accessible for 508
readers.

      In addition, most stakeholders consulted in 1994, when we evaluated the level of
detail, format, and function of the preamble as part of the government wide streamlining
initiative, indicated that they prefer for the preamble to contain a succinct summary of
the science, economic and technical analyses and other considerations that went into
the rulemaking. This allowed those responsible for or interested in the different
disciplines to obtain a general understanding of all of these considerations, as well as
the details of the one of most interest to them.  Since the primary audience for the
rulemaking is not the scientists, including the detailed scientific information in the
preamble  would not serve as an effective way to communicate the scientific information
to the primary audience.
                                       68                       Report 2003-P-00003

-------
                                      Exhibit 10
Office of Policy, Economics, and
      Innovation Comments
      The full text of the comments follows.
                 69                 Report 2003-P-00003

-------
                     UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                                    WASHINGTON, D.C. 20460
                                                                   OFFICE OF POLICY,
                                                                ECONOMICS, AND INNOVATION
                                  September 22, 2002


MEMORANDUM

SUBJECT:   Review of Draft Report "Science to Support Rulemaking"

FROM:   Thomas J, Gibson
          Associate Administrator

TO:      Nikki Tinsley, Inspector General
          Office of the Inspector General

Thank you for the opportunity to review the draft report "Science to Support Rulemaking." This
report is important since it explicitly describes the role of science in achieving the Agency's
mission.  It is especially relevant in light of the Administrator's efforts to improve the Agency's
decision making by promoting the full integration of science, including economics, into the
regulatory process.  In addition, the report's suggestions support many areas for improvement
that are targeted in the Agency's new Information Quality Guidelines.

Having carefully reviewed the report, we offer the attached comments and suggestions on ways
in which we believe the report could be improved, as well as our reactions to each enumerated
suggestion made in the report. We hope that our suggestions will help strengthen the study's
findings and the conclusions you draw from them.

Our general comments on the report can be found in Attachment 1. Attachment 2 contains our
reactions to the report's suggestions, and Attachment 3 is detailed comments organized by page
number.

Again, thank you for the opportunity to review the Draft Pilot Study.  My congratulations on a
well-written report that clearly articulates the need to explicitly define the role of science in the
rule making process.  I look forward to seeing your revised report.

Attachments
                                          70                         Report 2003-P-00003

-------
                                                                           Attachment 1

                               GENERAL COMMENTS

Recognizing Institutional Mechanisms

The report does an excellent job of recognizing Agency institutional mechanisms which ensure
that regulations are based on sound science. The role of peer review and the peer review process
in the development of credible science is discussed in depth, and the Office of Policy, Economics
and Innovation (OPEI) agrees with the heavy emphasis the report places on the utility and
importance of independent peer review. Not emphasized are two other key "good science"
processes: Analytic Blueprints and the Risk Characterization Policy. The former was designed,
in part, to ensure that critical science needs are identified early in the process and developed in
time to inform regulatory decisions, and the latter requires that both the risk assessment process
and risk the analyses are transparent, clear, reasonable and consistent.  Taken together, these
three existing mechanisms can assure that:

12.     Critical science is identified early, and developed in time to inform decisions (Analytic
       Blueprint),

13.     Critical science is of sufficient quality for regulatory decision making (Peer Review
       Process),

14:     The quality of the science and the associated uncertainty is clearly described (Risk
       Characterization Policy).

Further, these three mechanisms  appear to directly address three of the four findings of your
report, i.e., that critical science supporting the rules often was not independently peer reviewed.
that more data and fewer "blind spots" could reduce assumptions, and that the role of science
(was) not made clear. Your report "determined that the oversight of peer review of the critical
science documents to support the pilot rules was limited and ineffective."  Applying the same
logic suggests that shortfalls in identifying critical data needs, and the lack of transparency and
clarity in science is due to inefficiencies or limitations in the two Agency processes intended to
identify, develop and make critical science transparent.

Definingjhe Scope  of the Science

The report refers to "science" quite broadly without ever offering a clear definition of what
specifically was considered. The report would benefit from a clearer discussion of what
categories of science were addressed and on which categories primary contacts were asked to
comment.  Page 5 of the report simply states: "We asked that [the contacts] consider several
categories of science that may be relevant to the rules..." but does not disclose what these
categories were. Presumably,  such categories included risk assessment, exposure modeling, and
                                           71                         Report 2003-P-00003

-------
epidemiology.  It is not clear from the text, however, whether engineering or any social sciences
were included in this discussion.

The extent to which economics and other social sciences were addressed in this report is an
important point to make especially in light of the renewed importance the Administrator and the
General Accounting Office have placed on the conduct of quality economic analyses in the
Agency.  While the review of economic analyses may be beyond the scope of this particular
report, the importance of this type of analysis as an input into the decision-making process should
not be overlooked!

Characterizing the Role of Economics in the Decision Making Process

The report mis-characterizes the role of economics in the decision-making process in some
places. The final point under suggestion number one lumps economics, a social science, together
with non-scientific aspects of rule making such as law and administrative discretion. This should
not be the case. Economics should be separated from other non-scientific considerations. Under
some statutes, such as the  Safe Drinking Water Act (amended 1996) findings from economic
science may be the basis for the standard, and therefore merit separate, detailed treatment
analogous to that given to  other science. Also, under the Federal Insecticide, Fungicide and
Rodenticide Act, the Agency must balance the risk posed by the use of pesticides against the
economic impacts on crop production of restricting pesticide use. Even in cases where a
standard may not be driven by economic findings, data on benefits, costs and impacts should be
presented with the same clarity and detail as "scientific findings."

Use of Risk Characterization Policy as a Framework  for Presenting Results and
Suggestions

Some of the science supporting rulemaking deals with health and environmental risks. EPA
adopted its policy on  "Risk Characterization" in February 1992, via a memorandum from Henry
Habicht, Deputy Administrator, and an accompanying document, prepared by a cross-office work
group. The policy was reiterated and elaborated in the mid 1990s. At its core, the policy states
that significant risk assessments should:

•      Describe how the estimated risk is expected to vary across population groups, geographic
       areas, or other relevant break-outs,

•      Describe the sources of uncertainties in the risk estimates, and quantify them, to the
       extent possible, and

•      Explicitly identify the impact of science and data, as opposed to policy choices, as the
       source of various elements of the risk assessment.
                                           72                        Report 2003-P-00003

-------
We have found that this standard has been followed in an incomplete fashion in documents
supporting regulations, as well as other EPA risk assessments. The draft Office of the Inspector
General (OIG) report refers repeatedly to the second and third elements of EPA's Risk
Characterization Policy, both in describing its findings and in its recommendations.  We
recommend that OIG examine this policy (in effect during most of the time period covered by
the pilot study), and use it as a framework for presenting its results and suggestions.

A Call for the Development of "Principals of Analytic Integrity"

Recently, the Administrator reaffirmed the "Principals of Scientific Integrity" establishing clear
and ethical standards that should govern the conduct of scientific studies within the Agency.  To
date, there is no parallel document establishing standards for the use of research in a policy
analytic setting. OIG may wish to recommend that such a document be developed expanding on
its recommendations for clarity of presentation, etc. and drawing on other Agency guidelines
such as The Guidelines for Preparing Economic Analyses.

Characterization of the RAPIDS Data  Base and its Capabilities

RAPIDS tracks all substantive rulemakings appearing in the Semi-Annual Regulatory Agenda as
well as a number of actions not in the Agenda, such as Reports to  Congress, Policy Plans, etc.
RAPIDS does not track every non-substantive rulemaking (SIPs, SNURs, Fffs, State Approvals,
etc.), but a sister database to RAPIDS (Federal Register Tracking Database - FR Dailies), also
maintained by OPEFs Regulatory Management Staff (RMS), tracks every EPA action sent to and
published in the Federal Register. These rules are not economically significant or normally
reviewed by OMB and therefore are classified as "not significant."

RAPIDS records go back a number of years (1996 forward) with some rulemaking records from
earlier years available. RAPIDS also tracks NPRMs published in many of those same years. The
Regulatory Management Staff (RMS) has built numerous views in RAPIDS and has a view (list)
of rules finalized each year.

The report seems to confuse or not clearly differentiate between "significant" rulemakings (those
OMB reviews) and "economically significant rulemakings" (economic impact of greater than
$100 million per year).  RAPIDS separates out those rules identified as "economically
significant." This designation has only been in effect for rules in the Semi-annual Regulatory
Agenda as Priority "A" (Economically Significant) since 1995.  Although for years before 1995,
it is more difficult to clearly identify economically significant rules, RAPIDS identifies 50 final
rules as economically significant for the years 1994-2001 and can produce lists of economically
significant rules published final for the years 1990 to the present.

For additional information regarding the  capabilities and content of the RAPIDS database or its
sister database, OIG staff may wish to contact RMS staff directly (Darryl Adams is the contact
for RAPIDS).
                                          73                         Report 2003-P-00003

-------
                                                                           Attachment 2

                          REACTIONS TO SUGGESTIONS

1)     Consider presenting the scientific findings that support a rule in specific sections of
       the preambles.  These findings should be organized according to the norms of
       science...

This suggestion is consistent with the Agency's efforts related to the use of and dissemination of
information covered by the new Information Quality Guidelines (IQ Guidelines). These
Guidelines have been developed in response to Section 515 of the Treasury and General
Government Appropriations Act for FY 2001 (often referred to as the Data Quality Act) that
directs the Office of Management and Budget (OMB) to issue guidelines that provide policy and
procedural guidance to Federal agencies and direct Federal agencies to:

•      adopt a basic standard of quality as a performance goal and take appropriate steps to
       incorporate information quality criteria into agency information dissemination practices;

•      issue guidance for ensuring and maximizing the quality,  objectivity, utility, and integrity
       of information disseminated by the agency; establish administrative mechanisms allowing
       affected persons to obtain  correction of information that  does not comply with the
       guidelines; and

*      submit an annual report, beginning January 1,2004, to OMB on the number and
       disposition of complaints received.

OMB published its guidelines  to Agencies on October 1,2001, and required agencies, including
EPA, to develop and publish their own information quality guidelines by October 1, 2002.  The
Office of Environmental Information has led development of the IQ Guidelines within EPA.
These Guidelines were  developed as a Tier  1 rulemaking, with broad participation across the
Agency and included briefings with the Deputy Administrator. EPA submitted its draft final IQ
Guidelines to OMB  on  August 1 and the new Guidelines become effective October 1, 2002. Per
OMB requirements,  EPA published draft guidance and received public comment on the draft
document.

The guidelines are consistent with many existing Agency practices and policies (e.g., EPA
considered its peer review and risk characterization policies while developing its guidelines).
However, the statutory  basis and underlying OMB guidelines provide additional emphasis on
EPA's implementation  of these information quality practices and policies. The complaint
resolution process will further intensify accountability for Agency staff, line managers, and
senior managers to ensure quality information products. Pre-dissemination review processes
need to be reviewed and there  is a plan to develop minimum standards for pre-dissemination.
review (product review) consistent with the Guidelines. OEI is developing a communication
strategy as well as the details and  procedures necessary to implement the complaint resolution

                                           74                        Report 2003-P-00003

-------
process, including appeals and the appeals panel. OEI is responsible for ensuring that sufficient
information is available to report complaint resolution in conformance with OMB deadlines.
Each program office and region are responsible for implementing the guidelines when they are
finalized and implemented on October 1, 2002.

OPEI believes that a full implementation of the IQ Guidelines will improve the Agency's
performance related to its discussion regarding the use of science in rulemakings.
This is also an area where  OPEI and ORD together can develop more complete recommendations
regarding the presentation  of scientific findings in preamble discussions. OPEI and ORD are
both increasing their presence in Agency rulemakings as a result of last year's Task Force on
Improving Regulation Development. OPEI believes that this increased participation by ORD and
OPEI analysts will improve the attention to and discussion of the results of the underlying
analysis, including but not limited to science, used to support EPA regulations could be
improved. This discussion would be consistent with the IQ Guidelines, existing policies such as
the risk characterization policy, and some of the key findings of your report.

2)     Focus more attention in the development phase of regulations on collecting data and
       doing research to  close "blind spots" to support rulemakings.

The purpose of an analytic blueprint is to identify research needs and guide data collection and
research studies during the development phase of regulations. While a requirement for
developing, updating, and  following an "analytic blueprint" has been a formal part of EPA's rule-
making process  for more than a decade, it has been OPEI's experience that most  analytic
blueprints are treated as little more than formalities. As a result of last year's review and
reassessment of EPA's rule-making process, OPEI  and the program offices are taking steps to
make the blueprints more central and relevant to actual rale-making decisions. We suggest that
the OIG report consider referring to the analytic blueprints as one means to achieve the results
desired in Suggestion 2.

3)     Take advantage of EPA's information technology...

OPEI is currently evaluating and enhancing RAPIDS in order to improve the management
information that is available or potentially obtainable.  To date, RAPIDS has focused on tracking
regulation development progress and facilitating EPA  's submission of its portion of the Semi-
Annual Regulatory Agenda to OMB. OPEI is interested in adding features that enhance
management accountability and improved performance metrics. RAPIDS currently links to
relevant guidance and policy documents. OPEI will continue to improve RAPIDS  and seek to
take advantage of other information technology capabilities over the next year. Much of this
work will be coordinated through the Regulatory Steering Committee or Regulatory Policy
Council. We will follow up with you over the next  several months to more fully understand these
recommendations and identify what specific changes or opportunities we can adopt.
                                          75                        Report 2003-P-00003

-------
4)     Reinforce EPA's current peer review policy ensuring that all EPA-generated
       documents critical to significant and substantive rulemakings are independently
       peer reviewed, and that the responses to the significant comments appear in the
       documents.

OPEI fully supports this recommendation on peer-review of critical documents and in fact has
recently extended this peer-review policy to include economic analyses.  OPEI is working
closely with the Agency's Program Offices to ensure that a full review of supporting economic
analyses for all economically significant rules occurs prior to the rule's submission to OMB. In
this way, the application of sound and consistent economic practices is ensured and the Agency's
position on the use of sound science strengthened.
                                          76                        Report 2003-P-00003

-------
                                                                           Attachment 3

                              DETAILED COMMENTS

Pace 1 ("What We Did and Whv")

It is unfortunate that economic analysis was not included in the pilot study, especially given the
importance the Administrator has recently placed on the role of economics in rule making. It
may be worth mentioning that although economics is important to the rule making process, you
chose to focus on chemistry, biology, health sciences, etc. for the purposes of the pilot study.

In the last paragraph, consider elaborating a bit more  regarding what was presented to the
Research Strategies Advisory Committee and the Regulatory Steering Committee as well as the
comments you received from them.

Page 2

In the first full paragraph, more exposition would be useful for those not familiar with the
Government's Auditing standards.  Do the standards  recommend the testing of controls?

A short summary of the methodology employed would also be helpful for the reader if provided
early in the report.  We recommend the following:

       "The pilot study was conducted in steps.  Once we learned more about the rule-making
       process, we began our pilot study by identifying all significant rules that were eligible for
       the study and then selected a small sample to pursue in case studies. For each  selected
       rule, we identified primary contacts involved  in the rule making and contacted each
       individual via email for assistance in identifying the critical science documents.  We then
       attempted to locate each primary and secondary science document underlying the rule-
       making process for each selected rule.  For each located document, we established who
       conducted the study, how the study was funded, and the level of peer-review the study
       received.  Each step is discussed in more detail below and the findings for each selected
       rule are summarized in the case studies located in the appendix."

In the second paragraph consider changing the wording so the sentence reads:

       "The remaining notices are for rules that primarily impact individual States, Tribes, or
       sites...."'

Page 3 ("We Identified Significant Rules Since 1990 and Selected 15 Case Studies")

For the title consider inserting "as" so that the title reads "We identified significant rules since
1990 and selected 15 as case studies"
                                          77                         Report 2003-P-00003

-------
The second paragraph could be made clearer by making the following changes (shown in italics):

       "Therefore, we used the Federal Register and EPA website to identify 88 significant rules
       finalized in 1990 through 2001. They are listed in Exhibit 1.  The list of rule promulgated
       before 1994 may be incomplete since EPA's web-based materials tended to be dated 1994
       and later. Focusing on the 74 rules finalized from 1994 on, we show in Figure 1 that
       more than half (38) of these rules were issued under the Clean Air Act...."

Page 4 ("We Sought to Identify the Critical Science Behind Each Rule")

Footnote 2: The composition of the pilot team is an important factor in how the study was
conducted and by whom.  This footnote should perhaps be moved to the text.

Page 5

Top of page: Is it the case that all primary contacts for the 15 rules responded?  Were all primary
contacts still at EPA? Presumably the 83 contacts mentioned at the end of the paragraph were
those identified by the primary contacts.  Are the primary contacts for each rule included in the
83?  Did the pilot study team identify the contacts or did the primary contacts identify the
contacts, or were the 83 contacts all primary contacts identified by the pilot study team? What
proportion of the 83 contacts were at EPA?

Second paragraph: In the email sent to contacts, was a description of the project included in the
email? Was any official endorsement of the study by a manager included to help gain
cooperation?

Third paragraph  (first full paragraph): In this paragraph you mention that follow-up contacts were
made with those individuals who had not responded. What kinds of follow-ups were made?
Were the follow-ups by email? Telephone? Did the follow-ups yield any additional
cooperation? What was the breakdown of the 83 contacts by role (EPA execs, peer reviewers,
etc.)? When stating, "We then turned to a combination of interviews and reading materials in  the
dockets," who was interviewed? Presumably, the report refers to those individuals who
responded to the initial  inquiries. Was a standard set of questions asked in each interview? How
many attempts were made to contact stakeholders and peer-reviewers?  Was accurate contact
information available for each individual or was it the case that the contact information had
changed and the person could not be found?

Page 6

First paragraph: The example of "think of laying a brick wall" was not as helpful as it could be
and  does not match the  situation entirely. Perhaps if the example were reworded to read "think
of abrick wall comprised of many individual pieces...."
                                           78                        Report 2003-P-00003

-------
Second paragraph: The second paragraph mentions an advisory from the Research Strategies
Advisory committee.  Does this refer to the recommendations received from the committee as a
result of the briefing they received about the study?  The suggestions they made should be
summarized earlier in the report as noted above.

The first sentence is not as clear as it could be.  Consider rewording it so that it reads:
"Identifying the critical science inputs to the various rules proved to be a much more difficult
task than expected given that the pilot study team members carrying out the identification process
were not involved in the original rulemaking. As a result, for some rules, the data are likely to be
incomplete. We encountered a particular problem as we traced...."

Readers of this paragraph may get get the health criteria documents and the critical documents
confused. Are the underlying studies primary or secondary critical documents?  According to the
text box, it appears they should be secondary. A few wording changes might clarify this.
Consider making the following changes:

             "These underlying studies then become critical secondary documents (there are
             usually more than one per health criteria document)".

Rewording the next few sentences may also help clarify the discussion here (changes noted in
italics):

       "Thirty-three of these underlying criteria documents were identified for Cases 2,  9 and
       11 alone. Because the  number of critical secondary documents increases exponentially as
       one goes backward through the citation  chain, locating these documents becomes a very
       time-consuming process"

Page 7 ("We Identified the Sources of the Critical Science")

First full paragraph: Add comma and remove "or" so that the sentence reads, "Who performed
the research was often identified on the title page of reports, the by-lines in journal articles, or in
the acknowledgements...."

Second paragraph:  Again, the report should establish who the respondents are. Are these the 7
people who responded to the initial inquiry?

Third paragraph: ("We asked the respondents about science gaps and science quality")
Last sentence: the rating scale  should be defined (what does one mean? What does five mean?)

Last paragraph: ("We identified the type of peer review undergone by the critical science")
This section may read better with some reordering:  cut the first paragraph in this section and
move the OMB quote and the sentence preceding it so that they follow the now second to last
paragraph in this section (before the paragraph  starting "Some documents indicated...").
                                           79                        Report 2003-P-00003

-------
Page 9 ("Science Played a Critical Role in the Rules, but That Role May Not Be Clear")

Title: Consider changing the title wording so that it reads " .. .but that role was not always clear"
The wording change should be carried over to the last sentence in the first paragraph.

Page 10

Second bullet: "We also identified critical science documents without which it is reasonable to
believe the section authorizing the rule might not have been included in the Clean Air Act?"
This seems to mean that you reviewed Congress' use of science, which is interesting, but seems
beyond the scope of the report.  Did you undertake similar efforts to review the science
underlying other relevant statutes passed during the study period?

Page 11 ("Role of Science Not Made Clear")

Second full paragraph: Consider changing "science" to "scientific" so that the sentence reads
"However, we saw little evidence of any of these conventions in communicating the scientific
underpinnings of the rules in the preambles..."

Last sentence of the second full paragraph: consider adding "described below" so that the
sentence reads "Two of the preambles described below provided examples of food practices in
the presentation of the data."

Page 12

The last paragraph on page 12 says that "only six [preambles] were well-referenced to the science
underpinnings."  Five bullets follow, and most of them illustrate inadequate referencing, although
the second bullet seems to offer no criticism of the preamble of Case 14. This list is confusing,  .
and contrasts to the two illustrative bullets on pages 11 and 12, which unambiguously detail
examples of "good practices in the presentation of data".

Page 15  ("Who Funded the Critical Work")

 "Intrastate" should be "interstate"

Table  3.  It is not clear from the text how the count for ORD critical science documents came to
75. The text states that ORD funded 74 of the  secondary documents and that EPA program
offices funded all but one of the primary documents.  Did ORD fund a primary document? If so,
this should be stated clearly in the text.  Also, it would be useful to break the  "Other" category
into its constituent parts (primarily State governments and industry, according to the text).

The text states on page 9 that 436 critical scientific studies were identified and classified in
various ways, in Tables 2, 3,4, and 5. Although footnote 5 on page 14 notes  that some
                                           80                        Report 2003-P-00003

-------
categories can be counted more than once (so the totals for Tables 2, 3, and 4 are greater than
436), it would be clearer if each of those three tables contained a note to that effect.

Page 17 ("More Data and Fewer "Blind Spots" Could Reduce Assumptions")

The finding that more data and fewer 'blind spots' could reduce assumptions seems reasonable,
but the text does not say that a great number of conservative assumptions were made where there
were "blind spots." Was this, in fact, the case?  It seems the text should support the finding more
clearly.

Consider refraining from referring to the rules as "pilot rules."  The rules themselves have been
finalized and are therefore not "pilot."  Perhaps you could refer to them as "the rules included in
this pilot study" or  "the selected rules" or simply "the rules."

Second full paragraph, first sentence: Consider changing the wording order so that the sentence
reads "Based on the responses, we concluded that having more data would have resulted in even
more efficient rules, because...."

Second full paragraph, second sentence: Remove the extra period at the end of the sentence.

Last paragraph ("Critical science supporting the rules was not always independently peer
reviewed"): Be consistent on the relative merits of peer review and public comment. This
paragraph states that public comment does not substitute for peer review, which seems sound and
reasonable. However, page A-75 of the appendix provides an extended editorial on the relative
merits of peer review  and public comment that suggests otherwise. The text in the appendix
even seems to question the validity of Agency peer review policy through the use of scare quotes:

       ...This rulemaking process must be substantially more rigorous than  the Agency's "peer
       review" process-

Last paragraph, last sentence: Consider giving an indication of how many total staff members
were in the sample. Add wording so that the sentence reads "Nonetheless, we were told by six of
the X EPA staff members..." where X = total number of EPA staff members with which you
were in contact.

Page 18

First sentence: Consider changing the wording so that sentence reads "A large number (290) of
the critical documents supporting the rules either were not peer reviewed (138)  or their peer-
review status was indeterminate (152)."  The "large number of the critical documents supporting
the pilot rules - 2,940" should apparently be the number "290."

Table 5: Consider changing last "Action" entry so that it reads, depending on the  intended
meaning, either "Independent internal EPA review done through the risk assessment forum, ORD

                                           81                         Report 2003-P-00003

-------
or a program office" or "Independent internal EPA review done of a program office document,
such as through the risk assessment forum or ORD."

Page 21 ("Pilot Lessons Learned")

Third bullet: The meaning in this bullet is not entirely clear. Consider rewording as follows:

       "A clear determination of a rule's relevant history should be made prior to the
       commencement of future studies, where "relevant history" is defined as the length of time
       preceding a rule's finalization during which review team members can be confident that
       identified science documents meeting the requirements of "critical documents", can in
       fact, be defined as such.  Any future studies of this sort should plan to conduct reviews
       over the rule's relevant history.

Fifth bullet: Consider rewording as follows:

       "Interviews should be conducted with as many people connected to the rulemaking as
       possible. Special effort should be made to interview peer-reviewers and stakeholders.
       Email is not an effective mechanism to elicit this kind of information."

Seventh bullet: The bullet should lead with the recommendation.  Consider rewording as follows:

       "There should be at least one research scientist on the team in spite of the fact that a
       science background is not necessary to identify critical science documents. A science
       background increases the efficiency of the identification process."

Exhibits

The  Exhibits would be more informative if they were reorganized. Specifically, Exhibit 1  should
highlight the rules that are part of the study sample.  For the other exhibits, so long as the name
and case number accompanies each study there is no need to present each list in chronological
order.  Exhibits 2-6 can be reordered to highlight completeness, aggregate numbers or other
interesting  findings along the dimensions displayed.

Appendices

Confirm coding on "who performed" the work.  Are EPA contracted reports 'private sector' or
'program office'? The body of the report suggests that EPA-contracted reports are performed by
the 'private sector,' but at least one the appendix indicates otherwise. See specifically A-45,
reference #2. This may be an isolated error;  each reference was not reviewed in detail.

Confirm coding for peer review: on pages A-78 and A-79, the peer review status of several
journal articles are coded as "unknown" although they appear to be from peer-reviewed journals
(including Fundamental and Applied Toxicology - now published as Toxicological Sciences:

                                           82                         Report 2003-P-00003

-------
Epidemiology, which is peer-reviewed; Toxicologic Pathology, which is peer-reviewed; and
others). These are simply examples. Each reference was not reviewed in detail.

There are a number of typographical errors and minor editorial errors that need to be addressed,
particularly in the appendices.
                                           83                        Report 2003-P-00003

-------
                                                                       Exhibit 11
	Report  Distribution	

Headquarters Officials
Associate Administrator for Policy, Economics, and Innovation (1804A) (paper copy)
Agency Followup Official (2710A) (paper copy)
Agency Audit Followup Coordinator (2124A) (paper copy)
Associate Administrator for Congressional and Intergovernmental Relations (1301 A)
Associate Administrator for Communications, Education, and Media Relations (1701 A)
Inspector General (2410)

Audit Liaisons
Pat Gilchriest, OA (1104A)
Pam Stirling, OPEI (1805T)
Peter Cosier, OAR (6102A)
Tom Coda, OAQPS (C40402)
Pat Keitt, OPPTS (7101M)
Cheryl Varkalis, ORD (8102R)
Johnsie Webster, OSWER (5103T)
Judy Hecht, OW (4101M)
Howard Levin, Region 5 (MF-10J)

Primary Contacts
Synthetic Chemicals Monitoring: Al Havinga OGWDW
Acid Rain Permits:  Larry Kertcher, OAP/OAR
Municipal Waste Combustors:  Walter Stevenson,  ESD
Land Disposal Restrictions: Rhonda Minnick, OSWER
Reformulated Gasoline: Paul Machiele, OTAQ
Great Lakes Water Quality: Mark Morris, OW
Polychlorinated Biphenyl Disposal: Tony Baney, OPPTS
Nonroad Diesel Engines: Karl Simon, OTAQ
Biotechnology: Elizabeth Milewski, OPPTS
Plant-Incorporated Protectants: Elizabeth Milewski, OPPTS
Disinfectants and Byproducts:  Tom Grubbs, OGWDW
Regional Ozone: Tom Helms,  OAQPS/OAR
Pulp and Paper (Air): Stephen Shedd, ESD/OAR
Pulp and Paper (Water): Donald Anderson, OW
Municipal Solid Waste Landfills:  Martha Smith, ESD/OAR
Note: Electronic distribution unless otherwise indicated.
                                        84                       Report 2003-P-00003

-------