^ฃDSX
ฎ J
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INvSPECTOR GENERAL
Catalyst for Improving the Environment
Procedural Review of EPA's
Greenhouse Gases Endangerment
Finding Data Quality Processes
Report No. 11-P-07Q2
September 26, 2011
-------
Report Contributors:
Rick Beusse
Bao Chuong
James Hatfield
Andrew Lavenburg
Abbreviations
ANPR
Advance notice of proposed rulemaking
AR4
Fourth Assessment Report (IPCC)
CAA
Clean Air Act
CCSP
U.S. Climate Change Science Program
CRU
Climatic Research Unit (University of East Anglia)
DQA
Data Quality Act
EO
Executive order
EPA
U.S. Environmental Protection Agency
IPCC
Intergovernmental Panel on Climate Change
NAS
National Academy of Sciences
NASA
National Aeronautics and Space Administration
NCDC
National Climatic Data Center
NO A A
National Oceanic and Atmospheric Administration
NRC
National Research Council
OAR
Office of Air and Radiation
OIG
Office of Inspector General
OMB
Office of Management and Budget
TSD
Technical support document
USGCRP
U.S. Global Change Research Program
Cover photo: An image of the Earth, (digital image created by NASA)
Hotline
To report fraud, waste, or abuse, contact us through one of the following methods:
e-mail: OIG Hotline@epa.gov write: EPA Inspector General Hotline
phone: 1-888-546-8740 1200 Pennsylvania Avenue NW
fax: 703-347-8330 Mailcode 8431P (Room N-4330)
online: http://www.epa.qov/oiq/hotline.htm Washington, DC 20460
-------
^ฃ0 STA>.
151. U.S. Environmental Protection Agency 11-P-0702
Office of Inspector General September 26, 2011
PRO^
At a Glance
Why We Did This Review
The Ranking Member, Senate
Committee on Environment
and Public Works, requested
that we determine whether the
U.S. Environmental
Protection Agency (EPA)
followed key federal and
Agency regulations and
policies in developing and
reviewing the technical data
used to make and support its
greenhouse gases
endangerment finding.
Background
On December 15, 2009, EPA
published its Endangerment
and Cause or Contribute
Findings for Greenhouse
Gases Under Section 202(a)
of the Clean Air Act. As the
primary scientific basis for
EPA's finding, the Agency
relied upon assessments
conducted by other
organizations. EPA
summarized the results of
these and other scientific
assessments in a technical
support document (TSD).
For further information,
contact our Office of
Congressional and Public
Affairs at (202) 566-2391.
The full report is at:
www.epa.aov/oia/reports/2011/
20110926-11 -P-0702.pdf
Catalyst for Improving the Environment
Procedural Review of EPA's Greenhouse Gases
Endangerment Finding Data Quality Processes
What We Found
EPA met statutory requirements for rulemaking and generally followed
requirements and guidance related to ensuring the quality of the supporting
technical information. Whether EPA's review of its endangerment finding TSD
met Office of Management and Budget (OMB) requirements for peer review
depends on whether the TSD is considered a highly influential scientific
assessment. In our opinion, the TSD was a highly influential scientific assessment
because EPA weighed the strength of the available science by its choices of
information, data, studies, and conclusions included in and excluded from the TSD.
EPA officials told us they did not consider the TSD a highly influential scientific
assessment. EPA noted that the TSD consisted only of science that was previously
peer reviewed, and that these reviews were deemed adequate under the Agency's
policy. EPA had the TSD reviewed by a panel of 12 federal climate change
scientists. This review did not meet all OMB requirements for peer review of a
highly influential scientific assessment primarily because the review results and
EPA's response were not publicly reported, and because 1 of the 12 reviewers was
an EPA employee.
EPA's guidance for assessing data generated by other organizations does not
include procedures for conducting such assessments or require EPA to document
its assessment. EPA provided statements in its final findings notice and supporting
TSD that generally addressed the Agency's assessment factors for evaluating
scientific and technical information, and explained its rationale for accepting other
organizations' data. However, no supporting documentation was available to show
what analyses the Agency conducted prior to disseminating the information.
Our evaluation examined the data quality procedures EPA used in developing the
endangerment finding. We did not assess whether the scientific information and
data supported the endangerment finding.
What We Recommend
We recommend that EPA (1) revise its Peer Review Handbook to accurately reflect
OMB requirements for peer review of highly influential scientific assessments,
(2) instruct program offices to state in proposed and final rules whether the action
is supported by influential scientific information or a highly influential scientific
assessment, and (3) revise its assessment factors guidance to establish minimum
review and documentation requirements for assessing and accepting data from
other organizations. EPA stated that its response to the final report will address our
recommendations.
-------
o^EDSrX
sS?
ฃ \ UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
1^7 S WASHINGTON, D.C. 20460
\po/
THE INSPECTOR GENERAL
September 26, 2011
MEMORANDUM
SUBJECT:
FROM:
Procedural Review of EPA's Greenhouse Gases
Endangerment Finding Data Quality Processes
Report No. ll-P-0702
Arthur A. Elkins, Jr.
Inspector General
TO:
Gina McCarthy
Assistant Administrator for Air and Radiation
Paul Anastas
Assistant Administrator for Research and Development
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends. This report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
Final determinations on matters in this report will be made by EPA managers in accordance with
established audit resolution procedures.
The estimated direct labor and travel costs for this report are $297,385.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. You should include a corrective actions plan for agreed-upon
actions, including milestone dates. Your response will be posted on the OIG's public website,
along with our memorandum commenting on your response. Your response should be provided
as an Adobe PDF file that complies with the accessibility requirements of Section 508 of the
Rehabilitation Act of 1973, as amended. The final response should not contain data that you do
not want to be released to the public; if your response contains such data, you should identify the
-------
data for redaction or removal. We have no objections to the further release of this report to the
public. We will post this report to our website at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact Wade Najjum,
Assistant Inspector General for Program Evaluation, at (202) 566-0827 or
naiium.wade@epa.gov; or Rick Beusse at (919) 541-5747 or beusse.rick@epa.gov
-------
Procedural Review of EPA's Greenhouse Gases
Endangerment Finding Data Quality Processes
11-P-0702
Table of Contents
Chapters
1 Introduction 1
Purpose 1
Background 1
Scope and Methodology 11
2 EPA's TSD Peer Review Methodology Did Not Meet
OMB Requirements for Highly Influential Scientific Assessments 13
EPA's Endangerment Finding Required a High Standard of
Information Quality 14
EPA Employed Procedures to Ensure Data Quality and
Fulfilled the Basic Requirements for Federal Rulemaking and
Other Statutory and Executive Order Requirements 15
EPA's Review of Its TSD Did Not Meet All OMB Peer Review
Requirements for Highly Influential Scientific Assessments 15
OAR Did Not Certify Its Compliance With OMB's Peer Review
Bulletin and EPA's Peer Review Policy 19
OAR Did Not Follow Some Steps in the Action Development
Process 20
Conclusions 22
Recommendations 22
Agency Comments and OIG Evaluation 23
OMB Comments and OIG Evaluation 24
3 EPA Procedures for Assessing External Data Are Unclear 26
EPA Guidance for Assessing Externally Generated Data
Can Be Improved 26
Conclusions 29
Recommendation 29
Agency Comments and OIG Evaluation 29
Status of Recommendations and Potential Monetary Benefits 31
Appendices
A Answers to Specific Questions From the Ranking Member 32
B Timeline of Key Events Leading to Issuance of Endangerment Finding 41
C Details on Scope and Methodology 43
-continued-
-------
Procedural Review of EPA's Greenhouse Gases
Endangerment Finding Data Quality Processes
11-P-0702
D OIG Request to OMB for Clarification of the Definition of a
Highly Influential Scientific Assessment 46
E OMB Clarification of the Definition of a Highly Influential
Scientific Assessment 48
F OMB's Views on the Application of OMB's Peer Review Bulletin
and Information Quality Guidelines to EPA's TSD for the EPA
Endangerment Finding 50
G Agency Comments on Draft Report and OIG Evaluation of
Agency Comments 52
H OMB Comments on Draft Report and OIG Evaluation of
OMB Comments 87
I Distribution 92
-------
Chapter 1
Introduction
Purpose
On April 7, 2010, the Ranking Member, Senate Committee on Environment and
Public Works, requested that the Office of Inspector General (OIG) conduct a
review of the process the U.S. Environmental Protection Agency (EPA) used to
make and support its greenhouse gases endangerment finding. Based on this
request, our objective was to determine whether EPA followed key federal and
Agency regulations and policies in obtaining, developing, and reviewing the
technical data used to make and support its greenhouse gases endangerment
finding. The Senator also posed seven questions he wanted us to address in
fulfilling our objective. Answers to those seven questions are in appendix A.
Background
On December 15, 2009, EPA published its Endangerment and Cause or
Contribute Findings for Greenhouse Gases Under Section 202(a) of the Clean Air
Act (CAA)1 (hereafter referred to as the endangerment finding). Specifically, the
EPA Administrator determined that:
Six greenhouse gases taken in combination endanger both the public
health and the public welfare of current and future generations.
Greenhouse gases from new motor vehicles and new motor vehicle
engines contribute to the greenhouse gas air pollution that endangers
public health and welfare under CAA Section 202(a).
This action was the culmination of an October 1999 rulemaking petition that
asked EPA to regulate greenhouse gas emissions under Section 202(a) of the
CAA. EPA denied this petition in September 2003, and the petitioners appealed
this decision to the Court of Appeals for the District of Columbia Circuit,2 which
upheld EPA's denial. The petitioners then appealed to the Supreme Court of the
United States. On April 2, 2007, in Massachusetts v. EPA,3 the Supreme Court
found that greenhouse gases are air pollutants covered by the CAA. Further, the
Supreme Court ruled that the Administrator must determine whether emissions of
greenhouse gases from new motor vehicles cause or contribute to air pollution,
which may reasonably be anticipated to endanger public health or welfare, or
1 "Endangerment and Cause or Contribute Findings for Greenhouse Gases Under Section 202(a) of the Clean Air
Act; Final Rule," Federal Register / Vol. 74, No. 239 / Tuesday, December 15, 2009 / Rules and Regulations,
p. 66496.
2 415 F.3d 50 (D.C. Cir. 2005)
3 549 U.S. 497 (2007)
11-P-0702
1
-------
whether the science is too uncertain to make a reasoned decision. Appendix B
shows the timeline of key events leading to the issuance of the endangerment
finding.
Clean Air Act Section 202 Requirements for Endangerment Findings
Section 202(a)(1) of the CAA states:
The Administrator shall by regulation prescribe (and from time to
time revise) in accordance with the provisions of this section,
standards applicable to the emission of any air pollutant from any
class or classes of new motor vehicles or new motor vehicle
engines, which in his judgment cause, or contribute to, air
pollution which may reasonably be anticipated to endanger public
health or welfare. Such standards shall be applicable to such
vehicles and engines for their useful life (as determined under
subsection (d), relating to useful life of vehicles for purposes of
certification), whether such vehicles and engines are designed as
complete systems or incorporate devices to prevent or control such
pollution.
information Used to Support the Endangerment Finding
The primary scientific basis for EPA's finding that greenhouse gases endanger
public health and welfare were assessments conducted by (1) the U.S. Global
Change Research Program (USGCRP) (formerly known as the U.S. Climate
Change Science Program (CCSP), (2) the Intergovernmental Panel on Climate
Change (IPCC), and (3) the National Research Council (NRC). EPA summarized
the results of these and other assessments in a technical support document (TSD).
U.S. Global Change Research Program
USGCRP coordinates and integrates federal research on changes in the
global environment and their implications for society. Thirteen departments
and agencies, including EPA, participate in USGCRP, which was known as
the CCSP from 2002 through 2008. The program is steered by the
Subcommittee on Global Change Research under the Committee on
Environment and Natural Resources, overseen by the Executive Office of
the President and facilitated by an Integration and Coordination Office.
According to EPA's TSD, the scientific information produced by CCSP
(now USGCRP) that EPA "relied upon most heavily" included 16 synthesis
and assessment products issued from 2006 through 2009.
11-P-0702
2
-------
According to its Guidelines for Producing CCSP Synthesis and
Assessment Products, issued December 2004, CCSP products will be
considered federal government disseminations and must be prepared in
conformance with the provisions of the Data Quality Act (DQA).
Intergovernmental Panel on Climate Change
IPCC was established in 1988 by the World Meteorological Organization
and the United Nations Environment Programme as an effort by the United
Nations to provide the governments of the world with a clear scientific
view of what is happening to the world's climate. IPCC periodically issues
assessment reports on climate change. Its most recent assessment, the
Fourth Assessment Report (AR4) was issued in 2007 and, according to
EPA's TSD, was heavily relied upon by EPA for its endangerment finding.
The AR4 included three working group reports4 and a Synthesis Report.
According to IPCC's Procedures for the Preparation, Review, Acceptance,
Adoption, Approval and Publication of IPCC Reports, IPCC reports
undergo a three-stage review process: (1) expert review; (2) government/
expert review; and (3) government review of summaries, synthesis reports,
and overviews. According to IPCC's procedures, the first draft of a report
is subject to expert review, which should include:
Experts who have significant expertise and/or publications in
particular areas covered by the report
Experts nominated by governments as coordinating lead authors,
lead authors, contributing authors, or expert reviewers as included
in lists maintained by the IPCC Secretariat
Expert reviewers nominated by appropriate organizations
After the initial draft report is revised in response to expert review
comments, the revised draft is sent out for government expert review. For
the IPCC's AR4 Working Group I and II draft reports, the U.S. government
review was managed by USGCRP. According to the USGCRP
representative coordinating the review, the U.S. expert review included
soliciting comments through a Federal Register notice. Additionally, a
panel of government representatives from the 13 U.S. agencies and
departments that USGCRP comprises was convened to review, approve,
and consolidate the comments received in response to the Federal Register
notice. These comments were submitted to IPCC and made available to the
public.
For the final U.S. government review, the USGCRP coordinating official
told us that comments were solicited from all 13 USGCRP participating
4 Working Group I report, The Physical Science Basis; Working Group II report, Impacts, Adaption, and
Vulnerability; and Working Group III report, Mitigation of Climate Change.
11-P-0702
3
-------
agencies and departments. According to the USGCRP representative, the
IPCC summaries for policymakers for all three working group reports were
accepted by the U.S. government at an IPCC plenary session. EPA
participated in the U.S. delegation that approved of the summary for
policymakers for the Working Group II report, Impacts, Adaptation, and
Vulnerability.
National Research Council
NRC is part of the National Academies, which also comprise the National
Academy of Sciences (NAS), National Academy of Engineering, and
Institute of Medicine. These private, nonprofit institutions provide science,
technology, and health policy advice under a congressional charter.
According to EPA's TSD, EPA primarily relied upon four NRC reports in
developing its finding. These reports were issued from 2001 through 2008.
According to NRC:
Any National Academies report (including meeting
summaries, signed papers, letter reports, or other study
products) must be reviewed by a diverse group of experts
other than its authors before it may be released outside the
institution. This independent, rigorous review is a hallmark
that distinguishes the National Academies from many other
organizations offering scientific and technical advice on
issues of national importance.
EPA's Technical Support Document
EPA's TSD for its endangerment finding summarized the results and
conclusion of the aforementioned scientific assessments and studies. The
draft TSD was originally prepared in 2007 in conjunction with EPA's
efforts to establish a rule controlling greenhouse gas emissions from light-
duty vehicles and transportation fuels. This draft rule was sent to the Office
of Management and Budget (OMB) but never finalized. Subsequently, EPA
issued an advance notice of proposed rulemaking (ANPR)5 asking for
public comment on its plans to issue a stand-alone endangerment finding
for greenhouse gases. A revised TSD was made publicly available by EPA
to accompany the ANPR. On April 24, 2009, EPA published its proposed
endangerment finding,6 which noted that EPA had updated its TSD to
reflect the results of 11 recent CCSP reports, as well as more recent climate
data from federal agencies. The final endangerment finding was published
in the Federal Register on December 15, 2009.7 A final TSD accompanied
the finding. The TSD reflected changes to include the most recent
5 Federal Register / Vol. 73, No. 147 / Wednesday, July 30, 2008 / Proposed Rules, p. 44354.
6 Federal Register / Vol. 74, No. 78 / Friday, April 24, 2009 / Proposed Rules, p. 18886.
7 Federal Register / Vol. 74, No. 239 / Tuesday, December 15, 2009 / Rules and Regulations, p. 66496.
11-P-0702
4
-------
comprehensive assessment of USGCRP, up-to-date observational data for a
number of key climate variables from the National Oceanic and
Atmospheric Administration (NOAA), and up-to-date emissions data from
EPA's annual inventory of greenhouse gases. Additionally, EPA made
other edits and updates to the TSD in response to comments received on the
proposed rule.
In its endangerment finding, the Agency recognized that scientific research
on climate change issues was an ongoing process. Since EPA's
endangerment finding was issued in December 2009, NRC has published a
series of climate change reports. In its May 2010 report, Advancing the
Science of Climate Change, NRC concluded:
A strong, credible body of scientific evidence shows that
climate change is occurring, is caused largely by human
activities, and poses significant risks for a broad range of
human and natural systems.
In addition, IPCC has begun preparations for its Fifth Assessment Report on
climate change. In June 2010, IPCC selected authors and reviewers for its
Fifth Assessment Report.
The Data Quality Act and Key Requirements for Information Quality
Congress enacted the DQA in 2000. The DQA directed OMB to issue guidelines
to federal agencies for ensuring and maximizing the quality, objectivity, utility,
and integrity of information disseminated by federal agencies in fulfillment of the
purposes and provisions of the Paperwork Reduction Act. The OMB guidelines
were to require each federal agency subject to the guidelines to:
Issue guidelines ensuring and maximizing the quality, objectivity, utility,
and integrity of information, including statistical information,
disseminated by the agency
Establish administrative mechanisms allowing affected persons to seek
and obtain correction of information maintained and disseminated by the
agency that does not comply with the guidelines
Report periodically to the OMB Director on (i) the number and nature of
complaints received by the agency regarding the accuracy of information
disseminated by the agency, and (ii) how such complaints were handled by
the agency
11-P-0702
-------
Accordingly, OMB issued its final Guidelines for Ensuring and Maximizing the
Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal
Agencies in January 2002.8
OMB Final Information Quality Bulletin for Peer Review
To improve the quality of scientific information used by federal agencies, in
addition to its January 2002 guidelines OMB also developed a Final Information
Quality Bulletin for Peer Review. This bulletin was published in the Federal
Register9 in January 2005, and established government-wide guidance for
enhancing the practice of peer review of government science documents. The
bulletin defined two types of information requiring peer review, "influential
scientific information" and "highly influential scientific assessments."
Influential Scientific Information
The OMB bulletin requires each agency to subject influential scientific
information to peer review prior to dissemination. The guidelines provide
agencies broad discretion in determining what type of peer review is
appropriate and what procedures should be employed to select appropriate
reviewers. Agencies do not have to subject influential scientific
information to additional peer review if the information has already been
subjected to adequate peer review. In determining whether prior peer
review is adequate, OMB's bulletin states that agencies shall give due
consideration to the:
Novelty and complexity of the science to be reviewed
Importance of the information to decisionmaking
Extent of prior peer reviews
Expected benefits and costs of additional review
Principal findings, conclusions, and recommendations in official reports of
NAS are generally presumed to have been adequately peer reviewed.
Highly Influential Scientific Assessment
OMB requires a more rigorous form of peer review for influential
scientific information products that are considered highly influential
scientific assessments. OMB defines a scientific assessment as:
... an evaluation of a body of scientific or technical
knowledge which typically synthesizes multiple factual
inputs, data, models, assumptions, and/or applies best
8 A February 5, 2002, Federal Register notice corrected errors in the January 2002 version. OMB published the final
corrected guidelines in the Federal Register on February 22, 2002, Volume 67, No. 36, p. 8452.
9 Federal Register / Vol. 70, No. 10 / Friday January 14, 2005 / p. 2664.
11-P-0702
6
-------
professional judgment to bridge uncertainties in the
available information.
A highly influential scientific assessment is a scientific assessment that:
Could have a potential impact of more than $500 million in any
year on either the public or private sector, or
Is novel, controversial, or precedent setting, or has significant
interagency interest
For highly influential scientific assessments, OMB guidance requires more
attention to peer review consideration such as individual versus panel
review, timing, scope of the review, selection of reviewers, disclosure and
attribution, public participation, and disposition of reviewer comments. If
the material to be disseminated falls within OMB's definition of highly
influential scientific assessment, OMB requires the agency to adhere to the
peer review procedures identified in Section III of its bulletin.
OMB guidance also requires that agencies certify compliance with the
requirements of the bulletin and information quality guidelines when using
influential scientific information or highly influential scientific
assessments to support a regulatory action. This certification and other
relevant materials should be included in the administrative record for the
action.
EPA's Information Quality Guidelines
In response to OMB's data quality guidelines, EPA issued its Guidelines for
Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of
Information Disseminated by the Environmental Protection Agency (hereafter
referred to as EPA information quality guidelines) in October 2002. Consistent
with OMB's guidelines, EPA's information quality guidelines state that the
quality of information is determined by the objectivity, integrity, and utility of
that information. EPA's information quality guidelines cite various policies and
procedures that EPA uses to ensure and maximize the quality of the information
that it disseminates.
EPA's information quality guidelines outline examples of existing Agency
guidelines and policies in place to address quality, objectivity, utility, and
integrity of information. These examples include EPA's Peer Review and Peer
Involvement at the U.S. Environmental Protection Agency (hereafter referred to as
EPA's peer review policy) and the Action Development Process: Guidance for
EPA Staff on Developing Quality Actions (hereafter referred to as action
development process guidance). EPA's guidance for implementing its peer review
policy is contained in EPA's Peer Review Handbook. EPA's action development
process and guidance are outlined in the action development process guidance.
Earlier versions of these guidance documents existed before Congress passed the
11-P-0702
7
-------
DQA. A third key guidance document, A Summary of General Assessment
Factors for Evaluating the Quality of Scientific and Technical Information,
identifies the factors that EPA generally considers in evaluating the quality and
relevance of scientific and technical information from outside sources. According
to the document's introduction, the document does not constitute new data quality
considerations, but provides transparency and information about the Agency's
existing procedures. Key congressional, OMB, and Agency information quality
requirements and guidance are shown in figure 1.
Figure 1: Key data quality requirements and guidelines
Data Quality Act
(December 2000)
Guidelines for Ensuring and Maximizing the Quality,
Objectivity, Utility, and Integrity of Information
Disseminated by Federal Agencies
(January 2002, amended February 2002)
Final Information
Quality Bulletin for
Peer Review
(January 2005)
Guidelines for Ensuring and Maximizing the Quality, Objectivity,
Utility, and Integrity of Information Disseminated by the
Environmental Protection Agency (October 2002)
1
1
1
EPA Action Development
Process: Guidance for
EPA Staff on Developing
Quality Actions
(June 2004)
A Summary of General
Assessment Factors for
Evaluating the Quality of
Scientific and Technical
Information (June 2003)
EPA Peer Review
Policy and Peer
Review Handbook,
3rd Edition
(January 2006)
Source: OIG analysis of key federal and EPA information quality criteria.
Notes:
1 - EPA had established action development and peer review processes before the
DQA. EPA's October 2002 guidelines incorporate these processes as
part of the Agency's overall system for ensuring and maximizing data quality. The
guidelines noted that EPA would develop assessment factors for reviewing the
quality of information developed by external parties before EPA uses that information.
2 - EPA had a peer review process in place before the DQA. The 3rd edition
of EPA's handbook incorporated the requirements of OMB's bulletin.
Descriptions of these three key EPA guidance documents follow.
11-P-0702
8
-------
Action Development Process: Guidance for EPA Staff on
Developing Quality Actions
EPA's guidance for developing quality actions, the Action Development
Process: Guidance for EPA Staff on Developing Quality Actions, is
intended to ensure that the Agency uses quality information to support its
actions and ensure that scientific, economic, and policy issues are
adequately addressed at the right stages in action development. EPA
actions subject to the action development process include rules, policy
statements, risk assessments, guidance documents, models that may be
used in future rulemakings, reports to Congress that are statutorily
mandated, and strategies related to regulations.
The action development process guidance requires EPA program offices
to classify actions as Tier 1, Tier 2, or Tier 3. Tier 1 actions are top actions
that demand the ongoing involvement of the Administrator's office and
extensive cross-Agency involvement on the part of the Assistant and
Regional Administrators. Scientific or economic issues that are precedent
setting or controversial, or economically significant per Executive Order
(EO) 12866, should be considered Tier 1 actions. All Tier 1 and Tier 2
actions are expected to include an "analytic blueprint." An analytic
blueprint lays out a workgroup's plans for the data collection and analyses
that will support development of a specific action. The analytic blueprint
spells out how this information will be collected, peer reviewed, and used
to develop the action within a specific budget and timeframe.
A Summary of General Assessment Factors for Evaluating the
Quality of Scientific and Technical Information
EPA's general assessment factors guidance document describes EPA's
quality expectations for scientific and technical information, including
information that is voluntarily submitted by or collected from external
sources. It draws from existing Agency information quality systems,
practices, and guidelines to describe the types of considerations EPA takes
into account when evaluating the quality and relevance of scientific and
technical information. According to the guidance, EPA typically takes into
account the following five general assessment factors when evaluating the
quality and relevance of scientific and technical information:
Soundness: The extent to which the scientific and technical
procedures, measures, methods, or models employed to generate
the information are reasonable for, and consistent with, the
intended application.
Applicability and utility: The extent to which the information is
relevant for the Agency's intended use.
11-P-0702
9
-------
Clarity and completeness: The degree of clarity and
completeness with which the data, assumptions, methods, quality
assurance, sponsoring organizations, and analyses employed to
generate the information are documented.
Uncertainty and variability: The extent to which the variability
and uncertainty (quantitative and qualitative) in the information, or
in the procedures, measures, methods, or models, are evaluated and
characterized.
Evaluation and review: The extent of independent verification,
validation, and peer review of the information or of the procedures,
measures, methods, or models.
EPA drew from existing Agency quality systems, practices, and guidelines
in creating its general assessment factors. According to EPA's guidance,
the assessment factors did not constitute a new process for evaluating
information.
EPA's Peer Review Policy and Peer Review Handbook
EPA's peer review policy consists of the following provisions:
Peer review is encouraged and expected for all scientific and
technical information that is intended to inform or support Agency
decisions.
Influential scientific information, including highly influential
scientific assessments, should be peer reviewed in accordance with
the Agency's Peer Review Handbook.
External peer review is the expected procedure for highly
influential scientific assessments.
External peer review is the approach of choice for influential
scientific information intended to support important decisions, or
for work products that have special importance in their own right.
All Agency managers are accountable for ensuring that Agency
policy and guidance are appropriately applied in determining
whether their work products are influential or highly influential,
and for deciding the nature, scope, and timing of their peer review.
11-P-0702
10
-------
Peer review is not restricted to the nearly final version of work
products, as peer review at the planning stage can often be
extremely beneficial.
The Agency's Peer Review Handbook incorporates the provisions of
EPA's peer review policy and outlines the Agency's procedures and
processes for peer review. As stated in the handbook, EPA's underlying
principle is that "all influential scientific and technical work products used
in decision making will be peer reviewed."
After a work product is determined to be an influential scientific or
technical work product, the next key determination is whether the product
is a highly influential scientific assessment. Highly influential scientific
assessments are expected to undergo external peer review in accordance
with the procedures outlined in OMB's Final Information Quality Bulletin
for Peer Review. According to the handbook, any scientific or technical
work product used in Agency decisionmaking should be considered a
candidate for peer review, even if EPA did not develop the work product.
If these work products have been previously peer reviewed, EPA can
accept the peer review if it meets the intent of the Agency peer review
policy and the Agency's intended use of the product. The handbook states
that the appropriate EPA office:
. . . should examine closely the particulars of the peer
review to ensure independence and a conscious effort to
incorporate the peer reviewers' comments into the final
work product. If there are perceived, or real, conflicts of
interest, this may preclude the use of that peer review and,
in those instances, another peer review would be needed.
In response to the provisions of OMB's bulletin on peer review, the Peer
Review Handbook recommends placing language in the preamble of
proposed or final rules, ANPRs, or other substantive actions stating that
EPA conducted a peer review of influential scientific information or
highly influential scientific assessments in accordance with OMB's Final
Information Quality Bulletin For Peer Review. Further, EPA's action
memoranda from the program office to the EPA signer of the rule should
state that EPA followed the EPA peer review policy with respect to the
information supporting the action.
Scope and Methodology
We conducted our work at EPA's Office of Air and Radiation (OAR), Office of
Atmospheric Programs, in Washington, DC. To assess EPA's compliance with
key regulations and guidance governing information quality, we reviewed the
following documents:
11-P-0702
11
-------
Data Quality Act
OMB, Guidelines for Ensuring and Maximizing the Quality, Objectivity,
Utility, and Integrity of Information Disseminated by Federal Agencies
OMB, Final Information Quality Bulletin for Peer Review
EPA, Guidelines for Ensuring and Maximizing the Quality, Objectivity,
Utility, and Integrity of Information Disseminated by the Environmental
Protection Agency
EPA, Action Development Process: Guidance for EPA Staff on
Developing Quality Actions
EPA, Peer Review and Peer Involvement at the U.S. Environmental
Protection Agency
EPA, Peer Review Handbook
EPA, A Summary of General Assessment Factors for Evaluating the
Quality of Scientific and Technical Information
In addition to reviewing and analyzing key federal and Agency requirements for
information quality, we also reviewed EPA's compliance with federal regulations
and EOs governing the regulatory development process.
We conducted our field work from April 2010 to April 2011. We conducted this
evaluation in accordance with generally accepted government auditing standards.
Those standards require that we obtain sufficient, appropriate evidence to provide
a reasonable basis for our findings and conclusions based on our evaluation
objectives. We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our objectives. Appendix C provides a
detailed description of our scope and methodology.
Limitations
Our evaluation focused only on EPA's process for developing the endangerment
finding and ensuring information quality. We did not evaluate the effectiveness of
IPCC's or other organizations' information quality procedures. We did not test the
validity of the scientific or technical information used by EPA to support its
endangerment finding, nor did we evaluate the merit of the conclusions or
analyses presented in EPA's endangerment finding. We did not make conclusions
regarding the impact that EPA's information quality control systems may have
had on the scientific information used to support the endangerment finding.
11-P-0702
12
-------
Chapter 2
EPA's TSD Peer Review Methodology
Did Not Meet OMB Requirements for
Highly Influential Scientific Assessments
EPA fulfilled the statutory requirements for notice and comment rulemakings
mandated in the Administrative Procedure Act and in Section 307 of the CAA,
and employed several of its processes designed to ensure data quality. Whether
EPA followed all key federal and Agency requirements and policies in peer
reviewing its endangerment finding TSD depends upon how the document is
defined according to OMB's bulletin on peer review. We interpreted OMB's
guidance to indicate that the TSD was a highly influential scientific assessment.
EPA's peer review did not meet all OMB requirements for such documents. EPA
had the TSD reviewed by a panel of 12 federal climate change scientists.
However, the panel's findings and EPA's disposition of the findings were not
made available to the public as would be required for reviews of highly influential
scientific assessments. Also, this panel did not fully meet the independence
requirements for reviews of highly influential scientific assessments because one
of the panelists was an EPA employee. Further, in developing its endangerment
finding, we found that OAR did not:
Include language in its proposed action, final action, or internal
memoranda that identified whether the Agency used influential scientific
information or highly influential scientific assessments to support the
action. OAR also did not certify that the supporting technical information
was peer reviewed in accordance with EPA's peer review policy.
Prepare a complete analytic blueprint outlining its approach for reviewing
the technical data needed to support its action as recommended by the
Agency's action development process. OAR also did not follow some of
the procedural guidelines in EPA's action development process.
OAR officials told us that a peer review of the TSD in accordance with Section III
of OMB's bulletin on peer review was not required because they did not consider
the TSD a scientific assessment. They noted that the TSD consisted only of
science that was previously peer reviewed and that these reviews were deemed
adequate under the Agency's policy. They also stated that, as described in the
final Federal Register notice, the Administrator primarily relied upon assessments
conducted by other organizations rather than the TSD, which summarizes the
conclusions and findings of these other assessments. OMB's Associate
Administrator, Office of Information and Regulatory Affairs, in response to our
draft report, stated that OMB believes that EPA reasonably interpreted the OMB
bulletin in concluding that the TSD did not meet the bulletin's definition of a
11-P-0702
13
-------
highly influential scientific assessment. We did not analyze the effect that any of
these conditions had on the TSD's presentation or the Administrator's decision.
EPA's Endangerment Finding Required a High Standard of
Information Quality
Because of the influential nature of the Agency's endangerment finding and the
supporting technical information, federal and Agency guidelines called for EPA
to apply a rigorous standard of quality to the information disseminated as part of
its endangerment finding. EPA's information quality guidelines identify its action
development process, peer review procedures, and other related procedures as
processes of particular importance for ensuring the quality, objectivity, and
transparency of influential information.
EPA's action development process requires the program office developing an
action to designate its action by tier level. Tier 1 actions are top actions that
demand the ongoing involvement of the Administrator's office and extensive
cross-Agency involvement on the part of the EPA Assistant and Regional
Administrators. A key requirement for Tier 1 actions is the development of an
analytic blueprint. An analytic blueprint lays out the approach for collecting and
reviewing the data that support the action.
As outlined in OMB and EPA guidance, an important method for ensuring data
quality is the external peer review process. The nature of the technical
information supporting an action determines the level and extent of peer review.
Influential scientific information is expected to be peer reviewed. 0MB's Final
Information Quality Bulletin for Peer Review and EPA's Peer Review Handbook
describe criteria for information to be classified as influential scientific
information. EPA's criteria include information that:
Focuses on significant emerging issues
Has significant cross-Agency/interagency implications
Highly influential scientific assessments are a subset of influential scientific
information that must undergo specific minimum requirements for external peer
review. OMB's peer review bulletin requires external peer review of highly
influential scientific assessments unless an agency employs an acceptable
alternative procedure, as provided in Section IV of the bulletin, or the assessment
meets the exemptions outlined in Section IX of the bulletin. Examples of the
listed acceptable alternative procedures include relying on scientific information
produced by NAS, commissioning NAS to peer review the work product, and
employing procedures approved by the OMB Office of Information and
Regulatory Analysis Administrator in consultation with the Office of Science and
Technology Policy. Examples of the listed exemptions include information related
to national security, financial information, and information involving a health or
safety dissemination that is time sensitive.
11-P-0702
14
-------
EPA Employed Procedures to Ensure Data Quality and Fulfilled the
Basic Requirements for Federal Rulemaking and Other Statutory and
Executive Order Requirements
EPA's Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility,
and Integrity of Information Disseminated by the Environmental Protection
Agency describe many policies and procedures that EPA uses to ensure and
maximize the quality of data it disseminates. Examples of processes that EPA
used to help ensure and maximize the quality of information used to support its
endangerment finding included:
Holding public hearings after issuing the proposed rule.
Obtaining reviews of the draft TSDs by federal experts.
Providing a predissemination review for other federal agencies.
Addressing information concerns through the public comment process for
its proposed rule. The Agency developed 11 volumes of responses to
comments to address significant public comments received in response to
the proposed finding.
Providing additional information and analysis in response to 10 notice and
comment petitions for reconsideration received after its final rule was
issued.
Placing the action in the appropriate tier (Tier 1) and establishing a
workgroup to develop the rule.
Addressing comments received on the TSD from workgroup members,
federal experts, OMB, and other agencies.
EPA also fulfilled the statutory requirements for notice and comment rulemakings
mandated in the Administrative Procedure Act and in Section 307 of the CAA.
Additionally, EPA fulfilled the requirements in other statutes and EOs affecting
EPA rulemakings, where applicable. A detailed list of the statutes and EOs we
examined is in appendix C.
EPA's Review of Its TSD Did Not Meet All OMB Peer Review
Requirements for Highly Influential Scientific Assessments
OAR had the TSD reviewed by a panel of climate change scientists. This review
did not meet all of OMB's peer review requirements for highly influential
scientific assessments. The methodology that OAR employed for this review was
within the discretion afforded by OMB guidance for peer reviews of influential
scientific information, but not for highly influential scientific assessments. In our
11-P-0702
15
-------
opinion, the TSD is a highly influential scientific assessment and thus it required a
peer review as described in Section III of OMB's Final Information Quality
Bulletin for Peer Review. OAR officials explained that, in their view, a more
formal review of the TSD was not needed because:
They did not consider the TSD to be a scientific assessment because it
only summarized existing findings and conclusions and provided no new
findings or conclusions.
The core references relied upon for the TSD had been peer reviewed in a
manner consistent with OMB's bulletin. OAR pointed out that the
information had been reviewed and vetted by the scientific community
through the IPCC, USGCRP/CCSP, and NRC review procedures.
OMB's peer review bulletin distinguishes between influential scientific
information and highly influential scientific assessments, and affords federal
agencies more discretion for the type of peer review mechanism and reviewer
selection process they use for influential scientific information. Section II of the
bulletin describes the minimum standards for peer review of influential scientific
information. Section III of OMB's bulletin prescribes additional and more
prescriptive standards for peer review of highly influential scientific assessments,
a subset of influential scientific information. Thus, an important determination for
peer review purposes is whether supporting influential scientific information is
considered a highly influential scientific assessment. OMB's peer review bulletin
defines a scientific assessment as:
... an evaluation of a body of scientific or technical knowledge
that typically synthesizes multiple factual inputs, data, models,
assumptions, and/or applies best professional judgment to bridge
uncertainties in the available information.
We contacted OMB, as the originator and creator of the Guidelines for Ensuring
and Maximizing the Quality, Objectivity, Utility, and Integrity of Information
Disseminated by Federal Agencies and the Final Information Quality Bulletin for
Peer Review, to clarify whether a document summarizing existing findings and
conclusions of other peer-reviewed scientific assessments, but not offering any
new analyses or conclusions, would meet OMB's definition of a scientific
assessment (see appendix D). We provided OMB with a copy of our assignment
notification letter to EPA, the Ranking Member's letter requesting our review,
and a set of questions pertaining to the guidance in OMB's Final Information
Quality Bulletin on Peer Review. OMB's Assistant General Counsel provided the
following interpretation to us via electronic mail (appendix E contains OMB's
complete response):
11-P-0702
16
-------
An annotated bibliography would generally not be considered a
scientific assessment; however, a document summarizing the "state
of the science" would be, as it implicitly or explicitly weighs the
strength of the available evidence.
Based on the above interpretation, by providing a summary of existing findings
and conclusions from IPCC, USGCRP/CCSP, NRC, and other reports, OAR
implicitly and explicitly weighed the strength of the available science by its
choices of information, data, studies, and conclusions included in and excluded
from the TSD. Also, in our judgment the TSD synthesizes multiple factual inputs,
data, models, and assumptions. The Agency stated in its endangerment finding
that it "gave careful consideration to all the scientific and technical information in
the record." EPA's TSD referenced multiple sources (some cited within the
assessment reports, and some not), including "up-to-date" data from sources other
than the "major scientific assessments." In evaluating the scientific information,
the Agency stated that it "placed limited weight on the much smaller number of
individual studies that were not considered or reflected in the major assessments."
EPA reviewed such studies "largely to see if they would lead EPA to change or
place less weight on the judgments reflected in the assessment report." The
Agency stated in the endangerment finding that "the studies did not change the
various conclusions or judgments EPA would draw based on the assessment
reports." Thus, in our opinion, the TSD is a scientific assessment.
Further, the TSD meets OMB 's definition of a highly influential scientific
assessment because it is "novel, controversial, or precedent-setting or has
significant interagency interest." Statements in the Agency's proposed and final
notices for the endangerment finding also suggest that the TSD is a highly
influential scientific assessment. For example, EPA's proposed Federal Register
notice stated:
EPA has developed a technical support document (TSD) which
synthesizes major findings from the best available scientific
assessments that have gone through rigorous and transparent peer
review.
Further, in describing the science on which the finding was based, the final
Federal Register notice stated:
In 2007, EPA initiated its assessment of the science and other
technical information to use in addressing the endangerment and
cause or contribute issues before it under CAA section 202(a).
This scientific and technical information was developed in the
form of a TSD in 2007. An earlier draft of this document was
released as part of the ANPR published July 30, 2008
(73 FR 44353).
11-P-0702
17
-------
We asked OAR if it sought guidance from EPA's Science Policy Council on its
choice of peer review methodology for the TSD; OAR confirmed that it did not.
OAR pointed out that OMB had coordinated the interagency review process for
the TSD and had cleared the document as part of EPA's endangerment finding.
However, OAR never formally categorized the nature of the document; i.e., OAR
never stated whether it was influential scientific information or a highly
influential scientific assessment.
OAR had the TSD reviewed by 12 climate change experts. This panel was
charged with reviewing the TSD to make sure it was "a fair and accurate
reflection of the current state of climate change science as embodied in the major
assessments reports such as IPCC, USGCRP/ CCSP, and NRC." The panel
conducted three such reviews (the same 12 experts were used for the three
reviews) at different stages and provided its comments to EPA. EPA maintained a
record of all reviewer comments. EPA also maintained a record of its response
and disposition of the reviewers' comments to the initial draft TSD that
accompanied the 2007 Office of Transportation and Air Quality rulemaking. EPA
did not maintain a record of its response and disposition of comments for the two
TSDs that accompanied the proposed and final rules. Additionally, the panel's
results and EPA's response to the panel's results were not made available to the
public as is required for a peer review of a highly influential scientific assessment.
We also noted that this panel did not fully meet the independence requirements
for reviews of highly influential scientific assessments because one of the
panelists was an EPA employee. The OMB bulletin for peer review states that
"scientists employed by the sponsoring agency are not permitted to serve as
reviewers for highly influential scientific assessments." See appendix A,
question 5, for a more detailed discussion of the expert panel process.
We discussed our conclusions with EPA and provided EPA with a draft
discussion document outlining our preliminary observations. Since our conclusion
was based on our interpretation of OMB peer review guidance, including the
clarification OMB provided to us (see appendix E), EPA sought our permission to
discuss our initial conclusions with OMB. We agreed that EPA could discuss our
results with OMB. OMB's Deputy General Counsel subsequently provided us
with an electronic mail correspondence stating:
OMB believes that EPA reasonably determined that the
Endangerment TSD itself (as opposed to the underlying peer-
reviewed scientific assessments of the NRC, IPCC, USGRCP
identified and discussed in the TSD) did not have the impacts or
characteristics required to meet the OMB Bulletin's definition of a
highly influential scientific assessment.
OMB noted that its original interpretative statement to us was general in nature
and was not specific to the endangerment finding TSD. OMB also requested to be
11-P-0702
18
-------
allowed to comment on the formal draft report. Appendix F contains OMB 's
subsequent correspondence.
Regardless of the interpretations applied to the endangerment finding TSD, we
noted that a flowchart on page 2 of EPA's Peer Review Handbook indicates that
highly influential scientific assessments do not require an external peer review if
the assessment only consists of science previously peer reviewed and the peer
review is deemed adequate under the Agency's policy. While this interpretation is
correct for influential scientific information, OMB's peer review guidance does
not provide for such an exception for highly influential scientific assessments.
OAR Did Not Certify Its Compliance With OMB's Peer Review Bulletin
and EPA's Peer Review Policy
OAR did not certify its compliance with OMB's peer review bulletin and EPA's
peer review policy. Further, EPA did not identify whether the information
supporting its action was influential scientific information or a highly influential
scientific assessment. Properly identifying the information is a prerequisite to
determining the applicable peer review procedures per OMB's peer review
bulletin and EPA's peer review policy.
OMB's I until Information Quality Bulletin for Peer Review requires an agency
relying on influential scientific information or a highly influential scientific
assessment to support a regulatory action to include in the administrative record
for that action a certification explaining how the agency complied with the
requirements of the OMB peer review bulletin and the applicable information
quality guidelines. Appendix C of EPA's Peer Review Handbook provides
detailed guidelines regarding the language and mechanisms EPA should use to
certify its compliance with OMB and EPA peer review requirements. When using
either influential scientific information or a highly influential scientific
assessment as the underlying information to support an Agency action, EPA's
Peer Review Handbook states that the Agency should insert language into the
preamble of the proposed and final rules that:
1. States that the action was supported by influential scientific information or
a highly influential scientific assessment
2. Certifies that EPA conducted a peer review of the supporting information
in accordance with OMB's Final Information Quality Bulletin for Peer
Review
Additionally, EPA's Peer Review Handbook directs the Agency to include a
statement in its action memorandum that the Agency followed its peer review
policy with respect to the influential scientific information or highly influential
scientific assessments supporting the action. If the policy was not followed, the
action memorandum should explain why not. If influential scientific information
11-P-0702
19
-------
or highly influential scientific assessments were not used to support the action, the
action memorandum should include a statement to that effect.
EPA did not provide statements in the preamble to either the proposed or final
endangerment findings describing whether influential scientific information or
highly influential scientific assessments were used to support the finding.
Additionally, EPA did not provide a statement certifying that it conducted a peer
review of the information supporting the findings in accordance with OMB's
Final Information Quality Bulletin for Peer Review. Briefing documents for the
EPA Administrator stated that the TSD would be reviewed by federal climate
change experts. However, this documentation also did not identify whether the
TSD was influential scientific information or a highly influential scientific
assessment, nor did it describe how the proposed review would comply with
OMB's peer review requirements and EPA's peer review policy.
Additionally, EPA's action memoranda for the proposed and final findings did not
state that the Agency followed its peer review policy with respect to the
underlying information supporting the endangerment findings. EPA also did not
provide any statements in its action memoranda for either the proposed or final
findings that identified whether the supporting information was influential
scientific information or a highly influential scientific assessment. In responding
to our draft report findings, the Agency characterized the TSD as influential
scientific information and thus not subject to OMB's peer review requirements for
highly influential scientific assessments.
OAR Did Not Follow Some Steps in the Action Development Process
The development of the endangerment finding did not follow all action
development process steps as outlined in EPA's action development process
guidance. This was due, in part, to the finding originally being included as part of
a proposed greenhouse gases transportation rule developed in 2007. EPA
withdrew this rule after enactment of the Energy Independence and Security Act.
In July 2008, EPA issued an ANPR asking for comment on the ramification of an
endangerment finding and on the underlying science. When EPA initiated a
formal action development process for the stand-alone greenhouse gases
endangerment finding in early March 2009, the TSD in support of the finding had
already been through several iterations and one round of review by a panel of
federal climate change scientists. Further, the OAR program office had already
briefed the Administrator on potential options for making an endangerment
finding. Thus, when the formal workgroup was convened to develop the proposed
stand-alone endangerment finding, OAR used the same approach it used for the
original 2007 action and the 2008 ANPR.
When OAR began the action development process for the stand-alone
endangerment finding in March 2009, it designated the action as a Tier 1 action.
According to EPA's action development process guidance, the workgroup should
11-P-0702
20
-------
prepare a preliminary analytic blueprint detailing its approach to developing the
supporting data for the action. This guidance states that the preliminary analytic
blueprint is revised and finalized once the workgroup receives "Early Guidance
from senior management." Among other things, the analytic blueprint identifies
the type of information EPA plans to use to support the action, and addresses the
question of whether external peer review will be needed to meet the requirements
of the Agency's peer review policy.
An analytic blueprint was developed as part of the 2007 rulemaking on
greenhouse gas emission control for light-duty vehicles and transportation fuels.
This blueprint contained an outline for the endangerment finding portion of the
rulemaking. The outline listed the IPCC, CCSP (USGCRP), and NRC reports as
core references for the development of OAR's TSD. However, the outline did not
explain what reviews were needed before accepting the other organizations' data
or how the TSD would be peer reviewed. For the stand-alone endangerment
finding, a revised analytic blueprint was not prepared. Instead, the action went
forward using the approach to developing and reviewing the TSD that had been
previously approved in 2007.
Although the analytic blueprint for the 2007 action did not describe how the TSD
would be peer reviewed, OAR prepared nine briefing documents for EPA senior
management that provided details on the Agency's plans for preparing and peer
reviewing the TSD. These briefings were conducted from May through
September 2007. These briefing documents outlined the Agency's approach but
did not explain why it chose not to have a formal external peer review of the TSD.
Two of the nine briefing documents were prepared for presentation to the former
EPA Administrator. The current Administrator was made aware of the approach
OAR was taking with the TSD in a February 2009 briefing. We also noted that the
former EPA Administrator provided early guidance to the initial 2007 workgroup
for developing the endangerment finding portion of the action. According to EPA,
early guidance was provided for the 2009 action during briefings with the
Administrator.
OAR also did not follow certain procedural guidelines outlined in the action
development process. These deviations included (1) not having a workgroup
meeting to discuss the options and policies to be considered at the October 29,
2009, options selection meeting; (2) reducing the time to review options selection
meeting materials; (3) not including all reviewing offices' positions in the options
selection meeting materials; (4) not documenting the options selection meeting
decisions; and (5) not providing a complete final Agency review package to the
Assistant Administrators/Regional Administrators, workgroup members, and
Regulatory Steering Committee representatives/regional regulatory contacts prior
to the final Agency review meeting.
In our opinion, the potential impact of these procedural deviations on the quality
of the action is debatable. The EPA Office of Policy, Regulatory Management
11-P-0702
21
-------
Division Director, who is also chair of the Regulatory Steering Committee, said
that the action development process guidance is supposed to be flexible.
However, in our view, reducing the time to review materials and not providing all
materials have the potential to hinder a reviewer's ability to comment on a
proposed action.
Conclusions
In our opinion, the endangerment finding TSD is a highly influential scientific
assessment that should have been peer reviewed as outlined in Section III of
OMB's Final Information Quality Bulletin for Peer Review. OAR never formally
designated the document as either influential scientific information or as a highly
influential scientific assessment in the preamble to the proposed and final
endangerment findings or in its internal documentation. EPA did not consider the
TSD to be a highly influential scientific assessment.
Additionally, OAR did not adhere to some of its internal processes established to
guide Tier 1 actions. We noted that OAR had completed many of the processes
and steps outlined in its guidance to ensure the quality of the information the
Administrator used in making her determination. Those processes are intended to
help ensure EPA develops quality actions and to provide assurance on data
quality. We concluded that the Agency did not complete some of these key
requirements and recommended actions. We did not analyze the quality of the
scientific information and data used to support the Administrator's decision.
Recommendations
We recommend that the Assistant Administrator for Research and Development
direct the EPA Science Policy Council to:
1. Revise the flowchart on page 2 of EPA's Peer Review Handbook to
ensure that the flowchart accurately depicts OMB requirements for
external peer review of highly influential scientific assessments.
2. Instruct program offices that, when using influential scientific
information or highly influential scientific assessments supporting an
action, to:
a. Include language in the preamble of proposed and final rules
that specifically states that the action was supported by
influential scientific information or a highly influential
scientific assessment, and certifies that EPA conducted a peer
review of the supporting information in accordance with
OMB's Final Information Quality Bulletin for Peer Review.
11-P-0702
22
-------
b. Include a compliance statement in its action memoranda stating
that the Agency followed its peer review policy.
Agency Comments and OIG Evaluation
The Agency requested that the final OIG report provide more context regarding
the extent of peer review conducted for the underlying scientific assessments and
the extent of EPA's peer and public review of its TSD. We added descriptions of
the peer review procedures of IPCC, USGCRP, and NRC to chapter 1 of the final
report. However, we believe the draft report contained sufficient information on
EPA's peer and public review of its TSD.
The Agency disagreed with our conclusion in the draft report that the TSD was a
highly influential scientific assessment and thus subject to certain peer review
requirements that EPA did not implement. EPA responded that the TSD does not
meet the OMB definition of a scientific assessment in that no weighing of
information, data, and studies occurred in the TSD. EPA maintained that this
process had already occurred in the underlying assessments, where the scientific
synthesis occurred and where the state of the science was assessed. EPA stated
that the TSD is not a scientific assessment, but rather a document that summarized
in a straightforward manner the key findings of NRC, USGCRP, and IPCC. In our
opinion, the TSD met the definition of a scientific assessment in that it evaluated
a body of scientific knowledge and synthesized multiple factual inputs. While we
agree that the primary information EPA relied upon were scientific assessments,
these assessments were voluminous and numerous. Thus, EPA had to weigh the
conclusions and information in those assessments in deciding which information
to present in its TSD. In addition, the TSD included information from outside
these assessments, such as greenhouse gas emissions data and statements from
government officials regarding national security consequences of climate change.
EPA disagreed with a heading in our draft report stating that "OAR Did Not
Identify the Specific Type of Information Supporting Its Action or Certify
Compliance with EPA's Peer Review Policy." EPA responded that the full
endangerment record, including the endangerment finding, TSD, and the response
to comments, provided extensive discussions regarding the type of information
supporting the finding. However, EPA acknowledged that the record did not
contain a specific discussion of the TSD and its peer review in terms of the peer
review requirements for influential scientific information. EPA agreed that
including a clearer discussion of this issue could have been helpful and could
have avoided confusion. We agree that the endangerment finding record provided
extensive discussion of the type of information supporting its finding, but not in
the manner required by OMB's peer review bulletin and EPA's peer review
policy. We changed the title of this subsection in the final report to more
accurately reflect our conclusion.
11-P-0702
23
-------
With respect to EPA's implementation of the procedures outlined in its action
development process, EPA responded that our draft report did not present a
balanced discussion of the actions EPA took to achieve the goals and
communicate the major components of an analytic blueprint to all program offices
and workgroup members. We made minor revisions to the final report in this
section. However, our draft report acknowledged that even though EPA did not
revise its analytic blueprint for the stand-alone endangerment finding action, it
prepared nine briefing documents for EPA senior management that provided
details on the Agency's plans for preparing and peer reviewing the TSD.
EPA did not propose any corrective actions for the recommendations in the draft
report. EPA stated that it will address any recommendations in its written
response to the final report. EPA suggested that we clearly state in our final report
that our conclusion is based on our opinion that the TSD is a highly influential
scientific assessment, and that if the TSD is in fact not a highly influential
scientific assessment, then such conclusions, and perhaps attendant
recommendations, would not apply. We did not make any clarifications to the
final report in response to this comment, because our recommendations address
the Agency's action development process and are not dependent upon whether the
endangerment finding TSD was considered influential scientific information or a
highly influential scientific assessment. We consider recommendations 1 and 2
unresolved. Detailed responses to EPA comments on the draft report are included
in appendix G.
OMB Comments and OIG Evaluation
OMB stated that it believes that EPA reasonably interpreted the OMB peer review
bulletin in concluding that the TSD did not meet the bulletin's definition of a
highly influential scientific assessment. OMB commented that EPA concluded
that it was the separate, underlying assessments of the IPCC, USGCRP, and NRC
that met OMB's definition of a scientific assessment. EPA's TSD, according to
OMB, provided a condensed form of the three underlying assessments. OMB
further stated that, rather than creating a new scientific assessment, it understood
EPA's TSD to include a "reader-friendly" version of the passages from the
underlying assessments on which EPA was relying. We agree that the primary
underlying assessments that EPA relied upon in developing its TSD were
scientific assessments. However, by synthesizing the findings, conclusions, and
other information from these assessment reports, as well as from other sources, in
its TSD, EPA evaluated the state of science and produced an entirely new and
separate document that met OMB's definition of a scientific assessment.
OMB also stated that it believes EPA reasonably concluded that it was the
underlying assessments that were identified in the TSD, and not the TSD itself,
that proved "highly influential" to EPA's determination in its endangerment
finding. We agree that the major assessments cited in EPA's TSD are "highly
influential." We also believe that the TSD is "highly influential" because it
11-P-0702
24
-------
provided scientific and technical information in support of a Tier 1, significant
regulatory action. The stated purpose of the TSD was . . to provide scientific
and technical information for an endangerment and cause or contribute analysis
regarding greenhouse gas (GHG) emissions. . . ." EPA documentation indicated
the TSD played a role in informing the Administrator's endangerment decision.
For example, the former Director of EPA's Climate Change Division told us in an
August 2010 e-mail that former EPA Administrator Johnson reviewed the entire
TSD in 2007 and found the science in it to be the basis for his core decision that
there was endangerment. Also, in the endangerment finding EPA described the
April 2009 TSD as the "underlying scientific and technical basis" for the
Administrator's proposed findings. Responses to OMB's comments on the draft
report are included in appendix H.
11-P-0702
25
-------
Chapter 3
EPA Procedures for Assessing
External Data Are Unclear
We found that EPA relied upon descriptions of other organizations' information
quality processes and U.S. government acceptance10 of work products in
determining that the scientific work products it used to support the endangerment
finding met OMB and EPA requirements for data quality. The Agency provided
statements in its final findings notice and supporting TSD that generally
addressed the Agency's assessment factors for evaluating scientific and technical
information. Additionally, EPA addressed public comments to the proposed rule
in an 11-volume response to comment document. EPA responded to specific
concerns related to the major assessment reports summarized in the TSD, as well
as data quality issues. However, no contemporaneous documentation was
available to show what analyses the Agency conducted prior to dissemination of
the information in its advance notice and proposed action. The Agency's guidance
for assessing outside sources of data does not include procedures for conducting
these assessments or require the Agency to document its assessments.
EPA Guidance for Assessing Externally Generated Data Can Be
Improved
EPA's A Summary of General Assessment Factors for Evaluating the Quality of
Scientific and Technical Information contains guidance on the factors that EPA
staff should consider in evaluating the quality and relevance of information,
regardless of source. However, this guidance document does not identify specific
steps or procedures EPA personnel should use in determining whether scientific
and technical information is of acceptable quality, nor does it identify the
documentation requirements for these determinations.
Section 2.2.17 of EPA's Peer Review Handbook contains peer review
considerations for products submitted to EPA for use in decisionmaking. These
considerations can be used to address the "evaluation and review" factor
identified in EPA's guidance, A Summary of General Assessment Factors for
Evaluating the Quality of Scientific and Technical Information. According to the
Peer Review Handbook, "It is hoped that if the other organization has the work
111 U.S. government representatives, led by the White House Office of Science and Technology Policy and U.S.
Department of State, approved the "Summaries for Policymakers" for each of the four volumes of the IPCC AR4 in
2007. IPCC procedures state that "approval" means "the material has been subjected to detailed, line by line
discussion and agreement." In addition to approval of the "Summaries for Policymakers," a director of Operations
with IPCC told us that participating governments, including the U.S. government, accepted the underlying working
group chapters by acclamation during IPCC plenary sessions. "Acceptance," according to IPCC procedures, means
that "the material has not been subject to line by line discussion and agreement, but nevertheless presents a
comprehensive, objective and balanced view of the subject matter."
11-P-0702
26
-------
product independently peer reviewed, the peer review will meet the intent of the
Agency's Peer Review Policy and EPA's proposed use of the product (i.e., the
peer review is basically equivalent to what EPA would do)." The handbook states
that "Agency staff from the appropriate office(s) should examine closely the
particulars of the peer review to ensure independence and a conscious effort to
incorporate the peer reviewers' comments into the final work product." The
handbook does not describe any documentation requirements for this review.
We found that EPA did not contemporaneously document how it applied and
considered the assessment factors in determining whether the IPCC and other
assessment reports were of sufficient quality, objectivity, utility, and integrity.
EPA described the IPCC review procedures and how they met EPA data quality
requirements in the proposed and final rulemakings. However, the Agency did not
conduct any independent evaluations of IPCC's compliance with IPCC
procedures, nor did EPA document any specific processes it employed to evaluate
the scientific and technical information included in IPCC's AR4 prior to EPA
disseminating that information. An OAR manager told us that OAR routinely
applies the assessment factors to information it receives or gathers, including the
IPCC assessment reports, but that the Agency did not have any procedures in
place to document how these factors are applied.
In its endangerment finding, EPA cited several reasons for relying on the findings
from the IPCC, USGCRP, and NRC assessments as the primary basis for the
endangerment decision. These reasons included the "rigorous and exacting
standard of peer review" employed by these organizations, as well as U.S.
government acceptance of IPCC's AR4. The Agency also stated in its response to
comments document that it was an active participant in the USGCRP and IPCC
assessments. According to EPA's response to comments document, it "was the
lead agency for three significant reports under the USGCRP and recently
completed an assessment addressing the climate change impacts on air quality in
the United States." EPA was also involved in a review of IPCC's AR4, and in
particular took part in the approval of the summary for policymakers for the
Working Group II volume, Impacts, Adaptation, and Vulnerability.
EPA stated in its endangerment finding that it had maintained "the highest level
of adherence to Agency and OMB guidelines for data and scientific integrity and
transparency." With respect to accepting and disseminating data produced by
other organizations, OMB told us:
If an agency uses another organization's data or analysis to support
their policy, they are disseminating that information. As such, that
information becomes subject to the Agency's Information Quality
Guidelines and the Bulletin for Peer Review.
11-P-0702
27
-------
Therefore, in evaluating whether to disseminate the information,
the agency must determine whether the information complies with
the Agency's Information Quality Guidelines.
and
If an assessment has been endorsed by one agency within the
U.S. Government, and is going to be used by another agency, the
agency using the information should ensure that the assessment is
consistent with its own Information Quality Guidelines.
Because EPA used information from other organizations to support its findings,
EPA, in evaluating whether to disseminate11 that information, should have
determined whether the assessments referenced in the TSD (e.g., IPCC's AR4)
complied with EPA's information quality guidelines, and whether the peer
reviews of these assessments met OMB's requirements for peer review of
scientific assessments. U.S. government acceptance of the documents did not
relieve EPA of its responsibility to determine whether the data met EPA's
information quality guidelines before disseminating the information. We reviewed
EPA's descriptions of how the IPCC review process met EPA quality guidelines
to determine whether EPA addressed all applicable OMB requirements in
Section III of its Final Information Quality Bulletin for Peer Review. These
descriptions did not address all applicable OMB requirements for highly
influential scientific assessments. Table 2 summarizes the results of our review.
Table 2: EPA's examination of IPCC's AR4 peer review
OMB criterion for peer review of highly
influential scientific assessments
Did EPA address the OMB criterion in
Federal Register notice or
response to comments?
Expertise and balance of peer reviewers
Yes
Conflicts of interest of peer reviewers
No
Independence of peer reviewers
Yes
Rotation of peer reviewers
No
Information access for peer reviewers
No
Opportunities for public participation
Yes
Transparency of review process
Partially3
Management of the peer review
Not Applicable13
Source: OIG analysis of EPA's response to comments to the proposed rule.
a Transparency was generally addressed but not for all elements. For example, EPA did
not discuss whether IPCC procedures required a description of the credentials and
relevant experiences of each peer reviewer.
b This criterion only applies to peer reviews managed by a federal agency.
11 OMB defines "dissemination" as "agency initiated or sponsored distribution of information to the public" (see
5 Code of Federal Regulations 1320.3(d) (definition of "Conduct or Sponsor")). Therefore, information submitted to
the public as part of an advance notice or proposed action has been "disseminated" according to OMB's definition.
11-P-0702
28
-------
After disseminating its endangerment finding, in response to petitions for
reconsideration, the Agency took further steps to examine and evaluate IPCC
procedures. As part of this process, the Agency evaluated evidence provided by
petitioners related to allegations that IPCC peer review and report development
procedures were designed inappropriately.
Conclusions
EPA determined that the IPCC assessment and other outside reports met EPA's
information quality guidelines and were sufficiently peer reviewed. EPA's
reasoning was described in its response to comments on the proposed rule.
However, no supporting analytical information was available to show how EPA
made its determination prior to disseminating the information. EPA's guidance
for assessing the quality of externally generated information does not provide
procedures or steps for assessing outside data or requirements for documenting
such analysis.
Recommendation
We recommend that the Assistant Administrator for Research and Development:
3. Revise EPA's guidance document, A Summary of General Assessment
Factors for Evaluating the Quality of Scientific and Technical
Information, to establish minimum review and documentation
requirements for assessing and accepting data from other
organizations.
Agency Comments and OIG Evaluation
The Agency did not concur with the conclusions in our draft report that EPA did
not conduct any independent evaluations of IPCC's compliance with IPCC
procedures, and that EPA did not document any specific processes it employed to
evaluate the scientific and technical information included in IPCC's AR4. EPA
stated that the scientific assessments it utilized, namely those from NRC, IPCC,
and USGRCP, underwent their own peer review processes, which are well known
and accepted by the U.S. government. According to EPA, these processes are
accepted by the U.S. government as appropriate mechanisms to ensure the quality
of the science assessments. We agree that EPA took additional steps to evaluate
and examine IPCC procedures in response to specific allegations made by
petitioners, and we added a paragraph to the final report to reflect this. However,
the conclusions presented in our draft report remain valid. Our conclusions
focused on how EPA evaluated the quality of externally generated information
prior to EPA disseminating that information.
11-P-0702
29
-------
Regarding the IPCC review process, EPA stated that it devoted significant staff
time to reviewing the IPCC assessment reports, providing comments during an
interagency process, participating in that interagency process to prioritize U.S.
government comments, and approving the summaries of the IPCC reports in a
detailed line-by-line process. EPA commented that our draft report did not include
the examples of instances in which EPA specifically discussed and documented
throughout the endangerment finding record how the IPCC peer review
procedures meet EPA and OMB guidelines. We added statements to chapter 1,
chapter 3, and appendix A of the final report to indicate EPA's involvement in the
IPCC process, including its participation in the U.S. delegation approving the
Working Group II report for the AR4. However, the only organization for which
OMB guidance specifically allows federal agencies to presume findings and
conclusions to be adequately peer reviewed is NAS. We believe that EPA's
guidance for assessing the quality of information generated by outside sources,
other than NAS, does not provide procedures or steps for assessing outside data or
requirements for documenting such analysis.
The Agency also disagreed with several of the points provided in table 2 of
chapter 3, and directed the OIG to its response to petitions document as evidence
of additional EPA review and analysis of IPCC procedures. We believe the
information in table 2 of chapter 3 to be accurate and did not make changes to
the table from our draft report. We consider recommendation 3 unresolved.
Detailed responses to EPA comments on the draft report are included in
appendix G.
11-P-0702
30
-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Action Officials
Planned
Completion
Date
Claimed
Amount
Ag reed-To
Amount
22 Direct the EPA Science Policy Council to revise the
flowchart on page 2 of EPA's Peer Review
Handbook to ensure that the flowchart accurately
depicts OMB requirements for external peer review
of highly influential scientific assessments.
22 Direct the EPA Science Policy Council to instruct
program offices that, when using influential
scientific information or highly influential scientific
assessments supporting an action, to:
a. Include language in the preamble of
proposed and final rules that specifically
states that the action was supported by
influential scientific information or a highly
influential scientific assessment, and certifies
that EPA conducted a peer review of the
supporting information in accordance with
OMB's Final Information Quality Bulletin for
Peer Review.
b. Include a compliance statement in its action
memoranda stating that the Agency followed
its peer review policy.
29 Revise EPA's guidance document, A Summary of
General Assessment Factors for Evaluating the
Quality of Scientific and Technical Information, to
establish minimum review and documentation
requirements for assessing and accepting data
from other organizations.
Assistant Administrator for
Research and Development
Assistant Administrator for
Research and Development
Assistant Administrator for
Research and Development
0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is unresolved with resolution efforts in progress
11-P-0702 31
-------
Appendix A
Answers to Specific Questions
From the Ranking Member
This appendix presents the results of our review for the questions posed by the Ranking Member,
Committee on Environment and Public Works, United States Senate.
1. Did EPA conduct an examination of the IPCC procedures, including the IPCC process
for handling review comments? How did EPA determine that the IPCC process satisfied
EPA's obligations to follow the Data Quality Act and the Agency's, as well as OMB's,
peer review guidelines? How was this determination documented?
As discussed in chapter 3, EPA examined IPCC procedures to the extent that EPA described
these procedures in its proposed and final endangerment finding packages. Although OAR did
not contemporaneously document its determinations regarding compliance with the DQA or
OMB's or the Agency's peer review guidelines prior to disseminating its endangerment finding,
the Agency stated in both the proposed and final findings that it adhered to its information
quality guidelines in developing its findings and associated TSD. EPA also identified several
reasons in its response to comments document as to why it believed the supporting information
in its TSD adhered to a basic standard of quality, including objectivity, utility, and integrity.
After disseminating its endangerment finding, in response to petitions for reconsideration, the
Agency took further steps to examine and evaluate IPCC procedures. As part of this process, the
Agency evaluated evidence provided by petitioners related to allegations that IPCC peer review
and report development procedures were designed inappropriately.
EPA Examination of IPCC Procedures
EPA provided detailed descriptions of the IPCC principles and procedures in documents
associated with its endangerment and cause or contribute findings for greenhouse gases. These
documents included:
The final findings (Section III. A)
The TSD accompanying the proposed and final findings
EPA's response to comment document
EPA's response to petitions document
According to EPA, the assessment reports summarized in the TSD, including IPCC's AR4,
"were prepared following rigorous and transparent processes addressing such issues as the
nomination and selection of authors, the caliber of literature reflected in the assessment, and the
processes for review and revision of reports."
EPA also stated in its TSD that the procedures employed by the IPCC, among others, provided
the Agency with assurances that the assessment material was well vetted by both the climate
change community and by the U.S. government.
11-P-0702
32
-------
How EPA Determined That IPCC Procedures Met Applicable Information Quality Criteria
EPA did not provide documentation to describe the specific processes it used to determine that
IPCC's procedures met the applicable information quality criteria. An OAR manager told us that
the Agency routinely applies its General Assessment Factors for Evaluating the Quality of
Scientific and Technical Information, and that these factors were considered when the Agency
evaluated the information generated by IPCC's AR4. However, the Agency did not have
procedures in place to document who evaluated the quality of information, or how the
assessment factors were considered by the Agency in evaluating the information from IPCC.
Although EPA did not clearly document how it examined the IPCC's procedures, the Agency
provided several reasons as to why it believed the information in IPCC's AR4 was of sufficient
quality, objectivity, utility, and integrity. According to EPA, the IPCC procedures met applicable
information quality criteria because:
IPCC's AR4 was reviewed and formally approved and accepted by the U.S.
government.
EPA's familiarity with procedures used by IPCC resulted in EPA's confidence that
these procedures were consistent with federal information quality guidelines for
quality, objectivity, utility, and integrity.
EPA stated that it "thoroughly reviewed and evaluated the author selection, report
preparation, expert review, public review, information quality, and approval
procedures of IPCC, USGCRP/CCSP, and NRC to ensure that the information
adhered 'to a basic standard of quality, including objectivity, utility, and integrity.'"
EPA was also involved in a review of IPCC's AR4, and in particular took part in the approval of
the summary for policymakers for the Working Group II volume, Impacts, Adaptation, and
Vulnerability. Additionally, in response to petitions for reconsideration of its endangerment
finding action, the Agency assessed specific critiques of IPCC procedures. In summary, EPA
concluded that:
In regard to concerns that some draft assessment reviewers should have also been
listed as report contributors, it was proper to not list certain scientists as contributing
authors as they did not contribute significantly to the writing and editorial decisions
in developing any AR4 chapter, including chapter 6 of Working Group I's
contribution (Jansen et al., 2007). Therefore, these reviewers' objectivity was not
compromised during the peer review process.
IPCC authors did not cite their own studies more frequently than what was acceptable
and reasonable.
IPCC authors were not directed to focus on policy-prescriptive conclusions, but rather
implemented IPCC guidelines by presenting policy-relevant and neutral findings.
IPCC authors did not alter the content of reports to eliminate suggestions of
nonconsensus.
Collaborations among IPCC authors and reviewers prior to the development of AR4
did not compromise objectivity or generate conflicts of interest.
11-P-0702
33
-------
IPCC's peer review processes are appropriate and adequate, and were properly
implemented.
IPCC authors did not manipulate deadlines for receipt of new literature.
IPCC's very limited use of gray literature12 does not call into question the quality and
objectivity of the assessment reports.
EPA Did Not Evaluate IPCC's Compliance With Its Procedures
EPA did not evaluate IPCC's compliance with its principles and procedures. According to the
chief of the Climate Science and Impacts Branch, EPA had no reason not to accept the IPCC
assessment reports as valid and of high quality. EPA was part of the IPCC process, and the U.S.
government was broadly involved with the IPCC, including the development, review, and
approval of IPCC's AR4. Therefore, EPA felt confident accepting IPCC's AR4 as valid and of
high quality.
2. IPCC procedures require that it consider all information and scientific viewpoints.
Examine how EPA evaluated and determined that the IPCC examined all viewpoints.
In general, EPA evaluated whether IPCC considered all scientific viewpoints during the
development of its AR4 through a review and examination of IPCC's procedures. Formal
comments that alleged IPCC did not consider all scientific viewpoints in developing its AR4
were submitted to EPA during the public comment period for the proposed findings, and in
petitions for reconsideration that were submitted after the final findings were published.
In responding to public comment for the proposed findings, the Agency relied upon its
understanding of the stated IPCC process to ensure that all viewpoints were considered. For
example, EPA stated in its response to comment document that it "carefully reviewed the IPCC
procedures" and concluded that there was no evidence to show that alternative perspectives were
not incorporated into the IPCC process. The Agency also noted the breadth and scope of IPCC's
review process, in interviews and in the endangerment finding itself, as a characteristic that
helped to include all viewpoints.
IPCC's AR4 was described by EPA as "robust." EPA noted the following about the IPCC
process:
IPCC's AR4 synthesized thousands of individual studies and conveyed the consensus
conclusions on the body of scientific literature.
More than 6,000 peer-reviewed publications were cited by IPCC Working Group I.
"In implementing [IPCC's] procedures across the three working groups of the IPCC's
AR4, 1,250 scientists (450 lead authors and more than 800 contributing authors) from
130 countries served as authors and more than 2,500 experts provided over 90,000
review comments."
12 "Gray literature" is written material, such as a report, that is not published commercially or is not generally
accessible.
11-P-0702
34
-------
Since issuing its final findings in December 2009, EPA received 10 petitions requesting that
EPA reconsider its findings. As part of their request for reconsideration, petitioners claimed that
IPCC suppressed dissenting views during the development of its AR4, and some of the
petitioners provided e-mails from University of East Anglia Climatic Research Unit (CRU)
scientists as part of the evidence to support this claim. Similar to the response EPA provided in
its response to comments document for the proposed findings, EPA again referred to the IPCC
procedures as a means of ensuring that all scientific views were considered during the
development of the AR4. According to EPA's response to petitions document:
... the IPCC processes have been developed for the express purpose of ensuring
integrity and avoiding any possibility of bias in the conclusions drawn from the
scientific literature. The level of detail and clarity of the requirementsespecially
concerning report review, the specific responsibilities of review editors to assess
the quality of author responses to comment, and the transparency of the record of
comments and responsesprovide the foundation for ensuring that the reports are
credible and sound.
In response to one particular petitioner comment, EPA reviewed the IPCC AR4 to determine
how two particular references were included in the assessment. The Agency found that the IPCC
AR4 report cited the two references, thus providing "... stronger support for the proposition
that the IPCC assessed and cited critical literature than for the petitioners' view that these events
demonstrate that the IPCC is biased." Further, EPA stated that its conclusions on this issue were
consistent with the findings of the Independent Climate Change E-mails Review investigation,
which stated that the panel has "not found any direct evidence to support the allegation that
members of CRU . . . misused their position on IPCC to seek to prevent the publication of
opposing ideas." In all instances, EPA found that the petitioners did not present sufficient
evidence to demonstrate that the IPCC did not consider all scientific viewpoints during the
development of its AR4.
3. Was EPA aware of editing of final IPCC assessment reports after the reviewers
submitted their final comments?
According to the counsel for the Air and Radiation Law Office of Office of General Counsel,
OAR was not aware of editing of the final IPCC AR4 after the reviewers submitted their final
comments. An OAR manager said that based on EPA's knowledge of the IPCC procedures,
OAR had no reason to think this was an issue. The manager also noted that the Agency had not
asked IPCC whether such editing had occurred. An OAR manager noted that a concern was
raised about improper edits being made to the second IPCC assessment report in 1995; however,
this concern was not substantiated. The OAR manager added that the concern about the second
IPCC assessment report prompted IPCC to change its review and editing procedures to the
current procedures.
11-P-0702
35
-------
4. Was the Endangerment Finding's Technical Support Document (TSD) subjected to
peer review as specified in the EPA Peer Review Handbook? If not, please provide EPA's
explanation for why it was not.
EPA did not conduct a peer review of the TSD that met all recommended steps in the Peer
Review Handbook for peer reviews of influential scientific information or highly influential
scientific assessments. EPA's peer review policy states that "for influential scientific information
intended to support important decisions, or for work products that have special importance in
their own right, external peer review is the approach of choice" and that "for highly influential
scientific assessments, external peer review is the expected procedure." According to the policy,
external peer review involves reviewers who are "independent experts from outside EPA." The
handbook provides examples of "independent experts from outside EPA," that include NAS, an
established Federal Advisory Committee Act mechanism (e.g., Science Advisory Board), and an
ad hoc panel of independent experts outside the Agency. The handbook lays out a number of
procedural steps involved in an external peer review.
EPA had the TSD reviewed by 12 federal climate change experts. While all but one of the
experts was from outside EPA, this review did not follow all recommended steps of an external
peer review outlined in the handbook. For example, EPA did not create a formal peer review
record, have the expert panel prepare a peer review report, prepare an Agency response to
reviewer comments on the TSD that accompanied the proposed and final rules, obtain written
management approval of EPA's response to the reviewers' comments on the TSD that eventually
accompanied the ANPR, or include peer review certification in the preamble to the rule. OAR
officials explained that an external peer review of the TSD was not needed because (1) they did
not consider the TSD to be a scientific assessment because it only summarized existing findings
and conclusions and provided no new findings or conclusions, and (2) the core references relied
upon for the TSD had been peer reviewed in a manner consistent with the OMB bulletin.
Although the Agency did not conduct all of the recommended steps in its Peer Review
Handbook, the handbook and OMB guidance allow discretion in the methodology used to peer
review influential scientific information. OMB does not provide such discretion for highly
influential scientific assessments, and peer review of these assessments must meet the minimum
peer review requirements outlined in Section III of OMB's I una I Information Quality Bulletin on
Peer Review. Both EPA and OMB disagreed with our conclusion that the TSD met the definition
of a highly influential scientific assessment and should have been reviewed in accordance with
the minimum requirements of Section III. For more information on this subject, refer to chapter 2
of our report.
5. EPA has acknowledged sending the Draft TSD to a group of federal climate change
experts for review. Apparently this was done for a number of versions of the Draft TSD.
Were changes made to the Draft TSD based on these federal reviewers' comments? Did
this process follow EPA's, as well as OMB's, peer review guidelines?
Changes were made to draft versions of the TSD in response to comments from 12 federal
climate change experts. The draft TSD was formally reviewed by the 12 federal climate change
experts on three occasions (August 16 to September 4, 2007; March 9-16, 2009; and
11-P-0702
36
-------
October 9-19, 2009). EPA made changes to the draft TSD after each of these reviews.
Specifically, changes were made to the draft TSD before it was published with the:
ANPR
Proposed endangerment finding
Final endangerment finding
EPA maintained documentation showing its response to and disposition of reviewer comments
for the first draft TSD that accompanied the ANPR. However, EPA did not maintain
documentation showing its response to and disposition of comments reviewers made to the
versions of the TSD that accompanied the proposed and final actions. While our review of
subsequent versions of the TSD indicated that changes were made after receipt of the reviewers'
comments, we did not attempt to trace all potential changes back to revised TSDs.
The federal climate change expert review of the TSD did not follow all the recommended
procedures outlined in the Peer Review Handbook or OMB's guidelines for peer review of
influential scientific information or highly influential scientific assessments. According to EPA's
Peer Review Handbook, an external peer review mechanism is the preferred choice for
influential scientific information, and highly influential scientific assessments are expected to
undergo external peer review. However, the following items specified in the handbook for
external peer review were not developed or obtained:
Formal peer review record
Peer review report
EPA's response to the reviewers' comments on the TSD that accompanied the
proposed rule, and comments on the TSD that accompanied the final rule
Written management approval of EPA's response to the reviewers' comments on the
TSD that eventually accompanied the ANPR
The review panel was predominantly an external panel, as 11 of the 12 panelists worked for
other federal agencies. One of the panelists was an EPA employee. None of the 12 panelists
participated in the drafting of the TSD. According to OAR managers, the TSD did not require an
external peer review because the scientific assessments it relied upon had already been
rigorously reviewed. Thus, EPA did not need to follow the Peer Review Handbook procedures
for external peer review in conducting the federal climate change expert review.
6. Assess the Interagency review process used in developing the Endangerment Finding.
Were there significant interagency comments on the finding? How were these resolved?
The TSD was subject to OMB/interagency review prior to publication of EPA's ANPR,
proposed findings, and final findings. For each review, EPA addressed and resolved interagency
comments on the TSD to OMB's satisfaction. In some cases, EPA made changes to the TSD in
response to comments, and in other cases, EPA explained why it did not agree with the comment
or did not make changes to the TSD.
The OMB/interagency review process for the version of the TSD that accompanied EPA's ANPR
on the Regulation of Greenhouse Gases under the CAA did not result in any significant comments.
11-P-0702
37
-------
The OMB/interagency review process for the version of the TSD that accompanied the proposed
findings resulted in 86 comments. In general, the comments related to:
Requesting certain TSD language be more consistent with cited literature
Inserting additional citations or clarifying existing citations
Requesting clarification or additional evidence for statements made in the TSD
More clearly stating uncertainties associated with some findings
Differentiating between U.S. and international impacts of climate change
EPA made changes to the draft TSD to address 39 comments. For 45 other comments, EPA did
not make changes to the draft TSD, but provided responses to OMB with its rationale for not
doing so. In two instances, EPA's actions to address the OMB/interagency comments did not
appear to be directly responsive to the comments; however, we did not find these two instances
to be of significance. OMB approved all EPA actions and responses to OMB/interagency review
comments.
There were no OMB/interagency comments provided for the version of the TSD that
accompanied the final findings.
7. In recent months a number of e-mails from the Climatic Research Unit ("CRU") of the
University of East Anglia in the United Kingdom were released. EPA has claimed that
these e-mails do not affect the fundamental findings of the IPCC assessment reports.
What independent analyses has EPA conducted to reach this conclusion, in particular its
conclusion regarding the HadCRUT temperature dataset and its relation to the other data
sets used in the endangerment finding from NOAA and NASA [National Aeronautics and
Space Administration]?
In November 2009, subsequent to publication of EPA's proposed finding, approximately 1,000
e-mails were hacked from the servers of the University of East Anglia CRU and made public.
CRU is recognized for its climate change research and, since 1978, had developed and
maintained a land-based temperature record widely used by climate change researchers.
According to CRU, its staff have been heavily involved in the IPCC assessments, and CRU's
work has been used by IPCC in construction of future climate projections. The content of the
e-mails caused some to challenge the work of CRU and the conclusions of the IPCC. Since EPA
relied heavily upon IPCC's AR4 in developing the TSD for its endangerment finding, concerns
have been raised about EPA's acceptance and use of this information in light of federal and
Agency information quality guidelines.
In response to allegations that CRU scientists acted inappropriately in handling their data and
improperly influenced the process of advising policymakers, the University of East Anglia
commissioned two studies. An April 2010 study, chaired by Professor Ron Oxburgh, examined
the integrity of the research published by CRU. A July 2010 study, chaired by Sir Muir Russell,
examined the conduct of the CRU scientists. In general, the studies found that there was no
deliberative scientific malpractice at CRU, and that there was no evidence of CRU scientists'
behavior that would undermine the conclusions of the IPCC assessments. However, the Oxburgh
report also concluded that there would be benefits to CRU collaborating with professional
11-P-0702
38
-------
statisticians to aid their research, and noted that there were unresolved questions relating to the
availability of environmental datasets. Further, the Russell report found that both CRU scientists
and the University of East Anglia failed to display the proper degree of openness regarding their
research.
In March 2010, the United Nations and IPCC requested the InterAcademy Council, a
multinational organization of science academies, to conduct an independent review of IPCC's
procedures and processes. The InterAcademy Council released its final report in August 2010,
concluding that the IPCC assessment process had been successful and "served society well."
However, the InterAcademy Council also found that some fundamental changes to the IPCC
processes and management structure were essential. Its report offered multiple recommendations
for improving the IPCC procedures. These recommendations primarily related to governance and
management; the review process; and characterizing and communicating uncertainty,
communications, and transparency in the assessment process.
EPA evaluated the concerns related to the CRU/HadCRUT13 temperature record as part of a two-
step process. The first step of EPA's evaluation was after the public comment period for its
proposed findings ended on June 23, 2009, and the second step was in response to 10 petitions
for reconsideration that EPA received after it issued its December 2009 final findings. EPA did
not provide the OIG with documentation for either step that showed it had independently verified
the temperature records for CRU, NOAA, or NASA. Instead, EPA relied on external peer review
and investigations of the datasets, as well as the larger body of scientific evidence, to ensure that
the data met federal and Agency information quality guidelines. However, EPA also reviewed
CRU e-mails, as well as underlying scientific literature, in addressing petitioner concerns related
to the CRU/HadCRUT temperature record.
The Agency's determination that the temperature records (from CRU, NOAA, and NASA) were
independent and valid, and that they were properly considered in the IPCC assessments, was
largely based on the following factors:
The temperature datasets had been subject to formal, independent, external peer
review.
The temperature records were part of a large body of scientific evidence that included
multiple lines of evidence showing a warming of the climate system.
The temperature records were considered by IPCC as "one line of evidence among
many" that presented a broad range of indicators that global warming was occurring.
An independent investigation issued in March 2010 by the UK House of Commons
Science and Technology Committee verified the CRU data and results and found that
the NOAA, NASA, and CRU datasets were independent.
To obtain an expert opinion on the relationship between NOAA temperature datasets and other
key datasets used in IPCC's AR4, in July 2010, we discussed these records with the Director of
NOAA's National Climatic Data Center (NCDC). According to the NCDC Director, while
NCDC, NASA, and CRU each employ different methodologies to analyze the raw temperature
13 HadCRUT is a global surface dataset maintained by the United Kingdom's Hadley Centre and CRU.
11-P-0702
39
-------
data they collect, the long-term temperature trend analyses produced by all three organizations
generally agree with each other. Further, he told us that IPCC recognized that the temperature
datasets generated by NCDC, NASA, and CRU were all quite important in demonstrating the
robustness of data for the temperature trends discussed in IPCC's AR4. The NCDC Director said
that even if the CRU temperature data were completely disregarded, the NCDC and NASA
temperature datasets, along with other indicators such as sea level data, sea ice melting, ice sheet
melting, subsurface ocean temperatures, lake and river freeze dates, glacier retreat, and
biological indicators, were capable of demonstrating that the weight of scientific evidence
overwhelmingly pointed toward a long-term trend of global warming.
Information quality guidelines from both OMB and EPA state that if data and analytical results
are subject to formal, independent, external review, then the information may generally be
presumed to be of acceptable objectivity. The presumption of objectivity ensured by peer review
is rebuttable, but the burden of proof lies with the complainant or petitioner. According to EPA,
the CRU, NOAA, and NASA datasets have all been widely reviewed and assessed within the
climate change community, thus ensuring objectivity. In its response to 10 petitions for
reconsideration of the endangerment finding, EPA concluded that "petitioners have failed to
consider or rebut" the body of evidence (which includes the temperature records) supporting the
IPCC and USGCRP conclusions that warming of the climate system in recent decades is
unequivocal.
11-P-0702
40
-------
Appendix B
Timeline of Key Events Leading to
Issuance of Endangerment Finding
Date
Event
10/20/99
Rulemaking petition filed asking EPA to regulate greenhouse gas emissions from new motor vehicles.
09/08/03
EPA denied rulemaking petition.
04/02/07
Supreme Court ruled that greenhouse gases are air pollutants under CAA, and that EPA must
determine whether greenhouse gases emitted from new motor vehicles cause or contribute to air
pollution, which may reasonably be anticipated to endanger public health and welfare.
05/14/07
President Bush issued EO 13432 on Cooperation Among Agencies in Protecting the Environment
With Respect to Greenhouse Gas Emissions From Motor Vehicles, Nonroad Vehicles, and Nonroad
Engines.
05/18/07
Acting Assistant Administrator for OAR briefed on the plan for developing the endangerment finding.
06/07
Office of Transportation and Air Quality workgroup formed to develop light-duty vehicles and
transportation fuels greenhouse gas emissions rule. Office of Atmospheric Programs tasked with
developing the endangerment finding portion of the rule.
06/01/07
Acting Assistant Administrator for OAR briefed on plan for developing the endangerment finding.
06/01/07
Administrator Johnson briefed on the approach and process for the endangerment finding.
06/18/07
Acting Assistant Administrator for OAR briefed on EPA's plan for developing the endangerment
finding.
06/27/07
Other federal agencies briefed on approach for developing TSD and findings.
07/05/07
Acting Assistant Administrator for OAR briefed on endangerment finding progress.
07/20/07
Acting Assistant Administrator for OAR given early guidance prebrief.
07/25/07
Administrator Johnson briefed on early guidance.
08/08/07
EPA senior management given update briefing.
08/08/07
Draft TSD sent to workgroup members for review.
08/08/07
Other federal agencies briefed on EPA's plans for the endangerment finding.
08/08/07
Administrator Johnson's early guidance memo distributed to workgroup.
08/16/07
Draft TSD sent to federal climate change experts for review.
08/28/07
Draft TSD sent to workgroup members for review.
09/25/07
Endangerment finding status update provided to Acting Assistant Administrator for OAR.
10/04/07
Final options selection meeting held with Acting Assistant Administrator for OAR.
10/22/07
Administrator Johnson briefed on endangerment finding.
11/08/07
Administrator Johnson briefed on endangerment finding.
12/04/07
Final Agency review meeting held for the endangerment finding preamble portion of the Office of
Transportation and Air Quality rulemaking.
12/05/07
Draft proposed rule sent to OMB. (Rule never reviewed by OMB because new corporate average fuel
economy standards were established in late 2007.)
07/11/08
EPA signed ANPR for regulating greenhouse gas emissions under CAA (published at 73 Federal
Register 44354, July 30, 2008).
01/27/09
Senior Climate Policy Counsel to Administrator briefed.
02/04/09
Administrator Jackson briefed.
11-P-0702
41
-------
Date
Event
02/27/09
Regulatory Steering Committee approved tiering of action.
03/03/09
Endangerment finding workgroup held its first meeting.
03/05/09
Draft options selection briefing sent to EPA workgroup.
03/06/09
Administrator Jackson briefed on options selection.
03/09/09
Draft TSD sent to workgroup members and federal climate change experts for review.
03/11/09
Workgroup meeting held.
03/16/09
OMB briefed.
03/18/09
Final Agency review meeting held for draft proposed endangerment finding.
03/20/09
Draft proposed endangerment finding and TSD sent to OMB for review.
03/27/09
EPA Office of Policy, Economics and Innovation approved tiering.
04/07/09
OMB sent comments on draft proposed findings and TSD to EPA.
04/08/09
Revised draft proposed findings sent to OMB for review.
04/09/09
Revised draft TSD sent to OMB.
04/10/09
OMB sent more comments on draft TSD and proposed draft findings.
04/13/09
OMB signed off on final proposal.
04/14/09
Revised draft TSD sent to OMB for review.
04/17/09
Administrator Jackson signed proposed rule.
05/18/09
Public hearing held in Arlington, Virginia.
05/21/09
Public hearing held in Seattle, Washington.
06/23/09
60-day public comment period on proposed findings ends.
10/07/09
Workgroup meeting held.
10/09/09
Draft final TSD sent to federal climate change experts for review.
10/13/09
and
10/16/09
Draft TSD made available to workgroup members for review.
10/16/09
Workgroup meeting held.
10/26/09
Workgroup meeting held.
10/29/09
Options selection meeting held with Administrator Jackson.
11/02/09
Draft final findings sent to Assistant Administrators of workgroup members for review.
11/05/09
Final Agency review held for draft final findings.
11/06/09
Draft final findings and TSD sent to OMB for review.
11/25/09
and
11/30/09
OMB sent comments on draft final findings to EPA. (No comments on the TSD.)
12/03/09
Revised draft final findings sent to OMB for final review.
12/07/09
Final findings signed by Administrator Jackson.
11-P-0702
42
-------
Appendix C
Details on Scope and Methodology
Our evaluation focused on the processes employed by EPA in developing EPA's Endangerment
and Cause or Contribute Findings for Greenhouse Gases Under Section 202(a) of the CAA.
Based on a request from the Ranking Member of the United States Senate Committee on
Environment and Public Works, our primary objective was to determine whether EPA followed
key federal and Agency regulations and policies in obtaining, developing, and reviewing the
technical data used to support its endangerment finding. To assess EPA's compliance with key
federal and Agency regulations, policies, and guidance governing information quality, we
reviewed the following documents:
Data Quality Act, also known as the Information Quality Act, as provided under
Section 515 of the Treasury and General Government Appropriations Act for Fiscal
Year 2001 (Public Law 106-554, December 2000).
Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and
Integrity of Information Disseminated by Federal Agencies, OMB, January 3, 2002
(as amended February 22, 2002).
OMB's Final Information Quality Bulletin for Peer Review, December 16, 2004.
Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and
Integrity of Information Disseminated by the Environmental Protection Agency,
EPA/260R-02-008, October 2002.
Action Development Process: Guidance for EPA Staff on Developing Quality Actions,
Office of Policy, Economics, and Innovation Regulatory Development Series, EPA,
June 30, 2004.
Peer Review and Peer Involvement at the U.S. Environmental Protection Agency,
EPA's peer review policy, EPA Administrator, January 31, 2006.
U.S. Environmental Protection Agency Peer Review Handbook, 3rd Edition,
EPA/100/B-06/002, Science Policy Council, EPA, 2006.
Assessment Factors: A Summary of General Assessment Factors for Evaluating the
Quality of Scientific and Technical Information, EPA 100/B-03/001, Science Policy
Council, EPA, June 2003.
Additionally, we reviewed the legislative history of the CAA to understand the congressional
intent of the statute with regard to a finding of endangerment.
We reviewed the following EPA documents to determine whether the Agency followed the
information quality guidelines provided above:
Action development package, including the following documents:
o Action initiation (tiering) form
o Documentation of EPA's early guidance on the preliminary analytic blueprint
o Options selection briefing package
o Documentation of the options selection briefing meeting
o Documentation summarizing the results of the final Agency review meeting
11-P-0702
43
-------
Documentation of any EPA reviews of IPCC's reporting process
Copies of any external peer review reports, including EPA responses to peer review
comments
Names and affiliations for the 12 federal climate change experts involved in the
review of the TSD
Comments received from the 12 federal climate change experts and EPA's response
to the comments
Rule and Policy Information Development System (RAPIDS) data for the action,
including the list of workgroup members
Documentation of analyses EPA has conducted to conclude that the use of HadCRUT
temperature data does not impact the fundamental findings of the IPCC report and
EPA's endangerment finding
Draft versions of TSD submitted to OMB for review, and OMB's comments
Comments received from other federal agencies during interagency review
We also requested the preliminary and detailed analytic blueprints for the endangerment finding,
as well as workgroup meeting minutes, from the Agency. However, the Agency did not create an
analytic blueprint for the stand-alone action, nor maintain documentation of workgroup meeting
minutes.
We also reviewed EPA's proposed and final endangerment findings, its TSD, EPA's response to
public comments on the proposed rule (EPA's response to comment document), and EPA's
response to petitions for reconsideration.
To evaluate EPA's review of its TSD, we reviewed multiple draft versions of the TSD, as well as
federal expert reviewer comments, and compared the process followed to the guidelines provided
in EPA's Peer Review Handbook and OMB' s Final Information Quality Bulletin for Peer
Review. We assessed the OMB/interagency review process by examining information contained
in the EPA docket, and requested additional documentation from EPA regarding interagency
comments for the TSD accompanying the ANPR, the proposed rule, and the final rule. We also
asked OMB to clarify whether a document summarizing existing findings and conclusions of
other peer-reviewed scientific assessments, but not offering any new analyses or conclusions,
would meet OMB's definition of a scientific assessment (see appendix D).
In addition to key federal and Agency requirements for information quality, we also reviewed
EPA's compliance with the following statutes and EOs governing the regulatory development
process:
Administrative Procedure Act (5 USC ง551 et seq. (1946))
Congressional Review Act (5 USC ง801 et seq. (1996))
National Technology Transfer and Advancement Act (15 U.S.C. ง3701 et seq.
(1996))
Paperwork Reduction Act (44 USC ง3501 et seq. (1980))
Regulatory Flexibility Act as amended by the Small Business Regulatory
Enforcement Fairness Act (5 USC ง601 et seq. (1980))
Unfunded Mandates Reform Act (2 USC ง658 et seq. and 2 USC 1501 et seq. (1995))
11-P-0702
44
-------
EO 12866: Regulatory Planning and Review, September 30, 1993
EO 12898: Federal Actions to Address Environmental Justice in Minority
Populations and Low-Income Populations, February 11, 1994
EO 13 045: Protection of Children from Environmental Health Risks and Safety Risks,
April 21, 1997
EO 13132: Federalism, August 4, 1999
EO 13175: Consultation and Coordination with Indian Tribal Governments,
November 6, 2000
EO 13211: Actions Concerning Regulations that Significantly Affect Energy Supply,
Distribution, or Use, May 18, 2001
EO 13272: Proper Consideration of Small Entities in Agency Rulemaking, August 13,
2002
We also conducted numerous interviews with EPA and non-EPA personnel to corroborate and
examine further the process EPA used to make and support the endangerment finding.
Review of Management (Internal) Controls
Generally accepted government auditing standards require that auditors obtain an understanding
of internal controls significant to the audit objectives and consider whether specific internal
control procedures have been properly designed and placed in operation. We examined
management and internal controls as they related to our objective. We reviewed the federal
statutes and guidance related to information quality, and focused on EPA's adherence to, and
implementation of, those statutes and guidance. We tested whether EPA's internal policies and
procedures were applied properly during the development of its endangerment finding, and how
they were applied. In general, we examined whether management controls were effective
through document review and analysis, corroborated with testimonial evidence. In cases where
documentation was lacking, we reviewed procedures to determine whether design or
implementation problems existed. The recommendations in our report reflect alternative actions
or additional steps that could be taken to improve the management control weaknesses we found.
Prior Reports
There were no previous EPA OIG or U.S. Government Accountability Office evaluations or
audits applicable to this evaluation.
11-P-0702
45
-------
Appendix D
OIG Request to OMB for Clarification of the
Definition of a Highly Influential Scientific Assessment
The following questions were submitted to OMB in an August 11, 2010, e-mail to the OMB
Assistant General Counsel.
This e-mail is a follow-up to your prior communication with [name] of EPA's Office of
Inspector General (OIG). As [name] may have expressed to you, the EPA OIG is currently
evaluating EPA's development of its endangerment and cause or contribute findings for
greenhouse gases. Our objective is to determine whether EPA followed key federal and Agency
regulations and policies in developing and reviewing the technical data used to support and make
its endangerment finding. This evaluation was initiated based on a request from Senator James
M. Inhofe, Ranking Member, Senate Committee on Environment and Public Works. Attached
below are copies of the Senator's request letter and the OIG's notification memo to EPA's
Assistant Administrator for Air and Radiation.
In order to assess the process that EPA used, we are interested in discussing certain criteria
issued by OMB. Specifically, we would like input from your office to address the following
questions:
1. In order to ensure that information meets the basic definition of "quality" provided in
OMB's Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and
Integrity of Information Disseminated by Federal Agencies and OMB's Final Information
Quality Bulletin for Peer Review,
a. To what extent should a Federal agency review another organization's data
quality and peer review processes (e.g., document all processes, document and
test processes, test selected processes, etc.) before disseminating information from
a peer reviewed scientific assessment published by that organization?
b. If such a scientific assessment has been reviewed and endorsed by the U.S.
Government, what impact does this have on the level of review (e.g., less review,
same level of review, no review, etc,) needed by a Federal Agency to ensure the
information in the assessment meets OMB's basic definition of "quality"?
2. OMB's Final Information Quality Bulletin for Peer Review defines a scientific assessment
as "...an evaluation of a body of scientific or technical knowledge, which typically
synthesizes multiple factual inputs, data, models, assumptions, and/ or applies best
professional judgment to bridge uncertainties in the available information." Is a document
that summarizes the results and conclusions of other peer reviewed scientific assessments,
but offers no new analysis or conclusions, considered a scientific assessment according to
OMB's definition?
11-P-0702
46
-------
We would be pleased to meet with you to discuss the above questions at your earliest
convenience. You may contact me at the number below. Alternatively, we can accept written
responses to our questions if a meeting is not feasible. In this event, we request written input
from your office on the questions above by Friday, August 20, 2010.
If you have any questions or concerns about our request, please feel free to contact me.
11-P-0702
-------
Appendix E
OMB Clarification of the Definition of a
Highly Influential Scientific Assessment
The following information was provided to the OIG in a September 10, 2010, e-mail from the
OMB Assistant General Counsel in response to questions the OIG submitted on August 11, 2010.
1. In order to ensure that information meets the basic definition of "quality" provided in OMB's
Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of
Information Disseminated by Federal Agencies and OMB's Final Information Quality Bulletin
for Peer Review,
a. To what extent should a Federal agency review another organization's data quality and peer
review processes (e.g., document all processes, document and test processes, test selected
processes, etc.) before disseminating information from a peer reviewed scientific assessment
published by that organization?
"If an agency uses another organization's data or analysis to support their policy, they are
disseminating that information. As such, that information becomes subject to the Agency's
Information Quality Guidelines and the Bulletin for Peer Review.
Therefore, in evaluating whether to disseminate the information, the agency must determine
whether the information complies with the Agency's Information Quality Guidelines. OMB
recognizes that information quality can be costly and encourages agencies to consider the
social value of better information in different contexts. OMB's guidelines recognize that
some government information may need to meet higher or more specific standards than
would apply to other types of government information. OMB's guidelines encourage
agencies to weigh the costs and benefits of higher quality information; the more important
the information, the higher the quality standards to which it should be held. Information that
is most likely to have influence on important public and private sector decisions requires a
higher level of quality1.
The second step is for the agency to determine whether a specific piece of information is
subject to the Bulletin for Peer Review. Not every article cited in a preamble or risk
assessment, for instance, is subject to the Bulletin. Only scientific information that is
important to the conclusion being drawn by the agency is subject to the Bulletin. For
information that is subject to the Bulletin, the agency must either conduct a new peer review
of that information or determine that the prior peer review meets the requirements of the
Bulletin for either a peer review or alternative process. When the Agency itself has reviewed
or otherwise synthesized or evaluated a body of peer reviewed literature, the Agency's
assessment would be subject to the Bulletin if it is deems to be 'influential scientific
information.'
11-P-0702
48
-------
Finally, in the case of EPA, we note that in 2003, EPA released A Summary of General
Assessment Factors for Evaluating the Quality of Scientific and Technical Information" (see:
http://www.epa.gov/OSA/spc/pdfs/assess2.pdf). The EPA document states: "The Agency
believes that the summary of general assessment factors provided in this document will serve
to increase the extent to which the information-generating public builds quality
considerations into the generation and documentation of their information products. The
Agency expects that the resulting improvements in the quality of such information will
enable the Agency to more fully utilize and disseminate such information."
b. If such a scientific assessment has been reviewed and endorsed by the U.S. Government, what
impact does this have on the level of review (e.g., less review, same level of review, no review,
etc,) needed by a Federal Agency to ensure the information in the assessment meets OMB's basic
definition of "quality"?
"If a scientific assessment is deemed subject to the Bulletin and has been reviewed by the
agency pursuant to the Bulletin, no additional peer review is needed. If a scientific
assessment was created by and peer reviewed by a different agency of the U.S. Government
than the agency that is using it in its policy, the agency using the document in its policy is
responsible for determining whether the prior review was consistent with the level of rigor
necessary for the use to which the assessment will now be put. Similarly, if an assessment
has been endorsed by one agency within the U.S. Government, and is going to be used by
another agency, the agency using the information should ensure that the assessment is
consistent with its own Information Quality Guidelines."
2. OMB's Final Information Quality Bulletin for Peer Review defines a scientific assessment as
"...an evaluation of a body of scientific or technical knowledge, which typically synthesizes
multiple factual inputs, data, models, assumptions, and/ or applies best professional judgment to
bridge uncertainties in the available information." Is a document that summarizes the results and
conclusions of other peer reviewed scientific assessments, but offers no new analysis or
conclusions, considered a scientific assessment according to OMB's definition?
"An annotated bibliography would generally not be considered a scientific assessment;
however, a document summarizing the 'state of the science' would be, as it implicitly or
explicitly weighs the strength of the available evidence."
'Per OMB's government-wide guidelines, "Influential" information is subject to higher standards of quality.
"Influential" means that information that the agency can reasonably determine that dissemination of the information
will have or does have a clear and substantial impact on important public policies or important private sector
decisions. Each agency is authorized to define "influential" in ways appropriate for it given the nature and
multiplicity of issues for which the agency is responsible. With several important exceptions and qualifications (e.g.,
privacy, intellectual property rights, and other confidentiality protections) influential information should be
reproducible by qualified third parties.
11-P-0702
49
-------
Appendix F
OMB's Views on the Application of OMB's
Peer Review Bulletin and Information Quality
Guidelines to EPA's TSD for the
EPA Endangerment Finding
The following information was provided to the OIG in an April 15, 2011, e-mail from the OMB
Deputy General Counsel.
We understand that your office authorized EPA to share with OMB information from a
preliminary draft of your report on this matter, and we accordingly reached out to your office to
discuss this matter further. Thus, in my email to [OIG staff] (separately attached), I had
explained that the questions that your office had posed to OMB last August were of a general
nature concerning OMB's Information Quality Guidelines and Peer Review Bulletin, and that
OMB's responses accordingly did not express any views regarding the application of the OMB
Guidelines and/or the Bulletin to any particular agency document. In his email to me (below),
[the OIG Assistant Inspector General for Program Evaluation] had indicated that it would be
appropriate for OMB to provide EPA with OMB's views regarding the application of the OMB
Bulletin and Guidelines to a particular EPA document: namely, the Technical Support Document
that EPA issued in connection with its Endangerment Findings (the "Endangerment TSD").
Consistent with that approach, OMB provided EPA with OMB's views regarding the application
of the Bulletin and Guidelines to the Endangerment TSD, and it is OMB's understanding that
EPA communicated OMB's views to your office. OMB would further like to take this
opportunity to provide directly to you OMB's views on this particular matter.
As communicated to EPA several weeks ago, OMB's views are as follows.
OMB's Peer Review Bulletin grants agencies "broad discretion to weigh the benefits and
costs of using a particular peer review mechanism for a specific information product."
With respect to the relationship between the Endangerment TSD and OMB's Bulletin, the
threshold question is whether EPA should have determined that the TSD met the
definition of a "highly influential scientific assessment." If so, the Endangerment TSD
would have been subject to the stricter minimum requirements for peer review contained
in Section III of the OMB Bulletin.
OMB believes that EPA reasonably determined that the Endangerment TSD itself (as
opposed to the underlying peer-reviewed scientific assessments of the NRC, IPCC,
USGRCP identified and discussed in the TSD) did not have the impacts or characteristics
required to meet the OMB Bulletin's definition of a highly influential scientific
assessment. In making its endangerment findings, EPA noted that the underlying
"scientific assessments of the IPCC,USGRCP, and the NRC represent the best reference
materials" on which it was prepared to rely; that it had determined not "to perform a new
11-P-0702
50
-------
and independent assessment of all of the underlying climate change science"; and that it
had "no reason to believe that putting this significant body of work aside and attempting
to develop a new and separate assessment would provide any better basis for making the
endangerment decision." Thus, it was not the Endangerment TSD prepared by EPA itself
but the conclusions of these three underlying peer-reviewed scientific assessments that
informed the agency's decision.
If the document does not meet the definition of a "highly influential scientific
assessment," but meets the definition of "influential scientific information," then it is
subject to more discretionary requirements contained in Section II of the OMB Bulletin.
OMB believes that the EPA complied with those requirements. More specifically, the
process through which EPA reviewed this document was a permissible exercise of the
discretion afforded agencies under Section II of the OMB Bulletin, which provides that
for such information "agencies need not have further peer review conducted on
information that has already been subjected to adequate peer review." Pages 2 through 8
of the Endangerment TSD transparently discuss the extent to which EPA relied on
information that has already been subjected to such peer review. The same section
discusses the process by which EPA subjected the document to public comment, a
technical review by federal climate change experts, and an interagency review. Finally,
EPA acknowledged that the agency must ensure consistency with its own Information
Quality Guidelines, and openly discussed how it prepared the document to be consistent
with those guidelines.
We would appreciate if your office would take into consideration, during its review, these OMB
views regarding the application of OMB's Bulletin and Guidelines in this particular context. As
[the OIG Assistant Inspector General for Program Evaluation] indicated in his email to me, we
would further appreciate the opportunity to review subsequent iterations of your office's
description of OMB's views with respect to this matter, so that we may provide further input as
needed.
11-P-0702
51
-------
Appendix G
Agency Comments on Draft Report and
OIG Evaluation of Agency Comments
MEMORANDUM
June 17.2011
SUBJECT: U. S. Environmental Protection Agency (EPA) Response to Draft OIG Report
Dated May 3, 2011, Procedural Review of Greenhouse Gases Endangerment
Finding Data Quality Processes, Project No. OPE-FY10-0017
FROM: Gina McCarthy
Assistant Administrator for Air and Radiation
Paul Anastas, PhD
Assistant Administrator for Research and Development
Malco lm D. Jackson
Assistant Administrator for Environmental Information and Chief Information
Officer
TO: Wade T. Najjum
Assistant Inspector General for Program Evaluation
EPA appreciates the opportunity to comment on the Office of the Inspector General (OIG) draft
report, Procedural Review of Greenhouse Gases Endangerment Finding Data Quality Processes,
dated May 3, 2011.
The endangerment finding that greenhouse gases in the atmosphere threaten public health and
welfare is based on the best available peer-reviewed science, and we appreciate OIG's statement
that this report does not purport to evaluate the quality of the underlying science supporting the
endangerment finding. The scope of OIG's analysis should be highlighted, as this report
addresses only procedural issues, and none of these procedural issues relate to the body of
science that formed the basis for the Administrator's decision. Although the draft report states
that the OIG did not assess the quality of the scientific information and data EPA used to support
the endangerment finding, we remain concerned about the potential for this report to mislead
readers about the scientific content underlying EPA's greenhouse gas endangerment finding.
Our comments here and in the enclosed attachment address six major concerns that we believe
require your attention and consideration before issuance of the final report.
11-P-0702
52
-------
First, we appreciate the efforts that the OIG report makes to provide context. We believe
additional context will provide the reader a more balanced picture. The report should clearly
state that:
All of the science used to support the endangerment finding is from peer-reviewed scientific
assessments;
OIG Response 1: Our report acknowledges that EPA relied on the major assessments of USGCRP, IPCC, and
NRC as the primary scientific basis for its endangerment finding, and describes the peer review processes for
these assessments. However, EPA also used sources other than scientific assessments to support its
endangerment finding. Thus we have not included this statement in our final report. See OIG Response 12,
below, for further details.
These scientific assessments utilized by EPA, namely those from the National Research
Council (NRC), the Intergovernmental Panel on Climate Change (IPCC), and U.S. Global
Change Research Program (USGRCP), underwent their own peer-review processes that are
well known and accepted by the U.S. Government;
OIG Response 2: We added descriptions of the peer review procedures of USGCRP, IPCC, and NRC to
chapter 1 of the final report. EPA states that the peer review processes of each of the organizations mentioned
are "accepted by the U.S. Government." However, the only organization for which OMB guidance specifically
allows federal agencies to presume findings and conclusions to be adequately peer reviewed is NAS.
As noted in chapter 3 of our report, with respect to accepting and disseminating data produced by other
organizations, OMB, in an e-mail to us from the Assistant General Counsel, told us:
If an agency uses another organization's data or analysis to support their policy, they are
disseminating that information. As such, that information becomes subject to the Agency's
Information Quality Guidelines and the Bulletin for Peer Review.
Therefore, in evaluating whether to disseminate the information, the agency must determine
whether the information complies with the Agency's Information Quality Guidelines.
and
If an assessment has been endorsed by one agency within the U.S. Government, and is going
to be used by another agency, the agency using the information should ensure that the
assessment is consistent with its own Information Quality Guidelines.
The scientific assessments of the NRC, IPCC, and the USGRCP, not EPA's technical support
document (TSD), served as the scientific underpinnings of the endangerment finding; and
OIG Response 3: We agree that the scientific assessments of NRC, IPCC, and USGRCP served as the
scientific underpinnings of the endangerment finding. However, the document that EPA created to present this
information in support of its endangerment finding was the TSD. We believe that EPA's comments to our draft
report understate the role of the TSD in the overall development of its endangerment finding. While the TSD
heavily relied on information from the major assessment reports of USGCRP, IPCC, and NRC, the TSD itself
played a role in informing the Administrator's endangerment decision. For example, the former Director of EPA's
Climate Change Division told us in an August 2010 e-mail that former EPA Administrator Johnson reviewed the
entire TSD in 2007 and found the science in it to be the basis for his core decision that there was endangerment.
Further, a briefing document provided to EPA Administrator Jackson in February 2009 stated that the TSD
"became the basis for discussions and eventually December 2007 proposal." Also, in the endangerment finding
itself, EPA described the April 2009 TSD as the "underlying scientific and technical basis for the Administrator's
Proposed Findings. . . ."
11-P-0702
53
-------
The TSD summarized material that had already undergone peer review, and was itself subject
to extensive expert, internal EPA, interagency, and public reviews.
OIG Response 4: In chapter 2 of our report (p. 16) we note that EPA obtained reviews of the draft TSDs by
federal experts, provided a predissemination review for other federal agencies, and addressed comments
received on the TSD from workgroup members, federal experts, OMB, and other agencies. Further discussion of
the review process that EPA used for its TSD is also provided in our answers to questions 4, 5, and 6 in
appendix A of the final report.
Further discussion regarding the review processes utilized by the assessment reports can be
found in section A. 1 of the attachment.
Second, EPA does not concur with the OIG's opinion that EPA's TSD is a "highly influential
scientific assessment" (HISA) according to the OMB Peer Review Bulletin. The TSD is
"influential scientific information" (ISI), and OMB, the agency charged with administering the
Data Quality Act (DQA), agrees. As shown in Appendix G of the draft OIG report, OMB
believes that EPA reasonably determined that the TSD itself (as opposed to the underlying peer-
reviewed scientific assessments of the NRC, IPCC, and USGRCP summarized in the TSD) did
not have the impacts or characteristics required to meet the OMB Peer Review Bulletin's
definition of a HISA. The OIG has mistakenly concluded that EPA's TSD is a HISA. A
"scientific assessment" (a prerequisite for being a HISA) is defined in OMB's Peer Review
Bulletin as "an evaluation of a body of scientific or technical knowledge, which typically
synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional
judgment to bridge uncertainties in the available information." The TSD did not conduct such an
evaluation. No weighing of information, data and studies occurred in the TSD. That had already
occurred in the underlying assessments, where the scientific synthesis occurred and where the
state of the science was assessed. The TSD is not a scientific assessment, but rather summarized
in a straightforward manner the key findings of the NRC, the USGCRP and IPCC. EPA is
confident that a comprehensive review of this issue leads to the conclusion that the TSD is not a
HISA, but rather ISI.
11-P-0702
54
-------
OIG Response 5: In our opinion, the TSD meets the definition of a scientific assessment provided in the OMB
peer review bulletin. The TSD synthesized multiple factual inputs, data, models, and assumptions. The primary
inputs to the TSD were the three major assessment reportsIPCC, NRC, and USGCRPbut EPA
synthesized the data, models, risk assumptions, and the general findings and conclusions from these
underlying reports, as well as other information from other sources, into a single documentits TSD.
While information from the "major scientific assessments" may have formed the foundation of the Agency's
endangerment finding, it should be noted that the Agency stated in its endangerment finding that it "gave
careful consideration to all the scientific and technical information in the record." EPA's TSD referenced
multiple sources (some cited within the assessment reports, and some not), including "up-to-date" data from
sources other than the "major scientific assessments." In evaluating the scientific information, the Agency
stated that it "placed limited weight on the much smaller number of individual studies that were not considered
or reflected in the major assessments." EPA reviewed such studies "largely to see if they would lead EPA to
change or place less weight on the judgments reflected in the assessment report." The Agency stated in the
endangerment finding that "the studies did not change the various conclusions or judgments EPA would draw
based on the assessment reports."
Descriptions in the endangerment finding and TSD show that the Agency placed greater emphasis on certain
sources of information. EPA synthesized conclusions and scientific findings from various assessment reports,
and other sources, in a single document that had as a stated purpose "to provide scientific and technical
information for an endangerment and cause or contribute analysis regarding greenhouse gas (GHG)
emissions." Therefore, we continue to believe that EPA's TSD meets OMB's definition of a "highly influential
scientific assessment," thus requiring the Agency to have met all applicable OMB peer review requirements for
such types of information.
The Agency's response states that OMB agreed with EPA that the TSD was influential scientific information
(but not a highly influential scientific assessment). We do not agree with this characterization of OMB's
response. OMB stated that it believed EPA reasonably interpreted OMB guidance in determining that the TSD
was not a highly influential scientific assessment. OMB's response does not specifically provide OMB's opinion
on whether the TSD is influential scientific information or a highly influential scientific assessment. Instead,
OMB opines on the reasonableness of the EPA's determination. We also note that EPA did not categorize the
TSD as influential scientific information during the action development process. EPA first described the TSD as
influential scientific information when it responded to the OIG draft report.
Furthermore, the OIG should clearly indicate that agencies are given discretion to make
determinations as to the level of review required. EPA went far beyond the requirements for peer
review of an ISI by conducting multiple expert, interagency and public reviews that were
reasonable and appropriate to ensure the credibility of the TSD. OMB also states that EPA
satisfied all peer review requirements for ISIs set out in the OMB Peer Review Bulletin.
OIG Response 6: Our draft report specifically stated that "the methodology that OAR employed for this review
was within the discretion afforded the Agency for peer reviews of influential scientific information." However, as
the draft report also stated in appendix B, "EPA did not conduct a peer review of the TSD that met all
recommended (emphasis added) steps in the Peer Review Handbook for peer reviews of influential scientific
information or highly influential scientific assessments." We believe our draft report to be accurate in these
statements, and thus made no further change on this matter in the final report. See OIG Response 55 for
further details.
Even assuming OIG continues in its current view that the TSD is a HISA, the discussion in the
draft OIG report indicates that OIG believes EPA failed to satisfy, in part, one of the several peer
review requirements for a HISA. If OIG continues with that view, then the final OIG report
should indicate that EPA did not consider the TSD to be a HISA, but OIG believes the extensive
peer review conducted by EPA satisfied all but one of the OMB peer review requirements for
HISAs, and partially satisfied that one requirement. There are substantial and reasonable grounds
11-P-0702
55
-------
for EPA's and OMB's views that the TSD is not a HISA, and OIG should at the most conclude
that in its view the TSD may or may not be a HISA. OIG needs to state its conclusion in a
manner that is accurate and will avoid misrepresentation. See section A.2 of the attachment for
more information.
OIG Response 7: OIG Responses 28 through 33 provide specific details regarding how EPA's review of its TSD
compared to each of the OMB peer review requirements for highly influential scientific assessments.
Third, EPA does not concur with the OIG statements that the Agency did not conduct any
independent evaluations of IPC C's compliance with IPCC procedures, nor did EPA document
any specific processes it employed to evaluate the scientific and technical information included
in IPCC's Fourth Assessment Report. EPA thoroughly documented the peer review processes
used by the organizations (e.g., IPCC, NRC, and USGCRP) that issued the underlying scientific
assessments. This information is in the docket for the endangerment finding itself and available
publically. These processes are well known and accepted by the U.S. government as appropriate
mechanisms to ensure the quality of the science assessments. Regarding IPCC, EPA devoted
significant staff time to review the IPCC assessment reports, provided comments during an
interagency process, participated in that interagency process to prioritize U.S. government
comments, and approved the summaries of the IPCC reports in a detailed line-by-line process.
The OIG draft report does not include the examples where EPA specifically discussed and
documented throughout the endangerment finding record how the IPCC peer review procedures
meet EPA and OMB guidelines. See section A. 3 of the attachment for more information.
OIG Response 8: We added statements to chapter 1, chapter 3, and appendix A of the final report to indicate
EPA's involvement in the IPCC process, including its participation in the U.S. delegation approving the Working
Group II report for the AR4. Further analysis of EPA comments regarding its analysis and documentation of
IPCC procedures is provided in OIG Responses 46 through 52.
Fourth, EPA identified the specific type of information supporting the endangerment finding.
The final endangerment finding devotes an entire section to "The Science On Which the
Decisions are Based" (Section III. A).1 There, after discussing why EPA chose to primarily rely
on the major scientific assessments of USGCRP, IPCC and NRC to inform the Administrator's
judgment, EPA states that these assessments "maintain the highest level of adherence to Agency
and OMB guidelines for data and scientific integrity and transparency" (74 FR 66511). The full
endangerment record, including the findings as well as the TSD and the Response to Comments,
provide extensive discussions regarding the type of information supporting the endangerment
finding. See section A.4 of the attachment for additional information.
OIG Response 9: We revised the title of this section to more accurately reflect our findings. Our finding was not
that EPA failed to indentify the sources of information it used to support its endangerment finding. Rather, we
found the Agency did not follow OMB and EPA requirements to discuss such information in terms of whether it
was influential scientific information or highly influential scientific assessments. Further, EPA did not certify in the
administrative record for the endangerment finding that its review of the TSD met applicable OMB peer review
requirements. Further analysis regarding EPA's comments related to this finding is provided in OIG
Response 42.
1 Endangerment finding available at http://epa.gov/climatechange/endangennent.html.
11-P-0702
56
-------
Fifth, EPA followed its Action Development Process. In 2007, EPA's Office of Transportation
and Air Quality initiated a rulemaking to determine whether greenhouse gas emissions from
sources covered under Section 202(a) of the Clean Air Act cause or contribute to air pollution
that endangers public health or welfare, and if so, to set new standards for certain motor vehicles.
As part of this effort, EPA held an early guidance meeting and outlined a "plan for developing
the Endangerment Finding" Management briefings (provided to OIG) regarding the
development plan communicated the topics covered in an Analytic Blueprint, including the
approach, scope, underlying science, and review mechanisms for the TSD and endangerment
finding process. In 2009, EPA management separated the elements of the original action into
separate actions. Hence, the 2009 action was not a new action that required the workgroup to -
start over. In 2009, EPA management chose to proceed using the same approach for the TSD
identified in 2007. This is a common agency practice. However, the draft OIG report makes no
attempt to take into account the flexibility and discretion provided by the Agency guidance itself.
Please see section A. 5 of the attachment for more information.
OIG Response 10: We believe that the report provides sufficient context for how EPA developed this action.
See OIG Responses 43, 44, and 45 for our analysis of specific EPA comments related to our evaluation of
EPA's action development process for its endangerment finding.
We have serious concerns about the information you have included in Appendix B of your draft
report. This is where you chose to include your answers to Senator Inhofe's questions. It appears
as though the responses may have been developed prior to the release of your preliminary
position paper. We ask that your responses to the Senator align with the information included in
the body of your report. See additional information in section B of the attachment.
OIG Response 11: We believe that our responses to the Senator's questions align with the information included
in the body of our report. Where we deemed appropriate, we made minor changes to the report to include
additional context in addressing the Senator's questions. Any instances where we made changes to our report
based on EPA comments are noted in OIG Responses 46 through 60 in our detailed responses to EPA's
comments on appendix B of the draft report.
We appreciate OIG's draft recommendations regarding revision of a figure in the EPA Peer
Review Handbook, inclusion of key terms regarding peer review in certain Agency documents
and memoranda, and revision of Agency guidance and documentation procedures for use of data
from other organizations. We understand that the findings and conclusions are the premise to
developing your final recommendations. As required by EPA Order 2750, our written response
to the final report will address any recommendations that may be included at that time. We will
consider any recommendations on their own merit and, if applicable, provide a corrective action
plan and/or offer alternative solutions to the report's recommendations.
If you or your staff have any questions, please contact David LaRoche at (202) 564-3926 or
Norman Adkins at (919) 541-0872.
Attachment
cc: Arthur Elkins
Mark Bialek
11-P-0702
57
-------
Rick Beusse
Lek Kadeli
Kevin Teichman
Jim Jones
Renee Wynn
11-P-0702
58
-------
Attachment
A. EPA's Comments on Context, Key Findings and Conclusions in the Draft OIG
Report
A.l More context is needed regarding the extent of peer review conducted for the
underlying scientific assessments, and extent of EPA's peer and public review of its
technical support document
As stated in OIG's draft summary "At a Glance" section: "We did not assess the quality of the
scientific information and data EPA used to support the endangerment finding." EPA appreciates
that the title of the inquiry makes clear that "procedural" matters are the exclusive subject of the
OIG review. However, because the OIG report is intended to focus on process matters, it should
provide more context regarding the peer-review processes carried out for the underlying
scientific assessments utilized by EPA.
The OIG correctly points out that the primary scientific basis for EPA's finding was the
assessments conducted by the U.S. Global Change Research Program (USGCRP), IPCC and the
National Research Council (NRC) of the National Academy of Sciences. OIG then provides
some background on these scientific institutions but provides no details about the peer-review
procedures conducted for these assessments. The major focus of the OIG inquiry is the review
procedures for the TSD. Nothing in the OIG report reflects on the quality of the peer-reviewed
science that supported the endangerment finding.
We therefore request that the final OIG report clearly state the following points which are
supported by EPA's record and by the brief information provided below:
1) All of the science used to support the endangerment finding is from peer-reviewed
scientific assessments.
11-P-0702
59
-------
OIG Response 12: Our report acknowledges that EPA relied on the major assessments of USGCRP, IPCC,
and NRC as the primary scientific basis for its endangerment finding. We added descriptions of the peer review
procedures of USGCRP, IPCC, and NRC to the final report. However, EPA used additional information from
sources other than scientific assessments to support its finding. For example, the TSD stated the following:
In some cases, this document [the TSD] references other reports and studies in addition to
the core references of IPCC, CCSP/USGCRP, NRC, and, for GHG emissions, EPA. These
references are primarily for major reports and studies produced by U.S. federal and state
government agencies. This document also references data made available by other
government agencies, such as NOAA and National Aeronautics and Space Administration
(NASA).
EPA recently completed and published an assessment of the literature on the effect of
climate change on air quality (U.S. EPA, 2009a). Therefore, because EPA evaluated the
literature in the preparation of that assessment, EPA does cite some individual studies it
reviewed in its summary of this topic in Section 8. Also, for Section 16a on the national
security implications of climate change, this document cites a number of analyses and
publications, from inside and outside the government, because IPCC and CCSP/USGCRP
assessments have not traditionally addressed these issues.
Additionally, a senior analyst in OAR's Climate Change Division provided similar information to the OIG, stating
in an August 12, 2010, e-mail that:
[W]e [EPA] have always stated from the beginning of this process and in every version of
the TSD that was submitted for expert, interagency and public comment that we were
relying "most heavily" or "primarily" on what we have been calling the major assessment
reports, such as those from IPCC, USGCRP and NRC. We have never stated that we
would only rely 100% exclusively on such assessments because there were particular and
limited cases where the assessments did not provide the kind of information that was
needed; for example, we used EPA reports to provide the detailed GHG emissions data.
2) EPA documented the peer review procedures of IPCC and the other underlying scientific
assessments throughout different stages of the endangerment finding process.
OIG Response 13: We acknowledge in appendix A, question 1, that EPA documented the peer review
procedures of IPCC.
3) EPA assessed the peer review procedures of IPCC and the other underlying scientific
assessments to show these procedures meet standards of quality, objectivity, transparency
and integrity such that they are consistent with EPA and OMB guidelines.
OIG Response 14: The degree to which EPA assessed these procedures, or provided assurances that such
procedures met OMB requirements for highly influential scientific assessments, is not clear from our evaluation
of supporting documentation. This issue is discussed in further detail in chapter 3 of the report, and in
appendix A.
4) The TSD summarized material that had already undergone peer review, and was itself
subject to expert, internal EPA, interagency, and public reviews.
OIG Response 15: The review process for the TSD is discussed throughout our report. For example, in
chapter 2 of our report (p. 16) we note that EPA obtained reviews of the draft TSDs by federal experts,
provided a predissemination review for other federal agencies, and addressed comments received on the TSD
from workgroup members, federal experts, OMB, and other agencies. Further discussion of the review process
that EPA used for its TSD is also provided in our answers to questions 4, 5, and 6 in appendix A of the report.
11-P-0702
60
-------
The underlying scientific assessments used by EPA went through lengthy and rigorous peer-
review procedures, and received a high level of review and vetting from the U.S. Government.
Chapter 1 of the TSD provides a brief overview of the "Peer Review, Publication, and Approval
Processes for IPCC, CCSP/USGCRP, andNRC Reports" in Box 1.1 on pages 4-5. EPA's
proposed endangerment finding stated that, "The IPCC and CCSP assessments base their
findings on the large body of many individual, peer reviewed studies in the literature, and then
the IPCC and CCSP assessments themselves go through a transparent peer-review process" (74
FR 18894).
In response to the public comment period that followed the proposed endangerment finding, EPA
provided detailed information about the peer review procedures of the National Academies,
USGCRP and IPCC in over 70 pages publically available at
http://epa.gov/climatechange/endangerment.html (see volume 1 of Response to Comments that
accompanied the endangerment finding). Within those documented responses to public
comments regarding IPCC procedures, EPA discussed at length the specifics of IPCC peer
review procedures and concluded:
The evidence is clear that IPCC's procedures are sufficient and effective for ensuring quality,
transparency, and consideration of multiple and diverse perspectives. Because the assessment
reports EPA used in developing the TSD represent the best available science, and because
supporting studies were conducted in accordance with sound and objective scientific
practices, were peer reviewed, and adhered to standards of quality based on objectivity,
utility, and integrity, we find that IPCC's information quality process is consistent with
EPA's Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility and Integrity
of Information Disseminated by the Environmental Protection Agency. (See Response 1-14
regarding "Specific Comments on EPA's Use of IPCC Reports").
We provide further context about EPA's documentation of IPCC peer review procedures in
section B.
OIG Response 16: EPA's response to the OIG draft report emphasizes EPA's reliance on the major scientific
assessments of the USGCRP, IPCC, and NRC as the primary scientific and technical basis for the
Administrator's endangerment finding. While the conclusions from these assessments may have formed the
foundation of the Agency's endangerment finding, it should be noted that the Agency stated in its
endangerment finding that it gave "careful consideration to all the scientific and technical information in the
record." EPA's TSD referenced multiple sources, including "up-to-date" data from sources other than the
"major scientific assessments." In evaluating the scientific information, the Agency "placed limited weight on
the much smaller number of individual studies that were not considered or reflected in the major
assessments." EPA reviewed such studies "largely to see if they would lead EPA to change or place less
weight on the judgments reflected in the assessment report." The Agency stated in the endangerment finding
that "the studies did not change the various conclusions or judgments EPA would draw based on the
assessment reports."
A.2 EPA does not concur with OIG's opinion that EPA's technical support document
(TSD) is a HISA according to the OMB Peer Review Bulletin
OMB is the agency charged with implementing the Data Quality Act. OMB believes (as shown
in appendix G of the draft OIG report) that, EPA reasonably determined that the Endangerment
11-P-0702
61
-------
TSD itself (as opposed to the underlying peer-reviewed scientific assessments of the NRC, IPCC,
USGRCP identified and discussed in the TSD) did not have the impacts or characteristics
required to meet the OMB Peer Review Bulletin's definition of a highly influential scientific
assessment.
The TSD is "influential scientific information" (ISI) under the OMB Peer Review Bulletin, and
OMB agrees. EPA satisfied all peer review requirements for ISIs set out in the OMB Peer
Review Bulletin, and OMB agrees in its letter of April 15, 2011. In this regard, the OIG draft
states incorrectly in appendix B that the peer review requirements for an ISI were not satisfied.
OIG Response 17: Appendix A lists the OIG's answers to the Senator's specific questions. The Senator's fourth
question asks whether the TSD was subjected to peer review as specified in the EPA Peer Review Handbook
(not the OMB peer review bulletin). The answer is that EPA did not conduct a peer review of the TSD that met
all recommended steps in the Peer Review Handbook for peer reviews of influential scientific information or
highly influential scientific information, as discussed in appendix A.
To support its opinion that the TSD is a "highly influential scientific assessment" (HISA), the
OIG does not give weight to the official, considered, written OMB response, and instead relies
on an email from an OMB official regarding a largely hypothetical and general question posed
by OIG.2
An annotated bibliogi'aphy would generally not be considered a scientific assessment;
however, a document summarizing the "state of the science " would be, as it implicitly
or explicitly weighs the strength of the available evidence.
OIG Response 18: The OIG has considered all of the information that was provided or available to us in forming
our conclusion. The intent of our August 1, 2010, e-mail to OMB was not to obtain OMB's opinion on whether
the endangerment finding TSD was a highly influential scientific assessment, but to obtain clarification on its
peer review bulletin, including its definition of a scientific assessment. Therefore, it was appropriate for OMB to
speak in hypothetical terms and for us to ask general questions. We used the additional interpretative
information provided by OMB, and the definition and explanations provided in the OMB peer review bulletin in
conjunction with the information obtained during our assessment, to reach our conclusion.
We also disagree with the statement provided in footnote 2 of EPA's response. The OIG did not make any
clarifications to EPA regarding the interpretative response provided by OMB, but only provided a copy of our
questions to OMB as an appendix to the draft report. The information provided as background before those
questions clearly demonstrates the reasons for our inquiry. However, whether the official responding to our
questions was familiar with the TSD is not important. As explained above, our intent was not to obtain OMB's
opinion on whether the TSD was a highly influential scientific assessment, but to obtain information to help us
reach our own independent conclusion as to the nature of the TSD.
2 It has since been clarified by the OIG that the writer of this e-mail was not familiar with the TSD and was not
provided by the OIG with the TSD or informed that the TSD was the document about which the OIG was ultimately
inquiring (see appendix E, p. 44, of the OIG's draft report).
11-P-0702
62
-------
The OIG goes on to conclude that, Based on the above interpretation, by providing a summary of
existing findings and conclusions from the IPCC, USGCRPZCCSP, NRC, and other reports, OAR
implicitly and explicitly weighed the strength of the available science by its choices of
information, data, studies, and conclusions included and excluded from the TSD. Thus, in our
opinion, the TSD is a scientific assessment.
This is not correct. A "scientific assessment" is defined in the OMB Peer Review Bulletin (2004)
as "an evaluation of a body of scientific or technical knowledge, which typically synthesizes
multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to
bridge uncertainties in the available information." The TSD did not conduct such an evaluation.
No weighing of information, data and studies occurred in the TSD. That had already occurred in
the underlying assessments. That is where the scientific synthesis occurred, that is where the
state of the science was assessed.
OIG Response 19: We continue to believe that the TSD is a scientific assessment based on the definition in
the OMB peer review bulletin for the reasons outlined in OIG Response 5.
Indeed, some commenters chastised EPA for not performing its own scientific assessment rather
than relying on the existing peer reviewed science. See section 1 of volume 1 of the Response to
Comments that accompanied the endangerment finding at
http://epa.gOv/climatechange/endangerment/comments/volumel.html#0.
EPA certainly agrees that the IPCC, USGCRP and NAS/NRC reports summarized in the TSD
are "scientific assessments." However, the scientific findings found in the TSD are not the result
of the TSD having gone through the scientific literature or assessments to synthesize multiple
factual inputs, data, models, assumptions. To the contrary, that synthesis occurred during the
development of the assessment reports. The TSD did not synthesize or alter the findings of the
underlying assessment reports. Nor does the TSD bridge uncertainties in the available
information or otherwise use "professional judgment" to resolve scientific issues. The TSD
simply summarizes in a straightforward manner the underlying assessments of the National
Academies, the USGCRP and IPCC.
OIG Response 20: A scientific assessment does not have to involve applying best professional judgment to
bridge uncertainties in the available information. We believe the TSD is a scientific assessment because a
synthesis occurred during the development of the TSD as EPA went through primarily the three major
assessment reportsIPCC, NRC, and USGCRPto synthesize multiple factual inputs. Also, as noted in OMB's
clarification on the definition of a scientific assessment, a document such as the TSD that summarizes the "state
of science" (climate change, in this case) would be a scientific assessment because it implicitly or explicitly
weighed the strength of the available science. For more details on other information relied upon for the
development of the TSD, see OIG Responses 5 and 16.
The initial response from OMB cited in the report is a paraphrase of an example from the
Handbook's definition of "scientific assessment" that includes a "state of the science report" as
one of several examples. The key predicate to this example in the Handbook is in the sentence
preceding it, which refers to evaluating a body of scientific knowledge, typically synthesizing
multiple factual inputs, data, models, assumptions, and/or applying best professional judgment to
bridge uncertainties in the available information. The reference to a "state of the science
document" cannot be seen in isolation, but refers to documents that involve the kind of weighing
11-P-0702
63
-------
of the evidence and exercising scientific judgment that is described in the Handbook. Although
the assessment reports involved weighing the data, studies and other information in order to
arrive at conclusions on the state of the science, the TSD did not do this.
The OIG concludes that EPA implicitly and explicitly weighed the strength of the available
science by its choice of information, data, studies, and conclusions included and excluded from
the TSD, but provides no basis for this conclusion. The TSD reported the results of the weighing
of the science that had been conducted by the assessment reports, but the TSD did not re-weigh
the strength of the available science. The decision to include in the TSD various scientific
conclusions and other information from the assessment reports was not based on weighing the
strength or otherwise evaluating the science. Inclusion in the TSD was based on relevance to the
science issues before the Administrator under section 202(a) of the Clean Air Act, and whether
the result was a fair and accurate summary of the conclusions and information from the
assessment reports. This is neither an implicit or explicit weighing of the strengths of the
available science.
OIG Response 21: We believe the TSD implicitly or explicitly weighed the strength of the available science by
relying primarily on three assessment reportsIPCC, NRC, and USGCRPrather than on other literature.
EPA noted in its response to the draft report that it "had to assess which information should and could be used
as the scientific basis." For more details on other information relied upon for the development of the TSD, see
OIG Responses 5 and 16.
OIG also cites two isolated sentences from the proposed and final endangerment finding as
support for the OIG opinion. These statements by themselves are not representative of the fuller
record and EPA's long-standing view that the TSD is not itself a new assessment of the science.
The first example is from the proposed finding:
EPA has developed a technical support document (TSD) which synthesizes major
findings from the best available scientific assessments that have gone through
rigorous and transparent peer review.
Here OIG is apparently emphasizing use of the word "synthesizes"; we recognize "summarizes"
is the more appropriate word because the TSD does not break new ground from the major
findings of the underlying scientific assessments.
OIG Response 22: Even if the word "summarizes" is used, we still consider the TSD a scientific assessment
based on OMB's clarification on the definition of a scientific assessment, which states that a document
summarizing (emphasis added) the "state of the science" would be a scientific assessment, as it implicitly or
explicitly weighs the strength of the available evidence. Another example in the proposed finding suggesting that
the TSD is a scientific assessment is the following:
The first section describes the approach EPA has taken in gathering and synthesizing the best
available scientific information to inform the Administrator's judgment. . .
The second example is from the final endangerment finding:
In 2007, EPA initiated its assessment of the science and other technical information
to use in addressing the endangerment and cause or contribute issues before it under
CAA section 202(a). This scientific and technical information was developed in the
11-P-0702
64
-------
form of a TSD in 2007. An earlier draft of this document was released as part of the
ANPR published July 30, 2008 (73 FR 44353).
OIG Response 23: By assessing "which information should and could be used as the scientific basis," EPA
weighed the strength of the available science in developing the TSD.
Other examples in the final finding suggesting that the TSD is a scientific assessment are the following:
Many of the comments received on the ANPR were reflected in the draft TSD released in April
2009 that served as the underlying scientific and technical basis for the Administrator's
Proposed Findings, published April 24, 2009 (74 FR 18886). The draft TSD released in April
2009 also reflected the findings of 11 new synthesis and assessment products under the U.S.
CCSP that had been published since July 2008.
In addition, the TSD incorporates up-to-date observational data for a number of key climate
variables from the NOAA, and the most up-to-date emissions data from EPA's annual
Inventory of U.S. Greenhouse Gas Emissions and Sinks, published in April, 2009.9 And finally,
as discussed in Section I.B of these Findings, EPA received a large number of public
comments on the Administrator's Proposed Findings, many of which addressed science issues
either generally or specifically as reflected in the draft TSD released with the April 2009
proposal. A number of edits and updates were made to the draft TSD as a result of these
comments.
EPA recognizes the potential importance of new scientific research, and the value of an
ongoing process to take more recent science into account. EPA reviewed new literature in
preparation of this TSD to evaluate its consistency with recent scientific assessments. We also
considered public comments received and studies incorporated by reference. In a number of
cases, the TSD was updated based on such information to add context for assessment
literature findings, which includes supporting information and/or qualifying statements. In other
cases, material that was not incorporated into the TSD is discussed within the Response to
Comments document.
Many comments focus on the scientific and technical data underlying the Proposed Findings,
such as climate change science and greenhouse gas emissions data. These comments cover
a range of topics and are summarized and responded to in the Response to Public Comments
document. The responses note those cases where a technical or scientific comment resulted in
an editorial or substantive change to the TSD. The final TSD reflects all changes made as a
result of public comments.
9 U.S. EPA (2009) Inventory of U.S. Greenhouse Gas Emissions and Sinks: 1990-2007. EPA-430-R-
09-004, Washington, DC.
Here OIG is apparently focusing on the word "assessment" in the first sentence. The intended
meaning here is simply that, in 2007 when EPA was first requested under Executive Order to
begin an endangerment finding process, we had to assess which information should and could be
used as the scientific basis. This is the time when EPA decided that it should primarily rely on
the scientific assessments of the major bodies of USGCRP (then CCSP), IPCC and NRC. So
"assessment" in the first sentence above refers to the process by which EPA determined what
kind of information should be brought in and summarized in the TSD. We were very clear what
scientific information we were relying on and believe that the OIG report picks two sentences
and uses them out of context.
OIG Response 24: The OIG did not pick two sentences out of context. There were more examples as noted
above in OIG Response 23.
We ask that the OIG also address and include other information from the record that clearly
shows that the TSD was not itself a new assessment of the science, including the introductory
11-P-0702
65
-------
chapter of the TSD and the entire sections of the proposed and final findings in the federal
register that are devoted to the "Science Upon Which the Decisions are Based." These provide
detailed rationale for the Agency's reliance on the existing assessment reports and our
determination to summarize the scientific information rather than develop our own scientific
assessment.
OIG Response 25: We believe sufficient details on the information used to support the endangerment finding
were provided in the draft report.
The OIG should consider all of the analysis and information provided on this issue, and the final
report should reflect a comprehensive evaluation in light of the above discussion. EPA is
confident that a comprehensive review of this issue leads to the conclusion that the TSD is not a
HISA. However, even if the OIG reaches a different conclusion in its final report, the OIG
should clearly indicate that there are substantial and reasonable grounds for EPA's and OMB's
views that the TSD is an ISI and not a HISA. The OIG should also make clear throughout their
report that their finding is premised on the TSD being a HISA, and that if the TSD is not a HISA
then such conclusions, and perhaps attendant recommendations, would not apply. This would
help to clarify the conditional nature of OIG's views, and allow readers to draw their own
conclusions in light of the very reasonable and, we believe, compelling grounds for determining
that the TSD is not a HISA.
OIG Response 26: We recognize that the TSD is a unique document in that it is primarily based upon the
results of other scientific assessments. Accordingly, others may reasonably interpret that the TSD does not fit
the OMB definition of a highly influential scientific assessment and, therefore, was not subject to peer review
requirements for highly influential scientific assessments. However, based on the criteria set forth in the OMB
peer review bulletin and OMB's clarification on the definition of a scientific assessment, and our assessment of
information obtained during our review, we continue to believe that the TSD is a highly influential scientific
assessment. Even if we concluded that the TSD was not a highly influential scientific assessment, our
recommendations are still applicable as they pertain to aspects of the action development process that were not
followed or areas where guidance should be clarified regardless of whether EPA considered the TSD to be
influential scientific information or a highly influential scientific assessment.
OIG has stated in their draft report that, EPA'speer review did not meet minimum OMB
requirements for such documents. We strongly object to the limited discussion of the peer review
actions of EPA and the use of the word minimum as it is misleading. It could be interpreted as
meaning EPA did little to satisfy the peer review requirements for the TSD. In fact, EPA went
far beyond the requirements for peer review of an ISI by conducting multiple expert, interagency
and public reviews that were reasonable and appropriate to ensure the credibility of the TSD. All
of the reviewer comments were maintained, multiple versions of the draft TSD were archived, all
peer reviewers were disclosed, and EPA submitted a memo to the record (see EPA-HQ-OAR-
2009-0171-11639) documenting all the changes to the TSD that were made in response to all
levels of comments. This detailed process was completed for all three rounds of federal expert
review, not just for the ANPR, as stated by the draft OIG report. EPA has maintained
documentation for all of the comments received from federal experts and its response to these
comments. Furthermore, the public was given two full opportunities to comment on the TSD:
once during the ANPR (a 120-day comment period) and again following the proposed
endangerment finding (a 60-day comment period including two public hearings).
11-P-0702
66
-------
OIG Response 27: We revised the text to say that EPA's review of its TSD did not meet all OMB's peer review
requirements for a highly influential scientific assessment.
The statement in the draft report that "EPA did not maintain a record of its response and disposition of
comments for the two TSDs that accompanied the proposed and final rules" is correct. We had asked EPA to
provide us copies of its response to the federal climate change expert reviewers' comments on the TSD that
accompanied the ANPR, proposed, and final rules. EPA provided us with a table of the reviewer comments,
EPA's response, and disposition of the comments for the TSD that accompanied the ANPR. However, EPA did
not prepare similar documentation for the TSD that accompanied the proposed and final rules.
The OMB Peer Review Bulletin has eight peer review criteria for a HISA (see table below). In
addition, "[e]ven for these highly influential scientific assessments, the Bulletin leaves
significant discretion to the agency formulating the peer review plan" (pg Bulletin at p. 2-3). In
light of the circumstances of the TSD, the extensive peer review and public review process that
was conducted for the TSD, and the discretion afforded under the peer review guidance, EPA
believes that it met the OMB peer review requirements for HISAs.
The OIG draft report does not discuss all of the criteria, nor does it clarify how the review EPA
conducted for the TSD met the criteria. This results in a significant lack of balance and context
for the conclusion in the draft report. The discussion in the draft OIG report indicates that the
OIG believes EPA did not fully comply with one of the peer review requirements, involving
preparation of a peer review report (number 7 in the table below). If OIG continues with that
view, then the final OIG report should indicate that EPA did not consider the TSD to be a HISA,
but OIG believes the extensive peer review conducted by EPA satisfied all but one of the OMB
peer review requirements for HISAs, and partially satisfied that one requirement.
OMB Peer Review
Criteria for HISA
EPA's Review of TSD
1
Peer reviewers should have
appropriate expertise and
balance
EPA had the TSD reviewed by 12 highly qualified and credentialed federal
government scientists whose expertise covered the range of topic areas.
OIG Response 28: The OMB peer review bulletin refers to balance as "a
diversity of scientific perspectives relevant to the subject." We did not question
the appropriateness of the 12 federal reviewers' expertise. We do note that all
were federal employees and all had leading roles in developing the assessment
reports cited in the TSD.
11-P-0702
67
-------
2
Conflict of interest
OMB Peer Review Bulletin states peer reviewers who are federal employees "are
subject to applicable statutory and regulatory standards for federal employees.
OIG Response 29: Under additional peer review requirements for highly
influential scientific assessments, the OMB peer review bulletin states, "The
agency - or the entity selecting the peer reviewers - shall (i) ensure that those
reviewers serving as Federal employees (including special government
employees) comply with applicable Federal ethics requirements . . . ." We asked
EPA how it ensured that the reviewers complied with applicable federal ethics
requirements. According to a senior analyst in OAR, the Agency had no reason
to believe that the federal reviewers would have any conflict of interest issues
because they were only reviewing the TSD for accuracy and not re-opening or
judging their prior work.
3
Independence from the
sponsoring agency
One of the 12 federal expert reviewers was an EPA employee. This person was not
one of the original authors of the TSD, and her expertise in the human health impacts
of climate change helped fill the balance of expertise needed.
OIG Response 30: The OMB peer review bulletin states that "scientists
employed by the sponsoring agency are not permitted to serve as reviewers for
highly influential scientific assessments." Thus, EPA did not fully meet the OMB
requirement for independence of reviewers.
4
Repeated use of same peer
reviewers for multiple
assessments
Group of 12 federal scientists was not used for peer review of other assessments. 12
federal scientists remained the same since 2007 so reviewers could clearly track, in
line with their charge, how the TSD was summarizing new assessments that had
become available.
OIG Response 31: While EPA met the requirement to not use the same
reviewers on multiple assessments, we note that the EPA Peer Review
Handbook states that it is preferable to use different people each time the
product is being reviewed again to provide a broader perspective. The handbook
notes that "if the same peer reviewers are used repeatedly, they may lose their
independence (or the appearance of independence) from the work product(s)."
5
Provide reviewers with
sufficient background
information
12 reviewers had full access to both the TSD and, more importantly for this criterion,
full access and knowledge of the underlying assessments (and the underlying
individual studies that were synthesized in those assessments) summarized in the TSD.
6
Public participation in peer
review
TSD was subject to public review and comment. Furthermore, the assessments of the
National Academies, USGCRP and IPCC had already been in the public domain, and
for USGCRP and IPCC assessments the public had previous opportunities to comment
on those reports.1(a)
OIG Response 32: While the public was afforded opportunities to comment on the
TSD, public opportunity to comment on a document is not public participation in the
peer review. EPA did not make the draft TSD that eventually accompanied the
proposed endangerment finding available to the public at the same time it went to
the peer reviewers for review. EPA also did not sponsor a public meeting or make
the public comments on the draft TSD that accompanied the proposed
endangerment finding available to the peer reviewers during their review of the draft
final TSD.
11-P-0702
68
-------
7
Preparation of a peer
review report
EPA submitted a memorandum to the record (see EPA-HQ-OAR-2009-0171-11639)
documenting all changes to the TSD in response to all levels of comments, both from
the expert reviewers and from the public.
EPA did not produce a separate memorandum describing changes only in response to
expert reviewer comments, as this did not seem necessary as we were not operating
under HISA procedural assumptions.
Expert reviewers were disclosed, and EPA has maintained documentation of all
comments received and before-and-after versions of the TSD.
OIG Response 33: According to the OMB bulletin, an agency managing the peer
review shall instruct peer reviewers to prepare a report that describes the nature of
their review and their findings and conclusions. The agency shall disseminate the
final peer review report on the agency's website along with all materials related to
the peer review (any charge statement, the peer review report, and any agency
response). The peer review report shall be discussed in the preamble to any
related rulemaking and included in the administrative record for any related agency
action. Additional requirements apply to peer reviews of highly influential scientific
assessment. The agency shall prepare a written response to the peer review
report explaining (a) the agency's agreement or disagreement with the views
expressed in the report, (b) the actions the agency has undertaken or will
undertake in response to the report, and (c) the reasons the agency believes those
actions satisfy the key concerns stated in the report (if applicable). The agency
shall disseminate its response to the peer review report on the agency's website.
For this action, a peer review report was not prepared. A record of the reviewers'
comments were maintained but not made available to the public. For the first
version of the TSD reviewed by the panel, EPA prepared written responses to the
reviewers' comments. However, for the two subsequent reviews of the TSD, EPA
did not prepare written responses to the reviewers' comments.
8
Authorization to have a
separate entity carry out
peer review
EPA reasonably chose to manage the review process itself.
(a) EPA did not conduct expert review and public review simultaneously, and did not make publically available the
expert review comments received on the TSD.
EPA is not implying that the level of peer review for the TSD generally would be considered
appropriate for a document determined by EPA to be a HISA. The TSD is not a HISA and its
review more than satisfies the EPA peer review requirements for influential scientific
information. When EPA conducts peer review of a scientific assessment that is a HISA, the peer
review often includes the use of independent third party panels such as the Science Advisory
Board and National Academies of Science for conducting the peer review, ensuring the balance
of the peer review panel, consideration of conflicts of interest, consideration of public
participation, and preparation of a peer review record.
OIG Response 34: The federal climate change expert review of the TSD did not follow the recommended
procedures outlined in the Peer Review Handbook for either influential scientific information or highly influential
scientific assessments, or OMB's guidelines for peer review of highly influential scientific assessments.
However, we recognize the level of review would be within EPA's discretion if the document was considered
influential scientific information, but not if considered a highly influential scientific assessment.
11-P-0702
69
-------
A.3 EPA does not concur with the OIG statements that "the Agency did not conduct any
independent evaluations of IPCC's compliance with IPCC procedures, nor did EPA
document any specific processes it employed to evaluate the scientific and technical
information included in IPCC's Fourth Assessment Report."
First, both in the main body of the OIG report and in the draft response to this question from
Senator Inhofe, the OIG fails to fully recognize or provide any documentation regarding EPA's
actual involvement, and at times lead role, in the U.S. Government process to review, comment
on, and assist in the U.S. Government approval of the IPCC assessment reports. Statements in
OIG's draft report give the inaccurate impression that EPA was a passive observer of U.S.
Government approval of the IPCC Fourth Assessment Report, stating, for example, that "U.S.
government acceptance did not relieve EPA of its responsibility to determine whether the data
met EPA's information quality guidelines before disseminating the information." The reality is
that EPA devoted significant staff time to review the IPCC assessment reports, provide
comments to an interagency process, participate in that interagency process to prioritize U.S.
government comments, and to approve the summaries of the IPCC reports in a detailed line-by-
line process, which involves detailed checking of how the summary is reflecting the underlying
report and therefore how U.S. government comments are being reflected.
OIG Response 35: We added statements to chapter 1, chapter 3, and appendix A of the final report to indicate
EPA's involvement in the IPCC process, including its participation in the U.S. delegation approving the Working
Group II report for the AR4.
Second, the OIG draft report overlooks the many examples where EPA specifically discussed
and documented throughout the endangerment finding record how the IPCC peer review
procedures meet EPA and OMB guidelines. We refer to our list of key examples provided under
the response to Senator Inhofe's first question.
OIG Response 36: OIG analysis of specific EPA comments regarding its analysis and documentation of
IPCC procedures is provided in OIG Responses 46 through 52.
The OIG summarizes EPA's "examination of IPCC's Fourth Assessment Report peer review" in
a table, but we do not concur with a number of OIG's conclusions in that table.
Stating that EPA did not address IPCC's conflict of interest procedures is incorrect. EPA
documented how IPCC review procedures are designed to avoid conflict of interest among
authors and peer reviewers in detail in Section 1 and Appendix A of Volume 1 of the Response
to Comments document following the 2009 proposed endangerment finding, and Section 2.2 of
the Response to Petitions document. EPA explains here that while IPCC procedures do not
explicitly contain "conflict of interest" language, there are sufficient checks and balances built
into the IPCC procedures such that there has been no evidence that the quality of IPCC reports
suffer from potential conflict of interest issues.
For example, IPCC's report development procedures state that the review should entail, a wide
circulation process, ensuring representation of independent experts (i.e. experts not involved in
the preparation of that particular chapter) f'om developing and developed countries and
countries with economies in transition should aim to involve as many experts as possible in the
11-P-0702
70
-------
IPCCprocess. Thirdly, the review process should be objective, open and transparent... To help
ensure that Reports provide a balanced and complete assessment of current information, each
Working Group Task Force Bureau should normally select two Review Editors per chapter...
Review Editors should not be involved in the preparation or review of material for which they
are an editor.
OIG Response 37: The example cited by EPA above addresses aspects of reviewer independence, as
described in OMB guidance. As noted in table 2 of our report, we believe that EPA examined the issue of
IPCC reviewer independence, and this is included in documentation supporting its endangerment finding.
OMB guidelines state that the NAS policy for committee selection should be used with respect to evaluating
the potential for conflicts. The NAS policy states that "at the time of appointment, each committee member is
required to list all professional, consulting, and financial connections, as well as to describe pertinent
intellectual positions and public statements by filling out a confidential form, 'Background Information and
Confidential Conflict of Interest Disclosure.' The committee appointment is not finalized until the institution
completes a review of information regarding potential conflicts of interest and bias." EPA referred us to
several documents for details on their review of IPCC's procedures. We were not able to find any statements
in the materials referred to us that EPA had examined this aspect of IPCC's procedures.
In response to specific questions raised about conflict of interest issues in EPA's denial of the
petitions to reconsider the endangerment finding, EPA stated:
See Sections 2.2.2 and 2.2.3 in this volume of the RTP document, which address our view
that the IPCC's procedures are sufficient to ensure that no one individualincluding the
IPCC chairmanor group of individuals can exert disproportionate influence in developing
any piece of the assessment reports. The IPCC has robust procedures to ensure that the
assessment reports are objective, are unbiased, and represent the state of the science
regarding climate change.
OIG Response 38: The conclusion presented in chapter 3 of the draft report states that "no contemporaneous
documentation was available to show what analyses the Agency conducted prior (emphasis added) to
dissemination of the information in its advance notice and proposed action." Therefore, for this particular
aspect of our evaluation, we did not consider EPA analyses provided in its response to petitions document
because such analyses were done after the Agency had already disseminated the information.
Stating that EPA did not document its examination of IPCC's procedures for including a rotation
of peer reviewers in its Fourth Assessment Report is incorrect. The IPCC' s peer review process
included multiple rounds of expert review involving more than 2,500 individuals as well as
governmental review by the 189 member countries. These rotation procedures are described in
Section 1.1.1 and Appendix A of Volume 1 of the Response to Comments document, which
describes the entire peer review process for the IPCC. Since the OMB criterion requires that
"Agencies shall avoid repeated use of the same reviewer on multiple assessments," the IPCC
rotation procedures require a far greater amount of rotation than the OMB guidance, as
thousands of reviewers are employed rather than just a handful.
OIG Response 39: EPA's response does not provide supporting evidence to show that the Agency
documented its examination of this issue in the Federal Register notice or its response to comments. EPA
does not address the issue of rotation of IPCC reviewers in documentation supporting its endangerment
finding.
11-P-0702
71
-------
Stating that EPA did not document its examination oflPCC's procedures to ensure all
appropriate information was provided to IPCC reviewers is incorrect. IPCC procedures provide
for dissemination of all relevant information to peer reviewers, and EPA documented this fact.
See Appendix A of Volume 1 (describing the IPCC information dissemination procedures):
"Working Group Co-Chairs should make available to reviewers on request during the review
process specific material referenced in the document being reviewed, which is not available in
the international published literature." We do not understand how OIG concludes that
"information access for peer reviewers" is not adequate or was not documented by EPA with
regard to IPCC.
OIG Response 40: We did not conclude that IPCC's information access to peer reviewers was not adequate.
Instead, we concluded that EPA did not provide a description of this matter, or an evaluation of this particular
aspect of IPCC procedure, in its endangerment finding or its response to comment document. We reviewed
EPA's descriptions of the IPCC review process in both its Federal Register notice and its response to
comments document. For this particular issue, EPA did not address how the Agency evaluated, or otherwise
considered, the IPCC procedures relative to the OMB peer review requirements for highly influential scientific
assessments. Instead, the Agency simply attached the IPCC procedures as an appendix to its response to
comments document and provided an overall conclusion that such procedures met all applicable information
quality requirements.
Stating that EPA only partially documented IPCC's procedures to ensure transparency of the
IPCC peer-review process is incorrect. A footnote to OIG's Table 2 states that "transparency
was generally addressed but not for all elements." The IPCC's procedures satisfy the OMB
criteria for transparency by clearly articulating the procedures for peer review, making the charge
to reviewers publically available and publishing peer review reports that document how the
authors addressed all comments received. EPA clearly documented IPCC's procedures for peer
review transparency in Appendix A of Volume 1 of the Response to Comments document:
All written expert and government review comments will be made available to reviewers on
request during the review process and will be retained in an open archive in a location
determined by the IPCC Secretariat on completion of the Report for a period of at least five
years.
EPA's evaluation of these procedures is further described in Section 1 of Volume 1 of the
Response to Comments document and Section 2.2 of the Response to Petitions document. The
OIG states that EPA did not discuss whether "IPCC procedures required a description of the
credentials and relevant experience of each peer reviewer." EPA finds that this element is not
applicable in this circumstance because the IPCC assessment reports are reviewed by thousands
of individuals. The fact that OIG is applying an element for peer-review transparency involving
the disclosure of credentials/experiences for thousands of peer reviewers suggests that the OIG
does not appreciate the nature and size of the IPCC's peer-review process. To suggest that the
credentials and relevant experiences be disclosed for thousands of individuals is inappropriate
and not reasonable.
11-P-0702
72
-------
OIG Response 41: As noted in OIG Response 40, we did not make any conclusions regarding whether the
IPCC process met OMB peer review requirements. Instead, we state that EPA did not make any explicit
statements in its endangerment finding or its response to comment document that evaluated the IPCC
procedures for this particular element of the OMB requirements. OMB requires that "the peer review report
shall include the charge to the reviewers and a short paragraph on both the credentials and relevant
experiences of each peer reviewer." EPA did not address this particular aspect of the OMB "transparency"
requirement in its endangerment finding or its response to comment document. If, as EPA stated in its
comments to the OIG draft report, "this element of the OMB requirement is not applicable in this
circumstance," such a statement, and justification, could have been provided in the endangerment finding
itself, or in EPA's response to comment document. As noted in table 2 of the report, EPA did not provide a
statement in the Federal Register notice or response to comments that addressed this issue.
A.4 EPA does not concur with the OIG's statement that "OAR did not identify the
specific type of information supporting its action or certify compliance with EPA's
Peer Review Policy."
The final endangerment finding devotes an entire section to "The Science On Which the
Decisions are Based" (Section III. A). There, after discussing in depth why EPA chose to
primarily rely on the major scientific assessments of USGCRP, IPCC and NRC as the science
informing the Administrator's judgment, EPA states that these assessments "maintain the highest
level of adherence to Agency and OMB guidelines for data and scientific integrity and
transparency" (74 FR 66511). Furthermore, the full endangerment record, including the
endangerment finding as well as the TSD and the Response to Comments, provide extensive
discussions regarding the type of information supporting the finding. This information was
provided to EPA management in various forms including briefings to two Administrators.
Despite the many pages of descriptive text in the public record describing the specific type of
information supporting the endangerment action, the OIG appears to focus on searching only for
key terms in internal memoranda. EPA has done due diligence ensuring that there was
transparency in our reliance on the scientific assessment reports for the finding. Furthermore, the
expert review approach used for the TSD was discussed with the EPA Workgroup, EPA senior
management and with other government agencies. In all instances, the approach was clear and
approved. However, EPA recognizes that the record does not contain a specific discussion of the
TSD and its peer review in terms of the peer review requirements for an ISI. EPA agrees that
including a clearer discussion of this could have been helpful and avoided confusion on this
issue.
11-P-0702
73
-------
OIG Response 42: We revised the title for this section in our final report (see p. 19). We did not find that EPA
failed to indentify the sources of information it used to support its endangerment finding. Rather, we found the
Agency did not follow OMB and EPA requirements to discuss such information in terms of whether it was
influential scientific information or highly influential scientific assessments. Further, EPA did not certify in the
administrative record for the endangerment finding that its review of the TSD met applicable OMB peer review
requirements. Both OMB requirements and EPA's Peer Review Handbook state that the Agency is supposed to
identify whether it used influential scientific information or a highly influential scientific assessment to support
the Agency's action. Further, OMB requires the Agency to certify compliance with OMB peer review
requirements, and the Peer Review Handbook recommends that the peer review report for the underlying
information be included in the docket for the action. These are information quality procedures that EPA did not
follow for this particular action.
The Agency did not describe the TSD in terms of influential scientific information or highly influential scientific
assessments in documentation associated with the endangerment finding. When we asked EPA whether it
characterized the TSD as either influential scientific information or a highly influential scientific assessment in
documentation supporting the endangerment finding, an OAR branch chief told us that EPA did not describe
the scientific information in such terms because it was "not the focus of the discussion." Further, the OIG did
not simply perform a search "only for key terms in internal memoranda" as EPA's comments suggest. Although
the OMB bulletin and EPA requirements call for the Agency to conclusively state whether the information being
disseminated is influential scientific information or a highly influential scientific assessment, throughout the
majority of the OIG's evaluation, EPA would not identify whether it considered the TSD to be influential
scientific information or a highly influential scientific assessment. The Agency did not provide this information
even in response to direct questions from the OIG. For example, the OIG sent an e-mail to a senior analyst in
OAR (who managed the expert review of EPA's TSD) on October 13, 2010, asking whether EPA characterized
its TSD as a scientific assessment. The analyst responded in writing that EPA simply called the TSD a
"technical support document" to "acknowledge it was the compilation of science to support the Administrator's
endangerment decision." The analyst did not characterize the TSD further, but did state that EPA did not call
the TSD an assessment report. When asked in a subsequent e-mail, on October 13, 2010, whether EPA
considered the TSD to be influential scientific information, as defined in the Agency's Peer Review Handbook,
the analyst stated that EPA "didn't formally designate the TSD as 'influential scientific information'." EPA first
described the TSD as influential scientific information when it responded to the OIG draft report.
A.5 EPA disagrees with the OIG's statement that "OAR deviated from the Action
Development Process in some areas."
Although we appreciate OIG's attempt to provide some context surrounding the 2007 origins of
the endangerment findings and the TSD, the final OIG report need to indicate the flexibility and
discretion provided by the Agency guidance itself. The OIG does not present a balanced
discussion of the actions EPA took to achieve the goals and communicate the major components
of an Analytic Blueprint to all program offices and workgroup members. In 2007, EPA's Office
of Transportation and Air Quality initiated a rulemaking for Greenhouse Gas Emission Standards
for certain motor vehicles. The rulemaking involved determining whether greenhouse gas
emissions from sources covered under section 202(a) of the CAA cause or contribute to air
pollution that endangers public health or welfare and if so, to set new standards for motor
vehicles to control such emissions. As part of this effort, EPA held an early guidance meeting
and outlined a "plan for developing the Endangerment Finding" based on the early guidance.
Management briefings (provided to OIG) regarding the development plan identified and clearly
communicated the topics covered in an analytic blueprint including approach, scope, underlying
science and review mechanisms for the TSD and endangerment finding process. In 2009, EPA
management separated the elements of the original action into separate actions. Hence, the 2009
action was not a new action that required the workgroup to start at the beginning of the Action
Development Process. In 2009, when the actions were separated, EPA management chose to
proceed using the same approach for the TSD identified in 2007. This decision was clearly
11-P-0702
74
-------
communicated to and supported by workgroup members. EPA shared with OIG the briefing
materials that provided this information. The approach laid out in 2007 remained valid.
OIG Response 43: We believe we have presented a balanced discussion on whether EPA fulfilled the
requirement to develop an analytic blueprint for the endangerment findinga Tier 1 action. We acknowledged in
the draft report that even though EPA did not develop a new blueprint or revise the analytic blueprint for the
stand-alone endangerment finding action, it prepared nine briefing documents for EPA senior management that
provided details on the Agency's plans for preparing and peer reviewing the TSD, including the early guidance
briefing.
The OIG appears to assume that the lack of a document in 2009 with the title "Analytic
Blueprint" means that the goals and key elements of such a document were not addressed and
communicated by EPA to management and other program offices. To the contrary, the content of
an Analytic Blueprint was discussed on numerous occasions with all program offices involved.
EPA thoroughly discussed at numerous management briefings in both 2007 and 2009 which
assessment reports were going to be relied upon and why (i.e., because they have already
undergone extensive peer review) as well as the approach for review of the TSD. It is
inappropriate for OIG to conclude that a 2009 decision to maintain a 2007 decision to continue
to rely primarily on and remain consistent with the existing peer-reviewed assessment literature -
is a "deviation" from the Action Development Process.
OIG Response 44: The draft report acknowledged that an analytic blueprint was prepared in 2007 for a
proposed greenhouse gases transportation rule that would include an endangerment finding. This analytic
blueprint, however, did not explain what reviews were needed before accepting the other organizations' data or
how the TSD would be peer reviewed. The draft report also acknowledged that although the analytic blueprint
for the 2007 action did not describe how the TSD would be peer reviewed, OAR prepared nine briefing
documents for EPA senior management that provided details on the Agency's plans for preparing and peer
reviewing the TSD. These briefings were conducted from May through September 2007. These briefing
documents outlined the Agency's approach but did not explain why it chose not to have a formal external peer
review of the TSD. We clarified our discussion in the final report to make it clear that we did not expect the
Agency to prepare an entirely new analytic blueprint for 2009.
The OIG notes that OAR "completed many of the processes and steps outlined in its guidance to
ensure the quality of the information the Administrator used in making her determination." Yet
the report also states that "OAR did not adhere to some of its internal processes established to
guide Tier I actions." Here again, we believe that the few examples cited by the OIG are within
the flexibility and discretion clearly provided within the agency guidance, are taken out of
context, are trivial in substance, and are inappropriately elevated by the OIG as "procedural
deviations". For example, the report asserts a "deviation" for "not including all reviewing
offices' positions in the options selection meeting materials," whereas the reviewing offices all
concurred with the options recommended. Furthermore, official reviewing offices' positions
were presented at the Final Agency Review meeting held on November 5, 2009, and formally
documented in the memorandum issued by the Office of Policy summarizing the results of the
meeting and provided to the OIG.
OIG Response 45: Our draft report acknowledged that the potential impact of these procedural deviations is
"debatable." Our draft report further acknowledged the comments of EPA's Office of Policy, Regulatory
Management Division Director, that the action development process guidance is supposed to be flexible.
11-P-0702
75
-------
B. EPA Comments on OIG Draft Answers to the Questions Posed by Senator Inhofe
We offer the following responses to the OIG's draft answers to Senator Inhofe, and strongly urge
the OIG to ensure consistency between its report and its responses to the Senator (Appendix B).
1. Did EPA conduct an examination of the IPCC procedures, including the IPCC process for
handling review comments? How did EPA determine that the IPCC process satisfied EPA's
obligations to follow the Data Quality Act (sic) and the Agency's, as well as OMB's, peer review
guidelines? How was this determination documented?
In response to Senator Inhofe, the OIG correctly states that "EPA felt confident accepting
IPCC's I'Oiirth Assessment Report as valid and of high quality." However, the draft OIG
response is incomplete and at times misleading regarding its characterization of what EPA
conducted and documented in order to be confident that the IPCC assessment reports complied
with EPA and OMB standards of quality, objectivity, utility, and integrity.
First, both in the main body of the OIG report and in the draft response to this question from
Senator Inhofe, the OIG fails to fully recognize or provide any documentation regarding EPA's
actual involvement in the U.S. Government process to review, comment on, and assist in the
U.S. Government approval of the IPCC assessment reports. Instead, the OIG makes a number of
statements that to most readers will imply that EPA passively took note of the U.S. Government
approval of the IPCC assessments, and used this as 'trust-without-verification' rationale for
making use of the IPCC Fourth Assessment Report in the endangerment finding. The reality is
that EPA devoted significant staff time to participate, and at times play leading roles, in the U.S.
Government review, comment and approval process for the IPCC Fourth Assessment Report. For
each of the three major volumes of the IPCC Fourth Assessment Report, EPA comments were
collated and submitted to the USGCRP office (the focal point for interagency review), which in
turn underwent an interagency process, of which EPA participated, to prioritize the comments
that would be submitted as U.S. Government comments from the State Department to the IPCC.
Furthermore, EPA staff were part of the small U.S. Government teams that attended week-long
IPCC meetings in 2007 to approve, in a line-by-line process, the Summary for Policymakers of
two of the three major volumes of IPCC's Fourth Assessment Report. There is documentation on
this process.
OIG Response 46: We agree that EPA's role in the IPCC process should be mentioned in the report. We added
statements to chapter 1, chapter 3, and appendix A of the final report to indicate EPA's involvement in the IPCC
process, including its participation in the U.S. delegation approving the Working Group II report for the AR4.
Second, the OIG draft report overlooks the many examples where EPA specifically discussed
and documented throughout the endangerment finding process how the IPCC peer review
procedures meet EPA and OMB guidelines. We list important examples here:
EPA's first internal briefings on the endangerment finding in 2007 document the conscious
effort to include only peer reviewed, well-vetted, consensus-based, and U.S.-approved
science for the endangerment finding. This decision lead to relying primarily on the
assessments of the USGCRP (previously the CCSP), the IPCC and the NRC.
11-P-0702
76
-------
The version of the TSD that accompanied the Advanced Notice of Proposed Rulemaking in
July 2008 described the general review procedures of IPCC, noting, for example, that:
Lead authors are nominated by governments and are selected by the respective IPCC
Working Groups on the basis of their scientific credentials and with due consideration
for broad geographic representation. For Working Group I there were 152 coordinating
lead authors, and for Working Group II 48 coordinating lead authors. Drafts prepared
by the authors are subject to two rounds of review; the second round includes
government review. For the IPCC Working Group I report, over 30,000 written
comments were submitted by over 650 individual experts, governments and
international organizations. For Working Group II there were 910 expert reviewers.
Review Editors for each chapter are responsible for ensuring that all substantive
government and expert review comments receive appropriate consideration. IPCC
documents how every comment is addressed.
OIG Response 47: Our report specifically states that EPA "provided detailed descriptions of the IPCC
principles and procedures in documents associated with its endangerment and cause or contribute findings
for greenhouse gases," and specifies that such descriptions were provided in the TSD accompanying the
proposed and final findings. The description of IPCC's procedures that is provided in EPA's TSD was
considered during the course of our evaluation and was noted in our report.
The proposed endangerment finding in April 2009 stated that, "The IPCC and CCSP
assessments base their findings on the large body of many individual, peer reviewed studies
in the literature, and then the IPCC and CCSP assessments themselves go through a
transparent peer-review process." (74 FR 18894). The proposed finding further stated, when
referring to the fact that the TSD relies on the assessments, that:
The information in the TSD has therefore been developed and prepared in a manner
that is consistent with EPA's Guidelines for Ensuring and Maximizing the Quality,
Objectivity, Utility and Integrity of Information Disseminated by the Environmental
Protection Agency.
OIG Response 48: In its proposed endangerment finding, EPA provides an overall conclusion regarding
the information quality of the assessments that EPA relied on in forming its TSD. However, the proposed
endangerment finding does not provide any description of specific IPCC procedures, nor does it provide any
language about specific evaluation procedures EPA may have employed to reach its overall conclusion that
the information in the TSD was developed and prepared in a manner consistent with information quality
guidelines.
EPA provided a description of the science used to support the endangerment finding in its final
endangerment finding, including specific references to IPCC procedures. Therefore, our report highlighted
the final findings, not the proposed findings, as a document in which EPA provided its detailed description of
IPCC procedures.
EPA's Response to Comments document in 2009, particularly Section 1 and Appendix A of
Volume 1, addressed IPCC's peer review procedures at length, and specially addressed how
they meet EPA and OMB guidelines. There, after considerable description of the IPCC
procedures and an examination of adherence to those procedures, EPA concluded:
11-P-0702
77
-------
The evidence is clear that the IPCC's procedures are sufficient and effective for
ensuring quality, transparency, and consideration of multiple and diverse perspectives.
[W]e find that IPCC's information quality process is consistent with EPA's Guidelines
for Ensuring and Maximizing the Quality, Objectivity, Utility and Integrity of
Information Disseminated by the Environmental Protection Agency.
OIG Response 49: We agree that EPA described the IPCC procedures in documents associated with the
endangerment finding. However, the documentation EPA referred us to did not show what analyses EPA
conducted prior to dissemination of the information.
As another example of EPA's response to a comment that questioned IPCC's reliance on
non-peer reviewed studies, EPA first described IPCC's procedures that "Contributions
should be supported as far as possible with references from the peer-reviewed and
internationally available literature." EPA further responded as follows:
EPA's review [emphasis added] of the studies included in the IPCC assessment reports,
which we used in developing the TSD, confirms that the use of non-peer-reviewed
literature was predominantly associated with the report of IPCC Working Group III
("Mitigation of Climate Change") and the treatment of adaptation issues in IPCC
Working Group II, two areas not assessed by the Administrator in this action.
The final endangerment finding in December 2009 spoke to the utility of the IPCC and other
assessments, stating, inter alia, that "these assessments address the scientific issues that the
Administrator must examine for the endangerment analysis." (74 FR 66510). The finding
further spoke of the high standards of peer review (74 FR 66511):
[T]hese assessment reports undergo a rigorous and exacting standard of peer review by
the expert community, as well as rigorous levels of U.S. government review and
acceptance. Individual studies that appear in scientific journals, even if peer reviewed,
do not go through as many review stages, nor are they reviewed and commented on by
as many scientists. The review processes of the IPCC, USGCRP, and NRC (explained
in fuller detail in the TSD and the Response to Comments document, Volume 1)
provide EPA with strong assurance that this material has been well vetted by both the
climate change research community and by the U.S. government. For example, with
regard to government acceptance and approval of IPCC assessment reports, the
USGCRP Web site states that: "When governments accept the IPCC reports and
approve their Summary for Policymakers, they acknowledge the legitimacy of their
scientific content."
No other source of information provides such a comprehensive and in-depth analysis
across such a large body of scientific studies, adheres to such a high and exacting
standard of peer review, and synthesizes the resulting consensus view of a large body of
scientific experts across the world.
11-P-0702
78
-------
OIG Response 50: We acknowledge in appendix A of our report that EPA provided detailed descriptions of
IPCC procedures in the final endangerment finding. Additionally, some of the relevant points from the two
paragraphs cited above were already included in the draft report. For example, appendix A of the report
notes the following:
According to EPA, the assessment reports summarized in the TSD, including IPCC's
AR4, "were prepared following rigorous and transparent processes addressing such
issues as the nomination and selection of authors, the caliber of literature reflected in
the assessment, and the processes for review and revision of reports."
EPA also stated in its TSD that the procedures employed by the IPCC, among others,
provided the Agency with assurances that the assessment material was well vetted by
both the climate change community and by the U.S. government.
In EPA's 2010 denial of petitions to reconsider the endangerment finding, EPA concluded
the following after considering and reviewing specific critiques raised against IPCC's Fourth
Assessment Report (75 FR 49558):
Petitioners' also point to a limited number of factual mistakes in IPCC AR4, some
confirmed, some alleged, to argue that the climate science supporting the
Administrator's Endangerment Finding is flawed. EPA's review confirmed two factual
mistakes. These two confirmed instances of factual mistakes are tangential and minor
and do not change the key IPCC AR4 conclusions that are central to the
Administrator's Endangerment Finding. While it is unfortunate that IPCC's review
process did not catch these errors, in the context of a report of this size and scope
(almost 3,000 pages), it is an inappropriate and unfounded exaggeration to claim that
these two confirmed mistakes delegitimize all of the scientific statements and findings
contained in IPCC AR4. To the contrary, given the scrutiny to which IPCC AR4 has
been subjected, the limited nature of these mistakes demonstrates that the IPCC review
procedures have been highly effective and very robust.
The accompanying Response to Petitions document further details at considerable length
EPA's review of and responses to a number of critiques raised about IPCC review
procedures, particularly volume 2 "Issues Raised by Petitioners on EPA's Use of IPCC." In
that summary, after considerable discussion, EPA concluded:
o It was proper to not list certain scientists as contributing authors as they did not
contribute significantly to the writing and editorial decisions in developing any
Fourth Assessment Report chapter, including Chapter 6 of Working Group I's
contribution (Jansen et al., 2007), and therefore did not compromise their objectivity
during the peer-review process.
o IPCC authors did not cite their own studies more frequently than what was acceptable
and reasonable.
o IPCC authors were not directed to focus on policy-prescriptive conclusions, but rather
implemented IPCC guidelines by presenting policy-relevant and neutral findings.
11-P-0702
79
-------
o IPCC authors did not alter the content of reports to eliminate suggestions of non-
consensus.
o Collaborations among IPCC authors and reviewers prior to the development of AR4
did not compromise objectivity or generate conflicts of interest.
o The IPCC's peer-review processes are appropriate and adequate, and were properly
implemented.
o IPCC authors did not manipulate deadlines for receipt of new literature.
o The IPCC's very limited use of gray literature does not call into question the quality
and objectivity of the assessment reports.
OIG Response 51: EPA's response to petitions document does show an additional degree of review and
evaluation that the Agency applied to IPCC procedures. Accordingly, we added a discussion of this review to
our response to question 1 in appendix A of the final report.
Despite this documented record, the OIG draft response makes the following two statements that
we think are misleading and inaccurate:
EPA examined IPCC procedures to the extent that EPA described these procedures in its
proposed andfinal endangerment finding packages.
EPA did not conduct any independent review or analysis of the IPCC's compliance with
these procedures.
The first statement is misleading, because it implies that EPA did nothing more than describe
IPCC's procedures. As discussed above, EPA has done more than that. EPA has been a
significant participant in the review of IPCC assessment reports, and EPA has documented
throughout the endangerment process not only the IPCC review procedures themselves, but how
IPCC procedures have addressed and adhered to high standards of transparency, objectivity and
integrity such that they meet EPA and OMB guidelines. We therefore request that OIG edit this
statement to reflect that EPA did more than simply describe IPCC's review procedures.
The second statement is inaccurate because it states that EPA conducted no review of IPCC
procedures. This statement overlooks the documentation described above. If there is a particular
kind of review that OIG thinks EPA should have conducted (e.g., an audit of all comments
submitted to IPCC and IPCC's responses to all of those (thousands of) comments), or if the OIG
thinks that EPA's documentation is not sufficiently consolidated into one place, or is not cross-
walked explicitly enough with each EPA and OMB data quality criterion, then the OIG should be
more explicit about that. We therefore request that OIG edit this second statement, and all similar
statements, to reflect the fact that EPA did review and documented its review of both IPCC
procedural and data quality issues at multiple stages during the endangerment finding process.
11-P-0702
80
-------
OIG Response 52: To address EPA's concerns with the first statement it quoted from the draft report, above,
we added the following statement to the introductory paragraph of question 1 of appendix A:
After disseminating its endangerment finding, in response to petitions for reconsideration,
the Agency took further steps to examine and evaluate IPCC procedures. As part of this
process, the Agency evaluated evidence provided by petitioners related to allegations
that IPCC peer review and report development procedures are designed inappropriately.
We deleted the second statement quoted from the draft report because the Agency does not have guidance
specifying the level of review required for such situations. Chapter 3 discusses the need for EPA to develop
guidance in this area.
2. IPCC procedures require that it consider all information and scientific viewpoints. Examine
how EPA evaluated and determined that the IPCC examined all viewpoints.
The OIG response to this question provides a reasonable yet limited overview of EPA's approach
to evaluating and determining that IPCC examined all view points. In addition to the current
response, OIG should also explain that EPA reviewed all studies received during the public
comment period following the proposed endangerment finding that were not considered or
reflected in the major assessments "to see if they would lead EPA to change or place less weight
on the judgments reflected in the assessment report. [T]he overall conclusion EPA drew from its
review of studies submitted by commenters was that the studies did not change the various
conclusions or judgments EPA would draw based on the assessment reports." (74 FR 66512)
We would also note that contained within the IPCC procedures themselves, under "Review,"
(see Volume 1, Appendix A of EPA's Response to Comments) it states that:
Three principles governing the review should be borne in mind. First, the best possible scientific
and technical advice should be included so that the IPCC Reports represent the latest scientific,
technical and socio-economic findings and are as comprehensive as possible. Secondly, a wide
circulation process, ensuring representation of independent experts (i.e. experts not involved in
the preparation of that particular chapter) fi'om developing and developed countries and
countries with economies in transition should aim to involve as many experts as possible in the
IPCC process. Thirdly, the review process should be objective, open and transparent.
OIG Response 53: We believe the draft report's response to this question is appropriate. We have not added
the additional information suggested by EPA since it is not clear how EPA's review of studies not included in
the major assessments pertains to a review of IPCC's procedures for including all viewpoints.
3. Was EPA aware of editing of final IPCC assessment reports after the reviewers submitted
their final comments?
EPA wishes to clarify that the question should be whether any IPCC edits occurred after USG
approval of the reports, not after reviewer comments were received. It would be expected that
the IPCC would make changes after receiving reviewer comments. Assuming the OIG intended
for the response to indicate that EPA was not aware of editing of the final IPCC report after
approval, EPA has no concerns with the OIG response.
11-P-0702
81
-------
OIG Response 54: We acknowledge EPA's comments, but are not providing further changes or clarifications
to the report. The question is written as provided to the OIG, and as we asked it to EPA personnel during the
course of our evaluation.
4. Was the Endangerment Finding's Technical Support Document (TSD) subjected to peer
review as specified in the EPA Peer Review Handbook? If not, please provide EPA's explanation
for why it was not.
As currently written, this response appears inconsistent with OIG's main report. For example, as
pointed out above, the first sentence of the OIG answer states that the peer review process for the
TSD would not satisfy requirements for "influential scientific information." This is inaccurate,
inconsistent with the OIG report itself, and should be corrected.
We refer OIG to our comments on the main body of the OIG report where we describe in detail
that each step undertaken for the expert review of the TSD followed EPA and OMB information
quality guidelines for peer review. We would also note that this response should include the
context that the assessments of the NRC, the USGCRP and the IPCC are themselves subject to
high standards of external peer review. The TSD, as described more fully earlier in EPA's
response, is a summary of the findings of these previously peer-reviewed assessment reports.
OIG Response 55: The OIG response to the Senator's question is consistent with other statements in the
report. The Peer Review Handbook, on pages 2-4, specifically recommends that EPA follow certain procedures
for planning (e.g., create peer review record), conducting (e.g., ask reviewers to prepare peer review report),
and completing (e.g., prepare Agency response to reviewer comments) peer reviews for both influential scientific
information and highly influential scientific assessments. While the Agency has discretion to choose the peer
review mechanism for influential scientific information (e.g., letter reviews, ad hoc panels), the handbook
recommends that the Agency follow certain procedures for planning, conducting, and completing peer reviews
for both influential scientific information and highly influential scientific assessments.
5. EPA has acknowledged sending the Draft TSD to a group of federal climate change experts
for review. Apparently this was done for a number of versions of the Draft TSD. Were changes
made to the Draft TSD based on these federal reviewers' comments? Did this process follow
EPA's, as well as OMB's, peer review guidelines?
The answer to this question is addressed in the main body of the report, and it is not clear that a
separate response here is needed if it does not provide a full accounting or context.
We disagree with OIG's statement that:
EPA did not maintain documentation showing its responses to and disposition of comments
reviewers made to the versions of the TSD that accompanied the proposed andfinal actions.
OIG Response 56: This statement is correct. The OIG asked EPA for its responses to the federal climate
change reviewers' comments on the versions of the TSD that accompanied the proposed and final actions,
which would include statements on whether it agreed or disagreed with the reviewers' comments and how it
addressed the comments (e.g., deleted/added a sentence, made edits to a sentence). EPA did not develop
responses to reviewers' comments for the last two reviews of the TSD performed by the federal climate change
reviewers. The OMB peer review bulletin and EPA Peer Review Handbook call for the preparation of responses
to reviewer comments.
11-P-0702
82
-------
It is also incorrect to say that:
The federal climate change expert review of the TSD did not follow the recommended
procedures outlined in the Peer Review Handbook or OMB 's guidelines for peer review of
influential scientific information or highly influential scientific assessments.
EPA believes the TSD is an ISI and that the reviews conducted for the TSD went beyond the
requirements for what is required for an ISI in the EPA Peer Review Handbook. The draft TSD
was reviewed by 12 federal climate change science experts. Each of these individuals has
significant scientific credentials and represents the balance of expertise needed to cover the range
of topics summarized in the TSD. The OIG report does not describe the credentials of the 12
reviewers, and we view this as a fundamental omission because it is one of the most important
criteria for a credible and robust peer review. As discussed above, the review provided for the
TSD was comprehensive and fully appropriate for the nature of the document - a summary of
existing peer-reviewed scientific assessments. EPA never viewed the TSD as a scientific
assessment and did not weigh the science, draw any new conclusions, nor identify or fill gaps in
the science. The charge to the federal expert reviewers was: ... to provide us with any general or
detailed comments whether or not the TSD is a fair and accurate reflection of the current state of
climate change science, as embodied in the major assessment reports such as IPCC,
USGCRP/CCSP and NRC.
OIG Response 57: The statement has been revised to say "the federal climate change expert review of the
TSD did not follow all recommended procedures outlined in the Peer Review Handbook or OMB's guidelines for
peer review of influential scientific information or highly influential scientific assessments." This statement
concerns whether EPA followed procedures for the peer review of influential scientific information and highly
influential scientific assessments as outlined in the EPA Peer Review Handbook and OMB peer review bulletin.
The handbook outlines recommended procedures for planning (e.g., create peer review record), conducting
(e.g., ask reviewers to prepare peer review report), and completing (e.g., prepare Agency response to reviewer
comments) a peer review for influential scientific information and highly influential scientific assessments. The
federal climate change expert review of the TSD did not follow all recommended procedures in the handbook for
influential scientific information or highly influential scientific assessments. For example, as noted in the draft
report, the following items specified in the handbook were not developed or obtained:
Formal peer review record
Peer review report
EPA's response to the reviewers' comments on the TSD versions that accompanied the proposed and
final rules
Written management approval of EPA's response to the reviewers' comments on the TSD that
eventually accompanied the ANPR
Similarly, the bulletin outlines procedures/requirements for the selection of reviewers (e.g., independence from
sponsoring agency), providing reviewers with sufficient background information, public participation in peer
review, preparation of a peer review report, and authorization to have a separate entity carry out peer review.
The federal climate change expert review of the TSD did not follow all procedures/requirements in the bulletin
for highly influential scientific assessments. For example, the Agency did not consider asking the public to
nominate reviewers nor did it ask the reviewers to prepare a peer review report.
The three rounds of expert review were consistent with the EPA Peer Review handbook, which
recommends the inclusion of a clear charge, that no conflicts of interest exist, a timeline for
review, and documentation of comments and responses. All of the reviewer comments were
maintained, multiple versions of the draft TSD were archived, all peer reviewers were disclosed,
and EPA submitted a memo to the record (see EPA-HQ-OAR-2009-0171-11639) documenting
11-P-0702
83
-------
all the changes to the TSD that were made in response to all levels of comments. This detailed
process was completed for all three rounds of federal expert review, not just for the ANPR, as
stated by the draft OIG report. EPA has maintained documentation for all of the comments
received from federal experts and its response to these comments. Furthermore, although EPA
did not docket the comments from the federal expert reviewers, the public was given two full
opportunities to comment on the TSD: once during the ANPR (a 120-day comment period) and
again following the proposed endangerment finding (a 60-day comment period including two
public hearings). Although EPA recognizes that a single peer-review report with one table
itemizing all of the comments received during the three rounds of federal expert review and how
they were responded to would have provided the OIG with a simpler way for the OIG to consider
all comments and responses at once, all basic core requirements have been met and the OIG
should so state in this question response. If the OIG's conclusion is that, although the
requirements are satisfied, the information should have been presented more clearly, it should
state so.
OIG Response 58: EPA did not develop responses to reviewers' comments for the last two reviews of the TSD
performed by the federal climate change reviewers, as required by the OMB Peer Review Bulletin. EPA
maintained documentation showing whether changes were made in response to reviewers' comments, but not
showing its assessment of the comments, including reasons for rejecting comments.
6. Assess the Interagency review process used in developing the Endangerment Finding. Were
there significant interagency comments on the finding? How were these resolved?
The OIG response contains an ambiguous statement that "In two instances, EPA's actions to
address the OMB/interagency comments did not appear to be directly responsive to the
comments." EPA is not provided with any information regarding what is being referred to by this
statement, and this appears to be the first time here that the OIG is questioning a substantive
response to an interagency comment. This seems to be outside the scope of OIG's procedural
review, and in any case is unfounded, since OMB cleared the document. This clearance meant
that OMB was satisfied that all interagency comments had been dealt with satisfactorily.
OIG Response 59: The draft report acknowledged that "OMB approved all EPA actions and responses to
OMB/interagency review comments." Our independent review of OMB/interagency comments and EPA's
responses to those comments found that two of EPA's responses were not directly responsive to
OMB/interagency comments. However, we do not believe these two instances to be significant. Thus, we added
further clarification to the final report to state that (1) we did not find these two instances to be significant, and
(2) OMB approved all EPA responses.
7. In recent months a number of e-mails f'om the Climatic Research Unit ("CRU") of the
University of East Anglia in the United Kingdom were released. EPA has claimed that these e-
mails do not affect the fundamental findings of the IPCC assessment reports. What analyses has
EPA conducted to reach this conclusion, in particular its conclusion regarding the HadCRUT
temperature dataset and its relation to other data sets used in the endangerment finding f~om
NOAA [National Oceanic and Atmospheric Administration] and NASA [National Aeronautics
and Space Administration] ?
The OIG correctly notes that EPA evaluated concerns related to the CRU/HadCRUT temperature
record as part of a two-step process. The first step of EPA's evaluation was after the public
11-P-0702
84
-------
comment period but prior to finalizing the Endangerment and Cause or Contribute Findings. The
second was in response to the Petitions for Reconsideration. The OIG also correctly notes that
even absent the CRU temperature dataset, the scientific evidence overwhelmingly points towards
a long-term trend of global warming, as communicated by the NOAA/NCDC Director.
The OIG's response to Senator Inhofe's question incorrectly states that "EPA did not provide
OIG with documentation for either step that showed it had independently verified the
temperature records for CRU, NOAA, or NASA." As correctly noted by the OIG in the report,
"EPA relied on external peer review and investigations of the datasets, as well as the larger body
of scientific evidence, to ensure that data met Federal and Agency information quality
guidelines." However, a complete answer should also indicate that EPA evaluated the
implications of the CRU E-mails in-depth in EPA's Response to the Petitions to Reconsider the
Endangerment Finding. Issues regarding the scientific implications of the emails for the
fundamental findings of the IPCC assessment reports were addressed in Volume 1 of the
Response to Petitions, including more than 70 pages of analysis based on questions by
petitioners regarding the HadCRUT, NOAA, and NASA temperature records.
In order to respond to the petitions, EPA staff read and reviewed all the CRU E-mails, evaluated
quotes highlighted by petitioners in their full context, and reviewed the underlying literature in
order to determine whether the petitioners had raised any issues that would change the
conclusions reached in the assessment reports on which the Administrator relied for the Findings.
EPA found that for the issues raised by petitioners, the assessment reports were consistent with
the underlying literature, and that the assessment reports and the literature recognized and
appropriately discussed the uncertainties that were highlighted by petitioners in the CRU E-
mails. EPA also reviewed the five recent inquiries and investigations and found that all five
investigations reached conclusions consistent with those reached independently by EPA.
With regards to the HadCRUT, NOAA, and NASA temperature records, EPA read the peer-
reviewed literature describing the methodologies used by the three research groups for collecting
and analyzing temperature data. EPA considered the CRU E-mails, the complete unadjusted and
adjusted data from NOAA and NASA as well as the available data from HadCRUT, the
publically available code from NASA, the reconstructions of the HadCRUT and NASA results
by independent groups, and the observations of a warming system based on other indicators such
as satellite data, retreating glaciers, and rising sea levels. Based on this analysis, EPA was able to
determine that objections by petitioners that the temperature records were flawed based on
"station dropout" and alleged inappropriate data adjustments were unfounded, and that the
consistency between different methodologies and indicators was strong confirmation of an
unambiguous warming trend over the last 100 years.
We also note that a recent NOAA OIG report examined all 1073 CRU emails in the context of an
investigation into whether inappropriate manipulation of temperature datasets occurred and
stated that they "did not find any evidence that NOAA inappropriately manipulated data."
OIG Response 60: We added a statement to our response to question 7 in appendix A of the final report that
notes EPA's review of CRU e-mails and underlying literature as part of its evaluation and response to petitions.
However, we believe that our report provides an appropriate and accurate response to the Senator's question.
Our review of EPA's response to petitions document found that EPA did not conduct its own independent
verification of the temperature datasets. Instead, it based its conclusions on the factors noted in appendix A.
11-P-0702
85
-------
C. Title of the Draft Report Leaves Room for Misinterpretation
We believe the title should reflect the procedural nature of the evaluation and should not imply
that OIG evaluated the inherent quality of the information supporting the Finding. We
recommend modifying the title to: Procedural Review of Greenhouse Gases Endangerment
Finding Information Evaluation Processes. This would help to avoid misinterpretation that the
Report examined the quality of the information itself.
OIG Response 61: Our evaluation focused on the data quality procedures EPA used in developing the
endangerment finding. These procedures are intended to provide assurances that the data are of sufficient
quality for their intended use. We believe the title appropriately describes the focus of our evaluation.
D. Appropriate EPA Office for Draft OIG Recommendation Regarding Document, A
Summary of General Assessment Factors for Evaluating the Quality of Scientific and
Technical Information
The Office of Environmental Information should not be named as the recipient of this
recommendation as OEI has no responsibility for the Assessment Factors document.
OIG Response 62: We removed the Assistant Administrator for Environmental Information and Chief
Information Officer as an action official for recommendation 3 of the report. The recommendation was
co-addressed to this official because the Office of Environmental Information helped develop and co-signed the
subject guidance document.
E. New Science Information
The last item we would request that the OIG consider for context is that the scientific
assessments published since the time of the 2009 endangerment finding have reaffirmed the
scientific conclusions EPA relied upon in making the endangerment finding. For example, the
2011 NRC report, "Climate Stabilization Targets," states, "Evidence now shows that the
increases in these [greenhouse] gases very likely (>90 percent chance) account for most of the
Earth's warming over the past 50 years." Another 2011 NRC report, "America's Climate
Choices," states that "Climate change is occurring, is very likely caused by human activities, and
poses significant risks for a broad range of human and natural systems. Each additional ton of
greenhouse gases emitted commits us to further change and greater risks. In the judgment of the
Committee on America's Climate Choices, the environmental, economic and humanitarian risks
of climate change indicate a pressing need for substantial action to limit the magnitude of climate
change and to prepare to adapt to its impacts."
OIG Response 63: These publications are outside of the scope of the OIG's evaluation. As stated in the
"Limitations" section of our Scope and Methodology, we did not examine the scientific merit of the information
supporting EPA's endangerment finding.
11-P-0702
86
-------
Appendix H
OMB Comments on Draft Report and
OIG Evaluation of OMB Comments
EXECUTIVE OFFICE OF THE PRESIDENT
OFFICE OF MANAGEMENT AND BUDGET
WASHINGTON, D C. 20503
June 17,2011
Wade T. Najjum
Assistant Inspector General for Program Evaluation
United States Environmental Protection Agency
Washington, D.C. 20460
Dear Mr. Najjum:
Thank you for the opportunity for the Office of Management and Budget (OMB) to
provide comments on your office's draft report entitled "Procedural Review of Greenhouse Gases
Endangerment Finding Data Quality Processes," dated May 3, 2011 (Draft Report).
The Draft Report notes that your office was asked to determine, among other questions,
whether EPA followed OMB's peer review guidelines in preparing the Technical Support
Document (TSD) accompanying EPA's "Endangerment and Cause or Contribute Findings for
Greenhouse Gases Under Section 202(a) of the Clean Air Act." 74 Fed. Reg. 66496 (2009)
(Endangerment Finding). The Draft Report notes (at page 15) that EPA had the TSD reviewed by
a panel of climate change scientists, and that the methodology employed for this review was an
appropriate exercise of the discretion afforded the agency for peer reviews of "influential
scientific information," as defined in OMB's Final Information Quality Bulletin for Peer Review
(OMB Bulletin). However, the Draft Report concludes that the TSD met OMB's definition of a
"highly influential scientific assessment," and that the TSD did not satisfy all of the peer review
procedures that OMB had established in the OMB Bulletin for a "highly influential scientific
assessment."
As the author of the OMB Bulletin, and as OMB explained in the comments that OMB
provided to your office regarding this TSD on April 15, 2011 (which are contained in Appendix
G of the Draft Report), OMB believes that EPA reasonably interpreted the OMB Bulletin in
concluding that the particular TSD that EPA prepared in this case did not meet the Bulletin's
definition of a "highly influential scientific assessment":
11-P-0702
87
-------
1. Section 1(7) of the OMB Bulletin defines a "scientific assessment" as "an evaluation
of a body of scientific or technical knowledge," including "state-of-science reports." In this
case, EPA concluded that it was the separate, pre-existing and peer-reviewed assessments by
IPCC, USGCRP, and NRC that constituted such evaluations of the state of the science. As EPA
explained at the time, these three preexisting peer-reviewed assessments "evaluate[d] the
findings of numerous individual peer-reviewed studies in order to draw more general and
overarching conclusions about the state of science," and "synthesize[d] literally thousands of
individual studies and convey[ed] the consensus conclusions on what the body of scientific
literature tells us." 74 Fed. Reg. at 66511.
OIG Response 1: We agree with OMB that the IPCC, USGCRP, and NRC assessments that EPA used to
support its endangerment finding did, in fact, meet OMB's definition of a "scientific assessment." However, in
synthesizing the findings, conclusions, and other information from these assessment reports (and other sources)
in its TSD, EPA was evaluating the state of science and producing an entirely new and separate document that
also met OMB's definition of a "scientific assessment."
We note that the Agency stated in its endangerment finding that it gave "careful consideration to all the scientific
and technical information in the record" but relied on the assessments of the USGCRP, ICPP, and NCR as the
primary scientific and technical basis for the finding. EPA's TSD referenced multiple sources, including "up-to-
date" data from sources other than the "major scientific assessments." Specifically, EPA cited 28 core
references in its TSD. In evaluating the scientific information, the Agency "placed limited weight on the much
smaller number of individual studies that were not considered or reflected in the major assessments." EPA
reviewed such studies "largely to see if they would lead EPA to change or place less weight on the judgments
reflected in the assessment report." The Agency stated in the endangerment finding that "the studies did not
change the various conclusions or judgments EPA would draw based on the assessment reports."
2. The TSD accompanying EPA's decision provided a condensed form of the three
underlying peer- reviewed assessments and, with respect to the key conclusions in the
Endangerment Finding, the TSD is in many respects simply a word-for-word transcription of
the summary conclusions that are contained in those peer-reviewed assessments. Rather than
creating a new assessment, we understand that EPA instead relied on the three pre-existing
peer-reviewed assessments, and - rather than requiring interested persons to read the entirety of
these lengthy assessments - EPA included in the TSD a reader-friendly version of those
passages (from those pre-existing peer-reviewed assessments) on which EPA was relying for
making its determination.
OIG Response 2: OMB's response to the OIG draft report emphasizes EPA's reliance on "the major scientific
assessments of the USGCRP, IPCC, and NRC" as the primary scientific and technical basis for the
Administrator's endangerment finding. However, EPA used more than "three pre-existing peer-reviewed
assessments," as OMB suggests, as references for its TSD. As explained in OIG Response 1 above, the
Agency considered and cited other sources of information in its TSD.
Descriptions in the endangerment finding and TSD show that the Agency placed value judgments on certain
sources of information. EPA synthesized conclusions and scientific findings from various assessment reports,
and other sources, in a single document that had as a stated purpose "to provide scientific and technical
information for an endangerment and cause or contribute analysis regarding greenhouse gas (GHG)
emissions." Nowhere in the TSD does it state that the purpose of the document is to "provide a reader friendly
version" of the underlying assessments. EPA also evaluated, and weighed, a broader universe of scientific
information than simply the major assessments of USGCRP, IPCC, and NRC.
11-P-0702
88
-------
3. We believe EPA reasonably concluded that it was the three pre-existing peer-
reviewed assessments that were identified in the TSD, and not the TSD itself, that proved
"highly influential" to EPA's determination in its Endangerment Finding.
In this regard, it is important to recognize that the OMB Bulletin does not characterize
as "highly influential" every agency document that identifies the evidence upon which an
agency relies in making its decision, including in the case of rulemaking decisions. Here, the
Endangerment Finding noted that EPA had concluded that these three pre-existing peer-
reviewed assessments "represent the best reference materials" on which EPA was prepared to
rely, and therefore that EPA had "no reason to believe that putting this significant body of work
aside and attempting to develop a new and separate assessment would provide any better basis
for making the endangerment decision." Accordingly, EPA determined that EPA would not
"perform a new and independent assessment of all of the underlying climate change science."
74 Fed. Reg. at 66511. Thus, this particular TSD served a very different purpose than have
other EPA-prepared documents such as Integrated Science Assessments (ISAs) that EPA has
developed when it is considering making changes to National Ambient Air Quality Standards
(NAAQS). For example, EPA describes the 2008 ISA that EPA developed for sulfur oxides as a
"concise review, synthesis, and evaluation of the most policy-relevant science" that "form[edl
the scientific foundation for the review of the primary (health-based) NAAQS" for sulfur oxides
and was subject to an external peer review process through the Clean Air Scientific Advisory
Committee. EPA, Integrated Science Assessment (ISA) for Sulfur Oxides - Health Criteria
(Sept. 2008) at 1-1, available at http://cfpub.epa.gov/ncea/cfm/recordisplav.cfm?deid=198843.
By contrast, the TSD that EPA prepared in connection with the Endangerment Finding
did not provide a synthesis of the state of the science, and this TSD did not serve as the
scientific foundation for the Endangerment Finding. Rather, in the case of the Endangerment
Finding, it was the pre-existing peer-reviewed assessments that provided a "state-of-science"
synthesis as well as the scientific foundation for the agency's finding.
OIG Response 3: We agree that the major assessments cited in EPA's TSD are "highly influential." We also
believe that the TSD is "highly influential." As noted in OIG Response 2 above, the stated purpose of the TSD
was "to provide scientific and technical information for an endangerment and cause or contribute analysis
regarding greenhouse gas (GHG) emissions." EPA classified its endangerment and cause or contribute
findings for greenhouse gases as a Tier 1, significant regulatory action because it raises novel policy issues.
The TSD provided scientific and technical information to support that action. For that reason, and because the
TSD synthesizes information from multiple sources (beyond just those of the USGCRP, IPCC, and NRC
assessments), the TSD itself should be considered a highly influential scientific assessment, subject to the
applicable OMB peer review requirements for that type of information.
4. The Draft Report relies, in support of its contrary application of the OMB Bulletin, on
a statement that OMB made to your office last September, which OMB provided in response to
questions that your office had posed to OMB last August. (The questions and responses are
contained in Appendix F to the Draft Report.) In this regard, we note that OMB made this
statement in response to questions posed by your office of a general nature, and not with
reference to any particular document. Neither your office's general question, nor OMB's general
response, referred to the TSD in this case. When evaluating whether a proposed course of action
complies with the Bulletin's requirements, the primary focus should be on the text of the
Bulletin, because it is the Bulletin that outlines the formal, public and authoritative direction by
11-P-0702
89
-------
OMB to agencies regarding these peer review requirements.
Moreover, the statement that OMB made last September, in response to your office's
general questions, is entirely consistent with EPA's conclusion that the TSD which EPA prepared
in connection with the Endangerment Finding was not a "highly influential scientific
assessment." In the statement from last September, OMB noted that a document "summarizing
the 'state of the science'" would qualify as a "scientific assessment," because such a document
"implicitly or explicitly weighs the strength of the available evidence" (see Appendix F of the
Draft Report, page 46). In this case, for the reasons stated above, that statement describes the
syntheses that are reflected in the pre-existing peer-reviewed IPCC, USGCRP, and NRC
assessments, which were "state of the science" assessments. But this statement does not describe
the TSD here, and that is because this TSD did not "implicitly or explicitly weigh[] the strength
of the available evidence." Rather, it was the three pre-existing peer-reviewed assessments that
did so, and - as noted above -EPA decided to make its Endangerment Finding based on those
pre-existing peer-reviewed assessments rather than taking the alternative course to "perform a
new and independent assessment of all of the underlying climate change science."
OIG Response 4: We continue to believe that the TSD is a scientific assessment based on the definition in the
OMB peer review bulletin. EPA evaluated a body of scientific knowledge and synthesized multiple factual
inputs in the development of the TSD for the stated purpose of "providing] scientific and technical information
for an endangerment and cause or contribute analysis regarding greenhouse gas (GHG) emissions from new
motor vehicles and engines under Section 202(a) of the Clean Air Act." Specifically, EPA's TSD lists 28
separate core references, with 24 of them coming from IPCC, USGCRP (CCSP), and NRC. The other four
references included information from NOAA and EPA's assessment of the literature on the effect of climate
change on air quality. By relying primarily on assessment reports from IPCC, USGCRP (CCSP), and NRC,
EPA placed a value judgment on these sources of information. It is clear that EPA pulled from multiple inputs
and synthesized them into a single documentthe TSD. EPA also made decisions as to which information
from these assessments to include in and exclude from its TSD.
OMB's comments to the draft report imply that the OIG was too vague or general in an attempt to clarify OMB
guidance. We contacted OMB in July 2010, requesting to speak to the contacts listed in the OMB information
quality guidelines guidance and the OMB peer review bulletin. We were subsequently told by OMB personnel
to work through the Assistant General Counsel. The Assistant General Counsel asked us to submit written
questions, and we submitted questions, along with the Senator's request letter. In an August 11, 2010, e-mail,
we requested clarification on OMB's information quality and peer review guidelines as they relate to
(1) reviewing another organization's data quality and peer review procedures prior to disseminating information
from a scientific assessment published by that organization, and (2) defining a scientific assessment. We were
transparent in explaining the assignment we were conducting. Our August 11, 2010, e-mail stated:
The EPA OIG is currently evaluating EPA's development of its endangerment and cause or
contribute findings for greenhouse gases. Our objective is to determine whether EPA
followed key federal and Agency regulations and policies in developing and reviewing the
technical data used to support and make its endangerment finding. This evaluation was
initiated based on a request from Senator James M. Inhofe, Ranking Member, Senate
Committee on Environment and Public Works. Attached below are copies of the Senator's
request letter and the OIG's notification memo to EPA's Assistant Administrator for Air and
Radiation.
The Assistant General Counsel e-mailed the responses to our questions with a cc to the Deputy General
Counsel on September 10, 2010. This e-mail exchange was just one component of many sources used in
forming our conclusion.
Given EPA's decision to rely directly on those pre-existing peer-reviewed assessments,
EPA could have reasonably (and justifiably) concluded that the OMB Bulletin does not require
11-P-0702
90
-------
an agency to perform an entirely new (duplicative) peer review - for statements which are
contained in pre-existing peer-reviewed assessments on which the agency is directly relying -
simply because the agency has decided to include, in an agency-prepared document for ease of
use by the public and stakeholders, summaries of the relevant passages from those peer-
reviewed assessments on which the agency is relying. Such a conclusion would be consistent
with the OMB Bulletin, which states that the Bulletin's requirements for more intensive peer
review "apply only to the more important scientific assessments" disseminated by agencies.
OMB Bulletin at 2.
OIG Response 5: We have not called into question EPA's charge to the federal reviewers that reviewed the
TSD. Further, it is not our position that EPA should have required a duplicative peer review of the science
included in the underlying references cited in EPA's TSD. Instead, it was our conclusion that a peer review of
the TSD should have been conducted according to the requirements outlined in OMB guidance, and EPA's
Peer Review Handbook, for highly influential scientific assessments.
Also, as explained in chapter 2 of the final report, EPA did not characterize the TSD as influential scientific
information (or as a highly influential scientific assessment) during the action development process. EPA first
characterized the TSD as influential scientific information in response to our draft report.
We hope that these views are helpful in your consideration of how EPA applied the
guidelines set forth in the OMB Bulletin to the TSD which EPA prepared in connection with its
Endangerment Finding.
Sincerely,
Michael A. Fitzpatrick
Associate Administrator
Office of Information and Regulatory Affairs
11-P-0702
91
-------
Appendix I
Distribution
Office of the Administrator
Assistant Administrator for Air and Radiation
Assistant Administrator for Research and Development
Agency Followup Official (the CFO)
Agency Followup Coordinator
General Counsel
Deputy Assistant Administrator for Management, Office of Research and Development
Deputy Assistant Administrator for Science, Office of Research and Development
Director, Office of Atmospheric Programs, Office of Air and Radiation
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for External Affairs and Environmental Information
Audit Followup Coordinator, Office of Air and Radiation
Audit Followup Coordinator, Office of Research and Development
11-P-0702
92
------- |