United States
Environmental Protection
Agency
Science Policy Council
       M

-------
                                          EPA 100/B-03/001
                                               June 2003
I
3)         2
 \
   U.S. Environmental Protection Agency

   A Summary of General Assessment Factors for
  Evaluating the Quality of Scientific and Technical
                    Information
    Prepared for the U.S. Environmental Protection Agency
by members of the Assessment Factors Workgroup, a group of the
               EPA's Science Policy Council
                 Science Policy Council
           U.S. Environmental Protection Agency
                 Washington, DC 20460

-------
Page ii	Assessment Factors
                                  DISCLAIMER

      This document has been reviewed in accordance with United States Environmental
Protection Agency policy and approved for publication and distribution. Mention of trade names
or commercial products does not constitute endorsement or recommendation for use.

-------
                                Assessment Factors	Page iii
                            TABLE OF CONTENTS

FOREWORD	Page iv

ACKNOWLEDGMENTS	Page vi

A Summary of General Assessment Factors for Evaluating the Quality of Scientific
      and Technical Information	 Page 1

      1.     Introduction	 Page 1
             1.1    Overview	 Page 1
             1.2    Purpose	 Page 1
             1.3    Background  	 Page 2

      2.     Assessment Factors  	 Page 4
             2.1    General Assessment Factors 	 Page 4
             2.2    Examples of Questions Raised by Consideration of the
                   Assessment Factors  	 Page 5
             2.3    Relationship Between the General Assessment Factors and the
                   Elements of Quality in EPA's Information Quality Guidelines .... Page 8

      3.     Summary 	 Page 10

References  	 Page 11

-------
Page iv
Assessment Factors
                                    FOREWORD

       This document was prepared under the auspices of the Science Policy Council (SPC) to
describe the assessment factors and considerations generally used by the Agency to evaluate the
quality and relevance of scientific and technical information. These general assessment factors
are founded in the Agency guidelines, practices and procedures that make up the EPA
information and quality systems, including existing program-specific quality assurance policies.
As such, the general assessment factors do not constitute new quality-related considerations, nor
does this document describe a new process for evaluating information.  This document is
intended to raise the awareness of the information-generating public about EPA's ongoing
interest in ensuring and enhancing the quality of information available for Agency use. Further,
it complements the Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and
Integrity of Information Disseminated by the Environmental Protection Agency (EPA
Information Quality Guidelines).  This summary of Agency practice is also an additional resource
for Agency staff as they evaluate the quality and relevance of information,  regardless of source.

       Consistent with the Agency's approach to the development of the EPA Information
Quality Guidelines, this document is  the product of an open, collaborative  process between EPA
and the public. During the development of this document, EPA obtained public comments on a
draft version of the document released in September 2002 and commissioned the National
Academy of Sciences to host a workshop in January 2003 to discuss key aspects of this document
from a scientific and technical perspective.

       We want to acknowledge and thank the Assessment Factors workgroup for its steady and
insightful work in assembling this document under stringent time constraints and scrutiny.  We
particularly appreciate the efforts of the co-chairs, Haluk Ozkaynak (ORD) and Greg Schweer
(OPPTS),  who successfully led and shepherded the workgroup.

       It is with great pleasure that we present the Summary of General Assessment Factors for
Evaluating the Quality of Scientific and Technical Information.
       Paul Oilman, Ph.D.
       Science Advisor to the Agency
       Chair, Science Policy Council
             Elaine Stanley
             Director, Office of Information Analysis
                and Access
             Office of Environmental Information

-------
                          Assessment Factors	Page v
                         Science Policy Council

Paul Oilman, Science Advisor to                 Stan Meiburg, Region 4
      the Agency, Chair                      Joseph Merenda, OPPTS
Devereaux Barnes, OSWER                    William Muszynski, Region 2
Robert Brenner, OAR                         Joanne Rodman, OCHP
Jerry Clifford, OIA                            Michael Ryan, OCFO
William Farland, ORD                         Michael Shapiro, OW
Geoffrey Grubbs, OW                         Elaine Stanley, OEI
Diana Love, OECA                            Ramona Trovato, OEI
Bharat Mathur, Region 5                       Vanessa Vu, SAB
Albert McGartland, OPEI                      Anna Wolgast, OGC

               Science Policy Council Steering Committee

Randolph Perfetti, Chair                       Kate Mahaffey, OPPTS
Thomas Baugh, Region 4                      Carl Mazza, OAR
Michael Brody, OCFO                         James Nelson, OGC
Reginald Cheatham, OEI                       Jennifer Orme-Zavaleta, ORD
Patricia Cirone, Region 10                     Rosemarie Russo, ORD
John Diamante, OIA                           Rita Schoeny, OW
Michael Firestone, OCHP                      Margaret Stasikowski, OPPTS
A. Robert Flaak, SAB                         Kevin Teichman, ORD
Jerri-Anne Garl, Region 5                      Mary Ellen Weber, OPPTS
Roland Hemmett, Region 2                     William Wood, ORD
Lee Hofmann, OSWER                        Tracey Woodruff, OPEI
Daniel Malloy, OCFO

                       Science Policy Council Staff

Edward Bender            Kerry Dearfield            Kathryn Gallagher

-------
Page vi
Assessment Factors
                            ACKNOWLEDGMENTS

      Many people worked on and contributed to the assessment factors effort that ultimately
resulted in this document. Many EPA employees from the Agency's Offices and Regions
provided input and we would like to specifically acknowledge the efforts and contributions made
by the following individuals:
      Haluk Ozkaynak, ORD, Co-Chair
      Mary Belefski, OPPTS
      Katherine Biggs, OECA
      Connie Bosnia, ORD
      Ming Chang, OEI
      Weihsueh Chiu, ORD
      Evangeline Cummings, OEI
      Kerry Dearfield, SPC Staff
      Thomas Eagles, OAR
      Kathryn Gallagher, SPC Staff
      Staci Gatica-Hebert, ORD
      Roland Hemmett, Region 2
      Annie Jarabek, ORD
      Stephen Kroner, OSWER
            Greg Schweer, OPPTS, Co-Chair
            Kevin Kubik, Region 2
            Karen Martin, OAR
            Amy Mills, ORD
            Barbara Pace, OGC
            Nicole Paquette, OEI
            Devon Payne-Sturges, OPEI
            Ward Penberthy, OPPTS
            James Quackenboss, ORD
            Mary Reiley, OW
            Bruce Rodan, ORD
            Terry Simpson, ORD
            Elaine Stanley, OEI
            Lanelle Wiggins, OPEI

-------
                                 Assessment Factors	Page 1
                   U.S. Environmental Protection Agency


    A Summary of General Assessment Factors  for Evaluating the
             Quality of Scientific and Technical Information

1.     Introduction

       1.1    Overview

       As part of the ongoing commitment of the United States Environmental Protection
Agency (USEPA) to ensure the quality of the information it uses, the Agency is publishing this
summary of general assessment factors in an effort to enhance the transparency about EPA's
quality expectations for information that is voluntarily submitted to or gathered or generated by
the Agency for various purposes. This Assessment Factors document is intended to inform
information-generating scientists about quality issues that should appropriately be taken into
consideration at the time information is generated. It is also an additional resource for Agency
staff as they evaluate the quality and relevance of information, regardless of source. The general
assessment  factors are drawn from the Agency's existing information quality systems, practices
and guidelines that describe the types of considerations EPA takes  into account when evaluating
the quality and relevance of scientific and technical information used in support of Agency
actions. As such, the general assessment factors do not constitute new quality-related
considerations, nor does this document describe a new process for evaluating information. This
document is intended to raise the awareness of the information-generating public about EPA's
ongoing interest in ensuring and enhancing the quality of information available for Agency use.

       1.2    Purpose

       The Agency believes that the summary of general assessment factors provided in this
document will serve to increase the extent to which the information-generating public builds
quality considerations into the generation and documentation of their information products. The
Agency expects that the resulting improvements in the quality of such information will enable the
Agency to more fully utilize and disseminate such information. Thus, this document is intended
to complement the Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and
Integrity of Information Disseminated by the Environmental Protection Agency (EPA
Information Quality Guidelines) (EPA, 2002) and other Agency efforts to ensure and enhance
information quality, as discussed below in Section 1.3. This document is not a regulation and is
not intended to create any legal rights or impose legally binding requirements or  obligations on
EPA or the  information-generating public.

-------
Page 2	Assessment Factors
       Although the assessment factors as presented are intended to most generally apply to
individual pieces of information, they can also be used as part of a broader evaluation of a body
of evidence that is collectively evaluated through a process typically referred to as a "weight-of-
evidence" approach. The weight-of-evidence approach considers all relevant information in an
integrative assessment that takes into account the kinds of evidence available, the quality and
quantity of the evidence, the strengths and limitations associated with each type of evidence and
explains how the various types of evidence fit together. Details as to the Agency's approach to
integrating a body of evidence depend on the type of decision or action being undertaken, and are
not the subject of this document. For instance, the Guidelines for Carcinogen Risk Assessment,
Review Draft (EPA, 1999) provides guidance on characterizing the weight-of-evidence for
carcinogenicity.  Similarly, the Guidelines for Ecological Risk Assessment (EPA, 1998) describes
the development of "lines of evidence" to reach a conclusion regarding an ecological risk
estimate.

       The general assessment factors are presented and discussed more fully in Section 2.1.
Section 2.2 presents illustrative examples of the types of questions that consideration of these
factors raise in the process of evaluating the quality and relevance of different types of
information for different uses. The relationship between these general assessment factors and the
elements of quality contained in the EPA Information Quality Guidelines is discussed in Section
2.3.

       1.3    Background

       In October 2002, EPA made available the EPA Information Quality Guidelines.  The
EPA Information Quality Guidelines were developed in response to guidelines issued by the
Office of Management and Budget (OMB, 2002) under Section 515 of the Treasury and General
Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554; H.R. 5658). The
EPA Information Quality Guidelines set forth the Agency's policy and procedural guidance for
ensuring and maximizing the quality of information disseminated by EPA, regardless of the
source of the information, and articulate the Agency's ongoing commitment to ensuring and
maximizing information quality through existing policies, systems and programs. Thus, the EPA
Information Quality Guidelines build upon the Agency's numerous existing systems, practices
and guidelines that address information quality, and provide new policies and administrative
mechanisms that respond to OMB's guidelines.

       The EPA Information Quality Guidelines also recognize that, as part of its efforts to
ensure information quality, the Agency does not wait until the point at which information is
disseminated to consider important quality principles.  Rather, the Agency recognizes that it is

-------
                                   Assessment Factors	Page 3
important to assure the quality of information through processes that incorporate quality
principles starting at the point at which information is generated.

       The Agency uses and disseminates information that is generated by a variety of sources,
including EPA itself as well as other parties that produce information through EPA contracts,
grants and cooperative and interagency agreements or in response to a requirement under a
statute, regulation, permit, order or other mandate. EPA generally has considerable control or
influence over the quality of this information at the time the information is generated. Existing
quality controls that EPA applies to the generation of information from these sources are based
on EPA's Quality System (EPA, 2000a; EPA, 2000b), Peer Review Policy (EPA, 1994), Risk
Characterization Policy (EPA, 1995) and other agency-wide and program-specific policies, as
well as specific provisions in contracts, grants, agreements, regulations and statutes. A few
additional useful web sites for obtaining further information on EPA's Quality System and
various regulatory policies and decisions are provided under the References section at the end of
this document.

       The Agency also receives information that is voluntarily submitted by or collected from
external sources, the generation of which does not come under the direct control of the Agency's
internal information quality systems. This information may include scientific studies published
in journal articles, testing or survey data, such as environmental monitoring or laboratory test
results, and analytic studies, such as those  that model environmental conditions or that assess
risks to public health.  Since EPA has placed great emphasis on the management of
environmental issues on a cooperative basis with its many stakeholders, the amount of
information submitted to EPA by external sources is increasing. Such sources include other
federal, state, tribal, local and international agencies; national laboratories; academic and
research institutions; business and industry; and public interest organizations.  Although EPA's
existing quality systems are not applied at  the time this information is  generated, EPA does apply
appropriate quality controls when evaluating this information for use in Agency actions and for
its dissemination consistent with the EPA Information Quality Guidelines. The Agency hopes
this document will inform the public of EPA's objectives and enlist them in its effort to
disseminate quality information and make quality decisions.

       During the development of this document, EPA requested public input in a variety of
ways. EPA distributed a draft document for public comment in September 2002 and hosted a
public meeting in Washington, DC. In January 2003, EPA commissioned the National Academy
of Sciences to host a workshop to discuss key aspects of this document from a scientific and
technical perspective.  EPA revised this document based on the input received through these
public outreach opportunities.

-------
Page 4	Assessment Factors
2.     Assessment Factors

       2.1     General Assessment Factors

       When evaluating the quality and relevance of scientific and technical information, the
considerations that the Agency typically takes into account can be characterized by five general
assessment factors:

       •      Soundness - The extent to which the scientific and technical procedures,
              measures, methods or models employed to generate the information are
              reasonable for, and consistent with, the intended application.

       •      Applicability and Utility - The extent to which the information is relevant for the
              Agency's intended use.

       •      Clarity and Completeness - The degree of clarity and completeness with which
              the data, assumptions, methods, quality assurance, sponsoring organizations and
              analyses employed to generate the information are documented.

       •      Uncertainty and Variability - The extent to which the variability and uncertainty
              (quantitative and qualitative) in the information  or in the procedures, measures,
              methods or models are evaluated and characterized.

       •      Evaluation and Review - The extent of independent verification, validation and
              peer review of the  information or of the procedures, measures, methods or
              models.

       These assessment factors reflect the most salient features of EPA's existing information
quality policies and guidelines. Whether the information consists of scientific theories, computer
codes for modeling environmental systems, environmental monitoring data, economic analyses,
social survey or demographic data, chemical toxicity testing, environmental fate and transport
predictions or a human health risk assessment, EPA generally evaluates information by weighing
considerations that fit within these five assessment factors. Thus, these factors encompass
considerations that are weighed in the process of evaluating the quality and relevance of
information.  The appropriate level of quality for any particular information product is
necessarily related to how and in what context the information is to be used. If EPA chooses to
later "disseminate" the information, that dissemination would be covered by the Information
Quality Guidelines which describe EPA policy and procedures for reviewing and substantiating
the quality  of information before EPA disseminates it.

-------
                                   Assessment Factors	Page 5
       When EPA considers using information for a particular purpose, careful judgment is
applied to evaluate the information for quality and relevance in the context of the specific
Agency action being developed. For instance, in the context of a given action, EPA may need to
weigh the appropriateness of using information with significant,  but known uncertainties to fill
"data gaps," relative to using default assumptions or committing additional resources to generate
new information.

       2.2    Examples of Questions Raised by Consideration of the Assessment Factors

       Example questions that could be raised by the consideration of each of the assessment
factors for various types of information are provided below. Given the very general nature of
these assessment factors, the Agency felt that a compilation of such illustrative questions would
most clearly convey the intended nature and breadth of the assessment factors,  and how they
would be reflected in an evaluation of various types of information. However, the applicability
of these factors depends on the individual situation, and EPA retains discretion to consider and
use factors and approaches on a case-by-case basis that may differ from the  illustrative
considerations presented below.

       2.2.1   Soundness

              The extent to which the scientific and technical procedures, measures, methods or
              models employed to generate the information are reasonable for, and consistent
              with, the intended application.

              a)    Is the purpose of the study reasonable and consistent with its design?

              b)     To what extent are the procedures, measures, methods, or models
                    employed to develop the information reasonable and  consistent with sound
                    scientific theory or accepted approaches?

              c)    How do the study's design and results compare with existing scientific or
                    economic theory and practice? Are the assumptions, governing equations
                    and mathematical descriptions employed scientifically and technically
                    justified? Is the study based on sound scientific or econometric
                    principles?

              d)    In the case of a survey, have the questionnaires and other  survey
                    instruments been validated (e.g., compared with direct measurement data)?
                    Were checks for potential errors made during the interview process?

-------
Page 6	Assessment Factors
              e)    How internally consistent are the study's conclusions with the data and
                    results presented?

       2.2.2   Applicability and Utility

              The extent to which the information is relevant for the Agency's intended use.

              a)     How useful or applicable is the scientific or economic theory applied in
                    the study to the Agency's intended use of the analysis?

              b)    How relevant are the study's purpose, design, outcome measures and
                    results to the Agency's intended use of the analysis (e.g., for a chemical
                    hazard characterization)?

              c)    Are the domains (e.g., duration, species, exposure) where the model or
                    results are valid useful to the Agency's application?

              d)    How relevant is the study to current conditions of interest? For example,
                    in the case of a survey, are conditions likely to have changed since the
                    survey was completed (i.e., is the information still relevant)? Is the
                    sampled population relevant to the Agency's current application? How
                    well does the sample take into account sensitive subpopulations?

       2.2.3   Clarity and Completeness

              The degree of clarity and completeness with which the data, assumptions,
              methods, quality assurance, sponsoring organizations and analyses employed to
              generate the information are documented.

              a)    To what extent does the documentation clearly and completely describe
                    the underlying scientific or economic theory and the analytic methods
                    used?

              b)     To what extent have key assumptions, parameter values, measures,
                    domains and limitations been described and characterized?

              c)     To what extent are the results clearly and completely documented as a
                    basis for comparing them to results from other similar tests?

-------
                            Assessment Factors	Page 7
       d)   If novel or alternative theories or approaches are used, how clearly are they
              explained and the differences with accepted theories or approaches
              highlighted?

       e)   Is the complete data set accessible, including metadata, data-dictionaries
              and embedded definitions (e.g., codes for missing values, data quality
              flags and questionnaire responses)? Are there confidentiality issues that
              may limit accessibility to the complete data set?

       f)   In the case of a modeling exercise, have the definitions and units of model
              parameters been provided?  To what extent have the procedures for
              applying the model been clearly and completely documented? How
              available and adequate is the information necessary to run the model
              computer code?

       g)     To what extent are the descriptions of the study or survey design clear,
              complete and sufficient to enable the study or survey to be reproduced?

       h)   Have the sponsoring organization(s) for the study/information product and
              the author(s) affiliation(s) been documented?

       i)     To what extent are the procedures for quality assurance and quality control
              of the data documented and accessible?

2.2.4   Uncertainty and Variability

       The extent to which the variability and uncertainty (quantitative and qualitative)
       in the information or in the procedures, measures, methods or models are
       evaluated and characterized.

       a)     To what extent have appropriate statistical techniques been employed to
              evaluate variability and uncertainty? To what extent have the sensitive
              parameters of models been identified and characterized?

       b)   To what extent do the uncertainty and variability impact the conclusions
              that can be inferred from the data and the utility of the study? What are
              the potential sources and effects of error and bias in the study design?

-------
Page 8	Assessment Factors
              c)    Did the study identify potential uncertainties such as those due to inherent
                    variability in environmental and exposure-related parameters or possible
                    measurement errors?

       2.2.5   Evaluation and Review

              The extent of independent verification, validation and peer review of the
              information or of the procedures, measures, methods or models.

              a)    To what extent has there been independent verification or validation of the
                    study method and results? What were the conclusions of these
                    independent efforts, and are they consistent?

              b)    To what extent has independent peer review been conducted of the study
                    method and results, and how were the conclusions of this review taken
                    into account?

              c)    Has the procedure, method or model been used in similar, peer reviewed
                    studies? Are the results consistent with other relevant studies?

              d)    In the case of model-based information, to what extent has independent
                    evaluation and testing of the model code been performed and documented?

       2.3     Relationship Between the General Assessment Factors and the Elements of
              Quality in EPA's Information Quality Guidelines

       The definition of quality in the EPA Information Quality Guidelines consists of three
components, consistent with the definition of quality in OMB's Guidelines: objectivity, utility
and integrity of disseminated information. "Objectivity" focuses on the extent to which
information is presented in an accurate, clear, complete and unbiased manner; and, as a matter of
substance, the extent to which the information is accurate, reliable and unbiased. "Utility" refers
to the usefulness of the information to the intended users. "Integrity" refers to security, such as
the protection of information from unauthorized access or revision, to ensure the information is
not compromised through corruption or falsification.

       The five general assessment factors presented in this document are consistent with the
quality elements of objectivity and utility, but do not extend to the distinct element of integrity
(which refers to the separate matter of security issues).  The assessment factor applicability and
utility is most directly related to the element of utility in the OMB and EPA Information Quality

-------
                                   Assessment Factors	Page 9
Guidelines.  The other four assessment factors relate to the element of objectivity, which itself
encompasses a number of issues related to both presentation and substance. In particular, the
factor clarity and completeness is most directly related to some aspects of the presentation of
information (including whether the information is "presented in an accurate, clear, complete and
unbiased manner"). The factors soundness, uncertainty and variability and evaluation and
review most directly relate to the substantive aspects of the element of objectivity (related to
whether the information itself is "accurate, reliable and unbiased"), although they also play a role
in enhancing aspects of the presentation of the information.  Thus, the general assessment factors
are fully consistent with the related information quality elements described in the OMB and EPA
Information Quality Guidelines, and do not constitute a conceptually different or unrelated basis
for evaluating information quality.

       It is important to note that the EPA Information Quality Guidelines apply to
"information" that EPA disseminates to the public.  The EPA Information Quality Guidelines
apply to information generated by third parties if EPA distributes information prepared or
submitted by an outside party in a manner that reasonably suggests that EPA endorses or agrees
with it; if EPA indicates in its distribution that the information supports or represents EPA's
viewpoint; or if EPA in its distribution proposes to use or uses  the information to formulate or
support a regulation, guidance, policy or other Agency decision or position (EPA 2002). Please
refer to the EPA Information Quality Guidelines for additional information regarding their
applicability to  information EPA disseminates.

-------
Page 10	Assessment Factors
3.     Summary

       This document describes the assessment factors and considerations generally used by the
Agency to evaluate the quality and relevance of the broad range of scientific and technical
information used by the EPA.  These factors are founded in the Agency guidelines, practices and
procedures that make up the EPA information and quality systems including existing
program-specific quality assurance policies. Consistent with the Agency's approach to the
development of the EPA Information Quality Guidelines, this document is the product of an
open, collaborative process between EPA and the public.

       The Agency believes that the summary of general assessment factors provided in this
document will serve to increase the extent to which the information-generating public builds
quality considerations into the generation and documentation of their information products. The
Agency expects that the resulting improvements in the quality of such information will enable the
Agency to more fully utilize and disseminate such information. Thus, this document is intended
to complement the EPA Information Quality Guidelines and other Agency efforts to ensure and
enhance information quality.

-------
                                  Assessment Factors	Page 11
                                     References

US Environmental Protection Agency, 1994, Peer Review and Peer Involvement at the U.S.
       Environmental Protection Agency, Science Policy Council,
       http://www.epa.gov/osp/spc/memo0607.htm

US Environmental Protection Agency, 1995, Policy for Risk Characterization, Science Policy
       Council, http://www.epa.gov/osp/spc/rcpolicy.htm

US Environmental Protection Agency, 1998, Guidelines for Ecological Risk Assessment,
       EPA/630/R095/002F, Risk Assessment Forum,
       http ://cfpub. epa. gov/ncea/cfm/recordisplay.cfm?deid= 12460

US Environmental Protection Agency, 1999, Guidelines for Carcinogen Risk Assessment,
       Review Draft, NCEA-F-0644, Office of Research and Development,
       http ://cfpub .epa. gov/ncea/raf/cancer. cfm

US Environmental Protection Agency, 2000a, "Policy and Program Requirements for the
       Mandatory Agency-wide Quality System," (EPA Order 5360.1 A2),
       http://www.epa.gov/quality/qs-docs/5360-l.pdf

US Environmental Protection Agency, 2000b, "EPA Quality Manual for Environmental
       Programs," (EPA Order 5360 Al), http://www.epa.gov/quality/qs-docs/5360.pdf

US Environmental Protection Agency, 2002, Guidelines for Ensuring and Maximizing the
       Quality,  Objectivity, Utility, and Integrity of Information Disseminated by the
       Environmental Protection Agency, EPA 260R-02-008, Office of Environmental
       Information,  http ://www. epa. gov/oei/qualityguidelines

US Office of Management and Budget, 2002, Guidelines for Ensuring and Maximizing the
       Quality,  Objectivity, Utility, and Integrity of Information Disseminated by Federal
       Agencies, (67 FR 8452), http://www.whitehouse.gov/omb/fedreg/reproducible2.pdf

                           Additional Useful Web Sites

EPA Quality System web site: http://www.epa.gov/quality

EPA Science Policy Council web site: http://www.epa.gov/osp/spc

EPA Information Quality Guidelines web site: http ://www. epa. gov/oei/quality guidelines

-------