Region 9 Superfund Data Evaluation/Validation Guidance
R9Q A/006.1
U.S. 1 ji\ iron mental Protection Agency
Region 9
Quality Assurance Office
75 Hawthorne Street
San Francisco, CA 94105
DRAFT
December 2001

-------
Region 9 Superfund Data Evaluation/Validation Guidance
Version 1
R9Q A/006.1
December, 2001
This guidance document is designed by the U.S. Environmental Protection Agency (I .IW)
Region 9 Quality Assurance Office to provide assistance to project officers. Superlund
contractors, and Superfund grantees in performing timely data evaluation and or \ alidation of
laboratory data. Data review is considered a necessary step to ensuring that data are of sufficient
quality to support decisions based on data quality objecti\es (DQOs) as defined in the appropriate
project or site specific Quality Assurance Project Plan (QAPjP) or Sampling and Analysis Plan
(SAP). The evaluation process which is described allows se\ eral tiers that require an increasingly
more stringent review of the data. The appropriate tier can he chosen hased on the data's
intended use. The guidance defines Region 9 policy and pro\ ides examples of the various
evaluation tiers as they might be applied to different types of Superlund projects.
Region 9 requires that the level of data quality re\ iew (i e . the tier chosen) must be defined
during the planning stage of the project and be documented in the QAPjP (or equivalent). It is
expected that QAPjPs will be prepared in accordance with the QA K-5 (EPA Requirements for
Quality Assurance Project Plan) guidance, and that SAPs will he prepared using one of Region 9's
two SAP guidances (Sampling and Analysis (iuidance and Template, Version 1, EPA Analytical
Services Used, or Sampling and Analysis Guidance and Template. Version 2, Private Analytical
Services Used), or if the SAP is prepared using another guidance or format, it will be consistent
with these guidances
This guidance is hased on the assumption that lull data packages are prepared in accordance with
the Contract Laboratory Program (CI.P) Organic or Inorganic Statements ofWork (SOWs), or
the Region ^ guidance. "I .ahoratory Documentation Required For Data Evaluation," R9QA/004.2
(August 2'in I) It is assumed that each data package should include twenty or less samples of the
same matrix analyzed hy a single analytical method. Each group of twenty samples is commonly
called a "Sample l)eli\ cry (iroup (SDG)," if it is provided by the CLP or Region 9 Laboratory.
This nomenclature may not be used for data generated by a private laboratory not under contract
with I -PA, but the assumption is that samples will be grouped together in lots of 20 or fewer
samples, and that laboratory quality control samples will also track these groups of 20.
The data e\ aluation process should be tailored to meet the project's or site's specific data quality
requirements The U.S. EPA Region 9 Quality Assurance Office is available to assist in
determining the appropriate evaluation tier(s) for each project. Please contact the Regional
Quality Assurance Manager at 415-972-3798.
i
Region 9 Data Evaluation Guidance (R9QA/006.1)	Data_validation_guidance./December_2001

-------
TABLE OF CONTENTS
1.0 Introduction .........	1
2.0 Evaluation Tier 1A . . . . . . . .	1
2.1	Scope .........	1
2.2	Example of Use . . . . . . .	2
3.0 Evaluation Tier IB . . . . . . . .	2
3.1	Scope .........	2
3.2	Example of Use .......	3
4.0 Evaluation Tier 2 ........	3
4.1	Scope .........	3
4.2	Examples of Use .......	4
5.0 Evaluation Tier 3	.......	5
5.1	Scope .........	5
5.2	Example of Use .......	6
APPENDIX A. Region 9 Data Validation Definition ....	7
APPENDIX B. Region 9 QA Office's (icncral (iuidelincs l-'or Superfund Data
Validation/Ucvicw .......	8/9
Region 9 Data Evaluation Guidance (R9QA/006.1)	Data_validation_guidance./December_2001

-------
1.0 Introduction
The U.S. EPA Region 9 Quality Assurance Office data evaluation approach includes evaluation
tiers that build on each other to become increasingly more inclusive and thorough. A combination
of tiers may be appropriate for most projects. This approach may mean that one or more data
packages produced using a particular analytical method is evaluated with a different degree of
thoroughness than other data packages produced using the same or methods. For the purposes of
this guidance, each tier will have the scope defined and then be followed by exam pk-s of when and
how data evaluated using this tier might be used for a project. Appendix 1} ( Region ^ ().\
Office's General Guidelines For Superfund Data Validation/Review) is meant lo pix>\ itic
additional examples of the minimum expected data evaluation requirements llial should he
considered during the data quality objectives (DQOs) as defined in the appropriate project or site
specific Quality Assurance Project Plan (QAPjP) or Sampling and Analysis Plan (SAP)
Each project has specific data quality needs; therefore, the pro\ ided examples should only be used
as suggestions or starting points. The project's specific data e\ aluation requirements should be
included in the Quality Assurance Project Plan and or the Sampling and Analysis Plan.
2.0 Evaluation Tier 1A
2.1	Scope
The goal of Evaluation Tier IA is to quickly pro\ ide a brief summary of key analytical
issues/deficiencies which might nil eel data quality, and. henee. user decisions based on the
data.
1a aluation Tier IA is employed w hen in-depth data review is not required as indicated in
1he QAI'jP I MssiMe applications include recurrent monitoring activities, emergency or
time-critical situations, data generated by field-based monitoring techniques, and priority
en\ ironmental management activities. Please refer to Section 2.2 and Appendix B for
more information
Such a re\ iew may include, hut is not limited to: review of the data package for
completeness. re\ iew of chain of custody forms (against laboratory reported information),
for signatures, sample condition upon receipt by the laboratory, and sample preservation;
review of holding times, review of Quality Control (QC) summaries; review of blank
results for possible lield or laboratory contamination; random checks of reported results
against raw data, and random checks of raw data for interference problems or system
control problems (e.g., baseline anomalies, baseline drifts, etc.).
The review deliverable from the QA Office for a 1A level review might include: a verbal
discussion of the sample results with the data user, a memo summarizing the evaluated
results, and/or a table of data showing data points (with associated qualifiers) that were
considered to be biased or outside acceptance criteria for various data quality indicators by
a large enough factor that use of the data might affect environmental decisions.
2.2	Example of Use
Region 9 Data Evaluation Guidance (R9QA/006.1)	1	Data_validation_guidance./December_2001

-------
As noted above, an example of the appropriate use of Evaluation Tier 1A would be during
a recurrent monitoring project. These projects usually have several years of data from the
same sampling locations, and are often sampled multiple times each year. The Quality
Assurance Project Plan and/or the Sampling and Analysis Plan should describe a
combination of data assessment tools (e.g., performance evaluation samples, split samples,
field QC samples, data validation, etc.) so that project decisions do not rely on validation
results alone. Once the initial sampling and analysis protocols have been established,
verified as appropriate and expected concent rati 011 le\els of analytes determined lor each
location, then Evaluation Tier 1A can be used lor a large percentage of (lie data packages.
The remainder of the data packages should still undergo a higher e\ aluation tier
Additionally, if any major data quality deficiencies are noted during the Ia aluation Tier IA
review, then the data package should undergo a more inclusi\ e re\ iew The percentage of
data which is evaluated at a higher evaluation tier many also need to be increased
3.0 Evaluation Tier IB
3.1	Scope
The goal of Evaluation Tier IB is to produce an automated summary which reflects
whether contract required QC criteria and or generic measurement quality objecti\ es ha\ e
been met. The summary can be quickly generated for the user The summary can also be
used to facilitate a full data quality re\ iew as discussed below in La aluation Tier 3 if such
a review is necessary.
Evaluation Tier IB approach is employed alone when an in-depth data review is not
required as indicated in the Quality Assurance Project Plan. Possible applications include
recurrent monitoring acti\ ities. prioritization of site work, environmental management
acti\ilies, and projects in which EPA is not the implementing agency as in EPA financial
assistance agreements including Brownfields and Tribal projects. Grantees may be
required to perform a more stringent data review tier depending on project requirements.
Please refer to Section 3 2 and Appendix B for more information.
Such a review may include, but is not limited to: review of laboratory electronic data
deliverables for completeness, review of holding times, review of QC summaries, review
of blanks for contamination, checks of reported results against raw data, and performance
checks of a majority of calculations used in the data set.
The Q.\ Office could provide a review deliverable consisting of: a computer generated
table of results and qualifiers (e.g., Computer-Aided Data Review and Evaluation
(CADRI-) or E-DATA report), a memo that discusses the evaluated results, and/or a table
of data that has some qualifiers associated with the data points identified as having a larger
error than permitted in the data review software system. Usually the data review software
is based on criteria defined in the organic or inorganic CLP SOWs.
3.2	Example of Use
Region 9 Data Evaluation Guidance (R9QA/006.1)	2	Data_validation_guidance./December_2001

-------
As noted above, an example of EPA use of Evaluation Tier IB would be for a project in
which EPA is not the lead agency as in EPA financial assistance agreements. These
projects usually are being further evaluated for project specific use by another party. The
grantees may be required to perform more stringent data review in addition to what EPA's
analytical services offer. For example, if a site were to be a candidate for listing on the
National Priorities List. The Quality Assurance Project Plan and/or the Sampling and
Analysis Plan should include a combination of data assessment tools (e.g., performance
evaluation samples, split samples, field QC samples, data validation, etc ) so ilrat project
decisions do not rely on validation results alone. The Quality Assurance Oflicc can
provide Evaluation Tier IB results to grantees or other project personnel upon request.
This tier is only viable if data are available in the appropriate electronic format
4.0 Evaluation Tier 2
4.1 Scope
The goal of Evaluation Tier 2 is to produce a Data Re\ ieu report hased on clearly
defined and documented project-specific data quality criteria The report identifies
significant and noticeable data quality issues'deliciencies and indicates whether data
quality is consistent with the intended use
Evaluation Tier 2 is based on a more focused e\ aluation of selected analytes, a limited
number of locations, or it may also focus on a selected aspect of a particular analysis (such
as only on tentatively identified coumpounds) It is con lined to data within a single data
package and is used in conjunction with a I A 1 B review of the remainder of the data. It is
employed u hen I a aluation Tier 3 is not required as indicated in the QAPjP. Logic on
why this l-\aluation Tier is appropriate should he included in the QAPjP or SAP. This
e\ aluation tier does not in\ olve an in-depth review of all raw data. Possible applications
include monitoring acti\ ities. delineation of environmental impacts caused by pollutants,
contaminants, or toxic constituents; data used in support of EPA or other regulatory
agency enforcement, possible litigation; public health and ecological assessments;
commitment of substantial I'I'A funds, etc. Please refer to Section 4.2 and Appendix B
for more information
Such a review may include, but is not limited to use of Evaluation Tier 1A/1B for some of
the data plus a more detailed evaluation of other data in the context of project DQOs
defined in the QAPjP or SAP. This evaluation may focus on specific target compounds or
classes of compounds, or, alternatively, data from areas identified as being of particular
concerns that show potential high/low bias or false positive/false negative potential where
results are close to action or regulatory levels may be the focus.
The QA Office could provide two types of review deliverables. For data not undergoing
extensive review, deliverables would be as described above for Tiers 1A or IB. For the
data that underwent the more comprehensive review, a memorandum would be provided
that discusses the evaluated results along with an attached table of data that has qualifiers
Region 9 Data Evaluation Guidance (R9QA/006.1)	J	Data_validation_guidance./December_2001

-------
associated with those data points identified as having a bias due to larger than expected
errors as established in the project's or method's QC criteria. These QC criteria should
have been defined in the QA Plan as data quality indicators.
4.2 Examples of Use
As noted above, Evaluation Tier 2 is more of an exception, than a rule. It is only used
when Evaluation Tier 1A/1B is not inclusive or thorough enough for the project
objectives, but Evaluation Tier 3 is not warranted. It should focus on specific items that
are detailed in the data evaluation section of the Quality Assurance Project Plan and or the
Field Sampling and Analysis Plan. These plans should also include a combination of data
assessment tools (e.g., performance evaluation samples, split samples, field QC samples,
data validation, etc.) so that project decisions do not rely on validation results alone If
any major data quality deficiencies are noted in Ia aliiation Tier 2 re\ iew. then the data
should undergo a more inclusive review.
An example of the appropriate use of Evaluation Tier 2 would he a site assessment
investigation. If the site is deemed to not need further l-IW in\ ol\ ement based on
preliminary data, then a focussed review may he sufficient The locus of this review could
be an Evaluation Tier 1A/1B review plus an e\aluation lor the potential of false negatives
and/or significant low bias for those chemicals or areas that would be most likely to impact
the scoring of the site. These would be the factors that would most significantly impact
the decision that no further EPA invoh ement is necessary
Similarly, a Brownllekls site investigation may be deemed to require no further action
based on the preliminary data A focussed approach including an Evaluation Tier 1A/1B
plus an evaluation for false negatives and or significant low bias may be sufficient to
support that decision
Other decisions at Brownllekls sites may require different focuses for data evaluation. For
example, if decisions will be based on specific compounds of concern and their acceptable
bias around a specific action level, than the focus of the data evaluation should reflect
those concerns. This scenario might be that only a few compounds were historically used
at the site. These compounds were included as part of a larger list of compounds that are
analyzed by a particular laboratory method. Therefore, only a few compounds may need
to be evaluated more stringently for the decision at hand. Evaluation Tier 3 (full
validation) could be applied on the few compounds of concern, while data for the other
compounds reported from that method for the same sample(s) are handled using
Ia aluation Tier 1A/1B. Since this combination of two levels of review is for the same
method for the same sample(s) (i.e., within the same data package), it is considered to be
an application of Evaluation Tier 2.
5.0 Evaluation Tier 3
Region 9 Data Evaluation Guidance (R9QA/006.1)	4	Data_validation_guidance./December_2001

-------
5.1 Scope
The goal of Evaluation Tier 3 is to produce a detailed Data Validation report based on
clearly defined and documented project-specific data quality control criteria and/or
measurement quality objectives. Such a report identifies significant and noticeable data
quality issues/deficiencies and indicates whether the quality of the data is sufficient for the
intended use. Note that this process only evaluates the data based on method defined or
QA Plan defined QC criteria. Whether the data meet the uncertainly criteria or confidence
criteria which may have been defined by the DQO process, ^ hclher there are sufficient
data points for decision making, or whether the samples collected adequately represent the
target area, must be determined through the data quality assessment process u liich is
outside the scope of this guidance.
Evaluation Tier 3 involves an in-depth review of raw data Possible applications include
monitoring activities; delineation of environmental impacts caused by pollutants,
contaminants, or toxic constituents; data used in support of I- PA enforcement or litigation;
data supporting a record of decision; data supporting ecological assessments; data related
to commitment of substantial EPA funds: data supporting inclusion of a site on the
National Priority List, etc. Please refer to Section 5 2 and Appendix li for more
information.
Such a review may include, Init is not limited to. application of Evaluation Tier 1A/1B
plus a random check (percentage determined hy the professional judgement of the data
evaluator on a project specific basis) of nil the \ arious calculations in the data set (e.g.,
verifying and recalculating concentrations of standards including checking of expiration
dates of standards from standard preparation logs, confirming calibration criteria were
met. \crilying (X' sample results ^ere as staled, etc.), checking raw data for correct
integration, confirming mass ion spectra matches (if applicable), and assessing interference
problems or system control problems (e.g., baseline anomalies, baseline drifts, etc.).
These checks would he conducted in the context of project data quality objectives. A
more in-depth e\ aluation will he performed on target compounds or analytes identified in
the appro\ ed Quality Assurance Project Plan and/or Sample and Analysis Plan. An
e\ aluation of potential high low bias or false positive/false negative results around project
thresholds of concern will be the primary focus. The attached validation definition
describes in more detail other aspects of the evaluation.
I'or the data that: underwent the more comprehensive review, a memorandum or detailed
report would be provided that discusses the evaluated results along with an attached table
of data that has qualifiers associated with those data points identified as having a bias due
to larger than expected errors as established in the project's or method's QC criteria. This
detailed report would discuss data point qualification, as well as any additional information
that may affect the use of the data. Method or project QC criteria should have been
defined in the QA Plan as data quality indicators either directly or by reference (such as to
the CLP SOW).
5.2 Example of Use
Region 9 Data Evaluation Guidance (R9QA/006.1)	5	Data_validation_guidance./December_2001

-------
Appendix A
As noted above, an example of the appropriate use of Evaluation Tier 3 would be data
used in support of EPA enforcement projects. The Quality Assurance Project Plan and/or
the Sampling and Analysis Plan should include a combination of data assessment tools
(e.g., performance evaluation samples, split samples, field QC samples, data validation,
etc.) so that project decisions do not rely on validation results alone. Evaluation Tier 3 is
the most inclusive and thorough review of data and is used to document the quality of data
to be directly used for decision making by the Agency, sepeciallv llic Supcilund Program.
APPENDIX A: Region 9 Data Validation Definition
Region 9 Data Evaluation Guidance (R9QA/006.1)	6	Data_validation_guidance./December_2001

-------
Appendix A
Data Validation is a sample- and analyte-specific process that extends beyond method or
contractual compliance. Data validation is based on regionally defined data quality criteria and
limits, professional judgment of the data validator, and (if available) the project-specific Quality
Assurance Project Plan (QAPjP), and/or Sampling and Analysis Plan (SAP). Its purpose is to
assess the usability of specific sample and analyte results for use in decision making at the
regional, site and/or project level.
Data validation may include the following items:
Determination of data usability
Qualification of data based on project-specific dala quality criteria and or professional
judgment
Interpretation and evaluation of raw dala. e u . chiomatourams and mass spectra
Assessment of data based on their intended use and compliance u illi the project or site
QAPjP and SAP
Verification of analyte identification and or quantification
Assessment and incorporation of site specific factors thai may affect data usability
• Determination of how the nature of the sample inhibits attainment of analytical
specifications
Assessment and application of dala from field duplicates, performance evaluation samples,
blind spikes, and blind blanks
E\ al nation and assessment of analytical problems that may be documented in the SDG
nairati \ e
I a aluation of the i mpact of multiple data issues on the final analytical result
Preparation of final validation narratives, reports, comments, or findings
7

-------
Appendix B
8

-------
ca
X
HH
O
Z
W
Ph
Ph
<
—
>
s
Z
O
—
H
<
O
HH
-
<
H
<
O
O
Z
&
to
W
Ph
&
tZ)
O
to
ifi
to
z
HH
-
O
HH
&
o
-
$
to
z
to
o
Cfl
CN
to
u
HH
to
to
o
a
ON
S3
O
'3d
a>
Tier 3



¦&
10%



1
5%

50%

50%
Tier 2
20%


100%
10%
100% **

• r.

—
l/~i

«r.
Tier IB
80%
vO
0s
vO
0s

nP
ox
00


vO
0s
iy^j
C
a>


100 %

Tier 1A





Intended Use/
Types of $F Decisions
Site Discovery; Hot Spot Id.; PA
w/sampling
Time-Critical Removal/Risk Reduction
Prioritization of Site Assessment
Site Assessment/HRS
Site Characterization/RI
Risk Assessment
Treatability Studies
Treatment Optimization
On-Going-Monitoring
No Further Action (Brownfields)
Site Close-out
Investigation Derived Wastes
Five Year Review/ Confirmation Studies
c
1
C/3
C
o
o
•*%

o
0
-------
Appendix B
10
-------