m tv
h '
Dat^
1 gju
Quality Assurance Office, Region!
¦ MI ZzJ
) — October, 2011 version 2
INTRODUCTION
Frequently the quality of results from data collection activities are difficult to assess due to the number of
reports one needs to review and digest to reach a conclusion (see Figure 1). These reviews may take
place months after data collection is conducted.
- GROUNDWATER SAMPLING
VALIDATION OF LABORATORY RESULTS
ENVIRONMENTAL LABORATORY
LABORATORY REPORT * 0701010-1 • REPORT DATED JANUARY I I. J007
LEVEL 4 - 1'LLL Qf DELIVERABLES EVALUATION FOR VOCl AMI
I'LRCHI.ORATE
INTRODUCTION
collected rovcii groundwater «a»nplc». aim! one quality control sample i>n
.lunuAty 4. 2007. The sumplcs we«c hund delivered to Environmental
l.Aboratory located in on Janu.w 4, 200 7 Aimlysct fur
piUcbloraic wore >uUcunlntvlcO tu Laboratory located in Sncrnmcnlo.
California I lit cample identification", mil rc.jucited mtalyirs are litlcil below
\l« \l»IN NallipU Ill
!.»«» It) Matflk
Ali»ly«*> : C'0iiiiu«nti
y.'A-iNiKo roj.'oo.-
• 'A t?** OIIM/UU?
OTOinlOt-OI A.11 ; A.I» Grounduwrn
Vf X"» r.-.rl frrrMixuIr
YVX~. »rv.I P-<.
VI « *»
• • A • 102-OIO42MI
Tltortl-Ot 04ivov
0701 AmVl.Ot II ( iTviuiMlwatrrr
070ll»104.0» II !.iou«|WiBl
G Groundwain
070IM04-0H t> W«K« Quality
"Vo<>
vo<;>
V(V. • TYtp nt.nl
Samples wore uihniiitnl for nn.dyxis r>f organic- nnd inorganic compound*. us lined
• Volatile Organic Compound* (VOC»l LP A Method 82» evaluated in the following cheek »»• iaWc> »»•)) or below (less than
(<)) the line. Where lines are paired, QC results should lie
between the limit lines. The limits represent laboratory
and field QC (precision, accuracy, and bias (PAB) criteria -
see pages 14-16). Environmentally influenced anomolies
will periodically cause deviations from these limits to occur.
Even with a flawless QA/QC program in place, scientific
uncertainty is inevitable due to the real variation in the
population being sampled. Deviations should always be
-------
monitored. A single deviation should not be cause for
concern provided overall long term performance meets
limits, and one is not making decisions based on data that
exceeded limits. Corrective action should take place where
trends and patterns show deviation will likely occur.
Trend charts are related to but differ from control charts.
Trend charts are project specific and represent non-
continuous QC checks. Points making up the trend chart
may be obtained from any laboratory performing analyses
for a project.
Control charts are laboratory and instrument specific;
they also must be continuous. Control charts con-
tinuously record a single laboratory's QC performance check
results obtained from a single instrument.
Due to the non-continuous nature of trend charts, they may
be produced from control chart points obtained from multi-
ple laboratories, but the reverse is not true, as control charts
are laboratory and instrument specific, and must be con-
tinuous.
Trend charts permit near real time evaluation to quanti-
tatively screen the quality of laboratory and field QC sam-
ple results for specific COC over time, and for monitoring
excursions from QC criteria established in the QAPP. For
the laboratory QC results checked, the charts communicate
what is covered in data validation reports in a clear, concise,
quantitative, and graphical format. Other activities covered
in QAPP e.g., field and laboratory audit findings and the
percentage of data validated over time may similarly be
charted.
• Laboratory QC results that may be tracked include:
initial and continuing calibration, laboratory control
sample, laboratory control sample duplicates, matrix spike
(MS)/matrix spike duplicates (MSD) (or deuterated
compound recoveries if MS/MSDs not performed), blanks,
internal standards, serial dilutions, system performance
check compounds, tunes and performance evaluation
(PE) sample results.
Figure 3 contains a subset of laboratory QC charts that
are available from US EPA's Contract Laboratory Program
(CLP) Trending Analytical Data system. It represents
laboratory QC results for initial and continuing calibration,
deuterated compound recoveries, and blank results over
three years from an actual site. Overall the QC results
look satisfactory with the following boxed exceptions as
CLP Lab QC Results for Site A, Carbon Tetrachloride
Initial Calibration Relative Response Factor s/b > 0.1
Deuterated Mon Ck Cmpd (DMC) % Recovery
78-129 for water and
Initial Calibration RRF RSD s/b < 20%
BICAL Relative Response Factor % RSD
Carbon Tetrachloride
Blank Contamination (s/b zero)
Continuing Calibration % Diff s/b +/- 25%
Q O
i»
s—
4 I
9*>
BP
F*T~
~
. X X X X. X X. X X "X X/X X.
Figure 3
-------
Performance evaluation results for
three years for two laboratories
U—
200
150
1
Ijr - V ^
IS A + n ~ ~~ ~~ ~ 2p
1 nPq£D ^ n&fh «* 1
1
* 1
o •
*<2-
they fail to meet greater than or paired limits for soil (red
data points and limit lines) and water (blue data points and
limit lines).
Figure 3 chart evaluations:
A) Initial Calibration Relative Response Factor (RRF).
Excursions occur for water (blue) in March, October,
December 2007.
B) Initial Calibration RRF Relative Standard Deviation
(RSD). No excursions.
C) Continuing Calibration. Excursions occur for soil (red
+/- 25% criteria) in September and November 2008,
For water, excursions occur (blue +/-30% criteria)
in September 2008.
D) Deuterated monitoring check compounds. Excursions
occur for soil in May, October and November 2008
and water in January, April, June, July, August, Sep-
tember and December 2008.
E) Blank contamination occurred in August and October,
2008.
Note also that excursions may follow a pattern, with more
appearing in certain months than others (e.g., September
- December 2008 in the above example provided). This
pattern may indicate a decline in performance that should
Figure 4
be looked into to minimize recurrence.
One should use sample results associated with QC excur-
sions with caution, consulting with a chemist or validating
associated data packages for increased confidence in
determining how data may be used, if at all.
Figure 4 provides examples of performance evaluation
sample results from two laboratories (yellow square and
blue diamond) from 2006-2009. The recovery criteria for
laboratory results represented by the yellow squares is 75-
120, and show relatively stable and acceptable performance,
with one borderline deviation from criteria in 2006. The
recovery criteria for laboratory results represented by the
blue diamonds is wider 70-130 and show some trending
(four-five down-up cycles), and three borderline deviations
(near the beginning of 2007, mid 2008, and early 2009).
Other elements covered in a QAPP may also be tracked:
• Field QC results for blank and duplicate recovery. Figure
5 provides examples of QC samples collected by two
different samplers, a novice and experienced trainer.
Although there are no QC criteria or red lines bounding
these QC results, samplers should strive to minimize error
from being introduced. Trainer results show less error
for duplicates (5% as opposed to 15% for the novice) and
blank samples (no contamination as opposed to some
contamination for the novice).
3
-------
Note that throughout the chart displays, actual sample results
may also be presented to track impact of deviations on
results.
• Field and Laboratory Audit Findings: Charts on
Figure 6 show the dates audits were conducted and the
number of findings made. Optimally, there should be
no findings. The number of field audit findings (approx-
imately 50) is cause for concern due to the impact of
sampling on sample results.
If there are numerous and significant findings, and
corrective actions are not effective (i.e., repeated au-
dits do not help to correct deficiencies), other sam-
plers should be considered.
• Data Validation. The red line in Figure 7 shows the
percentage of validation committed to in the QAPP, and
the bars represent the percentage of data validated over
the year. The graph on the right shows they did not
meet the commitment of 10%.
Tracking performance by activity will allow one to focus on
and isolate areas needing improvement, e.g., if a laboratory
QC results meet criteria, while field sampling criteria are
exceeded, corrective action should focus on the field
activity.
Alternatively, if laboratory QC results fail criteria while field
QC show acceptable results, initial review and corrective
action should focus on the laboratory. The effect of field ac-
tivities should not be excluded as contributing to error; how-
ever, initial investigation should begin with the laboratory.
Effective Use of Trend Charts
Figures 8, 9, and 10 are further examples of how to inter-
pret the trend charts by activity. Some of the charts are
presented twice; e.g., some charts from Figure 8 and 9
were included in Figure 10 to demonstrate the different
conclusions that may be reached based on what appears in
the charts.
Begin by aligning the charts by date to determine 1) whether
results are in conformance with plan requirements;
2) whether data quality objectives are being met; and
3) source of deficiencies (laboratory, field, or validation).
Whenever reviewing trend charts, look at the overall perfor-
mance over time. A single divergence may be acceptable if
the overall performance meets criteria, provided a critical
decision is not based on that data point, a satisfactory
explanation has been provided and corrective action has
been taken. Assistance may be obtained from the QA
Office to investigate any point(s) that diverge from QAPP
criteria.
One can check for trends such as in Figure 11, which depicts
a systemic error (high bias) that could be corrected. Upon
correction, nearly all data meets acceptance criteria.
Field QC Samples
Duplicate, Blank, and Sample Results from Two Samplers
Novice
Trainer
1
Field Duplicate Results
Field Duplicate 2
1
rf
H
J 10 j
1 8 1 1 I 8 1 8 1 I I 1
1 ! 1 » 1 « ' 1 1 i 1 I
Fielrt Blank
8 11118 8 18 118
FkjW Blank 2
, 08 J
^ j
f J
f *5 i
888181888188
42I»!*Y?|S£I
! 1 I ! I f I I 1 i ! 1
Laboratory Analytical Results in ug/1
Laboratory Analytical Results 2 in ug/l
M "j
1?
1? '0 1 ^ ^ ^
12 i
\ 10 J
3 |
1:
2 |
1 I 1 8 i 8 I I 8 S I §
t ! i i M M il S 1 1
1 1 1 1 1 1 1 I 1 1 1 1
i i I M ^ M i S £ 1
-------
Field and Lab Audit Findings
4=
Figure 6
Percentage of Data Validated
Consistent with QAPP?
Tier 1 = Screening
Tier 3 = Full Validation
Figure 7
-------
Figure 8a, Laboratory
QC Check results for
TRICHLOROETHENE
Field QC, Number of
Audit Findings, and
Data Validation Charts
MulU-Pomt Calibration forTCE
Oxidation Coefficient Vb >= 0,995
Points snouKJ be awwe red une
Field QC
Figure 8. Shows all charts - laboratory, field, audit,
and validation results aligned by date for deter-
mining if/when QC criteria were met, and when
specific activities were completed. Results from
these QC checks show good performance (within
red error bars) or above or below criteria specified.
Corrective action should be performed on audit
findings and where validation falls short. On
December 1, 2008, only 9% validated whereas
performance criteria calls for 10% to be validated.
Figures 8b and 8c are expanded views of Figure
8a.
Percentage Tier 3 Validation Performed
Should bt 10% tout over
I! I! 11111111
Figure 8a
-------
Figure 8b, Laboratory QC Check results for TRICHLOROETHENE
§11
So
: <
•S«4
Multi-Point Calibration for TCE
Correlation Coefficient sfb> - 0.995
Points should be above red line
Deuterated Compound
Accuracy Cntena +/• 40%
Matrix Spike
Accuracy Criteria * /- 40%
-
:»
;» •
• ®'
OSW ,
c 15
*10
• 5
a 9-
Matrix Spike Duplicate
Precision Criteria (Percent Difference) <30%
Laboratory Blanks
Should be Zero
Laboratory Control Sample Results for TCE
Accuracy Criteria ~/- 20%
Jia
Z IK
* K
- fit
" N
- »
* t
«5" t? f* t*
£! •? e {? e;
SZ Tz
Continuing Calibration
Correlation Coefficient s/b >• 0.9
c '
I or
104
" 0.1
S I S S S I 8 I I I I I
-Si?>555k55o^S
Figure 8b
Figure 8c, Field QC, Number of Audit
Findings, and Data Validation Charts
Field QC
Number of Audit Findings
Validation Perf
60
. 60
I 40
I30
i 20
J 10
0
Field Blank for TCE
Should be Zero
Number of Field Audit Findinqs
Should be Zero
?60
120
100
sa
t 40
w 80
¦o
a 30
e 60
2 40
"» 10
o
20
mnmntBmoimnman
0
Field Duplicate for TCE
Precision (Percent Difference J Criteria < 30%
§ § §
a
B S 5
o o o
o o o
fr S £
5 5 S
f« (J f! C! r!
Number af Lab Audit Findings
Should be Zero
» 25
§»
= 15
"O
:3
0
£ «
S 5
s s §
* s «
Percentage Tier i validation Performed
Should be 100% Each Time
Percentage Tier 3 Validation Performed
Should Total 10% over year
Figure 8c
7
-------
Figure 9a, Laboratory QC Check results for TRICHLOROETHENE
Figures 9a arid 9b,
Results from these QC
checks show poor per-
formance often above or
outside of red error bars
both in the field and
laboratory. There were
numerous audit findings
both in the field and lab-
oratory; validation also
not performed in confor-
mance with QAPP. Cor-
rective action is need-
ed in all areas charted.
The source of error may
be identified through
pattern identification. In
the charts, field QC re-
sults for January and
March appear to be im-
pacted by laboratory per-
formance as they seem
to follow laboratory QC
results that exceeded
criteria (e.g., deuterated
compound, matrix spike,
matrix spike duplicate,
and laboratory control
sample).
It is less clear, but ap-
pears to occur again in
October and November
in laboratory QC samples
for deuterated com-
pound, matrix spike du-
plicate, laboratory control
sample, and laboratory
blanks. At other times
the laboratory results
appear to be within lim-
its, but field results do
not track or follow labor-
atory QC results.
Although initially thought
that corrective action
should be with the lab-
oratory (January and
March), additional eval-
uation of the laboratory
and field charts support
corrective action for
both activities. The num-
ber of audit findings both
in the field and labor-
atory support this con-
clusion, as there is no
indication that corrective
actions were taken to
address findings. Per-
cent validation also did
not meet the QAPP com-
mitment.
Figure 9a
Figure 9b, Field QC, Number of Audit
Findings, and Data Validation Charts
Field QC
Field Blank for TCE
Should be Zero
Number of Audit Findings
Number of Field Audit Rndings
Should be Zero
Validation Perf
Percentaqe Tter 1 Validation Performed
5hould be 100% Each Time
c c c
Field Duplicate fur TCE
Precision (Percent Difference) Criteria < 30%
Results should be below red line
Number of tab Audtt Findings
Should he Zero
l.ll In-- I
Percentage Tter 3 Validation Performed
Should Total 10% over year
Figure 9b
3
-------
Figure 10a, Laboratory QC Check results for TRICHLOROETHENE
MulU-Poirit Calibration for TCE
Correlation Coefficient s/b >¦ 0.995
P0W5 should be above red Hoe
507
~04
|fl>-
Out?rated Compound
Accuracy Criteria +/- 40%
-2W r
Matrix Spike
Accuracy Criteria +/- 40%
Matrix Spike Duplicate
Precision Criteria (Percent Difference) <30%
5 5
laboratory Blanks
Should be 2flro
Laboratory Control Sample Results for TCE
Accuracy Criteria */- 20%
c? c; cs t? £«
Continuing Calibration
Correlation Coefficient s/b >=
1.9
- 19
ju
1u
i '
= 07
U
u 01
1 1
C2 5
1111111
ill
e e cs
- Si
?l ¦# ifl to 5 «a cS
© -• pj
Figure 10a
Figure 10b, Field QC, Number of Audit
Findings, and Data Validation Charts
Field QC
Number of Audit Findings
Validation Perf
Field Blank for TCE
Should be Zero
a i# 2 a
a a a ?
Field Duplicate for TCE
Precision (Percent Difference ) Criteria < 30%
Results should be below red line
Number of Field Audit Findings
Should be Zero
•t 30
a M
9 r r c
Number of Ub Audit Findings
Should be Zero
120
100
Percentage Tier 1 Validation Performed
Should be 100% Each Time
§ ?
e 5
ess
see*
Percentage Tier 3 Validation Performed
Should Total 10% over year
¦h,
ill
0000
0. "2 rZ 2
Figure 10b
Figures 10a and 10b. With exception of laboratory deuterated compound and matrix spike results, shows good laboratory control. Field
QC results and number of audit findings are significant. Validation performed in near conformance with plan. Corrective action is
needed in the field (overcome sampling deficiences captured in field blank, field duplicate and field audit charts) and laboratory
(overcome effects from matrix [based on laboratory deuterated compound and matrix spike, results likely impacted by matrix, which
may necessitate selection of another method or modification of existing method to overcome interferences, and that perform within
acceptable limits] and laboratory audit findings). Validation also fell short by one percent and should also be addressed.
9
-------
Figure 11, HIGH BIAS in Laboratory QC Samples
Laboratory Control Sample Results
Accuracy Criteria W- 20".
88888S88S888
S J i
Graphs on left show a
high bias. The
laboratory should
check if this is a
systemic problem that
may be easily
corrected.
Deuteratod Compound
Accuracy Criteria *-40%
- 240
> 200
| ,60-
- 120
40
0
SSSgSSSsS
H 5 5 j I H j
Sartple Results
When corrected, graphs
on right show improved
performance and
reliability in results,
though some points still
borderline (e.g., Lab
Control Sample results
for Apr, May and Dec 08)
and outside of limits (Feb
and Sep 08).
This is significant if
results near action or
regulatory limits of 13
ug/l (results now ~ 12 as
opposed to ~ 14 ug/l).
Doutcrated Compound
Accuracy Criteria +A40'/i
?
240
o
200
&
160
120
V
•
80
40
0
SSSgSSSgSSSS
3 £ 2 2 2 •?
a 6 <> o o
1 i is j a
Ssinple Re$ults
sssssssgssss
Figure 11
Figure 12, Effective Use of Resources
Validation erroneously
performed on highlighted
results where QC shown
within limits.
4
Trend charts can identify areas
to perform validation (i.e., areas
out of performance criteria) for
resource savings and to
determine how the data
produced out of performance
criteria may be used, if at all.
Laboratory Control Sample Results
Acc uracy Criteria 20%
Deuterated Confound Sample Results
Accuracy Criteria *t- 40\
!i /
ssssssssssss
Sample Results
§ s
§ n
10
Figure 12
-------
One can also check if validation resources were effectively
deployed. For example, Figure 12 shows validation was
performed on results that met performance criteria. The
validation should focus on those results that did not meet
criteria.
Once QC results are in a database, one can examine the
data for use in many different ways. For example, Figure
13 depicts QC data, associated chart, and site conditions
associated with the QC data set. The ability to see develop-
ments, e.g., plume expansion or contraction on a near real
time basis, is valuable for decision making, particularly if
trying to control the plume with remediation.
One can also chart two line graphs on one chart, provided
criteria and scales are similar (Figure 14). Or one can chart
all QC results on one page for a comprehensive view of data
quality (Figure 15).
Source of Trend Chart Points and
How They Are Produced
When the laboratory performs analysis on project samples,
they generate QC samples for determining whether the
analytical system is performing within method or project
PAB specifications (Figure 16).
With the exception of blanks, which should always be non-
detect, a known concentration of a standard is typically
spiked into the QC sample composed of laboratory reagent
water or into actual sample (MS/MSD, deuterated monitoring
compound, or surrogate). The QC sample is analyzed and
results compared against what was spiked into the QC
sample, and should be close to what was spiked into it.
The laboratory performs PAB calculations (see Figure 17)
on the QC sample results to determine their performance
and whether they met QC limits of the method or QAPP
specific PAB criteria.
Sensitivity, PAB are central to determining data quality,
understanding what the trend charts are communicating
and how to effectively use them.
Sensitivity is the capability of a method or instrument to
discriminate between measurement responses represent-
ing different levels or amounts of the variable of interest.
Accuracy measures how close QC results are the "true"
value. One spikes in a known concentration of Compound
X into the QC sample. The QC sample is is then analyzed to
determine the concentration of Compound X. The QC
result (amount recovered) is then compared against the
"true" value to determine how close the laboratory recov-
ery is to "true". The assessment of accuracy includes both
accuracy and precision and is usually expressed as bias or
percent bias. See Figure 18 for example accuracy/bias
calculation.
Bias describes the degree of accuracy and assigns a
"direction" relative to the "true" value or expected result.
It manifests itself in the systematic or persistent over or
underreporting of a QC test results and may be positive
(high) or negative (low). When interpreting trend charts,
the closer the accuracy result is to a 100% recovery, the
better. However, one will likely encounter accuracy results
over 100% or under 100% recovery, indicating a high or
low bias, respectively, from the true value (see figure 19).
Value of Trend Charts
• Ease of tracking QAPP implementation over time for
laboratory and field QC, audits conducted, and vali-
dation using visual charts rather than numerous text
reports.
• Improve oversight and control of data quality due to
ease in interpretation by those responsible for over-
sight and implementation of QAPP.
• Permit efficient self monitoring and tracking of QC
results by parties responsible for implementing QAPP
e.g., laboratory and field staff for determining
excursions from QAPP criteria (e.g., +/-15% accu-
racy criteria).
• Assimilate meaning of QC results and impact on
sample results quickly.
• Spot out of control events/trends for performing
corrective action.
• Convey information to others succinctly and trans-
parently to enable their immediate understanding
of the important characteristics of the data.
• Select data for validation based on charts, resulting
in resource savings.
• Evaluate data collected from other sources, potential-
ly with different objectives and criteria, if QC data
available for charting. Data collected from other
sources is often referred to as secondary data.
• Promote transparency and open government.
Limitations
• Trend charts are an effective broad brush tool. Fine
tuned oversight still is necessary to determine cause
ofexceedances.
• Inappropriate use of trend chart results. Project
chemist should be consulted on the use of QC data
that exceed limits before making decisions with
sample results associated with that exceedance.
• Limited to a set of COC, not entire target compound
list in a method (e.g., EPA Method 8260). Use of
charts may be cumbersome if COC exceed 10 at
this time.
• Conventional validation should be performed for a
one time sampling event where the target compound
is unknown, and decisions will be based on that one
event.
11
-------
Data, Chart, and Site Conditions
Analysis Date Laboratory Control Sample Results
0U)1«
102
oaoira
98
C&0UB
100
O4JO14D0
37
c&oi«e
100
36.01.OS
101
07«i«e
98
33.0S.08
99
3ao!«e
100
iaoi.ce
101
103
mm
100
Laboratory Control Sample Results forTCE
Accuiacy Criteria ~/- 20 S
> no
| no
g 100
| 80
1 60
3 40
o ^ oi
Figure 14
12
-------
—
; -a
5 <
-r »~-
O o
"O o
a> z
9
4
{
* E
V
HOttf*
1
}
J
el r
1!
/
¦n&uv
i
{
uxKi*
1
\
¦bum
SJKIMI
8 3SSS!'
tJiS$
ifMSb
2
o>
LU
I
. !
Keen®
weuii
s.
W»IW
a
3
nxui
1
~
5
E
8UEk«
Si
ii
KCBJ.4
KWV
|i
3 1
wceru;
SCCE-hH
r
WOSK
sl
BttKTLS
|
P«I'»
want
I
|
SUJCVT
mc»-e
X
KHSI«
MO*
:
HGMfl
a
M
8i«ap«
Bl»»"
"•wpnrf#
lis
* -
• •* -
. * i
:
* *
* «' *
>• :
* • M
..
: f( :
* * *
%*
V
1 *
I • 1
» cr
!
\.\0- ft *
!
¦ c
MR 1
1
1
o o
£¦ &
ro o
£ g
E 5
3 <5
w -1
Si
31
ll
? |
!
cj |
I
si
<
1
>
/
.
1
i
¦HafflN
W4&Lft>|i
II
S 8 H 8 ®
ats^sKsaEsa
MKMHj
WK.UI
WMM®
WWJirU
WKfi'9
scam*
~ 5
j:
&
2
If
If
$ $f|f «*£»
A±ny|
Wtfti
W9JCM
13
-------
What Can You Chart?
Batch of Samples
Samples sent to the lab
QC Samples Lab Generates
er batch received
¦ I/O Calibration - accuracy cai
¦ I/O Continuing Calibration - precision
• I/O Lab Control Sample (LCS) - accuracy
I/O Matrix Spike (MS) - accuracy and matrix effects/interference
¦ I/O Matrix Spike Duplicate (MSD) - precision
¦ O Surrogate Spike - accuracy, method bias and extraction efficiency
¦ surrogates are spiked into every sample, not per batch
¦ I/O Blank - contamination
I Duplicates - Precision
9
Legend I = inorganic; O = organic; with exception of blanks, all QC samples are spiked with
a known concentration of compound or element.
Figure 16
Determining Data Quality
Types of Measurements on QC Samples
¦ ACCURACY/BIAS
Percent Recovery = Amount Recovered (Results') x 100
Amount Spiked (True Value)
Measures how close you are to the "True Value;" the closer the number, the better.
¦ PRECISION
Relative Percent Difference (RPD) = | Dud 1* - Dud 2* I x 100
[(Dup 1* + Dup 2*)/2]
*Dup = results from lab duplicates
Smaller RPDs the better, results reproducible
Larger RPDs, the more unpredictable is the resulting data
Figure 17
-------
Example Accuracy/Bias
Calculations
x 100 = 67%
30 ug/l
x 100 =
30 ug/l
ACCURACY/BIAS ASSOCIATED WITH SPIKED SAMPLES
Measures how close you are to the "True Value." The closer the
results to the true value, the better (i.e., recovery of 100%).
Figure 18
| (a) high bias
+ low precision
= low accuracy
| (b) low bias
+ low precision
= low accuracy
| (c) high bias
+ high precision
= low accuracy
| (d) low bias
+ high precision
= high accuracy
15
-------
Example Precision Calculations
120-501 x 100 = 86 RPD
[(20 + 50)/2]
Equation 3
149-501 x 100 =
[(49 + 50)/2]
Equation 4
PRECISION ASSOCIATED WITH DUPLICATES
Smaller RPDs the better, results reproducible.
Larger RPDs, the more unpredictable is the resulting data
Precision is defined as a measure of agreement among
repeated measurements of the same property under identi-
cal, or substantially similar, conditions. The equation for
calculating precision is presented in Figures 17 and 20.
Results from PAB calculations (Figures 17,18, and 20) are
plotted on the charts. These criteria, for the most part, are
dependent on analytical method criteria; they may also be
based on project specific criteria. In no case, should PAB
criteria be less stringent than those identified in the method.
It may be challenging to meet PAB criteria with some
analytes, e.g., emerging compounds where methods are
in development. QA Office representatives should be con-
tacted to request assistance with QC data interpretation in
these cases.
CONCLUSION
Trend charts produce a quick visual method for use in
assessing QC results and QA oversight of QAPP implemen-
tation over time. Core QC elements may be tracked in the
field and laboratory. Laboratories produce the data used in
preparing these charts, with most producing charts upon
request.
Trend charts use may be extended to quantify results of
performance evaluation samples, field laboratory audits, and
data validation. They permit one to see trends in a timely
manner, for corrective action when needed. Due to their
ease in interpretation, they permit improved oversight and
1 Data Quality Assessment: A Reviewer's Guide, EPA QA-G9R, EPA/240/B-06/
002, February 2006.
Figure 20
control of data quality, for corrective action when needed.
The chemist should be consulted when making a critical
decision based on the data provided in the trend charts.
The discussion in this paper is based on recent work
completed for Region 9 sites and is subject to revision
as more QC information becomes available. Mention
of trade names or commercial products does not
constitute endorsement or recommendations for use.
Acknowledgements
John Warren, Quality Staff, Washington, D.C.
Gary Johnson, Quality Staff, Research Triangle Park, NC
Charles Ritchey, Division Quality Assurance Officer,
Multimedia Planning & Permitting Division, Region 6
Michael S. Johnson, Analytical System Branch, Technology
Innovation & Field Services Division, Office of Superfund
Remediation & Technology Innovation, Office of Solid Waste
& Emergency Response
R. Paul Swift, PhD, PE, Techlaw, Inc.
Nazy Abousaedi, Science & Engineering Mission Support, CSC
Ken Moura, Science & Engineering Mission Support, CSC
Eugenia McNaughton, Quality Assurance Office (QAO),
Region 9 (R9)
Roseanne Sakamoto, QAO, R9
Steve Remaley, QAO, R9
Jan Byers, Softec Solutions Inc., Graphics
16
-------
FREQUENTLY ASKED QUESTIONS (FAOsl
1. What software was used to prepare the charts
and may I have access to it?
Any graphical package may be used. Microsoft Excel was
used to produce the majority of figures in this paper, prim-
arily due to its ability to accept data and present in graphi-
cal format.
2. What is the cost of producing charts?
Minimal, if anything, as QC data is already being produced
by the laboratory.
Trend charts monitoring laboratory performance are already
available to Regional QA Offices for the Superfund Con-
tract Laboratory Program (CLP) at website: http://
epasmoweb.dvncsc.com/scstr/. They cover initial and con-
tinuing calibration, deuterated compound recovery (similar
to surrogate recovery), and blank results.
Region 9 laboratory will produce QC results for the follow-
ing: laboratory control sample, matrix spike, matrix spike
duplicate, blank, and surrogates. Region 9's experience with
PRP laboratories has also been successful.
3. Who will be responsible for producing the charts?
Laboratories, contractors, grant, cooperative, and interagency
agreement recipients performing data collection.
4. Who will be responsible for managing the charts?
RPM's and the QA Office will have equal responsibility in
managing data uploading due to the security firewall. The
data system will house trend charts for laboratory QC, field
QC, number field and laboratory audit, validation, and per-
formance evaluation sample results by site, with ability to
append charts from prior years for a full documentation of
QC results obtained over the life of the project.
5. How frequently should charting results be
reported?
For those immediately involved in producing data and
reporting to oversight parties, it should be reported daily to
enable tracking of the source of error and corrective action.
For those in an oversight role, results may be charted quar-
terly or more frequently, if needed.
6. Can trend charts be used to demonstrate
laboratory proficiency at the beginning of a project?
Yes, however, one should ask for control charts, as opposed
to trend charts, for the COC at concentrations of concern at
the beginning of a project or whenever using a new lab-
oratory. Once charts have been obtained, determine whether
they meet project completeness criteria. For example, if
the charts show excursions outside of project limits 25% of
the time over the past 100 days and the project requires a
completeness of 95% over the same period, the laboratory
may not be suitable as the charts show they only meet 75%
completeness (75/100 x 100% = 75%, see completeness
definition in glossary). Results may not be usable 25% of
the time and will not meet completeness criteria.
One should require the laboratory's commitment to meet
project specific criteria on the method specified instrument
used by the laboratory to produce project results. "Best
Practices for Detection and Deterrence of Laboratory Fraud,"
1997, California Military Environmental Coordination
Committee contains contains additional including reviewing
laboratory Quality Assurance Project Plans, Standard
Operating Procedures, recent audit reports performed by
credible organizations, self audits, review of control charts
for past 100 days for analytical method and concentrations
of interest, performance evaluation sample results, etc.
7. How often should field and laboratory audits be
conducted?
Ideally, audits should be conducted when beginning to work
with a new field contractor or laboratory, and as frequently
as necessary thereafter. This is to ensure the "systems" are
within control criteria to improve confidence in field and lab-
oratory ability to produce data of the quality specified in the
QAPP. If one doesn't have ability to conduct audits, be sure
to use a contractor with credible performance history; i.e.,
obtain control charts for prior QC results for COCs and con-
centrations of concern, PE sample results, audit reports from
other reputable organizations, etc.
8. My laboratory is stating that it cannot produce
the charts suggested by Figures 8b, 9a, and 10a. Is
there a core set of QC elements that should be ob-
tained?
All laboratories should already be producing the following:
Initial and continuing calibration, laboratory control sample,
laboratory control sample duplicates, matrix spike (MS)/
matrix spike duplicates (MSD) (or deuterated compound re-
coveries if MS/MSDs are not performed), blanks, and per-
formance evaluation (PE) sample results. One should re-
quire an explanation be provided where they are not. If
other QC checks are sought (samples properly preserved,
holding time met), one may chart these responses as well.
9. Can trend charts be used in lieu of validation re-
ports for contaminants of concern?
No. As stated in the title of this document, trend charts are
a screening tool. Validation should coninue to be conducted
where QC results deviate from criteria. The charts pre-
sented in this paper cover some of the core laboratory QC
checks that are performed, and are limited to those checks
specifically. There are many other QC checks reviewed in
the validation process beyond those identified in the previ-
ous question or the charts presented in this paper; e.g.,
holding time, proper preservation, chain of custody, system
performance check compounds, serial dilutions, tuning,
internal standards, and others which may also be charted.
Greater certainty is achieved with charting of additional QC
checks for determining if validation should be performed.
This question merits revisiting if the additional QC checks
performed during validation are also captured in trend
charts, as QC results do not change whether captured in
validation reports or trend charts, and the objectives of both
are to bring deficiencies to light for the data user.
17
-------
10. What other field sampling activity should I track
besides blanks and duplicates?
Anything that is quantifiable and specified in the QAPP
including:
• calibration standards (expiration, stability);
• daily instrument calibration results (to monitor changes
and need for potential instrument maintenance);
• well depth;
• well stabilization results for each well (monitor changes
[e.g., changes in pH affects chemical form {mobility} and
microbial activity. Dissolved oxygen affects aerobic and
anaerobic metabolism of chlorinated compounds such as
tricholoroethene and affects activity kinetics] and need
for well maintenance due to silting and corrosion).
• pH,
• conductivity,
• temperature,
• redox (oxidation-reduction),
• dissolved oxygen,
• turbidity.
11. Why do the charts look different, some with data
points being connected by lines and others not? Is
one way more appropriate than the other?
The charts may be represented in either format and should
always be referred to as trend charts. The lines connecting
data points were used mainly to emphasize and more easily
track the temporal progression of QC results.
12. How do you use the charts for screening data
obtained from other sources, potentially with
different data and method quality objectives?
Comparability is of vital importance for projects using exist-
ing data. It is a qualitative term that expresses the confidence
that two data sets can contribute to common interpretation
and analysis.
When using data from a variety of sources or sampling
events, it is important to be sure that the data are similar.
This response limits itself to the analytical sensitivity and
QC elements (including holding time, proper preservation)
for determining data comparability using trend charts.
Comparability determination of field sample collection, de-
sign (collected at a certain depth, time of year) and collec-
tion QC should be determined separately as they will have
a direct impact.
Trend charts may be used to efficiently and quantitatively
determine analytical data comparability whether the
analytical methodology is the same or differs (e.g., per-
formance based methods used).
Data obtained from other sources must meet current sen-
sitivity and QC acceptance limits. Sample and associat-
ed QC results must come from the same "batch" (i.e., set of
QC results [initial and continuing calibration, laboratory
control sample, DMC, blank results] reflect those analyzed
concurrently with the sample) when screening. Do not mix
or match other source QC data sets, selecting only those
QC results that meet current criteria or limits, e.g., select-
ing control sample QC results from Laboratory 1, DMC QC
results from Laboratory 2, blank recovery from Laboratory
3, etc.
Once batch QC results are obtained, analytical trend charts
may then be created using QC results only, no matter wheth-
er they were obtained directly or from other sources. Simi-
lar to data obtained directly, one needs to chart the date,
QC acceptance limits (single or paired), and QC results to
screen for acceptability.
Example 1, Current QC Limits (Criteria) Wider or
Greater than Data Acquired from Other Sources
Figure AA.
Laboratory Control Sample Results for
Trichloroethylene (TCE)
Other Source Accuracy Criteria +/- 25% (Red Line)
£>140 ¦
S 120 ¦
a;
o: so ¦
.......
§ 60 ¦
£ 40 ¦
-------
Figure CC.
Laboratory Control Sample Results for
Trichloroethylene (TCE)
Other Source Accuracy Criteria +/- 25% (Red Line)
£>140
S 120
8 100
•3>
tK 80
1 60
S 40 i
-------
Deuterated Monitoring Compounds (DMCs):
Compounds added to every calibration standard, blank, and
sample used to evaluate the efficiency of the extraction/
purge-and-trap procedures, and the performance of the Gas
Chromatograph/Mass Spectrometer (GC/MS) systems.
DMCs are isotopically labeled (deuterated) analogs of native
target compounds. DMCs are not expected to be naturally
detected in the environmental media.
Field QC: Any QC samples submitted from the field to the
laboratory. Examples include, but are not limited to: Field
blanks, field duplicates, and field spikes.
Initial Calibration: Analysis of analytical standards for a
series of different specified concentrations; used to define
the quantitative response, linearity, and dynamic range of
the response of the mass spectrometer (MS) or electron
capture detector (ECD) to the target compounds.
Internal Standards: Compounds added to every standard,
blank, matrix spike and matrix spike duplicate (MS/MSD),
sample (for volatiles), and sample extract (for semivolatiles)
at a known concentration, prior to analysis. Instrument
responses to internal standards are used as the basis for
quantitation of the target compounds.
Laboratory Control Sample (LCS): An internal laboratory
QC sample used to monitor the capability of the laboratory
to perform the analytical method.
Matrix Spike (MS): Aliquot of a sample (water or soil)
taken from one of the field samples to be analyzed within
an SDG, fortified (spiked) with known quantities of specific
compounds, and subjected to the entire analytical proce-
dure in order to indicate the appropriateness of the method
for the matrix by measuring recovery.
Matrix Spike Duplicate (MSD): A second aliquot of the
same sample as the Matrix Spike (above) that is spiked in
order to determine the precision of the method.
Opening Continuing Calibration Verification: First
analytical standard run every 12 hours to verify the initial
calibration of the system.
Performance Evaluation (PE) Sample: A sample of
known composition and concentration used to evaluate
Laboratory performance.
Reagent Water: Water in which the compounds of con-
cern or interferants are not observed at the method detection
limit.
Representativeness: The measure of the degree to
which data accurately and precisely represent a character-
istic of a population, parameter variations at a sampling
point, a process condition or environmental condition. Cen-
tral to representativeness is assurance that both the sam-
pling and measurement processes are free from known
biases.
Surrogates: For pesticides and aroclors, compounds
added to every blank, sample, matrix spike and matrix spike
duplicates (MS/MSDs), and standard. Surrogates are used
to evaluate analytical efficiency by measuring recovery.
Surrogates are not expected to be detected in environmental
media.
20
------- |