EPA747-R-01-004
                                                   March, 2001
NATIONAL LEAD LABORATORY ACCREDITATION PROGRAM:
    DOUBLE-BLIND PROFICIENCY-TESTING PILOT STUDY

                    FINAL REPORT
        Program Assessment and Outreach Branch
           National Program Chemicals Division
         Office of Pollution Prevention and Toxics
          U.S. Environmental Protection Agency
                  401 M Street, S.W.
                 Washington, DC 20460

-------
                           DISCLAIMER

The material in this document has been subject to Agency technical and
policy review and approved for publication as an EPA report.  Mention of
trade names, products, or services does not convey,  and should  not  be
interpreted   as  conveying,  official  EPA   approval,   endorsement,  or
recommendation.
                      FURTHER INFORMATION

Additional copies of this report can be obtained by calling the National Lead
Information Center at 1-800-424-LEAD.  Information about other technical
reports on lead can be found on the Internet at the following address:
                      http://www.epa.gov/lead

                This report is copied on recycled paper.

-------
                         CONTRIBUTING ORGANIZATIONS

      The study documented in this report was funded and managed by the U.S.
Environmental Protection Agency  (EPA).  The study was conducted collaboratively by
the following organizations:

                                     Battelle

      Under contract to the U.S.  Environmental Protection Agency,  Battelle was
responsible for managing contractor efforts, establishing the study design, preparing
and distributing the Quality Assurance  Project Plan, recruiting study participants,
establishing  necessary standard operating procedures, preparing correspondence and
instructions  to study participants, reimbursing study participants for sample analysis
costs, summarizing and  analyzing the study data, and producing the final report.

                   American Industrial Hygiene Association (AIHA)

      Under contract to Battelle,  AIHA was responsible for providing consultation on
accreditation programs,  overseeing efforts to provide proficiency-test samples to
study participants, acting as the "proficiency testing service" in this pilot (i.e., liaison
with study participants), collecting the  analytical results of the laboratory analyses,
and providing necessary data to Battelle for data summary and analysis.

                          Research Triangle Institute (RTI)

      Under contract to AIHA,  RTI was responsible for preparing and storing
proficiency-test samples for use in the  pilot, receiving and storing sample containers
from study participants, and distributing proficiency-test samples  (with appropriate
data tracking and reporting materials) to the study participants.

                    U.S. Environmental Protection Agency (EPA)

      U.S. EPA was responsible for oversight in developing the study plan, managing
and coordinating the study, performing recruitment of study participants, and
reviewing and editing this report.  The EPA  Work Assignment Manager was John
Scalera.  Brion Cook was the  Branch Chief of the Technical Branch under whose
direction the study was conducted.  The Project Officer was Sineta Wooten.

-------
This page left blank intentionally.
              IV

-------
            NATIONAL LEAD LABORATORY ACCREDITATION PROGRAM:
               DOUBLE-BLIND PROFICIENCY-TESTING PILOT STUDY
                             TABLE OF CONTENTS
                                                                          Page

EXECUTIVE SUMMARY	xiii

1.0   INTRODUCTION	 1
      •     Background  	 1
      •     Purpose for Work	 1
      •     Objectives  	 2
      •     Report Organization  	 2

2.0   STUDY DESIGN AND OPERATION  	 5
      2.1    Study Participants and Testing Rounds	 5
      2.2   Types of Proficiency-Test Samples	 7
      2.3   Numbers of Proficiency-Test Samples Provided
            to Each Client  	 9
      2.4   Sample Transfer to Clients and Laboratories  	 9
            2.4.1  Transferring Samples to Client-Supplied
                  Sample Containers 	 9
            2.4.2  Shipping Samples to Clients  	  10
            2.4.3  Receiving and Shipping of Samples by Clients	  11
      2.5   Reporting Results	  12

3.0   STUDY PARTICIPANTS	  13
      3.1    Recruitment Approaches	  13
      3.2   Client Eligibility  	  16
      3.3   Client Selection and  Enrollment	  18
      3.4   Recruitment Experiences	  22
      3.5   Interaction With Clients During the Study	  24

4.0   DATA ANALYSIS PROCEDURES 	  27
      4.1    Data Analysis Endpoints and Data Handling  	  27
      4.2   Identifying and Handling Statistical Outliers  	  28
      4.3   Summary of Reported Lead Amounts 	  29
      4.4   Characterizing Components of Variation	  30
            4.4.1  The Full Statistical Model  	  31
            4.4.2  Reduced Forms of the Full Statistical Model  	  32
            4.4.3  Comparisons of Lab-to-Lab Variability Between
                  the Pilot Study and ELPAT Round 22  	  33
            4.4.4  Presenting Results of Statistical  Analysis 	  34

-------
                            Table of Contents (cont.)
                                                                           Page

5.0   RESULTS	  35
      5.1    Study Data  	  35
            5.1.1  Data Corrections	  35
            5.1.2  Plots of the Data Values  	  36
            5.1.3  Identifying Statistical Outliers and Data Exceeding
                  Various Types of Limits	  40
      5.2   Summary of Reported Lead Amounts  	  49
      5.3   Characterizing Components of Variation	  57
      5.4   Comparing Lab-to-Lab Variability Between the ELPAT
            Program and the Double-Blind Study	  73

6.0   QUALITY ASSURANCE	  77
      6.1    Sample Fabrication and Transfer	  77
            6.1.1  Obtaining and  Preparing  Bulk Source Material  	  77
            6.1.2  Preparing Proficiency-Test Samples	  77
            6.1.3  Verifying That Client-Supplied Sample Containers
                  Are Uncontaminated  	  78
            6.1.4  Transferring Samples to Client-Supplied
                  Sample Containers  	  79
      6.2   Data Management and Sample Tracking  	  79
            6.2.1  Types of Data	  79
            6.2.2  Data Storage and Transfer	  80
            6.2.3  Sample Identification	  80
      6.3   Data Quality Checking  	  81

7.0   CONCLUSIONS AND RECOMMENDATIONS  	  83

8.0   REFERENCES	  89
                                 List of Tables
Table                                                                      Page

ES-1  Average (and Standard Deviation) of Measured Lead Amounts
      in the Double-Blind Pilot  Study  (by Testing Round) and in
      Round 22 of the ELPAT Program, by Sample Type	  xv

ES-2. Percentage of Total Variability in Double-Blind Pilot Study Results
      That Can Be Attributed to Lab-to-Lab Variation   	xvii
                                       VI

-------
                            Table of Contents (cont.)

                              List of Tables (cont.)
Table
2-1    Information on the Three Types of Dust and Paint Proficiency-Test
      Samples Prepared in Round 22 of the ELPAT Program and Used in
      the Double-Blind Pilot Study	8

3-1    Clients Recruited to Participate in This Pilot Study, the Type(s)
      of Proficiency-Test Samples That They Would Submit in Each
      Testing  Round, and the NLLAP-Recognized Laboratory(ies)
      To Which They Would Submit the Samples	  20

5-1    Double-Blind Pilot Proficiency-Test Sample Results Identified as
      Outliers by Applying the Method of Rosner (1983) on All Results by
      Sample Type  	  44

5-2a  List of Reported Lead Amounts in Low-Spiked Dust Proficiency-Test
      Samples That Exceed At Least One of the Various Acceptance
      Ranges  Determined from Methods Used in Round 22 of the ELPAT	  46

5-2b  List of Reported Lead Amounts in Mid-Spiked Dust Proficiency-Test
      Samples That Exceed At Least One of the Various Acceptance
      Ranges  Determined from Methods Used in Round 22 of the ELPAT	  47

5-2c  List of Reported Lead Amounts in Spiked Paint Proficiency-Test
      Samples That Exceed At Least One of the Various Acceptance
      Ranges  Determined from Methods Used in Round 22 of the ELPAT	  48

5-3a  Descriptive Statistics of Reported Lead Amounts (//g) in  Low-Spiked
      Dust Proficiency-Test Samples, Calculated Across Participating
      Clients and Their Laboratories, by Testing Round  	  50

5-3b  Descriptive Statistics of Reported Percent Recoveries in  Low-Spiked
      Dust Proficiency-Test Samples, Calculated Across Participating
      Clients and Their Laboratories, by Testing Round  	  50

5-4a  Descriptive Statistics of Reported Lead Amounts (jug) in  Mid-Spiked
      Dust Proficiency-Test Samples, Calculated Across Participating
      Clients and Their Laboratories, by Testing Round  	  51

5-4b  Descriptive Statistics of Reported Percent Recoveries in  Mid-Spiked
      Dust Proficiency-Test Samples, Calculated Across Participating
      Clients and Their Laboratories, by Testing Round  	  51

                                       vii

-------
                            Table of Contents (cont.)

                               List of Tables (cont.)
Table                                                                       Page

5-5a  Descriptive Statistics of Reported Lead Amounts (% by weight) in
      Spiked Paint Proficiency-Test Samples, Calculated  Across
      Participating Clients and Their Laboratories, by Testing Round	  52

5-5b  Descriptive Statistics of Reported Percent Recoveries in Spiked Paint
      Proficiency-Test Samples, Calculated Across Participating
      Clients and Their Laboratories, by Testing Round  	  52

5-6a  Summaries of Reported Lead Amounts (//g) in Low-Spiked Dust
      Proficiency-Test Samples, by Laboratory and Testing Round	  54

5-6b  Summaries of Reported Percent Recoveries in Low-Spiked Dust
      Proficiency-Test Samples, by Laboratory and Testing Round	  54

5-7a  Summaries of Reported Lead Amounts (//g) in Mid-Spiked Dust
      Proficiency-Test Samples, by Laboratory and Testing Round	  55

5-7b  Summaries of Reported Percent Recoveries in Mid-Spiked Dust
      Proficiency-Test Samples, by Laboratory and Testing Round	  55

5-8a  Summaries of Reported Lead Amounts (% by weight) in Spiked Paint
      Proficiency-Test Samples, by Laboratory and Testing Round	  56

5-8b  Summaries of Reported Percent Recoveries in Spiked Paint
      Proficiency-Test Samples, by Laboratory and Testing Round	  56

5-9a  Descriptive Statistics of Average Reported Lead Amounts (//g) in
      Low-Spiked Dust Proficiency-Test Samples for Each
      Client/Laboratory Combination,  by Double-Blind Testing Round 	  58

5-9b  Descriptive Statistics of Average Reported  Percent Recovery in
      Low-Spiked Dust Proficiency-Test Samples for Each
      Client/Laboratory Combination,  by Double-Blind Testing Round 	  58

5-1 Oa Descriptive Statistics of Average Reported Lead Amounts (jug) in
      Mid-Spiked Dust Proficiency-Test Samples for Each
      Client/Laboratory Combination,  by Double-Blind Testing Round 	  59
                                       VIM

-------
                             Table of Contents (cont.)

                               List of Tables (cont.)
Table
5-10b Descriptive Statistics of Average Reported Percent Recovery in
      Mid-Spiked Dust Proficiency-Test Samples for Each
      Client/Laboratory Combination, by Double-Blind Testing Round  	  59

5-11a Descriptive Statistics of Average Reported Lead Amounts
      (%  by weight) in Spiked Paint Proficiency-Test Samples for
      Each  Client/Laboratory Combination, by Double-Blind Testing  Round  	  60

5-11b Descriptive Statistics of Average Reported Percent Recovery in
      Spiked Paint Proficiency-Test Samples for Each
      Client/Laboratory Combination, by Double-Blind Testing Round  	  60

5-12  Estimates of Lab-to-Lab Variability and Within-Lab Variability,
      Expressed Absolutely and Relative to Total Variability, Associated
      with Lead Measurements Reported in the Double-Blind Pilot Study,
      by Sample Type and Testing Round	  71

5-13  Summaries, Calculated Across  Laboratories, of Laboratory Average
      Lead  Measurements for the Three Types of Proficiency-Test Samples,
      Within Each Round of the Double-Blind Pilot Study and in Round 22
      of the ELPAT Program   	  74
                                  List of Figures
Figure                                                                       Page

2-1    Organization of the Double-Blind Proficiency  Testing Pilot
      Study, and Relationships Between Participating Organizations  	 6

3-1    News Brief on the Double-Blind Proficiency Testing Pilot Study  	  15

3-2   Histograms of the Proficiency-Test Sample Analysis Results
      from Round 22 of the ELPAT Program for 118 NLLAP-Recognized
      ("Reference") Laboratories, with the Results  for the 12 Laboratories
      Participating in the Double-Blind Pilot Study  Highlighted 	  21

5-1    Measured Lead Amounts (//g) in the Low-Spiked Dust
      Proficiency-Test Samples, by Double-Blind Testing Round  	  37
                                       IX

-------
                            Table of Contents (cont.)

                              List of Figures (cont.)
5-2   Measured Lead Amounts (//g) in the Mid-Spiked Dust
      Proficiency-Test Samples, by Double-Blind Testing Round  	 38

5-3   Measured Lead Amounts (% by Weight) in the Spiked
      Paint Proficiency-Test Samples, by Double-Blind Testing Round  	 39

5-4   Measured Percent Recoveries in the Low-Spiked Dust
      Proficiency-Test Samples, by Double-Blind Testing Round  	 41

5-5   Measured Percent Recoveries in the Mid-Spiked Dust
      Proficiency-Test Samples, by Double-Blind Testing Round  	 42

5-6   Measured Percent Recoveries in the Spiked Paint Proficiency-Test
      Samples, by Double-Blind Testing Round 	 43

5-7   Histogram of Low-Spiked Dust Sample Results (//g) Reported in
      the Double-Blind Pilot Study (unshaded bars) and  in Round 22 of
      the ELPAT Program  (shaded bars) by Laboratories Participating in the Pilot   . 62

5-8   Histogram of Mid-Spiked Dust Sample Results (//g) Reported in
      the Double-Blind Pilot Study (unshaded bars) and  in Round 22 of
      the ELPAT Program  (shaded bars) by Laboratories Participating in the Pilot   . 63

5-9   Histogram of Spiked Paint Sample Results (% by weight) Reported
      in the Double-Blind Pilot Study (unshaded bars) and in Round 22 of
      the ELPAT Program  (shaded bars) by Laboratories Participating in the Pilot   . 64

5-10  Estimated Percentage of Total Variability in Reported Lead Amounts
      Within Low-Spiked Dust Samples That is Attributable to Lab-to-Lab  and
      Within-Lab Sources, Based on Analyses Performed With and Without
      Statistical Outliers Included	 65

5-11  Estimated Percentage of Total Variability in Reported Lead Amounts
      Within Mid-Spiked Dust Samples That is Attributable to Lab-to-Lab and
      Within-Lab Sources, Based on Analyses Performed With and Without
      Statistical Outliers Included	 66

-------
                            Table of Contents (cont.)

                              List of Figures (cont.)
5-12  Estimated Percentage of Total Variability in Reported Lead Amounts
      Within Spiked Paint Samples That is Attributable to Lab-to-Lab and
      Within-Lab Sources, Based on Analyses Performed With and Without
      Statistical Outliers Included	 67

5-13  Estimates of Lab-to-Lab and Within-Lab Sources of Variability in
      Reported Lead Amounts Within Low-Spiked Dust Samples,  Based
      on Analyses Performed With and Without Statistical Outliers Included   .... 68

5-14  Estimates of Lab-to-Lab and Within-Lab Sources of Variability in
      Reported Lead Amounts Within Mid-Spiked Dust Samples, Based
      on Analyses Performed With and Without Statistical Outliers Included   .... 69

5-15  Estimates of Lab-to-Lab and Within-Lab Sources of Variability in
      Reported Lead Amounts Within Spiked Paint Samples, Based
      on Analyses Performed With and Without Statistical Outliers Included   .... 70
                                       XI

-------
This page left blank intentionally.
               XII

-------
            NATIONAL LEAD LABORATORY ACCREDITATION PROGRAM:
                 DOUBLE-BLIND PROFICIENCY-TESTING PILOT STUDY

EXECUTIVE SUMMARY

       This report documents the design, execution, results, and conclusions of a pilot study to gather
information on how a double-blind proficiency-testing program could be incorporated within EPA's
National Lead Laboratory Accreditation Program (NLLAP).  Currently, the proficiency-testing
program used within the NLLAP is the Environmental Lead Proficiency Analytical Testing (ELPAT)
Program (NIOSH, 1994). The ELPAT Program is single-blind, in that the laboratories are aware that
they have received a batch of proficiency-test samples for analysis, but they are unaware of the
amounts of lead in these samples. In a double-blind program, the laboratory is unaware that its
proficiency is being evaluated. Neither is the laboratory aware that proficiency-test samples are
included within a batch of samples received for analysis. Therefore, a double-blind program is more
likely than a  single-blind program to characterize the overall performance of routine field sample
analyses.

       Twelve clients of NLLAP-recognized laboratories provided proficiency-test samples to
laboratories in this double-blind proficiency testing pilot study and reported the results of analytical
testing on these samples.  These laboratories were found to be representative of NLLAP-recognized
laboratories in their performance within Round 22 of the ELPAT Program. In each of three double-
blind testing  rounds, these clients were instructed to include the proficiency-test samples within their
next available batch of field samples for shipment to one or more NLLAP-recognized laboratories for
lead analysis. Within each double-blind testing round, 9 of the 12 clients who  routinely collected dust
wipe samples for lead analysis were provided with from 4 to 8 dust wipe proficiency-test samples to
send to a particular laboratory. These samples were spiked with lead-dust, with half (i.e., from 2 to 4
samples) at a lower nominal lead amount than the other half. In addition, 10 of the 12 clients who
routinely collected paint chip samples for lead analysis were provided with from 2 to 4 paint chip
proficiency-test samples to send to a particular laboratory  in a given double-blind testing round, where
all paint samples contained the same nominal amount of lead.

       The three  types of dust-wipe proficiency-test samples analyzed in this  double-blind pilot study
(i.e., low-spiked dust wipes, mid-spiked dust wipes, paint chips) originated from three of the batches of
proficiency-test samples that were prepared for Round 22 of the ELPAT Program. Therefore, the
laboratories in this pilot study had previously analyzed one of each type of proficiency-test sample
within Round 22 of the ELPAT Program. Generally, laboratories analyzed proficiency-test samples in
the first testing round in this double-blind pilot study within 30 days of analyzing proficiency-test
samples in Round 22 of the ELPAT Program.

       Client recruitment involved submitting  a news brief for publication in a monthly trade journal to
the lead hazard control industry documenting the study and its objectives and inviting interested clients
to contact the study team, and contacting risk assessors directly for their participation.  The recruitment
                                             XIII

-------
process revealed a need to develop a strategy to educate laboratory clients on the benefits of a
double-blind proficiency-test program, which would likely increase their participation in such a
program.

       Many clients encountered in the recruitment process were small organizations that either
collected relatively few samples on a monthly basis for analysis, or had limited resources that made it
difficult to participate in this study.  For these reasons, and to improve the overall success rate of
recruitment, clients were reimbursed for analysis costs associated with the proficiency-test samples they
were provided in this study. In addition, materials such as  sample containers and dust wipes were
provided to some clients.  The policy of providing such incentives to get laboratory clients to participate
needs to be reviewed when establishing a double-blind program.

       Although the three types of double-blind proficiency-test samples originated from the same
batches of samples used in Round 22 of the ELPAT Program, the variability in the double-blind pilot
study data is greater than the variability in the ELPAT Round 22 data for the same sample types and
laboratories.  In fact, while all of the laboratories that analyzed proficiency-test samples in the double-
blind pilot study reported acceptable results within Round 22 of the ELPAT Program, 29
(approximately 11%) of the double-blind pilot study results were outside of the ranges that were
considered acceptable within the ELPAT Round 22. Of these results, 10 were for low-spiked  dust-
wipe samples from four of the ten laboratories analyzing dust-wipe samples, 8 were for mid-spiked
dust-wipe samples from four of the ten laboratories analyzing dust-wipe samples, and 11 were for paint
chip samples  from three of the ten laboratories analyzing paint chip samples. Thus, double-blind pilot
study data for some laboratories may be more likely than single-blind study data  to exceed the
acceptance limits determined from data within the (single-blind) ELPAT Program.

       Twelve of the double-blind pilot data values were labeled as  statistical outliers (i.e., very high or
very low values relative to other data of the same sample type in the given  testing round).  While some
of these outliers were later determined to be the result of laboratories reporting invalid lead amounts,
they were values that the  laboratories reported to the clients and, therefore, would be used by the
clients to make decisions  on lead contamination. Therefore, statistical summaries and analyses were
performed both with and  without the outliers included. Other extreme data values that were reported
inaccurately by the clients were revised upon obtaining laboratory reports and noting how the
laboratory reported these  values to the client.

       Table ES-1  summarizes average lead amounts reported in the double-blind proficiency-test
pilot study, by sample type and double-blind testing round.  The target lead amounts associated with
each sample type, as determined within Round 22 of the ELPAT Program, are the means specified in
the last row of this table.  Also included in Table ES-1 are ELPAT Round 22 data summaries for the
group of laboratories participating in this pilot study, as well as for all 118  participating NLLAP-
recognized laboratories (labeled as "reference laboratories" within ELPAT Round 22). Note that
statistical outliers occasionally contribute to inflated standard
                                              XIV

-------
Table ES-1.  Average (and Standard Deviation) of Measured Lead Amounts in the
              Double-Blind Pilot Study (by Testing Round) and in Round 22 of the
              ELPAT Program, by Sample Type

Mean (Standard Deviation) (Sample Size)
Low-Spiked Dust
Samples (//g lead)
Mid-Spiked Dust
Samples (//g lead)
Paint Samples
(% lead by weight)
Double-Blind Pilot Data
DB Round 1
DB Round 2
DB Round 3
124.1 (39.4) (28)
132.7 (29.8) (30)
129.4 (24.5) (24)
265.6 (81.6) (28)
276.2 (50.6) (30)
282.1 (43.7) (24)
0.639 (0.124) (32)
0.749 (0.340) (32)
0.710(0.204) (29)
Double-Blind Pilot Data, With Statistical Outliers Excluded (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
133.4 (20.1) (26)
128.9 (21.8) (29)
129.4 (24.5) (24)
285.6 (36.8) (26)
281.3 (43.1) (29)
282.1 (43.7) (24)
0.658(0.070) (31)
0.659 (0.066) (29)
0.656 (0.039) (27)
ELPAT Round 22 Data1
Labs participating in the
double-blind pilot
NLLAP-recognized labs
NLLAP-recognized labs
(Winsorized data)2
135.5 (10.1) (10)
127.9 (17.4) (118)
129.0(11.6) (118)
284.2 (19.8) (10)
294.5 (270.2) (118)
272.0(24.4) (118)
0.655 (0.056) (11)
0.726 (0.881) (118)
0.6454 (0.040) (118)
1 For a given sample type, one sample was tested per laboratory in Round 22 of the ELPAT Program.
2 The specified means represented target lead levels for the specified sample types in Round 22 of the ELPAT Program.
deviations both in the double-blind testing rounds and in Round 22 of the ELPAT Program among
NLLAP-recognized laboratories.

       Even when statistical outliers were excluded, the standard deviation of the dust wipe
proficiency-test sample data in each double-blind testing round was more than twice that reported in
Round 22 of the ELPAT Program for the same group of laboratories (Table ES-1).  While Levene's
test indicated that, for each dust-wipe sample type, the differences in these standard deviations across
the four testing rounds (the three double-blind testing rounds plus ELPAT Round 22) was not
significantly different at the 0.05 level, the standard deviations of the log-transformed dust-wipe data
(i.e., the data used in statistical analyses within this report) were significantly different at the 0.05 level
across testing rounds. This was true even when statistical outliers were not excluded when calculating
standard deviations within the double-blind testing rounds. The significant difference was primarily due
                                             xv

-------
to the lower variability observed in Round 22 of the ELPAT Program relative to the double-blind
testing rounds.

        Compared to the dust-wipe data, the double-blind proficiency-test paint chip sample data had
standard deviations that more closely matched the standard deviation for paint chip sample data from
ELPAT Round 22 (Table ES-1) when statistical outliers were excluded. These standard deviations did
not differ significantly (at the 0.05 level) across testing rounds.

        While it appears from Table ES-1 that the average double-blind dust-wipe proficiency-test
sample results more closely matched the ELPAT Round 22 target levels in double-blind rounds 2 and 3
compared to double-blind round 1, the differences in average results for dust-wipes relative to their
target levels were not significantly different across double-blind testing rounds at the 0.05 level for either
sample type, based on tests performed within an analysis of variance. Furthermore, the analysis of
variance concluded that when statistical outliers were excluded, the extent of variability in average
laboratory results (i.e., lab-to-lab variability) did not differ significantly among the double-blind testing
rounds, nor did the extent of variability in the results of multiple sample analyses within the same
laboratory (i.e., within-lab variability) differ significantly among the double-blind testing rounds, at the
0.05 level.  The same statistical conclusions were made on the results of paint chip sample analyses.
This suggests that for each type of proficiency-test sample, with the exception of a few sample results
that were labeled statistical outliers, the  overall performance of the laboratories in this double-blind pilot
study did not differ significantly across the three double-blind testing rounds.

        Table ES-2 presents estimates of the percentages of total variability in the double-blind pilot
study dat that was associated with lab-to-lab variation, according to double-blind testing round. This
table shows that in testing rounds containing statistical outliers, lab-to-lab variability represented at least
90% of total variability. However, when statistical outliers were not present,  lab-to-lab variability was
generally about 70% of total variability for both types of dust-wipe proficiency-test samples, and
slightly less than 50% of total variability for the paint chip proficiency-test samples.  These latter
percentages represent data over the entire study, as these percentages did not differ significantly (at the
0.05 level) across testing rounds when statistical outliers were excluded. However, these percentages
were slightly higher when calculated for a specific double-blind testing round. Statistical acceptance
criteria in a double-blind program  designed similarly to this pilot study should consider both lab-to-lab
and within-lab components of variation.

        For each proficiency-test sample type, the overall average measurement reported by the
laboratories within a double-blind testing round did not differ significantly across the three testing rounds
(at the 0.05 level), and deviation of this  average from the target level associated with the proficiency-
test sample type was not statistically significant overall.  This finding, along with the finding mentioned
above that both lab-to-lab and within-lab components of variation did not differ significantly across
testing rounds when statistical outliers were excluded from analysis, suggests that overall performance
of the laboratories in this double-blind pilot study did
                                               XVI

-------
Table ES-2.   Percentage of Total Variability in  Double-Blind Pilot Study Results That
               Can Be Attributed to Lab-to-Lab Variation
Double-Blind Testing
Round
Low-Spiked Dust
Samples
Mid-Spiked Dust
Samples
Paint Samples
All Double-Blind Pilot Data Included
DB Round 1
DB Round 2
DB Round 3
99.6%
64.7%
85.2%
99.8%
69.1%
84.0%
55.8%
94.0%
98.8%
Statistical Outliers Excluded (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
All DB Rounds*
70.9%
84.3%
85.2%
68.6%
80.5%
80.8%
84.0%
68.7%
73.2%
53.4%
49.5%
46.4%
  Provided as the percentages did not differ significantly (at the 0.05 level) across DB testing rounds.
not differ significantly across the three double-blind testing rounds for each proficiency-test sample type
(when statistical outliers were disregarded).

       Additional research is needed on identifying appropriate dust-wipe, paint, and soil materials for
use in preparing proficiency-test samples for a double-blind program.  The different types of wipes
available for field dust collection can make it difficult to select a specific type of wipe for use in the
proficiency-testing program. Paint-chip proficiency-test samples are finely ground, which makes them
easily distinguishable from paint chip samples collected in the field, as field samples tend to be larger
chips attached to pieces of substrate.  Soil samples were not considered in this pilot study, because the
small amounts of soil (1-gram) used to prepare proficiency-test samples are considered easily
distinguishable from composite soil samples (5-10 grams) typically collected in the field.
                                              XVII

-------
This page left blank intentionally.
             XVIII

-------
1.0   INTRODUCTION

1.1    BACKGROUND

       Lead exposure has a well-known association with adverse health effects in humans, especially
young children. To identify and control or abate lead hazards in the nation's housing, lead inspections
and risk assessments are conducted under guidelines dictated by regulation under Section 402 of the
Toxic Substances Control Act (TSCA) (40 CFRPart 745, August 29, 1996). In these activities,
samples are collected of those environmental media that would most likely provide lead exposure to
children when a lead source is present (e.g., dust, soil, paint chips). These samples are then
transported to analytical laboratories to determine the amount of lead that is present in these samples.

       In the early 1990s, the U.S. Environmental Protection Agency (EPA) began activities to
develop a national laboratory accreditation program for laboratories involved in the analysis of lead in
dust, soil, and paint chips. EPA's official recognition as the responsible federal agency for developing
such a program was made within Section 405(b) of Title IV of TSCA. This program, known as the
National Lead Laboratory Accreditation Program (NLLAP) (68 FR 38656; July 19, 1993), recognizes
laboratories for their ability to analyze lead in dust, soil, and paint chips in support of efforts to identify,
abate, and control lead-based paint and lead-based paint hazards.

       In order to be recognized by EPA under the NLLAP, laboratories must achieve the following:

       •      Successful participation in the Environmental Lead Proficiency Analytical Testing
              (ELPAT) Program (NIOSH, 1994), a cooperative effort of the National Institute for
              Occupational Safety and Health (NIOSH) and the American Industrial Hygiene
              Association (AIHA).

       •      A successful systems audit of laboratory operations, conducted by a laboratory
              accreditation organization participating in the NLLAP.

Successful participation in the ELPAT Program is required for NLLAP-recognized laboratories. The
ELPAT Program is a single-blind proficiency testing program, where the proficiency testing service
sends proficiency-test samples directly to the laboratories for analysis.  Therefore, while the
laboratories are unaware of the amount of lead in samples that are part of the program's performance
evaluation, they are aware of when they are analyzing such  samples.

1.2    PURPOSE FOR WORK

       The need to perform lead analyses within the rapid-response, high-volume, cost-cutting
environment common in many laboratories can negatively impact a laboratory's routine performance.
Therefore, laboratories may try to minimize this impact when  analyzing a batch of proficiency-test
samples. This action is possible in a single-blind proficiency-testing program, such as the current

-------
ELPAT Program, in which the laboratories are aware of when they are analyzing proficiency-test
samples.

       By contrast, a double-blind proficiency testing program would incorporate proficiency-test
samples blindly within batches of field samples provided by clients of the laboratory. Because the
proficiency-test samples would not be identified as such in these batches, laboratories would ideally not
recognize such samples as proficiency-test samples and would therefore treat these proficiency-test
samples with the same degree of care and attention they use in handling and analyzing routine field
samples. Thus, variability associated with field sample testing may be better estimated by the results of
double-blind testing rather than single-blind testing. In addition, a double-blind program can increase
the level of alertness by the laboratories concerning quality, as they would not know when double-blind
proficiency-test samples may arrive for analysis.

       In determining whether to consider a double-blind proficiency testing program as a supplement
to the ELPAT Program within the NLLAP, it was necessary to conduct a pilot version of a double-
blind program to evaluate logistical considerations, to develop appropriate protocols, and to establish
statistical performance criteria.

1.3    OBJECTIVES

       The objectives of this pilot study were to:

       •      Design and test a protocol for double-blind proficiency testing under the NLLAP

       •      Compare analytical performance of NLLAP-recognized laboratories between the
               double-blind pilot study and the (single-blind) ELPAT Program.

Note that this report does not suggest appropriate acceptance criteria for double-blind proficiency-test
samples, but instead provides important information to be used by those who must determine such
criteria.

1.4    REPORT ORGANIZATION

       Chapter 2 documents the study design used in the pilot and how it was implemented.
Procedures used to recruit study participants, the outcome of the recruitment process, and experiences
encountered in recruiting and working with the study participants and in obtaining the study data are
presented in Chapter 3. The information in Chapter 3 provides valuable information in determining how
to organize and operate a double-blind proficiency test study. Chapter 4 presents the methods used to
perform data analyses on the study data, and results of executing these methods on the pilot study data
are presented in Chapter 5. Quality assurance issues, such as sample fabrication, data management
and sample tracking, and data quality checking are discussed in Chapter 6. Finally, conclusions and

-------
recommendations made from executing this pilot study and in support of the study objectives are
presented in Chapter 7.

       Research Triangle Institute (RTI) prepared the proficiency-test samples and performed sample
transfer and shipment to the study participants. A report on RTFs responsibilities on this pilot study is
found in Appendix A.

-------
This page left blank intentionally.

-------
2.0   STUDY DESIGN AND OPERATION

       A double-blind proficiency-testing program makes every effort to conceal from the laboratory
the identity of proficiency-test (PT) samples, presenting them as field samples. Therefore, it was
necessary to design this pilot study and to establish procedures to minimize the likelihood of laboratory
recognition of its participation in this study.  This chapter presents the study's underlying design to
support the objectives presented in Chapter 1. This chapter also presents how procedures such as
sample distribution and analysis reporting were implemented.

       The overall organization of the double-blind pilot study, including the organizations involved in
implementing the study, is illustrated in Figure 2-1. This figure indicates that the "proficiency-testing
service" (PTS) of this pilot study, AIHA with support from RTI, was the primary point of contact with
the study participants ("Clients"), who in turn were the points of contact with the laboratories who
analyzed the proficiency-test samples. As a means of comparison, this figure also contains the flow of
materials and information that occurs in each round of the single-blind ELPAT Program, which involves
the above participants as well  as NIOSH.

       To allow for comparisons to single-blind results and to alleviate any  sample integrity concerns
among laboratories, operation of this double-blind pilot study was coordinated with Round 22 of the
ELPAT Program.  The proficiency-test samples analyzed within the pilot study consisted of the same
batches of materials used to prepare the proficiency-test samples for Round 22 of the ELPAT
Program.  The proficiency-test samples analyzed within the pilot study consisted of the same bulk test
material that was used in preparing the proficiency-test samples for Round 22 of the ELPAT Program.
Furthermore, the PaceWipe™ brand of dust-wipe, used in the ELPAT Program,  was also used in this
pilot study.  The PaceWipe™ is similar in composition to a pre-packaged moist towelette provided by
restaurants for hand cleaning.  This would be likely to reduce concerns from laboratories that the
double-blind samples were prepared differently from single-blind samples, which could have led to
more highly variable results.

2.1    STUDY PARTICIPANTS AND TESTING ROUNDS

       As discussed in Chapter 3, this pilot study recruited lead-based paint inspectors and risk
assessors who contract with NLLAP-recognized laboratories to perform lead analysis on the
environmental samples (dust and paint) that they collect from residences. The initial goal of the study
design was to enroll enough laboratory clients so that from 10-15 different laboratories would each
have two clients submit a given type of proficiency-test sample for analysis for each test round. This
would permit lab-to-lab variation in the analytical results to be estimated and would provide information
on how performance at a given laboratory may differ from client to client. It would also reflect a design
option for a double-blind program that would reduce a laboratory's vulnerability to having its
performance negatively impacted by a specific client's actions. However, while each type of
proficiency-test sample was analyzed by at least 10 different laboratories in this study, most
laboratories received samples from only one client.

-------
                                                   ELPAT and doub e-b md resu ts
                                                         ELPAT results
                                                  Double-blind results
           Sample containers
                Double-blind PT samples
                     Double-blind PT samples
Results of field and
                           with field samples
double-blind PT samples
         ELPAT PT samples
           ELPAT results
Figure 2-1.   Organization of the Double-Blind Proficiency Testing Pilot Study, and
              Relationships Between Participating Organizations
       This pilot study consisted of three testing rounds. The first testing round was initiated within
one month of laboratories analyzing samples in Round 22 of the ELPAT Program, and the second and
third rounds occurred approximately one and three months, respectively, following the first round.
Proficiency-test samples analyzed in the second and third rounds were analogous to the samples
submitted in the first round. Within each round, each recruited client was supplied with proficiency-test
samples, with instructions to place these samples into their next available batch of regular field samples
(of the given sample type) and to submit them to the laboratory for analysis. Incorporating multiple

-------
testing rounds in this study provided information on how laboratory performance in a double-blind
proficiency-testing program may vary over time and over different analysis conditions (e.g., different
technicians, calibrations).  In particular, it was of interest to determine whether laboratory performance
improved over time, especially if a given laboratory eventually suspected that they were being "double-
blinded."

2.2    TYPES OF PROFICIENCY-TEST SAMPLES

       Two different types of proficiency-test samples were considered in this pilot study: dust-wipes
and paint chips. The wipe method for dust collection was considered for two reasons:

       •       The wipe method is most often used by risk assessors to collect dust samples within a
               residence.

       •       The rule (in response to Section 403 of Title IV of TSCA) that establishes criteria for
               determining when household dust is considered lead-contaminated (40 CFR Part 745;
               January 5, 2001) assumes wipe techniques for sampling household dust for lead
               analysis.

Paint chip samples were considered for three reasons:

       •      They are usually collected for laboratory analysis whenever in situ methods (e.g., x-ray
               fluorescence, or XRF) provide inconclusive results  on the presence of lead in paint, or
               when such methods cannot be used (e.g., when paint is found on certain curved or
               ornate surfaces).

       •      Risk assessors may collect paint chip samples from deteriorated paint surfaces while
               doing a risk assessment.

       •      To evaluate the feasibility of including paint chip samples within a double-blind testing
               program.

       Within Round 22 of the ELPAT Program, proficiency-test samples were prepared (using
methods described in Section 6.1) at four different lead levels within each of three matrices (dust, soil,
paint chip), thereby representing 3x4=12 sample types. Proficiency-test samples prepared within three
of these sample types were selected for use in the double-blind pilot program.  These sample types are
documented in Table 2-1  and consist of dust-wipe samples at the two lowest lead levels  and paint-chip
samples at the second-lowest lead level. The lower two dust-lead  levels were selected because the
ELPAT Program typically observes an increased failure rate at low lead levels (Schlecht et al,  1996),
and the Federal action level at the time of ELPAT Round
                                              7

-------
Table 2-1.    Information on the Three Types of Dust and Paint Proficiency-Test
               Samples Prepared in Round 22 of the ELPAT Program and Used  in the
               Double-Blind  Pilot Study
Sample Type ID
Within ELPAT
Round 22
22W2
22W3
22P4
Matrix/Lead Level
Dust-wipe/lower level
Dust-wipe/higher level
Paint-chip
Target
Lead
Amount1
129//g
272 //g
0.6454%
Acceptance Range for
Individual PT Sample
Results Within ELPAT
Round 222
94 - 164//g
199 - 345 //g
0.5264- 0.7645 %
       1 Mean of Winsorized data for 118 NLLAP-recognized ("reference") laboratories within ELPAT Round 22 (as documented
       in the round's individual laboratory reports), with each laboratory analyzing one sample of the given matrix and lead level.
       2 Plus and minus three standard deviations of the target lead amount, as calculated from Winsorized data for the 118
       reference laboratories within ELPAT Round 22 and as documented in the round's individual laboratory reports.
22 was 100 |ig/ft2. The selected paint-lead level is within the range at which some in situ XRF
instruments yield inconclusive results for the presence of lead (USEPA, 1995; USHUD, 1995).

       Table 2-1 also includes the following for each sample type:

       •      the "target lead amount," or the lead amount to which each result of the given sample
               type was compared in this study's statistical analysis

       •      the "acceptance range," within which ELPAT Round 22 proficiency-test sample results
               were categorized as "acceptable" for the given sample type.

These two columns were calculated within Round 22 of the ELPAT Program using Winsorized data for
thel 18 laboratories in this testing round that were NLLAP-recognized (labeled as "reference
laboratories)."1  The target lead amount is the mean of these data, and the acceptance range is plus and
minus three standard deviations of this mean.

       Paint chip samples used in the ELPAT Program can be difficult to use as double-blind
proficiency-test samples because they are ground to a small particle size (<120 iim) during the material
homogenization stage of sample development.  Paint chips collected in the field are typically coarser
and more likely to contain larger pieces of the substrate compared to proficiency-test samples.
Therefore, to reduce laboratory suspicion of the paint-chip proficiency-test samples, clients within this
        1 Reference laboratories are the NLLAP-recognized laboratories going into the given testing round and are
accredited by AIHA. Winsorization involves replacing the highest 5% of the 118 data points (for a given sample
type) with the maximum of the remaining data, and the lowest 5% of these data with the minimum of the remaining
data.
                                               8

-------
double-blind pilot study were instructed to refer to these samples as either "paint chips removed from a
brick- or concrete-painted surface" or as "dust contaminated by paint."

       While the ELPAT Program also includes proficiency-test soil samples, and soil is collected by
many risk assessors, soil samples were not included among the proficiency-test samples in the double-
blind pilot study. This is because proficiency-test soil samples in the ELPAT Program are typically less
than one-half of the mass of soil samples collected in the field and, therefore, could be readily identified
as proficiency-test samples. As field soil samples often exceed 10 grams, proficiency-test soil samples
of mass similar to typical field samples would be prohibitively expensive to prepare.

2.3    NUMBERS OF PROFICIENCY-TEST SAMPLES  PROVIDED
       TO EACH CLIENT

       Within each testing round, each client received at least two proficiency-test samples of a given
type (dust wipe, paint chip). Clients were provided with one or both types of samples, depending on
what types of samples they typically provide to NLLAP-recognized laboratories monthly. By having at
least two analytical results for each type of proficiency-test sample  for a given client in  a given testing
round, within-laboratory variability in the analytical results could be estimated within each testing round.

       In some situations where a laboratory received a given type of proficiency-test sample from
only one client, that client was provided with four samples of the given type, rather than two. The
additional  samples provided additional information on within-laboratory variability and permitted each
laboratory to analyze the same total number of samples in a given round, and therefore, to provide the
same amount of information.

       Specific numbers of proficiency-test samples provided to each participating client (and
forwarded to each laboratory) are provided in Section 3.3.

2.4    SAMPLE TRANSFER TO CLIENTS AND LABORATORIES

       The flow diagram in Figure 2-1 illustrates how the proficiency-test samples in this double-blind
pilot study were transferred from representatives of the proficiency-testing service (RTI) to the
laboratory clients, who then sent the samples to the laboratory for analysis.  The study design had to
include procedures for the various transfers of these samples from one participant to another, while
retaining their proper identification, limiting their ability to be identified as proficiency-test samples, and
not compromising their lead content. These procedures are discussed briefly below.

2.4.1  Transferring  Samples to Client-Supplied Sample Containers

       Placing proficiency-test samples in the same type of sample container that a client uses for
regular field samples was necessary to ensure that the proficiency-test samples remained blind within a
batch of regular field samples. As a result, the clients were requested to provide the proficiency-testing
service with a specified number of uncontaminated sample containers that they  normally used when

-------
submitting samples to an NLLAP-recognized laboratory.  Clients were notified of this requirement
during the recruitment process. The number of containers to supply was included in the letter the clients
received when they were selected for the study (Appendix C).  The types of sample containers
received from the clients included glass jars, plastic jars, centrifuge tubes, and plastic bags.

       Upon verifying that sample containers from participating clients were free of lead contamination
(Section 6.1.3), the proficiency-test samples (paint chips and PaceWipes™ spiked with leaded dust)
were transferred from the plastic scintillation vials in which they were initially stored after preparation to
the client-supplied sample containers.  An analytical method was applied to verify the quantitative
transfer of samples from one container to another (Section 6.1.4).

2.4.2 Shipping Samples  to Clients

       After placing each proficiency-test sample into client-supplied sample containers, each sample
was assigned a unique identification number (the "PTS Sample ID"). The form of the PTS Sample ID
is provided in Section 5.1 of Appendix A. This ID number was printed on a label which was placed
onto a clean, plastic bag.  Each sample was then placed into its plastic bag, and the bag was sealed.  At
this point, the double-blind proficiency-test samples were ready for shipment to the clients. All samples
shipped in the three rounds of the study were prepared and identified prior to the first testing round.

       The proficiency-testing service shipped the proficiency-test samples to the recruited clients via
priority mail in three distinct testing rounds. This approach differed from the initially-proposed
approach which was to  provide samples for all three testing rounds in one shipment at the beginning of
the study. In the original  plan, the clients were to store the samples appropriately until they were
notified to take a given  set and incorporate them into their next  available batch for analysis. This one-
shot approach to shipping samples was later revised to consist  of three separate shipments, when
concerns were raised as to whether the clients had sufficient storage facilities (e.g., cold room for
storing dust-wipe samples) and whether the clients could ensure the integrity of the samples during
storage. In addition, it was uncertain whether all clients would correctly retrieve the appropriate
samples at the right time for analysis if they received all  of the samples at one time.

       Each shipment  of proficiency-test samples was  accompanied by copies of the Sample Tracking
and Analysis Report Form and a cover letter. The Sample Tracking and Analysis Report Form,
included in Appendix C, contained a list of PTS Sample IDs, information on the client receiving these
samples, information on the laboratory to whom the client would provide these samples for analysis
(only the laboratory name was specified; additional laboratory information was provided by the client),
sample matrices and weights,  and dates when the samples and results were shipped or received.
Samples that were sent to different laboratories appeared on different copies of the form.

       The recruitment staff provided the proficiency-testing service with necessary information for
making sample  shipments to clients (i.e., information to be placed on the Sample Tracking and Analysis
Forms), including the clients'  addresses, responsible parties, the laboratory(ies) to which a client would
                                              10

-------
send samples, and the number and types of samples to send to each client.  For ten clients, samples for
double-blind (DB) Rounds 1, 2, and 3 were shipped on February 27, April 3, and June 1, 1998,
respectively. These dates corresponded to approximately one, two, and four months following the date
that the proficiency-testing service shipped proficiency-test samples to laboratories in Round 22 of the
ELPAT Program.  One of the 10 clients reported that they did not receive the first round of samples; a
new batch of DB Round 1 samples was sent to this client on April 3,  and DB Round 2 samples were
sent on May 1.  DB Round 1 samples for one client were sent on March 10. Samples for one client
who enrolled late in the study were sent on April 21, May 18, and June 15.  Samples for two clients
who enrolled late in the study were sent on April 21, May 18, and June 1.

2.4.3  Receiving and Shipping of Samples by Clients

       Upon being selected for participation in the study (and again with the batch of DB Round 1
samples), the clients were provided with written instructions and procedures for proper handling and
shipping of the proficiency-test samples that they received in this study. These included the following:

       •       Samples should be stored in a locked area with limited access (e.g., cabinet, closet)
               until they are placed within the next available batch of field samples.

       •       Sample containers and their  contents should not be opened  or otherwise tampered
               with.

       •       Clients were to incorporate the samples randomly within their next available batch of
               field samples that are earmarked for analysis at that laboratory (or place  the samples in
               their own batch if no field samples are available within one month of receiving the
               proficiency-test samples).

       •       Clients were to assign identifications (IDs) to all samples in a batch.  For the
               proficiency-test samples, the clients were to remove the sample container from the
               outer plastic bag, noting the PTS Sample ID that is on this outer bag, and place a label
               on  the sample container containing their assigned ID.

The instructions also contained detailed information on how to complete the  Sample Tracking and
Analysis Report Forms  during the entire shipping and analysis process. As the proficiency-test samples
were received by the client, the samples were given different IDs by the client, the samples were
shipped to the laboratories, and the analytical results received from the laboratories. The clients were
instructed to store these tracking forms in a secure location. See Appendix C for all instructions and
procedures provided to the clients upon their enrollment in the study.

       Once a client had incorporated the proficiency-test samples for a particular laboratory within its
next available batch of regular field samples of the same type (dust wipe, paint chip), the client shipped
the samples to the laboratory for analysis via its routine procedures, and ideally within one month of
                                             11

-------
receiving the proficiency-test samples. The client was instructed to fax the tracking forms to the
proficiency-testing service after recording the client sample IDs and the date that the samples were
shipped to the laboratory in the tracking forms.

2.5     REPORTING RESULTS

        When laboratory clients received proficiency-test samples analytical results from the
laboratories, they transcribed the analysis results onto the Sample Tracking and Analysis Report Form,
next to the corresponding sample IDs. The client then faxed the completed Sample Tracking and
Analysis Report Form to the proficiency-testing service, along with any other pertinent information that
the client felt might provide important insights about the analysis.   After the proficiency-testing service
had received all laboratory results that it expected within a given testing round, the results were entered
into a spreadsheet and were forwarded for statistical summary and analysis.  The proficiency-testing
service also forwarded copies of the Sample Tracking and Analysis Report Forms and any
accompanying ancillary information from the clients and laboratories. The proficiency-testing service
also provided results of Round 22 of the ELPAT Program, which used proficiency-test samples created
from the same batch as those used in the double-blind pilot study and which preceded the first round of
the double-blind pilot study by approximately one month, along with the information from the ELPAT
Program found in Table 2-1 for each type  of proficiency-test sample.  See Section 6.2 on details
concerning types of data,  data management, and data tracking issues.
                                             12

-------
3.0   STUDY PARTICIPANTS

       This chapter documents the approaches taken to recruit risk assessors and lead-based paint
inspectors for this pilot study who regularly submit dust-wipe and/or paint chip samples to NLLAP-
recognized laboratories for analysis. As discussed in Chapter 2, the double-blind nature of this pilot
study required that laboratory clients, rather than laboratories themselves, be recruited for the study.

       As will be discussed in detail below, the recruitment process resulted in 19 clients being
selected for participation in the study. Seventeen were privately-owned lead service providers, and two
were state-operated health departments. Of the 19 clients selected, 12 provided analytical results
(from 11 privately-owned laboratories and one state-operated health department's laboratory) on the
proficiency-test samples they  were provided.

       Sections 3.1 and 3.2 present the general approaches used to perform recruitment and the
criteria used to determine study eligibility.  Section 3.3 documents the results of the recruitment process,
including reasons why some potential clients did not wish to participate or were not selected for
participation.  Sections 3.4 and 3.5 present notes on client recruitment and communication throughout
the course of the study.

3.1    RECRUITMENT APPROACHES

       To recruit privately-owned risk assessors and lead-based paint inspectors, this study used two
approaches:

       •      Publishing a news brief on the study in a newspaper widely-read by the lead hazard
               control industry, inviting interested parties to contact EPA.

       •      Making direct calls to lead-based paint inspectors and risk assessors.

Also, recruitment occurred at a meeting on the NLLAP held in December 1997,  whose attendees
included officials of several state-operated lead programs. Note, however, that it was not of interest to
recruit many state-operated agencies for the study because of the potential that they might perform
more QA/QC activities than private firms, resulting in more accurate laboratory results compared to the
results associated with privately-owned risk assessors.

       In both approaches to recruitment, recruiters interviewed potential study participants using the
script found in Appendix B. In the interview, recruiters provided information on the double-blind pilot
study, determined the client's interest in participating, and obtained information from interested clients to
help assess whether a given client should be selected with higher or lower priority relative to other
clients. Information obtained from interested clients included the following:
                                              13

-------
        •      whether they submit dust-wipe or paint chip samples on at least a monthly basis to one
               or more NLLAP-recognized laboratories

        •      names of NLLAP-recognized laboratories with which they contract, along with
               information on their testing volume, analytical methods, and detection limits

        •      approximate sizes of sample batches

        •      information on sample containers used and types of dust-wipes used.

Details on each approach to recruiting study participants and the outcome of implementing these
approaches are as follows.

Recruitment Approach #1

        In the first recruitment approach, a news brief on the pilot study was prepared and submitted to
a trade journal for the lead hazard control industry for publication in an upcoming edition. This news
brief, found in Figure 3-1, summarized the need for a double-blind proficiency testing program and how
recruited clients of laboratories would assist with the program. Interested participants were asked to
contact  EPA or its contractor, Battelle.

        A total of eight laboratory clients contacted EPA or Battelle as a result of reading the news
brief, indicating their interest in participating. Seven of these clients were selected for the study.

Recruitment Approach #2

        The second recruitment approach involved telephoning a sample of lead evaluation service
providers (lead inspectors, risk assessors, and abatement contractors)  who were included in the Lead
Listing (the National Lead Service Providers' listing system) dated November 21, 1997. The service
providers in the Lead Listing are grouped according to the state(s) in which they provide services.
Service  provider information in this list included name, telephone numbers, states in which the provider
is certified as a lead inspector,  and states in which the provider is certified as a risk assessor.2

        Only service providers included within the New Jersey, Ohio, Pennsylvania, and Virginia
segments of the Lead Listing were considered for recruitment in this study. This was done to
       2 The Lead Listing is operated by QuanTech, Inc., andlCF Information Technology, Inc., for the U.S.
Department of Housing and Urban Development (HUD)'s Office of Lead Hazard Control, and is supported by the
National Lead Assessment and Abatement Council (NLAAC). It can be accessed atwww.leadlisting.org or by
calling 1-888-LEADLIST.

                                              14

-------
 Published in the "Newsline" column of Lead Detection and Abatement Contractor. January 1998 edition:
 EPA Seeks Lab Clients for "Double Blind" Test Program

         EPA is seeking clients of laboratories accredited by the National Lead Laboratory Accreditation Program
 (NLLAP) to participate in a double-blind proficiency testing pilot study.
         At the present time, laboratories recognized by the NLLAP are required to successfully participate in the
 Environmental Lead Proficiency Analytical Testing Program (ELPAT).  The ELPAT is a single-blind proficiency
 testing study. In a single-blind proficiency testing study, the participating laboratories are aware of when they
 receive proficiency-test samples.  Because laboratories know when they are analyzing proficiency-test samples
 under the ELPAT, there is a potential to deviate from the routine analysis procedures provided for field samples,
 providing more attention to the analysis of the test samples.  In a double-blind proficiency testing study,
 laboratory clients would submit proficiency-test samples to laboratories as routine field samples. Ideally, the test
 samples would be indistinguishable from the field samples. As laboratories would be unaware of when they
 would be analyzing proficiency-test samples, a double-blind study would give a more accurate evaluation of a
 laboratory's routine performance.
         Clients selected to participate in the pilot study would be asked to insert proficiency-test samples in
 with batches of field samples that they are submitting for analysis to an NLLAP-recognized laboratory. A
 proficiency testing service would provide proficiency-test samples to the client at no charge.  Laboratory clients
 interested in participating in the double-blind pilot study are asked to contact John Scalera of the EPA's Office of
 Pollution Prevention and Toxics by facsimile at 202/260-0001, or Robert Lordo of Battelle by phone at 614/424-
 4516 by Jan. 20.
Figure 3-1.    News Brief on the Double-Blind Proficiency Testing Pilot Study
improve the overall recruitment success rate, as these four states were among those that required
service providers to use NLLAP-recognized laboratories at the time of this pilot study.

        Telephone recruitment of laboratory clients in the Lead Listing occurred from January through
April, 1998, with most of the recruitment completed before March, 1998.  A total of 55 service
providers were contacted. Results of these contacts were as follows:

        •      21  contacts indicated an interest in participating. (As discussed in Section 3.3, not all
               of these contacts were found to be eligible for participation, and two contacts later
               declined interest.)

        •      10 contacts indicated that they were not interested in participating,  for reasons such as
               a lack of sufficient dust and/or paint sampling for lead testing, they were too busy, the
               study would require too much effort, or they did not have the resources to participate.

        •      24 providers either could not be reached, did not reply to messages left by recruiters,
               were no longer in the lead evaluation business, or no longer responded after requesting
               time to review study information provided by the recruiters.
                                                 15

-------
Some of the 21 interested contacts became interested only when a policy was adopted in this study to
reimburse study participants for analytical costs associated with the proficiency-test samples. The
reimbursement issue is discussed further in Section 3.4. Of the 21 interested clients, 11 were accepted
into the study (with one client dropping out prior to the start).

Other Recruitment Approaches

        The EPA Work Assignment Manager discussed the double-blind pilot study at a meeting held
on December 17, 1997, where state and federal government representatives interested in
environmental-lead testing met to discuss NLLAP-related subjects. At this meeting, five
representatives of state health departments (from five different states) who collected dust-wipe and/or
paint chip samples in household risk assessments and had the samples analyzed for lead by NLLAP-
recognized laboratories indicated an interest in participating in this pilot study.  Once all recruitment
efforts in this study were completed, two states remained interested and were selected to participate in
the study.

        When initially formulating the idea of a double-blind pilot study and discussing it with various
representatives of the lead inspection industry, the EPA Work Assignment Manager identified two lead
inspectors who were interested in participating. However,  these two inspectors were not available to
participate once the study was ready to begin.

3.2     CLIENT ELIGIBILITY

        During the initial telephone contact with a potential study participant, responses to the questions
in the telephone script (Appendix B) were used to determine if the laboratory client was eligible for the
pilot study and to prioritize those clients found to be eligible. To participate in the pilot study, a client
recruited via telephone must have met the following criteria regarding dust and/or paint sample
collection and analysis:

        •      Contract with an MLLAP-recognized laboratory to analyze the given sample type - The
               client must have routinely submitted dust and/or paint samples to NLLAP-recognized
               laboratories for lead testing.  This criterion allowed an evaluation of the current  routine
               performance of a subset of NLLAP laboratories.

        •      Use the PaceWipe™ or similar towelettes to collect dust samples (only for clients
               recruited to submit dust-wipe samples^ - All  proficiency-test dust-wipe samples in this
               study consisted of the PaceWipe™. Because of the need to have proficiency-test
               samples indistinguishable from regular field samples, it was necessary to enroll clients
               who collected dust samples in the field using the PaceWipe™ or a towelette that is
               similar in appearance to the PaceWipe™.
                                              16

-------
Other criteria that were used to prioritize clients for selection in this pilot study but were not absolutely
necessary for the clients to meet included the following:
       •       Submit batches of the given sample type at least monthly - Because of the desire to
               have all samples for this pilot study analyzed between scheduled rounds of the ELPAT
               Program, it was desired to enroll clients who submitted dust-wipe and/or paint chip
               samples to an NLLAP-recognized laboratory at least once per month.  This enabled
               the double-blind samples to be sent to the laboratories within one month after the clients
               received them.  While clients met this criterion during enrollment, some occasionally did
               not have the field samples to ship to a laboratory in a given month, and therefore,
               submitted only the proficiency-test samples in that round.

       •       Do not perform a double-blind procedure on their contracted laboratories, or do not
               contract with laboratories in the HUD Grantee program - Laboratories involved in
               double-blind testing either through the efforts of their clients or through their
               participation in HUD's Lead-Based Paint Hazard Control Program (HUD Grantee)
               may have been already aware that they were being evaluated via double-blind
               procedures.  This may enhance their routine performance relative to laboratories that
               participate only in single-blind proficiency testing. Therefore, higher priority was placed
               on clients who did not perform double-blind procedures or who contracted with
               laboratories that were not affiliated with the HUD Grantee program.

       •       Contract with larger laboratories - Initially, clients who contracted with NLLAP-
               recognized laboratories with high testing volumes were to have a higher priority for
               selection than clients who used exclusively small laboratories. As large laboratories
               tended to do the majority of sample analyses, this pilot study aimed to have more
               proficiency-test sample analyses performed by large laboratories compared to small
               laboratories.  However, in the final selection process, it was desired to have as many
               different laboratories involved as possible, and so this criterion was relaxed.

       •       Use centrifuge tubes for shipping dust-wipe samples to the laboratory - A dust sample
               result was reported as a total lead amount in the sample. To ensure that a laboratory
               would not cause a quantitative loss of lead or dust when removing a dust-wipe sample
               from its shipping container, clients receiving dust-wipe proficiency-test samples needed
               to ship such samples in centrifuge tubes, rather than plastic bags. As paint chip samples
               were re-weighed prior to analysis and their results reported as a percentage of lead by
               weight, use of plastic bags for shipping paint chip samples was deemed acceptable.

In addition, to protect any one laboratory from being over-represented in this study, some clients
enrolled later in the study were prioritized based on the NLLAP-recognized laboratory(ies) to which
                                              17

-------
they contract.  Clients who contracted with a laboratory that was considered sufficiently represented by
other clients enrolled earlier were occasionally given a low priority for selection, while those contracting
with a laboratory not yet represented in the study were given a high priority.

3.3    CLIENT SELECTION AND ENROLLMENT

       As mentioned in Section 3.1, 29 privately-owned laboratory clients expressed an interest in
participating in this pilot study, either through responding to the news briefer being contacted by the
recruitment staff.  Once these clients were identified, the study team used the criteria mentioned in
Section 3.2 to  select those clients who would participate in the study, determined which type(s) of
proficiency-test samples (dust-wipes, paint chips) each selected client would be submitting in the study,
and determined to which laboratory(ies) the clients would submit these samples.

       Of the 29 privately-owned clients who showed an interest in participating, 11 were not selected
for the study for the following reasons:

       •      Before receiving study participation instructions and proficiency-test samples, 3 clients
               decided that they did not want to participate.

       •      3  did not submit dust or paint samples to an NLLAP-recognized laboratory  at least
               once per month.

       •      2  used baby wipes rather than towelettes to collect dust samples and did  not collect
               paint samples.

       •      2  did not provide sufficient information to allow them to be considered.

       •      1  used baby wipes rather than towelettes to collect dust samples, but did  collect paint
               samples.  However, they contracted paint sample analysis with a laboratory which was
               already considered to have sufficient participation in this study by the time this client
               was interviewed.

The remaining 18 privately-owned clients, in addition to two state health departments, were  selected to
participate in the study. One of the privately-owned clients dropped out of the study prior to its start
(due to the need to redirect staff to other work), for a total of 19 clients.  Based on information that
these clients provided on their dust and paint sampling volume, 11 were selected to submit both dust
and paint proficiency-testing samples for analysis in this study, 2 were to submit only dust samples, and
6 were to submit only paint samples.

       Table 3-1 documents the 19 clients successfully recruited for this pilot study, the  type(s) of
proficiency-test samples that they would submit in each testing round, and the NLLAP-recognized
laboratory(ies) to which they would submit the samples. For this report, the clients are identified
                                              18

-------
according to a unique alphabetic ID, while the laboratories are identified by a unique two-digit numeric
ID. From this table, the following should be noted:

       •     The 19 clients were associated with 16 different NLLAP-recognized laboratories.

       •     7 clients were to submit samples to two different laboratories.

       •     2 laboratories were to have received samples from three different clients, and 5
              laboratories were to have received samples from two different clients.

Perhaps of most importance, as indicated by the footnotes to Table 3-1, 7 of the 19 clients (clients M
through S) decided not to participate once the  pilot study was begun, by either not providing sample
containers or not reporting back the results of the proficiency-test sample analyses in any testing round.
Therefore, only 12 clients (clients A through L) actually participated in the pilot study, representing 12
laboratories (laboratories 01 through 12). The issue of client non-response during the study is
discussed in Section 3.5.

       All but one (laboratory 08) of the 12 laboratories who analyzed proficiency-test samples in the
double-blind pilot study were NLLAP-recognized at the time that they analyzed proficiency-test
samples in Round 22 of the ELPAT Program.  Thus, they were among the 118 reference laboratories
whose results determined the target lead amounts associated with the proficiency-test samples in Round
22. For each of the three proficiency-test sample types that were included in the double-blind pilot
study, Figure 3-2 contains a bar chart of the observed distribution of the ELPAT Round 22 results for
the 118 laboratories plus laboratory  08, with results for the laboratories participating in the double-blind
pilot study highlighted in black within the bars. These charts show that while most of the ELPAT
Round 22 data for the double-blind pilot study laboratories were in the upper half of the  distribution of
reference laboratory data for each sample type, the data for the double-blind laboratories were good
representations of the reference laboratory data.  This was confirmed when statistical comparisons
were made between ELPAT Round 22 data for the double-blind laboratories and ELPAT Round 22
data for the remaining NLLAP-recognized laboratories. When both a two-sample t-test and the
Mann-Whitney nonparametric test were applied to these data, no significant differences  were observed
between these two groups of laboratories at the 0.05 level for each of the three sample types.

       Once the 19 clients in Table 3-1 were selected for the pilot study, letters of acceptance were
prepared and sent to them.  These letters were accompanied by an attachment providing
                                             19

-------
Table 3-1.    Clients Recruited to Participate in This Pilot Study, the Type(s) of
              Proficiency-Test Samples That They Would Submit in Each Testing
              Round, and the NLLAP-Recognized  Laboratory(ies) To Which They Would
              Submit the Samples
Client ID
A
B
C1
D
E
F
G
H
I
J
K
L
Number and Type of Proficiency-Test Samples
Submitted in Each DB Testing Round
2 dust samples
2 dust, 4 paint samples
2 dust samples
2 dust, 2 paint samples
2 dust, 2 paint samples
2 dust, 2 paint samples
2 dust, 2 paint samples
2 dust, 2 paint samples
2 dust samples
2 dust, 2 paint samples
2 dust, 2 paint samples
4 dust, 4 paint samples
2 dust, 2 paint samples
2 dust, 2 paint samples
4 paint samples
2 paint samples
4 paint samples
Laboratory ID
10
05
08
06
15
03
04
04
08
02
09
02
01
07
08
11
12
The following clients were recruited but did not participate once the study begun
M2
N2
O2
p3
Q2
R3
S2
2 dust, 2 paint samples
2 dust, 2 paint samples
2 dust, 2 paint samples
2 dust, 2 paint samples
4 dust, 4 paint samples
2 dust, 4 paint samples
2 paint samples
2 paint samples
2 paint samples
06
13
04
14
not determined
09
11
16
16
1 This client ended its contract with Laboratory 15 prior to the start of the study, and any samples earmarked to
be sent to Laboratory 15 were sent to Laboratory 06 instead.
2 Withdrew from the study as they did not submit sample containers to the proficiency-testing service.
3 Withdrew from the study as they did not report analytical results on the proficiency-test samples provided to
them, despite several attempts to contact them for the results.
                                             20

-------
                 Low—Spttad Duet Samp*
Trom ELRAT Round 22 ( /4j Pb)
                  Mld-SpMd Dint Swnpl*
from ELFAT Round 22 ( /zg Pb)
  A| NLLAP—
                                       fram ELPAT Rcuid 22 (96 Pb by
Figure 3-2.   Histograms of the Proficiency-Test Sample Analysis Results from Round
             22 of the ELPAT Program for 118 NLLAP-Recognized ("Reference")
             Laboratories,  with the Results for the 12 Laboratories Participating in the
             Double-Blind Pilot Study Highlighted

                                       21

-------
details on the pilot study which the participating clients needed to know (such as objectives and point of
contact), the tasks which the clients would perform, and other necessary instructions. The letters were
sent early enough to give clients sufficient time to submit sample containers to the proficiency-testing
service, so that they had sufficient time to transfer the proficiency-test samples for the first DB testing
round to these containers and to ship these samples to the clients within the specified time schedule.
The generic form of the acceptance letter and the attachment are found in Appendix C.

        Once the acceptance letters were sent, the proficiency-testing service was supplied with the
following information for each selected client:  name, address for shipping the proficiency-test samples,
and the name(s) of the laboratory(ies) to receive these samples from the client.  At this point, the
proficiency-testing service became the primary point of contact with these clients. The proficiency-
testing service recorded this information on copies of the Sample Tracking and Analysis Report Form
(Section 2.4.2).

3.4    RECRUITMENT EXPERIENCES

        The following issues and experiences arose while conducting the recruitment process of
laboratory clients in this pilot study:

"What'sin it for me?"

        When initially contacted, some laboratory clients did not recognize how a double-blind program
would allow them to have greater confidence in the accuracy of the lead levels being reported back
from the laboratories on the samples that they collect. As a result, they either did not have a desire to
participate due to the responsibilities involved, or wished to be reimbursed in some way for the resulting
expense.

Need for reimbursement

        When the recruitment effort began, it was apparent that the interest of clients to participate in
this study would increase if they were reimbursed for the analytical costs of the proficiency-test samples
that were provided to them.  While this was particularly true for small organizations who had limited
cash flow, some larger organizations stipulated that compensation for these costs was a requirement for
their participation.  Therefore, to improve the success rate of the recruitment effort, the study team
decided to adopt a policy of reimbursing the clients for costs billed to the client by the laboratory to
analyze the proficiency-test samples.  In order to receive this reimbursement, clients had to provide
copies of the laboratory's invoice, showing either the cost per sample or the number of samples
analyzed and the total analysis cost material, thereby documenting that the proficiency-test samples
were analyzed.  Clients would not be reimbursed for other costs to participate in the study, such as
shipping costs and costs associated with providing sample containers to the study.
                                              22

-------
       While the decision to reimburse clients increased the rate of interest in the study, the issue of
whether compensation is a viable part of an actual double-blind program needs to be addressed (e.g.,
what would be the source of the compensation money). In addition, providing compensation would
require a support staff, and therefore, additional resources.

Difficulty in making a contact

       It was occasionally difficult to make contact with a potential study participant based on just the
name and telephone number information. For example, frequent changes in a client's name and
telephone number from that specified in the Lead Listing sometimes made it difficult to reach some
clients.

       Recruiters were often intercepted by answering machines or voice mail when making calls.
When someone did answer the telephone, this person was not always knowledgeable on NLLAP
issues, or the most appropriate contact was occasionally not available (e.g., was in the field doing
sample collection). In these situations, recruitment staff left telephone messages, describing the study
and requesting an appropriate contact to call back. However, in many cases, the client did not call
back.

Desire to review responsibilities

       Some clients wished to have materials on the study sent to them for review prior to giving
notification that they were interested in participating. The materials prepared for faxing to these clients
included a copy of the news brief in Figure 3-1 and selected materials in the attachment to the
acceptance letter found in Appendix C.  Discussions with others at the client's headquarters were
occasionally required before a client indicated a desire to participate.

Limited work involving environmental sampling for lead

       Some clients did not feel that they could participate due to the small number of dust and/or paint
chip samples that they submit to an NLLAP-recognized laboratory  in a given month. This should not
be a major concern if the client is asked to include only two proficiency-test samples in a given batch.
In fact, there is one issue that supports having proficiency-test samples placed in small batches.  If the
proficiency-test sample results are considerably different than those for other samples within the batch
(e.g., are very high), then some laboratories may suspect that the samples are some kind of reference
material.  This is more likely to happen in large batches than in small batches, especially if the lead
content is consistent from sample to sample within a batch.

Use of an appropriate dust wipe

       While many potentially-interested clients used a pre-moistened wipe to collect dust-wipe
samples, it is important in a double-blind program that this wipe closely resemble the type of wipe used
                                              23

-------
in the proficiency-test samples. As mentioned earlier, the ELPAT Program (and, therefore, this pilot
study) uses the PaceWipe™, which resembles a small towelette like those stored in individual wrappers
and provided to customers in many fast-food restaurants. However, several clients use baby wipes
rather than the PaceWipe™ or a towelette. Baby wipes are generally larger, thicker, and have a
different consistency and aroma than the PaceWipe™ or a towelette. Therefore, to limit the likelihood
that a laboratory would distinguish a difference between the proficiency-test samples and regular field
samples on the basis of the type of wipe used, interested clients who used baby wipes were not
considered for submitting dust proficiency-test samples in this pilot study.

Need to provide materials for selected clients

       As discussed in Section 2.4.1, clients were asked to provide sample containers to ensure that
the proficiency-test samples would be placed in the same containers that clients used to store field
samples. However, two clients in this study were provided default sample containers (centrifuge tubes
that are different from the containers used in the ELPAT Program) to use in this study in order for them
to agree to participate in the study. In this instance, the clients were also supplied with a sufficient
number of empty containers for all of their regular field samples for the batch in which the proficiency-
test samples would be placed.

       In addition, to ensure participation, the study provided PaceWipes™ to one client to use in
collecting field samples that would be included with the dust proficiency-test samples.

3.5    INTERACTION WITH CLIENTS DURING THE STUDY

       During the pilot study, the proficiency-testing service was the primary point of contact with  the
participating laboratory clients. It was the proficiency-testing service's responsibility to ensure that  the
clients received their proficiency-test samples on time within each testing round, to answer any
questions that the clients had during the  study, and to collect analytical results on the proficiency-test
samples from the clients within each testing round. The following are issues that were encountered
when interacting with the clients during the course of the study.

Some clients did not provide information/materials on time or at all

       At the start of the study, some clients did not meet the prescribed deadline for submitting clean,
empty sample containers to the proficiency-testing service. During the study, some clients did not
report their results back to the proficiency-testing service in a given DB testing round within the one-
month period after receiving the proficiency-test samples. Client delays in reporting results or materials
were due to a number of reasons,  such as when clients had not yet collected a sufficient number of field
samples to accompany the proficiency-test samples within a batch, or when clients simply needed
reminding. In these situations, it was necessary for the proficiency-testing service to follow-up with the
clients to check on the status of the materials or information and reasons for delay.  As was observed in
the recruitment process, these follow-up calls frequently resulted in  a request to call back, either by
                                              24

-------
reaching voice mail or a secretary/ receptionist.  This delayed the receipt of materials or information
even more.

       As of the scheduled release of the DB first-round proficiency-test samples to the clients, six
clients had not provided the proficiency-testing service with sample containers.  (These clients were not
scheduled to receive default containers.)  As seen in Table 3-1, five of these clients never provided
containers, leading to their removal from the study.

       Despite several attempts to contact them, the proficiency-testing service was unable to
determine whether two clients submitted proficiency-test samples for analysis in any DB testing round
and that they received analytical results back from the laboratory.  Therefore, as seen in Table 3-1,
these clients were removed from the study.

       In one instance, after repeatedly trying to get results from one client for a given DB testing
round, the client finally called and read the results to the proficiency-testing service over the telephone,
rather than by providing a completed Sample Analysis and Tracking Form. However, as the sample
IDs that the client provided over the telephone did not match the IDs that the proficiency-testing service
had for these samples, these data were considered unreliable and were not used in the final analysis.

Occasional shipping of exclusively proficiency-test samples for analysis

       In situations where it was uncertain whether certain clients would be able to create a batch of
regular field samples into which the proficiency-test samples could be placed, clients were instructed to
submit the proficiency-test samples within a batch by themselves and submit the batch to the laboratory.
Here, the samples would still need to be disguised by the client as field samples by specifying fabricated
information on where and how the sample was collected in the field.

       In one situation occurring in DB Round 1, a client had no field samples to place in a batch.
Therefore, the client added blanks to the batch to help disguise the batch from containing exclusively
proficiency-test samples. However, the client asked to be reimbursed for the analysis of all samples in
the batch, including the blanks. The pilot study granted this request. The client also indicated that in
DB Rounds 2 and 3, the proficiency-test samples would be labeled as "follow-up" samples to the DB
Round 1 batch.

Non-receipt of samples

       In two instances, a client reported that they did not receive a batch of proficiency-test samples
for a given testing round. This occurred for one client in DB Round 1 and a different client in DB
Round 3.  For the DB Round 1 situation, sufficient samples were prepared (and sample containers
acquired from the clients) to allow for a second batch to be provided to the client for that round. This
required, however, an extension on the amount of time that a client had to report results back to the
proficiency-testing service.  Because the selected period for sample analysis  and reporting was nearly
                                              25

-------
over, no replacement samples were shipped to the client who reported no Round 3 samples were
received.

Reported measurement units associated with sample results

       As indicated in the instructions to the clients, the proficiency-test samples were to be
accompanied to the laboratory with fabricated information on where the sample was "collected" within
some residence.  Some clients also provided fabricated sampling areas (e.g., 0.5 ft2) associated with
the "sample." This practice caused considerable confusion on when a lead measurement being
reported on the Sample Tracking and Analysis Forms represented a total lead amount in the sample or
a lead amount per unit of sampling area. When clients specified lead  measurement as lead per unit area
rather than as total lead amount, it was necessary for the proficiency-testing service to follow-up with
these clients to verify the sample areas that were specified to the laboratory. Often, several calls were
necessary to get the needed information. Also in this pilot study, a client occasionally reported the
wrong units on the Sample Tracking and Analysis Form, such as reporting a result as micrograms when
it actually represented micrograms per square foot.  Such errors were found at the end of the study
when the proficiency-testing service obtained copies of the laboratory report forms from the clients or
laboratories to investigate the validity of certain unusual data values. Proper reporting of measurement
units is one example on how using the client as a "middleman" in reporting laboratory results can
increase the likelihood that results will be reported inaccurately to the proficiency-testing service.
                                             26

-------
4.0   DATA ANALYSIS PROCEDURES

       As presented in Section 1.3, the primary objectives of this double-blind pilot study were to

       •      Design and test a protocol for double-blind proficiency testing under the NLLAP,
              which involves evaluating logistical issues, identifying responsibilities and investigating
              minimum performance standards for laboratories

       •      Compare analytical performance of NLLAP-recognized laboratories between the
              double-blind pilot study and single-blind testing (i.e., Round 22 of the ELPAT
              Program).

Accomplishing both of these objectives required statistical analysis of the data collected during the pilot
study.  The corresponding statistical objectives were to

       •      Summarize the measured lead levels for each laboratory and across all laboratories

       •      Perform similar summaries on lead levels expressed relative to the ELPAT Round 22
              target levels (Table 2-1), and to compare a laboratory's analytical results with what the
              laboratory reported in ELPAT Round 22.

       •      Assess quantitative information (e.g., variability characterization) needed to determine
              appropriate statistical proficiency criteria for a double-blind program.

The methods used to prepare the data for analysis and the analysis methods used to accomplish the
statistical objectives are described in the sections that follow. Results of implementing these methods
through use of the SAS® System on the collected data in this study are provided in Chapter 5.

4.1    DATA ANALYSIS ENDPOINTS AND DATA HANDLING

       Once the proficiency-testing service provided the pilot study data in electronic format, these
data were placed into SAS® datasets for statistical summary and analysis.  Prior to beginning the
analysis, however, the data in these datasets were hand-checked for accuracy against the completed
Sample Tracking and Analysis Report Forms (also provided by the proficiency-testing service) as
discussed in Section 6.3, and any necessary corrections were made to the  datasets.

       As discussed in Section 6.1, a common batch of bulk source material was used to generate
each proficiency-test sample of a given type in both the double-blind pilot study and Round 22 of the
single-blind ELPAT Program.  Therefore, if the samples of a given type were homogenous and
laboratories are proficient, the measured lead amounts should be approximately the same across
samples, regardless of which laboratory performed the analysis, and should be approximately equal to
the target lead  amounts as reported in Round 22 of the ELPAT Program. Therefore, data summary
                                             27

-------
and analysis focused on the following two endpoints, whose values were determined for each
proficiency-test sample:

       •      The measured lead amount in the sample, as reported by the laboratory analyzing the
               sample and forwarded by the client who was provided the sample (expressed in
               micrograms of lead for dust-wipe samples and percent lead by weight for paint-chip
               samples).

       •      Percent recovery associated with the sample, equal to the measured lead amount
               divided by the target lead amount associated with the bulk source material, as
               determined in Round 22 of the ELPAT Program (Table 2-1). multiplied bv 100%.

These data were categorized according to testing round, client, and laboratory.

4.2    IDENTIFYING AND HANDLING STATISTICAL  OUTLIERS

       Prior to statistical summary and analysis, the data were investigated for values that appeared to
be extreme (e.g., very high or very low) compared to other values for proficiency-test samples that
were created from the same batch of bulk materials. Initially, this was done by plotting the data and
noting those data points whose values were considerably different from the others for the given sample
type. These data points were reported to the proficiency-test service, who attempted to contact the
appropriate clients to determine the cause of the extreme values (e.g., reporting errors, mislabeled
samples,  invalid analysis).  In some cases, if the client contact could not completely address the validity
of the extreme data values, the proficiency-test service contacted the laboratories (once all laboratory
testing was completed).

       While client and/or laboratory contacts occasionally resulted in corrections to certain extreme
data values,  other values were found to be legitimate as reported for purposes of this analysis.
However, their presence can overly inflate variability estimates within the statistical analyses. Therefore,
a statistical procedure was used to identify "statistical outliers," or those data values identified as
unusually high or low according to some  pre-specified statistical criteria, and the statistical summaries
and analyses were performed both with and without the statistical outliers included.  The "generalized
extreme-Studentized deviate (BSD) many-outlier" procedure documented in Rosner (1983) was
applied independently to both the untransformed  and log-transformed lead amounts for each of the
three sample types (low dust, mid dust, paint), with data for all three testing rounds combined. Within
each application of the outlier detection procedure, up to 10 outliers could be identified at an overall
significance level of 0.05 (i.e., the probability of identifying more outliers than are actually present is no
higher than 0.05).  This procedure assumes that the data have either a normal or lognormal distribution,
which has historically been observed in previous  studies.

       Section 5.1 contains those reported lead amounts that were identified as being statistical outliers
according to the procedure by Rosner (1983).
                                              28

-------
4.3    SUMMARY OF REPORTED LEAD AMOUNTS

       To begin to see how well the laboratories performed relative to the target amount and to each
other, simple summary statistics of the reported lead amounts and percent recoveries were generated.
Note that these initial summaries were meant primarily to document the reported data and to provide
preliminary, observational information on data variability and on overall data accuracy relative to target
levels from ELPAT Round 22.  Statistical comparisons to evaluate whether differences in variability
from one testing round to the next were significant were made in subsequent sections of Chapter 5,
using methods documented in Section 4.4 below.

       Initially, the data were portrayed graphically by sample type and testing round to provide initial
information on how the data varied across laboratories (due to multiple laboratories analyzing the same
type of sample) and within laboratories (due to each laboratory analyzing at least two samples of a
given type for the client). In addition, the graphs showed how lab-to-lab variation differed among  the
double-blind testing rounds as well as with the variability in data from Round 22 of the ELPAT
Program for the same laboratories and sample batches.

       The following types of tabular summaries of the above two endpoints were prepared for each
proficiency-test sample type:

       •       Summaries of individual sample results, performed by double-blind testing round across
               all samples, regardless of client or laboratory

       •       Summaries of individual sample results, performed by double-blind testing round and
               laboratory

       •       Summaries of the average sample result for a client/laboratory combination, performed
               by double-blind testing round across all such combinations.

The tabular summaries specified by the first and third bullets included the number of results, arithmetic
mean, standard deviation, minimum value, maximum value, and quartiles (i.e., 25th, 50th, and 75th
percentiles). The summaries specified by the second bullet included the number of results, arithmetic
mean, and standard error of the mean.

       For comparison purposes, the tabular summaries specified by the first bullet above also
included summaries of data from Round 22 of the ELPAT Program. Three subsets of ELPAT Round
22 data were summarized: 1) data for only those laboratories participating in the double-blind pilot
study, 2) data (non-Winsorized) for the 118 NLLAP-recognized laboratories participating in ELPAT
Round 22, and 3) data for all laboratories participating in ELPAT Round 22.  These summaries were
included to allow initial comparisons of various types between the double-blind pilot study data and
ELPAT Round  22 data for a given proficiency-test sample type.
                                             29

-------
        Tabular data summaries were generated using the UNIVARIATE procedure in the SAS®
System.  In addition, summaries of the double-blind pilot study data were calculated both including and
excluding the statistical outliers identified by the methods in Section 4.2.

4.4    CHARACTERIZING COMPONENTS OF VARIATION

        Statistical criteria for evaluating a laboratory's proficiency within a double-blind program can be
established only after characterizing the  different sources of data variation that are present in the
proficiency-test sample analysis results.  Variability that needed to be considered in the single-blind
ELPAT Program (e.g., lab-to-lab variability) must also be considered in a double-blind program.
However, a double-blind program may need to consider additional sources of variability, such as
variability due to analyzing samples originating from different clients and due to laboratories being
unaware of when a sample they are analyzing is a proficiency-test sample. Furthermore, as laboratories
may test multiple samples of a given type in a double-blind testing round, within-laboratory variability
can also be characterized.

        The initial design of the double-blind pilot study permitted up to three sources of variation in the
analytical results to be measured:

        •      lab-to-lab variation (due to the presence of multiple laboratories)

        •      client-to-client variation within a laboratory (due to multiple clients  sending samples to
               the same laboratory)

        •      within-client and laboratory variation (due to laboratories receiving at least two
               replicate proficiency-test samples  from a client within a testing round)

Furthermore, the study included multiple double-blind testing rounds to investigate how these sources of
variation contribute differently to total variability in the reported results from one testing round to
another.

       A lack of sufficient data prohibited client-to-client variation within a laboratory (i.e., the second
bullet in the previous paragraph) to be characterized appropriately within this pilot study. Therefore,
the statistical analysis only took into consideration two components of total variability in the double-
blind pilot study data: lab-to-lab variability and within-lab variability. Within-lab variability, therefore,
actually represents variability in results for different clients as well as the same client within a common
laboratory in a given testing round.

       Analysis of variance procedures were used to characterize lab-to-lab and within-lab variability
in the double-blind pilot study data for each DB testing  round. The data analyzed in these procedures
were log-transformed results for dust-wipe samples and untransformed paint sample results.  A log
transformation was made to the dust-wipe data as these data represented a total measured lead amount
                                               30

-------
in the sample.  Such measurements have been found to originate more likely from a lognormal
distribution than a normal distribution (i.e., majority of low values, with an extended right tail to
represent occasional high values). Paint sample results were not log-transformed, as these
measurements were expressed as percentages (i.e., percent lead by sample weight), and typically,
percentages do not require a logarithmic transformation to achieve assumptions of a normal distribution
more accurately.

4.4.1  The Full Statistical Model

       To characterize total variability in the double-blind pilot study data, the contribution of lab-to-
lab variability to this total variability, and how the variability changes across the testing rounds, a mixed-
effects model was developed.  This model had the following form:

                                7^ = 11 + Rj + G>t(J> + Ł                                   (1)
where Yyk is the measured lead amount (log-transformed for dust samples, untransformed for paint
       samples) for the kth sample that the ith laboratory analyzed in the jth DB testing round (j=l, 2,
       3);
       |i is the overall mean response across all samples, laboratories, and DB testing rounds;
       RJ is the (fixed) effect of the jth DB testing round;
       u i(j) is the (random) effect of the ith laboratory within the jth DB testing round; and
       •k(ij) is random error  associated with the k"1 sample analyzed by the ith laboratory in the jth
       testing round (i.e., that portion of the measured lead amount that is not explained by the model).

Furthermore, the effect u JQ was assumed to be a random variable having a normal distribution with
mean zero and variance Lj2 (j=l, 2, 3). This variance Lj2 is a measure of lab-to-lab variability within a
DB testing round and is allowed to differ across testing rounds.  The effect '^^ was assumed to be a
random variable having a normal distribution with mean zero and variance 6j2 (j=l, 2, 3). This variance
6j2 is a measure of within-lab variability within a DB  testing round and is also allowed to differ across
testing rounds.

       Therefore, within a given DB testing round, model  (1) assumes that a given measurement is
equal to an overall predicted  value (given by n+Rj), plus or minus some random amount that comes
from a normal distribution with mean zero and variance Lj2+6j2.  The variability associated with this
additional random amount has components corresponding to lab-to-lab (Lj2) variability and within-lab
(6j2) variability  measures. Model (1) is called a "full" model as the overall predicted value and the two
variance  components can differ from one DB testing  round to another.
       Model (1) was fitted, and the variances of each component estimated, using restricted
maximum likelihood routines within the MIXED procedure in the SAS® System. For each sample
type, two fittings of the model were made: one including and one excluding the statistical outliers
                                              31

-------
identified by the methods in Section 4.2. With three sample types in the double-blind pilot study, model
(1) was fitted a total of six times.

4.4.2 Reduced Forms of the  Full Statistical Model

       The key feature of model (1) was that it allowed both the lab-to-lab variability and within-lab
variability to differ from one DB testing round to another.  To determine whether lab-to-lab variability
differed significantly across DB testing rounds (and also whether within-lab variability differed
significantly across rounds), a likelihood ratio test approach was taken.  This approach involved fitting
reduced versions of model (1) to the same set of double-blind pilot study data that were used in fitting
model (1), where the reduced models did not allow one or both of the variance components to differ in
their estimates from one round to another. Three different reduced models were considered:

       1.      The lab-to-lab variability estimate did not differ from round to round, but the within-lab
               variability estimate could (i.e., 1^=1^ =Ls2, but 6j2 (j=l, 2, 3) could differ).

       2.      The within-lab variability estimate did not differ from round to round, but the lab-to-lab
               variability estimate could (i.e., 612=622 =632, but L,2 (j=l, 2, 3) could differ).

       3.      Both lab-to-lab variability and within-lab variability estimates were consistent across
               testing rounds (i.e.,  L^L^2 =1^ and 612=622 =632).

For model (1) and for each of the three reduced models, the estimate of the likelihood function was
obtained from the model fit. Then, the likelihood estimates for two specific models, where one model
was reduced from the other, were used in a likelihood ratio test to determine whether the reduced
model was associated with a statistically worse fit compared to the other. This test determined whether
the additional model parameters associated with the more complete model (e.g., different variability
estimates for random effects between testing rounds) were statistically necessary.

       To describe the likelihood ratio test approach, consider the example that tests whether or not
both Lj2=L22 =L32 and 612=622 =632 in model (1). In this test, the ratio of the likelihood for model (1)
to the likelihood for reduced model #3 above was calculated. The test statistic T, equal to -2 times the
log-transformed likelihood ratio, was calculated and compared to the 95th percentile of a chi-square
distribution with degrees of freedom equal to the difference in the number of model parameters between
the two models (4 in this example).  If T exceeded this percentile, then it was concluded that the fit
associated with the full model was significantly better (at the 0.05 level) than the fit associated with
reduced model #3 (and, therefore, that some difference in variability estimates exists either for lab-to-
lab or within-lab variability). Other likelihood ratio tests that compare model (1) with reduced models
#1 and #2 specifically test for equal variability estimates between DB testing rounds for lab-to-lab
variability and within-lab variability, respectively.
                                               32

-------
       As with model (1), the likelihood ratio approach was applied separately by proficiency-test
sample type and whether or not statistical outliers (as identified through the procedures documented in
Section 4.2) were included in the analysis.

4.4.3  Comparisons of Lab-to-Lab Variability Between the
       Pilot Study and ELPAT Round 22

       While the likelihood ratio test approach discussed in the previous subsection tested for
significant differences in lab-to-lab variability and in within-lab variability between the three DB testing
rounds, it was also of interest to test whether lab-to-lab variability in the DB pilot study differed
significantly from the variability across laboratories in Round 22 of the ELPAT Program for the same
proficiency-test sample type.  This objective could not be addressed by adding ELPAT Round 22 data
to the double-blind data in the above statistical modeling analyses, as each laboratory analyzed only one
proficiency-test sample of a given type in ELPAT Round 22, thereby preventing within-lab variability
from being estimated in the statistical modeling procedures for ELPAT Round 22.  Instead, the
following approach was performed (on log-transformed lead amounts in dust-wipe samples and on
untransformed paint-lead measures given as percent lead by weight):

       1.      Arithmetic averages were calculated of the reported results for each laboratory within
              each DB testing round (ignoring whether the results were associated with different
              clients).

       2.      ELPAT Round 22 results were identified for the same group of laboratories.

       3.      Levene' s test of homogeneity of variance (Snedecor and Cochran, 1989) was applied
              to the combined set of measures in #1  and #2 to test whether variability in these
              measures differed between the four testing rounds (i.e., the three DB testing rounds and
              ELPAT Round 22).

Because a laboratory's result from ELPAT Round 22 was for a single sample, and its result in a given
DB testing round was an average of multiple samples (where the number of samples can differ from one
laboratory to another), the data used in this analysis for a given laboratory were not generated from a
source having equal variability across all four testing rounds.  This is a statistical concern that must be
recognized when interpreting the results of this analysis.

       The analysis was applied separately to each proficiency-test sample type and whether or not
statistical outliers from the double-blind study (as identified through the procedures documented in
Section 4.2) were included. Results of this analysis are presented in Section 5.4.

       A  similar application of Levene's test of homogeneity of variance was performed on the
individual  sample results from the three double-blind testing rounds and from Round  22 of the ELPAT
Program for the laboratories participating in the double-blind pilot study, in order to evaluate
                                              33

-------
differences in total variability of individual sample results across these four testing rounds.  In this
application, results in each testing round were analyzed without regard to the laboratory which
generated them, in order to obtain a general idea of whether double-blind testing data have a different
underlying variability compared to single-blind data from the ELPAT Program. Note that this is an
approximate test, as the laboratory effect on these data within a testing round was ignored.  Results of
this statistical test are presented in Section 5.2.

4.4.4  Presenting Results of Statistical Analysis

        Estimates of lab-to-lab variability and within-lab variability obtained from applying the above
statistical modeling procedures are presented within bar charts and tables.  Also presented is the
percentage that each variance component contributes to total variability (i.e., their sum).  Results are
presented for each DB testing round, as determined from fitting model (1) in Section 4.4.2 above. In
addition, if the likelihood ratio tests described in Section 4.4.3 above determined that the estimates for a
specific variance component did not differ significantly across DB testing rounds, the summaries for this
component are also presented across the entire pilot study.
                                               34

-------
5.0   RESULTS

       This section reports the results of the proficiency-test sample analyses performed in this double-
blind pilot study.  Methods discussed in Chapter 4 were used to summarize the data and to conduct
statistical modeling to characterize various sources of total variability in the reported data.  All
references to clients and laboratories in this chapter are made according to the identification protocol
introduced in Table 3-1  of Section 3.3.

       As discussed in Section 3.3, a total of 12 clients (clients A through L) reported analytical results
in this pilot study. These results were obtained from 12 different NLLAP-recognized laboratories: 10
laboratories reporting dust sample results (laboratories 01 through 10), and 11 laboratories reporting
paint sample results (laboratories 01  through 09, 11, and 12). Note that nine of the laboratories
analyzed both dust and paint samples in this study.  Of the 12 laboratories providing analytical results,

       •      7 laboratories (01, 03, 05, 06,  07, 09,  10) each received dust samples from one client
       •      3 laboratories (02, 04, 08) each received dust samples from two clients
       •      9 laboratories (01, 03, 05, 06, 07, 08, 09, 11, 12) each received paint samples from
               one client
       •      2 laboratories (02, 04) each received paint samples from two clients

As mentioned in Section 4.1, this chapter considers the following types of results:

       •      The measured lead amount in the sample, as reported by the laboratory analyzing the
               sample and forwarded by the client who was provided the sample (expressed in
               micrograms of lead for dust samples and percent lead by weight for paint samples).

       •      Percent recovery associated with the sample, equal to the measured lead amount
               divided by the target lead amount associated with the bulk source material, as
               determined in Round 22 of the ELPAT Program (and provided in Table 2-1 of Chapter
               2), multiplied by 100%.

5.1    STUDY DATA

5.1.1  Data Corrections

       In this pilot study, analytical results associated with the double-blind proficiency-test samples
were occasionally reported incorrectly by either the client or the laboratory. When a laboratory found
it necessary to correct a result reported earlier to the client but did not identify the error on its own, the
original incorrect result was used in the data analyses.  This situation occurred with the four dust sample
results in DB Round 1 for  Client I/Laboratory  07. The laboratory did not consider the dilution factor
when originally reporting lead amounts for these samples, and therefore, the results were unusually low
relative to the target amount.  However, while the laboratory reported corrected values back to the
                                              35

-------
proficiency-testing service at the end of the pilot study after the proficiency-testing service inquired
about these results, the original incorrect values were used in the analysis rather than the corrected
results.  This is because the laboratory would not have corrected these values without the proficiency-
testing service's inquiries.

       When evidence existed that a client reported results incorrectly relative to the laboratory
reports, these results were corrected prior to performing data summaries and analyses.  For example,
as discussed in Section 3.5, clients occasionally reported the units of analytical results incorrectly on the
Sample Tracking and Analysis Report Forms, thereby requiring an adjustment to the affected values.
The proficiency-testing service identified such errors upon contacting the clients about unusually high or
low results, and the  correct results were used in the analysis. However, when evidence did not exist
that a given data value was reported incorrectly by the client, the data value was not corrected.

5.1.2 Plots of the Data Values

       Figures 5-1  through 5-3 contain plots of measured lead amounts for low-dust, mid-dust, and
paint proficiency-test samples, respectively, as reported by the laboratories and their participating
clients and after any necessary data corrections were made. Each figure contains three plots, one for
each testing round.  The horizontal axis of each plot indicates the client/laboratory combination
associated with a set of measurements (client ID, followed by lab ID). Each plot includes three
horizontal reference  lines, with the middle line corresponding to the target lead amount as determined in
Round 22 of the ELPAT Program and  the other two lines corresponding to the lower and upper limits
of the acceptance regions for this ELPAT round. (These numbers were given in Table  2-1 in Chapter
2).  The measured lead amounts for proficiency-test samples in the double-blind pilot study are
represented by asterisks in these plots, while a laboratory's measured lead amount for the proficiency-
test sample it analyzed in Round 22 of the ELPAT Program (a sample produced from the same bulk
source material as the double-blind pilot samples) is represented by an open circle. Note that the same
ELPAT Round 22 result is portrayed in each set of data corresponding to different clients of the same
laboratory.

       The vertical axis ranges in Figures 5-1 through 5-3 were selected to allow the variability in
results to be seen clearly across the plots. Occasionally, the vertical axis range could not include some
very large or very small data values without considerably distorting the plot.  Those few results that fell
outside of the axis ranges are plotted at the upper (or lower) range limit by an up (or down) arrow and
the actual data value(s) in parentheses.

       All of the ELPAT Round 22 results plotted in Figures 5-1 through 5-3 were within the
acceptance limits determined for that round (i.e., the upper and lower dashed lines in the plots).
However, 29 of the  257 double-blind proficiency-test sample results (11%) were outside of these
limits. Of these 29 results, 10 were for low-spiked dust-wipe samples from four of the  ten
                                              36

-------
                                    CVDH  Q*» Q/D4 &O* F/DB Q/D2
                     A«JB AHQ
                                                                                V07
Figure 5-1.
Measured Lead Amounts (|jg) in the Low-Spiked Dust Proficiency-Test
Samples, by Double-Blind Testing Round
(Double-blind results are plotted with asterisks, and results for ELPAT Round 22 are plotted with
open circles.  Extreme values outside of the vertical axis limits are specified by an arrow and the
values in parentheses. The target lead amount (129 ug) and the lower and upper acceptance
limits in ELPAT Round 22 are plotted as horizontal dashed lines.)
                                             37

-------
    3BQJ1G
    3BOJDO
    ZBQJH
a
    aoojjo
    1SQJX)
      DJJO

0
0
* ° f
*
*

1 Round 1 I
* *
• * *
* o o o*0
* * * *
» _ _ * n
4k
0
I
(S.O, S.*)*
                                    Q/O4 QD4 F/DB Q/D2 Q/CB H/CB

                                     C|M1t/UdO P
    200JO
    1OQJX1
                                                              i-??"!!?-'-!--
                                                              0        °
                                                             	o	
                   { 1 3O.O)
           A/OE >S/1O
                                    D/O4 BO4 F/D8
                                     dont/Lab P
                                                                  POI
    aaujio
    eaojn
    eoojw
    laojoo
                                                                Round 3
                                         nfi-
           /VOB AHQ
                          cyan a/os 0/04 e/a*
                                                   ca/ne
                                                             I-VDB
                                                                       VD7
Figure 5-2.
    Measured Lead Amounts (|jg) in the Mid-Spiked Dust Proficiency-Test
    Samples, by Double-Blind Testing Round
    (Double-blind results are plotted with asterisks, and results for ELPAT Round 22 are plotted with
    open circles.  Extreme values outside of the vertical axis limits are specified by an arrow and the
    values in parentheses. The target lead amount (272 ug) and the lower and upper acceptance
    limits in ELPAT Round 22 are plotted as horizontal dashed lines.)
                                    38

-------
          I
                                                                       FtDLnd 1
              a.Tao	:r	
          -"=•                        *        *                       4

          *•  a'7aa   *                      °        °   i         O
              fojBBO-  O             O   Q             S   ®
              Qfl4o._;_	o	*_	*	_s___*___o	
                               Sw
              0.000 -                               *                      *

          ff  O.SBQ -
          •a                                      o                           9
          •B  O.SBO -                                                            "if

              0.4BO-                          *                           *
                    A/OB  A/1O  B/DB  CVDB  ryn« CVD4 &O4 FVDB  O/D2


              °*4°"                       (1.09,2.27)^.   f.(D,S17}         - pjg,^^ g

          |  Oim
          _g>  O.7BO

          Ł  °i7aoi                          oo**
              OyHBO-  O        A    O   O                  Q         X
                                    *   *   *                  O        A
              0.040-1                 ~~                                 «    _

              OjBDO


              T                               o             *             t
              n^gm


                              B/DB  CVD8  OfOS OIM &O4 F/QB  C3/OB  CVOB
                                             dent/Lob E>

                                          (1 .*a. 1.4S) f                  | Round 3 ~|

          f  Ofl00'
          _gw  O.7HO

          ^  0.720
                                                                      O
              OJOBO^  es   A            o                  5    o
                                                                 _       a_i

          „  oj«H                                              *    w

          ff  O^BD .
          ,       I
                                       o/oa
                                                           GUIK  a/oe  I-VDB  IDI   VD7
Figure 5-3.
Measured Lead Amounts (% by weight) in the Spiked Paint Proficiency-
Test Samples, by Double-Blind Testing Round
(Double-blind results are plotted with asterisks, and results for ELPAT Round 22 are plotted with
open circles.  Extreme values outside of the vertical axis limits are specified by an arrow and the
values in parentheses. The target lead amount (0.6454%) and the lower and upper acceptance
limits in ELPAT Round 22 are plotted as horizontal dashed lines.)
                                             39

-------
laboratories analyzing dust-wipe samples, 8 were for mid-spiked dust-wipe samples from four of the
ten laboratories analyzing dust-wipe samples, and 11 were for paint chip samples from three of the ten
laboratories analyzing paint chip samples.  This suggests that for some (but not necessarily all) of the
participating laboratories and for each sample type, the double-blind pilot study data are more likely
than single-blind study data to exceed the acceptance limits determined from data within the (single-
blind) ELPAT Program.

       Figures 5-4 through  5-6 contain plots of the percent recoveries (relative to the target lead
amount from Round 22 of the ELPAT Program) associated with the low-dust, mid-dust, and paint
proficiency-test samples, respectively. The horizontal reference lines in each plot correspond to a
percent recovery of 100% and to the ELPAT Round 22 acceptance limits, equal to the acceptance
limits plotted in Figures  5-1 through 5-3, respectively, divided by the corresponding target values.

       Details on the statistical characterization of the observed levels of variability in these plots and
how this variability differs statistically across testing rounds are provided in Sections 5.2 through 5.4.

5.1.3 Identifying  Statistical Outliers and Data Exceeding
       Various Types of  Limits

       Figures 5-1 through 5-3 show that some data values are very high or very low relative to the
other measurements of that sample type (e.g., those values denoted by arrows).  Such measurements
can greatly influence the results of characterizing variability in the pilot data.  To identify those
measurements which can be labeled as statistical outliers, the  methods of Section 4.2 were applied to
all data for each sample type separately. The method could identify up to ten outliers (either high or
low) for each sample type.

       Table 5-1 contains a list of those data values labeled as statistical outliers by the methods of
Section 4.2. These values correspond exactly to those represented by arrows in Figures 5-1 through
5-3 (i.e., were outside of the vertical axis ranges in these plots).  The dust sample outliers included the
four results from Client I/Laboratory 07 where the dilution factor was not taken into account and two
results from Client A/Laboratory 10 whose samples were suspected of having their ID labels  switched
(although no evidence existed that this actually occurred).

       Five of the six paint sample outliers in Table 5-1 were results from Client G/Laboratory 09;
these results were as the laboratory reported to the client. Note that all but one of the six paint sample
results for Client G/Laboratory 09 were identified as  statistical outliers. Client G also sent paint
samples to Laboratory 02, but the results for those samples were consistent with other client/laboratory
pairs (Figure  5-3).  However, as no other clients sent paint samples to Laboratory 09, the extent to
which the highly variable results of Laboratory 09 would occur for other clients could not be observed.
Note from Figure 5-3 that Laboratory 09 was slightly above the lower limit of the acceptance range for
paint samples of the same bulk source material in Round 22 of the ELPAT Program.
                                              40

-------
          1
16O;
140
13D-
120
BO
00
TO
BJ

1bU
140
130
120
BO
BO
TO'
BO

1BO
14O
130
1ZO
tlQ
00
00
70
BO


* 0
o 3
*
*
«

fi/OS CyOO D/QS D/Q4-
t

O * O
* o o
*
*

AA6 CM» DV03 EMX-


e o
•*
•* *
*
| Round 1 |
*
* * *
*
* 6 O O
* *
o * o o
* * *
*
<2.O. 2.1) *•
E/CM Qfte ^00 HAH VDI to? *lfd& Kfl1 L/lz
demA^ p
I Round 2 I
*
*
^ O *
A- ft O
0 * ° o
*
*
*

EAM GI/O2 GKJ9 HVD2 |OI IDT JWB Kfl1 U12
[Round 3 1

* *
009 *
g * * *
O o
*

A/OB  tyoa a/aa
O/be
HJ7
                                                                          K/n  L/IE
Figure 5-4.
                                        H/DE  |bi
                                        ID
Measured Percent Recoveries in the Low-Spiked Dust Proficiency-Test
Samples, by Double-Blind Testing Round
(Double-blind results are plotted with asterisks, and results for ELPAT Round 22 are plotted with
open circles. Extreme values outside of the vertical axis limits are specified by an arrow and the
values in parentheses. The percent recovery of 1 00% and the lower and upper acceptance
limits of 73% and 127% in ELPAT Round 22 are plotted as horizontal dashed lines.)
                                             41

-------
               16O ,
                                                                          Round 1
               140

               13D


               **<                            *               *
                                                   *   o   o         o   *
                                                   *              *    *    *
               1001	~=	O-—-f*	A	*	            ~

               BO

               oo'                  *    *

               TO

               «H                                                         
-------
1BD

140

13D

120
                             100 -

                              BO

                              00

                              70

                              60
                                                                                        Round 1
5
                                    *

                                   _JO_
A/09
DAM
                              QAJ2
                                         H/O2  KM
                                         p
                                                                                          Kfl1   L/12
                             140

                             130

                             120



                             100

                              n

                              BO

                              TO
                                                                                        RBund2
                                          *
                                         -*-.

                                          O
                                  MX  cm oaa EMX-  EAM 0/02  aw  HAS  101  IDT
                                                                                          K11   U12
                             1BO

                             14O

                             130
                         I
•no

100-|—

 00

 00

 70
                          <220.0. 22B.7) f
                          8
                         "*"
                               0         "
                               ^___0	

                                     J    *
A/OB tyoa a/aa
               O/be
                                                   HJ7
                                                                                                L/IE
               Figure 5-6.
                                         H/DE  |bi
                                         ID
Measured Percent Recoveries in the Spiked Paint Proficiency-Test
Samples, by Double-Blind Testing Round
(Double-blind results are plotted with asterisks, and results for ELPAT Round 22 are plotted with
open circles. Extreme values outside of the vertical axis limits are specified by an arrow and the
values in parentheses.  The percent recovery of 1 00% and the lower and upper acceptance
limits of 82% and 118% in ELPAT Round 22 are plotted as horizontal dashed lines.)
                                                             43

-------
Table 5-1.     Double-Blind Pilot Proficiency-Test Sample Results Identified as Outliers
               by Applying the Method of Rosner (1983) on All Results by Sample
               Type1
Client/Lab
Testing Round
Reported Lead
Amount
Low/High
Outlier
% of Target
Value2
Low-Spiked Dust Samples (n = 82)
1/07
1/07
A/10
1
1
2
2.60/yg
2.70/yg
242. 5 /yg
Low
Low
High
2.0%
2.1%
188.0%
Mid-Spiked Dust Samples (n = 82)
I/07
I/07
A/10
1
1
2
5.00/yg
5.40/yg
130.0/yg
Low
Low
Low
1.8%
2.0%
47.8%
Spiked Paint Samples (n = 93)
G/09
H/02
G/09
G/09
G/09
G/09
1
2
3
3
2
2
0.073%
0.917%
1 .42%
1.45%
1.69%
2.27%
Low
High
High
High
High
High
11.3%
142.1%
220.0%
224.7%
261.9%
351.7%
1 The method identifies up to 10 outliers for each sample type at a significance level of 0.05 (i.e., the probability of
identifying more outliers than are actually present is no higher than 0.05) and assumes a normal distribution.
2 Target lead amounts are 1 29 /jg for low-dust samples, 272 /jg for mid-dust samples, and 0.6454% lead by weight
for paint samples.



       As a first step to identifying those pilot study data values in Figures 5-1 through 5-6 that would

exceed an appropriate acceptance range for a double-blind proficiency-testing program, these data
values were compared to a series of ranges that resembled the acceptance limits calculated in Round
22 of the ELPAT Program. These ranges, calculated for each sample type, were as follows:


       •     the actual acceptance range from Round 22 of the ELPAT Program (documented in

              Table 2-1 of Chapter 2; the method used to calculate this range is given in Section 2.2)


       •     a range equal to ±27% (for dust) or ±18.5% (for paint) of the Winsorized mean of the

              double-blind pilot study data in a given testing round (corresponding to the sizes of the
              acceptance ranges calculated in Round 22 of the ELPAT Program)
                                              44

-------
       •      a range obtained by applying the ELPAT Program method of calculating acceptance
              ranges on the double-blind pilot study data of a given testing round (i.e., ±3 standard
              deviations of the mean, calculated after Winsorizing the data)

The latter two ranges were calculated by double-blind testing round and by whether or not the
statistical outliers in Table 5-1 were included.  Note that these ranges do not necessarily represent
appropriate acceptance ranges within a double-blind program, but were considered simply to
investigate how the double-blind pilot data may compare to various types of acceptance criteria that are
considered in single-blind proficiency testing.

       Those double-blind pilot study data values that exceeded at least one of the above ranges are
documented in Tables 5-2a through 5-2c for the low-spiked dust, mid-spiked dust, and paint samples,
respectively. According to column 4 of Table 5-2a (and as mentioned in Section 5.1.2), 10 of the 82
low-spiked dust sample results exceeded the acceptance range from Round 22 of the ELPAT Program
(94-164 |ig). Also, 8 of the 82 mid-spiked dust sample results  (Table 5-2b) and 11 of the 93 paint
sample results (Table 5-2c) exceeded their corresponding acceptance ranges from Round 22 of the
ELPAT Program (199-345 jig and 0.5264-0.7645%, respectively). Thus, as discussed in Section
5.1.2, the double-blind pilot study data were more likely than the single-blind data from the ELPAT
Program to exceed the acceptance ranges determined from data collected in the (single-blind) ELPAT
Program. Note that the inclusion or exclusion of outliers in determining the ranges found in the last two
bullets above had only a minor effect on whether non-outliers exceeded these ranges, as the ranges
were calculated from Winsorized data. When the methods used to determine acceptance ranges in the
ELPAT Program were applied to the double-blind pilot study data, only data values that were among
the statistical outliers in Table 5-1 exceeded these ranges.

       If the amount of lead spiked into a dust-wipe sample is known, Page 5-33  of the HUD
Guidelines (USHUD, 1995) specifies that in a double-blind setting, the laboratory results for these
spiked dust-wipe samples should be within 20% of the spiking level. This is more stringent than what
was used in ELPAT Round 22 (20% versus 27%).  Twenty-seven of 82 low-spiked dust sample
results (32.9%) and 17 of 82 mid-spiked dust sample results (20.7%) are more than 20% beyond their
respective ELPAT Round 22 target levels.  These are considerably more dust sample results than the
numbers indicated in the previous paragraph that exceed 27% of the target level (i.e., the ELPAT
Round 22 criteria).
                                             45

-------
Table 5-2a.   List of Reported Lead Amounts in Low-Spiked Dust Proficiency-Test
               Samples That Exceed At Least One of the Various Acceptance Ranges
               Determined from Methods Used in Round 22 of the ELPAT Program
Client/ Lab
Reported Lead
Amount (fjg)
Is Result
Above or
Below the
Mean?
Outside of
ELPAT Round
22 Acceptance
Range?1
All Low-Spiked Dust Sample Data
Considered
Exceeds
±27% of
Winsorized
Mean of DB
Data?2
Outside of
Acceptance
Range as
Determined by
ELPAT
Methods?3
Low-Spiked Dust Sample Data
with Outliers Removed
(see Table 5-1)
Exceeds
±27% of
Winsorized
Mean of DB
Data?2
Outside of
Acceptance
Range as
Determined by
ELPAT
Methods?3
Double-Blind Round 1 (n = 28)
I/07
I/07
H/02
G/02
G/09
G/09
2.60
2.70
156.9
158.1
160.0
173.0
Below
Below
Above
Above
Above
Above
•
•



•
•
•
•
•
•
•
•
•




Outlier removed
Outlier removed



•




Double-Blind Round 2 (n = 30)
C/06
G/09
A/10
93.0
168.0
242.5
Below
Above
Above
•
•
•
•
•
•


•
•
•


Outlier removed
Double-Blind Round 3 (n = 24)
C/06
C/06
C/06
G/09
83.0
83.0
92.0
184.0
Below
Below
Below
Above
•
•
•
•
•
•
•
•




•
•
•
•




1 Acceptance range in ELPAT Round 22 for these samples was (94 ^g, 164^g), or ±3 standard deviations from the mean (as
determined from the Winsorized data of 11 8 reference laboratories). This acceptance range corresponds to ± 27% of the mean.
2 The observed size of the acceptance range in ELPAT Round 22 for samples generated from this sample batch.
3 The acceptance range is determined by Winsorizing the lowest and highest 5% of the sample results in the given round, calculating
the mean and standard deviation of the Winsorized data, and taking ±3 standard deviations of the mean.
                                                46

-------
Table 5-2b.   List of Reported  Lead Amounts in Mid-Spiked Dust Proficiency-Test
               Samples That Exceed At Least One of the Various Acceptance Ranges
               Determined from Methods Used  in Round  22 of the ELPAT Program
Client/ Lab
Reported Lead
Amount (fjg)
Is Result
Above or
Below the
Mean?
Outside of
ELPAT Round
22 Acceptance
Range?1
All Mid-Spiked Dust Sample Data
Considered
Exceeds
±27% of
Winsorized
Mean of DB
Data?2
Outside of
Acceptance
Range as
Determined by
ELPAT
Methods?3
Mid-Spiked Dust Sample Data
with Outliers Removed
(see Table 5-1)
Exceeds
±27% of
Winsorized
Mean of DB
Data?2
Outside of
Acceptance
Range as
Determined by
ELPAT
Methods?3
Double-Blind Round 1 (n = 28)
I/07
I/07
5.00
5.40
Below
Below
•
•
•
•
•
•
Outlier removed
Outlier removed
Double-Blind Round 2 (n = 30)
A/10
I/07
C/06
C/06
130.0
170.0
194.0
199.0
Below
Below
Below
Below
•
•
•

•
•
•
•
•



Outlier removed
•
•
•



Double-Blind Round 3 (n = 24)
C/06
C/06
G/02
189.0
192.0
350.0
Below
Below
Above
•
•
•
•
•




•
•




1 Acceptance range in ELPAT Round 22 for these samples was (199 /jg, 345 ^g), or ±3 standard deviations from the mean (as
determined from the Winsorized data of 11 8 reference laboratories).  This acceptance range corresponds to ± 27% of the mean.
2 The observed size of the acceptance range in ELPAT Round 22 for samples generated from this sample batch.
3 The acceptance range is determined by Winsorizing the lowest and highest 5% of the sample results in the given round, calculating
the mean and standard deviation of the Winsorized data, and taking ±3 standard deviations of the mean.
                                                47

-------
Table 5-2c.    List of Reported Lead Amounts in Spiked Paint Proficiency-Test Samples
                That Exceed At Least One of the Various Acceptance  Ranges  Determined
                from Methods Used in Round 22 of the ELPAT Program
Client/ Lab
Reported Lead
Amount (%
lead by
weight)
Is Result
Above or
Below the
Mean?
Outside of
ELPAT Round
22 Acceptance
Range?1
All Spiked Paint Sample Data
Considered
Exceeds
±18. 5% of
Winsorized
Mean of DB
Data?2
Outside of
Acceptance
Range as
Determined by
ELPAT
Methods?3
Spiked Paint Sample Data with
Outliers Removed
(see Table 5-1)
Exceeds
± 18.5% of
Winsorized
Mean of DB
Data?2
Outside of
Acceptance
Range as
Determined by
ELPAT
Methods?3
Double-Blind Round 1 (n = 32)
G/09
L/12
L/12
G/02
0.073
0.493
0.519
0.779
Below
Below
Below
Above
•
•
•
•
•
•
•
•
•



Outlier removed
•
•
•



Double-Blind Round 2 (n = 32)
L/12
L/12
I/07
L/12
H/02
H/02
H/02
G/09
G/09
0.539
0.547
0.550
0.554
0.792
0.804
0.917
1.69
2.27
Below
Below
Below
Below
Above
Above
Above
Above
Above




•
•
•
•
•
•
•
•
•


•
•
•







•
•




•
•






Outlier removed
Outlier removed
Outlier removed
Double-Blind Round 3 (n = 29)
G/09
G/09
1.42
1.45
Above
Above
•
•
•
•
•
•
Outlier removed
Outlier removed
1 Acceptance range in ELPAT Round 22 for these samples was (0.5264%, 0.7645%), or ±3 standard deviations from the mean (as
determined from the Winsorized data of 11 8 reference laboratories). This acceptance range corresponds to ± 18.5% of the mean.
2 The observed size of the acceptance range in ELPAT Round 22 for samples generated from this sample batch.
3 The acceptance range is determined by Winsorizing the lowest and highest 5% of the sample results in the given round, calculating
the mean and standard deviation of the Winsorized data, and taking ±3 standard deviations of the mean.
                                                 48

-------
5.2    SUMMARY OF REPORTED LEAD AMOUNTS

       This section presents numerical summaries of the data plotted in Figures 5-1 through 5-6. As
discussed in Section 4.1, three types of summaries were prepared and presented in tables:

       •     Individual sample results, summarized across all samples, regardless of client or
              laboratory

       •     Individual sample results, summarized by laboratory

       •     Averages of individual sample results within each client/laboratory combination,
              summarized across all such combinations.

Summaries across all samples by testing round

       Measured lead amounts are summarized across all samples, by double-blind testing round, in
Tables 5-3a (low-spiked dust), 5-4a (mid-spiked dust), and 5-5a (paint). The corresponding
summaries for percent recovery are found in Tables 5-3b (low-spiked dust), 5-4b (mid-spiked dust),
and 5-5b (paint).   Summaries are presented both with and without the outliers identified in Table 5-1.
Note that the double-blind pilot data summaries in these tables do not consider that some samples were
analyzed by the same laboratories but for different clients, while other samples were analyzed by
different laboratories.

       Also included in Tables 5-3a through 5-5b are data summaries for Round 22 of the ELPAT
Program for the same batch of proficiency-test samples, calculated across the following three groups of
laboratories: 1) all laboratories participating in ELPAT Round 22, 2) only those laboratories which
were NLLAP-recognized (and which acted as "reference laboratories"), and 3) only those laboratories
analyzing samples within the double-blind pilot study.

       Tables 5-3a through 5-5b show that average percent recoveries for the three sample types and
three DB testing rounds ranged from 96.2% to  116.1% when outliers were  included, and from 99.9%
to 105% when outliers were removed.  While excluding the outliers tended to result in mean percent
recoveries closer to 100%, this was not the case for mid-dust samples (Table 5-4b), where the percent
recoveries in DB Rounds 1  and 2 (in which outliers were identified) averaged higher and slightly farther
away from  100% than when the outliers were included. For dust samples, slightly better performance
on average was observed in DB Rounds 2 and 3 compared to DB Round 1. Also, when outliers were
included, average percent recoveries for paint samples in DB Rounds 2 and 3 were high (Table 5-5b).
In general, average percent recoveries for the double-blind testing rounds compared favorably with
those reported in ELPAT Round 22.
                                             49

-------
Table 5-3a.   Descriptive Statistics of Reported Lead Amounts (//g) in Low-Spiked
              Dust Proficiency-Test Samples, Calculated Across Participating Clients
              and Their Laboratories, by Testing Round
Testing Round
#
Samples
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Low-Spiked Dust Sample Data
DB Round 1
DB Round 2
DB Round 3
28
30
24
124.1
132.7
129.4
39.4
29.8
24.5
2.6
93.0
83.0
1 1 1.7
115.0
1 14.0
137.0
131.8
135.9
147.2
145.0
141.9
173.0
242.5
184.0
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
26
29
24
133.4
128.9
129.4
20.1
21.8
24.5
100.0
93.0
83.0
1 15.4
115.0
1 14.0
138.3
131.0
135.9
147.5
144.0
141.9
173.0
168.0
184.0
ELPAT Round 22 Results for This Sample Batch
DB labs only1
All NLLAP-recognized labs2
All ELPAT Round 22 labs
10
118
303
135.5
127.9
125.9
10.1
17.4
18.1
122.4
2.6
1.3
124.0
120.1
1 18.7
135.6
129.0
127.0
145.0
138.0
136.0
147.2
157.0
180.0
1 ELPAT Round 22 data for only those laboratories analyzing the given sample type in the double-blind pilot study.
2 Data are NOT Winsorized.

Table 5-3b.   Descriptive Statistics of Reported Percent  Recoveries in Low-Spiked Dust
              Proficiency-Test Samples, Calculated Across Participating Clients and
              Their Laboratories, by Testing Round1
Testing Round
#
Samples
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Low-Spiked Dust Sample Data
DB Round 1
DB Round 2
DB Round 3
28
30
24
96.2
102.9
100.3
30.5
23.1
19.0
2.0
72.1
64.3
86.6
89.1
88.4
106.2
102.2
105.3
114.1
112.4
110.0
134.1
188.0
142.6
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
26
29
24
103.4
99.9
100.3
15.6
16.9
19.0
77.5
72.1
64.3
89.4
89.1
88.4
107.2
101.6
105.3
114.3
111.6
110.0
134.1
130.2
142.6
ELPAT Round 22 Results for This Sample Batch
DB labs only2
All NLLAP-recognized labs3
All ELPAT Round 22 labs
10
118
303
105.0
99.1
97.6
7.9
13.5
14.0
94.9
2.0
1.0
96.1
93.1
92.0
105.1
100.0
98.4
112.4
107.0
105.4
114.1
121.7
139.5
1 Percent recoveries are calculated relative to the target of 1 29 /jg Pb.
2 ELPAT Round 22 data for only those laboratories analyzing the given sample type in the double-blind pilot study.
3 Data are NOT Winsorized.
                                            50

-------
Table 5-4a.   Descriptive Statistics of Reported Lead Amounts (//g) in Mid-Spiked Dust
              Proficiency-Test Samples, Calculated Across Participating Clients and
              Their Laboratories, by Testing Round
Testing Round
#
Samples
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Mid-Spiked Dust Sample Data
DB Round 1
DB Round 2
DB Round 3
28
30
24
265.6
276.2
282.1
81.6
50.6
43.7
5.0
130.0
189.0
240.0
250.0
267.7
287.5
292.4
284.8
318.1
310.0
312.1
331.0
335.0
350.0
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
26
29
24
285.6
281.3
282.1
36.8
43.1
43.7
219.0
170.0
189.0
260.0
268.0
267.7
289.0
293.0
284.8
318.3
310.0
312.1
331.0
335.0
350.0
ELPAT Round 22 Results for This Sample Batch
DB labs only1
All NLLAP-recognized labs2
All ELPAT Round 22 labs
10
118
303
284.2
294.5
276.4
19.8
270.2
170.9
242.3
5.7
2.6
273.0
257.0
251.0
288.0
273.8
271.5
299.3
291.1
288.0
304.9
3180
3180
1 ELPAT Round 22 data for only those laboratories analyzing the given sample type in the double-blind pilot study.
2 Data are NOT Winsorized.

Table 5-4b.   Descriptive Statistics of Reported Percent Recoveries in Mid-Spiked Dust
              Proficiency-Test Samples, Calculated Across Participating Clients and
              Their Laboratories, by Testing Round1
Testing Round
#
Samples
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Mid-Spiked Dust Sample Data
DB Round 1
DB Round 2
DB Round 3
28
30
24
97.7
101.6
103.7
30.0
18.6
16.1
1.8
47.8
69.5
88.2
91.9
98.4
105.7
107.5
104.7
116.9
114.0
114.7
121.7
123.2
128.7
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
26
29
24
105.0
103.4
103.7
13.5
15.9
16.1
80.5
62.5
69.5
95.6
98.5
98.4
106.3
107.7
104.7
117.0
114.0
114.7
121.7
123.2
128.7
ELPAT Round 22 Results for This Sample Batch
DB labs only2
All NLLAP-recognized labs3
All ELPAT Round 22 labs
10
118
303
104.5
108.3
101.6
7.3
99.3
62.8
89.1
2.1
1.0
100.4
94.5
92.3
105.9
100.7
99.8
110.0
107.0
105.9
112.1
1169
1169
1 Percent recoveries are calculated relative to the target of 272 /jg Pb.
2 ELPAT Round 22 data for only those laboratories analyzing the given sample type in the double-blind pilot study.
3 Data are NOT Winsorized.
                                            51

-------
Table 5-5a.   Descriptive Statistics of Reported Lead Amounts (% by weight) in Spiked
              Paint Proficiency-Test Samples, Calculated Across Participating Clients
              and Their Laboratories, by Testing Round
Testing Round
#
Samples
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Spiked Paint Sample Data
DB Round 1
DB Round 2
DB Round 3
32
32
29
0.639
0.749
0.710
0.124
0.340
0.204
0.073
0.539
0.600
0.617
0.637
0.626
0.664
0.664
0.653
0.696
0.695
0.690
0.779
2.270
1.450
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
31
29
27
0.658
0.659
0.656
0.070
0.066
0.039
0.493
0.539
0.600
0.624
0.632
0.620
0.668
0.661
0.652
0.700
0.681
0.690
0.779
0.804
0.755
ELPAT Round 22 Results for This Sample Batch
DB labs only1
All NLLAP-recognized labs2
All ELPAT Round 22 labs
11
118
323
0.655
0.726
0.679
0.056
0.881
0.566
0.542
0.510
0.063
0.656
0.624
0.611
0.673
0.651
0.646
0.686
0.675
0.677
0.708
10.21
10.21
1 ELPAT Round 22 data for only those laboratories analyzing the given sample type in the double-blind pilot study.
2 Data are NOT Winsorized.

Table 5-5b.   Descriptive Statistics of Reported Percent Recoveries in Spiked Paint
              Proficiency-Test Samples, Calculated Across Participating Clients and
              Their Laboratories,  by Testing Round1
Testing Round
#
Samples
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Spiked Paint Sample Data
DB Round 1
DB Round 2
DB Round 3
32
32
29
99.1
1 16.1
110.0
19.2
52.7
31.7
11.3
83.5
93.0
95.6
98.7
97.0
102.9
102.9
101.2
107.8
107.6
106.9
120.7
351.7
224.7
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
31
29
27
101.9
102.1
101.7
10.9
10.2
6.1
76.4
83.5
93.0
96.7
97.9
96.1
103.5
102.4
101.0
108.5
105.5
106.9
120.7
124.6
117.0
ELPAT Round 22 Results for This Sample Batch
DB labs only2
All NLLAP-recognized labs3
All ELPAT Round 22 labs
11
118
323
101.5
1 12.5
105.2
8.7
136.6
87.7
83.9
79.0
9.7
101.6
96.7
94.6
104.3
100.8
100.2
106.3
104.6
104.9
109.7
1581
1581
1 Percent recoveries are calculated relative to the target of 0.6454% lead by weight.
2 ELPAT Round 22 data for only those laboratories analyzing the given sample type in the double-blind pilot study.
3 Data are NOT Winsorized.
                                            52

-------
       Laboratory performance in the double-blind testing rounds can also be inferred from Tables 5-
3a through 5-5b by observing the variability of the sample results by testing round. Initial review of
Figures 5-1 through 5-3 in Section 5.1.2 suggested that for each proficiency-test sample type, data in
the double-blind testing rounds had higher variability than for ELPAT Round 22 for the given set of
laboratories.  For dust-wipe samples, Tables 5-3a and 5-4a confirm this conclusion by noting that even
when statistical outliers were excluded, the standard deviation of the dust wipe sample data in each DB
testing round was more than twice that reported in ELPAT Round 22 for the same group of
laboratories.  To compare these variability estimates across the four testing rounds (the three double-
blind testing rounds plus ELPAT Round 22), Levene's test for homogeneous variance was applied to
these data separately for the low-spiked and mid-spiked dust-wipe samples. While Levene's test
indicated that, for both dust-wipe sample types, the differences in these standard deviations across the
four testing rounds was not significantly different at the 0.05 level, the standard deviations of the log-
transformed dust-wipe data (i.e., the data used in statistical analyses within this report) were
significantly different at the 0.05 level across testing rounds.  This was true regardless of whether
statistical outliers were included when calculating standard deviations within the double-blind testing
rounds. The significant difference was primarily due to the lower variability observed in Round 22 of
the ELPAT Program relative to the double-blind testing rounds.

       Compared to the dust-wipe data, the double-blind proficiency-test paint chip sample data had
standard deviations that more closely matched the standard deviation for paint chip sample data from
ELPAT Round 22 (Table 5-5a). These standard deviations did not differ significantly (at the 0.05
level) across testing rounds, according to Levene's test.

       A more complete statistical evaluation of how variability in proficiency-test sample results differ
across testing rounds and of how accurately the laboratories report results  near the samples' respective
target levels takes into account the different components of variability (lab-to-lab, within-lab) that could
be estimated from the double-blind pilot study data.  Results of this analysis, and further conclusions
that could be made from this analysis on laboratory performance, are presented within Sections 5.3 and
5.4.

Summaries by laboratory for each double-blind testing round

       Summaries by laboratory of the double-blind pilot study data are presented according to
double-blind testing round in Tables 5-6a (low-spiked dust), 5-7a (mid-spiked dust), and 5-8a (paint)
for the measured lead amounts, and in Tables 5-6b (low-spiked dust), 5-7b (mid-spiked dust), and
5-8b (paint) for percent recoveries. Summaries excluding the outliers in Table 5-1 are presented in
italics below the summaries that include outliers, for the combinations of laboratories and DB testing
rounds that contained outliers.  Each laboratory's analysis result for the proficiency-test sample of the
same batch that it analyzed in ELPAT Round 22 is included in these tables.
                                              53

-------
Table 5-6a.   Summaries of Reported Lead Amounts (//g) in Low-Spiked Dust
              Proficiency-Test Samples, by Laboratory and  Testing Round
Laboratory ID
01
02
03
04
05
06
07
08
09
10
ELPAT Round 22
Result
123.2
147.2
124.0
130.4
145.0
145.0
122.4
135.7
146.7
135.4
Double-Blind Pilot Study: Mean (Standard Error) (tt Samples)1
Double-Blind Round 1
125.0(15.0) (2)
150.6 (2.6) (6)
108.5 (8.5) (2)
128.0 (8.2) (4)
No data
109.8 (4.7) (4)
2.7(0.1)(2)
No data
132.4 (4.5) (4)
166.5 (6.5) (2)
142.5 (5.0) (2)
Double-Blind Round 2
150.0(10.0) (2)
145.2 (3.4) (6)
122.0(7.0) (2)
130.8(4.4) (4)
132.0(4.0) (2)
95.3 (0.9) (4)
99.5 (2.1) (2)
125.8(7.8) (4)
165.5 (2.5) (2)
182.1 (60.4) (2)
121.7 (-) (1)
Double-Blind Round 3
135.0(5.0) (2)
141.5 (3.0) (6)
No data
126.0(7.9) (2)
141.0(5.0) (2)
89.8 (4.3) (4)
133.1 (12.2) (2)
137.6 (1.8) (2)
171.0(13.0) (2)
105.0(5.0) (2)
1 Results in italics represent summaries with outliers removed (see Table 5-1).

Table 5-6b.   Summaries of Reported Percent  Recoveries in Low-Spiked Dust
              Proficiency-Test Samples, by Laboratory and Testing Round1
Laboratory ID
01
02
03
04
05
06
07
08
09
10
ELPAT Round 22
Result
95.5
114.1
96.1
101.1
112.4
112.4
94.9
105.2
113.7
105.0
Double-Blind Pilot Study: Mean (Standard Error) (# Samples)2
Double-Blind Round 1
96.9 (11.6) (2)
116.7 (2.0) (6)
84.1 (6.6) (2)
99.2 (6.3) (4)
No data
85.1 (3.7) (4)
2.1 (0.0) (2)
No data
102.6 (3.5) (4)
129.1 (5.0) (2)
110.5 (3.9) (2)
Double-Blind Round 2
116.3 (7.8) (2)
112.6 (2.6) (6)
94.6 (5.4) (2)
101.4(3.4) (4)
102.3 (3.1) (2)
73.8 (0.7) (4)
77.1 (1.6) (2)
97.5 (6.1) (4)
128.3 (1.9) (2)
141.2 (46.8) (2)
94.3 (-) (1)
Double-Blind Round 3
104.7 (3.9) (2)
109.7 (2.3) (6)
No data
97.7 (6.2) (2)
109.3 (3.9) (2)
69.6 (3.3) (4)
103.2 (9.5) (2)
106.6 (1.4) (2)
132.6 (10.1) (2)
81.4 (3.9) (2)
1 Percent recoveries are calculated relative to the target of 1 29 /jg Pb.
2 Results in italics represent summaries with outliers removed (see Table 5-1).
                                           54

-------
Table 5-7a.   Summaries of Reported Lead Amounts (//g) in Mid-Spiked Dust
              Proficiency-Test Samples, by Laboratory and Testing Round
Laboratory ID
01
02
03
04
05
06
07
08
09
10
ELPAT Round 22
Result
273.6
303.1
273.0
271.2
299.3
281.0
298.9
304.9
242.3
294.9
Double-Blind Pilot Study: Mean (Standard Error) (# Samples)1
Double-Blind Round 1
295.0 (5.0) (2)
314.7 (5.6) (6)
222.0 (3.0) (2)
307.0 (7.8) (4)
No data
233.3 (5.1) (4)
5.2 (0.2) (2)
No data
294.3 (15.4) (4)
309.0 (22.0) (2)
274.0 (9.0) (2)
Double-Blind Round 2
315.0(5.0) (2)
316.2 (6.3) (6)
271.5 (3.5) (2)
292.3 (7.0) (4)
280.5 (6.5) (2)
214.5 (11.0) (4)
205.2 (35.2) (2)
294.8 (3.4) (4)
329.5 (2.5) (2)
190.0(60.0) (2)
250.0 (-) (1)
Double-Blind Round 3
305.0 (5.0) (2)
301.8(11.3) (6)
No data
277.8 (0.2) (2)
292.0 (5.0) (2)
201.8(7.1) (4)
286.6 (11.2) (2)
322.6 (8.3) (2)
332.5 (9.5) (2)
260.0 (0.0) (2)
1 Results in italics represent summaries with outliers removed (see Table 5-1).

Table 5-7b.   Summaries of Reported Percent Recoveries  in Mid-Spiked Dust
              Proficiency-Test Samples, by Laboratory and Testing Round1
Laboratory ID
01
02
03
04
05
06
07
08
09
10
ELPAT Round 22
Result
100.6
111.4
100.4
99.7
110.0
103.3
109.9
112.1
89.1
108.4
Double-Blind Pilot Study: Mean (Standard Error) (# Samples)2
Double-Blind Round 1
108.5 (1.8) (2)
115.7 (2.0) (6)
81.6 (1.1) (2)
112.9 (2.9) (4)
No data
85.8(1.9) (4)
1.9(0.1) (2)
No data
108.2 (5.7) (4)
113.6 (8.1) (2)
100.7 (3.3) (2)
Double-Blind Round 2
115.8(1.8) (2)
116.3 (2.3) (6)
99.8(1.3) (2)
107.5 (2.6) (4)
103.1 (2.4) (2)
78.9 (4.0) (4)
75.4 (12.9) (2)
108.4(1.3) (4)
121.1 (0.9) (2)
69.9 (22.1) (2)
91.9 (-) (1)
Double-Blind Round 3
112.1 (1.8) (2)
111.0(4.2) (6)
No data
102.1 (0.1) (2)
107.4(1.8) (2)
74.2 (2.6) (4)
105.3 (4.1) (2)
118.6 (3.1) (2)
122.2 (3.5) (2)
95.6 (0.0) (2)
1 Percent recoveries are calculated relative to the target of 272 /jg Pb.
2 Results in italics represent summaries with outliers removed (see Table 5-1).
                                           55

-------
Table 5-8a.   Summaries of Reported Lead Amounts (% by weight) in Spiked Paint
              Proficiency-Test Samples, by Laboratory and Testing Round
Laboratory ID
01
02
03
04
05
06
07
08
09
11
12
ELPAT Round 22
Result
0.6858
0.7081
0.6650
0.6811
0.6790
0.6730
0.6653
0.7000
0.5418
0.6555
0.5493
Double-Blind Pilot Study: Mean (Standard Error) (tt Samples)1
Double-Blind Round 1
0.695 (0.005) (2)
0.708(0.018) (6)
0.625 (0.001) (2)
0.699 (0.026) (4)
0.664(0.014) (4)
No data
0.655 (0.005) (2)
0.709 (0.021) (4)
0.338 (0.265) (2)
0.603 (-) (1)
0.600(0.010) (2)
0.523 (0.011) (4)
Double-Blind Round 2
0.690 (0.010) (2)
0.760 (0.044) (6)
0.728 (0.037) (5)
0.674 (0.001) (2)
0.653 (0.007) (4)
0.667 (0.006) (4)
No data
0.605 (0.055) (2)
0.687 (0.019) (4)
1 .980 (0.290) (2)
No data
0.645 (0.015) (2)
0.552 (0.006) (4)
Double-Blind Round 3
0.680 (0.010) (2)
0.700 (0.016) (6)
No data
0.661 (0.035) (2)
0.663 (0.008) (4)
0.690 (-) (1)
0.610 (0.010) (2)
0.622 (0.007) (4)
1.435 (0.015) (2)
No data
0.635 (0.015) (2)
0.631 (0.010) (4)
1 Results in italics represent summaries with outliers removed (see Table 5-1).

Table 5-8b.   Summaries of Reported Percent Recoveries in Spiked Paint Proficiency-
              Test Samples, by Laboratory and Testing  Round1
Laboratory ID
01
02
03
04
05
06
07
08
09
11
12
ELPAT Round 22
Result
106.3
109.7
103.0
105.5
105.2
104.3
103.1
108.5
83.9
101.6
85.1
Double-Blind Pilot Study: Mean (Standard Error) (# Samples)2
Double-Blind Round 1
107.7 (0.8) (2)
109.8(2.8) (6)
96.8 (0.2) (2)
108.4(4.0) (4)
102.9 (2.1) (4)
No data
101.5 (0.8) (2)
109.9 (3.2) (4)
52.4(41.1) (2)
93. 4 (-) (1)
93.0(1.5) (2)
81.0(1.7) (4)
Double-Blind Round 2
106.9 (1.5) (2)
117.7 (6.8) (6)
112.9 (5.8) (5)
104.4(0.1) (2)
101.2 (1.1) (4)
103.3 (0.9) (4)
No data
93.7 (8.5) (2)
106.4(2.9) (4)
306.8 (44.9) (2)
No data
99.9 (2.3) (2)
85.5 (1.0) (4)
Double-Blind Round 3
105.4 (1.5) (2)
108.4 (2.6) (6)
No data
102.4 (5.4) (2)
102.7 (1.3) (4)
106.9H (1)
94.5 (1.5) (2)
96.4(1.1) (4)
222.3 (2.3) (2)
No data
98.4 (2.3) (2)
97.7 (1.5) (4)
1 Percent recoveries are calculated relative to
2 Results in italics represent summaries with
the target of 0.6454% lead by weight.
outliers removed (see Table 5-1).
                                           56

-------
       The laboratory summaries (Tables 5-6a through 5-8b) show occasional gaps in the data for
certain DB testing rounds, due to reasons such as omitting outliers from analysis (e.g., Laboratory 07 in
DB Round 1 for dust) or the client did not report analytical results back for a given set of samples (e.g.,
Laboratory 03 in DB Round 3 for dust). In general, average lead amounts for a specific laboratory
were either consistently above or consistently below the target lead amount across the three DB testing
rounds. For example, average percent recoveries for both types of dust samples exceeded 100% in
each DB testing round for laboratories 02, 05, and 09, while averages were below 100% in each DB
testing round for laboratories 03 and 06 (Tables 5-6b and 5-7b).  The unusually high paint results for
laboratory 09 relative to the other laboratories are evident throughout the study in Tables 5-8a and 5-
8b. Thus, laboratories tended to be consistent in their reported results within and across testing rounds,
while results from different laboratories could differ considerably.

Summaries of client/laboratory averages by DB testing round

       Summaries of the averages of individual sample results within each client/laboratory
combination are presented according to DB testing round in Tables 5-9a (low-spiked dust), 5-10a
(mid-spiked dust), and 5-1 la (paint) for the measured lead amounts, and Tables 5-9b (low-spiked
dust), 5-1 Ob (mid-spiked dust), and 5-1 Ib (paint) for percent recoveries.  These tables are similar to
Tables 5-3a through 5-5b, but instead of summarizing individual sample results, these results are first
averaged within each of the 10-13 client/laboratory combinations, then summarized.  Thus, each
client/laboratory combination has equal weight, despite one combination being associated with more
sample results than another.  As in Tables 5-3a through 5-5b, summaries are presented both with and
without the outliers in Table 5-1 included.

       While the reported means in these tables are similar to those in Tables 5-1 and 5-2, the
variability in the results is slightly lower due to summarizing average lead measurements rather than
measurements for individual samples. (Averages have lower variability than the individual data values
used to calculate them, as more information is used to determine the average.) The summaries show
that averages tended to be higher than their respective target amounts in each DB testing round when
outliers were removed, with similar performance noted from one DB testing round to the next by the
laboratories.

5.3    CHARACTERIZING COMPONENTS OF VARIATION

       The tables and figures in  Sections 5.1 and 5.2 provided preliminary information on how results
in the double-blind pilot study differed from one laboratory to another and from one testing round to
another. In addition, the figures provided some initial information on the two sources of variability  in the
data within a testing round: lab-to-lab variability and within-lab variability.  This section uses statistical
modeling techniques documented in  Section 4.4 to characterize lab-to-lab and within-lab variability in
the double-blind pilot study data and to determine how this variability differs across the three double-
blind testing rounds for a given proficiency-test sample type.

                                              57

-------
Table 5-9a.   Descriptive Statistics of Average Reported Lead Amounts (//g) in Low-
             Spiked Dust Proficiency-Test Samples for Each Client/ Laboratory
             Combination, by Double-Blind Testing Round
Testing Round
# Client/Lab
Combination
s
Average Measured Lead Amount in Low-Spiked Dust Samples
for Each Client/Laboratory Combination (//g)
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Low-Spiked Dust Sample Data
DB Round 1
DB Round 2
DB Round 3
12
13
10
123.2
134.8
132.4
41.7
24.5
22.3
2.7
95.3
89.8
117.4
122.0
126.0
128.0
135.2
136.3
145.9
150.0
141.0
166.5
182.1
171.0
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
11
13
10
134.1
130.1
132.4
18.0
20.1
22.3
108.5
95.3
89.8
125.0
121.7
126.0
130.1
132.0
136.3
149.3
142.8
141.0
166.5
165.5
171.0
Table 5-9b.   Descriptive Statistics of Average Reported Percent Recovery in Low-
             Spiked Dust Proficiency-Test Samples for Each Client/Laboratory
             Combination, by Double-Blind Testing Round1
Testing Round

DB Round 1
DB Round 2
DB Round 3
# Client/Lab
Combination
s
Average Percent Recovery for Low-Spiked Dust Samples
for Each Client/Laboratory Combination
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Low-Spiked Dust Sample Data
12
13
10
95.5
104.5
102.7
32.3
19.0
17.3
2.1
73.8
69.6
91.0
94.6
97.7
99.2
104.8
105.6
113.1
116.3
109.3
129.1
141.2
132.6
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
1 1
13
10
104.0
100.9
102.7
14.0
15.6
17.3
84.1
73.8
69.6
96.9
94.3
97.7
100.9
102.3
105.6
115.7
110.7
109.3
129.1
128.3
132.6
1 Percent recoveries are calculated relative to the target of 129 //g Pb.
                                        58

-------
Table 5-10a.  Descriptive Statistics of Average Reported Lead Amounts (//g) in Mid-
             Spiked Dust Proficiency-Test Samples for Each Client/ Laboratory
             Combination, by Double-Blind Testing Round
Testing Round
# Client/Lab
Combination
s
Average Measured Lead Amount in Mid-Spiked Dust Samples
for Each Client/Laboratory Combination (//g)
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Mid-Spiked Dust Sample Data
DB Round 1
DB Round 2
DB Round 3
12
13
10
264.5
278.0
289.7
88.2
46.0
39.1
5.2
190.0
201.8
250.9
271.5
277.8
295.5
292.0
289.3
314.4
314.4
322.6
322.8
329.5
332.5
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
11
13
10
288.1
282.7
289.7
34.8
38.9
39.1
222.0
205.2
201.8
268.5
271.5
277.8
296.0
292.0
289.3
318.1
314.4
322.6
322.8
329.5
332.5
Table 5-1 Ob. Descriptive Statistics of Average Reported Percent Recovery in Mid-
             Spiked Dust Proficiency-Test Samples for Each Client/ Laboratory
             Combination, by Double-Blind Testing Round1
Testing Round
# Client/Lab
Combination
s
Average Percent Recovery for Mid-Spiked Dust Samples
for Each Client/Laboratory Combination
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Mid-Spiked Dust Sample Data
DB Round 1
DB Round 2
DB Round 3
12
13
10
97.3
102.2
106.5
32.4
16.9
14.4
1.9
69.9
74.2
92.2
99.8
102.1
108.6
107.4
106.4
115.6
115.6
118.6
118.7
121.1
122.2
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
1 1
13
10
105.9
103.9
106.5
12.8
14.3
14.4
81.6
75.4
74.2
98.7
99.8
102.1
108.8
107.4
106.4
116.9
115.6
118.6
118.7
121.1
122.2
1 Percent recoveries are calculated relative to the target of 272 //g Pb.
                                        59

-------
Table 5-11a.  Descriptive Statistics of Average Reported Lead Amounts (% by weight)
             in Spiked Paint Proficiency-Test Samples for Each Client/Laboratory
             Combination, by Double-Blind Testing Round
Testing Round
# Client/Lab
Combination
s
Average Measured Lead Amount in Spiked Paint Samples
for Each Client/Laboratory Combination (% by weight)
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Spiked Paint Sample Data
DB Round 1
DB Round 2
DB Round 3
12
12
1 1
0.638
0.772
0.728
0.114
0.385
0.237
0.338
0.552
0.610
0.613
0.643
0.631
0.660
0.665
0.661
0.702
0.688
0.690
0.763
1.980
1.435
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
12
1 1
10
0.660
0.659
0.657
0.066
0.058
0.034
0.523
0.552
0.610
0.614
0.641
0.631
0.660
0.662
0.657
0.702
0.687
0.680
0.763
0.787
0.723
Table 5-11b. Descriptive Statistics of Average Reported Percent Recovery in Spiked
             Paint Proficiency-Test Samples for Each Client/Laboratory Combination,
             by Double-Blind Testing Round1
Testing Round
# Client/Lab
Combination
s
Average Percent Recovery for Spiked Paint Samples
for Each Client/Laboratory Combination
Mean
Standard
Deviation
Minimum
Percentiles
25th
50th
75th
Maximum
All Spiked Paint Sample Data
DB Round 1
DB Round 2
DB Round 3
12
12
1 1
98.8
119.6
112.7
17.7
59.7
36.7
52.4
85.5
94.5
94.9
99.6
97.7
102.3
103.0
102.4
108.8
106.6
106.9
118.2
306.8
222.3
Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
12
11
10
102.2
102.2
101.8
10.3
8.9
5.3
81.0
85.5
94.5
95.1
99.3
97.7
102.3
102.6
101.8
108.8
106.4
105.4
118.2
121.9
112.0
1 Percent recoveries are calculated relative to the target of 0.6454% lead by weight.
                                        60

-------
       To portray the observed distributions of the entire pilot study data graphically, Figures 5-7
through 5-9 contain histograms of these data, with data for the entire study (i.e., all three testing rounds)
summarized within a single histogram for each sample type.  The double-blind pilot data are
represented in these histograms within unshaded bars. To compare variability in the results between the
double-blind pilot and Round 22 of the ELPAT Program for the participating laboratories, the
histograms include the ELPAT Round 22 data within shaded bars.

       Note from Figures 5-7 through 5-9 that although the double-blind pilot study has considerably
more data than Round 22 of the ELPAT Program for the participating laboratories, total variability in
the double-blind pilot data is considerably greater than in ELPAT Round 22 for each sample type.
While this is especially true for the dust sample results, the outliers identified in Table 5-1 are clearly
visible within the paint histogram and result in most of the variability observed in that histogram.  These
figures suggest that additional variability is present in the double-blind pilot study data compared to the
single-blind data for the group of laboratories participating in this pilot study. However, as seen in
Tables 5-3a through 5-5a when considering the larger group of 118 NLLAP-recognized laboratories, it
is possible to observe extreme and wide-ranging data values even within a testing round of the single-
blind ELPAT Program.

Statistical modeling results

       Section 4.4 presented the random effects modeling approach taken to characterize lab-to-lab
and within-lab variability across testing rounds in the reported lead amounts for the proficiency-test
samples analyzed in this double-blind pilot study.  Model (1) in Section 4.4 was fitted to these data for
each of the three proficiency-test sample type, both with and without the outliers in  Table 5-1 included,
for a total of six fits. Recall that this model assumed that each sample result was equal to an overall
predicted value, which could change from one testing round to the next, plus or minus some random
amount that comes from a normal distribution with mean zero and positive variance.  This variance was
assumed to contain lab-to-lab and within-lab components which were each allowed to change from one
testing round to the next.  To test whether the estimates of lab-to-lab variability and within-lab
variability differed from one double-blind testing round to the next, reduced versions of model (1) were
fitted to the same set of data, and the estimated likelihoods were statistically compared among model
(1) and the reduced models.  Dust-lead measurements were logarithmically-transformed prior to
performing this analysis.

       The results of fitting model (1) to the double-blind pilot study data are presented graphically in
Figures 5-10 through 5-15 and are documented in Table 5-12. Figures 5-10 through 5-15 each
contain two sets  of bar charts, one representing an analysis performed without the statistical outliers in
Table 5-1 included, and the other representing an analysis performed on all data.  For each DB testing
round, Figures 5-10 through 5-12 plot the percentages of total variability attributable to each of its two
assumed components, lab-to-lab variability (solid portion of each bar) and within-lab variability
("cross-hatched" portion of each bar), for low-

                                              61

-------
      20
    Round a
bower Ltr« en
ELRKT Round ass
upper mitt on
Accepted* Ranee
       15
       10
          100              200

        Lew Spiked  Dust Sampte ResuDts (
                                                            300

                                                              Pb )
                                      400
Figure 5-7.   Histogram of Low-Spiked Dust Sample Results (ug) Reported in the
             Double-Blind Pilot Study (unshaded bars) and in Round 22 of the ELPAT
             Program (shaded bars) by Laboratories Participating in the Pilot
                                       62

-------
     20
                ELFAT Round 22;
                Lovwi LJnft on
     15
     10
      5
ELMT Round 22;
Reterarae Vaftw
                          n
                                        n
                          JHUllll
ELMT Round 22:
UppafLJitton
                     100           200          300
                    MDd Spiked Dust SampDa ResuDts ( /ug Pb )
                                                   400
Figure 5-8.
Histogram of Mid-Spiked Dust Sample Results (ug) Reported in the
Double-Blind Pilot Study (unshaded bars) and in Round 22 of the ELPAT
Program (shaded bars) by Laboratories Participating in the Pilot
                               63

-------
      20
       10
               ELRW Round 22;
               liMM-UnBon
             n
          BJW Fkuid22i
            n

n
n
          OJO
0,5           1JD            1J5           2JO
Spiked Palht Sampfe Results {% Pb by Watjght)
                     =*=r
                      2J5
Figure 5-9.   Histogram of Spiked Paint Sample Results (% by weight) Reported in the
             Double-Blind Pilot Study (unshaded bars) and in Round 22 of the ELPAT
             Program (shaded bars) by Laboratories Participating in the Pilot
                                        64

-------
CD
cn
        1
        I
                                                                                         DBRoundl  DBRx«J2   DBRounUS
   Figure 5-10.  Estimated Percentage of Total Variability in  Reported  Lead Amounts Within  Low-Spiked Dust Samples
                 That is Attributable to Lab-to-Lab and Within-Lab Sources, Based on Analyses Performed With and
                 Without Statistical Outliers Included
   (Note: "All DB Rounds" represents estimated percentages over the entire pilot study. These estimates were calculated only when statistical outliers were
   removed, as the percentages differed significantly from one DB testing round to another when the outliers were included.)

-------
CD
CD
        1
        I
                                                                                         DBRoundl   DBRx«J2   DBRounUS
   Figure 5-11.  Estimated Percentage of Total Variability in Reported Lead Amounts Within Mid-Spiked Dust Samples That
                 is Attributable to Lab-to-Lab and Within-Lab Sources, Based on Analyses Performed With and Without
                 Statistical Outliers Included
   (Note: "All DB Rounds" represents estimated percentages over the entire pilot study.  These estimates were calculated only when statistical outliers were
   removed, as the percentages differed significantly from one DB testing round to another when the outliers were included.)

-------
CD
        1
        I
                                                                                         DBRoundl   DBRx«J2  DBRounUS
   Figure 5-12.  Estimated Percentage of Total Variability in Reported Lead Amounts Within Spiked Paint Samples That is
                 Attributable to Lab-to-Lab and Within-Lab Sources, Based on Analyses Performed With and Without
                 Statistical Outliers Included
   (Note: "All DB Rounds" represents estimated percentages over the entire pilot study.  These estimates were calculated only when statistical outliers were
   removed, as the percentages differed significantly from one DB testing round to another when the outliers were included.)

-------
CD
CO
                                   (wl** 5-1)
      QflM)
             AIDBRnunh
               9wnw of ^Mb^n
DBRundl
CB Round?
DR Round!
    lUb-b-l*  88H W|*i L*
                                                                  ADahMidbd
                                                                             00m
     DBfeundl
9wnw of Wsb^R
  DBfound2
|L*-to-l*
   Figure 5-13.  Estimates of Lab-to-Lab and Within-Lab Sources of Variability in Reported Lead Amounts Within Low-
                 Spiked Dust Samples, Based on Analyses Performed With and Without Statistical Outliers Included

   Note: Vertical axis is in (Iog0t/g))2.
   Note: "All DB Rounds" represents estimates over the entire pilot study.  These estimates were calculated only when statistical outliers were removed, as the
   estimates differed significantly from one DB testing round to  another when the outliers were included.
   * Value of estimate is 1.7075.

-------
CD
CD
                                   (wl** 5-1)
AIDBRnunh
                            DBRundl
                                                                                 ADahMidbd
                                                                             00m
     DBfeundl

9wnw of WBto^R
  DBfound2

|L*-to-l*
   Figure 5-14.  Estimates of Lab-to-Lab and Within-Lab Sources of Variability in Reported Lead Amounts Within Mid-
                 Spiked Dust Samples, Based on Analyses Performed With and Without Statistical Outliers Included

   Note: Vertical axis is in (Iog0t/g))2.
   Note: "All DB Rounds" represents estimates over the entire pilot study. These estimates were calculated only when statistical outliers were removed, as the
   estimates differed significantly from one DB testing round to another when the outliers were included.
   * Value of estimate is 1.7752.

-------
                         DBRMCh       t

            Sura of fettly;   ^L*-fc-L4>  BSBWNiLA
     DBBwidi
SQUQB of IMPEW^F!
   MRIWI2

|Lab-D-ld>   B889
Figure 5-15.  Estimates of Lab-to-Lab and Within-Lab Sources of Variability in Reported Lead Amounts Within Spiked
              Paint Samples, Based on Analyses Performed With and Without Statistical Outliers Included

Note: Vertical axis is in (%)2.
Note: "All DB Rounds" represents estimates over the entire pilot study. These estimates were calculated only when statistical outliers were removed, as the
estimates differed significantly from one DB testing round to another when the outliers were included.

-------
Table 5-12.  Estimates of Lab-to-Lab Variability and Within-Lab Variability, Expressed
              Absolutely and Relative to Total Variability, Associated With Lead
              Measurements Reported in the Double-Blind Pilot Study,  by Sample Type
              and Testing Round
Sample Type
Testing Round
Lab-to-Lab
Variability (% of
Total Variability)
Within-Lab
Variability (% of
Total Variability)
Total Variability
Statistical Outliers Removed (see Table 5-1)
Low-Spiked
Dust
Samples
Mid-Spiked
Dust
Samples
Spiked Paint
Samples
DB Round 1
DB Round 2
DB Round 3
Across All Rounds
DB Round 1
DB Round 2
DB Round 3
Across All Rounds
DB Round 1
DB Round 2
DB Round 3
Across All Rounds
0.0189 (70.9%)
0.0274 (84.3%)
0.0331(85.2%)
0.0208 (68.6%)
0.0162 (80.5%)
0.0251 (80.8%)
0.0212 (84.0%)
0.0165 (68.7%)
0.0034 (73.2%)
0.0022 (53.4%)
0.00075 (49.5%)
0.0016 (46.4%)
0.0078(29.1%)
0.0051 (15.7%)
0.0057 (14.8%)
0.0095 (31.4%)
0.0039 (19.5%)
0.0060(19.2%)
0.0040(16.0%)
0.0075 (31.3%)
0.0012(26.8%)
0.0020 (46.6%)
0.00076 (50.5%)
0.0018(53.6%)
0.0267
0.0325
0.0388
0.0303
0.0201
0.0311
0.0252
0.0240
0.0046
0.0042
0.00151
0.0034
All Data Included
Low-Spiked
Dust
Samples
Mid-Spiked
Dust
Samples
Spiked Paint
Samples
DB Round 1
DB Round 2
DB Round 3
DB Round 1
DB Round 2
DB Round 3
DB Round 1
DB Round 2
DB Round 3
1.7075 (99.6%)
0.0308 (64.7%)
0.0331 (85.2%)
1.7752 (99.8%)
0.0369 (69.1%)
0.0212 (84.0%)
0.0097 (55.8%)
0.1709 (94.0%)
0.0614 (98.8%)
0.0074 (0.4%)
0.0168(35.3%)
0.0057 (14.8%)
0.0039 (0.2%)
0.0165 (30.9%)
0.0040(16.0%)
0.0077 (44.2%)
0.0108(6.0%)
0.00076 (1.2%)
1.7149
0.0476
0.0388
1.7791
0.0534
0.0252
0.0174
0.1817
0.0621
Note: "Across All Rounds" represents estimates over the entire pilot study. These estimates were calculated
only when statistical outliers were removed, as for each proficiency-test sample type, the estimates did not differ
significantly from one DB testing round to another (at the 0.05 level) when the outliers were removed, but they
did differ significantly when the outliers were included.
                                             71

-------
spiked dust, mid-spiked dust, and spiked paint samples, respectively. Each bar in these charts
represents 100% of the total variability. Figures 5-13 through 5-15 present the estimated variance
components for low-spiked dust, mid-spiked dust, and spiked paint samples, respectively.

       Note that the portion of Figures 5-10 through 5-15 associated with analyses performed while
excluding statistical outliers from Table 5-1 include bars for estimates across DB testing rounds (i.e.,
over the entire pilot study).  When statistical outliers were omitted from the analysis, the observed
differences in the variance component estimates did not differ significantly (at the 0.05 leveH from
round-to-round for either component or for any of the three proficiency-test sample types. In these
cases, common estimates of lab-to-lab variability and within-lab variability were generated across the
entire study, without regard to DB testing rounds, and are included in these figures. However, when
statistical outliers were not omitted from the analysis, the variance component estimates differed
significantly from one testing round to another (p < 0.0011 for each type of proficiency-test sample and
for both variance components.  Thus, variance component estimates over the entire pilot study were not
generated when all study data were included in the analysis.

       The following additional conclusions could be made from the analyses documented in Figures
5-10 through 5-15 and Table 5-12:

       •     According to all forms of the model and for each proficiency-test sample type, the
              overall model-predicted value for the lead measurement in a given DB testing round
              (represented by the term |i+Rj in Model (1)) did not differ significantly across testing
              rounds at the 0.05 level. Furthermore, a predicted value did not differ significantly from
              its corresponding target level at the 0.05 level.  This finding was observed regardless of
              whether the statistical outliers in Table 5-1 were included or excluded from the analysis.
              This result implies that general bias in the laboratory-reported measurements on the
              double-blind proficiency-test samples did not differ significantly across the DB testing
              rounds (at the 0.05 level! and deviation from their respective target levels (Table 2-1}
              was not statistically significant overall.

       •     When all data, including statistical outliers, were included in the analysis, lab-to-lab
              variability for both low-spiked and mid-spiked dust-wipe samples was greatest in DB
              Round 1, primarily due to the results for one laboratory (laboratory 07). For paint chip
              samples, lab-to-lab variability was greatest in DB Rounds 2 and 3, again primarily due
              to the results for one laboratory (laboratory 09).  In each instance, the proportion of
              total variability associated with lab-to-lab variability exceeded 90%.  This percentage
              decreased considerably when these and other statistical outliers were omitted from the
              analysis, especially for paint samples (Figure 5-12). Because some laboratories that
              reported unusually high or low measurements within a DB testing round did so for all
              samples in that round and with relatively good precision  (as noted in Figures 5-1
                                              72

-------
              through 5-31 the presence of statistical outliers highly influenced lab-to-lab variability
              within a DB testing round.

              When excluding outliers from the analysis, lab-to-lab variability tended to represent
              approximately 70% of total variability associated with the two types of dust-wipe
              samples and slightly under 50% of total variability associated with the paint chip
              samples.  These percentages were slightly higher when estimated for only a single
              testing round, as combining a laboratory's round-to-round variability with its variability
              associated with analyzing multiple samples within the same testing round contributes to
              an increased within-laboratory variability estimate.
5.4    COMPARING LAB-TO-LAB VARIABILITY BETWEEN THE ELPAT
       PROGRAM AND THE DOUBLE-BLIND STUDY

       For each of the three sample types included in the double-blind pilot study, results for Round
22 of the ELPAT Program were plotted in Figures 5-1 through 5-3 and listed in Tables 5-6a through
5-8a for the participating laboratories. As each laboratory analyzed only one sample of a given sample
type in each testing round within the ELPAT Program, all sources of variability in data from the ELPAT
Program are confounded with lab-to-lab variability. Nevertheless, to investigate how statistical
acceptance criteria developed for the ELPAT Program may be applied within a double-blind program,
it was of interest to compare the variability across laboratories in Round 22 of the ELPAT Program
with each testing round of the double-blind pilot study. The approach used to make this statistical
comparison was presented in Section 4.4.3.

       Initially, for a given double-blind proficiency-test sample type, all sample results reported by a
given laboratory within a given DB testing round were averaged.  Table 5-13 presents the means and
standard deviations of these laboratory averages for each DB testing round (both including and
excluding the statistical outliers in Table 5-1), and of the single-sample results reported in Round 22 of
the ELPAT Program (and documented in Tables 5-4a through 5-6a).  Thus, for the participating
laboratories, this table gives an indication of how lab-to-lab variability differs between Round 22 of the
ELPAT Program and each round of the double-blind  pilot study.

       As discussed in Section 4.4.3, Levene's test of homogeneity of variance was used to determine
whether the variability estimates presented in Table 5-13 (i.e.,  the standard deviations)  differed
significantly across the four testing rounds (the three DB testing rounds and ELPAT Round 22).
Significant differences in variability were observed across testing rounds at the 0.05 level only for the
two dust-wipe sample types, when the statistical outliers were included in the analysis. This is reflective
of the highly-inflated variability observed in DB Round 1 versus the other testing rounds, which resulted
from the presence of the statistical outliers. No other incidences of significant differences across testing
                                              73

-------
rounds were observed, despite slightly lower variability estimates occurring in ELPAT Round 22 versus
the DB testing rounds.

Table 5-13.   Summaries, Calculated Across Laboratories, of Laboratory Average Lead
               Measurements for the Three Types of Proficiency-Test Samples, Within
               Each  Round of the Double-Blind Pilot Study and in Round 22 of the
               ELPAT Program

ELPAT Round 22
# Samples
Analyzed Per
Lab
1
Summary of Average Sample Result Per Laboratory
Average (Standard Deviation) (# Laboratories)
Low-Spiked Dust
(//g Lead)
135.5 (10.1) (10)
Mid-Spiked Dust
(//g Lead)
284.2 (19.8) (10)
Paint
(% Lead by Wgt.)
0.655 (0.056) (11)
All Double-Blind Pilot Data
DB Round 1
DB Round 2
DB Round 3
2 to 6
2 to 6
1 to 6
118.4 (47.2) (9)
134.8(27.1) (10)
131.1 (23.1) (9)
250.5 (97.8) (9)
270.9 (50.1) (10)
286.7 (38.7) (9)
0.622 (0.116) (10)
0.791 (0.421) (10)
0.733 (0.249) (10)
Double-Blind Pilot Data with Outliers Removed (see Table 5-1)
DB Round 1
DB Round 2
DB Round 3
1 to 6
1 to 6
1 to 6
132.9 (19.8) (8)
128.8(21.5) (10)
131.1 (23.1) (9)
281.2 (35.4) (8)
276.9 (42.3) (10)
286.7 (38.7) (9)
0.648(0.061) (10)
0.656 (0.052) (9)
0.655 (0.032) (9)
Note: Statistics in this table for ELPAT Round 22, are based on one sample analyzed per laboratory and considers only those
laboratories involved in the double-blind pilot study. Statistics in this table for the double-blind (DB) pilot study are based on averages
of multiple samples analyzed per laboratory.
       Caution must be exercised when interpreting the results in Table 5-13 and the results of
Levene's test.  For each laboratory, the averages in this table for the DB testing rounds are calculated
from laboratory averages of from up to six results each, while averages for Round 22 of the ELPAT
Program are calculated from individual sample results.  The average of multiple observations from a
common distribution has lower variability than any one observation from this distribution (Snedecor and
Cochran, 1989), and so, the two sets of data would be expected to have different underlying variability.
To extend this point further, the double-blind averages should have  a lower standard error than the
ELPAT Program average, if the double-blind data originate from the same distribution as the ELPAT
Program data and a constant number of laboratories is assumed. However, in most instances,  the
opposite is seen in Table 5-13. Regardless of whether statistical outliers were included or not, the
                                              74

-------
standard deviations in Table 5-13 for the two dust sample types were higher (by at least 79%) in each
of the double-blind pilot testing rounds compared to Round 22 of the ELPAT Program.  Only for paint
sample results in Rounds 2 and 3 when outliers were excluded were the standard deviations of the
laboratory averages below what was observed in Round 22 of the ELPAT Program.

       Therefore, while Table 5-13 implies that averages in the DB testing rounds tended to vary more
considerably across laboratories than did the individual sample results within Round 22 of the ELPAT
Program, the extent that this lab-to-lab variation differed across testing rounds was not necessarily
statistically significant. However, as the table summarized averages of multiple sample results for each
laboratory in the DB testing rounds, while single-sample results were summarized from ELPAT Round
22, one expected to see lower variability in the DB testing rounds if, in fact, the data for the double-
blind and ELPAT Program testing rounds originated from the same underlying distribution.  Therefore,
even if a double-blind program evaluates a laboratory based on an average result across multiple
samples analyzed by the laboratory (of the same type), rather than on the result of analyzing an
individual sample, the evaluation criteria should consider that the results of double-blind testing may
have greater lab-to-lab variability compared to the  results of single-blind testing.
                                             75

-------
This page left blank intentionally.
              76

-------
6.0   QUALITY ASSURANCE

       A successful proficiency-test program must apply quality assurance procedures to ensure the
overall integrity of the proficiency-test samples throughout the course of the program, from preparation
to analysis.  In addition, the program must ensure that analytical results involving the proficiency-test
samples are reported accurately.  This chapter discusses how these issues were addressed in this
double-blind pilot study. More details on quality assurance issues in this pilot study are included in
Appendix A.

6.1    SAMPLE FABRICATION AND TRANSFER

       As  discussed below, the dust and paint source materials used in the double-blind pilot study
were obtained as part of the ELPAT Program.  The proficiency-test samples were prepared from these
materials within the ELPAT Program. Some of the procedures used to prepare the proficiency-test
samples are discussed below, with an emphasis on quality assurance practices.

6.1.1  Obtaining and Preparing Bulk Source Material

       The paint and household dust source material used to prepare the proficiency-test samples
were collected in the ELPAT Program following the procedures set forth  in Standard Operating
Procedure for Source Material Collection (Appendix C of RTI, 1994). A network of contractors
associated with abatement and risk assessment projects in public housing, military housing, and private
dwelling units contributes bulk paint source material for use in the ELPAT Program, while dust sample
material is obtained from vacuum bags in households conducting normal cleaning routines, from HEPA
vacuums used in post-abatement cleaning efforts, and from street sweeping. This material is then
classified according to lead content. Specifically, the low-lead dust used in this study came from a
Milwaukee (WI) exposure intervention program, the medium-lead dust came from a North Carolina
household,  and the paint came from an old hospital in Raleigh, NC.  See  Section 2 of Appendix A for
additional information on material selection.

       In the ELPAT Program, a given batch of source material is homogenized with respect to lead
concentration and particle size distribution. The method for preparing the paint source material is
detailed in Standard Operating Procedure for Preparation of Lead-in-Paint Proficiency
Analytical  Testing Material (Appendix C of RTI, 1994) and summarized in Section 4.3.1 of RTI,
1994. The  method for preparing the dust source material is detailed in Standard Operating
Procedure for Preparation of Proficiency Analytical Testing Material for Lead in Dust (Appendix
D of RTI, 1994) and is summarized in Section 4.3.2 of RTI, 1994.

6.1.2  Preparing Proficiency-Test Samples

       The dust and paint proficiency-test samples were prepared within the ELPAT Program. For
the double-blind pilot study, 180 proficiency-test samples of each sample type were prepared (Table
2-1 of Section 2.2). Additional samples at the rate of  5% per sample type were also prepared to act

                                            77

-------
as QC samples in the final verification process, which verified the lead content in the proficiency-test
samples, as described in Section 3.3 of Appendix A. The methods used to prepare the proficiency-test
samples are detailed in Sections 4.3.1 and 4.3.2 of RTI, 1994, and in Section 3 of Appendix A.

       Each dust proficiency-test sample consisted of a PaceWipe™ with 0.1 (±0.0005) grams of the
appropriate dust source material, as described in Appendix D of RTI, 19943. The PaceWipe™ is
preferred in the ELPAT Program as it contains no detectable background lead when using flame atomic
absorption spectrometry, such as NIOSH Method 7082. As presented in Section 4.1 of RTI
(Appendix A), analyses of 13 blank PaceWipes™ each reported <0.001 mg lead/wipe.  In addition,
the PaceWipe™ has a consistent moisture level and is covered by a solution that tends to retard
molding over time when stored under proper environmental conditions.

       Paint proficiency-test samples consist of one-gram aliquots of the paint source material.

       When dust (at each lead level) and paint proficiency-test samples were tested in duplicate, the
difference  in results agreed within 5% of the initial result.  Potential matrix interferences were evaluated
for both dust-wipe and paint-chip proficiency-test materials by evaluating the recovery of lead spiked
into replicate samples before analysis. The lead recoveries of the spiked samples were within the target
recovery range of 90-110%. These results are discussed in Section 4.1 of Appendix A.

       Once prepared, the proficiency-test samples were placed into individual plastic scintillation
vials, capped, and stored until they were ready to be transferred to client-supplied sample containers
(Section 2.4.1). Dust samples were stored in a 4°C cold room.  All vials were stored according to the
type of sample (dust, paint) and the level of lead in the sample.

6.1.3 Verifying That Client-Supplied  Sample Containers  Are Uncontaminated

       From each batch of sample containers received from a client, one container was used in a lead-
background test to verify that the batch of containers were uncontaminated. This test involved
swabbing the interior of the container with a PaceWipe™, then analyzing the wipe for lead
contamination (Section 4.3 of Appendix A).  While a default sample container (e.g., plastic centrifuge
tubes different from the containers used in the ELPAT Program) would have been used in place of a
client's sample containers  if they were found to be contaminated, this corrective action was not
necessary  (i.e., all analysis results were <0.001 mg lead/container).
       3 While RTI, 1994, indicates that Whatman™ No. 40 filters are used, the PaceWipe™ was used in this pilot
study and in the ELPAT Program.

                                             78

-------
6.1.4  Transferring Samples to Client-Supplied Sample Containers

       To verify the quantitative transfer of dust-wipe samples from the plastic scintillation vials to the
client-supplied sample containers, seven blank PaceWipes™ were spiked with NIST Standard
Reference Material 2711.  The samples were digested and analyzed for lead, as detailed in Appendix
D of RTI1994, and in Section 4.2 of Appendix A. The recovery percentage averaged 83.3%
(±1.6%) for these seven samples, compared to an 85% nominal recovery percentage for the SRM and
the recovery percentages of 82.6% - 86.0% recorded in Round 22 of the ELPAT Program.

       To investigate whether PaceWipes™ could become contaminated with lead as a result of the
sample transfer process, three blank PaceWipes™ were stored in plastic scintillation vials for two
hours and stored in a 4°C cold room, transferred to the default centrifuge tube, and removed for
analysis.  The blank recoveries for these three samples were each <0.001 mg lead/wipe (Section 4.1 of
Appendix A).
6.2    DATA MANAGEMENT AND SAMPLE TRACKING

       Data collected in this double-blind pilot study were generated by multiple sources specified
within Figure 2-1 of Section 2.0. This section describes the data management procedures that were
used in this pilot study and the methods used for sample tracking through the study.

6.2.1  Types of Data

       The majority of data in the pilot study were the quantitative and tracking data that were taken
from the Sample Tracking and Analysis Report Forms (Appendix C). Other types of data included the
analytical results for the participating laboratories from Round 22 of the ELPAT Program (provided by
the proficiency-testing service), qualitative information on the participating clients (used to select clients
in Chapter 3), and any feedback that the clients (or laboratories) had as a result of their participation in
the study.

       Information on the recruited clients was recorded during the recruitment process onto copies of
the telephone recruitment script in Appendix B. While most of this information was used to determine
the eligibility of the client for this pilot study (Sections 3.2 and 3.3) and to determine which laboratories
were testing proficiency-test samples, selected information was used for correspondence and sample
shipment throughout the study.

       The proficiency-testing service recorded the PTS Sample ID (Section 5.1 of Appendix A),
sample type, sample weight, sample shipped date, and sample received date on the Sample Tracking
and Analysis Report Forms.  They also prepared the report in Appendix A detailing the results of
verification testing and other information on the proficiency-test samples such as sample-to-sample
variation determined within the sample preparation stage.
                                            79

-------
       The participating clients recorded their own Client Sample IDs for cross-reference, dates of
sample shipment and receipt of analysis results, and the analysis results for the proficiency-test samples
on the Sample Tracking and Analysis Report Forms. Clients also could report protocol violations and
quality assurance issues to the proficiency-testing service when necessary.

6.2.2 Data Storage and Transfer

       Throughout the pilot study, the following hardcopy documents were stored by the organizations
responsible for their completion and reporting: completed telephone recruitment scripts, copies of the
Sample Tracking and Analysis Report Forms that were sent to the clients with the sample shipments,
and copies of these same forms as received from the clients via fax as they shipped the samples to the
laboratories and as they received the analytical results.

       Section 2.5 discusses how data were reported from the various organizations involved in the
pilot study and how these data were stored electronically.

6.2.3 Sample Identification

       Identifications were placed on proficiency-test samples at two distinct points in the double-blind
pilot study: when samples were prepared by the proficiency-testing service ("PTS Sample IDs") and
when samples were incorporated into regular field sample batches by the participating clients ("Client
Sample IDs").

       The method that the proficiency-testing service used to specify PTS Sample IDs is discussed in
Section 5.1 of Appendix A.  These sample IDs were placed onto labels which were affixed to plastic
bags. Then, the appropriate sample containers were placed in their appropriate plastic bags.  The
proficiency-testing service recorded the PTS Sample IDs  on Sample Tracking and Analysis Report
Forms and included the forms with the samples when shipping to the clients.

       When the proficiency-test samples were received  by the clients, they assigned Client Sample
IDs to the samples when placing them in a batch for shipment to the laboratory.  The identifiers were
assigned in a manner that the laboratory could not distinguish the proficiency-test samples from the
other field samples in the batch based on its ID or label.

       After recording the Client Sample IDs in Section B of the Sample Tracking and Analysis
Report Form next to the PTS Sample ID, the client removed the proficiency-test sample container from
the plastic bag and affixed a label containing the Client Sample ID onto the container. The type  of label
used by the client, the manner of recording Client Sample  IDs to the labels, and the manner of affixing
the labels to the sample containers were consistent  across all samples in the batch to ensure blindness of
this pilot study.
                                             80

-------
6.3    DATA QUALITY CHECKING

       The proficiency-testing service performed some verification of hand-entered data before
releasing the final spreadsheet of analytical results for statistical analysis.  The proficiency-testing service
also provided the originals of the Sample Tracking and Analysis Report Forms and laboratory reporting
forms that the participating clients provided. Once the organization performing the statistical analysis
received these materials, they performed  a 100% verification of the data in the spreadsheet for each
testing round, comparing the recorded results with what was recorded on the forms. Any deviation
from the forms was reported back to the proficiency-testing service for verification.  This process was
completed before preparing final versions of the results presented in Chapter 5.

       The organization performing the statistical analysis also notified the proficiency-testing service of
any results that appeared to be extreme (i.e., unusually high or low) relative to other results for the given
sample type, or relative to the target lead  level as determined from the reference labs in Round 22 of
the ELPAT Program.  The proficiency-testing service investigated the correctness of these extreme
data values by contacting the laboratories that analyzed the samples in question to obtain the analytical
results as reported on the laboratory report forms, and/or contacting the clients associated with these
samples to verify that they reported the results correctly and in the proper units.  The proficiency-testing
service provided all information obtained in this investigation to the organization performing the
statistical analysis.  Any necessary data corrections that were identified in this investigation into extreme
data values were made prior to generating the final data summaries and analyses presented in Chapter
5.
                                              81

-------
This page left blank intentionally.
              82

-------
7.0   CONCLUSIONS AND RECOMMENDATIONS

       The following findings and conclusions were made from this double-blind proficiency-testing
pilot study and from the summaries and analyses of the data generated by this study:

Design and participant issues

       •      It can be difficult to get clients of laboratories (e.g., risk assessors) to cooperate in a
               double-blind proficiency-testing program (through receiving and distributing
               proficiency-test samples and reporting back the results of the analysis of these samples)
               without providing some kind of incentive or reimbursement.

       •      Reimbursement for client participation in a double-blind program would require funding
               and a small amount of management staff effort to administer the funds.

       •      It is apparent that some laboratory clients may not recognize the benefits that a double-
               blind proficiency-test program would have for them, over and above current
               proficiency-testing programs.

       •      Many laboratory clients are small operations that do not have the cash flow or staff to
               contribute resources to a double-blind program, making their participation a hardship to
               them.  In some cases, materials (e.g., sample containers, PaceWipes™) were provided
               to clients in this pilot study to ensure their participation.

       •      Simple, yet explicit, instructions are necessary for the clients to ensure proper storage,
               handling, and identification of proficiency-test  samples while in their control. These
               factors  can affect the ongoing integrity of the samples.

       •      Frequently, clients go out of business, reorganize, change their organizational identity
               and/or mission, and change their telephone numbers and staff.  In addition, points of
               contact are frequently unavailable when needed (typically in the field).  This hinders the
               ability of clients to give a long-term commitment to a double-blind program, as well as
               the ability of a proficiency-testing service to contact participating clients during the
               course of a double-blind program.

       •      Several clients participating in this pilot study (and many who were attempted to be
               recruited) had a low monthly volume of field samples, making it difficult to generate a
               batch containing field samples and proficiency-test samples within a month of receiving
               the proficiency-test samples.  This problem may be especially acute in time periods
               when less environmental sampling occurs (e.g., winter months).
                                              83

-------
•      The finely-ground nature of paint-chip proficiency-test samples, along with the absence
       of substrate particles, make these samples easily distinguishable from field samples.
       Further research may be necessary to reformulate the physical characteristics of paint-
       chip proficiency-test samples.

•      As soil is recognized as a potential source of lead in indoor dust, soil sampling can play
       an important role in a risk assessment. However, while field soil samples typically
       exceed 10 grams, the proficiency-test soil samples in the ELPAT Program are typically
       less than 5 grams.  More study is needed to determine how soil proficiency-test
       samples can be prepared for  a double-blind program.

•      Currently, the use of PaceWipes™ or a similar hand-towelette in preparing dust-wipe
       proficiency-test samples makes it easy to distinguish them from field samples that
       consist of baby wipes, which many risk assessors use. This  can hinder the participation
       of clients who use baby wipes. However, it is expected that laboratory requests to use
       the smaller, thinner towelettes rather than baby wipes will result in more clients adopting
       towelettes for dust sampling.

•      Future double-blind programs need to take into account situations where proficiency-
       test sample results are reported as a lead amount per "unit area." When blindly
       including proficiency-test samples within a batch of regular field samples, the clients
       fabricated field sampling information associated with the proficiency-test samples to aid
       in their disguise. This fabricated information included sampled areas associated with the
       samples. This posed a problem when clients reported back the results of the
       proficiency-test samples in terms of a lead amount per unit area.  In order to convert
       this to a strict lead amount, it was necessary to verify the "area" that the client fabricated
       for this sample.

•      Page 5-33 of the 1995 HUD  Guidelines recommends that risk assessors use double-
       blinding techniques on their laboratories, where risk assessors obtain spiked dust-wipe
       samples (in the range of 50-300 jig/wipe) from laboratories and insert them into their
       field sample batches for analysis (at  1 spiked wipe per 50 samples). However, this is a
       recommendation and not a requirement. Except when double-blinding is mandatory
       (e.g., certain government programs such as the HUD Grantees program), the added
       resources that double-blinding requires on the part of the risk assessor often keep some
       from performing (and understanding) double-blinding  on a voluntary basis.  This issue
       must be addressed when determining the feasibility of a double-blind proficiency testing
       program.
                                      84

-------
Laboratory performance
               While all data for Round 22 of the ELPAT Program fell within the round's acceptance
               limits for the 12 participating laboratories in the double-blind pilot study, 11% of the
               double-blind pilot data for these laboratories exceeded these limits. This suggests that
               for some (but not necessarily all) of the participating laboratories and for each sample
               type, the double-blind pilot study data are more likely than single-blind study data to
               exceed the acceptance limits determined from data within the (single-blind) ELPAT
               Program.

               For each proficiency-test sample type, the overall average measurement reported by
               the laboratories within a double-blind testing round did not differ significantly across the
               three testing rounds (at the 0.05 level), and deviation of this average from the target
               level associated with the proficiency-test sample type was not statistically significant
               overall.

               Some statistical evidence exists that the variability in log-transformed dust-wipe
               proficiency-test sample measurements differs significantly (at the 0.05 level) across
               double-blind testing rounds, primarily due to the presence of unusually large or small
               data values.  Variability in the dust-wipe measures also tended to be higher in the
               double-blind testing rounds compared to Round 22 of the ELPAT Program. These
               observations were less evident for the paint chip proficiency-test sample measures.

               Analysis of the double-blind pilot study data characterized variability into two
               components: lab-to-lab variability and within-lab variability.  For both of these
               components, when statistical outliers were omitted from the analysis, the observed
               differences in the variance estimates did not differ significantly (at the 0.05 level) from
               round-to-round for any of the three proficiency-test sample types.  This finding, along
               with the conclusion made in the previous bullet, suggests that general laboratory
               performance did not differ significantly across the three double-blind testing rounds.

               When excluding outliers  from the analysis, lab-to-lab variability tended to represent
               approximately 70% of total variability associated with the two types of dust-wipe
               samples and slightly under 50% of total variability associated with  the paint chip
               samples. Thus, lab-to-lab variability constituted a greater percentage of total variability
               for the dust-wipe proficiency-test samples than for the paint chip samples.  These
               percentages were slightly higher when calculated within each double-blind testing
               round.

               When statistical outliers were not excluded from the analysis, the estimates of both lab-
               to-lab variability and within-lab variability differed significantly from one testing round to
               another (p < 0.001), for each type of proficiency-test sample.  However, as those
                                               85

-------
               laboratories that occasionally reported unusually high or low results did so for multiple
               samples of a given type within a testing round, the presence of statistical outliers
               affected lab-to-lab variability considerably more than within-lab variability, often
               resulting in lab-to-lab variability representing over 90% of total variability in the given
               testing round.

               When calculating laboratory averages within each double-blind testing round
               (disregarding statistical outliers) and the single-sample results from Round 22 of the
               ELPAT Program for the same group of laboratories and the same proficiency-test
               sample types, no statistically significant differences (at the 0.05 level) were observed in
               the variability of these data across the double-blind testing rounds and Round 22 of the
               ELPAT Program.  However, the observed variability associated with the double-blind
               laboratory averages was slightly higher than the observed variability associated with the
               single-sample results for these laboratories in Round 22 of the ELPAT Program.
               Because statistical theory specifies that averages have lower variability than the data
               entering into their calculation, this finding suggests that for these laboratories and
               proficiency-test sample types, the results of double-blind testing may have higher
               variability compared to the results of single-blind testing.

               The data for the group of laboratories participating in this pilot study suggest that
               additional variability may be present in double-blind testing data compared to single-
               blind testing data. However, while such a finding may influence how acceptance ranges
               in a double-blind program are determined, the criteria for determining acceptance in a
               double-blind program should not be relaxed simply because laboratories may be more
               likely to exhibit reduced performance compared to within a single-blind program.
               Instead, the criteria should address a laboratory's typical performance level in the test
               setting.
The following recommendations can be made as a result of conducting this study:

        •      An education strategy is needed for laboratory clients to recognize the benefits of a
               double-blind program.

        •      Consideration should be given on whether clients should be reimbursed for costs
               associated with proficiency-test sample analyses in a double-blind program. If the
               decision to reimburse is approved, sources of funding must be identified, such as
               charging laboratories a fee for participating in a double-blind program.

        •      In a double-blind proficiency-testing program, it may be beneficial to have proficiency-
               test samples placed in small batches rather than large batches.  If the proficiency-test
               sample results are considerably different from those for other samples within the batch
                                               86

-------
(e.g., are very high), then some laboratories may suspect that the samples are some
kind of reference material. This is more likely to happen in large batches than in small
batches, especially if the lead content is consistent from sample to sample within a
batch.

Statistically-based laboratory performance criteria in a double-blind program may need
to consider that lab-to-lab variability in proficiency-test sample results may naturally
differ between a double-blind setting and a single-blind setting, as some laboratories
may perform differently when aware of analyzing proficiency-test samples.

If statistical evaluation criteria in a double-blind program will be made based on
individual sample results (as is done in the single-blind program), then within-laboratory
variability in these results should be considered, in addition to lab-to-lab variability.

Risk assessors can use several different types and brands of wipes for collecting dust
samples for lead analysis.  Therefore, it is necessary to work with laboratories, their
clients, and other interested agencies to standardize the type of dust-wipe that should
used in lead inspections and risk assessments, so that the same type of wipe can be
used to develop proficiency-test samples.

Further research should be considered to develop more appropriate paint materials that
can be used in proficiency-test samples and that more closely resemble paint samples
collected in the field than the finely-ground material used in this pilot study.  A similar
recommendation can be made for soil samples, which were not considered in this pilot
study.

When laboratory proficiency-test results are suspected to have been reported
inaccurately, and it is determined that the laboratory client did not contribute to the
inaccuracy, it may be necessary for the proficiency testing service to contact the
laboratories to resolve any issues contributing to the inaccuracy, such as by reviewing
laboratory QA/QC results for the given batch (e.g., results of analyzing laboratory
control samples), and to determine whether the client's field sample results within the
same batch may have been compromised in any way.
                                87

-------
This page left blank intentionally.
              88

-------
8.0    REFERENCES

"EPA Seeks Lab Clients for 'Double Blind' Test Program."  Lead Detection and Abatement
       Contractor, January 1998,'Newsline' column, page 2.

NIOSH (1994) Laboratory Evaluations and Performance Reports for the Proficiency Analytical
       Testing (PAT) and Environmental Lead Proficiency Analytical Testing (ELPAT)
       Programs. Prepared by the National Institute for Occupational Safety and Health, Centers for
       Disease Control and Prevention, November, 1994.

Rosner, B. (1983) "Percentage Points for a Generalized BSD Many-Outlier Procedure."
       Technometrics 25(2): 165-172.

RTI (1994) Environmental Lead Proficiency Analytical Testing Program:  Quality Assurance
       Project Plan, Revision No. 1. Prepared by the Center for Environmental Measurements and
       Quality Assurance, Research Triangle Institute, for the American Industrial Hygiene
       Association, 3 May 1994.

Schlecht, P.C., Groff, J.H., Feng, A., and Song, R. (1996) "Laboratory and Analysis Method
       Performance of Lead Measurements in Paint Chips,  Soils, and Dusts." American Industrial
       Hygiene Association Journal 57:1035-1043.

Snedecor, G.W., and Cochran, W.G. (1989) Statistical Methods, Eighth Edition. Ames, IA: Iowa
       State University Press.

USEPA (1995) "A Field Test of Lead-Based Paint Testing Technologies: Summary Report."  Office of
       Prevention, Pesticides,  and Toxic Substances, U.S. Environmental Protection Agency, EPA
       747-R-95-002a, May  1995.

USHUD (1995) "Guidelines for the Evaluation and Control of Lead-Based Paint Hazards in Housing."
       Office of Lead Hazard Control, U.S. Department of Housing and Urban Development. HUD-
       1539-LBP.
                                           89

-------
APPENDIX A

-------
Report on the Preparation and Distribution of Samples
                          for the
         Double-Blind Proficiency Testing Pilot
                          in the
   National Lead Laboratory Accreditation Program
                        Prepared for

                      Fred I. Grunder
             American Industrial Hygiene Association
                2700 Prosperity Ave., Suite 250
                     Fairfax, VA 22031
                        Prepared by

       Laura L. Hodson, David A. Binstock and Ann R.Turner
              Environmental Chemistry Department
    Center for Environmental Measurements and Quality Assurance
                         June 1998
                           A-l

-------
                                       SECTION 1.0
                                     INTRODUCTION
1.1    PROJECT OVERVIEW

       This report describes the preparation and distribution of lead-containing dust and paint samples
for a pilot version of a double-blind proficiency testing program. The double-blind program is currently
being considered for inclusion within the National Lead Laboratory Accreditation Program (NLLAP)
operated by the U.S. Environmental Protection Agency to test the ability of analytical laboratories to
analyze dust, soil, and paint chip samples for lead content.

       The  current laboratory proficiency testing program within NLLAP is the Environmental Lead
Proficiency Analytical Testing (ELPAT) program which is administered by the American Industrial
Hygiene Association (AIHA).  Since 1992, AIHA has contracted with Research Triangle Institute
(RTI) to produce lead-containing paint, soil and dust wipe samples for distribution to laboratories
wishing to participate in the ELPAT program.  The ELPAT program is a single-blind proficiency testing
program, so while the participating laboratories are unaware of the amount of lead in samples that are
part of the program's performance evaluation,  they are aware of when they are analyzing such  samples.

       This double-blind study will have the field inspectors (clients) receiving lead-in-dust and lead-
in-paint samples from RTI, and incorporating these proficiency-testing samples as blind samples within
batches of field samples submitted by the clients of the laboratories. Since these proficiency samples
would not be identified as such in these batches, laboratories would ideally treat these samples as
routine field  samples.

       To accomplish such a task, RTI prepared additional Round 22 ELPAT materials, and mailed
these samples to participating clients. Samples were provided to the clients three times over a three
month period.  It was the clients responsibility to report the laboratory analysis results back to AIHA.

1.2    PROJECT ORGANIZATION AND MANAGEMENT STRUCTURE

       The  double-blind pilot program was sponsored by the Office of Pollution Prevention and
Toxics of the U.S. EPA (EPA/OPPT) under Work Assignment 3-30 of EPA Contract Number 68-
D5-0008.  Mr. John  Scalara was the EPA Work Assignment Manager.  Three contractors were
involved in the design and conduct of this program.  Battelle is under contract by the EPA, AIHA was a
subcontractor to Battelle and RTI was a subcontractor to AIHA.

       Battelle had responsibility for the establishment of the program design, preparation of the
Quality Assurance Project Plan, recruiting participants, establishing statistical performance criteria,
preparing SOPs, letters and forms to be used in the pilot study, and preparing a final report for EPA.
                                            A-2

-------
       AIHA was responsible for serving as the proficiency testing service, acting as the client's
primary contact, collecting the analytical results and providing them to Battelle, and providing data on
the ELPAT Round materials.

       RTI was responsible for preparing and storing the proficiency test materials used in the double-
blind study, receiving the clients sample containers and testing these containers for any possible lead
contamination, conducting verification procedures on sample transfers among storage devices and
centrifuge tubes, distributing samples to clients using the client-supplied containers, (or a default
centrifuge tube if the client did not supply a container), coordinating the method for ensuring blindness
when the clients incorporate these samples with their regular field samples, and reporting information on
the test samples that is necessary for determining laboratory performance.

       This report describes the preparation and distribution of the double-blind samples.
                                              A-3

-------
                                      SECTION 2.0
                   PROFICIENCY TESTING MATERIAL SELECTION

2.1    MATERIAL SOURCES

       The low-lead dust used in this study came from a Milwaukee exposure intervention project and
the medium-lead dust came from a North Carolina household. The paint came from an old hospital in
Raleigh, NC.  The double-blind samples were prepared from the same bulk processed materials as
those used for Round 22 in the ELPAT program. The double-blind samples were prepared
immediately preceding Round 22.

2.2    SCREENING ANALYSIS

       The dusts were sent to Neutron Products, Inc., in Dickerson, MD for sterilization by gamma
irradiation and then returned to RTI. Upon receipt at RTI, the bags of sterilized raw dust were sieved
using a 250-um sieve, and three 0.1 g aliquots were taken from the sieved dust and subjected to
analysis using microwave/acid digestion and inductively coupled plasma emission spectroscopy1. This
concentration was the screening value for each dust.

       The paint was ground using a ball-mill jar, and three 0.1 g aliquots were subjected to the lead
analysis as described for the dust samples. The lead concentration was the screening value for the
paint.

2.3    TARGET CONCENTRATION

       The QAPP for the double-blind project (written by Dr. Bob Lordo et al., of Battelle, Feb.
1998) specified that the dust-wipe proficiency-test samples will have lead levels in each of the following
ranges  : 70-120 wg/wipe (Low-level dust); and 200-600 wg/wipe (Md-level dust); and that the paint
chip sample be in the range of 0.2-1.2 percent lead.  Samples were then  selected from the Round 22
ELPAT proficiency samples to achieve these target values. The following ELPAT samples were
selected: 22W2 (Low-level dust), 22W3 (Mid-level dust), and 22P4 (Paint).
                                           A-4

-------
                                       SECTION 3.0
                   PREPARATION OF DOUBLE-BLIND MATERIALS
3.1    PRELIMINARY VERIFICATION

       The bulk dust samples, having been sieved to 250 um for screening analysis, were then passed
through a 150 • m sieve using the Ro-Tap apparatus in preparation for preliminary verification analysis.
The paint having been previously ground using the ball mill jar was passed through a 125 • m sieve.

       Five 0.1 g grab samples of each dust and paint were then taken manually from each batch of
the processed materials, and analyzed using the microwave/ nitric acid, hydrochloric acid extraction and
measurement by plasma emission spectroscopy.  The preliminary verification values as presented in
Table 1 came within the acceptable range of the target concentrations.

                                          Table 1
                           Screening and Preliminary Verification
Sample
Low-level
Dust
Mid-level
Dust
Paint
Target
Value
70-120
ug/wipe
200 - 600
ug/wipe
0.2- 1.2%
ELPAT
Number
22W2
22W3
22P4
Screening
Value
1180-g/g
2270 • g/g
0.642 %
Preliminary
Verification
118±5.04
• g/wipe
243 ± 8.44
• g/wipe
0.602 ± 0.044 %
Source
Milwaukee,
WI
Intervention
Program
North
Carolina
Household
Raleigh, NC
Hospital
3.2    SAMPLE LOADING

       PaceWipes™ were loaded with 0.1000 ± 0.0005 g portions of dust. Only one analytical
balance was used to weigh the dust onto the PaceWipes™ and a single 0.1 g Class 1 weight was used
for a daily calibration check. The dust jar was first tumbled, then allowed to settle briefly and the
container was opened.  Material was taken using a spatula and transferred to the tared weighing paper.
If more material was needed, it was taken from the bulk container and added to the material on the
paper.  If excess (>0.1005 g) was placed on the weighing paper, it was carefully removed with the tip
of the spatula and discarded. A PaceWipe™  was prepared for receiving the dust by opening  the foil
pouch, removing the wet folded wipe and squeezing the excess moisture out by hand over a trash can.
                                           A-5

-------
The wipe was then unfolded and briefly set on a Kimwipe to soak up excess moisture.  The
PaceWipe™ was then transferred to a flat plastic board to await the dust.  The weighing paper
containing the pre-weighed dust was then removed from the balance and the dust gently tapped out
onto the PaceWipe™ . The wipe was then folded and placed in a capped, plastic scintillation vial. All
vials containing the spiked wipes were stored in a cold room (4O F) as a secondary means of
retarding mold growth until shipment.

       The paint was riffled out  into  1.0 gram aliquots using a spinning riffler. A 20 gram portion was
introduced into the hopper of the riffler, and the paint was slowly vibrated down a chute leading from
the hopper to 20 trays slowly turning  under the end of the chute. After all of the 20 gram portion was
split into 20 samples, the trays were removed and the paint samples transferred to plastic scintillation
vials.  The process was repeated 9 times to obtain 180 samples.

3.3    FINAL VERIFICATION
       After the vials were filled with the appropriate dust or paint samples, they were returned to their
positions in divided boxes holding 10 rows of 10 vials each.  Samples were selected for final
verification across the entire set of samples at a rate of 5% of the total number of samples. One dust or
paint sample was selected at random from each batch of 20 samples, for a total of 9 samples from each
set of 180 dusts and paint.  The final verification values are presented in Table 2.  Values within 20% of
the target values were achieved.

       Homogeneity of the samples is indicated by the relative standard deviations (RSD). The RSDs
based upon analysis of 9 samples met the goal of a RSD less than  10%.

                                           Table 2
                                      Final Verification
Sample
Low-level Dust
Mid-level Dust
Paint
Final Verification
(n=9)
101 ±5.29* g/wipe
215 ± 10.4* g/wipe
0.658 ± 0.029 %
Relative Standard
Deviation (RSD)
5.24
4.86
4.40
                                            A-6

-------
                                      SECTION 4.0
                                 QUALITY ASSURANCE

4.1    QUALITY CONTROL OPERATIONS

       Quality control/quality assurance is an essential component of the ELPAT program, and
was continued for the double-blind study. An earlier study to determine the blank value of the
PaceWipe™ (Lot number 1296-01) showed that 13 blanks all contained <0.001 mg lead/wipe. Three
more blank wipes which had been placed in plastic scintillation vials for two hours and stored at 40 °F
prior to removal and transfer to the centrifuge tube, were subsequently analyzed. The blank recoveries
for these three samples were also <0.001 mg lead/wipe.

       A duplicate analysis was conducted for each of the dusts and paint.  The duplicates agreed
within 5 %, as shown in Table 3.  Spike solutions were prepared from a 1000-wg/ml stock solution of
Pb(NO3)2 obtained from PE Pure, Atomic Spectroscopy Std and added to the dust samples prior to
digestion. One ml of a 50 wg/ml Pb solution was added to the low-level dust and one ml of a 100 wg/ml
Pb solution was added to the mid-level dust prior to digestion. The paint solution concentration was
too high to effectively add a spike solution prior to digestion; therefore one ml of a 50 wg/ml spike
solution was added to the diluted paint solution following digestion. The spike recoveries for the dusts
and paint were within the goal of 90-110, as shown in Table 4.
                                          Table 3
                                    Duplicate Analysis
Sample
Low-level Dust
Mid-level Dust
Paint
Vial ID
462/482
471/491
465
Initial
104 • g/g
210 • g/g
0.676 %
Repeat
102 • g/g
218- g/g
0.649 %
Percent
Difference
1.9%
3.8%
4.4 %
                                           A-7

-------
                                         Table 4
                                Results of Spike Analysis
Sample


Low-level
Dust
Mid-level
Dust
Paint
Added
Amount (• g)

50

100

50
Amount
Recovered
(•g)
45.6

92.4

50
% Recovery


91.2

92.4

100.0
Unspiked
Amount (• g)

101

214

32.0
4.2    QUANTIFICATION OF DUST-WIPE SAMPLE TRANSFER

       To verify the quantitative transfer of dust-wipe samples from the scintillation vials to the client-
supplied sample containers, seven aliquots of MST SRM 2711 were weighed and transferred to the
PaceWipes™ as described for the dust loading. These scintillation vials were placed in the cold room
overnight, then the wipes were transferred from the scintillation vials to the centrifuge tubes used for the
analysis. The wipes were then digested, analyzed and the results compared to the historic recovery of
lead from SRM 2711, which is nominally 85%. The recovery of these seven wipes was 83.3 ± 1.6 %
which can be compared to the Round 22 SRM 2711 recovery of  82.6 - 86.0%.

4.3    CLEANLINESS DETERMINATION OF CLIENT-SUPPLIED CONTAINERS

       Each client supplied an extra container which was tested for lead contamination by swabbing
the interior of the container with a PaceWipe™. These PaceWipes were subsequently analyzed for
lead contamination.  As can be seen on Table 5, all containers contained <0.001 mg lead/container.
                                           A-8

-------
                Table 5
Cleanliness of Client-Supplied Containers
Client ID
A
B
C
D
E
F
G
H
I
J
K
L
Container Type
Glass Jars
Centrifuge Tubes
Plastic Bags
Plastic Bags
Plastic Bags
Centrifuge Tubes
Plastic Bags
Centrifuge Tubes
RTI Centrifuge Tubes
Centrifuge Tubes
Plastic Bags
Plastic Bags
Centrifuge Tubes
Lead from Container
<0.001 mg
<0.001 mg
<0.001 mg
<0.001 mg
<0.001 mg
<0.001 mg
<0.001 mg
<0.001 mg
<0.001 mg
<0.001 mg
<0.001 mg
O.001 mg
<0.001 mg
                 A-9

-------
                                       SECTION 5.0
                           PACKAGING AND DISTRIBUTION

5.1    SAMPLE IDENTIFICATION

       RTl assigned a sample identification number (PTS Sample ID) to each sample using the
following form:
                                       R-MM-L-NNN

where  R is a single-digit indicator of the testing round in which the sample was to be analyzed (R=l, 2
       or 3),
       MM was a two-letter indicator of the sample matrix and lead level (DL for dust/low, DM for
       dust/medium, PT for paint),
       L was an unique identifier of the laboratory to which the batch was sent by the client (L=l, 2,...;
       L=l if the client will be sending  samples of the given matrix/lead level to only one laboratory),
       and
       NNN was a three-digit sequential ID number that uniquely identifies each sample having
       common values for R and MM  (example : 001-999).

        The PTS Sample ID was copied onto standard self stick labels which were subsequently
attached to the inside of a Ziplock™ bag.  The labels were placed on the inside of the bags so that in
the event the label fell off, it would still be associated with the sample bag.  The  double-blind dust
and/or paint was transferred to the client-supplied containers and the containers placed into the pre-
labeled plastic bags.

       After assigning sample labels to the sample containers, RTI recorded the PTS Sample ID on a
copy of the Sample Tracking and Analysis Report Form (Figure 1). The sample matrix of each sample
type (dust or paint),  the weight of the sample placed into the sample container, and the date each
sample was shipped  to the clients was recorded. One copy of this form was made for retention at RTI,
and the original was  sent with the samples to the clients.

5.2     DISTRIBUTION

       A total of 14 clients participated. The original mail-out of materials to 10 clients was on
February 27, April 3, and June 1, 1998.  Four clients joined the program late and received materials in
April, May and June.

       A copy of the Material Safety Data Sheet (MSDS), (Appendix A) for the lead containing
materials was sent once to each client along with the instruction that it was for the clients use only and
not to forward the MSDS with the double-blind samples to their laboratories.
                                           A-10

-------
Testing Round:
                      Page     of
                           NLLAP Double-Blind Proficiency Testing Pilot Program
                                 Sample Tracking and Analysis Report Form
               Client Information: (1)
Laboratory Information: (2)
Client:
Address:
City:
Telephone:
State: 1 Zip:
Fax:
Responsible Party:
Lab: | Lab ID#:
Address:
City:
Telephone:
State: 1 Zip:
Fax:
Responsible Party:
                                                                       Analysis Method: (17)
                                                        Paint: D ICP  D GFAAS D FAAS  D Other
                                                        Dust:  D ICP  D GFAAS D FAAS D Other
 Section A: Sample Distribution from Proficiency-Testing Service (PTS) to the Client
PTS Sample ID
(3)









Sample Matrix (4)
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
Sample
Weight
(grams) (5)









Date Shipped
to Client
(mm/dd/yy)
(6)









Initials
(7)









Date Received
by Client
(mm/dd/yy) (8)









Initials (9)









 Section B: Client Cross-Reference and Shipment to Laboratory
PTS Sample ID (10)









Client Sample ID
(11)









Initials
(12)









Date (mm/dd/yy)
(13)









Date Shipped to
Laboratory
(mm/dd/yy) (14)









Initials (15)









Section C: Laboratory Results
                                                  A-ll

-------
Client Sample ID (16)









Reported Lead
Concentration
(18)









Units of
Concen-
tration
(19)









Date Results
Received by
Client
(mm/dd/yy) (20)









Initials
(21)









Verification
(22)









Date
(mm/dd/yy)
(23)









Numbers in parentheses in the column headings refer to citations in the Standard Operating Procedure for Proficiency Sample
Tracking and Data Reporting.

                                       SECTION 6.0
                                       REFERENCES

1.     Binstock, D.A., D.L. Hardison, P.M. Grohse, and W.F. Gutknecht, "Standard Operating
       Procedures for Lead in Paint by Hotplate- or -Microwave-based Acid Digestions and Atomic
       Absorption or Inductively Coupled Plasma Emission Spectrometry." NTIS Publication N. PB
       92-114172, EPA Contract No. 68-02-4550, September 1991.
                                            A-12

-------
        APPENDIX A




MATERIAL SAFETY DATA SHEET
           A-13

-------
APPENDIX B

-------
                NLLAP Double-Blind Proficiency Testing Pilot Program

               Script for Telephone Recruitment of Laboratory Clients
 State:
 Client name:                                        Client ID #:
 Telephone number:
 Date of call: 	           Time of call: 	 AM PM

 Caller:
Hello, this is (state your name). I understand that (you/your organization) send environmental samples
to analytical laboratories to evaluate whether a home contains lead hazards. Who can I speak with
who is responsible for overseeing the collection of dust or paint samples in homes and the shipping of
these samples to a laboratory?

Obtain the contact name:
Obtain the title/position of contact:  	

Continue once the contact is on line.

Hello, my name is (state your name).  I (work for/am a contractor to) the EPA's Office of Pollution
Prevention and Toxics in Washington, DC. I understand that in the process of evaluating whether lead
hazards are present in a home, you collect dust or paint chip samples from the home and send them to
an analytical laboratory to determine the amount of lead in these samples.

Question #1:   Is this true?    D   No       Terminate the call
                            D   Yes      Continue

Then you may be familiar with EPA's National Lead Laboratory Accreditation Program, or NLLAP.
NLLAP recognizes analytical laboratories which have demonstrated and meet the minimum standards
for analysis of lead in dust, soil, and paint chips.  The EPA is currently investigating whether to
implement a new procedure within the NLLAP which would improve the way by which laboratories
                                           B-l

-------
are tested on their ability to analyze and report amounts of lead in environmental samples, and you may
be able to help us in this study.

Question #2: Would you mind if I fill you in very briefly on this new procedure?

              D   No       Proceed             D  Yes       Terminate the call

Currently in the NLLAP, approximately every three months, a proficiency testing service sends a
special batch of proficiency-test samples to a laboratory, then the laboratory analyzes these samples
and reports back  the results. The proficiency testing service knows how much lead is in  each sample,
so they can compare what the laboratory reports with what is actually in the sample. However, one
disadvantage to this procedure is that the laboratory knows that they are being evaluated based on the
results they report for these special samples. The evaluation would be better if the laboratory did not
know when they  were analyzing these proficiency-test samples.

The new procedure which EPA is considering would have the proficiency testing service send the
proficiency-test samples to clients of the laboratories, rather than directly to the laboratories.  The client
would place these samples within batches of their own field samples and send the batches off to the
laboratory for analysis.  The proficiency-test samples would remain anonymous within the batches, and
so the laboratory would not know that they are testing proficiency-test samples. This approach is
called a double-blind approach and leads to a more accurate evaluation of a laboratory's routine
performance in analyzing samples that they receive from their clients.

Here's where you come in.  We need to identify clients of NLLAP-accredited laboratories who would
be willing to participate in a pilot study that will evaluate the feasibility of developing a double-blind
program. Each client in this study will receive no more than 18 proficiency-test samples  (perhaps less)
free of charge around the end of February.  They would be given explicit instructions on how to
incorporate these samples into three batches of their field samples, submit these batches to the
laboratory, and report the results of the proficiency-test samples back to us.  These clients would be
involved in the study for about three months, and it would take a minimal amount of effort on their part
to participate.

Question #3: Does this sound like  a study that you may be interested in participating in?

                      D   Yes       Proceed             D   No      Terminate the call

In order to determine which role, if any, you may be able to play in this pilot study, I need to ask you a
few questions about the types and numbers of field samples you submit to a laboratory for analysis.

Question #4: First, do you send either dust-wipe or paint chip samples to an NLLAP-accredited
laboratory for analysis?  D   No       Terminate the call
                             D   Yes      Proceed

                                             B-2

-------
Question #5a: Do you currently run a double-blind test of your own on an NLLAP-accredited
laboratory?    D   No        Go to Question #6a
              D   Yes       Proceed

Question #5b: In the double-blind test, do you submit blank samples to the laboratory, or do you spike
samples with known amount of lead prior to sending the samples to the laboratory?
       D  Submit blank samples                  Go to Question #6a
       D  Submit spiked samples, or don't know    Proceed

Question #5c: Are there any NLLAP-accredited laboratories that you contract with in which you do
not run double-blind tests?           D  No       Terminate the call
                                  D  Yes      Proceed

In the remaining questions, we are interested in only NLLAP-accredited laboratories that you send
samples to for which you do not run double-blind tests.
                                    Dust-Wipe Samples

Question #6a: Do you send at least one batch of dust-wipe samples every month to an NLLAP-
accredited laboratory?        D No  Proceed to Question 7a     D  Yes

Question #6b: Approximately how many dust-wipe samples
do you place in a typical batch?  	

Question #6c: Can you tell me the type or brand of dust wipe you use in the field? If they don't
know, ask if it is a hand-tow elette versus a baby wipe.
Question #6d. What type and brand of sample container do you place the dust-wipe sample in when
sending the sample to the laboratory?
Question #6e:  In general, do you use only one dust-wipe to collect a dust sample, or do you use more
than one wipe to collect a single sample?

       D  Use only one wipe per sample    D  Use multiple wipes per sample

-------
                                    Paint Chip Samples

Question #7a: Do you send at least one batch of paint-chip samples every month to an NLLAP-
accredited laboratory?  D No  If Question #6a was yes, then skip Questions #7b and#7c
                           Otherwise, terminate the call
                           D Yes

Question #7b: Approximately how many paint-chip samples
do you place in a typical batch?  	

Question #7c: What type and brand of sample container do you place the paint chip sample in when
sending the sample to the laboratory?
Question # 7d: Do you prepare the paint-chip sample in any way prior to placing them in sample
containers for shipment to the laboratory, such as grinding them to a powder?

       D  No             D  Yes
                                       Laboratory

Question #8a: What are the names and locations of NLLAP-accredited laboratories that you send
dust-wipe and/or paint chip samples to on at least a monthly basis? Note that these should not be
laboratories that the client is currently sending double-blind samples to already.

       Lab #1: 	

       Lab #2: 	

       Lab #3: 	

If the answers to Question #6a and#7a are both yes, then ask Question #8b.  Otherwise, skip to
Question #8d.

Question #8b: Do you submit both dust-wipe and paint chip samples to the same laboratory?

       D   No      Skip to Question #8d         D   Yes      Proceed

Question #8c: To which labs do you send dust-wipe samples, and to which do you send paint chip
samples? Respond according to responses to Questions #6a and # 7a.
                                          B-4

-------
       Dust-wipe samples:   D  Lab #1    D  Lab #2    D  Lab #3
       Paint-chip samples:   D  Lab #1    D  Lab #2    D  Lab #3

Question #8d: Do you know approximately how many samples the laboratory tests at a given time,
like per month? Respond according to responses to Questions #6a and # 7a.  We want to get a
basic idea on whether the lab is large or small.

       Lab #1: Dust-wipe	Paint-chip	

       Lab #2: Dust-wipe	Paint-chip	

       Lab #3: Dust-wipe	Paint-chip	
Question #8e: Do you know the analytical method used by the laboratory to analyze the samples?
Respond according to responses to Questions #6a and #7a.

       Lab #1: Dust-wipe	Paint-chip	

       Lab #2: Dust-wipe	Paint-chip	

       Lab #3: Dust-wipe	Paint-chip	
Question #8f.  Do you know the analytical method's detection limit for lead? Respond according to
responses to Questions #6a and # 7a.

       Lab #1: Dust-wipe	Paint-chip	

       Lab #2: Dust-wipe	Paint-chip	

       Lab #3: Dust-wipe	Paint-chip	
Those who participate in the pilot study will be required to submit as many as 30 empty, unused sample
containers to the proficiency testing service in which they will place the proficiency-test samples.

Question #9:  Would you agree to do this if you were involved in this study?

                     D   Yes                    D   No

Thank you very much for your time and for your interest in this pilot study. We will get back to you by
the end of the month on whether or not you will be selected to participate in this study. Could I get
your address?

Street Address: 	

                                           B-5

-------
City: 	   State: 	  Zip Code:
Verify name of contact.
                                          B-6

-------
APPENDIX C

-------
January 26, 1998

(Name of Contact)
(Name of Business)
(Address)
(City, ST ZIP)

Dear (Name):

Thank you for expressing an interest in participating in a double-blind proficiency-testing pilot program
being conducted by the U.S. Environmental Protection Agency.  We are happy to inform you that
you have been selected to participate in this program!

As we discussed with you on the telephone, the primary objective of this pilot is to evaluate the
feasibility of implementing a double-blind program as a supplement to the current single-blind laboratory
proficiency-testing program within the National Lead Laboratory Accreditation Program (NLLAP). In
a double-blind program, the proficiency-testing service would supply proficiency-test samples to the
clients of laboratories, rather than directly to laboratories as a single-blind program does.  The clients
would then incorporate these samples within batches of field samples, submit the batches to the
laboratories, and report the analytical results of the proficiency-test samples back to the proficiency-
testing service.  Therefore, unlike a single-blind program, a double-blind program does not allow the
laboratories to know when they are analyzing proficiency-test samples, thereby allowing their routine
performance to  be more accurately measured.

The attachment  to this letter provides instructions for you to follow as a participant in this pilot program.
Through the course of the program, we will be providing you with a total of (specify: #) dust-wipe
samples and (specify: #) paint chip samples for you to submit to (specify: name of laboratory
here) for analysis. We will provide you with these samples over three testing rounds: in March,
April, and June, 1998. Within each testing round, we will ask you to incorporate the samples you
receive into the next available batch of field samples that you will submit to (specify:  this laboratory,
these laboratories) for analysis

In order to ensure that the laboratory cannot discern the proficiency-test samples from the field
samples, we need to place the proficiency-test samples in the same containers that you will use for the
field samples  Therefore, please submit (specify: #) empty, unused sample containers by
February 13,1998, to Ms. Laura Hodson at the address specified on the attachment, so that we can
place our samples in the same containers you will be using.

During the pilot program, please regard all aspects of the program as confidential and program-
sensitive. You will be able to find out the results of the analysis of these proficiency-test samples at
the  end of the pilot program, upon release of the program's final report.

                                            C-l

-------
(Name of contact)                              2                              January 26, 1998

We will be contacting you by telephone in the next week to review the instructions in this letter and on
the attached sheet and to verify your participation in the program. We look forward to talking with you
again and answering any questions that you may have.  Meanwhile, please call Bob Lordo of Battelle at
(614) 424-4516 if you have any questions on your acceptance in the program. Thank you again for
your participation!

Sincerely,
John Scalera
Office of Pollution Prevention and Toxics
                                             C-2

-------
                   NLLAP Double-Blind Proficiency-Testing Pilot Program

               Information and Instructions to Participating Laboratory Clients
                                         REVISED

Thank you for agreeing to participate in this double-blind proficiency-testing pilot program! The
information that we can gather in this program will be very useful in developing a double-blind program
within the NLLAP.  Such a program will eliminate the need for laboratory clients to perform their own
double-blind procedures, which will greatly benefit all lead inspectors and risk assessors who  submit
environmental samples to laboratories for lead analysis.

During the pilot program, you will interact with the program's proficiency-testing service who will
manage the shipping of proficiency-test samples and the collecting of analysis results on these samples.
Your primary contact at the proficiency-testing service on any questions you may have,  issues that
arise, and quality assurance issues to report, is:

       Fred Grunder, Cffl
       Manager, Laboratory Accreditation Programs
       American Industrial Hygiene Association (AIHA)
       2700 Prosperity Avenue, Suite 250
       Fairfax, VA 22031
       phone:  703/849-8888
       fax: 703/207-3561
       e-mail: fgrunder@aiha.org

If you are unable to reach the primary contact, you may contact Mr. Carl Bell  at AIHA (at the same
address and telephone numbers as above), or the NLLAP staff at the U.S. Environmental Protection
Agency at 202/260-6709.

The proficiency-testing service will provide you with proficiency-test samples in each of three testing
rounds:  in March, April, and June, 1998. This attachment provides you with information such as
how to place the proficiency-test samples for a given testing round within your next available batch of
field samples, how to properly track these samples while in your possession, and how to report the
results of these samples to the proficiency-testing service.

Although we are unable to reimburse you for shipping/postage costs and for costs to supply the
proficiency-test service with sample containers, we will be happy to reimburse you for the cost of
laboratory analysis of the proficiency-test samples we supply to you in this program. To receive
reimbursement of the analysis costs, please provide a copy of the laboratory's invoice, showing either
1) the cost per sample, or 2) the number of samples analyzed and the total analysis cost, to:

       Robert Lordo, Ph.D.
       Battelle

                                            C-3

-------
       505 King Avenue
       Columbus, OH 43201
You can also fax this information to Dr. Lordo at 614/424-4516.

In order to preserve the double-blind nature of this pilot program, while ensuring proper sample
tracking and data reporting, we have identified seven primary tasks for you to follow in the program.
Each task will require only a minimal effort on your part.  You need perform Task 1 only once; Tasks 2
through 6 will be accomplished three times, once in each testing round.
Task 1:    Please provide clean, unused sample containers in which the proficiency-testing service
          will place your proficiency-test samples. The number of containers to submit is specified in
          the letter accompanying these instructions. Please submit these containers via priority mail
          or first-class mail by February 13,1998, to the following address:

                 Laura Hodson, CIH
                 Center for Environmental Measurements and Quality Assurance
                 Research Triangle Institute
                 Research Triangle Park, NC 27709

          Within your shipment of the sample containers, please notify us on the proper address
          that we should use in submitting the packages of proficiency-test samples to you.

Task 2:    Each shipment of proficiency-test samples that you will receive will be accompanied by the
          attached Sample Tracking and Analysis Report Form., with one copy included for every
          laboratory to whom you will be shipping the samples for analysis. This form is used to
          properly track the proficiency-test samples through the program and to report the analytical
          results of these samples.

          C.     When you receive the proficiency-test samples, please store them in a locked area
                 with limited access (e.g., cabinet, closet) until you place them within your next
                 available batch of field samples.
          D.     Do not open or otherwise tamper with the sample containers or their contents.
          E.     Please store the accompanying Sample Tracking and Analysis Report Forms in a
                 locked area when not in use.

Task 3:    For the Sample Tracking and Analysis Report Forms that you receive,

          A.     Please review the information which the proficiency-testing service provides
                 in the block titled "Client Information (1)" and make any necessary changes.
                 (Note that the proficiency-testing service will also provide sample information in
                 columns (3) through (7) on the form.)
                                           C-4

-------
          B.     In the upper right of the form, within the block labeled "Laboratory Information
                 (2)", please supply information on the specified laboratory that will receive the
                 proficiency-test samples for analysis.
          C.     Please specify the date you received the proficiency-test samples in the
                 column of the form labeled "Date Received by Client (8)" and place your initials
                 (indicating that this date was recorded correctly) in the column labeled "Initials (9)".

Task 4:    Upon your receipt of proficiency-test samples in a given testing round, please use the
          Sample Tracking and Analysis Report Forms to identify which samples are to go to
          which laboratories for analysis (if you are to submit samples to multiple laboratories).  Then,
          place a laboratory's proficiency-test samples randomly within your next available batch of
          field samples that are earmarked for analysis at that laboratory. To ensure that the
          proficiency-test samples remain properly identified through the program, please perform the
          following while treating the proficiency-test samples no differently from the field
          samples:

          A     Please assign identifications (IDs) to all samples in the batch
          B.     For each proficiency-test sample, please record the PTS  Sample ID (i.e., the ID
                 specified on the label when you receive the samples) in the column labeled "PTS
                 Sample ID (10)" within Section B of the Sample Tracking and Analysis Report
                 Form, and the ID which you assign to the sample in the column labeled "Client
                 Sample ID (11)."  When recording these two IDs, please ensure that the two IDs
                 on a given row of the table are for the same sample.
          C.     For each proficiency-test sample, please remove the sample container from the
                 outer plastic bag, noting the PTS Sample ID that is on this outer bag, then
                 place a label on the  sample container containing the ID you assign to the
                 sample.
          D.     Once you have verified that each proficiency-test sample is properly identified in the
                 batch and on the Sample Tracking and Analysis Report Form, please initial and
                 date the  columns labeled "Initials (12)" and "Date (13)".

Task 5:    Please perform the following when shipping a batch of field and proficiency-test samples to
          the laboratory for analysis:

          A.     Please include any forms you would routinely send with the batch, such as
                 your chain-of-custody form. (Do not send the Sample Tracking and Analysis
                 Report Form with the samples!)  If your form includes such information as
                 sample area, location, and substrate for each  sample, please specify this information
                 for the proficiency-test samples (even if you have to make up the information) so
                 that the laboratory believes that these are actual field samples. For example, you
                 may want to specify that a dust-wipe proficiency-test sample was "collected" from
                 a one square-foot area on the living room floor. If you are receiving paint-chip
                 proficiency-test samples, please refer to these samples as "paint chips from brick-

                                            C-5

-------
                 or concrete-surfaces" or "dust contaminated with paint" due to their small
                 particle sizes.
          B.     Please specify the date shipped to the laboratory in the column labeled "Date
                 Shipped to Laboratory (14)" on the Sample Tracking and Analysis Report Form
                 and place your initials in the column labeled "Initials (15)" on the form.
          C.     On the day that you ship the samples, please fax a copy of the Sample Tracking
                 and Analysis Report Form (completed through Section B) to Fred Grunder
                 of AfflA (fax number:  703/207-3561).
          D.     A successful pilot program will provide important information on double-blind
                 laboratory accreditation, which will ultimately benefit you and others who employ
                 laboratories for accurate analysis of lead in environmental samples.  Therefore,
                 please do not discuss the proficiency-test samples with the laboratory in any
                 way that would indicate that they are not routine samples, and do not
                 divulge to the laboratory your participation in the pilot program.

Task 6:    Once the laboratory has provided you the analytical results, please perform the following
          for each proficiency-test sample in the batch:

          A.     Please record the sample ID that you assigned to each proficiency-test
                 sample in the column labeled "Client Sample ID (16)" in Section C of the Sample
                 Tracking and Analysis Report Form., then record the date you received the
                 results, along with your initials, in the columns labeled "Date Results Received
                 by Client (20)" and "Initials (21)" on the form.
          B.     Please specify the analysis method used by the laboratory in the box labeled
                 "Analysis Method (17)" on the Sample Tracking and Analysis Report Form
                 (ICP=inductively coupled plasma-atomic emission spectroscopy, GFAAS=
                 graphite furnace atomic absorption spectroscopy, FAAS=flame atomic absorption
                 spectroscopy).
          C     Please specify the lead concentration reported by the laboratory for the
                 proficiency-test sample, as well as the units of measurement, in the columns
                 labeled "Reported Lead Concentration (18)" and "Units of Concentration (19)" in
                 Section C  of the Sample  Tracking and Analysis Report Form.
          D.     Please review all entries on the form for any transcription errors, then initial and
                 date the column of the Sample Tracking and Analysis Report Form labeled
                 "Verification (22)" and "Date (23)".
          E.     Please send a copy of the final completed form to Fred Grunder of AIHA via
                 fax and mail within three working days after receipt of analysis results from
                 the laboratory (fax number: 703/207-3561; mailing address given on the first page
                 of this attachment).

Task 7:    Throughout the pilot program, please feel free to report any quality assurance issues
          (e.g., compromised proficiency-test samples, errors in tracking or reporting) to Fred
                                           C-6

-------
          Grunder or Carl Bell of AfflA (phone number: 703/849-8888; fax number: 703/207-
          3561; mailing address given on the first page of this attachment).

The Sample Tracking and Analysis Report Form is attached for your reference.
Thank you again for your participation!
                                           C-7

-------
Testing Round:
                      Page     of
                           NLLAP Double-Blind Proficiency Testing Pilot Program
                                 Sample Tracking and Analysis Report Form
               Client Information: (1)
Laboratory Information: (2)
Client:
Address:
City:
Telephone:
State: 1 Zip:
Fax:
Responsible Party:
Lab: | Lab ID#:
Address:
City:
Telephone:
State: 1 Zip:
Fax:
Responsible Party:
                                                                       Analysis Method: (17)
                                                        Paint:  D ICP D GFAAS  D FAAS  D Other
                                                        Dust:  D ICP  D GFAAS  D FAAS  D Other
 Section A: Sample Distribution from Proficiency-Testing Service (PTS) to the Client
PTS Sample ID
(3)









Sample Matrix (4)
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
D Paint D Dust
Wipe
Sample
Weight
(grams) (5)









Date Shipped
to Client
(mm/dd/yy)
(6)









Initials
(7)









Date Received
by Client
(mm/dd/yy) (8)









Initials (9)









 Section B: Client Cross-Reference and Shipment to Laboratory
PTS Sample ID (10)









Client Sample ID
(11)









Initials
(12)









Date (mm/dd/yy)
(13)









Date Shipped to
Laboratory
(mm/dd/yy) (14)









Initials (15)









Section C: Laboratory Results

-------
Client Sample ID (16)









Reported Lead
Concentration
(18)









Units of
Concen-
tration
(19)









Date Results
Received by
Client
(mm/dd/yy) (20)









Initials
(21)









Verification
(22)









Date
(mm/dd/yy)
(23)









Numbers in parentheses in the column headings refer to citations in the Standard Operating Procedure for Proficiency Sample
Tracking and Data Reporting.
                                                         C-9

-------