EPA-R4-73-028-e
August 1973
Environmental Monitoring Series
I
55
V
S3S2
al
O
i
-------
EPA-R4-73-028-e
GUIDELINES FOR DEVELOPMENT
OF A QUALITY
ASSURANCE PROGRAM
MEASURING POLLUTANTS FOR WHICH
NATIONAL AMBIENT AIR QUALITY
STANDARDS HAVE BEEN PROMULGATED
FINAL REPORT
by
Franklin Smith and A. Carl Nelson, Jr.
Research Triangle Institute
Research Triangle Park, North Carolina 27709
Contract No. 68-02-0598
ROAP No. 26BGC
Program Element No. 1HA327
EPA Project Officer: Dr. Joseph F. Walling
Quality Assurance and Environmental Monitoring Laboratory
National Environmental Research Center
Research Triangle Park, North Carolina 27711
Prepared for
OFFICE OF RESEARCH AND DEVELOPMENT
U.S. ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
August 1973
-------
This report has been reviewed by the Environmental Protection Agency
and approved for publication. Approval does not signify that the
contents necessarily reflect the views and policies of the Agency,
nor does mention of trade names or commercial products constitute
endorsement or recommendation for use.
11
-------
TABLE OF CONTENTS
Section Page
I INTRODUCTION 1
GENERAL 1
PROGRAM OBJECTIVES . 2
PROGRAM APPROACH 2
II MEASUREIOT FETHODS 4
GENERAL 4
REFERENCE METHOD FOR THE DETERMINATION OF SUSPENDED
PARTICIPATES IN THE ATMOSPHERE (HIGH VOLUME METHOD) 6
General 6
Functional Analysis 7
Arriving At Suggested Performance Standards 25
Recommendations 30
REFERENCE METHOD FOR DETERMINATION OF SULFUR DIOXIDE
IN THE ATMOSPHERE (PARAROSANILINE METHOD) 32
General 32
Identification and Modeling of Important Parameters 33
Performance Model 41
Arriving at Suggested Performance Standards 44
Recommendations 48
REFERENCE METHOD FOR THE MEASUREMENT OF PHOTOCHEMICAL
OXIDANTS CORRECTED FOR INTERFERENCES DUE TO NITROGEN
OXIDES AND SULFUR DIOXIDE 49
General 49
Arriving at Suggested Performance Standards 49
Recommendations . 56
111
-------
TABLE OF CONTENTS (Cont'd)
Section Page
TENTATIVE METHOD FOR THE CONTINUOUS MEASUREMENT OF
NITROGEN DIOXIDE (CHEMILUMINESCENT) , 57
General 57
Arriving at Suggested Performance Standards 57
Recommendations 58
REFERENCE METHOD FOR THE CONTINUOUS MEASUREMENT OF
CARBON MONOXIDE IN THE ATMOSPHERE (NON-DISPERSIVE
INFRARED SPECTROMETRY) 67
General 67
Arriving at Suggested Performance Standards 67
Recommendations 72
111 BACKGROUND INFORMATION APPLICABLf TO THE MANAGEMENT
WNUALS OF FIELD DOCIfBTTS 73
GENERAL 73
SELECTING AUDITING SCHEMES 75
Computation of the Probability of Accepting a Lot as
a Function of the Sample Size and the Acceptable
Number of Defectives 75
Confidence Interval Estimate for the Percentage of
Good Measurements in a Lot 77
Computation of the Percent Defective in the
Reported Data 81
COST IMPLICATIONS 85
Bayesian Scheme for Computation of Costs 86
Computation of Average Cost 89
Cost Analysis 92
1V
-------
TABLE OF CONTENTS (Cont'd)
Section Page
Cost Trade-Off Procedures 97
DATA QUALITY ASSESSMENT '103
Data Presentation 103
Assessment of Individual Measurements 104
Overall Assessment of Data Quality 105
Estimated Variance of Reported Data Using the
Test Statistic: s2/a2 = x2/f 107
IV REOWENDATIONS AND CONCLUSIONS 110
REFERENCES ILL
APPENDIX 1 - COFPUTER SIMULATION 114
APPENDIX 2 - SENSITIVITY ANALYSIS " 128
APPENDIX 3 - FREQUENCY DISTRIBUTIONS 139
APPENDIX 4 - ESTIMATION OF THE MEAN AND VARIANCE OF VARIOUS
COMPARISONS OF IfiTEREST Wl
APPENDIX 5 - CONTROL CHARTS 158
-------
LIST OF ILLUSTRATIONS
Figure No. Page
1 Particulate Concentration and Flow Rate as
Functions of Time 16
2 Symmetrical Diurnal Concentration Pattern 16
3 Sensitivity of Measured Concentrations to Various
Parameters for Different Sampling Environments 24
4 Experimental Plan for Selected Factors 31
5 Estimated Standard Deviation of Analyzer Reading
(volts) Versus N0? Concentration (ppm) 66
6 Probability of d Defectives in the Sample If the
Lot (N = 100) Contains D% Defectives 78
7 Probability of d Defectives in the Sample If the
Lot (N = 50) Contains D% Defectives 79
8 Percentage of Good Measurements Vs. Sample Size
for No Defectives and Indicated Confidence Level 82
9 Percentage of Good Measurements Vs. Sample Size for
1 Defective Observed and Indicated Confidence Level
Lot Size = 100 83
10 Average Cost Vs. Audit Level (Lot Size N = 100) 91
11 Added Cost ($) Vs. MSB (%) for Alternative Strategies 99
12 Precision of Reported Measurements Vs. Cost of Quality
Control Procedure 100
13 Precision of Reported C vs. Total Added Cost, C_ 102
14 Critical Values of Ratio s./a. Vs. n 109
1-1 Simulated Distribution of Particle Concentration
0
(Mean Concentration = 100 yg/m") and
Normal Distribution Approximation 120
3-1 Sketch of Normal Frequency Distribution 140
3-2 Sketch of Uniform Frequency Distribution 143
VI
-------
LIST OF ILLUSTRATIONS (Concl'd)
Figure No. Page
4-1 Comparison of Individual Analyzer Reading
With Value From Calibration Curve 149
4-2 Confidence and Tolerance Limits for the Example 153
5-1 Control Chart for Means 159
5-2 Control Chart for Ranges 159
5-3 Subdivision of Control Chart Into Zones and
Assigned Scores 161
5-4 Standard Control Chart Comparing Techniques 162
-------
LIST OF TABLES
Table No. Page
1
2
3
4
5
6
7
8
9
10
11
12
13
14
PARAMETER EFFECTS MODELS AND VARIABLE MODELS
AS USED IN SIMULATION AND SENSITIVITY ANALYSES
SYSTEM SIMULATION RESULTS
COLLABORATIVE TESTING DATAHIGH VOLUME METHOD
FOR PARTICIPATES
SUGGESTED PERFORMANCE STANDARDS
PARAMETER EFFECTS MODELS AND VARIABLE MODELS AS USED
IN SIMULATION AND SENSITIVITY ANALYSIS (S02)
SYSTEM SIMULATION RESULTS (S02>
SENSITIVITY ANALYSIS RESULTS (SO^
SUGGESTED PERFORMANCE STANDARDS
SUGGESTED PERFORMANCE STANDARDS
CALIBRATION DATA FOR CHEMILUMINESCENT OZONE MONITOR
ANALYSIS OF VARIANCE OF KI DATA
ANALYSIS OF VARIANCE OF CHEMILUMINESCENT MONITOR DATA
ANALYSIS OF VARIANCE OF KI DATA FOR EACH SLEEVE SETTING
ESTIMATE OF STANDARD DEVIATION FOR REPEATABILITY
»J + 4>1/2
19
22
26
28
40
42
43
45
50
51
52
52
54
54
15 ANALYSIS OF VARIANCE OF CHEMILUMINESCENT OZONE
MONITOR DATA FOR EACH SLEEVE SETTING 55
16 ESTIMATE OF STANDARD DEVIATION FOR REPEATABILITY
"2 2 1/2
(&B + V 55
17 SUGGESTED PERFORMANCE STANDARDS 59
18 DATA FOR ANALYZER A 60
19 DATA FOR ANALYZER B 61-62
20 ANALYSIS OF VARIANCE OF ANALYZER A DATA 63
21 ANALYSIS OF VARIANCE OF ANALYZER B DATA 64
viii
-------
LIST OF TABLES (Cont'd)
Table No. Page
22 SUMMARY OF DATA ANALYSES - ANALYZER A 65
23 SUMMARY OF DATA ANALYSES - ANALYZER B 65
24 SUGGESTED PERFORMANCE STANDARDS 68
25 P(d defectives) 77
26 COSTS VS. DATA QUALITY 86
27 COSTS IF 0 DEFECTIVES ARE OBSERVED AND THE
LOT IS REJECTED 87
28 COSTS IF 0 DEFECTIVES ARE OBSERVED AND THE
LOT IS ACCEPTED 87
29 COSTS IN DOLLARS 88
30 OVERALL AVERAGE COSTS FOR ONE ACCEPTANCE-REJECTION SCHEME 90
31 OVERALL AVERAGE COSTS 90
32 QUALITY OF INCOMING DATA AND ASSOCIATED AVERAGE COST 92
33 SUMMARY INFORMATION 93
34 EFFECT OF VARYING THE COST OF ACCEPTING POOR QUALITY DATA 94
35 EFFECT OF VARYING THE COST OF REJECTING GOOD QUALITY DATA 95
36 EFFECT OF VARYING THE PERCENT DEFECTS IN POOR
QUALITY LOTS 95
37 ASSUMED STANDARD DEVIATIONS FOR ALTERNATIVE STRATEGIES 98
38 CRITICAL VALUES OF s./a. 108
1-1 INPUT DATA 116
1-2 PORTION OF SIMULATED DATA 117
1-3 CHECK OF SIMULATED VALUES 118
1-4 LISTING OF SIMULATED VALUES IN ASCENDING ORDER 119
1-5 MOMENTS OF SIMULATED DISTRIBUTION 121
IX
-------
LIST OF TABLES (Concl'd)
Table No. Page
1-6 INPUT DATA (S02> 123
1-7 PORTION OF SIMULATED DATA (S02) 124
1-8 CHECK OF SIMULATED VALUES (SO^ 125
1-9 LISTING OF SIMULATED VALUES IN ASCENDING ORDER (SO,,) 126
1-10 MOMENTS OF SIMULATED DISTRIBUTION (SO ) 127
2-1 PARTIAL OUTPUT OF SENSITIVITY ANALYSIS 133
2-2 PRINTOUT OF SENSITIVITY ANALYSIS OF S.P. 134
2-3 PARTIAL OUTPUT OF SENSITIVITY ANALYSIS (S02) 137
2-4 PRINTOUT OF SENSITIVITY ANALYSIS (S02) 138
4-1 COMPUTATION OF CONFIDENCE AND TOLERANCE LIMITS
FOR THE EXAMPLE 152
5-1 TABLE OF FACTORS FOR CONSTRUCTING CONTROL CHARTS 160
-------
SECTION I INTRODUCTION
GENERAL
Surveillance of air quality is an important function of any air pollution
control agency and an integral part of the total effort to control air
pollution.
Air quality measurements are required to
(1) determine the presence and extent of air pollution
problems within a given jurisdictional area,
(2) develop implementation plans,
(3) document progress towards attainment of standards
and,
(4) provide immediate air quality data when air pollution
episode conditions exist.
Air pollution control agencies at all levels of government, (i.e., federal
state, and local), are currently monitoring a wide range of gaseous and
particulate pollutants. Primary responsibilities for. monitoring rest with
state and local agencies. The Environmental Protection Agency (EPA) provides
assistance through issuance of monitoring guidelines, development and field
testing of instruments and analytical methods, promulgation of reference
methods, issuance of recommended calibration, operation, and maintenance
procedures for continuous analyzers, and training activities.
National primary and secondary ambient air quality standards, reference
measurement methods, and sample averaging times have been promulgated for
suspended particulates, sulfur dioxide, carbon monoxide, photochemical
oxidants, and non-methane hydrocarbons. Regulations have been published
requiring implementation plans which will allow achievement of these
standards. These plans require the operation of air surveillance networks
which routinely measure the specified pollutants.
In order to insure that the results reported by the various networks are
valid and that non-compliance with standards will be detected when it
occurs, the Quality Assurance and Environmental Monitoring Laboratory of
the EPA has developed quality assurance programs and procedures applicable
to each of the reference methods. Implementation of a quality assurance
program will result in data that are more uniform in terms of precision
and accuracy and will enable each monitoring network to continuously
generate data that approach the highest level of quality possible for a
particular measurement method.
-------
PROGRAM OBJECTIVES
The objectives of this program were to provide guidelines for the
development of quality assurance programs applicable to measuring pollutants
for which National Ambient Air Quality. Standards have been promulgated.
Specifically, guidelines were written for the reference methods for measuring
suspended particulates, carbon monoxide, photochemical oxidants, sulfur
dioxide, and one tentative method for the continuous measurement of nitrogen
dioxide (chemiluminescent).
The objectives of a quality assurance program for any measurement method are
to:
(1) provide routine indications of unsatisfactory performance
of personnel and/or equipment,
(2) provide for prompt detection and correction of conditions
which contribute to the collection of poor quality data,
and
(3) collect and supply information necessary to describe the
quality of the data.
To accomplish the above objectives, the guidelines for each quality assur-
ance program include directions for the:
(1) routine monitoring of the variables and parameters which
may have a significant effect on data quality,
(2) development of statements and evidence to qualify data
and detect defects,
(3) evaluation of relevant action strategies to vary the level
of precision/accuracy in the reported data as a function
of cost, and
(4) routine training and evaluation of operators.
PROGRAM APPROACH
The program approach is discussed in terms of the content, the means of
prescribing the guidelines to achieve maximum utility, and scope of the
guidelines for each method.
In presenting quality assurance guidelines, an individual, field-useable
document was written for each method. 'Backup data, analyses, and discussions
to further elucidate, in a more rigorous manner, the contents of the field
documents are included in this final report.
-------
Each field document consists of three major parts. They are:
(1) OPERATIONS MANUAL - The Operations Manual sets forth
recommended operating procedures, instructions for
performing control checks designed to give an indication
or warning that invalid or poor quality data are being
collected, and instructions for performing certain
special checks for auditing purposes.
(2) SUPERVISION MANUAL - The Supervision Manual contains
directions for 1) assessing air quality data,
2) collecting information to detect and/or identify
trouble, 3) applying quality control procedures to
improve data quality, and 4) varying the auditing or
checking level to achieve a desired level of confidence
in the validity of the outgoing data. Also, monitoring
strategies and costs as discussed in the Management
Manual are summarized in this manual.
(3) MANAGEMENT MANUAL - The Management Manual presents
procedures designed to assist the manager in 1) detecting
when data quality is inadequate, 2) assessing overall
data quality, 3) determining the extent of independent
auditing to be performed, 4) relating costs of data
quality assurance procedures to a measure of data quality,
and 5) selecting from the options available the alter-
native(s) which will enable one to meet the data quality
goals by the most cost-effective means. Also, discussions
on data presentation and personnel requirements are
included in this manual.
This final report contains sections on:
(1) the individual measurement methods which include any data
and analyses used to arrive at suggested performance
standards in the respective field documents and discussions
of the methods used to treat areas where sufficient field
data were not available.
2) background information directly applicable to the Management
Manual sections in the field documents in the form of a
broader and more rigorous mathematical treatment of the
subject areas in that manual, and
(3) recommendations for implementation of quality assurance
programs, areas requiring further study, and conclusions
arrived at as the result of this program.
-------
SECTION II ' PEASUREIW ICTiODS
GENERAL
A systems approach was employed in writing guidelines for developing a
quality assurance program for each measurement method. The approach was
basically the following.
A functional analysis of the method was performed to identify and, if
possible, quantify sources of variability. Total variability of a measure-
ment process is a combination of two error components; namely, determinate
and indeterminate. Determinate errors are defined as those that can be
avoided once they are recognized. This type of error is caused by such
factors as:
(1) improper calibration of glassware or instruments, or
improper standardization of reagents;
(2) personnel errors, such as the tendency of an operator
to read a meter too high or too low; and
(3) a constant error in the method.
Determinate errors introduce a bias into the measured values. Indeterminate
errors, or chance variations, have a random nature and cannot be completely
eliminated from a measurement process, but can possibly be contained in a
narrow zone when their sources are well known.
The second step in writing the guidelines was to delineate operating
procedures, quality control checks, and suggested performance standards
designed to eliminate determinate errors and to minimize indeterminate
errors. Also, independent audit checks encompassing all phases of the
measurement process were presented to allow for an independent estimate of
data quality in terms of precision and accuracy.
The third and final step was to evaluate certain monitoring strategies for
improving data quality and different auditing schemes to vary the level of
confidence in the estimated quality of the reported data as functions of
cost.
In all cases suggested performance standards and control limits were arrived
at from results of a collaborative test of the particular method, published
data from special tests, or from engineering judgment. All such values must
be evaluated and adjusted as field data become available. To employ any
said value as a' "hard" standard without first evaluating its validity for
field conditions would be contrary to the intent of the authors and would,
in their opinion, significantly reduce the utility of the documents.
-------
Suggested performance standards given in the individual field documents,
which were derived from actual data, are treated in detail in this report.
Suggested standards which resulted from engineering judgments or estimates
are adequately described in the field documents and are not addressed in
this report. Suggested standards for auditing and for operation are
somewhat common to all methods and are summarized here and not repeated
for each method.
Audit rates are discussed in the field documents and in Section III of this
report. For normal, routine monitoring purposes an auditing level of
7 checks out of a lot size of 100 samples was recommended. For the conditions
spelled out in the field document this level of auditing allows one to say
at the 50 percent confidence level that:
(1) If no defects are found in the 7 checks, at least
90 percent of the samples in the lot are good
(read from the 50% curve in Figure 8, page 82, for
a sample size of 7).
(2) If no more than 1 defect is observed in the 7 checks.
at least 76 percent of the samples in the lot are
good (read from the 50% curve in Figure 9, page 83,
for a sample size of 7).
This turned out to be the most cost effective auditing level for the costs
assumed in Section III, under Computation of Average Cost, page 89.
Suggested standards for operation involve evaluating auditing data to
determine if the measurement process is "in" or "out" of control. In all
cases the suggested standards for defining defects were given at the 3o
level in the field documents. It was assumed that the datathat is, the
difference between the value derived from the audit and the operator's
measured valueare normally distributed and that the means and standard
deviations as estimated or computed from a limited amount of test data
represent the true mean and standard deviation of the total population in
question (see Appendix 3 for a detailed discussion of the normal distri-
bution) . Under these assumptions and for an auditing level of 7 checks out
of a lot size of 100, the measuring process was judged to be "out" of control
and in need of trouble shooting and corrective actions if any of the
following conditions were observed:
(1) One check value exceeds the + 3a limits (99.7 percent of
the checks should be within + 3a of the mean; in this
case .997 * 7 = 6.979, or approximately all 7 checks,
should be within + 30 if the process in "in" control).
(2) Two checks exceed the + 2a limits (under normal conditions
95 percent of the checks would be within + 20 of the mean).
(3) Four checks out of 7 exceed the + la limits (under normal
conditions 68 percent of the checks, or 4.76, checks should
be within + la of the mean).
Each measurement method is discussed separately in the following subsections.
5
-------
REFERENCE METHOD FOR THE DETERMINATION OF SUSPENDED PARTICULATES IN
THE ATMOSPHERE (HIGH VOLUME METHOD)
General
Measurement of the mass of suspended particulate matter in the ambient
atmosphere by the High Volume Method requires a sequence of operations
and events that yields as an end result a number that serves to represent
the average mass of suspended particulates per unit volume of air over
the sampling period. Techniques for dynamic calibration of high volume
samplers using test atmospheres containing known concentrations of
particulates are not available. Therefore, there is no way of knowing
the accuracy of the values derived from high volume sampling. However,
numerous experiments and studies have been performed to identify and
evaluate factors which influence the final results. Also, a collaborative
study has been made to determine the repeatability and reproducibility of
the method under controlled conditions (Ref. 1).
The High Volume Method is dependent on several parameters such as operator
effects, environmental conditions, calibration procedures, variation in
instrumentation, and other variables and effects, some of which may be
unknown at this time. With the relatively large number of parameters/
variables involved, it was felt necessary to develop a mathematical model
(hereafter referred to as a performance model) of the process which could
be subjected to statistical analysis in order to:
(1) generate accuracy data for comparison with published
data as a check on the reasonableness of essential
assumptions made concerning important variables,
(2) estimate the process variability for various monitoring
strategies,
(3) rank the variables according to their influence on the
measured concentration, and
(4) evaluate the effect of various control procedures on the
measurement process.
The development and use of the performance model are discussed in the
following paragraphs as part of a functional analysis of the measurement
process.
A functional analysis of the High Volume measurement process starting with
the specification of the filter media and extending through the docu-
mentation and data reporting steps was performed to identify all parameters,
variables, and errors which could be possible contributors to the variation
in the measurements. Data sufficient to allow an estimate of the error
range associated with each parameter in the measuring process were collected
-------
from published documents, private communications with EPA personnel, and
from RTI in-house projects. If data were not available, engineering
judgements were used to estimate variable limits. These data were used
in developing a performance model for computer simulation.
ie performance model was designed to account for the major parameters
in the application of the method and to characterize their variations.
It was subjected to statistical analysis by means of a computer program
designed to introduce the kinds of variations to be expected of each of
the parameters and to execute simulation for a sequence of different
choices of conditions so that the effects of each parameter and any
significant interactions between the parameters would be identified and
evaluated. Results from the model study provided a basis for decisions
concerning the quality control technology applied to the different steps
in the measurement procedure and the suggested performance standards
given in the field document.
The functional analysis is discussed in terms of parameter evaluation and
modeling, the performance model, and results from the simulation and
sensitivity analyses. Discussions of the computer programs written to
perform the simulation and sensitivity analyses are given in Appendices 1
and 2, respectively.
Functional Analysis
A functional analysis consisting of identification and modeling of important
variables and parameters, development of a performance model of the measure-
ment process, and simulation and sensitivity analyses using the performance
model was performed as a means of evaluating the overall measurement
process. Each phase of the analysis is discussed below.
Identification and Modeling of Important Parameters - The parameters are
grouped according to whether they influence particulate weight, flow rate,
sampling time, or the measured concentration directly.
Factors which could influence particulate weight include: 1) filter
surface alkalinity, 2) relative humidity of the filter conditioning
environment, 3) elapsed time between sample collection and analysis, and 4)
weighing errors.
Factors influencing the precision and accuracy of the calculated average
flow rate are: 1) method of calculating the average flow rate, 2) flow
rate calibration, and 3) temperature and pressure effects on flow rate.
Sampling time is usually measured by an elapsed time indicator operated from
an electronic timer capable of being pre-set to start and stop the sampler.
One factor affects the measured concentration directly; that is, unequal
sampling rates. Unequal sampling rates result when particulate concen-
tration and sample air flow vary with time.
-------
4uAj(ace atkaLiyicfy - Flash fired glass-fiber filters are the most
frequently used filters for collecting suspended particulate matter for
gravimetric analysis. It has been shown (Refs. 2-4) that solid matter is
deposited on the fiber surfaces by oxidation of acid gases in the sample
air. It was also observed that the quantity of such matter deposited in
a given sampling period was not the same for all commercially available
glass-fiber filters. Although other reactions are conceivable, the
formation of sulfate was studied. It occurs during the first 4 to 6 hours
of sampling, and very little is formed after 6 hours (Ref. 2).
Tests conducted with filters of pH-6.5 and pH-11 showed a significantly
larger sulfate to total particulates ratio for the filters of pH-11 (Ref.
3). Additional tests (Ref. 4) have shown that alkaline filter media can
yield erroneously high results for total particulate matter, sulfates,
nitrates, and other species existing as acid gases in the sample air.
Samplers operating side by side, one equipped with a filter of pH-11 and the
other with a filter of pH-6.5, showed after 9 sampling periods that the average
total particulate matter was higher by 18 percent, sulfates by 40 percent, and
nitrates by 60 percent for filters of pH-11 than for filters of pH-6.5.
The quantity of solid matter deposited during a sampling period is a
function of filter pH, length of sampling period or volume of air sampled,
and the concentration of acid gases in the sample air. However, even
background levels of NO,, and S0? well below national air quality standards
can induce significant errors when alkaline filters are used.
In modeling this variable, the following points were considered. First, the
effect can be controlled or eliminated by specifying neutral filters when
ordering a batch of filters. Also, the data from references 3 and 4 show
a large difference in the magnitude of the effect. Using data from either
reference independently would result in the effect being insignificantly
small (Ref. 2) or being so large as to mask the effects of other parameters
(Ref. 4). The values used to model the pH effect, E(pH), for this analysis
were selected so that the effect will always show up as being significant
but still not mask the effects of other parameters. Also, it was assumed
that there are always sufficient concentrations of acid gases in the
atmosphere to effect this reaction.
The pH effect was modeled as a function of filter pH and of sampling
period (T) time in minutes by:
1/2
E(pH) =33.5 (PH-7)(T)i/Z
where E.(pH) = weight of acid gases converted to particulate
matter during the sampling period in yg,
pH = the pH of the filter, and
T = sampling time in minutes.
-------
This model shows a zero effect for a neutral filter (pH-7) and a conversion
of 5,000 yg (5 rag) for a filter of pH-11 and a 24-hour (1440-minute)
sampling period.
For simulation purposes in guideline development, values of pH were
generated from a uniform distribution ranging from 7 to 12 (see Appendix 3
for a discussion of the uniform distribution). However, in an actual
application the pH value or range should be determined and used in the
model.
humidity e^ect - Collected particulates are hygroscopic in
varying degrees. Samples collected from suburban, urban, and industrial
atmospheres were weighed after being conditioned for a minimum of 4 days
at relative humidities varying from 0 to 100 percent (Ref. 5). The results
show less than a 1 percent increase in particulate weight in going from
0 to 55 percent relative humidity. However, the relationship is exponential
for relative humidities greater than 55 percent, showing a 5 percent
increase in particulate weight at a relative humidity of 70 percent and
approximately 15 percent weight increase at 80 percent relative humidity.
The industrial sample proved most hygroscopic with a 90 percent weight
increase at a relative humidity of 100 percent.
The above results point out the importance of maintaining the conditioning
environment at a relative humidity less than 55 percent. Also, the
humidity level should be the same for conditioning the exposed filter as
that used to condition the clean filter. In instances in which the
exposed filter has to be removed from the conditioning environment for
weighing, the time interval between removal and weighing should be kept to
a minimum. An interval of less than 5 minutes is recommended.
It must be remembered that the exposure period in this test was 4 days.
Also, in filter-conditioning environments the relative humdity is probably
never over 60 to 65 percent. It also seems unlikely that samples would be
out of the conditioning environment for over an hour, if at all, before
weighing. Therefore, it appears reasonable to assume that under normal
operating conditions the humidity effect would seldom exceed that of a
24-hour exposure to a 60 to 65 percent relative humidity. The relative
humidity effect was modeled as
E(RH) = (W ) 0.00013 exp(0.087 x RH)
where E(RH) = increased particulate weight due to adsorption of
moisture in yg,
W = true weight of collected particulates in yg, and
RH = relative humidity of the conditioning environment
expressed as a percent.
-------
For simulation purposes the relative humidity was modeled as a normal
distribution with a mean of 50 and a standard deviation of 5 (see Appendix 3
for discussion of the normal distribution). This would result in an error
of 1 percent if the exposed filter was conditioned at 50 percent RH and a
2.7 percent error if conditioned at 60 percent RH.
EtapAe.d time. b&twe.e.n Aampte. c.otie.ctLon and anaty&iA - During the time
between sample collection and final weighing volatile matter having
substantial vapor pressures may evaporate resulting in a significant
reduction in particulate weight.
Results from one set of tests (Ref. 6) indicate that the weight loss is
approximately proportional to the percent of organic matter initially
present in the collected sample. The greatest rate of loss is experienced
during the first 24 hours after collection. A lower but somewhat constant
rate of loss continues for several days, the number of which is again a
function of the initial content of organic matter.
The test involved samples collected from three different sources.
Particulates from these sources averaged 1.8, 9.3, and 61.5 percent initial
organics. Weight losses were measured at elapsed times of from less than
24 hours up to 26 days for several samples.
In modeling the results of this test, the weight of the particulates after
the 24-hour conditioning period was taken as the reference weight. That
is, no error was associated with the losses occuring during the first
24 hours after sample collection. This approach was taken because a
24-hour conditioning period is specified by the reference method and the
manner in which the above test was conducted may have biased the indicated
weight losses during this initial 24-hour period. For example, the samples
were placed in a desiccator for only 30 minutes to remove extraneous
moisture. The sample was stabilized for another 30 minutes in a controlled
environment at 75° + 3°F and 54 +_ 3 percent relative humidity, and then
weighed. Thus, the conditioning period was only 1 hour. The high rate of
weight loss during the first 24 hours may have been due to increased loss of
moisture as well as the evaporation of organics.
Weight loss due to evaporation of organic matter was modeled as
E(OM) = W [0.000054 (% of OM)(D)]
where E(OM) = weight loss in ng,
W = true weight of collected particulates in yg,
% of OM = initial organic content of collected
particulates in percent, and
D = time between collection and weighing in
days minus 1 day.
10
-------
This model gives a weight loss of 1 percent per 10 percent organic content
for a delay of 12 days and a zero loss when no delay is involved.
For simulation purposes the initial organic content was modeled as a
normal distribution with a mean of 20 and a standard deviation of 5 (see
Appendix 3 for definitions of normal distribution) . Values for delays
in days were randomly selected from a uniform distribution with a range
of 0 to 21.
- Two weighing processes are involved in the High Volume
Method. They are the weighing of clean filters and the weighing of
exposed filters. If not properly monitored, the weighing process can be
a source of significant error in the final result derived from the High
Volume Method. Fifty tare-weight weighings for each of five filters made
over a period of time in which the relative humidity of the conditioning
chamber was varied from 20 to 50 percent showed a maximum variation in
tare-weight weighings of 1.2 mg (Ref. 5). Another test showed a standard
deviation of approximately 0.8 mg for weighing clean filters after
successive 24-hour conditioning periods.
These data point out the importance of performing the weighings at the
appropriate time, i.e., just after the 24-hour conditioning period, and
the necessity of performing the audit or check within a few minutes either
before or after the regular weighing in order to expect good agreement
between the two weighings.
It is suggested that if the weighing and auditing procedures are properly
carried out, the variation between the original and check weights of clean
filters should not exceed + 1.0 mg and not more than + 2.7 mg for exposed
filters. The combined weighing error is modeled as a normal distribution
with a zero mean and a standard deviation of 1000 yg symbolized by
E(W) = N(0,1000)
where E(W) = weighing error in yg, and
N(0,1000) = represents a normal distribution with a zero
mean and a standard deviation of 1000 yg (the
accepted notation for a normal distribution is
N(y,a).
Ftow-Mite. tivuLinQ HWWti - The general feeling of people reading the rotameter
Q O
is that they can read it to within +0.03 m /min (+ 1 ft /min). For an
3 3
average flow rate of 1.13 m /min (40 ft /min), this would represent a
+2.5 percent error. Under field conditions the largest component of
reading error is probably due to operator technique (e.g., parallex error,
11
-------
rotameter not held in a true vertical position, or estimating an average
reading while the float is in oscillatory motion) and, if not monitored,
would in some instances be much larger than 2.5 percent of the true value.
Flow-rate reading error was modeled as
E(Qr) = N(0, 0.025)
where E(Q ) = flow-rate reading error as a decimal fraction, and
N(0, 0.025) = a normal distribution (see Appendix 3) with
a zero mean and a standard deviation of 0.025.
Ca£cu£cutLng average. ££ow /tote - Calculating the average flow rate from
initial and final values assuming a constant rate of change throughout the
sampling period can result in large errors. One report (Ref. 7) shows an
average bias ranging from -1.2 to +8.1 percent of the true average flow
rate for 3 sets of data with 6 samples each. Bias is defined as the
difference in average flow rate as computed from the initial and final
measurements compared to the average derived from several measurements made
throughout the sampling period. These errors can result from particulates
plugging the filter resulting in a nonuniform decrease in the flow rate
over the sampling period or from variations in source voltage. .. Nonuniform
changes in flow rate are probably greatest in industrial areas due to
sticky particulates and can result in a -2 to +10 percent error range in
average flow-rate values.
A sampler equipped with a continuous flow-rate recorder does not have the
above problem. The true average flow rate can be estimated or calculated
3 3
by hourly values to within 0.03 m /min (1 ft /min) from the recorder chart.
This represents a significant improvement in system accuracy.
For simulation purposes the error in computing the average flow rate using
initial and final flow-rate measurements and assuming a constant rate of
change was modeled as a function of the change in flow rate by the
relationship
E(Q) = Q- [UC-0.02 to +0.10)]
where E(Q) = the error in Q in percent,
AQ = change in flow rate for the sampling
period in m /min/24-hour
3
Q. = initial flow rate in m /min, and
U(-0.02 to +0.10) = uniform distribution with lower limit
of -0.02 and upper limit of +0.10. The
notation for uniform distributions used
in this document is U (lower limit to
upper limit) , see Appendix 3 for a dis-
cussion of the uniform distribution.
12
-------
The model is such that samples experiencing a change in flow rate as
great as 50 percent of the initial flow rate would be subject to biases
ranging from -2 percent to +10 percent of the average flow rate. As the
magnitude of change in flow rate decreases, the magnitude of the possible
error decreases also. It should be pointed out that the bias may be
negative. In actual practice the final flow rate may be greater than the
initial flow rate due to voltage variations or extreme changes in the
relative humidity during the sampling period.
The final flow rate, Q,, was allowed to vary according to a uniform
distribution (see Appendix 3 for discussion of uniform distribution) with a
3 3
range of from 1 to 1.7 m /min resulting in a AQ range of from 0 to 0.7 m /min/
24-hour. For a specific situation the drop in flow rate should be directly
related to the total weight of particulates collected. However, for a large
population of sites it was felt that the type of particulates were more
important. That is, large flow-rate drops can result from a relatively
small quantity of sticky particulates.
catLbfuition - Calibration of 12 new orifice units by well-
qualified individuals using positive displacement meters as primary
standards under laboratory conditions showed a standard deviation from the
mean of 2.1 percent (Ref . 1) . Calibration of samplers in the laboratory and
in the field by less qualified people using the orifice unit would be_ expected
to yield a much larger standard deviation. Previous experience with 'high
volume samplers indicates that +3 percent of the mean is a reasonable value
to use as a standard deviation for calibration error for a well-monitored
operation. There is also a possible change in the calibration with time.
Such a term is not incorporated in this treatment but could be once
sufficient field data are available to allow an estimate of its magnitude
and characteristics.
For simulation purposes the calibration effect was assumed to be normally
distributed with a zero mean and a standard deviation of 3 percent of the
flow rate, i.e. ,
E(Q ) = N(0,0.03)
c
where E(Q ) = calibration error expressed as a decimal
fraction, and
N(0,0.03) = a normal distribution (see Appendix 3) with
a zero mean and standard deviation of 0.03.
and ptLeAAute. ej^eoti on ^tow Mute. - in most regions in the
United States and for a specific site location, temperatures usually range
from -4°C (25°F) to 38°C (100°F) and barometric pressure variations are on
the order of + 12.7 mmHg (0.5 in. Hg) (Ref. 8). Tests on a sampler
13
-------
equipped with a flow-rate recorder showed a maximum deviation from the
calibration curve of + 7 percent to -10 percent in the indicated flow rate
when going from the extremes of 38°C and 736 mmHg to -4°C and 762 mmHg.
Calibration conditions were 70°F and 750 mmHg.
The above data point out the need for either calibrating on site or making
corrections for temperature and pressure if the ambient site conditions are
significantly different from the laboratory conditions.
No similar data were available for a sampler equipped with a rotameter, but
it was felt that the variability would be at least as large. The tempera-
ture and pressure effect was modeled as a normal distribution with a zero
mean and standard deviation of 3 percent of the measured flow rate, i.e.,
E(QBT) = N(0,0.03)
where E(Q T) = error in measured flow rate due to changes
in temperature and pressure as a decimal
fraction, and
N(0,0.03) = normal distribution (see Appendix 3) with
a zero and standard deviation of 0.03.
The error levels from this model would be less than +_ 9 percent 99.7 percent
of the time, less than +_ 6 percent 95 percent of the time, and less than
+ 3 percent 68 percent of the time.
This model is only valid for a specific site- where the sampler has been
calibrated at or near the average temperature and pressure. It does not
represent the error that would result if the sampler were calibrated at a
given temperature and pressure then used in a location where the average
temperature or pressure is significantly different from the calibration
conditions. Such a situation would require a bias term that is, the mean
of the distribution would no longer be zero.
£im&. - The results of high volume sampling are not very sensitive
to the normal magnitudes of timing errors. For example, a 14-minute error
in a 24-hour sampling period results in a 1 percent error in the measured
concentration. The reference method specifies that start and end times be
determined to the nearest 2 minutes. This can be accomplished with the
operators' watch or by using an elapsed time indicator on the sampler. In
the first instance there is no way of knowing of or compensating for power
failures or other interruptions occurring during the sampling period.
This could result in significant error in the sampling period time even
though the start and end times are accurate. Samplers equipped with an
elapsed time indicator or a continuous flow-rate recorder would indicate
such power interruptions and allow for corrections to be made.'
14
-------
In generating simulation and sensitivity data, the error in sampling period
time was assumed to be normally distributed about a mean value of zero with
a standard deviation of 7 minutes. The model is
E(T) = N(0,7)
where E(T) = error in sampling period time in minutes, and
N(0,7) = normal distribution (see Appendix 3) with a
zero mean and a standard deviation of 7 minutes.
- In certain instances, when both flow rate and
particulate concentration vary during the sampling period, significant
errors in the measured average concentration can occur. The example given
in Figure 1 is proposed as an extreme condition which is not likely to be
exceeded in actual operations. In this case the concentration of suspended
particulates varies from 353 yg/m3 (10 yg/ft3) to 70.6 yg/m3 (2 yg/ft3)
according to the following equation
S.P. = 141.2 (f + cos ^ t)
3
where S.P. is the instantaneous concentration in yg/m , and
t is the time in hours.
33 3
Also, the flow rate decreases from 1.7 m /min (60 ft /min) to 1.02.m /min
3
(36 ft /min) in a linear fashion according to the following relationship
Q = 1.7 - (0.03 m3/hr)t
where Q is the flow rate in m /min, and
t is the time in hours.
The true average concentration, S.P., is seen to be the value at the point
where the concentration curve crosses 12 on the time axis, or
1 /
[ 2 S.P. dt [ 141.2(| + cos -^ t)dt
' t o
=
15
= 212 yg/m3 (6 yg/ft3).
-------
- 1.70 (60)
cC 353 (10)
| 318 (9)
-2 283 (8)
*% 247 (7)
2 212 (6)
| 177 (5)
jij 141 (4)
g 106 (3)
1 71 (2)
° 35.3 (1)
8 12 16 20 24
Time (Hours)
Figure 1: Particulate Concentration and Flow Rate as Functions of Time
CO
>H
00
a.
"i
~eo
g
rt
P
§
U
1059 (30)
883 (25)
706 (20)
530 (15)
353 (10)
177 (5)
0
Flow Rate
Concentration .
1.70 (60)
1.13 (40)
0.57 (20)
«
8 12 16
Time (Hours)
20
24
Figure 2: Symmetrical Diurnal Concentration Pattern
16
-------
However, since the flow rate also varies with time, the average concentration
that would be measured by the high volume sampler, assuming no other errors
are involved, is expressed as:
. O /
[ 2 S.P. Qdt I 141.2(|+ cos |^-.t)(1.7 - 0.03 t)dt
t, 'o
S.P.1
t, (-24
z Q dt (1.7 - 0.03 t)dt
t >o
= 227 yg/m3 (6.42 ug/ft3).
This value differs from the true average concentration by + 7 percent. The
reverse case in which the concentration increases as the flow rate decreases
is illustrated by the dashed curve in Figure -1 and results in a -6.7 percent
deviation from the true average.
In a situation such as that shown in Figure 2 in which the concentration
exhibits a diurnal pattern which is symmetrical about the midpoint of the
sampling period, the true average concentration is realized by the High
Volume Method as long as the flow rate is a linear function of tljne. In
this case
S.P. = 353 [2 + sin(jj t + |)
12
and
Q = 1.7 - (0.03 m3/hr)t .
Performing the same calculations as those done in the example in Figure 1
shows that the "measured" value is the same as the true value.
As a result of the above calculations, it is felt that a deviation greater
than + 7 percent from the true average concentration due to this effect
alone should be very rare. An estimate of the possible error for a given
site could be made by using the local diurnal pattern of suspended partic-
ulate concentration and normal or average drop in the flow rate over a
24-hour sampling period to perform the above calculations. The error would
not be significant unless the change in flow rate is greater than 20 percent
of the initial flow rate, the diurnal pattern is extremely nonsynmetrical
about the midpoint of the sampling period, and the maximum concentration
is at least four times as great as the minimum.
17
-------
For simulation and sensitivity analyses the effect due to unequal sampling
rates was modeled as
E[Q(t), S.P.(t)] = S.P. QQ [U(-0.09 to +0.09)]
where E[Q(t), S.P.(t)] = concentration error due to unequal
sampling rates in yg/m^,
S.P. = true concentration in ..yg/m ,
AQ = change in flow rate in m^/min/''-hour,
Q = initial flow rate in m^/min, and
U(-0.09to +0.09) = uniform distribution over the range
-0.09 to +0.09 (see Appendix 3 for
discussion of the uniform distribution).
This is consistent with the calculated results, namely, that a maximum
error of + 7 percent in the average concentration may result when the change
in flow rate is as great as 0.4 Q., depending on the variability in tha
particulate concentration. It also shows that the magnitude oJf the possible
error decreases as the change in flow rate becomes smaller.
Stonmo/U/ 0(J pasuun&toic and vafLUlble. modeLing - Parameters and variable
relationships and models as used in the simulation and sensitivity analyses
are summarized in Table 1.
Performance Model - A mathematical performance model of the measurement
process incorporating the error terms discussed in the previous subsection
and summarized in Table 1 was derived from the basic equation
Sp =
* w
where S.P. = average concentration in yg/m ,
W = measured weight of collected particulates in yg,
and V = the measured volume of air sampled in m .
18
-------
Table 1: PARAMETER EFFECTS MODELS AND VARIABLE MODELS AS USED
IN SIMULATION AND SENSITIVITY ANALYSES
Parameter Effects
Variable Models
E(pH)
E(RH)
E(W)
E(Qr)
E(Q)
'33.5 (X(l) - 7)
N*(45, 5)
N(0, 1000)
N(0, 0.025)
Q± - X(13)
0.5
E(Qc) = N(0, 0.03)
E(QBT) = N(0, 0.03)
Q. - X(13)
E[Q(t),C(t)] = S.P. Ys [X(9>]
i
E(OM) =W [0.000054 (X(10))(X(11))]
P
E(T) = N(0, 7)
X(l) = pH = U(7.0 to 12.5)
X(2) = E(RH)
X(3) = E(W)
X(4) = E(Qr)
X(5) = U**(-0.02 to +0.10)
X(13) = Qf = U(l to 1.7)
X(6) = E(Qc)
X(7) = E(QBT)
X(8) extra variable, set equal to zero
for computer runs
X(9) = UC-0.09 to +0.09)
X(10) = D = U(l to 21)
X(ll) = % OM = N(20, 5)
X(12) = E(T)
**
N(y,a) represents a normal distribution with a mean y and standard deviation a, see Appendix 3.
U(a to b) represents a uniform distribution with lower limit a .and upper limit b, see Appendix 3.
-------
For an actual measurement where error terms are involved, the measured
particulate weight, W, would be
W = 0.5 (Qi + Qf) (S.P.)T + E(W) + E(pH) + E(RH) - E(OM)
where 0.5 (Q. + Qf) (S.P.) = the weight of particulates that would
be collected for a true concentration,
(S.P.)_, of suspended particulates if
no errors were involved.
The remaining error terms are as described in the previous subsection.
The measured volume is
V = Q T
m
where Q = the measured average flow rate in m /min given by
Q = 0.5(0^ + Qf) [1 + E(Qr) + E(Q)
and T = the measured sampling period time in minutes given by
T = T + E(T).
m t
The term 0.5(Q. + Qf) represents the true average flow rate, and T is the
true sampling period time.
Combining the above relationships plus the error due to unequal sampling
rates, i.e., E[Q(t), S.P.(t)], gives the following
qf) (S.P.)T + E(W) + E(pH) + E(RH) - E(OM)
S'P' = 0.5(Qi+Qf)[l + E(Qr) + E(Q) + E(Qc) + E(QBT)] [Tt+E(T)]
+ E[Q(t), S.P.(t)]
o
where S.P. is the "measured" average concentration in yg/m for a sampling
period.
The performance model was programmed for. the Bunker Ramo 340 computer and
simulation and sensitivity analyses performed as described in Appendices 1
and 2, respectively.
20
-------
Simulation Results - Simulation results for 6 computer runs are summarized
in Table 2. Computer runs 1 and 2 with input concentrations of 200 and
3
100 pg/m , respectively, are -representative of the results obtainable from
a large population of samples with parameters and variables modeled as in
'able 1. The results show no overall bias, i.e., the average measured
/alue is the same as the true value, and standard deviations of 7.5 and
7.2 percent of the mean for mean values of 200 and 100 pg/m , respectively.
Computer runs 3 and 4 were made with the major bias terms controlled.
That is, neutral filter media is assumed eliminating the E(pH) term and a
pressure transducer and continuous recorder replaces the rotameter reducing
the possible error in estimating the average flow rate, E(Q), and flow-rate
reading error, E(Q ). The results show a zero bias and a standard
deviation of 5 percent of the mean value for both concentration levels.
This is a reduction of 16 percent or greater in the standard deviations of
runs 1 and 2.
Computer run 5 indicates what could be achieved if all quality control
checks recommended in the field document were followed, i.e., a continuous
flow-rate recorder is used, and there is no delay between sample collection
and analysis other than the 24-hour conditioning period. This approximates
what is felt to be the best data quality achievable under field conditions.
The results show a standard deviation of 4 percent of the mean value and a
slight positive bias (1 percent).
Simulation results do not substantiate the accuracy value given for the
reference method (Subsection 4.2, page 8191, of the Federal Register,
Vol. 36, No. 84, Part II, Friday, April 30, 1971). The stated value of
+ 50 percent of the true average concentration appears to be based primarily
on the results of a computation of error due to unequal sampling rates
performed in Reference 9. That computation is in error. Unequal sampling
rates alone should seldom result in inaccuracies greater than 4^ 7 percent
of the true average according to the calculations in the subsection on
"Unequal sampling rates" starting on page 15 of this document. Simulation
results indicate that a standard deviation of about 7 percent, or less, of
the true value should be easy to realize. This implies that inaccuracies
greater than about 21 .percent of the mean (+3a limits) should seldom occur.
Sensitivity Results - The sensitivity analysis technique used in this study
is described in Appendix 2. A sample calculation for estimating the
variability of S.P. using assumed values of variability for each parameter
in the basic equation is given in Appendix 4, page 144,-as an example of
calculating the mean and variance of nonlinear functions.
Sensitivity analyses performed with the variables modeled as given in
Table 1 show that the relative ranking of the most important variables in
terms of their influence on data quality varies as a function of the
concentration level of S.P. being measured, characteristics of the
21
-------
Table 2: SYSTEM SIMULATION RESULTS
Computer
Run
1
2
3
4
5
T IT , Simulation
Input Values -
Results
(S.P.)
yg/m
200
100
200
100
200
Q.
3
m /min
1.7
1.7
1.7
1.7
1.7
T
t
min
1440
1440
1440
1440
1440
S.P.
yg/m
200
100
200
101
202
a
yg/m
15
7.2
10
5
8
Min
yg/m
170
84
181
89
179
Max
yg/m
259
120
241
116
223
Remarks
Parameters modeled as shown in
Table 1.
E(pH), E(Q), and E(Q ) controlled by using
neutral filters and a continuous flow-rate
recorder.
Same as run 4 with no delay between
collection and analysis.
N3
S3
-------
particulates, and environmental conditions. Figure 3 summarizes the
results in terms of the percent change in the average concentration, S.P.,
when the parameter takes on its -2a and +2a values. For parameters
modeled as uniform distributions, the lower and upper values of their
ranges were used as the +2a values.
Figure 3 shows that the effects of four parameters are independent of
monitoring conditions as indicated in the box identified as S . The
importance of errors due to filter surface alkalinity, E(pH), and filter
weighing, E(W), depends on the concentration level being measured as
shown in boxes S_, S.,, and S, ; both parameters show an increasing influence
on S.P. as the concentration level being measured decreases.
Boxes Sc and S, show that the possible error in calculating the average
J D
flow rate, E(Q), and the possible error due to unequal sampling rates,
E[Q(t), 'S.P.(t)], are greater for particulates that tend to plug the
filter resulting in large flow-rate drops over the sampling period.
Assuming that the sampler is calibrated at a temperature, T, of 21°C and
used in the field at different temperatures, boxes S , S , and S0 show
/ o y
the error ranges which would result if the average temperature of the
sampling site was within the interval of -4°C to 40°C.
Loss of organic matter is a function of time as well as the original
organic content of the sample. The error levels given in boxes S and
S result from the model of E(OM) as given in Table 1 for the delay
intervals specified in each box.
Taking a specific example, if data from a particular sampling site show that
3
the average concentration is usually greater than 150 yg/m , the drop in
flow rate is usually greater than 20 percent of the initial flow rate, the
site temperature is consistently higher than the temperature of calibration,
and the delay between collection and analysis is generally greater than
14 days; then, path S, - S, - S, - Sq - S is applicable and the possible
error sources can be ranked. In this case a listing of the four most
important potential error terms in order of decreasing effect would be
E(Q), E[Q(t), S.P.(t)], E(Qc), and E(Qr).
3
It can be seen that for S.P. concentrations less than 75 yg/m (box S.)
filter surface alkalinity, E(pH), and filter weighing errors, E(W), both
become important as the first and third largest sources of possible errors.
See Appendix 2 for directions in evaluating control procedures and/or
monitoring strategies using the linear terms of a Taylor's series expansion
as an approximate model for the measurement process.
23
-------
Concentration
All Monitoring Conditions
E(QC)
E(RH)
E(T)
E(Qr)
-6 to +6% of S.P.
0 to +5% of S.P.
-1 to +1% of S.P.
-5 to +5% of S.P.
ion
rvals
E(pH)
E(W)
S, S S
2 3 4 i
25 < S.P. < 75 Mg/m3
(50 to 150 mg)
+3.3 to +10% of S.P.
+1.9 to +5.6% of S.P.
75 < S.P. < 150 pg/m3
(150 to 300 mg)
1.7 to +3.3% of S.P.
+1 to +1.9% of S.P.
S.P. > 150 yg/m3
(300 mg up)
-0 to +1.7% of S.P.
-0 to + 1% of S.P.
Flow Rate Change per
Sampling Period
Range
E(Q)
E[Q(t)S.P.(t)]
-------
Arriving At Suggested Performance Standards
Use of Collaborative Test Data - The primary documented source of data
available for use in providing standards of measurement or baseline data
for classifying measurements as either defective or nondefective is the
Collaborative study (Ref. 1) . A few remarks about these data and the
conclusions of the report are given here to aid in relating the use
of the data in the corresponding guidelines for development of a quality
assurance program.
There was no standardized procedure to guide the laboratories in the use
of the High Volume Method, e.g., no standard method for calibration of the
High Volume Sampler. The remarks in Ref. 1 concerning the sampling vari-
ation relative to location of sources and to ground level indicate the
variability of results and hence the difficulty in sampling the "same"
atmosphere for checking purposes. This remark is important relative to
the use of a mobile reference monitor as discussed in the field document.
The selection of the laboratories and the analysts was on basis of
experience with the High Volume Method of sampling suspended particulate
matter; thus, the data would be expected to show less variability than
if they were based on a random selection of field stations in a sampling
network and/or a random selection of the typical operator. Even under these
conditions of selection, data from one laboratory were considered too
deviant from the others to be included in the final analysis.
The statistical analysis of the data provided estimates of the sources of
variation of results within laboratories and among laboratories. A true
replicate cannot be accomplished. Data used in the analysis are given in
Table 3. The standard deviation of the data for each day varied with the
mean, and hence a logarithmic transformation of the data was performed
prior to further analysis. This is consistent with the fact that
particulate concentrations have been observed to be lognormally distributed.
The transformation not only makes the variances of the transformed values
homogeneous, but it also yields data which are more nearly normally
distributed.
A least squares analysis was performed on the transformed data, resulting
in a linear fit to the four data points for each laboratory. As there is
no reference value for the concentration on a given day, the mean for all
laboratories was used. This value was then used with the observed values
for each laboratory in determining the best-fit line to the concentrations
for the four days. An analysis of the variation of these data (transformed)
indicates that there is no significant variation in the means or slopes of
the fitted lines among the 11 remaining laboratories used in the analysis.
Thus one infers that the within-day variation among the laboratories is
sufficiently large so that the variations in the resulting means and
slopes can be explained by chance variation (excluding the outlier
laboratory). The fact that there is one laboratory considered an outlier
should not be totally ignored in predicting how field monitoring operations
will behave. There may be a reasonably small percentage of stations which
would behave like the outlier laboratory, that is, until sufficient
experience is gained in the use of the techniques.
25
-------
Table 3. COLLABORATIVE TESTING DATAHIGH VOLUME
METHOD FOR PARTICULATES
2
Particulate Concentration yg/m
.Laboratory wumoer
222
311
320
341
345
509
572*
575*
578*
600
787*
799
2
Mean (yg/m )
3
Standard deviation (yg/m )
Day 1
138
125
128
126
127
128
128
108
126
125
125
131
126.3
6.8
Day 2
*
80
72
75
78
74
82
73
77
72
76
76
75.9
3.2
Day 3
87
82
81
83
87
86
84
72
83
80
83
86
82.8
4.1
Day 4
114
113
112
114
124
121
112
93
111
110
117
120
113.4
7.8
Missing data and recalculated results as described in Ref. 1.
Some specific results of the analysis used in arriving at the suggested
performance standards given in the field document are as follows. The
relative standard deviation (coefficient of variation) of measurements
within a laboratory was estimated as 3 percent; this is referred to as a
measure of repeatability. The reproducibility (measure of variation among
laboratories) was 3.7 percent. The standard deviation of weight measure-
ments was about 0.8 mg (0.7 to 0.9 mg) for unexposed filters, 1.7 mg for
exposed filters. These are considered to be sufficiently different to be
explained"by other than chance occurrence. The weight losses over 4 days
were about 10 mg or 5 percent, assumed to be due to loss of volatile
material on the filters. Flow-rate calibration errors were estimated to
be 2.1 percent. Intuitively it is felt that these specific values may tend
to be on the small side relative to that expected in the field data
collection of the monitoring sites.
It is important to note that in comparing two laboratories or two
measurements within a laboratory, the variance of the difference is
the sum of the variances. In comparing a measurement to a standard or
known value, the variance of the difference is the variance of the single
measurement (see Appendix 4, page 144 for a more detailed discussion).
26
-------
Use of Modeling Data - Data from the collaborative test (Ref. 1) are a
good indication of the precision obtainable with the High Volume Method.
At this time no such collaborative test has been conducted to obtain
accuracy data. The development of a performance model and subsequent
simulation and sensitivity analyses as discussed previously was in one
respect an attempt to generate accuracy data.
The performance model contains several bias terms such as filter surface
alkalinity, loss of organic material, ambient temperature, and ambient
pressure which would not influence variability in the collaborative test
data but are important functions of accuracy.
Simulation and sensitivity data generated from the performance model were
used in evaluating alternative monitoring strategies as given in the
field document, and from these results performance standards were
suggested.
The computer simulation results show the standard deviation varying from
approximately 7.5 to 4 percent of the mean value. This is about the same
range as that of the mean square error (MSB = /TZ + crz) associated with
the 7 alternative strategies given in Table 14 of the field document
(Ref. 10). Since it is desirable to encourage high quality data, the
lower value of 4 percent of the mean was used in arriving at standards for
defining defects. A defect can result from any one of three"audit checks.
They are 1) flow-rate check, 2) calibration check, and 3) a check of the
possible loss of volatile matter between collection and analysis. The
suggested performance standards are estimates of what can be achieved
under field conditions and yet are restrictive enough to ensure the
collection of good quality data. They were not derived from actual data.
The suggested performance standards as given in the field document are
summarized in Table 4.
All suggested performance standards, control limits, etc., used in the
field document are estimated at the 3a level. If the measurement process is
operating at a level resulting in a standard deviation of 4 percent of the
actual value, then it would be desirable to define a defect as any
measured S.P. concentration deviating more than + 12 percent from the
actual value. A defect is now declared any time one audit check indicates
a possible error greater than 9 percent of the actual value (standard
deviations for the three checks were all estimated at 3 percent of the mean
and were not derived from actual data). If all three checks are in control
(i.e., each has a relative standard deviation of 3 percent or less), then
the overall standard deviation would be
V2 2 2 I
°1 + °2 + °3 = >F «5-2-
The 3aT value would be 15.6 percent of the mean which is larger but still
comparable in magnitude to the desired + 12 percent given above.
27
-------
Table 4: Suggested Performance Standards
Parameter
Definition for
Defining Defects.
Suggested Minimum
Standards for Audit Ratea
1. Flow-Rate Check
.2. Calibration Check
3. Elapsed Time Between
Collection and Analysis
'3J1
> 9
n=7, N=100; or (n-3, N-15)
n=7, N=100; or (iy-3, N-15)
n=7, N=100r or (n=3, N=15)
Parameter
Standards for
Corrigible Errors
Suggested Minimum
Standards for Audit Rates
4. Filter Surface
Alkalinity
5. Weighing of Clean
Filters
6. Weighing of Exposed
Filters
7. Data Processing
Check
6.5 <_ pH <_ 7.5
+ 1 mg
+ 2.7 mg
+ 3% of S.P.
n=7, N=100
n=7, N=100
n=7, N ^50; or (n=4, N<50)
n=7, N > 50; or (n=4, N<50)
Standards for Operation
8. If at any time d=l is observed (i.e., a defect is observed for either
d , d2 , or d3 ), increase the audit rate to n=20, N=100 or n=6, N=l
until the cause has been determined and corrected.
). If at any time d=2 is observed (i.e., two defects are observed in the
same auditing period), cease operation until the cause has been deter-
mined and corrected. When data collection resumes,'use an auditing
level of n=20, N=100 (or n=6, N=15) until no values greater than + 6
are observed in three successive audits.
10. If at any time two (2) values of d.. ., d_., or d,. exceeding + 6 or
four''' values exceeding + 3 are observed, 1) increase the audit rate to
n=20, N=100 or n=6, N=15 for the remainder of the auditing period,
2) perform special checks to identify the trouble area, and 3) take
necessary corrective action to reduce error levels.
d without a subscript as used here represents the number of defects
observed in a sample of size n.
^The number three was used in the field document; it should be changed
to four.
28
-------
The extreme possibilities with this method are:
(1) all three audits have values just equal to 9 and
of the same sign, and
(2) a defect is declared because of one audit check
exceeding 9 when in fact one or both of the other
audits are of opposite sign and magnitude resulting
in a very small overall error in the measurement.
In the first instance a total error of 27 percent would result and would
be indicative of the process being out of control when compared to the
desired +12 percent. The probability of this happening should be very
small. The second case shows that good data could be invalidated as the
result of the method used to define defects. The probability is again
small, but certainly larger than the first probability.
Use of a Mobile High Volume Reference Sampler - An alternative method for
checking the precision and accuracy of a field sampler presented for
consideration in the field document was by use of a mobile reference sampler.
Under the reasonable assumption that two adjacent samplers are sampling
approximately the same atmosphere and that the differences in the measure-
ment should result only from instrument and operator errors, the mean and
standard deviation of the difference
AS.P. = s.P.F - S.P.R
would be
Mean (AS.P.) = 0(i.e., both samplers are measuring the
same value of C by assumption)
/ 2 2
Standard deviation [0(AS.P.)] = Wo^ + °T> 5 where a and o
f r K. r K.
are the standard deviations of the measured values for field sampler and
reference sampler respectively.
Based on the results of a collaborative test (Ref. 1) showing a relative
standard deviation for repeatability of 3.0 percent of the mean value, and
assuming that the variability in the data from both samplers is about
equal, then
a(AS.P.) = \(3.0) + (3.0) = 4.2 percent of the mean value.
29
-------
A defect, defined as any check in which the difference is equal to or
greater than the 3cr value, then would be declared anytime the percent
difference, d, is equal to or greater than 12.6 (~13) as computed by
AC x x 100 > 13.
0.5(CF
Recommendations
Collaborative Test to Obtain Accuracy Data - In implementing a quality
assurance program, it is recommended that a collaborative test be designed
and conducted to determine more precisely the accuracy of the High Volume
Method. Such a test design is discussed in general terms.
VeAJgn ofi fia&tosUjCit expe/ujngjtt - One means of estimating the effects of
various alternatives or strategies on the data quality is to plan an
experimental program to assess these effects similar to the manner in which
one assesses the measurement errors/variations in a collaborative test.
Some key factors in the data quality for suspended particulate matter are
the surface pH of the filter, the environment, type of flowmeter (rotameter
or continuous recorder), and calibration technique. Other factors could
become important for a particular agency (region or site) due to the
peculiar environmental effects and their interaction with the operator
and/or equipment. The selected factors are used to illustrate a means of
studying a set of typical factors.
In order to efficiently study several factors as those above, it is
desirable to design an experiment to consider all of the factors simul-
taneously. Such an experiment is referred to in the literature as a
factorial experiment. Figure 4 below indicates how the experimental plan
can be approached for the selected factors.
This experiment would require 3x2x2x2 = 24 tests to be made, (i.e., samples
3
to be collected and analyses of the concentration in yg/m to be determined)
This number of tests would seem to be practical; however, it is possible to
reduce this number through the judicious selection of the combinations of
levels of the several factors. Such an experiment is called a fractional
factorial experiment. This fractional approach certainly could be
advisable when considering a greater number of factors and/or levels of the
factors, resulting in a larger number of tests.
The analysis of the concentration data for the above experiment follows a
reasonable standard statistical method known as analysis of variance, in
which the total variation in the data is subdivided into meaningful
components due to the various factors of interest : environment, flowmeter,
pH level, sampler (or calibration error), and the appropriate interactions
of these factors. The effect of the sampler will really include all of
30
-------
NVIRONME
l
1
YTJ)
(ENVIRONMENT
i
$
J, A
C FLOWMETER A
V 1 J
\
>i~
1
Ir
( FLOWMETER ^
V 2
i
1
(ENVIRONMENTS)
^FILTER pfA . r FILTER pH A
f SAMPLER 1 A (\
VQALIBRATIONy y
SAMPLER 2
ALIBRATION
Figure 4: Experimental Plan for Selected Factors
those errors associated with two independently operated and calibrated
samplers at the same site. Even if all of the effects could be absolutely
controlled, there would be some expected variation in the results from
one sampler to another. Although the environments cannot be controlled,
it is desirable to select them for varying conditions, e.g., degree of
stickiness of the particulates or the percent of volatile matter. This
planned environmental effect would then be studied as to how it interacts
with or affects the other factors to be studied.
Statistical Analysis of Field Data - Alternative to a collaborative test,
audit, data, and other operational data should be evaluated and analyzed in
a statistical manner as soon as possible in order to adjust, eliminate, or
add checks and standards as given in the field document.
Further Use of the Performance Model - Also, once data from the factorial
experiment or a broad spectrum of field data from quality assurance programs
are available, they should be used to model the variables in the performance
model which in turn could be used to evaluate known and hypothesized
alternative monitoring strategies.
31
-------
REFERENCE METHOD FOR DETERMINATION OF SULFUR DIOXIDE IN THE
ATMOSPHERE (PARAROSANiLINE METHOD)
General
The manual (non-continuous) measurement of atmospheric SOj by the
Pararosaniline Method is subject to the same type errors as the High Volume
Method in the previous subsection. Unlike the High Volume Method, however,
the Pararosaniline Method from sample collection through sample analysis
can be checked by sampling from a test atmosphere containing a known con-
centration of SC^. These test atmospheres can be generated with calibrated
SOy permeation tubes. But permeation tube calibration setups are expensive
and sensitive to many parameters which may prove difficult to control in
other than well-equipped laboratories. Permeation tubes are not widely
used at this time for calibrating the Pararosaniline Method.
Without a permeation tube to generate test atmospheres, a calibration curve
is developed by measuring control samples containing a known quantity of
sulfite solution. This method only validates the analysis portion of the
measurement. It will not in any way serve to detect or indicate errors
made in sample collection, sample handling, or performing calculations.
A collaborative test of the method was conducted (Ref. 11) using permeation
tubes'to generate the test atmospheres. Each participant "prepared and
measured the test atmosphere in his own laboratory with his own equipment.
All sampling periods were 30 minutes.
Results from the collaborative test probably do not reflect the variability
characteristic of field conditions and particularly 24-hour sampling periods,
Major components of variabilty which are not accounted for in the collabora-
tive test results include:
(1) Deterioration of S0_ in the collected sample which is
a function of time and temperature.
(2) Sampling system problems such as temporary plugging of
the sample line, loss of critical flow, system leaks,
or temporary loss of electrical power.
(3) Error in determining and correcting for temperature.
(4) Possible deterioration of S02 due to exposure of the
sample to direct sunlight during collection or handling.
All the above terms except (3) would act as negative biases in the measured
results. The collaborative test results show no bias.
32
-------
For much the same reasons then, the same approach was used in writing the
guidelines for developing a quality assurance program for this method as
was used for the High Volume Method. This involved performing a functional
analysis of the measurement process which consisted of identification and
modeling of important variables and parameters, development of a performance
model of the measurement process, and simulation and sensitivity analyses
using the performance model was performed as a means of evaluating the
overall measurement process. Each phase of the 'analysis is discussed
below.
Identification and Modeling of Important Parameters
Measurement of sulfur dioxide in the ambient atmosphere by the manual
pararosaniline method requires a sequence of onerations and events that
yields as an end result a number that serves to represent the average mass
of sulfur dioxide per unit volume of air over the sampling period.
The measurement process can be roughly divided into three phases. They
are: 1) sample collection, 2) sample analysis, and 3) data processing.
Sample Collection - The sample collection phase of the measurement method
contains numerous sources of error. Some errors can be eliminated by a
conscientious operator, while others are inherent and can only -be
controlled. The sources of error are:
(1) flow-rate calibration,
(2) determination of the volume of air sampled,
(3) elapsed time between sample collection and analysis,
(4) exposure of sample to direct sunlight, and
(5) entrainment or loss of sample other than by evaporation.
calibration - Flow rates for critical orifices calibrated against
a wet test meter at different times, by different individuals, but with the
same equipment showed a standard deviation of less than 2 percent of the
mean (Ref. 12). When a large population of laboratories is considered, the
variability would undoubtedly increase significantly. Rotameter calibra-
tions would not be expected to be any more- precise than critical orifice
calibrations. Large flow-rate calibration errors will be detected by the
volume/ flow-rate check as part of the auditing procedure. However, to
maintain the error at or near a minimum, the wet test meter or calibrated
rotameter used for calibration should be checked against a standard such
as a good quality soap-bubble meter at least once a quarter.
The flow-rate calibration error was modeled as a normal distribution with
a zero mean and a standard deviation of 2 percent of the mean, as
E(Q ) = N(0, 0.02)
33
-------
where E(Q ) = calibration error expressed as a decimal
fraction of the mean value, and
N (0,0. 02) = a normal distribution (see Appendix 3 for
a discussion of the normal distribution)
with a zero mean and a standard deviation
of 0.02.
& tke. votumt ofi cuA Ampt&d - The volume of air sampled is
estimated from the calibration of the critical orifice before and after
sampling (flow-rate readings before and after sampling if a rotameter is
used) and the sampling period time. Such an estimate assumes that any change
in flow rate during the sampling period is linear with time. Nonlinear
changes due to such things as temporary plugging of the line or critical
orifice from condensed moisture are not detected by this method. Also,
a system leak between the absorber and the critical orifice would introduce
an error in the calculated volume unless detected and corrected by the
operator prior to sample collection.
Errors in the calculated volume are checked as part of the recommended
auditing process by using a wet test meter on-site to measure the integrated
volume. A discrepancy in the integrated volume measured by the wet test
meter and the volume calculated in the usual manner implies that there
could be:
(1) system leaks,
(2) nonlinear changes in flow rate,
(3) an error in the sampling period timer, or
(4) flow- rate calibration error.
Each item should be checked and verified.
For the 24-hour sampling period where a calibrated rotameter is used to
read the flow rate before, during, and after the sampling period, as part
of the auditing process, only items (1) and (4) above can be detected.
There is only a small possibility of detecting a temporary change in flow
rate (item 2). The timer should be checked independently against an
elapsed time indicator at least every six months. Flow-rate calibration
error was discussed in the previous subsection.
An additional source of error in estimating the sample volume is due to the
inability to determine an average temperature or pressure for the sampling
period. This is more important for 24-hour samples than it is for the
shorter periods. The recommended method for estimating the average
temperature is to average the minimum and maximum temperatures as read
from a minimum-maximum thermometer stored in the sampling train box. The
average pressure can be taken as the average pressure for that location.
34
-------
For modeling purposes characteristic distributions and variances were
assumed for each of the error sources. The means and standard deviations
of the parameters are given in percent. Errors in the sampled volume due
to the various parameters are modeled as follows:
(1) System leaks as a uniform distribution (see Appendix 3
for a discussion of the uniform distribution) -0.04 to 0,
symbolized by
U(-0.04 to 0).
(2) Intermittent plugging of lines, loss of critical flow,
or loss of power were combined and treated as a uniform
distribution symbolized by
U(-0.04 to 0).
(3) Inaccuracies in the sampling period timer or elapsed
time indicator are not critical for 24-hour samples.
For example, a 14-minute error in the sampling period
time would represent a 1 percent error in measured
concentration. In the computer analyses the error in
the sampling period time was simulated with a normal
distribution with a zero mean and a standard deviation
of 7 minutes. It is symbolized by
E(T) = N(0, 7)
where E(T) = error in sampling period time in minutes, and
N(0,7) = normal distribution (see Appendix 3) with a
zero mean and a standard deviation of 7.
(4) Error in determining the actual average temperature of
the sampling period was modeled as a normal distribution
with a zero mean and a standard deviation of approximately
1 percent. It is symbolized by
N(0, 0.0112).
(5) Error in determining the actual average pressure of the
sampling period was modeled as a normal distribution with
a zero mean and a standard deviation of approximately
0.6 percent. It is symbolized by
N(0, 0.0057).
Ela.pA£d time, between Aampte. col£e.c£ion and anaZy^-u - It has been shown that
exposure of the sample to temperatures above about 5°C results in SO^ losses.
A loss of about 1.8 percent per day occurred at 25°C and no losses were
observed at 5°C (Ref. 13).
35
-------
It is obvious that long delays between sample collection and analysis at
unrefrigerated or uncontrolled conditions will result in sizeable errors.
Every means should be employed to minimize the time that a sample is
exposed to temperatures above 5°C.
The difference in the weight of S02 collected and the weight of S02 in the
sample at analysis was modeled as a function of the elapsed time in days
and the average temperature of exposure in centigrade. The mathematical
model for the loss of S02> E(L), is
!0; for t
0.00059 (
0.025 (t-
5°C
E(L) = ^0.00059 (t-5) (d); for 5°C £ t £ 22°C
(t-18) (d); for 22°C < t < 40°C
where E(L) = loss of SO- expressed as a decimal fraction of
the initial S02 content in the collected sample,
t = average exposure temperature in °C and modeled as
a normal distribution with a mean of 15°C and a
standard deviation of 5°C, written as N(15,5), and
d = the elapsed time in days and is mode.led as a uni-
form distribution with a range of 0 to 10, written
as U(0 to 10).
This is essentially a piecewise linear approximation of an exponential
function. When more data are available, a simple exponential model probably-
should be used.
Exp04uAe 0$ bcunptd to ctiAe.ct Au.nLigkt - Exposure of the sample to direct
sunlight during or after collection can result in deterioration of the
sample. Losses from 4 to 5 percent were experienced by samples exposed
to bright sunlight for 30 minutes (Ref. 13). Therefore, it is recommended
that the bubbler be wrapped in tin foil anytime it is exposed to the
direct sunlight for more than the 1 or 2 minutes that might occur in the
normal transfer of 24-hour samples from the sampling train box to the
shipping block.
This error (see X(15) in Table 5, page 40) was treated as a percent weight
loss in the collected SO^- It was modeled as a uniform distribution
ranging from -0.05 to 0, U(-0.05 to 0), which is equivalent to a 0 to
30 minute exposure according to the above referenced data.
36
-------
EyitMU.me.nt OH to&& o& Ample. - Any loss of sample by means other than
evaporation during sample collection should result in the operator's
invalidating the sample. Large errors can result when:
(1) the operator fails to distinguish between sample loss
and evaporation and brings the absorber to the mark
with fresh TCM,
(2) the operator fails to bring the absorber to the mark
and the laboratory assumes that he did and that part
of the sample was lost during shipment, or
(3) the operator is careless in bringing the exposed absorber
to the mark (e.g., ^f 1 mil represents a 2 percent error in
a 24-hour sample).
This error (see X(12) in Table 5, page 40, was also treated as a percent
weight loss and modeled by a uniform distribution (see Appendix 3) ranging
from -0.05 to 0 symbolized by
U(-0.05 to 0).
Sample Analysis - To realize a high level of precision and accuracy from the
analysis phase of the method, the analyses must be done carefully, with
close attention to temperature, pH of final solution, purity of the chemicals
and water, and standardization of the sulfite solution (Ref. 13).
Important parameters in the analysis phase include:
(1) purity of the chemicals and water,
(2) spectrophotometer calibration,
(3) pH of the solution being analyzed,
(4) temperature at analysis compared to temperature
at calibration, and
(5) volumetric measurement of exposed sample solution
and aliquot.
Pu/Ufy ofa chmicjaLb and wateA - Purity of the chemicals and water is
extremely important in obtaining reproducible results because of the high
sensitivity of the method. As recommended in the Operations Manual of the
field document (Ref. 14), the purity of the pararosaniline dye is checked and,
if necessary, purified and assayed before use. The water used must be free
of oxidants and should be double-distilled when preparing and protected from
the atmosphere when stored and transferred from container to container. All
other chemicals should be ACS grade and special care exercised not to contam-
inate the chemicals while in storage or when removing portions for reagent
preparation. Errors due to these sources would appear as excessive vari-
ability in the calibration curve, measurement of reagent blanks, and
measurement of control and/or reference samples.
37
-------
- The spectrophotometer should be adjusted for
100 percent transmittance when measuring a sample cell of pure water prior
to each set of determinations. Also, the calibration of the wavelength
scale and the transmittance scale should be checked periodically or any time
reference samples cannot be measured within prescribed limits after checks
of the reagents and water have failed to identify the trouble.
The calibration of the wavelength scale can be checked by plotting the
absorption spectrum (in the visible range) of a didymium glass which has .
been calibrated by the National Bureau of Standards.
The transmittance scale can be checked using a set of filters from the
National Bureau of Standards.
Spectrophotometer calibration error would result from deterioration of the
spectrophotometer between calibrations and should be detected by an
increased variability in measuring reference and/or control samples.
pH 0^ tilt &oJLu£icm biung analyzid - For this method the maximum sensitivity
occurs when the pH of the final solution is in the region of 1.6+0.1
(Ref. 13). If care is exercised in reagent preparation and the analysis
phase, the pH should always be within the above limits.. Phosphoric acid
provides the pH control; hence, it should be checked first when the pH is
detected outside the above range.
- Temperature affects tne rate of color formation and fading
of the final color. Also, the reagent blank has a very high temperature
coefficient. Therefore, it is important that the temperature at analysis
be within + 2°C of the temperature at which the calibration curve was
developed. If the normal room temperature varies more than + 2°C from a
set value, it is recommended that a constant-temperature bath be used if
a high degree of accuracy is desired.
Large variations in temperature would be detected as errors in measuring
control samples and/or reference samples.
All of the above topics are part of the analysis phase of the measurement.
Therefore, it appeared feasible to incorporate these error terms into the
measured absorbance of a sample through the variability in the slope and
intercept of the calibration curve, the error in measuring control samples,
and the variability in reagent blank measurements. All values used in the
models were taken from collaborative test results (Ref. 11) as explained in
the subsection Use of Collaborative Test J)ata. page 44. The individual
models are:
(1) The variability in the slope of the calibration curve
was treated as a normal distribution with a mean of
0.03 absorbance unit and a standard deviation of 0.00082
absorbance unit. It is symbolized by
N(0.03, 0.00082).
38
-------
(2) The calibration curve intercept on the absorbance axis
was modeled as a normal distribution (see Appendix 3 for
a discussion of the normal distribution) with a mean of
0.163 absorbance unit and a standard deviation of 0.012
absorbance unit. It is symbolized by
N(0.163, 0.012).
(3) A standard deviation of 0.4 yg SO- for the measurement
of control samples multiplied by the average slope
(0.03 absorbance unit/yg SO-) yields a value of 0.012
absorbance unit as the standard deviation for the vari-
ability in the measured absorbance of control samples.
This was modeled as
N(0, 0.012).
Measured absorbance, A, of field samples was modeled as
A = N(0.03, 0.00082) x (yg SO^ + N(0.163, 0.012) + N(0, 0.012)
where yg SO- is the actual quantity of SO- in the aliquot of the sample.
Vo£ume.&u.c. me.cu>u/Lmen&> - In the analysis of a 24-hour sample, a 5 mX,
aliquot is taken from the sample for analysis. The results are then
multiplied by a dilution factor of 10 to give the SO- concentration of
the field sample. Treating the dilution factor as
50 + a-.
10 + 0 = r-
2 2
means that a = a. + a- . Assuming that a. = a_ = 2 percent of the average
value, i.e., 5 m£, + .1 m£ and 50 m£ + 1 m£, then the standard deviation of
the dilution factor is 2.8 percent of 10 or a = 0.28. The dilution factor
was modeled as a normal distribution with a mean of 10 and a standard
deviation of 0.28, written as
D = N(10, 0.28)
where D = a dimensionless dilution factor (normally 10
for a 24-hour sample) including error, and
N(10,0.28) = a normal distribution (see Appendix 3) with a
mean of 10 and a standard deviation of 0.28.
39
-------
Table 5: PARAMETER EFFECTS MODELS AND VARIABLE MODELS AS USED
IN SIMULATION AND SENSITIVITY ANALYSES (SO,,)
Parameter Effects Models
Variable Models
A' = X(l) x W + X(2) 4- X(3)
a
A' = X(2) + X(4)
o
D'
= N(10, 0.28)
= Q [1 + X(6)
K.
where
X(7) + X(8) + X(9) + X(10)
Q = actual average flow rate
through absorber,
and
_
measured (calculated)
flow rate
E(T) = N(0, 7)
(yg SO./m3)
X(12) + E(L)
where W = actual weight of SO.
collected,
; V = Q XT = Q x[l440+X(ll)],
3. a a 3.
(yg S00/m ) = true S00 concentration, and
2. ?£ £
(Q; for X(13) <_ 5°C,
E(L) =<0.0059(X(13)-5)(d) for 5°C
-------
Data Processing - Data processing, starting with the raw data through the
act of recording the-measured concentration on the SAROAD form, is subject
to many types of errors. The approach recommended in the Operations Manual
of the field document means that one can be about 55 percent confident that
no more than 10 percent of the reported concentrations are in error by more
than + 3 percent due to data processing error.
The magnitude of data processing errors can be estimated from, and controlled
by, the auditing program through performance of periodic checks and making
corrections when large errors are detected. A procedure for estimating the
bias and standard deviation of data processing errors is given in the
Management Manual of the field document. It was not included in the per-
formance model for this analysis, but could be included in the overall
assessment of data quality as explained in the field document.
All parameter and variable models are summarized in Table 5.
Performance Model
A mathematical performance model of the measurement process incorporating the
error terms discussed in the previous subsection and summarized in Table 5
was derived from the basic equation
3 (A - A ) (103) (B )
(yg SO /mJ) = ?- 5- x D
z m v
o
where (pg S0,/m ) = measured S00 concentration,
^ m ^
A = absorbance of sample,
A = absorbance of reagent blank,
B = reciprocal of the calibration curve slope,
s
D = dilution factor, and
V = volume of air sampled.
Introducing the error terms gives
, (A1 - A') (BJ (D1)
where A1, A1, and D' are defined in Table 4,
o
B = reciprocal of the calibration curve slope,
S
QR = correct average flow rate, and
T = correct sampling period time.
41
-------
Simulation Results - Simulation and sensitivity data were obtained for
o
3 levels of S02 concentration; namely, 800, 400, and 100 yg S02/m .
These data are believed to be representative of the results obtainable
from a large population of sampling sites. The parameter and variable
models are as given in Table 5. Simulation results are summarized in
Table 6. The results show a negative bias of about 5 percent of the true
value, i.e., the average measured S02 concentration. Standard deviations
of 8.9, 9.5, and 24 percent of the true S02 concentrations were obtained
for concentration levels of 800, 400, and 100 yg/m , respectively.
Table 6: SYSTEM SIMULATION RESULTS (SO )
Computer
Run
1
2
3
Input Values Simulation Results
Y (true)
yg S02/m3
800
400
100
Q
cm3 /min
200
200
200
Tt
Min
1440
1440
1440
Y (measured)
yg S02/m3
757
376
92
a
yg S02/m3
71
38
24
Min
yg/m3
589
290
30
Max
yg/m3
954
491
148
Sensitivity Results - The sensitivity analysis technique used in this study
is described in Appendix 2. A sample calculation for estimating the mean
and variance of nonlinear functions is presented in Appendix 4, page 154,
using suspended particulates as an example.
Table 7 summarizes the sensitivity results by listing the 5 variables
effecting the largest variability in the measured data for each of the
three levels of SO. concentration.
In all three runs the 5 most significant variables accounted for the major
portion of data variability (see Appendix 2). The order of ranking changed
significantly from one concentration level to another for the 5 most
important variables while the order of the 8 variables having the least
effect did not change.
See Appendix 2 for directions in evaluating control procedures and/or
monitoring strategies using the linear terms of a Taylor's series expansion
as an approximate model for the measurement process.
42
-------
Table 7: Sensitivity Analysis Results (S0_)
Computer
Run
1
2
3
Input Values
Y (true)
800
400
100
cm /min
200
200
200
Tt
mm
1440
1440
1440
Variables in Decending Order of Importance
1
X13
X13
X4
2
X5
X4
X3
3
XI
X3
X13
4
X4
X5
X5
5
X6
XI
XI
6
X14
X6
X6
7
X3
X14
X14
8
X15
X15
X15
9
X12
X12
X12
10
X9
X9
X9
11
X7
X7
X7
12
X8
X8
X8
13
X10
X10
X10
14
Xll
Xll
Xll
15
X2
X2
X2
Variables appearing in the top 5 for each run are:
X13 = Average temperature that the sample is exposed to between collection and analysis.
X4 = Variability in measuring reagent blanks.
X5 = Dilution factor.
X3 = Variability in measuring control samples.
XI = Slope of calibration curve.
X6 = Calibration error.
-------
Arriving at Suggested Performance Standards
The suggested performance standards as given in the field document are
reproduced in Table 8.
The standard proposed for the volume/flow-rate check was not based on
actual data. The equipment error (i.e., wet test meter, rotameter or
critical orifice) should be small in comparison to system errors such
as leaks, lines plugging, etc. It was felt that the difference in the
volume as measured by the audit and that calculated in the normal manner
could not be lower than 3 percent of the audited value on the average.
A relative standard deviation of 3 percent was assumed, making the value
for defining a defect 9(3a).
Also, the suggested standard for correcting data processing'errors was
an estimate. It was desired to use a standard large enough to accept
round-off errors yet small enough to prevent data processing errors from
becoming a major component of overall system error.
Use of Collaborative Test Data - The results of a collaborative test of
the manual pararosaniline method of analyzing SO- are given in Reference
11. Some specific remarks concerning this study and its use for the field
document on S0_ are contained herein. Fourteen laboratories provided the
results, based on analysis of pure synthetic atmospheres using the 30-
minute sampling procedure and the sulfite calibration method. The
permeation tubes provided the method of generating test atmospheres. It
should be remarked that this procedure was not recommended in the guide-
line .document because of its high cost. Furthermore, it appears that
there may have been considerable difficulty in the generation/measurement
of the test atmospheres resulting in a large variability between labs (or
large reproducibility measures). Seventeen (17) labs participated in the
first test. Based on the results of this test and the lack of uniformity
in test procedure, this test was voided and a second test program with 14
participants was run. A familiarization session of three days in length
was conducted between the first and second tests. This experience in
running the test twice indicates that until the field laboratories gain
the needed experience, their results can be subject to large variability
and inaccuracies.
The procedure required that a separate calibration curve be prepared for
each day. Also certain reagents were to be made fresh each day. The
concentrations were nominally 150, 275, and 820 yg S02/m3. A total of 378
determinations were made (14 labs * 3 concentrations x 3 days * 3 replications
= 378). Two cases of atypical data were noted and corrected by an appropriate
statistical procedure. Four instances of inadvertent errors in arithmetic
operations were noted and corrected. (A checking procedure should uncover
such errors in routine field operations, e.g. , such as the procedure suggested
in the field document).
44
-------
Table 8. SUGGESTED PERFORMANCE STANDARDS
Standards for Defining Defects
1. Volume/Flow-Rate Check; |(L | > 9
II &
d.. I > 1.2 yg SO-
Standard for Correcting Data Processing Errors
3. Data Processing Check; |cU.| > 3
Standards for Audit Rates
4. Suggested minimum auditing rates for data error;
number of audits, n = 7; lot size, N = 100; allowable number of
defects per lot, d = 0.
Standards for Operation
5. If at any time d = 1 is observed (i.e., a defect is observed) for
either d or d_., increase the audit rate to n = 20, N = 100 until
the cause has been determined and corrected.
6. If at any time d = 2 is observed (i.e., two defects are observed in
the same auditing period), stop collecting data until the cause has
been determined and corrected. When data collection resumes, use
an auditing level of n = 20, N = 100 until no defects are observed
in three successive audits.
7. If at any time either one of the two conditions listed below is
observed, 1) increase the audit rate to n = 20, N = 100 for the
remainder of the auditing period, 2) perform special checks to
identify the trouble area, and 3) take necessary corrective action
to reduce error levels. The two conditions are:
(a) two (2) d values exceeding ± 6, or
j- J
four (4) d . values exceeding + 3.
J-J
(b) two (2) d?. values exceeding ± 0.8 yg S07, or
t
four (4) d,. values exceeding +0.4 yg S00.
A3 2
TS
yg S02 = total quantity of S02 in the reference sample.
A value of three was used in the field document; it should be changed
to four.
45
-------
Forty-two independent calibrations of absorbance vs.
concentration were run (14 labs x 3 days). From these calibrations
the mean standard error of estimate, mean slope and its standard
deviation, mean intercept and its standard deviation were determined.
Mean std. error of estimate = 0.010 absorbance units.
M , . , , n -- . (0.00082 absorbance units/yg S00 (within lab)
Mean slope + std. dev.: 0.03 +
-------
Rep£/tcu£e, iiod.u.cAbJJLitij HM.OI - The errors of
replication (within day- lab) , repeatability (between day-within lab) ,
and reproducibility (between lab) are given below for convenient reference.
All errors are assumed to be linear functions of the concentration of S0_
(denoted by Y) and are expressed in pg SCL/m .
a (replication) = 10(0.7 + 0.001Y)yg S02/m3
0 (repeatability) = 21(0.7 + 0.001Y)pg S02/m3
a (reproducibility) = 41(0.7 + 0.001Y)yg S02/m3
In order to compare differences in two measurements corresponding to the
above, each of the above must be multiplied by /2 (1.96) = 2.77, for
including 95 percent of the differences, and /2~ (3) = 4.23 for including
99.7 percent of the differences.
Sta£it>£i.c.aJt anaJtyAiA - The dependence of the variability of the measure-
ments on the concentration suggest a transformation of all of the data
prior to refined analysis. A logarithmic transformation of Y (observed
concentration) was performed,
Z = 1000 logg(7 + 0.01Y) .
A linear effect model was employed, i.e., the observed concentration was
assumed to be equal to an overall mean + a lab effect + material
(concentration) effect + day effect + lab x material interaction effect
+ day x material interaction effect + replicate effect. All the effects
were found to be significant.
In all, three methods of statistical analysis were performed, and the
resulting estimates of replication, repeatability, and reproducibility
errors were plotted. (See Fig. B-3 in Reference 11.) These results were
in good agreement.
No statistically significant systematic error or bias was determined from
these data.
One should refer to Reference 11 for the detailed data and results.
47
-------
Recommendations
The auditing scheme as given in the field document does not effectively
check the overall data quality unless the other special checks are made
and the suggested control limits are adhered to.
An improvement in the auditing scheme would be realized if equipment were
available to measure directly the sampled volume without invalidating the
sample and at flow rates as low as 200 cm /min as is re-quired for the
24-hour sampling period.
Data derived from the implementation of a quality assurance program should
be analyzed in a statistical manner as soon as practical to readjust,
eliminate, or add to the special checks and control limits suggested in
the field document.
Also, the performance model should be verified using data of known quality
from the quality assurance program and used for evaluating known and
hypothesized alternative monitoring strategies.
48
-------
REFERENCE METHOD FOR THE MEASUREMENT OF PHOTOCHEMICAL OXIDANTS CORRECTED
FOR INTERFERENCES DUE TO NITROGEN OXIDES AND SULFUR DIOXIDE
General
The reference method for the continuous measurement of ozone utilizes the
chemiluminescent reaction between ethylene gas and ozone. Chemiluminescence
produced by mixing ethylene gas and the sample air is detected and amplified
by a photomultiplier tube. The magnitude of the chemiluminescent signal is
proportional to the ozone concentration.
Other pollutants normally found in ambient air do not interfere with this
reaction. Therefore, the total measurement process can be audited with two
checks. The two checks are 1) measurement of control samples, .i.e., mea-
suring samples of known 0~ concentrations, and 2) data processing checks.
Results from the two audit checks are used to estimate the quality of the
reported data.
Arriving at Suggested Performance Standards
The performance standards suggested in the field document (Ref. 15) are
given in Table 9. Each standard is discussed individually.
Standard for Defining Defects - A defect is declared anytime a control
sample cannot be measured within ±(0.01 + 0.075 x ppm 0_) of its true value.
This standard was derived from a provisional report of a collaborative test
of the reference method (Ref. 16) and from the analysis of a set of data
provided by EPA. Specifically, the collaborative test results showed a
standard deviation of (0.009 + 0.07 x ppm 0_) for repeated measurements
of the same sample by a given laboratory. The standard for defining a
defect is given at the 3a level. This can be changed as the manager or
supervisor wishes. Analysis of the ozone data as given below shows that
if the coefficient of variations as computed in Tables 14 and 16 (pages 54
and 55) are combined to arrive at a standard deviation for measuring con-
trol samples, agreement with the standard deviation from the collaborative
test is very good.
Analysis of Ozone Data - Calibration data for a chemiluminescent ozone
monitor were taken on three daysOctober 31, November 1, and November 15,
1972for six sleeve setting positions of the ozone generator. Three repeat
or replicate measurements were taken each day (with the exception of one day
for one sleeve setting) with both the KI Method and the monitor. These data
were provided to RTI by EPA for analysis and used in developing standards for
the ozone field document. The data are given below, followed by an analysis
of the variability of the measurements between and within days (replicates)
for each method.
49
-------
Table 9. SUGGESTED PERFORMANCE STANDARDS
Standards for Defining Defects
1. Measurement of Control Samples; d >_ + (0.01 + 0.075 x ppm* 0 )
Standard for Correcting Data Processing Errors
2. Data Processing Check; |d, | >_0.01 ppm
2j
Standards for Audit Rates
3. Suggested minimum auditing rates for data error; number of audits,
n=7; lot size, N=100; allowable number of defects per lot, d"*" = 0.
4. Suggested minimum auditing rates for data processing error;
number of audits, n=2; lot size, N=24; allowable number of defects
(i.e.,
Standards for Operation
5. Plot the values of d
0.05
I 0.04
Q.
0.03
o
d. . I >_ 0.01 ppm) per lot, d=0.
lj
on the graph below.
0.02
to
:D
-------
Table 10. CALIBRATION DATA FOR CHEMILUMINESCENT OZONE MONITOR
Sleeve
Setting
0
10
10
10
20
20
20
30
30
30
50
50
50
75
75
75
10-30-72
ppm 03
(KI)
0.000
.039
.033
.035
.086
.088
.085
.140
.139
.141
.254
.255
.249
.'369
.360
Monitor
DC pa
0.0
7.5
7.7
7.7
17.3
17.3
17.7
27.5
27.5
28.0
47.5
48.0
47.8
71.5
71.5
11-1-72
ppm 03
(KI)
0.000
.040
.045
.045
.097
.094
.094
.149
.150
.155
.259
.257
.258
.384
.384
.388
Monitor
DC pa
0.0
7.8
8.0
8.0
17.7
17.6
17.6
27.8
27.9
27.9
47.0
47.0
46.9
70.2
70.2
70.3
11-15-72
ppm 03
(KI)
.000
.043
.048
.047
.096
.096
.097
.148
.149
.145
.253
.251
.251
.386
.383
.379
Monitor
DC pa
0.0
8.3
8.4
8.5
19.0
18.9
18.9
29.3
29.3
29.2
49.4
49.6
49.5
74.3
74.3
74.6
An analysis was first performed for all of the data, excluding the zero
sleeve setting and the third measurement in each set of three measurements.
This last omission was for convenience of having the same number of measure-
ments for each day for each method. It is obvious from examination of the
data that the within-day, variation is small compared to day-to-day variation.
It is recognized that the within- and between-day variations depend on the
sleeve setting, i.e., the ozone concentrations; and thus a second set of
analyses were performed for each sleeve setting. In this second set of
analyses all of the data were used. The results of both sets of analyses
are tabulated below along with some summary remarks.
The analysis of variance, Tables 11 and 12, subdivides the total variation of
the data into its component parts. There are 30 measurements, and the total
variation corrected for the mean has an associated 30-1=29 degrees of free-
dom. Similarly, there are 3-days, 3-1=2 d.f., 5 levels, 4 d.f. The inter-
action D x L is a measure of the variation of the measurements from day-to-
day for each level. If for example, the day-to-day effects were identical
for each level in that the same added effect could be used for each day for
all levels to obtain the results on the next day, this interaction effect
would be zero or negligible. However, it is expected that the effect will
51
-------
Table 11. ANALYSIS OF VARIANCE OF KI DATA
Source of Variation (SV)
Total (Corrected for Mean)
Days (D)
Levels (L)
D x L
Replicates Within Days
Table 12.
SV
Total (Corrected for Mean)
Days (D)
Levels (L)
D x L
Residual
Sum of Squares (SS)xlO
436,783.5
578.4
435,808.3
296.3.
100.5
ANALYSIS OF VARIANCE OF
SS x 102
1,567,899.4
2,316.1
1,564,740.9
821.9
20.5
Degrees of Freedom (DF) Mean
29
2
4
8
15
CHEMILUMINESCENT MONITOR DATA
DF
29
2
4
8
15
Square (MS)xlO
15,061.5
289.2
108,952.1
37.0
6.7
MS x 102
54,065.5
1,158.1
391,185.2
102.7
1.35
-------
be different depending upon the magnitude of the measurement and hence the
second set of .analyses. The replicates within days have 15 d.f., one for
each set of duplicates. The relative magnitudes of the mean squares
(obtained by dividing the sum of squares by the respective d.f.) are as
expected. Obviously the effect of levels was expected and is really not of
interest. The effect of day exceeds that of D x L, and the variation
between duplicates is very small.
The second set of analyses, Tables 13 and 15, provides estimates of the
variation between and within days, and then total for each sleeve setting
and for each method. The expected mean squares, E(M.S.), are derived and
used to obtain the estimates of the two components of variation. In each
case, except for sleeve setting 75, there were 9 measurements, -three days
and three within each day. Thus, there are 8 d.f. for total corrected for
the mean (not shown in table), 2 d.f. for between days, and 6 d.f. for within
days. In all cases the between-day variation was considerably larger than
the within-day variation. The results are summarized in Tables 14 and 16
giving the estimate of the total variance of measurements made on an arbi-
trary day, i.e., the between-day effect and the' within-day effect are added
to obtain an estimate of repeatability. The total variance d2 is given in
the first row, followed by the standard deviation 6 = ', the mean level
(ppm 0» for the KI method and ya for the chemiluminescent ozone monitor) ,
and lastly the coefficient of variation, 100 6/x, in %. The coefficient of
variation enables a direct comparison of data in different units.
One can readily observe that for the KI method, 6 is almost constant over
the sleeve settings (ranges of ppm 0 ) , and hence the coefficient of variation
decreases from 14.1 to 2.9% (1.5% at 50 sleeve setting). On the other hand,
for the ozone analyzer data, d increases steadily from 0.40 to 2.12 via DC,
and the coefficient of variation is reasonably constant, ranging from 5.0%
at sleeve setting 10 to 3.1% at 75 sleeve setting (2.7 at 50). If these
results can be found to be consistent with results at field monitoring
stations, it would suggest restricting the use of the KI Method to high
concentrations (above about 0.1 ppm 0~) and using the ozone generator cali-
bration curve (i.e., the average calibration curve as. discussed in the
field document) for low concentrations when calibrating analyzers. Further
data would be needed to check out this inference. In any case the coeffi-
cients of variation of the two measurements methods are about the same order
of magnitude for sleeve settings 20, 30, 50 and 75. These estimates can be
used for standards until additional field data are obtained.
Standard for Correcting Data Processing Errors - In the field document it
was assumed that the largest source of error in data processing was in reading
the hourly average from a strip chart recording. For a dynamic strip chart
trace, estimating the hourly average can be highly subjective. The standard,
0.01 ppm, represents 2 percent of full scale for an analyzer range >of 0 to
0.5 ppm. This, in turn, represents two divisions on the strip chart trace.
This standard is no more than an estimate.
53
-------
Table 13. ANALYSIS OF VARIANCE OF KI DATA FOR EACH SLEEVE SETTING
Sleeve Setting 10
SV_ SS x IQ6
Days 172.7
Within Days 49.3
DF_
2
MS_ x_10
86.3
8.2
6
cr,
W
E (MS) Estimates (x 10 )
CTW+3°B ^=26.0
= 8'2
Sleeve Setting^ 20
Days
Within Days
Sleeve Setting 30
Days
Within Days
176.9
11.2
198.3
31.3
2
6
2
6
88.4
1.9
99.1
5.2
Same
Same
= 28.8
= 1.9
= 31.3
= 5.2
Sleeve Setting 50
Days 69.7 2 34.8
Within Days 25.3 6 4.2
Same
= 10.2
= 4.2
Sleeve Setting 75
Days 581.2 2 290.6
Within Days 75.7 5 15.14
a" = 104.9
D
- 15.14
~ 2 ~2 1/2
Table 14. ESTIMATE OF STANDARD DEVIATION FOR REPEATABILITY (afi + a>
-2 -2-2, . .2
a°B W(pP 3)
a(ppm 03) =
Mean=x (ppmO_) =
100 0/x (%) =
Sleeve Setting
10 20
34.2xlO~6
5.85xlO~3
41.7xlO~3
14.1
30.7x10 6 36
5.54xlO~3 6
92.6xlO~3 146
6.0 4
30
.Sxio"6
.04xlO~3
.2xlo~3
.1
50
I4.4xl0~6
3.8xlO~3
254.1xlO~3
1.5
75
120.0xlO~6
11.0xlO~3
379.1xlO~3
2.9
54
-------
Table 15. ANALYSIS OF VARIANCE OF CHEMILUMINESCENT
OZONE MONITOR DATA FOR EACH SLEEVE SETTING
Sleeve Setting 10
SV SS x 10
Days 89 . 6
Within Days 7.3
Sleeve Setting 20
Days 398
Within Days 12
Sleeve Setting 30
Days 456
Within Days 18
Sleeve Setting 50
Days 1006 . 3
Within Days 15.3
Sleeve Setting 75
Days 2568.8
Within Days 0.7
DF MS x 10 E (MS).
2 44.8 aj+3cj
" 4
2 199 Same
6 2 Same
2 228 Same
6 3 Same
2 503.1 Same
6 2.55 Same
2 1284.4 a2 + 2.62502
5 0.14 a2
Table 16. ESTIMATE OF STANDARD DEVIATION FOR REPEATABILITY
10
099 *) 9
a =a +atT(ya) 5.7x10
5(ya) = .40
Mean=x (via) = 8.0
100 o/x (%) = 5.0
Sleeve Setting
20 30 50
67.7xlO~2 78.0xlO~2 169.4xlO~2
.82 .88 1.30
18.0 28.3 48.1
4.6 3.1 2.7
Estimate x 10
a2 = 14.5
o2 = 1 2
65.7
2.0
75
3
166.85
2.55
a2 = 489.2
D
a2 = 0.14
-2 -2 1/2
75
489.4xlO~2
2.21
72.1
3.1
55
-------
Recommendations
When the collaborative test results (Ref. 17) are published, they should be
studied and adjustments made, if deemed necessary, in the standards and/or
limits given in the field document.
Data derived from the implementation of a quality assurance program should
be analyzed in a statistical manner as soon as possible so as to readjust,
eliminate, or add to the special checks and control limits given in the
field document. One area needing further work involves the method of
calibrating the ozone generator. A primary calibration using the KI method
to monitor the output of an ozone generator is recommended in the reference
method. The calibration data provided by EPA and presented in the previous
subsection are data taken by experienced personnel in the laboratory under
reasonably well controlled conditions. These data yield coefficients of
variation of 14 percent and 6 percent at ozone concentration levels of
about 0.04 ppm and 0.09 ppm, respectively. Under field conditions and
with less experienced personnel the variability would be expected to
increase significantly.
An ozone generator calibrated against the primary standard (i.e.,. the
buffered KI method) is being supported as a secondary standard for
calibrating ozone monitors (Ref. 17). Analysis of the set of calibration
data presented previously indicates that for ozone concentrations less
than about 0.1 ppm the coefficient of variation (Table 14) for the KI
method is larger than the coefficient of variation (Table 16) of the
generator output as monitored by a chemiluminescent analyzer. If these
data can be verified on a larger scale, it would appear that developing
an average calibration curve for an ozone generator fron at least 5 primary
calibrations, then using the generator as a secondary standard for cali-
brating the monitors would increase the precision over that obtained by
performing a single primary calibration of the analyzer.
56
-------
TENTATIVE METHOD FOR THE CONTINUOUS MEASUREMENT OF NITROGEN DIOXIDE
(CHEMILUMINESCENT)
General
A nitrogen dioxide analyzer operating on the chemi luminescent principle
contains a photomultiplier tube for measuring the light energy resulting
from the chemiluminescent reaction of nitric oxide (NO) and ozone (0,) .
The measurement cycle begins with a direct measure of NO in the sample for
a short period of time. During the second part of the cycle, total nitro-
gen oxides (NO ) is determined by first converting N0_ in the sample air
X £
to NO and measuring total NO. Subtracting the original value for NO from
the NO value gives a
A
ment cycle (Ref. 18).
the NO value gives an estimate of the NO- concentration for that measure-
A £
Overall data quality from this measurement method can be evaluated with
two special checks. They are: 1) measurement of control samples and
2) data processing checks (Ref. 19) .
Arriving at Suggested Performance Standards
Suggested performance standards given in the field document are reproduced
in Table 17. This method has not been collaboratively tested. The fol-
lowing RTI in-house data were used to arrive at a suggested performance
standard for measuring control samples.
A limited amount of precision data were obtained for two NO- analyzers from
different manufacturers. They will be referred to as analyzer A and
analyzer B in the following discussion. A series of zero and span determi-
nations was obtained. The data are given in Tables 18 and 19. These data
were analyzed to determine the variation between days and within days
(three replicates or voltage readings were taken at each setting or level
of N0? concentration). All of the analyses are in volts. The results are
converted to ppm in the final stage of the analysis.
Table 20 contains an analysis of analyzer A data for 0.0, 0.10, and 0.40 ppm
NO-. The total variation of the analyzer readings corrected for the mean
is subdivided into variation between days and within days. The sum of
squares (S.S.), degrees of freedom (D.F.) and mean square (M.S.) are given
for each source. From these values of the mean squares, one can estimate
the variance of the analyzer readings within and between days. These
estimates are summarized in Table 22. Tables 21 and 23 contain similar
analyses for analyzer B.
The summary Tables, Nos. 22 and 23, give an estimate of the within-day
standard deviations. (That among the three readings at one setting or
level of N02 concentrations, d , the estimate of the between day standard
deviation, & , the total d = /d£ + dz , the mean analyzer reading, X, and
15 li W
57
-------
the coefficient of variation, C.V. The 3's can be converted from volts to
ppm by dividing each value by 2 (i.e., the relative variation in scales).
One readily observes that for both analyzers 8 increases with the level of
N02 concentration; this is the primary reason for breaking down the analysis
into each level. However, the coefficient of variation (ratio of the esti-
mated standard deviation to the mean expressed in percent) does not appear
to be constant. In fact, it appears to decrease. The d's are plotted as a
function of the true concentration levels as shown in Figure 5.
There is a considerable variation between the precisions of the two analy-
zers. In order to obtain some tentative standards, the two values (one
for each analyzer) were averaged. A line was fitted to the resulting
averages, and the following equation relating the precision to concentra-
tion was derived:
6 = (0.003 + 0.015CNQ ) ppm .
The coefficients are one-half of those given in Figure 5 in order to relate
the variation in ppm to that expressed in volts.
The suggested performance standard for measuring control samples is given
in Table 17.
Recommendations
This is a relatively new method for measuring N0_. It has not been sub-
jected to a collaborative test. This should be the most logical step in
evaluating the method.
An integrating flask in the sampling line of sufficient size to integrate
NO spikes of duration less than or equal to the cycle period should be made
standard equipment.
As an alternative to installing an integrating flask, a study should be
made of the best way to treat negative values of NO- when computing hourly
averages. Presently there is no recommended method. In some instances,
all negative values are dropped (i.e., treated as zero). This would gen-
erally result in a higher-than-true measured hourly average concentration.
58
-------
Table. 1.7. SUGGESTED PERFORMANCE STANDARDS
Standards for Defining Defects
1. Measurement of Control Samples; + (0.009 + 0.045 x fr \ *)ppm
-i-J "
Standard for Correcting Data Processing Errors
2. Data Processing Check; | d« . | ^0.01 ppm
Standards for Audit Rates
3. Suggested minimum auditing rates for data error; number of audits,
n=7; lot size, N=100; allowable number of defects per lot, d+ = 0.
4. Suggested minimum auditing rates for data processing error; number
of audits, n=2; lot size, N=24; allowable number of defects (i.e.,
Id-.| j^ 0.02 ppm) per lot, d=0.
Standards for Operation
5. Plot the values of d
on the graph below.
Q.
0-
0.03 -
^0.02
-o
u_
o
0.01
0.1 0.2 0.3 0.4
MEASURED N02 CONCENTRATION (ppm)
0.5
6. If at any time during the auditing period
(a) one defect (d=l) is observed (i.e., a plotted value
of cL . is in the defect region of the graph),
(b) two plotted points fall in the region between the 2a
and 3o lines, or
(c) four d . values fall outside the acceptable region,
increase the audit rate to n=20, N=100 until the cause has been
determined and corrected.
7. If at any time two defects (d=2) are observed (i.e., two d
values plot in the defect region in the same auditing period),
stop collecting data until the cause has been determined and
corrected. __
is the calculated output concentration of the calibration system
used for the jth audit.
' W.
x /oj
An unsubscripted d represents the number of defects observed from n audits
of a lot size of N.
59
-------
Table 18. DATA FOR ANALYZER A
Date
4/25/73
4/27/73
5/ 7/73
5/ 8/73
5/ 9/73
5/11/73
5/14/73
N02 Level
(ppm)
Zero NO
.10 N02
Zero N02
.10 N02
Zero N02
.10
Zero NO-
.10
.40
Zero NO-
.10
.40
Zero NO-
.10
Zero NO
.40
Zero NO-
.10
.40
Zero NO.
.10
.40
Analyzer Reading (Volts)
Replicate
1
.003
.188
.001
.205
-.005
.190
-.001
.185
.802
.002
.184
.784
.001
.185
-.001
.786
.000
.185
.811
.002
.189
.798
2
.003
.186
.001
.204
-.005
.188
.000
.183
.803
.002
.184
.783
.001
.186
.000
.784
-.001
.186
.812
.000
.189
.802
3
.005
.187
.000
.204
-.004
.186
.001
.185
.805
.001
.183
.785
.001
.185
.000
.785
-.001
.184
.798
.001
.191
.799
60
-------
Table 19. DATA FOR ANALYZER B
Date
5/ 3/73
NO, Level
fppm)
Zero N02
.40 N02
Zero
.40
Zero
.40
Zero
.40
Zero
.40
Zero
.40
Zero
.40
Zero
.40
Zero
.40
Analyzer Reading (Volts)
Replicate
1
.003
.650
-.015
.629
.007
.632
.009
.640
-.003
.644
-.006
.640
-.003
.658
.002
.667
.004
.678
2
-.014
.643
ft
-.011
.658
.001
.643
.003
.629
.012
.650
.004
.671
-.002
.676
.011
.632
-.010
.636
3
Tool
.642
.012
.630
.006
.637
.007 '
.640
.013
.675
.011
.639
-.002
.657
.012
.655
.008
.679
61
-------
Table 19 (CONTINUED)
Date
5/ 4/73
5/ 7/73
*5/ 8/73
5/ 9/73
5/11/73
5/14/73
NO, Level
fppm)
Zero
.10'
.40
Zero
.10
.40
Zero
.10
.40
Zero
.40
Zero
.10
.40
Zero
.40
Analyzer Reading (Volts)
Replicate
1
.005
.149
.799
-.013
.136
.744
.006
.135
.814
-.003
.642
.014
.158
.795
-.003
.797
2
-7025"
.152
.825
-.011
.132
.760
.012
.151
.811
.014
.632
.007
.141
.791
-.004
.756
3
.TJ07
.152
.816
-.022
.147
.759
.017
.157
.805
.007
.634
.016
.138
.793
.007
.753
"at 115 Volts.
fat 125 Volts.
62
-------
Table 20. ANALYSIS OF VARIANCE OF ANALYZER A DATA
Source of
Variation (S.V.J
Total (Corrected
for Mean)
Between Days
Within Days
Total (Corrected
for Mean)
Between Days
Within Days
Total (Corrected
for Mean)
Between Days
Within Days
Zero N02 Data
Sum of Degrees of
Squares (S.S.) Freedom (D.F.)
127.6 x 10~6 .26
117.6 x 10~6 8
10 x 10~6 18
0.10 N00 Data
972 x io~6 23
952.7 x 10"6 7
19.3 x 10~6 16
0.40 N00 Data
1498 x io~6 14
1359 x i(f6 4
139 x io~6 10
Mean Square (M.S.I
-
14.7 x io~6
0.6 x io"6
-
136 x io"6
1.2 x io~6
_.
339.8 x io~6
13.9 x io~6
63
-------
Table 21. ANALYSIS OF VARIANCE OF ANALYZER B DATA
V.
Source of
Variation (S.V.)
Total (Corrected
for Mean)
Between Days
Within Days
Total (Corrected
for Mean)
Between Days
Within Days
Total (Corrected
for Mean)
Between Days
Within Days
0.40
Total (Corrected
for Mean)
Between Replicates
Within a Day
Within Replicates
Zero N0n Data
Sum of Degrees of
Squares (S.S.) Freedom (D.F.) Mean Square (M.S.)
2937 x 10"6 20
1761 x 10"6 6 293.5 * 10"6
1176 x 10"6 14 84 x 10"6
0.10 NO, Data
876.7xlO~6 11
258.7xio~6 3 86.2 * 10~6
618.0xlO~6 8 77.3 x io"6
0.40 NO,, Data
9741. 7xio"6 14
7973.7xlO~6 4 1993.4 x io~6
1768 x io"6 10 176.8 x io"6
N00 Data (All data taken on one day)
6757 x io~6 26
2767 x 10~6 8 346 x lo~6
.
-6 -6
3990 x 10 ° 18 222 x 10 °
64
-------
Table 22. SUMMARY OF DATA ANALYSES - ANALYZER A
Volts
A
w
A
o
X
CV
ZERO NO,,
.00075 Volts
.0022
.0023
.0008
-
.10 NO,,
«L
.0011
.0067
.0068
.188
3.6%
.40 NO,,
«
.0037
.0104
.011
.796 '
1.4%
Table 23. SUMMARY OF DATA ANALYSES - ANALYZER B
B
0
X
CV
ZERO N0;
.0092
.0084
.012
.10 NO
.0088
.0017
.009
.145
6.2%
2-
.40 NO.
2
.0133
.0246
.028
.788
3.6%
.40 N02 (Within one day)
.015
.0064
.016
.796
2.0%
65
-------
o
-------
REFERENCE METHOD FOR THE CONTINUOUS MEASUREMENT OF CARBON MONOXIDE
IN THE ATMOSPHERE (NON-DISPERSIVE INFRARED SPECTROMETRY)
General
Data quality of the reference method for the continuous measurement of
carbon monoxide in the atmosphere as promulgated in the Federal Register
can be estimated from three special checks. These checks are:
1) measurement of control samples, 2) water vapor interference checks,
and 3) data processing checks (Ref. 20).
Arriving at Suggested Performance Standards
Suggested performance standards as given in the field document are reproduced
in Table 24. Data used in arriving at performance standards and certain
other control limits given in the field documents are discussed below.
The primary source of data on the precision and accuracy of the NDIR method
for the continuous measurement of CO concentrations is given in Reference
21. A few remarks are given herein concerning the peculiarities of the test
and their implications on the use of the data provided. Furthermore, the
use of these data in the CO field document is described.
The collaborators- were selected from experienced laboratories having a person
with previous experience with the NDIR method of measuring the concentration
of CO. Furthermore, each participant was given a preliminary test sample for
analysis as a check of his capability; all "passed the test."
Test gases were provided to the test participants; these gases were analyzed
both before and after the test. Good agreement between these analyses was
indicated for all but 3 or 4 test gases. This indicates a need to check the
test gases which are furnished by another laboratory prior to their extensive
use in system calibration. Considerable precaution was taken during the test
to obtain gases of sufficient accuracy and to safeguard these materials from
contamination or deterioration in storage or use.
2
Test concentrations of CO were selected at 8, 30, and 53 mg/m . Effect of
humidity was studied by analyzing the dry test gases after humidification.
The test yielded 810 separate determinations54 for each laboratory, 3
concentrations, 3 days, 3 determinations per day under both dry and humid
conditions (3x3x3x2=54).
Of the 16 laboratories, 15 satisfactorily completed the tests. These labor-
atories constitute a sample of experienced rather than typical field
laboratories in the onitoring network. Calibration curves were prepared
independently on each of the three days. All arithmetic errors were
corrected; few errors were noted as the method is relatively simple and not
subject to such errors.
Statistical Analysis - Replication error is defined as the variation among
successive determinations with the same operator and instrument on the same
sample within time intervals short enough to avoid change of environmental
factors and with no instrument manipulations other than zero adjustments.
67
-------
Table 24: Suggested Performance'Standards
Standards for Defining Defects
1. Measurement of Control Samples; dn. > + 2.2 ppm
J.i
2. Water Vapor Interference Check; d_ ^.1.7 ppm
Standard for Correcting Data Processing Errors
3. Data Processing Check;
Standards for Audit Rates
3i
>_ 1 ppm
4. Suggested minimum auditing rates for data error; number of
audits, n = 7; lot size, N = 100; allowable number of defects
per lot, d = 0.
5. Suggested minimum auditing rates for data processing error;
number of audits, n = 2; lot size, N = 24; allowable number
of defects (i.e.,
Standards for Operation
>_ 1 ppm) per lot, d = 0.
6. If at any time d = 1 is observed (i.e., a defect is observed)
for either d.. . or d?., increase the audit rate to n <= 20,
N = ' 100 until the cause has been determined and corrected.
7. If at any time d = 2 is observed (i.e., two defects are observed
in the same auditing period), stop collecting data until the
cause has been determined and corrected. When data collection
resumes, use an auditing level of n = 20, N = 100 until no
defects are observed in three successive audits.
8. If at any time either one of the two conditions listed below is
observed, 1) increase the audit rate to n = 20, N = 100 for the
remainder of the auditing period, 2) perform special checks to
identify the trouble area, and 3) take necessary corrective
action to reduce error levels. The two conditions are:
a) two (2) d.. . values exceeding +_ 1.4 ppm, or
four* (4) d1. values exceeding +0.7 ppm.
b) two (2) d values exceeding +_ 1.0 ppm, or
four* (4) d_. values exceeding +0.5 ppm.
.
A value of three was given in the field document; it should be changed to
four. 6g
-------
The results indicate that this error is not affected by concentration or
by humidity within the limits of the test, and thus all individual esti-
mates are pooled to yield
a(repl) = 0.17 mg/m (0.148 ppm)(286 degrees of freedom).
All further analyses are made on averages of the three determinations
made within one day on a given sample.
The use of a suitable drying agent or of refrigeration for maintaining
constant humidity was satisfactory, whereas the use of narrow band optical
filters alone was not satisfactory. In conclusion, humidity had no
measurable effect on the accuracy of the method and did not appear to
contribute significantly to the precision.
A linear model analysis was performed to estimate the several effects
which contribute to the overall variation, precision, and accuracy of the
reported results. The effects are assumed to be modeled by a linear
additive effects model.
An analysis was performed to estimate the within day precision vs. the
between- or among-day precision. These estimates are a = a (within day)
3 3
= 0.17 mg/m , and
-------
Two errors are defined and estimated in the analysis using the method given
in Ref. 22. Repeatability is defined as a quantity that is exceeded only
about 5 percent of the time by the difference of two randomly selected
test results obtained in the same laboratory. This is a different quantity
than that used in the collaborative report on suspended particulate matter
in that it is the product of 2.77 times the standard deviation. The factor
2.77 is the product of 1.96 (5 percent value of the standard normal deviate)
and v^. (The difference of two observations has a variance of twice
that of a single observation or a standard deviation /2~ times that of a
single value.) The reproducibility is a quantity exceeded about 5
percent of the time by the difference, larger less the smaller value, of
two single results made on the same material (concentration) in two dif-
ferent randomly selected laboratories. These measures of repeatability
and reproducibility are in some cases dependent on the concentration. In
this case
A *j
a (repeatability) = 0.55 mg/m
( 2
a (reproducibility) = <0.001 x - 0.039 x + 1.1
where x is the concentration. For example, for x = 8 mg/m , a (reproduci-
3 3
bility) =0.9 mg/m , or reproducibility = 2.77(0.9) = 2.5 mg/m . The
results must be interpreted carefully because of the controlled conditions
under which they were obtained and also because several outlying observa-
tions were eliminated in the analysis.
Interpretation of the Parameters - Interpretation of the results are:
(1). It is indicated that replication, taking more observations on the
same day on the same sample, will in general be a waste of time. This is
very often the case in practice; that is, the day-to-day variation often
exceeds by a great deal the within-day sample variation, and hence taking
additional samples on a day will in general not be cost beneficial. There
are procedures for relating cost and variance; e.g., see Reference 23, which
gives a procedure for estimating how many replicates, repeats (day-to-day),
etc., should be taken to obtain an estimate of desired precision with minimum
cost of measurements. In an auditing program in which a deviation is
obtained between two measurements taken under conditions of replicates, the
absolute deviation between two such measurements should be less than
/2 (1.96) 0.17 = 0.47 mg/m3, 95% of the time, or
O
/2 (2.58) 0.17 = 0.62 mg/m , 99% of the time.
The values 1.96 and 2.58 assume, that the estimate of the standard deviation
0.17 is known or based on an infinite number of degrees of freedom (D.F).
If the D.F. is relatively small, say less than 25 or 30, then one should
use the studentized range test, Reference 24.
(2). The day-to-day variation, involving changing environmental effects, as
well as the short-time effects of instruments, as measured by the standard
deviation is 0.57 mg/m^. This measure of repeatability (on similar samples
analyzed on different days) is useful in performing any audit check involving
measurements made on different days but within the same laboratory. Thus
if two samples are measured under these conditions, the measured values
should differ by less than
70
-------
72 (1.96) 0.57 = 1.58 mg/m3, 95% of the time, or
/2 (2.58) 0.57 = 2.08 mg/m3, 99% of the time.
(3). The precision between laboratories is complicated by its dependence
on the concentration level. This dependence was not evident in the
previous measures. The comparison of measurements across laboratories
such as that of a collaborative test used by an agency to check several
labs involves not only the replication effect, and day-to-day effects,
but also the variation between or among laboratories. The standard
deviation is dependent on the concentration level ranging from 0.83 at 20
3 3 33
mg/m to 1.53 at 60 mg/m (and to 1.04 mg/m at 0 mg/m ). These values
must be multiplied by the appropriate factors as given above for 95% and
99% limits of variation.
The suggested standard of 2.2 ppm for measuring control samples as given
in Table 24 was arrived at as follows. The actual CO concentration of the
sample is determined by the supplier and the measured value by the laboratory.
Therefore, the standard deviation of the difference of two obs rvations is
applicable (see Appendix 4, page 144). The expected variability then is
assumed to be greater than the repeatability of a single laboratory and less
than or equal to the reproducibility value given above for among-laboratories
since two independent measurements are involved. As a standard to be used
when a quality assurance program is first started, the standard deviation
for reproducibility at the point of highest precision, 20 mg/m3, was suggested.
The standard deviation of reproducibility at this CO concentration is 0.72
ppm (0.83 mg/m3). Assuming that the variability in the known value of the
control samples is much less than the variability of the measured value, the
standard deviation of the difference was taken as the standard deviation of
the measured value. The suggested performance standard for defining a defect
when measuring control samples was 2.2 ppm (at the 3a level). (All standards
were based on a 0 to 50 ppm analyzer scale.)
The suggested standard for water vapor interference check was an estimate
and not based on real data. There were two reasons for suggesting this
value. They are:
(1) If a value much smaller than the normal measurement variability
is used, there is an increased probability of declaring a defect due
to water vapor interference when actually it is normal measurement
error.
(2) If a larger value is allowed, it approaches the value used for
measuring control samples and begins to significantly influence
overall data variability.
The standard for correcting data processing errors was suggested as 1 ppm for
a 50 ppm full scale range. This standard was suggested with dynamic strip
chart traces in mind. If an intergrating flask is used to smooth out the
CO spikes, a much lower standard should be used.
71
-------
Accuracy - According to the results of Reference 21, a definite bias
exists in the measurements. It also seems to depend on the concentra-
tion. On the average the measured results were 2.5% high. It is
expected that this bias results from the use of calibration gases which
exhibit significant variation with respect to their specified concentra-
tion. Thus it is concluded in Reference 21 that care must be taken in
obtaining high quality calibration gases and protecting ther t^om
deterioration. On this basis it was recommended in the field document
for CO that a check be made initially of a new calibration gas, and if
the check indicates a possible discrepancy, further action should be
taken to verify the stated concentration, e.g., a retest or a new
calibration.
Recommendations
Water vapor control units which operate at a specific dew point should
not be used in areas where the ambient dew point is frequently lower
than that of the control unit.
The water vapor interference check as part of the auditing process is a
must to ensure that the control units are properly used and maintained.
An integrating flask should be made a standard item of equipment on CO
monitors. It is important in reducing data reduction errors when reading
strip charts and is perhaps even more critical when an automatic digital
data acquisition system is used.
72
-------
SECTION III BACKGRUND INFORMATION APPLICABLE TO
THE WNAGEMENT MANUALS OF FIELD DOCUMENTS
GENERAL
This section contains a series of brief descriptions of the procedures
given in the management manual of the individual field documents. The
organization of this section follows that of the last three field docu-
ments, namely, 0,, N0_, and SCL. In addition the figures in these
manuals have been included in this final report for the convenience of
the reader. An attempt is made here to give the necessary background
information or reference to same for development of the tables and
figures of the field documents. As appropriate, the assumptions and
limitations concerning the statistical techniques are provided. In the
case of the average cost computation, the sensitivity was determined of
the results to the assumed costs and the percent of good quality data.
Section III is subdivided into three major subsections entitled SELECTING
AUDITING SCHEMES, COST IMPLICATIONS, and DATA QUALITY ASSESSMENT. These
are briefly described in the following paragraphs.
Two approaches to assessing or evaluating data quality were described in
the field documents. One approach was to define a defective measurement
in terms of some standard or permitted deviation from audited value.
Once such a definition has been made, the problem becomes one of sampling,
a selected number of measurements from a defined "lot" of measurements,
e.g., 100 days, and making a decision based on the number of defective
measurements in the sample. This approach was suggested because it would
be simple to implement and would require very few computations and mini-
mize the reporting of data. There is considerable flexibility in this
audit approach. The curves and tables provided^herein, and in the field
documents, allow the manager to select the audit level and the number of
permissible defects (0 or 1 is the extent of the tables, and is considered
adequate for practical sampling levels). The audit level determines the
confidence level concerning the percent of good measurements in the lot.
The definition of a defective measurement should be consistent with the
required data quality and the expected variation in the measurements,
subject to the conditions given in the reference method. Background infor-
mation pertaining to the selection of auditing schemes is given in the next
section.
The cost data will be most difficult to estimate. However, the basic
sampling approach is not very sensitive to the specific cost data as
'shown in the section entitled COST IMPLICATIONS. Thus it is not critical
to estimate the costs ^n order to implement an auditing program. This cost
analysis is given to provide a methodology for considering the overall or
average cost of auditing the measurement process and then making a decision
concerning the lot based on the number of defective measurements contained
in the sample drawn from the lot.
73
-------
An important approach for improving the data quality is to consider
alternative strategies for improving the precision/accuracy of the
measured concentrations. The alternative strategies may include changes
in operating procedures, training programs; equipment changes or modi-
fications; or improved environmental control. Each of the strategies
can be evaluated as to the expected improvement in data quality and the
expected additional cost. An example of this approach is provided in
each field document. That is, four or five alternative strategies for
improving the quality of the data are proposed, the expected improvement
in data quality is hypothesized and the cost per 100 samples is estimated.
The expected improvement in data quality as given in the field documents
is typically based on judgment, since there was no opportunity to run
experiments to estimate the improvements in the data quality. However,
this opportunity exists for the monitoring agencies and it should be
encouraged. Suggested improvements in procedures should be discussed and
encouraged among the laboratory analysts, site operators, and supervisory
personnel; and when judged to be a potential improvement, tested experi-
mentally. The fact that an improved procedure costs dollars but results
in improved quality data may ultimately reduce the average cost per item
of valid data and save a great deal in costs relative to air quality
control decisions to be based on these data.
The second approach to the assessment of data quality is one which estimates
the variation in the results due to various aspects of the measurement
process and then combines these results into one value or overall assessment.
This approach is described in the section entitled DATA QUALITY ASSESSMENT
and it is suggested in addition to the previous one, after experience has
been gained in reporting the data on number of defectives and further
knowledge has been gained concerning the more critical variables in the
process. In this approach an attempt is made to assess the process by
use of control samples and checking of data processing errors (reading
charts, transcribing and computational errors), and making appropriate
repeat determinations. By use of these data, the standard deviation of the
measured results can be estimated by methods given in Appendix 4 entitled
ESTIMATION OF THE MEAN AND VARIANCE OF VARIOUS COMPARISONS OF INTEREST.
In the case of two of the pollutants, suspended particulate matter and SO,
a model was formulated for relating the measured concentration to the
several variables or factors of the measurement process. The models were
developed starting with the deterministic equations for the concentrations
as given in the reference methods. This modeling approach was taken
because there was no simple, reliable, and cost efficient audit method to
make an overall assessment. An example computation is given on pages 154
through 157 under the heading, "Means and Variance of Nonlinear Functions."
Also see Appendices 1 and 2 for further details on these models and the
subsequent analyses.
The use of control charts in improving the data quality is briefly
described in Appendix 5. Appendix 5 does not attempt to give a complete
discussion of the subject of quality control, but it does give appropriate
references to some standard texts and indicates potential uses of control
74
-------
charts. The discussion also considers the consequences of "over
correcting" the data. This occurs, for example when a check or calibration
point is run daily and data are corrected daily in accordance with the
measured value of the check. This results in a greater variation in
reported results than if a more standard reference method were used in
making data adjustments (Ref. 25).
SELECTING AUDITING SCHEMES
Three concepts pertinent to the first approach in the field documents are
described in this section. The first is the computation of the probability
of accepting a collection of measurements, referred to as a>lot of N items,
as valid given that there are D defective measurements in the lot, n items
are sampled, and the number of defects permitted in the sample is c or
less. The second concept is that of the level of confidence concerning the
percentage of good measurements in the lot of N given that either 0 or 1
defect has been obtained in the sample of n measurements. Thirdly, the
computation of the percent of defects in data reported as valid is computed
as a function of the incoming data quality (prior to auditing) and the
parameters of the auditing scheme.
Computation of the Probability of Accepting a Lot as a Function of the
Sample Size and the Acceptable Number of Defectives
Auditing procedures, as recommended in the field documents, where certain
check measurements are made and where the results of each check identified
the air quality data as good or defective a"re analogous to what is commonly
referred to as "sampling without replacement from a population in which each
element of the population belongs to one of two classes". The hypergeometric
probability distribution provides the probabilities of an item belonging to
either class (in this case the two classes are good and defective) when drawn
from a lot (population) of known size containing a given ratio of good to
defective items. The parameters of a hypergeometric distribution are:
(1) the sample size n,
(2) the lot size (or population size) N, and
(3) the number of defective items in the lot D.
To illustrate how the hypergeometric distribution is used, suppose that a
lot of N items contains D defective items (or measurements) and that a
sample of n items is selected at random from the lot without replacing an
item prior to drawing the next item. Assume that if c or less defectives
are drawn, the lot is considered acceptable in quality, otherwise not
acceptable. The probability of drawing a sample of size n with d defectives
is given by
/d defectives in
a sample of n
75
-------
the hypergeometric distribution. The notation
d d! (D- d)!
For example, if n = 7, N = 100, D = 5, d = 0, we have
0.6903.
(ioo)
If N is large with respect to n, say N/n > 10, then the binomial approxi-
mation to the above computation is adequate for most problems; i.e., if
we let
we have
D ,
- = p, and
N-D
-5- = q say>
. o defectives^ ^ ln\ _o _n '" ~vn
V in n
In the above example, this approximation gives
C
p ( o defectivesj ,, (0.95)7 = 0.70
which is very nearly equal to the above exact value. If the items are
replaced in the lot before the next item is drawn (sampling with replace-
ment) , then the above formulation is exact; i.e., the binomial distribution
applies.
If one permits more than 0 defectives, say c or less, for the lot to be
acceptable, the probability of accepting the lot is the sum of the
probabilities of c or less defectives i.e., if d is the observed number
of defectives, we compute
P(d <_ c) = P(d = 0) + P(d = 1) + ... + P(d = c).
76
-------
If c = 0, the computation in the example above for n = 7 becomes the
probability of accepting the lot of items.
Table 25 below gives the results of the probability of d defectives for
d=0, 1, 2, ...,6, n=7, N=100, D=5 and 15%. Figures 6 and 7 give similar
results for varying sample size n(l to 25), for N=50 and 100, respectively.
Table 25. P(d defectives)
Data Quality
D=5% Defectives D=15% Defectives
0
1
2
3
4
5
6
0.6903
0.2715
0.0362
0.0020
0.00004
: 0
= 0
0.3083
0.4098
0.2152
0.0576
0.0084
0.0007
=0
Confidence Interval Estimate for the Percentage.of Good Measurements
in a Lot
For this discussion, it is assumed that a sample of n measurements
is selected at random from a lot of N measurements, and that the definition
of a defective measurement has been made explicit, e.g., within + 2 units
of a prescribed value. It is desired to determine a confidence interval
estimate of the percentage of good measurements in the lot (or vice versa,
the percentage of defects in the lot). The standard statistical procedure
employed for obtaining this confidence interval is similar to that for
the binomial variable. (For example, see A. Hald, Ref. 26, pages 697-
699). The procedure is to determine the percentage of defects D in the
lot such that the probability of observing d or less defects in the sample
is equal to the quantity, one minus the level of confidence. For example,
suppose that 0 defects are observed in a sample of n=10 from a lot of
N=10 measurements, and that an 80% confidence interval for the percentage
of good (or defective) measurements is to be determined. If the percentage
of defects in the lot is D, then the probability of 0 defects in the
sample is
D\/N-D
77
-------
1.0
I
en
01
3
00
o
V
01
T3
O
s
CM
0.8
0.6
0.4
0.2
d s 1, D
d - 0,, D «-
d * 1. D - 15Z
d - 0, D - 15Z
10 15
Sample Size (n)
2C
25
Figure 6: Probability of d Defectives in the Sample If
the Lot (N « 100) Contains DZ Defectives
78
-------
0)
JS
4J
8-
o 0.4
14-t
o
H
s
g
0.2
10 15
Sample Size (n)
Figure 7: Probability of d Defectives in the Sample If the
Lot (N = 50) Contains D% Defectives.
This graph is for a lot size of N = 50. Only whole numbers of defectives
are physically possible; therefore, even values of D (i.e., 6, 10, and
20 percent) are given rather than the odd values of 5 and 15 percent as
given in Figure 6.
79
-------
The value of D is then determined for which P(d=0) is equal to or smaller
than 1 - 0.80 = 0.20. Because D is a discrete value, 0, 1, 2, ..., one
increases D until the P(d=0) becomes less than 0.20. (In some cases, the
value of D was chosen corresponding to P(d=0) being close to 0.20.) For
example,
/LA /87)
If D = 13, P(d=0) = ^ "ty = 0.231 .
/100\
Vw
/14\ /86\
If D = 14, P(d=0) = V0/ Xl0' = 0.204 .
OS)
f!5\ /85\
If D-15, P(d=0) = V°Al°y =0.181 .
/100\
In this case D=14 or 14% defective measurements was taken as an upper
confidence limit on the percent of defects in the lot of N=100 measurements.
If one defective is observed; a similar approach is employed to determine
the value of D using the sum of the probabilities of 0 and 1 defective, i.e
P(d=0) + P(d=l) = P(d < 1), or
<1 flffl, fflCS)
" (n)
D is increased until the sum reaches the desired value, one minus the level of
confidence. For example, if D=27, n=10, N=100
P(d=0) + P(d=l) = - + X XnV = -0359 + 0.1515 = 0.184.
80
-------
Thus, D=27 is taken as the upper 80% confidence limit for the percent
defectives in a lot of N=100. Several computations of this type were
made in order to generate the corresponding curves of percent of good
measurements vs. n(sample size) and the specified confidence levels.
Figure 8 and 9 give these results for 0 and 1 observed defective
measurements, respectively.
Computation of the Percent Defective in the Reported Data
In order to estimate the percent of defects in reported data on a particular
measurement, an assumption must be made about the quality of the data prior
to the sampling and corrective actions taken following the sampling. Thus,
if the distribution of percentage defects in the incoming lots of size N
being sampled, the actions taken if either (1) c or less defects are
found in the sample of n measurements, or (2) more than c defects are
detected, are known, it is a straightforward but tedious calculation to
determine the quality of the reported data.
Consider .first a simple example of the incoming data quality being
subdivided into two categories.
50 percent of lots of
N = 100 measurements
are 5% defective
50 percent of lots of
N = 100 measurements
are 15%' defective
n =
Sample
7 measurements
If d = 0 defects are
observed in the
sample, accept data
as valid
If d = 1 or more defects
are found, consider
data as invalid (do
not report any data)
The probability of accepting (rejecting) the lot of data (100 measurements)
is given by the probability of 0 (1 or more) defectives; hence
P(0 defects|5% defects)
= 0.690
P(0 defects 115% defects)
- 0.308.
81
-------
100
10 15
Sample Size (n)
Figure 8:. Percentage of Good Measurements Vs. Sample Size
for No Defectives and Indicated Confidence Level
82
-------
100
10 15
Sample Size (n)
25
Figure 9: Percentage of Good Measurements Vs. Sample Sice
for 1 Defective Observed and Indicated Confidence Level
Lot Size - 100
83
-------
Thus the average quality level, p, of the reported data under the assumption
that all lots declared invalid are eliminated is given by
- _ [0.5 (0.05 x p(Q def15% def) + 0.5 (0.15 x p(Q def115% def)]
P ~ [0.5 P(0 def|5% def) + 0.5 P(0 def|l5% def)]
[0.5 (0.05 x .69) + 0.5 (0.15 * 0.308)3
0.5(.69 + 0.308)
0.081 or ..«.
In general if there are g% of lots of 100 which are good quality,
q = (100 - g)% of lots which are poor quality, the above computation would
be as follows.
LlOO V"'"" " xw def 15% def)) + (^(o.lS x P(0 def 115% def))]
def 15% def)) + (-^\ P(0 def 115% def)
Several values of g are used below and the corresponding p determined for
n = 7 and 17(c = 0 is assumed for both cases).
n = 7 _jg_ p(%)
90 5.5
80 6.0
70 6.6
60 7.3
50 8.1
n = 17 g p(%)
90 5.1
80 5.3
70 5.5
60 5.8
50 6.1
84
-------
In assuming that all good lots contain 5% defects and all bad lots contain
15% defects, a rather restrictive assumption has been made. Now consider
how the p is altered for a case in which the lot quality varies in
accordance with the following distribution of percent defects.
% defective
Percentage of Lots
Good Lots(<10% defects)
13579
10 10 10 10 10
Bad Lots (>10% defects)
11 15 19
15 20 15
With the assumed distribution, the average percent defects in the "good
lots" is 5%, and for the "bad lots" is 15%.
Good Lots (<10% defects)
Bad Lots (>1Q% defects)
% defective
Prob(0.def in n=7)
13579
0.92 0.81 0.69 0.60 0.52
11 15 19
0.45 0.308 0.24
Thus p = 9.1% vs. 8.1% obtained when 50% of the lots were 5% defective
and 50%, 15% defective.
COST IMPLICATIONS
Pertinent cost considerations are described in this section for the purpose
of illustrating methodologies. The conditional cost of accepting or reject-
ing a lot of data given a specified number of defects in the sample is
considered in the following section. The concept of average cost is defined
and the method given for computing this cost as a function of the quality
of the data, costs of accepting poor quality data, rejecting good quality
data, and the auditing cost. It is recognized that the assumed costs may
not apply to a particular agency and that changes in the costs may affect
the criteria used in the auditing scheme for accepting a lot of data as
valid. Thus, a cost analysis is performed to determine the effect of
changes in the costs as well as the assumed lot quality on the criterion
for acceptance. No significant effect on the criterion was determined for
the parameter changes employed in the analysis.
In the last subsection a methodology is given for performing cost tradeoffs,
that is, a procedure for comparing alternative strategies for improving
data quality (precision/accuracy) Subject to estimated cost of implementing
these strategies. When the costs and potential improvements in data quality
are determined by a specific agency, the best strategy for improving the
data quality may be selected using the methodology as suggested.
85
-------
Bayeslan Scheme for Computation of Costs
The auditing scheme can be translated into costs using the costs of
auditing, rejecting good data, and accepting poor quality data. These
costs may be very different in different geographic locations. There-
fore, purely for purposes of illustrating a method, the cost of auditing
is assumed to be directly proportional to the auditing level. For n = 7
it is assumed to be $155 per lot of 100. The cost of rejecting good
quality data is assumed to be $600 for a lot of N = 100. The cost of
reporting poor quality data is taken to be $800. To repeat, these costs
given in Table 26 are assumed for the purpose of illustrating a methodology
of relating auditing costs to data quality. Meaningful results can only
be obtained by using correct local information.
JT.able 26. COSTS VS. DATA QUALITY
Data Quality
Reject Lot of
Data
"Good"
"Bad"
D <_ 10%
Incorrect Decision
Lose cost of performing
audit plus cost of reject-
ing good quality data.
(-$600 - $155)
D > 10%
Correct Decision
Lose cost of performing
audit, save cost of not
permitting poor quality
data to be reported.
($400 - $155)
Accept Lot of
Data
Correct Decision
Lose cost of performing
audit.
(-$155)
Incorrect Decision
Lose cost of performing
audit plus cost of de-
claring poor quality
data valid.
(-$800 - $155)
Suppose that 50 percent of the lots have more than 10 percent defectives
and 50 percent have less than 10 percent defectives. For simplicity of
calculation, it is further assumed that the good lots have exactly
5 percent defectives and the poor quality lots have 15 percent defectives.
86
-------
Suppose that n = 7 measurements out of a lot of N = 100 have been audited
and none found to be defective. Furthermore, consider the two possible
decisions of rejecting the lot and accepting the lot and the relative
costs of each. These results are given in Tables 27 and 28.
Table 27. COSTS IF 0 DEFECTIVES ARE OBSERVED AND THE LOT IS REJECTED
Reject Lot
D = 5%
D = 15%
Correct
Decision
P2 = 0.31
C2 = 400 - 155
Incorrect
Decision
P1 = 0.69
C^ = -600 - 155
Net Value ($)
P1C1 = -$521
P2C2 = $76
Cost =
Table 28. COSTS IF 0 DEFECTIVES ARE.OBSERVED AND THE LOT IS ACCEPTED
Accept Lot
D = 5%
D = 15%
Correct
Decision
Pj^ = 0.69
C3 = -155
Incorrect
Decision
P2 = 0.31
C4 = -800 - 155
Net Value ($)
PlC3 = -$107
P2C4 = -$296
Cost =
= -$403
The value P-,(po) in the above table is the probability that the lot is
5% (15%) defective given that 0 defectives have been observed. For
example ,
87
-------
Pi = /
/"probability that the lot is 5% defective\
i, and 0 defectives are observed /
/lot is 15% defective and\
\ 0 defectives observed /
/lot is 5% defective and\
p\ 0 defectives observed /
0.5(0.69)
0.5(0.69) + 0.5(0.31)
= 0.69
/probability that the lot is 15% defective)
\ and 0 defectives are observed /_
/lot is 5% defective and\
p\ 0 defectives observed /
/lot is 15% defective and\
p\ 0 defectives observed /
0.5(0.31)
0.5(0.31) + 0.5(0.69)
0.31
It was assumed that the probability that the lot is 5% defective is 0.5.
The probability of observing zero defectives, given the lot quality is
5% or 15%, can be read from the graph of Figure 6. .
A similar table can be constructed for 1, 2, ..., defectives and the net
costs determined. The net costs are tabulated in Table 29 fo.r 1, 2, 3, and
4 defectives. The resulting costs indicate that the decision preferred
from a purely monetary viewpoint is to accept the lot if 0 defectives are
observed and to reject it otherwise. The decision cannot be made on this;
basis alone. The details of the audit scheme also affect the confidence
which can be placed in the data qualification; consideration must be
given to that aspect as well as to cost.
Table 29. COSTS IN DOLLARS
Decision
Reject Lot
Accept Lot
d =
0 1
-445 -155
-403 -635
number of
2
+101
,^839
defectives
3
+207
-928
4
+244
-952
88
-------
Computation of Average Cost
The average cost associated with a given audit procedure, associated
decision criterion, and assumed costs for each decision is obtained
by multiplying the probability of each possible decision by the
associated cost and summing over all the possible cases. The general
computation procedure is given by the formula.
Average Cost = P (lot is good quality and is accepted) Cost (Audit)
+ P (lot is good quality and is rejected) Cost (rejecting good data
and cost of audit)
+ P (lot is poor quality and is accepted) Cost (accepting poor quality
data and cost of audit)
+ P (lot is poor quality and is rejected) Cost (Savings of rejecting
poor quality data
less cost of audit)
Each of the probabilities in the above is computed by a formula similar
to the following.
P (lot is good quality and is accepted) = P (lot is good quality) x
P (lot is accepted|lot is good quality)
The notation P(A|fl) is read "probability that A occurs given that B has
occurred; e.g., in the last expression above, the probability that the
lot is accepted given that it is of good quality, say 5% defective
measurements in the lot. Each probability is the product of the proba-
bility' that the lot quality is as specified by the conditional probabi-
lity that the lot is accepted (or rejected) given the specified lot
quality. The four probabilities must add up to one (1) since all the
possible situations are enumerated.
Table 30 below contains the detailed computations for an example in
which the following assumptions are made.
Data Quality
Poor Quality Data: D = 15% defects.
Good Quality Data: D = 5% defects.
50% of the Lots are Good and 50% Poor Quality Data.
Decision Criterion
Sample 7 measurements from a lot of N = 100 measurements.
Accept lot if d = 0 defectives are observed.
Associated Costs
The costs are given in Table 26, page 86.
89
-------
Table 30: Overall Average Costs for One
Acceptance-Rejection Scheme
Decision
Reject any lot of data
if 1 or more defects
are found.
Accept any lot of data
if 0 defects are found.
Good Lots
D = 5%
q;L=0.5(0.31)=0.155
Cj_ = -$755
q =0.5 (0.69) =0.345
C3 = -$155
Bad Lots
D = 15%
4^=0. 5 (0.69) =0.345
C2 = $245
q4=0.5(0.31)=0.155
C4 = -$955
Average
Costs
q1C1+q2C2
= -$32
q3C3+q4C4
= -$202
Average Cost = -$234
The cost -$234 is the (weighted) average cost of the four possible situations.
The weights are the corresponding probabilities of each occurrence.
The average cost -$234 can also be obtained from the results of Table 29,
page 88, multiplying the cost associated with each decision for the
observed number of defectives d by the probability that d occurs and then
summing the products over all the possible cases. This computation is
given below in Table 31.
Table 31: Overall Average Costs
No. of
Defectives
d = 0
1
2
3
4
Decision
Rule
Accept
Reject
Reject
Reject
Reject
Costs ($) from
Table 3
- 403
- 155
101
207
244
Totals
Prob(d)
0.50
0.34
0.1255
0.030
0.0042
j 0.9997
Cost x Prob(d)
-$201.5
- 52.7
12.6
6.2
1.0
-$234.4
Using the method of Table 30, the average cost was computed for sample sizes
n = 0 through n = 20. These results and other relevant information is
summarized in Figure 10.
90
-------
$400
CO
o
o
So $300
cd
1-1
i-H
«J
$200
Cost if
d = 0
I
Probability
if d - 0
I
I
5 10 15
Audit Level (Sample Size)
1.0
.9
.8
.7
.6
.5
.4
.3
.2
.1
20
0)
N
O
0
c
H
Vj
01
(0
rt
/J
o
p
Oi
Figure io: Average Cost Vs. Audit Level
(Lot Size N = 100)
91
-------
Cost Analysis
In this subsection the effect of varying certain assumptions (parameters)
used to obtain the average cost are considered. Recall that from the
previous subsection the average cost is given as a function of (1) the
percent defects in the good and poor quality lots (5 and 15% were assumed),
(2) the percent of lots which are of good quality (50% was assumed),
(3) the individual costs (audit, cost, cost of rejecting good data, cost
of accepting poor quality data, etc.)> and (4) the variables associated
with the sampling (lot size, sample size, and number of permissible
defects in the sample). To vary all of the parameters simultaneously
would require extensive computation. Hence one parameter at a time
will be varied just to indicate the sensitivity of the results to each
assumption.
Case 1; Vary the Percentage of Lots of Good Quality
Consider first the percentage of the lots which are of good quality. Let
this percentage vary from the 50% assumed in the analysis in the field
documents to 90%, using one intermediate value, 70%. All other parameters
will be held constant. The effect of this change on the average cost is
given in the following table.
Table 32: Quality of Incoming Data and
Associated Average Cost
Percentage of Lots
Which are of Good
Quality
50
70
90
Average
Cost
-$234
-$277
-$320
The above costs assume that the acceptance rule is to sample n = 7 measure-
ments from N = 100 and to accept all data in the lot if no defects are
observed; otherwise, reject the lot as unsatisfactory data quality.
Now examine the costs associated with the Bayesian scheme for acceptance,
conditional on the number of defects, following the procedure described
in the previous subsection starting on page 86. In the notation of the
text, two probabilities are computed, p (and p?), for the conditional
probabilities that the sample is from a good lot (poor quality lot) given
that d defects have been observed in the sample. Assuming 70% of the lots
are good, then
Pl =
0.7(0.69)
0.7(0.69) + 0.3(0.31)
= 0.84
92
-------
0.3(0.31)
2 ~ 0.3(0.31) + 0.7(0.69)
If 0 defects are observed, and if the decision is to reject the lot, then
the conditional cost is
Cost
= 0.84(-755) + 0.16(245) = -$595.
If the decision is to accept the lot, the cost is
Cost =
+
0.84(-155) + 0.16(-955) = -$283.
On the basis of these costs, one would prefer to accept the lot if zero
defects are observed (the same as in the field document) .
If 1 defect is observed, the probabilities p. and p« become 0.61 and 0.39,
respectively, and the corresponding costs associated with rejection and
acceptance become -$365 and -$467, respectively, indicating that it is
cost effective to reject the lot. Continuing the analysis in this manner
for d = 2,3 and 4 defects we summarize the results in the following table
for d = 0, 1, 2, 3, and 4.
Table 33: Summary Information
d = number of defects observed in sample
n = 7 from a lot of N = 100
01 23
Reject the Lot -$595 -$365 -$35 +$170
Accept the Lot -$283 -$467 -$731 -$895
P1 0.84 0.61 0.28 0.075
p 0.16 0.39 0.72 0.925
P (d defects) 0.576 0.312 0.090 0.019
Cost* -$283 -$365 -$35 +$170
Cost x P(d defects)
4
«+$245
w-$955
»0
wl
Total
0.003 1.000
+$245
-$276
Cost is the value corresponding to the preferred decision rule, i.e.,
accept for d=0, reject otherwise.
93
-------
The above table gives the costs conditional on rejecting/accepting the
lot, the values of p., and P2, the probability that d defects will be
observed, and the computation of the average cost.
In summary the change from 50% good quality lots to 70% good quality lots
has not altered the general conclusions; only the relative costs have
changed. Similar conclusions hold for 90% good quality lots. It should
be noted that the average cost is increasing.
Case 2: Vary the Cost of Accepting_P£or_ Quality Data
The cost of accepting poor quality data was taken to be -$800 in the field
documents. Let us vary this cost to take values of -$400 and -$600. The
cost of rejecting the lot given d defects will not change from that given
in the field document. However, this cost is repeated here with that for
the three costs of accepting the lot.
Table 34: Effect of varying the Cost of Accepting Poor Quality Data
d = number of defects in sample
Decision
Reject the Lot
Accept -the lot (C,
Accept the lot (C,
Accept the lot (C,
= -$800)
= -$600)
= -$400)
d=0
-$445
-403
-341
-279
d=l
-155
-635
-515
-395
_
"+101"
-839
-669
-497
In summary the effect of varying the cost of accepting poor quality data
from -$400 to -$800 has not altered the preferred decision rule, that is,
to accept the lot if 0 defects in the sample and reject otherwise.
Case 3: Vary the Cost of Rejecting Good Data
In the field document the cost of rejecting good quality lots was taken to
be -$600, in this study we vary the cost from -$400 to -$800, similar to the
manner above. In this case the costs of accepting the lot for
various observed number of defects is not changed from that obtained in
the field documents. The audit cost is -$155 and is added to the cost of
rejecting good quality lots to obtain C , the cost used in Table 35.
94
-------
Table 35: Effect of Varying the Cost of Rejecting Good Quality Data
Decision
Reject the lot
Reject the lot
Reject the lot
Accept the Lot
d = number c
Rule
(Ci = -$555)
(C-L = -$755)
((^ = -$955)
>f defects in sample
d=0
-$307
-$445
-$583
-$403
d=l
-$75
-$155
-$235
-$635
Case 4: Vary the Percent Defects in the Poor Quality Lots
It was assumed in the field documents that all poor quality lots contained
15 percent defects and the good quality lots contained 5 percent defects.
Now let the percent defects in the poor quality lots take values of 20
and 30 percent, respectively. The results of these computations are
summarized in the following table.
Table 36: Effect of Varying the Percent Defects in Poor Quality Lots
Good Quality Lot
P(d|D=5)
Poor Quality Lot
P(d|D=15)
P(d|D=20)
P(d|D=30)
pl
P2
Cost(reject lot)
Cost(accept lot)
d
0
0.69
0.31
0.20
0.075
0.775
0.225
-$530
*
[-$335 J
= number of defects in sample
1234
0.27
0.410
0.375
0.246
0.418
0.582
[-$175]
-$619
0.036
0.215
0.285
0.330
0.113
0.887
{+$132J
-$865
0.002 ^0
0.058 0.008
0.113 0.025
0.233 0.094
0.017
0.983
-
~ ~
5
^0
fsQ
«0
.021
-
-
-
~
Preferred decision rule implied by lesser costs denoted by brackets.
95
-------
In summary the decision rule was altered in only one instance; that is,
when the cost of rejecting good quality data was lowered to -$400, the
cost of rejecting the lot for d = 0 was less than that of accepting the
lot. Hence the preference to reject the lot even if d = 0. However,
this would be an unacceptable decision rule for there would be no need
to take any data. In practice this particular cost, i.e., for rejecting
all.data, should be large particularly because there is no evidence of
poor quality data, i.e., d = 0 defects were observed.
Case 5: Vary the Audit Level or Sample Size
The variation of cost with sample size while holding fixed the following
parameters: N, percent of good quality lots, and costs of sampling,
rejecting good quality data, accepting bad quality data, etc., is given in
Figure 10, page 91.
96
-------
Cost Trade-9ff Procedures
This section illustrates a methodology for comparing alternative strategies
for improving the precision/accuracy of data. In practice it is necessary
to experiment with the alternative strategies in order to obtain specific
estimates of how much improvement, if any, can be expected and to relate
the precision to the cost.
In order to illustrate the methodology, the strategies proposed for the
measurement of the concentration of SO. are repeated here for convenience.
The assumed values of the standard deviations and biases for each strategy
and audit are not based on actual data, except where such data are given
for the reference method. Four alternative strategies (Al, A2, A3 and A4)
and various combinations of these actions are considered. The added cost
(above that of the reference method AO) for implementing each strategy is
estimated on the basis of each set of 100 24-hour samples. The assumed data
are given in Table 37 and the plot of added cost versus the estimated mean
square error (expressed in percent) is given in Figure 11. Procedures for
combining the biases and standard deviations to obtain an overall assessment
are given in the next section.
Suppose that it is desired to make a statement that the true SO- concentra-
tion is within 12% of the measured concentration (for simplicity of ,
discussion all calculations are made at a true concentration of 380 yg/m )
with approximately 95 percent confidence. Minimal cost control equipment
and checking procedures are to be employed to attain this desired precision.
Examining the graph in Figure 11 of cost versus precision, one observes
that A2 is the least costly strategy that meets the required goal of
2MSE <_ 12 or MSB _<_ 6 percent. The mean square error (MSB = /a* + TZ is
used in this analysis as a means of combining the bias (T) and standard
deviation (a) to obtain a single measure of the overall dispersion of the
data. The assumed values of the MSB's of the measured concentrations of S0_
for the alternative courses of action are given in Table 37. The costs for
the various alternatives are given as the ordinate axis in Figure 11.
Suppose that it is desired that MSB be less than 4% and that the cost of
reporting poor quality data increases rapidly for MSB greater than 4%, then
strategy A6 = (A2 + A4) appears best because it meets the goal of MSB being
less than 4% its costs of implementation is $400/100 samples. However,
strategy A5 costs $215 to implement and results in a cost of about $40 for
reporting poor quality data; an overall cost of $255 compared to $400 for
A6. Based on the assumed values A5 would be best. This example demon-
strates the need for the manager to obtain estimates of the improvements
in data quality which can be attained through various actions. This
assumption is illustrated by the cost curve given, by the solid line in
Figure 11. For any alternative strategy, the cost of reporting poor
quality data is given by the ordinate of this curve (the solid curve)
corresponding to the strategy.
97
-------
I/
Table 37. ASSUMED STANDARD DEVIATIONS FOR ALTERNATIVE STRATEGIES-
1. Flow rate d
°1
2. Control d
sample
°2
3. Data d
processing
a3
CTT (%)
Negative bias = -(%)
MSE(%)-/
Added cost ($) per
100 samples
AO
-4
°1
0
°2
0
V
5.0
4
6.5
0
Al
-4
0.6a1
0
02
0
°3
4.7
4
6.2
15
A2
-4
°1
0
0.7a2
0
°3
5.1
1
5.2
200
A3
-4
01
0
0.8a2
0
°3
4.7
4
6.2
260
A4
-3
0.7a1
0
0.8a2
0
0.7a3
3.8
3
4.8
200
A<2/
A5
-3
0.42a1
0
0.8a2
0
0.7a3
3.5
3
4.6
215
Aft*/
Ao
-1
0.7a
0
0.56a2
0
0.7a3
3.8
1
3.9
400
A7*/
-1
0.420
0
0.56a2
0
0.7a3
3.5
1
3.7
415
I/ 33
a. = 0.4 yg S02, for a 24-hour sample at 380 yg S02/m where 0.32 m of air
is sampled, a_ = 0.4 yg SO is equivalent to a standard deviation of 3.3%.
0 and o. are assumed to be 2.5% and 3% respectively of the average or mean
- 3
value, X = 380 yg S02/m .
d is also expressed as the bias in % of the mean concentration,
X = 380 yg S02/m3.
-/A5 = Al + A4, A6 = A2 + A4, A7 = Al + A2 + A4.
3/ / 2 2 2
0 = -t/a.. + a« + a- = percent variation in measured concentration of S0_,
J. i _L ^ J Q ^
calculated at a mean concentration of 380 yg/m .
-7MSE (%) =-J0J
98
-------
VO
V£>
CO
LJ
CO
8
a:
LJ
Q.
Q
O
400
300
200
CO
o
o
100
A7 =
A2 + A4 )
\ *A6 = (A24-A4)
BEST
STRATEGIES
COST OF REPORTING
POOR QUALITY DATA
A3
\A5 = (AI+A4)
V. A2 /
1
4
" V
/\
\
\
\
\
\A.
>-4°
5678
I
9
MSE (PERCENT)
Figure 11: Added Cost ($) vs. MSE (%) for Alternative Strategies
-------
To Illustrate a technique which aided in the development of Table 37
and Figure 11, consider the following linear approximation to the model
as derived from the sensitivity analysis (see Appendix 2 for a detailed
discussion) which includes the most important variables and explains
almost all of the variation in the measured concentration, C, of S0~:
C = 378.8 + 12,630X(1) + 1156X(3) - 1156X(4) + 37.9X(5) + 394.6X(6)
- 4.795X(13) + ... .
To improve the precision/accuracy of the measured concentrations
requires implementation of data quality control procedures applicable
to any one of the important variables or to a combination of them. For
example, the latter would result from implementation of action A7
(Figure 11) which is the sum of 3 actions and improves the precision
of more than one variable. Assume that the cost of implementing a
control procedure for any one variable X(j) is C , and for two or more
variables, C , , e.g., if X(j) and X(k) were being controlled. In
jK
practice there may be several levels of control which are desirable to
consider, and a function such as that shown below in Figure 12 is applicable-
.REFERENCE METHOD
^^^/v
TO
ADDED COST ,.OF IMPLEMENTING QUALITY CONTROL PROCEDURES
Figure 12: Precision of Reported Measurements Vs. Cost
of Quality Control Procedure
100
-------
It is desired to describe a methodology to select which quality control
procedures to implement for a given level of resources or which procedures
to implement to minimize cost while achieving a fixed degree of overall
precision/accuracy. For simplicity of discussion, assume that there is
one level of implementation for each variable, with associated cost C.
for variable X(j); i.e., one value other than the reference value for the
function in Figure 12.
With a small number of variables and a small number of control strategies,
it is easy to enumerate all cases, including combinations, as desired and
obtain a relationship of overall precision of C versus the added cost C .
The overall precision is given by the variance estimate
s2(C) = 12.6302 s2{X(l)} + 11562(s2{X(3)} + s2{X(4)}
+ 37.92 s2{X(5)} + 394.62 s2{X(6)} + 4.79S2 S2{X(13)}+...
or the standard deviation of the estimated C, s(C)
s(C) =
where s (X(j)} is the precision (standard deviation) of the measurement
of X(j). Figure 13 gives the form of the relationship.
To use the result of Figure 13 to determine the total cost corresponding to
a desired level of precision s_(C), one uses the procedure corresponding to
cost C . (See solid lines.) On the other hand if one has resources C ,
the best precision that can be obtained is s*(C). (See dashed lines.)
Although the above structure and example are very simple, the approach
illustrated is applied to a more complex problem. In general terms
the problem might be formulated as follows.
Problem 1
2
minimize s (C) or s(C); subject to given total added cost CT £ A
or
Problem 2
minimize C ; subject to s(C) <_S.
The mean square error may be substituted for s(C) in the above problem
statements.
101
-------
If the relationship of C. to s{X(j)} is known for each variable or
combinations thereof, then one needs a cost trade-off procedure, better
known in the literature as a mathematical programming procedure, for
solving such a problem. The cost is a linear function (additive) of the
corresponding cost elements, and the precision s(C) is a nonlinear
function but a very simple one. A computer program can easily be
structured to solve the problem for all cases of interest if the possi-
bilities are discrete in number, or a LaGrangian multiplier approach can
be used if a function form is given relating C. to s{X(j)}. More complex
problems can be solved using heuristic search techniques and dynamic
programming methods. Nonlinear programming techniques are also available
for complex problems and functions. However, it is felt that a reasonably
simple computer program which enumerates the function of s(C) versus C_,
would be adequate for problems 1 and 2 as stated above considering the
state-of-the-art in estimating the costs and associated precisions.
v>
5
| S.CT
Figure 13: Precision of Reported C vs. Total Added Cost, C,,
102
-------
DATA QUALITY ASSESSMENT
This section contains a discussion of the use of audit data to assess
individual measurements and of techniques for the combination of these
results to yield an overall assessment. The precision and biases of
the individual measurements and operational procedures are first esti-
mated. These results are then used to make the overall assessment
as required in the second method or approach. The following section
discusses a method of data presentation from a general point of view.
This is followed by some specific statistical procedures to be
employed in the assessment analysis.
Data Presentation
A reported value whose precision and accuracy (bias) are unknown is of
little, if any, worth. The actual error of a reported valuethat is,
the magnitude and sign of its deviation from the true valueis usually
unknown. Limits to this error, however, can usually be inferred, with
some risk of being incorrect, from the precision of the measurement
process by which the reported value was obtained and from reasonable
limits to the possible bias of the measurement process. The bias, or
systematic error, of a measurement process is the magnitude and direction
of its tendency to measure something other than what was intended; its
precision refers to the closeness or dispersion of successive independent
measurements generated by repeated applications of the process under
specified conditions, and its accuracy is determined by the closeness to
the true value characteristic of such measurements.
Precision and accuracy are inherent characteristics of the measurement
process employed and not of the particular end result obtained. From
experience with a particular measurement process and knowledge of its
sensitivity to uncontrolled factors, one can often place reasonable
bounds on its likely systematic error (bias). This has been done in the
model for the measured concentration as indicated in Table 37. It is
also necessary to know how well the particular value in hand is likely to
agree with other values that the same measurement process might have
provided in this instance or might yield on measurements of the same
magnitude on another occasion. Such information is provided by the
estimated standard deviation of the reported value, which measures (or is
an index of) the characteristic disagreement of repeated determinations
of the same quantity by the same method and thus serves to indicate the
precision (strictly, the imprecision) of the reported value.
A reported result should be qualified by a quasi-absolute type of statement
that places bounds on its systematic error and a separate statement of its
standard deviation, or of an upper bound thereto, whenever a reliable
determination of such value is available. Otherwise, a computed value of
the standard deviation should be given together with a statement of the
number of degrees of freedom on which it is based.
103
-------
As an example, consider strategy AO in Table 37. Here, the assumed
standard deviation and bias for a true SO- concentration of 380 yg S00/m
3 3 1
are OT = 19 yg S02/m (5.0% of 380 yg/m) and T = -15 yg S02/m ,
respectively. The results would be reported as the measured concentration
(yg S0_/m ) minus the bias and with the following 2a limits along with
the audit level and lot size N; e.g.,
(yg S0,/m3) + 15 + 38, n = 7, N = 100.
z m ~ .
3
For concentrations other than 380 yg S02/m , the overall standard deviation
is obtained by
crT(%) = Jcr]L + a,, + a3 ,'and
3 3
a_(yg/m ) = a (%) x yg S0./m
i i z
Assessment of Individual Measurements
The data collected during the audit program can also be used to assess the
precision/accuracy of the individual measurements. For example, suppose
that n = 7 independent audits are made of a particular measurement (e.g. ,
the concentration of a calibration gas) and that it is desired to estimate
the standard deviation of the reported data. In the first approach to
auditing the data, the difference between the audited value and the
original measured value by the operator or analyst was compared to the
permitted deviation or suggested standard, and the measurement was classi-
fied as defective or non-defective on this basis. In this second approach
the difference between the two corresponding measurements is treated
quantitatively to obtain the standard deviation and bias.
Let d be the difference between the two measurements, say operator
measurement less the audited measurement. Then
n
_/-J
n
E d-
is the mean difference and a measure of bias in the particular measurement.
The standard deviation of the differences is given by
w. - d)2
104
-------
This Is a measure of precision of the differences of two measurements.
As described in Appendix 4, this can be converted to the standard
deviation of a single measurement by dividing the result by V2 if each
measurement can be assumed to have the same precision. If the audited
measurement is made by a much more precise means than that of the routine
data, it is not necessary to divide by /2 as the a, will be almost equal
to that of the original measurement.
The quantity s, is also useful in checking for bias. The procedure is to
compute
d_
s,
and compare this value with the tabulated value of t for n-1 degrees of
freedom (See Ref. 27) , and if it is larger in absolute value than the t
value for a preselected level of significance, then it is inferred that
the mean difference is not due to chance but to some difference in the
two instruments, operators, or conditions under which the measurements are
taken, suggesting a bias in the resulting data.
The value of s, or the corresponding standard deviation of an individual
measurement s_,//2 can be compared to the suggested standard as indicated
in the subsection entitled Estimated Variance of Reported Data Using
the Test Statistic s2/a2 = *2/f.
Overall Assessment of Data Quality
Suppose that the bias and standard deviation of each of m important
variables in the measurement process can be obtained as suggested in the
previous subsection. These results are then combined to obtain an overall
estimate of the bias and standard deviation as follows. An estimate of
the overall bias is given by
, + d0 + ... + d
12 m
if an additive relationship exists between the individual measurements
and the desired measured concentration. In some cases a multiplicative
relationship exists and percentage biases were added.
A similar approach is employed in combining the standard deviations of
the individual measurements. The overall estimate
s2 + ...s2
L m
105
-------
where estimates of the standard deviations are obtained for m measure-
ments. See Appendix 4 for the assumptions made in deriving the above
results.
In modeling a complex relationship, computer programs exist for
simulating the distribution of the measured concentration and for
determining the sensitivity of the measurements to instrument errors,
operator variation, and environmental effects. These are described
in Appendices 1 and 2. With the use of these programs, errors/variations
in the several process variables can be propagated through the use of
the mathematical relationships to obtain the expected variation/bias
in the measured concentration. This approach does not require making
the assumptions concerning the additivity of the biases and variances
as employed above.
In cases where biases are expected, it is usually desirable to combine
the bias and the standard deviation to obtain a mean square error (MSB),
actually the square root of the value, as follows:
MSB
V2 2
(bias) + (standard deviation) »
MSB = y?2 + a2
This approach seems justified because the bias is expected to vary
from one agency-laboratory to another resulting in a distribution of
:biases rather than all laboratories having the same bias. This
laboratory bias is observed from examination of the collaborative test
results, where very often a laboratory which obtains high test results
on one day under given conditions continues to do so every day, for a
laboratory obtaining low concentrations, all results tend to be low,
even though both .laboratories are measuring samples having the same
concentration.
These measures of bias and precision are used in a cost versus
precision analysis as described in the previous section under the
heading "Cost Trade-Off Analyses." The methods of reporting or
presenting the data are given in the subsection entitled Data
Presentation.
106
-------
22 2
Estimated Variance of Reported Data Using the Test Statistic; s /a = X
In estimating the variance of the measured concentration of a pollutant,
the standard deviations of each of the variables are either hypothesized
or estimated from available intra- and inter-laboratory tests. It is
frequently desired to compare an estimated standard deviation based on
field data with the assumed value based on limited test data for the
purpose of determining if the true standard deviation is larger than the
hypothesized value. In order to make this comparison, the ratio of the
estimate s to the assumed value o is obtained. If this ratio is larger
than would be explained by chance, then corrective action should be taken
to determine the assignable cause of the large value. It is also possible
that the assumed value is too small because it was based on limited data
or an engineering guess in some cases where no data are available.
Methods are given in each of the field documents for estimating the variance
of a particular measurement, and the assumed variances (or standard deviations)
2 2
are given. Thus, if one computes the ratio, s /a , this value can be
2
compared with a tabulated value of x /f (read chi-square divided by degrees
of freedom f = n-1, i.e., sample size less one) which is given in Ref. 27.
If we are interested in detecting only unusually large values, then the
2
critical value of x /f roay be taken as one exceeded only 5% or 16% of the
2
time by chance. For example, if n=10, f=9, we read from the x /f table
that value exceeded 5% of the time (95 percent value) is 1.88. Hence, if
22 2
s la exceeds 1.88, one would infer that a is larger than the assumed
value with 5% risk of being incorrect.
If on the other hand we are interested in detecting large deviations
from the assumed value, i.e., unexpected large or small values, a two-
sided test can be made. For example, if n=10, f=9, the two critical values
are 0.30 and 2.114, outside of which 5% of the values fall. Hence, if the
2 2
ratio s /a is less than 0.3 or larger than 2.114, one would infer that
the variance is either smaller or larger than the assumed value with a 5%
chance of being incorrect .
This same distribution can be employed to set up a control chart for
the variance of a measurement of interest or in obtaining a confidence
interval estimate of the true variance. In the latter case a 95% confidence
2 2
interval estimate of 0 may be obtained by using the result that a falls
within the interval,
2,2
s 2 s
'SO <
2
.975 .025
107
-------
^
(x /f| 'denotes the p percentile of the distribution) 95% of the time".
IP
For example, if n=10, f=9, we have
2.2
_s 2 _s
2.114 - ° - 0.300
with 95% confidence.
2
Critical values of x /f vs. sample size are tabulated in Table 38 and
plotted in Figure 14 .
Table 38. CRITICAL VALUES OF s±/a±
Level of
Confidence
90%
95%
Statistic
si/0i
Si/ai
Audit Level
n=5 n«=10 n=15
1.40 1.28 1.23
1.54 1.37 1.30
n=20
1.20
1.26
n=25
1.18
1.23
'108
-------
1.60
1.50
o)
cd
-------
SECTION IV RECOfENDATIONS AND CONCLUSIONS
In addition to the recommendations given in Section II for the individual
measurement methods, the following recommendations and conclusions pertain
to the implementation of a quality assurance program in general.
In the field documents all control limits were set at the estimated 3o
value. If the monitoring application requires higher quality data, the
control limits can be adjusted to the 20 level by multiplying by 2/3. Some
limits as given may be more nearly what can be achieved in the field than
others; however, once sufficient data are obtained and all limits are
adjusted, the overall measurement process will begin to function in such a
manner as to allow for a valid and accurate assessment of outgoing data
quality.
Successful implementation and maintenance of a quality assurance program
will require special training periods for the supervisors and managers
responsible for conducting the program and decision making concerning data
quality versus cost. This training should include the subject areas
treated in Section III of this document.
Also, the managers and supervisors must converse with the operator. He
should be: (1) well trained on the measurement method of interest,
(2) continuously informed of what is expected of him, (3) informed of the
quality of the data that he is generating and (4) encouraged to improve
the quality of his work.
The present guideline documents should be continuously reviewed and updated
as more data and new control techniques/equipment become available.
110
-------
FINAL REPORT
1. Herbert C. McKee et al., "Collaborative Study of Reference Method for
the Determination of Suspended Particulates in the Atmosphere (High
Volume Method)," Southwest Research Institute, Contract CAP 70-40,
SwRI Project 21-2811, San Antonio, Texas, June 1971.
2. Robert E. Lee, Jr. and Jack Wagman, "A Sampling .Anomaly in the
Determination of Atmospheric Sulfate Concentration," American Indus-
trial Hygiene Association Journal 27, pp. 266-271, May-June 1966.
3. Robert M. Burton et al., "Field Evaluation of the High-Volume Particle
Fractionating Cascade ImpactorA Technique for Respirable Sampling,"
presented at the 65th Annual Meeting of the Air Pollution Control
Association, June 18-22, 1972.
4. Peter K. Mueller et al., "Selection of Filter Media: An Annotated
Outline," presented at the 13th Conference on Methods in Air
Pollution and Industrial Hygiene Studies, University of California,
Berkeley, California, October 30-31, 1972.
5. G. P. Tierney and W. D. Conner, "Hygroscopic Effects on Weight
Determinations of Particulates Collected on Glass-Fiber Filters,"
American Industrial Hygiene Association Journal 28, pp. 363-365,
July-August, 1967.
6. John F. Kowalczyk, "The Effects of Various Pre-Weighing Procedures on
the Reported Weights of Air Pollutants Collected by Filteration,"
presented at the 60th Annual Meeting of the Air Pollution Control
Association, Cleveland, Ohio, June 11-16, 1967.
7. C. D. Robson and K. E. Foster, "Evaluation of Air Particulate Sampling
Equipment," American Industrial Hygiene Association Journal 23,
pp. 404-410, 1962.
8. John S. Henderson, "A Continuous-Flow Recorder for the High-Volume
Air Sampler," presented at the 8th Conference on Methods in Air
Pollution and Industrial Hygiene Studies, Oakland, California,
February 6-8, 1967.
9. Walter K. Harrison et al., "Constant Flow Regulators for the High-
Volume Air Sampler," American Industrial Hygiene Association Journal
21, pp. 115-120, 1960.
10. Franklin Smith and A. Carl Nelson, "Guidelines for Development of a
Quality Assurance Program for Reference Method For the Determination
of Suspended Particulates In The Atmosphere (High Volume Method),"
Research Triangle Institute, Contract EPA-Durham 68-02-0598, RTI
Project 43U-763, Research Triangle Park, North Carolina, April 1973.
Ill
-------
11. H. C. McKee et al., "Collaborative Study of Reference Method for
Determination of Sulfur Dioxide in the Atmosphere (Pararosaniline
Method)," Southwest Research Institute, Contract CPA-70-40, SwRI
Project 21-2811, San Antonio, Texas, September 1971.
12. J. P. Lodge, Jr., et al., "The Use of Hypodermic Needles as Critical
Orifices in Air Sampling," Journal of the Air Pollution Control
Association 16 (4), April 1966, pp. 197-200.
13. F. P. Scaringelli, B. E. Saltzman, and S. A. Frey, "Spectrophpto-
metric Determination of Atmospheric Sulfur Dioxide," Analytical
Chemistry 39, page 1709, December 1967.
14. Franklin Smith and A. Carl Nelson, Jr., "Guidelines For Development
of a Quality Assurance Program for Reference Method For Determination
of Sulfur Dioxide In the Atmosphere (Pararosaniline Method),"
Research Triangle Institute, Contract EPA-Durham 68-02-0598, RTI
Project 43U-763, Research Triangle Park, North Carolina, June 1973.
15. Franklin Smith and A. Carl Nelson, Jr., "Guidelines For Development
of a Quality Assurance Program For Reference Method for the Measure-
ment of Photochemical Oxidants Corrected for Interferences Due to
Nitrogen Oxides and Sulfur Dioxide," Research Triangle Institute,
Contract EPA-Durham 68-02-0598, RTI Project 43U-763, Research Triangle
Park, North Carolina, April 1973.
16. Herbert C. McKee, and Ralph S. Childers, Provisional Report of
Collaborative Study of Reference Method for Measurement of Photo-
chemical Oxidants Corrected for Interference Due to Nitrogen Oxides
and Sulfur Dioxide, Contract CPA 70-40.
17. J. A. Hodgeson, R. K. Stevens, and B. E. Martin, "A Stable Ozone
Source Applicable as a Secondary Standard for Calibration of
Atmospheric Monitors," Air Quality Instruments Vol. .1, Instrument
Society of America, Pittsburg, 1972, pp. 149-158.
18. "Tentative Method For the Continuous Measurement of Nitrogen Dioxide
(Chemiluminescent)," Federal Register Vol. 38, No. 110, Part II,
Friday, June 8, 1973, pages 15177-715180.
19. Franklin Smith and A. Carl Nelson, Jr., "Guidelines For Development
of a Quality Assurance Program for the Continuous Measurement of
.Nitrogen Dioxide in the Ambient Air (Chemiluminescent),"
Research Triangle Institute, Contract EPA-Durham 68-02-0598,
RTI Project 43U-763, Research Triangle Park, North Carolina, .
June, 1973^ ;
20. Franklin Smith and A. Carl Nelson, Jr., "Guidelines for Development
of a Quality Assurance Program for Reference Method for the
Continuous Measurement of Carbon Monoxide in the Atmosphere (Non-
Dispersive Infrared Spectrometry)," Research Triangle Institute,
Contract EPA-Durham 68-02-0598, RTI Project 43U-763, Research
Triangle Park, North Carolina, March 1973.
112
-------
21. Herbert C. McKee et al., "Collaborative Study of Reference Method
for the Continuous Measurement of Carbon Monoxide in the Atmosphere
(Non-Dispersive Infrared Spectrometry)," Southwest Research Insti-
tute, Contract CPA 70-40, SwRI Project 01-2811, San Antonio, Texas,
May 1972.
22. John Mandel, The Statistical Analysis'of Experimental Data, Interscience
Publishers, Division of John Wiley & Sons, New York, N. Y., 1964. :
23. Anderson, R. L. and Bancroft, T. A., Statistical Theorjjf in Research
McGraw-Hill Book Company, Inc., New York, 1952.
24. C. A. Bennett and N. L. Franklin, Statistical Analysis in Chemistry
and the Chemical Industry, John Wiley & Sons, Inc., New York,. 1954.
s
25. Bartholomew P. Hsi, "Optimization of Quality Control in the Chemical
Laboratory," Technometries Vol. 8, No. 3, August 1966, pp. 519-534.
26. A. Hald, Statistical Theory With Engineering Applications, John
Wiley & Sons, Inc., New York, 1952.
27. A. Hald, Statistical Tables and Formulas, John Wiley and Sons, Inc.,
New York, 1952.
28. W. A. Wallis, "Tolerance Intervals for Linear Regression," Second ;
Berkeley Symposium.
1
29. W. A. Shewhart,. Economic Control of Quality of Manufactured Product,
D. Van Nostrand Company, Inc., 1931.
30. S. W. Roberts, "A Comparison of Some Control Chart Procedures,"
Technometrics, Vol. 8, No. 3, August 1966, pp. 411-430.
113
-------
COWER SIMULATION
GENERAL
Performance models were developed and utilized in writing guideline
documents for the reference methods for measuring suspended particulates
and sulfur dioxide. Simulation analyses were performed for both methods
using the performance models. The simulation process is described using
the high volume method as an example with sample computer printouts of
the results. Only sample computer printouts are presented for the
pararosaniline method. For details on the performance models or
individual variable models, refer to the appropriate subsection of
Section II of this document.
SIMULATION OF THE HIGH VOLUME METHOD OF MEASURING S.P.
In order to estimate how the concentration of particulate matter in air
varies as a function of the errors and/or variations in weighing, cali-
bration, reading flow rates, "pH of filter," relative humidity, etc., two
types of analyses are performed. They are a simulation analysis as
described in this Appendix and a sensitivity analysis as described in
Appendix 2. In the simulation analysis the parttoulate concentration is
denoted by S.P. and expressed as a function of thirteen pertinent variables
denoted by X(i), i = 1, ..., 13. These variables and the estimates of their
mean values and variations are required to perform the simulation. For
example, for a normally distributed variable its nominal or mean value is
given along with its standard deviation (see Appendix 3 for a discussion
of the normal distribution).
Some of the variables of the model for particulate concentration were
assumed to be uniformly distributed, i.e., all deviations from their nominal
values were considered to be equally likely over the.range of variation
indicated (see Appendix 3 for a discussion of the uniform distribution.)
All other variables were assumed to be normally distributed. This input
information is given in Table 1-1.
The next step in the process is to generate the random numbers with the
specified distributions and insert them into the model to obtain simulated
values of S*P. A sample of the first few values of the 100 values generated
is given in Table 1-2. In Table 1-3 is a printout of the computed means and
standard deviations of each of the variables in order to provide a gross
check on the simulated values. In Table 1-4 is given a ranking of the 100
simulated S.P. values in ascending order.
In summary, there are certain significant results to be derived from the
tabulated results. One is that the values of the S.P.'s shown in ascending
order can be plotted on normal probability paper (see Figure 1-1) to indicate
the expected variation of the concentration as a function of the accumulation
of errors of analytical method. A normal distribution with the mean and
114
-------
standard deviation computed from the simulated values is also shown on
Figure 1-1 for comparison with the simulated distribution. The moments as
computed and given in Table 1-5 can be used to test the deviation from
3
normality if it is desired. The standard deviation, 7.1 yg/m , is 7.1% of
the mean value.
The simulation analysis is clearly no better than the model and the
estimated values of the input data. These inputs are based on some actual
inter-laboratory tests and intra-laboratory tests; and when data are not
available, they are based on engineering and statistical judgments.
115
-------
TABLE 1-1: Input Data
MODEL i. SEOI VAR. NAMES
i
2
3
4
5
6
7
8
9
10
11
12
13
XI
X2
X3
X4
X5
X7
X8
VO
XlO
X13
NOMINAL VAL.UE
.65000E
.5SOOOE
.ooonoE
-. SOOOOE
-.20000E
.nnnnoE
.OOOOOE
.OOOOOE
lOOOOE
.20000E
.nnnnnE
. '.80000E
i
2
0
-1
-1
-. 0
0
0
1
2
n
0
DEVIATION
.60000E
.SOOOOE
lOOOOE.
.lOOOOE
.12000E
. .'30000E
'.20000E
.20000E
.SOOOOE
.84000E
DISTRIBUTION
1
1
0
0
-1
-1
-1 .
0
2
1
0
UNIFORM*
NORMAL**
NORMAL
UNIFORM,
UNIFORM
NORMAL
NORMAL
NORMAL
UNIFORM
UNIFORM
NORMAL
'NORMAL
UNIFORM
M
I-1
For all uniformly distributed variables the lowest value (6.5 in this case) and the
range of the distribution (6 in this case) are given; thus X(l) ranges from 6.5 to
12.5, and all deviations in this interval are equally likely... .
**
For all normally distributed variables the mean (nominal in table) and standard
deviation are given. Thus X(2) has mean 55 and a standard deviation of 5.
-------
.TABLE 1-2: Portion of Simulated Data
X(2) X(3) X(4) X'(5) X(6) X(7) X(8)
.7830E i .4782E 2 .1218E -2 .3853E -1 .1152E -1 .365lE -i .2990E -3 -.1627E -
_^09JiE._-l ^12Q.7_£ 2 .,JL62.9E..__. 2 ^655lE_.l____J>.8J775E._JI _,_99J57E,.._2 _. . __.__ .
.7068E 1 .5970E 2 --1612E "2 .7080E -2 .8629£ -1 .2536£ -6 .4795£ -1 .2783£ "1
2 .i*qs E £__^i_6J13£_f.__ .J9_9.81E. 0
.1149E 2 ,6?84E 2 .6437£ -4 -.4106E ~1 .997?£ "1 -.5824E "1 ".4.985E '2 ~.l075E ~2
.6980E 1 .58i6E 2 -.5803E "3 -.350lE "1 .9953E ~1 '.1802E "1 .5694E ~2 .3536E "1
_=^-4 3.59-^-1- -5 2^7-f 1 ^343S-E__2 -...932-4-E i ^9JJ.Mg 0
,8029E 1 ,5rifr2E 2 .4584E -4 -.3940g xi .455i£ -1 .6223E -1 .3764E -1 ,3962E -1
-1 ,27386..-! -,.l9.0.4g.__.2 _j.l842E- i ,14835 - -1 ,9083E .. 2 , , ,
.1177E 2 .5432E 2 -.2776£ ,2 .4l50E -2 .9o27E -1 -.5874E -1 .22,3E -1 .5693E -2
,1_ J.JJ.40E3 :
.9441E 1 .5H9E 2 -.1662E -2 .4351E ±1 .8875E -1 -.2245E -1 -.jre43E -2 -.3oo4E -2
L_ 2 .21.3.0 E.2 .1314£.1 .-1549E.. ..1 _i_r9.Z74 E..._2 1 . ^
.943QE 1 .4fc80E 2 .9699E -3 .2456E -1 .'5852E -1 .12g5E -1 ,2;22E -.1 .1Q34E -1
_.l .5_94 If __i .-212.2E -2 -,J.6_6.7£._._0 _^lO.?.2E. 1 ,-909.9E__2
.9729E ! ,46j.OE 2 -.3780E -3 .7666E *z --lOSiE -j, -.2684E -j_ .ii40E -j --i659E -t
_^-622.8E_Tl ^.1.840Ei._.l . 20.36.E..._.2 -_...36.53E £ , 1.45.8E._ i.. j.lOilE 3, .
1197E 2 .6340E 2 --1214E -2 -2622E J-i .4258E -2 -1642E -1 -.2222E -1 --1173E -3
_1J_032E.._T.l u2.06.5^__.2 ..19.32E 2 .r_._6Q.O_9.E.__l t.l369E__l .,.10.19E.__3__._ , :
889.1E 1 .5369E 2 --.1603E -2 -.4692£ -1 --1227£ -i -.4097£ -1 .3728£ -1 -.2776£ "1
-ilfi80E_.jJ. t.2_OjSIE _2 _. 1.439E ..._2 :3.702E__i. a2.1 d6 1 __.,.U02E ._3__
7379E 1 .5580E 2 .4721E -3 .9131E -2 .4750E "1 .3343E "2 .2967E -1 .9069£ -3
.1265E -1 ._! 16_1 F 2 __-_21.67E.._ 2 .3372E _. 1 ,.1488E __.1 _, 9_590E.._2
.6928E 1 .5932E: 2 -,50l4£ -3 -.2026£ '1 .83486 -1 ,6269£ -1 -.5l43E '2 '.23UE '1
JliiA38e._^l. iA9filE .2 a.-T.^E _._2._ .JLZ23E _ j, _,..8Q_3_7E__...P._t ^ 8_616E.._..2
.7666E 1 .5277E 2 .Il59£ -2 -.267i£ -1 ;6824£ -1 -.7494E -1 .7235E -2 -.1628E "1
.-.1128E -1 .1881E 2 iJ.,93.2F ?_ -_.J5842£.._ i .1449£ 1 .1109E 3
.984QE 1 .5237E 2 .8311E -3 -.1235E -1 ;3672E -1 -.5127E -1 -.2259E -1 .11Q3E -1
-.3780E -1 .1377E 2 .,2372E 2 .9432E 1 .lOgQE 1 .1044E _j :
-------
TABLE 1-3 Check of .Simulated Values
INPUT CHECK
MODEL 1. SEQ1 VAR. NAMES
1
2
3
4
5
6
7 .
8
0
10
11
12
13
H
. I-1
00
XI
X2
X3
X4
X5 -
X6
X7
X8
XQ
X10
Xll
X12
Xi3
Mean
NOMINAL VALUE
.94695E 1
.54879E 2
-.93897E -4
.46280E -3
.40245E -1
.14006E -2
.17010E -2
:i!067E 2
.20141E 2
V12168E 1
In the check the means and standard deviations are given
the mean is 9.47 and the
compared to the expected
the interval 6.5 to 12.5
standard
mean and
Standard
DEVIATION DISTRIBUTION
.17188E 1 UNIFORM*
.50161E 1 NORMAL
.1Q161E -2 NORMAL
.28397E -1 UNIFORM
.'35160E -1 UNIFORM
.'48554E -1 NORMAL
.30998E -1 NORMAL '
'.19365E -1 . NORMAL
,4Q94:*F -i UNIFORM i
;57i98E i UNIFORM
.45560E 1 NORMAL
. :?539Sf i NflRMAI
;23924E o UNIFORM
/
i
for each variable. For XC1;
deviation is 1.72; these values can be readily
standard deviation of
, which would have a mean of 9.5
/3 = 1.73. This is unusually good
agreement .
a uniform variable on
i
and a standard deviation ., '
-------
TABLE 1-4: Listing of Simulated Values in Ascending Order
I
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
£ 20
vo 21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
10
41
42
43
44
I/N
.010
.020
.030
.040
.050
.060
.070
.080
.090
.100
.110
.120
.130
.140
.150
.160
.170
.180
.190
.200
.210
.220
.230
.240
.250
.260
.270
.280
.290
.300
.310
.320
.330
.340
.350
.360
.370
.380
.390
.400
.410
.420
.430
.440
S.I.
.8416E
.8616E
.8731E
.8876E
.8892E
.8923E
.9034E
.9049E
».9069£
.9083E
,909C/E
.9116E
.9146E
.922f'E
.9240E
.9317E
.9339E
.9386E
.9409E
.9417E
.9421E
.9426E
.9456E
.9463E
.9493E
.952?E
.9554E
.9557E
.955BE
.9590E
. 962^5
.9660E
.9iS65E
" . 9686E
.9719E
.9729E
.97J4E
.973^E
.975?E
.9774E
.9784E
.978fE
.978F.E
.97956
2
2
2 , . .
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
?
2
2
?
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2 .
2
I
45
46
47
48
49
50
51
52
53
54
55
Sfi
57
58
^0
60
61
ft?
63
64
ft>5
66
67
hfl
69
70
7_1
72
73
74
7=5
76
77
78
79
80
31
82
03
84
85.
86
87
88
89
I/N
.450
.400
.4.70.
.480 '
.490
.500
.510
.520
_,530
.540
.550
.560
.570
.580
.590
.600
.610
.6^0
.630
.640
, .650
.660
.670
.630
.69-0
.700
.710 .
.720
.730
.740
.750
.7oO
.770
.780
.790
.800
.310
.920
.830
.340
.850
.860
.870
.880
.890
.9797~E
.9834E
.9853E
2
2
. .2.
.9854E 2
.9860E 2
.9866E 2
.9885E
.9936E
....9.93.7.E.
.9937E
.9954E
.997nE
.1002E
.1005E
1006E.
.lOoeE
1009E
.100..9E
lone
.1017E
..1019E
1021E
.1026E
lr'3?E
103?E
.instE
.1040E.
.1040E
.1042E
, 1044E
.10 4 'IE
.1047E
,1051E
.1053E
.106.71E
.106c:-E
.in??E
1074E
,1074E
.1076E
.107RE
.107PE
.1083E
.108BE
.loase
2
£.
> >
2
2
?
3
3
^
3
3
3
3
3
3
3 .
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3 ,
3
3
3
3
3
3
I I/N S.P.
90 .900 .1090E 3
91 .910 .1096E 3
92 .920 .HOOE 3
93 .930 .1101E 3
94 .940 .1102E 3
95 .950 .1109E 3
96 .960 .1136E 3
97 .970 .1143E 3
98 .980 .114BE 3
99 .990 .1179E 3
100 1.000 .1196E 3
-------
99.99
99.9 99.8
99 98 95 90 80 70 60 50 40 30 ZO
120
115
110
105
100
95
90
85
Figure 1-1
Simulated Distribution of
Particle Concentration
q
(Mean Concentration = 100 jjg/m )
and Normal Distribution Approximation
2 1 0.5 0.2 0.1 0.05 O.OI
oo
3.
1-g
<0
o
c
o
o
J-l
cd
PM
IkOl 045 0.1 0.2 OJ 1 2 5 10 20 30 40 50 60 70 80 90 95 98 99
Percentile of Distribution of Particle Concentration
99.8 99.9
99.99
-------
TABLE 1-5: Moments of Simulated Distribution
MOMENTS
S.P.
.FIRST 99.9661E 2
SECOND - .502728E 4
THIRD
.127074E 5
FOURTH
SKEWNESS
.723892E 6
_..7_1 Z6.0 4 £_.!-
.356498E 0
KURTOSIS
.286422E -1
VARIANCE
.515757E 2
-------
SIMULATION OF THE PARAROSANILINE METHOD OF MEASURING S02
Simulation analysis of the pararosaniline method of measuring SO. .was
performed in the same manner as described in the preceeding section of
this appendix for the High Volume Method. The SO. concentration is
denoted by Y and expressed as a function of fifteen variables denoted
by X(i), 1=1, 2, ..., 15. These variables and the estimates of their
mean values are required to perform the simulation. For example, for
a normally distributed variable its nominal or mean value is given along
with its standard deviation (see Appendix,3 for a discussion of the
normal distribution).
q
Some of the variables of the model for SO- concentration in ygSO_/m were
assumed to be uniformly distributed; i.e., all deviations from their
nominal values were considered to be equally likely over the range of
variation indicated (see Appendix 3 for a discussion of the uniform
distribution). All other variables were assumed to be normally distributed.
This input information is given in Table 1-6.
The next step in the process is to generate the random numbers with the
specified distributions and insert them into the model to obtain simulated
values of SO.. A sample of the first- few values of the 100 values generated
is given in Table 1-7. In Table 1-8 is a printout of the computed means and
standard deviations of each of the variables in order to provide a gross
check on the simulated values. In Table 1-9 is given a ranking of the 100
simulated SO. values in ascending order.
In summary, there are certain significant results to be derived from the
tabulated results. One is that the concentration values of SO- shown in
ascending order can be plotted on normal probability paper (see Figure 1-1
as an example) to indicate the expected variation of the concentration as
a function of the accumulation of errors of analytical method. A normal
distribution with the mean and standard deviation computed from the
simulated values was also shown on Figure 1-1 for comparison with the
simulated distribution. Also, the moments of the distribution as
computed and given in Table 1-10 can be used to test the deviation from
3
normality if it is desired. The standard deviation, 23.8 yg/m , is 26%
of the mean value. Results from a collaborative test of the method (Ref. n)
showed a standard deviation of 17 and 33 percent of the mean for repeati- .
bility and reproducibility, respectively, at a true concentration of 100 yg/m .
The simulation analysis is clearly no better than the model and the
estimated values of the input data. These inputs are based on some actual
inter-laboratory tests and intra-laboratory tests; and when data are not
available, they are based on engineering and statistical judgments.
122
-------
TABLE 1-6: Input Data (SCO
MODEL 1,
VAR. NAMES
1
2
3
4
5
6
7
8
9
10
H
12
13
14
15
XI
X2
X3
X4
X5
X6
X7
X8
X9
xio
Xll -
X12
Xi4
Xl5
NOMINAL VA12UE
.30000E
.16300E
.OOOOOE
.OOOOOE
»ioo9&E-
.OOOOOE
".40080E
".40060E
. OOdSOE
001)06 E
. . ..- i oodfloE-
-.50000E
1 5 0 0 0 E
_ _ . OOOfiOE-
-.50080E
-i
6
0
6
a_
C"4- -
6
-i
-1
0
6
--5- ---
-i
2
__5
"i
DEVIATION
.82QOOE -3
il2000E -rl
U20aOE -1
;l50dO£ "1
i?flnftnc . n
^-tcfwvwc "
J20000E -1
740800E «!
'.40000E '1
.11200E -1
579fiOE -2
- '«7000&E 1-
iSOOOOE -1
.50800E 1
lOOOOE 2
.50000E *l
DISTRIBUTION
NORMAL*
NORMAL-
NORMAL;
NORMAL
NORMAL- -- -'--
NORMAL-
UNIFORM**
UNITORM -
NORMAL
NORMAL
- NORMAL;
UNIFORM
NORMALS
- UNIFORM--
UNIFORM
For all normally distributed variables the mean (nominal in table) and standard deviation are given.
Thus X(l) has a mean of 0.03 and a standard deviation of 0.00082.
For all uniformally distributed variables the lower limit and the range of the distribution are
given. Variable X(7) has a lower limit of -0.04 and a range of 0.04.
-------
TABLE 1-7: Portion of Simulated Data (SO,)
XI X2 X3 X4 X5 X6 17 X8
X9 X10 Xll X12 XlJ ' XH X15 SE01
.?87BE *i .j696E n -.
-.8548E -2 -.7000E -3 .4493E t -.3274E -f .8906E t "-,'.44146 1 -.l92lE«-l .8279E 2
_ .3037E_-i __ .j79lE 0 __. __ _ ___ __.
..9844E _2
3213E -i* ".16946 0 .3374E -2' --.SlOSl -^1 ;ifll4E "2 -.2085E -1 -.3l35E' -\. -.32506 -1
.1?50E -1 -.7JB7E -g .449SE \ -.S9gaE < .tBTBE g
!
.2944E -1
-.9447E -Z -
I7~7 -.278lE -ill-
.3099E -i
.1660E II
.1007E
6694S -2 -*7867E i
.. .1523E D_
1394E _--2 .
-1 .?f43§
-.49126 -i
._-.4246E_-_2 .6387E.
5798E. i.__^.30iOE rj.
-.3562E
2 .32051=
-.7.7HE -4
*? .JOOOE 2
1557E 2
1...2593E ... 2
1 J1047E 2
-.S4??E
.6294E 1
.4089E
-.4307E
. i.^np <
*2 ..pOS^g .
-.903.36. -8 .
7.T..47B3E .'-i~
-1 -.2945E -
1 -.3479E -t
.9B45E 2
i -.3395E_-l __^
-J7403E 2. '..._-_: -
1 -.9980E -2
. _ . - -
«
.3042^ "1
-.2110E -1
.3066E -i
-.lOlOE -1 -
.17506 L
1392E -2
1529E 0
2768E -2.. -.
203lE i
.3719E
3145E i
j .*?«;£
-.5347E -2
-7 -.85D7E
1283E 2
-? .IflSOE 2
. .5408E ^1 _
- lAOnE
.5205E 1
.1537E
.307l£ 1
-i -.953«t -
-.3870E -i
-1 -.2590E -
. -,4077E...-l._
2 --??AOF -i
.8B85E 2
1 -.i?»8E -i
_.1167E 3 -
.2925E -i -1419E 0 -.1877E -i .3629E A2 .9818E 1 -.8235E -2 -.1943E -1 -.3934E -1
.41B5E -3 -.5784E »g JLttAAE 0 -.J497E -f .1443E ? .1?HE 1 -.1394E -j .62Q9E2
.3044E -i MtSSE 8 -.ggiBE -? -.84806 jg .969&E i -.33gQE *1 -.9844E -9 -.g87iE -2
.9800E -2 -.3594E -3 -.2383E 1 -.2642E ~i .1613E 2 .6147E 1 -.3320E -i .9678E 2
-------
TABLE 1-8: Check of Simulated Values (S02)
INPUT CHECK
MODEL 1, S£Ql
1
2
3
4
5
6
7
8
9
in
it
l?
13
14
15
VAR. NAMES
XI
X2
X3
X4
X5
X6
X7
X8 - _
X9
XlO
-X11-
XI?
X13
X14
X15
NOMINAL VAtlUE
.30103E -i
- ... .16306E 0
-.19097E -2
.96563E -3
.99832E i
-.19044E..-2-
-.20329E -1
.l97i9E -i
-;12091E -2
.40084E -3
-'.63892E 0
-.26241E -i
.14678E 2
.469J.5E i
-.23934E -l
BEVlATTON
J90222E -3
il0523E -1
*10127E -1
*15640E -1
.29516E 0
.20955E -1
. il0525E -1
,1 2615E -1
DISTRIBUTION
NORMAl*"
NORMAI
NORMAL
NORMAL
NORMAL5 " i
NORMAL
UNirORM
iiwirnnM
ao373E -i NORMAL
>65i02E -2 NORMAL:
t72386E 1 wnOMAi
'.13890E -1
.54267E 1
.30703E 1
il4035E -1
UNIFORM
NORMAL
UNIFORM
UNIFORM
In the check the means and standard deviations are given for each variable. For X(l) the mean
or nominal value is 0.03 and the standard deviation is 0.0009. This is checked against the
input data of Table 1-6 which shows a mean of 0.03 and a standard deviation of 0.00082 for
X(l). The agreement is sufficient to accept the simulation data for X(l) as valid.
-------
TABLE 1-9: Listing of Simulated Values in.Ascending Order (SO.)
DEPENDENT
1 IV
1 .
p
3 .
'. . 4
S ,
;L .'6... _.-.
7 :
R T
9 .
10
- 12 .
.13
15
16
17
18 - .
19 ..
2"
21
22
24 - .
25
P7
28
"3 o
30 .
31
32 .
33
34
36 »
37 .
3B t
39
40
'1
42 *
43 .
44 t
DATA
N v
olo
030
040
050_
060,
070
090
100
110-
120.
130
150
160
180.
190-
200
210
220
240 .
250
260
270
280
300
310
320
330
340
35(1
360 -
370
380
390
400
420
430
440-
LISTED IN
SE01
2980E
V -,4645E
..4843E
.SfiQSE
, *J?*E
541*F
' .5B80E
,:;, .. .6?09E
.63976
-. .6403E
.6575E
.6583E
*R9*E
.6979E
.7?13£
.7307E
7T99£
.7403E
.7739E
. .7B24E
.8I197E
.613BE
.8158E
,8?70E
. .8779E
.838PE
- .8451E
.8605E
.862JE
. AAHOE
.8773E
.6797E
ASCENDING
. .
2 '/'A .. ".-. ,-",
' .y ' '."
.' 2 ' ".-'V J
? '
9
9
.2 - . ;.-.
..Z"-: :.":.-. ;
y
J
9
'2 -.'.'".
.2 '.: 'f
2
2
J
2 .
j
2
-
2 ' -.'.
2 ' .' ."
M
?. .__
2
2
2
2
2
2
2
7
7
o
J
ORDER
45 -'. ' - .
-47'lV'V/oi
49 'r
sn
>! '.
53 ,
53 X J--.
54 :;:. :.'..-..;
5"' - - " i
56
57 ,
59 .-.-:..
60 ,:,. .
fl ; ' i
62 t
A4
65. ...-'.".
66 -" .
67 ' i
68 ,
69
70 i
71
72
73
74 i
75 -. .
Tf.
77
78
79
Rfl
81
83
84
85 .
86
87
HH
89
450
470 t'-'Z
480 ..
49"
sin
K5A
530 :.-r:;;
540 ^ -.<;.
JIIBV -:v.
560
570 .
^Rn
590 ,: :
600 ,,' ..
*in
62n
641
650
640
670
680
690
700
710
720
730
740
750 -.-.
741)
770
780
791)
Ran
R\n
B2fl
830
840 :
850
860
870
R80
890
.BB2BE
.8885E
.8987E
9ll*iE
^9l25E
.9189E
,9?57E
.'9?89E
.9P92E
.949iE
.9622E
i067"E
.9767E
.9A44E
.9889E
.9927E
.inoiE
.1003E
.1027E
. 1031E
1037E
.1037E
.1054E
.tn68E
.1070E
.1070E
.1091E
.1107E
-1119E
.1133E
.1152E
.1167E
.1188E
,1197E
..1201E.
.1703E
.1720E
2. .. ; 90 ''. .900 .1P29E 3 ;
».. . '. . . ". . .'.' ..** -' TT- ->*B : -rlZJOC J I ...
2 ..-;-,:."..;. \-: 92.,:.. ^;.::,..;92n ' .1737E 3 f-
2- :'; .!;'-:?:** »3 vV"/ .930- ' .1?60E 3 I.
J- 9* -950 , 1?R*F 3 !
? »« -960 ,110
-------
TABLE 1-10: Moments of Simulated Distribution (SO,)
MOMENTS
FIRST .918318*= 2
SECOND .561767E 5
IM1RD =u» 1352-15S6
FOURTH .834260E 8
.STD. DEV. .238210E 2
SKEMNESS ^.101§52E 0
KURTOSIS .264356E -1
VARIANCE - GOVARUNCE MATRIX, ORDER
VARIANCE .568121E 3
127
-------
APPENDIX 2 SENSITIVITY ANALYSIS
GENERAL
Performance models were developed and utilized in writing the guideline
documents for the reference methods for measuring suspended particulates,
and sulfur dioxide. Sensitivity analyses were performed for both methods
using the performance models. The sensitivity analysis process is
described using the High Volume Method as an example with sample computer
printouts of the results. Computer printout and a short discussion of the
results are presented for the pararosaniline method. For details on the
performance models and individual variables, refer to the appropriate
subsection of Section II of this document.
SENSITIVITY ANALYSIS OF THE HIGH VOLUME METHOD OF MEASURING
SUSPENDED PARTICULATES
A sensitivity analysis is performed on the model using as input the mean
(X) and deviation (DX) of each of the variables as given in the last two
columns of the upper part of Table 2-1. Note that these inputs are
identical to those used in the simulation analysis for the normal variables?
however, in the case of the uniform variables the mean and 0.25 x r-ange are
used as inputs. The computer program then calculates the first and second
order partial derivatives of the concentration of suspended particulates
(S.P.) with respect to each of the variables. These are given in Table 2-2
along with measures of sensitivity, the measure of variation in S.P. with
respect to that variable over the expected range of variation. ,Thusx for
example, variable X(6) explains the largest variation""in"S.P. as sensi-
tivity is defined herein. Very briefly, the linear sensitivity is defined
as the ratio of the expected change in S.P. to the mean value of S.P. for
the expected change in'x(i), i = 1, ..., 13. Hence, it is the product of
the first partial evaluated at the nominal or mean values of each of the
variables multiplied by the expected change in X (say, 2DX) and divided by
the nominal value of S.P. This sensitivity measure corresponds to the
first order terms in a Taylor series expansion of the model, S.P.=f(Xl, ...,
X13). The non-linear sensitivity is the contribution of the second order
terms in the Taylor series, omitting the mixed terms.
A step-by-step explanation of the procedure used in the sensitivity
analysis for estimating the mean and standard deviation using a simple model
containing only 3 variables is presented here to help clarify the above
discussion. In the modeling of S.P. as a function of the several variables,
an approximate linear model is obtained by numerically differentiating the
function
S.P. = f(X(l), X(2), ..., X(13)),
j\O p CiC p JiC p
obtaining the partial derivatives 3x/-iN . aym 3X(13) , and then using
128
-------
a Taylor series approximation with only the first order partial
derivatives, thus
S.P. * S.P.-+
3S.P.
3X(1)
DX(1) +
3S.P.
3X(2)
DX(2) + . .. +
X
3S.P.
3X(13)
DX(13) +
X
where S.P. is the average value of S.P. calculated using the
nominal values of all the variables in the
performance model, and
3S.P.
3X(i)
is the value of the i partial derivative with
X each variable at its mean value.
From this equation, the mean and variance of S.P. are obtained in the
computerized approach as follows. Suppose that for example and simplicity
of computation, the following approximation adequately describes the
performance model
S.P. = 200 + 2DX(1) + 5DX(2) - 3DX(3) , Eq(2-l)
and that the means and variances of the variables are as follows: __
Mean Variances
DX(1)
DX(2)
DX(3)
0
0
3
42 = 16
52 = 25
22 = 4
Then by substituting the mean values of the variables into Equation 2-1 and
performing the calculation, an estimate of 191 is obtained for the mean by
Mean (S.P.) = 200 + 2(0) + 5(0) - 3(3) = 191 .
To estimate the variance of S.P., multiply the square of the coefficients in
Equation 2-1 by the respective variances as
Variance (S.P.) = 0 + 22{DX(1)}2 + 52{DX(2)}2 + 32{DX(3)}2
129
-------
and by Inserting the standard deviations of the variables and performing
the' calculations,
Variance (S.P.) * 0 + 22(42) + 52(52) + 32(22) ,
that is, the sum of squares of the coefficients by the variances,
Variance (S.P.) = 64 + 625 + 36 = 725
or standard deviation of S.P. = /725 - 27. Note how one variable plays a
dominant role, in this case X(2). Typically only a few variables signif-
icantly affect the variation of the function. The above computation assumes
that the variables are independent; i.e., variation of one variable toward
a high (or low) value does not alter knowledge concerning the variation of
another variable. If the variables are correlated, an estimate of the
variance of S.P. can still be made, given the degree of correlation.
Caution should be observed when using this type approximation. Some of the
coefficients of the standard deviations change as the true concentration
changes. Until more information is available, it is advisable to derive a
new approximation through a computer run any time the concentration level
of interest differs more than about + 20 percent from the concentration
used to derive the original approximation. Also, large changes (e.g.,
+ 20 percent) in the standard deviation will cause a change in the coeffi- .
cient if the relationship between that variable and the average concentration
is nonlinear.
Returning to the analysis discussion using all thirteen variables, the
nominal value of S.P. and the standard deviation of S.P. computed by using
the first order terms in the Taylor series and performing a usual "error
analysis" are given at the bottom of Table 2-2. Referring to Table 2-1
again, there are several additional and useful computations. For example,
the "worst case limits" are defined as the largest and smallest values
of S.P. using the appropriate combination of the largest and smallest
values of each of the variables. Each limit of X(i) is defined by
X(i) + 2DX(i), 1=1, 2, ..., 13. The value of X(i) + 2DX(i) for which
S.P. is larger is used in computing the upper (lower) limit. The worst
case limits and the nominal value are given below this table of limit values.
The next few lines of the table give a comparison of the "worst case" values
of S.P. computed from the model versus those computed from the Taylor series
using (1) the first and second degree terms and (2) the first degree terms
only. The bottom part of the table contains a check of the individual terms
in the Taylor series. The columns headed by S.P.(X - 2DX)/S.P.(X) and
S.P.(X + 2DX)/S.P.(X) are the values of S.P. at the respective values
X(i) + 2DX(i) for i = 1, .... 13, divided by the nominal value of S.P.
130
-------
The columns headed by 1 + SENS, 1 + SENS + NON LIN are the predicted values
using the first, and first and second degree terms, respectively, of the
series. Note the good agreement throughout this tabulation.
The significant feature of this analysis is that it provides a measure of
where the significant variations in the S.P.'s result from the analysis or
measurement process. In this example, it is clear that (see Table 2-2)
X(6), X(7), and X(4) are the three largest contributing variables to the
variation in S.P. Hence, quality control and assurance techniques which
will minimize these variations; namely, errors in flow-rate calibration,
variations resulting from changes in temperature end/or pressure, and
errors in flow-rate readings should be looked at first. In this
particular run, it is assumed that the pH of the filter and the relative
humidity of the conditioning environment, RH, are controlled within narrow
limits. Further uses of the results of the sensitivity analysis in the
decision process are discussed in the following paragraphs.
The previous steps in the analysis have identified the important variables
in the measurement process and have estimated to what degree they affect
the value of S.P. The sensitivity analysis also provides an approximate
model of the process which may be written directly from the printout
(Table 2-2) as
S.P. = 100 + 0.531 DX(1) + 0.133 DX(2) + 466.8 DX(3)
- 98.2 DX(4) - 47.1 DX(5) - 98.2 DX(6) - 98.2 DX(7)
- 98.2 DX(8) + 48.0 DX(9) - 0.106 DX(10) - 0.0583 DX(ll)
- 0.0895 DX(12) + 3.38 DX(13).
The coefficients of the DX(i), i = 1, ..., 13 are the first order partial
derivatives read directly from Table 2-2 under S.P.'. This model approxi-
mation may be used to relate the variation in S.P. as estimated by the
standard deviation in S.P. to given variations or errors in the X(i)'s.
The adequacy of this approximation is checked thoroughly in the sensi-
tivity analysis. By introducing more control or improved instrumentation,
one can reduce the standard deviation in the X(i)'s, and hence, the standard
deviation of S.P, The relative cost of providing the additional control
or instrumentation must be weighed against the improved precision and/or
accuracy of the results. The computer program can be used in the decision
process as follows: Change the variation DX(i) of the particular X(i) of
interest to the value expected with improved instrumentation or introduction
of a quality control procedure, and compute the estimated standard deviation
in S.P. as was illustrated previously using a simplified model.
131
-------
Coefficients in the above approximation were obtained for a true average
3
concentration of 100 yg/m (approximated by the constant term) and with the
variables modeled as shown in Table 2-1, page 133, the two columns titled
"Mean" and "Deviation". A new approximation must be generated by a
computer run for other concentration levels or when a variable model is
significantly changed.
This approach gives an objective procedure for making decisions relative
to equipment needs, need for further data on critical variables, and where
to introduce more quality control and assurance, sampling, etc., in order
to obtain the desired improvements in the results. The day-to-day variation
in concentration of particulate matter are not included in the analysis
given by the model.
The decision model derived in the sensitivity analysis is subject to the
validity and adequacy of available data and statistical and engineering
judgments when data are not available. If the model is based on too many
assumptions without any supporting data, it is desirable to plan an experi-
mental program to estimate the variation in S.P. resulting from variations
or errors in the measurement process. The sensitivity analysis should be
used to identify the important variables; interactions, if any, between
the variables; and the expected effect of the variables on S.P. This
information will be useful in designing an efficient experimental program
to improve the model. Such an approach should have most value in the
initial stages of evaluating new instruments and/or analytical procedures.
132
-------
TABLE 2-1: Partial Output of Sensitivity Analysis
WORST CASE LIMITS
VALUE OF VARIABLE AT LOWER LIMIT AND
XI .65000E 1
X2
X3
X4
X5
X6
X7
X12
*l 3
WORST CASE L
SP
INTERACTION
G sp
w INTERACTION
SP "
GOODNESS OF
VARIABLES
''''' X4
X5
X6
X7
..... ;.- Xg :. .;
'' X9 ' . /
XiO
Xll
X12
X13
IMITS AND
.45000E
-.20000E
50000E
.10000E
itiOOOE
L60000E
'.40000=
-.70000E
^2lOOOEL
30000E
lOOOOE
,8(5000p
2
-2
- 1
0
0
~\
-1
-1
.. 2 '
2
2
n
AT UPPER LIMIT .
.12500E 2
' -
%
^
NOMINAL VALUE
. t. A o * n F n
CHECK USING jST ANQ 2ND DEGREE
. 64J42F: 2
CHECK USING iSt DEGREE TERMS QF
...617.24E '2. '"':.
FIT USING
SP(X-2DX),
.9R407E
.99067E
.1051 6E
.10291E
.11008E
.10625E
: , .10408E
: .96642E
10106E
.10058E
.10070E
*9864lE
1ST AND
/SHX)
Q ' ^
0
0 . .
2ND TERMS OF
I.-SENS i:
98407E Q
986746
99067E
10490E
1 .10283E
1 .10981E
1 .10589E
1 ' '
o ;
1 '''''?.
1
1
0
10392E
96642E
101066
10058E
10069E
98580P
0
0
1
1
1
1
1
0
1 , -
1
1
0
.65000E 2
'.200006 ^2
.50000E -1
.200006 -1
.lOOOOE 0
.400 OOE -i """
.70000E -1
.100006 2
.lOOOOE 2
.-.14476F' :3V' ' -
TERMS OF TAYLOR
.I4nfl3p 3
TAYLOR SERIES
.13841E 3
TAYLOR SERIES
-SENS+NON LIN
.9B404E 0
'.9925£E 0
.99067E n
.10515E 'i
.10290E 1
.11077E 1
.10623E 1
.10408E .. 1
.9664lE 0 ..,
V.10106E -1 ;
.1005BE l
.16070E l
,98635E 0
Mean
X
.95000E
:.55000E
.OOOOOE
* .OOOOOE
.40000E
.OOOOOE
.OOOOOE
.OOOOOE
.OOOOOE
.11000E
.20000E
..OOOOOE
>10007fc
SERIES
lOi59E i
"-. i o 21 2 E i ;
.10093E . 1
.-95324E 0
.97252E 0
.91065E 0
.94440E 0
"">' .96223E 0 ;
;.l0336E 1 >
' .98941E 0
.99417E . 0
99310E 0
.10147C ±
1
2
0
0
-1
0
Q
0
0
?
2
0
1
.
Deviation
DX
.15000E 1
.500006
.lOOOOE
.25000E
.JOOOOE
.50000E
.30000E
1
-2
-1
-1
-1
.20000E -l
.35000E -l
.50000E 1
l.+SENS
lOl5?E_
10133E
10093E
95095E
.97174E
.90189E
.94113E
,.
.' - '
.
96076E
10336E
96941E
99417E
99305E
10142E
.50000E
.50000E
..2.1JCLO.OE
L*
1
1
1
0
0-"
0
0
0
1
0
0
0
1
0
SENS+NON
_-lOi59E
.10093E
..95335E
.97254E
.9H51E
,94460E .
.'96230E
10336E
.96940E
.99417E
.99309E
10147E
LIN , '
1 '
1
0 ;V:.".v
0
0
0
0 . ;;;:;;. . \
. . -. ;'- --.-- - ' - '
0
o - ..-..
-------
TABLE 2-2: Printout of Sensitivity Analysis of S.P.
FIRST AND. SECOND PARTIAL DtRivATivEs (s.p.1 and S.P.") OF S.P. WITH RESPECT TO x " ~~~
PARTIALS SENSITIVITY Ranking of Linear
X S.P.(X-2DX) S.P.(X-IDX) S.P. (X+1DX) S.P.(X+2DX) S.P.' S_.P." ; LINEAR NON-LIN Sensitivities'
XI .98477E 2 .992746 2 .10087E 3 :iOl67E 3 .53134.E 0 -.13563E -3 .15929E -1 -.60991E -5
X2 .99184E 2 .99533E 2 .10090E 3 .10219E 3 .13264E Q .11535E -1 .13254E -1 .57632E -2
X3 .99138E 2 .99604E ? .1Q054E 3 '.lOjOOE 3 .46684E 3 '.203455 2 .933005 -2 -.40661E -6
X4 .10523E 3 .10259E 3 v?676fc * .95392E ? -.98176E 2 .19242E 3 -.49053£ -1 .24035E -2
X5 .10298E 3 .10151E 3 .98677E 2 .97321E 2 -.47125E 2 .445iOE 2 -.28255E -1 ^8006'iE -3
X6 .11096E 3^-10523F_3 ,95392E_ 2 '.9113QE 2 ».98l8lE 2 «1.?2556 3;'-.98llOE -1 .9620BE -2
3 .103HE 3 .97210E 2 .94508E 2 -.98186E 2 .19271E 3 -.58869E -1 ..34664E -2
X8 .10416E 3 .102ME 3 .98146E ' .96292E 2 -.98178E 2 .19267E 3 -.3.92436 -i .15402E "-2
X9 .967HE 2 .98391E 2 .10175E 3 ;10343E 3 .48002E 2 -.28234E 0 .33577E -1 -.69124E -5
X10 .10113E 3 .10060= 3 .99541E ' 99012E 2 -,10599c « -.13021E -4 -.105
-------
SENSITIVITY ANALYSIS OF THE PARAROSANILINE METHOD OF MEASURING S02
The same type of sensitivity analysis as was described in the previous
section of this Appendix for the High Volume Method was performed for the
Pararosaniline Method of measuring S0».
The input values of the mean (X) and standard deviation (DX) of each of
the fifteen variables are given in the last two columns of the upper part
of Table 2-3. These are the same values used in Table 5 of Section II of
this document.
The column titled Y@ (Y@ represents the first derivative) in Table 2-4
contains the coefficients of the linear terms of the Taylor series approxi-
mation of the performance model at a true SO concentration of 800 yg S02/m .
This represents computer run number 1 given in Table 7 of Section II. The
variables can be ranked according to their effect on the measured concen-
tration by ordering the values given in the next to last column of Table 2-4.
In this case only linear terms are important since all non-linear (second
degree) terms are zero (last column in the table). An approximate model
for. estimating the average measured value composed of the five most
important variables is
Y = 757.6-9.59 DX(13) + 75.76 DX(5) + 25254 DX(1)-1156DX(14) + 789 DX(6).
To see how much of the total variance is accounted for by these five vari-
ables, substitute their individual standard deviations, as given in Table 2-3,
in the following relationship
a2(Y) = 0+(9.59)2(5)2+(75.76)2(.28)2 + (25254)2(.00082)2 + (1156)2(.015)2
+(789)2(.02)2,
a2(Y) = 4118,
o(Y) = 64.
Then 64 is approximately 95 percent of 67.44 which is the standard deviation
given in the last line of Table 2-4 and was computed using all 15 variables.
In this instance then quality control procedures should be directed toward
these 5 variables, specifically X13 which accounts for approximately
50 percent of the total variability.
The above approximate model is only valid for SO, concentrations around
3 3
800 yg S09/m . Approximate models for concentrations of 400 yg SO./m and
100 yg 30,,/m were obtained from computer runs 2 and 3, respectively, as
given in Table 7 of Section II.
135
-------
For S0_ concentrations close to 400 tag SCK/m the following model
containing the five most influential variables is
Y = 379-4.8 DX(13)-1156 DX(4) + 1156 DX(3)+38DX(5)+12627 DX(1).
o
A similar approximate model for concentrations around 100 yg SO./m is
Y = 95-1156 DX(4)+1156 DX(3) - 1.2 DX(13)+9.5 DX(5)+3157 DX(1).
The changes in the coefficients for the above models result from the inter-
actions in the original model; that is, it contains several product terms
of the form X(l) X(2).
136
-------
TABLE 2-3: Partial Output of Sensitivity Analysis (S02)
WORST CASE .LIMITS
VALUF OF VARIABLE AT LOWER L'MfT AfjO AT UPPER |_lMl
XI .28361E -1 .31640E
X2 .1630DE 0 ' .16300E
X3 -.24000E -1 .24000E
X4
r:- .;^ ;/:'..' x5 ' "
X7
X8
X9
; " , xi o ;
'';,. - ."', Xt1 ':
'. ' :".'''.,":" .'xi? ' ' v
X13
X14
XI 5
WORST CASE LIMITS AND
* * , * SFOi :
r M*TERAC'TTON cHECK^Tl'STI
SF-01
3000nE -1 .
.94400E 1 "v
-.4000nE -1
-.4000I1E -1
-.40000E -1
-.22 4,0,0 E -1
.'-.-1.14005 -1
-.1400ftE 2
-.sooonE -i
2500I1E 2 .
.oobbnE o ~
-.50oonE -1
NOMINAL VALUE
.4789?E 3 -
flKsT-TRITTNin
.35113E 3
-.30000E
.10560E
.40000E
.OOOOOE
.OOOOOE
.22400E
.11400E
.14000E
.OOOOOE
.50000E
.IOOOOE
.ObofloE
\ , 1
. . >.13446E
iEGRF.E-TF.RMS~ Of
.11641E
IT ,
-1
0
-1
-1
-i- ,-' >
0
0
-1
1
0
i
2
0
} \
TAYUOFT
4
X
,3000nE -
,!630nE
.nobonE
.nononE
,1 OOOOE :
. 0 0 0 0 0 F. "
-.20000E -
-.?ononE -
.nononE
\ v .nooonE
"" , .nooonE .
""-.?5nOOE :'-
.15000E
.50000E
-.?500flE ~
> - i^ -o f
"""- ,7576?E
SFHIES
1
0
0
n
2
n
l
1
0
n
0
i
?
i
1
3
nx .
.8?onoe.-3
.iPonoR -i
.i?onoE -i
.loonoE -i
.inonoE -i
-i nonoE -i .
11200E -1
' ; .57000E -2 ' -:". '
' V" ' '.;.700noF 1 '"'' - '; ' ''"'**'
" '..1350 OF -1 :
.51000E 1
. 2 5 0 h 0 E 1
. 1?510E. -l
. . '/:?:, \ ::^i
INTERACTION CHECK USING 1ST DEGREE TERMS OF TAYLOR SERIES
SF.Ot .35111E 3 ' .11641E 4
GOODNESS OF FIT USING 1ST AND
VARIABLES
xi
- X4 - '''. '
X5
X6
X7
X9
xlt
X12
Xi3
,f;j-g;|;xi5 1'
Y(X-2DXj/1
. 94534F
10000F
.96337F
.10458F
.94400^
' .95833F
979i7F
.97667F
"f .98813F
.99028^
.97468F
Vii266F
.97468F-:
f(X>
0
1
0 ~
1 *"
0
0
0
0
: ^
0
0
1
O.-'V-*"
;-0-'---^-..
?ND TERMS OF TA
1 .-SF.NS
.94533"E~
.1 OOOOE
.96337E
.10458E
.94400t
r95833E
. 9-7-9^7 E~
.979i7t
.97666E
.98812E
.99027t
. 9 7T6~BE~
.112"615E~
.962036^
l.-S
0
' l;^A5.-<.:
;^':3il':
0
0
0
0
0
0 - ' ,
0 .
0
1
\0-:''-.-$:- .'«
YLOR SER
ENS + NOTT
"945 33E
lOOOOfi
94400^
958336
9 79 17?
V79l7t
97666E
99027&
V74~68f >
112666
J?5?8|%
IEl>
LIN
6
J-
1
0
0
0
U
0
0 "
0
0
1
1 'j'-i
Y{ X»2bx )/
.10547F
lonooE
"" -10366F.
",' -'.95421 E;
.10560E
10417E
.10208E
.10208F
" ' -10233E
. ' ,.l6li9E
lonvyt
10253E
..8734iE
^jff^'\ 0 25 3 .E^
»(
1,
^1
1
1
1
1
1
1
1
1
U
-------
TABLE 2-4: Printout of Sensitivity Analysis (SO )
CO
00
t"IKST 4NP bECUND HARIKL DFRIVATIVbS lYd AND Iff) Of SfcOl H1TH RFSPFfl TO X
PARTIAIS SFNSITJVITY
X . Y(X-ZnX) YfX-lDX) Y(X + inX) . Y LINEAR NON-LIN
XI .71621F 3 .73691E 3 .77833F 3 .79904E 3 .25254F 5 .OOOOOF 0 .54666F -1 .onOilOE 0
X? .75762?
.000006
T T<3. .7P987F-
1 .138766
X4 .79231F
-.17344F
X? ./1519F
21Z13F
X6 .75.606F
.. 157846.
X7 .74184F-
./HOBOS
X8 .74184F
'.7fl888F
i
X9 '.73995F
.8B3V8K
X10 .74863F
.44V93F
Xll .75026F-
.360386
Xl2 .73844F
.9S923F
iT13 -.85353F-
Xt4 .7J886F
.14384F
»lb ./3844F
3
u
3-
3
2
3
3
3-
1
3
3
i
3
1
3 '
i
3
T"
-3-
3
2
-3-
.95923F i
»i L x AT NOMINAL.
RTf) DEV OF"YTSn
nr-
.070006 *
.75762E
ipoooe. -
-74374F
.77496E
.1SOOOE -
.73641E
.74iS4E
.200006. *
-.74973E-
.74973E
ihOOOt -
74878E
3
1
T
3
1
3
0
3
1
3
3
1
J
112006 -1
,5/UUOE
-.7'5394E
.700006
.74803?
1?5006 -
-805X76
.500006
743?4E
.290006
-774803^
i?500!-
Y »
.67437E
£
3
3
1
7
1
3
1
y
.
000006
.75762?
.000006
T77150F-
000006
74028E
.OOOOOF
.77883E
000006
.77341F
.doOnof
T7655IF"
OOOOQf
.OOOOOfc
T766-46R
. ooouuh
.76212F
, onouoF
776i3fE-
.000006
.76722^
.ooonoF
7709g7F
000006
7720lF
.000006
.76722E
.DOODOb
75762E 3
2
0
3
0
3 ;
0
3 .
0 .
-3 v
0
3
4 '..
-3-T
0
3
0
3
D .
3 .
U
~3
V
3
0 .
-37
0
3
0 .
3
0
75762E
14400F
78537E-
144006
72293E
2250 OE
80005E
78400t
789j9E
400006
77341E
lOOOOt
77341F
10"OOT)6~
77530E
125446
76662E
77* 9TTE"
490001:
77680E
15625fc
6«172E"
78639E
625uot
77680E
iS625fc
o
3 .OOOOOF 6 .OOOOOF o .ooonoF o .onoooE o
-3
-3 i 11364 F 4 TO n 0 0 0 F ^0 T3 6 ^ I15" *1 TO n 0 0 06 0
-3 ...-'..
3 -.11562F 4 .OOOOOF 0 -.4<>784F -1 .OOOOOE 0
-3
'
3 .75760F 'i .OOOOOF' 0 .5b998F -j .01000E 0
"1
3 .78922F 3 .OOOOOF 0 .4t668F -j .OOOOOE 0
-3
-3 T78888F 3 rOOOOOF 0 T2n8?5f:~st TOnOODE 0
3 .78888F 3 .OOOOOF 0 .208?5F -1 .OOOtlOF 0
-3
3 .78927F 31 .00000-f 0 .zJ3j6F -\ .OflflnflE 0
3 .78936F 3 .onOOOF' 0 .11877F -1 .ODOOOE 0
-4
'» -
2
3 .76738F 3 .OOOOOF : 0 .253P2F -1 .OOOOOE 0
-3 ' "
3 -.95899F 1 '. OOOOOF 0 -.'1?698F 0 .UIIO'IUE 0
Z
3* .57534F i .OOOOOF 0-., .37970F -1 .OOOOOE 0
1
3 .76738F- 3 .OOOOO^ 0 .2^32ZE -1 .000005 0
J . ' :-.
-------
APPENDIX 3 FEQUENCY DISTRIBUTIONS
NORMAL (GAUSSIAN) FREQUENCY DISTRIBUTION
A brief discussion concerning the normal or gaussian frequency distribution
is given herein. The normal frequency distribution has been used frequently
throughout the field documents and in this final report to describe the
variation of a conceptual population of measurements. For example, suppose
that measurements are being made of the concentration of a specific pollu-
tant in a control gas sample. Conceptually a very large number of such
measurements could be made, generating a frequency distribution of these
values. These data will have an average or mean value, the true value of
the concentration, and a dispersion about this value measured by the root
mean square of the deviations from the mean. The latter statistic
(corrected for bias in the variance estimate) is referred to as the
standard deviation. The mean and the standard deviation completely specify
the normal distribution. That is, from the use of these parameters one
..can answer any question concerning the likelihood of a measurement taken
at random falling within any specific interval. (Notation used throughout
this document to represent a normal distribution with a mean value y and a
standard deviation 0 is N(u,0).)
If the mean and standard deviation are denoted by y and a, respectively, the
normal frequency distribution is given by the mathematical formula
p(x) = exp
o/2ir
y
where x is the measurement, exp {y} denotes e , where e - 2.7128. This
frequency distribution is bell-shaped, symmetrical with the mean as the
point of symmetry. Figure 3-1 is a sketch of the curve.
In order to apply this frequency distribution to a specific problem, it is
first necessary to decide that the distribution is an adequate represen-
tation of the data. This can be decided on the basis of data, if available,
or by intuition and the fact that so many measurements are adequately
approximated by such a frequency distribution. It is next necessary to
estimate the mean and standard deviation of the measurements. With these
values one can estimate the relative frequency of measurements in any
specified interval, say (a,b) , by obtaining the proportion of the total area
under the curve which falls between the lines x=a and x=b; see Figure 3-1.
These areas are tabulated in tables for areas under the normal
curve. For example, in Reference 27, one obtains the area to the left of b
and subtracts from this the area to the left of a. To enter the table, one
must compute standard normal variables by computing
b-p , a-y
- and -
o o
139
-------
L 68%J
95% H
>>x
99.7%
Figure 3-1: Sketch of Normal Frequency Distribution
and then, looking up the corresponding probability (area) that x is less than
b, a, respectively.,
Prob(a < x < b) = Prob (x < b) - Prob(x < a)
For example, if b = 25, a = 10, y = 15, a = 5, we have
P(10 < x < 25) = P(x < 25) - P(x < 10) .
Converting the values 25 and 10 to standard normal values u, one obtains
P(10 < x < 25) = 0.97725 - .1587 - 0.82
A few useful values are given below for convenient reference.
140
-------
Standard
Normal
Variable
Probability that x falls between
- u and + u
1.0 0.68
1.645 0.90
1.96 0.95
2.58 0.99
3.0 0.997
For example, the probability that a measurement falls between y + 20 (+ 3a)
is approximately 0.95 (0.997).
In some cases there is very definite evidence that the data are not
normally distributed. For example, the concentration of particles in
ambient air is very adequately approximated by the logarithmic normal
frequency distribution. That is, the logarithm of the concentration is
normally distributed. Thus, one cannot take the normal frequency distribu-
tion on complete faith without checking its adequacy. There are statis-
tical tests for making such an adequacy check. (Plotting the data on normal
probability graph paper is a simple visual check.)
There is one fundamental theorem which gives considerable justification for
assuming the normal frequency distribution. This is referred to as the
central limit theorem. It indicates that the average or mean of a large
number of measurements tends to be normally distributed even when an indivi-
dual measurement may not be very well approximated by this same distribution.
Further discussion of the implications of this theorem and of the normal
frequency function is given in many standard texts; e.g., see Ref 26.
The normal distribution has another important characteristic: a linear
combination of normally distributed variables is also normally distributed.
See the section on Means and Variances of Linear Combinations of Variables
for a more detailed discussion of how one can obtain or compute the mean
and variance of the linear combination. This result is used frequently
in explaining the overall variation of results. For example, the measure-
ments of the concentration of a control gas sample by several monitoring
stations (laboratories) might be considered as a sum of the variation amonf
laboratories and that within laboratories, say,
141
-------
where C . = j measured concentration of the sample by
the 1th lab,
\i = the overall (true) mean concentration,
i. = deviation of the mean concentration for
laboratory i from the overall mean,
w. = deviation of the j individual (within)
laboratory measurement from the lab mean.
Now if 9,. and w. are assumed to be normally distributed, C. . is also normally
J. 77
distributed with mean p and variance a + a , the sum of the variances of the
Si's and the w's. The A, might also be considered as a laboratory bias; the
above model assumes these biases have mean zero and standard deviation a..
Such a simple model may also be generalized to include operator effects,
day effects, instrument errors or variation, etc.
UNIFORM FREQUENCY DISTRIBUTION
The uniform distribution with the parameters a and b is defined by the
equation
f( . = fl/(b-a) for a < x < b
* ' \ 0 elsewhere
and its graph is shown in Figure 3-2. (Notation used throughout this docu-
ment to represent a uniform distribution with a lower limit a and an upper
limit of b is U(a to b).) All values of x from a to b are "equally likely"
in the sense that the probability that x lies in a narrow interval of width
Ax entirely contained in the interval from a to b is equal to Ax/(b-a),
regardless of the exact location of the interval.
The governing equations for a uniform distribution are:
(1) f(x) = l/(b-a)
(2) y = (a+b>/2, and
(3) a = (b
142
-------
f(x)
I I
I
b-o
0 a
Figure 3-2: Sketch of Uniform Frequency Distribution
143
-------
APPENDIX 4 ESTIMATION OF THE I^EAN AND VARIANCE OF VARIOUS
COMPARISONS OF INTEREST
MEAN AND VARIANCE OF LINEAR FUNCTIONS
Difference of Two Observations
In 'the development of standards for comparison of measured values with
audited values and 'of measured values with the predicted values given by a
calibration, it is necessary to estimate the variance of simple linear
combinations of observations. For example, in comparing a measured value
with an audited value such as in a data processing check, the variance of
the difference is
a2(X1 - X2) = a2 + a2,
2
where a. is the variance of the first measured value, and
2
a is the variance of the second measured value, the
audited value.
In practice these two variances may be assumed to be equal to the same
value, a^say, particularly if the measurements are made by two different
operators using the same instrument. Thus under this assumption,
a2(X1 - X2) = 2a2,
and if one obtains the estimated variance of the differences of n pairs of
2 2
measurements, the result would estimate 2a.. rather than a^, or
would estimate a-^. This is the reason for inserting a 2 in the denominator
of some of the expressions for s, the computed standard deviation, in the
field documents..
The mean of the difference of two observations is the difference of their
means, i.e., if u(X) denotes the mean or expected value of X, then
- y(X2) or PX - W2
144
-------
It is expected that p, would equal y. if both measurements are of the
same characteristic such as the final filter weight, unless an
operator is biased,, e.g., tends to read a scale too high for all
measurements. If v^ = V2 then p^ - X2) = 0 by assumption. This
assumption can be checked by a standard statistical test, called the t
test (Ref. 26).
Suppose that a comparison is being made of a measured value and an audited
value which is considered to be very precise, e.g., a case in which a very
precise instrument or "known" control sample (i.e., with a precisely
determined concentration) is being used in the audit check. In this case
the variance of the difference will almost be equivalent to the variance
2
of the measured value with the larger variance a.., say, because the addition
22
of a, to a will alter the total result very little. For example, if
22 2
a. =5, a_ = 1, the a. + a_ = 26 compared to a. = 25, assuming the measurements
V~22
a. + 0? - 5.2. In some
cases in the field documents it is assumed that one is comparing a measured
value with a relatively precise value, and the estimated variance desired for
computing the precision of the measurement process is that of the measured
value. Hence, the assumption is made that the variance of the difference
is essentially equal to the variance of the less precise measured value.
The mean difference will be zero unless some bias is present in the
measurement method, e.g., the interference from water vapor in the CO
measurement.
Mean of n Observations
The most frequent application of estimating the variance of a linear
combination is that of the mean of n observations. For n=2,
2
a
/Xl * X2\ a! + £i ai
\ 2 ^ ~ 4 4 2
'or in general
+ X, + ...+ X9\
2 2\
n /
a2/Xl + X2+'-^X2\__2^_a2
This is a well-known result of the standard error for the mean, i.e.,
o(X) = a/t/n .
145
-------
The mean of the average of n measurements is the overall mean, or
"population" mean, if there is no measurement bias, i.e., y(X) = p. This
application is used in the :field. documents in the case ojf,..the data processing
check, where averages of two differences are reported if required by the manager
2
Each difference has a variance of 2a , and hence the average of two
2 2
differences has the variance 20 /2 = a , the same variance as that of an
individual measurement. This fact is used in the data processing check in
the fie.ld documents.
. General Case
In general, if Y is a linear combination of n values X.., X , ..., X , such
as
Y = £ + AX, + A-X. + ...+ i X ,
o 11 2. i n n
then if the X's are independent or uncor related* the variance of Y is given by
(1) a2(Y) = £2a2 + A2a2 +. . .+ £2a2
22
and if a = a
(la) a2(Y) = a2 x
The mean of the linear combination is the same linear combination of the
means, i.e.,
(2)
if all p. = p, i = 1, 2, ..., n. If the X. are not uncorrelated but have
correlation coefficients p.. for X , X , the variance of Y becomes
2 22 22 22
(3) a (Y) = VJa. + £,a, +...+ JTo
11 f- f. n n
2V2p12CTla2 + 2£lSP13CTlCT3 +'"+ 2VlVn-l,n VlV
146
-------
If the a.'s are all equal, then
(3a) a2(Y) - 02(I^) + 2a2 Z^^ZA^ p
For example, suppose that several measurements are obtaine.d on the
3
concentration of suspended particles (yg/m ) at neighboring stations and
that it is known that they are correlated with p = 0.5. The variance of
the difference of two such measurements would be
2
- 2p
If a. = a , then for p = 0.5,
20
/ v \ n ^ -|A1 i
a (Xj^ - X2) - 2a - 2(7Jo
2
= a
2
and not 2a as it would be for uncorrelated measurements.
Comparison with Calibrated Value
Suppose that a calibration is made at 5 levels. A procedure for comparing
a single observation with a calibrated value given by a straight line
relationship, using tolerance intervals, is given in Ref. 28 and presented
briefly here.
Consider an example in which an analyzer is calibrated using a series of
five samples of "known" concentrations distributed over the range of
interest. Let y be the analyzer reading in volts and x the concentration
of the sample in yg/m . Assume that
y = 3 + B, (x. - x) + e,
that is, the observed analyzer reading is a linear function of x with unknown
coefficients to be estimated from the data and E is the deviation of_the
observed analyzer reading from the true value given by 3 + 3 (x. - x).
147
-------
In this example it is assumed that x is without error. The unknown
coefficients 8 and $ can be estimated by the method of least squares.
Let their estimates be b and b.., respectively, then
n 5
fi yi SL y±
b = = r = y
o n 5 3
and
b, =
1 5
Then the predicted mean value of y for given x, say Y, will be given by
Y±=bo+b1 (x. - x),
and y - Y. is the deviation of the predicted value from the observed value.
The b's were chosen so that the sum of squares of these deviations for the
n = 5 values would be a minimum. Now s, the estimated standard deviation of
the y's for a given x is given by
n 9
2 S ''I - V
S n- 2 °r
where n = 5. It has been implicitly assumed that the variance of the y's
for a given x is independent of x. This is not always the case and, in fact,
there is considerable evidence to indicate otherwise in the analysis of
pollutant data. In order to consider the case of unequal variances, it is
suggested that one refer to Ref. 26, pages 551-7. In this example we will
assume constant variance for simplicity of discussion. The modification of
the techniques for the heterogeneous variance case is reasonably straight-
forward.
Now consider the comparison of another determination of y for a given x,
say x . For example, suppose that a control sample is given to the operator
to check his instrument, calibration, etc. , and that he estimates the concen-
tration of the sample by the analyzer reading and the calibration curve he
is currently using. Let his reading be y . Determine if the predicted
concentration, x., given by the calibration curve and the known concentration
are consistent. This is equivalent to requiring that the observation y
does not deviate too far from the Y given by the calibration curve.
See Figure 4-1 below. The variance of a single value y for a given x,
148
-------
pj y
o
S Y
N O
Calibration
Curve
Deviation of Measured From
| Known Value
X X,
o 1
Sample Concentration
Figure 4-1:" Comparison of Individual Analyzer Reading
With Value From Calibration Curve
say x , is given by a (Y + e), i.e.,
a (b + b, (x - x) + e)
o 1 o
(e denotes the "error" associated with the individual measurement). This
formula is appropriate if one is comparing a single determination of y
with a single "calibrated" value. However, if one wishes to do this
repeatedly, a tolerance interval must be obtained (Ref. 28).
For information concerning the fact that the x's are not without error,
see Ref. 22 for discussion of this assumption and some appropriate
techniques.
149
-------
An example computation is given below for illustrating the computations
suggested ab'ove.
(concentration in ppm) (analyzer reading in volts)
10 .039
20 .086
30 .140
50 .254
75 .369
Computations
x = 37 y = 0.1776
bQ = 0.1776, b^ = 0.0052
Y = 0.1776 + 0.0052 (x - 37)
s2 = 0.000042
s = 0.0065
2 2 / 1 (x~ ~ *>
s (yn) = sMl +i +
'o' ° I " ' n 5
i-1
_
Now n=5, and (x. - x) = 2010, and let x be 30, then
1=1
s2(y) =0.000042 I +
= 0.0001 .
s(y ) = 0.01
150
-------
A 90% confidence interval estimate for the mean of the y's for x = 30
is given by
Yo±2.353s(yo)
where Y = 0.1776 + 0.0052 (30-37) = 0.1412 and 2.353 is the t-value
o
(t-distribution) for 90% confidence and for 3 degrees of freedom (3 = n-2
in this example) associated with the estimate of s. This procedure is
sufficient if we are interested in making just one such statement using
one given calibration. However, if we wish to repeat this many times, a
tolerance interval is required, as indicated above. The computation of
the tolerance limits requires a value of k such that
Y + ks
o
includes, say, 95% of the observations with a desired confidence.
Following the procedure in Ref. 28 yields k * 5 (it ranges from 4.89 at
x=30.to 5.74 at x=75) to include 95% of the observations with 90% confidence.
These two intervals are wide due to the small number of observations (5) and
the .corresponding 'small number of degrees of freedom. If five calibrations
were made and .averaged, then n=25, the degrees of freedom would be n-2=23,
and the intervals corresponding to the above would .be
Confidence Interval: Y + 1.71 s(y )
o o
and
Tolerance Interval: Y + 2.49 s, at x = 30,
o
respectively, where s = 0.006
s2(yQ) = 0.000042 |l + -|j + y (2010)) = 0.000044
or
s(y ) = 0.0066.
Thus, not only s(y ) has decreased, but the t and k values have decreased.
Some of the pertinent computations are given in Table 4-1. The resulting
interval for N = 25 is about one-half as wide as that for N = 5.
Figure 4-2 graphically illustrates a calibration curve with confidence
limits for a single analyzer reading and tolerance limits for 95% of all
analyzer readings.
151
-------
Table 4-1: COMPUTATION OF CONFIDENCE AND TOLERANCE LIMITS
FOR THE EXAMPLE
N = 5 Observations, Single Calibrations
1 1 (x - 37)2
k H» 5 2010
10 0.563
20 0.344
30 0.224
50 0.284
75 0.918
N = 25 Observations, Five
1 1 , (x - 37)2
x N' 25 10'°5°
10 0.133
20 0.069
30 0.045
50 0.057
75 0.184
k
5.41
5.10
4.89
5.00
5.74
t
2.353
2.353
2.353
2.353
2.353
ks
0.035
0.033
0.032
.0.033
0.037
s(y ) = (1 + )s
° N'
.0100
.0087
.0080
.0083
.0125
Y
0.037
0.089
0.141
0.245
0.375
ts(yo)
.024
.021
.019
.020 .
.029
Y - ks
.002
.056
.109
.212
.338
Y - ts(yQ)
.013
.068
.122
..225
.346
Y + ks
.072
.122
.173
.278
.412
Y + ts(yQ)
.061
.110
.160
.265
.404
Calibrations
k
2.57
2.52
2.49
2.50
2.64
t
1.71
1.71
1.71
1.71
1.71
ks*
0.017
0.016
' 0.016
0.016
0.017
s(y ) = (1 + )s
0 N
.0072
.0069
.0068
.0069
.0077
Y*
0.037
0.089
0.141
0.245
0.375
ts(yo)
.012
.012
.012
.012
.013
Y - ks
0.020
0.073
0.125
0.229
0.358
Y - ts(yQ)
.025
.077
.129
.233
.362
Y + ks
0.054
0.105
0.157
0.261
0.392
Y + ts(yQ)
.049
.101
.153
.257
.388
The same values of Y and s were used in both computations since there were no.
actual data for five calibrations.
152
-------
0.4
GO
0.3
u no
Q: O.2
a:
LU
<
iz.
O.I
0
Y = 0.1776+0.0052 (X-37)
Y±ks (TOLERANCE LIMITS)
FOR 95% OF ALL
ANALYZER READINGS
Y±ks(yo) (CONFIDENCE LIMITS)
FOR A SINGLE
ANALYZER READING
0 10 20 30 40 50 60 70 80
CONCENTRATION IN ppm
Figure 4-2: Confidence and Tolerance Limits for the Example.
-------
MEAN AND VARIANCE OF NONLINEAR FUNCTIONS
In the previous examples, the variable of interest was expressed as a linear
function of several variables with known coefficients. However, it is
often necessary to estimate the mean and variance of a nonlinear
combination such as that for the concentration of suspended particles in
air as given below,
SP =
(wf -
x 10
6
Q.+Q
x T
where SP denotes the concentration in yg/m ,
w_ = final weight of filter in grams,
w = initial (tare) weight of filter in grams,
Q = initial flow rate in m /min,
3
Q, = final flow rate in m /min, and
T = sampling time in min.
For relatively small variations (errors of measurement) in the variables
wf, w , ..., T in the above expression, it is usually sufficient to obtain
an approximate linear expression of SP as a function of each of the variables
and then proceed to use the formulas given previously. To obtain a linear
approximation, differentiate SP with respect to each of the variables and
expand the function in a Taylor Series as follows:
SP « SP
mean
, 3(SP)
9Wf
mean
mean
dT
mean
where SP
denotes the value of SP where each variable is
mean replaced by its mean or expected value,
3(SP)
ax
denotes the value of the partial derivative of
mean SP with respect to variable "X" and evaluated
for each variable at its mean value, and
dx denotes the deviation of the variable "X" from
its mean.
154
-------
The variable X may be any one of wf, w., ..., T. Using the results of
equations (1) and (2) above, the approximate mean and variance of SP is
obtained by
(5)
Mean (SP) = SP + 0 ,
mean '
(6) Variance
mean/
2 ,/3(SP)
°«f +PWi
mee
w.
3(SP)
meanJ
provided the deviations dw,, dw , ..., dT are uncorrelated or independent.
If they are not uncorrelated, one would use equation (3) and also have
estimates of the correlations. An estimate of the variance (SP) is
obtained by substituting estimates of the a's throughout equation (6).
Another useful approach to obtaining the estimated variance of a function
such as that for SP is to first take the logarithm of SP and then differ-
entiate the resulting expression. In this case we obtain
SP
d(wf * wj
wf -w±
d((?
V
The variance of an expression
dX
X
X 2
= CV (X), i.e., the square of the coefficient of
is given by
Mean'(X)
variation. Hence, the variance of SP is given by
where
(7) CV2(SP) = CV2(Wf - W±) + CV2(Qi + Qf) + CV2(T)
(8) o2(SP) * CV2('SP) Mean2(SP).
155
-------
This approach to obtaining an estimate of the variance of SP is particularly
useful if the CV's are expected to be constant (i.e., the standard deviation
is some fixed percentage of the mean) and/or when the expression is a
product (or quotient) of several variables.
The differentiation and algebraic calculations become very tedious for
several variables and complex expressions. A computer program has been
written not only to do the above computations using the first approach
(the computer uses a numerically obtained derivative), but also to make
certain checks on the adequacy of the approximation. Another computer-
program, in parallel with this one, simulates the measurement process by
using assumed distributions of the variables in the expression and
empirically obtains a distribution of SP. This approach was used in
studying the distribution of SP as a function of several variables. The
expression given above for SP may be expanded to include variables other
than just those given. For example, the final weight of the filter might
be expressed as a function of the pH of the filter surface and the weight
of the collected particles. The concentration of the particles will vary
with time of day, flow rate, etc. Hence, the expression can be made more
specific and be useful provided the appropriate data are available to perform
the analysis. These computer programs are fully described in Appendices 1 and 2.
An example computation is given below for illustrating the computational
approach with equations (4), (5) and (6).
Parameter
Assumed
, Means
4031.2 mg
3700 mg
1.20 m3/min
1.10 m3/min
24 x 60 = 1440 min
,3
Assumed
Standard Deviations
0.7 mg
0. 7 ing
0.03 m /min
o
0.03 m /min
7 min
SP
331.2 x 1Q-
+1-1)
x 1440
200 yg/m
Computation of Derivatives
3(SP)
10"
mean
= .604 = -
9(SP)
3w.
mean
mean
156
-------
8(SP)
3Q,
3(SP)
mean
(Wf - w±)
= - 87.0
mean
t(Q± + Qf)
2 x 10 (wf -w±)
3T
- .139 .
(Q± + Qf)T
Thus,
SP = 200 + 0.604 dwf - 0.604 dw± - 87 dQ± - 87 dQf
-0.139 dT.
Hence,
y(SP) = 200 yg/mJ,
a2(SP) = (0.604)2 (0.7)2 + (0.604)2 (0.7)2 + (87)2 (0.03)2
(87)2 (0.03)2 + (.139)2 72,
o(SP) = 3.86 ng/m .
Using formulas (7) and (8)
CV2(SP) = (0.003)2 +.(0.0183)2 + (0.005)2
0.000368
CV(SP) = 0.0192
o(SP) = CV(SP) x y(SP) = (.0192)(200) = 3.84 yg/mj
which is the same as the above except for roundoff errors.
157
-------
APPENDIX 5 CONTROL CHARTS FOR USE IN IITOVING OJALITY OF
FEASUREMT PROCESS
There are several potential applications of quality control chart procedures
which can be applied to aid in detecting deviations in the measurement
process from expected values. For example, it may be desired to use a
control chart for zero and span calibration checks. When dealing with
measurements, it is customary to exercise control over the average quality
of a process as well as its variability. Control over the average quality
is accomplished by plotting the means of periodic samples on a so-called
control chart for means, or an x chart. Variability is controlled by
plotting the sample ranges or standard deviations on an R chart or a a chart,
respectively, depending on which statistic is used to estimate the population
standard deviation. A control chart consists of a central line corresponding
to the average quality at which the process is to perform, and lines corres-
ponding to the upper and lower control limits. These limits are chosen so
that values falling between them can be attributed to chance, while values
falling outside them are interpreted as indicating a lack of control.
Hence, if a zero or span check point deviates from the mean value by more
than expected, action should be taken to determine and.correct the cause.
The control limits can be established from prior knowledge of the process,
level of error that is acceptable, or by collecting data over a period of
time and letting the process establish its own limits as to what can be met
in actual practice. Some useful quality control charts techniques are
discussed briefly below.
STANDARD CONTROL CHART
No attempt will be made to give the detailed background concerning the
charting techniques. This information is available in many textbooks (Refs.
22, 26, 29). Also see Refs. 25, 30 for discussion of specific charts and
comparisons of their efficiencies. The final results are given here
without background material.
Let the notation be as follows:
n = sample size,
o = standard deviation of a single measurement (in the applications
to .pollutant measurements the standard deviation is usually the
combination of among-day effects and within-day effectse.g.,
see analyses of NO and 0- data in the respective sections of
this report),
a- = = standard deviation (error) of the average of n measurements,
X t^rT
y = mean or expected value of the measurements (central line of
the control chart),
k = multiple of a- used in obtaining the limits of the control chart
for the average of n measurements,
158 '
-------
R = range of the sample of n measurements, the largest value less
the smallest value,
d = multiple of 0 to be used in the construction of the control
chart for ranges,
UCL = upper control line,
LCL = lower control line, and
R = mean value of the range R of k samples of size n drawn from a
normal population.
Using the above definitions and information, control charts for both the
mean and ranges can be readily obtained. These two charts are illustrated
below in Figure 5-1 and 5-2. The appropriate factors for construction of the
charts are given in Table 5-1 below.
y+ka-
V- ka-
UCL
Central Line
123456
Figure 5-1: Control Chart for Means
LCL
Sample Number
d a
u
(R/a)a
123456
Figure 5-2: Control Chart for Ranges
UCL
Central Line
LCL
Sample Number
159
-------
To illustrate the construction of a control chart for ranges as shown in
Figure 5-2_, assume a sample size of 7 (i.e., n=7). From Table 5-1 the
value of (R/o) for n=7 is 2.70, and 2.70a becomes the value of the central
line of the control chart. Obtain values for d and d for computing UCL
and LCL, respectively, by specifying the upper and lower percentile to be
used. If the 95 percentile is specified for the UCL, Table 5-1 shows
that d has a value of 4.17 for a sample size of 7. The UCL is
d a
u
4.170.
Using 05 as the specified percentile for the LCL, Table 5-1 gives a value
of 1.44 for d , or 1.44a for the LCL.
Table 5-1: TABLE OF FACTORS FOR CONSTRUCTING CONTROL CHARTS
Sample size (n) R/O
2 1.13
3 1.69
4 2.06
7 2.70
Values of
d for Selected Percentiles
d d
u £
95
2.77 3
3.31 4
3.63 4
4.17 4
Values of
k( two- sided)
k(one-sided)
99 05
01
.64 .09 .02
.12 .43 .19
.40 .76 .43
.88 1.44 1.05
k for Selected Percentiles
95
1.96 ( 2)
1.645
99 99.7
2.58 3.00
2.33 2.75
If for a given sample the mean x exceeds the upper control limit and/or if
the range R exceeds the control limit da, the inference is made that the
magnitude of the measurements and/or their dispersion, respectively, has
increased. The risk of an incorrect inference that the measurement process
is not in control when in fact it is stable or in control is controlled by
the risk level selected according to Table 5-1.
There is also the risk of inferring that the process is still in control
when in fact its level of dispersion has changed. This risk can be
controlled by increasing the sample size or frequency of sampling and can
be calculated, see e.g., Ref. 26.
160
-------
RUN LENGTH AND RUN-SUM TESTS
There are frequent applications in the literature which suggest supple-
menting the standard quality control chart with either a run test or a
run-sum test. The run test simply counts the number of consecutive values
exceeding, e.g., the mean level. In the case of the chart for the mean, a run
of length r would occur when there are r consecutive measurements exceeding
the mean level, p. If r = 7, then it is usually inferred that the mean
has changed. Note that a run of seven, either below or above the mean,
should occur with probability 2 * (1/2) = 1/64 if there is, in fact, no
change.
Another test which is suggested is a run-sum test. The standard quality
control chart for the mean is subdivided into 8 zones, say; and an integral
score is assigned to each zone weighted according the distance of the zone
from the mean. See Figure 5-3 below illustrating the assignment of a score.
Score
u+ 3o-
2a-
y- a-
u- 2a-
p- 3a-
+ 0
-0
-1
-2
-3
Figure 5-3: Subdivision of Control Chart Into Zones and Assigned Scores.
161
-------
If a run of positive scores occurs and adds to 5, say, then it is inferred
that a positive shift in the mean measurement has occurred. Similarly for
a negative run, hence the reason for the 0 and 0 values. A run is
interrupted once a sign change occurs, and the score returns to 0. The
following figure illustrates the use of such a chart.
3a-
M- 30-
123456789 10
Sample Number
Run-Sum Score +0-0+0+011235 8
Run-Length 111234567 8
Figure 5-4: Standard Control Chart Comparing Techniques
It is observed in Figure 5-4 that the run-length and run-sum tests indicate
out of control at sample number 9, and the standard chart so indicates at
sample number 10. These supplementary tests are easy to maintain and aid
in detecting out of control when there are trends toward the limits without
exceeding them. Reference 3'0 compares these techniques of quality control
for abrupt changes in the process mean, \i. It is our desire to detect
gradual changes which may result from the drift in the data points caused
by instrument drift or component degradation. It is expected that the
general comparisons given in Reference 30 would still be applicable.
One useful application of these quality control techniques would be in
zero and span checks. For example, an operational rule might be to not
make adjustments in the readings of concentrations of 0- from the strip-
chart unless the zero and/or span calibration points fall outside the
range or limits given by a quality control chart for these readings.
If the zero or span checkpoints do fall outside the limits, one would take
162
-------
action to determine if some degradation has occurred and/or make an
appropriate adjustment in the calibration, e.g., complete the multipoint
calibration and use the new calibration. This approach would avoid the
result of "over correcting" the measurement process, the latter would
yield resulting concentrations which would be more variable than the
original uncorrected values. For example, the extreme case of making
a daily adjustment in the mean based on measurement of a standard or control
sample, where the standard deviation of the measurement of the standard is
the same as that for an unknown sample, would result in a variance twice
as large for the predicted values for the calculated concentrations of the
unknown samples. To see that this is the case, consider the following simple
example. Let the measured concentration of the unknown sample on the ith
day be y. = y + e. and that for the standard sample
x. = y + 6. .
is i
Suppose the concentration of the unknown sample is reported as y. - x.;
i.e., it is corrected for the standard sample. Now y. has variance
22 1
a , and x. has variance ag. Assuming these to be independent, we have
a2(y. -x.)=a2+a2=2a2, if a2 = a2 - a2 .
i i e o o .e
If y. is corrected using an average of n, x. values, e.g., for the last
1 - 2 a2 X
n days, then y. - x has variance a + under the assumption that
r\ f\ * *+
More complex considerations can be treated as suggested in Reference 25.
163
-------
TECHNICAL REPORT DATA
(Please read Instructions on the reverse before completing)
1. REPORT NO.
EPA-R4-73-028-6
3. RECIPIENT'S ACCESSION>NO.
4. TITLE AND SUBTITLE
Guidelines for Development of a Quality Assurance
Program. Measuring Pollutants for Which National Am-
bient Air Quality Standards Have Been Promulgated.
5. REPORT DATE
August 1973
6. PERFORMING ORGANIZATION CODE
7.AUTHORIS)(Final Report)
Franklin Smith and A. Carl Nelson, Jr.
8. PERFORMING ORGANIZATION REPORT NO,
9. PERFORMING ORGANIZATION NAME AND ADDRESS
Research Triangle Institute
Research Triangle Park, N. C.
27709
10. PROGRAM ELEMENT NO.
1HA327
11. CONTRACT/GRANT NO.
68-02-0598
12. SPONSORING AGENCY NAME AND ADDRESS
Office of Research and Development
U.S. Environmental Protection Agency
Washington, D. C. 20460
13. TYPE OF REPORT AND PERIOD COVERED
Final
14. SPONSORING AGENCY CODE
15. SUPPLEMENTARY NOTES
16. ABSTRACT
Five method specific field usable quidelines for quality assurance for ambient
air pollutants were produced under the contract and published separately. This
document is not a field document. It supplies some background information used
in the development of the five documents.
KEY WORDS AND DOCUMENT ANALYSIS
DESCRIPTORS
b.IDENTIFIERS/OPEN ENDED TERMS C. COSATI Field/GlOUp
Quality Assurance
Quality Control
Air Pollution
13H
14D
13B
8. DISTRIBUTION STATEMENT
Unlimited
19. SECURITY CLASS (ThisReport)'
21. NO. OF PAGES
174
20. SECURITY CLASS.
22. PRICE '
EPA Form 2220-1 (9-73)
164
------- |