United States
                    Environmental Protection
                    Agency
 Environmental Sciences Research
 Laboratory
 Research Triangle Park NC 27711
                   Research and Development
EPA-600/S3-83-023c June 1983
&EPA          Project  Summary

                    Houston  Oxidant  Modeling
                    Study—1978:
                    Volume  III.  Characterization
                    of  Data  Quality
                    J.R. Martinez
                     During the period 15 September
                   through 12 October 1978, the U. S.
                   Environmental Protection Agency (EPA)
                   conducted a special program that mon-
                   itored air quality and meteorology in
                   the Houston, Texas  area. The objec-
                   tives of the program were to obtain a
                   comprehensive data  base suitable for
                   use with photochemical air-quality sim-
                   ulation models and  to provide a de-
                   tailed body of data that can be used to
                   investigate  Houston's air quality in
                   general and photochemical oxidants
                   (HOMS Study) and  aerosols (HACS
                   Study)  in particular.
                     The objectives of this contract were
                   (a) to evaluate the suitability of the
                   1978  Houston data base for  photo-
                   chemical modeling application, (b) to
                   analyze spatial and temporal patterns
                   of pollutant concentrations,  (c) to
                   archive data in a manner suitable for
                   use with air quality simulation models,
                   (d) to  analyze and  characterize the
                   quality of the gaseous pollutant mea-
                   surements, and (e) to use the data to
                   evaluate the performance of the EKMA
                   photochemical model. Results from
                   the data evaluation and archiving work
                   are reported in a three-volume report.
                   Results from the EKMA evaluation
                   work are reported in a separate report
                     This report. Volume III of the three-
                   volume  report, covers the effort to
                   analyze and characterize the quality of
                   the gaseous pollutant measurements
                   obtained in the 1978 HOMS Study.
                   The analysis is based on data obtained
                   in a  number of  field audits of the
                   instruments used in  the HOMS. The
                   audits were performed independently
                   by the EPA and by the Research Triangle
Institute under the sponsorship of the
EPA. The audit data were analyzed
statistically to derive overall bounds
for measurement accuracy and to de-
fine the accuracy and precision of indi-
vidual instruments.

   This Project Summary was developed
by EPA's Environmental Sciences Re-
search Laboratory, Research Triangle
Park. NC, to announce key findings of
the research project that is fully docu-
mented in a separate report of the
same title (see Project Report ordering
information at back).

Introduction
  This is the  third volume  of a three-
volume report  describing the results of a
study of the quality of the pollutant mea-
surements made during the 1978 Houston
Oxidant Modeling Study (HOMS). In the
course of the  HOMS, field audits were
conducted to check the performance of
the instruments that measured gaseous
pollutants. In this study, the audit data will
be analyzed to determine the accuracy and
precision of the HOMS air quality data.
  The  HOMS  was conducted from 1 5
September through 12 October 1978. Its
primary objective was to obtain a data base
suitable for use with photochemical air
quality simulation models. The description
of the data base and the analysis of the t.ir
quality and meteorological data  are re-
ported in Volumes I and II.

Audit Procedures
  Concern was with the instruments that
monitored the  following gaseous  pollu-
tants:
  • Ozone (03)
  • Nitric oxide (NO)

-------
     Nitrogen dioxide (N02)
     Oxides of nitrogen (NOX)
     Carbon monoxide (CO)
     Total hydrocarbons (THC)
     Methane (CH4)
     Sulfur dioxide (S02)
Field audits  of these  instruments were
performed independently by two organiza-
tions: the U.  S. Environmental Protection
Agency (EPA) and the Research Triangle
Institute (RTI). RTI  performed the audits
under  the sponsorship of the Quality
Assurance Branch of EPA.
  The audit procedure consisted of feeding
a known concentration (reference concen-
tration) of the gaseous pollutant to the
instrument and recording the instrument's
response.  The reference concentrations
usually included a zero level  and several
nonzero values.
  Reference  ozone concentrations were
produced using an  ultraviolet ozone gen-
erator. The EPA determined the reference
ozone  level  using IMBKI,  whereas  RTI
established the reference concentration
using gas phase titration.
  Test  atmospheres of NO, NOX, and N02
were generated  by both EPA and  RTI
using standard-gas cylinder dilution  and
gas phase titration. The standard  gases
were referenced to NBS-SRM 1683 (RTI)
and to NBS-SRM 1684 (both  EPA  and
RTI). Standard-gas  cylinders were also
used to generate reference concentrations
of CO,  CH4, and  S02.  EPA used CO and
CH4 cylinders from Scott Environmental.

Statistical Analysis of Audit
Data
  The  statistical  analysis of quality as-
surance audit  data  from the HOMS in-
cludes: (1) obtaining frequency distribu-
tions for percent relative error (percent
relative error = 100 x [measured - ref-
erence] / reference); and (2) performing
regression analyses of measured and ref-
erence concentrations  for individual in-
struments. The analysis was performed
separately for the EPA  and  RTI data
because of the differences in their audit
procedures.
  The  audit  data have been analyzed in
two ways. One type of analysis  examines
the total variation of measurement errors
for each variable (e.g., ozone) by aggre-
gating  the data for all the instruments. This
allows us to  investigate the error  dis-
tribution and to estimate overall bounds of
measurement error. Although these error
bounds do  not  necessarily apply to  a
specific instrument at a particular time,
they quantify the  size and frequency of the
errors  likely  to be found in the totality of
measurements.   Aggregating the audit
data also provides an easy way to identify
large errors and possible anomalies.
  The  second analysis  considers each
instrument individually, using regression
techniques to define the relationship be-
tween measured and reference concentra-
tions.

Conclusions and
Recommendations
  The premise of an audit program is that
random checks of an  instrument can pro-
vide a good indication of the performance
of that instrument. During the HOMS,
most  instruments were audited at least
once but not more than twice by either
EPA or RTI or both. Such an audit rate for
individual  instruments is too low to pro-
vide a data base  from which to draw
statistically valid conclusions regarding
the overall performance  of a particular
instrument
  Because the audits were relatively few
in number, the data were considered in
two ways:  by pooling all the audits for each
type of instrument and by treating each
instrument individually. Pooling the data
enlarges the data base and allows general
conclusions to be drawn about the overall
quality of the data. Aggregating the audit
data also  helps to identify  performance
anomalies. By its nature, the results of the
analysis of pooled  audit data cannot  be
applied to specific instruments Thus, we
can make statements such as, "The audits
indicate that about 70% of the measure-
ments of  pollutant X were accurate to
within  20%."  But we cannot state that
70% of the measurements of a specific
instrument were accurate to within 2096.
  Analyzing the audit performance of in-
dividual instruments can only serve either
to  assuage or raise  doubts  about the
quality of the measurement of a given
instrument On the one hand, a good audit
performance may cause us to feel con-
fident about the  instrument, recognizing,
however,  that this does not necessarily
imply that all the measurements of that
instrument are equally good. On the other
hand,  a poor  audit  performance  raises
doubts about the instrument, but again
does not necessarily imply that other mea-
surements are equally poor. Hence,  one
should consult historical calibration and
repair records to obtain a better indication
of whether the poor (or good) audit repre-
sents a persistent or a one-time condition.
The results of this study provide guidance
about the potential performance of indi-
vidual instruments, but because the anal-
ysis is  restricted to  the audit data,  this
study cannot separate persistent cone
tions from one-time events. Each audit is
one-time event and only many audits c<
fill in the picture.
  Table 1  is a summary  of the  oven
measurement  accuracy for the  varioi
pollutants,  based  on the analysis of tt
pooled audit data. The table shows <
estimated range for the percentage of tf
measurements that are accurate to with
±10%, ±20%,  and ±30%. For exampl
for ozone Table 1 shows  that betwee
50% and 60% of the data were accurate)
within±20%, and85% to95% haderroi
bounded by ±30%. Table 1 indicates thi
for ozone, NO, NOX, N02, and S02 at leas
70% of the data were accurate to withi
±20%. Methane has the largest fraction <
the data accurate to within ±20%. Carbo
monoxide and THC can be considered 1
be the least accurate because they hav
the smallest fraction of the data in th
±20% error band The THC measurement
were  found to be  subject to large  ir
accuracies, which explains the relative!
low percentage of the  data within th
±30%  error  band.  There is  cause fc
concern  about  the quality of the THi
measurements  and,  hence, the  NMHl
measurements  in  the HOMS.
  The  regression analysis of  individuj
instruments showed that almost all dem
onstrated exceptionally good linearity c
response. The precision of the measure
ments, as indicated by the standard erro
of the regressions, was also good in nearh
all cases.  The accuracy, as reflected in thi
slope of the regression line, was variable
Specifically, the regression analyses  in
dicated potential accuracy problems at th<
following monitoring sites:
  • 03:  Aldine, Pasadena, Channelview
         Seabrooke
     NO:  Parkhurst, Fuqua, Pearland
     NOX:  Mae Drive, Parkhurst, Fuqua
     NO2: Fuqua
     CO:  Parkhurst Pearland, Jackrabbit
          EPA Mobile Lab.
     CH4:  Parkhurst
     THC:  Crawford, Clinton, Parkhurst
     S02:  Aldine

All the instruments named above hac
regression lines whose slope was greate
than 1.25 or smaller than 0.75 in at leas
one of the audits. The history of these
instruments should be investigated tc
establish whether or not the bias was i
recurring phenomenon. If the bias recurred
then it will be  necessary  to define cor-
rection factors to be applied to the routine
ly collected data.                     ,
   Future  monitoring programs   should
continue to include field  audits of th«

-------
Table 7.    Characterization of Overall Measurement Accuracy
                                      Percent of Measurements Accurate to
Variable
Ozone
Nitric oxide
Nitrogen oxides
Nitrogen dioxide
Carbon monoxide
Methane
Total hydrocarbons
Sulfur dioxide
±10%
50-60
50-70
60-70
45-55
35-65
65-80
20-45
30-40
±20 %
70-80
70-80
70-85
70-75
60-70
85-90
50-60
70-80
±30%
85-95
80-95
85-95
80-90
80-95
90-95
55-60
80-90
instruments. However, the procedures fol-
lowed should be changed to facilitate the
interpretation of the audit data  In par-
ticular, the  audit procedure should  be
designed to allow the analyst to draw
statistically sound general inferences a-
bout the accuracy of individual instruments
from the audit data. It was not possible to
do so in this study because the number of
audits was  too small. Hence, it is recom-
mended that future  audit programs  in-
clude in their design a determination of the
number of audits required to describe the
performance of individual  instruments
over the life of the monitoring program.
This is a problem in statistical experiment
design that is beyond the scope of our
investigation.

  Although it is desirable to have more
than one  agency conduct  field  audits,
differences between  EPA and RTI audit
protocols produced some inconsistent re-
sults (e.g.,  EPA ozone audits showed a
tendency toward underestimation,  and
RTI audits showed the opposite). Thus, in
this case procedural differences obscured
the effects that the audits were intended
to uncover. This  situation should  be
avoided in future programs by having all
parties adhere to a standard audit protocol.
In this context, the audit procedures used
in the present program resulted in several
cases of same-day audits  of the same
instruments by both  EPA and RTI. Such
overlap is desirable  to check the con-
sistency, or lack thereof, of the results
obtained by the two auditing  agencies.
However, same-day  audits of the same
instrument essentially constitute a single
audit, which acts to reduce the total num-
ber of tests. Future programs should en-
sure that overlapping audits do not lower
the total number of checks performed.
  J. R. Martinez is with SRI International, Menlo Park. CA 94025.
  B. Dimitriades is the EPA Project Officer (see below).
  The complete report, entitled "Houston Oxidant Modeling Study—1978: Volume
    III. Characterization of Data Quality," (Order No. PB 83-194 217; Cost: $11.50,
    subject to change) will  be available   only from:
          National Technical Information Service
          5285 Port Royal Road
          Springfield, VA22161
          Telephone: 703-487-4650
  The EPA Project Officer can be contacted at:
          Environmental Sciences Research Laboratory
          U.S. Environmental Protection Agency
          Research Triangle Park, NC 27711

-------
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45268
Postage and
Fees Paid
Environmental
Protection
Agency
EPA 335
Official Business
Penalty for Private Use $300
     PS   0000329
     U S ENVIR PROTECTION  AGENCY
     REGION 5  LIBRARY
     330 S  DEARBORN  STREET
     CHICAGO  IL 60604

-------