United States
Environmental Protection
Agency
Environmental Sciences Research
Laboratory
Research Triangle Park NC 27711
Research and Development
EPA-600/S3-83-023c June 1983
&EPA Project Summary
Houston Oxidant Modeling
Study—1978:
Volume III. Characterization
of Data Quality
J.R. Martinez
During the period 15 September
through 12 October 1978, the U. S.
Environmental Protection Agency (EPA)
conducted a special program that mon-
itored air quality and meteorology in
the Houston, Texas area. The objec-
tives of the program were to obtain a
comprehensive data base suitable for
use with photochemical air-quality sim-
ulation models and to provide a de-
tailed body of data that can be used to
investigate Houston's air quality in
general and photochemical oxidants
(HOMS Study) and aerosols (HACS
Study) in particular.
The objectives of this contract were
(a) to evaluate the suitability of the
1978 Houston data base for photo-
chemical modeling application, (b) to
analyze spatial and temporal patterns
of pollutant concentrations, (c) to
archive data in a manner suitable for
use with air quality simulation models,
(d) to analyze and characterize the
quality of the gaseous pollutant mea-
surements, and (e) to use the data to
evaluate the performance of the EKMA
photochemical model. Results from
the data evaluation and archiving work
are reported in a three-volume report.
Results from the EKMA evaluation
work are reported in a separate report
This report. Volume III of the three-
volume report, covers the effort to
analyze and characterize the quality of
the gaseous pollutant measurements
obtained in the 1978 HOMS Study.
The analysis is based on data obtained
in a number of field audits of the
instruments used in the HOMS. The
audits were performed independently
by the EPA and by the Research Triangle
Institute under the sponsorship of the
EPA. The audit data were analyzed
statistically to derive overall bounds
for measurement accuracy and to de-
fine the accuracy and precision of indi-
vidual instruments.
This Project Summary was developed
by EPA's Environmental Sciences Re-
search Laboratory, Research Triangle
Park. NC, to announce key findings of
the research project that is fully docu-
mented in a separate report of the
same title (see Project Report ordering
information at back).
Introduction
This is the third volume of a three-
volume report describing the results of a
study of the quality of the pollutant mea-
surements made during the 1978 Houston
Oxidant Modeling Study (HOMS). In the
course of the HOMS, field audits were
conducted to check the performance of
the instruments that measured gaseous
pollutants. In this study, the audit data will
be analyzed to determine the accuracy and
precision of the HOMS air quality data.
The HOMS was conducted from 1 5
September through 12 October 1978. Its
primary objective was to obtain a data base
suitable for use with photochemical air
quality simulation models. The description
of the data base and the analysis of the t.ir
quality and meteorological data are re-
ported in Volumes I and II.
Audit Procedures
Concern was with the instruments that
monitored the following gaseous pollu-
tants:
• Ozone (03)
• Nitric oxide (NO)
-------
Nitrogen dioxide (N02)
Oxides of nitrogen (NOX)
Carbon monoxide (CO)
Total hydrocarbons (THC)
Methane (CH4)
Sulfur dioxide (S02)
Field audits of these instruments were
performed independently by two organiza-
tions: the U. S. Environmental Protection
Agency (EPA) and the Research Triangle
Institute (RTI). RTI performed the audits
under the sponsorship of the Quality
Assurance Branch of EPA.
The audit procedure consisted of feeding
a known concentration (reference concen-
tration) of the gaseous pollutant to the
instrument and recording the instrument's
response. The reference concentrations
usually included a zero level and several
nonzero values.
Reference ozone concentrations were
produced using an ultraviolet ozone gen-
erator. The EPA determined the reference
ozone level using IMBKI, whereas RTI
established the reference concentration
using gas phase titration.
Test atmospheres of NO, NOX, and N02
were generated by both EPA and RTI
using standard-gas cylinder dilution and
gas phase titration. The standard gases
were referenced to NBS-SRM 1683 (RTI)
and to NBS-SRM 1684 (both EPA and
RTI). Standard-gas cylinders were also
used to generate reference concentrations
of CO, CH4, and S02. EPA used CO and
CH4 cylinders from Scott Environmental.
Statistical Analysis of Audit
Data
The statistical analysis of quality as-
surance audit data from the HOMS in-
cludes: (1) obtaining frequency distribu-
tions for percent relative error (percent
relative error = 100 x [measured - ref-
erence] / reference); and (2) performing
regression analyses of measured and ref-
erence concentrations for individual in-
struments. The analysis was performed
separately for the EPA and RTI data
because of the differences in their audit
procedures.
The audit data have been analyzed in
two ways. One type of analysis examines
the total variation of measurement errors
for each variable (e.g., ozone) by aggre-
gating the data for all the instruments. This
allows us to investigate the error dis-
tribution and to estimate overall bounds of
measurement error. Although these error
bounds do not necessarily apply to a
specific instrument at a particular time,
they quantify the size and frequency of the
errors likely to be found in the totality of
measurements. Aggregating the audit
data also provides an easy way to identify
large errors and possible anomalies.
The second analysis considers each
instrument individually, using regression
techniques to define the relationship be-
tween measured and reference concentra-
tions.
Conclusions and
Recommendations
The premise of an audit program is that
random checks of an instrument can pro-
vide a good indication of the performance
of that instrument. During the HOMS,
most instruments were audited at least
once but not more than twice by either
EPA or RTI or both. Such an audit rate for
individual instruments is too low to pro-
vide a data base from which to draw
statistically valid conclusions regarding
the overall performance of a particular
instrument
Because the audits were relatively few
in number, the data were considered in
two ways: by pooling all the audits for each
type of instrument and by treating each
instrument individually. Pooling the data
enlarges the data base and allows general
conclusions to be drawn about the overall
quality of the data. Aggregating the audit
data also helps to identify performance
anomalies. By its nature, the results of the
analysis of pooled audit data cannot be
applied to specific instruments Thus, we
can make statements such as, "The audits
indicate that about 70% of the measure-
ments of pollutant X were accurate to
within 20%." But we cannot state that
70% of the measurements of a specific
instrument were accurate to within 2096.
Analyzing the audit performance of in-
dividual instruments can only serve either
to assuage or raise doubts about the
quality of the measurement of a given
instrument On the one hand, a good audit
performance may cause us to feel con-
fident about the instrument, recognizing,
however, that this does not necessarily
imply that all the measurements of that
instrument are equally good. On the other
hand, a poor audit performance raises
doubts about the instrument, but again
does not necessarily imply that other mea-
surements are equally poor. Hence, one
should consult historical calibration and
repair records to obtain a better indication
of whether the poor (or good) audit repre-
sents a persistent or a one-time condition.
The results of this study provide guidance
about the potential performance of indi-
vidual instruments, but because the anal-
ysis is restricted to the audit data, this
study cannot separate persistent cone
tions from one-time events. Each audit is
one-time event and only many audits c<
fill in the picture.
Table 1 is a summary of the oven
measurement accuracy for the varioi
pollutants, based on the analysis of tt
pooled audit data. The table shows <
estimated range for the percentage of tf
measurements that are accurate to with
±10%, ±20%, and ±30%. For exampl
for ozone Table 1 shows that betwee
50% and 60% of the data were accurate)
within±20%, and85% to95% haderroi
bounded by ±30%. Table 1 indicates thi
for ozone, NO, NOX, N02, and S02 at leas
70% of the data were accurate to withi
±20%. Methane has the largest fraction <
the data accurate to within ±20%. Carbo
monoxide and THC can be considered 1
be the least accurate because they hav
the smallest fraction of the data in th
±20% error band The THC measurement
were found to be subject to large ir
accuracies, which explains the relative!
low percentage of the data within th
±30% error band. There is cause fc
concern about the quality of the THi
measurements and, hence, the NMHl
measurements in the HOMS.
The regression analysis of individuj
instruments showed that almost all dem
onstrated exceptionally good linearity c
response. The precision of the measure
ments, as indicated by the standard erro
of the regressions, was also good in nearh
all cases. The accuracy, as reflected in thi
slope of the regression line, was variable
Specifically, the regression analyses in
dicated potential accuracy problems at th<
following monitoring sites:
• 03: Aldine, Pasadena, Channelview
Seabrooke
NO: Parkhurst, Fuqua, Pearland
NOX: Mae Drive, Parkhurst, Fuqua
NO2: Fuqua
CO: Parkhurst Pearland, Jackrabbit
EPA Mobile Lab.
CH4: Parkhurst
THC: Crawford, Clinton, Parkhurst
S02: Aldine
All the instruments named above hac
regression lines whose slope was greate
than 1.25 or smaller than 0.75 in at leas
one of the audits. The history of these
instruments should be investigated tc
establish whether or not the bias was i
recurring phenomenon. If the bias recurred
then it will be necessary to define cor-
rection factors to be applied to the routine
ly collected data. ,
Future monitoring programs should
continue to include field audits of th«
-------
Table 7. Characterization of Overall Measurement Accuracy
Percent of Measurements Accurate to
Variable
Ozone
Nitric oxide
Nitrogen oxides
Nitrogen dioxide
Carbon monoxide
Methane
Total hydrocarbons
Sulfur dioxide
±10%
50-60
50-70
60-70
45-55
35-65
65-80
20-45
30-40
±20 %
70-80
70-80
70-85
70-75
60-70
85-90
50-60
70-80
±30%
85-95
80-95
85-95
80-90
80-95
90-95
55-60
80-90
instruments. However, the procedures fol-
lowed should be changed to facilitate the
interpretation of the audit data In par-
ticular, the audit procedure should be
designed to allow the analyst to draw
statistically sound general inferences a-
bout the accuracy of individual instruments
from the audit data. It was not possible to
do so in this study because the number of
audits was too small. Hence, it is recom-
mended that future audit programs in-
clude in their design a determination of the
number of audits required to describe the
performance of individual instruments
over the life of the monitoring program.
This is a problem in statistical experiment
design that is beyond the scope of our
investigation.
Although it is desirable to have more
than one agency conduct field audits,
differences between EPA and RTI audit
protocols produced some inconsistent re-
sults (e.g., EPA ozone audits showed a
tendency toward underestimation, and
RTI audits showed the opposite). Thus, in
this case procedural differences obscured
the effects that the audits were intended
to uncover. This situation should be
avoided in future programs by having all
parties adhere to a standard audit protocol.
In this context, the audit procedures used
in the present program resulted in several
cases of same-day audits of the same
instruments by both EPA and RTI. Such
overlap is desirable to check the con-
sistency, or lack thereof, of the results
obtained by the two auditing agencies.
However, same-day audits of the same
instrument essentially constitute a single
audit, which acts to reduce the total num-
ber of tests. Future programs should en-
sure that overlapping audits do not lower
the total number of checks performed.
J. R. Martinez is with SRI International, Menlo Park. CA 94025.
B. Dimitriades is the EPA Project Officer (see below).
The complete report, entitled "Houston Oxidant Modeling Study—1978: Volume
III. Characterization of Data Quality," (Order No. PB 83-194 217; Cost: $11.50,
subject to change) will be available only from:
National Technical Information Service
5285 Port Royal Road
Springfield, VA22161
Telephone: 703-487-4650
The EPA Project Officer can be contacted at:
Environmental Sciences Research Laboratory
U.S. Environmental Protection Agency
Research Triangle Park, NC 27711
-------
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45268
Postage and
Fees Paid
Environmental
Protection
Agency
EPA 335
Official Business
Penalty for Private Use $300
PS 0000329
U S ENVIR PROTECTION AGENCY
REGION 5 LIBRARY
330 S DEARBORN STREET
CHICAGO IL 60604
------- |