United States Environmental Protection Agency Environmental Sciences Research Laboratory Research Triangle Park NC 27711 Research and Development EPA-600/S3-83-023c June 1983 &EPA Project Summary Houston Oxidant Modeling Study—1978: Volume III. Characterization of Data Quality J.R. Martinez During the period 15 September through 12 October 1978, the U. S. Environmental Protection Agency (EPA) conducted a special program that mon- itored air quality and meteorology in the Houston, Texas area. The objec- tives of the program were to obtain a comprehensive data base suitable for use with photochemical air-quality sim- ulation models and to provide a de- tailed body of data that can be used to investigate Houston's air quality in general and photochemical oxidants (HOMS Study) and aerosols (HACS Study) in particular. The objectives of this contract were (a) to evaluate the suitability of the 1978 Houston data base for photo- chemical modeling application, (b) to analyze spatial and temporal patterns of pollutant concentrations, (c) to archive data in a manner suitable for use with air quality simulation models, (d) to analyze and characterize the quality of the gaseous pollutant mea- surements, and (e) to use the data to evaluate the performance of the EKMA photochemical model. Results from the data evaluation and archiving work are reported in a three-volume report. Results from the EKMA evaluation work are reported in a separate report This report. Volume III of the three- volume report, covers the effort to analyze and characterize the quality of the gaseous pollutant measurements obtained in the 1978 HOMS Study. The analysis is based on data obtained in a number of field audits of the instruments used in the HOMS. The audits were performed independently by the EPA and by the Research Triangle Institute under the sponsorship of the EPA. The audit data were analyzed statistically to derive overall bounds for measurement accuracy and to de- fine the accuracy and precision of indi- vidual instruments. This Project Summary was developed by EPA's Environmental Sciences Re- search Laboratory, Research Triangle Park. NC, to announce key findings of the research project that is fully docu- mented in a separate report of the same title (see Project Report ordering information at back). Introduction This is the third volume of a three- volume report describing the results of a study of the quality of the pollutant mea- surements made during the 1978 Houston Oxidant Modeling Study (HOMS). In the course of the HOMS, field audits were conducted to check the performance of the instruments that measured gaseous pollutants. In this study, the audit data will be analyzed to determine the accuracy and precision of the HOMS air quality data. The HOMS was conducted from 1 5 September through 12 October 1978. Its primary objective was to obtain a data base suitable for use with photochemical air quality simulation models. The description of the data base and the analysis of the t.ir quality and meteorological data are re- ported in Volumes I and II. Audit Procedures Concern was with the instruments that monitored the following gaseous pollu- tants: • Ozone (03) • Nitric oxide (NO) ------- Nitrogen dioxide (N02) Oxides of nitrogen (NOX) Carbon monoxide (CO) Total hydrocarbons (THC) Methane (CH4) Sulfur dioxide (S02) Field audits of these instruments were performed independently by two organiza- tions: the U. S. Environmental Protection Agency (EPA) and the Research Triangle Institute (RTI). RTI performed the audits under the sponsorship of the Quality Assurance Branch of EPA. The audit procedure consisted of feeding a known concentration (reference concen- tration) of the gaseous pollutant to the instrument and recording the instrument's response. The reference concentrations usually included a zero level and several nonzero values. Reference ozone concentrations were produced using an ultraviolet ozone gen- erator. The EPA determined the reference ozone level using IMBKI, whereas RTI established the reference concentration using gas phase titration. Test atmospheres of NO, NOX, and N02 were generated by both EPA and RTI using standard-gas cylinder dilution and gas phase titration. The standard gases were referenced to NBS-SRM 1683 (RTI) and to NBS-SRM 1684 (both EPA and RTI). Standard-gas cylinders were also used to generate reference concentrations of CO, CH4, and S02. EPA used CO and CH4 cylinders from Scott Environmental. Statistical Analysis of Audit Data The statistical analysis of quality as- surance audit data from the HOMS in- cludes: (1) obtaining frequency distribu- tions for percent relative error (percent relative error = 100 x [measured - ref- erence] / reference); and (2) performing regression analyses of measured and ref- erence concentrations for individual in- struments. The analysis was performed separately for the EPA and RTI data because of the differences in their audit procedures. The audit data have been analyzed in two ways. One type of analysis examines the total variation of measurement errors for each variable (e.g., ozone) by aggre- gating the data for all the instruments. This allows us to investigate the error dis- tribution and to estimate overall bounds of measurement error. Although these error bounds do not necessarily apply to a specific instrument at a particular time, they quantify the size and frequency of the errors likely to be found in the totality of measurements. Aggregating the audit data also provides an easy way to identify large errors and possible anomalies. The second analysis considers each instrument individually, using regression techniques to define the relationship be- tween measured and reference concentra- tions. Conclusions and Recommendations The premise of an audit program is that random checks of an instrument can pro- vide a good indication of the performance of that instrument. During the HOMS, most instruments were audited at least once but not more than twice by either EPA or RTI or both. Such an audit rate for individual instruments is too low to pro- vide a data base from which to draw statistically valid conclusions regarding the overall performance of a particular instrument Because the audits were relatively few in number, the data were considered in two ways: by pooling all the audits for each type of instrument and by treating each instrument individually. Pooling the data enlarges the data base and allows general conclusions to be drawn about the overall quality of the data. Aggregating the audit data also helps to identify performance anomalies. By its nature, the results of the analysis of pooled audit data cannot be applied to specific instruments Thus, we can make statements such as, "The audits indicate that about 70% of the measure- ments of pollutant X were accurate to within 20%." But we cannot state that 70% of the measurements of a specific instrument were accurate to within 2096. Analyzing the audit performance of in- dividual instruments can only serve either to assuage or raise doubts about the quality of the measurement of a given instrument On the one hand, a good audit performance may cause us to feel con- fident about the instrument, recognizing, however, that this does not necessarily imply that all the measurements of that instrument are equally good. On the other hand, a poor audit performance raises doubts about the instrument, but again does not necessarily imply that other mea- surements are equally poor. Hence, one should consult historical calibration and repair records to obtain a better indication of whether the poor (or good) audit repre- sents a persistent or a one-time condition. The results of this study provide guidance about the potential performance of indi- vidual instruments, but because the anal- ysis is restricted to the audit data, this study cannot separate persistent cone tions from one-time events. Each audit is one-time event and only many audits c< fill in the picture. Table 1 is a summary of the oven measurement accuracy for the varioi pollutants, based on the analysis of tt pooled audit data. The table shows < estimated range for the percentage of tf measurements that are accurate to with ±10%, ±20%, and ±30%. For exampl for ozone Table 1 shows that betwee 50% and 60% of the data were accurate) within±20%, and85% to95% haderroi bounded by ±30%. Table 1 indicates thi for ozone, NO, NOX, N02, and S02 at leas 70% of the data were accurate to withi ±20%. Methane has the largest fraction < the data accurate to within ±20%. Carbo monoxide and THC can be considered 1 be the least accurate because they hav the smallest fraction of the data in th ±20% error band The THC measurement were found to be subject to large ir accuracies, which explains the relative! low percentage of the data within th ±30% error band. There is cause fc concern about the quality of the THi measurements and, hence, the NMHl measurements in the HOMS. The regression analysis of individuj instruments showed that almost all dem onstrated exceptionally good linearity c response. The precision of the measure ments, as indicated by the standard erro of the regressions, was also good in nearh all cases. The accuracy, as reflected in thi slope of the regression line, was variable Specifically, the regression analyses in dicated potential accuracy problems at th< following monitoring sites: • 03: Aldine, Pasadena, Channelview Seabrooke NO: Parkhurst, Fuqua, Pearland NOX: Mae Drive, Parkhurst, Fuqua NO2: Fuqua CO: Parkhurst Pearland, Jackrabbit EPA Mobile Lab. CH4: Parkhurst THC: Crawford, Clinton, Parkhurst S02: Aldine All the instruments named above hac regression lines whose slope was greate than 1.25 or smaller than 0.75 in at leas one of the audits. The history of these instruments should be investigated tc establish whether or not the bias was i recurring phenomenon. If the bias recurred then it will be necessary to define cor- rection factors to be applied to the routine ly collected data. , Future monitoring programs should continue to include field audits of th« ------- Table 7. Characterization of Overall Measurement Accuracy Percent of Measurements Accurate to Variable Ozone Nitric oxide Nitrogen oxides Nitrogen dioxide Carbon monoxide Methane Total hydrocarbons Sulfur dioxide ±10% 50-60 50-70 60-70 45-55 35-65 65-80 20-45 30-40 ±20 % 70-80 70-80 70-85 70-75 60-70 85-90 50-60 70-80 ±30% 85-95 80-95 85-95 80-90 80-95 90-95 55-60 80-90 instruments. However, the procedures fol- lowed should be changed to facilitate the interpretation of the audit data In par- ticular, the audit procedure should be designed to allow the analyst to draw statistically sound general inferences a- bout the accuracy of individual instruments from the audit data. It was not possible to do so in this study because the number of audits was too small. Hence, it is recom- mended that future audit programs in- clude in their design a determination of the number of audits required to describe the performance of individual instruments over the life of the monitoring program. This is a problem in statistical experiment design that is beyond the scope of our investigation. Although it is desirable to have more than one agency conduct field audits, differences between EPA and RTI audit protocols produced some inconsistent re- sults (e.g., EPA ozone audits showed a tendency toward underestimation, and RTI audits showed the opposite). Thus, in this case procedural differences obscured the effects that the audits were intended to uncover. This situation should be avoided in future programs by having all parties adhere to a standard audit protocol. In this context, the audit procedures used in the present program resulted in several cases of same-day audits of the same instruments by both EPA and RTI. Such overlap is desirable to check the con- sistency, or lack thereof, of the results obtained by the two auditing agencies. However, same-day audits of the same instrument essentially constitute a single audit, which acts to reduce the total num- ber of tests. Future programs should en- sure that overlapping audits do not lower the total number of checks performed. J. R. Martinez is with SRI International, Menlo Park. CA 94025. B. Dimitriades is the EPA Project Officer (see below). The complete report, entitled "Houston Oxidant Modeling Study—1978: Volume III. Characterization of Data Quality," (Order No. PB 83-194 217; Cost: $11.50, subject to change) will be available only from: National Technical Information Service 5285 Port Royal Road Springfield, VA22161 Telephone: 703-487-4650 The EPA Project Officer can be contacted at: Environmental Sciences Research Laboratory U.S. Environmental Protection Agency Research Triangle Park, NC 27711 ------- United States Environmental Protection Agency Center for Environmental Research Information Cincinnati OH 45268 Postage and Fees Paid Environmental Protection Agency EPA 335 Official Business Penalty for Private Use $300 PS 0000329 U S ENVIR PROTECTION AGENCY REGION 5 LIBRARY 330 S DEARBORN STREET CHICAGO IL 60604 ------- |