Batteiie The Bu sincss ii/ I iinov.it ion Environmental Technology Verification Program Advanced Monitoring Systems Center Test/QA Plan for Verification of Ozone Indicator Cards ElVft/ ElV ------- TEST/QA PLAN for Verification of Ozone Indicator Cards November 10, 2009 Prepared by Battelle 505 King Avenue Columbus, OH 43201-2693 ------- Ozone Indicator Cards Test/QA Plan Page 1 of 45 Version 1 November 10, 2009 SECTION A PROJECT MANAGEMENT A1 Vendor Approval Page ETV Advanced Monitoring Systems Center Test/QA Plan for Verification of Ozone Indicator Cards November 10, 2009 Oeoijc Indicator Cards Test-QA Plan Page J of45 Version I November 10, 2009 VENDOR ACCEPTANCE: Name Company fcHiy/iVO Ln. i i/l C Date OQ gj Ayi A_^| ------- Ozone Indicator Cards Test/QA Plan Page 2 of 45 Version 1 November 10, 2009 A2 TABLE OF CONTENTS Section Page A PROJECT MANAGEMENT 1 A2 Table of Contents 2 A3 Acronyms and Abbreviations 4 A4 Distribution List 5 A5 Verification Test Organization 6 A6 Background 12 A7 Verification Test Descrition and Schedule 14 A8 Quality Objectives 15 A9 Special Training/Certification 15 A10 Documentation and Records 16 B MEASUREMENT AM) DATA ACQUISITION 18 B1 Experimental Design 18 B2 Reference Sample Collection 30 B3 Sample Handling and Custody Requirements 31 B4 Reference Method 31 B5 Quality Control Requirements 31 B6 Instrument/Equipment Testing, Inspection, and Maintenance 32 B7 Instrument Calibration and Frequency 32 B8 Inspection/Acceptance of Supplies and Consumables 32 B9 Non-Direct Measurements 32 BIO Data Management 32 C ASSESSMENT AND OVERSIGHT 36 CI Assessments and Response Actions 36 C2 Reports to Management 38 D DATA VALIDATION AM) USABILITY 39 D1 Data Review, Validation, and Verification Requirements 39 D2 Validation and Verification Methods 39 D3 Reconciliation with User Requirements 39 E REFERENCES 41 ------- Ozone Indicator Cards Test/QA Plan Page 3 of 45 Version 1 November 10, 2009 Appendices Page APPENDIX A: Field Data Sheet for Use by SCAQMD 44 APPENDIX B: Field Data Sheet for Use by BCLA 46 Figure Page Figure 1. Organization Chart for the Verification Test 7 Figure 2. Examples of Ozone Indicator Cards 19 Tables Page Table 1. Planned Verification Schedule 16 Table 2. Summary of OIC Verification Samples 21 Table 3. Summary of Data Recording Process 34 ------- Ozone Indicator Cards Test/QA Plan Page 4 of 45 Version 1 November 10, 2009 A3 ACRONYMS AND ABBREVIATIONS ADQ audit of data quality AMS Advanced Monitoring Systems BCLA Breathe California of Los Angeles EPA U.S. Environmental Protection Agency ETV Environmental Technology Verification FEM Federal Equivalent Method LRB laboratory record book Mn02 manganese dioxide nm nanometer OIC ozone indicator card ppbv parts per billion by volume PE performance evaluation QA quality assurance QC quality control QMP quality management plan r2 coefficient of determination RH relative humidity SCAQMD South Coast Air Quality Management District TSA technical systems audit T temperature UV ultraviolet VOC volatile organic compounds ------- Ozone Indicator Cards Test/QA Plan Page 5 of 45 Version 1 November 10, 2009 A4 DISTRIBUTION LIST Vendors Dieter F el db erg EnviroScan, Inc. 689 Unionville Road Kennett Square, PA 19348 EPA Michelle Henderson U.S. Environmental Protection Agency ETAV/NRMRL/ORD 26 W. Martin Luther King Dr., MS 208 Cincinnati, OH 45268 Collaborators/Subcontractors Dan Witzling Environmental Program Manager Breathe California of Los Angeles 5858 Wilshire Blvd, Suite 300 Los Angeles, CA 90036 Rudy Eden Laboratory Services and Source Testing South Coast Air Quality Management District 21865 E. Copley Drive Diamond Bar, CA 91765 John McKernan, ScD, CIH Commander, USPHS Senior Environmental Health Officer U.S. Environmental Protection Agency ETAV/NRMRL/ORD 26 W. Martin Luther King Dr., MS 208 Cincinnati, OH 45268 Battelle Amy Dindal Thomas Kelly Brad Goodwin Zachary Willenberg Battelle 505 King Ave. Columbus, OH 43201 Peer Reviewers Dennis Mikel U.S. Environmental Protection Agency Office of Air Quality Planning and Standards E-143-02 Research Triangle Park, NC 27711 Will Ollison American Petroleum Institute 1220 L Street Washington, D.C. 2005-4070 ------- Ozone Indicator Cards Test/QA Plan Page 6 of 45 Version 1 November 10, 2009 A5 VERIFICATION TEST ORGANIZATION The verification test will be conducted under the auspices of the U.S. Environmental Protection Agency (EPA) through the Environmental Technology Verification (ETV) Program. It will be performed by Battelle, which is managing the ETV Advanced Monitoring Systems (AMS) Center through a cooperative agreement with EPA. The scope of the AMS Center covers verification of monitoring technologies for contaminants and natural species in air, water, and soil. This verification will address ozone indicator cards (OICs) that provide short-term semi- quantitative measures of ozone concentration in ambient air. The performance of OICs will be judged in part by comparison with measurements made with an EPA-designated reference measurement method (the Federal Equivalent Method [FEM]) for ozone. The day to day operations of this verification test will be coordinated and supervised by Battelle personnel, with in-kind involvement from personnel of the California South Coast Air Quality Management District (SCAQMD), Breathe California of Los Angeles (BCLA), and the vendors who will be having the performance of their OICs verified. Laboratory testing will be conducted at Battelle in Columbus, Ohio. Field testing will be carried out by SCAQMD and BCLA under Battelle's oversight at multiple locations in southern California. The vendors will provide Battelle with instructions for use of their OICs and enough OICs to make up to 1,000 ozone readings. The organization chart in Figure 1 identifies the responsibilities of the organizations and individuals associated with the verification test. Roles and responsibilities are defined further below. Quality Assurance (QA) oversight will be provided by the Battelle Quality Manager and also by the EPA AMS Center Quality Manager, at her discretion. ------- Ozone Indicator Cards Test/QA Plan Page 7 of 45 Version 1 November 10, 2009 Figure 1. Organization Chart for the Verification Test A5.1 Battelle Dr. Thomas Kelly is the AMS Center's Verification Test Coordinator for this test. In this role, Dr. Kelly will have overall responsibility for ensuring that the technical, schedule, and cost goals established for the verification test are met. Specifically, Dr. Kelly will: Prepare the draft test/QA plan, verification reports, and verification statements. Revise the draft test/QA plan, verification reports, and verification statements in response to reviewers' comments. ------- Ozone Indicator Cards Test/QA Plan Page 8 of 45 Version 1 November 10, 2009 Assemble a team of qualified technical staff to conduct the verification test. Establish a budget for the verification test and manage staff to ensure the budget is not exceeded. Coordinate with the vendors for provision of OICs for testing. Direct Battelle technical staff in the laboratory testing and coordinate with BCLA personnel for performance of the field testing Direct the team in performing the verification test in accordance with this test/QA plan. Hold a kick-off meeting approximately one week prior to the start of the verification test to review the critical logistical, technical, and administrative aspects of the verification test. Responsibility for each aspect of the verification test will be confirmed. Ensure that all quality procedures specified in this EPA Quality Level III test/QA plan and in the AMS Center Quality Management Plan1 (QMP) are followed. Serve as the primary point of contact for SCAQMD, BCLA, and vendor representatives. Ensure that confidentiality of sensitive vendor information is maintained. Assist vendors as needed during verification testing. Become familiar with the operation of the OICs through instruction by the vendors, if needed. Prepare a deviation report for any departure from the test/QA plan during the verification, obtain the requisite EPA and vendor approvals, and distribute the approved report as specified in the AMS Center QMP. Respond to any issues raised in assessment reports, audits, or from test staff observations, and institute corrective action as necessary. Coordinate distribution of the final test/QA plan, verification reports, and verification statements. Ms. Amy Dindal is Battelle's Manager for the AMS Center. As such, Ms. Dindal will oversee the various stages of verification testing. Ms. Dindal will: Review the draft and final test/QA plan. Attend the verification test kick-off meeting. Review the draft and final verification reports and verification statements. ------- Ozone Indicator Cards Test/QA Plan Page 9 of 45 Version 1 November 10, 2009 Ensure that necessary Battelle resources, including staff and facilities, are committed to the verification test. Maintain communication with EPA's technical and quality managers. Issue a stop work order if Battelle or EPA QA staff discover adverse findings that will compromise test results. Technical staff from Battelle will support Dr. Kelly in planning and conducting the verification test. The responsibilities of the technical staff will be to: Assist in planning for the test, and making arrangements for the receipt of and training on the OICs. Attend the verification test kick-off meeting. Conduct verification testing using the vendors' OIC technology. Conduct FEM measurements during the testing. Perform statistical calculations specified in this test/QA plan on the technology data as needed. Provide results of statistical calculations and associated discussion for the verification reports as needed. Support Dr. Kelly in responding to any issues raised in assessment reports and audits related to statistics and data reduction as needed. Mr. Zachary Willenberg is Battelle's Quality Manager for the AMS Center. Mr. Willenberg will: Review the draft and final test/QA plan. Attend the verification test kick-off meeting. Conduct a technical systems audit (TSA) at least once during the verification test, or designate other QA staff to conduct the audit. That TSA will address both the laboratory and field components of OIC testing. Audit at least 10% of the verification data or designate other QA staff to conduct the data audit. Prepare and distribute an assessment report for each audit. Verify implementation of any necessary corrective action. Request that Battelle's AMS Center Manager issue a stop work order if audits indicate that data quality is being compromised. ------- Ozone Indicator Cards Test/QA Plan Page 10 of 45 Version 1 November 10, 2009 Provide a summary of the QA/quality control (QC) activities and results for the verification reports. Review the draft and final verification reports and verification statements. A5.2 OIC Vendor The responsibilities of an OIC vendor are as follows: Review and provide comments on the draft test/QA plan. Accept (by signature of a company representative) the final test/QA plan prior to test initiation. Initially provide enough units of their OIC technology to carry out up to 1,000 measurements of ambient ozone during the verification test, and additional units should the need arise during testing. Supply instructions on the use of the technology, and written consent for test staff to carry out verification testing. Review and provide comments on the draft verification report and verification statement for their respective technology. A5.3 South Coast Air Quality Management District The primary point of contact for SCAQMD will be Mr. Rudy Eden. The role of SCAQMD is to assist Battelle in conducting a quantitative comparison of OIC readings with FEM readings. Mr. Eden will lead SCAQMD personnel in meeting the following responsibilities of SCAQMD in this verification: Review and provide comments on the draft test/QA plan. Select established SCAQMD ambient ozone monitoring sites that exhibit a range of ozone concentrations and are available for this verification. Provide sufficient SCAQMD personnel to carry out ambient ozone measurements with OICs at the selected sites. Provide ozone and meteorological data from those sites. Review and provide comments on the draft verification report and verification statement for each tested OIC technology. ------- Ozone Indicator Cards Test/QA Plan Page 11 of 45 Version 1 November 10, 2009 A5.4 Breathe California of Los Angeles The primary point of contact for BCLA will be Mr. Dan Witzling. The role of BCLA is to assist Battelle in conducting a qualitative evaluation of OIC use, including an assessment of OIC duplication and user agreement. The operators of OICs in BCLA testing will be non-technically trained adult volunteers who lead after-school programs in the Long Beach, CA area. The test sites of interest will be elementary and/or secondary schools in Long Beach where potential exposures of school children to ozone are of concern. Mr. Witzling will lead in meeting the following responsibilities of BCLA in this verification: Review and provide comments on the draft test/QA plan. Identify and coordinate sufficient volunteer personnel to carry out ambient ozone measurements with OICs at sites of interest to BCLA. Provide the results of those OIC measurements to Battelle along with user observations about the OICs. Conduct wearer acceptance testing of the OICs when worn as a personal exposure badge, and report wearer observations to Battelle. Review and provide comments on the draft verification report and verification statement for each tested OIC technology. A5.5 EPA EPA's responsibilities in the AMS Center are based on the requirements stated in the "Environmental Technology Verification Program Quality Management Plan" (EPA ETV QMP).2 The roles of specific EPA staff are as follows: Ms. Michelle Henderson is EPA's AMS Center Quality Manager. For the verification test, Ms. Henderson will: Review the draft and approve the final test/QA plan. Attend the verification kickoff meeting. Review checklists, reports, report responses, and closure statements of TSA, PE audits, and quality systems audits (QSAs) conducted by Battelle. ------- Ozone Indicator Cards Test/QA Plan Page 12 of 45 Version 1 November 10, 2009 Perform at her option an external TSA of field and/or laboratory activities, performance evaluation (PE) audit, and/or an audit of data quality during the verification test. Notify the EPA AMS Center Project Officer of the need for a stop work order if evidence indicates that data quality is being compromised. Prepare and distribute an assessment report summarizing results of the external audit performed. Review the draft and approve the final verification reports and verification statements. Dr. John McKernan is EPA's Project Officer for the AMS Center. Dr. McKernan will: Review the draft test/QA plan. Approve the final test/QA plan. Attend the verification kickoff meeting. Be available during the verification test to authorize any test/QA plan deviations by phone and provide the name of a delegate to the Battelle AMS Center Manager should he not be available during the testing period. Review the draft verification reports and verification statements. Oversee the EPA review process for the test/QA plan, verification reports, and verification statements. Coordinate the submission of verification reports and verification statements for final EPA approval. Post the test QA plan, verification reports, and verification statements on the ETV web site. A6 BACKGROUND A6.1 Technology Need The ETV Program's AMS Center conducts third-party performance testing of commercially available technologies that detect or monitor natural species or contaminants in air, water, and soil. The purpose of ETV is to provide objective and quality assured performance data on environmental technologies, so that users, developers, regulators, and consultants can make informed decisions about purchasing and applying these technologies. Stakeholder committees ------- Ozone Indicator Cards Test/QA Plan Page 13 of 45 Version 1 November 10, 2009 of buyers and users of such technologies recommend technology categories, and technologies within those categories, as priorities for testing. Verification reports from previous tests are available at http://www.epa.gov/nrmrl/std/etv/verifiedtechnologies.html. Ozone is a widespread pollutant that is formed by photochemical processes involving sunlight, nitrogen oxides, and volatile organic compounds (VOC) in air. The U.S. Clean Air Act and its Amendments established air quality standards for ozone, and pollution control strategies that require state and local authorities to regulate for compliance with the standards. Ozone is regulated because of its effect on human health when air containing elevated concentrations of ozone is inhaled. Because of the costs associated with emission control programs and penalties for those regions that are not in compliance, it is essential that ozone measurements that determine compliance with standards be accurate. For that purpose EPA has established Federal "3 Reference and Equivalent Methods (FRM and FEM) of monitoring ozone. The method currently widely used is the FEM, which makes use of the ozone molecule's strong absorption band in the ultraviolet region of the spectrum, with a maximum coinciding with the strong mercury vapor emission line at 254 nanometers (nm). The FEM has completely supplanted the FRM for all compliance monitoring in the U.S., because of the greater complexity of the FRM and its requirement for flammable ethylene gas. Commercial FEM instruments measure the transmission of UV light through an air sample and compare the intensity with that obtained along the same pathlength through air containing no ozone. A scrubber (typically Mn02or a heated metal scrubber [HMS]) designed to selectively remove ozone from the air, is used to determine the background absorption of UV light by species such as aromatic hydrocarbons. Potential interferences in the FEM ozone measurement, due to removal of UV-absorbing aromatic hydrocarbons by the ozone scrubber, have been identified and may be significant in highly polluted conditions that produce elevated ozone levels of regulatory significance.4 For assessment of human exposure to ozone in air, less expensive and complex measurements of ozone can be useful. Qualitative methods that approximately indicate the ozone concentration in air can be helpful to people with ailments that cause respiratory sensitivity to ozone. A6.2 Technology Description One simple and inexpensive approach to ozone measurement is the use of colorimetric indicator cards for ozone, which incorporate a reagent that undergoes a color change when exposed to ozone in air. To use the card, a protective film over a reagent spot is removed and the spot is exposed to ambient air for a specified period of time (e.g., 10 minutes). Then the intensity of the ------- Ozone Indicator Cards Test/QA Plan Page 14 of 45 Version 1 November 10, 2009 resulting color change in the reagent spot is visually compared to a color index printed on the card, providing an estimate of the ozone concentration. Such OICs typically indicate ozone concentrations in a few broad concentration ranges from near zero to over 100 parts per billion by volume (ppbv). The ETV verification described in this test/QA plan will explore the effectiveness and ease of use of OICs and quantify their response relative to the response of FEM instrumentation in laboratory and outdoor testing. A7 VERIFICATION TEST DESCRIPTION AND SCHEDULE A7.1 Verification Test Description The purpose of this test/QA plan is to specify procedures for a verification test applicable to commercial OICs. One aspect of the verification test is to compare ozone readings made by the OICs to those made by the FEM UV method for ozone. In performing the verification test, Battelle will follow the technical and QA procedures specified in this test/QA plan and will comply with the data quality requirements in the AMS Center QMP.1 This is an EPA QA Level III project. A7.2 Verification Schedule Table 1 shows the planned schedule of activities for the verification testing and data analysis and reporting. Both laboratory and field testing of OICs will be initiated upon final approval of this test/QA plan, and will take place simultaneously. As shown in Table 1, testing activities are planned to begin in the fall of 2009. Laboratory testing will require approximately one month. Field testing will be split into fall and spring/summer episodes to assure capture of a range of ambient ozone levels. Each of those two field periods will require approximately one month to complete. A separate ETV verification report will then be drafted for each OIC tested, and the report will be reviewed simultaneously by the technology vendor, SCAQMD, and BCLA, and subsequently by peer reviewers. The final reports will be submitted to EPA for final signature, and the final reports will be made publicly available on the EPA/ETV web site. A7.3 Test Facility Laboratory analyses will be conducted in Battelle laboratories in Columbus, Ohio. In performing this verification test, Battelle will follow the procedures specified in this test/QA plan and will comply with quality requirements in the AMS Center QMP.1 The air quality laboratory that will be used is fully equipped for production, delivery, and calibrated measurement of ozone in test atmospheres of controlled temperature and humidity. ------- Ozone Indicator Cards Test/QA Plan Page 15 of 45 Version 1 November 10, 2009 A8 QUALITY OBJECTIVES This verification test is designed to evaluate the performance of the OICs for determining ozone in air. Laboratory evaluation will include a comparison of the OIC results to a commercial ozone monitor designated by EPA as an ozone FEM (Thermo Environmental Model 49C, FEM EQOA-0880-047). Ozone will be generated at known concentrations to challenge the OICs, and will be monitored simultaneously by the FEM instrument. The ozone will be generated using an Environics Model 6400 ozone generator that has itself been calibrated against a Dasibi Model 1800 UV calibration photometer. In field testing OIC readings will be compared to those of ozone analyzers operating at SCAQMD air monitoring sites. To the extent possible, SCAQMD sites will be used at which FEM monitors equipped with HMS technology, or FRM monitors, are deployed. However, the choice of field sites and ozone monitors will be subject to SCAQMD discretion. The quality of the ozone data will be documented by calibration data, and documentation of the QA activities carried out by the relevant air quality agencies at the field monitoring sites. QA/QC requirements will include a TSA, a PE audit, and an audit of data quality (ADQ). These QA procedures will be carried out by Battelle. The planned audit procedures are described in Section CI. The EPA Quality Manager also may conduct an independent TSA, PE audits, and/or ADQ at her discretion. A9 SPECIAL TRAINING/CERTIFICATION Documentation of training related to technology testing, field testing, data analysis, and reporting is maintained for all Battelle technical staff in training files at their respective locations. The Battelle Quality Manager may verify the presence of appropriate training records prior to the start of testing. Battelle technical staff involved in this verification will have experience in operation of ozone calibration and monitoring equipment, and specifically with the Environics, Dasibi, and Thermo Environmental equipment noted in Section A8. SCAQMD personnel who use the OICs at SCAQMD field sites will be site operators experienced in calibration and performance of FEM ozone measurements at those sites. All Battelle, SCAQMD, and BCLA personnel will receive training in use of the OICs based on the instructions which are printed on every OIC. ------- Ozone Indicator Cards Test/QA Plan Page 16 of 45 Version 1 November 10, 2009 Table 1. Planned Verification Schedule Month and Year Verification Activity Testing Data Analysis and Reporting November 2009 Initiate laboratory and field testing Begin preparation of ETV report template December 2009 Completion of laboratory testing Completion of first phase field testing Compile data from laboratory testing Compile data from first phase field testing Review and summarize laboratory operator observations January-April 2010 Analyze laboratory test data Analyze field test data Review and summarize field operator observations Prepare partial draft reports May-June 2010 Completion of second phase field testing Compile data and operator observations from second phase field testing Analyze field test data July 2010 Analyze field test data Prepare draft reports Internal review of draft reports August 2010 Vendor/BCLA/SCAQMD/EPA review of draft reports Revision of draft reports Peer review of draft reports September 2010 Revision of draft reports Submission of final reports for EPA approval A10 DOCUMENTATION AND RECORDS The records for this verification test will be contained in the test/QA plan, laboratory record books (LRB), data collection forms, electronic files (both raw data and spreadsheets), the final verification report, or assessment reports. All of these records will be maintained in the Verification Test Coordinator's office during the test and will be transferred to permanent storage at Battelle's Records Management Office within two months of the finalization of the verification reports. All Battelle LRBs are also stored indefinitely, either by the Verification Test Coordinator or within two months of the finalization of the verification reports in Battelle's Records Management Office. EPA will be notified before disposal of any files. Section BIO further details the data recording practices and responsibilities. ------- Ozone Indicator Cards Test/QA Plan Page 17 of 45 Version 1 November 10, 2009 All written records will be in ink. Any corrections to notebook entries, or changes in recorded data, will be made with a single line through the original entry. The correction is then to be entered, initialed, and dated by the person making the correction. In all cases, strict confidentiality of data from the vendor's technology will be maintained, i.e., separate files (including manual records, printouts, and/or electronic data files) will be kept for each vendor's technology, and no intercomparison of data from different OIC technologies will be conducted. ------- Ozone Indicator Cards Test/QA Plan Page 18 of 45 Version 1 November 10, 2009 SECTION B MEASUREMENT AND DATA ACQUISITION B1 EXPERIMENTAL DESIGN An OIC consists of a playing-card-size piece of cardboard on which one or more spots of a solid reagent are placed, each spot being approximately 5 mm in diameter. Before use each reagent spot is covered and completely sealed with a thin strip of flexible foil. To use the OIC, the foil is removed from a reagent spot and discarded, and the card is then left exposed to the ambient air for a period of time specified by the vendor (e.g., 10 minutes for use in outdoor air). The reagent reacts with ozone in the air and changes color, typically darkening from white or pale yellow before exposure to darker yellow or brown. The extent of the color change depends on the ozone concentration in the air and the duration of exposure; close adherence to the vendor-specified exposure time allows the color change to be used as an indicator of the ambient ozone concentration. Determination of the ozone concentration is done by visually comparing the color of the exposed reagent spot to a printed color standard immediately after the end of the exposure period. The color standard may be separate from the OIC or printed on the OIC itself, and in the latter case can often be positioned over the exposed reagent spot by folding the OIC so that the reagent spot shows through a hole around which the color standard is printed. The correspondence of color with ozone concentration is qualitative, in that OIC ozone readings fall into broad ranges. For example, one common OIC has four color comparison standards (numbered 1 through 4) which correspond to ozone concentration ranges of 10 to 45, 45 to 75, 75 to 105, and >105 ppbv, respectively. Figure 2 shows examples of commercially available OICs, illustrating the arrangement of reagent spots and the positioning of the color standard. Instructions for use of the OIC are printed on the card. ------- Ozone Indicator Cards Test/QA Plan Page 19 of 45 Version 1 November 10, 2009 (hone IMeetor Cord Figure 2. Examples of Ozone Indicator Cards This test/QA plan addresses the verification of OICs through laboratory testing and monitoring of ozone in ambient air. Specifically the OICs will be evaluated for the following performance parameters: Accuracy Variability of readings OIC duplication and user agreement Effect of light intensity on OIC color development Effect of ambient temperature, relative humidity (RH), and wind speed on OIC readings Operational factors including ease of use, readability, and use of OICs as personal monitors. ------- Ozone Indicator Cards Test/QA Plan Page 20 of 45 Version 1 November 10, 2009 Accuracy will be determined by comparing OIC readings to reference ozone measurements made in laboratory and field testing. Variability will be assessed by observing the spread of OIC readings made at constant ozone concentrations. OIC duplication will be assessed by comparing ozone readings made by a single user with duplicate OICs exposed simultaneously. User agreement will be assessed by comparing visual readings made by two different users on the same individual OIC reagent spots. Use of the OICs requires visual reading of the color of a reagent spot; therefore all users will be asked to indicate if they are color-blind or have other visual impairment before taking part in testing, and to remove sunglasses before making OIC visual readings. The effect of light intensity on OIC color changes will be assessed in the laboratory by comparing OIC readings made by a single user on OICs exposed at a constant ozone level but under varying degrees of illumination. Temperature, RH, and wind speed effects will be estimated by recording the ambient conditions at ozone monitoring sites in the field at the time that OIC and reference FEM measurements are obtained. Operator observations will be recorded in all laboratory and field testing to assess operational factors of the OICs. In all testing, each OIC and reagent spot will be uniquely numbered to facilitate data recording. Bl.l Test Procedures The following sections describe the test procedures that will be used to evaluate each of the performance parameters listed above. Table 2 summarizes the types and numbers of samples that will be used to verify performance of each OIC. For laboratory testing, test atmospheres will be generated in a small environmental chamber or continuous flow system. The OICs undergoing testing will be placed in the chamber or flow system, and exposed to air of normal temperature and RH (i.e., approximately 22 °C and 50% RH) and controlled ozone content. The temperature, RH, and ozone concentration of the test air will be monitored during each test, using standard commercial temperature and RH sensors and a Thermo Environmental Model 49C UV ozone monitor (FEM EQOA-0880-047). The signals from the ozone monitor and temperature and RH monitors will be recorded using a Campbell Scientific datalogger or similar data acquisition system. ------- Ozone Indicator Cards Test/QA Plan Page 21 of 45 Version 1 November 10, 2009 Table 2. Summary of OIC Verification Samples Sample Type Responsible Organization Number of Samples or Readings" Associated QC Samples Uses Laboratory OIC readings Battelle 200 30 NA Accuracy, variability, OIC duplication user agreement Light intensity effect Laboratory FEM readings Battelle 200 30 Initial multipoint calibration, daily zero/span check Accuracy, variability, OIC duplication user agreement Light intensity effect Laboratory light intensity readings Battelle 30 Battelle Instrumentation Laboratory calibration dark (zero) checks Light intensity effect Laboratory temperature and RH readings Battelle Continuous Battelle Instrumentation Laboratory calibration Documentation of laboratory test conditions Field OIC Readings SCAQMD 120b NA Accuracy, OIC duplication, user agreement, effect of ambient conditions Field FEM Readings SCAQMD 120b Per SCAQMD compliance monitoring program Accuracy, OIC duplication, user agreement, effect of ambient conditions Field temperature, RH, wind speed readings SCAQMD 120b Per SCAQMD maintenance Effect of ambient conditions Field OIC Readings BCLA 120b NA OIC duplication, user agreement User observations BCLA/Battelle/S CAQMD As needed NA Operational factors a: Estimated number of samples/readings per each OIC technology tested, b: Rough estimate, testing will be subject to responsible organization's discretion. Field testing will consist of use of the OICs by SCAQMD or BCLA representatives at multiple sites in southern California. SCAQMD representatives will use OICs at sites at which continuous ozone monitoring is carried out for compliance purposes using the FEM. At such sites OIC readings will be recorded and compared to the average FEM readings over the time period of the OIC exposure to ambient air. Temperature, RH, and wind speed data will be recorded by SCAQMD staff during each OIC exposure interval in the field, and those data will ------- Ozone Indicator Cards Test/QA Plan Page 22 of 45 Version 1 November 10, 2009 be evaluated to assess the effects of these conditions on OIC performance. BCLA staff will use OICs at schools in Long Beach, CA where personal exposure of children to ozone may be of particular concern. At both types of sites, multiple OICs will be exposed and read at the same time, and to the extent feasible multiple users will record their visual judgment of the OIC readings. All field testing will be conducted outdoors, and in all testing each OIC will be tested with a single exposure duration (e.g., 10 minutes) specified by the respective vendor. The total number of OICs required from each vendor for testing will be less than 200; the exact number to be used will depend on field testing factors such as personnel availability, site issues, etc., that cannot be entirely defined in this plan. Laboratory testing is expected to use approximately 40 OICs if each OIC has five reagent spots, so approximately 50 will be required to account for practice runs, damage, etc. The number of OICs to be used by SCAQMD for accuracy testing similarly is planned to be about 25 at five reagent spots per OIC, but practice runs, damage, and greater than expected availability of SCAQMD personnel may double that number. The number of OICs to be used by BCLA will depend on the interest and involvement of volunteer personnel; approximately 60 OICs will be provided to BCLA initially, and the vendor will be prompted for more OICs if volunteer activities consume that initial allotment. The quality and objectivity of the field evaluations conducted by SCAQMD and BCLA will be assured by the efforts of Battelle technical and QA staff. The Battelle Verification Test Coordinator (Dr. Kelly) will communicate the procedures and expectations for OIC evaluation to both the SCAQMD and BCLA lead representatives (Mr. Eden and Mr. Witzling, respectively), by verbal and written means including this test/QA plan. That communication will be maintained before, during, and after the periods of field testing, and will continue in the data analysis phase of the verification. In addition, the Battelle AMS Center QA Manager will conduct on-site TSAs at both SCAQMD and BCLA field sites during OIC testing. In those efforts he will compare test procedures and data recording efforts to the requirements of this test/QA plan, and direct any necessary improvements. Bl. 1.1-Accuracy The accuracy of the OIC will be determined by comparing OIC readings determined visually to simultaneous measurements made using the FEM. The comparison of accuracy will be made graphically, by plotting OIC readings in indicated OIC ranges (e.g., 10 to 45, 45 to 75, 75 to 105, >105 ppbv etc.) against the ozone concentration measured by the FEM. ------- Ozone Indicator Cards Test/QA Plan Page 23 of 45 Version 1 November 10, 2009 In laboratory testing, ozone will be produced using an Environics Model 6400 ozone generator (transfer standard) that has itself been calibrated against a Dasibi Model 1008 UV calibration photometer. Dry high purity air will be used as the feed gas to the Environics Model 6400 ozone generator. The dry air exiting the ozone source will then be mixed with humidified air to produce the challenge ozone concentrations. The ozone concentrations delivered to the OICs will be monitored in the OIC exposure chamber by the FEM instrument. The ozone concentrations delivered to the OICs will be adjusted depending on the OICs being tested so that ozone concentrations both in the middle and near the edges of the OIC indication ranges are used in testing. For example, with the OIC indication ranges noted in the previous paragraph, ozone challenge concentrations of 30, 50, 70, 90 and 120 ppbv would be suitable. The use of ozone concentrations near the outer limits of OIC ranges will provide information on OIC variability that might not be evident with ozone concentrations centered in the ranges. Ozone concentrations that coincide with the transition points from one OIC range to the next will not be used. The laboratory ozone challenges will be delivered in air at normal temperature and RH conditions (i.e., approximately 22 °C and 50% RH). For each OIC tested, once reference method readings have stabilized, four cards will be exposed in each laboratory testing step, and each OIC will be read by two different users, who record their readings on separate sheets. Those users will not know the ozone concentration being delivered to the OICs, i.e., the ozone concentration will be changed between OIC exposures and will be known only by the Battelle test operator. As Table 2 shows, 20 OIC readings will be recorded by each of two users at each of five ozone levels, producing 200 total data points for comparison of OIC and reference method results. In field testing, SCAQMD personnel will take OICs to established SCAQMD ozone monitoring sites during normal site operations, and expose multiple reagent spots on multiple cards to ambient air while at the sites. Personnel will visually judge each OIC reading, and record it on a data sheet provided by Battelle (an example is shown in Appendix A), along with the date, time, and location of sampling, and the FEM ozone reading, temperature, RH, and wind speed at the site during the OIC exposure period. Potential sites to be used for this purpose by SCAQMD are located in Azusa, Crestline, Glendora, Rubidoux, and Santa Clarita. These sites are all in ------- Ozone Indicator Cards Test/QA Plan Page 24 of 45 Version 1 November 10, 2009 mountain foothill locations, and are frequently subject to elevated ambient ozone levels. The relative locations of the FEM inlet and the OICs during exposure at each site will be documented. In each such site visit up to five such readings will be taken with two OICs simultaneously. The intent of this plan is that 12 such site visits will be conducted, producing up to 60 ozone readings with duplicate OICs (120 OIC readings total). Duplicate OIC readings will be obtained by exposing one or more spots on separate OICs over the same 10-minute period, and recording the readings on pre-established data sheets (Appendix A). The unique OIC number, the reagent spot number (1 through 5), and the visual readings of multiple users (if applicable) will be recorded along with the start-stop times for every OIC reading taken. The results will be entered from the data sheets into Microsoft Excel for evaluation of OIC duplicates (i.e., different OIC numbers, same start-stop times, same user) and user duplicates (i.e., same OIC number and reagent spot number, different users). This effort will be carried out as an in-kind contribution to the verification by SCAQMD, so the actual number of OICs used and readings taken may be subject to other demands on SCAQMD personnel. At a minimum, 6 such site visits, producing up to 30 ozone readings with duplicate OICs (60 OIC readings total) are planned. Also, it is desirable that the OIC visual readings be made by more than one SCAQMD user at the field sites; the feasibility of this may similarly be subject to the demands on SCAQMD personnel. B. 1.1.2 Variability of Readings Variability of OIC readings refers to the consistency, or lack thereof, in visually determined OIC results with a constant ozone concentration. Variability will be assessed using the multiple readings made by each of two users at five ozone concentrations, in the laboratory accuracy testing described in Section B1.1.1. Variability will be expressed as the number of OIC indication ranges into which the OIC readings fall at each of those concentrations. Bl. 1.3Duplication The degree of agreement of ozone measurements made simultaneously on duplicate OICs will be assessed using all data in which duplicate OICs are read by a single user. Similarly, the degree of agreement of ozone measurements made by separate users will be assessed using all data in which the same exposed OIC reagent spot is read by more than one user. These two measures of performance are termed OIC duplication and user agreement, respectively, and are intended to address two types of variation in OIC readings. OIC duplication addresses within-user variation, ------- Ozone Indicator Cards Test/QA Plan Page 25 of 45 Version 1 November 10, 2009 whereas user duplication addresses between-user variation. Within-user variation arises from differences in the color development of different reagent spots that have been exposed under identical conditions. Between-user variation arises from visual perception differences in readings made by different individual users on the same exposed reagent spot. These two forms of duplication are distinguished in testing by unique numbering of OICs and reagent spots, and by separate recording of readings made by different users. In the laboratory testing, data to assess both OIC and user agreement will arise from the tests conducted to assess accuracy (Section B1.1.1). In field testing, such data will arise whenever either SCAQMD or BCLA representatives report visual readings for duplicate OIC cards, or visual readings for the same OIC reagent spot from two different users. Such duplicate readings will be recorded on the field data sheets to be used by SCAQMD and BCLA representatives (see Appendices A and B, respectively). In all cases duplicate readings (whether of duplicate OICs by a single user or of a single OIC by multiple users) will be taken in close succession and under identical lighting conditions immediately after the OIC exposure period. B. 1.1.4Effect of Light Intensity The effect of light intensity on OIC response to ozone will be evaluated in laboratory testing by exposing OICs to a constant ozone concentration under different conditions of illumination. In this test, all reagent spots on two OIC cards will be exposed simultaneously to approximately 60 ppb of ozone in air at 22 °C and 50% RH, with the test conducted under normal indoor illumination and the OICs lying horizontal (flat) during testing. The total light intensity reaching the OICs during the test will be monitored by appropriate commercial meters. The same test will be repeated with the OICs in darkness during O3 exposure, and then repeated once more with the test apparatus illuminated by sunlamps to achieve a light intensity of approximately 100 mW/cm , equivalent to a full overhead sun. The OIC results from the three trials (i.e., 10 readings at each of three light intensities) will be compared to assess whether light intensity affects OIC reading. In all three tests, the same single user will make the visual readings on both OICs under the same normal laboratory illumination conditions. The light source used in the high light intensity test will be a Class A solar simulator (certified to International Electrotechnical Commission 904-9 and American Society for Testing and Materials E927-05 standards). This light source mimics the entire solar spectrum including the low levels of UV-A (320-400nm) and UV-B (280-320 nm) in the solar spectrum at ground level. ------- Ozone Indicator Cards Test/QA Plan Page 26 of 45 Version 1 November 10, 2009 A lamp that provides the solar spectrum and intensity characteristic of an overhead sun at sea level (i.e., 1.0 sun intensity with air mass =1.0) will be used. Light intensity will be monitored using Greenlee Model 93-1065F photometers, or similar, obtained from Battelle's Instrumentation Services Laboratory. Bl.1.5Effect of Ambient Conditions The effect of ambient temperature, RH, and wind speed will be evaluated based on information provided by SCAQMD personnel from the field sites. The OIC and FEM field ozone results used to determine OIC accuracy (Sections B 1.1.1 and B 1.2.1) will be segregated into those showing agreement between OIC and FEM results and those showing disagreement. Then the temperature, RH, and wind speed conditions associated with these two data sets will be compared. HI. 1.6Operational Factors Operational factors associated with use of the OICs will be evaluated based on the comments and observations of all users (Battelle, SCAQMD, and BCLA) in the laboratory and field testing. Such observations may address the convenience of the OICs, their readability under differing conditions, the apparent consistency of OIC readings, and acceptability as a personal monitor. In particular the observations of BCLA users will be important, as those users are likely to be the relatively non-technically trained users for which the OICs are designed. In a part of the field testing that is unique to BCLA's effort, BCLA will conduct trials of the use of each OIC as a personal monitor. Exposure of an OIC worn as a personal monitor will be conducted over the same time period as exposure of other OICs at a field site, so that comparisons of OIC results can be made between personal and non-personal measurements. Also, the BCLA representatives will report on the degree to which they could accept the OIC as a personal monitor to be worn; comments on convenience, annoyance while worn, obstruction of movement, conspicuous appearance, and any other user observations will be obtained. In testing of OICs as personal monitors, the manner in which an OIC is worn by a BCLA user will be controlled to avoid erroneous ozone readings. General recommendations for use of the OICs include: The OIC must be fully opened; Foil covering must be fully removed from the reagent spot, and discarded; ------- Ozone Indicator Cards Test/QA Plan Page 27 of 45 Version 1 November 10, 2009 The OIC must be fully exposed to ambient air without surface proximity that can remove ozone from air; The visual reading must be made immediately after the OIC exposure period. Specific recommendations for use of OICs as personal monitors include: Attach the OIC so that minimal interference results from surface contact; Expose the OIC as a personal monitor simultaneously with exposure of other OICs used as local (i.e., non-personal) air monitors. Placement of the OIC in a horizontal (flat) orientation atop a hat or cap is preferable to attachment in a vertical orientation hanging from clothing. Recommended order of preference of location of an OIC worn as a personal monitor (the first two provide horizontal orientation, the others are vertical): top of cap > bill of cap > sleeve (shoulder) > shirt pocket > lapel . B1.2 Statistical Analysis The semi-quantitative nature of the OIC readings determines the types of statistical comparisons that can be done to evaluate the performance parameters. The planned statistical comparisons are described in the following sections. B. 1.2.1A ccuracy The accuracy of the OIC with respect to the FEM will be assessed as a percentage of readings in the correct OIC indication range, i.e.: Accuracy = (iOIC in Range/Number of Trials) x 100 (1) where OIC in Range is the number of total OIC readings reported as being in the OIC indication range that encompasses the corresponding FEM ozone reading, and Number of Trials is the total number of such comparisons for which FEM readings fell within that range. This calculation of accuracy will be performed with the data from each of the four challenge ozone concentrations in the laboratory testing, and with the data obtained by SCAQMD at field sites. Readings from OICs exposed simultaneously, and readings from OICs read in the laboratory by more than one ------- Ozone Indicator Cards Test/QA Plan Page 28 of 45 Version 1 November 10, 2009 user, will be included in this calculation as separate trials (i.e., all OIC/FEM comparisons will be treated as independent data for this calculation). Bl.2.2Variability of Readings Variability of the OIC readings will be evaluated using only the data from the repeated laboratory trials at five ozone challenge concentrations. For each of those concentrations, variability will be determined as the number of OIC indication ranges into which the user readings fall. That is, at ozone concentration X (e.g.): Variability X= (# of OIC Ranges with Readings at Ozone Concentration X) (2) Bl. 2.3Duplication OIC duplication will be assessed in terms of the percentage of readings in which a single user reports a result in the same OIC indicator range from two duplicate OICs exposed simultaneously (regardless of whether that range agrees with FEM ozone results). OIC duplication will be calculated as: OIC Duplication = (Number Same Range J Number DuplicatesJ) x 100 (3) where Number Same Ranged is the number of cases in which a user reading simultaneously exposed reagent spots on duplicate cards reports the same OIC indication range from each card, and Number Duplicatesd is the total number of cases in which one user reads such duplicate exposed reagent spots. This calculation will be done for the data from the five ozone concentrations in laboratory testing, and separately for any cases of duplicate cards used in the field testing performed by SCAQMD and BCLA. User agreement will be calculated in the same manner, except using data from multiple users reading the same exposed reagent spot on a single OIC, i.e.: User Agreement = (Number Same RangeJNumber Duplicatesu) x 100 (4) where Number Same Rangeu is the number of cases in which two users reading the same exposed reagent spot on a card report the same ODC indication range, and Number Duplicatesu is the ------- Ozone Indicator Cards Test/QA Plan Page 29 of 45 Version 1 November 10, 2009 total number of cases in which two users read the same exposed reagent spot. This calculation will be done for the data from the duplicate users with five ozone concentrations in laboratory testing, and separately for any cases of duplicate users in the field testing performed by SCAQMD and BCLA. HI. 2.4Effect of Light Intensity The effect of light intensity on OIC performance will be assessed using the data from the laboratory testing (section Bl.1.4), by calculating the accuracy, variability, and OIC duplication of readings (Sections B 1.2.1 through Bl.2.3) of the test data at each of the three light intensity conditions. Those results will be compared to indicate whether light intensity has any apparent effect on the OIC performance at a constant ozone concentration. Accuracy or OIC duplication results that differ by more than 20% accuracy or 20% duplication will be taken as evidence of a significant light intensity effect. Similarly, variability results that differ by one or more OIC indication ranges will be taken as evidence of a significant light intensity effect. B.l.2.5Effect of Ambient Conditions The effect of temperature, RH, and wind speed on OIC readings will be assessed by segregating the SCAQMD results into those that are accurate relative to the FEM (as defined in Section B 1.2.1), and those that are inaccurate. The temperature, RH, and wind speed data provided by SCAQMD for these two data sets will then be compared to assess whether the conditions associated with inaccurate OIC readings are significantly different from those associated with accurate OIC readings. Comparison of the data sets for significance of differences will be based on t-test comparisons of means. Should data sets contain small numbers of samples (n < 10), comparison of ranges will be used as an alternative procedure, based on tabulated values of Cn and the equation: H = x ± CnR (5) where |i is the population mean, x is the mean of a small set of samples, and R is the range of values.5 ------- Ozone Indicator Cards Test/QA Plan Page 30 of 45 Version 1 November 10, 2009 B1.3 Reporting The statistical comparisons described above will be conducted separately for each OIC being tested, and user comments on the operational factors will be compiled and reported. The data for each OIC will be kept separate from data for all other OIC's, and no intercomparison of the data from different vendors' OICs will be performed at any time. A verification report will be prepared for each OIC tested, that presents the test procedures and test data, as well as the results of the statistical evaluation of those data. Operational aspects of the OICs will be recorded by testing staff at the time of use, and summarized in the verification report. The verification report will briefly describe the ETV program, the AMS Center, the test equipment and test conditions, and the procedures used in verification testing. The results of the verification test will be stated quantitatively, without comparison to any other OICs tested or comment on the acceptability of the technology's performance. The draft verification report will first be subjected to review by the technology vendor, BCLA, and SCAQMD, and then revised and subjected to a review by EPA and/or other peer reviewers. The EPA comments and the peer review comments will be addressed in further revisions of the report, and the comments and responses will be tabulated to document the peer review process. The reporting and review process will be conducted according to the requirements of the AMS Center QMP.1 B2 REFERENCE SAMPLE COLLECTION Reference sample collection in this verification consists of monitoring ozone in field and laboratory testing using a properly calibrated FEM analyzer. OIC readings of ozone in air will then be compared with the corresponding response of the FEM. The procedures and records of reference method calibrations will be reviewed for both laboratory and field testing as part of the Technical Systems Audit procedure (Section Cl.l). ------- Ozone Indicator Cards Test/QA Plan Page 31 of 45 Version 1 November 10, 2009 B3 SAMPLE HANDLING AND CUSTODY REQUIREMENTS Field data sheets (Appendices A and B) will be filled in by SCAQMD and BCLA representatives. Copies of those sheets will be retained by the respective organization that conducted the field testing, and the originals will be sent to Battelle by tracked shipment (FedEx or similar) for compilation of the data. B4 REFERENCE METHOD The reference method used for ozone determination in this test is the EPA-established UV absorption FEM, implemented for laboratory testing in the form of a Battelle-owned commercial continuous real-time ozone analyzer. In laboratory testing the FEM analyzer will be operated by Battelle staff according to the manufacturers' instructions, including those for warm- up and stabilization time before testing. In field testing, comparable FEM analyzers will be operated by appropriate state or local air quality agencies, with adherence to quality requirements specified for such monitoring in the Code of Federal Regulations. The laboratory FEM ozone analyzer will be operated according to requirements that exceed those for ambient ozone monitoring.6 Specifically, the laboratory FEM ozone analyzer will be calibrated prior to testing, and then subjected to an ozone zero/span check at the start of each day of testing, using the Environics 6400 ozone transfer standard noted in Section A8. This transfer standard itself will be calibrated against the Dasibi Model 1008 UV primary calibration photometer at the start of testing. In that calibration ozone output of the Environics 6400 will be monitored using the Dasibi photometer at several settings of the Environics 6400, and a calibration curve of output ozone concentration versus Environics setting will be established. B5 QUALITY CONTROL REQUIREMENTS Quality of the laboratory reference ozone measurements will be assured by a calibration of the FEM ozone analyzer before any testing, and a daily one-point check of the FEM analyzer at the start of each day of testing, using the ozone transfer standard. The pre-testing calibration slope must equal 1 ± 0.02 and the coefficient of determination (r ) must be greater than 0.95 before testing can proceed. The FEM reading on the one-point daily check must be within 5% of the ozone concentration from the transfer standard before testing can begin on that day. The FEM will be adjusted if necessary to meet that daily requirement. Quality of the field reference measurements will be assured by SCAQMD records of the calibration and maintenance of FEM monitors at the field sites. ------- Ozone Indicator Cards Test/QA Plan Page 32 of 45 Version 1 November 10, 2009 B6 INSTRUMENT/EQUIPMENT TESTING, INSPECTION, AND MAINTENANCE The Thermo Environmental FEM ozone analyzer owned by Battelle and used as the reference ozone measurement will be tested, inspected, maintained, and calibrated per the manufacturer's operating instructions, so as to meet the quality control requirements stated in Section B5. Other equipment such as the temperature and RH monitoring instruments will be obtained from the Battelle Instrument Services Laboratory and will have been calibrated within the past year. B7 INSTRUMENT CALIBRATION AND FREQUENCY Prior to the start of the OIC laboratory testing a multipoint calibration will be performed on the Battelle FEM analyzer using an Environics Model 6400 ozone generator (transfer standard) that has itself been calibrated against a Dasibi Model 1008 UV primary calibration photometer. This calibration will include ozone concentrations in the range of 10 to 150 ppbv; that range spans all the ozone challenge concentrations to be used in laboratory testing and the full useful range of the OICs . The ozone calibration standards will be generated in dry high purity air. Also, on each day of testing the FEM analyzer will be challenged with zero air and a single point ozone span check. Consistent with CFR requirements,6 the daily ozone checks will be in the range of 10 to 100 ppbv and will be supplied to the analyzer through the entire normal sample flow path, including the inlet line, particle filter, and internal analyzer plumbing including the ozone scrubber. However, to assure consistent data quality in laboratory testing of OICs, the daily frequency of these ozone checks greatly exceeds the biweekly frequency required for continuous ambient ozone monitoring.6 B8 INSPECTION/ACCEPTANCE OF SUPPLIES AND CONSUMABLES All materials, supplies, and consumables will be ordered by the Verification Test Coordinator or designee. Where possible, Battelle will rely on sources of materials and consumables that have been used previously as part of ETV verification testing without problems. B9 NON-DIRECT MEASUREMENTS Data published previously in the scientific literature will not be used to evaluate the vendor's technology during this verification test. B10 DATA MANAGEMENT Various types of data will be acquired and recorded electronically or manually by Battelle, SCAQMD, and BCLA during this verification test. Table 3 summarizes the types of data to be ------- Ozone Indicator Cards Test/QA Plan Page 33 of 45 Version 1 November 10, 2009 recorded, how and by whom the data will be recorded, and how the data will be treated or used. All observations relevant to the laboratory testing of the OICs will be documented by Battelle staff in laboratory record books. Results from the laboratory FEM and temperature and RH monitors will be recorded in electronic format. Preprinted data sheets (Appendix A) will be used by SCAQMD to record OIC readings, ozone, temperature, RH, and operator observations in the field. It will also be requested that electronic files be provided by SCAQMD to document ozone, temperature, and RH. Preprinted data sheets (Appendix B) will be used by BCLA to record OIC readings and operator observations in the field. All Battelle LRBs, record books and files are stored indefinitely, either by the Verification Test Coordinator or Battelle's Records Management Office. EPA will be notified before disposal of any files. Records generated by any Battelle staff during the verification test will be reviewed by a Battelle staff member within two weeks of generation, before the records are used to calculate, evaluate, or report verification results. This review will be performed by a Battelle technical staff member involved in the verification test, but not the staff member who originally received or generated the record. The review will be documented by the person performing the review by adding his/her initials and date to the hard copy of the record being reviewed. Data entered from hard copy records into spreadsheets will be reviewed in hard copy within the two-week window. Spreadsheet entries will be checked as part of the ADQ (Section CI.3). In addition, any calculations performed by Battelle staff will be spot-checked by another Battelle technical staff member to ensure that calculations are performed correctly. All such calculations will be done by incorporating Equations 1 to 5 from Section B1.2 into electronic spreadsheets as formulae, and this spot-checking will be done by reviewing the correctness of the entered formulae. The data obtained from this verification test will be compiled and reported independently for each COA tested. ------- Ozone Indicator Cards Test/QA Plan Page 34 of 45 Version 1 November 10, 2009 Table 3. Summary of Data Recording Process Data to Be Recorded Where Recorded How Often Recorded Disposition of Data Dates, times, and details of laboratory test events Battelle record books Start/end of test procedure, and at each change of a test parameter Used to organize/check test results; manually incorporated in electronic spreadsheets as necessary Laboratory FEM QA results Battelle record books, or electronically At FEM calibration or re- calibration Incorporated in verification report as necessary Temperature, RH, light intensity data in laboratory testing Recorded electronically and noted in Battelle record books Start/end of test procedure, and at each change of a test parameter Incorporated in verification report as necessary Laboratory OIC readings Battelle record books At each reading Converted to spreadsheet for statistical analysis and comparisons OIC readings at field sites Field data sheets by SCAQMD or BCLA At each reading Converted to spreadsheet for statistical analysis and comparisons FEM readings, temperature, RH, wind speed at field sites Field data sheets or recorded electronically by SCAQMD Coincident with OIC exposures to ambient air Converted to spreadsheets for statistical analysis and comparisons User observations at field sites Field data sheets by SCAQMD or BCLA At each use of OICs Incorporated in verification report as necessary Field data sheets filled out by SCAQMD or BCLA representatives will be reviewed by the lead SCAQMD or BCLA representative for completeness and consistency within one week after they are filled out. SCAQMD and BCLA will then retain copies of their respective data sheets and send the originals to Battelle. The Battelle Test Coordinator or Quality Manager will review all such data sheets immediately upon receipt. Any corrections needed in how SCAQMD or BCLA representatives fill in the forms will be implemented immediately. Among the QA activities conducted by Battelle QA staff will be an audit of data quality. This audit will consist of a review by the Battelle Quality Manager of at least 10% of the test data. During the course of any such audit, the Battelle Quality Manager will inform the technical staff of any findings and any immediate corrective action that should be taken. If serious data quality problems exist, the Battelle Quality Manager will notify the AMS Center Manager, who is ------- Ozone Indicator Cards Test/QA Plan Page 35 of 45 Version 1 November 10, 2009 authorized to stop work. Once the assessment report has been prepared, the Verification Test Coordinator will ensure that a response is provided for each adverse finding or potential problem, and will implement any necessary follow-up corrective action. The Battelle Quality Manager will ensure that follow-up corrective action has been taken. Field data sheets, laboratory record book entries, and other hard copy data will be converted to Portable Document Format (.pdf) files and submitted to the EPA Quality Manager along with compiled Excel files of test data within two weeks after Battelle's ADQ is completed. The EPA Quality Manager may conduct an ADQ, at her discretion, and appropriate action will be taken in response to any findings. ------- Ozone Indicator Cards Test/QA Plan Page 36 of 45 Version 1 November 10, 2009 SECTION C ASSESSMENT AND OVERSIGHT CI ASSESSMENTS AND RESPONSE ACTIONS Every effort will be made in this verification test to anticipate and resolve potential problems before the quality of performance is compromised. One of the major objectives of this test/QA plan is to establish mechanisms necessary to ensure this. The procedures described in this test/QA plan, which is peer reviewed by a panel of outside experts, implemented by the technical staff and monitored by the Verification Test Coordinator, will give information on data quality on a day-to-day basis. The responsibility for interpreting the results of these checks and resolving any potential problems resides with the Verification Test Coordinator. Technical staff have the responsibility to identify problems that could affect data quality or the ability to use the data. Any problems that are identified will be reported to the Verification Test Coordinator, who will work with the Battelle Quality Manager to resolve any issues. Action will be taken to control the problem, identify a solution to the problem, and minimize losses and correct data, where possible. Independent of any EPA QA activities, Battelle will be responsible for ensuring that the audits described below are conducted as part of this verification test. Cl.l Technical Systems Audits The Battelle Quality Manager will perform a technical systems audit (TSA) at least once during this verification test, in both the laboratory and field testing locations. The purpose of this audit is to ensure that the verification test is being performed in accordance with the AMS Center QMP,1 this test/QA plan, and the published reference method. In the TSA, the Battelle Quality Manager or a designee may review the reference method used, compare actual test procedures to those specified or referenced in this plan, and review data acquisition and handling procedures. In the laboratory, the Battelle Quality Manager may inspect the ozone challenge delivery system, observe the ozone monitoring and test procedures, and review data records. He may also check calibration certifications for test measurement devices. In the field the Battelle Quality Manager may inspect SCAQMD site QA records, observe SCAQMD and BCLA test procedures, and review data records. At his discretion, the Battelle Quality Manager may conduct a TSA at both SCAQMD and BCLA sites in both the Fall 2009 and Summer 2010 field periods (see Section A7.2). A TSA report will be prepared, including a statement of findings and the actions taken to ------- Ozone Indicator Cards Test/QA Plan Page 37 of 45 Version 1 November 10, 2009 address any adverse findings. The EPA AMS Center Quality Manager will receive a copy of Battelle's TSA report. At EPA's discretion, EPA QA staff may also conduct an independent TSA during the verification test. The TSA findings will be communicated to technical staff at the time of the audit and documented in a TSA report. C1.2 Performance Evaluation Audit A PE audit of the ozone measurements will be done to confirm the accuracy of the Battelle- owned Dasibi 1008 UV photometer as the basis for the FEM calibration in this verification. A side-by-side comparison will be done to establish the traceability of the Battelle Dasibi 1008 UV photometer relative to the standard photometer owned by the Ohio EPA, which is also a Dasibi 1008, and which is traceable to the primary ozone standard reference photometer located at EPA Region 5, Chicago, Illinois. In the side-by-side comparison, agreement of the Battelle and Ohio EPA photometers within 2% or 1 ppbv ozone (whichever is larger) will be the target. Should agreement fall outside this tolerance, the Battelle photometer data will be corrected to match the Ohio EPA photometer readings. C1.3 Audit of Data Quality The Battelle Quality Manager will audit at least 10% of the verification data acquired in the verification test. The Battelle Quality Manager will trace the data from initial acquisition, through spreadsheet entry, reduction and statistical comparisons, to final reporting. All calculations performed on the data undergoing the audit will be checked. CI.4 QA/QC Reporting Each assessment and audit will be documented in accordance with Sections 3.3.4 and 3.3.5 of the AMS Center QMP.1 The results of all audits will be submitted to EPA. Assessment reports will include the following: Identification of any adverse findings or potential problems Response to adverse findings or potential problems Recommendations for resolving problems Confirmation that solutions have been implemented and are effective Citation of any noteworthy practices that may be of use to others. ------- Ozone Indicator Cards Test/QA Plan Page 38 of 45 Version 1 November 10, 2009 C2 REPORTS TO MANAGEMENT The Battelle Quality Manager, during the course of any assessment or audit, will identify to the technical staff performing experimental activities any immediate corrective action that should be taken. If serious quality problems exist, the Battelle Quality Manager will notify the AMS Center Manager, who is authorized to stop work. Once the assessment report has been prepared, the Verification Test Coordinator will ensure that a response is provided for each adverse finding or potential problem and will implement any necessary follow-up corrective action. The Battelle Quality Manager will ensure that follow-up corrective action has been taken. The test/QA plan and final report are reviewed by EPA AMS Center QA staff and EPA AMS Center program management staff. Upon final review and approval, both documents will then be posted on the ETV website (www.epa.gov/etv). ------- Ozone Indicator Cards Test/QA Plan Page 39 of 45 Version 1 November 10, 2009 SECTION D DATA VALIDATION AND USABILITY D1 DATA REVIEW, VALIDATION, AND VERIFICATION REQUIREMENTS The key data review requirements for the verification test are stated in Section BIO of this test/QA plan. The QA audits described in Section CI of this document, including the audit of data quality, are designed to assure the quality of the data. Data will be verified for completeness, correctness, and compliance with the procedures as written in this test/QA plan. D2 VALIDATION AND VERIFICATION METHODS Section C of this test/QA plan provides a description of the validation safeguards employed for this verification test. Data validation and verification efforts include the use of an EPA-certified FEM and appropriate ozone calibration equipment, and the performance of TSA and data audits as described in Section C. An audit of data quality will be conducted by the Battelle Quality Manager to ensure that data review and validation procedures were completed, and to assure the overall quality of the data. Any findings will be communicated to technical staff at the time of the audit and documented in a report. D3 RECONCILIATION WITH USER REQUIREMENTS The purpose of this test is to evaluate the performance of OICs. The data obtained should include thorough documentation to allow verification of the performance of each OIC. Potential limitations of the data are primarily expected to take the form of incomplete recording of data on test forms, e.g., card numbers, reagent spot numbers, site or user identification. Such limitations will be minimized by instructions for use of the forms, review of the data forms, and prompt correction of recording procedures. The data review and validation procedures described in the previous sections will assure that data meet these requirements and are accurately presented in the evaluation reports generated from this test. Data for which incomplete recording cannot be resolved will be retained but will be flagged and not used for OIC verification. This test/QA plan and the resulting ETV verification report(s) will be subjected to review by the OIC vendors, BCLA, SCAQMD, EPA, and expert peer reviewers. These reviews will assure that this test/QA plan and the resulting report(s) meet the needs of potential users of OICs. The ------- Ozone Indicator Cards Test/QA Plan Page 40 of 45 Version 1 November 10, 2009 final report(s) will be submitted to EPA in Word and Adobe PDF compliant format and subsequently posted on the ETV website. ------- Ozone Indicator Cards Test/QA Plan Page 41 of 45 Version 1 November 10, 2009 SECTION E REFERENCES 1. Quality Management Plan (QMP) for the ETV Advanced Monitoring Systems Center, Version 7.0, Environmental Technology Verification Program, Battelle, November 2008. 2. Environmental Technology Verification Program Quality Management Plan. Version 3.0, EPA/600/R-08/009, January, 2008. 3. Federal Reference and Equivalent Methods established by Code of Federal Regulations, Title 40, Part 53; a list of commercial devices designated as Reference or Equivalent Methods is available at http://www.epa.gov/ttn/amtic/criteria.html. 4. Leston, A.L., W.M. Ollison, C.W. Spicer, and J. Satola, J. Air Waste Manage. Assoc., 55, 1464-1472, 2005. 5. Statistics in Quantitative Analysis, in Chemical Analysis. H.A. Laitinen, McGraw-Hill, New York, 1960. 6. Code of Federal Regulations, Title 40 Chapter 1 Part 58 Appendix A - Quality Assurance Requirements for SLAMS, SPMs, and PSD Air Monitoring, Environmental Protection Agency, July 1, 2009 edition. ------- Ozone Indicator Cards Test/QA Plan Page 42 of 45 Version 1 November 10, 2009 Appendix A Field Data Sheet for Use by SCAQMD ------- Ozone Indicator Cards Test/QA Plan Page 43 of 45 Version 1 November 10, 2009 Field Data Sheet for Use by SCAQMD in Verification of Ozone Indicator Cards (OICs) Date Field Site User(s) Initials 1) 2) OIC Reagent Spot Time Reading FEM O3 Temp RH Wind Spd. Number Number Start Stop Init/Fnla (ppb) (F) (%) (mph) / / / / / / / / / / a: Init = initial reading when foil first removed from spot; Fnl = reading after spot exposed to air. User Observations: ------- Ozone Indicator Cards Test/QA Plan Page 44 of 45 Version 1 November 10, 2009 Appendix B Field Data Sheet for Use by BCLA ------- Ozone Indicator Cards Test/QA Plan Page 45 of 45 Version 1 November 10, 2009 Field Data Sheet for Use by BCLA in Verification of Ozone Indicator Cards (OICs) Date Field Site User(s) Initials 1) 2) OIC Reagent Spot Time Reading Personal Monitor Reading Number Number Start Stop User la Location User 2a __/ /_ __/ /_ __/ /_ __/ /_ __/ /_ __/ /_ __/ /_ _/ /_ _/ /_ / / a: Record initial reading (when foil first removed from reagent spot) and final reading (after exposure to air). Additional User Observations: ------- |