O Bafleiie . . . Putting Technology To Work Environmental Technology Verification Program Advanced Monitoring Systems Pilot Test/QA Plan for Verification of On-Board Vehicle Emissions Monitors et^Wetv ------- TEST/QA PLAN FOR VERIFICATION OF ON-BOARD VEHICLE EMISSIONS MONITORS April 26, 2001 Prepared by Battelle 505 King Avenue Columbus, OH 43201-2693 ------- APPROVAL Page ii Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Vendor Company (Print) Signature Date ------- Page iii Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 DISTRIBUTION Battelle has provided a copy of this Test/QA Plan to the following individuals. EPA Robert Fuerst, Pilot Manager, Elizabeth Hunike, Pilot Quality Manager, Elizabeth Betz, Pilot Quality Manager, Battelle Karen Riggs, Pilot Manager Thomas Kelly, Verification Testing Leader Charles Lawrie, Quality Manager Kenneth Cowen, Verification Testing Coordinator Jeffrey Myers, Verification Testing Coordinator Peer Reviewers Jeff Cook, California Air Resources Board Clifford Glowacki, Casting Emissions Reduction Program Donald Stedman, University of Denver Test Facility Walt Dudek, Automotive Testing Laboratory ------- Page iv Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Table of Contents 1. Introduction 1 1.2. Test Objective 1 1.3 Test Applicability 2 2. Technology Description 2 3. Verification Approach 3 3.1. Scope of Testing 3 3.2. Experimental Design 4 3.3. Reference Methods 7 3.4. Test Facility 7 3.5. Roles and Responsibilities 8 3.5.1. Battelle 8 3.5.2. Vendors 12 3.5.3. EPA 12 3.5.4. Test Facility 13 4. Test Procedures 14 4.1. Vehicle Recruitment and Inspection 14 4.2. Monitor Installation 15 4.3. Dynamometer Testing 15 4.4. Road Testing 17 5. Quality Assurance/Quality Control 18 5.1. Calibration 18 5.2. Audits 18 5.2.1. Pre-test Facility Audit 18 5.2.2. Technical Systems Audits 19 5.2.3. Performance Evaluation Audits 19 5.2.4. Audits of Data Quality 20 5.3. Reporting of Audits 20 5.4. Corrective Action 21 6. Data Handling and Reporting 21 6.1. Data Review 21 6.2. Statistical Calculations 22 6.2.1 Bias 22 6.2.2. Precision 23 ------- Page v Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 6.2.3 Other Comparisons 25 6.3. Reporting 25 7. Health and Safety 26 8. References 26 List of Figures 1. Organizational Chart for On-Board Vehicle Emissions Monitor Verification Test 9 List of Tables 1. Summary of Chassis Dynamometer Runs 5 2. Schedule for Dynamometer Runs 16 ------- Page vi Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Acronyms AMS - Advanced Monitoring Systems CL - chemiluminescence analyzer CO - carbon monoxide C02 - carbon dioxide EPA - United States Environmental Protection Agency ETV - Environmental Technology Verification FTP - Federal Test Procedure GC - gas chromatography g/mi - grams per mile FID - flame ionization detector LDV - light duty vehicle LRB - laboratory record book mph - miles per hour NDIR - non-dispersive infrared spectrometry NMHC - non-methane hydrocarbons NOx - nitrogen oxides OBD - on-board diagnostic OEM - on-board emissions monitor QA - quality assurance QC - quality control QMP - quality management plan rpm - revolutions per minute THC - total hydrocarbons ------- Page 1 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 1. Introduction 1.1. ETV Background This test/quality assurance (QA) plan provides detailed procedures for a verification test of monitors that continuously measure the concentration of various gaseous components of vehicle emissions under normal operating conditions (i.e., on-road driving), and may provide quantitative vehicle emissions data (e.g. g/mi emission rates) based on live engine data scanning. The verification test will be conducted under the auspices of the U.S. Environmental Protection Agency (EPA) through its Environmental Technology Verification (ETV) program. The purpose of the ETV program is to provide objective and quality assured performance data on environmental technologies, so that users, developers, regulators, and consultants can make informed decisions about these technologies. ETV verification does not imply approval, certification, or designation by EPA, but rather provides a quantitative assessment of the performance of a technology under specified test conditions. The verification test will be coordinated by Battelle, of Columbus, Ohio, who is EPA's partner in the ETV Advanced Monitoring Systems (AMS) pilot through cooperative agreement CR 826215-01-1. The scope of the AMS pilot covers verification of monitoring technologies for contaminants and natural species in air, water, and soil. In performing the verification test, Battelle will follow the procedures specified in this test/QA plan, and will comply with the data quality requirements in the "Quality Management Plan for the ETV Advanced Monitoring Systems Pilot" (QMP).1 1.2. Test Objective The purpose of this verification test is to evaluate the performance of on-board vehicle emissions monitors (OEM) under realistic operating conditions. Specifically, these monitors will ------- Page 2 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 be evaluated by comparisons with chassis dynamometer test results to assess accuracy and reproducibility, and through operation during normal on-road driving to assess performance under realistic conditions. The effect of temperature on the performance of the OEMs will be assessed by varying the dynamometer cell conditions under which selected tests are performed. The unit-to-unit reproducibility of the OEM will be evaluated from comparisons between duplicate monitors during both the dynamometer tests and the on-road driving tests. General performance characteristics of these OEMs, such as reliability and ease of use, will also be evaluated through observations by the test personnel. 1.3 Test Applicability This test/QA plan is applicable to the verification testing of vehicle emissions monitors which can be operated on-board a vehicle under normal driving conditions. The OEMs to be tested are capable of providing real-time concentration measurements of several key pollutants in vehicle exhaust, and in some cases, quantitative emission rates (e.g., g/mi) as calculated from the concentration measurements and live engine parameter data. In accordance with the intent of the ETV program, the OEMs to be tested are commercially available, and not developmental products or prototypes. 2. Technology Description This document details the procedures for the verification testing of technologies which can be mounted in a variety of light or heavy duty vehicles and are capable of monitoring emissions from these vehicles under routine driving conditions. These OEMs are portable, weighing up to approximately 60 lbs., and can be installed in the passenger seat or trunk of most ------- Page 3 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 vehicles without modifications to the vehicle. The OEMs can measure exhaust emissions of vehicles during their regular operation. In some cases the OEMs also simultaneously collect live engine data from vehicles equipped with on-board diagnostic (OBD) ports, to monitor fuel consumption as well as operating characteristics of the vehicles. These OEMs use the live engine data to determine real-time emission rates. The basic component of the OEMs to be tested in this verification test is a multi-gas analyzer for measuring the composition of the vehicle exhaust. Additionally, these OEMs may include other components such as a laptop computer to collect, process, and store data, or an engine diagnostic scanner to monitor engine data. For those OEMs with engine scanning capabilities, the computer uses live engine data to compute exhaust mass flow, which, when multiplied by the measured concentrations of exhaust gases, yields grams per second data. Grams per mile emissions are then calculated from the vehicle speed and time data. The primary components of the OEMs are contained in a single case, which can be installed in the passenger seat of most domestic cars and trucks. The OEMs are designed for automatic operation such that no user attention is needed during operation. 3. Verification Approach 3.1. Scope of Testing The objective of this test is to evaluate the performance of OEMs under realistic operating conditions. It is not the intent of this test to determine emissions data for the vehicles used in the test. Rather, the intent is to establish the performance capabilities of these OEMs under normal driving conditions (i.e., on the road) as well as in standard vehicle test cycles using dynamometers. To achieve this goal, this verification test will involve two phases. The first phase of testing will involve comparisons between the OEMs and the current standard for vehicle ------- Page 4 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 emissions monitoring, namely dynamometer testing. The second phase of verification will involve performance evaluation of the OEMs during on road testing. In all tests two duplicate units of each OEM will be operated side-by-side. 3.2. Experimental Design This section describes the experimental design for this OEM verification test. The approach is based on the primary objectives of this test: to assess the overall accuracy (i.e., bias and precision) of an OEM relative to a chassis dynamometer, and to assess OEM performance under real-world conditions. The approach to the verification test is summarized below, and the statistical methodology for establishing bias and precision are described in Section 6.2. For this test, three gasoline-powered light duty vehicles (LDV) will be recruited by the test facility for both chassis dynamometer testing and road testing. The vehicles will be chosen to include current models of vehicles that popular in the on-road fleet. Furthermore, the vehicles will be chosen by the test facility such that they include a wide range of emission rates and engine sizes (i.e., 4, 6, 8 cylinder). The vehicles will also be chosen such that both the mass flow, and speed density methods used by the engine diagnostic system to determine exhaust mass are included in the test. In order to establish intra-method precision (i.e., unit-to-unit relative error), it is necessary to include duplicate OEMs as part of the study design (see Section 6.2). This approach is analogous to the use of co-located monitors for establishing the precision of ambient air monitoring sampling and analytical techniques. Duplicate OEMs will be operated in both phases of the verification test. In the first phase of the verification test, the vehicles will be operated on a chassis dynamometer and the vehicle emissions will be monitored by both the OEMs being verified and by laboratory reference methods. Given the goal of evaluating OEM performance under real- world conditions, it is important to perform an array of dynamometer test runs. Consequently, ------- Page 5 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 the vehicles will each be operated on two test cycles: the Federal Test Procedure (FTP),2 and the US062 cycle. The use of different vehicle emission rates, along with the different test cycles, will provide a range of real-world conditions under which bias and precision results are to be obtained. Three runs under each test scenario (i.e., under each test cycle and vehicle combination) will be conducted. There are two important reasons for including multiple runs in the design of the test. First, multiple runs provide information on test-to-test repeatability. Such data are essential for establishing the reliability of the reference method results, which are treated as the "truth" or "gold standard" to which OEM results are to be compared. Second, triplicate runs allow for statistically testing whether or not interactions between vehicle type and test cycle have an impact on observed bias and precision. For example, it may be the case that levels of bias and/or precision differ from vehicle to vehicle, but only when the FTP test cycle is run and not the US06. As another example, one vehicle type may show consistent bias and precision across both test cycles, while the other two vary in bias and precision depending on test cycle. Without multiple runs, the statistical significance of such interaction cannot be determined. A summary of the dynamometer runs to be performed in this verification test is provided in Table 1. Table 1. Summary of Chassis Dynamometer Runs Test cycle Vehicle 1 Vehicle 2 Vehicle 3 Total FTP 1 3 3 3 9 US06 3 3 3 9 Total 6 6 6 18 1 FTP produces multiple bags, so more than 18 observations will be obtained for bag-level calculations. ------- Page 6 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 After the dynamometer runs shown in Table 1 are completed, a series of four US06 test cycles will be performed on the vehicle with mid-range emissions. In this series, a single US06 dynamometer run will be performed with the vehicle accessories off at each of three different temperatures (i.e., 30°F, 75°F, and 100°F). The fourth US06 run will be performed at 100°F with the vehicle's air conditioner operating at maximum capacity, to assess whether use of the vehicle's accessories influences the performance of the OEMs. For all the dynamometer runs, the test facility will measure vehicle emissions by the reference methods described in Section 3.3. The results obtained from these reference methods will be used as the basis of comparison for establishing bias. During each dynamometer run the vehicle emissions will be monitored in real time by the reference methods and by the duplicate OEMs, for total hydrocarbons (THC), carbon monoxide (CO), carbon dioxide (C02), and nitrogen oxides (NOx). Methane will be monitored by the reference method at the bag level and used to determine non-methane hydrocarbons (NMHC) as needed. Bias and precision will be determined independently for each of these analytes. The performance of the two duplicate OEMs will be based on comparisons between the test level results (e.g., average concentration or g/mi) from each of the duplicate OEMs and the results from the reference methods. These comparisons will be made primarily on the test level, and also on the per-bag level (e.g., the three bags comprising the FTP schedule). Graphical comparisons of the second-by-second data will be used as a secondary comparison to illustrate the transient response of the OEMs. In the second phase, duplicate on-board monitors will be installed in a test vehicle and the vehicle will be driven over two different routes for at least 15 minutes each. The two routes will be different in nature such that one includes predominantly stop-and-go traffic, and the other includes predominantly sustained high speed traffic. While the test vehicle is driven over these two routes, second-by-second data will be collected by the duplicate on-board emissions monitors. Results from the duplicate monitors will be compared to establish the unit-to-unit reproducibility of the OEMs being verified. ------- Page 7 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Furthermore, general issues of performance, including reliability and ease of use, will be assessed based on observations recorded during the verification test, and will be reported for the OEMs in addition to the overall cost of the monitors. 3.3. Reference Methods During this verification test, various analytical methods will be used by the test facility to measure the concentrations of THC, CH4 CO, C02, and NOx in vehicle emissions. THC concentrations will be measured using a flame ionization detector (FID), CH4 will be determined using a gas chromatography (GC) with FID, CO and C02 concentrations will be determined by non-dispersive infrared spectroscopy (NDIR), and NOx concentrations will be measured using a chemiluminescence (CL) analyzer. These methods are described in 40 CFR Part 862 and will be the reference methods for this verification test. Results from these methods will serve as the basis of comparison for evaluating the accuracy of the on-board emissions monitor. These analyses, with the exception of CH4, will be performed both in real-time and on collected bag samples. CH4 will be determined only by analysis of collected bag samples. Non-methane hydrocarbon (NMHC) concentrations will be deduced at the bag level from the difference between the bag level THC and CH4 readings. 3.4. Test Facility The test facility to be used for this verification test will be a recognized emissions testing laboratory with facilities appropriate for chassis dynamometer testing according to 40 CFR Part 86. The test facility will have standard operating procedures in place for the dynamometer runs and laboratory analyses to be performed in this verification and will have trained personnel capable of performing these activities according to those standard procedures. Documentation of ------- Page 8 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 the staff qualifications will be provided to Battelle in the form of training records prior to test initiation. 3.5. Roles and Responsibilities The verification test will be coordinated and supervised by Battelle personnel and will be conducted at a recognized test facility with documented QA procedures in place. Staff from the test facility will participate in this test by operating the dynamometer, and providing the reference measurements. Vendor representatives will install, maintain, and operate their respective technologies throughout the test. Quality assurance oversight will be provided by the Battelle Quality Manager, and, at her discretion, the EPA Pilot Quality Manager. The organizational chart below shows the individuals from Battelle, the vendor companies, EPA, and the test facility who will have responsibilities in the verification test. The specific responsibilities of these individuals are detailed in Figure 1. 3.5.1. Battelle The Verification Test Coordinator will have the overall responsibility for ensuring that the technical, schedule, and cost goals established for the verification test are met. The Verification Test Coordinator will: Prepare the draft test/QA plan, verification reports, and verification statements Revise the draft test/QA plan, verification reports, and verification statements in response to the reviewers' comments Coordinate testing at testing site Ensure that all quality procedures specified in the test/QA plan and in the QMP are followed ------- Page 9 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Battelle Quality Manager Test Facilities Management Battelle Management Battelle AMS Pilot Manager Battelle Verification Test Leader Battelle Verification Test Coordinator Battelle Staff Statistician EPA Pilot Manager EPA Pilot Quality Manager Vendor Figure 1. Organizational Chart for On-Board Vehicle Emissions Monitor Verification Test ------- Page 10 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Respond to any issues raised in assessment reports and audits, including instituting corrective action as necessary Serve as the primary point of contact for vendor and test facility representatives Establish a budget for the verification test and monitor staff effort to ensure that the budget is not exceeded Ensure that confidentiality of vendor information is maintained. The Verification Testing Leader for the AMS pilot will provide technical guidance and oversee various stages of the verification test, and will: Support the Verification Test Coordinator in preparing the test/QA plan and organizing the testing Review the draft test/QA plan Review the draft verification reports and statements Ensure that confidentiality of vendor information is maintained. Battelle's AMS Pilot Manager will: Review the draft test/QA plan Review the draft verification reports and statements Coordinate distribution of the final test/QA plan, verification reports, and verification statements Ensure that necessary Battelle resources, including staff and facilities, are committed to the verification test Ensure that vendor confidentiality is maintained Support the Verification Test Coordinator in responding to any issues raised in assessment reports and audits ------- Page 11 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Maintain communication with EPA's pilot and quality manager Battelle will provide a Staff Statistician who will support statistical and data analysis activities for this verification test. As needed, the Staff Statistician will: Assist in the conversion of verification data from electronic spreadsheet format to appropriate file format for statistical evaluation Support the Verification Test Coordinator in performing statistical calculations specified in this test/QA plan on the verification data Provide results of statistical calculations and associated discussion for the verification reports Support the Verification Test Coordinator in responding to any issues raised in assessment reports and audits related to statistics and data reduction. Battelle's Quality Manager for this verification test will: Review the draft test/QA plan Conduct a technical systems audit once during the verification test Audit at least 10% of the verification data Prepare and distribute an assessment report for each audit Verify implementation of any necessary corrective action Issue a stop work order if self audits indicate that data quality is being compromised; notify Battelle AMS Pilot Manager if stop work order is issued Provide a summary of the audit activities and results for the verification reports Review the draft verification reports and statements Have an overall responsibility for ensuring that the test/QA plan and QMP are followed ------- Page 12 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Ensure that Battelle management is informed if persistent quality problems are not corrected Interface with EPA's Pilot Quality Manager. 3.5.2. Vendors Vendor representatives will: Review the draft test/QA plan, and provide comments and recommendations Approve the revised test/QA plan Work with Battelle to commit to a specific schedule for the verification test Provide duplicate commercial ready monitors for testing Provide an on-site operator(s) throughout the verification test period to install the monitors in the test vehicles, and operate and maintain the monitors during testing Remove monitors and other related equipment from test facility upon completion of the verification test Review and comment upon their respective draft verification report and statement. 3.5.3. EPA EPA's responsibilities in the AMS pilot are based on the requirements stated in the "Environmental Technology Verification Program Quality and Management Plan of the Pilot Period (1995-2000)" (QMP)3. The roles of the specific EPA staff are as follows: EPA's Pilot Quality Manager will: Review the draft test/QA plan ------- Page 13 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Perform, at her option, one external technical system audit during the verification test Notify the Battelle AMS Pilot Manager to facilitate a stop work order if external audit indicates that data quality is being compromised Prepare and distribute an assessment report summarizing results of external audit, if performed Review draft verification reports and statements. EPA's Pilot Manager will: Review the draft test/QA plan Approve the final test/QA plan Approve the final verification reports Review the draft verification statements. 3.5.4. Test Facility The responsibilities of the test facility are: Assist in developing the test/QA plan for the verification test Allow facility access to vendor, Battelle, and EPA representatives during the scheduled verification test including set-up and tear-down operations Select, secure, and operate vehicles for the dynamometer and road testing • Perform all reference emissions measurements Provide all test data to Battelle electronically, in mutually agreed upon format Assist vendor in installation of the OEMs in the test vehicles Perform dynamometer runs and associated vehicle preconditioning according to the procedures and schedule described in this test/QA plan ------- Page 14 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 Provide EPA and Battelle staff access to and /or copies of appropriate quality assurance documentation of test equipment and procedures Assist in Battelle's reporting of the test facility's QA/QC procedures Review portions of the verification report to assure accurate descriptions of the test facility operations, and to provide technical insight on verification results Provide safety instructions to test and QA personnel for operations at the test facility. 4. Test Procedures 4.1. Vehicle Recruitment and Inspection Three gasoline-fueled vehicles will be recruited and will be inspected by the test facility to ensure suitability for use in the verification test. The test vehicles should include currently popular models which are representative of the on-road fleet. The vehicles should be in good working condition, however, at least one should be a high exhaust emitter, based on the experience of the test facility. The recruited vehicles must have on-board diagnostic ports which are compatible with the OEMs being tested, and should be capable of providing data sufficient to determine vehicle speed in miles per hour (mph), engine speed in revolutions per minute (rpm), and engine torque (or its surrogate). The vehicle identification number for each vehicle will be recorded by Battelle staff in a laboratory record book (LRB) in addition to a general description of the vehicle (i.e., make, model, year, etc.). The vehicles will be inspected for fuel and exhaust leaks by the test facility prior to testing. Any required vehicle repairs will be documented in the LRB. ------- Page 15 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 4.2. Monitor Installation The on-board emissions monitors to be verified will be installed by a vendor representative who will ensure that each monitor is calibrated and operating properly before testing begins on each day of testing. The duplicate OEMs will be installed with appropriate plumbing to split the exhaust stream for analysis by the on-board emissions monitors and by the test facility. A leak check will be performed before road testing and each series of dynamometer runs to ensure the integrity of the exhaust sampling assembly. Any observed leaks will be repaired before testing begins. The vehicle battery will be used to power one of the two OEMs and a secondary supply (independent of the vehicle battery) will be used to power the other OEM. The installation activities (including on-site calibration, repairs, etc.) will be documented by Battelle staff in the LRB. Observations regarding installation time and simplicity, ease of use, practicality, passenger safety, etc., in the verification report will be based on the installation of a single unit. 4.3. Dynamometer Testing Dynamometer runs will be performed according to the schedule shown in Table 2, and will be conducted with the vehicle accessories off, except where noted. Documentation of the run conditions will be performed by the test facility and will be in accordance with 40 CFR Part 86. This documentation will be provided to Battelle. As this test is not designed to determine emission rates for the test vehicles, strict adherence to soak and preconditioning procedures described in 40 CFR Part 86 is not necessary. However, conditions should be consistent for replicate runs of each test cycle. After the vehicle soak (12-36 hours), the test vehicle will be placed on the dynamometer and prepared for testing. An FTP cycle will be performed with the intent of immediately performing a US06 cycle within ------- Page 16 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 10 minutes of the completion of the FTP test. If the US06 cycle is not started within 10 minutes of completion of the previous cycle, the first 505 seconds of the FTP driving cycle will be performed to condition the vehicle. Alternate FTP and US06 cycles will be performed in sequence on the three vehicles on each of three test days. On the fourth day of testing a series of three US06 test cycles will be performed including one at each of the following temperatures: 30°F, 75°F, and 100°F. These tests will be conducted using the vehicle with the mid-range emissions as established by the previous testing. After this sequence of temperature tests, an additional US06 cycle will be performed at 100°F with the vehicles air conditioner operating at maximum capacity. Table 2. Schedule for Dynamometer Runs Day 1 Day 2 Day 3 Day 4 Vehicle 1 - FTP Vehicle 1 - FTP Vehicle 1 - FTP Vehicle 2 - US06 @ 30°F Vehicle 1 - US06 Vehicle 1 - US06 Vehicle 1 - US06 Vehicle 2 - US06 @ 75°F Vehicle 2 - FTP Vehicle 2 - FTP Vehicle 2 - FTP Vehicle 2 - US06 @ 100°F Vehicle 2 - US06 Vehicle 2 - US06 Vehicle 2 - US06 Vehicle 2 - US06 w/AC Vehicle 3 - FTP Vehicle 3 - FTP Vehicle 3 - FTP Vehicle 3 - US06 Vehicle 3 - US06 Vehicle 3 - US06 For each driving cycle, the exhaust emissions and engine activity data will be monitored by both the test facility reference methods and the duplicate OEMs. The test facility will record and report the data on THC, CH4, CO, C02, and NOx emissions at the test, bag, and second-by- second level. Similarly, the OEM will record THC/NMHC, CO, C02, NOx, and 02 at the test, bag, and second-by-second level. Summary values of the second-by-second values will be compared with the corresponding bag values to assess agreement for the reference measurements ------- Page 17 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 of C02, NOx and THC. Agreement between the two values should be within 5.0%, 10%, and 15%), respectively. Tests not meeting these criteria will be reviewed as to their validity or impact on the verification results and repeated if possible and if necessary. 4.4. Road Testing Each of the three test vehicles used in the dynamometer tests described above will be driven on two separate routes over public roads while the duplicate OEMs record second-by- second data for THC/NMHC, CO, C02, 02, and NOx. Engine data will be recorded either by the OEMs being tested or by a laptop computer. Weather conditions and observations concerning traffic and vehicle operation will be recorded in a LRB by the vehicle operator. The vehicles will begin the road testing with a full tank of suitable locally available gasoline and will complete the two driving routes in succession (i.e., on the same trip). The routes to be driven should be such that they include the following conditions: a) at least 15 minutes of stop and go traffic through a central business district b) at least 15 minutes of sustained high speed driving on a freeway. Tests routes will be consistent from vehicle to vehicle and from test to test (i.e., different OEM verification tests). Effort will be made to conduct testing under similar driving conditions (i.e., time of day, weather conditions) for the verification tests of different OEMs. ------- Page 18 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 5. Quality Assurance/Quality Control 5.1. Calibration The dynamometer and laboratory instrumentation to be used in this verification test will be calibrated by the test facility according to the standard operating procedures and schedules in place at the test facility. These calibration specifications must meet or exceed those described in 40 CFR 86.2 Documentation of these calibrations will be provided to Battelle by the test facility prior to test initiation. If not required by the reference methods or by the standard operating procedures of the test facility, calibration verifications of specific instrumentation will be performed at the request of Battelle during the verification test. The results of the calibration verifications will be provided to Battelle. 5.2. Audits Independent of test facility and EPA QA activities, Battelle will be responsible for ensuring that the following audits are conducted as part of this verification test. 5.2.1. Pre-test Facility Audit At least two weeks prior to verification testing, the Verification Test Coordinator and/or the Battelle Quality Manager may conduct an audit of the test facility chosen to conduct the verification test. If performed, this audit will be conducted to ensure that the test facility has the equipment necessary to perform the verification test and that a satisfactory QA/QC program is implemented at the test facility. The audit should include at least a tour of the dynamometer ------- Page 19 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 facilities and review of appropriate standard operating procedures and calibration records. If possible the audit will also include observation of on-going dynamometer testing. 5.2.2. Technical Systems Audits Battelle's Quality Manager will perform a technical systems audit at least once during this verification test. The purpose of this audit is to ensure that the verification test is being performed in accordance with the AMS QMP, this test/QA plan, referenced methods, and any standard operating procedures used by the test facility. In this audit, the Battelle Quality Manager will review the reference methods used, compare actual test procedures to those specified or referenced in this plan, and review data acquisition and handling procedures. This effort will include reviewing the actual procedures used at the test facility for compliance with this test/QA plan and with the standard operating procedures for the test facility. A Technical Systems Audit (TSA) report will be prepared, including a statement of findings and the actions taken to address any adverse findings. The EPA Pilot Quality Manager will receive a copy of Battelle's TSA report. At EPA's discretion, the EPA Pilot Quality Manager may also conduct an independent TSA of the verification testing procedures. 5.2.3. Performance Evaluation Audits A performance evaluation audit will be conducted to assess the quality of the reference measurements made in this verification test. This audit will address only the emissions measurements provided by the reference methods. The audit will be performed by analyzing a NIST-traceable calibration gas standard that is independent of those used by the test facility during the testing. The acceptance criteria for the results of this audit are identical to those ------- Page 20 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 already in place at the test facility for calibration verification. This audit will be performed once during the verification test. 5.2.4. Audits of Data Quality Battelle's Quality Manager will audit at least 10 percent of the verification data acquired during the verification test. The Battelle Quality Manager will trace the data from initial acquisition, through reduction and statistical comparisons, and to final reporting. All calculations performed on the data undergoing the audit will be checked. 5.3. Reporting of Audits Each audit will be documented in accordance with the AMS pilot QMP.1 Audit reports will include the following: Identification of any adverse findings or potential problems Corrective actions that address adverse findings or potential problems Confirmation by Battelle's Quality Manager that the corrective actions have been implemented and are effective Citation of any noteworthy practices that may be of use to others. All audit reports will be reviewed by the AMS Pilot Manager, and Verification Testing Leader. A copy will be sent to the EPA Pilot Quality Manager and the EPA Pilot Manager. ------- Page 21 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 5.4. Corrective Action The Battelle or EPA Pilot Quality Manager during the course of any audit will identify to the technical staff performing experimental activities any immediate corrective action that should be taken. If serious quality problems exist, the Battelle Quality Manager is authorized to stop work. Once the audit report has been prepared, the Verification Test Coordinator will ensure that a response is provided for each adverse finding or potential problem, and will implement any necessary follow-up corrective action. The Battelle Quality Manager will ensure that follow-up corrective action has been taken. 6. Data Handling and Reporting 6.1. Data Review Data generated by the test facility and vendors in the verification test will be provided to Battelle and will be reviewed by the Verification Test Coordinator before they are used to calculate, evaluate, or report verification results. These data will include electronic data; entries in laboratory record books; operating data from the test facility; and equipment calibration records. The review will be documented by the person performing the review by adding his/her initials and date to a hard copy of the record being reviewed. This hard copy will be placed in the files of this verification test by the Verification Test Coordinator. In addition, data calculations performed by Battelle will be spot-checked by Battelle technical staff to ensure that calculations are performed correctly. Calculations to be checked include determination of accuracy, intra-method precision, and other statistical calculations as identified in Section 6.2 of this test/QA plan. ------- Page 22 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 6.2. Statistical Calculations Performance verification is based, in part, on statistical comparisons of the average concentration results or g/mi results from the on-board emissions monitor to results from the reference methods. A summary of the calculations to be made is given below. 6.2.1 Bias The bias of each of the duplicate OEMs will be assessed at the test level based on the percent difference between the average concentration measurements or the g/mi emission rates from the OEM relative to the reference method. For each individual dynamometer run the percent difference, dt, between the OEM and the reference method results will be calculated as: Z-X, d. = x 100 X, (1) where Y, represents the test level results from the OEM, and Xt represents the test level results of the reference method for a given analyte. The average, I), and standard deviation, 5, of these individual bias results will be calculated from: Id, and D i=i n (2) V £d? -(«r>2) v 7 = 1 ' (3) n- 1 ------- Page 23 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 where n is the total number of dynamometer runs. The standard deviation and average difference will be used to calculate the upper (UL) and lower (LL) 95% confidence limits for the bias of each monitor according to: UL = D+ t0 975 0) and LL — D — ^0.975 (**0 where t0975 is the 0.975 quantile of the Student's t distribution with n-1 degrees of freedom. Bias will be calculated independently for each of the duplicate monitors and each analyte. Additionally, bias will be calculated independently for each vehicle and for each test cycle (i.e., FTP, US06). (4) (5) 6.2.2. Precision Intra-method precision will be calculated based on the percent difference in the readings of the duplicate monitors relative to the mean of the readings, as shown below: Y-Y d. = x 100 (6) ' (X + Y,)/2 where Y, and Y \ are the test level results for a given analyte from the two duplicate monitors for each test cycle i. The coefficient of variation, CVh for each dynamometer run and vehicle will be calculated according to Equation (7). CK = d\ 42 (7) ------- Page 24 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 The individual coefficients of variation for all test cycles and vehicles will be pooled according to Equation (8) to determine the overall precision of the monitors. CV-- 1 t(cr,)2 (8) t = 1 n The upper (UL) and lower (LL) 90% confidence limits for the monitor's CV are given by, ll=cv .l—p— m X0.95, n and, UL = CV n Xo.05 ,n (10) 2 2 where n is the number of degrees of freedom, and %0 95 n and %0 05 n are the 0.95 and 0.05 quantiles, respectively, of the c2 distribution with n degrees of freedom. Precision will be assessed independently for each analyte, as well as for each vehicle and each test cycle. Supplemental comparisons will be made at the second-by-second level to determine the instantaneous unit-to-unit reproducibility of the duplicate monitors. As with the test level results, these comparisons will be made based on a percent difference calculation. ------- Page 25 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 6.2.3 Other Comparisons Second-by-second data from the OBD port and the on-board emissions monitor will be compared graphically to illustrate temporal correlations between the vehicle operational parameters and the measured components in the vehicle exhaust. Likewise, second-by-second data from the reference analyzers will be compared visually against those from the OEM to illustrate temporal correlations. No statistical evaluations will be made of these second-by- second comparisons owing to likely differences in the lag times and response times between the reference analyzers and the OEMs. For the on road tests, second by second comparisons will be made between the results of the duplicate analyzers. 6.3. Reporting The statistical comparisons that result from each of the tests described above will be conducted separately for each of the two units of each technology being verified, and information on the additional performance parameters will be compiled and reported. Separate verification reports will then be prepared, each addressing a technology provided by one commercial vendor. Each report will show separate verification results from the two units undergoing testing, along with calculations of the unit-to-unit reproducibility of the technology. For each test conducted in this verification, the verification report will present the test procedures and test data, as well as the results of the statistical evaluation of those data. The verification report will briefly describe the ETV program and the AMS pilot, and will describe the procedures used in verification testing. These sections will be common to each verification report resulting from this verification test. The results of the verification test will then be stated quantitatively, without comparison to any other technology tested, or comment on the acceptability of the technology's performance. The preparation of draft verification reports, the review of reports by vendors and others, the revision of the reports, final approval, and the ------- Page 26 of 26 Test/QA Plan for Verification of On-Board Emissions Monitors Version: 1.0 distribution of the reports, will be conducted as stated in the Generic Verification Protocol for the Advanced Monitoring Systems Pilot.4 Preparation, approval, and use of Verification Statements summarizing the results of this test will also be subject to the requirements of that same Protocol. 7. Health and Safety The test facility will provide appropriate safety instructions regarding potential hazards during the verification testing to Battelle and vendor staff. Installation and operation of the OEMs will be such that the OEM operators or the drivers of the vehicles are not endangered, nor is the integrity of the test vehicle compromised. Testing performed on-road will be conducted in accordance with local traffic laws and speed limit restrictions. 8. References 1 U.S. Environmental Protection Agency,"Quality Management Plan for the ETV Advanced Monitoring Systems Pilot", Environmental Technology Verification Program, prepared by Battelle, Columbus, Ohio, September 1998. 2 U.S. Environmental Protection Agency, "Control of Air Pollution from New and In-Use Motor Vehicles and New and In-Use Motor Vehicle Engines: Certification and Test Procedures," 40 CFR Part 86. 3 U. S. Environmental Protection Agency, "Environmental Technology Verification Program Quality and Management Plan for the Pilot Period (1995-2000)", EPA-600/R-98/064, Cincinnati, Ohio, May 1998. 4 U.S. Environmental Protection Agency, "Generic Verification Protocol for the Advanced Monitoring Systems Pilot", Environmental Technology Verification Program, prepared by Battelle, Columbus, Ohio, October 1998. ------- |