Revision No.: 07
Date: January 22, 2002
DRAFT
GENERIC VERIFICATION PROTOCOL FOR
DIESEL EXHAUST CATALYSTS, PARTICULATE FILTERS, AND ENGINE
MODIFICATION CONTROL TECHNOLOGIES
FOR HIGHWAY AND NONROAD USE DIESEL ENGINES
EPA Cooperative Agreement No. CR826152-01-3
RTI Project No. 93U-7012-015
Prepared by: ' I * TT
BAR II
INTERNATIONAL
APPROVED BY:
APCTVC Director: J. R. Farmer Signature on File Date: 1/24/02
APCTVC Quality Manager: R.S.Wright Signature on File Date: 1/24/02
APCTVC Task Leader: D. W. VanOsdell Signature on File Date: / 24 02
APCTVC Quality Leader: C. E. Tatsch Signature on File Date: / 24 02
EPA Project Manager: T. G. Brna Signature on File Date: 1/29/02
EPA Quality Manager: P. W. Groff Signature on File Date: 1/29/02
-------
Revision No.: 07
Date: January 22, 2002
Table of Contents
1.0 INTRODUCTION 1
1.1 Environmental Technology Verification 1
1.2 Air Pollution Control Technology Verification Center 1
1.3 Mobile Sources Air Pollution Control Technologies 2
1.4 The APCTVC Mobile Sources Verification Program 2
1.5 Quality Management Documents 3
2.0 OBJECTIVE AND SCOPE 5
2.1 Objective 5
2.2 Scope 5
2.2.1 Highway Engines 6
2.2.2 Nonroad Engines 7
2.2.3 Control Technologies 7
2.2.4 Relationship of ETV Program to EPA-OTAQ VDRP Verified Technology
List 8
2.3 Data Quality Objectives (DQOs) 8
3 .0 ETV TESTING RESPONSIBILITIES 10
4.0 TECHNOLOGY CAPABILITIES AND DESCRIPTION 11
4.1 Basic Technology Identification and ETV Requirements 11
4.2 Technology Descriptive Information 12
4.3 Technology Performance 13
4.4 Technology/Engine Interactions 13
5.0 ETV TESTING 13
5.1 ETV Test Design 14
5.1.1 Determination of Number of Tests Required to Meet ETV Requirements .. 14
5.1.2 Confidence Interval on the ETV Result 16
5.2 General ETV Test Considerations 18
5.2.1 Submittal of Technologies and ETV Sequence 18
5.2.2 Highway Engine ETV 19
5.2.3 Highway Test Engines 19
5.2.4 ETV Testing of Nonroad Engines - General 20
5.2.5 Nonroad Test Engines 20
5.2.6 De-greening 20
5.2.7 Regeneration 21
5.2.8 Device Scaling 21
5.2.9 Durability 21
5.2.10 Standard Diesel Test Fuel 22
5.2.11 Engine Performance and Power 23
5.2.12 Fuel Consumption 24
5.2.13 Back-pressure 24
11
-------
Revision No.: 07
Date: January 22, 2002
5.2.14 Control Technology Operating Temperature 24
5.2.15 Other Measurements and Conditions 24
5.3 ETV of DECs and PM Filters 25
5.3.1 Technology Description 25
5.3.2 Test Considerations Specific to DECs and PM Filters 25
5.4 ETV of Engine Modifications 25
5.4.1 Technology Description for Engine Modifications 25
5.4.2 Test Considerations Specific to Engine Modifications 26
6.0 REPORTING AND DOCUMENTATION 26
6.1 Reports 26
6.2 Data Reduction 27
7.0 DISSEMINATION OF ETV REPORTS AND STATEMENTS 27
8.0 APPLICANT'S OPTIONS SHOULD A TECHNOLOGY PERFORM BELOW
EXPECTATIONS 28
9.0 LIMITATIONS ON TESTING AND REPORTING 28
10.0 REQUIREMENTS FOR TEST/QA PLAN 29
10.1 Quality Management 29
10.2 Quality Assurance (QA) 29
10.3 Additional Requirements to be Included in the Test/QA Plan 30
11.0 ASSESSMENT AND RESPONSE 31
11.1 Assessment Types 31
11.2 Assessment Frequency 32
11.3 Response to Assessment 32
12.0 SAFETY MEASURES 33
12.1 Safety Responsibilities 33
12.2 Safety Program 33
13.0 REFERENCES 33
APPENDIX A: EXAMPLE VERIFICATION STATEMENT A-l
APPENDIX B: EXAMPLE CALCULATION OF NUMBER OF TESTS REQUIRED B-l
APPENDIX C: SENSITIVITY OF NUMBER OF TESTS CALCULATION TO
MEASUREMENT VARIABILITY C-l
iii
-------
Revision No.: 07
Date: January 22, 2002
List of Tables
Table 1. Number of replicate tests (on both baseline and controlled engine) to achieve a 90%
probability of detecting a PM emission reduction at a confidence level of 95% .... 16
Table 2. 95% confidence intervals on example mean emission reductions relative to
baseline engine emissions at 1990 certification value 17
Table 3. Minimum durability testing periods (CARB, 2001) 23
iv
-------
Revision No.: 07
Date: January 22, 2002
1.0 INTRODUCTION
1.1 Environmental Technology Verification
The U.S. Environmental Protection Agency (EPA) through its Office of Research and
Development (EPA-ORD) has instituted the Environmental Technology Verification (ETV)
Program to verify the performance of innovative and improved technical solutions to problems
that threaten human health or the environment. EPA created the ETV Program to substantially
accelerate the entrance of new and improved environmental technologies into the domestic and
international marketplaces. It is a voluntary, non-regulatory program. Its goal is to verify the
environmental performance characteristics of commercial-ready technologies through the
evaluation of objective and quality-assured data so that potential purchasers and permitters are
provided with an independent and credible assessment of what they are buying and permitting.
The ETV Program does not conduct technology research or development. ETV test results are
always publically available, and the applicants are strongly encouraged to ensure, prior to
beginning an ETV test, that they are satisfied with the performance of their technologies. Within
the ETV Program, this state of development is characterized as "commercial-ready ," and the
ETV test is conducted on production units or prototypes having the major characteristics of
commercial units.
The provision of high-quality performance data on fully-developed commercial technology
encourages more rapid implementation of those technologies and consequent protection of the
environment with better and less expensive approaches. The ETV Program is conducted by six
ETV centers that span the breadth of environmental technologies.
1.2 Air Pollution Control Technology Verification Center
EPA's partner in the Air Pollution Control Technology Verification Center (APCTVC) is RTI, a
non-profit contract research organization with headquarters in Research Triangle Park, NC. The
APCTVC verifies the performance of commercial-ready technologies used to control air
pollutant emissions. The emphasis of the APCTVC is currently on technologies for controlling
particulate matter, volatile organic compounds, nitrogen oxides (NOx), and hazardous air
pollutants from both mobile and stationary sources. The activities of the APCTVC are
conducted with the assistance of stakeholders from various interested parties. Overall APCTVC
guidance is provided by the Stakeholders Advisory Committee (SAC), while the detailed
development of individual technology ETV protocols is conducted with input from Technical
Panels (TP) focused on each technology area.
The APCTVC develops generic verification protocols and specific test/quality assurance
(test/QA) plans, conducts independent testing of technologies, and prepares ETV test reports and
statements for broad dissemination. Testing costs are ultimately borne by the technology
applicants, although initial tests within a given technology area may be partially supported with
government funds.
1
-------
Revision No.: 07
Date: January 22, 2002
1.3 Mobile Sources Air Pollution Control Technologies
Control of emissions from mobile sources continues to be of great national importance. Several
areas of the country are not able to attain ambient air quality standards. The mobile source
provisions of the 1990 Clean Air Act Amendments (CAAA) are intended to reduce most
vehicle-related air pollutants by more than 40 percent for 1996 and later model year vehicles and
engines. Earlier engine models emit pollutants at higher levels, and as these engine are durable
and have long useful lives, they will continue to emit pollutants at higher levels many years into
the future. For these reasons, the EPA funded and the APCTVC SAC recommended inclusion of
air pollution control technologies for mobile sources as a priority for ETV.
One important group of mobile source air pollution control initiatives is concerned with highway
and nonroad use diesel engines. The diesel particulate standard for urban buses was reduced in
1993 by 60 percent, from 0.33 to 0.13 g/kWh (0.25 to 0.10 g/bhp-hr). The standard, which
applies to urban transit buses, dropped to 0.094 g/kWh (0.070 g/bhp-hr) in 1994 and to 0.067
g/kWh (0.050 g/bhp-hr) in 1996. While existing engine technologies can meet these standards,
future standards are expected to be increasingly stringent and will require the use of post-
combustion emissions control technologies. New technologies are being developed to meet these
goals. In addition, since a NOx emission level below the level mandated allows the generation of
credits through the Voluntary Diesel Retrofit Program (VDRP), pollution prevention becomes
more cost effective, and innovations in less-polluting alternatives and control technologies are
encouraged.
Retrofit mobile source control technologies are principally exhaust treatment emission control
devices and engine modifications. Some require no mechanical changes to engines, while others
will involve some modification of the engine or its control system.
Filters for particulate matter (PM) control and diesel exhaust catalysts (DECs) may make use of
or require some integration with engines. Engine modifications, in the context of this protocol,
refer to pollution reduction technologies integral to the engine or the engine control systems. All
these technologies have the potential to affect engine performance, and the concurrence of the
engine manufacturer that the changes are compatible with safe, efficient, and reliable operation in
the engine is an important element in demonstrating commercial readiness and suitability for
ETV. The technologies and their testing are discussed in more detail below.
Selective Catalytic Reduction (SCR) nitrogen oxides (NOx) control technologies, fuels, fuel
additives, reformulated fuels, and lubricants are specifically excluded from consideration under
this protocol for testing as retrofit emissions control devices. Protocols for these technologies
may be developed in the future.
1.4 The APCTVC Mobile Sources Verification Program
This generic verification protocol (GVP) provides the requirements for APCTVC's ETV of the
performance of DEC, PM filter, and engine modification air pollution control technologies
applied to highway and nonroad use diesel engines. Other technologies will be addressed
2
-------
Revision No.: 07
Date: January 22, 2002
similarly. Other organizations [e.g., EPA's Office of Transportation and Air Quality (EPA-
OTAQ) and the California Air Resources Board] also 'verify' the performance of mobile source
emissions control devices under different protocols to meet the needs of those organizations.
Specifically, EPA-OTAQ has established, for applicants who meet particular criteria, a generally
parallel path that does not require the level of external QA that is required for APCTVC ETV.
The technology applicant should discuss the intended application of the technology with the
appropriate regulatory body to determine the most suitable path for "verification."
This GVP is intended to apply only to DECs, PM filters, and engine modifications and their
combinations. The APCTVC reserves the right to evaluate each technology submitted for ETV
and to determine the applicability of this protocol to that specific technology. Special testing
may be required in some cases to maintain the integrity and credibility, and, therefore, the value
of ETVs. The critical data quality objectives (DQOs) in this document were chosen to provide
emissions measurements sufficient to support the VDRP and its emissions credit provisions.
Emissions credit allowances will be set by the appropriate state regulatory authority or EPA-
OTAQ. (As stated above, the VDRP approving agencies may have data requirements that are in
addition to the ETV test report.)
This protocol was developed and has been reviewed by a technical panel composed of a broad
group of stakeholders who have expertise in mobile source controls and come from the vendor,
user, and regulatory spheres. Technical panel membership is dynamic, and its composition is
expected to change over time as technical emphases change. The APCTVC will maintain
membership balance on the panel.
The basic APCTVC ETV test will measure and report baseline emissions concentrations and
rates for an engine prior to use of the retrofit technology and also the emissions concentrations
and rates for this engine following retrofit. It will be conducted at an independent, third-party
test laboratory. The data quality requirements of this generic protocol will be applied at
approved test laboratories through the preparation of test-specific test/QA plans. Other
laboratory-, application-, or technology-specific information may also need to be addressed in the
test/QA plan, which is described in Section 10.0. In general, test/QA plans prepared by test
laboratories will not be reviewed by the entire technical panel. However, because specific
technology areas may require special expertise or emphasis, input and review will be obtained
from an ad hoc subcommittee of the technical panel and/or outside experts when deemed
appropriate by the APCTVC. Test results will be presented as ETV reports and statements.
1.5 Quality Management Documents
Management and testing in this APCTVC program are performed in accordance with procedures
and protocols defined by the following:
1) EPA's ETV Quality and Management Plan (ETV QMP) (EPA, 1998a or the quality
management plan current at the time of testing),
2) the APCT Quality Management Plan (QMP) (RTI, 1998),
3
-------
Revision No.: 07
Date: January 22, 2002
3) the Generic Verification Protocol for Diesel Exhaust Catalysts, Particulate Filters,
and Engine Modification Control Technologies for Highway and Nonroad Use
Diesel Engines (this document), and
4) the test/QA plan prepared for each specific test or group of tests.
EPA's ETV QMP lays out the definitions, procedures, processes, inter-organizational
relationships, and outputs that will ensure the quality of both the data and the programmatic
elements of the ETV Program. Part A of the ETV QMP contains the specifications and
guidelines that are applicable to common or routine quality management functions and activities
necessary to support the ETV Program. Part B of the ETV QMP contains the specifications and
guidelines that apply to test-specific environmental activities involving the generation, collection,
analysis, evaluation, and reporting of test data.
APCT's QMP describes the quality systems in place for the overall APCTVC. It was prepared
by RTI and approved by EPA. Among other quality management items, it defines what must be
covered in the GVPs and test/QA plans for technologies undergoing ETV testing.
Generic Verification Protocols (GVPs) are prepared to describe the general procedures to be
used for testing a type of technology and define the critical data quality objectives (critical
DQOs). The GVP for retrofit air pollution control technologies for highway and nonroad use
diesel engines was written by the APCTVC with input from a technical panel and approved by
EPA.
A test/QA plan is prepared for each test or group of tests. The test/QA plan describes, in detail,
how the testing organization will implement and meet the requirements of the GVP. The test/QA
plan also sets DQOs for any planned measurements that were not set in the GVP. The test/QA
plan addresses issues such as the test organization's management structure, the test schedule, test
procedures and documentation, analytical methods, record keeping requirements, and instrument
calibration and traceability, and it specifies the QA and quality control (QC) requirements for
obtaining ETV data of sufficient quantity and quality to satisfy the DQOs of the generic
verification protocol. Section 10 of this GVP addresses requirements for the test/QA plan.
Because multiple testing organizations will be conducting the tests and the desirability to ensure
comparability, the APCTVC will develop a prototype test/QA plan (not part of this GVP) for
each type of technology. This prototype will be customized by the testing organization to meet
its specific implementation of the Federal Test Procedures (FTPs) as defined in 40 CFR Parts 86
and 89, and the secondary measurements, subject to approval by the APCTVC and EPA-ORD.
Testing arrangements that do not meet the requirements of the FTP will not be approved, and test
instrumentation or test procedures that the APCTVC determines will compromise data reliability
or comparability between testing organizations will not be approved.
4
-------
Revision No.: 07
Date: January 22, 2002
2.0 OBJECTIVE AND SCOPE
2.1 Objective
The objective of this GVP is to establish the parameters within which DECs, PM filters, and
engine modification air pollution control technologies for highway and nonroad use diesel
engines will be tested to verify their performance with uniform and consistent methodologies
within the APCTVC. The protocol addresses the requirements for technology submission,
outlines the test conditions and procedures to be used, and states the critical DQOs for ETV and
states reporting requirements. The control technologies will be verified within a specified range
of applicability, and ETV reports and statements will be produced for dissemination to the
public.
2.2 Scope
APCTVC testing will be performed to quantify the effectiveness of commercially ready
emissions control technologies that are intended for use on diesel engines considered mobile
sources of air pollutants. Emissions testing under this ETV Program is based on the applicable
FTPs for highway and nonroad engines. As performed by manufacturers to certify their engines,
FTPs are fully defined by the regulations and consist of the engine-appropriate cold- and hot-start
engine dynamometer tests conducted under specified load cycles. For purposes of this protocol,
the number and type of FTP tests may be different from certification requirements, as determined
by the applicant and the APCTVC to best verify the performance of the technology under
consideration. However, the individual FTP test module — the cold-start test or the hot-start test
— will be performed completely in accordance with the regulations. (The low NOx and PM
emissions from post-2001 engines are prompting development of new test procedures better
suited to these emission levels. As these test procedures become standardized and acceptable to
EPA, they will be acceptable for use in ETVs.) The pollutants of major interest are NOx,
hydrocarbons (HCs), PM, and carbon monoxide (CO). Other measurements will also be made of
emissions (for instance, carbon dioxide[C02]) and operating parameters of the test engine and the
control technology. Additional measurements may also be required if the technology might be
reasonably expected to generate secondary emissions. Emissions will be measured along with
other data useful for evaluating the performance of the technologies and the technologies'
associated environmental and efficiency impacts. In this context, the effects of control
technologies on fuel economy and engine performance and power will be of particular interest.
Also fundamental to this ETV protocol is providing emissions control efficiency information
needed for applicants to participate in the EPA Voluntary Diesel Retrofit Program (VDRP). The
VDRP is managed by EPA-OTAQ. The data quality objectives given in Section 2.3 were set to
provide emissions control device performance data that can be used in submittals to the VDRP.
VDRP determinations are made by EPA-OTAQ, which may require information not included in
the ETV.
The APCTVC will consider applications for ETV of air pollution control technologies on test
engines specified by the applicant. Depending on the expected application, the data collected,
5
-------
Revision No.: 07
Date: January 22, 2002
and the test design, the results obtained may be valid only for that single engine or to groupings
of multiple engines. If application of single engine test results to an engine group is made by the
regulatory body, the decision will be based on an expectation that the control device, if
appropriately sized, will perform similarly on all the engines in that group. Generally, similar
emissions control performance is expected from engines that have the same engine technology
and use. The sensitivity of emissions control performance to engine type and usage is not known
to be the same for all technologies, and the applicability decision will be made separately for
each technology type and possibly for each technology. Previously obtained performance data
may affect the decision. In all cases, the decision regarding applicability will be made by EPA-
OTAO. which should be contacted at the time a technology is submitted to obtain current
information.
Implicit in the potential use of single engine ETV tests as predictive of performance over
segments of the total engine population is the availability of non-proprietary application sizing
information and that the technology be labeled. The purpose of this requirement is to allow the
APCTVC and the potential technology user to evaluate the applicability of the verified
technology to other applications. Applicants who utilize application sizing charts must verify
their technologies on an engine at the most challenging end of the range of applicability. ETV
reports and statements will report the results and the design guidance required to independently
extend application of the technology to other engines in the same engine grouping.
To request ETV, the applicant applies to the APCTVC for the technology to be tested and
proposes the test engine(s) to be used. The APCTVC will review the application and provide
input as required to ensure that the engine(s) proposed are suitable and that the emission control
equipment sizing is clear and correct. The equipment/technology will be applied to the engine(s)
according to the applicant's instructions. ETV test results for the engine(s) will be reported in
the ETV report and statement. The results will provide the emissions performance data required
for the VDRP reviewing regulatory authority to determine the emission reduction capability of
the technology.
2.2.1 Highway Engines
Applicants may choose to have their technologies verified on a single highway engine or multiple
selected engines, depending on the applicant's intended use of the data. EPA-OTAQ will
determine the extent of applicability of data from any given engine and should be consulted by
the applicant for that purpose. Testing of emissions control technology intended to control
emissions from highway diesel engines will be conducted generally under the provisions of 40
CFR, Part 86, Subpart N. The primary emissions measurements will be of NOx, PM, HC, CO,
and other FTP requirements. C02 and the engine operating parameters are required secondary
measurements. Each engine in the test will be loaded by a dynamometer as described in the
transient FTP. Each ETV test will consist of at least a single full FTP test (cold- and hot-start)
and two additional hot-start tests (total of three hot-starts) on an emission-stabilized baseline
engine and on the engine with the control technology in use. Additional hot-start tests may also
be required to meet the number of test requirements described in Section 5, or the applicant may
wish to run additional tests to narrow the confidence interval on the result. [An ETV
6
-------
Revision No.: 07
Date: January 22, 2002
requirement for multiple hot-start tests may be met using the hot-start results from full FTP test
data (both cold- and hot-start), at the applicant's option.] Durability testing , which consists of
ETV of aged control technology units as described in Section 5.2.9, is part of the ETV process.
Emissions reductions will be computed relative to the baseline engine emissions.
2.2.2 Nonroad Engines
As with highway engines, applicants select the nonroad engines on which to test their
technologies. The ETV testing of emissions from nonroad diesel engines will be conducted
generally within the requirements of 40 CFR, Part 89, Subpart E. As for highway engines, the
primary emissions measurements for nonroad engines will be of NOx, PM, HC, CO, and other
FTP requirements. C02 and engine operating parameters are required secondary measurements.
The ETV testing will consist of three or more nonroad steady-state mode tests (40 CFR, Part 89,
Subparts D and E), as described in Section 5. (Concerns about transient emissions from nonroad
engines are prompting development of new test procedures. As these test procedures become
standardized and are codified in the FTP, they will be included in the ETV test protocol.) The
test must be conducted over all applicable modes of the test cycle appropriate to the engine (e.g.,
8-mode test cycle for variable speed engines or 5-mode cycle for constant speed engines) or may
be conducted over all modes at the applicant's request. The ETV report will reflect the extent of
the test conducted. Each ETV test will consist of the same assemblage of FTP tests on the
broken-in baseline engine and the engine with the control technology in use. Durability testing is
required as described in Section 5.2.9. Composite emissions reductions are computed relative to
the baseline engine emissions. In addition, the emissions results will be reported in composite
and mode-by-mode to allow flexibility in use of the data for different engine applications.
2.2.3 Control Technologies
This GVP is specifically intended to include the following emissions control technologies:
1) DECs regenerated continuously or periodically,
2) PM filters, and
3) Engine modifications.
The basic ETV test remains the same for all emissions control technologies. However, the
technologies interact differently with the various engines. Therefore, the engine grouping
concept may be applied differently by EPA-OTAQ depending of the nature of the technology. In
addition, where technologies require particular fuel characteristics or have other restrictions or
interactions, applicants must specify these and any associated requirements in their ETV
applications.
In all cases, the ETV test will be designed to reasonably isolate the effects of the retrofit
technology from coincident engine adjustments and tune-ups. For example, a retrofit technology
that required an engine tune-up following installation would be tested against a tuned-up baseline
engine.
7
-------
Revision No.: 07
Date: January 22, 2002
Some technologies, by their nature, may reasonably require engine adjustments following
installation. To be allowed, such adjustments must be part of the description of the technology,
be included in the installation instructions, be described in the application, and be made by the
laboratory technician and not by the applicant. They will be examined to determine the effect of
those engine adjustments alone, and the baseline engine may be tuned to the same operating
condition if the adjustments alone are considered to have a positive effect on emissions. This
requires that the APCTVC and the testing laboratory be provided with a full disclosure of the
working principles of the technology prior to developing the test. In these cases, the ETV report
and statement will define the retrofit technology as consisting of both the device and the engine
adjustments, with both required to achieve any emissions reduction observed.
2.2.4 Relationship of ETV Program to EPA-OTAO VDRP Verified Technology List
EPA-OTAQ is charged with establishing a Verified Technology List of technologies capable of
providing emissions reductions. The test results EPA-OTAQ will utilize to evaluate a
technology may be generated following the ETV process, with the ETV report and verification
statement serving as the data package to EPA-OTAQ. Other paths to the verified technology list
also exist. The VDRP program is described and appropriate contacts are identified at
http://www.epa.gov/otaq/retrofit/. The technology applicant should discuss the intended
application of the technology with EPA-OTAQ to determine the most suitable evaluation path for
the applicant's technology.
As of the date of this protocol, emissions test data from a full FTP test (whether highway or
nonroad), that was generated by a manufacturer to obtain a certificate of conformity may be
determined by EPA-OTAQ to constitute a valid data set for VDRP purposes. Certification data
do not pass though the ETV Program. The certification and VDRP technology listing processes
are independently performed by EPA-OTAQ. In addition, EPA-OTAQ has established for some
applicants a technology evaluation pathway whose test requirements parallel those of this GVP,
but with reduced QA requirements.
2.3 Data Quality Objectives (DQOs)
The data of primary interest in this ETV are the reduction in emissions of NOx, HC, PM, and CO.
(Numerous other measurements, including C02 emissions, will also be made as described in later
sections. However, these are not considered critical, and the methods and data quality objectives
will be stated in the test/QA plan for each technology.) The DQOs of this GVP are the
requirements of the test methods specified in 40CFR Parts 86 or 89 for highway or nonroad
diesel engines, respectively, when conducting the number and type of FTP tests called for by the
approved test/QA plan for the technology. ETV tests that do not meet the FTP QA requirements
must be repeated.
The number of and type of FTP tests (cold- and/or hot-start) required for ETV is determined from
the following criteria:
8
-------
Revision No.: 07
Date: January 22, 2002
First, a minimum of three tests is required to provide the basic ETV result of a mean
emission reduction and the 95% confidence interval on that mean based on measured
variability for each of the measured emissions and test parameters. For highway engines
this minimum is satisfied with one cold start test and three hot start tests. For nonroad
engines three replicates of the appropriate test sequence (i.e., three 8-mode tests, or three
6-mode tests) are required. A three test minimum is currently the same as is required by
the State of California for verification.
Second, additional tests may be required to meet the ETV requirement that the test/OA
plan provide a 90% probability of detecting the expected emissions reductions when
computed using the expected experimental errors for the various measurements. These
criteria become controlling for low emissions reductions and/or high test variability. This
is a planning requirement for the test/QA plan. The procedure to determine the
appropriate number of tests is given in Section 5.
Third, additional tests may be desired by the applicant to reduce the width of the 95%
confidence interval on the mean emission reduction. Section 5 provides additional
explanation and example scenarios. This third criterion is a consequence of applying
standard statistical procedures to the ETV test design and data analysis. At a fixed
measurement variability, normal statistical procedures lead to a small number of tests
giving a broader 95% confidence interval than a larger number of tests. To any regulator
or potential technology user, an emission reduction of 40% ± 5% is better than 40% ±
20% and will be given more credence.
An applicant may conduct privately sponsored tests at a test laboratory for development
purposes with the same test engine prior to and/or after conducting ETV tests. Such testing is
understood to be common and important to ensure the technology is properly adjusted and
tuned to the application. However, two important principles of ETV are that the verified
technology be commercial ready when tested and that all data obtained under the ETV
process be reported.
Therefore, preparation for the ETV test (submittal of the technology to the APCTVC,
discussion of engine selection, and preparation of the test/QA plan) must be completed prior
to conducting the ETV test itself. In particular, declaration of the test run which is to be the
ETV test must be made prior to starting the test, the engine must be brought to a starting
point in accordance with the test/QA plan, and the results of that test will be documented and
reported in accordance with the test/QA plan. The data from all ETV tests will be retained
and reported to the APCTVC, including invalid FTP test results. Data that meet the QA
requirements of the FTP are considered valid and will be used to compute emissions
reductions for ETV purposes. Section 8.0 addresses a technology applicant's options should
the technology perform below expectations.
Tests conducted at qualified laboratories will be accepted as ETV tests provided the
technology is submitted to the APCTVC and all requirements of this GVP are met, including
approval of the test/QA plan prior to testing. This requires that the technology be submitted
9
-------
Revision No.: 07
Date: January 22, 2002
to the APCTVC, that a test/QA plan be prepared by the laboratory in conjunction with the
APCTVC, and that the test/QA plan be approved by the APCTVC prior to testing. All QA
requirements of the FTP methods must be met, and the test and test laboratory must meet the
QA requirements of ETV as established in this GVP and the test/QA plan.
Engine emissions are expressed in grams of pollutant per kilowatt-hour (g/kWh) or in units
of grams per brake horsepower-hour (g/bhp-hr). The primary measurement of HC and NOx is
normally concentration in the exhaust stream in parts per million by volume (ppmv), which is
then converted to g/kWh in accordance with the FTP. For calculation of the technology's
removal efficiency, the baseline emissions of the tested engine must also be known. The
control technology performance will be reported as both absolute emissions in g/kWh and as
percentage emissions reduction for a specific engine or engine family. The percentage
emission reduction reported will be the mean emission reduction (relative to the baseline
emission) with attendant upper and lower 95% confidence limits on that mean.
3.0 ETV TESTING RESPONSIBILITIES
This ETV program is conducted by the APCTVC under the sponsorship of the EPA-ORD
and with the participation of technology applicants. The APCTVC is operated under a
cooperative agreement by RTI, EPA's ETV partner. RTFs role as ETV partner is to provide
technical and administrative leadership and either conduct or manage the conduct of ETV
testing and reporting. Various subcontractors have roles in the APCTVC under RTFs
management.
ETV tests are conducted by qualified test laboratories as contractors to RTF Test laboratory-
specific ETV test/QA plans are prepared by the testing laboratories to meet the requirements
of the GVP.
The test/QA plan includes a chart that presents the test program organization and major lines
of communication. The organizations involved in the ETV of mobile diesel engine air
pollution control technologies are the EPA, RTI, testing laboratory, and technology applicant.
The primary responsibilities for each organization involved in the test program are:
1) The EPA-ORD, following its procedures for ETV, reviews and approves GVPs,
test/QA plans, ETV reports and statements, and conducts QA audits.
2) The APCTVC prepares the GVP, provides oversight of the testing organization,
provides a template for test/QA plans, and jointly with EPA-ORD reviews and
approves the ETV reports and statements.
3) The testing organization prepares the test/QA plan in accordance with the GVP,
coordinates test details and schedules with the applicants, conducts the tests, and
prepares and revises draft ETV reports and statements. The testing organization QA
staff is responsible for conducting internal QA on test results and reports.
4) EPA-ORD and/or APCTVC QA staff, at their discretion, will conduct assessments of
the test organization's technical and quality systems.
10
-------
Revision No.: 07
Date: January 22, 2002
5) The technology applicant provides complete, commercial-ready equipment for ETV
testing; provides logistical and technical support, as required; and assists the testing
organization with operation and monitoring of the equipment during the ETV testing.
The applicant's responsibilities are defined by a contract or letter of agreement with
RTL.
4.0 TECHNOLOGY CAPABILITIES AND DESCRIPTION
This section outlines the information to be submitted by the ETV applicant, who is the basic
source of the information regarding its mobile source air pollution control technology. This
information is used by the test laboratory and APCTVC to prepare and review a test/QA plan
that meets the requirements of the applicant and potential users of the ETV. No general
outline can anticipate the data requirements for all possible technologies, so the applicant
may be requested to supplement this outline when additional information is needed. This
protocol has been developed with an awareness of the California Air Resources Board
(CARB) "verification" program, and an application to CARB for retrofit "verification" is
expected to provide adequate information. The basic application form to be used for ETV is
that posted by EPA-OTAQ for the voluntary retrofit program, which can be found at:
http://www.epa.gov/otaq/retrofit/documents/forms.xls (as an Excel file)
http://www.epa.gov/otaq/retrofit/documents/forms.wk4 (as a Quattro file)
EPA-OTAQ has also posted application form guides, which can be found by replacing
"forms" with "forms guide" in the links above. The ETV test/QA plan will summarize this
information in a form that clearly describes the technology being verified and states, for
potential users of the technology, it's intended range of applicability as stated by the
applicant. The test/QA plan will also include a draft ETV statement, based on Appendix A,
customized to the specific technology being verified and measurements being made. The
balance of this section independently describes the information requirements that led to
development of these forms.
4.1 Basic Technology Identification and ETV Requirements
The basic technology identification should be a concise description of the technology that
describes what is being verified and summarizes the emissions control performance expected.
This information will be used to describe the technology in the ETV statement.
It should include such information as: technology name, model number, manufacturer's
name and address, serial number or other unique identification, warning and caution
statements, capacity or throughput rate, specific installation instructions, adjustments, limits
on flow, temperature, pressure, and other information necessary to describe the specific
technology and its intended use. Warranty information on the technology should be
provided, including a sample warranty statement.
In addition, the applicant must state the goal of the ETV in terms of range of applicability
11
-------
Revision No.: 07
Date: January 22, 2002
(single or multiple engine, fuel requirements, equipment sizing, etc.), expected emissions
reduction, and anticipated data use. This information will be used to develop the ETV
test/QA plan.
4.2 Technology Descriptive Information
This section of the application is intended to describe the technology more fully. It will be
used to prepare the test/QA plan and as a more complete description of the technology in the
ETV report.
The applicant should describe the technology in a brief (300 word) statement, referring to the
scientific principles of the technology's operation. Depending on the complexity of the
technology, this may be followed by more extensive information. When applicable, the
inclusion of schematic drawings explaining operation of the technology is encouraged.
Examples of descriptive information that should be included are:
Installation requirements. Space occupied, installation time, any physical
modifications, placement issues, identification of ancillary equipment, if any,
recommended duration of degreening as well as any special start-up procedures, and
any other special requirements.
Operation. Description to include identification of any favorable or unfavorable
operating conditions and their effect on performance, fuel requirements (such as fuel
sulfur limit), consequences of misfueling, identification of any chemicals or other
consumable reactants, regeneration requirements, and expected engine back-pressure.
Any requirement for energy sources (e.g., heat, electricity, compressed air, hydraulic
fluid, pressure) external to the engine must be identified so they can be accounted for
in the test/QA plan.
Maintenance and warranty. Identification of recommended maintenance procedures,
cleaning instructions, and spare parts and supplies. The warranty being offered
should be stated.
Operator or mechanic qualifications/training/safety. Qualifications needed to operate
and service the technology, amount and type of training needed for operation and
maintenance, and special safety considerations. Use of personal protective equipment
(eyeglasses, hearing protection, etc.) should be noted.
Secondary emissions. Identification of any secondary emission to the air that impacts
water quality, or liquid or solid waste.
Technology's life expectancy. Special attention should be given to the technology's
regeneration requirements, if any, including a description of the regeneration process,
any operating condition requirements for successful regeneration (e.g., temperature),
and any manual or computer control of regeneration.
In the case of combinations of independent technologies that are being submitted for ETV,
the description of the combined technology should completely identify and describe those
technologies being combined and fully state the nature of the combined test and expected
result.
12
-------
Revision No.: 07
Date: January 22, 2002
4.3 Technology Performance
The technology applicant must quantify the performance expected from the technology and
support that expectation with background test data, including both initial performance and
technology durability test descriptions and results. In general, these data will not be used to
verify the performance of the technology, but will be used to develop an ETV test/QA plan
that can adequately support the expected performance. The level of descriptive detail
required for existing tests and their results cannot be generally stated because it is dependent
on the ETV goals. Applicants are encouraged to supply pertinent data supporting their
claims, which will include test facility descriptions, procedures, quality assurance and
control, and any deviations from the FTP.
All aspects of the technology's performance should be addressed, beginning with
performance during routine steady-state operation, but also including unsteady or unfavorable
operation such as extended idling periods, uncontrolled regeneration, in-place maintenance,
etc. Any known or expected impact the technology might have on the character of the
emissions (gas composition, temperature, particle size or composition, etc.) should be
identified.
4.4 Technology/Engine Interactions
The applicant must describe interactions between the proposed engine(s) and the technology,
to include the known or expected effects of the technology on the engine(s). Backpressure,
additional parasitic loads, fuel consumption, effects on engine durability, and oil
consumption are of special concern. The relationship of the technology, engine control logic,
and engine operation should be addressed for the engine(s) under consideration.
The applicant should defend the applicability of the technology to the applicable engine(s) by
including exhaust temperature profiles, duty cycles, and other relevant performance
information. The operating conditions that are favorable for the control technology should be
compared with those of the engine.
The opinion of the engine(s)'s manufacturers regarding compatibility of the technology with
their engine should be solicited, summarized, and the correspondence included in the
application. Any known or anticipated impact of the technology on engine warranty is to be
identified. The relationship between technology warranty and engine warranty should be
discussed.
5.0 ETV TESTING
As stated in Section 2.3, Data Quality Objectives, an ETV test consists of a minimum of
three FTP tests, and may include additional FTP tests to differentiate a low emission
reduction technology from zero or to narrow the confidence limit on the mean emission
reduction. Section 5.1 presents the methods that are used to design an ETV test to
accomplish the applicant's goal, beginning from an expected emissions reduction for a
13
-------
Revision No.: 07
Date: January 22, 2002
particular pollutant and progressing through the methods used to generate the 95%
confidence interval on the mean emission redutions. Section 5.2 presents other general ETV
test considerations that apply to most retrofit air pollution control technologies. The
remaining sections of this chapter are specific to the indicated types of technologies.
5.1 ETV Test Design
The data of primary interest in this ETV are the reduction in emissions of NOx, HC, PM, and
CO. (Numerous other measurements, including C02 emissions, will also be made as
described in later sections. However, these are not considered critical, and the methods and
data quality objectives will be stated in the test/QA plan for each technology.) NOx
emissions will be used to compute the number of tests required unless the applicant
specifically requests that the reduction in another of the pollutants be used to control test
design.
The number of and type of FTP tests required for ETV will be as follows:
A minimum of three tests is required to provide a mean emission reduction and the 95%
confidence interval on that mean based on measured variability for each of the measured
emissions and test parameters. For highway engines this minimum is satisfied with one
cold start test and three hot start tests. For nonroad engines the minimum is satisfied with
three replicates of the test sequence appropriate for the engine classification for which the
technology is intended (i.e., three 8-mode tests for applicable engines, three 5-mode test
for constant speed engines, three 6-mode test for variable speed engines under 19 kW.)
Additional tests may be required to meet the ETV requirement that the test/QA plan
provide a 90% probability of detecting the expected emissions reductions when computed
using the expected experimental errors for the various measurements. These criteria
become controlling for low emissions reductions and/or high test variability for the
controlling pollutant, which will be NOx unless another pollutant is selected. This is a
planning requirement for the test/QA plan. Section 5.1.1 describes the calculation
procedure.
Relative to the minimum number, additional tests may be desired by the applicant to
reduce the width of the 95% confidence interval on the mean emission reduction. Section
5.2 provides additional explanation and example scenarios.
5.1.1 Determination of Number of Tests Required to Meet ETV Requirements
Minimizing the cost of ETV testing is important, and reducing the number of tests conducted
is one way to minimize cost. However, if too few tests are conducted, normal experimental
variability could prevent the ETV from producing a useful outcome. For measurements
having known variability and for stated expected reductions in emissions, the probability that
a given number of tests can detect the expected reduction can be computed from statistical
theory based on normal distributions. This ETV protocol requires that all ETV test/QA plans
14
-------
Revision No.: 07
Date: January 22, 2002
include sufficient tests to have a high probability of detecting the emissions reductions
expected by the applicant. This requirement was implemented to ensure, as much as was
practical, that the ETV test would accomplish the applicant's goals. This requirement will
generally only be important to ETV test design for low emissions reductions and/or
measurements having high variability. Specifically, the ETV test is to be designed such that
there is 95% confidence that there is a 90% probability of detecting the emission reductions
expected by the applicant, computed from the expected experimental variability for the
various measurements. In this context, "detecting" means that the 95% confidence interval
does not include zero emissions reduction. (This requirement is for test design purposes, and
does not require that the test/QA plan be modified should actual test data show that the
assumptions that went into the calculation were incorrect. However, as explained below,
insufficient replication can result in inability to issue a verification and publication of an ETV
report stating that a technology had no statistically significant benefit.)
An equation was derived to explicitly compute the number of tests required for normally
distributed measurements having known standard deviations. Some results of applying that
calculation are presented in Table 1 for PM and NOx emission reductions. The equation
used to compute the number of tests required is presented in Appendix B.
In Table 1 the baseline engine is assumed to be at a fixed emission value (the 1990
certification emission level for highway diesel engines) for PM (top of table) and NOx
(bottom three rows). PM and NOx measurements are used in Table 1 because the
measurement standard deviations were known for one laboratory. The results in Table 1 are
limited by the requirement in the derivation that the measurements be normally distributed
with known variability. As will be shown in Section 5.1.2, the determination of the
measurement variability from data, as will be done during the ETV data analysis, leads to a
broadening of the confidence interval. Consequently, more tests than shown in Table 1 may
be required under those conditions, and careful examination of the number of tests needed
will be required during test/QA plan development.
The column headings in the first row of the PM and NOx sections of the table give the
assumption that installing the control technology gives emissions reductions of 5%, 20%,
50%), or 85%) from the baseline emissions levels, which might be the applicant's estimate of
emission reduction for different technologies. The second row gives the constant and known
baseline engine (without control technology) variability when emitting at the 1990
certification level. The third row is the known measurement variability for the baseline
engine after having been fitted with control technology that achieves the reduction shown in
the column heading. (Variability is expressed as the known standard deviation at the
emission level divided by the corresponding baseline engine mean emission and multiplied
by 100%).) The variability increases as the emission reduction increases from left to right
because the absolute error is roughly constant, while the magnitude is decreasing. The fourth
row gives the calculated number of tests required to have 90% confidence that the 95%
confidence interval about the mean expected emission reduction would not include zero.
15
-------
Revision No.: 07
Date: January 22, 2002
Table 1. Number of replicate tests (on both baseline and controlled engine) to achieve a
90% probability of detecting a PM emission reduction at a confidence level of
95%
Expected emission reduction (relative to the respective 1990
certification levels for PM and NOx, respectively)
5%
20%
50%
85%
Variability for baseline engine PM measurement at 1990 levels
2%
2%
2%
2%
PM measurement variability at specified emission reduction
2%
2%
3%
10%
Number of tests required to have 90% probability (ETV
requirement) of detecting expected PM emission reduction®
3
1
1
1
Variability for baseline engine NO, measurement at 1990 levels
2%
2%
2%
2%
NO, measurement variability at specified emission reduction
2%
2%
7%
NAb
Number of tests required to have 90% probability (ETV
requirement) of delecting expected NOx emission reduct ion''
3
1
1
NAb
Computed under the assumption of normally distributed measurements having known variability.
Additional tests may be required for sample standard deviations obtained during ETV.
^ = Not Available.
The fifth through the seventh rows (shaded) repeat the calculation for the NOx measurement.
No estimate of variability was available for the NOx measurement at an 85% reduction from
the 1990 level. Table 1 shows that, for both PM and NOx, three tests are sufficient to
distinguish between zero and emissions reductions as low as 5%, provided the estimates of
measurement variability are met in the actual ETV. One test would be sufficient to detect the
other emissions reductions, but three are required by ETV to provide a measured standard
deviation. No specific estimates of variability for HC emissions were available, but at high
emissions reductions (low absolute levels) the HC variability may be on the order of 30%.
Three tests would not be sufficient in that case. In the development of the test/QA plan, the
test laboratory will report its known measurement variability, and the expected number of
tests required. This estimate will be reviewed by the APCTVC.
Appendix C presents a sensitivity analysis for additional levels of measurement variability.
As explained in section 5.1.2, below, the number of tests may be increased further if an
applicant desires to reduce the width of the confidence interval on the mean emission
reduction.
5.1.2 Confidence Interval on the ETV Result
A consequence of the unavoidable variability in emissions measurements is that the mean of
a small number of measurements has a small probability of giving the true emission
reduction. In some cases it may not be possible to make a statistically-significant
determination that the measured emission reduction is different from zero. The results of
ETV measurements will be presented in verification statements and reports as the mean and
16
-------
Revision No.: 07
Date: January 22, 2002
95% confidence interval on that mean, calculated using the sample standard deviations
determined from the ETV test data. Because sample standard deviations are available, the t-
statistic is applied to compute the confidence intervals. This results in a broader error
distribution than indicated in Table 1.
Table 2 illustrates the results that would be obtained for the PM emissions measurement
variability numbers that were presented in Table 1 should those results be obtained during an
ETV. Two additional 20% reduction columns have been added as examples of the effects of
still higher variability at moderate reductions. (As was true in Table 1, the measurement
variability in Table 2 is expressed as a measurement standard deviation divided by the
baseline engine mean emission. However, relative to Table 1, the use of the t-distribution
broadens the confidence interval.) The first three rows of Table 2 are the same as those in
Table 1. Rows 5 through 7 give the two-sided 95% confidence intervals on the indicated
mean percent emissions reduction from the 1990 certification emission value for three, six,
and nine tests. For the 5% reduction column, Table 2 shows that the three test ETV
minimum does not detect the 5% reduction in that a zero reduction is within the 95%
confidence interval of -1.7 to 11.7% reduction. Six tests do detect the difference. Therefore,
between three and six tests would be required.
The fourth and fifth columns of Table 2 illustrate the effects of higher variability levels for
the controlled engine than were presented in Table 1 for a 20% emissions reduction, and
more clearly show the potential benefits of additional testing. This hypothetical additional
variability would arise from some interaction between the engine and the control device,
because the variability of the test method is affected by modest changes in emissions levels
such as are contemplated here. With 3 tests and an 8% test variability, the confidence
interval on the mean reduction would be ± 16.4%. Obtaining the same mean from 6 tests, at
the same variability, would give a confidence interval of ±6.9%, less than half the width of
that with 3 tests.
Table 2. 95% confidence intervals on example mean emission reductions relative to
baseline engine emissions at 1990 certification value.
Emission reduction, % of baseline engine
emission
5
20
20
20
50
85
PM measurement variability for baseline engine,
% of baseline engine emission
2
2
2
2
2
2
PM measurement variability for controlled
engine, % of baseline engine emission
2
2
4
8
3
10
Number of replicate verification tests
95% confidence intervals on emission reduction,
% of baseline engine emission
3
5±6.7
20±5.6
20±8.9
20±16.4
50±4.5
85±3.8
6
5±2.8
20±2.4
20±3.8
20±6.9
50±1.9
85±1.6
9
5±2.1
20±1.7
20±2.8
20±5.1
50±1.4
85±1.2
17
-------
Revision No.: 07
Date: January 22, 2002
When planning a test, the ETV program will use the applicant's estimated emissions
reduction, available test results, and experience with similar technologies to design an ETV
that is capable of detecting the emission reduction expected and meeting the applicant's goals
for confidence interval. However, the technology may not perform as expected. Section 8.0
addresses these circumstances.
5.2 General ETV Test Considerations
Unless specified otherwise, the general test considerations in this section will apply to all
technologies covered by this protocol.
5.2.1 Submittal of Technologies and ETV Sequence
The ETV process begins with the applicant indicating to the APCTVC or to a test laboratory
an interest in the ETV process. In either case, the second step is agreement between the
APCTVC, the applicant, and the test laboratory regarding the general form of the ETV
process and the responsibilities of all parties. This second step culminates with a formal
application for ETV to the APCTVC, providing the information about the technology
described in Section 4. The approximate cost of the ETV will be estimated and a contractual
relationship between the APCTVC and the applicant will be started.
The test laboratory will then develop the ETV test/QA plan, reviewing it with the applicant
and the APCTVC to ensure that it meets the requirements of the APCTVC, and, to the extent
possible, accomplishes the applicant's goals for ETV. The test/QA plan will then be
reviewed formally by the APCTVC and EPA. Test/QA plan development consists of a series
of sequential activities, and the test/QA plan must be approved before testing begins.
The applicant will provide the technology to be tested. The applicant has the responsibility to
arrange for provision of the engine on which the test will be conducted. Scheduling of the
ETV test will be done by the test lab. De-greening of the technology is the applicant's
responsibility, as is providing a de-greened unit and an aged unit for durability testing. The
applicant must also provide any special ancillary equipment required for operation of the
technology.
The applicant may choose to conduct private, proprietary test work at the test lab during the
same time period the ETV is taking place. Such arrangements are between the test lab and
the applicant. However, a clear differentiation is required between such private test work and
the ETV test itself. In addition, the APCTVC must have adequate notice regarding an ETV
test to be able to audit those tests if desired. The manner in which the start and end of the
ETV test will be established and when the APCTVC will be notified must be specified in the
test/QA plan. All ETV data must be reported.
18
-------
Revision No.: 07
Date: January 22, 2002
Following the ETV test, the test lab will report the data to the APCTVC, which will
incorporate those data into the ETV report and statement. These documents will be reviewed
with the test lab and the applicant to address comments and concerns as appropriate. The
draft report will then be submitted to the multi-step EPA review and revised as needed. The
ETV report and signed statement will be provided by the APCTVC to the applicant and EPA,
and posted on the ETV and APCTVC websites.
5.2.2 Highway Engine ETV
Testing of technology intended to control emissions from highway diesel engines will be
conducted generally under the provisions of 40 CFR Part 86 Subpart N. The primary
emissions measurements will be of NOx, PM, HCs, CO, and other FTP specified emissions.
C02 and other emissions and measurements described by this protocol are secondary
measurements. Each ETV test will consist of the same number of tests on the baseline
broken-in engine, the de-greened emission control technology, and the technology after
aging. The number of tests will include a single cold- and hot-start highway transient FTP
engine dynamometer test and may include additional hot-start tests in numbers determined
using the test design procedure described above. The emissions reductions will be reported
as the composite of the cold and mean of the hot starts, as required by the FTP, and as the
mean and confidence interval on multiple hot-start tests. The smoke test is not required. The
test parameters will be derived from the baseline engine mapping procedure. Emissions
reductions will be computed relative to the baseline engine emissions using the same test
cycle used during the baseline testing. As required by the FTP, tests that do not meet the FTP
QA requirements must be repeated.
The test/QA plan for a specific technology will specify the engines that will be tested and any
other aspects of the test that are specific to the technology or the test laboratory.
5.2.3 Highway Test Engines
The emissions characteristics of diesel engines of the same PM certification level are
arguably similar because the engine technologies required to achieve those certification levels
are likely to be similar. The emissions characteristics are also similar for engine families that
have a breadth of technology similar to the tested engine. These similarities may provide an
opportunity to group engines and reduce the ETV testing burden on applicants. As of the
date of this protocol, EPA-OTAQ has proposed allowing the ETV test results from a single
engine test to apply to multiple engines of different manufacturers. This is a decision that has
been reserved by EPA-OTAQ, who should be consulted for details regarding the engines
required to be tested as part of the ETV, the applicability of that ETV, and the magnitude of
the associated emissions reduction credit under the VDRP.
19
-------
Revision No.: 07
Date: January 22, 2002
5.2.4 ETY Testing of Nonroad Engines - General
The basic ETV test for nonroad engines will consist of the nonroad steady-state mode tests as
described in 40 CFR, Part 89, Subpart E. The primary emissions measurements will be of
NOx, PM, HC, CO and other FTP required measurements. C02 emissions and other
measurements described by this protocol are required secondary measurements. Each ETV
test will consist of one or more full steady-state multimode FTP test(s) on the baseline
broken-in engine, the de-greened technology, and the technology after aging. The test
condition will be derived from the baseline engine mapping procedure. The test may be
conducted over all modes of the Part 89 test or over portions selected by the applicant. The
emissions reductions will be reported as the composite mean and confidence interval of the
multiple test modes, computed as stated in the FTP.
Composite emissions reductions are computed relative to the composite baseline engine
emissions and will be reported in composite and mode-by-mode to allow flexibility in use of
the data for different engine applications.
5.2.5 Nonroad Test Engines
Following an approach similar to that used for the highway engines, the results of ETVs
conducted on individual nonroad engines may be applicable to broader groups of engines.
Technology applicants may select the engines on which to test their technologies.
Coordination with EPA-OTAQ is encouraged because the decision regarding acceptable
engine grouping has been reserved by EPA-OTAQ, who should be consulted regarding the
engines required to be tested as part of the ETV, the applicability of that ETV, and the
magnitude of the associated emissions reduction credit under the VDRP.
5.2.6 De-greening
For many hardware technologies, a brief period of use (de-greening) is needed to achieve a
stable emissions reduction that allows representative testing. For instance, the pressure drop
across a continuously regenerating PM filter reaches an equilibrium value. The de-greening
time period required varies for different technologies, but is on the order of 25 to 125 hours.
In all cases, the technology applicant must propose and justify the extent of the de-greening
process in the ETV application. The APCTVC office will review and comment on this
proposal, advise regarding the documentation requirements, and append the de-greening
process description to the technology test/QA plan. When complete, the actual process used
must be documented. A description of the de-greening process will be included in the ETV
report.
For purposes of this protocol, the de-greening time requirement will be specified by the
technology applicant as indicated by either previous testing or the requirements of the data
user. To allow flexibility for the applicant, de-greening is not required to be conducted at the
test lab. It may be done and documented by the applicant or conducted by the test lab by
arrangement with the applicant. In either case, the applicant will ensure that such de-
20
-------
Revision No.: 07
Date: January 22, 2002
greening activity documentation as start and stop date and time, engine and fuel description,
and operating conditions are available and signed by the person responsible for the activity.
De-greening may occur in a laboratory or during in-use field operations on an engine that is
equivalent to the proposed ETV test engine, or another engine of the same size which utilizes
the same engine technology (and thus falls within the range of the technology's stated
applicability).
5.2.7 Regeneration
Emissions control devices whose normal operation includes a periodic regeneration will be
tested over sufficient test cycles (described below) until a test cycle includes a "regeneration"
event. The verified emissions rate and emissions reduction achieved will be computed as the
time-weighted average of the emissions rate and reduction achieved over this complete
operating cycle.
Technologies that are continuously regenerated in normal operation are not the subject of this
paragraph. Continuous cleaning PM filters should reach a stable operating range (as
evidenced by near constant pressure drop) before testing, which may be accomplished in the
de-greening activity.
5.2.8 Device Scaling
The performance of most retrofit air pollution control technologies is affected by the relative
sizes of the tested technologies and the engines. Therefore, the use of single engine ETV
tests as predictive of performance over segments of the total engine population requires the
availability of non-proprietary application sizing information and that the technology be
labeled. Sizing information may be provided in sizing charts [e.g., model AA1 is suitable for
37 to 74 kW (50 to 100 hp) engines, model AA2 to 74 to 112 kW (100 to 150 hp) engines] or
through the release of basic design information (e.g., space velocity, catalyst loading). The
purpose of this requirement is to allow the APCTVC and the potential technology user (the
public) to evaluate the applicability of the verified technology to other applications.
Applicants who utilize application sizing charts must verify their technologies on an engine at
the most challenging end of the range of applicability. ETV reports and statements will
report the results and the design guidance required to independently extend application of the
technology to other engines in the same engine grouping. Device scaling and engine
selection decisions will be explained and documented in the test/QA plan to the extent
possible and will be reported in the ETV report. The ETV report and statement will report
the results and the design guidance required to independently extend application of the
technology to other engines in the same engine grouping.
5.2.9 Durability
The emissions reductions measured for the de-greened air pollution control device will not
account for changes in product performance that may occur as the device ages. For
21
-------
Revision No.: 07
Date: January 22, 2002
participation in the VDRP, additional testing of an "aged" control device is required by EPA-
OTAQ. If done as part of the ETV program, the "aged" device test will meet the same data
requirements as a new device test. This testing is followed by extrapolation of the "initial de-
greened" and "aged" verified performance measurement to the end of the warranted life of
the control technology using a method determined by EPA-OTAQ. The aging process details
and the extrapolation of the data to an "end of life point" are not part of this protocol.
Provided for information only, the description below is current as of December. 2001 and
may change. The details should be confirmed prior to ETV.
1) Aging entails subjecting the control device to operating conditions that cause normal
wear equivalent to at least 33% of the Minimum Durability Testing Period established
in the California "verification" procedure for diesel retrofit systems (CARB, 2001).
Table 3 provides the testing periods that were current as of the date of this protocol.
The applicant should verify the testing period current at the date of ETV. The
technology applicants must conduct the aging process on their technology. They have
discretion to tailor this process to product requirements. Applicants may age an
exhaust catalyst by using it during real-world operation, or through accelerated bench
testing. It is expected that applicants will submit identical parts (one in a de-greened
state, one aged to at least 33% expected full-life) so that testing with the baseline may
occur sequentially. However, applicants may conduct the de-greened and aged
technology tests as separate tests, in which case the baseline engine test must be
repeated. All aging protocols must accompany the ETV application and explain the
technical basis for stating the aging protocol results in at least 33% full-life aging. If
real-world aging is performed, the application must describe and provide
documentation of the usage and maintenance history of the aged unit as well as the
engine with which it was aged.
2) Emissions testing using the aged device will be part of the ETV process and will
follow the same procedures applied to the de-greened technology. First, baseline
emissions testing shall be conducted, followed by testing with the de-greened device,
and then finally testing the aged device with a re-test of the baseline engine if
technically feasible. An applicant may elect to age a unit to its full useful life prior to
testing and request ETV on that basis.
3) The full useful-life reduction capabilities will be estimated by EPA-OTAQ following
procedures they will provide to applicants.
5.2.10 Standard Diesel Test Fuel
The standard diesel test fuel for highway engines should meet the EPA specifications
outlined in 40 CFR Part 86.1313-98 with the exception of the sulfur content. For nonroad
engines, the test fuel
22
-------
Revision No.: 07
Date: January 22, 2002
Table 3. Minimum durability testing periods (CARB, 2001)
Engine Type
Engine Size
Minimum Durability Testing Period
Highway
Light heavy-duty, generally 70 to 170 hp,
GVWRa normally less than 19,500 lb.
5 yr or 60,000 miles (or equivalent
time in hours)
Medium heavy-duty, generally 170 to 250 hp,
GVWRa normally from 19,500 to 33,000 lb.
5 yr or 100,000 miles (or equivalent
time in hours)
Heavy heavy-duty, generally exceeds 250 hp,
GVWRa normally exceeds 33,000 lb.
5 yr or 150,000 miles (or equivalent
time in hours)
Nonroad and
Stationary
Under 25 hp and for constant speed engines rated
under 50 hp with rated speeds greater than or
equal to 3,000 rpm
3 yr
1,600 hr operation
At or above 25 hp and under 50 hp
4 yr
2,600 hr operation
At or above 50 hp
5 yr
4,200 hr operation
aGVWR = Gross Vehicle Weight Rating
should be that described in 40 CFR Part 89.330 or another fuel as specified by the control
technology applicant. Because the performance and durability of many types of diesel retrofit
technologies are affected by the sulfur content of the diesel fuel, applicants should specify the
maximum sulfur level of the fuel for which their technologies are designed. The sulfur
content of the ETV test fuel should be no less than 66% of the stated maximum sulfur
content. (Because refinery and blending operations are such that very low sulfur content
control is difficult, test fuel with a sulfur content of 15 ppm or below is not constrained to the
"66% rule". The actual sulfur content of the test fuel batch is to be reported.) Other test fuels
should meet the applicable EPA specifications outlined in 40CFRPart 86.1313. Adding
sulfur to the fuel by the test laboratories is permissible if necessary to achieve the required
fuel sulfur content for either baseline or controlled engine tests.
During ETVs, baseline engines should be fueled with the standard fuels that are
representative of nominal in-use fuels and controlled engines with low sulfur versions of the
standard fuels that are representative of the applicant's recommended or required fuel.
5.2.11 Engine Performance and Power
Engine performance and power will be measured and reported for both the baseline engine
(without the control device installed) and the engine with the control technology. Engine
performance measurements will be made with the engine operating at maximum power (rated
conditions) and at peak torque as defined in the applicable FTP.
23
-------
Revision No.: 07
Date: January 22, 2002
5.2.12 Fuel Consumption
Fuel consumption will be measured for both the baseline engine (without the control device
installed) and the engine with the control device installed to determine the effect of the
technology on fuel consumption. The engine fuel consumption measurements will be made
at maximum power at rated conditions and at peak torque at intermediate speed. Multiple
tests, if available, will be averaged. The results will be reported as a fractional increase or
decrease relative to the baseline engine. Also reported will be fuel economy and brake
specific fuel consumption from testing by the applicable FTP.
5.2.13 Back-pressure
Within the test cell, baseline engine back-pressure will be set to the value required by the
applicable FTP (highway or nonroad). Once installed for the ETV test, the back-pressure of
the retrofit control technology may be greater than the FTP requirement. If so, the ETV test
will be conducted without adding additional back-pressure; if not, the test cell will be
adjusted to meet the FTP requirements. Because back-pressure of a retrofit control
technology may affect the performance of an engine, the ETV test will measure and report
back-pressure with the control device at full-load and rated speed. Back-pressure will be
measured and reported for both the baseline engine (as set for the FTP test without the
technology installed) and the engine with the de-greened or aged control technology installed.
5.2.14 Control Technology Operating Temperature
Inlet (engine exhaust) and discharge temperatures must be measured for technologies that are
either dependent on specific operating temperature ranges or affected by engine or exhaust
temperatures.
5.2.15 Other Measurements and Conditions
ETV of technologies that may produce secondary pollutants or have other secondary effects
must include measurement of those pollutants in the ETV test. C02 emissions, while not
primary, must be measured using instrumentation and/or estimated from a carbon balance
from the fuel usage. ETV must include the appropriate measurements for technologies that
require other specific operating conditions or affect emissions over only a limited range of a
particular pollutant. For example, because the long-term operation of DECs and PM filters is
affected by the soluble organic fraction (SOF) of the PM, SOF must be measured during ETV
testing for these and similar technologies. The details of these non-critical measurements and
their QA goals will be part of the test/QA plan.
24
-------
Revision No.: 07
Date: January 22, 2002
5.3 ETV of DECs and PM Filters
5.3.1 Technology Description
For the purposes of this GVP, DECs are defined as devices made of active catalyst material
(often containing precious metal) deposited on a support medium. The engine exhaust passes
through the device, where the pollutants catalytically react and more environmentally
acceptable reaction products are exhausted. PM filters mechanically trap PM emissions and
subsequently oxidize them. Filters may also employ a catalyst for gaseous emissions control
and to enhance the oxidation of the collected PM.
Physically, DECs and PM filters have the general appearance of a large muffler and are
placed in the engine exhaust at approximately the same location. They are therefore well-
suited to retrofit applications. No external liquid or gaseous reactants are required for them
to function.
DECs have been reported to decrease HC and CO emissions by over 50%. They decrease
PM emissions by about 20%. PM filters have been reported to decrease PM and HC
emissions by 80% or more and to decrease CO at about the level achieved by exhaust
catalysts. NOx emissions may be decreased slightly by exhaust catalysts.
Operational issues with DECs and PM filters are primarily achieving or maintaining adequate
temperature within the device to complete the reactions and functioning satisfactorily with an
accumulation of catalyst poisons and/or non-combustible lubricant ash residue. Lubricant
ash and wear metals can largely be removed by infrequent "blowing" with compressed air.
Guidance on technique and frequency of such cleaning should be provided in the ETV
application.
5.3.2 Test Considerations Specific to DECs and PM Filters
The general considerations in Section 5.2 apply to DEC and PM filter ETV testing. ETVs of
DECs and PM filters require measurement of the SOF content of the exhaust stream prior to
entering the DEC. (The SOF may be measured for PM collected during the baseline engine
test. The method to be used is the Test Methodfor Soluble Organic Fraction (SOF)
Extraction [CARB, 1989] or a equivalent method recognized by EPA-OTAQ.) Both DECs
and PM filters require measurements of inlet and outlet temperatures.
5.4 ETV of Engine Modifications
5.4.1 Technology Description for Engine Modifications
Engine modifications are defined as a change integral to an engine (e.g., component
modifications or change or engine control system calibration change) that is proposed as an
emissions reduction technology.
25
-------
Revision No.: 07
Date: January 22, 2002
5.4.2 Test Considerations Specific to Engine Modifications
The general considerations in Section 5.2 apply to engine modification ETV. Baseline
engine performance will be measured prior to modification, followed by modification of the
baseline engine in accordance with the application instructions, operation for a de-
greening/stabilization period, and then measurement of the de-greened technology's
performance. An aged engine modification technology's emissions performance will be
evaluated as described in Section 5.2.9. However, there is no requirement to re-run the
baseline engine which was modified. It cannot be shown that a modified engine can be
returned to its original state to again serve as a baseline.
Engine modifications whose use is warranted by engine manufacturers will be taken to have
no adverse impact on engine durability, and that endorsement will be noted in the verification
report (VR) and verification statement (VS) along with the verified emissions reductions for
the aged engine. The lack of any verified knowledge regarding the impact on engine
durability of technologies not warranted by the engine manufacturer will similarly be noted.
6.0 REPORTING AND DOCUMENTATION
This section describes the procedures for reporting data in the VR and VS. The specifics of
what data must be included and the format in which the data must be included are addressed
in this section (e.g., QA/QC summary forms, raw data collected, photographs/slides/video
tapes). The VR for each technology will include near it's beginning the VS. The VS is a
short summary of the ETV results. An example draft is attached as Appendix A. The VR,
including the VS, will be written by the APCTVC based on the test report submitted by the
testing organization. The VR and VS will be reviewed by the APCTVC and the technology
applicant before being submitted to EPA for review and approval as specified in the ETV
QMP.
6.1 Reports
Based on the test report from the laboratory, the APCTVC will prepare the draft VR, which
includes the following topics:
1) VS;
2) Introduction;
3) Description and identification of product tested;
4) Procedures and methods used in testing;
5) Statement of operating range over which the test was conducted;
6) Summary and discussion of results as required to:
a) Support the VS,
b) Explain and document necessary deviations from test plan, and
c) Discuss QA issues;
7) Conclusions and recommendations;
8) References; and
26
-------
Revision No.: 07
Date: January 22, 2002
9) Appendices:
a) QA/QC activities and results,
b) Raw test data, and
c) Equipment calibration results.
The VS will include the following:
10) Technology applicant's name and technology's descriptive information,
11) Summary of ETV test program,
12) Results of the ETV test,
13) Any limitations of the ETV results, and
14) Brief QA statement.
Review and approval of the draft ETV report and statement are as described in Section 3.0.
A draft VS is attached to this protocol as Appendix A.
6.2 Data Reduction
Data from measurements made as part of the ETV test will be reported as emissions rates in
g/kWh (g/bhp-hr) and as percentage emission reductions from the baseline engine. The
confidence limits will be presented as well as the mean emissions reduction, as discussed in
Section 5.1.2. When they would be helpful to the mobile sources community because of
established usage, the appropriate English Engineering units will be supplied parenthetically.
7.0 DISSEMINATION OF ETV REPORTS AND STATEMENTS
After a retrofit control technology has been tested and the draft VR and VS received from the
testing organization, the APCTVC will send a draft of both to the applicant for review prior
to submission to EPA-ORD and release of the approved report to the public. This gives the
applicant opportunity to review the results, test methodology, and report terminology while
the drafts remain working documents and are not publically accessible. The applicant may
submit comments and revisions on the draft statement and report to the APCTVC. The
APCTVC will consider these comments and may suggest revisions of its own.
After incorporating appropriate revisions, the draft final VR and VS will be submitted to
EPA-ORD for review and approval. A signed original VS and VR will be filed and retained
by the APCTVC, and signed originals will also be provided to the applicant and to EPA.
Three additional paper copies of the ETV report will be provided to the applicant. Further
distribution of the ETV report, if desired, is at the applicant's discretion and responsibility.
However, approved VSs and VRs will be posted on the ETV web site for public access
without restriction. The VR report appendices will not be posted on the website, but will be
publically available from the APCTVC.
27
-------
Revision No.: 07
Date: January 22, 2002
8.0 APPLICANT'S OPTIONS SHOULD A TECHNOLOGY PERFORM BELOW
EXPECTATIONS
ETV is not a technology research and development program; technologies submitted for ETV
are to be commercial-ready and with well-understood performance. Tests that meet the ETV
data quality requirements (a valid FTP test) are considered valid and suitable for publishing.
However, a technology may fail to meet the applicant's expectations. Based on limited
testing, for instance, the applicant might expect an emission reduction of 30% ± 7% result.
However, the actual ETV result from the more complex FTP test cycle might be 20% ± 15%.
The APCTVC will use its experience to avoid this situation, but it is possible. In this case,
the applicant may choose to schedule additional tests, may accept the result and complete the
verification, or may request that a VS not be issued. However, ETV reports are always in the
public domain. VRs will be written and will be available from EPA-ORD for review by the
public regardless of a request not to issue a verification statement, and summaries of the
results of ETV testing may be published by ETV and/or EPA-OTAQ.
Another example of a technology not meeting expectations would be an expected result of a
mean of 10% reduction with a confidence interval of ± 5% reduction, but an actual ETV
mean reduction of 5% with a confidence interval of ± 7% reduction. In this case, the data are
insufficient for ETV to verify that the technology does anything at all. Additional tests must
be scheduled and a statistically significant reduction obtained for a VS to be issued. Inability
to detect a statistically significant emission reduction (equivalently, failure to have sufficient
tests) will prevent completion of the ETV, and the results of the ETV will be reported
publically stating that performance could not be distinguished from 0% reduction. An ETV
statement will not be issued in these cases.
In either of these cases, the applicant may improve the product and resubmit it under a new
model identification for ETV testing. ETV Statements for acceptable tests of the new
product will be issued as they are processed by the APCTVC and EPA-ORD (except that the
results for several identical tests performed in rapid succession will all be released at the
same time).
9.0 LIMITATIONS ON TESTING AND REPORTING
To avoid having multiple ETV reports for the same product and to maintain the ETV testing
as a cooperative effort with the applicant, the following restrictions apply to ETV testing
under this protocol:
1) Applicants may submit only products they manufacture or whose distribution
they control. Applicants may not submit for ETV testing control devices whose
use is not in their control except with the agreement of the manufacturer or
vendor.
2) For a given product (e.g., brand and model), APCTVC policy is that only one
ETV report and statement will be issued for any single application.
3) Air pollution control technology frequently performs differently in different
28
-------
Revision No.: 07
Date: January 22, 2002
applications. Applicants may request additional tests of essentially identical
technology if it is being applied to pollution sources that are clearly different
from those for which ETVs have been obtained.
10.0 REQUIREMENTS FOR TEST/QA PLAN
10.1 Quality Management
All testing organizations participating in this ETV Program must meet the QA/QC
requirements defined below and have an adequate quality system to manage the quality of
work performed. Documentation and records management must be performed according to
the ETV Quality and Management Plan for the Pilot Period (1995-2000) (ETV QMP, EPA,
1998a). Testing organizations must also perform assessments and allow audits by the
APCTVC (headed by the APCT QA Officer) and EPA corresponding to those in Section 11.
All testing organizations participating in the Retrofit Air Pollution Control Technologies for
Highway and Nonroad Use Diesel Engines Program must have an ISO 9000-accredited (ISO,
1994) or ANSI E4-compliant (ANSI, 1994) quality system and an EPA- or APCTVC-
approved QMP.
10.2 Quality Assurance (QA)
All ETV testing will be done following an approved test/QA plan that meets EPA
Requirements for Quality Assurance Project Plans (EPA, 2001a) and Part B, Section 2.2.2 of
EPA's ETV QMP (EPA, 1998a). These documents establish the requirements for test/QA
plans and the common guidance document, Guidance for Quality Assurance Project Plans
(EPA, 1998b), provides guidance on how to meet these requirements. The APCT Quality
Management Plan (RTI, 1998) implements this guidance for the APCTVC.
ETVs conducted under this generic protocol utilize test procedures described in the FTP (40
CFR Part 86 for highway engines and 40 CFR Part 89 for nonroad engines). The test/QA
plan must describe, in adequate detail, how the FTP test methods are implemented by the
testing organization. Replication of the FTP text is neither expected nor desired. The
test/QA plan should reference the FTP in detail, by section and subsection, as appropriate for
the topic under consideration. Any deviations from the FTP must be identified and
explained. Internal standard operating procedures (SOPs) may be referenced provided they
are available for audit review. (SOPs need not be incorporated into the test/QA plan except
by reference. If considered proprietary to the test organization, they should be clearly
marked.) When the FTP offers alternative test procedures or equipment, the test/QA plan
must identify the alternative implemented. Similarly, if a range of operating parameters is
allowed by the FTP, the specifics of the particular implementation must be provided. For a
test organization with multiple test cells, these details may be tabulated and incorporated by
attaching a table and identifying the test cell on the test report. Steps the testing organization
will take to ensure acceptable data quality in the test results are also identified in the test/QA
plan. As above, detailed reference to SOPs, the calibration portions of the FTP, or other
29
-------
Revision No.: 07
Date: January 22, 2002
available documents is encouraged. Any needed SOPs will be developed in accordance with
Guidance for Preparing Standard Operating Procedures (SOPs) (EPA, 2001b.)
The testing organization must prepare a test/QA plan and submit it for approval by the
APCTVC. The test/QA plan must be approved before the test organization can begin ETV
testing.
A test/QA plan contains the elements listed below, the contents of which may be stand alone
or include references to the FTP or other widely distributed and publically available sources.
Legible hand-notated diagrams from the FTP are acceptable. If specific elements are not
included, an explanation for not including them must be provided.
9
10
11
12
13
14
15
16
17
18
19
20
Title and approval sheet;
Table of contents, distribution list;
Test description and test objectives;
Identification of the critical measurements, data quality objectives (DQOs) and
indicators, test schedule, and milestones;
Organization of test team and responsibilities of members of that team;
Documentation and records;
Test design;
Sampling procedures;
Sample handling and custody;
Analytical procedures;
Test-specific procedures for assessing data quality indicators;
Calibrations and frequency;
Data acquisition and data management procedures;
Internal systems and performance audits;
Corrective action procedures;
Assessment reports to EPA;
Data reduction, data review, data validation, and data reporting procedures;
Reporting of data quality indicators for critical measurements;
Limitations of the data; and
Any deviations from methods from this generic verification protocol.
The APCTVC will provide a test/QA plan template that illustrates it's expectations.
10.3 Additional Requirements to be Included in the Test/QA Plan
The test/QA plan must include or reference a diagram and description of the extractive
gaseous measurement system to be used for the testing and a list of the reference analyzers
and measurement ranges to be used for quantifying the concentrations of all gaseous
compounds to be measured, including both primary and ancillary pollutants.
The test/QA plan must include or reference a schematic drawing showing all sample and test
locations, including the inlet and outlet to the technology sampling locations. The location of
30
-------
Revision No.: 07
Date: January 22, 2002
flow disturbances and the upstream and downstream distances from the sampling ports to
those flow disturbances must be noted. The number of traverse points that will be sampled
must be provided.
The test/QA plan must include or reference the appropriately detailed descriptions of all
measuring devices that will be used during the test.
The test/QA plan must explain or reference the specific techniques to be used for monitoring
process conditions appropriately for the source being tested. It must also note the techniques
that will be used to estimate any other operational parameters.
The test/QA plan must include and document estimates of historical measurement variability
that will be used, as discussed in Section 5.1.1, to compute the number of tests required and
provide confidence intervals on single test ETVs.
11.0 ASSESSMENT AND RESPONSE
Each independent test laboratory must conduct internal assessments of its quality and
technical systems and must allow external assessments of these systems by the APCTVC QA
personnel and by EPA QA personnel. After an assessment, the test laboratory will be
responsible for developing and implementing corrective actions in response to the
assessment's findings.
As appropriate, the APCTVC and/or EPA will conduct assessments to determine the testing
organization's compliance with its test/QA plan. The requirement to conduct assessments is
specified in EPA's Quality and Management Plan for the Pilot Period (1995 - 2000) (EPA,
1998a), and in RTI's QMP (RTI, 1998). EPA will assess RTI's compliance with RTI's
test/QA plans. RTI will assess the compliance of other organizations with their test/QA
plans. The assessments will be conducted according to Guidance on Technical Audits and
Related Assessments for Environmental Data Operations (EPA, 2000) and Guidance on
Assessing Quality Systems (EPA, 2001.)
11.1 Assessment Types
Quality system assessment - Qualitative assessment of a particular quality system to
establish whether the prevailing quality management structure, policies, practices, and
procedures meet EPA requirements and are adequate for ensuring that the type and quality of
measurements needed are obtained.
Technical systems audit - Qualitative on-site audit of the physical setup of the test. The
auditors determine the compliance of testing personnel with the test/QA plan.
Performance evaluation audit - Quantitative audit in which measurement data are
independently obtained and compared with routinely obtained data to evaluate the accuracy
(bias and precision) of a measurement system.
31
-------
Revision No.: 07
Date: January 22, 2002
Audit of data quality - Qualitative and quantitative audit in which data and data handling
are reviewed and data quality and data usability are assessed.
Surveillance audit - Observation of ongoing work to document conformance with specified
requirements and/or procedures, such as those given in a test/QA plan or SOP.
11.2 Assessment Frequency
Activities performed during ETV performance operations that affect the quality of the data
shall be assessed regularly and the findings reported to management to ensure that the
requirements stated in the generic verification protocols and the test/QA plans are being
implemented as prescribed.
The types and minimum frequency of assessments for the ETV Program are listed in Part A
Section 9.0 of EPA's Quality and Management Plan for the Pilot Period (1995 - 2000).
Tests conducted by the APCTVC will have at a minimum the following types and numbers of
assessments:
1) Technical systems audits and surveillance audits: Self-assessments (qualitative)
by test laboratory as provided for in the test/QA plans and at least one
independent assessment of the test laboratory.
2) Performance evaluation audits: Self-assessments (quantitative) by test laboratory
as provided for in the test/QA plans and at least one independent assessment of
the test laboratory.
3) Audits of data quality: Self-assessments (quantitative and qualitative) by the test
laboratory of at least 10% of all the ETV data with detailed reports of the audit
results to be included in the data packages sent to the APCTVC for review.
4) Assessements of quality systems: Self-assessments by the test laboratory as
provided for in the test/QA plans and at least one independent assessment of the
test laboratory.
The independent assessments of tests conducted by RTI will be performed by EPA. The
independent assessments of other organizations will be by RTI.
11.3 Response to Assessment
When needed, appropriate corrective actions shall be taken and their adequacy verified and
documented in response to the findings of the assessments. Data found to have been taken
from non-conforming technology shall be evaluated to determine its impact on the quality of
the required data. The impact and the action taken shall be documented. Assessments are
conducted according to procedures contained in the APCT QMP. Findings are provided in
audit reports. Responses by the test laboratory to adverse findings are required within 10
working days of receiving the audit report. Follow up by the auditors and documentation of
responses are required.
32
-------
Revision No.: 07
Date: January 22, 2002
12.0 SAFETY MEASURES
12.1 Safety Responsibilities
The test laboratory's project leader is responsible for ensuring compliance with all applicable
occupational health and safety requirements. Each individual staff member is expected to
follow the requirements and identify personnel who deviate from them and report such action
to their supervisor.
12.2 Safety Program
The test company must maintain a comprehensive safety program and ensure that all test
personnel are familiar with and follow it.
13.0 REFERENCES
ASQC. AMERICAN NATIONAL STANDARD Specifications and Guidelines for Quality
Systems for Environmental Data Collection and Environmental Technology Programs.
ANSI/ASQC E4-1994. American Society for Quality Control. Milwaukee, WI. 1994.
CARB. Proposed Regulation Order (DRAFT) Verification Procedure for Diesel Retrofit
Systems, http://www.arb.ca.gov/msprog/mailouts/msc0114/msc0114attl.pdf. California Air
Resources Board, El Monte, CA. August, 2001.
CARB. Test Methodfor Soluble Organic Fraction (SOF) Extraction, California Air
Resources Board, El Monte, CA. April, 1989.
EPA. Environmental Technology Verification Program, Quality and Management Plan for
the Pilot Period (1995-2000). EPA 600/R-98/064. http://www.epa.gov/etv/qmp.htm.
National Risk Management Research Laboratory - National Exposure Research Laboratory,
Office of Research and Development, U.S. Environmental Protection Agency. Cincinnati,
OH. May 1998a.
EPA. EPA Guidance for Quality Assurance Project Plans. EPA QA/G-5, EPA/600/R-
98/018, http://es.epa.gov/qualitv/qs-docs/g5-final.pdf. Office of Research and Development,
U. S. Environmental Protection Agency. Washington, DC. February 1998b.
EPA. EPA Requirements for Quality Assurance Project Plans. EPA QA/R-5, EPA/240/B-
01/003. http://www.epa.gov/qualitv/qs-docs/r5-final.pdf. Office of Environmental
Information, U. S. Environmental Protection Agency. Washington, DC. March 2001.
EPA. Guidance on Technical Audits and Related Assessments for Environmental Data
Operations, EPA QA/G-7, EPA/600/R-99/080, http://www.epa.gov/quality/qs-docs/g7-
final.pdf. Office of Environmental Information, U.S. Environmental Protection Agency.
Washington, DC. January 2000.
33
-------
Revision No.: 07
Date: January 22, 2002
EPA. Guidance for the Preparing Standard Operating Procedures (SOPs). EPA QA/G-6.
EPA 240/B-01/004. http://www.epa.gov/qualitv/qs-docs/g6-final.pdf. Office of
Environmental Information, U.S. Environmental Protection Agency. Washington DC.
March, 2001.
EPA. Guidance on Assessing Quality Systems (Quality Staff Draft). EPAQA/G-3. Office
of Environmental Information, U.S. Environmental Protection Agency. Washington, DC.
August, 2001.
ISO. ISO 9001-1994, Quality Systems Model for Quality Assurance in Design, Development,
Production, Installation, and Servicing. International Organization for Standardization.
Geneva, Switzerland. In USA, American National Standards Institute, New York, NY. 1994.
RTI. Verification Testing of Air Pollution Control Technology - Quality Management Plan.
Air Pollution Control Technology Program. J. R. Farmer, Program Director, Research
Triangle Institute, Research Triangle Park, NC. 1998.
U.S. Government. Protection of Environment. Title 40, Part 86, Code of Federal
Regulations, as of July 1, 1999. Federal Register. Washington, DC. 1999.
U.S. Government. Protection of Environment. Title 40, Part 89, Code of Federal
Regulations, as of July 1, 1999. Federal Register. Washington, DC. 1999.
34
-------
Revision No.: 07
Date: January 22, 2002
APPENDIX A: EXAMPLE VERIFICATION STATEMENT
Appendix A is an example verification statement written for a generic PM filter control
technology. The technology is assumed to be directed at a highway use engine. It is assumed
to be an efficient control device, requiring only a single test by the minimum-number-of-tests
calculation. The PM values in Table A-2 are taken from the 85% reduction column of Table
3, while the other values are completely hypothetical.
This generic verification statement is intended only to show the form of a verification
statement. It will require modification for each technology verified, depending on the details
of that technology's design, construction, and operation. The test/QA plan written for each
test will include a draft verification statement customized for the technology actually being
tested. The text of that specific verification statement will address the significant parameters
that apply to the technology tested.
A-l
-------
THE ENVIRONMENTAL TECHNOLOGY VERIFICATION
PROGRAM
SEFA CTl/ BRTI
U.S. Emironmental Protection Agency ^ ^ INTERNATIONAL
ETV Joint Verification Statement
TECHNOLOGY TYPE:
MOBILE DIESEL ENGINE AIR POLLUTION CONTROL
APPLICATION:
CONTROL OF EMISSIONS FROM MOBILE DIESEL
ENGINES IN (HIGHWAY) (NONROAD) USE BY
(TECHNOLOGY TYPE)
TECHNOLOGY NAME:
TECHNOLOGY NAME
COMPANY:
COMPANY NAME
ADDRESS:
ADDRESS
PHONE: (000) 000-0000
CITY, STATE ZIP
FAX: (000) 000-0000
WEB SITE:
http://www.company.com
E-MAIL:
some.one@company.com
The U.S. Environmental Protection Agency (EPA) has created the Environmental Technology
Verification (ETV) Program to facilitate the deployment of innovative or improved environmental
technologies through performance verification and dissemination of information. The goal of the
ETV Program is to further environmental protection by substantially accelerating the acceptance and
use of improved and cost-effective technologies. ETV seeks to achieve this goal by providing high
quality, peer reviewed data on technology performance to those involved in the design, distribution,
financing, permitting, purchase, and use of environmental technologies.
ETV works in partnership with recognized standards and testing organizations; stakeholder groups
which consist of buyers, vendor organizations, permitters, and other interested parties; and with the
full participation of individual technology developers. The program evaluates the performance of
innovative technologies by developing test plans that are responsive to the needs of stakeholders,
conducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing
peer-reviewed reports. All evaluations are conducted in accordance with rigorous quality assurance
protocols to ensure that data of known and adequate quality are generated and that the results are
defensible.
The Air Pollution Control Technology Verification Center (APCTVC), one of six centers under the
ETV Program, is operated by the RTI, in cooperation with EPA's National Risk Management
Research Laboratory. The APCTVC has evaluated the performance of a TYPE
emission control technology for mobile diesel engines, the TECHNOLOGY by COMPANY NAME.
A-2
-------
Revision No.: 7
Date: January 22, 2002
ETV TEST DESCRIPTION
All tests were performed in accordance with the APCTVC Generic Verification Protocol for Diesel
Exhaust Catalysts, PM Filters, and Engine Modification Technologies for Highway and Nonroad
Use Diesel Engines and the specific technology test plan "ETV Test/QA Plan for TECHNOLOGY
NAME' These documents include requirements for quality management, quality assurance,
procedures for product selection, auditing of the test laboratories, and test reporting format.
The mobile diesel engine air pollution control technology was tested at TEST LABORATORY. The
performance verified was the percentage emission reduction achieved by the technology for
particulate matter (PM), nitrogen oxides (NOx), hydrocarbons (HCs), and carbon monoxide (CO)
relative to the performance of the same baseline engine without the technology in place. Operating
conditions were documented and ancillary performance measurements were also made. The basic
modules of the test procedure are found in the federal test procedures (FTPs) for highway engines
(40CFR, Part 86, Subpart N) and nonroad engines (40CFR, Part 89, Subpart E). For highway use, a
single full FTP test was conducted, augmented by additional hot start transient tests as needed to
meet the requirements of the generic verification protocol (GVP). For nonroad use, one or more
multimode tests were conducted as described in the GVP. A summary description of the ETV test is
provided in Table A-l.
Table A-l. Summary of the conditions for ETV test of TECHNOLOGY NAME on
ENGINE DESCRIPTION.
Test Conducted
Highway Transient FTP
Engine Family
ENGINE MFGR NAME Series XXXYYY, 111 operating hours prior to test
Engine Size
YYY kW (XXX hp)
Technology
ACME Mark If PM Trap, Model AA1 for diesel engines up to 150 hp on
standard fuel
Technology description
Honeycomb PM filter packaged in an muffler-sized can for retrofit
installation by a moderately skilled mechanic. No engine modifications
required.
Test cycle or mode
description
1 full FTP test (1 cold start and 1 hot start)
Test fuel description
EPA standard diesel per 40 CFR Part 86.1313-98
Critical measurements
PM, NOx, HCs, and CO per the FTP
Ancillary measurements
C02, backpressure at engine exhaust port, exhaust temperature, fuel
consumption, regeneration requirements
VERIFIED TECHNOLOGY DESCRIPTION
This verification statement is applicable to the TECHNOLOGY NAME (to include model number and
other identifying information as needed), which is an cordierite-based PM filter manufactured by
MANUFACTURER NAME. TECHNOLOGY NAME is packaged and marketed for particular engine
families (for example, Model AA1 is properly sized for the A1A1 engine) or as a unit suitable for use
on engines below a particular diesel power rating. The unit whose performance was verified was the
Model AA1, which is rated for engines up to YYY kW (XXX hp) fueled by standard diesel fuel.
This verification statement describes the performance of TECHNOLOGY NAME on the diesel engine
A-3
-------
Revision No.: 7
Date: January 22, 2002
identified in Table A-l. The retrofit device, TECHNOLOGY NAME, is expected to provide similar
emissions control performance on other engines having similar exhaust stream characteristics
(similar fuel and engine technology) when properly sized for the application.
VERIFICATION OF PERFORMANCE
TECHNOLOGY NAME achieved the emissions reduction shown in Table A-2 at the stated
conditions. The number of required ETV tests was estimated to be one complete FTP test, and this
Table A-2. Verified emissions reductions for hypothetical TECHNOLOGY NAME
Test Engine: Manufacturer name
Model No. AA1
Degreened (XX hours)
Technology Test
Aged (XXXX hours)
Technology Test
Baseline
Engine
Controlled
Engine
Emissions
Reduction
O/a
/O
Baseline
Engine
Controlled
Engine
Emissions
Reduction
O/a
/O
Critical Measurements of Emissions
Hot Start PM, g/bkWh (g/bhp-hr)
0.804
(0.600)
0.012
(0.090)
85±3
0.790
(0.590)
0.012
(0.091)
84.6±3
Composited PM, g/bkWh (g/bhp-hr)
Hot Start NOx, g/bkWh (g/bhp-hr)
Composited NOx, g/bkWh (g/bhp-hr)
Composited HC, g/bkWh (g/bhp-hr)
Composited CO, g/bkWh (g/bhp-hr)
Ancillary Measurements
Engine Power, kW (hp)
Peak Torque, N-m (lbrft)
Composited C02, g/bkWh (g/bhp-hr)
PM SOF, g/bkWh (g/bhp-hr)
Exhaust Flow, L/min (ft3/min)
Exhaust Temperature, °C (°F)
Backpressure, kPa (in. Hg)
Fuel Usage, % reduction
Technology In/Out Temp., °C (°F)
Regeneration
brief description
Maintenance Schedule
brief description
Comments
3 units of rows as indicated except shaded columns in %
estimate was confirmed by the ETV test results. Table A-2 may include ETV results for both the
initial operation (degreened) and for the technology following the stated period of aging. For the
A-4
-------
Revision No.: 7
Date: January 22, 2002
purposes of determining the status of the technology in regard to EPA's voluntary retrofit program,
the prospective user is encouraged to contact EPA-Office of Transportation and Air Quality (OTAQ)
or visit the retrofit program web site at http://www.epa. gov/otaa/retrofit/.
The APCT QA Officer has reviewed the test results and quality control data and has concluded that
the data quality objectives given in the generic verification protocol and test/QA plan have been
attained.
During the ETV tests, EPA or APCTVC quality assurance staff conducted technical assessments at
the test laboratory. These confirm that the ETV tests were conducted in accordance with the test
laboratory's EPA-approved test/QA plan.
This verification statement verifies the emissions characteristics of TECHNOLOGY NAME within
the stated range of application. Extrapolation outside that range should be done with caution and an
understanding of the scientific principles that control the performance of TECHNOLOGY NAME.
This verification focused on emissions. Potential technology users may obtain other types of
performance information from the manufacturer.
In accordance with the generic verification protocol, this verification statement is valid commencing
on DATE indefinitely for application of TECHNOLOGY NAME within the range of applicability of
the statement.
E. Timothy Oppelt Date
Director
National Risk Management Research
Laboratory
Office of Research and Development
United States Environmental
Protection Agency
Jack R. Farmer Date
Program Director
Air Pollution Control Technology Verification
Center
RTI
NOTICE: ETV verifications are based on an evaluation of technology performance under specific,
predetermined criteria and the appropriate quality assurance procedures. EPA and RTI make no expressed
or implied warranties as to the performance of the technology and do not certify that a technology will
always operate as verified. The end user is solely responsible for complying with any and all applicable
federal, state, and local requirements. Mention of commercial product names does not imply endorsement.
A-5
-------
Revision No.: 7
Date: January 22, 2002
APPENDIX B: EXAMPLE CALCULATION OF NUMBER OF TESTS REQUIRED
This appendix presents example calculations to illustrate calculation of the minimum number
of tests for ETV.
The calculation is based on:
1) PM and NOx hot-start FTP emissions measurement variability experience for a diesel
emissions laboratory, which is a function of both the type of measurement and the
emissions level. The emissions variability is taken to be known for the population of
measurements. Therefore the normal distribution value is appropriately used in the
calculations. Eq. B-l is not strictly applicable for a set of measurements for which only
sample standard deviations are available;
2) A known baseline mean emission; and
3) A mean controlled emission (the product of the baseline emission and the expected
emissions reduction).
The criterion being used is that the minimum number of hot-start FTP tests required for ETV
is the number of tests required to have a 90% probability of detecting the specified emission
reduction at the 95% confidence level. The basic equation being used for the calculation is:
n ~ (za + zp)2 i°2i + Q - S/100)2 • a22] / 52
where:
n = sample size in each group, rounded up to the next integer;
za = normal distribution value corresponding to upper-tail probability of a; and
zp = normal distribution value corresponding to upper-tail probability of (3.
1- (X = confidence coefficient on comparison of means (0.95 minimum for ETV);
1 - P = probability of detection of reduction (0.90 minimum for ETV);
021 = squared standard deviation of baseline-engine emission data, expressed as a
percent of the baseline emission value;
022 = squared standard deviation of controlled-engine emission data, expressed as a
percent of the baseline emission value;
8 = difference between baseline and controlled engine emissions mean, expressed
as a percent of the baseline emission value;
Note that ax, a2 and 8, are expressed as percentages of the baseline emission. This
transformation was used to simplify the equation and allow its use for Table 1. For Table 1,
the baseline emission value was chosen as the 1990 EPA certification value. The parameter
'z' is tabulated under different names in statistics reference texts. It is the 'z' value
corresponding to 'the tail area of the unit normal distribution' in Box, Hunter, and Hunter
(1978). In the Standard Mathematical Tables (CRC, 1968), 'z' is known as 'x', and the tail
area is labeled ' 1 - F(x)', where F(x) is the cumulative distribution function of a standardized
normal random variable. To obtain the value of 'z' from the statistical tables, a (or (3) is the
probability (ranging from 0.5000 to 0.0000) in the body of the table, and 'z' is read (or
interpolated) from the appropriate column and/or row, ranging from 0.00 to 4.00 over that
B-l
-------
Revision No.: 7
Date: January 22, 2002
range. The value 'z' can be calculated within an EXCEL® or QUATTRO PRO® spreadsheet
as the absolute value of the function returning the inverse of the standard normal cumulative
distribution, NORMSINV(probability), by setting probability equal to CL or (3, as appropriate.
Table B-l gives a step-by-step calculation example drawn from Table 1 for a 90% probability
(P = 0.10) of detecting a 5% and a 50% emission reduction at the 95% confidence level (a =
0.05).
Table B-l. Example calculation of minimum number of ETV tests.
Table 1,
Col.l
Table 1,
Col. 3
Emission reduction relative to baseline (certification) emission, 8, %
5
50
Known measurement variability at specified emission reduction, a2,
%
2
3
Known measurement variability for baseline engine, ax, %
2
2
a
0.05
0.05
1.645
1.645
P
0.10
0.10
ZP
1.282
1.282
(a2x+ (1 - 5/100)2 • a22) / 52
0.304
0.0016
(za+zp)2 (a2x+ (1 - 6/100)2 • a22) / 52
2.61
0.014
Number of tests requireda, n
3
la
Vfote: a The minimum number of ETV tests is three. If'n'is less than three,'n'equa
s three.
Substitution of the t-statistic for z in Equation B-l implicitly violates some of the
assumptions made in the derivation and is therefore not strictly valid. An iterative
calculation results, and it may not converge. However, it does provide a number of tests
estimate that is larger and closer to the result of the population statistics confidence interval
described in Section 5.1.2.
References for Appendix B
CRC. Standard Mathematical Tables, 16th Ed. S. Shelby, Ed. The Chemical Rubber
Company, Cleveland, OH. 1968.
Box, G. E. P., W. G. Hunter, and J. S. Hunter. Statistics for Experimenters. John Wiley &
Sons, New York, NY. 1978.
EXCEL®. Microsoft Corporation, Redland, WA. 2001.
QUATTROPRO®. Corel Corporation, Ottawa, Ontario, CANADA. 2001.
B-2
-------
Revision No.: 7
Date: January 22, 2002
APPENDIX C: SENSITIVITY OF NUMBER OF TESTS CALCULATION TO
MEASUREMENT VARIABILITY
Table C-l presents the results of a sensitivity analysis of the calculation of the required
number of tests in the particular cases of high (85%) and low (10% and 5%) reductions in
emissions. The number of tests reported in the final column is the number required to have a
90% probability of detecting the emissions reduction with 95% confidence. The equation in
Appendix B was used to compute the required number of tests. As in these sections, all
calculations utilize the normal distribution under the assumption that the test measurement
error is known and constant.
Within Table C-l, the variability of the baseline engine measurement ranges from 2 to 30%,
and the controlled engine measurement variability from 10 to 30%. All of the percentage
numbers in the table are referenced to a baseline engine emission. To convert the percentages
to an absolute emission rate, they must be multiplied by a baseline engine emission rate, and
at 30%) variability the standard deviations are twice the emission rate in g/bhp-hr. The
emissions and standard deviations in Table C-l have all been calculated for a baseline engine
emitting PM at the 1990 certification level of 0.6 g/bhp-hr. For example, from the first row,
an 85%) reduction means an absolute PM emission of 0.09 g/bhp-hr. A 10% controlled
engine measurement variability means the standard deviation for that measurement is 0.06
g/bhp-hr. For the baseline engine, the variability is 2%, so the baseline engine standard
deviation is 0.012 g/bhp-hr. The same approach can be used to make a similar table for any
other emission rate by multiplying the percentages by the desired baseline emission rate.
While the number of required tests increases as the test variability increases, Table C-l shows
that the increase is modest at high emissions reduction levels. While higher variability is
expected at higher levels of control because the absolute emissions concentrations are low,
the large reduction in emissions is easily detected.
On the other hand, lower variability is expected for low emissions reductions, but the smaller
changes are harder to detect and more tests are required. Therefore even modest variability
levels (relative to those in the top block of Table C-l) lead to very large numbers of tests.
As was discussed in Section 5.1.2, calculation of the confidence intervals on the mean
emissions reduction will be based on population statistics and will utilize the t-distribution.
Thus the number of tests calculated using Equation B-l (population statistics) may be fewer
than will be required by the data itself based on sample statistics. As mentioned in Appendix
B, an approximate calculation can be made for sample variability utilizing the Student's t-
distribution rather than the normal (z-distribution) in Equation B-l. This use of Equation B-l
is not strictly valid, but may be useful for low emission reduction and/or high variability test
designs to make a better estimate of the number of tests required.
C-l
-------
Date:
Revision No.: 7
January 22, 2002
able C-l. Sensitivity of number of tests to measurement variability
©X
cT
_o
o
=3
s-h
.s -S
top *3
Ch c«
cs
T3
e
o
U
e
o o
^ rj
-§ .2
T5 w
5-H • -H
c3 S3
T3 « ,
3 U ij
a o •*?
"& .5 &
.. —^ i-fi
Sbj^ 60
^ <2
T3
S-H
O ^
S3
O 60
u
£
£
5
S5
^T1
ctf
e
o
"S
o
•§
"S
>
#g
'H
cl>
u °
e o
^ o\
to G\
oo i—H
ctf
PQ
Qh
=5 to
too g
§1^
'> -"S
-8 s ^
^ p-
o e
On O
to *f?
> a
u ¦£
P
toO
I go'
-rt -2 <+H
C/2 O
G oo
c/3
-------
------- |