United States Air and Radiation EPA420-P-01-003
Environmental Protection August 2001
Agency
&EPA Draft Guidance on
Use of In-Prog ram Data
for Evaluation of I/M
Program Performance
y£u Printed on Recycled
Paper
-------
EPA420-P-01-003
August 2001
Draft Guidance on Use of In-Program Data for
Evaluation of I/M Program Performance
Certification and Compliance Division
Office of Transportation and Air Quality
U.S. Environmental Protection Agency
Technical Guidance
Jim Lindner
(352)332-5444
lindner.jim@epa.gov
NOTICE
This technical report does not necessarily represent final EPA decisions or positions.
It is intended to present technical analysis of issues using data thatC are currently available.
The purpose in the release of such reports is to facilitate the exchange of
technical information and to inform the public of technical developments which
may form the basis for a final EPA decision, position, or regulatory action.
-------
2 1. INTRODUCTION 4
3 2. BACKGROUND HISTORY OF I/M 4
4 3. GENERAL APPROACHES TO I/M PROGRAM EVALUATION 8
5 3.1 Defining Program Evaluation 8
6 3.2 Process vs. Results Based Analysis 8
7 4. PROCESS BASED MEASURES OF EFFECTIVENESS 10
8 4.1. Participation Rate ~-.«v^>«m«%i«r"lF»>i^W ^
10 4.1.2 Matching Registration Records with I/M Records 11
11 4.1.3 Using Year-to-Year Trends !"W 12
12 4.1.4 Using Multi-Year Trends Ilk Tpr It 12
13 4.1.5 Recommended Best Practice 14
14 4.2. I/M Effectiveness :*.A..*M.lJL*. 14
15 4.2.1 QA/QC 14
16 4.2.1.1 Instrument Calibrations 15
19 4.2.1.4 Inspection-Repair Sequence ^^.^*f*^T?:?~7*^. 21
20 4.2.1.5 VTD Check 22
21 4.2.2 Test Data 23
22 4.2.2.1 Measurement Error 24
23 4.2.2.2 Cutpoints 24
24 4.2.2.3 Recommended Best Practices 27
25 4.2.3 Out-of-Program Comparison Data 27
26 4.2.3.1 Vehicle Round Robin Testing 28
27 4.2.3.2 Test Crew Round Robin Testing 28
28 4.2.3.3 Recommended Best Practices 29
29 4.3. Effectiveness of Repairs 29
30 4.3.1 Number and Type 29
31 4.3.2 Emission Reductions 29
32 4.3.3 Repair Lifetimes 33
33 4.3.4 Other Measures 34
34 4.3.4 In-Program Studies to Measure Repair Effectiveness 35
35 4.3.5 Repair Data Collection 36
36 4.3.6 Recommended Best Practices 36
37 5. RESULTS BASED MEASURES OF EFFECTIVENESS 37
38 5.1 ECOS Method 38
39 5.2 EPA Tailpipe I Method 38
DRAFT August 2001 - 2 -
-------
1 5.3 Use of Data Trends 39
2 5.3.1 Fleet Average Emissions Analysis fora Single Program Year 40
3 5.3.1.1 Recommended Best Practices 45
4 5.3.2 Fleet Average Emissions Analysis for Multiple Program Years 46
5 5.3.2.1 Recommended Best Practices 48
6 5.3.3 Emissions Changes in Individual Vehicles Over Multiple Program Years 50
7 5.3.3.1 Recommended Best Practice 55
8 5.3.4 Comparisons with Other Programs 55
9 5.3.4.1 Recommended Best Practice 58
10 5.3.5 Tracer Vehicles 59
11 5.3.5.1 Recommended Best Practices 62
12 5.4 Evaporative Emission Reductions 62
13 5.4.1 Estimate of Single Vehicle Gas Cap I/M Benefit 62
14 5.4.2 Fleet I/M Evaporative Benefit 66
15 5.4.3 Other Evaporative Control Measures 69
17 7. REFERENCES 70
22
23
24
18 APPENDIX A: DEVELOPMENT OF A MODEL TO PREDICT IM240 EMISSIONS
19 CONCENTRATIONS FROM TWO-SPEED IDLE DATA 72
20 A. 1 Data Collection .4^....!!..,jl ...... 72
Ti i _ _
21 A.2 Model Development 73
A.3 Limitations of the Models in Applications ....»». .. 81
A il VIIH
A.4 Accuracy of the Models in Their Application 83
DRAFT August 2001 - 3 -
-------
1
2 1. Introduction
3
4 This document is intended to provide guidance for performing I/M program evaluations using
5 operating program data. The next section is a background of EPA regulation of state I/M
6 programs and a history of methods used to evaluate these programs*. Section 3 describes general
7 approaches to I/M program evaluation. Section 4 focuses on Process-Based measurements and
8 how they relate to I/M program effectiveness and evaluation studies, while Section 5 deals with
9 Results-Based program evaluation analyses.
10
11 Equipment specifications, Quality Control and Quality Assurance procedures, test procedures,
12 vehicle pre-conditioning and other details specific to performing emission measurements in a
13 centralized or decentralized network can be found in the EPA guidance documents "EVI240 and
14 Evap Technical Guidance1" and "ASM Technical Guidance2. The importance of proper vehicle
15 pre-conditioning should not b% overlooked gndboth of the guidance documents cited provide
16 information on this topic. It should be noted that is pre-conditioning is not addressed, it is likely
17 that the estimation of program benefits will be underestimated as the resulting emissions
18 measurements will be higher.
19
20 It is strongly recommended that any state considering the use of in-program data for program
21 evaluation purposes work closely with their respective regional EPA office and the Office of
22 Transportation and Air Quality (OTAQ) to ensure the most up-to-date practices are incorporated
23 into the evaluation. Methods other than those outlined in this guidance document may be
4 f*^- k/3^
24 acceptable; however, close coordination with the appropriate EPA regional office and OTAQ
25 will be even more critical if a state intends to develop program evaluation protocols and analyses
26 not discussed in this document.
27
28 It should also be recognized given the difficulties associated with I/M program evaluations, that
29 an evaluation based on both out-of-program data (e.g. RSD or roadside pullover) and in-program
30 data will provide a more accurate estimate of overall program performance than simply relying
31 on one method alone. For instance, at this time there is no proposed method of estimating the air
32 quality benefit of pre-test repair using in-program data; however, analyses of RSD may provide
33 information on this important element of an I/M program.
34
35
36 2. Background History of I/M
37
38 The Environmental Protection Agency (EPA) has had oversight and policy development
39 responsibility for vehicle inspection and maintenance (I/M) programs since the passage of the
40 Clean Air Act (CAA) in 19703, which included I/M as an option for improving air quality. The
41 first I/M program was implemented in New Jersey in 1974 and consisted of an annual idle test of
42 1968 and newer light-duty gasoline-powered vehicles conducted at a centralized facility. No
43 tampering checks were performed and no repair waivers were allowed.
* This section is identical to Section 2 of "Guidance on Use of Remote Sensing for Evaluation of I/M Program
Performance July 2001 DRAFT". It is included in this document because it provides a short history of I/M program
development that many may find useful.
DRAFT August 2001 - 4 -
-------
1
2 I/M was first mandated for areas with long term air quality problems beginning with the Clean
3 Air Act Amendments of 19774. EPA issued its first guidance for such programs in 19785; this
4 guidance addressed State Implementation Plan (SIP) elements such as minimum emission
5 reduction requirements, administrative requirements, and implementation schedules. This
6 original I/M guidance was quite broad and difficult to enforce, given EPA's lack of legal
7 authority to establish minimum, Federal, I/M implementation. This lack of regulatory authority -
8 - and the state-to-state inconsistency with regard to I/M program design that resulted from it
9 was cited in audits of EPA's oversight of the I/M requirement conducted by both the Agency's
10 own Inspector General, as well as the General Accounting Office.
11
12 In response to the above-cited deficiencies, the 1990 Amendments to the Clean Air Act (CAAA)6
13 were much more prescriptive with regard to I/M requirements while also expanding I/M's role as
14 an attainment strategy. The CAAA required EPA to develop Federally enforceable guidance for
15 two levels of I/M program: "basic""l/M for areas designated as moderate non-attainment, and
16 "enhanced " I/M for serious and worse non-attainment areas, as well as for areas within an Ozone
17 Transport Region (OTR), regardless of attainment status. This guidance was to include
18 minimum performance standards for basic a"nd enhanced I/M programs and was also to address a
19 range of program implementation issues^uciras network design, test procedures, oversight and
20 enforcement requirements, waivers, funding, etc. The CAAA further mandated that enhanced
21 I/M programs were to be: annual (unless biennial was proven to be equally effective), centralized
22 (unless decentralized was shown to be equally effective), and enforced through registration
23 denial (unless a pre-existing enforcement mechanism was shown to be more effective).
24
25 In response to the CAAA, EPA published its I/M rule on November 5, 19927, which established
26 the minimum procedural and administrative requirements to be met by basic and enhanced I/M
27 programs. This rule also included a performance standard for basic I/M based upon the original
28 New Jersey I/M program and a separate performance standard for enhanced I/M, based on the
29 following program elements:
30
31 Centralized, annual testing of MY 1968 and newer light-duty vehicles (LDVs) and light-
32 duty trucks (LDTs) rated up to 8,500 pounds GVWR.
33
34 Tailpipe test: MY1968-1980 - idle; MY1981-1985 - two-speed idle; MY1986 and newer
35 - IM240.
36
37 Evaporative system test: MY1983 and newer - pressure; MY1986 and newer - purge test.
38
39 Visual inspection: MY1984 and newer - catalyst and fuel inlet restrictor.
40
41 Note that the phrase "performance standard" used above was initially used in the CAA and is
42 misleading in that it more accurately describes program design. Adhering to the "performance
43 standard" does not guarantee an I/M program will meet a specific level of emissions reductions.
44 Therefore, the performance standard is not what is required to be implemented, it is the bar
45 against which a program is to be compared.
46
DRAFT August 2001 - 5 -
-------
1 At the time the I/M rule was published in 1992, the enhanced I/M performance standard was
2 projected to achieve a 28% reduction in volatile organic compounds (VOCs), a 31% reduction in
3 carbon monoxide (CO), and a 9% reduction in oxides of nitrogen (NOx) by the year 2000 from a
4 No-I/M fleet as projected by the MOBILE model. The basic I/M performance standard, in turn,
5 was projected to yield a 5% reduction in VOCs and 16% reduction in CO. These projections
6 were made based upon computer simulations run using 1992 national default assumptions for
7 vehicle age distributions, mileage accumulation, fuel composition, etc., and were performed
8 using the most current emission factor model then available for mobile sources, MOBILE 4.1.
9 That version of the MOBILE model was the first to include a roughly 50% credit discount for
10 decentralized I/M programs, based upon EPA's experience with the high degree of improper
11 testing found in such programs. This discount was incorporated into the 1992 rule, and served to
12 address the CAAA's implicit requirement that EPA distinguish between the relative effectiveness
13 of centralized versus decentralized programs.
14
15 The CAAA also required that enhanced I/M programs include the use of on-road testing and that
16 they conduct evaluations of program effectiveness biennially (though no explicit connection was
17 made between these two requirements). In establishing guidelines for the program evaluation
18 requirement, the 1992 I/M rule specified that enhanced I/M programs were to perform separate,
19 state-administered or observed EVI240's"on IT random sample of 0.1% of the subject fleet in
20 support of the biennial evaluation, pnfqrtunately, the program evaluation procedure for
21 analyzing the 0.1% sample was never dlvllo^ed with sufficient detail to actually be used by the
22 states. In defining the on-road testing requirement, the 1992 rule required that an additional
23 0.5% of the fleet be tested using either remote sensing devices (RSD) or road-side pullovers.
24 Furthermore, the role that this additional testing was to play i.e., whether it was to be used to
25 achieve emission reductions over and above those ordinarily achieved by the program, or
26 whether it could be used to aid in program evaluation - was never adequately addressed.
27
28 At the time the 1992 I/M rule was being promulgated, EPA was criticized for not considering
29 alternatives to the EVI240. California in particular argued in favor of the Acceleration Simulation
30 Mode (ASM) test, a steady-state, dynamometer-based test developed by California, Sierra
31 Research, and Southwest Research Institute. In fact, this test had been considered by EPA while
32 the I/M rule was under development, but the combination of EVI240, purge, and pressure testing
33 was deemed sufficiently superior to the ASM that EPA dismissed ASM as a credible option for
34 enhanced I/M programs. Nevertheless, EPA continued to evaluate the ASM test in conjunction
35 with the State of California and by early 1995, sufficient data had been generated to support
36 EPA's recognizing ASM as an acceptable program element for meeting the enhanced
37 performance standard (even though the ASM itself was still deemed marginally inferior to the
38 EVI240, in terms of its emission reduction potential).
39
40 In early 1995, when the ASM test was first deemed an acceptable alternative to EVI240, the
41 presumptive, 50% discount for decentralized programs was still in place. Even at that time,
42 however, the practical importance of the discount was waning, in large part due to program
43 flexibilities introduced by EPA aimed at allowing enhanced I/M areas to use their preferred
44 decentralized program designs. This flexibility was created by replacing the single, enhanced
45 I/M performance standard with a total of three enhanced performance standards:
46
DRAFT August 2001 - 6 -
-------
1 * High Enhanced: Essentially the same as the enhanced I/M performance standard originally
2 promulgated in 1992.
3
4 * Low Enhanced: Essentially the basic I/M performance standard, but with light trucks and
5 visual inspections added. This standard was intended to apply to those areas that could
6 meet their other clean air requirements (i.e., 15%, post-1996 ROP, attainment) without
7 needing all the emission reduction credit generated by a high enhanced I/M program.
8
9 * OTR Low Enhanced: Sub-basic. Intended to provide relief to those areas located inside the
10 OTR which if located anywhere else in the country would not have to do I/Mat all.
11
12 Despite the additional flexibility afforded enhanced I/M areas by the new standards outlined
13 above, in November 1995 Congress passed and the President signed the National Highway
14 Systems Designation Act (NHSDA)8 which included a provision that allowed decentralized I/M
15 programs to claim 100% of the State Implementation Plan (SIP) credit that would be allowed for
16 an otherwise comparable centralized I/M program. These credit claims were to be based upon a
17 "good faith estimate" of program effectiveness, and were to be substantiated with actual program
18 data 18 months after approval. The evaluation methodology to be used for this 18-month
19 demonstration was developed by the Environmental Counsel of States (ECOS), though the
20 criteria used are primarily qualitative, a| opposed to quantitative. As a result, the ECOS criteria
21 developed for the 18-month NHSDA evaluations were not deemed an adequate replacement for
22 the CAAA and I/M rule required biennial program effectiveness evaluation.
23
^ ^ j jMyTi^yi^rt /%
24 In January 1998, EPA revised the I/M rule's original provisions for program evaluation by
25 removing the requirement that the evaluation be based on EVI240 or some equivalent, mass-
26 emission transient test (METT) and replacing this with the more flexible requirement that the
27 program evaluation methodology simply be In October 1998, EPA published a
28 guidance memorandum that outlined what the Agency considered to be acceptable, "sound,"
29 alternative program evaluation methods10. All the methods approved in the October 1998
30 guidance were based on tailpipe testing and required comparison to Arizona's enhanced I/M
31 program as a benchmark using a methodology developed by Sierra Research under contract to
32 EPA. Even though EPA recognized that an RSD-based program evaluation method may be
33 possible, a court-ordered deadline of October 30, 1998 for release of the guidance prevented
34 EPA from approving an RSD-based approach at that time.
35
36 The focus of this document is to provide methods states may use to estimate I/M program
37 benefits using program data. A separate guidance document is devoted to program evaluations
38 using RSD. As its operating premise, EPA recognizes that every program evaluation method
39 will have its limitations, regardless of whether it is based upon an RSD approach or more
40 traditional, tailpipe-based measurements. Therefore, no particular program evaluation
41 methodology is viewed as a "golden standard." Ideally, each evaluation method would yield
42 similar conclusions regarding program effectiveness, provided they were performed correctly.
43 Unfortunately, it is unlikely we will see such agreement among methods in actual practice, due
44 to the likelihood that different evaluation procedures will be biased toward different segments of
45 the in-use fleet. Therefore, it is conceivable that the most accurate assessment of I/M program
46 effectiveness will result from evaluations which combine multiple program evaluation methods.
DRAFT August 2001 - 7 -
-------
1
2
3 3. General Approaches to I/M Program Evaluation
4
5 3.1 Defining Program Evaluation
6 Aside from the technical challenges involved in gathering I/M program evaluation data, there are
7 also subtleties regarding what data is necessary that must be understood. The evaluation of Basic
8 I/M programs is strictly qualitative as per standard SIP policy protocols used to evaluate
9 stationary source emission reductions. Historically, these type of qualitative evaluations have
10 included verification of such parameters as waiver rates, compliance rates, and quality assurance/
11 quality control procedures, but they have not involved quantitative estimates of emission
12 reductions using in-program or out-of-program data.
13
14 The evaluation of Enhanced I/M programs is not as clearly defined and is left to the discretion of
15 the Regional EPA based on the data available. In some instances, it may be possible to estimate
16 the cumulative emission reductions, that is the current fleet emissions are compared to what that
17 same fleet's emissions would bt it mo,I/M program were in existence. However, directly
18 measuring the fleet's emissions to determine the No-I/M baseline is not possible in an area that
19 has implemented an I/M program. Therefore, in order to determine quantitatively whether the
20 level of SIP credit being claimed is being achieved in practice, it becomes necessary to rely on
21 modeling projections to estimate the No-I/M fleet emissions or measure the emissions of a
22 surrogate fleet that is representative of the I/M fleet. Obtaining emission estimates from a No-
23 I/M test fleet based on in-program data would obviously require a traditional tailpipe test be
24 performed on a fleet of No-I/M vehicles; however«4t4siecognized that this may not be possible
25 to do in all cases due to time, resource or operational constraints.
26
27 Two other analyses are also possible that can provide useful information regarding program
28 performance. The first method may be thought of as "one-cycle" since it compares the current
29 I/M fleet emissions to the same I/M fleet's emissions from a previous year or cycle. An analysis
30 such as this would yield information with regard to how the program is improving or declining
31 from year to year. The other method should be considered "incremental" in that it compares the
32 current I/M fleet's emissions to that same fleet's emissions while being subjected to a different
33 I/M program, for instance, comparing a fleet's emissions in an area that has just implemented an
34 IM240 program to that same fleet's emissions the previous year when a Basic Program was in
35 operation. It should be noted, that there is a window of opportunity prior to and during the start-
36 up of any I/M program, or program change, to actually analyze the fleet emissions that would
37 provide empirical data on the No-I/M fleet emissions. If resources and time permit, it is
38 recommended that these baseline data be analyzed in order to reduce I/M program evaluation
39 dependency on modeling projections and provide the most accurate measure of I/M program
40 performance.
41
42
43 3.2 Process vs. Results Based Analysis
44 Analysis of I/M program performance can be thought of in two distinct ways: Results-Based or
45 Process-Based. A Results-Based analysis is more commonly used for looking at the
46 performance of I/M programs, including comparisons of emissions reductions, pass/fail/waiver
47 rates, and other uses of the data collected within the program. Out-of-program data may also be
DRAFT August 2001 - 8 -
-------
1 used, such as remote sensing or roadside tests to determine the emission levels of vehicles
2 between and independent of regular I/M tests.
3 In a Process-Based analysis of I/M program effectiveness, each of the major steps in the I/M
4 process is evaluated separately:
5 achievement of proper fleet coverage
6 performance and documentation of accurate emissions inspections
7 documentation of repair operations on failing vehicles
8 The underlying concept of a Process-Based analysis is that if one step in the process is
9 ineffective, then the I/M program is ineffective. A single ineffective process can become the
10 bottleneck of the entire program. On the other hand, even if all processes in an I/M program are
11 operating as designed, the overall effectiveness is not guaranteed; the program is just more likely
12 to be effective.
13 For example, greater fleet coverage means more vehicles are receiving tests and possible repairs.
14 Similarly, factors such as the test method used, instrument calibration and operation, choice of
15 cutpoints, absence of inspection station fraud, and the effectiveness of vehicle repairs contribute
16 to the effectiveness of an I/M program. Results-Based analysis may show significant fleet
17 emissions reductions resulting from the program, but if the tests were done with uncalibrated
18 instruments, the repairs last only for a short time, or only a small portion of the fleet is actually
19 being tested, then the I/M program may not be effective.
I
20 When Process-Based analysis is used in combination with Results-Based analysis, a much more
21 thorough understanding of the effectiveness of an I/M program may be achieved. If a Results-
22 Based analysis indicates that an I/M program is ineffective, a state can have difficulty in
23 determining the cause. In this situation, a Process-Based analysis can help identify where the loss
24 of program effectiveness occurs.
25 For Process-Based measures to be used to evaluate an I/M program, some methods or standards
26 for evaluation are needed. Unfortunately, EPA is not in a position to provide these standards as
27 the standards should be based on actual operating data, although EPA may provide broad
28 guidelines and/or standard calculation procedures for performing these Process-Based analyses
29 as needed. Nonetheless, EPA recognizes that in many instances, judging the Process-Based
30 performance of an I/M program may be performed by states operating similar programs
31 exchanging results from their analyzer, dynamometer and OBDII Tester audits, as well as repair
32 data relating to number and type of repair, etc. This sharing of knowledge is occurring
33 informally in many forums such as EVI Solutions, Clean Air Conference, monthly status calls
34 between states and routine phone calls and Emails. It is not clear at this time if the EVI
35 community would support routinely providing this information to an agreed upon clearing house
36 to facilitate the exchange of this information, or if the program information is felt to be too
37 sensitive to permit its free distribution.
38 In the following sections, methods are described and examples presented for both Process-Based
39 (Section 4) and Results-Based (Section 5) analyses. Many of the examples presented use actual
DRAFT August 2001 - 9 -
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
I/M program data taken from several regions; however, the locations will be identified simply as
State 1, State 2, etc.
4. Process Based Measures of Effectiveness
4.1. Participation Rate
The fleet for an I/M program area may be defined either as the set of vehicles registered in the
area, or as the set of vehicles driven in the area. Results from various RSD programs have
shown that the two fleets are often quite different. Figure 4-1 is a diagram of a typical mix of
vehicles for an I/M Program area. The vehicles driven in the area may be registered in the area,
or may originate outside the area. Some of the vehicles registered in the I/M program area may
no longer be driven there, if the vehicle owner moves or the vehicle is sold. The set of vehicles
that participate in the I/M program may include most ofthe vehicles that are both registered in
the area and are located (driven) there. The greatest emissions reduction benefit would be
achieved if the set of vehicles that are driven in the program area all participated in the I/M
program. This goal is more difficult to achieve in some areas than others; for example, the
Kansas City metropolitan region is partlvinthe state pf Missouri and partly in the state of
Kansas, so many ofthe vehicles driven in Kansas City, Kansas are registered in Kansas City,
Missouri, and vice-versa.
Set of vehicles
registered outside of
I/M Program Area
Set of vehicles registered
in I/M Program Area
Vehicles driven in
I/M Program Area
Vehicles participating in
I/M Program
Figure 4-1. Mix of Vehicles within an I/M Program Area
To evaluate the performance of an I/M program, a first basic step is to define the participation
rate of vehicles eligible for the program. Even the most carefully administered I/M program may
be undermined if a significant portion ofthe fleet avoids the tests. The goal here is to compare a
set of vehicles participating in the I/M program to both the registered fleet and the driven fleet.
Emphasis should be placed on comparison to the registered fleet, since location of registration is
almost always used to define the program area; however, even greater emissions reductions
could be achieved in any area by expanding the program to include all vehicles driven in the
area.
DRAFT August 2001
-10-
-------
1 The most basic measure of fleet coverage is to compare counts of the number of vehicles in the
2 registered fleet, the driven fleet, and the I/M fleet. Although these rough estimates will contain
3 errors, given the minimal effort required to obtain these estimates, they should be performed and
4 recorded. For example, the registered fleet (usually taken from a state registration database)
5 often includes large numbers of vehicles that have been sold or moved out of area. A registration
6 database that is not consistently updated as vehicles migrate makes the I/M program participation
7 rate appear to be lower than it is, and makes it difficult to identify vehicles that really are located
8 in the area but not participating in the program. License plate readers, such as those used by
9 RSD and pneumatic vehicle counting devices can be used to estimate the driven fleet. However,
10 such readers and counters can have sampling errors depending on the locations for the readers.
11 Because newer vehicles are usually driven more than older vehicles, the RSD data may actually
12 catch more of the "travel fraction" than the "registration fraction" in an area.
13
14 The analysis described in this and the following sections is based on data from states' Vehicle
15 Inspection Databases, registration databases, and repair databases. Datasets may have several
16 million records and require multiple gigabytes of computer memory to process. The EPA
17 contractor (Eastern Research Group) who performed these analyses used a Digital Alpha DS20
18 Unix system with 100 GB of hard drive space and 1 GB of RAM with SAS statistical analysis
19 software.
20
21 4.1.1 Comparing Vehicle Age Distributions \
22 One method of assessing the participation rate is to compare the vehicle age distribution of the
23 registered fleet, the I/M fleet, and the driven .fleet. Distributions are used in place of counts due
24 to the large differences in the fleets. In the absence of a fully updated registration database,
25 distributions may still be compared to determine whether the registered and tested fleets are
26 qualitatively the same. This type of comparison is shown in Figure 4-2 using data from State 2.
27 From this figure it may be seen that the set of registered vehicles has a larger proportion of early
28 1980's vehicles than does the I/M set, which might indicate that owners of older vehicles are
29 avoiding inspections. The driven fleet that was observed on the roads by RSD contains even
30 fewer vehicles from the oldest model years than the I/M set, indicating that some of the older
31 vehicles that are registered but not participating in the I/M program may not be driven often.
32 The registration fleet has a mean age of 9.4 years, the I/M fleet, 8.2 years, and the RSD fleet, 7.0
33 years.
34
35 4.1.2 Matching Registration Records with I/M Records
36 Comparisons between the registered fleet and the I/M fleet could be done directly by attempting
37 to match each registration record with an I/M record. However, the registration database may
38 not be updated each time a vehicle is sold outside the area, leading to overstatement of the
39 difference between the two fleets. Figures like 4-2 include the implicit assumption that these
40 sales are evenly distributed over the model years; if this is not the case, then bias may be
41 introduced.
DRAFT August 2001 - 11 -
-------
1
2
3
4
5
6
1
8
9
10
11
12
13
14
15
16
17
12
10
w
-9>
o
5 6
Registered, 1998
l/M, 1998
RSD, 1998
1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998
Vehicle Age
Figure 4-2. Distribution of Vehicles in I/M Program, Registration Database, and Observed
through Remote Sensing
B
4.1.3 Using Year-to-Year Trends
Year-to-year trends in the age distribution bf the I/M fleet may also be informative even though
there can be many reasons for shifts. For example, if a fleet had a larger portion of new vehicles
each year, it might be concluded that an improving economy was helping encourage the
replacement of old vehicles with new ones. This doesn't seem to be the case for State 3, shown
in Figure 4-3. The average vehicle age increases from 7.3 years in the first program year shown
to 8.0 by the fourth program year.
10
9
8
0>
o
-II
c
0
8
10
12
14
16 18
20
Vehicle Age
Figure 4-3. Vehicle Age Distribution over Four Years of I/M Tests
4.1.4 Using Multi-Year Trends
Multiple years of I/M program data may also be used to find the rate at which vehicles leave the
program between test cycles. Vehicles that leave the program may have been sold and removed
DRAFT August 2001
-------
1
2
3
4
5
6
7
8
9
10
1 1
12
13
14
from the fleet, or they may remain in the area without participating in the I/M program. For
State 3, vehicles were tracked over the four years of data being used. It was found that almost
80% of the vehicles tested each year returned for testing the next year, as shown in Figure 4-4.
From the data available, it is not possible to determine whether the other 20% of vehicles were
sold outside the program area or simply dropped out. Figure 4-4 shows that the percentage of
vehicles returning the next year decreases significantly for vehicles aged 10 years or greater.
These vehicles are also the most likely to fail the I/M test, possibly leading the owners to avoid
further testing. In Figure 4-5, the percentage of vehicles that return the year following a failed
I/M test is presented. Since the return rate is considerably lower than the overall average shown
in Figure 4-4, it seems reasonable to conclude that fear of failing the test has led some vehicle' s
owners to drop out of the program.
100
c
3
"S
OL
-t-«
ro
.c
)
8 10 12 14 16
Vehicle Age
18 20
Figure 4-4. Percentage of Tested Vehicles Tnat Return for Testing the Following Year
100
a:
"ro
_c
-t-«
u)
^
o
o
CL
15
10
Vehicle Age
15
20
DRAFT August 2001
-13-
-------
1 Figure 4-5. Percentage of Failing Vehicles that Return for Testing the Following Year
2
3
4 4.1.5 Parking Lot Sticker Surveys
5 Data from parking lot sticker surveys have been used by states as a cost-effective method to
6 estimate I/M program compliance rates11'12. Care must be taken to ensure that the surveys
7 capture a representative sample which will require appropriate geographic coverage. Also,
8 procedures must be documented and in place to minimize the opportunity for fraudulent stickers
9 to be obtained by those motorists seeking to avoid the program.
10
11 4.1.6 Recommended Best Practice
12 One of the five methods described above should be used to verify compliance rate estimates used
13 in the SIP, as well as for estimating average emission reductions when used with failure rate and
14 emission data. The primary goal is to diligently update and maintain the accuracy of the vehicle
15 registration database, so that direct comparison between the sets of vehicles registered and
16 participating in the I/M program may be made. License plate reading equipment like that used in
17 RSD studies may be used to confirm thfe with which the vehicle registration database
18 represents the fleet. Until a high level of cdffidence in the accuracy of the registration database
19 is developed, comparisons of distributions such as those shown in Figures 4-2 and 4-3 should be
20 used to qualitatively compare the set of vehicles that undergoes I/M testing to the registration
21 database. Figures like 4-4 and 4-5 should be used to estimate the rate at which vehicles drop out
22 of the I/M program. Parking lot surveys have been used by many states as a cost-effective way
23 to estimate compliance rates also.
24
25
26
27 4.2. I/M Effectiveness
28
29 4.2.1 QA/QC
30 The effectiveness of the inspection processTtselflnay "Be influenced by many factors. The
31 inspection is primarily based on the measurement of vehicle emissions. Any factors that degrade
32 the accuracy of the emissions measurement contribute to the degradation of the I/M program.
33 Such factors might include improper analyzer calibrations, analyzers that require maintenance,
34 inaccurate data entry of vehicle information, emissions cutpoints that are too loose or too
35 stringent, emissions tests with excessively large measurement errors, and inspection station
36 fraud.
37
38 The following sub-sections provide a discussion and examples of ideas for techniques that can be
39 used to evaluate many of the factors that contribute to ineffective I/M programs. Passing grades
40 on all factors does not necessarily guarantee a successful I/M program. On the other hand, a
41 poor grade on one factor can act as a bottleneck preventing an I/M program from being effective.
42 Beyond merely using these techniques to demonstrate I/M program effectiveness, a state can use
43 these techniques to identify for itself areas of inspection effectiveness that are good and areas
44 where improvements need to be made.
45
46 This analysis of in-program I/M data should also be performed prior to any analysis of emissions
47 reductions so that emissions reduction calculations will be based on the data of known quality.
48
DRAFT August 2001 - 14 -
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
4.2.1.1 Instrument Calibrations
Records of I/M program analyzer calibrations can be used to measure the drift of analyzers
between calibrations. If many analyzers in a state's I/M program drift substantially, the results of
measurements are suspect. Ideally, all analyzers should drift no more than the specification of the
analyzers.
For example, in State 3 analyzers must be calibrated at least every 72 hours. Before calibration,
each analyzer is checked for drift by measuring the calibration gas mixture, whose concentration
is known within a specified precision. If the analyzer has not drifted since the last calibration, its
readings for the calibration gas will be close to the bottle label value, and little calibration
adjustment will be necessary. The difference between this pre-calibration analyzer reading and
the label concentration in the gas mixture is a direct measure of instrument drift. Analyzers that
consistently drift little from calibration to calibration can be expected to produce more accurate
measures of vehicle emissionsjthan ^>se that drift grea^
Six months of instrument pre-calibration data containing 90,781 calibrations from 2,324
instruments was examined. We examined the analyzers' drift characteristics on readings for HC,
CO, CO2, and O2 for zero, mid-span, and high-span g|ses. For this example, the CO high-span
gas is analyzed, which had a label value of 4.0%. The BAR90 analyzers, which were used in this
I/M program, have an accuracy specification of ±0.15 % for a 4% CO gas. Accordingly, it is
expected that most of the 90,781 pre-calibrations should fall within about ±0.15 % of 4.00%.
Any pre-calibrations that fall greatly outfideithis range would cause concern.
Figure 4-6 shows a histogram of the 90,7^
this period. About 86% of the values are wiffiin'±TT.15"%l)f 4.00%. However, 3.7% of the values
are zero, and 0.5% of the values are between 0.1^ and 3.5%. These unexpected values raise
concern and should be investigated. Several explanations may exist for these unexpected values.
In any case, states that have tighter distributions of pre-calibration values and have a system in
place for addressing out-of-spec values have a better chance of having an effective I/M program.
-calibrations for all instruments in the state during
3500
f> Qnnn
0
2 2500
~ni
o onnn
0
J2 1500
E
z 1000
0 -
nil
n
Ln___n
0.0 0.4 0.8 1.2 1.6 2.0 2.4 2.8 3.2 3.6 4.0 4.4 4.8
High-Span CO Pre-Calibration Reading (%)
DRAFT August 2001
-15-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Figure 4-6. Distribution of Values for High-Span CO Pre-Calibrations
Instrument Calibrations Recommended Best Practice
Instrument calibration data, especially the pre-calibration readings, are a good indicator of
instrument drift and should be tracked regularly. Instruments that consistently drift more than
the instrument specifications should be repaired.
4.2.1.2 Instrument Audits
Independent instrument audits of I/M program emissions analyzers with certified bottled gas can
also be used to evaluate analyzer accuracy. This additional instrument check is valuable because
instruments can experience periods when they are out of calibration even if the pre-calibration
data shows that the instrument has little drift. One possible cause is problems with the line
leading from the tailpipe probe to the instrument. Instrtp^nt calibrations introduce gas at the
instrument; instrument audits and vehicle tests introduce gas at the tailpipe probe. Obstructions,
leaks, or contamination might cause audits (and emissions measurements) to be out of
calibration.
For example, I/M analyzers were calibrated as normal using a station's normal supply of
calibration gas. Nothing abnormal was seen in the calibration data recorded in the VID. The
instruments were routinely challenged using a supply of bottled gas separate from the station's
calibration gas. Most instruments passed the audits for zero, low-span, and high-span gases, as is
shown for CO2 in Figure 4-7. However, one instrument showed varying behavior from day to
day with values biased low by about 30% on several days, as is shown in Figure 4-8.
C02 (%)
14 H
10 :
9 :
8 :
7 -
6 '-
5 \
4 \
3 :
2 :
1 '-_
0
H 1 1 Hh
H h
H 1 1 Hh
H h
310CT99 12NOV99 23NOV99 05DEC99
DATE
16DEC99
28DEC99
26
27
DRAFT August 2001
-16-
-------
1
2
Figure 4-7. Good Analyzer Results for 3 Audit Gases
DRAFT August 2001
-17-
-------
C02 (%)
14 H
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
-1 4
310CT99
12NOV99 23NOV99 05DEC99 16DEC99
Figure 4-8. Poor Analyzer Results for 3 Audit Gases
28DEC99
These audits indicated a recurring instrument problem that was not caught by the station staff or
the VID data. The problem was serious even though the measured quantity (CO2) is not a
pollutant of interest (HC, CO, or NOJ, since CO2 is used to correct for exhaust dilution;
inaccurate corrections would be made with an erroneous CO2 value. The result would be
inaccurate determinations of dilution-corrected HC, CO, and NO .
States that have some sort of instrument audit program in their I/M program would potentially be
able to identify instruments that are out of calibration by analyzing the data as described above.
Instrument Audits Recommended Best Practice
A standardized method of instrument audit program provides an added level of confidence that
instruments are accurate. We have found cases where instruments calibrated well and showed no
drift between calibrations, but provided inaccurate results when challenged with a separate
source of gas.
4.2.1.3 DCFCheck
The measurement of exhaust emissions concentrations can be confounded by the dilution of the
exhaust gas by non-optimal probe placement, leaking exhaust systems, cylinder misfires, and
excess oxygen from air pumps. Some I/M program emissions analyzers use measured CO and
CO2 concentrations to calculate a dilution correction factor to correct raw exhaust emissions
concentration values for this dilution to arrive at emissions values on an undiluted basis.
Assuming stoichiometric combustion of gasoline, an exhaust dilution correction factor (DCF)
can be estimated using a carbon mass-balance and the measurements of CO and CO2. These
DRAFT August 2001
-18-
-------
1 constituents are measured in the non-dispersive infrared bench of the analyzer. The equations are
2 based on the average composition of gasoline.
3
4 First, define the variable x:
CO-
5 x= ^
co2 + co
6
7 where the CO2 and CO values are in percent.
8
9 Then the dilution factor, dcfco/G02, is as follows:
10 drfco/co =100x/(464 + 188x)
co/co2 CQ^
12 If a fuel other than standard gasoline*!stised, the 4.64 constant will be different. For example,
13 the constants for methane (CNG), propane (LNG), methanol (M-100), and ethanol (E-100) are
14 6.64, 5.39, 4.76, and 4.76, respectively. The constants for reformulated gasoline and oxygenated
15 gasoline will depend on gasoline compofitionj, but are generally not far from 4.64.
16
17 In addition, many emissions analyzers also measure exhaust gas oxygen concentration with an
18 electrochemical cell. Assuming an ambient air oxygen concentration of 20.9%, the exhaust
19 oxygen measurement can also be used to estimate dilution in the exhaust. A dilution correction
20 factor based on the measured oxygen concentration O2 is:
21
22 ^
23 This relationship assumes that the tailpipe oxygen concentration for stoichiometric combustion
24 and no air in-leakage is 0.0% O2. Field measurements indicate that new vehicles with no exhaust
25 system leaks and operating at stoichiometric air/fuel ratio have 0.0% tailpipe oxygen
26 concentrations.
27
28 If CO, CO2, and O2 are measured correctly, the independent DCFs (CO/CO2 and O2) for each
29 vehicle inspection should agree well with each other. Emissions results for two-speed idle tests
30 in State 3 were examined and the DCFs were calculated for each test on each vehicle. Figure 4-9
31 shows a plot of the high-speed idle DCF based on CO/CO2 versus the high-speed idle DCF based
32 on O2for each emissions test. The plot shows that many of the points fall near the 1:1 line as
33 expected; however, many also fall far off the 1:1 line. Those points that fall off the line represent
34 analyzer sensors for CO, CO2, or O2 that are broken or out of calibration, data entry errors, or
35 tests on vehicles that use fuels far different from gasoline. Ideally, all points would fall near the
36 1:1 line.
DRAFT August 2001 - 19 -
-------
2.5-
CN 2.4 H
O
O 2.3 H
2.2 :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
o
o
I 2'°
W 1-9
E 1-8
fc "
O 1 6
CD
CD 1.3
Q_
^ 1.2
gi
X 1-0
eA ldle DCF based on 02
Figure 4-9. Comparison of High-Speed Idle DCFs in State 3.
i, <||
Each state could use this evaluation of CO, CO2, and O2 data for every emissions inspection to
demonstrate the fraction of inspections that meet a minimum requirement. Tolerances for
agreement between the two types of DCFs can be determined from the I/M analyzer accuracy
specifications for CO, CO2, and O2 and the local composition. The plot with this data
indicates that the difference between the two DCFs should be no larger than about ±0.14.
The dilution correction factor relationships are a consequence of gasoline combustion
stoichiometry. Therefore, it also follows that a relatively constant relationship exists among the
undiluted exhaust gas concentrations of O2, CO, and CO2 from a gasoline-fueled engine even if
the engine produces significant concentrations of HC, CO, and NOX. Analyzer manufacturers
could use this relationship to provide a check of each emissions test as it was being performed. If
the relationship was not satisfied, the analyzer operator would see a flag to indicate that analyzer
maintenance should be performed.
DCF Check Recommended Best Practice
The raw (before any corrections) concentration measurements of all emissions tests should
indicate that combustion of gasoline (or of whatever fuel is used) is the source of emissions. One
way to check this is to compare calculated dilution correction factors based on CO/CO2 against
those based on O2. For every emissions test they should agree within about ±0.14. If they do
not, the emissions test may be inaccurate. DCF checks can be made on records in the VID, but it
may be best to incorporate them in the analyzers so that inspection stations can address the
problem immediately.
DRAFT August 2001
-20-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
4.2.1.4 Inspection-Repair Sequence
An analysis of vehicle inspection/repair records from the VID can be used to evaluate the
accuracy and completeness of data in the VID system. States that have better VID systems have
more reliable inspection and repair data and therefore can better support their claims of effective
I/M programs. The following is an example of an analysis of inspection/repair sequences for
State 3. In this state no repair records were kept, and so the example cannot make use of repair
information. One of the key items of Section 4.3 is the strong recommendation for states to
maintain good records for vehicle repairs performed as part of an I/M program.
Each vehicle was tested at an I/M station on one or more occasions. On each occasion, the VID
contains a variable that gives the type of test (Initial or Re-test) and a variable that gives the
result of the emissions test (Pass or Fail). The Test Type variable has special rules for
designating whether a test is an Initial or a Re-test. For Initial tests, customers are charged for the
inspection. If they fail the inspection but return after repairs within 5 days, then the second test is
designated a Re-test, and the customer is not charged for the Re-test. If more than 5 days have
elapsed, then the second test is designated an Initial test and the customer is charged again.
Consequently, a test that is designated Initial may actually be a follow-up test in an effort to get
the vehicle to meet I/M requirements. In any case,"four combinations of these two variables are
possible for each occasion. For analysis^urp^es, tha^four combinations were given designators
as shown in Table 4-1.
Table 4-1. Designators for Test Type and Result
Designator Test Type Emission Test Result
Then, for each unique VIN, the designators were concatenated in chronological order to create a
sequence number that describes the testing sequence that each vehicle experienced during I/M
testing. For example, for a vehicle that initially failed and then passed on a re-test, the test
sequence would be IF, RP. The frequency distribution of the resulting test sequences is shown in
Table 4-2.
The distribution shows that the top ten most frequently found sequences accounted for 99.64% of
the vehicles tested. Although it is recognized that some of the vehicles may have incomplete test
cycles because the test cycle was begun in the last few days of the data set period, some of these
sequences raise questions. Why are 1.37% of the vehicles tested a second time after they pass?
Why do 0.82% of the vehicles undergo no further testing when they failed initially? An
important part of an analysis of inspection/repair sequences is to document the explanation for
these apparent anomalies.
Table 4-2. Frequency Distribution of Test Sequences in State 3
Test Sequence
Vehicle Frequency
% of Vehicles
DRAFT August 2001
-21-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
IP
IF, RP
IP, IP
IF
RP
IF, IP
IF, RF, RP
IF, RF
IP, RP
IF, RF, IP
450 Other Test Sequences
Total
3,413,802
66,987
49,771
29,682
21,509
9,183
7,037
4,790
4,365
2,192
13,093
3,622,411
94.24
1.85
1.37
0.82
0.59
0.25
0.19
0.13
0.12
0.06
0.36
100.00
Approximately, 450 less frequently used sequences accounted for the remaining 0.36% of the
tested fleet. Many of these remaining sequences seem to be unlikely. For example, what could be
the reason for 21 vehicles having the sequence IP, IP, IP, IP, IP, IP, IP? It is suspected that these
sequences represent database data entry problems instead of real situations. Better inspection
database systems should be able to reduq^the occurrence of these unlikely test sequences.
w
Inspection Repair Sequence Recommenced Best Practices
When a good inspection data set is combined with a good repair data set, the sequences of
inspections and repairs should make sense. Cross-checking between these data sets can identify
many errors in VID data sets. The sequence of each vehicle should tell a simple story. If it does
not, data entry problems probably exist^,
4.2. 1.5 VID Check
Since the in-program data is the primary basis of the I/M program evaluation, a series of basic
data checks should be used to demonstrate the accuracy and completeness of the data in the
database. The following list may serve as a starting point for basic validation checks in future
I/M program evaluations.
1) The beginning and ending dates of the VID data under consideration should be
specified.
2) A frequency distribution of almost all database variables should be provided to
demonstrate the accuracy and completeness of data entry. Missing and
nonsensical values should be included in the distribution to show the frequency of
improper entry.
3) A distribution of the emissions measurements is a special case of the above.
Ideally, no observations with missing values should be present. Also, all
observations should have a CO2 concentration between about 6% and 17%, since
a combustion process must be present.
4) The fraction of observations with both the license plate and the VIN missing
should be determined.
DRAFT August 2001
-22-
-------
1
2 5) The validity of each VIN should be checked in some manner. In the simplest
3 method, the check digit in 1981+ VINs can be checked. More extensive VIN
4 checking efforts could involve comparison of the recorded vehicle description
5 information with the corresponding information from a VIN decoder.
6
7 6) Each license plate should be associated with only a single VIN.
8
9 7) Within a single I/M cycle, each vehicle should have a recognizable and
10 reasonable test and repair sequence. For example, a vehicle with a "fail, repair,
11 fail, repair, pass" sequence is reasonable, but one with a "fail, repair, pass, pass,
12 pass, repair, fail, fail" sequence is not. Data entry problems by test stations and
13 repair stations can produce unreasonable sequences. Accordingly, a frequency
14 distribution of sequences can be an indicator of the extent of data entry problems.
15 ir%
16 VID Check Recommended Best Practices
17 These checks are probably the most fundamental VID data checks. They involve sanity checks
18 on every field in the VID. Distributions of numeric variables, frequency distributions of
19 categorical fields, x*y plots, and range chfckj^n alLjbe used to find how data is improperly
20 entered in the database.
21
22 4.2.2 Test Data
23 A discussion of the effectiveness of emissions inspections is necessary to evaluate their
24 contribution to the overall I/M program of a state. If a state's I/M program covers the fleet well
25 and has great repair stations, but the emissions inspection stations cannot properly identify high
26 emitting vehicles, the overall effectiveness of the I/M program will suffer.
27
28 Perhaps the most fundamental part of the discussion of emissions measurement is a definition of
29 the inspection flow sequence. The inspection sequence would first define the vehicles that are
30 subject to I/M testing. For example, this might be 1975 to 1995 light-duty, gasoline-fueled
31 vehicles. Then, perhaps all-wheel-drive vehicles get a two-speed idle test, and all remaining
32 vehicles get an ASM test. All of the steps in the inspection flow would be defined, including
33 station type (e.g. test only, test and repair, centralized, decentralized), test type (e.g. IM240,
34 ASM, gas cap check) and associated cutpoints, model year group selections, waiver thresholds,
35 and exemption criteria. This inspection sequence should be presented as a flow diagram.
36
37 Next, the flow diagram should be annotated to show the number of vehicles and inspections that
38 occurred in the state for the evaluation period. This would allow a between-state comparison to
39 be made of corresponding parts of the emission inspection sequence. For example, one state
40 might have a waiver threshold of $200 with 2% of vehicles waived, while another state has a
41 waiver threshold of $500 with only 0.3% of vehicles waived.
42
43 Next, the important characteristics of the emissions tests used should be defined. This would
44 include emissions test type and emissions pass/fail criteria (i.e. cutpoints).
45
46 Correlations can be built to use short emissions test results (e.g. ASM, two-speed idle, IM240) to
47 predict reference emissions test results (e.g. EVI240, FTP). The importance of vehicle pre-
48 conditioning in any correlation study or program evaluation effort must not be overlooked as
DRAFT August 2001 - 23 -
-------
1 inconsistent pre-conditioning will have an adverse impact on the test program. The IM240 test
2 can be the reference test, or it can be a short test when the FTP is the reference test. Studies that
3 apply these correlations indicate that the greatest source of error for a vehicle receiving an
4 incorrect pass/fail designation by the short test is the difference in the responses of vehicles to
5 the short and reference tests13'14. These studies indicate that measurement errors of the short test
6 and of the reference test are small contributors to incorrect pass/fail designations. Therefore,
7 states should report the variance of the deviations between their short test (if they use one) and a
8 reference test. A state could measure this variance by performing out-of-program reference tests
9 on a sample of program-eligible vehicles. Alternatively, a state could simply quote the variance
10 measured by other states. However, states that can demonstrate a smaller variance will tend to
11 have the better inspection effectiveness.
12
13 4.2.2.1 Measurement Error
14 The measurement error of an emissions test is an estimate of the uncertainty in the reported
15 emissions of a single measurement. Tests that have large measurement errors will cause the
16 pass/fail status of some vehicles to be improperly designated; however, studies have shown that
17 such tests can still provide emission reduction benefits for the fleet as a whole (14 above). For
18 each emissions test type, the measurement (as determined by replicate testing of vehicles)
19 should be reported. States may choose to report measurement error calculated from data taken in
20 other states, or they may choose to calculate measurement error based on their own data of repeat
21 emission measurements.
23 This measurement error for an emission^ tes| can be calculated from repeat emissions
24 measurements on a sample of vehicles. A state could obtain repeat measurements by performing
25 them on vehicles that are being inspected as part of the normal I/M program. The vehicles that
26 receive repeat measurements should be selected to cover the range of emissions levels
27 represented in the fleet. In general, a stratified sampling technique will provide the most useful
28 information from the fewest measurements. The measurement error is calculated by pooling the
29 variance of each repeated vehicle's measurements. However, the variance for each vehicle must
30 be calculated after transforming all emissions measurements to a space where measurement error
31 is relatively constant for all emission levels. We have found that the natural log transformation
32 provides this attribute for most emissions tests. An example of the calculation of measure error
33 is provided in Reference 13 above and is briefly outlined in Appendix A.
34
35 4.2.2.2 Cutpoints
36 The cutpoints applied to emissions measurements to designate a vehicle as a pass or fail also
37 have an important influence on the correctness of the designation and thereby on the overall
38 measurement effectiveness. An analysis of cutpoint effectiveness could be performed on in-
39 program data. States should already have an understanding of the role that cutpoint selection
40 plays in identifying vehicles that need repair versus vehicles that are sent to repair. The following
41 conceptual discussion is meant to reinforce that understanding, and it will lead to suggestions for
42 evaluating and optimizing cutpoint selection. With regard to optimizing cutpoints there are those
43 who believe there should be methods to get information on the emissions and repair rates of
44 vehicles below current cutpoints. The rational for this approach is that without this information,
45 state I/M program administrators would only be able to look to higher cutpoints to search for an
46 optimum.
47
DRAFT August 2001 - 24 -
-------
1 Figure 4-10 qualitatively shows the emissions distributions of vehicles in a state's I/M program
2 fleet subject to a common cutpoint. All vehicles that have a properly functioning emission
3 control systems are in the lower emitting distribution (shown by the thin line); these vehicles are
4 non-repairable since they have no problems. All vehicles that have problems with their emission
5 control systems are in the higher emitting distribution (shown by the thick line); these vehicles
6 could be repaired if the I/M program could identify them. The two distributions have a
7 significant overlap in emissions. This overlap is a consequence of the emissions characteristics
8 of specific non-repairable and repairable vehicles. For vehicles of the same age and technology,
9 some broken vehicles will have emissions lower than some properly operating vehicles.
10
11 Wherever the cutpoint is chosen (shown by the dashed vertical line in the figure), some vehicles
12 will be properly designated and some vehicles will be improperly designated as pass or fail.
13 Improper designations include two types: non-repairable vehicles called a fail, and repairable
14 vehicles called a pass. Where should the state set its cutpoint? If a state sets a high (loose)
15 emissions cutpoint, most failures will be repairable, few failures will be non-repairable, but only
16 a small fraction of all repairable,vehicles will be sent for repairs. The state's airshed incurs an
17 environmental cost from these false passes. If a state sets a low (stringent) emissions cutpoint, a
18 larger fraction of all repairable vehicles will be sent for repairs, but many non-repairable vehicles
19 will also be sent for repairs. In this casey^eh^cle owners incur an expense for taking their vehicle
20 to get a repair for a problem that does nc
DRAFT August 2001
-25-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
(A
0)
O
Non-Repairable
Vehicles
Outpoint
Emissions
Figure 4-10. Conceptual Emissions Distributions ofRepairable and Non-Repairable Vehicles
With estimates of the cost of false passes, the cost of false fails, and the distributions of
repairable and non-repairable vehicles (as shown conceptually in Figure 4-10), a state could
optimize the selection of the cutpoint by minimizing the total cost, which is the sum of false pass
and false fail costs.
Although the actual accounting in this situation will likely be difficult, a state could estimate the
costs of false passes and false fails by an economic analysis. False pass costs would be driven by
the influences of excess emissions released and would include health costs, cost of further
stationary source limits, and costs of not achieving SIP goals. False fail costs would be driven
by inconvenience costs including time and repair costs lost by owners taking vehicles to repair
shops for problems that do not exist. Excess emissions identified estimates would have to be
obtained from paired testing using a state's I/M test and a suitable reference test, and in many
instances, such as with idle, IM240 and ASM tests, data sets exist that could aid in this effort.
For a state to optimize the location of the cutpoint, knowledge of the shape of the non-repairable
and repairable distributions at emissions above and below the cutpoint is also required. An
analysis of the distributions above the current cutpoint should be performed first since the I/M
program will already have the data. The parts of the distributions for emissions above the
cutpoint value can be determined by an analysis of in-program emissions and repair data. The
state should analyze these distribution shapes and report them; they are an indication of the
ability of the emissions measurements to resolve (i.e. separate) the repairable from the non-
repairable vehicles. Some emissions tests may be better able to resolve repairable and non-
repairable vehicles than other tests. Later, the discovery of repairable and non-repairable
distributions below the current cutpoint using typical in-program I/M data could be made, but it
is more difficult. There are two potential problems: fast-pass and fast-fail emissions
DRAFT August 2001
-26-
-------
1 measurements and the unknown repair needs of vehicles with emissions below the outpoint.
2 Without accurate emissions and repair data for at least a sample of vehicles with emissions
3 below the current cutpoint, the search for a more cost-effective cutpoint below the current
4 cutpoint cannot be made with in-program data.
5
6 The use of fast-pass and fast-fail emissions measurements increases throughput at I/M stations
7 but impedes determination of the emissions distributions. Whenever an emissions test is cut
8 short by invoking fast-pass or fast-fail criteria, the emissions level of the full test is obviously
9 lost. In some I/M programs, whether a test result is from a fast test or a full test may not be
10 recorded. Use of fast-pass algorithms contaminates emissions measurements below the cutpoint;
11 fast-fails contaminate measurements above the cutpoint. If in-program data is to be used for
12 optimizing cutpoints, fast-fail algorithms should be used only above some high emissions value,
13 where cutpoints would never be considered, and fast-pass algorithms should be used only if the
14 instantaneous emissions measurement of a vehicle is at a fraction (e.g. 50%) of the standard
15 cutpoint value. This would allow an analysis of the full cycle emissions data for all inspections in
16 a window, for example, between 50% of the cutpoint and 400% of the cutpoint.
17
18 The second area of information required to optimize the cutpoint is the distribution of the
19 repairable and non-repairable vehicles below the current cutpoint; however, there is no
20 unobtrusive, cost-effective method to obtain such data. Normally, no vehicles that are
21 designated pass are sent to repair, and therefore, the fractions that are repairable and non-
22 repairable are not known. Therefore, the ol^ly way To find these fractions is to try to repair or to
23 diagnose a sample of the passing vehicles. This could be done with a random sample of the fleet
24 that passed by offering the vehicle owner an incentive to participate. The cost of the incentive
25 would be paid for by the increased cost-effectiveness of the I/M program after cutpoints are
26 adjusted. Given the anticipated difficulties of such a study, it may be best left for a joint study
27 between EPA and interested states to perform a pilot study that would provide insight into this
28 question. But it does seem clear that states that had access to such cutpoint optimization
29 procedures would tend to have better I/M programs than states that did not, and their I/M
30 programs would benefit from the optimization.
31
32 4.2.2.3 Recommended Best Practices
33 A state should provide a process flow diagram of the flow of vehicles through its I/M program.
34 The diagram should show vehicle counts at all points. The emissions tests used should be
35 defined and evaluated in terms of measurement error and vehicle-vehicle response differences
36 with respect to a reference test (FTP or IM240). A definition and effectiveness evaluation of
37 cutpoints should be made. Effectiveness should be evaluated in terms of false fails and false
38 passes based on the repairs performed whenever possible.
39
40 4.2.3 Out-of-Program Comparison Data
41 States also may be able to use out-of-program comparison data to demonstrate inspection
42 effectiveness*. Only in-program data can be used to demonstrate the I/M program data quality of
43 a state's particular program as discussed in Section 4.2. However, the quality of the emissions
44 inspections themselves may be judged using out-of-program comparison data . Two techniques
* The term out-of-program comparison data is used here to distinguish from the term out-of-program data that is
typically used to refer to RSD or road side pullover data.
DRAFT August 2001 - 27 -
-------
1 for doing so are discussed below. States may be able to suggest other techniques to help put the
2 inspection effectiveness of a state's I/M program in perspective.
3
4 A round robin is a technique commonly used by laboratories to cross check analytical methods
5 among a group of laboratories. For example, diesel fuel samples taken from a single bulk
6 quantity are sent to different labs for analysis of aromatics. The labs may analyze the aromatics
7 by their method of choice (e.g. FIA Hydrocarbon, HNMR, CNMR, GC-MS, Aniline Point, etc.)
8 or by all the methods each lab has available. Analysis of the round robin results from all labs
9 reveals which labs reported results that were significantly different from the participants in the
10 round robin. Those "outlying" labs can then investigate the details of their analytical methods. If
11 several different types of samples are sent to each lab, the results can also be used to look for
12 biases among the analytical methods. The same round robin technique may be applied to
13 emissions inspections as well and is commonly used by auto manufacturers and regulatory
14 laboratories.
16 4.2.3.1 Vehicle Round Robin Testing
17 The first technique might be to send test vehicles to different I/M stations for testing. Shipment
18 could be done using vehicle transporters so that the emissions characteristics are not changed
19 greatly as a result of mileage accumulation; same states already do this. The vehicles would be
20 selected to cover a range of technologies, model years, and emissions levels. The emissions of
21 these vehicles could be tested at different I/M stations within the state. Analysis of results would
22 indicate the variability among I/M stations in the state. If repeat tests were performed on the
23 vehicles at each station, the variability Q£ emissions testing at participating stations could be
24 determined.
25
26 A slight variation of this application might be even more useful. Vehicles could be transported
27 for testing at I/M stations in neighboring^states. Where large populations are near a state border,
28 private vehicle owners could be paid an incentive to participate in a state-to-state I/M program
29 comparison effort. Since neighboring states may use different emissions measurement methods
30 (e.g. IM240, ASM, two-speed idle, pressure, purge and pressure, gas cap check, etc.), these
31 results would provide data to evaluate emissions measurement effectiveness of the different
32 techniques and to establish relationships among the different methods. If the transport of
33 vehicles is not possible, at a minimum, gas bottles of known concentration could be measured at
34 the respective test facilities within a give state or among neighboring states to assess analyzer
35 accuracy and judge the relative effectiveness of the slight differences that will invariably exist
36 between analyzer QA/QC procedures.
37
38 4.2.3.2 Test Crew Round Robin Testing
39 In a second technique, instead of transporting vehicles, I/M instruments and test crews could be
40 sent to neighboring states. The crews would set up at neighboring state I/M stations and inspect
41 some of that state's vehicles. Vehicles would be inspected by their state's crew and then would
42 be offered an incentive to undergo I/M testing by the out-of-state crew. Reciprocal agreements
43 among neighboring states would provide for reciprocal testing visits and sharing of data. This
44 technique would provide a much large sample of vehicles tested by two emissions measurement
45 methods than the first technique.
46
DRAFT August 2001 - 28 -
-------
1 4.2.3.3 Recommended Best Practices
2 The quality of the emissions inspections themselves can be judged using out-of-program
3 comparison data. Round robins of vehicles or I/M analyzers with crews sent to I/M stations of
4 adjacent states can be sources of data for comparisons. Emissions measurements of vehicles or
5 gas bottles of known concentration analyzed by two different I/M programs will reveal
6 measurement bias between the programs. If resources permit, the information provided by such
7 efforts is believed to be worth pursuing.
8
9
10 4.3. Effectiveness of Repairs
11
12 4.3.1 Number and Type
13 State 3 requires all state-certified repair stations to record in the Vehicle Information Database
14 the repairs that were made to each vehicle. For each repair event, the repair station records all
15 repair actions that were made to the vehicle from a list of 34 repair types. Supporting information
16 is also entered for station identification, vehicle identification, repair cost, repair date and time,
17 etc.
19 Tables 4-3 and 4-4 show the frequency of repair station actions taken for each repair type for
20 passenger cars in two different model year groups. Table 4-3 shows results for 2,486 repair
21 events on 1976-1980 model year vehicles, and Table 4-4 shows results for 2,593 repair events on
22 1991-1995 model year vehicles. These model year groups were chosen to show the differences
23 in repair types and frequencies for vehicles of different technologies and ages. The 34 repair
24 types are described in the first column of each table, last column of each table gives the
25 percent of repair events that involved the item indicated.
26
27 In general these tables show the level of repairs that were made to these vehicles. Such data
28 documents that repairs are being made and therefore, on the simplest level, the I/M program is
29 causing repairs to be made to vehicles in the fleet. A state that has a larger fraction of its
30 vehicles undergoing repairs in comparison to another state can, all other things being equal, be
31 expected to have a more effective I/M program. Obviously, stations that perform repairs where
32 none are needed will decrease effectiveness. Additionally, whether these repairs are effective at
33 reducing emissions must also be demonstrated. This is the subject of the next sub-section.
34
35 4.3.2 Emission Reductions
36 A state can demonstrate the effectiveness of its I/M program by performing an analysis ofin-
37 program emissions measurements before and after repairs. At the simplest level, this can be
38 demonstrated by the average emissions of repaired vehicles before and after repair and the
39 average emissions change. State 3 used the ASM2525 test for its I/M program. Table 4-5 shows
40 the averages for the repaired vehicles in the two chosen model year groups.
41
42 Initially, such a table seems to indicate that the I/M program is producing real emissions
43 reductions. However, because of the "regression toward the mean" effect, any emissions
44 reductions based on the same measurements used to declare vehicles as emission failures are
45 biased. Thus, even if no repairs were made to failing vehicles, the average change of measured
DRAFT August 2001 - 29 -
-------
Table 4-3. Repair Station Actions for 1976-1980 Cars
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
SPLUGS
IGWIRE
DISTR
SPAADV
SPATIM
VACLEA
IDLMIX
IDLSPE
CARINJ
AIRFIL
CHOKE
TAG
PCV
AIRINJ
EGR
EVAP
GASCAP
CATCON
FFR
O2SENS
IPS
WOT
MAP
MAE
CIS
TVS
OTHSEN
PROM
ENGINE
PVALVE
CFLOAT
EGRPAS
EGRCTL
OTHER
spark plugs
ignition wires
distributor
spark advance
spark timing
vacuum leaks
idle mixture
idle speed
other carburetor or fuel Sinjection work
air filter
choke
thermostatic air cleaner
positive crankcase ventilation
air injection
exhaust gas recirculation "^^
evaporative control
gas cap
catalytic converter
fuel filler restrictor
oxygen sensor
throttle position switch
wide open throttle sensor
manifold absolute pressure sensor
mass air flow sensor
coolant temperature sensor
thermal vacuum switch
other sensors /y±
engine management computer U '
engine management computer I 1
carburetor power valve %1
carburetor float ~**
egr passages
egr controls
other repair items
Reconnected
0
5
0
9
0
34
0
0
1
0
1
28
5
22
81
14
0
3
0
0
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
SPLUGS
IGWIRE
DISTR
SPAADV
SPATIM
VACLEA
IDLMIX
IDLSPE
CARINJ
AIRFIL
CHOKE
TAG
PCV
AIRINJ
EGR
EVAP
GASCAP
CATCON
FFR
O2SENS
IPS
WOT
MAP
MAE
CIS
TVS
OTHSEN
PROM
ENGINE
PVALVE
CFLOAT
EGRPAS
EGRCTL
OTHER
spark plugs
ignition wires
distributor
spark advance
spark timing
vacuum leaks
idle mixture
idle speed
other carburetor or fuel Sinjection
work
air filter
choke
thermostatic air cleaner
positive crankcase ventilation
air injection
exhaust gas recirculation
evaporative control
gas cap ~wr
catalytic converter
fuel filler restrictor
oxygen sensor ^.*M.
throttle position switch
wide open throttle sensor
manifold absolute pressure sensor
mass air flow sensor
coolant temperature sensor
thermal vacuum switch
other sensors
engine management computer
engine management computer
carburetor power valve
carburetor float
egr passages
egr controls
other repair items
Reconnected
1
2
0
0
0
13
0
0
1
0
0
1
5
2
9
1
0
1
0
9
~0
^0
BT?
0
r fl
3
i
0
jJNd l|0
0
0
0
5
0
Defective and
Not Repaired
10
15
8
1
2
3
9
5
35
6
0
5
1
4
14
1
T
342
1
48
2
0
2
5
1
0
5
21
143
^^
1
* !V>
12
17
Not
Applicable
214
274
883
398
401
10
681
531
535
312
1875
1585
454
1685
982
517
k 793
154
795
170
486
1282
1033
1355
452
1137
654
537
762
2449
2469
1170
1134
1156
Item is
OK
1614
2086
1479
2152
1741
2384
1508
1690
1670
1798
704
989
1900
883
1370
2060
1795
1512
1786
1445
2027
1305
1549
1207
2103
1450
1910
1994
1544
133
117
1309
1359
1223
Replaced
697
213
187
3
2
6
0
0
27
425
6
1
169
6
99
4
3
582
8
872
16
1
3
9
22
1
16
31
1
1
2
1
26
49
Repaired,
Cleaned, or
Adjusted
55
1
34
37
445
175
393
365
323
50
6
10
62
11
117
8
0
0
1
47
60
3
2
15
12
2
5
8
141
6
2
101
55
146
% Replaced,
Repaired,
Cleaned, or
Adjusted
29.1
8.3
8.5
1.5
17.3
7.5
15.2
14.1
13.5
18.3
0.5
0.5
9.1
0.7
8.7
0.5
0.1
22.5
0.3
35.8
2.9
0.2
0.3
0.9
1.4
0.2
0.8
1.5
5.5
0.3
0.2
3.9
3.3
7.5
1
2
3
4
5
6
7
9
10
11
12
13
emissions for the fleet would show a decrease. The reason for this is that vehicles that are
declared failures tend to have measurements with positive emissions measurement errors.
Therefore, states need to use a technique for producing the data for a table such as Table 4-5
manner that corrects for regression toward the mean. Section 4.3.4 describes such a method.
Table 4-5. Observed Average Emissions Before and After Repairs
ma
1976-1980 Cars
1991-1995 Cars
N
2486
2591
Average ASM2525
Concentration Before
Repair
HC
(ppm)
187
87
CO
(%)
1.58
0.84
NOX
(ppm)
1143
902
Average ASM2525
Concentration After
Repair
HC
(ppm)
106
35
CO
(%)
1.05
0.14
NOX
(ppm)
870
511
Average ASM2525
Concentration
Change
HC
(ppm)
-81
-52
CO
(%)
-0.52
-0.70
NOX
(ppm)
-273
-391
By combining the repair data with the emissions data, an analysis will reveal the emissions
effects of different combinations of repair types. For example, Table 4-6 shows the most
frequent combinations of repair types for the two chosen model year groups. The 15 most
DRAFT August 2001
-31-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
frequent repair combinations for the 1976-1980 cars accounted for 33% of the repair events for
this vehicle group. For the 1991-1995 car group, the 8 most frequent repair combinations
accounted for 33% of the repair events.
An examination of individual repair combinations, their associated average emissions before
repair, and the emissions changes that the repairs produced shows expected effects of repairs on
emissions. For example, for the 1991-1995 car group, Repair Slate D5 (EGR) was applied to
vehicles with very low HC, very low CO, and very high NOX emissions and resulted in small
changes in HC and CO, but large decreases in NOx. On the other hand, Repair Slate D3
(Catalytic Converter and O2 Sensor) was applied to vehicles with moderately high HC, CO, and
NOx emissions and resulted in relatively large decreases in HC, CO, and NOx emissions. For the
1976-1980 car group, Repair Slate A10 (major carburetor work) was applied to vehicles with the
highest average HC and CO and just about the lowest NOX and resulted in large decreases in HC
and CO and large increases in NOx.
It Tfl Tf^
Each state is encourage to collect repair data in a similar way, then comparison of results such as
those shown in Table 4-6 could be part of a repair program evaluation. For example, it would be
expected that repair stations perform the same repair slates on corresponding technology vehicles
in different states, although the frequency distribution will vary with test type and cutpoints. In
addition, the average before-repair emissions and emissions changes for those repair slates
should be similar among different states with comparable repair programs. If one state's repair
stations applied repair slates more indiscriminately than another state's, the differences among
before-repair emissions averages woukLbe spiajler and emission decreases would be smaller.
aid be sm
.m
Table 4-6. Emission Reductions Associated with Combinations of Repairs
MJ
J5
VI
'«
o.
tf
N
Type of Repair
TTOAIT
01]
3
0.
_
o
t/3
i>
u
o
<
_
o
t/3
UD
c
S
H
"S
o
Vl
^>
y$
04
_
s
a
i*?
HH
o
1
O
,Q
«
U
J.
,Q
«
U
ll
-------
A10
All
A12
A13
A14
A15
22
19
18
18
17
16
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
219
165
89
118
216
98
3.78
3.50
0.42
2.76
1.01
0.58
598
589
2265
757
1246
989
-79
-74
-1
-41
-99
-30
-1.73
-2.11
0.18
-1.04
-0.07
0.16
333
483
-1278
-165
-67
-351
1991-1995 Cars
Dl
D2
D3
D4
D5
D6
D7
D8
301
237
80
58
55
49
44
38
X
X
X
X
TR
X
X
X
X
X
X
101
69
85
85
25
45
80
, 120
1.75
0.25
0.60
0.34
0.11
0.19
0.63
1.79
581
1317
1159
783
1589
570
1176
559
-74
-48
-70
-40
7
-19
-41
-89
-1.66
-0.20
-0.55
-0.23
0.04
-0.13
-0.36
-1.80
-169
-812
-728
-273
-719
-225
-446
-194
1
2
O
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
4.3.3 Repair Lifetimes
Once a state has shown that its I/M program is causing repairs to be made and the repairs are
causing emissions reductions, the final effect the state should quantify is the lifetime of the
repairs. If repairs last only a short time, the emissions benefits may only last a short time. If the
repairs last many years, then it is at least possible that the emissions benefits may last many
years. In addition, long lasting repairs help reduce the number of repairs that will be expected in
future years. In other words, one reason the number of repairs is low in a given year may not be
because of a failure of the vehicle inspections to identify them. Instead, it may be because
repairs made in previous years are durable.
The duration of repairs can be evaluated by analyzing a good repair database. For this example,
the repair data from State 3 was analyzed. For this state the I/M program repair data for five
consecutive years was available and subset of the vehicles that had any repair performed in the
first year was selected. The number of days between that first repair and the next repair of any
kind was calculated. If the vehicle did not get a second repair in the five-year data set, then the
duration was set to 1825 days for plotting purposes. Figure 4-11 shows the result of that
distribution.
DRAFT August 2001
-33-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
_CD
O
~o
CD
L_
"o
o_
CD
a:
M
o
CD
u
i_
CD
Q_
^^
Second Repair (days)
Figure 4-11. Distribution of Intervals from Any Year 1 Repair to Any Next Repair
The cumulative distribution shows that about 25% of the vehicles that had a repair in the first
year had at least a second repair by the fifth year. This leaves 75% of the vehicles that had a
repair in the first year and did not receive a repair (or at least did not have a second repair
recorded in the database) for the next four years. Perhaps more importantly, the plot also
indicates that by the end of the second year abou| 2jG%|of the vehicles already had a second
repair. This rapid rise in subsequent repair intervals suggests that some vehicles require frequent
repairs.
The programmatic implications will depend on an analysis by repair type. Some repairs may be
routine adjustments that are not really the result of serious degradation. Examples are idle speed
and idle mixture adjustments on carbureted vehicles. This contrasts with catalytic converter
replacements, which should not be performed routinely on any vehicle.
A more detailed analysis of this repair data by vehicle age, vehicle technology, and repair type
should illustrate the situations where repair durability is strong and where it needs to be
increased. Such an analysis could help a state improve its repair stations' performance. From an
I/M program evaluation perspective, an analysis of overall repair duration for the repaired
vehicles and a targeted analysis for different repair types would demonstrate that, beyond simply
making repairs, the repair stations are making repairs with a quantifiable durability.
4.3.4 Other Measures
In an effective I/M program the vast majority of vehicles that initially fail the emissions
inspection will require only a single repair event to pass the emissions inspection. In less
effective I/M programs, some vehicles will make repeated trips between inspection and repair in
DRAFT August 2001
-34-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
an effort to meet annual I/M emissions requirements. The cause for such "ping-ponging" may be
emissions measurement error, faulty repair diagnosis, or poor repair quality. Whatever the
cause, the vehicle owner will be frustrated. Emissions measurements and repair events with date
and time stamps are required to evaluate "ping-ponging" events.
I/M program inspection and repair databases also reveal that some owners of failing vehicles will
go from inspection station to inspection station to try to find a station that will pass their vehicle.
This so-called "shopping around" is distinguishable from "ping-ponging" because for "shopping
around" consecutive inspections do not have repair events between them.
In this example, State 3 apparently recorded all repair types, even if they occurred at different
repair events, as a single repair event. Accordingly, separate "ping-ponging" from "shopping
around" cannot be filtered out. Table 4-7 shows the distribution of repeated fails for State 3 as
an example of the type of result that could be expected from an analysis of "ping-ponging."
IT ₯%
Table 4-7. Distribution of Repeated Adjacent Inspection Failures Prior to a Pass
I
Number of Vehicles
Another measure of repair effectiveness is a comparison of the cost of all repairs to the reduction
of all emissions. Cost-effectiveness values ($/ton) could be calculated for the I/M program
overall and for individual repair slates. The calculations would require the logging of the repair
bill for each repair event. Calculated cost-effectiveness values can then be compared with
reference values from sources such as U.S. EPA and California BAR or other states.
4.3.4 In-Program Studies to Measure Repair Effectiveness
Various methods can be used to quantify repair effectiveness using modifications to normal in-
program procedures. The effectiveness of repairs can be examined by comparing the change in
emissions of failing vehicles when they are repaired with changes in emissions of vehicles that
are not repaired but just tested again. This comparison must be performed to avoid so-called
regression toward the mean, which would cause repair emission benefits to be over-estimated.
An example is described below. Other such methods for measuring repair effectiveness may be
devised.
A subset of vehicles failing the initial emissions test (Test A) would be assigned to the
Evaluation Group or the Control Group. These vehicles would be selected from the set of all
failing vehicles using a stratified random sampling method. Vehicles in the Evaluation Group
would immediately receive a second emissions test (Test B) and would then be sent for repairs
DRAFT August 2001
-35-
-------
1 based on their result on Test A (i.e., even if they passed Test B). When these vehicles returned
2 from repair, they would be given the repair follow-up emissions test (Test C). Following Test A,
3 vehicles in the Control Group would immediately receive a second emissions test (Test D) and
4 would immediately be given a third emissions test (Test E). Then, these vehicles would be sent
5 to repair and would return for their repair follow-up emissions test.
6
7 The actual emissions benefit of repairs is (C-B) - (E-D). This is the change in emissions before
8 and after repairs of the Evaluation Group vehicles less the change in emissions of vehicles in the
9 Control Group that did not receive repairs. It is critical that Test A results not be used to
10 calculate repair benefits. Doing so would introduce a bias in the calculated benefits. It is also
11 critical that all vehicles that fail Test A and pass Test B be sent for repairs. To not do so would
12 also introduce a bias in the calculated benefits.
13
14 4.3.5 Repair Data Collection
15 Repair data needs to be collected by I/M programs for analyses of repair effectiveness to be
16 made. Development of data collection requiremerJft'tan begin with the approaches used in states
17 that are currently collecting such data. Then, improvements to the approaches can be made as
18 states gain experience collecting and analyzing the data.
19
20 Repair stations should enter vehicle, emissions, and station information for each repair they
21 make:
22 station identification,
23 vehicle identification,
24 repair date and time,
25 repair cost, and
26 repair codes for standarcTrepairs such as those in Table 4-la.
27
28 This repair information should be entered each time a vehicle enters a repair station for work. In
29 most states, most repair work is done in repair stations that are not connected to the VID, or
30 repairs are done by the vehicle owner. Therefore, to allow the VID to achieve completeness and
31 accuracy targets for repair data, techniques need to be developed for acquiring repair data.
32
33 4.3.6 Recommended Best Practices
34 Section 4.3 discussed methods for evaluating the effectiveness of repairs in an I/M program.
35 Unfortunately, most current I/M programs place greater emphasis on accurately measuring
36 vehicle emissions and designating vehicles as pass or fail than on ensuring or even monitoring
37 the quality of vehicle repairs. This natural emphasis is probably a consequence of the more
38 quantifiable aspect of emission measurement over vehicle maintenance. As the discussions in
39 Section 4.3 demonstrated, acquiring a database of vehicle repairs would provide information and
40 opportunities that are not currently available in most I/M programs. Therefore, one the most
41 important recommendations is for states to develop database systems which are capable of
42 monitoring vehicle repairs so that the beneficial aspects of the analysis of those databases can be
43 realized. The list below summarizes the key aspects in this regard:
44
45 Repair data collection. States need to make a concerted effort to collect repair information on
46 all vehicles participating in the I/M program. The data should be collected in a manner such that
47 it can be matched to emissions data for each vehicle. Each visit of a vehicle to a repair station
48 should generate a record in the database. The record would include vehicle identification, codes
DRAFT August 2001 - 36 -
-------
1 for the types of repairs performed, and the cost of the repairs. Strategies must be developed to
2 ensure that all repairs performed would be recorded in the database. One possibility worth
3 consideration is for states to certify repair stations.
4 Number and type of repairs. Once the database is created, simple counts of the number and
5 type of repairs demonstrate that repairs are being performed. Analysis of the data would show
6 what types of repairs are common for different types of vehicle technologies.
7 Repair lifetimes. Analysis of the repair data set could be used to quantify the duration of
8 repairs. While some repairs are routinely performed as vehicles go out of adjustment, others
9 reflect the lifetime of repair components and the general competence of repair stations. Repair
10 lifetimes should be compared among different states to determine the typical repair lifetimes in
11 different I/M programs.
12 Emissions reductions for repairs. By combining the emissions database with the repair
13 database, it would be possiblejo denjgnstrate that repairj_are actually reducing emissions. More
14 specifically, an analysis would quantify how emissions are being reduced for each type of repair.
15 Such analyses from different states sould be compared to arrive at a consensus estimate of the
16 reductions that can be achieved by certain types of repairs. As a side benefit, the fingerprint of
17 emissions on vehicles that have failed thejns|)ection could be associated with the types of repairs
18 that successfully caused the vehicle to pass the follow-up emissions test. Such relationships
19 could be used to develop diagnostic guidance for repair stations to use.
20 Measures of customer inconvenience and repair cost. The combined repair and emissions
21 databases could be used to determine the extent of customer inconvenience produced by repeated
22 visits between inspection and repair stations at the time of the annual or biennial inspection.
23 Such so-called ping-ponging can be produced by excessively stringent cutpoints, inspection
24 emissions test measurement error, faulty repair diagnosis, or poor repair quality. When repair
25 costs are included in the repair database, the total customer repair dollars can be determined.
26 Also, the repair costs for each type of repair can be determined with respect to the emissions
27 reductions that are achieved.
28 In-program studies to measure repair effectiveness Slight modifications to the inspection
29 sequence for a subset of vehicles in the I/M program can produce data that will provide an
30 estimate of the effectiveness of the I/M program. The modifications are used to eliminate biases
31 produced by the so-called regression toward the mean effect.
32
33
34 5. Results Based Measures of Effectiveness
35
36 This section will outline procedures for analyzing the data in I/M vehicle inspection records.
37 Previous methods developed by stakeholders, contractors and EPA for this analysis will be
38 reviewed in Section 5.1 and 5.2. Section 5.3. contains descriptions of a new set of analysis
39 procedures as well as a brief discussion of the use of out-of-program data. Section 5.5 discusses
40 the testing of evaporative emissions. None of the procedures use MOBILE modeling;
41 comparisons are made between different years of test data and between different programs, but
42 projections to no-I/M levels are not attempted. The significance of any results obtained through
43 analysis of the I/M test records must be weighted by the findings from the procedures in Sections
DRAFT August 2001 - 37 -
-------
1 3 and 4. Additionally, the data validation methods described in Section 4 must be applied prior
2 to analysis. It is also important to realize that the model year results described in this section
3 should be weighted by vehicle miles traveled or some other travel fraction weighting.
4 5.1ECOS Method
5 The Environmental Council of States (ECOS) Group was formed in 1996 to develop an
6 evaluation process for state I/M programs with test and repair networks15. The primary objective
7 of the group was to develop common criteria to demonstrate equivalency to EPA's I/M program
8 standard. Twelve criteria were developed for a short-term qualitative evaluation that was to be
9 performed 6 months after program start-up. A successful completion of each criteria conferred a
10 set number of points that counted toward a successful fulfillment of the ECOS program
11 evaluation requirements. However, the focus of the criteria was on the comparison of test-and-
12 repair I/M stations to test-only stations, so that other differences that might exist between
13 programs, such as test type, data quality assurance, or cutpoint stringency, were not evaluated. A
14 second longer-term quantitative evaluation was then to be performed 18 months after program
15 start-up. One of the difficultiei'Mith the implementation of the ECOS method was that each state
16 chose a set of criteria from the twelve options to apply to their program, so it was possible to
17 choose analyses that provided favorable results, and ignore other analyses with unfavorable
18 results. Use of the ECOS criteria was dij^on^aued ia 1999.
19
20 5.2 EPA Tailpipe I Method
** =*
21 This method was based primarily on work done for EPA by Sierra Research, Inc. in 199716. The
22 original study done by Sierra was focused on comparing designer I/M tests to known reference
23 tests such as the IM240. However, in response to a court ordered deadline that required EPA to
24 establish program evaluation protocols, this study was used and modified so that it could meet
25 this need.
26
27 Under this method, a small sample of vehicles that has already met the I/M program
28 requirements is recruited for an additional I/M test. Emissions data from these vehicles is
29 compared to a baseline program that closely matches EPA's requirements for an "Enhanced I/M"
30 program. Regions that use I/M tests other than the IM240 are required to develop and apply a
31 correlation to relate emissions data from their program to equivalent EVI240 results. The
32 MOBILES model is used to correct for regional differences between the two programs, such as
33 altitude, climate, or fuels. The specific steps that have been taken to apply the method for
34 several I/M programs17'18 are listed in Table 5-1. The final result of the comparison between the
35 program under evaluation and the benchmark program is a ratio of the effectiveness of the two
36 programs.
37
38 The benefit of the Tailpipe I method is its capacity to condense comparisons between the I/M
39 program and the benchmark program into a single ratio. Also, the concept of developing a
40 correlation between the program test (TSI, ASM, etc.) and the IM240 test is a valuable tool for
41 comparing in-program data from programs using different tests. However, the reliance on the
42 MOBILES model to make the regional corrections and determine the no-I/M levels (see Table 5-
43 1) may introduce error to the results. The method also requires the use of an I/M program
44 compliance rate, which can be difficult to determine. Finally, while the use of a single
45 comparison between the two programs is convenient, it may result in some loss of detail, and
46 relevant information that might be found through a multi-faceted approach could be missed.
DRAFT August 2001 - 3 8 -
-------
1
2
Table 5-1. Steps for Application of the EPA Tailpipe I Method for an I/M Program Using the
Two-Speed Idle Test
1
Y
A random, stratified sample of about 800 vehicles is selected for use in developing a
relationship between the state's two-speed idle test results and IM240 test results.
Back-to-back EVI240 and two-speed idle tests are conducted on the sample of vehicles. This
dataset is used to develop a correlation between the results of the two speed idle test and
EVI240 emissions.
An estimated IM240 result was calculated for each I/M test record, using the correlation
between two-speed idle test results and EVI240 emissions that was developed according to
Step 2.
4
T
The 2% random sample of complete IM240 tests that is collected annually by Phoenix,
Arizona is obtained, representing data from a benchmark program.
Separately for each program (program under evaluation and benchmark program): An average
EVI240 emissions level is calculated by model year. , t
Separately for each program: Travel fractions based
MOBILES annual mileage accrual rates are used to <
astration distributions and
a single average emissions level.
The Arizona model year average emission levels are converted to match the program under
evaluation by correcting for any differences in fuel, altitude, climate, and calendar year
effects.
MOBILESb is used to model Arizona's average emission levels with and without an I/M
program in place. Inputs are based on local area parameters for the program under evaluation.
The results of this modeling are used to calculate a percent reduction in emission levels, or
benefit, achieved by the benchmark Arizona program.
Average EVI240 emissions levels for Arizona were calculated in Step 5. The benefit of the
Arizona program was calculated in Step 8. These two results are used to calculate the average
IM240 emissions level for Arizona without an I/M program in place (No-I/M levels).
10
The No-I/M emission levels calculated in Step 9 are compared to the average estimated
EVI240 emission levels in program under evaluation that were calculated in Step 5. These
results are used to calculate the percent reduction, or benefit, of the program under evaluation.
The benefits of the two programs are then compared.
11
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
5.3 Use of Data Trends
The use of data trends can be used to highlight differences between programs that may provide
useful information if investigated further. Several different types of analysis using I/M program
records are considered:
Fleet average emissions changes for a single I/M program year,
Fleet average emissions changes comparing multiple years of testing,
Emissions changes in individual vehicles over multiple years of testing, and
Comparisons with other I/M program results (different states or regions).
Fleet average emissions changes over a single year are computed in order to determine whether
the I/M program results in emissions reductions over a single program cycle, after any failing
vehicles are identified, repaired, and re-tested. Without an I/M program in place, the vehicle
would deteriorate and emissions would increase over time. With an I/M program in place,
DRAFT August 2001
-39-
-------
1 deterioration should be identified and the vehicle repaired at each test cycle. Looking at vehicle
2 emissions over multiple years where overall fleet emissions are being reduced as new vehicle
3 emissions technologies are introduced into the fleet makes this problem of identifying I/M
4 program effectiveness even more difficult. The goal in investigating fleet average emissions
5 changes over a single year is to determine whether deterioration is actually being identified and
6 reduced through repairs. A lack of emissions reductions in one year of program data would
7 indicate that any long-term fleet average emissions reductions are attributable to fleet
8 composition changes, rather than I/M program results. This type of analysis is demonstrated in
9 Section 5.3.1.
10
11 If an I/M program benefit within a single year is shown, then the emissions averages of the fleet
12 over time should be examined for long-term effects. Due to the problems associated with
13 determination of no-I/M emissions levels (i.e., moving away from empirical data with MOBILE
14 modeling, or attempting to project next years' emissions levels from this years' failed test
15 results), analysis methods are presented herein that are based pn year-to-year data. These year-
16 to-year comparisons are included in Section 5.3.2. Section 5.3.3 contains a similar analysis, but
17 fleet changes are eliminated by tracking individual vehicles that participated in the program over
18 multiple years.
20 In Section 5.3.4, program data from three different areas is compared. The comparisons are
21 made using two-speed idle data from two areas and IM240 data from a third. An additional
22 discussion of using a correlation to predict IM240 emissions levels from TSI results, as proposed
23 in the EPA Tailpipe I method, is included there. However, none of the analysis suggested
24 requires use of a correlation to compare data from states that use different types of tests.
26 5.3.1 Fleet Average Emissions Analysis for a Single Program Year
27 The single-cycle effect of an I/M program on a fleet may be found by comparing average
28 emissions levels at the beginning and the completion of the test cycles (a test cycle includes all
29 tests and retests for a vehicle, until it completes or drops out of the program). In Figure 5-1, for
30 State 1, the initial and final IM240 HC emissions of all passenger cars are presented. The State 1
31 program allows vehicles to fast-pass the EVI240 test, so results for the shorter tests must be
32 projected to full test results. Methods for projecting full test results from fast-pass data may be
33 found in the literature19'20; however, care must be taken to fully understand the implications of
34 using such algorithms as they may bias the results of the program evaluation analysis. The data
35 in Figure 5-1 is grouped by initial and final test result. It can be seen that the average emissions
36 of the vehicles that initially failed but were eventually repaired and passed decreased
37 significantly, almost to the level of the vehicles that passed on the first attempt. Vehicles that
38 dropped out of the program before being repaired and passing an inspection show almost no
39 reduction (the two lines are difficult to differentiate because they lie almost on top of each other).
DRAFT August 2001 - 40 -
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
82
84
86
88 90
Model Year
92
94
96
Initial Emissions of Final Pass Vehicles
Initial Emissions of V\feived Vehicles
Initial Emissions of Incomplete Sequences
Initial Pass Vehicles
Final Emissions of Final Pass Vehicles
Final Emissions of V\feived Vehicles
Final Emissions of Incomplete Sequences
Figure 5-1. Initial and Final All Passenger Cars, IM240 HC, State 1
While Figure 5-1 gives a good visual representation of the emissions reductions, it could be
misleading on its own. For example, the figure shows extremely high emissions for 1992
vehicles that received waivers, but it doesn't show that this group includes only one vehicle. A
minimum of 25 records per bin is often considered to be a cutoff below which averages are
unreliable (as for the 1992 waived vehicles in Figure 5-1). Table 5-2 provides additional
information about the data presented graphically in Figure 5-1, for the vehicles that initially or
ultimately passed the I/M test. From the table, it may be seen that sample sizes vary greatly
among the model years. It may also be seen that the standard deviation of the results is often as
large or larger than the mean value; this large spread is not apparent from Figure 5-1.
Table 5-2. Initial and Final Emissions for All Passenger Cars, EVI240 HC, State 1
Model
Year
82
83
84
85
86
87
88
89
90
91
92
93
Initial Pass
Number of
Vehicles
4831
12760
9885
23440
14504
32629
18189
41190
19388
45202
18782
44006
MeanHC
[g/mile]
1.66
1.37
1.37
0.98
0.87
0.72
0.73
0.57
0.54
0.41
0.36
0.29
Std.
Dev.
0.97
0.83
0.86
0.65
0.62
0.54
0.56
0.46
0.45
0.36
0.34
0.27
Initial Fail (for Vehicles that
Ultimately Pass)
Number of
Vehicles
889
1689
1508
3910
1935
3028
1495
1707
812
1351
533
792
MeanHC
[g/mile]
4.67
3.91
3.96
2.75
2.88
2.42
2.73
2.30
2.27
1.73
1.80
1.41
Std.
Dev.
5.38
4.03
4.62
3.12
3.30
2.94
4.17
3.79
3.67
2.91
2.63
2.47
Ultimate Pass (After Initially
Failing)
Number of
Vehicles
889
1689
1508
3910
1935
3028
1495
1707
812
1351
533
792
MeanHC
[g/mile]
2.15
1.79
1.77
1.18
1.16
0.94
0.98
0.81
0.78
0.62
0.64
0.47
Std.
Dev.
1.18
1.05
1.07
0.82
0.82
0.71
0.75
0.64
0.68
0.58
0.57
0.45
DRAFT August 2001
-41-
-------
94
95
96
97
38857
22329
15457
9327
0.21
0.16
0.12
0.11
0.22
0.18
0.12
0.09
393
231
159
57
1.09
0.67
0.43
0.38
2.32
1.26
1.37
1.79
393
231
159
57
0.31
0.18
0.14
0.09
0.39
0.27
0.19
0.12
1
2
3
4
5
6
7
8
9
10
11
12
13
14
The other point of information not shown in either Figure 5-1 or Table 5-2 is that the emissions
data that is averaged to generate each data point does not exhibit a normal (Gaussian)
distribution. Figure 5-2 shows the distribution of values of IM240 HC in all records for the
initial test on 1990 vehicles that ultimately passed. The data does not have a symmetric normal
distribution: the vast majority of vehicles have emissions near zero, while the high emitting
vehicles form a long "tail." When plotted on a logarithmic scale, the distribution is more nearly
symmetric, as shown in Figure 5-3. Because the log-normal distribution includes only positive
values, and because it condenses high values while spreading out the lower values, it is often
used to transform emissions data. Averages of emissions should still be performed on the raw
(linear space) values as for Table 5-2_since those averages represent the emissions of the model
year average vehicle.
400
10 15
20 25 30
IM240 HC
35
40 45
50
Figure 5-2. Distribution of Records for Single Data Point, EVI240 HC, State 1
E
^
15
16
IM240 HC
Figure 5-3. Log-Scale Distribution of Records for Single Data Point, IM240 HC, State 1
DRAFT August 2001 - 42 -
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Figure 5-1 presented only the IM240 HC emissions levels for passenger cars. If the purpose of
this report were to analyze the program effectiveness of State 1, additional figures would be
given for light duty trucks (and heavy duty trucks, if covered by the program), and results for
IM240 CO and NOx would be presented as well. This level of detail is useful in identifying
groups of vehicles with anomalous results, but larger trends may be easier to see in a more
general presentation such as Figure 5-4. This figure presents the overall emissions reductions for
passenger cars, as a percent decrease from initial to final test. Vehicles with only one test (i.e.,
initially passed or initially failed and dropped out of program) are included in the averages for
both the initial and final tests. It is clear that the vast majority of emissions reductions result
from the older vehicles.
-B-IM240CO
-*-IM240HC
-e-IM240NOx
1982 1984 1986 1988 1990 1992 1994 1996 Ave
Model Year
Figure 5-4. Overall Emissions Reductions for Passenger Cars, Incomplete Sequences Included,
State 1
One assumption behind the data in Figure 5-4 was that the vehicles that left the program before
passing a test (dropping out before completing their test sequence) remained in the area; data for
their last inspection is included in the average. However, if these vehicles were sold or otherwise
moved outside the program area, then they are no longer part of the fleet and the data for their
last inspection should be removed from the final test average. This change was made for Figure
5-5, but resulted only in slightly greater average reductions.
DRAFT August 2001
-43-
-------
1
2
3
4
5
6
7
8
9
10
11
12
-IM240CO
-IM240HC
-IM240NOX
1982 1984 1986 1988 1990 1992 1994
1996
Ave
Model Year
Figure 5-5. Overall Emissions Reductions for Passenge:
"Inched, State 1
, Incomplete Sequences Not
In addition to emissions reductions, the rale at which vehicles fail the inspection can be
informative; for example, a very low fail rate may indicate that cutpoints are too high to identify
some vehicles that would benefit from rfpiirfsee discussion of cutpoints in Section 4.2.2.2).
The rate at which vehicles failed their initial test in State 1 is shown in Figure 5-6. The overall
height of each bar indicates the total percentage of vehicles that failed their initial test; the
different sections within the bars divide the vehicles by the result they finally achieved before
leaving the program. Vehicles that recervt-waivers comprise a very small percentage of the
total.
13
14
15
Initially Failed, Ultimately
Received Waiver
D Initially Failed, Passed on
Retest
Initially Failed, Did Not
Complete Sequence
82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97
Model Year
Figure 5-6. Fail Rate for Initial Test, IM240, State 1
DRAFT August 2001
-44-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Finally, the number of tests required for vehicles to complete the program is shown in Figure 5-
7. Vehicles that passed their initial test are not included on this figure, since they each had
exactly one test. This information is somewhat related to the repair information presented in
Section 4.3, i.e. more effective repairs require fewer repeat tests before a vehicle passes. It is
interesting to note that vehicles that eventually drop out of the program before passing tend to
average almost as many repeat tests as vehicles that eventually pass. However, from Figure 5-1
it was seen that the emissions levels of these vehicles were almost unchanged from initial to final
failed test. It is possible that these vehicles are not being repaired between tests, or that the
owners leave the program in discouragement when repairs show no emissions benefit.
B Initially Failed,
Did Not
Complete
Sequence
X Initially Failed,
Passed on
Retest
-Initially Failed,
Ultimately
Received
Waiver
1982 1984 1986 1988 1990 1992 1994 1996 1998
Model Year
Figure 5-7. Average Number of Tests Required to Complete Program, State 1
5.3.1.1 Recommended Best Practices
It is recommended that analyses as illustrated in Figure 5-1 be used with vehicle miles traveled
(VMT) data to obtain average emissions by model year and test sequence. Figures 5-4 and 5-5
demonstrate how in-program data may be used to estimate average emissions reductions by
model year. Analyses such as those in Figure 5-6 should be used to track the rate at which
waivers are issued; the rate at which vehicles are repaired, resulting in an air quality benefit; and
the rate at which vehicles drop out the program, resulting in a lost air quality benefit, while
Figure 5-7 type of analyses provide information to track the progression of vehicles through the
program.
Each of these analyses uses only a single year of program data; this is the most basic level of
emissions results analysis. Whenever possible, the use of analyses such as those depicted in
Figures 5-1 and 5-4 through 5-7 should be combined with the multiple-year and multiple-state
analyses described in 5.2.2 through 5.2.4.
DRAFT August 2001
-45-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
5.3.2 Fleet Average Emissions Analysis for Multiple Program Years
The comparison of multiple years of I/M test records allows the observance of fleet emissions
trends over time.
Trend analyses as portrayed in Figure 5-8 have been used by others21 may be used to examine the
changing emissions of the fleet over different program years. Each line shows the average
emissions for the initial test of a different model year vehicle, plotted against the age of the
vehicle at the time of test. Without an I/M program in place, the emissions of each model year
would be expected to increase as the vehicles age. For the two-speed idle HC data of State 3,
shown in the figure, the average emissions in the newest model years actually do show
increasing emissions over time. However, the emission levels may still be so low that the
vehicles are not yet affected by the I/M program. The significant increase in emissions levels
between 1988 and 1987 illustrates the significance of cutpoints in fleet emissions as the 1987 HC
cutpoints are almost twice as high as the 1988 cutpoints. For other fleets without a similarly
large change in cutpoints, the gap in emissions between 1987 and 1988 is not seen, indicating
that the gap on Figure 5-8 is not due to vehicle technology changes. For the model years older
than 1987, the decrease in emissions as the vehicles age is clear, possibly indicating that the
program is having an effect on this component of the fleet, or that high-emitting vehicles drop
out of the program or are sold out of thejarogjarn area^to avoid further testing.
In summary, the primary purpose of Figure 5-8 is to look for potential problems, such as gaps
between the model years that indicate inadequate cutpoints, or large increases in emissions
within a model year as the vehicles age^indicaljng unchecked deterioration.
200
180
140
Q.
Q.
C
1 100
'E
LU
O
80
P
01 2345
6 7 8 9 10 11 12 13 14 15 16
Vehicle Age
-82
-83
-84
-85
-86
-87
-88
-89
-90
-91
-92
94
-95
-96
-97
Figure 5-8. Emissions Averages at Different Vehicle Ages, TSI HC, State 3
Figure 5-9 shows the percent emissions reduction from initial to final test for State 3 over four
years of I/M testing (similar to the single year of data shown in Figure 5-4). The x-axis of the
figure is the vehicle age, so a vehicle with an age of five years in the first year of program data
will be shown as six years old in the second year of program data and seven years old in the third
DRAFT August 2001
-46-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
year of data. This age-based presentation allows the emissions reductions over the different I/M
program years to be compared based on the length of time the vehicle has had to deteriorate.
Thus Figure 5-9 may be used to investigate whether the effectiveness of the I/M program
changes over time. For example, an ideal case would be a fleet with no immigration of vehicles
from outside the program area, covering a fleet of well-maintained vehicles. In this situation it
would be possible for all vehicles to eventually be repaired to passing emissions levels, and no
further emissions reductions would be achieved. While reductions didn't drop to zero for State
3, as shown in Figure 5-9, it does appear that the reductions decrease over the four years
presented. This may indicate that many high-emitting vehicles have been repaired, fewer
vehicles are failing the test, and the program is having a benefit. Conversely, it is possible that
the emissions levels for the initial tests are increasing. Figure 5-8 is based on initial emissions,
and indicates that while initial emissions for the 1988 and newer vehicles increase slightly from
year to year as they age, the initial emissions of the older vehicles do not increase as they age
from year to year.
10 12
Vehicle Age
14
16 18
20
Figure 5-9. Percent TSIHC Reduction Over Four Years, State 3
The initial fail rate for the vehicles of State 3 is shown in Figure 5-10, for four years of program
data. The trends correlate to what was seen in Figure 5-9 as the high fail rate for the older
vehicles, which decreases over the four program years, fits well with the high reductions seen in
Figure 5-9. Any inconsistencies between these two figures (i.e., a very low fail rate but high
emissions reductions) might be an indication of a problem with the I/M program data.
DRAFT August 2001
-47-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
! 10
Vehicle Age
12
14
16
18
20
Iff
Figure 5-10. Initial Fail Rate, State 3
Figure 5-11 illustrates another type of analysis donatly McClintock22, looking at the skewness of
State 3's TSIHC emissions. This figure shows the percent of total emissions that are contributed
by the dirtiest 10% of vehicles in each model year, which is as high as 50% for State 3. Since
they contribute such a large portion of the total emissions, repair of these vehicles provides a
large portion of the emissions reductions an I/M program achieves. From Figure 5-11, it can be
seen that the emissions contributed by the dirtiest 10% of the vehicles remains relatively constant
over the program years. Again, there could be more tlan one explanation. For example, the
overall fleet emissions may be decreasing, with emissions from the dirtiest 10% decreasing in
approximately the same proportions, or the highest emitters result may be due to new vehicle
equipment malfunctions or immigration of high-enjittipg vehicles.
Another way of looking at the skewness is to look at the emissions contributed by 10% of the
dirtiest vehicles of the overall fleet, chosen without stratifying by model year. These vehicles are
concentrated in the oldest model years, as shown in Figure 5-12, with very few newer vehicles in
the group. The percent of the emissions that the overall 10% dirtiest vehicles in the fleet
contribute to each model year is shown in Figure 5-13. For the oldest model years, this
contribution is over 80%. This type of information could have several uses. For example, if a
high-emitter identification program is being considered, Figures 5-12 and 5-13 could help
identify model years with the greatest number of target vehicles. Also, changes in the
distribution shown in Figure 5-13 from year to year of program data could identify cutpoint
problems. For instance, if high emitters were increasingly concentrated at a certain age range,
the cutpoints at that age might be too lax.
5.3.2.1 Recommended Best Practices
The analyses shown in Figures 5-8 through 5-13 should be used when multiple years of program
data are available. Figure 5-8 may be used to look for potential problems, such as gaps between
the model years that indicate inadequate cutpoints, or large increases in emissions within a model
year as the vehicles age, indicating unchecked deterioration. The percent reductions over the
program years shown in Figure 5-9 should be used to confirm that the program retains its
DRAFT August 2001
-48-
-------
1 effectiveness over time. The initial fail rates shown in Figure 5-10 should be analyzed in
2 conjunction with Figure 5-9; high fail rates that are not coupled with high emissions reductions
3 indicate problems with the program.
10 12 14 16
18 20
5
6
1
8
9
10
11
Vehicle Age
Figure 5-11. Percent TSIHC Emissions Contributed by Dirtiest 10% of Each Model Year, State
12
13
2 4 6 8 10 12 14 16 18 20
Vehicle Age
Figure 5-12. Distribution of the Dirtiest 10% of Vehicles in the OverallFleet, State 3
DRAFT August 2001
-49-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
0 2 4 6 8 10 12 14 16 18 20
Vehicle Age
Figure 5-13. Percent of Contributed by Dirtiest 10% of Overall Fleet
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
9DD
1RD
1RO
~~ 14D
D) I*HJ
C 120
> 100 -
E
LU Rn
o
1 fin
CO
l~~ AH
9n >
n
c
^
* *+-^
.^-H
S*^*^*-*^^
D 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1
Vehicle Age
6
-82
-» 83
-A- 84
85
QR
187
-a 88
-e 89
-A- 90
-091
-92
QQ
i. 9-1
-95
-x-96
197
Figure 5-14. Emissions Averages at Different Vehicle Ages, Vehicles Tested All Four Years,
3
The same type of plot is repeated again m Figure 5J15. However, this plot includes only vehicles
in the year after they failed an I/M inspection initially and passed on a retest (i.e., if a vehicle
failed and then passed on retest in Year 3, it is included in the Year 4 data here). The sample
sizes are smaller so more scatter is evident in the data, but it is clear that the emissions are
considerably higher than for the fleet as a whole; either these vehicles are poorly maintained and
new problems arise each year, or repairs are not lasting a full year. The initial fail rate for these
vehicles that previously failed and then passed is shown in Figure 5-16. Especially for the
newest vehicles and years three and four of the program, the initial fail rate is significantly
higher than the rate for the entire fleet, shown in Figure 5-10. This type of information might
indicate that the program could achieve greater emissions reductions over the year if the test
interval is shortened for vehicles that failed an earlier test. The collection and analysis of repair
data described in Section 4.3 should be used to determine whether such changes could result in
increased emissions reductions.
DRAFT August 2001
-51-
-------
250
200
Q.
150
w
w
E
U 100
O
H 50
0 1 2 3 4 5
6 7 8 9 10 11 12 13 14 15 16
Vehicle Age
1
2
3
4
5
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
Figure 5-15. Emissions Averages at Different Vehicle Ages, Vehicles That Failed in Previous
Year" TSIHC, State 3
6
7
8
9
10
11
12
13
1 3 5 7 9 11 13 15 17 19 Ave
Vehicle Age
Figure 5-16. Initial Fail Rate, Vehicles That Failed in Previous Year, State 3
Figure 5-17, showing total reductions from initial to final test for the set of vehicles tested in
each of the four years, correlates to Figure 5-9 for the whole fleet. The decreasing reductions
seen in Figure 5-9 are seen again here, so it can be concluded that they were not caused by
immigration or emigration of vehicles. Figures 5-17 and 5-9 are very similar overall.
DRAFT August 2001
-52-
-------
1
2
4
5
6
7
8
9
10
11
12
13
14
15
16
8 10 12 14 16 18 20
Vehicle Age
_
Figure 5-17. Percent TSI HC Reduction for Vehicles Tested All Four Years, State 3
If initial -to-final test emissions reductions are the same fleet year after year, a
question arises as to whether the fleet is simply getting cleaner each year, or whether some of the
gains made in one test cycle are lost by the start of the'next test cycle. The change in emissions
from final test one year to initial test the next year should be investigated as shown in Figure 5-
18. Unlike Figure 5-17, this figure shows the percent increase. Vehicles that initially passed as
well as those that initially failed and were repaired are included in the figure. Figure 5-18 shows
that initial test scores are indeed higher each year than the previous year's final test scores,
indicating that year-to-year vehicle deterioration does provide opportunities for I/M programs to
achieve air quality improvements.
DRAFT August 2001
-53-
-------
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
120
100
1 3 5 7 9 11 13 15 17 19 Ave
Figure 5-18. Increase in Emissions fro:
a Cycle to Initial Test in the Next Cycle,
Comparison of Figure 5-15 and 5-8 indicated that emissions levels are somewhat lower for the
vehicles in Figure 5-15 that participated in the I/M program for four years in a row compared to
vehicles that were not tested in one or more of those years. The higher emissions of vehicles
new to the program are shown in Figure 5-19; the bars indicate the ratio of the TSIHC emissions
of vehicles new to the program to emissions for vehicles tested in at least one previous year. The
new-to-program vehicles exhibit consistently higher emissions for Years 2, 3, and 4 of I/M
program data examined. The lower emissions of the fleet that was tested yearly, as compared to
the emissions of immigrating new vehicles, may indicate that the I/M program is providing a
lasting benefit to the vehicles in the program, outweighing the effects shown in Figure 5-18.
The new-to-program vehicles comprise about 10% of the total tested vehicles for each model
year, which might be a large enough sample to use them as a "No-I/M" fleet. However, since the
origin of the vehicles is unknown (they may have just migrated from an I/M program in another
area), it wouldn't be certain that the new-to-program vehicles would really represent "No-I/M"
vehicles. If the I/M program could determine the location of prior registration for these vehicles,
then only those from non-I/M areas could be used to estimate the emissions of the No-I/M fleet
while operating under the state's local area parameters.
DRAFT August 2001
-54-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
3 5 7 9 11 13 15 17 19 Ave
Figure 5-19. Ratio of Emissions of
:o Returning Vehicles, TSI HC, State 3
5.3.3.1 Recommended Best Practice
The value of the analyses described in this section is that they permit the examination of program
trends without any effect of changes in fleet composition over the years. As seen by comparing
Figure 5-14 to 5-8 the effects of vehicle immigration or emigration on emissions levels can be
examined. Figures 5-15 and 5-16 should be used in conjunction with repair effectiveness
analysis to determine repair durability between tests, and used to indicate if a change in testing
frequency should be made. Figure 5-17 should be used to determine whether emissions
reductions are achieved after several years of testing the same group of vehicles, while analyses
depicted in Figure 5-18 should be used to determine how much of the reductions within a
program year illustrated in Figure 5-17 are negated out by increases between program years.
5.3.4 Comparisons with Other Programs
The following comparisons between different I/M programs are qualitative only, due to the
numerous differences in the programs being compared. Quantitative estimates may be possible,
but would require the programs being compared be much more similar.
In this example, State 1 uses an IM240 test with a fast-pass component and a two-year test cycle;
States 2 and 3 uses a TSI test with a yearly cycle and different cutpoints. Regional factors such
as climate, altitude, and fuel are also different. However, these comparisons can be used to
identify unusual trends that might not otherwise be noticed. For example, earlier Figure 5-8 for
State 3 showed a large jump in TSI HC emissions between 1988 and 1987, when the cutpoints
changed. In Figure 5-20 below, total percent reductions (similar to Figures 5-4 and 5-9) for three
different I/M programs are presented. Total percent reductions are calculated from the change in
average emissions from initial to final test. The data used is comprised of IM240 HC results for
State 1, and TSI HC results for States 2 and 3. From the figure, it can be seen that the emissions
DRAFT August 2001
-55-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
reductions of State 3 excelled for the newest 10 model years, after which the cutpoints were
increased to much higher levels, and the emissions reductions dropped well below the reductions
achieved by the other states. The comparison to other programs, at first, suggests that State 3
might benefit from more stringent cutpoints for the older model year vehicles. However, upon
further reflection, it could also mean that State 3 has achieved more significant reductions from
more durable repairs in past years.
These two very different interpretations of Figure 5-20 demonstrate a key concept in I/M
program evaluation, i.e. an I/M program must be evaluated using many different and
complementary analysis tools to provide a balanced view. For example, to look only at the
emissions reductions achieved during the inspection/repair cycle, but ignore emissions increases
during the rest of the year, may lead to an inaccurate evaluation of the I/M program.
8 10
Vehicle Age
12
14
16
Figure 5-20. Comparison of Percent Reductions
Using the percent emissions reduction as the basis for comparison, as was done in Figure 5-20,
eliminates the effect of the different units used by the TSI and EVI240 tests. Thus comparisons
may be made without having to convert the results of the two different tests to a common basis.
However, the magnitude of the emissions reductions may differ for the different types of tests, so
the information obtained from figures such as Figure 5-20 is only useful for identifying trends.
It should be noted that in general, comparing mass emission reduction estimates between
programs is preferred to comparing percent reductions. Reporting reductions in units of mass
would allow direct comparisons between programs to be made with less misunderstanding. For
instance, an idle program study could report a 15% reduction in CO, while an EVI240 program
could report an 8% CO reduction and one may be led to believe that the idle program was twice
as effective. However, this is not necessarily the case because the CO excess mass emissions for
an idle test could be 25 g/mi, with I/M yielding a 3.75 g/mi reduction, while the EVI240 area
could have a CO excess mass emission of 80 g/mi, that would translate an 8% reduction into 6.4
g/mi.
DRAFT August 2001
-56-
-------
1
2 Appendix A outlines procedures used for predicting IM240 mass emissions from TSI data in
3 State 4. Input parameters for the correlation included TSI test result data, vehicle type, age, and
4 engine size, as well as information about the emissions equipment of the vehicle. The
5 correlation was applied to statewide TSI data. In Figures 5-21 and 5-22, the percent emissions
6 reductions are shown when calculated using measured TSI data as compared to predicted IM240
7 data. It can be seen from the figures that the reductions are smaller when calculated using the
8 IM240 data. Thus, comparison of TSI reductions in one state to IM240 reductions in another
9 state may overstate the relative benefit of the TSI program. This is why it is preferred to report
10 and compare emission reductions on a mass basis.
30
11
12
-B-TSI CO, Low Idle
$K Predicted IM240 CO
0
1981 1983 1985 1987
1989 1991
Model Year
1993 1995 1997 Ave
Figure 5-21. Percent Reduction of Measured TSI CO and Predicted IM240 CO
DRAFT August 2001
-57-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
-B-TSI HC, Low Idle
-*- Predicted IM240 HC
-©Predicted IM240 NOx
1981 1983 1985 1987 1989 1991
Model Year
1993 1995 1997 Ave
Figure 5-22. Percent Reduction of
HC and Predicted IM240 HC and NO,
Figures 5-23 and 5-24 show the initial fail rate in each of the three states, and the average
number of tests from the initial failed test to the final passed test. From Figure 5-23, it is
apparent that State 2 has a high failrate when compared to the other two states. However, Figure
5-24 shows that the average number of tests required to progress from the first failed test to the
final test is comparatively low in State 2. Possible explanations might be that repairs made in
State 2 are not holding between tests and must be repeated each year, or that motorists are
learning to "beat the test" after they have failed once. Whatever the reason, the combination of
information given by Figures 5-23 and 5-24 should be used by a state to highlight areas for
further investigation of an I/M program.
5.3.4.1 Recommended Best Practice
The analyses presented in this section are very qualitative, since differences between the
programs under comparison are not accounted for (i.e., climate, fuel, altitude, test type). A
correlation to convert all tests to an equivalent basis could be used, but without additional
corrections for program differences, results will still be qualitative. The use of a model like
MOBILE would be required to completely bring the results of the three areas to an equivalent
basis. However, Figure 5-20, 5-23, and 5-24 should be used as a tool to identify discrepancies in
emissions reductions trends between different states. Differences in trends may indicate a
weakness in one of the programs that would not appear without comparison to another program.
DRAFT August 2001
-58-
-------
2
3
Figur
6 8 10
Vehicle Age
il Rate
12
14
16
4
5
6
7
6 8 10
Vehicle Age
12
14
Figure 5-24. Average Number of Tests To Pass
16
9 5.3.5 Tracer Vehicles
10 The data analysis methods described in Section 5.3 include useful tools for understanding the
11 effects an I/M program is having on one fleet, and some basic methods for comparing fleet-
DRAFT August 2001
-59-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
average results to different fleets are provided. A different approach for comparing fleets would
be to select "tracer vehicles." Tracer vehicles are make/model/engine combinations chosen
because of their prevalence in most areas. Emissions comparisons from fleet-to-fleet based on
these tracer vehicles would be used to highlight differences between the fleets. Since all tracer
vehicles of a given make/model/engine combination should have had the same emissions levels
when they were new, differences as they age may be attributable to the I/M program. Comparing
the I/M program effects on tracer vehicles instead of on the entire fleet eliminates the effects of
different fleet composition and allows a more direct comparison.
This comparison is made below, using data from States 3 and 5, both of which administer a
yearly IM240 test. The EVI240 HC emissions distributions of three late model year
make/model/engine combinations from each state are presented in Figure 5-25. Model year 1994
vehicles are used, since that is the newest model year that is fully represented in both of the state
data sets. The three make/model/engine combinations, which are the same for both states, were
chosen as the three that are most heavily represented in both of the fleets. These distributions are
intended to represent the emissions of vehicles in the two states when they are new. Similarly,
IM240 HC emissions distributions for three 1984 make/model/engine combinations for both
states are shown in Figure 5-26. These represent vehicles that have been affected by the I/M
program as they have been operated within the state over many years. The make/model/engine
combinations and sample sizes are listed in Table 5-3. The sample sizes are reasonably large,
ranging from 400 to 1500 vehicles per combination.
Figure 5-25 does appear to show some differentiation between the two fleets. The curve for
vehicles of Combination 4, State 3 is shifted to the right of the curve for Combination 4 vehicles
from State 5; the same is true for Combinations 5 and 6. This may indicate that regional effects
such as altitude, fuel, and climate are causing the emissions distributions of the two states to
differ, since these vehicles are nearly new and should not be affected by deterioration. However,
the emissions levels for these new vehicles are very low, and in State 3 a sharp peak is seen
instead of a smooth distribution. Since the values are so clustered, it seems possible that this
peak is located at some minimum measurable concentration. Both State 3 and State 5 allow a
fast-pass and then use the results to project full test scores and many of the newest model year
vehicles achieve a fast-pass at the earliest allowable second of the test, making it difficult to
accurately project full-test emissions. Difficulties associated with projecting full test scores from
fast-pass results were mentioned earlier in Section 5.3.1. As a result, it is not entirely clear
whether the difference between the two distributions is real or is an artifact of data collection and
processing methodology. The emissions distributions for the 1984 vehicles in Figure 5-26 do not
show this effect. The distribution traces for each combination are now very similar for either
state. If the new-vehicle differences shown in Figure 5-25 did represent real emissions
differences between the two areas, the lack of difference in Figure 5-26 would indicate that
greater deterioration is occurring in State 5 than in State 3, since the State 5 emissions
distributions are lower when the vehicles are new but not lower after the vehicles have aged.
Table 5-3. Make/Model/Engine Combinations for States 1 and 5
Combination
1
Model
Year
1984
Make
Chev.
Model
Cavalier
Engine
Displacement
[Ll
2.0
Count,
State 1
556
Count,
State 5
1339
DRAFT August 2001
-60-
-------
2
3
4
5
6
1984
1984
1994
1994
1994
Chev.
Ford
Ford
Honda
Toyota
Celebrity
Tempo
Escort
Accord
Corolla
2.8
2.3
1.9
2.2
1.6
422
435
722
1220
624
745
615
1843
685
1490
2
3
4
5
6
^nn
Acri
Nu 400 _
mb
__ -jc;n -
er oou
of onn
Ve
. . ocn
hie ^ou
les pnn -
in
_. -| en
Bi IOU
n inn -
cn _
o -
0.0
A l\
/ V
/ » \
/ \
State 5, 1994, Combo 4
State
State
State
- - - -State
- - - -State
/.. -^p^' «
1 ^^^^^^ 1
001 0.001 0.01 0.1
IM240 HC [g/mi]
; 1, 1994, Combo 4
s5, 1994, ComboS
; 1, 1994, ComboS
; 5, 1994, Combo 6
; 1, 1994, Combo 6
1 10
Figure 5-25. IM240 HC Emissions Distributions for 1994 Vehicle Combinations
States, 1984, Combo 1
State 1, 1984, Combo 1
States, 1984, Combo 2
State 1, 1984, Combo 2
- - - -State 5, 1984, ComboS
- - - -State 1, 1984, ComboS
0.001 0.01 0.1 1
IM240 HC [g/mi]
10
100
Figure 5-26. IM240 HC Emissions Distributions for 1984 Vehicles Combinations
DRAFT August 2001 - 61 -
-------
2 5.3.5.1 Recommended Best Practices
3 The concept of tracer vehicles could be a valuable tool for benchmarking the results of one
4 program against another, assuming states are willing to coordinate their efforts and support this
5 concept. The effectiveness of I/M programs from two different areas may be compared using the
6 emissions distributions of the tracer vehicles, without the need for correcting for regional
7 differences (altitude, fuel, etc). However, fast-pass/fast-fail options seem to obscure the results.
8 Additional work will be needed to determine the value of this type of analysis.
9
10
11 5.4 Evaporative Emission Reductions
12 This section outlines recent data from EPA and CRC studies to develop a first order estimate for
13 the possible emission reductions from evaporative emissions for vehicles identified and repaired
14 for evaporative emission control problems. Very small numbers of vehicles are included in these
15 studies and clearly more data is needed to more accurately quantify the possible emission
16 reductions from gas cap and pressure test results. A detailed discussion about the methodology
17 used in the EPA and CRC studieaisivailable elsewhere23. U
18
19 5.4.1 Estimate of Single Vehicle Gas Cap I/M Benefit
20 The first study (Reference XXX), conducted in 1997/1998 by Automotive Testing Labs, was
21 performed under an EPA contract on vehicles recruited from the Arizona I/M Program. Vehicles
22 were tested with the following conditions:
23 ^4 «
24 FuelRVPof6.3psi.
25 38 hour 72-96°F diurnal.
26 1 hour hot soak at 95°F.
27 3x LA4 Running loss at 95°F.
29 The volatility of the fuel is described by Reid Vapor Pressure (RVP) with units of pounds per
30 square inch. Diurnal emissions were measured with a 38-hour ambient temperature profile made
31 up of a 72°F to 96°F increase, a 96°F to 72°F decrease, and another 72°F to 96°F increase. The
32 specific temperatures are taken from the EPA 72-hour enhanced diurnal profile used for diurnal
33 evaporative emissions testing. The hot soak emissions were measured for one hour with an
34 ambient temperature of 95°F following an FTP driven at 95°F. Running losses were measured
35 while driving three consecutive LA4 cycles. An LA4 cycle is the 1372-second cycle used for the
36 first two bags (cold start + warm stabilized) of the FTP.
37
38 These conditions were considered appropriate for Arizona conditions in 1997/1998 because they
39 were thought to be representative of in-use evaporative emissions generation. They are different
40 from new vehicle certification test conditions, which are designed to be severe test conditions
41 under which new vehicle emission control hardware and purge strategies must control emissions.
42 Data from the EPA study includes the before-repair and after-repair evaporative emissions of the
43 26 vehicles tested. The estimated total evaporative emissions reduction was calculated using the
44 24-hour diurnal, hot soak, and running loss measurements before and after repair and using
45 assumptions of 3 hot soaks per day and 30 miles traveled per day for each test vehicle.
46 Evaporative emissions reductions for pressure, purge, or fuel cap based repairs are assigned for
47 all the vehicles considered in the study.
DRAFT August 2001 - 62 -
-------
1
2
3
4
5
6
7
8
9
10
Table 5-4 presents a summary of the emission reductions associated with the following
categories:
Pressure system repair;
Purge system repair; and
Gas cap repair.
Table 5-4. Summary of EPA WAI-8
Carbureted
Vehicles
Fuel
Injected
Vehicles
All Vehicles
Evaporative Emission Reductions (g/mile)
Running Losses Only
Repairing
Pressure
Problems
0.88(1)*
1.83(10)
1.75(11)
Repairing
Purge
Problems
2.50(1)
Repairing
^ ^Gas Cap
Problems
J).56'(3)'
is iFfsy
H
2.50(1) 3.07(12)
All Evap Emissions (Hot Soak, Diurnal,
and Running Losses)
i-Jlepairing
Problems
1.01(1)
2.47 (10)
2.34(11)
Repairing
Purge
Problems
2.50(1)
2.50(1)
Repairing
Gas Cap
Problems
1.07(3)
4.37 (9)
3.55 (12)
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
*Numbers in parenthesis denote vehicle sample size.
i Tiirh/nWrtflVAA
The table shows the emission reductions for carbureted and fuel-injected vehicles. In addition,
running loss and total evaporative emissions reductions are shown. For this analysis the gas cap
emission benefits are most important because State 1 uses a gas cap test to identify gas cap
failures. The sample size for the data represented in Table 5-4 is small. EPA, California ARB,
California BAR, and CRC all plan to conduct more SFffiD tests to quantify the change in
evaporative emissions due to evaporative system repair. However, SFffiD tests are expensive
and time consuming (compared to EVI240 tests), so not as many tests are performed. The data in
Table 5-4 is a best-estimate of evaporative emissions given the available data at this time.
The scatter of before- and after-repair evaporative emissions can provide additional insight
beyond simply comparing means. Figures 5-27 and 5-28 show plots of the total evaporative
emissions after repair versus before repair. The plots use different symbols for repair type and
fuel metering system type, respectively. Logarithmic scales are used so that the data scatter can
be seen more clearly. Figure 5-27 indicates that the scatter of data points for pressure fails and
fuel cap fails is about the same. On the other hand, Figure 5-28 indicates that different but
overlapping regions may characterize different fuel metering types. The 1:1 line in the figures is
drawn to assist the readers in interpreting the data. Data points below the line represent vehicles
whose emissions prior to repair were higher than emissions after repair. The magnitude of the
distance of the points away from the 1:1 line denotes the amount of emission reduction caused by
evaporative system repair.
DRAFT August 2001
-63-
-------
10.00 n
CD
I
D)
^ 1.00-
+
CO
i
+
Q
| 0.10-
CD
1
n n-i
/
X
f
)
t
i;
X
A
t)J
X
i
A4
(
^r
k
A
^
)
^
^
^
|
0.10 1.00 10.00
Before-Repair DI+HS+RL (g/mile)
100.00
A Pressure Fails o Cap Fails
1:1
Figure 5-27. Effect of Repair Type on Evaporative Emissions System Repairs
= 10.00 -,
E
O)
_i
₯ 1.00-
Q
.!= 0.10-
CO
Q.
CD
o:
CD n m
^
x^
x
y
x
^
^
^
-------
1 In the CRC study, only running losses were considered in the evaporative emissions estimate.
2 The fuel cap related failure was diagnosed but post-repair emissions are not available. A
3 comparison of the measured before-repair running loss emissions from the CRC and EPA studies
4 is shown in Figure 5-29. A logarithmic scale was used to help see the data scatter more clearly.
5 The figure shows that pre-repair running losses were about 7 times higher in the EPA study, but
6 the amount of scatter was comparable in the two studies. Higher running loss emissions can be
7 expected from the EPA study, which used 3 LA-4s for its test cycle, in comparison with the CRC
8 study, which used 1 LA-4.
10.00
&)
or
1.00
0.10
0.01
I
CRC
EPA
9 Figure 5-29. Comparison of Before-Repair Running Losses from EPA and CRC Studies
10
11 To calculate the emission reductions from the CRC data, it was assumed that the average post-
12 repair running loss emissions for the CRC test vehicles is the same as that for the EPA test
13 vehicles, i.e. it was assumed the CRC vehicles could be repaired to the same levels as the
14 repaired vehicles in the EPA study. This is shown in Table 5-5. Average post-repair running
15 loss emissions were calculated from the EPA data described in Table 5-4. Finally, the estimated
16 running loss emission reductions associated with gas cap repair is calculated by difference for the
17 CRC sample in Table 5-5. Because the CRC test fleet and the EPA test fleet are not the same,
18 the subtraction of the EPA post-repair average from the CRC pre-repair average provides large
19 uncertainty. However, failure to subtract some estimate of post-repair emission values would
20 surely over-estimate the size of emissions reductions due to gas cap repairs.
21
22 Table 5-6 presents calculations to estimate the total evaporative emissions reductions for gas cap
23 repair. The results of both studies are combined to arrive at estimated reductions. The top of
24 Table 5-6 presents the emission reduction estimates from the two studies. The table shows the
25 measured running loss reductions from the EPA study, the estimated running loss reductions
26 from the CRC study, and the total evaporative emissions reductions from the EPA study. Total
27 evaporative emission reductions are not available from the CRC study. The average running loss
28 reductions for carbureted and fuel-injected vehicles are calculated by averaging the average
29 running loss reductions for the CRC and EPA studies with the number of vehicles (in
30 parentheses) as weighting factors. Then, total evaporative emissions (for hot soak, diurnal, and
31 running losses) are calculated by using the weighted running loss estimates instead of only the
32 EPA estimates. As shown, the best estimate of possible evaporative emissions reductions for
33 vehicles that fail the gas cap test are 1.00 g/mile for carbureted emissions and 3.25 g/mile for
DRAFT August 2001
-65-
-------
1
2
3
4
5
6
7
8
9
10
11
12
13
fuel-injected vehicles. It would be beneficial to provide error bars on these estimates but due to
the small sample it is difficult to quantify these with any degree of confidence. EPA and
California are conducting additional studies to improve these estimates.
It is readily recognized that the sample sizes used to arrive at these estimated evaporative
emissions are small; however, these are the only measurements available to states to make these
estimates. Accordingly, the uncertainties of these estimates are large.
Table 5-5. Estimate of Running Loss Emission Reductions from Gas Cap Repairs in CRC Study
(g/mile)
Carbureted Vehicles
CRC Pre-Repair Running Loss
Average
0.538 (25)
EPA Post-Repair Runningtoss
Average
Estimated CRC Running Los
Reduction
=-
Fuel-Injected Vehicles
0.334 (4)
0.088 (20)
0.25
*Numbers in parentheses denote vehicle|^m|lf size.4
P
Table 5-6. Total Evaporative Ernj_s|ion Reduction Calculation for Gas Cap Repairs
Carbureted
Vehicles
Fuel -Injected
Vehicles
Running Loss Reductions for Gas
Cap Repair
EPA
0.56(3)
3.90(9)
CRC LA,
0.48 (25) "
1
Total (RL + DI + HS) Evap
Reductions for Gas Cap Repair
EPA
1.07
4.37
CRC
-
-
14
15
16
17
18
19
20
21
22
23
24
25
*Numbers in parenthesis denote vehicle sampTtrsr
Carbureted Vehicles
Fuel-Injected Vehicles
Weighted Running Loss Reductions
0.49 g/mile
2.78 g/mile
Carbureted Vehicles
Fuel-Injected Vehicles
Estimated Total Evaporative Reductions
1.07 - 0.56 + 0.49 = 1.00 g/mile
4.37 - 3.90 + 2.78 = 3.25 g/mile
5.4.2 Fleet I/M Evaporative Benefit
In the last section, two studies were discussed to estimate the evaporative emissions benefit
associated with the repair following a gas cap test failure. Before this data is used to project fleet
benefits, several issues need to be discussed. These include the following:
Emissions Deterioration
Repair Effectiveness
Collateral Defects
DRAFT August 2001
-66-
-------
1 Evaporative Emissions Control Technology/OBD.
2
3
4 Emissions Deterioration: The previous section estimates the emissions reduction that are
5 achieved immediately after repair. As vehicles go back into their normal usage following this
6 repair, the emissions can creep up as the emissions control system degrades. Emissions can also
7 increase if the fuel cap is not tightened or replaced following refueling. The frequency of
8 occurrences of these events is not fully known at this time.
9
10 Repair Effectiveness: In the real world not all identified defects get repaired. Based on
11 conversations with state I/M staff, it is assumed that 90% of the emissions reductions estimated
12 for the roadside fleet associated with gas cap repair will actually be realized by the I/M program;
13 however, this estimate is not based on any observed data. As more VID and roadside data is
14 collected this assumption will be re-considered.
15
16 Collateral Defects: Vehicles which have a gas cap defect can also have other evaporative
17 emissions control problems. In a small sample of roadside data in which 1992 and older vehicles
18 were considered, 62.6% of the vehicles that failed the gas cap test also failed the fuel evaporative
19 pressure test. Since the pressure test is conducted after removing the gas cap, this implies that
20 these vehicles had other pressure leaks in addition to a gas cap defect. It is possible that some of
21 these vehicles, which have gas cap and pressure defects, would benefit from a gas cap repair.
22 For this analysis, State 1 assumed that 70% of the possible emissions reduction from gas cap
23 repair will be achievable. This implies that 30% of the emissions reduction will be negated due
24 to other evaporative emissions problems with the vehicl,%
25
26 Evaporative Emissions Control Technology/OBD: Newer vehicles have more robust
27 evaporative control systems and have fewer defects. In addition, 1996 and newer vehicles with
28 OBDII system checks set an engine malfunction indicator light (MIL) if the evaporative control
29 system fails the on-board test. The evaporative system monitors were optional/experimental on
30 1996-1997 Federal vehicles; monitors were required on at least 20% of 1996 model year vehicles
31 and on at least 40% of 1997 model year vehicles. It is expected that future gas cap benefits may
32 be reduced as more vehicles with OBD systems penetrate the fleet. New issues with OBD
33 systems may occur over time but this issue will need to be studied as OBD equipped vehicles
34 age. In this analysis, no gas cap emissions benefit is assumed for 1996 and newer vehicles.
35
36 Fleet Emissions Reduction Calculation: Figure 5-30 shows the gas cap failure rates observed
37 in State 1 roadside data. Vehicles which had undergone an inspection were observed to have a
38 lower fail rate than vehicles which were tested prior to their inspection. The results of the gas
39 cap repair benefit from Table 5-6 and fail rates from Figure 5-30 were used to estimate the fleet
40 emissions benefit. This calculation is shown in Table 5-7. The table shows the evaporative
41 emission calculations for each model year. The fail rates shown in Figure 5-30 and repeated in
42 this table are calculated from the roadside data. The percent of fuel injected vehicles are taken
43 from EPA estimates. The evaporative benefit for carbureted and fuel injected vehicles is
44 calculated as follows:
45 Evaporative emissions benefit for carbureted vehicles in any model year =
46
47 (l-FI)*(FRB-FRA)*EVcaib
48 Where:
DRAFT August 2001 - 67 -
-------
1
2
3
4
5
FI
FRB
EV
Fraction of fuel injected vehicles in the model year
Failure rate for roadside vehicles before and after Smog Check
Evaporative benefit for carbureted vehicles estimated in Table 3.
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
70%
60%
-. 50%
40%
OS
LL.
a.
OS
o
(0
OS
o
30%
20%
10%
0%
Q-ti
1970
1975
1980 1985 1990
Model Year
1995
2000
2005
-Before Smog Check - - -D- - - After Smog Check
Figure 5-30. Gas Cap Failure Rat
Rates from California Roadside Data
The emissions benefit associated with fuel injected vehicles is calculated in a similar fashion
using the emissions estimates for fuel injected vehicles from Table 5-6. The total model year
evaporative emissions estimate is then calculated and multiplied by the travel fraction to estimate
the weighted model year emissions benefit. The total calendar year 1999 estimate is then
calculated by summing the emissions benefit for all the model years. This is calculated to be
0.076 g/mile in Table 5-7. The net evaporative emission estimate is then calculated by using the
assumptions discussed above that 90% of the emissions associated with vehicles with gas cap
defects are actually repaired and that 30% of the emissions benefit is negated due to collateral
defects. The net evaporative estimate was hence calculated to be 0.048 g/mile.
Other states that do not conduct roadside tests could use gas cap fail rates in their states and
compare them to a No-I/M area gas cap fail rates. Data for No-I/M areas would have to be
developed by EPA and other stakeholders in order for this to be viable.
DRAFT August 2001
-68-
-------
1 Table 5-7. Calculation Summary for Estimating California Fleet Evaporative Emissions Benefit
2 for Gas Cap Repairs
Model
Year
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
Travel
Fraction*
0.00326
0.00275
0.00428
0.00632
0.00816
0.00979
0.00857
0.01000
0.01275
0.01622
0.02713
0.03274
0.03907
0.04284
0.04621
0.05222
0.04967
0.05314
0.04733
0.05763
0.06222
0.07303
0.06497
0.08405
0.10832
0.07732
Gas Cap Fail
Before
44.44%
64.71%
37.04%
41.27%
32.00%
33.94%
26.17%
29.57%
22.07%
15.76%
14.33%
14.63%
12.50%
11.46%
9.40%
5.38%
7.30%
5.25%
1.80%
1.63%
3.62%
1.42%
4.17%
0.00%
0.00%
0.00%
After
14.29%
14.29%
33.33%
9.38%
9.38%
6.82%
8.33%
4.08%
0.00%
4.30%
5.16%
6.06%
5.00%
3.37%
5.26%
4.07%
4.63%
1.91%
1.80%
0.00%
1.87%
0.00%
4.17%
0.00%
0.00%
0.00%
% of Fuel-
Injected
Vehicles
0.000%
0.000%
0.000%
0.000%
0.000%
0.000%
0.000%
9.000%
16.800%
27.100%
39.200%
51.500%
67.600%
74.100%
89.900%
87.200%
98.100%
99.800%
99.800%
100.00(7%
loo.oofp
100.000%
100.000%
100.000%
100.000%
100.000%
Evap Benefits
for Carbureted
Vehicles
(g/mile)
0.30159
0.50420
0.03704
0.31895
0.22625
0.27127
0.17835
0.23190
0.18361
0.08354
0.05575
0.04158
^"^2T0.02430
0.02094
^ TF1).00418
W |ka.00167
P. UJ).00051
0.00007
_0.00000
L "'Jt-00000
0.00000
0.00000
0.00000
0.00000
0.00000
0.00000
Evap Benefits
for Fuel-Injected
Vehicles
(g/mile)
0.00000
0.00000
0.00000
0.00000
0.00000
0.00000
0.00000
0.07454
0.12050
»> 0.10093
0.11682
0.14350
' ! 0.16478
0.19469
0.12092
0.03708
0.08513
0.10825
0.00000
0.05285
0.05701
0.04599
0.00000
0.00000
p o.ooooo
0.00000
i/JiAU. Evap benefit g/mile =
Discount for repairs =
Discount for Collateral Defects =
Total Model
Year Evap
Benefits
(g/mile)
0.30159
0.50420
0.03704
0.31895
0.22625
0.27127
0.17835
0.30644
0.30411
0.18447
0.17258
0.18508
0.18908
0.21563
0.12510
0.03875
0.08564
0.10832
0.00000
0.05285
0.05701
0.04599
0.00000
0.00000
0.00000
0.00000
90.0%
30.0%
Weighted
Evaporative
Emissions Benefit
0.00098
0.00139
0.00016
0.00202
0.00185
0.00266
0.00153
0.00306
0.00388
0.00299
0.00468
0.00606
0.00739
0.00924
0.00578
0.00202
0.00425
0.00576
0.00000
0.00305
0.00355
0.00336
0.00000
0.00000
0.00000
0.00000
0.07564
0.06808
0.04766
4
5
6
7
8
9
10
11
12
13
14
15
16
17
*From California BAR May 1999 Travel Fraction Calculator.
5.4.3 Other Evaporative Control Measures
In addition to pressure tests and gas cap tests, recent CRC and EPA studies have also led
researchers to identify and repair liquid leaking vehicles (Reference XXX). The CRC study,
CRC-E35, has pointed towards the existence of a very small fraction of vehicles which can be
designated as liquid leakers. Drops of fuel are seen to be leaking from these vehicles.
Experienced mechanics can usually identify these vehicles due to the strong gasoline smell
emanating from these vehicles. California BAR is developing a testing protocol to identify,
repair, and quantify the emission reductions possible from repairing such vehicles. EPA's
MOBILE6 model also includes these vehicles in the fleet and includes estimates of both the
frequency and the evaporative emissions estimates of these vehicles. However, procedures for
quantifying the emission benefits realized by identifying and repairing these vehicles must still
be developed.
DRAFT August 2001
-69-
-------
1 6. Summary
2 A number of methods for estimating I/M program effectiveness using in-program data were
3 outlined in this guidance. Effort was made to document, reference or provide examples for data
4 collection procedures, QA/QC protocols, analysis methods, and sources of error or possible bias
5 associated with a given method; however, it is recognized that improvements to the methods
6 outlined in this document will continue to evolve. Therefore, it is strongly recommended that
7 any state considering the use of in-program data for program evaluation purposes work closely
8 with their respective regional EPA office and the Office of Transportation and Air Quality to
9 ensure the most up-to-date practices are incorporated into the evaluation. Furthermore, states
10 interested in using in-program data for program evaluation must recognize the need within their
11 own agencies to develop a minimum level of expertise with the technology and procedures to
12 ensure reliable data are collected and analyses performed.
13
14 It should also be recognized, given the difficulties associated with I/M program evaluations, that
15 an evaluation based on both out-of-program data (e.g. RSD or roadside pullovers) and in-
16 program data, will provide a more accurate estimate of overall program performance than simply
17 relying on one method alone. J
18
19
20
21 7. References
1 IM240 & Evap Technical Guidance, April 2000, EPA420-R-00-0007
available on-line at www.epa.gov/oms/im.htfti "i
2 ASM Technical Guidance DRAFT, July
available on-line atwww.epa.gov/oms/im.htm
3 Clean Air Act, 1970
Clean Air Act Amendments, 1977
5 EPA Guidance, 1978
Clean Air Act Amendments, 1990
7 IM Rule, November 5, 1992
National Highway Systems Designation Act, 1995
9 EPA Rule, January, 1998
10 EPA Memo, October 30, 1998
Klausmeier, Evaluation of Test Data Collected in 1999 from Connecticut's I/M Program,
"DRAFT", July 2001.
New York State Enhanced I/M Program Evaluation Report for the Period of 11/16/98 to
12/31/98, New York Department of Environmental Conservation, January 2001.
13 T.H. DeFries, C.F. Palacios, and S. Kishan, "Models for Estimating California Fleet FTP
Emissions from ASM Measurements," December 25, 1999, Eastern Research Group, Inc.,
Austin, Texas, ERG report BAR-991225
14 T.H. DeFries and D.A. Westenbarger, "Evaluation of I/M Programs and Modeling
Techniques to Predict Fleet Average FTP and EVI240 Emission Rates," presented at Tenth
CRC On-Road Vehicle Emissions Workshop, March 27-29, 2000, Coordinating Research
Council, Atlanta, Georgia.
15 ECOS/STAPPA/EPA Inspection and Maintenance Workgroup, Final Meeting Summary for
March 25, 1998, Workgroup Meeting and Attached Background Materials.
DRAFT August 2001 - 70 -
-------
T.C. Austin, L.S. Caretto, T.R. Carlson, and P.L. Heirigs, "Development of a Proposed
Procedure for Determining the Equivalency of Alternative Inspection and Maintenance
Programs," prepared for U.S. EPA, Regional and State Programs Division, by Sierra
Research, Inc., Sacramento, California, November 10, 1997.
T.H. DeFries, A.D. Burnette, S. Kishan, and Y. Meng, "Evaluation of the Texas Motorist's
Choice Program," prepared for the Texas Natural Resource Conservation Commission, by
the Eastern Research Group, Inc., Austin, Texas, May 31, 2000, ERG report TNRCC-
000531.
S. Kishan, C.F. Palacios, "Comparison of California's I/M Program with the Benchmark
Program," prepared for California Bureau of Automotive Repair, by the Eastern Research
Group, Inc., January 19, 2000.
A.W. Ando, W. Harrington, and V. McConnell, "Estimating Full IM240 Emissions from
Partial Test Results: Evidence from Arizona", J. Air & Waste Management Association
(49), 1153, 1999.
T. Wenzel, "Analysis of LI NL'S S^crt Test A dp^ir et i Factors," Lawrence Berkeley
National Laboratory, Apri: ?H
P. McClintock, Environmental Analy sit private communication with J. Lindner, US EPA,
2000.
P. McClintock, Environmental Anabs s private communication with J. Lindner, US EPA,
2000.
"Evaporative Emissions Impact of
September 2000.
report by California BAR and ERG,
DRAFT August 2001
-71-
-------
Appendix A: Development of a Model to Predict IM240 Emissions Concentrations
from Two-Speed Idle Data
Although the performance standard for an I/M Program is the IM240 test, many states
choose to administer different types of tests such as the two-speed idle test (TSI) or the
ASM test. Sierra Research proposed that if results of alternative types of tests are to be
compared to baseline program results or results from other states, models must be built to
predict IM240 emission rates from the measured alternative test emissions
concentrations. This appendix contains an outline of the specific procedures to develop
such a correlation, based on work done by Eastern Research Group for the Texas Natural
Resource Conservation Commission.1 Since Texas uses a two-speed idle test,
development of a correlation between IM240 measurements and a different type of test
will differ in the details of the fundamental procedures outlined below.
The correlation of TSI and EVI240 results is based on emissions data from a sample of
Texas vehicles that received both the TSI and IM240 tests. Procedures for selecting a
suitable vehicle sample, developing a correlation model, testing the model for bias,
quantifying uncertainty, and the limitations of applying the model to the fleet will be
described.
P B
A. 1 Data Collection
Two types of data were acquired for the developmenTof the dataset on which models can
be developed: two-speed idle (TSI) daja, and EVI240 dynamometer data. The two-speed
idle measurements were made at two I/M inspection stations according to normal station
procedures. Since valid data is critical for successful model development, the TSI
instruments were calibrated and zeroed as usual, and then independently checked at the
beginning of each workday with zero and span audit gases separate from the I/M station's
normal supply. EVI240 tests were performed using a portable dynamometer located just
outside the I/M station. All equipment was calibrated and operated according to EPA
specifications.
Selection of vehicles to participate in the test program was based on a stratified random
sampling scheme using model year group, TSI test results, and vehicle type.
Stratification is used to prevent selection of predominantly new, relatively low-emitting
vehicles. While a stratified random sample does not represent the vehicle distribution in
the fleet, it does provide a model-building dataset containing the full range of emissions
levels.
Model year groups are used as a stratification category instead of individual model years
to reduce the number of stratification levels. For the TNRCC work, four model year
groups were used: 1981 to 1984, 1985 to 1988, 1989 to 1992, and 1993 to 1997. Also,
for each of the four TSI measures (high-speed idle HC and CO and low-speed idle HC
and CO), bins were created based on these model year groups and TSI concentration
groups. Historical Texas TSI data were used to define TSI concentration groups so that
each represented approximately a quintile of the TSI distribution for each model year
group. The goal of vehicle selection was to achieve an equal number of vehicles in each
model year group/TSI concentration group bin for each type of TSI. In addition, the
vehicles in each bin were targeted to be 64% passenger cars and 36% light-duty trucks
DRAFT August 2001 -72-
-------
(trucks, vans, MPVs, SUVs) as in the Texas fleet. When the TSI results, model year, and
vehicle type of a vehicle at the I/M station indicated that the vehicle would be a suitable
candidate for the stratified sample, the vehicle owner was offered an incentive in
exchange for allowing the vehicle to be receive an IM240 test following the TSI test. A
smaller stratified sample of repeat two-speed idle measurements was also collected to
cover the range of HC and CO low-speed idle and high-speed idle. Additional incentive
was offered to the vehicle owner for allowing a second EVI240 and TSI test to be
performed.
TSI measurements were performed using the I/M station's BAR90 analyzers. All TSI
and I/M CO and CO2 measurements were determined by non-dispersive infrared (NDIR).
IM240 NOX was determined by chemiluminescence. In the case of hydrocarbons, the
IM240 hydrocarbon was measured by flame ionization detector (FID) and the TSI
hydrocarbon was measured by NDIR. Major differences in response factors to different
types of hydrocarbon compounds are known to exist between FID and NDIR. Therefore,
proper application of the models that were developed requires that TSI hydrocarbon be
measured by NDIR.
^L A^L °°^L ^^L ^^^n^ff^L^^nfc^ti ^^E^ «n^^r^FV
The overall goal for the TSI/IM240 data set was to acquire test pairs of test results for
800 vehicles, divided among the four model year groups, five emissions level quintiles,
and two vehicle types.
A.2 Model Development
The steps involved in developing the models for TNRCC were:
General quality assurance of the raw data including review of the TSI
analyzer calibration and jjasjuidit results;
Data preparation consisting of humidity corrections for EVI240 NOX
values, correction of TSI values for vehicle exhaust system dilution,
removal of suspect observations from the dataset, and special handling for
low TSI values;
Investigation of transformations of the variables to be used in the models
to make the variance across the range of values homogeneous;
Various types of variable screening techniques to determine variables
which could be expected to be important to the prediction of EVI240 values
and to discover any major curvature that might be present;
Variable screening through the use of model building using ordinary least
squares modeling techniques. With ordinary least squares modeling, the
independent variables are assumed to have no measurement error;
Estimation of the error variances of EVI240 measurements and the error
variances and covariances of TSI measurements; and
DRAFT August 2001 -73-
-------
Using the independent variables which produced the best ordinary least
squares models, to develop the final models using the measurement error
model building technique. In this technique, the error variances and
covariances of the TSI measurements and the error variances of the EVI240
measurements were used to build models which are less biased than the
ordinary least squares models.
Each of these different steps in the modeling approach is discussed below.
Data Preparation
After an exhaustive quality assurance check was performed on the TSI and EVI240 data,
the TSI data was corrected for dilution. EVI240 data does not require a dilution
correction, although the NOX values are corrected for ambient humidity when collected.
Adjustment of Low Two-Speed Idle Values
The presence of negative two-speed idle values is known to exist in the Texas "VTD
system. Therefore, during field data collection in this project, we were aware that
negative values might occur. Negative values can be expected in any instrumental
measurement. Even though negative concentration values make no physical sense, it is
important to remember that the output of Jhstruments is simply a voltage or current which
can have negative values. Thus, a smalljgiyor in zeroing the instrument can produce
negative values in the dataset. During model building, negative and zero values need to
be handled appropriately to arrive at ajspdei which is unbiased on the low concentration
end.
In the dataset collected in this study, no negative two-speed idle values were obtained.
The smallest non-zero values reported by the TSI analyzers were 1 ppm HC and 0.01%
CO. Many zero values (0 ppm HC and 0.00% CO) for two-speed idle concentrations
were measured (11 low-speed idle HC zeroes, 246 low-speed idle CO zeroes, 55 high-
speed idle HC zeroes, and 324 high-speed idle CO zeroes) for the modeling dataset. For
model building purposes, zero two-speed idle HC values were set to 1 ppm, and zero
two-speed idle CO values were set to 0.01%. These changes are well within the
measurement error of the TSI method and instruments. The changes are necessary to
allow logarithmic transformations of TSI values for model building purposes.
Negative and zero IM240 values were not reported on the test vehicles.
Selection of Appropriate Variable Transformations
Plots of IM240 emission rates versus dilution-corrected TSI concentrations indicate that
the values of both variables are highly positively skewed and the variance of any
relationship between the two variables is inhomogeneous. Inhomogeneous variance
means that the scatter at high emissions levels is much different than the scatter at low
emissions levels. This difference in scatter can be seen in the sample plot in Figure A-l
for IM240 CO versus high-speed idle CO in linear space. The figure shows much larger
scatter at high emissions than at low emissions. Another serious problem with building
variables in linear space for this data is a result of the "kite and string" nature of the data.
Because of the highly skewed distribution for the dependent and independent variables,
as is seen in Figure A-l, any regression line will be anchored near the origin by the large
DRAFT August 2001
-------
number of data points there. Then, the presence or absence of the few high values on the
upper right portion of the plot will influence the position of the regression line far out of
proportion to their abundance in the data set.
DRAFT August 2001
-75-
-------
Figure A-1. IM240 CO versus High-Speed Idle CO
4UUH
S3
Q
6 7
(%)
/dipstick/data/tnrcc/mett/TX2 In.sas
10
el
Other transformations were sought to help correct these problems. The natural logarithm
of both the IM240 emission rates and the, TSJ concentrations was chosen. Figure A-2
shows the scatter plot in log-log space for IM240 CO versus high-speed idle CO. The
plot shows that the data for both variables in log space is not highly skewed and that the
variance (the scatter of points) is nearly homogeneous across the range of the variables.
The log-log plots for all combinations qfJM240 emission rates and TSI emission
concentrations were also examined.
DRAFT August 2001
-76-
-------
Figure A-2. Comparison of IM240 CO and High-Speed Idle CO
High-Speed Idle CO (%)
/dipstick/data/tnrcc/mett/TX2 In.sas
Investigate Independent Variables
As the first step in model building, correlation coefficients were calculated and plots were
made to investigate the relationships among the different variables in the dataset. The R2
values were tabulated and the strongest relationships noted. The R2 between IM240
emission rates and different variables which are candidates for predictors were also
calculated.
Statistical Variable Selection Using Conventional Regression
The second step in the selection of variables to be-ctseJ-to predict IM240 emission rates is
the development of ordinary least squares regression models. Unlike correlation
coefficients and scatter plots that can only consider the influence of one independent
variable at a time on the IM240 emission rate, multiple linear regression can consider the
influences of many variables at the same time on IM240 emission rates.
In the process of performing ordinary least squares regression, dozens of models were
created and evaluated in an effort to find the best model for predicting EVI240 emission
rates. The PROC REG procedure in SAS was used with the stepwise option to select
input variables from the TSI measurements and vehicle characteristic descriptors. Main
effects, two-factor interactions, and squared effects of the following variables were
considered for inclusion as terms in the models:
High-Speed Idle HC (ppm)
High-Speed Idle CO (%)
Low-Speed Idle HC (ppm)
Low-Speed Idle CO (%)
Engine Displacement (L)
Age (year)
DRAFT August 2001
-77-
-------
Truck/Car Indicator (+0.5, -0.5)
Carbureted/Fuel-Injected Indicator (+0.5, -0.5)
Oxy-Catalyst Indicator (+0.5, -0.5)
Three-Way Catalyst Indicator (+0.5, -0.5)
Exhaust Gas Recirculation Indicator (+0.5, -0.5)
Air Injection Reactor Indicator (+0.5, -0.5)
Only terms which had coefficients that were significant at the 99.9% confidence level
were retained for further consideration. The terms which survived this test were then
used to develop the measurement error models.
Estimation of IM240 and TSI Measurement Error
The measurement error variances of EVI240 HC, CO, and NOx and of TSI HC and CO are
needed for development of measurement error models and for evaluation of the
influences of measurement error on model predictions. In the context of this study,
measurement error is used in the statistical sense and includes all sources of error that
would cause the emissions measurement of a vehicle to be different if the vehicle were
tested at different I/M stations. Correctly determining the measurement error would
involve measuring the emissions of a seyrf vehicles at different times and at different
stations and instruments. Instead of using this type of comprehensive effort, we used
repeat measurements on a set of vehicles to estimate measurement error. IM240 repeat
measurements were performed following each other on the same dynamometer. TSI
repeat measurements were performed at the same I/M station within about one hour of
each other; some repeats were performed on the same BAR90 analyzer, and some were
performed on different BAR90 analyzers. In any case, the repeat measurements will
under-estimate the true measurement error since variability contributions of different
stations, dynamometers, and days are not present. Nevertheless, the use of estimated
measurement error values is significantly better than ignoring measurement error in
model development, which would essentially be assuming all measurement errors are
zero.
For the emissions of each repeat-tested vehicle, the variance of each repeat pair was
calculated, and then the variances for all vehicles getting repeat tests were pooled to
arrive at the overall variance for the test. EVI240 measurement errors for HC, CO, and
NOx were calculated using 127, 127, and 125 repeat pairs, respectively. In a somewhat
similar manner, the TSI measurement error variances were calculated for the TSI HC and
CO values using the repeat TSI data. High-speed idle HC and CO and low-speed idle HC
and CO had 146, 101, 159, and 111 repeat pairs, respectively.
The pooling of measurement variances for the repeat-tested vehicles must be performed
in a transformed space where measurement error is homogeneous, that is, where the
scatter from measurement error is constant across the range of emissions levels. We
searched for the optimum transformation using the following procedure. Each set of
repeat pairs was divided into low-valued pairs and high-valued pairs. Pairs were assigned
to low if their transformed-space average was below the transformed-space value
corresponding to 100 ppm for HC or 1.0% for CO; otherwise, they were assigned to high.
Then, we considered different power transformations from A,= 0.1 to 0.9 until the pooled
DRAFT August 2001 -78-
-------
standard deviations of the within-pair differences were the same for the low set and the
high set.
The same approach was used to estimate the measurement variance of the IM240 HC,
CO, and NOx. Table A-l shows the measurement variances for TSI and EVI240 tests.
Table A-l. Measurement Variances for EVI240 and TSI Measurements
IM240 HC (g/mile)
IM240 CO (g/mile)
IM240MX (g/mile)
High-Speed Idle HC (ppm)
High-Speed Idle CO (%)
Low-Speed Idle HC (ppm)
Low-Speed Idle CO (%)
Space
natural log
natural log
natural log
0.38 power
0.60 power
0.32 power
0.75 power
Variance
0.0798
0.284
0.126
1.30
0.042
0.92
0.037
To put the measurement error variances in perspective, the variances given in Table A-l
have been converted to the 95% confidence limits in linear space shown in Table A-2.
The confidence limits can be interpreted as follows. The exact value of a vehicle's
emission rate is unknown; the measured value is just an estimate of the emission rate.
The probability that the exact value falls within the confidence limits in the table is 95%.
For example, if a measured IM240 CO^alue'w^re 10 g/mile, we would be 95% confident
that the exact IM240 CO would be between 3.5 and 28 g/mile.
DRAFT August 2001
-79-
-------
Table A-2. Measurement Error 95% Confidence Limits for IM240 and TSI
in Linear Space
Measur
ed
Emissio
n Value
0.01
0.1
1
10
100
1000
IM240
HC
(g/mile)
|
J
0.06
0.57
5.75
a
P"
0.17
1.72
17.2
0
IM240 CO
(g/mile)
|
J
0.04
0.35
3.52
35.2
1
a .
0.28
2.84
28.40
284.0
0
IM240
NOx
(g/mile)
|
J
0.05
0.50
5.00
a
P"
0.20
2.00
20.00
1
J i -
High-
Speed
HC
(ppm)
1 ,
o
0
0
27
628
a
P"
Low-
Speed
HC
(ppm)
1 ,
o
22 p
57
23*7
1485
0
17
486
a
P"
27
74
306
1796
High-
Speed CO
|
J
0.00
0.00
0.42
8.38
a .
0.28
0.49
1.76
11.74
Low-
Speed
CO (%)
|
3
0.0
0
0.0
0
0.5
3
9.1
2
a .
0.30
0.46
1.53
10.90
I
l-^-J^J
An examination of the resulting measurement error magnitudes might lead the reader to
question the ability of the TSI (especially at low TSI values) to be useful to predict the
average IM240 emission rate of a fleet. In fact, the evaluation of sources of error when
applying these models to a fleet reveals that the TSI measurement error is one of the
smaller sources of error.
In any case, the non-negligible error variances for the TSI values, which were used as
predictor variables, provide a motivation for using measurement error models. This topic
is discussed in the following subsection.
Measurement Error Method for Final Models
In conventional regression analysis, it is assumed that the dependent variable (the IM240
HC, CO, or NOXvalue in this study) has error, but the independent variables have no
error. The TSI variables included as predictor variables in the models have, as we have
shown above, non-negligible measurement errors. Since the assumptions of conventional
regression analysis are not satisfied for this problem, if this method had been used to
develop the final models, there would have been biases in the regression coefficients. To
avoid this problem, statistical methods designed to handle situations with errors in both
the dependent and independent models were used. The type of model in which there are
errors in both the dependent variable and one or more of the independent variables are
called "measurement error models." Measurement error models were developed using
EV CARP software. This program is a product of the Statistical Laboratory at Iowa State
University.
DRAFT August 2001
-------
As with conventional regression, EV CARP requires the value of the dependent variable
and the values of the independent variables for each observation to be used in the model
development. Other inputs are required also, depending on the option of EV CARP that
is selected. The option used by ERG is called EV1. We elected to supply the variance of
the measurement error in the dependent variable, which is an optional input with EV1.
Additionally, the variances of the measurement errors in the predictor variables were
input. Covariances quantify the relationships between measurement errors for different
variables. Error covariances were also calculated from the repeat emissions tests and
supplied to the software.
The EV1 option is especially suited for this application because it accounts for several
separate sources of variability. Disagreement between individual measured EVI240
values and the IM240 values predicted by the model occurs for three reasons. First,
measurement errors in the dependent variable (the EVI240 value) cause data scatter.
Second, the TSI values measured with error are used in the model, so TSI measurement
error also causes differences between measured and predicted IM240 values.
There is a third reason for data scatter. EveWf the TSI values and EVI240 values were
measured with no error, there would still be some disagreement between the measured
and predicted EVI240 values. This is because of idiosyncrasies of individual vehicles that
cannot reasonably be captured perfectly by the modelj m
EV CARP is especially appropriate for this application, since it provides an option that
accounts for all three sources of data scatter mentioned above.
Ideally, the measurement variances and covariances of the predictor variables would be
calculated for input into EV CARP in the transform space where the variances were
homogeneous. These spaces were determined in the analysis described in the previous
subsection. However, we found that when the measurement error models were built
using these transformations for the input variables, the regression results for the
measurement error models were unstable. This instability was characterized by large
changes in the regression coefficients compared to the values obtained with the
conventional regression analysis. In some cases, the regression coefficients changed
sign. We found that to achieve a stable measurement error model it was necessary to
change the transformations used for the two-speed idle measurements. We found that the
natural log of the two-speed idle measurements produced measurement error models
which were stable. Unfortunately, this means that the two-speed idle variances and co-
variances used to develop the models were the average variances for the dataset when we
know that the variances are not homogeneous in log space. By using these average
variance values, the model "believes" that low TSI values carry more information and
high TSI values carry less information then they actually do. Nevertheless, the use of
these average variance values will provide models that should be superior to models built
without considering measurement error at all.
A.3 Limitations of the Models in Applications
The models developed for TNRCC relate emissions from TSI concentrations to EVI240
emission rates as they were determined: 1) in two specific Texas I/M stations for TSI
measurements and in a portable IM240 dynamometer environment for EVI240
DRAFT August 2001 -81-
-------
measurements; and 2) on a specific set of vehicles. Therefore, as with any models,
application of these models to other situations may result in the introduction of biases in
the results. Biases can be introduced through the application of the model in situations
with different TSI test conditions and/or different vehicle characteristics from those used
in the dataset used to develop the models. Nevertheless, the variety of model years,
technologies, vehicle types, and vehicle ages used in the model building data set should
be sufficiently diverse to allow the model to be used successfully in many real situations.
In the discussion in this section, we present a summary of the test conditions and vehicle
characteristics under which these models were built. The model user should consider
how the model application dataset differs with respect to test conditions and vehicle
characteristics when he uses the models reported in this study.
The following test conditions were used to acquire the model training dataset:
TSIs were measured with Texas I/M station grade BAR90 equipment and
procedures;
TSIs were measured at ambiWt temperature and relative humidity;
TSIs and IM240s were dstermined on vehicles with as-received fuel; and
IM240s were measured op,a portable dynamometer system.
JK t,
If TSIs are collected for an application dataset with equipment and procedures other than
those at the Texas I/M stations used to develop the model training dataset, then there is a
possibility of a bias or a different variance for the TSI measurements between the training
dataset and the application dataset.
The effects of ambient temperature and relative humidity on TSI HC and CO results to
our knowledge, are not known. Therefore, TSI results at conditions other than the
ambient temperature and relative humidity used for the training dataset could produce
TSI values which are systematically differeriC'
There are several vehicle characteristics of the training dataset which could affect the
applicability of the models developed:
Model year and vehicle age;
Vehicle type;
I/M program in place at the time of the training dataset collection; and
Small, specific fractions of the fleet.
Application of the models to datasets which differ significantly from the training dataset
in model year could be a step outside prudent application limits. This would also include
application to datasets where vehicle ages were significantly different from those in the
training dataset even though the model year distribution was similar. The model user
should also be aware of the emission control technologies used on the vehicles in the
application dataset although attention to the model year distribution should be adequate
given the high correlation between emission control technology and model year.
DRAFT August 2001 -82-
-------
Consequently, we expect that it will be beneficial to update the models as newer TSI and
IM240 measurements on a set of vehicles become available.
The Texas models were built on vehicles with models years from 1981 to 1997. Smaller
numbers of vehicles in the oldest model years mean that the uncertainty in the predicted
IM240 values for vehicles in those model years is relatively larger than for the IM240
emissions in the later years. As far as predicting fleet emissions is concerned, for the
middle 1990's model year vehicles, the very low IM240 and TSI emissions of these
vehicles make the measurement and prediction of EVI240 emissions with small relative
errors difficult.
Perhaps a more subtle limitation on the application of the models developed in this study
is the effect of the I/M program in force at the time of data collection. For the training
dataset, the I/M program at the time was based on two-speed idle testing. Therefore, the
vehicles which were tested for TSI and IM240 emissions were subject to a two-speed idle
I/M program. As long as the models developed in this study are applied to vehicles
subject to the same two-speed idle I/M program and cutpoints, there should be no
question that the model application is appropriate from this perspective. However, if a
different I/M program is instituted, then it is possible that the relationship between TSI
and EVI240 could be different. Under a new I/M program, vehicles would be tested and
repaired based on other emissions results. There is no guarantee that the resulting
changes in the emissions characteristics of the vehicle population would preserve the TSI
to EVI240 relationships discovered in this, stufly.
The correlation models are intended to be used to estimate the average IM240 emissions
of a large fleet of vehicles such as the Texas fleet. The estimates can be made for
different cities and for different model years. The uncertainty of the average will
increase for small fractions of a fleet since small fractions could not have been well
represented in the model training dataset. For example, we would expect larger
uncertainties for predicted EVI240 emissions for 1985 light-duty carbureted trucks. Thus,
as an investigator further sub-divides the application dataset when applying these models,
the uncertainty of the mean predicted IM240 emission rates increases. In the extreme, the
largest uncertainties are those for a single vehicle based on its TSI measurement.
A.4 Accuracy of the Models in Their Application
This section discusses application of the models and the roles of various sources of
variability. Issues pertaining to model precision and bias and the effect on the estimation
of the fleet average by using the models are also covered.
The role of several types of variance in using the model to estimate a fleet average is
discussed below. The estimation of the fleet average involves first estimating the average
IM240 emission rate in model year strata. These stratum-specific averages are weighted
by their travel fractions and summed to obtain the estimated fleet average. Model
refinement is necessary to achieve zero or insignificant biases in the strata. This in turn
produces a zero or insignificant bias in the estimate of the fleet average.
Refinements in the model precision may not change the estimated precision in the fleet
average. Refinements improve the estimate of the emissions of a specific vehicle. Thus,
DRAFT August 2001 -83-
-------
the unexplained part of the variance in the IM240 values decreases. However, the
explained part of the variance of the EVI240 values increases by an equal amount. The
uncertainty in the estimate of an average emission level is a function of both types of
variance. The details of these relationships are discussed further below.
Even if an enhancement to the model does not change the estimated precision in the fleet
average, improving the model is still beneficial. As is mentioned above, enhancements in
the model reduce the possibility of bias in the strata and therefore reduce the possibility
of bias in the estimate of the fleet average. A detailed set of plots may be used to
determine if any bias remains in the models and if it is small compared to the random
scatter.
What is meant by bias in this context is lack of fit between the model and the data that
could be eliminated by modifying the terms in the model in some manner (including
additional terms or changing the functional forms of the existing terms). The issue here
does not pertain to biases in the data jQj to biases resulting from inappropriate use of the
models.
Variability as it Influences Precision a^cl E^s
We will briefly review the different sources of variability and indicate the role of each
source. The primary emphasis of this settion pertains to the application of the models.
However, this cannot be adequately discussed without some reference to the model
development.
IM240 measurement errors, TSI measurement errors, and vehicle-to-vemcle
idiosyncrasies that are not captured by the models all affect the model development. All
of these sources of variability contributejojscatter of data points in the model-
development dataset about the IM240 values predicted by the model.
Of these sources of variability, the IM24
-------
The terms involving TSI values are self-explanatory. The terms without error include
predictor variables such as vehicle age, vehicle type, fuel metering type, and
displacement that are considered to be known essentially without error. The vehicle-to-
vehicle term represents the effect of the vehicle's specific characteristics that are not
captured by the model.
If one were interested in predicting the IM240 value for an individual vehicle, the
predominant errors of concern would include (1) the effect of measurement errors in the
TSI values used in the prediction and (2) the unexplainable vehicle-to-vehicle term.
Since the EVI240 value for a specific vehicle is needed, variability among true IM240
values in the fleet does not contribute to the relevant error in the estimate.
Alternatively, suppose we want to estimate the average emissions for a fleet or stratum
within a fleet. Even if there were no TSI measurement error and no unexplainable
vehicle-to-vehicle term, the average EVI240 value based|On a sample of size n would still
have an error. The sample will not perfectly represent the population from which it is
drawn. The imperfect representajjonLoithe population by the sample occurs because of
random variability of the true IM240 values among vehicles in the fleet and because of
the random sampling process.
To summarize, the following three sources of variation affect the estimation of IM240
average emissions on the basis of predictions made using one of the models:
(1) The effect of TSI measurement errors on the predictions;
(2) True variability of the EVI240 values that is captured by the model; and
(3) True variability of the IM240 values that is not captured by the model.
The first two errors listed above are represented in Ine predicted EVI240 values. If we
compute the variance of the predicted IM240 values for our sample of size n, this
variance will represent the effect of these two sources of variability. We call the variance
of the n predicted values s2xplaimd . This is the "explained" variance in the sense that it can
be computed directly on the basis of the predicted values.
But, as is discussed above, the emission rate of a given vehicle deviates from the
predicted value because of the vehicle-to-vehicle idiosyncrasy effect. Since the vehicle-
to-vehicle effect is not "explained" in terms of the predicted IM240 values, we denote its
variance s2 laimd . The total variance of a single IM240 prediction is as follows:
s2 =s2 +s2
total explained unexplained
If we average the predicted EVI240 values for n vehicles, the result will differ from the
true average for the population sampled because of both the unexplained and the
explained errors discussed above. The variance of the error in the mean associated with
the explained part of the variance is s2explained In. The variance of the error in the mean
DRAFT August 2001 -85-
-------
associated with the unexplained part of the variance is s2unexplained/n. Thus, the total
variance of the error in the mean is as follows:
s2 +s2
2 explained unexplained
mean
n
Now, suppose we make some improvements to the model that allow the model to
"explain" some of the variance that was previously unexplained and therefore included in
the vehicle-to-vehicle term. The improvement, therefore, reduces s2unexplained to some
extent.
The total variance among the true EVI240 values in the sampled population does not
change as a result of the change to our model. That is, changing the model does not
change s2total. The s2otal is the same (except for the small effects of TSI measurement
error and EVI240 measurement error) as if IM240s had actually been measured. Thus,
Explained ^s increased by the amount that s2unexplained is decreased. Thus, the error variance
smean > which is an estimate of the precision inthe mean, is not changed by improvements
in the model (one can contrive exceptions to this statement on the basis of trivial models).
This does not, however, imply that improvements in the model lead to no improvement in
the estimation of the fleet average EVI240 emission rate; model improvements lead to
reduced biases. The process of making this estimation will be briefly summarized here.
The average and error variance of the average is computed within each stratum, where a
stratum consists of all data for a specific model year. The estimated average emission
rate for the fleet equals the sum of stratum-specific averages, each weighted by its travel
fraction.
Now, suppose we omitted a variable, such as model year, from the model. The mean
residual (observed minus predicted value) in the model development dataset would still
be zero, since this is a property of regression analysis. However, there would be biases in
the strata. Similar comments would apply if model year were included in the model, but
the functional form of the term involving model year did not fit the data. Avoiding
prediction bias is much more complicated than simply being sure that all the necessary
variables are included in the model in their simplest forms.
In this exercise, the sample size of the model development datasets HC, CO, NOx were
897, 921, and 918 observations, respectively. Despite this large sample size, the counts
in the strata can be small. For example, the largest number of vehicles for any model
year is 66 for 1988 and 1993. Much smaller counts exist for some years. For example,
1981 has only 8 counts.
Even if the model development dataset were selected randomly from the fleet, because of
sampling variability, one would not expect the travel fractions in the fleet to be exactly
matched by the fractions of vehicles in the strata in the model development dataset.
Thus, even if the biases in the different strata in the model development dataset produce
DRAFT August 2001 -86-
-------
an average residual value of zero, the biases will in all likelihood not balance when the
fleet average is computed on the basis of the application dataset.
The solution is to develop the models so that the biases in the strata are zero or
insignificant. If this is achieved, the bias in the fleet average is likely also to be zero or
insignificant. There will be no necessity for the biases in the different strata to "balance"
each other for the bias in the fleet average to be unbiased.
Evaluation of Bias in the Models
The importance of avoiding model bias is stressed in the discussion above. The
considerable steps taken to avoid significant prediction biases are discussed in this
subsection. Again, bias in this context refers to a systematic difference between the
observed and predicted EVI240 values, such that this systematic difference could be
eliminating by including more or different terms in the models. Biases in the data or
prediction biases resulting froifi imp^yper use of the models are addressed in Section A. 3.
Evaluation of the models to ensure that no significant biases exist is an important
additional step and was performed by examining a large number of plots. Table A-3
presents a list of plots that was prepared for this purpose for the Texas models. Recall
that a residual is the observed minus predicted value. The variables are in natural-log
space unless otherwise noted. In addition to the scatter plots listed in Table A-3, several
histograms were also prepared.
DRAFT August 2001
-87-
-------
Table A-3. List of Model Validation Scatter Plots Examined for All Three Pollutants, for
All Vehicles Combined, for Trucks and Cars Separately, and for Carbureted and Fuel-
Injected Vehicles Separately
Y- Variable in the Plot
Residual
Residual
Residual
Residual
Residual
Residual
Residual
Measured Value of the Pollutant
Ratio of Average Predicted^) Ave/age
Measured Value by Model Year
Ratio of Average Predicted to Average
Measured Value by Model Year Group
Mean Residual by Model Year
Measured Value in Linear Space
Measured versus Predicted Value in Linear
Space, Expansion Showing Smaller Values
X- Variable in the Plot
Model Year
Natural Log of Displacement
Natural Log of High-Speed Idle HC Value
Natural Log of High-Speed Idle CO Value
Natural Log of Low-Speed Idle HC Value
Natural Log of Low-Speed Idle CO Value
Natural Log of Predicted Value of the Pollutant
Estimated Value of the Pollutant
Model Year
i
Model Year Group
r]
Model Year
Predicted Value in Linear Space
'/Expansion around Smaller Predicted Values in
T'Wi** Linear Space
Table A-3 lists 13 types of plots. These were all produced for all three pollutants (HC,
CO, and NOX), resulting in 39 separate plots in a set. A complete set of plots was
produced for five cases: All vehicles combined, trucks and cars separately, and
carbureted vehicles and fuel-injected vehicles separately. Five sets times 39 plots per set
results in 195 separate plots. The ERG staff examined all of these plots and reasonable
sampling of the plots, which present the majo'r're'sults. A major objective in plotting
residuals is to determine whether any remaining trend exists in the data. If so, it is
possible that further improvement in the models can be made.
Further Discussion of the Role of Model Year
The performance of the models as a function of model year is important and warrants
some discussion. One way to address this issue is to examine the average residual for
each model year, as in the figures described above. However, the number of vehicles
varies as a function of model year. The mean residuals for years with small numbers of
data points are highly variable. One way to address this issue is to account for the
different sample sizes for different years by using the t-statistic.
To address this problem, the t-statistic was computed for each model year. The t-statistic
for a particular model year is as follows:
t =
DRAFT August 2001
-------
where
t = t-statistic;
f = mean log-space residual for this model year;
n = number of vehicles for this model year; and
s = pooled standard deviation.
The pooled standard deviation s is an estimate of the variability within a model year.
However, the separate estimates from all model years were combined to obtain the most
reliable common estimate. Pooling was necessary, since otherwise the standard
deviations for some years with small numbers of vehicles were unreliable.
The t-statistic accounts explicitly for the different sampl*«sizes in the different years. The
residuals are expressed in units that are much more comparable for different years with
different numbers of data poin|sxi
We have shown that the means of the log space residuals versus model year appear to be
unbiased. However, when the predicted values of EVI240 are considered in linear space,
it is possible that biases with respect to model year can be present.
To evaluate the potential for bias in the linear space predictions as a function of model
year, we calculated the average predicted and measured EVI240 value for each model year
in the dataset. Then we took the ratio of the average predicted IM240 value and the
average measured IM240 value. These ratios were plotted as a function of model year,
with a horizontal reference line at 1.0 on each graph. If there is an insignificant bias with
respect to model year, the data points that apply should be scattered more or less
randomly about this line. These plots showed data points that were scattered randomly
about the 1.0 reference line.
Histograms for Residuals Revealing the Roles of Additional Variables
Additional plots were produced to reveal the role of other variables. These include the
vehicle type (car or truck), the presence of a carburetor or fuel-injection, and the presence
of exhaust gas recirculation. As is indicated in Section A.2, variables to account for the
vehicle type, the carburetor versus fuel-injection dichotomy, and the exhaust gas
recirculation dichotomy are included in the models. In view of this, a remaining bias
with respect to these variables was not expected.
DRAFT August 2001
------- |