Analysis of Invalid Emission Testing
in the California Smog Check Program
prepared for:
U.S. Environmental Protection Agency
April 27, 1994
principal authors:
Thomas C. Austin
H. Anthony Ashby
Thomas R. Carlson
Sierra Research, Inc.
1801 J Street
Sacramento, California 95814
(916) 444-6666
-------
Although the information described in this report has been funded
wholly or in part by the United States Environmental Protection
Agency under Contract No. 68-C1-0079, it has not been subjected to
the Agency's peer and administrative review and is being released
for information purposes only. It therefore may not necessarily
reflect the views of the Agency and no official endorsement should
be inferred.
-------
Analysis of Invalid Emission Testing
in the California Smog Check Program
Table of Contents
1. Summary 1
2. Introduction 4
3. Analysis of I/M Evaluation Data 7
4. FTP Emissions Impact of Invalid Testing 19
-i-
-------
List of Tables
1. Smog Check Tailpipe Idle Emissions From Undercover
Vehicles Before and After Smog Check Repair 13
2. FTP Mass Emissions From Undercover Vehicles Before
and After Smog Check Repair 17
3. Undercover Fleet - All Vehicle Emissions
By Model Year Range 19
4. Undercover Fleec - "Suspect" Vehicle Emissions by Model
Year Range 20
5. Undercover Fleet - "Other" Vehicle Emissions by Model
Year Range 20
6. Undercover Fleet - With 54X of Randomly Selected "Suspect"
Vehicles Repaired by Model Year Range 21
7. Mass Emission Reductions With and Without 54X of Randomly
Selected "Suspect" Vehicles Repaired 21
8. Undercover Fleet - With the 54X Dirtiest of "Suspect"
Vehicles Repaired by Model Year Range 22
9. Mass Emission Reductions With and Without 54X Dirtiest
"Suspect" Vehicles Repaired 23
10. Undercover Fleet - With the 54X Cleanest of "Suspect"
Vehicles Repaired by Model Year Range 23
11. Mass Emission Reductions With and Without 54X Cleanest
"Suspect" Vehicles Repaired 24
-ii-
-------
List of Figures
1. Average Idle CO Readings for Undercover Vehicles
With Lasc Smog Check Test Confirmed by ARB 2
2. Average Idle CO Readings for Undercover Vehicles
With Lasc Smog Check Test Much Lower Than ARB Retest 3
3. Average Idle HC Readings for All Undercover Vehicles 5
4. Average Idle CO Readings for All Undercover Vehicles 5
5. Distribution of Differences in Idle HC Readings:
ARB Initial Test minus Screening Test 9
6. Distribution of Differences in Idle CO Readings:
ARB Initial Test minus Screening Test 9
7. Distribution of Differences in Idle HC Readings:
ARB After-Smog Test minus Last Smog Check Test 10
8. Distribution of Differences in Idle CO Readings:
ARB After-Smog Test minus Last Smog Check Test 10
9. Distribution of Differences in Idle HC Readings:
Normal Distribution (ARB Initial less Screening) vs.
ARB After-Smog Test less Last Smog Check Test 11
10. Distribution of Differences in Idle CO Readings:
Normal Distribution (ARB Initial less Screening) vs.
ARB After-Smog Test less Lasc Smog Check Test 11
11. Average Idle HC Readings for Undercover Vehicles
With Last Smog Check Test Confirmed by ARB IS
12. Average Idle CO Readings for Undercover Vehicles
With Last Smog Check Test Confirmed by ARB 15
13. Average Idle HC Readings for Undercover Vehicles
With Last Smog Check Test Much Lower Than ARB Retest 16
14. Average Idle CO Readings for Undercover Vehicles
With Last Smog Check Test Much Lower Than ARB Retest 16
IS. Mass Emissions From Undercover Vehicles Before
and After Smog Check 18
16. Frequency of "Suspect" Exhaust Measurement on the
Last Smog Check Test, By Vehicle Model Year 18
-iii-
-------
Analysis of Invalid Emission Testing
in the California Smog Check Program
1. Summary
Emissions measurements reported by licensed automotive repair
dealerships participating in the California motor vehicle inspection and
maintenance (I/M) program (called "Smog Check") have been compared to
measurements made on the same vehicles at che California Air Resources
Board (ARB) laboratory. Based on an experimental program under which
1,100 defective vehicles were taken to randomly selected repair
facilities (called "Smog Check stations") by ARB employees posing as
ordinary motorists, the Smog Check station emission test results appear
to be invalid for about 17.52 of the vehicles. This is the fraction of
defective vehicles for which the lower readings reported by Smog Check
stations cannot be explained by test-to—test variability. These
vehicles usually failed the emission standards at ARB after reportedly
passing the standards at the Smog Check station.
Although the computer-controlled emissions analyzers used in the
California program are designed to minimize che occurrence of erroneous
emission measurements, there are several ways in which invalid results
can be recorded. One technique, referred co as "clean piping", involves
measuring the emissions from a known clean vehicle and reporting the
results as if they were obtained from another vehicle. Another
technique involves partially removing the sampling probe from the
tailpipe to dilute the exhaust sample enough to gee the vehicle to pass
the test without exceeding the dilution thresholds programmed into the
analyzer. Temporarily increasing idle speed is a technique used on some
vehicles. Specific techniques that may have been used on the vehicles
involved in the ARB experimental program are unknown.
All of the vehicles in the experimental program should have failed a
properly conducted Smog Check test, and most of them did fail the first
test conducted aC a randomly selected Smog Check station. However, some
of the vehicles returned from the Smog Check stations with idle emission
test results that were substantially different from the results of a
confirmatory test conducted by ARB. On average, the vehicles showing
the greatest discrepancy between the last test at the Smog Check station
and the retest at ARB had significantly higher Federal Test Procedure
(FTP) and idle emissions during the initial (pre-Smog Check) testing at
ARB. The final test results reported by the Smog Check stations
indicate that idle emissions were reduced to the same level as other
vehicles, presumably due to the effect of repairs. However, based on
ARB's test results before and after the Smog Check, no significant
change in either FTP or idle emissions actually occurred. The lack of
an FTP emission reduction is consistent with the conclusion that the
-1-
-------
idle emission reduction reported by the Smog Check stacion was not che
result of any actual repairs.
Figures 1 and 2 highlight the differences between the vehicles for which
ARB's test was much higher than the last Smog Check test and the
vehicles for which the last Smog Check test was confirmed by ARB's test.
As shown in Figure 1, in cases where the last Smog Check test was
confirmed by ARB, CO emissions were reduced substantially. Based on the
Before and After Smog Check tests conducted by ARB, idle CO emissions
were reduced from 1.63X to 0.73X, a 55X reduction. As shown in
parentheses, FTP emissions were reduced from 3.86 g/mi HC, 40.5 g/mi CO,
and 2.20 g/mi NOx to 2.67 g/mi HC, 32.7 g/mi CO, and 1.92 g/mi NOx,
reductions of 3U for HC, 19X for CO, and 13X for NOx.
Figure 2 shows a radically different trend for the vehicles with a large
discrepancy between the results reported by the Smog Check station and
the results measured by ARB after Smog Check. Not only were the idle CO
emissions higher to begin with (3.OX), the idle emissions after the
vehicle was returned to ARB from the Smog Check stacion were actually
higher (3.36X). In contrast, the Smog Check stations reported that the
idle CO emissions were reduced to 0.69X. As shown in parentheses, FTP
emissions before Smog Check - 7.41 g/mi HC, 64.6 g/ml CO, and 2.00 g/mi
NOx - were substantially higher than for the other vehicles. After
return from the Smog Check stacion, emissions had been reduced by only
10X for HC. OX for CO, and 11X for NOx.
Figure 1
Average Idle CO Readings for Undercover Vehicles
With Last Smog Check Test Confirmed by ARB
(FTP Results, g/mi HC/CO/NOx, in parentheses)
*,
o
o
3*
ARB's lasts show that
average kft* CO emissions
before Smog Check were) 1.63%
Smog Check stations reported
that CO emissions were reduce
Rate** at ARB show that
CO emissions war* in fact
reduced to 0.73%.
0.73
ARB HIM
Last Smog Check
ARB After Smog Check
(Z67/3Z7/1.92)
ARB Altar-Smog readings do not exceed
Last Smog CtMck reaolngs by • margin ol
400 ppm HC or 2.00% CO.
-2-
-------
Figure 2
Average Idle CO Readings for Undercover Vehicles
With Last Smog Check Test Much Lower Than ARB Retest
(FTP Results, g/mi HC/CO/NOx, in parentheses)
.3
O
O
ARB's tests show that idle CO omissions
before Smog Check were 3.00%.
100
Retests at ARB indicate mat
idle CO emissions actually increased.
Smog Check stations feporteU
that CO emissions were reduced
to 0.69% after repair.
ARBtaHM
(7.41/84.8/2.00)
ARB After-Smog reeoTngs meesd Last Smog Check
reedbtgs by more than 400 pom HC / Z00% CO.
Lot Smog Check
ARB After Smog Check
(0.88/88.0/1.79)
Because the occurrence of apparently invalid emission testing was
determined to be proportional to the emissions of vehicles, it is
estimated that invalid testing reduces the benefits of the Smog Check
program by at least 30X for hydrocarbons and 39Z for carbon monoxide.
NOx emissions are not significantly affected. These estimates do not
account for any additional loss in benefits that might be associated
with testing of vehicles owned by motorists who are regular customers of
the facility that performs the Smog Check. (In this study, the vehicles
were taken to randomly selected facilities not familiar with either the
driver or the vehicle.) The estimated reduction in benefits appears to
validate EPA's discount for I/M programs that allow vehicles to be
tested in the same facility that is responsible for repairing vehicles
that fail the test.
-3-
-------
2. Introduction
Deliberate falsification of test results to get vehicles through the
California I/M (Smog Check) program has been positively identified in
previous investigations by the California Bureau of Automotive Repair
(BAR). One form of test falsification is commonly referred to as
"clean-piping", which involves measuring the emissions from a known
clean vehicle and reporting the results as if they were obtained from
another vehicle. Other forms of falsifying the emission test results
include partially removing the sampling probe from the tailpipe to
deliberately dilute the sample enough to cause the vehicle to pass the
test but not exceed the dilution limits specified by BAR. Another
technique involves temporarily increasing the idle rpm. The net effect
of such techniques is that vehicles that should have failed the test do
not receive additional repairs and the emissions after Smog Check are
higher than they should be. The extent to which such testing techniques
are reducing the effectiveness of the California I/M program has not
been previously quantified. The comprehensive evaluation of the
Smog Check program that was conducted during 1992 quantified the
emission reductions achieved by the'.program without specifically
addressing the extent to which invalid test results reduced program
effectiveness.* That evaluation program involved the testing of 1,100
defective vehicles that were taken to randomly selected repair
facilities (called "Smog Check stations") by ARB employees posing as
ordinary motorists.
One indication of a possible problem with the validity of the test
results from Smog Check stations is that the idle emission rates of
vehicles stopped during random roadside inspections are higher than the
idle emission results obtained for the same vehicle during the prior I/M
test." Some have attributed this difference in emissions to action
that motorists have taken, e.g., restoring vehicles to a defective
condition almost immediately after passing an I/M test. However, a
preliminary analysis of data from the 1,100 vehicle ARB test program
indicated that the correlation problem between roadside inspection
results and Smog Check station test results exists before the vehicle is
ever returned to the motorist after it passes a Smog Check test. As
shown in Figure 3, average idle emissions for the vehicles were 76 ppm
HC during their last tesc at a Smog Check station (which was typically
the after-repair test) before they were returned to ARB. When retested
at ARB, the actual idle emissions (196 ppm) were more than 100X higher.
Figure 4 shows similar results for the idle CO emission measurements.
' "Evaluation of the California Smog Check Program and Recommendations
for Program Improvements, Fourth Report to the Legislature," California
I/M Review Committee, February 16, 1993.
" D.R. Lawson, "Passing the Test - Human Behavior and California's
Smog Check Program," Air & Waste. Vol. 43, December 1993.
-4-
-------
Figure 3
Average Idle HC Readings
for All Undercover Vehicles
(FTP results, g/mi HC/CO/NOx, in parentheses)
800
700
600
Q.500
a
Q" 400
200
100
0
329
196
ARBtaMM
(4.«7/4a.
-------
Task 2 of Work Assignment 2-01 of Sierra's support contract with EPA
(Contract No. 68-C1-0079) requested the performance of a more detailed
analysis of the data collected by ARB during the 1,100 vehicle
undercover program to determine the extent to which fraud or testing
error might be causing the true emissions of vehicles to exceed the
results reported by Smog Check stations.
-6-
-------
3. Analysis of I/M Evaluation Data
To determine the frequency of invalid exhaust measurements and to
determine the effects of such measurements on I/M program benefits, it
is necessary to have a robust sample of data from vehicles that have
gone through the Smog Check program and have been tested before and
after the process. In addition to the mass emissions tests before and
after Smog Check using the Federal Test Procedure (FTP), independent
measurements of emissions at idle and 2500 rpm are needed for comparison
with the results reported by Smog Check stations. The 1,100-car
undercover tesc program conducted by ARB for the 1992 Smog Check
evaluation provided the required data base. The undercover program
test/repair sequence was as follows:
0) Screening Smog Check tailpipe test at ARB contractor
1) Baseline Smog Check visual/functional inspection and tailpipe
test at ARB laboratory
2) Baseline FTP at ARB laboratory
3) Baseline (Initial) Smog Check visual/functional inspection and
tailpipe test at Smog Check station
4) After-Repair Smog Check visual/functional inspection and
tailpipe test at Smog Check station
5) After-Smog Check visual/functional inspection and tailpipe test
at ARB Laboratory
6) After-Smog Check FTP at ARB laboratory
Analysis of I/M Test Variability
The initial focus of the analysis effort was based on the presumption
that invalid exhaust measurements could be determined by comparing
concentration readings from the last (or After-Repair) Smog Check
tailpipe test (Step 4 above) with those from the test conducted when the
vehicle returned to the ARB laboratory (Step 5 above). If the ARB
After-Smog readings on a parcicular vehicle were sufficiently higher
than the vehicle's Last Smog Check readings, then it could be said that
the vehicle had apparently been tested in an invalid manner in the
Smog Check shop. In this context, however, the determination of what
"sufficiently higher" should be requires consideration of the normal
variability associated with site-to-site and test-to-test variation.
To determine the expected variation in tailpipe readings from one site
to another, the readings from the Initial test at the ARB laboratory
(Step 1) were compared with those from the Screening facility (Step 0).
-7-
-------
Since the operator of che Screening facility knew that ARB would be
conducting confirmatory tests, there was no reason to suspect that there
would be any bias deliberately induced during the testing process at the
Screening facility.
The distribution of the differences in che readings (ARB Initial test
minus Screening test) is presented in Figures 5 and 6, for Idle HC
readings and Idle CO readings, respectively. The differences appear to
be normally distributed, with about as many positive as are negative. A
similar examination was made of the differences between the ARB After-
Smog test and the last Smog Check tesc, which was the test on which the
vehicle would receive its certificate of compliance. The distributions
of those HC and CO differences (ARB After-Smog test readings minus last
Smog Check test readings) are presented in Figures 7 and 8,
respectively.
Estimation of Invalid Test Results
Figures 7 and 8, when compared with Figures 5 and 6, show a bias toward
positive difference values, chat is,- they show chac more of che ARB
After-Smog readings were higher Chan che lasC Smog Check readings,
rather than being normally discribuced with similar numbers higher and
lower. One example will illustrate this apparent bias: Referring Co
Figure 8, on 12.52 of che sample, che ARB AfCer-Smog Idle CO reading was
higher by more Chan 2.00X CO Chan che Idle CO reading on che last test
at a Smog Check shop. In che more normally discribuced differences in
readings between che ARB IniCial Cesc and che Screening test, che
difference was greater Chan 2.00X on only 4.3X of che sample, as shown
in Figure 6.
The bias in the comparison of ARB After-Smog readings can be seen more
clearly in Figures 9 and 10. These show, for HC and CO respectively,
che ARB After-Smog minus Last Smog Check differences plocced as bars
against a line plot of che more normally discribuced ARB IniCial minus
Screening Cesc results. Comparing ARB resulcs wich che Screening Cesc,
23.IX of che vehicles had HC emissions ac lease SOppm higher when
measured ac ARB. 17.IX of che vehicles had CO emissions ac lease 0.25X
higher.* In contrast, comparing ARB resulcs with Che last tesc at a
Smog Check scacion, 39.3X of Che vehicles had HC emissions at lease
SOppm higher when measured ae ARB and 34.6X of che vehicles had CO
emissions aC lease 0.2SX higher. The ARB vs. Smog Check distribution is
skewed coward higher readings ae ARB by 16.2X of che sample in Che case
of idle HC and 17.SX of Che sample in Che case of idle CO. These
comparisons indicate that a minimum of 17.SX of the vehicles had higher
emissions in che after-Smog Check cese ac ARB Chan can be explained by
cesc-co-cesc variabilicy.
* The SO ppm HC and 0.25X CO Colerance were used Co eliminace
consideration of test resulc differences due Co Che accuracy of che
analyzers used in BAR-approved Test Analyzer Systems. The BAR analyzers
have an HC accuracy of 30-80 ppm and CO accuracy of 0.15-0.4X on che
mid- and high-range.
-8-
-------
Figure 5
Distribution of Differences in Idle HC Readings:
ARB Initial Test minus Screening Test
o
o>
C7
-------
40
35
30
Figure 7
Distribution of Differences in Idle HC Readings:
ARB After-Smog Test minus Last Smog Check Test
§ 20
D
STlS
10
5
0
1.7
0.4 0.3 0.0 0.0 0.2 0.2 m
Differences in Idle HC Readings
Figure 8
Distribution of Differences in Idle CO Readings:
ARB After-Smog Test minus Last Smog Check Test
40
35
30
20
10
5
0
27.4 27.3
19 i« 2-a
0.3 0.5^ 0.4 _&?_ r=l [Sil
3.1
12.5
1
5.1
2.0 10 1S 1.8
Differences in Idle CO Readings
-10-
-------
Figure 9
Distribution of Differences in Idle HC Readings:
Normal Distribution (ARB Initial less Screening)
vs ARB After-Smog test less Last Smog Check test
<£
40
35
30
si 25
® 20
S"l5
ARB Initial minus
Screening test
ARB After-Smog minus
Last Smog Check test
40
35
tf30
>125
O 20
§"15
Differences in Idle HC Readings
Figure 10
Distribution of Differences in Idle CO Readings:
Normal Distribution (ARB Initial less Screening)
vs ARB After-Smog test less Last Smog Check test
ARB InfflaJ minus
Screening test
ARB Aftsr-Smog minus
Last Smog Check test
Differences in Idle CO Readings
-11-
-------
Because the increased frequency of higher measurements at AR3 is not
restricted to extreme differences, it is difficult Co separate those
vehicles that were improperly tested from vehicles that were higher at
ARB due only to sice-Co-site/test-to-test variability. In addition, in
cases where the differences between ARB's measurements and the values
reported by che Smog Check station were relatively small, there is less
concern about the loss in potential emission benefits associated with
the improper or falsified test results. For that reason, the analytical
approach focused on those vehicles where the ARB After-Smog Check
readings were more than 400 ppm HC or 2.00X CO higher than the "Last
Smog Check" reading. Basing the analysis on this level of difference
was somewhat arbitrary because there was no level of difference which
cleanly divided the vehicles into those with high variability and those
with invalid results reported by the Smog Check station. However, as
described below, the analytical approach used accounted for the fact
that some of the vehicles with differences exceeding the 400 ppm/2.00X
criteria were expected to have been properly tested.
Based on the comparison between ARB and Screening Facility test results
(as shown earlier in Figures 1 and 2), a total of 7.8X of the vehicles
would have been expected to have idle test results ac ARB that exceeded
their Smog Check test results by more Chan 400 ppm HC or 2.01 CO. When
both idle and 2500 rpm emission tests (used on 1980 and later models)
are considered, a total of 9.41 of che vehicles would have been expected
Co have test results ac ARB that exceeded their Smog Check test results
by more than 400 ppm HC or 2.OX CO. Because 20.4Z of che undercover
vehicles exceeded these margins, 11.OX of che vehicles exceeded che 400
ppm/2.0X margins and have apparently invalid test results, based on this
analysis. Expressed another way, 54X (11.0+20.4) of che vehicles chac
exceed che 400 ppm/2.0X margins are projecCed co have invalid CesC
results.
Characteristics of Vehicles Based on ARB/Smog Check Correlation
In che undercover car test program were a Cocal of 1,027 vehicles chac
received a LasC Smog Check test (ac a Smog Check shop) and a subsequent
Smog Check CesC (After-Smog test) ac Che ARB laboracory. Of these, 209,
or 20.4X, exceeded che 400 ppm HC or 2.00X CO exhausc measuremenc
difference criCerion. In che remainder of che analysis, vehicles for
which che lasc ARB cesc exceeded che Lasc Smog Check CesC by more chan
Che 400 ppm/2.0X margins are referred Co as "Suspecc" vehicles because,
as explained above, 54X of chem are projected co have invalid cesc
results. Of chese 209 vehicles, 96.7X Passed che Last Smog Check cesc
and Chen Failed che Afcer-Smog cesc ac Che ARB laboracory. In cases
where che ARB Cesc resulcs did noc exceed Che Lasc Smog Check results by
the 400 ppra/2.0X margins, some invalid cescing is also suspected.Co have
occurred; however, Chese vehicles are colleccively referred Co as
"Ocher" vehicles or vehicles for which ARB cesc resulcs and Lasc Smog
Check cesc resulcs are in reasonable agreemenc.
All 1,027 vehicles were noc suicable for an analysis inCended Co address
che effeccs of pocencially invalid cescing on che effecciveness of che
California I/M program. Only 831 vehicles received a Lasc Smog Check
-12-
-------
cesc and an ARB After-Smog test, plus a Smog Check tailpipe cest in
Baseline condition ac ARB's laboratory, and an Initial test at a Smog
Check shop (Steps 1. 3, 4, and 5), and received FTP mass emissions tests
in Baseline (Step 2) and After-Smog Check conditions (Step 6). ARB
recasting after the return of the undercover vehicles from the Smog
Check stations indicated results exceeding those reported by the Smog
Check station by 400 ppra HC or 2.OX CO in 22.9X (190 out of 831) of the
cases.
Idle emissions - Table 1 presents average Idle emissions concentrations
of HC and CO measured during the tailpipe test portion of the Smog Check
inspection. In cases where the ARB test after Smog Check and the Last
Smog Check test were in reasonable agreement, the vehicles had roughly
similar readings in both facilities: average HC readings were 75 ppm in
the Last Smog Check test and 110 ppm in the After-Smog test at the ARB
laboratory. The CO readings were also similar: the average Idle
Table 1
Smog Check Tailpipe Idle Emissions
From Undercover Vehicles
Before and After Smog Check Repair
Condition
ARB— Smog Check Results Agree:
Initial (Baseline) Test at ARB
After-Smog Test at ARB
Z Change, Initial (Baseline)
to After-Repair
Initial (Baseline) Test
ac Smog Check
Last Test ac Smog Check
X Change, Initial (Baseline)
to After-Repair
ARB Results Much Higher:
Initial (Baseline) Test at ARB
AfCer-Smog Test at ARB
Z Change, Initial (Baseline)
Co After-Repair
Inicial (Baseline) Test
at Smog Check
Last Tesc at Smog Check
Z Change, Inicial (Baseline)
co Afcer-Repair
Idle Emissions
HC, ppm
267
110
-58.8
210
75
-64.3
537
484
-9.9
347
81
-76.7
CO, Z
1.63
0.73
-55.2
1.20
0.66
-45.0
3.00
3.36
+12.0
2.04
0.69
-66.2
-13-
-------
reading of 0.66% in che Last cest ac che Smog Check shops is close Co
the average 0.737. recorded in the After-Smog test at the ARB laboratory.
These vehicles also showed Initial (Baseline) to After-Repair Idle
emission reductions of similar magnitudes in both the ARB lab and the
Smog Check stations. Tailpipe HC readings, for instance, were reduced
58.8% in the ARB lab, and 64.3% in the Smog Check shop. Tailpipe CO
reductions were also of the same order of magnitude, i.e., 55.2% in the
ARB lab and 45.0% in the Smog Check shop.
For the "Suspect" vehicles (where the ARB test results were much
higher), the results were significantly different, comparing both
measured emissions concentrations and claimed reductions at the Smog
Check shop with the actual reductions measured at the ARB laboratory.
Referring still to Table 1, average Smog Check Last-Test HC was 81 ppm,
but 484 ppm at ARB (After-Smog). Average Last-Test CO was 0.69X at Smog
Check shops, but 3.36X at ARB. These readings are reflected in the
disparity between claimed and actual emission reductions also. A
reduction of 76.7% in Idle HC emissions was claimed to have been
achieved at the Smog Check station, but the same tests conducted at ARB
indicated that the HC reduction was only 9.9X. The CO situation is even
worse: the Smog Check stations claimed to reduce emissions by 66.2%,
but ARB tests indicated an actual emissions increase of 12.OX.
Figures 11 through 14 are the graphical presentations of the information
provided in Table 1. Figures 11 and 12 are HC and CO readings,
respectively, for vehicles where the ARB results are in reasonable
agreement with the Last Smog Check test; Figures 13 and 14 present HC
and CO for vehicles with ARB results exceeding the Last Smog Check test
by more than the margins. In each figure, the tailpipe emissions
concentrations are represented by vertical bars, arranged left to right
in the order in which the tests were performed.
The charts make it clear that there was not much difference in the
results reported by the Smog Check station regardless of whether the ARB
results were in agreement with the Smog Check station results. However,
tests conducted at the ARB laboratory cell another story. The "Suspect"
vehicles had higher emissions from beginning to end than the other
vehicles. The "Suspect" vehicles also emerged from che process wich
little or no reduction in their Idle emissions. Average Idle HC
emissions were reduced about 10X, from 537 ppm Co 484 ppm, and Idle CO
emissions actually increased, from 3.00Z to 3.36X.
Even though the other vehicles began che sequence of tests with lower
average emissions, they still delivered significant emission reductions.
As discussed earlier, and shown in Che chares, when there was reasonable
agreement between che ARB and Lasc Smog Check resulcs, Idle emission
reductions of 58.8X for HC and 55.2X for CO were observed comparing che
ARB Inicial Test (before Smog Check) wich che AfCer Smog Check ARB test.
Mass emissions - Table 2 below summarizes Che mass emissions resulcs,
derived from tests conducced ac ARB's El Honce laboratory using the 1975
Federal Test Procedure (FTP). The information presented in Che Cable
shows that when che ARB test resulcs were much higher Chan Che Lasc Smog
Check test, the vehicles had higher mass emissions of HC and CO than the
other vehicles. In addition, the reductions in mass emissions of HC and
-14-
-------
Figure 11
Average Idle HC Readings for Undercover Vehicles
With Last Smog Check Test Confirmed by ARB
(FTP Results, g/mi HC/CO/NOx, in parentheses)
800
ARBfaMMTMl
LolSmogCtMck
ARB AJtw-Smog CtMCk
(Z67/32.7/1.92)
ARB /dtar-Srnog rwflnga do net «xcMd Urt
Smog Qwek rvtOnqt by man ttMn 400 ppnVZOO *.
641 voWctav nwl ttw critafl&
Figure 12
Average Idle CO Readings for Undercover Vehicles
With Last Smog Check Test Confirmed by ARB
(FTP Results, g/mi HC/CO/NOx, in parentheses)
ARBMMTMl
MtMSmoflChKk UMSmagQMCk ARBAftw-SmogOwc*
(2.67/32.7/1.82)
ABB Mtw-Smeg r««J»B« do net «owd LMl
Smog Chock rMdlngi by man itan 400 Dpm/2.00%.
841
-15-
-------
800
700
600
E500
a.
0 400
1 300
200
100
Figure 13
Average Idle HC Readings for Undercover Vehicles
With Last Smog Check Test Much Lower Than ARB Retest
(FTP Results, g/mi HC/CO/NOx, in parentheses)
537
Actual Reduction (9.9%)
484
347
V-:'I. f'Sfc, V\^"
S^p::r.l|^
ARBMMTMl
(7.41/84. W2.00)
maw Smog ctwdc
LMtSmogOMek
ARBAftw^mogQwefc
(6.05/6S.(yi.79)
ARB Altar-Smog nodlngi nmd Imt Smog Qwck
rMdng» by «(MM 400 ppm HC or 2.00% CO.
190 v«NdM nwt ttM ottwla.
Figure 14
Average Idle CO Readings for Undercover Vehicles
With Last Smog Check Test Much Lower Than ARB Retest
(FTP Results, g/mi HC/CO/NOx, in parentheses)
4 -
*3
8
«
i 2
Actu* Reduction (1ZO% IncrMM)
3.36
100
2.04
ARBMtMTMt
(7.41/84.B«.00)
MMSmogQMCk
LMtSmogCtw*
ARB AAv-Smog Qwck
(8.66788.0/1.79)
ARB Altar-Smog rMdlngs «OMd LMl Smog ChKk
rMdngt by tt IMA 400 pom HC or 2.00% CO.
-16-
-------
Table 2
FTP Mass Emissions From Undercover Vehicles
Before and Afcer Smog Check Repair
Condicion
ARB-Smoe Check Tests Agree:
Initial (Baseline) Test at ARB
After-Smog Check Test at ARB
X Change, Initial (Baseline)
co After-Smog
ARB Test Results Much Higher:
Initial (Baseline) Test at ARB
Last Test at ARB
X Change, Initial (Baseline)
to Last Test
Mass Emissions, g/mi
HC
3.86
2.67
-30.8
7.41
6.65
-10.3
CO
40.5
32.7
-19.1
64.6
65.0
+0.5
NOx
2.20
1.92
-12.7
2.00
1.79
-10.5
CO for Che vehicles that had much higher test results at ARB were less
than the reductions achieved for the other vehicles. For example,
average HC mass emissions were reduced by 30.81 for the vehicles for
which the ARB and Last Smog Check results were in reasonable agreement,
but by only 10.32 for the "Suspect" vehicles. The results are even more
disparate for CO: mass emissions increased 0.5X on Che "Suspect"
vehicles, but were reduced an average of 19.12 for the other vehicles.
NOx mass emissions were reduced relatively little for both groups of
vehicles. However, "Suspect" vehicles showed smaller reductions Chan
Che other vehicles.
Figure 15 presents the mass emissions results graphically. Noce that
Che scale for CO emissions is on the right side of Che chare.
Model Year Trends - The incidence of "Suspecc" exhausc measuremenc as a
function of vehicle model year was investigated in an attempt Co
determine whether any pactems were apparent. As shown in Figure 16,
chere appears Co be only a weak correlacion between che occurrence of
"Suspecc" emission measuremencs ac Smog Check scations and model year
(and cherefore control technology).
Figure 16 shows that while che overall "Suspect" exhaust measurement
race was 20.41 (209 out of 1,027), it ranges from 8.3Z among 1987 model
year vehicles to 42.91 among 1970 model year vehicles. (None of the six
1989 models received "Suspect" exhaust measurements.) The average
"Suspect" exhausc measuremenc race for pre-1975 models is somewhac above
che overall race, ac 30.05Z. 1975-79 model year vehicles are obviously
below che overall race, che average for those five years being 13.7Z.
-17-
-------
Figure 15
Mass Emissions From Undercover Vehicles
Before and After Smog Check
AftvSmogQwck
• AR8and SmogOwck RnUtsXgra* —
MMTM MttrSmogQMck
— Afl8 Test Rasuds Mucti HigTMr —
60
50
40
I
30
20
10
0
Figure 16
Frequency of "Suspect" Exhaust Measurement
On the Last Smog Check Test
By Vehicle Model Year
Illlllllll
60
50
40
30
20
10
0
Model Year
-18-
-------
4. FTP Emissions Impact of Invalid Testing
To the extent that vehicles go unrepaired because of invalid test
results, the benefits of the I/M program are diminished. A first-order
approximation of the impact of invalid testing has been developed by
randomly replacing the after-repair emission levels of 54X of the
"Suspect" vehicles with the after-repair emission levels of the "other"
vehicles (54X being the fraction of vehicles in Che "Suspect" category
that cannot be explained based on test-to-test variability)". Tables 3-
7 show the results of this analysis and the intermediate steps in the
calculation.
Table 3 shows the FTP emissions by model year group and the VMT-weighted
composite emissions before and after Smog Check for all of the
undercover vehicles used in the analysis. As expected, the older
vehicles are shown to have significantly higher emissions. The VMT-
weighted emissions are close to the "Post-1979" average emissions
because of the large fraction of the fleet represented by that category
and the relatively high annual VMT rate for that category.
Table 3
Undercover Fleet — All Vehicle Emissions
By Model Year Range
Model Year
Range
Pre-75
75-79
Post-79
Sample
Size
167
238
427
Weighted Composite
Mass Emissions (g/mi)
Before Smog Check After Smog Check
HC
9.91
4.34
2.80
3.54
CO
70.94
48.20
35.05
39.78
NOx
2.76
2.51
1.71
1.93
HC
7.31
3.51
2.16
2.74
CO
66.11
42.88
28.41
33.50
NOx
2.50
2.14
1.51
1.69
* This analytical approach deals with all vehicles that passed the I/M
test because of invalid testing, not Just those vehicles that initially
failed the I/M test. The "Suspect" vehicles include vehicles that
passed the initial I/M test. Under the ARB program, defective vehicles
that initially passed were taken to a second Smog Check station. The
results of those second chance tests have been ignored in this analysis.
-19-
-------
Table 4 shows that the HC and CO FTP emissions of che vehicles where
invalid testing is suspected were higher than average to begin with and
not reduced nearly as much as the result of going through the Smog Check
program. Table 5 shows the corresponding results for the other
vehicles, which have significantly lower HC and CO FTP emissions to
begin with and greater emission reductions due to I/M.
Table 4
Undercover Fleec - "Suspect" Vehicle Emissions
By Model Year Range
Model Year
Range
Pre-75
75-79
Post-79
Sample
Size
61
43
101
Mass Emissions (g/mi)
Before Smog Check After Smog Check
HC
12.20
7.77
4.48
CO
77.55
85.07
53.34
NOx
2.74
2.18
1.54
HC
10.96
7.34
3.82
CO
76.97
84.23
52.74
NOx
2.57
1.90
1.43
Table 5
Undercover FleeC - "Other" Vehicle Emissions
By Model Year Range
Model Year
Range
Pre-75
75-79
Post-79
Sample
Size
106
195
326
Mass Emissions (g/mi)
Before Smog Check After Smog Check
HC
8.59
3.59
2.28
CO
67.14
40.07
29.38
NOx
2.77
2.58
1.77
HC
5.22
2.67
1.65
CO
59.85
33.77
20.87
NOx
2.47
2.19
1.54
Table 6 shows the effect of randomly substituting the After Smog Check
emission rates of the "other" vehicles for the After Smog Check emission
rates of 541 of the vehicles suspected of passing due to invalid
testing. Table 7 shows how the net emission reductions are affected by
this substitution. With no substitution (using the results presented in
Table 3), the emission reductions are 231 for HC, 161 for CO, and 12X
for NOx. Randomly replacing the after-repair emission levels of 54X of
-20-
-------
Che "Suspect" vehicles with the after-repair emissions from the other
vehicles increases the reduction in FTP emissions to 33X for HC and 26%
for CO. NOx emission reductions decline slightly, from 12X to 11X. It
should be noted that these emission reduction estimates are the single
I/M cycle estimates for should-fail vehicles only. Coincidentally, the
results are close to the overall benefits expected from the program
after accouncing for passing vehicles and the compounding of benefits
caused by multiple program cycles.
Table 6
Undercover Fleet - With 54Z of Randomly Selected
"Suspect" Vehicles Repaired
By Model Year Range
Model Year
Range
Pre-75
75-79
Post-79
Should
Fail
VMT
Frac
0.064
0.185
0.751
Weighted Composite
Mass Emissions (g/mi)
Before Smog Check After Smog Check
HC
9.91
4.34
2.80
3.54
CO
70.94
48.20
35.05
39.78
NOx
2.76
2.51
1.71
1.93
HC
6.18
3.06
1.89
2.38
CO
62.73
37.96
24.34
29.32
NOx
2.49
2.17
1.53
1.71
Table 7
Mass Emission Reductions
With and Without 541 of Randomly Selected
"Suspect* Vehicles Repaired
Model Year Range
Pre-75
75-79
Post-79
Weighted Composite
Mass Emission
Reductions (g/mi)
With "Suspect"
Vehicles Included
HC
26Z
19%
23Z
23Z
CO
7Z
111
19Z
16Z
NOx
91
15Z
12Z
12Z
: Mass Emission •
Reductions If 54Z
of "Suspect"
Vehicles Eliminated
HC
38Z
30Z
33Z
33Z
CO
12Z
21Z
31Z
26Z
NOx
1QZ
14Z
111
HZ
-21-
-------
Based on the results shown in Table 7, the overall benefits of the I/M
program would be expected to increase by at least 1.43 times for HC and
1.63 times for CO if apparently invalid testing could be eliminated.
Expressed in another way, invalid testing is estimated to cause a loss
in I/M benefits of 30X for HC and 39X for CO.
The maximum benefits of eliminating invalid testing are expected to be
higher than shown in Table 7 because this analysis only accounted for
the benefits of failing and repairing vehicles for which the apparent
falsification of results exceeded 400 ppm HC and 2.OX CO. Based on the
analysis of test-to-test variability, 11X of the defective vehicles
going through the I/M program are in this category. Based on the
variability analysis discussed earlier, another 6.5X of the defective
vehicles receive invalid test results in cases where the discrepancy
between the reported results and the actual results does not exceed the
400 ppm HC/2.0X CO margins. The benefits of correctly failing and
repairing these vehicles is probably somewhat less on a per-vehicle
basis.
The computations summarized above have also been performed for two other
substitution scenarios. Instead of randomly substituting results for
the "Other" vehicles for 54X of the "Suspect" vehicles, we have also
calculated what the effect would be of replacing the 54X dirtiest
"Suspect" vehicles in each model year group. This might be considered
an upper bound of what the effect of eliminating invalid testing might
be. At the other end of the spectrum, we have also computed the results
of replacing the 54X cleanest "Suspect" vehicles with the average
emissions from the "Other" category. The "dirtiest" and "cleanest"
vehicles were determined based on the sum of HC and NOx emissions plus
CO emissions divided by seven (the standard averaging technique used by
the California Air Resources Board). Tables 8 and 9 show the results
for replacing the 54X dirtiest. Tables 10 and 11 show the results for
replacing the 54Z cleanest.
Table 8
Undercover Fleet - With the 54X Dirtiest
of "Suspect" Vehicles Repaired
By Model Year Range
Model Year
Range
Pre-75
75-79
Post-79
Should
Fail
VMT
Frac
0.064
0.185
0.751
Weighted Composite
Mass Emissions (g/mi)
Before Smog Check After Smog Check
HC
9.91
4.34
2.80
3.54
CO
70.94
48.20
35.05
39.78
NOx
2.76
2.51
1.71
1.93
HC
5.41
2.76
1.70
2.13
CO
58.67
35.01
21.71
26.54
NOx
2.47
2.19
1.52
1.70
-22-
-------
Table 9
Mass Emission Reductions
With and Without 54X Dirtiest "Suspect" Vehicles Repaired
Model Year Range
Pre-75
75-79
Post-79
Weighted Composite
Mass Emission
Reductions (g/mi)
With "Suspect"
Vehicles Included
HC
26X
19*
23X
23X
CO
7X
11X
19X
16X
NOx
9X
15X
12X
12X
Mass Emission
Reductions If 54X
of "Suspect"
Vehicles Eliminated
HC
45X
36X
39X
40X
CO
17X
27X
38X
33X
NOx
10X
13X
11X
11X
Table 10
Undercover Fleet - With the 541 Cleanest
of "Suspect" Vehicles Repaired
By Model Year Range
Model Year
Range
Pre-75
75-79
Posc-79
Should
Fail
VMT
Frac
0.064
0.185
0.751
Weighted Composite
Mass Emissions (g/mi)
Before Smog Check After Smog Check
HC
9.91
4.34
2.80
3.54
CO
70.94
48.20
35.05
39.78
NOx
2.76
2.51
1.71
1.93
HC
7.02
3.38
2.07
2.63
CO
66.18
40.88
26.86
31.97
NOx
2.51
2.15
1.54
1.71
-23-
-------
Table 11
Mass Emission Reductions
With and Without 54% Cleanest "Suspect" Vehicles Repaired
Model Year Range
Pre-75
75-79
Post-79
Weighted Composite
Mass Emission
Reductions (g/mi)
With "Suspect-
Vehicles Included
HC
26X
19X
23X
23X
CO
7X
11X
19X
16X
NOx
9X
15X
12X
12X
Mass Emission
Reductions If 54X
of "Suspect"
Vehicles Eliminated
HC
29X
22X
26X
26X
CO
7X
15X
23X
20X
NOx
9X
14X
11X
11X
As shown in Table 9, replacing the 54X dirtiest vehicles increases the
composite emission reductions from 33X to 40X for HC and from 26X to 33X
for CO. NOx emission reductions are the same as in the random
substitution case (11X). As shown in Table 11, replacing the 54X
cleanest vehicles reduces the composite emission reductions to from 33X
to 26X for HC and from 26X to 20X for CO. Again, NOx emission
reductions are the same as in the random substitution case.
-24-
------- |