EPA-AA-TSS-I/M-87-l Technical Report A Discussion of Possible Causes of Low Failure Rates in Decentralized I/M Programs By Eugene J. Tierney January 1987 NOTICE Technical Reports do not necessarily represent final EPA decisions or positions. They are intended to present technical analysis of issues using data which are currently available. The purpose in the release of such reports is to facilitate the exchange of technical information and to inform the public of technical developments which may form the basis for a final EPA decision, position or regulatory action. Technical Support Staff Emission Control Technology Division Office of Mobile Sources Office of Air and Radiation U. S. Environmental Protection Agency ------- ABSTRACT This technical report reviews six possible explanations for low reported failure rates in manual, decentralized I/M programs. The report analyzes and discusses random roadside idle survey data, reported I/M program data and data collected during audits of I/M programs. The data indicate that five of the explanations: quality control, fleet maintenance, differences in fleet mix or emission standards, anticipatory maintenance, and pre-inspection repair, do not sufficiently explain low reported failure rates. The report concludes that the major problem contributing to low reported failure rates in decentralized, manual I/M programs is improper inspections by test station personnel. ------- TABLE OF CONTENTS SECTION PAGE Abstract i Table of Contents ii List of Tables iii List of Figures iv Introduction 1 Quality control issues 4 Better maintained fleet 6 Standards and coverage differences 6 Anticipatory maintenance 7 Pre-inspection repair 9 Improper Inspection 9 Conclusions 25 References 26 11 ------- LIST OF TABLES TABLE TITLE PAGE 1 Failure Rates in I/M Programs 2 2 Potential Causes For Lower Than Expected Failure Rates 3 3 Potential Impact of Quality Control Deficiencies on I/M Failure Rates 5 4 Non-I/M Vehicle Failure Rates Using Consistent Emission Standards 6 5 Reported Failure Rates vs. Survey Failure Rates 10 111 ------- LIST OF FIGURES FIGURE TITLE PAGE 1 Comparison of Idle Surveys 11 2 Frequency Distribution of Emission Scores 13 3 Cumulative Distribution of Emission Scores 14 4 Initial Test Scores Cumulative Distribution Model Year 1977 15 5 Retest Scores for Connecticut versus Initial 16 Test Scores, Model Year 1977 6 Initial Test Scores Cumulative Distribution Model Year 1980 17 7 Retest Scores for Connecticut versus Initial 18 Test Scores, Model Year 1980 8 Initial Test Scores Cumulative Distribution Model Year 1982 19 9 Retest Scores for Connecticut versus Initial 20 Test Scores, Model Year 1982 10 Repeat Index - Colorado 22 11 Repeat Index - Virginia 23 IV ------- INTRODUCTION Inspection and maintenance (I/M) programs are currently operating in thirty-one states and affect approximately one third of all light duty cars and light duty trucks in the country. The annual inspection cost of these programs is in the neighborhood of $500 million. Given the significant impact, it is important to carefully assess the outcomes of these programs on an individual basis with an eye toward effectiveness and cost-efficiency. The object of an I/M program is to identify vehicles that are "gross emitters" of hydrocarbons (HC) and/or carbon monoxide (CO) and require emission-related repairs of those vehicles such that emissions are reduced. Thus, one important indicator of program effectiveness is the percentage of vehicles that are identified as gross emitters (i.e., fail the emissions test). Two basic approaches to inspecting vehicles have been implemented among I/M programs: centralized and decentralized. In centralized programs, motorists bring their vehicles to high volume test facilities operated by the state or local government or by a contractor hired by the state or local government. The repair function is independent of the test function and the centralized facilities are generally highly automated and systematic. Decentralized I/M programs generally have few or no high volume stations. The state or local government licenses service stations, automobile dealerships and the like to do inspections. Motorists have the option of obtaining repairs at the licensed facility or going elsewhere. Two distinct types of decentralized inspection programs exist: ones that use manual emission analyzers and ones that use computerized analyzers. In the latter case, a computer is built into the analyzer that controls the test procedure, the selection of emission standards, the pass/fail decision, data recording, and quality control. In the case of manual analyzers, no computer is available; so, the inspector is responsible for quality control, chooses emission standards, reads meters or digital displays for emission levels, decides pass/fail status, and records the test data. In its role as an oversight agency, EPA has been conducting audits under the National Air Audit System guidelines and has been gathering data collected by individual I/M programs. EPA has also been conducting random roadside tampering and idle surveys in cities throughout the country for many years. Analysis of these data has revealed some significant findings: first, emission test failure rates in I/M programs vary widely, from a low of about 2% to a high of 28% (see Table One); second, failure rates vary by program type. Decentralized programs with manual analyzers tend to have very low failure rates while centralized programs and decentralized programs with computerized analyzers tend to have much higher failure rates. ------- Table One EMISSION TEST FAILURE RATES IN I/M PROGRAMS* REPORTED EXPECTED RATIO CENTRALIZED (CO Arizona 20.2 36.8 .55 Connecticut 17.2 33.0 .52 Delaware 13.7 7.7 1.00 Kentucky 15.7 16.2 .97 Maryland 14.6 14.0 1.00 Memphis, TN 8.1 3.7 1.00 Nashville, TN 24.5 25.4 .97 New Jersey 26.1 27.8 .94 Oregon 24.0 38.3 .63 Washington, D.C. 18.4 13.4 1.00 Washington 19.0 28.1 .68 Wisconsin 15.3 19.3 .79 DECENTRALI ZED Computerized Analyzers (DC) Alaska Fairbanks 19.4 22.7 .85 Anchorage 15.7 24.7 .63 California 27.7 28.7 .96 Michigan 15.8 12.9 1.00 New York** 5.1 33.4 .15 Pennsylvania 17.6 19.5 .90 Manual Analyzers (DM) Georgia 6.6 25.0 .26 Idaho 9.8 16.9 .58 Missouri 6.7 20.5 .33 North Carolina 5.6 21.1 .27 Nevada Clark County 9.5 29.4 .32 Washoe County ll.O 29.4 .37 Utah Davis County 8.7 21.3 .41 Salt Lake County 10.0 21.3 .47 Virginia 2.3 15.6 .15 FOP all model yea^S, including light duty trucks. New York's analyzers are only partially computerized ------- Table One lists reported failure rates from the most recent year available for each program, expected failure rates and the ratio of reported to expected failure rates. The reported failure rates are provided to EPA in several ways: as lump sum failure rates for all model years, as failure rates by model year, or as failure rates by model year group. In the latter two instances, national default registration distributions were used to weight separate model year or group failure rates together into one overall failure rate. As a result, the reported overall failure rate here may differ slightly from the actual overall failure rate experienced in each program. EPA does not have available to it the distribution of vehicles tested to make more precise calculations. The expected failure rates are based on the emission standards used in the program applied to the 1984 Louisville I/M data base. This data base was chosen because it is the best available data base for this purpose. It represents the non-I/M fleet and covers light-duty cars and light-duty trucks. The failure rate ratio is the reported failure rate divided by the expected failure rate (in the few cases where this yields a number greater than one, the result was rounded down to one). This report discusses six potential causes for the differences in reported failure rates among I/M programs (see Table Two) and cites available data to support or cast doubt upon them. The data come from a variety of sources: audit reports conducted under the National Air Audit System guidelines; reported I/M program failure rates, emissions scores and other data; random roadside idle emission surveys conducted by EPA; and, various contractor studies conducted for EPA. Analysis of I/M program data is limited by the program design: how data is handled and reported to EPA limits the kinds of analyses that can be conducted. Since resource constraints naturally exist, contractor studies have not generally involved analyses of all I/M programs but rather selected programs that are representative of the different types of programs. As a result, the findings must be extrapolated to other programs of a similar type. Table Two POTENTIAL CAUSES FOR LOWER THAN EXPECTED FAILURE RATES 1) Quality control issues 2) Better maintained fleet 3) Standards and coverage differences 4) Anticipatory maintenance 5) Pre-inspection repair 6) Improper inspection ------- QUALITY CONTROL ISSUES Non-dispersive infra-red analyzers are used in all I/M programs to determine emission levels of hydrocarbons and carbon monoxide from motor vehicles subject to the program requirements. Proper calibration of emission analyzers is essential for obtaining accurate test results. Quality control requirements vary somewhat from program to program but several common elements exist: 1) Weekly calibration of analyzers 2) Low range calibration gas 3) Weekly leak check 4) Periodic audit of analyzers by program officials 5) Zero and span within one hour of each test In programs with computerized analyzers, some quality control functions are done automatically and software protection exists that prevents use of the analyzer unless the leak check and gas calibration check have been conducted and passed within the past seven days. The analyzer software guides the inspector through the steps necessary to complete quality control checks thereby insuring consistency and accuracy. EPA audits of computerized analyzer programs usually show few analyzers failing quality control checks. In programs with manual analyzers, quality control is done manually and it is up to the inspector to insure that it gets done. Typically, programs specify that the weekly gas calibration and leak check take place each Monday morning. Nothing exists to prevent use of the analyzer if the quality control functions are not performed except periodic audits by program officials. EPA audits have shown that analyzers in manual programs are frequently out of calibration, possess leaks, or have other problems that can severely compromise test quality (e.g. clogged filters). Quality control for calibration gas is accomplished two ways in I/M programs: through periodic station audits and calibration gas specifications. Most programs conduct monthly audits and the auditors carry gas cylinders to check analyzer accuracy. This also accomplishes a check on calibration gas accuracy because once erroneous calibration is eliminated continued failure of an analyzer usually indicates a gas problem. Calibration gas specifications are fairly consistent among I/M programs. Most specify an accuracy of + 2% and require specific concentrations (i.e., a zero blend tolerance) of 1.6% CO and 600 ppm HC in nitrogen. Quality control lapses will diminish the accuracy of emission scores and, to some degree, will alter the pass/fail outcome. The question is whether quality control lapses could explain the low reported failure rates experienced in some I/M programs. There are three sources of data that will help answer this question: audit data, idle survey data, and operating data. ------- If analyzers are typically reading low or .have significant leaks in the sample system, vehicles with emission scores close to standards may pass the test when they should fail. A review of the gas audit results obtained during EPA audits in Georgia, North Carolina, Idaho and Missouri'1' shows that, on average, analyzers were 2% out of calibration on the high side. This means that vehicles that have emission scores within that range will tend to incorrectly fail rather than incorrectly pass. An analysis of the Virginia I/M data was also conducted to determine the impact of increasing all emission scores by 5%. Five percent was chosen because the audit data showed that over 80% of the emission analyzers checked were within the 5% tolerance; it is also the audit tolerance used in most I/M programs. The overall failure rate for the 1975 through 1984 vehicles sampled for this analysis increased from 3% to 4.2% when emission scores were increased 5%. Finally, random, roadside idle survey data(2' were analyzed to determine the impact of lowering emission scores by 100 ppm HC and 1% CO to simulate the results which would have been obtained if the analyzers were severely out of calibration or had gross leaks in the sampling system. Table Three shows, for all model years, the idle survey failure rates and the idle survey failure rates with the cushion added. Note that no dramatic drop in failure rates occurs as a result of a cushion and that survey failure rates with the cushion are still much higher than reported rates in manual I/M programs. These three analyses show that low reported failure rates are not explained by typical quality control deficiencies in manual I/M programs. STATE Connecticut Missouri New Jersey New York North Carolina Pennsylvania Virginia Table Three POTENTIAL IMPACT OF QUALITY CONTROL DEFICIENCIES ON I/M FAILURE RATES PROGRAM TYPE CC DM CC DC DM DC DM REPORTED FAIL RATE 17.2% 6.7% 26.1% 5.1% 5.6% 17.6% 2.3% SURVEY FAIL RATE 16 16 34 22 17 18 16 0% 2% 1% 2% 6% 4% 0% SURVEY RATE WITH GUSH I ON 13.2% 11.2% 27. 1% 19. 1% 14.7% 15.3% 13.5% ------- BETTER MAINTAINED FLEET There is no evidence to indicate that mechanic effectiveness varies significantly from region to region or that vehicle owners are more conscientious about getting repairs in one state or another. Nevertheless, it is conceivable that better general maintenance in an area could result in a cleaner fleet, overall, and lower than expected I/M failure rates. If this were the case, then there should be a significant difference between failure rates of non-I/M survey vehicles (i.e. those registered outside the program boundaries but surveyed while operating within the I/M boundaries) among I/M areas. Table Four illustrates the survey failure rates at constant cutpoints (pre-1981: 3.0% CO/300 ppm HC; post-80: 1.2% CO/220 ppm HC) for non-I/M vehicles in two groups of areas where EPA has conducted random, roadside idle surveys. The data were grouped due to the small sample size of non-I/M vehicles in I/M areas. The members of the group were determined based on reported failure rates, VA, et.al. being low and PA, et.al. being high. Note that the failure rates are very similar between the two groups. This indicates that maintenance differences between areas does not seem to be influential on non-I/M vehicle failure rates. By extension, this is likely to be true of I/M vehicles in these areas as well. Table Four NON I/M VEHICLE FAILURE RATES USING CONSISTENT EMISSION STANDARDS MODEL YEARS VA,NC,MO,NY PA,CT,NJ,OR Post 1980 Pre-1981 Overall 7.1% 50.6% 20.7% 6.4% 54.7% 24.2% EMISSION STANDARD AND VEHICLE COVERAGE DIFFERENCES Emission standards are established by each program and are used to determine whether a vehicle must be subjected to repairs to bring emission levels down to acceptable levels. No two I/M programs have identical emission standards for all model years, although most programs use the same standards for 1981 and later vehicles (1.2% CO and 220ppm HC) . The difference in standards from program to program should result in different failure rates. Two other important factors that contribute to expected differences in overall failure rates among programs are model year coverage and vehicle type coverage. Thus, it is conceivable that low failure rates in some I/M programs may be due to one or a combination of these factors. ------- To evaluate this question, the emission standards and vehicle coverage from every I/M program were applied to a common data base consisting of emission distributions from the Louisville I/M program'3'. The Louisville program data is for calendar year 1984 and represents the non-I/M fleet since the Louisville program started January l, 1984. Table One lists the expected failure rates for twenty-one I/M programs. The expected failure rates range from a low of 3.7% in Memphis, Tennessee to a high of 38.3% in Portland, Oregon. Expected failure rates for most I/M programs fall into the 20% to 40% range. In particular, the expected failure rates for manual decentralized programs also fall into this range, with the exception of Virginia and Idaho which have expected failure rates of 15.6% and 16.9%, respectively. It is clear that, based on this analysis, the combination of emission standards, vehicle coverage and model year coverage yields a range of expected failure rates but does not explain the low reported failure rates of decentralized manual I/M programs. To evaluate this question further, the ratios of reported failure rates to the expected failure rates were calculated. The third column in Table One shows the ratio of reported to expected failure rates, hereafter referred to as failure rate fractions. The failure rate fractions in manual decentralized programs are all under 0.4, except Idaho and Utah. Most other I/M programs have failure fractions over 0.7. Some differences in failure rate fractions are anticipated due to variations in historical emission standards, tampering program coverage, waiver rates, pre-conditioning, and length of operation of the program. For example, the emission standards used throughout the life of the program are very significant. Note in Table One that, among the centralized programs, the three with the lowest failure rate fractions also have the highest expected failure rates. These programs have a history of tight emission standards which has led to lower current failure rates than otherwise expected. The failure rate fractions in centralized programs reflect normal program variations. However, these variations are never large enough to cause the low failure rates in manual, decentralized programs. Thus, the normal range of failure rate fractions also shows that differences in emission standards and model year coverage do not explain low failure rates in manual decentralized I/M programs. ANTICIPATORY MAINTENANCE Anticipatory maintenance occurs when a motorist obtains repairs on a vehicle due to have an inspection in the near future with the intent of avoiding test failure but without the knowledge that the vehicle would, in fact, fail the test. There is no doubt that anticipatory maintenance occurs, but it is not clear to what extent it occurs and there is no evidence to show it occurs more frequently in one program or another. It is conceivable that anticipatory maintenance could contribute to low initial test failure rates. ------- Data collected by EPA indicate that anticipatory maintenance may not achieve its intended goal; in fact, some evidence indicates it will increase chances of test failure! In 1979 and 1980, EPA conducted several studies(4•s • *} to determine the potential emission benefits of a mandatory vehicle maintenance program. In these studies, mechanics were asked to adjust vehicles to manufacturer specifications. The mechanics were not aware that they were being tested; some of the vehicles were out of adjustment (i.e., Federal Test Procedure (FTP) failures) and some were not. In Houston, adjustment by mechanics resulted in a 2.3% increase in FTP mass HC emissions and a 2.7% increase in FTP mass CO emissions. In a St. Louis study, an 87% increase in idle HC and a 30% increase in idle CO emissions were observed in 83 shops. In general, vehicles that were FTP failures before "repairs" showed some reduction in emissions while clean cars typically suffered emission increases. The two situations studied here are analogous to that of anticipatory maintenance: mechanics are not being asked to fix the vehicle in response to an I/M failure. Even when repairs occur in response to an I/M failure, successful emission reductions are not assured. Retest failure rates in centralized I/M programs are typically in the 30-40% range. Thus, it is reasonable to assume that anticipatory maintenance may not be very successful, especially when applied to "clean" cars. Another reason anticipatory maintenance does not seem to be a satisfactory explanation for low failure rates in decentralized manual I/M programs is that its effect, if important, should be felt in other types of programs as well. In fact, it is arguable that the effect should be greater in centralized programs. In decentralized programs, the testing and repair functions are combined in the service station environment. The normal sequence of events is for a customer to visit the garage, get an emissions test and if a failure occurs, to obtain repairs at that facility. Given this scenario, there is little motive for anticipatory maintenance. In centralized programs, where repair functions are separate from testing functions, the motorist has to make at least three trips if an initial test failure occurs. This provides an incentive to avoid initial test failure especially when a failure was experienced in a previous year. Data from the Arizona and Seattle, Washington I/M programs show that, in those centralized programs, vehicles that failed in the previous year fail at higher than average rates. This implies that motorists are not attempting to avoid failures through anticipatory maintenance or that such maintenance is unsuccessful. ------- PRE-INSPECT I ON REPAIR Pre-inspection repair may occur in two basic ways: first, a vehicle is brought in for an emission test and an unofficial initial test is conducted to determine if the vehicle will pass. If the vehicle fails, repairs are conducted such that it' will pass and then the official initial test follows. The second scenario occurs when a vehicle is brought in for a tune-up plus an emission test and, again, repairs precede the official test. It is likely that this phenomenon occurs in both manual and computerized decentralized I/M programs. To the extent that this is the case, it would lower initial test failure rates but is a phenomenon confined to decentralized programs. The structure and requirements in manual and computerized decentralized I/M programs are essentially the same with exception of the analyzer. Thus, the opportunities and incentives for pre-repair are about the same in both types of program. So, the impact of pre-repair in terms of failure rates should be the same as well. However, the data in Table One show that decentralized programs with fully computerized analyzers do not experience the very low failure rates of decentralized manual programs. This indicates that pre-inspection repair does not seem to be a big issue in computerized programs. The software prompts in the Michigan computerized analyzer include a question regarding repairs within a week preceding the initial test. The data* } for the first and second quarters of 1986 show that for vehicles passing the initial test, about 6-7% were known to have received repairs within the past week. It is not known how many of these vehicles received anticipatory maintenance as opposed to pre-inspection repair. It is also unknown whether the repairs were actually needed to pass the initial test or effective at reducing emissions. In any case, these data indicate that, taken together, pre-inspection repair and anticipatory maintenance are not common phenomena and therefore are not satisfactory explanations for low reported failure rates. IMPROPER INSPECTION Improper inspection is believed to occur in several ways: inspectors skip the emissions test and invent passing emission scores; inspectors conduct the test but still invent passing emission scores without doing repairs; inspectors conduct the test, do repairs but, failing to bring the vehicle into compliance, they invent passing emission scores. We can imagine a host of variations on these three basic scenarios, but all have one important factor in common: emission reductions are not achieved. Improper inspection has been found to occur in I/M programs through covert audits. What is unclear is the magnitude of the problem. ------- One way to assess whether emission reductions are not actually occurring is to randomly test I/M vehicles. EPA has conducted random idle surveys throughout the country. Figure One, on the next page, illustrates the results for four eastern states: North Carolina, Connecticut, Pennsylvania and Virginia. The graph lines illustrate the failure rate of 1981 and later vehicles (using each program's own outpoints) over time since last inspection. Note that the rates for North Carolina and Virginia are relatively flat - the failure rates start high and end high. On the other hand, the failure rates for Pennsylvania and Connecticut start low and end high. The symbols on the right Y axis are the reported failure rates for the four programs. The reported failure rates in Pennsylvania and Connecticut are essentially identical to the 12 month survey failure rates. The reported rates for Virginia and North Carolina are much lower than the survey rates. Two things are apparent from these results: vehicles sampled in the survey in Virginia and North Carolina do not seem to have their emissions lowered after inspection, and the reported failure rates do not accurately reflect the actual idle emissions of sampled vehicles. The data presented in Table Five show a comparison of overall I/M failure rates and reported failure rates for cars in the same four states. The reported failure rate in Pennsylvania and Connecticut is very similar to the survey failure rate. In North Carolina and Virginia, the reported failure rates are much lower than the survey failure rates. Again, the data indicate that I/M cars are not achieving significant emission reductions. Table Five REPORTED FAILURE RATES VS. SURVEY FAILURE RATES STATE North Carolina Virginia Pennsylvania Connecticut PROGRAM TYPE DM DM DC CC REPORTED 5.6% 2.3% 17.8% 17.2% LOCAL SURVEY I/M CARS 17.8% 14.9% 18.2% 16.4% An EPA contractor was given a work assignment in 1986 to analyze I/M program data to determine if significant differences were present between different I/M programs. The draft report'8) from this work has been completed. The contractor analyzed and compared reported data from I/M programs in Washington, Virginia, New York, Colorado and Connecticut (an analysis of Massachusetts data is in progress and will be available in early 1987). 10 ------- Figure One Comparison of Idle Surveys 1981 And Newer Passenger Cars Site O CT © a PA ffl o NC » A VA A 23456789 Months Since Last Inspection 11 12 Survey Reported ------- Three representative model years were analyzed to reduce the enormity of the task: 1977, 1980 and 1982. Manufacturers were broken up into three groups: Gl consists of Chrysler, Ford and AMC; G2 consists of only GM vehicles; and, G3 consists of all imports. Figure Two provides an example frequency distribution of carbon monoxide failure rates from four programs. The X axis is the CO emission score and the Y axis is the percent with that score in the sample. The Virginia and Colorado data are very "spiky" since the emission readings are manually recorded and mechanics appear to round off the readings. Note also that the decentralized programs show a step change in the distributions exactly at the program cutpoints. Additionally, the distributions show a very large number of vehicles (relative to Washington) just below the cutpoints. This distribution is typical of what was found in other model years, in other vehicle groups, and for HC. The contractor also produced cumulative distributions to overcome the distortion created by the spikes in the data. In these curves, the gradient of the curve is proportional to the number of vehicles at the particular emission level. Figure 3 shows the distinctive kink in the curves at the program cutpoints except in Washington, which shows a smooth distribution. To further assess the question of whether the "kink in the curves could be due to pre-repair, the contractor compared the first test results from the decentralized programs with two sets of results: 1) the first test results of centralized programs. 2) the first test results of centralized programs for passing vehicles and the retest results for failed vehicles combined into one distribution. The latter comparison is intended to simulate the distribution resulting from pre-repair. Figures 4 through 9 show the results of these analyses. The contractor's analysis of these figures succinctly states the case: From Figure Five it is obvious that Connecticut's vehicles have much lower emissions after repair, and the New York, Colorado and Virginia curves lie between the Connecticut first test (Figure Four) and after repair distributions (Figure Five) . This may indicate that a fraction of the cars are being pre-repaired and that the average lies between the two Connecticut distributions. However, examination of the 1980 and 1982 distributions in Figures Six to Nine shows that this hypothesis is unlikely to be correct. For both model years the Connecticut's initial test distributions shows lower CO values than the decentralized program distributions. Moreover, a substantial portion of the population in each decentralized program appears to have CO emissions just below the cutpoint (as indicated by the steepness of the line just below the CO cutpoint). It is [usually] not possible to repair 1980 and 1982 cars to "just meet" the 12 ------- 20.00 -r 18.00 16.00 14.00 4- 12.00 4- 10.00 4- 8.00 + 6.00 4- 4.00 4- 2.00 4- o.oo Figure Two FREQUENCY DISTRIBUTION OF REPORTED EMISSION SCORES IN THREE DECENTRALIZED AND ONE CENTRALIZED I/M PROGRAM Carbon Monoxide Frequency Distribution WA'77 G1 NY77G1 VA'77G1 CO77G1 0-50 1.50 2.50 3.50 4.50 5.50 6.50 7.50 8.50 9.50 10.50 11.50 Encfeoint ------- 100.00 -r 95.00 + 90.00 4- 85.00 4- Figure Three CUMULATIVE DISTRIBUTION OF REPORTED EMISSION SCORES Cumulative CO Frequency Distribution ^ Cumulative sum of - WA77G2 •"• Cumulative sum of NY77G2 ••• Cumulative sum of VA77G2 esss* Cumulative sum of CO77G2 1.50 2.50 3.50 4.50 5.50 BCPCNT 6.50 7.50 8.50 9.50 10.50 11.50 ------- 100,00 -r 98.00 4- 96.00 4- 94.00 4- 92.00 4- 90.00 4- 88.00 4» 86.00 Figure Four INITIAL TEST SCORES CUMULATIVE DISTRIBUTION MODEL YEAR 1977 Cumulative CO Frequency Distribution NY CO •" Cumulative sum of CT77G2 •« Cumulative sum of NY77G2 "•• Cumulative sum of CO77G2 *"" Cumulative sum of VA77G2 I I I I I I I I I I I I I I 0.50 1.50 2.50 3.50 450 5.50 6.50 7.50 8.50 9.50 10.50 11.50 ------- Figure Five PERCENT 100 -r CONNECTICUT RETEST SCORES VERSUS INITIAL TEST SCORES IN DECENTRALIZED PROGRAMS MODEL YEAR 1977 Cumulative CO Frequency Distribution 98 4- 96 4- 94 + 92 + 90 + 88 + 86 * i i i — CT77COG2- MCQOJM. «««• Cumulative sum of NY77G2 •"•• Cumulative sum of CO77G2 SSM* Cumulative sum of VA77G2 I I I I I I I I I I I 0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 ------- Figure Six PERCENT 100.00 -r INITIAL TEST SCORES CUMULATIVE DISTRIBUTION MODEL YEAR 1980 Cumulative CO Frequency Distribution 98.00 -• 96.00 -• 94.00 - - 92.00 90.00 -• Cumulative sum of CT '80 G2 Cumulative sum of NY '80 G2 . 88.00 Cumulative sum of CO '80 G2 Cumulative sum of VA '80 G2 I I I I I I 0.25 0.75 1.25 1.75 2.25 2.75 3.25 3.75 4.25 4.75 5.25 5.75 6.25 6.75 BCPONT ------- Figure Seven PEFCBJT 100 -r CONNECTICUT RETEST SCORES VERSUS INITIAL TEST SCORES IN DECENTRALIZED PROGRAMS MODEL YEAR 1980 Cumulative CO Frequency Distribution 98 - - 96 -• 94 -• 92 90 -- 88 CT'SOCO G2- MOO.CIM Cumulative sum of NY '80 G2 Cumulative sum of CO '80 G2 Cumulative sumofVA '80 G2 0.25 0.75 1.25 1.75 2.25 2.75 3.25 3.75 BCPONT 4.25 4.75 5.25 5.75 6.25 6.75 ------- Figure Eight PERCENT 100.00 - INITIAL TEST SCORES CUMULATIVE DISTRIBUTION MODEL YEAR 1982 Cumulative CO Frequency Distribution 98.00 -- 96.00 -- 94.00 -- 92.00 -- 90.00 -• 88.00 Cumulative sum of CT82CQG2 Cumulative sum of NV82COG2 Cumulative sum of C082COG2 Cumulative sum of VAB2COG2 0.125 0.625 1.125 1625 2.125 BCPCNT 2.625 3.125 3.625 ------- PERCENT 100.00 Figure Nine CONNECTICUT RETEST SCORES VERSUS INITIAL TEST SCORES IN DECENTRALIZED PROGRAMS MODEL YEAR 1982 Cumulative CO Frequency Distribution 98.00 96.00 4- "CTB2COG2- MCO.OJM. ™" Cumulative sum of NYB2COG2 94.00 4- 92.00 4- 90.00 88.00 *•• Cumulative sum of COB2COG2 «•«. Cumulative sum of VAB2COG2 0.125 0.625 I I I I I I I I I I I I 1.125 1.625 2.125 B^DPONT 2.625 3.125 3.625 ------- CO standard as they have sealed adjustments for the carburetor and, in many cases, sophisticated electronic controls that are either operational or malfunctioning, with no "in-between" states. Finally, it must be noted that Connecticut's post-repair distribution for any of three model years considered do not show a large group of cars below the applicable cutpoint. This indicated that even if some form of pre-repair occurs in decentralized programs, the repairs do not appear to be complete or related to the defects in the emission control system. Two important conclusions drawn from this data analysis are that: 1) Emission distributions from decentralized programs do not resemble those from centralized programs. In decentralized programs the distributions exhibit a distinctive discontinuity at program cutpoints. 2) Pre-repair is not a satisfactory explanation for the shape of the emission distribution curves, particularly for newer model years, in decentralized programs. The contractor devised two indices to attempt to further distinguish pre-repair and improper inspection. The abnormal freguency (ABF) index is the percent of cars that have emission scores 0.7 to 1.0 times the cutpoint. This index is based on the finding that an abnormally large percentage of the fleet in decentralized programs is reported to have emission scores in a narrow emissions range just below the cutpoints. The index for Washington and Connecticut was approximately 0.15 for 1975 to 1980 vehicles using the 3.0% CO/300 ppm HC cutpoints, dropping to 0.09 using 6.0% CO/600 ppm HC cutpoints. Analysis of New York, Colorado and Virginia data showed that 40-60% of stations analyzed have high ABF indices. The second index devised is the repeat index. In reviewing I/M station records, EPA auditors have observed reported emission scores repeated again and again at a given station. The contractor devised a repeat index by counting the number of times each HC and CO value is repeated and the three highest counts for HC and CO (six in all) are added. This number is then normalized by the sample size to derive the Repeat Index. By way of example, if all emission readings are reported as one of any three values for HC and CO (an extreme case), the calculated Repeat Index would equal 2. The contractor suggests a level of 0.5 to 0.6 as criteria for identifying potentially dishonest stations. Analysis of the Colorado and Virginia data showed indices ranging as high as 1.2. Figures Ten and Eleven illustrate the results for all stations examined. 21 ------- Figure Ten REPEAT INDEX - COLORADO Frequency Bar Chart MIDPOINT RPT.IDX 0.025 0.075 0.125 0.175 0.223 0.275 0.325 0.375 0.4Z5 0.475 0.525 0.575 0.625 0.675 0.725 0.775 0.825 0.875 0.923 0.973 1.025 1.075 1.125 1.175 FREQ 0 0 1 •••••* 16 ••••••.•••••••••••ft.** 56 •••••••»•»•••••••••••••••••€••• 78 ••••••••••••••••••••••••••••••••I*** 91 152 170 195 164 ••••••••••••••••••••••••••••»« »**•**•»«•*•*»•*•**••»* 132 •••••••••••••••••••••••t***»*« **»**•«**•*****»** 119 • 70 70 *«••*•«•«§*••*»**•* 47 *•*••***•**** 32 ••••••*** 23 • •## 10 • •*• 9 ** 6 • • 3 • 3 • * 4 4 * » » * •» « * * * * 4 +___*__-* .«-..>. 4.---*-- CUtt. FREQ 0 0 1 17 73 151 242 394 564 759 923 1055 1174 1244 1314 1361 1393 1416 1426 1433 1441 1446 1449 1453 PERCENT 0.00 0.00 0.07 1.10 3.85 5.37 6.26 10.46 11.70 13.42 11.29 9.08 8.19 4.82 4.SC 3.23 2.20 1.58 0.69 0.62 0.41 0.34 0.21 0.28 CUM. PERCENT 0.00 0.00 0.07 1.17 3.02 10.39 16.66 27.12 38.82 52.24 63.32 72.61 80.80 85.62 90.43 93.67 95.87 97.45 98.14 98.76 99.17 99.32 99.72 100.00 FREQUENCY ------- Figure Eleven REPEAT INDEX - VIRGINIA MIDPOINT RPT_IDX 0.025 0.075 0.125 0.175 0.225 0.275 0.325 0.375 0.4T3 0.47b 0.525 0.575 0.625 0.675 0.725 0.775 0.825 0.875 0.925 0.975 1.025 1.075 1.125 1.175 Frequency Bar Chart FREO 0 0 ****** 3 »•*••**•»**• 6 «*** •••* 10 .... 2 **•••«*••*•»*• 7 ••••••••••••i* 7 22 •••••••••••I***...***********.**.***.*********.*** 25 «•*•««•»*•**•••••••***••••*•**•***•**•**••***•*•*» 25 31 •••«.••••«••«*••***••*•*»•******•*•*•*.«*«•*••«•*•• 25 * * 35 .........*.*. 33 ••••A*******************.****..**... 16 ••*••**••**•***••***»•••*•*•»•**•*«***»* 20 •**»•*****•***•»****.* II **••*•**••*»••••*« 9 •••••******* . 6 A******************* 10 •**••*•» 4 **»• 2 •••••••ft************ 10 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 CUM. FREQ 0 0 3 9 19 21 28 35 57 82 107 138 163 198 231 :47 269 280 289 295 305 309 311 321 PERCENT 0.00 0.00 0.93 1.87 3.12 0.62 2.18 2.18 6.85 7.79 7.79 9.66 7.79 10.90 10. re V6I 6.23 3.43 2.80 1.87 3.12 1.25 0.62 3.12 CUM. PERCENT 0.00 0.00 0.93 Z.80 5.92 6.54 8.72 10.90 17.76 25.55 33.33 42.99 50.78 61.68 71.96 77.97 83.80 87.23 90.03 91.90 95.02 96.26 96.88 100.00 FREQUENCY ------- In addition to the extensive data analysis just reviewed, practical experience in the field leaves a strong impression that improper inspection is occurring. The procedures used by EPA to audit I/M programs include a review of records and usually a demonstration by station inspectors of test procedures. Auditors have often witnessed a lack of knowledge, on the part of inspectors, of how to correctly conduct the emission test or calibrate the instrument. Unfortunately, most decentralized programs have limited resources and place insufficient emphasis on identifying stations that improperly inspect vehicles. Nevertheless, there are examples in every manual I/M program where undercover work is conducted of inspectors who got caught issuing a certificate of compliance without conducting the emission test. There are plenty of reasons why it is advantageous to improperly inspect rather than conduct inspections properly. Mechanic/ inspectors are in business to make money, not necessarily to reduce air pollution. Customers want to meet the program requirements with as little time and expenditure as possible. If a regular customer goes in for an emission test, there is strong incentive for the mechanic to report a passing result regardless of actual performance of the vehicle. If a tune-up had been done recently on the vehicle, reporting a failure would call into question the mechanic's ability and trustworthiness. There is also the situation where the station does not have any competent mechanics. These stations are in the business to collect the test fee and sell gasoline. They fail a few vehicles here and there to avoid attracting suspicion. Undercover work has verified that such stations exist; customers come to realize that they will always pass at this station, word spreads and business is up. On the other hand, there is the competent mechanic who runs a good station and is faced with increasingly complex automotive systems for which training is difficult and time consuming to obtain. This mechanic may try very hard to fix the vehicle but in failing to do so may report a pass anyway. To say the least, customers would not be very happy with the mechanic charging them the test fee, repair costs and not getting the vehicle to pass! Good mechanics are in high demand these days, and when the workload is heavy, they may issue a certificate without doing the test, simply for lack of time. In that failure rates among new technology vehicles are very low, some mechanic/inspectors may come to believe that they never fail so they don't bother testing them after a while. These motives and incentives are very real problems. While there is little in the way of hard evidence to support these ideas, they are based on observations and discussions with mechanics and program officials during audits. 24 ------- CONCLUSIONS This paper has reviewed a variety of possible causes for the low reported failure rates found in manual, decentralized I/M programs. It was shown that quality control lapses could, at most, only make a small difference in the failure rate outcome in an I/M program. Another potential explanation, better local maintenance, does not seem to be the case when emission scores of non-I/M vehicles from different areas are analyzed using uniform cutpoints. Cars in one area seem to fail at rates similar to that of other areas. Differences in emission standards and model year coverage among I/M programs do not seem to explain the low reported failure rates. By estimating the expected failure rates using a common data base, it was shown that reported failure rates in manual, decentralized I/M programs differ radically from expected failure rates. This was not the case for other program types. Anticipatory maintenance and pre-inspection repair, while likely to exist to some limited extent, do not seem to be prevalent phenomena and therefore cannot contribute significantly to low failure rates. Improper inspection seems to be the primary cause of' low failure rates in decentralized, manual I/M programs. The random, roadside idle surveys conducted by EPA show that I/M vehicles are not "clean" after inspection. The surveys show that failure rates of these vehicles are much higher than the reported failure rates. Analysis of I/M data shows that the reported emission scores from manual decentralized I/M programs are very unusual. Instead of having a smooth distribution of emission scores, the distributions from manual programs show higher scores close to the cutpoints. A distinct kink in the distribution occurs right at the cutpoint, where the scores drop off dramatically. This indicates that inspectors are not entering real test scores; the scores they invent are more often right below the cutpoint. The abnormal frequency index attempts to quantify this phenomenon and shows that in fact, there are unusually large numbers of vehicles just below the cutpoints in these programs. Patterns of emission scores have been observed during audits of inspection stations in which scores are repeated again and again. The repeat index was devised to quantify this in relation to the cutpoints and it lends support to the observed pattern of emission scores. Finally, practical experience during the audit process and the results of undercover inspection work show that improper inspection is a problem. In conclusion, low reported failure rates in manual, decentralized I/M programs seem to be primarily a function of improper inspection rather than simple variations in vehicle coverage, test procedures or other program characteristics. In order to correct this problem, programs will have to either eliminate the manual aspect of the program or put sufficient resources into oversight and undercover efforts. ------- REFERENCES 1 National Air Audit System Reports. 1984, 1985, 1986. U.S. EPA, Office of Air Quality Planning and Standards. 2 Motor Vehicle Tampering Surveys - 1983, 1984, 1985, 1986. U.S. EPA, Office of Air and Radiation. 3 Vehicle Exhaust Testing 1984 Annual Report, Air Pollution Control District of Jefferson County, January 25, 1985. 4 "Summary of Programs Simulating a Mandatory Maintenance Program." Memorandum from R. Bruce Michael, I/M Staff to Tom Cackette, Chief, I/M Staff. U.S. EPA, February 27, 1980. Effectiveness of Idle Adjustment on Light Duty Trucks at Commercial Repair Facilities. John C. Shelton, Test and Evaluation Branch, U.S. EPA, June 1980, EPA-AA-TEB-81-17. Effectiveness of Idle Adjustment on Passenger Cars at Commercial Repair Facilities. John C. Shelton, Test and Evaluation Branch, U.S. EPA, October 1980, EPA-AA-TEB-81-5. 7 Auto Exhaust Testing Inspection Statistics" First and Second Quarters 1986. Michigan Department of State. 8 Development of Data Analysis Systems for Decentralized I/M Programs, Energy and Environmental Analysis, Draft Report, September 1986. 26 ------- |