EPA-AA-TSS-I/M-85-09
EPA AUDITS OF STATE AND LOCAL
INSPECTION/MAINTENANCE PROGRAMS
By:
Philip A. Lorang
Jane A. Armstrong
John M. Cabaniss, Jr.
This paper was presented, at 78th Annual Meeting and
Exhibition of the Air Pollution .Control Association in
Detroit, Michigan, on June 18, 1985.
Technical Support Staff
Emission Control Technology Division
Office of Mobile Sources
Office of Air and Radiation
U. S. Environmental Protection Agency
-------
-2-
Abstract
EPA AUDITS OF STATE AND LOCAL INSPECTION/MAINTENANCE PROGRAMS
P.A.Lorang, J.A. Armstrong, J.M.Cabaniss
U.S.Environmental Protection Agency, Ann Arbor,MI 48105
Many State and local agencies have implemented vehicle emissions
I/M programs in the last few years. EPA began to audit these
programs in federal FY1984. The audit process for I/M was
developed in conjunction with STAPPA/ALAPCO; the I/M audit became
part of the National Air Audit System in FY1985. Results of the
I/M audits indicate that (1) enforcement is a problem in some
programs with sticker based enforcement; (2) low reported failure
rates are a problem in many decentralized programs, especially
those that -use manual analyzers, and in some centralized,
government run programs; (3) high waiver rates are a problem in
some programs, both centralized and decentralized; (4) analyzer
quality assurance ranges from excellent in centralized, contractor
programs to marginal in decentralized programs with manual
analyzers and in some centralized government run programs; (5)
data analyses are not being effectively used in most programs to
monitor and improve program performance and the performance of
individual inspection stations; (6) the quality of I/M repairs is
a problem, to some extent, in every program audited. EPA believes
that the resolution of these problems generally rests with each
State/local I/M program developing an overall I/M quality
assurance program to ensure that problems are identified and
resolved in a timely manner. .
-------
-4-
As a direct result of. the survey/workshop and related
comments received from State and local officials, EPA's Office
of Mobile Sources (QMS) decided early in 1984 that there was a
definite need for a national I/M audit system and national I/M
audit guidelines. EPA and State and local agencies could use
such an audit program (1) to ensure that statutory and State
Implementation Plan (SIP) requirements are being met; (2) to
assist in developing an acceptable level of I/M program
guality; (3) to account for the achievements, shortcomings, and
needs of the various I/M programs; (4) to identify programs
needing further technical support or other assistance; and (5)
to identify technical issues common to all or many programs
which need further investigation.
At the time that QMS made this decision, EPA's Office of
Air Quality Planning and Standards (OAQPS) was in the midst of
a cooperative effort with the State and Territorial Air
Pollution Program Administrators (STAPPA) and the Association
of Local Air Pollution Control Officials (ALAPCO) to develop
and implement the National Air Audit System (NAAS). The NAAS
was being developed to serve generally the same desired
objectives for the air guality . management programs across the
country. In NAAS, the EPA Regional Offices are responsible for
conducting the individual audits and. issuing related audit
reports, and OAQPS is responsible for assembling national audit
results and issuing an annual national audit report.
After consultations among officials of STAPPA/ALAPCO,
OAQPS, and OMS, it was decided that the best approach for
developing the I/M audit program would be to add an I/M element
to NAAS in federal FY1985. This necessitated the development
of the national I/M audit guidelines during federal FY1984. In
keeping with the NAAS process, a STAPPA/ALAPCO I/M subcommittee
was formed to work with OMS in developing the national I/M
audit guidelines. With the help of STAPPA/ALAPCO and I/M
officials around the country who served as reviewers, the I/M
audit guidelines were developed on schedule and appear as
Chapter 6 of the National Air Audit System Guidance Manual for
FY85 (EPA-450/2-84-008; December, 1984). The I/M audit
guidelines were developed in conformance with generally
accepted government auditing standards as contained in the
report: "Standards for Audit of Governmental Organizations,
Programs, Activities, and Functions," (U.S. General Accounting
Office, 1981 Revision).
-------
-6-
The FY1985 I/M audit schedule appears in Table 3. The
majority of the FY1985 I/M audits will involve decentralized
programs (six of eight). All of the programs, except the Texas
(Harris County) program, will have been in operation for at
least one year at the time of the audits. In the case of
Texas, the program will have been in operation only nine
months. The Texas program was originally included in the
FY1985 audit schedule because of its planned implementation
date of April 1984. Although, program start-up slipped to July
1984, it was decided to keep Texas in the FY1985 schedule
because of its special characteristics. The Texas I/M program
is the only I/M program implemented to date which relies
exclusively on the inspection and repair of tampered and
otherwise defective emission control system components rather
than basing the determination of the need for maintenance on
emissions short tests.
Table 3
FY1985 I/M Audit Schedule
Program
Location Pates Type*
Nevada 10/15 - 10/19/84 C
New York 12/10 - 12/14/84 D
Georgia 1/22 - 1/25/85 C
Missouri 3/04 - 3/08/85 C
Delaware 3/07 - 3/08/85 B
North Carolina 3/18 - 3/22/85 C
Texas 3/26 - 3/27/85 C
4/02 - 4/04/85
Oregon . 4/15 - 4/19/85 B
*B - centralized, government-run
C = decentralized
D = decentralized with computerized analyzers
-------
-8-
3.0 Audit Process for I/M Programs
As embodied in the NAAS, the I/M audit process contains
four elements:
1. Audit preparation.
2. Audit visit.
3. Development of the audit report •.
4. Audit follow-up.
The preparation for the audit allows the auditors to
familiarize themselves with the design and operations of the
program under review and to identify those particular aspects
of the program which may need special emphasis during the audit
visit. Proper preparation will allow the auditors to establish
priorities for various audit activities in order to make
efficient use of the limited time available during the audit
visit. It will also reduce the State/local resources needed to
cooperate with EPA in the audit and the disruption of the I/M
program itself.
The auditors assigned to perform the audit must
collectively possess as much knowledge as possible about the
operations of I/M programs in general and about the specific
details of the program under review. The goal of audit
preparation is to learn the basic design of the I/M program, to
identify as much as possible about the operating
characteristics of the program, and to determine as much as
possible about the potential strengths and weaknesses of the
program. To accomplish this, EPA auditors assemble and review
documentation on the program to be audited including (1)
program rules and regulations, (2) operating manuals or
procedures, (3) operating reports, (4) documentation of
previous audits or investigations, and (5) routine
correspondence to and from program officials, citizens, and
other interested parties. To facilitate audit preparation, the
I/M audit guidelines include an I/M program questionnaire which
contains questions covering the relevant aspects of an I/M
program.
The audit visit is for the purpose of verifying and
documenting whether the program is being properly administered,
operated, and enforced according to established laws, rules and
regulations, and procedural requirements in .the SIP.
Additionally, the audit visit allows a better evaluation of
program effectiveness and efficiency. The audit visit is
planned and coordinated with State/local agencies in order to
minimize the level of intrusion and disruption of normal
program activities and to make the best use of both EPA and
-------
-10-
EPA staff will plan follow-up activities as necessary to
encourage and assist State or local implementation of the
improvements discussed in the audit report. In cases where
deficiencies must be corrected, follow-up audit visits may be
necessary after corrective actions have been implemented.
-------
-12-
Another problem area in some programs was sticker
accountability. EPA auditors found that some programs
have very thorough sticker accountability procedures while
others do not. In order to ensure the proper disposition
of stickers, State/local agencies need to confirm that
each approval sticker has a matching inspection record
showing passing results. (This same confirmation is
needed in non-sticker programs for approval certificates,
except where the approval certificates and official
inspection reports are printed automatically by machine.)
This confirmation is accomplished in some current programs
by correlating sticker serial numbers to inspection
reports and then reviewing inspection records, sticker
records, and sticker supplies during audits. In programs
with automated data collection, sticker serial numbers can
be easily recorded in the inspection report and reviewed
through routine data analysis. Because of the potential
for data loss in some of these systems, State/local
agencies need to be cautious about totally relying on the
automatic records.
2. Reported failure rates - EPA auditors found reported
failure rates as high as 35 percent and as low as 2
percent.
Reported failure rates were consistently much lower than
(less that half) the designed failure rates in
decentralized programs. (One exception was a
decentralized program with computerized analyzers.) Also,
there were low failure rates reported in one government
run, centralized program. The contractor, centralized
programs all had reported failure rates in the designed
range.
There are several reasons for the low failure rates. In a
few cases, the failure rates are low because the I/M
cutpoints are too lenient. However, most of the problems
with low reported failure rates, especially in
decentralized programs, are caused by either
pre-inspection repairs, cheating by inspectors, or some
combination of these two factors. EPA believes that a
strong inspection station surveillance program is needed
to ensure proper station performance. This surveillance
program should include regular station audits, spot checks
with unmarked vehicles, and the ability to gauge and track
station performance through data analyses. Spot checks
with unmarked vehicles set to' fail inspection, should be
considered an indispensable part of the oversight function
in a decentralized program, particularly a program with
manual analyzers.
-------
-14-
Analyzer quality assurance - EPA auditors found quite a
variation in analyzer quality assurance among the audited
programs. In the centralized, contractor programs,
analyzer quality assurance was excellent. However, in
decentralized programs with manual analyzers and in some
centralized, government run programs, analyzer quality
assurance was marginal. These problems were caused by
several factors:
a. Lack of a comprehensive program of preventive
and corrective maintenance.
b. Lack of thorough audit/surveillance activities
for inspection stations.
c.. In some cases, existing analyzers are rather
old.
Data analyses - EPA auditors found that, with only a few
exceptions, I/M programs are failing to effectively use
available program data to monitor and take steps to
improve program performance and performance of individual
inspection stations. Of the programs audited, only one had
routine and timely data analyses which included more than
failure rates by model year and a few other overall
statistics.
In cases where inspection data are collected manually,
there are a number of problems which prevent, or at least
limit, the collection of accurate data. In some cases,
records are illegible and therefore unusable. A more
serious problem, and one that is more difficult to
resolve, is that in many cases inspectors do not correctly
record data. EPA auditors found that manually collected
data records often contain easily identified patterns of
record keeping (or other) abuses. In most cases, however,
the State/local agencies tend to categorize the problem as
inevitably poor record keeping practices rather than
attempting to resolve them. While true record keeping
abuses of themselves may not be serious, there is no way
to analyze the data to distinquish between record keeping
errors and more serious infractions, such as falsification
of test results in order to improperly pass a vehicle with
high emissions or to avoid inspecting a vehicle at all.
By screening inspection data, agency field investigators
should be able to identify questionable transactions or
problem stations. Therefore, EPA considers emphasis on
collecting and analyzing valid data to be a high priority.
In cases where data are collected automatically, the data
are generally available to program officials, although
there have been problems with data loss in some cases.
The problem in these cases tends to be an inability to use
-------
-16-
5.0 Conclusion
EPA believes that the resolution to the problems in
operating I/M programs generally rests with each State/local
I/M program developing an overall I/M guality assurance program
to ensure that problems are identified and resolved in a timely
manner. Through such systems, program managers need to track:
1. The level of non-compliance among vehicle owners.
This is particularly important in non-registration
enforcement systems. However, even registration
enforcement is not necessarily exempt from problems.
2. The performance of inspection stations to make sure
that vehicles are receiving fair, equitable, and
accurate inspections.
This involves inspection station audit and
surveillance activities as well as tracking
performance through data analysis. The latter is
possible only if accurate data are collected.
3. The performance of the program itself.
Program data need to be summarized and analyzed to
ensure that cutpoints, failure rates, waiver rates,
and other program statistics are within acceptable
limits.
4. The guality of repairs.
Quality repairs .are the backbone of I/M. Program
data need to be reviewed to ensure that vehicles are
not being improperly or incorrectly repaired.
Retest failure rates and comparisons of before- and
after-repair emissions levels can be useful
indicators to assess the guality of I/M repairs.
Identifying operating problems is as important first step
in providing guality I/M programs. The results of completed
audits ' are already- being used by State/local agencies and EPA
to improve I/M programs. EPA believes that the I/M audit
system and guidelines will continue to be a dynamic process for
achieving environmental results.
------- |