EPA-AA-TEB-82-7
The Portland Study
by
H. Anthony Ashby
June 1982
Test and Evaluation Branch
Emission Control Technology Division
Office of Mobile Sources
U.S. Environmental Protection Agency
-------
Summary
The Portland Study encompassed a series of test programs conducted for
EPA by a contractor in Portland, Oregon from September 1977 through April
1982. The objectives of the major programs were to study the correlation
between short inspection tests and the Federal Test Procedure, to study
the emission-reduction effectiveness of an operating vehicle inspection-
maintenance (I/M) program, and to study post-I/M deterioration. Other
important tasks completed in the Portland Study included an Emission
Factors test program, an evaluation of the effectiveness of mechanic
training, a study of the effect of I/M on light-duty trucks, and
evaluations of the in-use performance of three-way catalyst emission
control systems.
A total of 8,465 emission test sequences were conducted (all of which
included at least the FTP and various short t«;sts), at a cost for the two
contracts of $5,258,328.
Data are on magnetic tape in Ann Arbor under the cognizance of either
Test and Evaluation Branch (TEB) or I/M Staff personnel. Included are
mass emissions and tailpipe concentration data, vehicle descriptions,
maintenance information, vehicle owner questionnaires, diagnostic and
repair information, and so on. Portland Study data have been discussed
in I/M Staff reports, TEB reports, and papers presented to the Society of
Automotive Engineers and the Air Pollution Control Association.
-------
Introduction
This report describes the two phases of the Portland Study, a
contractor-operated vehicle emission test program that was conducted in
Portland, Oregon, over a five-year period beginning in 1977. The report
includes information on the history of the program, the objectives of the
various tasks that were completed, program design, and some of the
problems encountered. It does not include any test results, which may be
found in contractor test reports available from the library at the EPA
Motor Vehicle Emission Laboratory in Ann Arbor, Michigan. Test data and
other information are also stored on magnetic tape under the cognizance
of I/M Staff and Test and Evaluation Branch personnel in Ann Arbor.
Motor vehicle emission inspection and maintenance has been recognized
for years by EPA as an integral part of the total approach toward
reducing exhaust emissions from motor vehicles. In late 1975, a study
was conceived by Characterization and Applications Branch (CAB) personnel
for examining two major areas, of interest with respect to I/M programs.
These areas, on which very little data were available, were: 1) the
emission data correlation between the offical EPA test for measuring
exhaust emissions (the cold start Federal Test Procedure) and the various
simple, short inspection tests that would be used in I/M programs; and 2)
the effectiveness of an operating inspection program in terms of
identifying high-emission cars, emission reductions due to repairs,
costs, and the emission histories of cars subject to I/M programs.
At the time the study was being designed, I/M programs were operating
in the state of New Jersey and in the metropolitan area of Portland,
Oregon. Both were considered to be acceptable sites for the study.
Portland was chosen because of slightly lower costs, and because a
question had arisen as to whether the New Jersey Inspection program was
going to continue.
-------
I. Short Test Correlation and Effectiveness Study
Contract Information
Title: Short Test Correlation and Effectiveness Study
Contract No.: 68-03-2513
Contractor: Hamilton Test Systems, Inc.
Cost: $3,135,983
Number of Tests: 5,874
Period of Performance: February 3, 1977 through July 9, 1979
(except for Final Report)
Objectives
The Portland Study as originally planned consisted of two parts or
elements. The objective of Element I was t:o determine the correlation
between the Federal Test Procedure, conducted under laboratory
conditions, and various short tests that might be used in actual
inspection programs. Such correlation was necessary in order to
implement the emission performance warranty provisions of the Clean Air
Act (Section 207(b)).
The objective of Element II was to evaluate the effectiveness of an
operating I/M program (that run by the Oregon Department of Environmental
Quality) (DEQ). The evaluation was in terms of the idle test's ability
to identify gross emitters, the effectiveness of the repair industry in
reducing emissions, and the deterioration ol: emissions over a one-year
period.
Well into the program a third element was added. The objective of
Element III, the post-I/M deterioration study, was to collect information
on deterioration of emissions over short periods of time after passing an
inspection test.
-------
Chronology
12-08-75
06-11-76
06-24-76
07-26-76
08-23-76
09-23-76
10-22-76
11-24-76
12-08-76
12-15/16-76
01-15-77
02-03-77
04-11-77
04 and 05-77
CAB personnel initiated Procurement
Request for "Short Test Correlation
and Effectiveness Study." Estimated
cost: $2.8 million.
Request for Proposal No. CI 76-0070
issued by Negotiated Contracts
Branch
Bidders' Conference held in Portland
Amendment to RFP issued; Contract
type changed from Cost Plus Fixed
Fee to Firm Fixed Price
Proposal submittal date moved back
two weeks, from 08-09-76 to 08-23-76
Minor Scope of Work changes in
response to questions raised in
Bidders' Conference
Proposal received (one offerer only)
CAB Technical Evaluation completed
Received HTS clarification of
questions raised in Technical
Evaluation
Cost Advisor's Report receiived
Meeting with HTS and NCB in
Cincinnati to discuss source of
Element II control fleet
Negotiations in Ann Arbor on Scope
of Work
Agreement achieved on Firm Fixed
Price: $2,730,000
Contract signed
EPA personnel arrived in Portland
to set up Project Office
Modification of test facility,
installation of test equipment,
training of personnel
-------
05-11/12-77
08-26-77
09-20-77
09-01-78
01-22-79
01-24-79
04-02-79
07-06-79
Meeting in Ann Arbor to resolve
data handling problems
HTS terminated contract with OLI
(testing subcontractor)
First emission test successfully
completed
Short tests by DEQ inspectors
halted.
Element III testing began
Element I testing halted (1950
tests)
Element II testing completed (3220
tests)
Element III testing completed (704
tests)
-------
Program Design
Element I
For the Element I correlation study, 59 t:est vehicle groups or cells
were chosen to represent the national fleet of 1975-77 model year
passenger cars. The cells were defined by model year, manufacturer,
engine, transmission and vehicle weight. Considering statistical
validity for the intended short test-FTP correlation, the minimum sample
size for each cell was set at 40 vehicles. The criteria by which the
cells were defined included: the widest possible coverage of vehicles and
engines, quantities produced, technological significance, likelihood of
future production, and availability in the Portland area. The test
sequence included idle and other short tests conducted at a DEQ
inspection station, the same tests at the HTS facility, and the FTP.
Originally, there were to be 2400 cars tested for Element I, of
which 220 would also be involved in the Element II effectiveness study.
As the program developed, it was necessary to recruit many replacements
for participants who dropped out of the Element II program. Also,
Element I testing was halted ahead of schedule when I/M Staff data
analysis indicated that sufficient data on FTP-short test correlation had
been collected. This resulted in a total of 2207 tests for Element I.
Of these, 1950 cars were tested exclusively for Element I, and 257 were
involved in both Element I and Element II.
Element II
For the Element II effectiveness study, a fleet of cars from
Portland (subject to I/M) and a similar fleet from Eugene, Oregon (non-
I/M area, to serve as the experimental control) were tested initially in
as-received condition and then at three month, i.e., quarterly intervals
for one year. An additional test was conducted on Portland cars
initially failing the DEQ inspection, following repair sufficient to
enable the car to pass the inspection. Thus, Element II testing produced
emission histories of cars subject to I/M for comparison with the
histories of cars not subject to I/M. In addition, the effectiveness of
repairs to failed cars, done in the repair Industry at the car owner's
expense, was clearly shown. The test sequence included the FTP, HFET,
hot start FTP, and various short tests.
The test fleet of 660 cars was divided into model year groups
1972-74 and 1975-1977, the composition of each group reflecting Oregon
registration figures. Within each model year group, about 110 Portland
cars passed the DEQ inspection, 110 Portland cars failed, and 110 were
from Eugene. Element II attrition was significant: 7% to 9% of the cars
remaining in the program dropped out each quarter. As a result, over 50
cars had to be recruited to replace those whos?e owners dropped out of the
program. A total of 715 cars were initially tested and 506, or 71%,
received the fourth quarterly re test. By contractual requirement the
total number of Element II tests was 3220.
-------
Early in the follow-on contract, second, third, and fourth quarterly
retests were conducted on 1975-77 Portland cars to improve the Element II
data base. This brought the completion total to 527 cars, or 74%.
Element III
The element III post-I/M deterioration jitudy was a negotiated-price
modification to the original contract. The Element III test fleet
consisted of 292 privately owned cars of 1975-77 model years in rough
proportion to the Oregon registered population of those model years, but
with sample sizes of five minimum and nine maximum. The recruiting
objective was to start the test program on each car at the beginning of
its biennial registration cycle rather than at an unknown point as in the
case of Element II. Owners were contacted at the time their vehicles
were due to be reregistered and therefore due for a "real" DEQ
inspection. This was accomplished by recruiting on the basis of vehicle
model year and the first letter on the license plate, which indicates the
month in which the car was registered.
The initial test was in as-received condition. Retests were at one,
two, or three month intervals for each third of the test fleet,
respectively. Cars were entered in the program without regard to their
DEQ pass/fail status, and it turned out thai: 113 (39%) were DEQ fails.
The owners of those cars were asked to have the cars repaired to pass the
DEQ inspection (108 did so) and the cars were retested after the repair.
If a vehicle showed significant deterioration of emissions on the
one, two, or three month retest, the owner was interviewed to possibly
establish the reason. With the owner's permission the car was tuned by
the contractor to manufacturer's specifications, tested, and retested
after one month. There were fourteen such cars. A total of 704 tests
were run in Element III.
DEQ Support
An important phase of the Element I correlation study was to have
short inspection tests conducted by state employees at a state inspection
station. To this end the Oregon DEQ was awarded a contract, "Support to
Correlation and I/M Study," to provide space at an inspection station for
test equipment installed by HTS, and to provide the services of state
inspectors to conduct inspection tests on Element I cars using that
equipment.
Because of changes in EPA policy on the need for correlation between
the FTP and loaded inspection tests, the contract with the Oregon DEQ was
terminated by EPA on September 1, 1978. By l:hat time 1182 cars had been
tested by the DEQ personnel. The total cost of the contract was £100,500.
Problems
As might be expected with a project of the size and scope of the
original Portland Study, there were numerous problems. The most
important ones will be discussed here, roughly in chronological order.
-------
Element I Vehicle List
The original vehicle list that appeared in the RFP and the contract
consisted of 2400 cars of 1975-76 model years. They represented the four
major U.S. manufacturers plus 20 foreign marques including Alfa Romeo, MG
and Triumph. Sample sizes for various vehicles ranged from 5 to 100.
Shortly after the contract was signed the CAB Branch Chief determined
that testing 1977 models would be desirable. Further, for statistical
validity of the desired FTP-short test correlation, the minimum sample
size for a given vehicle type (defined as a combination of manufacturer,
engine, transmission, and weight class) had to be 40.
The Element I test fleet was completely redesigned, using criteria
including: the widest possible coverage of vehicle and engines,
quantities produced, technological significance, likelihood of future
production, and availability in the Portland area. Because the contract
was for a Firm Fixed Price, this change had to be accomplished by a
contract modification.
Data Formatting and Handling
Early on, a major problem developed over EPA's requirements for data
collecting and formatting as revealed in the EPA proposed data formats.
These had the effect of increasing the contractor's equipment and
personnel requirements beyond its contractual responsibilities. After
considerable discussion and negotiation, the data collecting, handling,
and formatting requirements were agreed to. The main elements of the
agreement were that HTS would collect the additional data required by EPA
and rearrange all data into EPA's formats, using an EPA-paid account on
the Michigan Terminal System. HTS would supply terminals and
teleprinters both for itself and the EPA Portland Study project office.
The agreement was documented as contract modification number 3. This
data handling system worked well for as long ,is it was used.
Program Delays
The contractor's original plan of performance was that equipment
would be installed in May 1977, with testing beginning in June at the
rate of 20 tests per day. The actual program startup was much more
time-consuming, and the testing reached 20 per day only by July 1978.
Problems were numerous. Calibration gases and chassis dynamometers were
delivered on schedule, but other test equipment was a month late.
Installation and checkout were complicated by wiring and plumbing
mistakes in the analyzers. These problems were aggravated by the
inability of the testing subcontractor (Olson Laboratories, Inc.) to
organize and manage the work it had to do. In late August 1977 HTS
terminated its contract with OLI and assumed the responsibility for
vehicle testing operations. A new plan of performance was submitted
showing a gradual buildup of the testing rate as equipment was made
operational and new test technicians were trained. The first official
test, which became part of EPA's data basu, was run on September 20,
1977. These delays caused the program to slip several months behind the
-------
10
original schedule. EPA thus had to spend additional money on its office
space in Portland and on its contract for support from the DEQ. To cover
EPA's extra expenses, a downward adjustment in the price of a later
modification to the HTS contract was negotiated.
Vehicle Recruiting
Vehicle recruiting went very well in terms of the public's response
to the HTS recruiting letter. Problems did occur with respect to the
makeup of the fleet in both Element I and Element II. In Element I, HTS
recruited off the top of the registration lists for each cell, which
resulted in a lot of vanity plates and letter-A license plates getting
into the program. HTS also recruited makes in bunches, in violation of a
contractual requirement. At one point 19 cars had been tested in one
cell and none, or only a few, in several other cells. The ideal was to
have the Element I cells be filled evenly, with cars of various license
plate first-letters, over the entire 10 months of Element I testing so
that climatic effects in the short test to FTP correlation could be seen
for all the cells. Technical directions and a contract modification
finally got HTS squared away but not before much distortion had
occurred. At the end of Element I testing, sample sizes ranged from 32
to 45, where the ideal would have been 37.
The same vanity plate and A-plate problems occurred in Element II.
A bigger problem had to do with the Element II program design. Vehicles
were recruited for Element II at any point: in their two-year Oregon
registration cycle; the ideal would have be:en to recruit them at the
beginning of that cycle. (An EPA remark in the Bidders' Conference
indicates that such was EPA's intent.) Unfortunately it was never
clearly stated in the RFP or the contract. The result was that cars were
recruited off registration lists without regard to when they were last
inspected by DEQ. Therefore, the emissions history of a car in Element
II could be interrupted by maintenance sometime during the one-year study
if the car needed it to pass the DEQ inspection. Thanks to Element III
and its continuation in the follow-on contract, data did become available
on cars in the first year after they passed i:he DEQ inspection. Also to
help with Element II data analysis the Oregon Motor Vehicle Division was
engaged to look up registration dates on Elemesnt II cars.
Data Packet Error Rate
The contractor's original proposal stated that the same
Hewlett-Packard computer used to present driving cycles and to switch
valves for CVS bag filling would also select gas analyzer ranges, zero
and span the analyzers, read sample bag concentrations and CVS inputs,
and calculate mass emissions. This capability was never achieved in
spite of attempts by both OLI and HTS. The idea of hiring the H-P
programming service was rejected by HTS on the grounds of cost. As a
result the HTS data flow system included a large number of manual entries
and interfacing, and manual checking. The HTS data review process was
insufficient to ensure delivery to EPA of error-free data.
-------
11
The error rate was very high for several months - from September
1977 through April 1978 over one third of the data packets submitted
contained error, and there was no trend of improvement. After EPA began
simply returning erroneous packets, instead of also including a
description of the error, HTS finally seemed to attend to the problem.
By June the error rate was down to an acceptable 10%.
Contract Modifications
Because this was a Firm Fixed Price contract, it was difficult to
change the Scope of Work, as EPA found it necessary to do several times
during the course of the program. For each change the entire procurement
cycle was involved: preparation of procurerient package, evaluation of
HTS response, price negotiation and so on. "his often led to delays in
the implementation of the desired changes.
II. Testing Support for Evaluation of I/M Issues
Contract Information
Title: Testing Support for Evaluation of I/M Issues
Contract No.: 68-03-2829
Contractor: Hamilton Test Systems, Inc.
Cost: $2,122,345
No. of Tests: 2591
Period of Performance: June 10, 1979 through April 14, 1982 (for
testing operations)
Objective
The objective of the program was primarily to provide EPA with the
capability of quickly obtaining emission data and other information to
answer questions and resolve issues that might arise as I/M programs were
planned and implemented around the country. In general the data were to
come from in-use, privately owned light duty vehicles recruited in
Portland, where an I/M program had been in operation since 1975. In
addition, the contract would provide a site for other testing
requirements which could not be handled by the EPA Motor Vehicle Emission
Laboratory at Ann Arbor, Michigan. A total of 1951 tests were conducted
for I/M support purposes.
As the program developed and testing requirements were defined by
IMS and TEB personnel, a significant number of tests were conducted for
which I/M was not the rationale. These included tests to study fuel
economy discrepancies between in-use and EPA values, tests on cars from a
non-I/M area (Vancouver, Washington) for the: Emission Factors data base,
and evaluation of a fuel economy retrofit device. Additional tests under
the Project Officer's direction were also run to evaluate alternate
fuels, retrofits and other . systems, primarily at the request of
government agencies in the Portland area. A total of 640 non-I/M
oriented tests were conducted.
-------
12
Chronology
12-19-78 TEB personnel initiate Procurement Request for
"Testing Support for Evaluation of I/M Issues"
03-28-78 RFP No. CI 79-0118 issued (Sole Source Procurement)
04-16-79 HTS proposal received
05-04-79 TEB Technical Evaluation completed
06-10-79 Work begun under Anticipatory Cost Letter
07-26-79 Effective date of contract
04-15-82 End of testing operations
Program Design
The contract type - Cost Plus Fixed Fee -• made it relatively easy to
give technical direction to HTS, within the general specifications of the
contract. The assigned tasks, called Test Groups in the contract, were
described in Technical Directions which included instructions in these
areas:
Vehicles to be recruited
Recruiting methods
Sequence of tests to be run
Data to be collected
Criteria for performing maintenance and retest
Seventeen Test Groups were specified. All except No. 15 required
the two-speed idle test conducted by the Oregon DEQ, the FTP, the Highway
Fuel Economy Test, two to five chassis dynamometer short tests, and
underhood and diagnostic inspections. In some sequences in Test Group 8,
the FTP was replaced with the hot start FTP.
On Test Groups 14, 16, and 17, test data including emissions
concentrations were entered on EPA-specifiec! forms and sent to Systex,
Inc., EPA's data reduction contractor. On all other Test Groups, test
information including mass emissions and. fuel economy data were
transmitted via telephone to MTS, reformatted, and made available to EPA.
Test Group Descriptions
Following are brief descriptions of the tasks completed in the
program.
Test Group 1 - Restorative Maintenance en Three-Way Catalyst Vehicles
Three-way catalyst cars of 1978-79 model years tested as received
and after restorative maintenance. 51 cars, 74 tests.
Test Group 2 - Element II Continuation Tests.
Quarterly retests were continued on 1975-77 Portland cars from
Element II to increase the number of cars completing the full
one-year series of tests. 22 cars, 50 l:ests.
-------
13
Test Groups 3 and 4 - Element III Retests
Element III cars were re tested at intervals of 6, 9 and 12 months
after the initial test. Cars failing the DEQ inspection at the
12-month point were given restorative maintenance and retested. 258
cars, 743 tests.
Test Group 5 - Effectiveness of EGR Repairs
Element II cars with defective EGR systems were tested as received,
after EGR repairs, and after six more months of service in the
owner's hands. 28 cars, 80 tests.
Test Group 6 - Heated Air Intake Disablement
Cars from the contractor's loaner fleet (1979-80 model years) were
tested to study the effect of disabling heated air intake systems.
10 cars, 20 tests.
Test Group 7 - Mechanic Training Evaluation Program
The objective of this program was to evaluate the effect of mechanic
training on emission reductions achieved in an I/M program. The
training consisted of the Colorado State University emission repair
course.
Cars failing the DEQ inspection were tested as-received, after
repair by the participating mechanics, and after restorative
maintenance by the contractor. Matched sets of cars were tested
this way before and after the training course was given, in a blind
study. 98 cars, 307 tests.
Test Group 8 - Evaluation of Potential Catalyst Diagnostic Tests
Cars from the contractor's loaner fleet were tested to evaluate the
usefulness of various short tests in identifying lead-poisoned
catalytic converters. 3 cars, 43 tests.
Test Group 9 - Vancouver Versus Portland Vehicle Emissions
Matching cars (1976 and 1978 model years) from the two cities were
tested as-received to compare emissions from I/M and non-I/M areas.
212 cars, 212 tests.
Test Group 10 - Effect of Specific Maintenance Tasks on Emission
Repairs on I/M Failed Vehicles
Test Group 9 Vancouver cars that failed the DEQ inspection
as-received were repaired in specified steps by the contractor and
retested. 35 cars, 41 tests.
-------
14
Test Group 11 - Fuel Economy Shortfall Tests
Twenty types of 1978-80 model year vehicles (including four light
duty truck models) were tested to provide information on the
discrepancies between in-use fuel economy and the EPA Buyer's Guide
values. The sample size for each type was between 5 and 7
vehicles. The vehicles were tested as-received; those failing to
meet Federal emission standards were aliio tested after restorative
maintenance. 129 vehicles, 219 tests.
Test Group 12 - Effect of I/M on Light Duty Trucks
Light duty trucks of 1976, 1978 and 1980 model years were tested in
as-received condition. Those failing the state inspection were
repaired commercially and retested. Selected vehicles were also
given restorative maintenance by the contractor and retested. 105
vehicles, 148 tests
Test Group 13 - Evaluation of 1981 Ford Open-Loop Three-Way Catalyst
Cars
Open-loop 1981 Ford products were tested in as-received condition.
Those exceeding specified levels on the FTP or certain short tests
were given restorative maintenance and retested. Four cars were
given a series of tests using various fuels and with disablement of
emission control systems. 112 cars, 179 tests.
Test Group 14 - A Study of Emissions Froia Light-Duty Vehicles in
Vancouver, Washington.
Vehicles of 1978-82 model years from Vancouver were tested in
as-received condition as part of l:he Emission Factors data
collection program. Those failing to rae.et certain emission criteria
were retested following restorative maintenance. 300 vehicles, 340
tests.
Test Group 15 - Evaluation of "Super-Mag Fuel Extender".
This device evaluation was conducted pursuant to Section 511 of the
Motor Vehicle Information and Cost Savings Act. 3 cars, 34 tests.
Test Group 16 - Problem Closed-Loop 3-way Catalyst Cars
As-received and after-maintenance tests were performed on
closed-loop 3-way catalyst cars with known or suspected problems in
those systems. 6 cars, 10 tests.
Test Group 17 - Disablement Testing of Closed-Loop Systems
Closed-loop cars were tested to evaluate the effects of various
disablements on FTP and short test emissions. 22 cars, 80 tests.
-------
15
In addition to the tasks described above, a total of 47 tests were
conducted by technical direction of the EPA Project Officer. Almost all
of these were in response to requests from two government agencies in the
area. For the Bonneville Power Administration, tests were run to
evaluate alcohol-gasoline blends, a methanol-fueled car, and an exhaust
system heat shield for a vehicle that travels off-road. For the Oregon
Department of Environmental Quality Vehicle Inspection Program, tests
were run on aftermarket turbochargers, a retrofit device, a gasohol fuel,
and a noncomplying imported car. A total of about 6 tests were run on:
a liquefied natural gas fueled pickup truck, an experimental safety
vehicle, an energy storage system, and a methanol-fueled car.
Program Conclusions
The last of several modifications to the contract for Testing
Support for Evaluation of I/M issues covered testing until April 15,
1982. The final test sequence in the program was conducted, as part of
the Test Group 17 task, on April 14.
No final laboratory qualification procedure was conducted.
Calibration and calibration checks on chassis dynamometers, constant
volume samplers, and gas analyzers had been conducted on a regularly
scheduled basis all through the life of the program, with EPA personnel
on site. The EPA Project Officer felt that a final laboratory
qualification, therefore, was unnecessary.
All test equipment was removed by the contractor and shipped to its
corporate headquarters in Arizona, and the facility was vacated on
May/15, 1982.
Test Results
Subject Contact
Contractor's Test Reports on MVEL Library
Element I, Element II,
Element III, and Test
Groups 1 through 17
I/M Staff Reports Phil Lorang, Chief
(See "Abstracts of I/M Tech- I/M Staff
nical Reports and Policy")
Emission Factors Data Base Lois Platte, TEB
------- |