U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
EPA Could Improve the
SmartWay Transport
Partnership Program by
Implementing a Direct Data
Verification Process
Report No. 12-P-0747
August 30, 2012

-------
Report Contributors:	John Bishop
Dan Howard
Geoff Pierce
Rick Beusse
Abbreviations

C02
Carbon Dioxide
DERA
Diesel Emission Reduction Act
EPA
U.S. Environmental Protection Agency
FY
Fiscal year
GAO
U.S. Government Accountability Office
GHG
Greenhouse gas
IFTA
International Fuel Tax Agreement
IRS
Internal Revenue Service
MIT
Massachusetts Institute of Technology
NHTSA
National Highway Traffic Safety Administration
NOx,
Nitrogen Oxides
OAR
Office of Air and Radiation
OIG
Office of Inspector General
OMB
Office of Management and Budget
OTAQ
Office of Transportation and Air Quality
PM
Particulate matter
Cover photo: A truck that is outfitted with fuel saving technologies as part of the SmartWay
Technology Upgrade Project. (EPA photo)
Hotline
To report fraud, waste, or abuse, contact us through one of the following methods:
e-mail: OIG Hotline@epa.gov	write: EPA Inspector General Hotline
phone: 1-888-546-8740	1200 Pennsylvania Avenue NW
fax:	202-566-2599	Mailcode 2431T
online:
http://www.epa.gov/oia/hotline.htm
Washington, DC 20460

-------
^tDsrx
• A v
iSi
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
12-P-0747
August 30, 2012
Why We Did This Review
We sought to determine how the
U.S. Environmental Protection
Agency (EPA) ensures the
validity of the SmartWay
Transport program results. EPA
established the SmartWay
Transport Partnership in 2004.
It is a voluntary collaboration
between EPA and the freight
industry (carriers, shippers,
logistics companies, etc.) to
improve fuel efficiency and
reduce environmental impacts
from freight transport. Almost
2,900 SmartWay partners,
employing about 650,000 trucks,
have traveled nearly 43 billion
miles on average each year,
according to EPA. Since the
program's inception, EPA
estimates it has saved about
50 million barrels of oil (as of
March 2011), resulting in reduced
air pollution. Also, envisioned
future carbon dioxide reductions
from EPA's September 2011
standards for heavy-duty 2014-
2018 model year vehicles will
depend heavily on EPA's
SmartWay technologies and
strategies.
This report addresses the
following EPA Goal or Cross-
Cutting Strategy:
• Taking action on climate
change and improving air
quality
For further information, contact
our Office of Congressional and
Public Affairs at (202) 566-2391.
The full report is at:
www.epa.qov/oiq/reports/2012/
20120830-12-P-0747.pdf
EPA Could Improve the SmartWay Transport
Partnership Program by Implementing a
Direct Data Verification Process
What We Found
Recent studies corroborate EPA's claims that its SmartWay Transport
Partnership program helps remove marketplace barriers in order to deploy fuel
efficient technologies faster. To calculate SmartWay program emission
reductions, EPA relies on self-reported industry data. EPA's Office of
Transportation and Air Quality performs some checks of data provided by
industry. However, there is no independent direct verification by EPA of data
submitted by SmartWay participants. The risk of false claims was highlighted in
2011 when EPA became aware of a case where a company was alleged to
have improperly used the SmartWay logo.
There is an incentive for carriers to obtain and maintain high scores. Carrier
performance scores are listed on EPA's SmartWay website. The carriers that
receive the highest scores are more likely to be selected by more shippers. As
more and more shippers join SmartWay, the economic incentives for carriers to
achieve higher scores on EPA's website may increase, which could also
increase the potential that a carrier would submit data that overstates its
scores.
In our view, the SmartWay Transport Partnership program may lose its value if
EPA does not protect the integrity of its program by implementing some form of
direct verification or other measures to deter companies from submitting data
that result in overstated scores.
Recommendations and Planned Agency Corrective Actions
We recommend that the Assistant Administrator for Air and Radiation develop
and implement direct verification or other measures to verify the accuracy of a
sample of the self-reported, industry data for the SmartWay Transport
Partnership. EPA agreed with the OIG on the importance of ensuring the
integrity of program results and proposed a five step process to better ensure
the accuracy of partner data. EPA's planned actions are a step in the right
direction. EPA should describe any additional planned corrective actions in its
90-day response to the final report.
Noteworthy Achievements
Representatives from environmental, retail, and trucking associations consider
EPA's SmartWay program an effective program for reducing fuel costs and the
environmental impact of freight movement. Further, the number of partners in
the SmartWay Transport Partnership has grown considerably since 2008.

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
THE INSPECTOR GENERAL
August 30, 2012
MEMORANDUM
SUBJECT: EPA Could Improve the SmartWay Transport Partnership Program by
Implementing a Direct Data Verification Process
Report No. 12-P-0747
FROM: Arthur A. Elkins, Jr.
TO:
Gina McCarthy
Assistant Administrator for Air and Radiation
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends. This report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
Final determinations on matters in this report will be made by EPA managers in accordance with
established audit resolution procedures.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. The recommendation is listed as open with corrective actions
pending. Your response should include a corrective action plan for agreed-upon actions,
including actual or estimated milestone completion dates. Your response will be posted on the
OIG's public website, along with our comments to your response. Your response should be
provided in an Adobe PDF file that complies with the accessibility requirements of Section 508
of the Rehabilitation Act of 1973, as amended. Please e-mail your response to Carolyn Copper at
copper.carolyn@epa.gov. We have no objections to the further release of this report to the
public. We will post this report to our website at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact Carolyn Copper,
Assistant Inspector General for Program Evaluation, at (202) 566-0829 or
copper.carolyn@epa. gov; or Rick Beusse, Director for Air and Research Evaluations, at
(919) 541-5747 or beusse.rick@epa.gov; or John Bishop, Project Manager, at (919) 541-1028 or
bi shop. i ohn@epa. gov.

-------
EPA Could Improve the SmartWay Transport
Partnership Program by Implementing a Direct
Data Verification Process
12-P-0747
Table of C
Purpose		1
Background		1
Noteworthy Achievements		3
Scope and Methodology		3
Results of Review		5
Other Matters		7
Conclusions		7
Recommendation		8
Agency Comments and OIG Evaluation		8
Status of Recommendations and Potential Monetary Benefits		10
Appendices
A General Criteria and Benefits of the SmartWay Program		11
B SmartWay Partner Recognition System and Its Relationship
to EPA's Partner Tools		13
C Prior OIG Reports Related to EPA Voluntary Programs		15
D Agency Comments to Draft Report		16
E Distribution		19

-------
Purpose
The purpose of our review was to determine how the U.S. Environmental
Protection Agency (EPA) ensures the validity of the SmartWay Transport
Partnership results.
Background
The SmartWay Transport Partnership is one of a number of voluntary programs
that EPA manages (e.g., the ENERGY STAR program, clean energy partnerships,
and multiple non-carbon dioxide (C02) greenhouse gases programs) that work to
remove barriers in the marketplace in order to deploy cost-effective technologies
faster. They work by overcoming widely acknowledged barriers to energy
efficiency and deployment of greenhouse gas (GHG) reduction measures such as
lack of clear, reliable information on technology opportunities; lack of awareness
of energy efficient products, services, and transportation choices; and the lack of
additional incentives for manufacturers to invest in efficiency research and
development. Recent studies corroborate EPA's claims that its SmartWay
Transport Partnership program helps remove marketplace barriers in order to
deploy fuel efficient technologies faster (see appendix A).
EPA's SmartWay Transport Partnership program, begun in 2004, is a voluntary
public/private collaboration between EPA and the freight industry to improve fuel
efficiency and reduce environmental impacts from freight transport by
accelerating the deployment of fuel saving, low emission technologies and
promoting GHG reductions across the global supply chain. The carriers, shippers,
logistics companies, and others who voluntarily participate are known as
SmartWay partners.
According to EPA, almost 2,900 SmartWay partners, employing about 650,000
trucks, have traveled nearly 43 billion miles on average each year. EPA also
stated in its fiscal year (FY) 2012 Congressional Justification for the Proposed
Budget that SmartWay helps reduce emissions from the existing 2.2 million
heavy-duty trucks currently in operation not covered by the Greenhouse Gas
regulation1 for medium-duty and heavy-duty vehicles. Appendix A provides
additional information about the program.
SmartWay Technologies and Strategies
Carriers, shippers, logistics companies, and others use SmartWay technologies
and strategies to help improve fuel efficiency and reduce environmental impacts
from freight transport.
1 Greenhouse Gas Emissions Standards and Fuel Efficiency Standards for Medium- and Heavy-Duty Engines and
Vehicles, Final Rules, Federal Registcr/Vol. 76/No. 179/ Thursday, September 15, 2011, page 57106, Environmental
Protection Agency and Department of Transportation, National Highway Traffic Safety Administration [EPA-HQ-
OAR-2010-0162; NHTSA-2010-0079],
12-P-0747
1

-------
•	SmartWay technologies include low rolling resistance tires, low viscosity
lubricants, idle reduction, automatic tire inflation systems, improved
aerodynamics, hybrid powertrain technology, and longer combination
vehicles.
•	SmartWay strategies include weight reduction (reductions in empty truck
weight), reducing highway speed, driver training, idle reduction, improved
freight logistics (load matching, improved routing and scheduling, etc.),
intermodal shipping, and maintaining proper tire inflation pressure.
In September 2011, EPA and the U.S. Department of Transportation's National
Highway Traffic Safety Administration (NHTSA) adopted complementary
standards for heavy-duty vehicles under their respective authorities covering
model years 2014-2018. EPA's and NHTSA's standards address C02 emissions
and fuel consumption, respectively. The envisioned C02 reductions depend
heavily on EPA's SmartWay technologies and strategies. The agencies noted that
SmartWay trucks are already available today which incorporate the technologies
on whose performance the final standards are based.
EPA Resources Devoted to SmartWay
EPA's SmartWay program received $2.42 million in FY 2011, had 11 full-time
equivalents2, and contracted for 10 Partner Account Managers. The Partner
Account Managers are staffed through contracts with Senior Service America,
Inc. EPA's FY 2012 budget for the SmartWay program totaled $2.7 million.
EPA's SmartWay program also received $16.9 million in Diesel Emission
Reduction Act (DERA) funds for FY 2008 through FY 2010. The SmartWay
program also received $30 million in American Reinvestment and Recovery Act
of 2009 funds. EPA awarded 12 grants to 9 grantees for SmartWay projects with
these funds. The grants were for loan guarantees, loans, subsidies, and leases to
retrofit, repower, or replace equipment to reduce emissions. The SmartWay
program did not receive any DERA funds for FY 2011 and FY 2012, according to
the Office of Transportation and Air Quality (OTAQ).
SmartWay Partner Tools and Recognition System
EPA has developed a number of software tools that are used by SmartWay
partners to assess the efficiency of their operations. Data from these applications
are reported to EPA annually and used to generate performance scores for each
SmartWay partner based on emissions rates developed by EPA in the various
sectors of the shipping industry. Truck carriers and shippers with high
performance scores, are eligible to use the SmartWay logo. Shippers' scores are
based on the amount of freight that is moved by high performing truck carriers. A
2Full-time equivalents are calculated based on the number of full-time and part-time employees in an organization.
Full-time equivalents represent these workers as a comparable number of full-time employees.
12-P-0747
2

-------
more detailed discussion of the tools used to generate performance scores for
truck carriers and shippers, as well as information for two other EPA-developed
SmartWay tools, is provided in appendix B.
Noteworthy Achievements
According to EPA's March 2011 SmartWay Program Highlights, SmartWay's
clean air achievements include emission reductions of 16.5 million metric tons of
C02, 235,000 tons of nitrogen oxides (NOx), and 9,000 tons of particulate matter
(PM). Between 2004 and 2011, SmartWay partners saved 50 million barrels of
oil, according to EPA's March 2011 SmartWay Program Highlights. These
savings are equivalent to taking over 3 million cars off the road for an entire year.
SmartWay has also helped U.S. businesses slash their fuel costs, saving $6.1
billion dollars to date, according to EPA.
Representatives in the shipping industry are complimentary of EPA's SmartWay
program. Representatives from environmental, retail, and trucking associations
consider EPA's SmartWay program an effective program for reducing fuel costs
and reducing the environmental impact of freight movement. According to a
survey conducted by American Shipper,3 SmartWay ranked first among all the
choices of supply chain sustainability programs4. Further, the number of partners
in the SmartWay Transport Partnership has grown considerably since 2008.
EPA's website lists almost 2,900 partners. According to the Center Director for
the SmartWay and Supply Chain Programs, there were approximately 500
partners in 2008.
Scope and Methodology
We conducted this evaluation in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the
evaluation to obtain sufficient, appropriate evidence to provide a reasonable basis
for our findings and conclusions based on our objectives. We believe that the
evidence obtained provides a reasonable basis for our findings and conclusions
based upon our objectives. We conducted our field work from May 2011 through
June 2012.
We conducted a design evaluation of the SmartWay program to determine
whether controls were in place to ensure the overall validity of claimed SmartWay
Transport Partnership results. We examined OTAQ's SmartWay guidance,
procedures, and the existing SmartWay Partner tools OTAQ developed to assess
3	Environmental Sustainability Benchmark Study: Leaders Prepare for the "Greening" Supply Chain, American
Shipper, published February 2011.
4	Nearly 200 shippers and third-party logistics providers answered some or all of the survey's 25 questions.
According to the survey report, 36 percent of respondents to one particular question in the study identified
SmartWay as a leading program, with "other" programs getting the second highest votes at 27 percent. Appendix A
provides additional information related to studies that indicate SmartWay helps to overcome marketplace barriers.
12-P-0747
3

-------
the effectiveness of EPA's oversight and management of the SmartWay program.
We reviewed peer review reports for the tools developed by OTAQ. We also
reviewed the results of prior EPA Office of Inspector General (OIG) reports on
other EPA voluntary programs and external reviews of the SmartWay program.
We reviewed the contents of EPA's SmartWay Transport website for information
related to claimed emissions reductions, policies, and procedures. We reviewed
guidance and reports issued by government agencies (including the Office of
Management and Budget (OMB) and the U.S. Government Accountability Office
(GAO)) regarding the requirements and steps taken to ensure that accurate data is
reported both to and by the government. We reviewed the 2010 Taxpayer Attitude
Survey completed by the Internal Revenue Service (IRS) Oversight Board.
We interviewed EPA OTAQ SmartWay program managers and staff located in
Washington, DC, and Ann Arbor, Michigan, for information regarding the goals
of the program, policies and procedures for the program, and EPA's process for
validation of self-reported industry data.
The SmartWay program tools and the logo recognition system for the SmartWay
partners were being developed and revised during the time we conducted our
evaluation. Thus, there was a lack of revised and newly developed data available
for us to review.
Prior Audit Coverage
In a 2010 EPA OIG summary report on another EPA voluntary program, the
ENERGY STAR program,5 the OIG concluded that the integrity of the ENERGY
STAR label remains at risk because it does not necessarily identify and promote
the most energy-efficient products. Further, the OIG found that:
•	Products historically qualified for the ENERGY STAR label based on
manufacturer self-certification, rather than EPA testing.
•	EPA conducted only minimal verification testing and assumed that in a
competitive market, manufacturers would test each other's products and
report failures to EPA. However, the Agency could not provide any
examples as evidence that self-policing occurred.
•	EPA had not conducted any verification testing for the first 10 years of the
program. When verification testing began, it accounted for only a small
component of the program's activities and budget.
•	EPA cannot be certain that it's reported savings claims are valid or
supportable, and that large amounts of GHG emissions are in fact being
avoided.
5 EPA OIG Evaluation Report No. 1 l-P-0010, ENERGY STAR Label Needs to Assure Superior Energy Conservation
Performance, A Summary Report, October 28, 2010.
12-P-0747
4

-------
The OIG has also issued eight other reports related to voluntary programs. These
reports are listed in appendix C.
Results of Review
In order to calculate SmartWay program emission reductions, EPA relies on self-
reported industry data. OTAQ performs some checks of data provided by
industry. EPA also has 10 Partner Account Managers under contract who review
data provided by the partners, along with any explanations, and who can question
data that appears incorrect. However, there is no direct verification by EPA of
data submitted by SmartWay participants. We believe this lack of direct
verification is a potential design weakness in the program, which affects the
Agency's ability to ensure the overall validity of claimed SmartWay Transport
Partnership results.
OMB Circular A-123 and the GAO Standards for Internal Control in the Federal
Government were issued to implement the Federal Managers' Financial Integrity
Act of 1982. OMB Circular A-123 (Attachment I: Introduction) calls for federal
agencies to develop management controls that provide reasonable assurance that
programs are achieving desired results. OMB Circular A-123 states that such
controls are the tools to help program managers "achieve results and safeguard the
integrity of their programs." For example, the IRS completed audits and
examinations of a sample of tax returns to provide a deterrent, or disincentive, to
taxpayers who otherwise might submit incorrect information to the IRS. In the
2010 Taxpayer Attitude Survey conducted by the IRS Oversight Board, 64
percent of the respondents reported that "fear of an audit" influenced whether they
report and pay their taxes honestly. Further, 66 percent of the respondents
reported that third-party reporting of data on income such as wages, dividends,
and interest to the IRS influenced their decision on whether to honestly report and
pay taxes.
According to OMB Circular A-l 1 (2011), Section 230:
Verification and validation of performance data support the general
accuracy and reliability of performance information, reduce the
risk of inaccurate performance data, and provide a sufficient level
of confidence to Congress and the public that the information
presented is credible.
GAO defines verification as a process of checking or testing performance
information to assess other types of errors, such as errors in keying data. GAO
defines validation as an effort to ensure that data are free of systematic error or
bias and that what is intended to be measured is actually measured.
12-P-0747
5

-------
Direct Verification of Reported Data Needed
While EPA performs some quality assurance checks of industry supplied data, in
our opinion, some kind of direct verification of data is needed. Since shippers'
scores are dependent on selecting carriers that receive high scores, shippers may
choose carriers with the highest scores. This will become more likely as more
shippers seek to demonstrate to customers, clients, and investors that they are (1)
taking responsibility for the emissions associated with goods movement, (2)
committed to corporate social responsibility and sustainable business practices,
and (3) reducing their carbon footprint. This may give carriers an economic
incentive to submit data that maximizes their scores in order to be included higher
up in the SmartWay listing of carriers, and thus be more likely to be selected by
more shippers.
Although EPA does not directly verify any of the partner-provided data, OTAQ
and the Partner Account Managers perform some quality assurance checks of data
provided by industry. The Truck Tool, an EPA-developed computer model, has
the capability to highlight data that are outliers from average industry data. EPA
has incorporated acceptable ranges into the Truck Tool model for data that deviate
from the industry averages. If the data are outside of the ranges, the tool notifies
the user to recheck their input data. Additionally, if the user continues to enter
data outside of the range, the user must provide an explanation for EPA's tool to
accept the data. In order to use the Truck Tool, the carrier is supposed to identify
the source of the data for the number of miles driven and the amount of fuel used.
For example, carriers can select International Fuel Tax Agreement (IFTA) Form
4416, IRS records, company electronic records, or other records as the source of
their data.
Also, OTAQ has 10 Partner Account Managers that review the data reports on an
exception basis that SmartWay partners input into the system and follow up with
questions for data that appear incorrect. According to OTAQ, reports are not
approved if there are questionable data. OTAQ developed a consolidated report
that allows Partner Account Managers to identify data that are outside normal
ranges. The tools also allow the Partner Account Managers to generate year-to-
year comparison reports to help them review completed Truck Tool reports
received from participants. Standardized and customized reports can be generated
from the database. There is no independent direct verification by EPA or an
independent third party to ensure that this data is accurate. Further, OTAQ
became aware of a case where a company was alleged to have improperly used
the SmartWay logo. The OTAQ SmartWay team forwarded the case to EPA's
Office of General Counsel in 2011. According to OTAQ, the company removed
the trademark from its website as a result of EPA's action. At the time we were
completing our field work, OTAQ had addressed approximately 35 alleged
6 IFTA simplifies reporting of fuel use taxes by commercial motor carriers. The agreement allows a
trucker/company to obtain one fuel tax license, issued by their base jurisdiction, authorizing them to travel in all
IFTA member jurisdictions. Tax reports containing detailed operations are submitted only to the base jurisdiction.
12-P-0747

-------
violations involving the SmartWay program. Almost all of these alleged
violations were resolved without having to refer the cases to the Office of General
Counsel.
Other Matters
OTAQ's Program Manager stated that OTAQ targeted the large carriers for
participation when they established the SmartWay program, and in the future,
OTAQ will focus on increasing the participation of mid-sized and smaller carriers
as the program grows. Based on a recent study (or "working paper") completed by
researchers from Colorado State University and Miami University (Ohio)7, EPA
could take actions to improve their outreach to the smaller carriers. For example,
the working paper identified some barriers that the SmartWay Transport
Partnership could address that could increase the participation rate of mid-sized
and smaller carriers through increased education. Specifically, the working paper
stated that shippers and carriers that were not SmartWay partners had varying
reasons for not being or planning to be a partner. These reasons included:
1.	The lack of resources or a lack of understanding
2.	The lack of time to spend collecting data and completing paperwork for
the tools
3.	The perceived costs to invest in new technologies being too high
4.	Not being sure what the value would be from the partnership
5.	Some misconceptions where certain types of equipment or technology
may not have been cost-beneficial
The working paper stated that the SmartWay Partnership could better inform the
industry about its organization. The working paper also suggested that the
SmartWay Partnership provide case studies to companies to educate them on the
costs and benefits of becoming a SmartWay partner. For example, the working
paper noted that there were some misperceptions about certain types of equipment
or technology that might be mandated by SmartWay and might not be cost-
beneficial. The working paper stated that some education on SmartWay's role
might also help clear up this misunderstanding. Also, companies could be
educated about innovative practices and technologies that can be implemented
with little to no investment which can achieve a quick return.
Conclusions
OTAQ performs some checks of data provided by industry. However, there is no
direct verification by EPA of data submitted by SmartWay participants. We believe
the SmartWay Transport Partnership program may lose its value if EPA does not
protect its data integrity. We also recognize that the SmartWay program has limited
1 Environmentally Sustainable Transportation: An Executive Summary of Research Findings, October 18, 2011.
This study is considered a working paper that will be finalized after peer review.
12-P-0747
7

-------
resources and that there may be other measures available to address our data
integrity concerns. We believe EPA should implement some form of direct
verification or take other measures to prevent or reduce the likelihood of companies
qualifying for logos by submitting data that overstate their scores.
Recommendation
We recommend that the Assistant Administrator for Air and Radiation:
1. Develop and implement direct verification or other measures to verify
the accuracy of a sample of the self-reported, industry data for the
SmartWay Transport Partnership.
Agency Comments and OIG Evaluation
EPA's Office of Air and Radiation (OAR) agreed with the OIG on the importance
of ensuring the integrity of program results. OAR also stated that it concurs with
the OIG's assessment that as the SmartWay Transport Partnership grows and
matures, its data systems must evolve as well.
In response to recommendation 1, OAR proposed a five step process to better
ensure the accuracy of partner data. OAR noted that it had recently started a
partner data quality project to address the OIG recommendation and enhance the
quality of SmartWay partner self-reported data. Among other things, OAR's
initiative includes site visits to observe how partners collect and quality assure the
data reported in their SmartWay partner submissions. OAR's onsite direct
observations will be accompanied by interviews with key staff involved in partner
data collection and review to clarify how these data management and quality
assurance measures ensure partner data validity. Using this information, OAR
plans to develop and publish a data quality assurance guidance document based
on best practices of a sample of SmartWay shipper, carrier, and logistics partners.
OAR also responded that it will conduct a series of training, communication, and
outreach activities to ensure that all SmartWay partners are aware of and have
access to this information, and fully understand the program's expectations for
partner data integrity. OAR estimated completion of these planned corrective
actions by December 31, 2013.
OAR's planned actions are a step in the right direction that should enhance the
quality of program data. As noted in Step 2 of OAR's response, the Agency is
planning to validate data quality assurance measures for a sample of SmartWay
partners. Additionally, OAR's planned data quality assurance guidance document
and the new training, communication, and outreach should improve quality
controls over the data and improve data accuracy. However, to better promote
partner implementation and use of the new data quality guidance, OAR should
make its use a condition of continued participation in the program. Further, OAR
should also periodically reassess through direct observation that these newly
12-P-0747
8

-------
established controls are still enhancing the integrity and quality of the data.
Therefore, we are keeping this recommendation open in our tracking system.
OAR should describe any additional planned corrective actions in its 90-day
response to the final report.
The Agency's written comments are in appendix D.
12-P-0747
9

-------
Status of Recommendations and
Potential Monetary Benefits
POTENTIAL MONETARY
BENEFITS (In $000s)
Claimed Agreed-To
Amount Amount
measures to verify the accuracy of a sample of the	Air and Radiation
self-reported, industry data for the SmartWay
Transport Partnership.
RECOMMENDATIONS
Planned
Rec. Page	Completion
No. No.	Subject	Status1 Action Official	Date
1 8 Develop and implement direct verification or other 0 Assistant Administrator for 12/31/13
1 0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is unresolved with resolution efforts in progress
12-P-0747
10

-------
Appendix A
General Criteria and Benefits of the
SmartWay Program
To meet SmartWay Transport Partnership goals, partners agree to:
•	Assess freight operations
•	Calculate fuel consumption and carbon footprint
•	Track fuel-efficiency and emission reductions annually
In exchange, EPA ranks and publicizes each partner's performance in the SmartWay Partner List
on EPA's website. EPA does this based on self-reported industry-supplied data. According to
EPA, the partners that are most effective in reducing greenhouse gases and improving air quality
are eligible to use the SmartWay Partner logo from EPA.
According to EPA, participation in SmartWay helps carriers:
•	Identify opportunities to improve efficiency (such as improved fuel economy, resulting in
reduced C02, NOx, and PM emissions)
•	Demonstrate efficiency to potential customers
•	Reduce fuel costs
Further, according to EPA, participation in SmartWay helps shippers and logistics companies:
•	Choose more efficient carriers
•	Assess optimal mode choices
•	Reduce their transport carbon footprint
Studies Indicate SmartWay Helps To Overcome Barriers
A 2009 study by Massachusetts Institute of Technology (MIT) researchers,8 an October 2011
working paper by researchers from Colorado State University and Miami University (Ohio)9
(scheduled to be peer reviewed), and presentations at the November 2011 United States Freight
Sustainability Summit10 corroborate EPA's claims that its SmartWay Transport Partnership
program helps remove marketplace barriers in order to deploy technologies faster. In general, the
program does this by working to overcome barriers such as: (1) lack of clear, reliable
information on technology opportunities; (2) lack of awareness of energy efficient products,
services, and transportation choices; and (3) the need for additional incentives for manufacturers
8	System Dynamics Modeling of the SmartWay Transport Partnership, Second International Symposium on
Engineering Systems, MIT, Cambridge, Massachusetts, June 15-17, 2009.
9	See footnote 7.
10	EPA, American Trucking Associations, Environmental Defense Fund, and the Retail Industry Leaders
Association co-hosted the United States Freight Sustainability Summit in Washington, DC on November 17-18,
2011.
12-P-0747
11

-------
to invest in efficiency research and development. For example, according to a June 2009
SmartWay report by MIT researchers:
•	The mileage of heavy-duty trucks has remained stagnant, between 5-6 miles per gallon
over the past 25 years, despite readily available, cost effective technologies that are able
to improve the efficiency of heavy-duty trucks by at least 12 percent.
•	Several factors account for this apparent market failure including the lack of accurate and
verifiable fuel economy information in the industry and the fragmented nature of industry
where smaller owner operators make up a sizable proportion but often lack the resources
and capital to test and implement technology opportunities.
An October 2011 working paper11 by researchers from Colorado State University and Miami
University (Ohio) also supports that EPA's SmartWay Transport Partnership program helps
improve awareness and is a source of reliable information (the working paper is draft until peer
reviewed). The researchers conducted 36 interviews with shippers and motor carriers associated
with SmartWay as well as shippers and motor carriers not associated with this program. Their
report noted that (1) SmartWay provides a way to standardize emissions measurements so that
they are meaningful; (2) an increasing number of SmartWay shippers appear to be mandating
that all of their carriers become SmartWay certified, or begin to pursue the certification process;
and (3) a few interviewees specifically noted how SmartWay has caused them to critically
examine their operations as they collect information for the tools submitted to EPA.
Presentations at the November 2011 United States Freight Sustainability Summit also support
that the SmartWay Transport Partnership program helps remove marketplace barriers. For
example, a senior manager from a large retailer indicated that his company only hires carriers
that participate in the SmartWay partnership. Also, according to EPA, the Environmental
Counsel to the American Trucking Association stated that the SmartWay Partnership can educate
mid-sized and smaller trucking companies that lack knowledge about the benefits of the
SmartWay Partnership. He said that SmartWay has allowed companies to increase their
profitability since fuel costs are one of their largest operating expenses, and SmartWay's
verification program helps prevent companies from investing in technologies that do not provide
financial benefits.
11 See footnote 7.
12-P-0747
12

-------
Appendix B
SmartWay Partner Recognition System and
Its Relationship to EPA'S Partner Tools
SmartWay Recognition System
Under the current scoring system, EPA ranks carriers in one of five levels of performance, or
bins, based on their emission rates for each pollutant.12 The performance "bins" are further
categorized by each of the various body types of truck carriers and for each pollutant addressed
(C02, NOx, and PM). For example, bin 1 is reserved for the highest performing carriers within a
particular pollutant category (such as C02), and truck body type. A high performing carrier
could be placed in the highest performing bin for having a low C02 emissions rate for a
particular type of truck, but could be placed in a lower bin for NOx emissions as a result of
having a fleet of older, higher NOx-emitting trucks.
SmartWay Truck Tool
Carriers use the SmartWay Truck Tool, an EPA-developed software tool, to report their input
data to EPA. The carriers input the requested data and information directly into the Truck Tool,
and then provide this to EPA so that the Agency can use it to assess their performance.
Information provided includes type/characteristics of fleet, use of particulate matter reduction
technologies, miles driven, gallons of fuel used by types of fuel, average payload, average truck
capacity, and average annual idle hours, etc. The Truck Tool requires carriers to provide the
source of the input data, such as IFTA Form 441, or IRS records, or company electronic, or other
records.
OTAQ uses EPA's emission factors and data provided by the carriers to compute each carrier's
C02, NOx, and PM emissions. To determine C02 emissions, OTAQ uses emissions factors
based on the grams of C02 produced by burning a gallon of the various types of fuels used by
trucks and the number of gallons used by the fleet. To determine NOx and PM emissions, OTAQ
uses emissions factors from EPA's mobile source model (MOVES2010) and carrier provided
data such as the model engine years and vehicle classes in the fleet, speed and operation mode
(running or idle) information, and the number of installed PM emissions control devices. Carriers
also provide activity level data such as fuel consumption, miles traveled, payload, and capacity
volume. This data is used to calculate emission rates for the vehicle classes in the fleet of trucks
in terms of grams of pollutant per mile or grams of pollutant per average payload ton-mile. This
tool was revised in early 2011.
12 OTAQ previously established a scoring system for carriers, logistics companies, and shippers that determined
which companies were eligible to use a SmartWay logo. With changes to the tools in the past year, OTAQ plans to
revise the scoring system for logo eligibility. The new recognition / logo requirements are currently under OTAQ
management review. Existing carriers are recognized on EPA's SmartWay website based on the old scoring criteria.
Until the new recognition system is developed, new carriers will not be awarded logos.
12-P-0747
13

-------
SmartWay Shipper Tool
Shippers use the SmartWay Shipper Tool, an EPA developed software tool, to submit data to
OTAQ on their activities in the past year. The shippers provide the names of carriers they used in
the past year, the number of miles driven each year by each carrier, and payload data on the
shipments.
A revised Shipper Tool has been finalized for use by SmartWay shippers in 2012. This tool had
been on hold for about a year and half, while a new version of the Shipper Tool and the rating
methodology was being finalized. During this time, OTAQ has not assessed the performance of
shippers because the performance recognition system for shippers was being revised to be
compatible with the Truck Tool that was revised in early 2011.
The new Shipper Tool is more sophisticated than the previous Shipper Tool. It allows shippers to
estimate their C02, particulate matter (including PM2.5 and PM10), and NOx emissions
associated with goods movement in the freight rail and trucking sectors. OTAQ revised the new
Shipper Tool to also allow shippers to track their freight-related emissions performance on a
yearly basis and assess different strategies for improving the emissions performance of their
freight operations, including selecting low-emissions carriers and implementing operational
strategies. The new tool allows the shipper to input specific information pertaining to strategies
that impact emissions. The strategies are based on reducing miles or weight from the system. For
example, the shipper can use the tool to calculate the impact of various strategies for reducing
miles traveled, such as: (1) distribution center relocation, (2) retail sales relocation, (3) routing
optimization, and (4) using larger vehicles and/or trailers. Other strategies related to removing
weight from the system include: (1) product weight reduction, (2) package weight reduction, and
(3) vehicle weight reduction.
As with the older editions of the Shipper Tool, trucker performance results calculated by the
Truck Tool will be used as input for the Shipper Tool. The shippers' scores will be computed
based on the performance bin scores of the trucks they use to ship their freight. The shippers that
use more of the higher performing carriers will receive higher rankings and will be eligible to
earn the SmartWay logo. An OTAQ official also said that shippers want to show the public that
they are reducing their carbon footprint by reducing GHG emissions from their business
operations.
Other EPA Developed Tools in Process
In addition to the Truck Tool and Shipper Tool, OTAQ has worked to improve its existing tools
used by its partners and to develop new tools. For example, OTAQ has released versions of tools
for logistic companies and a multi-modal tool that addresses both truck and rail freight emissions
for Class I railroads.13 OTAQ also plans to develop a tool for the emissions for the smaller
railroads. Also, OTAQ announced a new Port Drayage Truck program on June 28, 2011. Under
the SmartWay dray truck initiative, carriers sign an agreement with EPA to track and reduce PM
2.5 emissions by 50 percent and NOx emissions by 25 percent below the industry average over a
three year period.
13 Class I railroads are the largest freight rail companies based on operating revenue.
12-P-0747
14

-------
Appendix C
Prior OIG Reports Related To
EPA Voluntary Programs
Prior OIG reports related to EPA voluntary programs included:
•	Evaluation Report, Report No. 08-P-0206, Voluntary Greenhouse Gas Reduction
Programs Have Limited Potential, July 23, 2008
•	Evaluation Report, Report No. 2007-P-00013, Performance Track Could Improve
Program Design and Management to Ensure Value, March 29, 2007
•	Evaluation Report, Report No. 2007-P-00003, Partnership Programs May Expand EPA's
Influence, November 14, 2006
•	Evaluation Report, Report No. 2007-P-00041, Voluntary Programs Could Benefit from
Internal Policy Controls and a Systematic Management Approach, September 25, 2007
•	Evaluation Report, Report Number: 2005-P-00007, Ongoing Management Improvements
and Further Evaluation Vital to EPA Stewardship and Voluntary Programs, February 17,
2005
•	Evaluation Report, Report No. 10-P-0040, ENERGY STAR Program Integrity Can Be
Enhanced Through Expanded Product Testing, November 30, 2009
•	Evaluation Report, Report No. 09-P-0061, Improvements Needed to Validate Reported
ENERGY STAR Benefits, December 17, 2008
•	Evaluation Report, Report No. 2007-P-00028, ENERGY STAR Program Can Strengthen
Controls Protecting the Integrity of the Label, August 1, 2007
12-P-0747
15

-------
Appendix D
Agency Comments to Draft Report
MEMORANDUM
SUBJECT: Office of Air and Radiation's (OAR) Response to OIG Draft Report: EPA Could
Improve the SmartWay Transport Partnership by Implementing a Direct
Data Verification Process, Project No. OPE-FY11-0011
FROM: Gina McCarthy
Assistant Administrator
TO:	Carolyn Copper
Acting Assistant Inspector General for Program Evaluation
Office of Inspector General
Thank you for the opportunity to comment on the Office of Inspector General (OIG) draft report,
EPA Could Improve the SmartWay Transport Partnership by Implementing a Direct Data
Verification Process, Project No. OPE-FY 11-0011, dated June 29, 2012, which focused on the
Agency's SmartWay Transport Partnership with the freight industry and how OAR could
continue to uphold the integrity of program results in the future. This review, which is one of
several OIG evaluations that have focused on EPA voluntary programs, is aimed at assisting
OAR in improving its oversight of information submitted to the SmartWay Transport
Partnership.
OAR appreciates the effort by the OIG to thoroughly understand the complexity of the
SmartWay partnership and its value to the shipping community, freight industry and the general
public. As the OIG noted, SmartWay is recognized across the industry and by EPA regulatory
programs for its leadership in identifying, enabling and encouraging cleaner and more efficient
goods movement practices and technologies. The report also cites a number of measures that
OAR has already implemented to strengthen the SmartWay program and the integrity of its data,
data collection methods, and reporting. OAR took these steps for reasons identified by the OIG
in its review - namely, to safeguard the integrity of the program; to protect the value of the
SmartWay brand; and, to ensure that the robustness and accuracy of partner reported data will
continue to keep pace with the program's growing impact on the shipping and freight
communities. These steps the OIG cited as already implemented include: utilizing previously
submitted data by businesses that already mandatorily submit to the government (e.g., IFTA
reports, motor vehicle registrations) as the basis for SmartWay program information;
development of rigorous internal data quality assurance controls including annual data
comparison reports to ensure consistency and to eliminate input errors; comprehensive review
and cross-checking of partner data before it is accepted by the program; and, diligence in
resolving any issues that may arise regarding improper use of the SmartWay brand. The OIG
12-P-0747
16

-------
review recommends that OAR build upon these improvements by direct verification of
SmartWay partner data or other measures.
OAR agrees with the OIG on the importance of the integrity of program results. OAR also
concurs with the OIG's assessment that as the SmartWay Transport Partnership grows and
matures, its data systems must evolve as well. OAR has already started a partner data quality
project that will address the OIG recommendation. A summary of the recommendation, its
associated actions and projected completion dates are provided below.
Recommendation:
Develop and implement direct verification or other measures to verify the accuracy of a sample
of the self-reported, industry data for the SmartWay Transport Partnership.
EPA response: In addition to the steps that OAR has already implemented to ensure the integrity
of the SmartWay program, OAR recently started an initiative designed to enhance the quality of
SmartWay partner self-reported data. This process consists of five steps that OAR anticipates
will be completed on or before December 31,2013.
•	Step 1: Identify a sample of SmartWay shipper, carrier, and logistics partners that have
demonstrated program compliance with a credible quality process or certification program,
such as ISO certification, Six-Sigma designation, or similar quality assurance system. Staff
will interview each candidate to assess its readiness and suitability for participation before
selecting the most appropriate candidates. This action has already been completed.
•	Step 2: Conduct site visits of this sample of partners. These visits will include first-hand
observation and recording of the processes and safeguards employed to collect, handle,
check, manage, track and preserve the data reported in their SmartWay partner submissions.
Partner site visits will be accompanied by discussions and interviews with key staff involved
in data collection and review to clarify how these data management and quality assurance
measures ensure partner data validity. This activity is occurring during July and August of
2012.
•	Step 3: Assess and synthesize the results of these interviews and site visits into a
comprehensive and consistent set of best practices available for all SmartWay partners to use
in their data collection, management and quality assurance procedures. OAR anticipates that
the first draft of this guidance document will be available by December, 2012.
•	Step 4: Obtain internal and external review of the draft document. The completed document
will clearly establish uniform and rigorous quality assurance measures and practices for
partner data. The SmartWay partner tool guides and technical guidances will be updated to
reflect the availability of the data quality assurance guidance. OAR anticipates that the
guidance document and related materials will be finalized on or before March 31, 2013.
•	Step 5: Publish the guidance document and conduct partner outreach and training. OAR will
publish the guidance document on the SmartWay website by August 2013. From September
through December of 2013, OAR will plan and conduct a series of training, communication
and outreach activities to ensure that all SmartWay partners are aware of and have access to
this information, and fully understand the program's expectations for partner data integrity.
12-P-0747
17

-------
We anticipate that implementing this partner data quality assurance measure is responsive to
OIG's recommendation. If you have any questions, please contact me or SmartWay Center
Director, Cheryl L. Bynum (734-214-4844).
12-P-0747

-------
Distribution
Office of the Administrator
Assistant Administrator for Air and Radiation
Director, Office of Transportation and Air Quality
Deputy Director, Office of Transportation and Air Quality
Agency Follow-Up Official (the CFO)
Agency Follow-Up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for External Affairs and Environmental Education
Audit Follow-Up Coordinator, Office of Air and Radiation
12-P-0747

-------