Content Analysis of Model Year 2014
Professional Automotive Reviews
£% United States
Environmental Protect
Agency
-------
Content Analysis of Model Year 2014
Professional Automotive Reviews
Assessment and Standards Division
Office of Transportation and Air Quality
U.S. Environmental Protection Agency
Prepared for EPA by
RTI International
EPA Contract No. EP-C-11-045
Work Assignment No. 3-01
This technical report does not necessarily represent final EPA decisions or
positions. It is intended to present technical analysis of issues using data
that are currently available. The purpose in the release of such reports is to
facilitate the exchange of technical information and to inform the public of
technical developments.
NOTICE
4>EPA
United States
Environmental Protection
Agency
EPA-420-R-16-015
October 2016
-------
CONTENTS
Section Page
1. Introduction 1-1
2. Methods 2-1
2.1 Unitizing 2-1
2.2 Sampling 2-2
2.2.1 Identification of Relevant Websites 2-2
2.2.2 Sampling Substantive Auto Reviews for Model Year 2014 2-3
2.2.3 Analysis and Recording Procedures 2-4
2.3 Recording/Coding 2-5
2.3.1 Coding Framework 2-5
2.3.2 Coder Training and Inter-Coder Reliability Evaluation 2-6
3. Results 3-1
3.1 Database Structure 3-1
3.2 Summary Statistics 3-2
3.3 Summary Tables of Content Analysis Output 3-4
4. Limitations (Sources of Uncertainty) 4-1
References R-l
Appendix
A Glossary and Coding Guidelines A-l
iii
-------
FIGURES
Number Page
2-1 WA 3-01 Coding Production Work Flow 2-5
iv
-------
TABLES
Number Page
2-1 Monthly Unique Views per Website, as Reported by Quantcast.com and
Compete.com 2-3
2-2 Coding Frame 2-7
3-1 List of Variables for Each Auto Review 3-1
3-2 Auto Reviews by Website 3-2
3-3 Auto Reviews by Manufacturers 3-2
3-4 Auto Reviews by Class 3-3
3-5 Efficiency Technology's Positive, Negative, or Neutral Evaluations by Auto
Reviews 3-5
3-6 Efficiency Technology's Total Number of Positive, Negative, or Neutral
Evaluations 3-6
3-7 Operational Characteristics: Positive, Negative, or Neutral Evaluations by Auto
Reviews 3-7
3-8 Operational Characteristics: Total Number of Positive, Negative, or Neutral
Evaluations 3-8
v
-------
[This page intentionally left blank.]
vi
-------
1. INTRODUCTION
Work Assignment 3-01 consists of a content analysis of professional vehicle reviews for
Model Year 2014. According to Krippendorff (2013), content analysis is "a research
technique for making replicable and valid inferences from texts to the contexts of their use."
RTI International assisted the U.S. Environmental Protection Agency (EPA) with developing
a methodology to perform the content analysis, assembled the auto reviews that would be
included in the analysis, coded a large scale of unstructured data in the form of 1,023
reviews and summarized the results.
EPA identified the three following research questions for this study:
1. How do professional auto reviewers assess their experiences of fuel-saving
technologies and vehicle operational characteristics? Do they have negative
assessments of technologies or characteristics, or do they view them positively?
2. Are there identifiable circumstances that distinguish positive from negative
assessments for each technology and characteristic (e.g., do negative
assessments occur in certain manufacturers, or certain review sources)?
3. What do the vehicle reviews say in the assessments about the particular
technologies and vehicle operational characteristics?
The results of the analysis are used to examine how professional auto reviewers assess their
experiences of fuel-saving technologies and vehicle operational characteristics, with the goal
of providing insight into potential hidden costs which may affect consumer acceptance of
vehicles equipped with fuel-saving technologies.1
1 Throughout this document, the terms "vehicle characteristics" and "vehicle operating
characteristics" are used interchangeably to refer to any attribute of the vehicle that can
only be evaluated by the reviewer based on sensory perception or instrumented
measurement while driving the vehicle.
1-1
-------
-------
2. METHODS
Content analysis is an established technique for summarizing written, spoken, or visual
information in a systematic manner (Moeller 1963, Carney 1972, Weber 1990, Berg & Lune
2012, Krippendorff 2013). This method is unobtrusive and can be replicated. The text is
coded and categorized by human or by computer, and results can also be quantified into
statistical data. Although computer coding is not as time consuming as human coding, the
human brain is well-suited to handle the coding of content beyond the accounting of words
and phrases. This is crucial when part of the content is contextually based or may contain
"hidden" meaning that is not readily apparent. For example, professional auto reviews often
provide an evaluation relative to another vehicle using language such as "quieter than," "not
as quiet as," or "not as noisy as." The coder needs to examine the context to determine if it
was a positive, negative, or neutral evaluation. Thus, we used human coders when
evaluating professional auto reviews. Yet, the human approach is subject to coder bias if left
unchecked. We minimized the inherent variability among human coders by (1) using
experienced coders, (2) conducting comprehensive training on reading and capturing
context of auto reviews and assigning appropriate codes, and (3) evaluating inter-coder
reliability. In addition, an RTI content analysis expert regularly met with the coders and
adjudicator to resolve unanticipated issues, such as when the auto review content did not
appear to correspond with the existing coding frame. In these cases, we provided additional
instructions in the use of existing codes or in some instances, EPA chose to revise or add to
the coding frame. A well-structured content analysis methodology has enabled us to
systematically evaluate a large quantity of information contained within the materials.
This section describes the details of the methodology used to perform the content analysis,
which generally includes the following steps (see Krippendorff 2013 for more details):
Unitizing: defining the units of analysis (in this case, auto reviews);
Sampling: defining a manageable subset of units that are statistically or conceptually
representative of the set of all possible units, the population, or universe of interest;
Recording/coding: relying on coding instructions, including identifying key words,
phrases, or ideas in a text (e.g., mention of a technology) and coding evaluations of
how they are used in the text using predetermined guidelines (e.g., "easy" or
"smooth" may indicate a positive evaluation of the technology, while "difficult" or
"choppy" may indicate a negative evaluation); and
Reducing or analyzing the data to manageable representations by relying on
established methods for summarizing or simplifying data.
2.1 Unitizing
For the purpose of this analysis, one auto review equals one sampling unit. A coded unit is a
phrase or sentence within a sampling unit (review) that is a description of one of the
2-1
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
characteristics of interest. Each coded unit is treated separately, and a review therefore
may contain multiple records/coding units. Consequently, the number of sampling units
does not necessarily equal the number of recording/coding units.
Some reviews (sampling units) include updates provided by the same author. In this case,
the updates and the initial review are treated as one sampling unit. Some Websites publish
reviews of the same model of vehicle but by different authors. In this case, the reviews
(sampling units) are treated separately. Finally, if the same reviewer provides assessments
of more than one vehicle model in the same article, the assessment for each model is
treated as a different sampling unit.
2.2 Sampling
The first part of the study included the selection of Websites and professional auto reviews
to be analyzed by applying inclusion and exclusion criteria. As described in more detail
below, EPA identified six Websites containing professional auto reviews. About 1,091
reviews were accessed for content analysis, with dates through end of July 16, 2014.2 Some
of these were discarded because they did not meet the criteria for significant reviews,
discussed below. As noted above, reviews were split if more than one model of vehicle was
included. Ultimately, the content analysis was performed on 1,023 auto reviews. The
articles were read and coded using a coding frame (see details in the next section).
2.2.1 Identification of Relevant Websites
EPA followed a set of specific procedures to identify which Websites' reviews would be
analyzed using content analysis methodologies. The goal of these procedures was to identify
Websites that consumers are most likely to consult when making car buying decisions.
Using Google and Yahoo search engines, EPA followed a conceptual hierarchy to
systematically narrow the search of relevant Websites in multiple stages, consistent with the
practice of relevance sampling described in Krippendorff (2013):
1. Searched by keywords "new cars," "buying a new car" and "auto reviews."
2. Excluded Websites that were not identified in the first page of the search result. We
note there may be bias in which Websites are on the first page of the search results,
but believe most people do not go beyond the first page. This returned a list of 30
Websites.
3. Among the 30 Websites identified in the first page of the search result, we excluded
advertisements, Websites that did not have reviews, or Websites that only contained
consumer reviews. This returned 8 Websites that have national and professional auto
reviews.
4. Obtained monthly unique views from Quantcast.com and Compete.com to gauge
Website popularity (accessed December 26, 2013), and then excluded 2 Websites
2 Most reviews of Model Year 2014 (MY2014) vehicles were written early in the model year.
We cut off collection of reviews at this point to begin the analysis.
2-2
-------
Section 2 — Methods
that had less than one million unique views in both Quantcast.com and
Compete.com.
5. Identified additional sites through Compete.corn's feature that provides competitive
ranking of similar Websites. This feature returned 1 additional Website that met
Criteria 3 and 4. Adding this step may help to alleviate the bias of only identifying
Websites that appear on the first page of search results.
6. The remaining 7 Websites were screened to include only Websites with reviews that
evaluated vehicles and technologies. Each reviewer must have gone beyond a basic
specification list, have an independent assessment of vehicle quality, and have test-
driven the vehicle. The six Websites shown in Table 2-1 satisfied each of these
criteria and constitute the final set of Websites used in this study.
Table 2-1. Monthly Unique Views per Website, as Reported by Quantcast.com
and Compete.com
Website Quantcast.com Compete.com
Monthly Unique Views (M=million; K=thousand)
Automobilemag.com
391 K
1 M
Caranddriver.com
1.2 M
847 K
Motortrend.com
1.6 M
1.8 M
Autotrader.com
3.8 M
6.1 M
Consumerreports.org
3.9 M
4.3 M
Edmunds.com
6.2 M
5.8 M
2.2.2 Sampling Substantive Auto Reviews for Model Year 2014
EPA chose to analyze only "substantive" reviews in the content analysis, as these reviews
are likely to contain more detailed assessments of fuel-saving technologies than reviews
which solely list vehicle specifications. EPA defined a substantive review using the following
criteria:
1. The vehicle is test driven.
2. The review evaluates the vehicle and its technologies. The reviewer must go beyond
a basic specification sheet and have some independent assessment of vehicle
operational characteristics.
3. Assessments of some technological features are included (e.g., transmission, engine,
aerodynamics, mass reduction, hybrid technologies, or stop-start).
4. The review identifies configuration or trim level (i.e., the specific version of the
model when there may be multiple versions of the model).
5. The vehicle is available in the United States for sale to the general consumer and is
not a prototype or development/concept vehicle.
2-3
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
6. The manufacturers are subject to EPA's greenhouse gas standards (except for small
volume manufacturers that produce fewer than 5,000 vehicles per year have more
flexibility in complying with EPA's greenhouse gas standards). Small vehicle
manufacturers are not included in the study, because they do not face the same
incentives to add fuel-saving technologies as manufacturers who are subject to the
standards.
2.2.3 Analysis and Recording Procedures
To manage the large number of reviews, RTI created "replicates"—several equal-sized,
stratified subsamples of auto reviews. To do this, all auto reviews were stratified (grouped)
by Make and then assigned randomly to each replicate. A total of 30 replicates containing
either 36 or 37 auto reviews each were created. This approach ensured adequate
representation of auto reviews in the event that there were insufficient resources to read
and assess all reviews; it enabled us to release a small, manageable set of replicates at
first, and then release more replicates as resources allowed. At the end of the coding phase,
we coded all replicates (i.e., the entire population of "substantive" auto reviews from these
Websites).
When using a replicate strategy, each open replicate should be fully coded before the next
one is opened. In addition, sample weights are required for any sample in which the
probability of selection is not equal for each selected unit (in other words only a random
sample of equal probability of selection does not require weights). Applying sample weights
allows us to reconfigure the sample as if it was a simple random draw of the population of
reviews and yield an accurate extrapolation of technologies. In this case, we did not create
and apply weights because the entire population of reviews were read and assessed, and
therefore there would be no need to apply weights.
Although we attempted to include only substantive reviews in the replicates, we discovered
during the process of reading and assessing that there were 197 reviews that did not meet
the inclusion and exclusion criteria for being "substantive" reviews, as discussed earlier.3
Once identified, they were excluded from the coding because they were out of scope for this
study. In addition, when the same reviewer provided assessment of more than one model of
a vehicle in the same article, the assessment for each model was coded. This adjustment
added 129 separate reviews. At the end of the coding phase, a total of 1,023 auto reviews
was coded for analysis. The articles in the analysis were accessed between July 8, 2014,
and August 14, 2014.
3 Nineteen (9.6%) of the 197 out-of-scope reviews were not test driven, or not available in
the United States for sale to the general consumer. Therefore, they were not "substantive"
reviews.
2-4
-------
Section 2 — Methods
2.3 Recording/Coding
Our method incorporates the use of human coders, who read each review and then
identified and coded the recording units (which are professional reviewers' comments in a
contiguous passage of text). To track the recording units, we used NVivo 10, one of the
leading software programs for coding and organizing unstructured data (QSR
International).4 Figure 2-1 shows the coding production workflow.
Figure 2-1. WA 3-01 Coding Production Work Flow
WA 3-01 Coding Production Work Flow
Last Updated: 09/24/2014
Coding (8 weeks)
Team lead
Coders
Coders
Coders
Make assignments by replicates 3
times during each two-week period
(i.e. a total of 3 replicates every 2
weeks; one replicate at a time.)
Receive, acknowledge,
and confirm
assignments.
Code auto
reviews in Nvivo
in independent
workstations.
Code on
average 50 auto
reviews a week.
Team lead to merge
coded auto reviews into master file
Quality Control and Continuous Improvements
Coders
Team lead
Work Assignment Leader
Compile recurring issues and
questions every 15th auto
reviews coded.
Provide
resolutions
within 24 hours.
Facilitate weekly meetings to:
• Confirm coding production goal the
coming week
• Confirm hours per case (HPC)
• Discuss outstanding coding issues
and questions
Receive, acknowledge
and confirm
understanding of
feedback and agreement.
Coders
Evaluate work and provide feedback:
• 100% of the first 40 coded reviews
per coder.
• 10% each week of coded reviews per
coder.
Team lead
2.3.1 Coding Framework
The coding frame was developed by EPA based on an assessment of the key characteristics
to address, and is shown in Table 2-2. Three new codes and two clarifications were added
during the coding phase, and are highlighted in yellow in the Table. During the coding
phase, human coders read each review, identified recording units, and assigned a code
4 For more information about the NVivo 10 software, see
http://www.qsrinternational.com/default.aspx.
2-5
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
(Positive, Negative, or Neutral) to the recording unit. As described earlier, a recording unit
is defined as the shortest possible passage of text that still captures the context of a
reviewer's evaluation of a vehicle characteristic. Because multiple ideas are often contained
within a single passage of text, a recording unit can contain evaluations of more than one
characteristic. In addition, positive and negative assessments are clear sentiments, while a
neutral assessment cannot be clearly described as positive or negative. The "Neutral" value
is coded when the reviewer does not demonstrate an intensity of opinion that can be clearly
discerned to be positive or negative.
Coding occurred at the lowest coding level ("child codes"), as shown in Table 2-2. An
example from an auto review of the Honda Accord Hybrid5 is as follows:
"Class leading energy conservation" is coded Positive-Fuel Economy for the vehicle's
operational characteristics.
"There is a prominent wail when the car accelerates briskly or climbs a grade" is
coded Negative-NVH-Noise-Powertrain for the vehicle's operational characteristics.
¦ "Low-rolling-resistance tires do sacrifice some grip, but the hybrid's 184-foot, 70-to-
zero stopping distance was only four feet longer than that of the last nonhybrid
Accord we tested" is coded Neutral-Low rolling resistance tires for vehicle technology
and also Neutral-Braking for the vehicle's operational characteristics.
Detailed coding guidelines and a glossary were developed for coder training and can be
viewed in Appendix A.
Once the recording units for each auto review were coded, we constructed a database
containing the count of Positive, Negative, and Neutral coded values. In addition to having
recording units, we assigned a code to the overall review sentiment. Based on the reading
of the review as a whole as well as the coded results, we assigned a verbal category
(Positive, Negative, or Mixed) to each review at the sampling unit.
2.3.2 Coder Training and Inter-Coder Reliability Evaluation
We used two coders and one adjudicator ("team lead" in Figure 2-1), each of whom is
experienced with coding a large amount of texts in a defined time period and has
demonstrated the ability to understand the coding instructions and apply them consistently
throughout the repetitive coding process. The adjudicator's skills had been approved by EPA
ahead of time by coding five auto reviews for EPA's direct evaluation and feedback. The
coder training included in-person and independent learning components. The coding team
attended a 2-day in-person training, with most of the time spent on coding exercises, how
to use various resources as references, and how to evaluate inter-rater reliability. After the
in-person training, the coders spent 5 days in independent learning. Training materials were
approved by EPA prior to training and included the following:
5 http://www.caranddriver.com/reviews/2014-honda-accord-hybrid-test-review
2-6
-------
Section 2 — Methods
Table 2-2. Coding Frame
Parent Hierarchy
Coding Level (Child Codes)
Low rolling resistance tires
Low rolling resistance tires
Electronic power steering
Electronic power steering
Powertrain Engine
Turbocharged
GDI
Cylinder deactivation
Diesel
Hybrid
>•
o>
o
Plug-in hybrid electric
Full electric
o
c
Stop-start
£
(J
General engine
V
1—
Transmission
High-speed automatic
>•
CVT
c
DCT
V
u
General transmission
IE
General
General powertrain
LJJ
Powertrain
Electric assist or low drag brakes
Electric assist or low drag brakes
Lighting-LED
Lighting-LED
Mass reduction
Mass reduction
Active ride height
Active ride height (active aerodynamics)
Active grill shutters
Active grill shutters (active aerodynamics)
Active air dam
Active air dam (active aerodynamics)
Passive aerodynamics
Passive aerodynamics
(A
U
(A
V
u
re
re
£
u
re
c
o
V
a
O
Drivability
General
Drivability
Handling
Acceleration
Braking
General drivability
General handling
Steering feel/controllability/responsiveness
Cornering ability/grip/balance/body control
General acceleration
Acceleration feel/smoothness/responsiveness
Acceleration capability/power/torque
General braking
Brake feel/responsiveness
Stopping ability
Noise
Tire/road
Wind
Interior
Powertrain
General noise
Vibration
Chassis
Powertrain
General vibration
Ride comfort
Ride comfort
Fuel economy
Fuel economy
Range
Range
Charging
Charging
2-7
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
Coding frame (see Table 2-2)
Glossary and guidelines (see Appendix A)
A list of the 2014 vehicles and the Websites that were sampled
Relevant technologies and vehicle operational characteristics
A set of 12 example coded auto reviews. They were used for lecture, coding
exercises, and "checkout" exams. Four of the 12 auto reviews were coded by EPA.6
Eight were coded by the RTI adjudicator and submitted to EPA for approval prior to
the training.7 These example auto reviews were selected purposely to represent
different vehicle makes, models, and Websites
During the training phase, we compared the codes assigned by the coders against the
example coded review as a way to render information on inter-coder reliability. It is
examined in two ways:
Percentage agreement: the number of units of agreement divided by the total units
of measure within the data item, displayed as a percentage.
6 EPA provided example codes for four auto reviews: (1) Honda Accord Hybrid (Accessed on
7/23/2014: http://www.caranddriver.com/reviews/2014-honda-accord-hybrid-test-review);
(2) Infiniti Q50S (Accessed on 7/10/2014:
http://www.motortrend.com/roadtests/oneyear/sedans/1402_2014_infiniti_q50s_37_arrival
&
http://www.motortrend.com/roadtests/oneyear/sedans/1404_2014_infiniti_q50s_37_updat
e_l/); (3) Corolla LE Plus (Accessed on 7/17/2014:
http://www.consumerreports.ora/cro/tovota/corolla/road-test.htm'): (41 Hyundai Elantra
(Accessed on 7/22/2014: http://www.edmunds.com/hvundai/elantra/2014/ratina-
details.html?sub=sedan)i
7 RTI adjudicator coded eight auto reviews that were approved by EPA for training: (1) Ford
Focus (Accessed on 7/21/2014: http://www.edmunds.com/ford/focus/2014/rating-
details.html?sub=sedan'): (2) Mitsubishi Mirage (Accessed on 7/25/2014:
t/): (3) Lexus IS 250 (Accessed on 7/24/2014:
http://www.motortrend.com/roadtests/onevear/sedans/1401
2014
lexus
is
250
lona ter
m arrival/:
http://www.motortrend.com/roadtests/onevear/sedans/1402
2014
lexus
is
250
update 1
Zi
http://www.motortrend.com/roadtests/onevear/sedans/1403
2014
lexus
is
250
update 2
/): (4) Nissan Maxima (Accessed on 7/24/2014:
http://www.autotrader.com/research/article/car-reviews/215792/2014-nissan-maxima-
new-car-review.isp): (5) BMW 435i (Accessed on 7/17/2014:
http://www.automobilemaa.com/reviews/driven/1307 2014 bmw 435i/): (6) Tovota
Corolla (Accessed on 7/23/2014: http://www.caranddriver.com/reviews/2014-tovota-
corolla-s-automatic-test-review'): (7) Porsche Cavman (Accessed on 7/18/2014:
http://www.automobilemaa.com/reviews/driven/1305 2014 Porsche cayman/): (81
Chevrolet Corvette 3LT (Accessed on 7/21/2014:
http://www.consumerreports.ora/cro/chevrolet/corvette/road-test.htm).
2-8
-------
Section 2 — Methods
Kappa coefficient8: a statistical measure that takes into account the amount of
agreement that could be expected to occur through chance. To calculate Kappa
coefficient, NVivo analyzes each code assigned in an auto review for instances of
coding agreement. Kappa is scored between 0 and 1, with 0=no agreement and
l = perfect agreement. Since some variation in the portion of text highlighted was
likely always going to happen (coders may determine to highlight more text for
contextual reasons), confidence standards at or above 0.75 are accepted as an
indicator of "excellent agreement."
Not surprisingly, the first evaluation showed that the agreement was not high. The
adjudicator also examined each code assigned by the coder and debriefed on whether and
why it was accurate or not accurate based on the guidance provided. This step is crucial to
provide feedback to the coders that they can use to improve their future coding. At the end
of the group training, the coders reached above 90% agreement and a Kappa coefficient of
0.6 (fair agreement), again followed by code-by-code review and debriefing. After the group
training, we continued with independent learning and assessment until the coders reached
above 90% agreement and a Kappa coefficient of 0.8. The coding operation officially started
after that.
At the onset of the coding operation, the first 40 auto reviews coded by each coder were
verified by the adjudicator. This achieved a 100% verification of 80 auto reviews that
facilitated early detection of problems.
Coder #1 assigned 799 codes, of which 35 codes and 20 sentiments were flagged for
correction and discussion by the adjudicator.
Coder #2 assigned 754 codes, of which 26 codes and 26 sentiments were flagged for
correction and discussion by the adjudicator.
After the 100% verification of those 80 auto reviews, the adjudicator randomly selected
about 10% of coded reviews to be verified each week. The coders and the adjudicator also
met after every 30 coded auto reviews to resolve coding issues. By the end of the coding
period, we achieved an overall 10% of the 1,023 coded reviews being verified using the
measures discussed in this section.
8 See Cohen (1960). For quick reference, see http://en.wikipedia.org/wiki/Cohen's_kappa.
2-9
-------
-------
3. RESULTS
We coded a total of 1,023 auto reviews for MY2014 vehicles from six Websites, representing
36 manufacturers and 14 official vehicle Class categories (using EPA Size Class definitions,
available from Fueleconomy.gov). In terms of publication dates, 43% of the auto reviews
were published before 2014, and the rest were published in 2014 (26%), or no dates were
recorded (31%).9
3.1 Database Structure
The coding results are reported in an Excel-based database. We maintained the database in
Excel to allow easy access and manipulation of the data by EPA. Each row represents one
review, columns are variables as shown in Table 3-1. The recording units and their
Positive, Negative, and Neutral evaluations are also shown in the Excel database, in a
separate tab.
Table 3-1. List of Variables for Each Auto Review
File name ¦ Engine type
Website name ¦ Transmission type
Website URL ¦ Overall assessment of the auto review: Positive,
Date of review Negative, or Neutral
Date accessed " A" eff'ciency technology types and their Positive,
Negative, or Neutral evaluations
Vehicl6 make
All operational characteristics and their Positive,
Vehicle model Negative, or Neutral evaluations
Vehicle class
Data recorded for the Engine and Transmission Type variables are those given in the
information provided in the auto reviews. If the information was not available in the auto
review, the variables were left blank. For Hybrid cars, EPA had instructed leaving the
Transmission variable blank and recording what the auto review provides for Engine, if
available. We also relied on the level of details provided in the auto reviews to record the
specific model and trim information (trim was recorded with the model).
9 MY2014 vehicles could have been available for test drive by professional auto reviewers
prior to 2014.
3-1
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
3.2 Summary Statistics
Table 3-2 below shows the number of auto reviews coded by Website. Auto reviews from
Autotrader.com (n=233), Motortrend.com (n=223), and Caranddriver.com (n=221)
combine to make up about 66% of the coded auto reviews, followed by Automobilemag.com
(n = 145), Edmunds.com (n = 113), and Consumerreports.org (n=88).
Table 3-2. Auto Reviews by Website
Website
Counts
Autotrader.com
233
Motortrend.com
223
Caranddriver.com
221
Automobilemag.com
145
Edmunds.com
113
Consumerreports.org
88
Total
1,023
Thirty-six makes were represented in the coded auto reviews, as summarized in Table 3-3.
Although there was only one auto review for Smart, our coding results included 88 auto
reviews of Chevrolet vehicles. The difference in the number of auto reviews reflects the
range of models/trims that were available for each make, as well as the professional
reviewer's decision to provide substantive reviews for a vehicle.
Table 3-3. Auto Reviews by Manufacturers
Make
Count
Make
Count
Make
Count
Chevrolet
88
Honda
34
Land Rover
15
Mercedes
74
Porsche
34
Ram
14
BMW
69
Jaguar
28
Mini Cooper
11
Toyota
63
Buick
27
Bentley
11
Mazda
49
Infiniti
25
Rolls Royce
9
Ford
47
Subaru
25
Fiat
8
Kia
44
Dodge
24
Ferrari
7
Nissan
42
Acura
24
Lincoln
6
Jeep
42
Lexus
23
Volvo
5
Audi
37
GMC
20
Chrysler
4
Volkswagen
37
Hyundai
19
Scion
4
Cadillac
36
Mitsubishi
17
Smart
1
3-2
-------
Section 3 — Results
Fourteen vehicle classes, as defined by Fueleconomy.gov, were represented in the auto
reviews, mostly in the Midsize (n=233), Compact (n = 173), and Small SUV (n = 145) classes.
As shown in Table 3-4, there were less than 10 Midsize Station Wagons (n=8), Van
Passengers (n=5), and Small Pickup Trucks (n = l) represented in the auto reviews. Twenty-
three auto reviews do not have information on class.
Table 3-4. Auto Reviews by Class (in order of the size of the vehicle class)
Class
Count
Not mentioned
23
Subcompact
79
Minicompact
13
Compact
173
Two Seater
82
Midsize
233
Midsize Station Wagon
8
Van Passenger
5
Minivan
15
Small Station Wagon
26
Small Pickup Truck
1
Small SUV
145
Standard SUV
89
Large Sedan
87
Standard Pickup Truck
44
Total
1,023
In addition, because publication dates of the reviews varied widely (from 5/8/2012 to
7/16/2014),10 we categorized them into two groups: "before calendar year 2014" (43% or
n=440) and "calendar year 2014" (25% or n=260). Note that 32% (n=323) of auto reviews
did not have dates, including 233 from Autotrader.com, 88 from Consumerreports.org, and
2 from Edmunds.com.
10 Some auto reviews of MY2014 vehicles were published as early as 5/8/2012, most likely
because those models were available for test drive at that time.
3-3
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
3.3 Summary Tables of Content Analysis Output
Table 3-5 provides the number of auto reviews that had Positive, Negative, or Neutral
evaluations of efficiency technology. For example, when low rolling resistance tires receives
positive evaluations multiple times in an auto review, we report that auto review only once
under Positive low rolling resistance tires. If the technology receives positive and negative
evaluations in the same auto review, we report that auto review once for Positive and once
for Negative evaluations. In comparison, Table 3-6 provides the total number of Positive,
Negative, or Neutral evaluations of efficiency technology based on individual codes.
Table 3-7 provides the number of auto reviews that had Positive, Negative, or Neutral
evaluations of operational characteristics. For example, when Steering
feel/controllability/responsiveness receives positive evaluations multiple times in an auto
review, we report that auto review only once under Positive Steering
feel/controllability/responsiveness. If the operational characteristic receives both positive
and negative evaluations in the same auto review, we report that auto review once for
Positive and once for Negative evaluations. In comparison, Table 3-8 provides the total
number of Positive, Negative, or Neutral evaluations of operational characteristics.
3-4
-------
Section 3 — Results
Table 3-5. Efficiency Technology's Positive, Negative, or Neutral Evaluations by
Auto Reviews
Efficiency Technology
Categories
Coding Level
Positive
Negative
Neutral
Total3
Active air dam
Active air dam (active
aerodynamics)
6
0
0
6
Active grill
shutters
Active grill shutters
(active aerodynamics)
1
0
0
1
Active ride
height
Active ride height
(active aerodynamics)
2
0
1
3
Low rolling resistance tires
Low rolling resistance
tires
8
4
5
17
Electronic power steering
Electronic power
steering
121
47
42
210
Powertrain Engine
Turbocharged
182
20
23
225
GDI
54
6
6
66
Cylinder deactivation
30
1
4
35
Diesel
53
9
11
73
Hybrid
45
16
10
71
Plug-in hybrid electric
18
4
6
28
Full electric
14
2
6
22
Stop-start
29
15
8
52
General engine
453
105
98
656
Transmission
High speed automatic
275
60
81
416
CVT
57
36
21
114
DCT
42
16
12
70
General transmission
112
32
29
173
General
Powertrain
General powertrain
80
9
20
109
Electric assist or low drag
brakes
Electric assist or low
drag brakes
3
1
3
7
Lighting-LED
Lighting-LED
17
1
2
20
Mass reduction
Mass reduction
67
0
9
76
Passive
aerodynamics
Passive aerodynamics
29
4
7
40
Total
1,698
388
404
2,490
a The Total column slightly overstates the number of reviews that mentioned the technology, because
one review could mention the same technology more than once. For example, if the technology
receives positive and negative evaluations in the same auto review, we report that auto review once
for Positive and once for Negative evaluations. That renders "2" for that row in the total column.
(But if the same technology receives positive [or negative] evaluations more than once, we report
that auto review only once as having received positive [or negative] evaluations.)
3-5
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
Table 3-6. Efficiency Technology's Total Number of Positive, Negative, or Neutral
Evaluations
Efficiency Technology Categories
Coding Level
Positive
Negative
Neutral
Total
Active air dam
Active air dam (active
aerodynamics)
6
0
0
6
Active grill
shutters
Active grill shutters
(active aerodynamics)
1
0
0
1
Active ride height
Active ride height (active
aerodynamics)
2
0
1
3
Low rolling resistance tires
Low rolling resistance
tires
8
4
5
17
Electronic power steering
Electronic power steering
129
53
43
225
Powertrain Engine
Turbocharged
287
23
25
335
GDI
63
7
7
77
Cylinder deactivation
35
1
4
40
Diesel
141
15
13
169
Hybrid
104
28
13
145
Plug-in hybrid electric
42
7
6
55
Full electric
24
4
7
35
Stop-start
32
16
9
57
General engine
751
156
116
1,023
Transmission
High speed automatic
390
117
101
608
CVT
97
58
32
187
DCT
67
27
14
108
General transmission
139
49
31
219
General
Powertrain
General powertrain
93
14
20
126
Electric assist or low drag
brakes
Electric assist or low drag
brakes
4
1
3
8
Lighting-LED
Lighting-LED
20
1
2
23
Mass reduction
Mass reduction
82
0
12
94
Passive
aerodynamics
Passive aerodynamics
30
4
7
41
Total
2,546
585
471
3,602
3-6
-------
Section 3 — Results
Table 3-7. Operational Characteristics: Positive, Negative, or Neutral Evaluations
by Auto Reviews
Parent Hierarchy
Coding Level (Child codes)
Positive
Negative
Neutral
Total3
Drivability Handling
Steering feel/controllability/
responsiveness
446
148
164
758
Cornering
ability/grip/balance/body
control
478
95
117
690
General drivability
546
119
147
812
General handling
457
87
131
675
Acceleration
Acceleration feel/smoothness/
responsiveness
349
77
74
500
Acceleration capability/power/
torque
643
167
232
1,042
General acceleration
89
24
27
140
Braking
Brake feel/responsiveness
251
46
59
356
Stopping ability
250
33
82
365
General braking
83
22
18
123
Noise
Tire/ road
156
72
76
304
Wind
140
29
48
217
Interior
28
16
6
50
Powertrain
333
146
108
587
General noise
334
58
33
425
Vibration
Chassis
0
7
3
10
Powertrain
7
10
8
25
General vibration
13
19
13
45
Ride Comfort
Ride comfort
469
151
175
795
Fuel
Economy
Fuel economy
402
166
180
748
Range
Range
27
7
11
45
Charging
Charging
7
3
0
10
Total
5,508
1,502
1,712
8,722
a The Total column slightly overstates the number of reviews that mentioned the operational
characteristics, because one review could mention the same operational characteristic more than
once. For example, if the operational characteristic receives positive and negative evaluations in the
same auto review, we report that auto review once for Positive and once for Negative evaluations.
That renders "2" for that row in the total column. (But if the same operational characteristic receives
positive [or negative] evaluations more than once, we report that auto review only once as having
received positive [or negative] evaluations.)
3-7
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
Table 3-8. Operational Characteristics: Total Number of Positive, Negative, or
Neutral Evaluations
Parent Hierarchy Coding Level (Child Codes) Positive Negative Neutral Total
Drivability
Handling
Steering feel/controllability/
responsiveness
603
202
178
983
Cornering ability/
grip/balance/body control
732
131
148
1,011
General drivability
1,062
150
208
1,420
General handling
790
135
177
1,102
Acceleration Acceleration feel/smoothness/
responsiveness
459
105
80
644
Acceleration capability/power/
torque
1,331
282
277
1,890
General acceleration
112
34
28
174
Braking
Brake feel/responsiveness
277
50
63
390
Stopping ability
267
37
83
387
General braking
93
24
19
136
Noise
Tire/ road
168
87
79
334
Wind
146
34
48
228
Interior
29
16
7
52
Powertrain
454
210
120
784
General noise
482
87
36
605
Vibration
Chassis
7
3
10
Powertrain
7
11
8
26
General vibration
14
21
13
48
Ride Comfort
Ride comfort
766
268
242
1,276
Fuel Economy
Fuel economy
770
261
219
1,250
Range
Range
49
10
11
70
Charging
Charging
9
5
14
Total
8,620
2,167
2,047
12,834
3-8
-------
4. LIMITATIONS (SOURCES OF UNCERTAINTY)
There are two major sources of uncertainty:
1. The inherent variability between human coders, which was minimized by conducting
comprehensive training on coding, evaluating inter-coder reliability, and regular
quality checks by the adjudicator. This is a limitation faced by all content analysis
studies using human coders (Gottschalk 1995, Krippendorff 2013); and
2. Inconsistencies in how vehicle details are reported in the auto reviews (e.g., BMW 3-
series is reviewed as a whole or more than one model is covered in one review).
Such inconsistencies are beyond analyst control and are not atypical in content
analysis. To address this limitation, we discussed these occurrences with staff at EPA
who had greater insight into the details. In some instances, the review comments did
not meet the inclusion criteria for "substantive" reviews and therefore were excluded
from coding.
4-1
-------
-------
References
Berg, B. L., & Lune, H. (2012) Qualitative research methods for the social sciences (8th
ed.)- Boston: Pearson.
Carney, T. F. (1972). Content analysis: A technique for systematic inference from
communications. Winnipeg, Canada: University of Manitoba Press.
Cohen, Jacob (1960). "A coefficient of agreement for nominal scales". Educational and
Psychological Measurement 20 (1): 37-46. doi:10.1177/001316446002000104
Gottschalk, L. A. (1995). Content analysis of verbal behavior: New findings and clinical
applications. Hillside, NJ: Lawrence Erlbaum Associates, Inc.
Krippendorff, K. (2013). Content Analysis: An Introduction to Its Methodology, SAGE
Publications, Inc.
Moeller, L. G. (1963). An introduction to content analysis—including annotated bibliography.
Iowa City: University of Iowa Press.
Weber, R. P. (1990). Basic content analysis, second edition. Newbury Park, CA: Sage
Publications.
R-l
-------
-------
Appendix A:
Glossary and Coding Guidelines
A-l
-------
>
I
nj
Table A-l. Efficiency Technology
Parent Hierarchy
Child Code
Definition of Code
Example
Low rolling resistance
tires
Electronic power steering
Low rolling
resistance
tires(acronym
LRR)
Electronic
power steering
Powertrain
Engine
NA
Turbocharged
(Turbo, Eco-
boost, Ecotec)
GDI
Tires designed with less rolling resistance to
improve vehicle fuel efficiency.
Power steering that relies on computer
controlled electric motor to assist steering.
This is in contrast to less efficient hydraulic
systems that have been almost universally
used until recently, (synonyms: EPS. Note,
Steer-by-Wire is just one type of EPS system,
in which there is no mechanical connection to
the steering wheel) .
Defined as the engine and transmission
combination.
Device that increases engine combustion
efficiency (in terms of power per fuel
consumption).
Gasoline Direct Injection (GDI) is a fuel-
delivery technology that allows gasoline
engines to burn fuel more efficiently,
synonyms: direct injection, direct-injection
engine, petrol direct injection (PDI), direct
petrol injection (DPI), spark ignited direct
injection (SIDI), fuel stratified injection (FSI),
turbo fuel stratified injection (TFSI), smart
charge injection SCi), direct-injection-spark-
ignition (DISI), Ecotec (LAF, LCV, or LTG),
LT1) Note: Port fuel injection (PFI/SPFI/MPFI)
is not a type of GDI system)
"Low-rolling-resistance tires do sacrifice some
grip, but the hybrid's 184-foot 70-to-zero
stopping distance was only four feet longer than
that of the last no hybrid Accord we tested."3
(Neutral)
"The steer-by-wire system has been divisive so
far, with senior features editor Jonny Lieberman
calling bits of it somewhat "disconcerting..."13
(Negative)
NA
"...the Regal GS boasted its own tune for the 2.0-
liter turbo four that provided cushions of 50
horsepower and 35 Ib-ft of torque over the same
engine in the mid-grade Regal Turbo." (Positive)0
Example not yet available
(continued)
-------
Table A-l. Efficiency Technology (continued)
Parent Hierarchy
Child Code
Definition of Code
Example
Cylinder
deactivation
Diesel
Hybrid (HEV)
Plug-in hybrid
electric (PHEV)
Full electric
(EV)
Technology used on bigger engines to help
improve fuel economy that shuts down one or
more the engine's cylinders under low demand
situations, (synonyms: Variable displacement,
engine deactivation, engine displacement
change, variable cylinder management, active
fuel management, multi-displacement system,
active cylinder control system, active cylinder
control)
Diesel internal combustion engine (synonyms:
Compression-ignition engine)
A Hybrid Electric Vehicle (HEV) uses one or
more electric machines (motor/generator) to
help propel the vehicle and recapture braking
energy, but only gasoline or diesel is used to
fuel the vehicle (no plug-in to recharge
battery)
Same as HEV, except that rechargeable
batteries can be restored by plugging into an
electrical source (synonyms: Range Extended
Electric Vehicle (REEV), Extended Range
Electric Vehicle (EREV))
Vehicles that run on rechargeable electric
battery packs only (synonyms: Battery Electric
Vehicle (BEV))
Example not yet available
"Engine noise is subdued, and clues that it's a
diesel are remarkably few."d (Positive)
"Compared with its arch rival, the Camry hybrid,
Honda's planet-saver doesn't sacrifice the driving
experience on the altar of eco frugality."
(Positive)
"Crucial to the Fusion Energi's appeal, the Ford's
EPA-rated 21-mile EV range before the gas
engine turns on is far less than the 2013-2014
Volt (38 miles), but more than the Prius (11
miles) and Accord Hybrid (13 miles)." (Neutral)
"Although we found electrification dulls one of
our favorite Hondas, the reverse seems true of
the Chevy: It makes a boring car intriguing."
(Positive)
(continued)
>
I
W
-------
>
I
JS>
Table A-l. Efficiency Technology (continued)
Parent Hierarchy
Child Code
Definition of Code
Example
Stop-start Engine shuts down while idling to reduce fuel
consumption/emission. Not applicable to HEV's
and PHEV's. (synonym: i-Stop)
Transmission
General Engine Engine technologies not specifically defined by
another category
High speed Automatic transmission with a torque
automatic converter that is 6 speeds or higher (4 and 5-
speeds are not fuel saving technologies and
should not be coded. If just "transmission" is
referenced without speed specification code as
general transmission.) Not applicable to DCT's
and CVT's which also change ratios
automatically.
CVT Continuously Variable Transmissions change
through an infinite number of effective gear
ratios. This contrasts with other mechanical
transmissions that offer a fixed number of gear
ratios. A vehicle with CVT is like driving an
automatic without the feel of "gear changes."
(Note: Some recent manufacturers have
designed CVTs to mimic the feel of gear
changes.)
DCT Dual-clutch transmission shifts automatically
(Porsche may using clutches, allowing greater efficiency and
refer to it as acceleration performance than the typical
PDK). automatic transmission with a torque
converter.
"The EPA rates the TDI versions of the A6 and A7
at 24/38 mpg city/highway, and after back-road
stints through Virginia and Maryland, the in-car
trip computers reported seriously impressive fuel
sipping: 35 mpg in the A6 and 34 mpg in the A7.
Both cars are helped by a standard engine stop-
start system that can be alternately discreet and
abrupt."
"Elantra's 4-cylinder engine becomes thrashy
when accelerating hard." (Negative)
"The 6-speed automatic is ultra-smooth..."
(Positive)
"We like the way the CVT paddle shifters help the
2.5-liter make the most of its power." (Positive)
"Happily, you can't miss with either transmission.
The PDK is clever enough that you can leave it in
auto even on the track" (Positive)
"Helping the Focus is a quick-shifting 6-speed
dual-clutch PowerShift transmission with launch
control" (Positive)
(continued)
-------
Table A-l. Efficiency Technology (continued)
Parent Hierarchy
Child Code
Definition of Code
Example
General Powertrain
Electric assist or low drag
brakes
Lighting-LED
Mass reduction
Active ride height (Active
aerodynamics)
Active grill shutters
(Active aerodynamics)
Active air dam (Active
aerodynamics)
Passive aerodynamics
General
Transmission
General
Powertrain
Electric assist
or low drag
brakes
Lighting-LED
Mass reduction
Active ride
height
Active grill
shutters
Active air dam
Passive
aerodynamics
Transmission technologies not specifically
defined by another category or when specific
transmission type is not specified.
Powertrain technologies (i.e. a combination of
engine and transmission) not specifically
defined by another category.
Electric brake boosters that increase breaking
pressure in emergency/quick braking
situations, which allows a greater distance to
be maintained between the brake pads and
rotors during normal driving for reduced drag),
(synonyms: Electric servo brake system)
Evaluations of LED headlights
Reducing weight of vehicle through the use of
alternative and lighter weight materials.
The vehicle automatically adjusts
suspension/height of vehicle to reduce drag,
increasing fuel efficiency (e.g. if vehicle lowers
to the ground when at higher speeds there is
less drag on the vehicle).
Grille shutters which open and close
automatically, controlling airflow to engine bay
to reduce drag on vehicle
Air dams which open and close automatically
to optimize aerodynamics.
Static characteristics of the vehicle's body
design and/or added components that work to
reduce vehicle drag. (e.g. specific body
shapes, spoilers, underbody covers etc.)
"...the transmission's seeming inability to find
and hold a gear." (Negative)
"...the engine and transmission live in relative
harmony." (Positive)
"The same goes for the Accord hybrid's new
electrically assisted brakes: While totally
reengineered to use mostly regenerative braking,
they give crisp top-of-pedal response that's easy
to modulate." (Positive)
"Standard low beam LEDs on all Corollas not only
allow versatile styling but provide some of the
best visibility we've ever tested. These LEDs also
have excellent levels of light intensity." (Positive)
Example not yet available
Example not yet available
Example not yet available
Fully extended, the clever air dam sits lower than
the GT3's nose, but, at low speeds, it's flush
against the bottom of the bumper, so drivers
don't have to worry about scraping against
driveways
The "air curtain" directs air through the corners
of the lower front fascia, around the front
wheels, and out the front fender scallops, thus
making them functional.
>
i
Ul
a 2014_Honda Accord Hybrid Car and Driver, 012
b 2014_Infiniti_Q50S_Compact_lab_Motortrend, 001
c 2014_Buick_Regal-Turbo-AWD-GS-AWD_Midsize_l_CarandDriver
-------
>
I
01
Table A-2. Operational Characteristics
Parent Hierarchy
Child Code
Definition
Example
Drivability
General Drivability
Handling
Refers to the vehicle's
lateral movements (i.e.
turning movements
controlled through
steering) and the degree
to which they follow a
desired path and speed
on the road, and are
controlled easily and
predictably by turning
the wheel . Better
handling means a car
can turn at higher
speeds with reduced risk
of losing control. A car
that doesn't handle well
will be quicker to lose
grip, or lose control. Do
not code evaluations of
parking maneuverability
and turning radius.)
NA
General
Drivability
General
handling
Steering
feel/Controllabi
lity/Responsive
ness (Driver
experience)
The degree to which a vehicle's lateral and
longitudinal movements follow a desired path
and speed on the road, and are controlled
easily and predictably by the driver.
Drivability characteristics that do not fall under
other child codes (handling, acceleration,
braking) (Keyword examples: drive,
drivability, driving)
Handling characteristics that do not fall under
other child codes (steering feel..., cornering
ability...) Refers to the degree to which a
vehicle's lateral movements (i.e. turning
movements controlled through steering) follow
a desired path and speed on the road, and are
controlled easily and predictably
Driver perception of steering via steering
wheel feel/Driver perception of how well the
vehicle responds to steering inputs by driver/
Drivers perception of how well vehicle can be
controlled via steering inputs, (e.g. When the
driver turns the wheel does it react the way
they want it to?) Keyword examples: Sharp,
crisp, (pos) vague, disconnected (neg).
NA
"There's nothing particularly fun about driving
the Elantra." (Negative)
"On the track, the Corolla was secure and
forgiving." (Positive)a
"By no means does the DAS interfere with
driving, but you end up doing so by other tactile
feel... and muscle memory. In other words, your
hands really don't feel connected to the road,
despite the artificial feedback."
(continued)
-------
Table A-2. Operational Characteristics
Parent Hierarchy
Child Code
Definition
Example
Acceleration
Refers to the rate of
increase in vehicle speed
(i.e. forward movement
controlled through the
accelerator pedal) and
the degree to which
speed increases
smoothly and predictably
in response to the
driver's application of the
pedal.
Cornering
ability/Grip/Bal
ance/Body
Control
(Performance
expectations)
General
acceleration
Acceleration
feel/Smoothne
ss/Responsiven
ess
(Driver
experience)
Acceleration
capability/Powe
r/Torque
(Performance
expectations)
Refers the cars ability to safely and quickly
handle corners, (i.e. How in control is the
driver of the car going around corners at
accelerated speeds? Does the car have the
ability to go through around the corners
quickly? How well does the car/tires grip the
road when going around corners? Do tires slide
out (balance), or does the car understeer
(steers less than the amount commanded by
driver) or over-steer (steers more than the
amount commanded by driver) around the
corner? ) Note: this code refers to the actual
performance of the vehicle, not how it "feels"
to the driver while maneuvering.
Acceleration characteristics that do not fall
under other child codes (acceleration feel...,
acceleration capability)
How the acceleration of the car feels to
driver/Acceleration smoothness (or lack of
smooth acceleration)/Responsiveness of pedal
(i.e. Does the vehicle respond as expected
when the pedal position is changed?.)
How fast can the car accelerate? Note: power
can only be perceived as an operational
characteristic through acceleration
performance. Be careful not to code
evaluations of power as a specification, i.e.,
when it is not experienced through
acceleration while driving. Example:"260
horsepower is low for its class" (do not code -
specification only)
"It steers nicely through turns and is totally
controllable, although there's limited grip and
power available." (Negative)
"Performance for our LE with the 132-hp engine
and the CVT was adequate. No complaints about
acceleration but no one raved, either."13 (Neutral)
"The Elantra's 4-cylinder engine becomes
thrashy when accelerating."0 (Negative)
"Quick and precise turn-in with excellent feel
through the thick and grippy steering wheel"
(positive)
"Power is weak at low revs, and its 0-60 mph
time of 9.7 sec is slow for the class."d (Negative)
>
i
(continued)
-------
>
I
00
Table A-2. Operational Characteristics (continued)
Parent Hierarchy
Child Code
Definition
Example
Braking
Refers to the slowing and
stopping of the vehicle
(i.e. forward movement
controlled through the
brake pedal) and the
degree to which the
vehicle can be stopped
quickly and responds
predictably to driver
application of the brake
pedal.
Noise (synonyms:
Sound, turbulence,
boisterousness, boom,
clang, clatter, discord,
disquiet, drumming,
racket, ring, thud, echo;
quiet or quietness
(antonym)
General
braking
Brake
feel/Responsiv
eness
(Driver
experience)
Interior
Powertrain
Braking characteristics that do not fall under
other child codes (brake feel..., stopping
ability)
Driver perception of braking via feel of
pedal/Driver perception of how well the car
responds to braking/pressing pedal.
For hybrid & EV in specific, refers to
regenerative braking and transition between
friction and regenerative.
Stopping ability
(Performance
expectation)
Tire/Road
Wind
Braking performance (i.e. How well the vehicle
can stop/responds to driver braking.) This
includes how well the tires grip while braking,
and stopping distance.
Noise generated from tires while car is in
motion, based on perceptions of the driver
inside the cabin. Also referenced as "road
noise."
Noise generated as the vehicle passes through
the air, based on perceptions of the driver
inside the cabin).
Noise generated from vehicle's interior (e.g.
squeaky instrument panel) [note different
from General noise]
Noise coming from powertrain components
(engine/trans), based on perceptions of the
driver inside the cabin.
"Overall braking performance was good with
relatively short stops on both surfaces. It's not a
top performer in its class, but at least on par."
(Neutral)
"The Elantra exhibits some initial brake
jumpiness when hitting the pedal at higher
speeds..."e (Neutral)
"The same goes for the Accord hybrid's new
electrically assisted brakes: While totally
reengineered to use mostly regenerative
braking, they give crisp top-of-pedal response
that's easy to modulate. The brakes are neither
grabby nor laggy like those in many other
brands' hybrids and electrics" (Positive)
"...its panic-stop distance of 126 feet from 60
mph is longer than average." (Negative)
"Staggered sized UHP run-flat tires do their
share in keeping the cabin noisy with plenty of
rumble and impact boom reverberating from the
rear." (Negative)
"The Corolla has suppressed levels of wind and
road noise." (Positive)
"We did notice one intermittent dash rattle."
"We were also mildly annoyed by a whirring
electric-motor sound at low speeds."a (Negative)
(continued)
-------
Table A-2. Operational Characteristics (continued)
Parent Hierarchy
Child Code
Definition
Example
General noise
References generically to "cabin noise" or
"interior noise", without reference to the
source of the noise, based on perceptions of
the driver inside the cabin.
"Elantra is impressively quiet at highway
speeds."11 (Positive)
Vibration
Chassis
Vibration originating from base/frame of the
vehicle
Example not yet available
Powertrain
General
vibration
Vibration originating from powertrain
components (transmission and engine)
"At idle, vibration transmitted to the cabin is
significant — so significant that senior production
director Zach Gale said, "Mitsubishi could almost
advertise this car as having a low-intensity
massaging seat setting." (Negative)
General
vibration
Vibration not specified by another category
Ride comfort
Ride comfort
How comfortable the ride of the car is for
passengers when there are bumps in the road
surface (Note: this does not include reviews of
seat cushion comfort or interior decoration)
"...surprisingly harsh over bumps and potholes,
feeling unrefined compared to several key
rivals."' (Negative)
Fuel economy
Fuel economy
The amount of fuel a vehicle uses. Fuel
economy can be evaluated through observed
fuel efficiency reported from test drive, based
on reviewer's thoughts of published
descriptions (specifications) of fuel economy,
or in comparison with other vehicles.
"Even with our typically heavy collective right
foot, we saw observed mileage in the low 40s—
compelling for a mid-size car."J (Positive)
Range
Range
The distance you can travel on a full charge in
an electric vehicle before the battery requires
a recharge. Range can be evaluated through
observed range reported from test drive or
from the reviewers thoughts on a published
specification of range.
(continued)
>
I
<0
-------
Table A-2. Operational Characteristics (continued)
o
Parent Hierarchy
Child Code
Definition
Example
Charging
Charging
Evaluations of observed charging times for
electric/hybrid electric vehicles.
a2014 Mercedes-Benz_E250-BlueTec-Diesel_Midsize_l_CarandDriver
b2014_Toyota_Corolla-LE-Plus-sedan_Midsize_l_Consumer-Reports
c2014_Toyota_Corolla-LE-Plus-sedan_Midsize_l_Consumer-Reports
d2014_Hyundai_Elantra-Limited-Sedan_Midsize_l_Edmund
e2014_Hyundai_Elantra-Limited-Sedan_Midsize_l_Edmund
f 2014_Hyundai_Elantra-Limited-Sedan_Midsize_l_Edmund
9 2014_Hyundai_Elantra-Limited-Sedan_Midsize_l_Edmund
h2014_Honda Accord Hybrid Car and Driver
' 2014_Hyundai_Elantra-Limited-Sedan_Midsize_l_Edmund
j 2014_Hyundai_Elantra-Limited-Sedan_Midsize_l_Edmund
-------
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
Additional Coding Guidelines
Relative Evaluations
An evaluation relative to another vehicle may be provided in which the positive or negative
rating of that other vehicle is not provided. In such cases, the following rules should be
followed for coding:
¦ A positive evaluation relative to another vehicle should be coded as positive, (e.g.
"quieter than...")
¦ A mitigated negative evaluation relative to another vehicle should be coded as
neutral.(e.g. "not as noisy as...")
¦ A mitigated positive evaluation relative to another vehicle should be coded as
neutral.(e.g. "not as quiet as...")
A negative evaluation relative to another vehicle should be coded as negative
(e.g. "noisier than ...")
¦ An equivalent evaluation relative to another vehicle where the evaluation of the
other vehicle is not described in the article, or relative to another trim level of
the same vehicle coded elsewhere in the article should not be coded (e.g. "same
noise level as...")
- Note: if the evaluation of the other vehicle is described in the article or can be
inferred from the surrounding text, then code according to that evaluation
(e.g. "Overall, the Corolla doesn't ride much worse than the larger Camry."
Would be coded as positive, with ride comfort being essentially the same as
the "larger" vehicle and its implied ride quality.)
Unobtrusive Technologies
Many efficiency technologies are intended to save fuel while not affecting other operational
characteristics of the vehicle. Statements which evaluate a technology as unobtrusive or not
noticeable should be coded as positive for that technology. Examples: "you'd never know
the Corolla has a CVT". "Testing director Kim Reynolds said if Jonny hadn't told him, he
would have guessed it was a normal steering setup"
Synonymous Evaluations of Technologies and Operational Characteristics
Some technologies are perceived through only one relevant operational characteristic. When
there is never a distinction between technology and operational characteristics, the
evaluation shall be coded in only one category, as defined below:
Evaluations of engine, transmission, or powertrain noise shall be coded in the
operational characteristic category of "Noise: Powertrain", and not under the
technology categories of "Engine" or "Transmission"
"We were also mildly annoyed by a whirring electric-motor sound at low speeds"
(Honda Accord)
-------
Appendix A
— Glossary and Coding Guidelines
"And my ears receive a sort of "achievement unlocked" reward, as the engine
sounds smooth, tight and athletic - if not race car-raucous - at higher revs."
(Infiniti)
Evaluations of transmission shift quality, smoothness, or frequency shall be
coded under the appropriate technology category for that transmission type.
Noise may be the only overlapping characteristic.
Specifications
Note text that is coded under the Efficiency Technology node should be coded only when a
review generated from observing a specific technology during test drive occurs, not simply
when the technology is listed. That is, do not code specifications. Example:
CODE: "We like the way the CVT paddle shifters help the 2.5-liter make the most of its
power."
- Coded as CVT: positive
DO NOT CODE: "The 2.5i Premium ($24,090) adds a CVT, 17-in alloy wheels, the Cold
Weather Package (heated seats, side mirrors and windshield defrosters), upgraded audio
with 4.3-in LCD display, fog lights and a 10-way power driver's seat with power lumbar
support."
- While this mentions the CVT, it is only highlighting it as a specification. Do not
code specifications
Tip: Ask yourself, is the reviewer evaluating a technology? An operational characteristic? If
they aren't evaluating/reviewing an aspect of the car based their experience of driving the
vehicle, then it is not a codeable passage.
General Codes
Passages should only be coded under "general" (e.g. acceleration, handling, braking)
categories after all other "specific" codes (acceleration capability...steering feel... etc.) for
that characteristic have been eliminated. That is, coders should first exhaust all other coding
possibilities under more "specific" codes for that category.
Coding Overall Evaluations
In addition to categorizing each individual code as positive, negative or neutral, the overall
sentiment of the review must be coded as well. Each review should be assigned a positive,
negative or mixed code for the whole review. Review summaries and conclusions can often
provide a general idea of the reviewer's attitude about the reviewed vehicle (e.g. reviewer
states if they feel consumers should/shouldn't purchase particular car etc.) Example:
"We like diesels. We like the big torque and the excellent real-world fuel economy. But the
notion that anyone is buying a $60,000 sedan with a heavy environmental conscience or a
pragmatic economic analysis is a stretch. Performance sells cars in this price range. As
civilized and torquey as it is, the TDI engine doesn't have the immediacy of the gas-fueled,
A-12
-------
Content Analysis of Professional Automotive Reviews: Work Assignment 3-01 (RTI3-01)
supercharged V-6, an engine that sets an industry standard for its instant throttle response,
supreme linearity, and excellent power delivery. Paying more money for fewer thrills doesn't
make much sense here." (Negative)
"Redesigned for 2014, the new Corolla has landed right on target, and it now ranks among
the top models in its class. It combines the practicality and frugal fuel economy that
compact-sedan buyers want with more interior room, upgraded amenities, and a sorely
needed shot of style. Overall, it's a reasonable alternative to a larger, midsized sedan."
(Positive)
Use of Synonyms
Reviews may not necessarily spell out the technology. Sometimes you will need to infer
based on the synonyms provided and the context of the passage. For example, "electrically
boosted steering" is the same as "electronic power steering."
------- |