2010 Consumer Reports Survey Analysis: Part Two: EcoBoost Oddity

Michael Karesh
by Michael Karesh

In Part 1, we found that, despite its large overall sample size, Consumer Reports’ has serious gaps in its coverage. But what about the reliability ratings they can provide? An FAQ asserts CR’s ability to split results by engines, drive types, and so forth. At first glance, this appears valuable, as CR’s reliability scores often differ from powertrain to powertrain. But are these differences valid? Should you avoid the V6 in the Camry or insist that your Flex be EcoBoosted?

In his review of CR’s latest results last week, Jack Baruth noted that the 2010 V6 Camry is rated worse than average. Dig a little deeper, and this rating appears based on problems with squeaks and rattles, power accessories, and the audio system…all involving parts shared with the other Camry variants. The implied problems with the V6 powertrain? They don’t exist. All of the powertrain-related systems receive top marks.

The Ford Flex EcoBoost that leads its class is predicted to have reliability 60 percent better than average. Not mentioned in the press release: the Ford Flex AWD sans-boost is predicted to have reliability 16 percent worse than average—nearly bad enough for the half-black blob and a non-recommend. A 76-point difference is huge. A solid red blob and a solid black blob, best and worst ratings, are 90 points apart. The source of this massive difference? It’s not a pair of turbos.

CR’s predictions are based on however many of the three most recent model years they have sufficient data for. The EcoBoost was new for 2010, so the prediction for 2011 is based entirely on the 2010. In contrast, the prediction for the sans-boost also incorporates data on the 2009. Should first-year glitches unrelated to the powertrain have any more bearing on the 2011 sans-boost than on the 2011 EcoBoost? In CR’s formula, they do.

Second, even looking at only the 2010s, the sans-boost fares much less well than the EcoBoost. This would justify a more pessimistic prediction, except that, just as with the Camry (and nearly every other case I checked), the differences between the powertrain-based variants have little or nothing to do with the powertrains. Owners of the 2010 sans-boost Flex reported far more problems with squeaks and rattles, body hardware, power equipment, and the audio system—all involving parts shared with the EcoBoost.

This is far from an isolated anomaly. For every case like BMWs with the turbocharged six (with its notoriously unreliable fuel pump), there are a number of others like the Toyota Camry, Ford Flex, Hyundai Genesis (a lower V8 score can be traced to the Technology Package offered with both engines, but more often ordered with the V8), and Mercedes-Benz C300 (where non-powertrain problems common enough to earn the 2010 a solid black blob go away when AWD is added). Key takeaway: the differences in CR’s ratings for different powertrains often are not due to powertrain-related parts. When there are such differences, it’s critical to check the system-level blobs.

In an FAQ, CR provides an explanation for a similar (though only half as large) discrepancy between the very closely related Chevrolet Equinox and GMC Terrain [brackets mine]:

The Terrain had slightly [about 40 percent] more [reported] electrical, audio, and paint and exterior trims [sic] problems… We believe, though, in the accuracy of our data, and we have a commitment to report the experiences our subscribers share with us. In some cases, they report different reliability experiences with closely related models.

In fewer words: our data are accurate because we believe in the accuracy of our data.

Unless Ford performs far more thorough quality control on the boosted Flex, an unexplained 76-point difference should not happen. A miniscule sample size might explain it, but CR’s sample size isn’t small. The problem, then, is their methods. Ask the wrong question, and it doesn’t matter how many people answer it.

The problem with CR’s key question: it asks car owners to report problems they considered serious. Letting each respondent decide whether or not a problem is serious enough to report opens the door wide for bias. Not CR’s bias, at least not directly. But any bias the car owner might have, and have honestly. Love the car? Treated well by the dealer? Warranty paid for the repair? Then even a failed transmission might not seem “serious.” Especially not if it happened almost a year ago—the impact of this subjective wording is magnified by the annual (in)frequency of the survey. At the other extreme, many CR subscribers report minor problems like rattles and squeaks. If EcoBoost owners love their Flex considerably more than sans-boost owners do, a large difference in reliability—as reported by CR—might result.

In response to a blog comment critical of CR’s methods, a staff member recently argued:

The reliability survey asks if the owner had a problem requiring repair. The way it is constructed, it is objective… To use your example, if Fox News questioned viewers about political views, it would yield a certain, slanted response. However, if Fox News asked viewers if their TVs needed repairs in the past year, either they did or didn’t, regardless of political persuasion.

This would be a valid defense—but only if the survey is truly constructed to maximize objectivity. CR’s is not. The way their key survey question has actually been worded—for years—introduces so much subjective variation that even large sample sizes cannot compensate. A massive 76-point difference can be elicited where none objectively exists—and then be ignored when reporting results. Most models differ from one another by much less.

Michael Karesh owns and operates TrueDelta, an online source of automotive pricing and reliability data.

Michael Karesh
Michael Karesh

Michael Karesh lives in West Bloomfield, Michigan, with his wife and three children. In 2003 he received a Ph.D. from the University of Chicago. While in Chicago he worked at the National Opinion Research Center, a leader in the field of survey research. For his doctoral thesis, he spent a year-and-a-half inside an automaker studying how and how well it understood consumers when developing new products. While pursuing the degree he taught consumer behavior and product development at Oakland University. Since 1999, he has contributed auto reviews to Epinions, where he is currently one of two people in charge of the autos section. Since earning the degree he has continued to care for his children (school, gymnastics, tae-kwan-do...) and write reviews for Epinions and, more recently, The Truth About Cars while developing TrueDelta, a vehicle reliability and price comparison site.

More by Michael Karesh

Comments
Join the conversation
2 of 35 comments
  • SSLByron SSLByron on Nov 05, 2010

    Seems to me like the real lesson here is that you shouldn't base your buying decision on a single resource, especially if you don't take the time to evaluate the information it offers. It's not too difficult to dig deeper into CR's rankings and see the root of various discrepancies. Some are probably valid. Some are probably not. And for all CR's supposed love from the media, they're just as often a whipping boy for enthusiasts.

  • Ciddyguy Ciddyguy on Nov 05, 2010

    As far as reliability goes, I like a reliable car as the next person but I think many may confuse reliability with build quality (although neither are mutually exclusive of each other as each reflects directly or indirectly of the other and of the entire vehicle as a whole). That means, a car with so, so build quality can have a very robust drive train that is reliable, easy to fix, lasts a long time and performs well to boot, but the overall body integrity and fit and finish may not be overly stellar or the reverse can be true, a car that uses expensive looking materials and such, is well put together but can't remain reliable for any length of time and thus out of the mechanic's garage to be enjoyed, but often one reflects upon the other to one degree or the other. But as far as reliability factors are concerned, I am more concerned with how the engine, transmission, steering and suspension components hold up, how effective is the cooling system and does it prevent overheating in hot weather adequately? I also look at the secondary components, such as the electrics, you know, wiring harnesses, electronic components, switch gear, door and window controls and motors and how reliable they are because any and/or all of them can cause a car or truck/van to go into the mechanic to be repaired/replaced and how often does this tend to take place? I know there will be the occasional clunker part, but the issue is, how often does this occur over a model year (or does it keep occurring) or was this a rash incident where a 1 off bad batch of parts made it into production and didn't show until much later (can happen) but at the end of the day, does the car start reliably, run reasonably smooth, performs well, adequately powered, enjoyable and reliable enough to stay out of the dealer's garage and on the road to be enjoyed - especially true in its first 5 years or so, or until it hits 100K miles or above. That to me is a sign of reliability and it is true for both mechanical but electrical as well. Once you get over 5-7 years old and/or 100K miles, it IS expected for parts to begin to wear out, get slow (electric window motors especially), interiors to get a little shabby and so on but in the car's first 5 years at least, is critical that it show itself as being reliable as it will often indicate, if taken care of of how reliable it'll remain as it ages beyond 100K miles. And I should say that I tend to agree with the others in that one should not rely on one source, but a variety of sources as an overall picture will form of the product, first off, do people enjoy it (yes, no) is it reliable (again, yes, no, and to what degree) as cars that proven to be reliable do not necessarily reflect owner's enjoyment as factors such as ride quality, noise, vibration, interior usefulness and all that can negatively affect an owner's overall satisfaction of said car.

  • ToolGuy This is the kind of thing you get when you give people faster internet.
  • ToolGuy North America is already the greatest country on the planet, and I have learned to be careful about what I wish for in terms of making changes. I mean, if Greenland wants to buy JDM vehicles, isn't that for the Danes to decide?
  • ToolGuy Once again my home did not catch on fire and my fire extinguisher(s) stayed in the closet, unused. I guess I threw my money away on fire extinguishers.(And by fire extinguishers I mean nuclear missiles.)
  • Carson D The UAW has succeeded in organizing a US VW plant before. There's a reason they don't teach history in the schools any longer. People wouldn't make the same mistakes.
  • B-BodyBuick84 Mitsubishi Pajero Sport of course, a 7 seater, 2.4 turbo-diesel I4 BOF SUV with Super-Select 4WD, centre and rear locking diffs standard of course.
Next