In Part 1, we found that, despite its large overall sample size, Consumer Reports’ has serious gaps in its coverage. But what about the reliability ratings they can provide? An FAQ asserts CR’s ability to split results by engines, drive types, and so forth. At first glance, this appears valuable, as CR’s reliability scores often differ from powertrain to powertrain. But are these differences valid? Should you avoid the V6 in the Camry or insist that your Flex be EcoBoosted?
In his review of CR’s latest results last week, Jack Baruth noted that the 2010 V6 Camry is rated worse than average. Dig a little deeper, and this rating appears based on problems with squeaks and rattles, power accessories, and the audio system…all involving parts shared with the other Camry variants. The implied problems with the V6 powertrain? They don’t exist. All of the powertrain-related systems receive top marks.
The Ford Flex EcoBoost that leads its class is predicted to have reliability 60 percent better than average. Not mentioned in the press release: the Ford Flex AWD sans-boost is predicted to have reliability 16 percent worse than average—nearly bad enough for the half-black blob and a non-recommend. A 76-point difference is huge. A solid red blob and a solid black blob, best and worst ratings, are 90 points apart. The source of this massive difference? It’s not a pair of turbos.
CR’s predictions are based on however many of the three most recent model years they have sufficient data for. The EcoBoost was new for 2010, so the prediction for 2011 is based entirely on the 2010. In contrast, the prediction for the sans-boost also incorporates data on the 2009. Should first-year glitches unrelated to the powertrain have any more bearing on the 2011 sans-boost than on the 2011 EcoBoost? In CR’s formula, they do.
Second, even looking at only the 2010s, the sans-boost fares much less well than the EcoBoost. This would justify a more pessimistic prediction, except that, just as with the Camry (and nearly every other case I checked), the differences between the powertrain-based variants have little or nothing to do with the powertrains. Owners of the 2010 sans-boost Flex reported far more problems with squeaks and rattles, body hardware, power equipment, and the audio system—all involving parts shared with the EcoBoost.
This is far from an isolated anomaly. For every case like BMWs with the turbocharged six (with its notoriously unreliable fuel pump), there are a number of others like the Toyota Camry, Ford Flex, Hyundai Genesis (a lower V8 score can be traced to the Technology Package offered with both engines, but more often ordered with the V8), and Mercedes-Benz C300 (where non-powertrain problems common enough to earn the 2010 a solid black blob go away when AWD is added). Key takeaway: the differences in CR’s ratings for different powertrains often are not due to powertrain-related parts. When there are such differences, it’s critical to check the system-level blobs.
In an FAQ, CR provides an explanation for a similar (though only half as large) discrepancy between the very closely related Chevrolet Equinox and GMC Terrain [brackets mine]:
The Terrain had slightly [about 40 percent] more [reported] electrical, audio, and paint and exterior trims [sic] problems… We believe, though, in the accuracy of our data, and we have a commitment to report the experiences our subscribers share with us. In some cases, they report different reliability experiences with closely related models.
In fewer words: our data are accurate because we believe in the accuracy of our data.
Unless Ford performs far more thorough quality control on the boosted Flex, an unexplained 76-point difference should not happen. A miniscule sample size might explain it, but CR’s sample size isn’t small. The problem, then, is their methods. Ask the wrong question, and it doesn’t matter how many people answer it.
The problem with CR’s key question: it asks car owners to report problems they considered serious. Letting each respondent decide whether or not a problem is serious enough to report opens the door wide for bias. Not CR’s bias, at least not directly. But any bias the car owner might have, and have honestly. Love the car? Treated well by the dealer? Warranty paid for the repair? Then even a failed transmission might not seem “serious.” Especially not if it happened almost a year ago—the impact of this subjective wording is magnified by the annual (in)frequency of the survey. At the other extreme, many CR subscribers report minor problems like rattles and squeaks. If EcoBoost owners love their Flex considerably more than sans-boost owners do, a large difference in reliability—as reported by CR—might result.
In response to a blog comment critical of CR’s methods, a staff member recently argued:
The reliability survey asks if the owner had a problem requiring repair. The way it is constructed, it is objective… To use your example, if Fox News questioned viewers about political views, it would yield a certain, slanted response. However, if Fox News asked viewers if their TVs needed repairs in the past year, either they did or didn’t, regardless of political persuasion.
This would be a valid defense—but only if the survey is truly constructed to maximize objectivity. CR’s is not. The way their key survey question has actually been worded—for years—introduces so much subjective variation that even large sample sizes cannot compensate. A massive 76-point difference can be elicited where none objectively exists—and then be ignored when reporting results. Most models differ from one another by much less.
Michael Karesh owns and operates TrueDelta, an online source of automotive pricing and reliability data.