2010 Consumer Reports Survey Analysis: Part Two: EcoBoost Oddity

Michael Karesh
by Michael Karesh
2010 consumer reports survey analysis part two ecoboost oddity

In Part 1, we found that, despite its large overall sample size, Consumer Reports’ has serious gaps in its coverage. But what about the reliability ratings they can provide? An FAQ asserts CR’s ability to split results by engines, drive types, and so forth. At first glance, this appears valuable, as CR’s reliability scores often differ from powertrain to powertrain. But are these differences valid? Should you avoid the V6 in the Camry or insist that your Flex be EcoBoosted?

In his review of CR’s latest results last week, Jack Baruth noted that the 2010 V6 Camry is rated worse than average. Dig a little deeper, and this rating appears based on problems with squeaks and rattles, power accessories, and the audio system…all involving parts shared with the other Camry variants. The implied problems with the V6 powertrain? They don’t exist. All of the powertrain-related systems receive top marks.

The Ford Flex EcoBoost that leads its class is predicted to have reliability 60 percent better than average. Not mentioned in the press release: the Ford Flex AWD sans-boost is predicted to have reliability 16 percent worse than average—nearly bad enough for the half-black blob and a non-recommend. A 76-point difference is huge. A solid red blob and a solid black blob, best and worst ratings, are 90 points apart. The source of this massive difference? It’s not a pair of turbos.

CR’s predictions are based on however many of the three most recent model years they have sufficient data for. The EcoBoost was new for 2010, so the prediction for 2011 is based entirely on the 2010. In contrast, the prediction for the sans-boost also incorporates data on the 2009. Should first-year glitches unrelated to the powertrain have any more bearing on the 2011 sans-boost than on the 2011 EcoBoost? In CR’s formula, they do.

Second, even looking at only the 2010s, the sans-boost fares much less well than the EcoBoost. This would justify a more pessimistic prediction, except that, just as with the Camry (and nearly every other case I checked), the differences between the powertrain-based variants have little or nothing to do with the powertrains. Owners of the 2010 sans-boost Flex reported far more problems with squeaks and rattles, body hardware, power equipment, and the audio system—all involving parts shared with the EcoBoost.

This is far from an isolated anomaly. For every case like BMWs with the turbocharged six (with its notoriously unreliable fuel pump), there are a number of others like the Toyota Camry, Ford Flex, Hyundai Genesis (a lower V8 score can be traced to the Technology Package offered with both engines, but more often ordered with the V8), and Mercedes-Benz C300 (where non-powertrain problems common enough to earn the 2010 a solid black blob go away when AWD is added). Key takeaway: the differences in CR’s ratings for different powertrains often are not due to powertrain-related parts. When there are such differences, it’s critical to check the system-level blobs.

In an FAQ, CR provides an explanation for a similar (though only half as large) discrepancy between the very closely related Chevrolet Equinox and GMC Terrain [brackets mine]:

The Terrain had slightly [about 40 percent] more [reported] electrical, audio, and paint and exterior trims [sic] problems… We believe, though, in the accuracy of our data, and we have a commitment to report the experiences our subscribers share with us. In some cases, they report different reliability experiences with closely related models.

In fewer words: our data are accurate because we believe in the accuracy of our data.

Unless Ford performs far more thorough quality control on the boosted Flex, an unexplained 76-point difference should not happen. A miniscule sample size might explain it, but CR’s sample size isn’t small. The problem, then, is their methods. Ask the wrong question, and it doesn’t matter how many people answer it.

The problem with CR’s key question: it asks car owners to report problems they considered serious. Letting each respondent decide whether or not a problem is serious enough to report opens the door wide for bias. Not CR’s bias, at least not directly. But any bias the car owner might have, and have honestly. Love the car? Treated well by the dealer? Warranty paid for the repair? Then even a failed transmission might not seem “serious.” Especially not if it happened almost a year ago—the impact of this subjective wording is magnified by the annual (in)frequency of the survey. At the other extreme, many CR subscribers report minor problems like rattles and squeaks. If EcoBoost owners love their Flex considerably more than sans-boost owners do, a large difference in reliability—as reported by CR—might result.

In response to a blog comment critical of CR’s methods, a staff member recently argued:

The reliability survey asks if the owner had a problem requiring repair. The way it is constructed, it is objective… To use your example, if Fox News questioned viewers about political views, it would yield a certain, slanted response. However, if Fox News asked viewers if their TVs needed repairs in the past year, either they did or didn’t, regardless of political persuasion.

This would be a valid defense—but only if the survey is truly constructed to maximize objectivity. CR’s is not. The way their key survey question has actually been worded—for years—introduces so much subjective variation that even large sample sizes cannot compensate. A massive 76-point difference can be elicited where none objectively exists—and then be ignored when reporting results. Most models differ from one another by much less.

Michael Karesh owns and operates TrueDelta, an online source of automotive pricing and reliability data.

Comments
Join the conversation
2 of 35 comments
  • SSLByron SSLByron on Nov 05, 2010

    Seems to me like the real lesson here is that you shouldn't base your buying decision on a single resource, especially if you don't take the time to evaluate the information it offers. It's not too difficult to dig deeper into CR's rankings and see the root of various discrepancies. Some are probably valid. Some are probably not. And for all CR's supposed love from the media, they're just as often a whipping boy for enthusiasts.

  • Ciddyguy Ciddyguy on Nov 05, 2010

    As far as reliability goes, I like a reliable car as the next person but I think many may confuse reliability with build quality (although neither are mutually exclusive of each other as each reflects directly or indirectly of the other and of the entire vehicle as a whole). That means, a car with so, so build quality can have a very robust drive train that is reliable, easy to fix, lasts a long time and performs well to boot, but the overall body integrity and fit and finish may not be overly stellar or the reverse can be true, a car that uses expensive looking materials and such, is well put together but can't remain reliable for any length of time and thus out of the mechanic's garage to be enjoyed, but often one reflects upon the other to one degree or the other. But as far as reliability factors are concerned, I am more concerned with how the engine, transmission, steering and suspension components hold up, how effective is the cooling system and does it prevent overheating in hot weather adequately? I also look at the secondary components, such as the electrics, you know, wiring harnesses, electronic components, switch gear, door and window controls and motors and how reliable they are because any and/or all of them can cause a car or truck/van to go into the mechanic to be repaired/replaced and how often does this tend to take place? I know there will be the occasional clunker part, but the issue is, how often does this occur over a model year (or does it keep occurring) or was this a rash incident where a 1 off bad batch of parts made it into production and didn't show until much later (can happen) but at the end of the day, does the car start reliably, run reasonably smooth, performs well, adequately powered, enjoyable and reliable enough to stay out of the dealer's garage and on the road to be enjoyed - especially true in its first 5 years or so, or until it hits 100K miles or above. That to me is a sign of reliability and it is true for both mechanical but electrical as well. Once you get over 5-7 years old and/or 100K miles, it IS expected for parts to begin to wear out, get slow (electric window motors especially), interiors to get a little shabby and so on but in the car's first 5 years at least, is critical that it show itself as being reliable as it will often indicate, if taken care of of how reliable it'll remain as it ages beyond 100K miles. And I should say that I tend to agree with the others in that one should not rely on one source, but a variety of sources as an overall picture will form of the product, first off, do people enjoy it (yes, no) is it reliable (again, yes, no, and to what degree) as cars that proven to be reliable do not necessarily reflect owner's enjoyment as factors such as ride quality, noise, vibration, interior usefulness and all that can negatively affect an owner's overall satisfaction of said car.

  • Jim Bonham Full EVs are not for everyone, they cannot meet all needs. Hybrids do a much better job of providing the benefits of EVs without most of the drawbacks. I have a hybrid sedan with plenty of room, plus all the bells and whistles. It has 360 hp, AWD, does 0-60 in just over 5 sec.(the instant torque is a real benefit), and I get 29 mpg, average. NOT driven lightly. I bought it used for $25k.Sure, it's a little heavier because of the battery, motor, etc., but not nearly as much as a full EV. The battery is smaller/lighter/cheaper and both the alternator and starter motor are eliminated since the motor assumes those functions. It's cool to watch the charge guage show I'm getting energy back when coasting and/or braking. It's even cooler to drive around part of the time on battery only. It really comes in handy in traffic since the engine turns off and you don't waste fuel idling. With the adaptive cruise control you just let the car slowly inch along by itself.I only wish it were a Plug-in Hybrid (PHEV). Then, I'd have A LOT more EV-only range, along with even more of that instant torque. The battery would be bigger, but still a fraction of the size of a full EV. I could easily go weeks without using much, if any gas (depending upon my commute) IF I plug it in every night. But I don't have to. The gas engine will charge the battery whenever it's needed.It's just not as efficient a way to do it.Electric companies offer special rates for both EVs and PHEVs which lower your operating cost compared to gasoline. They'll even give you a rebate to offset the cost of installing a home charger. You can still get federal (up to $7,500, plus some state) tax credits for PHEVs.What's not to like? My next daily driver will be a PHEV of some kind. Probably a performance-oriented one like the new Dodge Hornet or one of the German Hybrid SUVs. All the benefits, sound, feel, etc., of a gas vehicle along with some electric assist to improve fuel economy, performance, and drivability. None of the inherent EV issues of cost, range anxiety, long charging times, poor charger availability, grid capacity issues, etc. I think most people will eventually catch on to this and go PHEV instead of going full EV. Synthetic, carbon-neutral eFuels, hydrogen engines, and other things will also prevent full EVs from being 100% of the fleet, regardless of what the politicians say. PHEVs can be as "clean" (overall) as full EVs with the right fuels. They're also cheaper, and far more practical, for most people. They can do it all, EVs can't.
  • Ron rufo there is in WaSHINGTON STATE
  • ToolGuy @Chris, your photography rocks.
  • ToolGuy No War for Oli.If you have not ever held a piece of structural honeycomb (composite sandwich) in your own hands, try it.
  • ToolGuy You make them sound like criminals.
Next