2010 Consumer Reports Survey Analysis: Part One: Insufficient Data

Michael Karesh
by Michael Karesh

Last week Jack Baruth reviewed the press release that attended Consumer Reports’ latest auto reliability survey results. But don’t run out and buy a Porsche for the sake of reliability just yet. And it might even be safe to buy a Chrysler.

Jack was surprised that Porsche ranked second among makes. On top of this, the Boxster was reported to be the most reliable car. What CR didn’t include in the press release about its coverage of Porsche models:

Number of 2009s with enough responses: 1

(a solid black blob for the 911)

Number of 2010s with enough responses: zero

Consumer Reports’ response to virtually any critique has long been the large size of their sample. Yet their coverage of recent Porsches is almost nonexistent. CR’s predictions are based on however many of the three most recent model years they have sufficient data for. The prediction for the 2011 Boxster is entirely based on the 2008, because that’s the only year they have enough data for. Yet the 2009 included significant revisions. They have no reliability ratings for the Panamera or the all-new Cayenne. So they have little basis for ranking the entire Porsche’s 2011 line. Even so, they rank Porsche second from the top.

Data limitations don’t end with Porsche. CR also did not receive enough responses for…

  • Most 2009 and 2010 Audis. For the A8 they can rate only the 2004. For the S4, only the 2005.
  • Many 2009 and 2010 BMWs, including the 135i and 535i singled out as unreliable in the press release. Consequently, BMW’s brand score is heavily based on the 2008 model year.
  • Most 2010 Cadillacs.
  • Six 2010 Chevrolets.
  • Many 2010 Hyundais, Kias, and Mazdas.
  • Any 2009 or 2010 Land Rover, including the new LR4.
  • Five of the last eight model years of the Merecedes S-Class.
  • The 2009 or the 2010 Mercedes GL-Class. Based on the 2008 alone they predict that the 2011 will be the least reliable SUV.
  • Any 2010 Mitsubishi. And among the 2008s and 2009s, they can rate only the Outlander.
  • Any 2009 or 2010 Saab.
  • The 2010 Scion tC and xD—even with Toyota products their coverage isn’t complete.
  • The 2010 Subaru WRX. They still single the WRX out as the one Subaru to avoid. From TrueDelta’s survey and forums I’ve learned that the engines in early 2009 WRXs have been prone to failure. But this problem was fixed during the 2009 model year, and should not affect the 2010s, much less the 2011s. Unfortunately, CR’s predictions don’t factor in known common problems that have been fixed.
  • Any 2010 Suzuki, including the new Kizashi.
  • Any 2010 Volvo aside from the XC60. And most 2009 Volvos. But the press release still mentions Volvo as one of the two consistently reliable European brands.

In general, coverage of recent model years is much less complete than for 2008 and earlier. The severe downturn in car sales two years ago appears to have severely impacted Consumer Reports’ ability to gather enough data on the 2009 and 2010 model years. As a result, they make predictions for many 2011s based entirely on the 2008 model year, but do not clearly note this. In these cases any improvements (or declines) over the last two years have no impact. And yet they still conclude that some manufacturers have improved over the past year, while others have not.

Chrysler allegedly falls in the latter camp, with the press release reporting that it “remains the lowest-ranked manufacturer.” Chrysler has responded that, based on warranty claims,the quality of its products has greatly improved over the past two model years. Who’s correct? According to CR’s own results, quite possibly Chrysler. By CR’s count, Chrysler offers 28 models.

Number of 2009s with enough responses: 14

Number of 2010s with enough responses: 7

The problem, once again: CR’s coverage is far less complete than their overall sample size (1.3 million) suggests it should be. Chrysler’s rating is heavily based on the 2008 model year. And their products were mostly unreliable that year.

In two cases for which CR has enough data, the minivans and the Dodge Journey, the ratings improve from “much worse than average” for the 2009s to “about average” for the 2010s. This said, if other models have similarly improved, and if CR had had enough data on them, it still wouldn’t have been enough. The predicted reliability formula (which is confidential) appears to equally weight the model years, even though the most recent year is most likely to predict the current year. So a bad 2008 and 2009 can easily outweigh a much better 2010, and do for the minivans and the Journey. Even when CR does have enough data for all model years it often takes three years before an improvement is fully reflected in their predictions. When they don’t have enough data on the most recent years, it can take forever.

With such sparse data on the 2009s and 2010s, and some indication that the reliability of Chrysler’s products has improved while at least one Porsche has gone in the other direction, Consumer Reports probably should have reported that Chrysler’s and Porsche’s relative positions are currently unclear. Instead, they applied a formula that doesn’t take trends into account and that ignores substantial holes in their data. Porsche benefits. Chrysler does not.

Coming in Part II: Should you EcoBoost?

Join the conversation
3 of 56 comments
  • Ronnie Schreiber Ronnie Schreiber on Nov 02, 2010

    Michael, how do the CR sample sizes compare to your own at True Delta? They have a much bigger brand name than you do, yet they have some really small sample sizes. I'm just wondering if you have better success than they do at attracting respondents.

    • Michael Karesh Michael Karesh on Nov 02, 2010

      They generally have much larger sample sizes, and their overall sample is far larger--1.3 million vs. 18,700 (in the latest round). This is why I was so shocked to find their coverage so limited. Even with my much smaller total sample, I have results for some models that they don't cover, such as the Land Rover LR4. A fairly small number of models must account for a large pecentage of their total sample.

  • Glenn Mercer Glenn Mercer on Nov 02, 2010

    Really excellent analytical work! Thanks for publishing this. One footnote to it all: I think the general public may be in some general aware of the CR data problems. A really nice academic study by Hoffer and Korenok done in 2009 on the determinants of OEM market share gains and losses (the paper is easily found by searching for Oleg Korenok) showed that the market share of a given vehicle model barely budged when CR either upgraded or downgraded its quality rating. Surprised me, in fact, but it seems that out there in the real world CR's ratings generate more news coverage than they do changes in buying behavior. That being said, the study just looked at year-on-year ratings changes... I bet that after two or three decades of praising Toyota (e.g.) there is definitely impact.

  • Jim Bonham Full EVs are not for everyone, they cannot meet all needs. Hybrids do a much better job of providing the benefits of EVs without most of the drawbacks. I have a hybrid sedan with plenty of room, plus all the bells and whistles. It has 360 hp, AWD, does 0-60 in just over 5 sec.(the instant torque is a real benefit), and I get 29 mpg, average. NOT driven lightly. I bought it used for $25k.Sure, it's a little heavier because of the battery, motor, etc., but not nearly as much as a full EV. The battery is smaller/lighter/cheaper and both the alternator and starter motor are eliminated since the motor assumes those functions. It's cool to watch the charge guage show I'm getting energy back when coasting and/or braking. It's even cooler to drive around part of the time on battery only. It really comes in handy in traffic since the engine turns off and you don't waste fuel idling. With the adaptive cruise control you just let the car slowly inch along by itself.I only wish it were a Plug-in Hybrid (PHEV). Then, I'd have A LOT more EV-only range, along with even more of that instant torque. The battery would be bigger, but still a fraction of the size of a full EV. I could easily go weeks without using much, if any gas (depending upon my commute) IF I plug it in every night. But I don't have to. The gas engine will charge the battery whenever it's needed.It's just not as efficient a way to do it.Electric companies offer special rates for both EVs and PHEVs which lower your operating cost compared to gasoline. They'll even give you a rebate to offset the cost of installing a home charger. You can still get federal (up to $7,500, plus some state) tax credits for PHEVs.What's not to like? My next daily driver will be a PHEV of some kind. Probably a performance-oriented one like the new Dodge Hornet or one of the German Hybrid SUVs. All the benefits, sound, feel, etc., of a gas vehicle along with some electric assist to improve fuel economy, performance, and drivability. None of the inherent EV issues of cost, range anxiety, long charging times, poor charger availability, grid capacity issues, etc. I think most people will eventually catch on to this and go PHEV instead of going full EV. Synthetic, carbon-neutral eFuels, hydrogen engines, and other things will also prevent full EVs from being 100% of the fleet, regardless of what the politicians say. PHEVs can be as "clean" (overall) as full EVs with the right fuels. They're also cheaper, and far more practical, for most people. They can do it all, EVs can't.
  • Ron rufo there is in WaSHINGTON STATE
  • ToolGuy @Chris, your photography rocks.
  • ToolGuy No War for Oli.If you have not ever held a piece of structural honeycomb (composite sandwich) in your own hands, try it.
  • ToolGuy You make them sound like criminals.