By on November 1, 2010

Last week Jack Baruth reviewed the press release that attended Consumer Reports’ latest auto reliability survey results. But don’t run out and buy a Porsche for the sake of reliability just yet. And it might even be safe to buy a Chrysler.

Jack was surprised that Porsche ranked second among makes. On top of this, the Boxster was reported to be the most reliable car. What CR didn’t include in the press release about its coverage of Porsche models:

Number of 2009s with enough responses: 1

(a solid black blob for the 911)

Number of 2010s with enough responses: zero

Consumer Reports’ response to virtually any critique has long been the large size of their sample. Yet their coverage of recent Porsches is almost nonexistent. CR’s predictions are based on however many of the three most recent model years they have sufficient data for. The prediction for the 2011 Boxster is entirely based on the 2008, because that’s the only year they have enough data for. Yet the 2009 included significant revisions. They have no reliability ratings for the Panamera or the all-new Cayenne. So they have little basis for ranking the entire Porsche’s 2011 line. Even so, they rank Porsche second from the top.

Data limitations don’t end with Porsche. CR also did not receive enough responses for…

  • Most 2009 and 2010 Audis. For the A8 they can rate only the 2004. For the S4, only the 2005.
  • Many 2009 and 2010 BMWs, including the 135i and 535i singled out as unreliable in the press release. Consequently, BMW’s brand score is heavily based on the 2008 model year.
  • Most 2010 Cadillacs.
  • Six 2010 Chevrolets.
  • Many 2010 Hyundais, Kias, and Mazdas.
  • Any 2009 or 2010 Land Rover, including the new LR4.
  • Five of the last eight model years of the Merecedes S-Class.
  • The 2009 or the 2010 Mercedes GL-Class. Based on the 2008 alone they predict that the 2011 will be the least reliable SUV.
  • Any 2010 Mitsubishi. And among the 2008s and 2009s, they can rate only the Outlander.
  • Any 2009 or 2010 Saab.
  • The 2010 Scion tC and xD—even with Toyota products their coverage isn’t complete.
  • The 2010 Subaru WRX. They still single the WRX out as the one Subaru to avoid. From TrueDelta’s survey and forums I’ve learned that the engines in early 2009 WRXs have been prone to failure. But this problem was fixed during the 2009 model year, and should not affect the 2010s, much less the 2011s. Unfortunately, CR’s predictions don’t factor in known common problems that have been fixed.
  • Any 2010 Suzuki, including the new Kizashi.
  • Any 2010 Volvo aside from the XC60. And most 2009 Volvos. But the press release still mentions Volvo as one of the two consistently reliable European brands.

In general, coverage of recent model years is much less complete than for 2008 and earlier. The severe downturn in car sales two years ago appears to have severely impacted Consumer Reports’ ability to gather enough data on the 2009 and 2010 model years. As a result, they make predictions for many 2011s based entirely on the 2008 model year, but do not clearly note this. In these cases any improvements (or declines) over the last two years have no impact. And yet they still conclude that some manufacturers have improved over the past year, while others have not.

Chrysler allegedly falls in the latter camp, with the press release reporting that it “remains the lowest-ranked manufacturer.” Chrysler has responded that, based on warranty claims,the quality of its products has greatly improved over the past two model years. Who’s correct? According to CR’s own results, quite possibly Chrysler. By CR’s count, Chrysler offers 28 models.

Number of 2009s with enough responses: 14

Number of 2010s with enough responses: 7

The problem, once again: CR’s coverage is far less complete than their overall sample size (1.3 million) suggests it should be. Chrysler’s rating is heavily based on the 2008 model year. And their products were mostly unreliable that year.

In two cases for which CR has enough data, the minivans and the Dodge Journey, the ratings improve from “much worse than average” for the 2009s to “about average” for the 2010s. This said, if other models have similarly improved, and if CR had had enough data on them, it still wouldn’t have been enough. The predicted reliability formula (which is confidential) appears to equally weight the model years, even though the most recent year is most likely to predict the current year. So a bad 2008 and 2009 can easily outweigh a much better 2010, and do for the minivans and the Journey. Even when CR does have enough data for all model years it often takes three years before an improvement is fully reflected in their predictions. When they don’t have enough data on the most recent years, it can take forever.

With such sparse data on the 2009s and 2010s, and some indication that the reliability of Chrysler’s products has improved while at least one Porsche has gone in the other direction, Consumer Reports probably should have reported that Chrysler’s and Porsche’s relative positions are currently unclear. Instead, they applied a formula that doesn’t take trends into account and that ignores substantial holes in their data. Porsche benefits. Chrysler does not.

Coming in Part II: Should you EcoBoost?

Get the latest TTAC e-Newsletter!

56 Comments on “2010 Consumer Reports Survey Analysis: Part One: Insufficient Data...”


  • avatar
    Steven02

    Really interesting data here Michael. Thanks for posting.
     
    Specifically, on the models that don’t have enough data, are these typically small volume sellers?  I assume Porsche models are pretty small sellers, but what about the Chevy’s for example.  Which ones were they?
     
    What about the Kias, Hyundais, and Mazdas?  Which ones? (especially with the Kia/Hyundai market share rising)
     
    Do you think that this is also because CR only sends information to its readers on the cars they bought?  I am sure they had enough Toyotas and Hondas there to fill up those, but I could also see that the deviation from having much smaller samples of other vehicle makes being very large.
     
    Does CR report how many responses per model they receive?

    • 0 avatar

      Actually, sales volume isn’t always a factor.

      For Chevrolet:
      Avalanche
      Aveo
      Cobalt
      Corvette
      Express
      HHR

      For Hyundai:
      Accent
      Azera

      For Kia:
      Optima
      Rio (insufficient data for ALL model years)
      Sedona
      Sportage

      For Mazda:
      CX-7
      Mazda6 V6
      MazdaSpeed3
      RX-8

      CR has repeatedly refused to divulge sample sizes at the model level. The only exception I can recall is that a few years ago they said they had 7,000 for a single year and perhaps even a single year/powertrain of the Camry.

      Part II will demonstrate that sample size is far from their largest problem, anyway.

    • 0 avatar
      SVX pearlie

      These are all small sellers, at least among CR’s self-selected reader base (the other big asterisk in CR’s methodology that they don’t admit to). I see CR’s readership as the Volvo demographic, so they don’t buy cheap things like the Aveo / Cobalt / Optima, no matter how popular the car might be, nor sporty things like the Corvette / RX-8 / Porsche.

      Plus, CR has been drumming for Toyota (and Honda & Subaru, by extension), it’s a wonder that any other manufacturers are in the sample at all.

    • 0 avatar

      You seem to have missed the line for Volvo. They had enough responses for only a single 2010 Volvo model, and only a few more 2009s. So they might have the used Volvo demographic, but not the new Volvo demographic (such as it still exists).

      In itself, I don’t see surveying only their subscribers as a major issue. It only becomes an issue when their subscribers don’t buy a broad range of car models, as seen here, and when it’s combined with a badly worded question, as discussed in Part II.

    • 0 avatar
      John Horner

      Volvo doesn’t actually have a demographic any more … but that is a rant for another time and another place :).
       

  • avatar

    re the Porsches: My understanding, verified by all the “insufficient data” marks in their charts with the red, black, and unfilled in circles, is that when they have insufficient data, they let you know. Am I missing something?
    The rest of your points are very well taken. But I still think CR has it way over JD Power.

    • 0 avatar

      You’re missing that most people–including, apparently, the entire automotive press–won’t dig deep enough to view those charts.

      You’re also missing that CR’s predictions and recommendations generally fail to note when they are based on partial data. They consistently write and make predictions as if they have complete data, even when they don’t.

    • 0 avatar

      Yeah, CR is way better than JD power, who rates squeaky brakes and glovebox rattles the same as catastrophic failures of the engine and transmission.

  • avatar
    ash78

    I agree with the idea of weighting, but ONLY as it applies to a consistent platform with no substantial refresh. Once a car has a new engine and/or tranny, plus other bits like suspension, the problems that plagued previous years should ideally be dimished or disappear completely. From my understanding (from these critiques, plus reading their recommendations), CR weighs heavily the manufacturer’s historical reputation when they lack concrete data.
     
    The problem is we’re talking about machines with a complex series of subcontracted assemblies, which are often replaced or upgraded. CR makes it seem like they’ve got a Framingham Heart Study on theirs hands. “Well, his daddy had a bad ICM, so he’s prolly gonna have a bad ICM, too”

    • 0 avatar
      HoldenSSVSE

      Correct.  This bit them in the ass in 2008 when the new Camry and Tundra both came out.  Sight unseen they gave them a “recommended” rating based on “historical performance” despite huge changes to the Tundra in particular.  What happened?  Both vehicles can in below average for 2008.  So much for predicted reliability.  At least they threw out their Toyota rubber stamp.

  • avatar
    Telegraph Road

    Until CR and JD Power allow for independent peer review of their “studies” by freeing their data, their opaque “analysis” should be seen merely as glorified anecdotes.   

    • 0 avatar
      jj99

      JD Power studies released to the public stop at 3 years.  At 3 years, most brands look fine, and the JD results agree with CR.  This is because Detroit does not pay for studies in the 5 to 10 year bracket.  I would bet this is because the JD result would match CR result, and this is not good for Detroit.  CR shows how poorly Domestic vehicles look compared to Toyota and Honda after 5 years. I think JD and CR are both valid.  They agree when the horizon is the same. 

    • 0 avatar

      Manufacturers aren’t interested in data on older cars because those cars are no longer in production, the next generation car is already done, and so there’s no way they could apply the data.

  • avatar
    HoldenSSVSE

    Thank you for exposing CR and their methodology.  I know when I’ve asked the question sample size was always trotted out as the validation.  As SVX Pearlie also points out above, CR’s same is a self-selected group.  Who answers the call?  Why CR readers do.  I don’t know a single publication that can say the cut across all ages, races, education level, genders, income, employment status, etc. etc.
     
    For almost my entire life I’ve had to listen to, “but CR says,” glad to see some digging to show that CR is as I thought, a rather flawed system for rating quality.

  • avatar
    jmo

    I’d like to see a post about proposals to weight faults.   How do you compare a crazy annoying dash rattle that they can’t find, to a power window that won’t go up in the middle of the winter after you get your coffee, to a transmission failure?

    • 0 avatar

      TrueDelta will take a step in this direction with the November survey. I am not sure how soon we will actually be able to apply the resulting data, as it will require larger sample sizes.

    • 0 avatar
      jmo

      What methodology will you be using?

    • 0 avatar

      A simplified version of an FMEA scale. Something like:

      1 Minor annoyance–squeak, rattle, trim, fit & finish

      2 Feature used infrequently functioning poorly or not at all

      3 Feature used daily functioning poorly or not at all

      4 Feature necessary for comfort functioning poorly or not at all (AC,, window stuck in down posn)

      5 Vehicle performance impaired, but could be dependably driven for another week

      6 Vehicle performance impaired, cannot be dependably driven for another week

      7 Vehicle had to be towed

      The hard part will be putting this in a form that people can easily answer.

      Also not sure where something like a major water leak should fall.

      I’ve had a questions that essentially separates out the 6s and covered towing since the beginning. There aren’t many 6s and 7s.

    • 0 avatar

      How about this:

      http://www.truedelta.com/blog/?p=663

  • avatar
    carve

    One other thing important to mention is I don’t believe usage is taken into account.  I’m sure one reason Porsche is so reliable is because, for most people, they aren’t daily drivers.  Any car will last a LOOONNNGGGG time getting driven 5,000 miles/year, on roadtrips and weekend outings rather than a daily commute.

    I heard they gave the 2010 Prius a mediocre reliability reading. Is that based on the performance they were getting back in 2008?

    Also, I’m an online subscriber to CR. Can I do the survey, or is that for print subscribers only? Maybe I shouldn’t though- don’t want my 335i to depreciate any faster than it needs to :)

    • 0 avatar

      Responses to TrueDelta’s Car Reliability Survey suggest that Boxsters do, in fact, average about 5k miles a year. We note this in our results.

      Consumer Reports claims to adjust for miles driven. But the top ranking of the Boxster and Corvette Z06 (well above even the regular Corvette) suggest that they don’t adjust them nearly enough.

      The average rating for the Prius stems from a brake issue that was fixed with a software update. TrueDelta has a much different result for the Prius because we don’t count free software updates as repairs–otherwise manufacturers might be discouraged from offering them. If we did included software updates, the reported repair frequency for some 2007 and 2008 GM models with the six-speed automatic would skyrocket.

    • 0 avatar
      ash78

      Another flaw that I’ve complained about for many, many years is that mileage doesn’t tell much of a story…maybe 50%-60% of a car’s wear & tear.
       
      1. Cycles — how many times turned on/off
      2. Cold starts
      3. Total running hours or total engine revolutions
      4. Intangible factors, such as driving style, temperatures, garaging (makes a huge difference with paint and rubber)
       
      It’s amazing to me we’ve come this far in such a mature industry, yet we still act like age and mileage are the only two things that matter. It’s almost insulting.

    • 0 avatar

      You’re necessarily limited by the data you can gather. Until manufacturers include far more sophisticated trip computers, this sort of information cannot be gathered outside of an experimental setting.

    • 0 avatar
      ash78

      MK, understand completely…my point was that, now that we all have sophisticated ECUs, there’s no reason to NOT report such data. Out of those suggestions, I think a total rev count would be the easiest and most useful, followed by cold starts.

    • 0 avatar
      golden2husky

      Ash78:  Well said.  I can’t believe the ignorance when it comes to mileage.  Cycles and trip distance are critical factors in wear.  A six year old car that gets used daily but has only 25,000 miles has far more wear than the average buyer thinks but forget trying to educate them on that point.  I’ve owned some 4 mile a day specials with 50,000 miles that I bought and pressed into 20K a year service.  Weird stuff fails – the spring in the driver door handle, the ignition switch, fan speed switch, a vacuum diaphragm in the climate control system, the brake light switch, my first starter motor job…  No doubt the cycles on these parts exceeded the design life built into them.  When it comes to low mileage older cars, the usage pattern is far more important.  Garage queens are the best; they get used at least monthly which is enough to keep atrophy at bay but keeps cycles really low.  That’s why a Porsche probably looks so good to the toaster testers at CR.

    • 0 avatar
      Lorenzo

      Actually, a lot of owners DO know that info, but there’s got to be a good way to ask for it.  Some of us crackpots keep a written log of mileage between fillups, and know under what conditions they’ve driven the car most of the time.
       
      For instance, I bought my ’95 Altima in April of ’96, with 7,117 miles on it, and I now have 50,238 miles on it. Over 14.5 years, I haven’t varied usage much from the long term average of about 3000 miles a year, or from the average of 10-12 miles a day. I know I use it for a short commute and it sits for a day or two a week, and when it’s in use it gets up to operating temperature and up to freeway speeds for a couple minutes nearly every trip. A long trip in this car is 25 miles each way in the same day, maybe once a month. For anything longer, I don’t use this car.
       
      How do you ask for such information in a survey, and how do you use it to judge the car’s reliability/durability? I’ve driven this car as long as a ’63 Dodge Dart wagon I owned, but under very different conditions. The Dart’s peripherals fell apart under heavy use, but the slant six and torqueflite were bulletproof. Maybe that’s all you need to know. The one thing surveys can’t answer is how are YOU going to use the car?

  • avatar
    jj99

    An important result here is no complaints about large volume Japanese and Domestic vehicles.  So, the general CR result that all Toyota and Honda brands are more reliable than all Domestic brands holds true.  I thought so.

  • avatar

    AHA! Nice digging, MK.
    @Michael Karesh: I tried to look for GTI reliability on TrueDelta, but it’s mashed in with Jetta and Golf for several years. What gives?

    • 0 avatar

      Not enough owners involved yet. For more recent model years I split out the 2.0T, but even for these we need more participants to provide precise stats, and I’m not sure the split is beneficial. Unless I see a clear pattern of engine-related repairs, any reported difference between engines is likely noise that would be reduced by combining the samples.

  • avatar
    getacargetacheck

    The need today for any reliability/durability index is overstated.  All vehicles today are built well.  What’s hardly reported is that the difference between “the best” and “the worst” is nearly negligible in the real world.  CR and Power and the rest are living off the memories of the 1980s when American carmakers were really struggling with quality.  Is it any wonder CR makes a recommendation even when it doesn’t have the data?  Their customers demand it.  Signed, the driver of a consistently low-ranked make with 75,000 trouble-free miles (so far).

    • 0 avatar
      HoldenSSVSE

      Yup.  Even some of the “worst” built cars today will go 150K to 200K miles relatively trouble free.  The gap from worst to best was as big as the Grand Canyon 30 years ago – that isn’t the case today with few outliers on either end of the spectrum.

    • 0 avatar

      This is the reason I started TrueDelta’s survey in the first place–to make the real size of the differences much clearer.
      Anyone interested in helping with the survey can read about it here:
      http://www.truedelta.com/reliability.php

    • 0 avatar
      18726543

      I definitely wouldn’t go as far as to say that reliability/durability indexes are over-rated or unnecessary anymore.  Sure, no one’s making cars in the states with motors that detonate in 35k miles, or vehicles that explode due to a golfball-sized impact, but it is still very helpful to keep track of repair records for the sheer annoyance factor of having a broken car! 

      For example, MANY recent model Dodge/Chrysler sedans have been suffering from breakage of a cheaply made plastic piece incorporated into the shift interlock system.  Sure, this failure won’t cause a flaming 9-car pile-up on the highway but it will leave you stranded for a while until the tow truck guy shows up.  Considering the number of vehicles currently experiencing this, it may be handy to know as you shop that 2-year-old Avenger. 

      Also, many systems on newer cars (fully electric power steering, adaptive headlights, SRS systems with 90 million incorporated bag units) are all serviced specifically by your dealer.  It isn’t as simple as bringing these cars to the shop down the street anymore.  You are pretty much forced to play the stealership game and that can be a real pain in itself. 

      I guess my main point is, “bad” cars don’t fail as drastically as they used to, true enough, but there are still inherant, wide-spread failures in vehicles that I would prefer to know about rather than buy into.

    • 0 avatar
      CarPerson

      Because of how some of the Consumer Reports articles are written, I have to wonder if the writer/reviewer actually lives 25 stories up and only sees the inside of a cab. New York City is not the first place I’d look for real expertise in the automobile.
       
      Detroit is too inbred. The Midwest doesn’t seem to make a good showing and the south is NASCAR where nothing is stock on their stock cars.   Maybe Eugene, OR?

  • avatar
    brandloyalty

    I’m happy to be a truedelta participant.
    As for CR, they must be fairly smart people. They must be aware of the severe limitations of their car reliability survey.  They say they’re dedicated to telling the truth about things such as car reliability.
    Do they have internal conflicts between staff who are comfortable with this state of affairs and those who are not?  How do they sleep at night, knowing the degree to which their sloppiness distorts the information, causing people to variously buy cars more and less reliable than they expected, and causing car manufacturers to have better or worse reputations than deserved, with consequent impacts on sales and staffing etc?
    This sort of thing doesn’t seem to diminish CR’s popularity.  Is CR just another mediocre operation, rather than being the cut above mediocrity as CR relentlessly markets itself?

    • 0 avatar

      I suspect it’s just another organization where it’s hard to make changes. The near total lack of competition or serious critiques probably hasn’t helped.

      I have no doubt that plenty of people within the organization fully recognize these problems, and probably did before I ever wrote about them.

      It’s the same within any car manufacturer. There’s nothing you might know that plenty of people within these companies don’t also know. They just can’t do anything about it.

  • avatar
    mpresley

    I participate in TD, but my vehicle has a small sample, so I’m not sure what any of the results really mean.  Also, as others have pointed out, how to rate a “failure” such as a faulty glove box hinge versus, say, a transmission failure?  Finally, what about driving styles.  A four or five year old GTI driven by a hot rod kid is not going to be the same as a four year old Avalon when it comes to reliability etc.

    • 0 avatar

      Contact me any time with specific questions, I’d be happy to answer them.

      The way a car is used and maintained will have more impact on older cars. But, by and large, the repairs reported could not have been avoided through different maintenance or use.

      The great majority of reported repairs are minor, even on CR’s survey which asks people to only report serious problems. Truly serious problems are so rare for most cars that you’d require a sample size in the hundreds to precisely measure them.

  • avatar
    TrailerTrash

    Thanks, Michael.
    Keep up the heavy lifting.
     
    I wish there was a way to help spread the word.
    Don’t you feel like you are swimming upstream at times?
     
    For the record, I have stopped my subscription to CR.  They constantly send out reminders, and I delete.
    The tires I purchased per their “advice”, the Goodyear tripletreads, were a disaster.
    Two refrigerators now occupy lower levels or garages.  I couldn’t take them anymore.
    They didn’t rate highly  any of the cars I got, and I love them.
    I could go on.
    But one day you wake up and realize they don’t know what they are talking about.
     

  • avatar
    pgcooldad

    Chrysler out to sue them for damages. The sooner the better.

  • avatar
    tklockau

    I’m finding it increasingly hard to give Consumer Reports much credibility when they recommend the Volvo C30, but tell you to avoid the S40 and V50, despite the fact that they have the same basic structure, drivetrain, electronics, and sheetmetal.

    Other family members owned a ’95 Blazer and an ’05 Trailblazer that were listed poorly in CR, but they were both good cars. There were/are three New Beetles, two A4′s, a TT coupe, and a V50 and XC90 in my family and all have been reliable cars. Certainly nothing that left them stranded. A VW Touareg, however was really bad; it was bought back by VW and the Trailblazer replaced it.

    • 0 avatar
      psarhjinian

      I’m finding it increasingly hard to give Consumer Reports much credibility when they recommend the Volvo C30, but tell you to avoid the S40 and V50, despite the fact that they have the same basic structure, drivetrain, electronics, and sheetmetal.

      To be fair, this happens in the real world for good reason.  There are real differences in assembly, location, engineering revisions, and so forth that do play a role.

      The classic example is the American-built Pontiac Grand Prix, which always showed a little less reliable than it’s Canadian-built siblings despite sharing quite a lot under the skin.  There are others, too, and while generally assembly location differs, there may be engineering changes that happened between the S40/V50 and later C30 that come into play.

    • 0 avatar
      Patrickj

      Actually, there are very sound reasons why a C30 would be more reliable than an S40 or V50.  The C30 is using the same powertrains, body structures, and brakes to run about 400 pounds less car–plus lighter passenger and cargo loads.
      Given owner demographics, hooning of the C30s is unlikely to make up the difference.

    • 0 avatar

      To be fair, when the sample size is one or two there’s a very good chance your experience will differ from the average.

  • avatar
    golden2husky

    CR does not like when people print or post information about their testing and quality surveys.  Go to ALLPAR and you will find that when they wrote an article questioning the validity of the CR survey, they were contacted by CR’s legal team threatening a suit.  The Emperor of Quality really has no clothes…I am glad to be a TD participant.  Of all the cars in our family for the past 30 years, only one was the black dot special that CR said it was.

    • 0 avatar

      They’re very insular and feel they deserve special treatment. And, for the most part, they get it.

      The use of CR’s results in order to critique these results is fair use, however much they might threaten otherwise. I thought at the time that allpar was on safe ground. But CR no doubt has attorneys on retainer and have much deeper pockets than Dave Z.

      Press stories have referred to them as a David fighting Goliaths. Right.

  • avatar
    Kendahl

    It should be no surprise that CR’s sample sizes are small for Porsches, Corvettes and the like. Their drivers don’t give a rat’s ass about CR or its ratings.
     
    CR is probably ok for toasters, washing machines, etc. However, they sometimes stray into areas where there is an established high end subculture. Examples are audio, video and photography. In those areas, their recommendations tend to be mediocre, mainstream products whose principal virtue is affordability rather than absolute performance. If they test high end stuff at all, they claim it’s not worth the money. Aficionados know better.

    • 0 avatar
      golden2husky

      I’ve been beating that drum for a long time as well.  But it does not have to be at the esoteric level.  Even when rating more mainstream brands, they trip up.  My first awaking to the limitations of CR was with audio products…they considered my choice of a Yamaha CR-640 (yeah, the golden age of quality-made consumer grade stuff) a worse buy than some Radio Shack Realistic branded thing…Ugh.  Can’t even imagine what they would say if somebody dropped a piece of Krell equipment by for testing….I can hear it now…”all this money and only one channel!!”…

  • avatar
    samdog

    Free software updates still require a trip to the dealer, with all of the scheduling/dropping/waiting that is entailed. This is a neccessary repair to make a car function as it was expected to by the customer and should absolutely be included as a “repair”.

    The 2007 and 2008 GM six-speed transmission update data should skyrocket. Ignoring it is hiding the truth. Repeated trips to the dealer for transmission updates would not only qualify under most Lemon laws, but would most certainly be of interest to potential buyers, dontcha think?  

    • 0 avatar

      I can certainly see your point. My reasoning:

      –these updates can often be done very quickly while you wait

      –they’re becoming increasingly common, and will probably eventually outnumber conventional repairs

      –the line between a software repair and a “performance enhancement” can get very fine, and I don’t want to have to make hundreds of such calls based on often minimal information from car owners.

  • avatar

    Michael, how do the CR sample sizes compare to your own at True Delta? They have a much bigger brand name than you do, yet they have some really small sample sizes. I’m just wondering if you have better success than they do at attracting respondents.

    • 0 avatar

      They generally have much larger sample sizes, and their overall sample is far larger–1.3 million vs. 18,700 (in the latest round). This is why I was so shocked to find their coverage so limited. Even with my much smaller total sample, I have results for some models that they don’t cover, such as the Land Rover LR4. A fairly small number of models must account for a large pecentage of their total sample.

  • avatar
    Glenn Mercer

    Really excellent analytical work!  Thanks for publishing this.  One footnote to it all: I think the general public may be in some general aware of the CR data problems.  A really nice academic study by Hoffer and Korenok done in 2009 on the determinants of OEM market share gains and losses (the paper is easily found by searching for Oleg Korenok) showed that the market share of a given vehicle model barely budged when CR either upgraded or downgraded its quality rating.  Surprised me, in fact, but it seems that out there in the real world CR’s ratings generate more news coverage than they do changes in buying behavior.  That being said, the study just looked at year-on-year ratings changes… I bet that after two or three decades of praising Toyota (e.g.) there is definitely impact.


Back to TopLeave a Reply

You must be logged in to post a comment.

Subscribe without commenting

Recent Comments

New Car Research

Get a Free Dealer Quote

Staff

  • Contributing Writers

  • Jack Baruth, United States
  • Brendan McAleer, Canada
  • Marcelo De Vasconcellos, Brazil
  • Vojta Dobes, Czech Republic
  • Matthias Gasnier, Australia
  • W. Christian 'Mental' Ward, Abu Dhabi
  • Mark Stevenson, Canada
  • Cameron Aubernon, United States
  • J Emerson, United States