Our Reporter Reports on Consumer Reports

Michael Karesh
by Michael Karesh

For decades, Consumer Reports has been the American automobile buyer’s primary source for vehicle reliability information. Tens of millions of highly-educated, independent-minded people have made their car purchase based on a brace of red dots. While I don’t care for the dots– they’re a blunt instrument that can hide as much information as they convey– I’ve always assumed that Consumer Reports’ (CR) underlying data was solid. And then I took their survey…

Of the survey’s 19 questions, only one collects the data that's ultimately responsible for Consumer Reports' final, all-important reliability dots: question number 13.

“If you had any problems with your car in the last year (April 1, 2005 through March 31, 2006) that you considered SERIOUS because of cost, failure, safety or downtime, click the appropriate box(es) for each car. INCLUDE problems covered by warranty. DO NOT INCLUDE 1) problems resulting from accident damage; or 2) replacement of normal maintenance items (brake pads, batteries, mufflers) unless they were replaced much sooner or more often than expected.”

CR’s form then lists the car’s major systems, with a simple checkbox next to each. That means that multiple problems with a single system, such as ongoing hassles with a car’s electrics, count once. Equally troubling, respondents are supposed to remember a car problem that may have occurred over a year ago. They also need to remember whether incidents near the cutoff happened in March or April. Respondents that err on the safe side and report problems that might have happened within the timeframe, and do this year after year, are likely to report some problems twice.

There's an even more profound methodological iceberg dead ahead. CR’s dots signal “SERIOUS” problems [note the caps], yet never defines the term. I’ve always wondered how CR staffers decided whether a problem is “serious” enough to include in their survey. They don’t. CR’s question 13 requires that individual respondents make the call, based on “cost, failure, safety or downtime,” or other entirely subjective criteria.


This is a buck that should not be passed. Anyone with a significant other knows that two people hardly ever agree on what constitutes a “serious” problem. As CR does not provides clear guidelines as to which problems qualify as SERIOUS and which do not, the resulting data is not reliable. Would it be so hard for CR to provide a definition of that includes a dollar amount or the number of days out of service? Apparently so.

Without unambiguous guidelines, extraneous influences intrude. First, there’s the respondent’s general opinion of the car. Things gone right can ameliorate things gone wrong. Why else would some people keeping buying those pricey “black dot” jobs? Second, the reliability of cars past shapes consumers’ expectations. If the participant’s previous car lost a transmission, then a bad alternator may not seem so SERIOUS. Unless the current car is the same brand, and the participant is starting to feel twice fooled. Then a burned-out turn signal may seem SERIOUS. And third, if the dealer was smart enough to play nice, maybe kicking in a free loaner, then a SERIOUS problem will seem less severe.

Finally, we come to the part of the question which cautions that replacement of “normal maintenance items” shouldn’t be reported “unless they were replaced much sooner or more often than expected.” This instruction lumps maintenance and repair items together, with no way for CR’s analysts to separate the data later (should they be so ambitious). And, once again, the respondent must define terms, deciding what items count as “normal” and assess the gap between their expectations and reality (usually called irony).

If CR is going to include wear items, it should specify how long they should last. But how long should brake pads last? Expectations are going to vary. A lot. Brake pad life is heavily affected by driving style, driving conditions, a tire shop’s financial goals and other factors that have nothing to do with reliability. And batteries? How many times were the lights left on? How much crud has been allowed to build up around the terminals? Asking average car owners to gauge their vehicle’s parts wear against an entirely subjective ideal does not a scientific study make. If they really want to know about brake pads and batteries, they should at least ask about them separately, to keep the nasty things from contaminating the entire data set. And provide some guidelines.

I’m no triskaidekaphobic. But Consumer Reports’ question 13 does nothing to instill confidence in their reliability ratings, and much to cast doubt on their value. Respondents and readers need a more scientific and, ultimately, more useful guide to automotive reliability. Until CR’s survey undergoes a major overhaul, readers will be misled and manufacturers won’t have the valuable feedback they need to make genuine improvements.

[Michael Karesh operates www.truedelta.com , a vehicle reliability and price comparison site.]

Michael Karesh
Michael Karesh

Michael Karesh lives in West Bloomfield, Michigan, with his wife and three children. In 2003 he received a Ph.D. from the University of Chicago. While in Chicago he worked at the National Opinion Research Center, a leader in the field of survey research. For his doctoral thesis, he spent a year-and-a-half inside an automaker studying how and how well it understood consumers when developing new products. While pursuing the degree he taught consumer behavior and product development at Oakland University. Since 1999, he has contributed auto reviews to Epinions, where he is currently one of two people in charge of the autos section. Since earning the degree he has continued to care for his children (school, gymnastics, tae-kwan-do...) and write reviews for Epinions and, more recently, The Truth About Cars while developing TrueDelta, a vehicle reliability and price comparison site.

More by Michael Karesh

Comments
Join the conversation
2 of 32 comments
  • EricGo EricGo on Jul 03, 2006

    M Karesh: I think your criticisms of the methods are valid if the goal is to predict repair frequences by system for any one model, but I at least do not use them in that fashion. I think their value is in comparing across models. It is analogous to the famous EPA fuel economy debate. While most people will not obtain the epa large print value, it is nonetheless true almost across the board that the relative ratios hold true. By that I mean that if my current car in my hands gets 80% of averaged city/highway EPA, I am very likely to get 80% of averaged city/highway EPA of my next car. Detroit spends a lot of money and effort in trying to discredit CR, arguing that import owners have a memory or value bias not present in domestic car owners. It is a logical question to ask, but I have *never* seen any data to back up the conjecture. The onus of proof is on you.

  • Michael Karesh Michael Karesh on Jul 04, 2006

    My original criticism of CR was that you really cannot compare cars using its ratings. You'll find it here: http://www.truedelta.com/pieces/shortcomings.php True, a "better than average" car is likely to be more reliable than an "average" car, but how much better? How and where does Detroit spend "a lot of money"? Can you provide a specific example? If they've been trying to discredit CR, they've done a poor, viritually invisible job of it.

  • CanadaCraig You can just imagine how quickly the tires are going to wear out on a 5,800 lbs AWD 2024 Dodge Charger.
  • Luke42 I tried FSD for a month in December 2022 on my Model Y and wasn’t impressed.The building-blocks were amazing but sum of the all of those amazing parts was about as useful as Honda Sensing in terms of reducing the driver’s workload.I have a list of fixes I need to see in Autopilot before I blow another $200 renting FSD. But I will try it for free for a month.I would love it if FSD v12 lived up to the hype and my mind were changed. But I have no reason to believe I might be wrong at this point, based on the reviews I’ve read so far. [shrug]. I’m sure I’ll have more to say about it once I get to test it.
  • FormerFF We bought three new and one used car last year, so we won't be visiting any showrooms this year unless a meteor hits one of them. Sorry to hear that Mini has terminated the manual transmission, a Mini could be a fun car to drive with a stick.It appears that 2025 is going to see a significant decrease in the number of models that can be had with a stick. The used car we bought is a Mk 7 GTI with a six speed manual, and my younger daughter and I are enjoying it quite a lot. We'll be hanging on to it for many years.
  • Oberkanone Where is the value here? Magna is assembling the vehicles. The IP is not novel. Just buy the IP at bankruptcy stage for next to nothing.
  • Jalop1991 what, no Turbo trim?
Next