Our Reporter Reports on Consumer Reports

Michael Karesh
by Michael Karesh
We’re committed to finding, researching, and recommending the best products. We earn commissions from purchases you make using links in our articles. Learn more here
our reporter reports on consumer reports

For decades, Consumer Reports has been the American automobile buyer’s primary source for vehicle reliability information. Tens of millions of highly-educated, independent-minded people have made their car purchase based on a brace of red dots. While I don’t care for the dots– they’re a blunt instrument that can hide as much information as they convey– I’ve always assumed that Consumer Reports’ (CR) underlying data was solid. And then I took their survey…

Of the survey’s 19 questions, only one collects the data that's ultimately responsible for Consumer Reports' final, all-important reliability dots: question number 13.

“If you had any problems with your car in the last year (April 1, 2005 through March 31, 2006) that you considered SERIOUS because of cost, failure, safety or downtime, click the appropriate box(es) for each car. INCLUDE problems covered by warranty. DO NOT INCLUDE 1) problems resulting from accident damage; or 2) replacement of normal maintenance items (brake pads, batteries, mufflers) unless they were replaced much sooner or more often than expected.”

CR’s form then lists the car’s major systems, with a simple checkbox next to each. That means that multiple problems with a single system, such as ongoing hassles with a car’s electrics, count once. Equally troubling, respondents are supposed to remember a car problem that may have occurred over a year ago. They also need to remember whether incidents near the cutoff happened in March or April. Respondents that err on the safe side and report problems that might have happened within the timeframe, and do this year after year, are likely to report some problems twice.

There's an even more profound methodological iceberg dead ahead. CR’s dots signal “SERIOUS” problems [note the caps], yet never defines the term. I’ve always wondered how CR staffers decided whether a problem is “serious” enough to include in their survey. They don’t. CR’s question 13 requires that individual respondents make the call, based on “cost, failure, safety or downtime,” or other entirely subjective criteria.

This is a buck that should not be passed. Anyone with a significant other knows that two people hardly ever agree on what constitutes a “serious” problem. As CR does not provides clear guidelines as to which problems qualify as SERIOUS and which do not, the resulting data is not reliable. Would it be so hard for CR to provide a definition of that includes a dollar amount or the number of days out of service? Apparently so.

Without unambiguous guidelines, extraneous influences intrude. First, there’s the respondent’s general opinion of the car. Things gone right can ameliorate things gone wrong. Why else would some people keeping buying those pricey “black dot” jobs? Second, the reliability of cars past shapes consumers’ expectations. If the participant’s previous car lost a transmission, then a bad alternator may not seem so SERIOUS. Unless the current car is the same brand, and the participant is starting to feel twice fooled. Then a burned-out turn signal may seem SERIOUS. And third, if the dealer was smart enough to play nice, maybe kicking in a free loaner, then a SERIOUS problem will seem less severe.

Finally, we come to the part of the question which cautions that replacement of “normal maintenance items” shouldn’t be reported “unless they were replaced much sooner or more often than expected.” This instruction lumps maintenance and repair items together, with no way for CR’s analysts to separate the data later (should they be so ambitious). And, once again, the respondent must define terms, deciding what items count as “normal” and assess the gap between their expectations and reality (usually called irony).

If CR is going to include wear items, it should specify how long they should last. But how long should brake pads last? Expectations are going to vary. A lot. Brake pad life is heavily affected by driving style, driving conditions, a tire shop’s financial goals and other factors that have nothing to do with reliability. And batteries? How many times were the lights left on? How much crud has been allowed to build up around the terminals? Asking average car owners to gauge their vehicle’s parts wear against an entirely subjective ideal does not a scientific study make. If they really want to know about brake pads and batteries, they should at least ask about them separately, to keep the nasty things from contaminating the entire data set. And provide some guidelines.

I’m no triskaidekaphobic. But Consumer Reports’ question 13 does nothing to instill confidence in their reliability ratings, and much to cast doubt on their value. Respondents and readers need a more scientific and, ultimately, more useful guide to automotive reliability. Until CR’s survey undergoes a major overhaul, readers will be misled and manufacturers won’t have the valuable feedback they need to make genuine improvements.

[Michael Karesh operates www.truedelta.com , a vehicle reliability and price comparison site.]

Michael Karesh
Michael Karesh

Michael Karesh lives in West Bloomfield, Michigan, with his wife and three children. In 2003 he received a Ph.D. from the University of Chicago. While in Chicago he worked at the National Opinion Research Center, a leader in the field of survey research. For his doctoral thesis, he spent a year-and-a-half inside an automaker studying how and how well it understood consumers when developing new products. While pursuing the degree he taught consumer behavior and product development at Oakland University. Since 1999, he has contributed auto reviews to Epinions, where he is currently one of two people in charge of the autos section. Since earning the degree he has continued to care for his children (school, gymnastics, tae-kwan-do...) and write reviews for Epinions and, more recently, The Truth About Cars while developing TrueDelta, a vehicle reliability and price comparison site.

More by Michael Karesh

Join the conversation
2 of 32 comments
  • EricGo EricGo on Jul 03, 2006

    M Karesh: I think your criticisms of the methods are valid if the goal is to predict repair frequences by system for any one model, but I at least do not use them in that fashion. I think their value is in comparing across models. It is analogous to the famous EPA fuel economy debate. While most people will not obtain the epa large print value, it is nonetheless true almost across the board that the relative ratios hold true. By that I mean that if my current car in my hands gets 80% of averaged city/highway EPA, I am very likely to get 80% of averaged city/highway EPA of my next car. Detroit spends a lot of money and effort in trying to discredit CR, arguing that import owners have a memory or value bias not present in domestic car owners. It is a logical question to ask, but I have *never* seen any data to back up the conjecture. The onus of proof is on you.

  • Michael Karesh Michael Karesh on Jul 04, 2006

    My original criticism of CR was that you really cannot compare cars using its ratings. You'll find it here: http://www.truedelta.com/pieces/shortcomings.php True, a "better than average" car is likely to be more reliable than an "average" car, but how much better? How and where does Detroit spend "a lot of money"? Can you provide a specific example? If they've been trying to discredit CR, they've done a poor, viritually invisible job of it.

  • Tassos Unlike Tim, I don't use this space as a wastebasket for ANYTHING BUT a proper used car.If you seriously need a car AND you are as destitute as Tim's finds imply, HERE IS A PROPER ONE FOR YOUR NEEDS:You can probably get it for only $4k, WITH Leather, Factory Navigation, plenty of room and a V6.https://www.cars.com/research/toyota-camry-2005/I even considered getting it myself as an extra reliable car.
  • Jeff Of all the EV trucks I like the Rivian the best but I am still years away if ever from buying an EV.
  • Kwik_Shift I definitely like the looks of the newest 300s over the Chargers.
  • SCE to AUX "Should car companies shack up with tech giants in order to produce legible infotainment systems and the like? Or should they go it alone?"Great question(s).The River Rouge days are gone, where Ford produced whole cars out of raw materials entering the plant at the other end. Nearly everything is outsourced these days - sometimes well, sometimes disastrously.But the problem with infotainment systems is that they are integrated with the car's operation. VW has delayed entire products for issues with infotainment.For me, the question boils down to a contractual arrangement - who owns and maintains the code forever? Since more and more of the car's function is tied to the infotainment system, I'd argue that the car mfr needs to own it - especially the larger ones.Do mfrs really want to share intellectual property with Huawei just to fast-track some code they've managed themselves in the past?
  • Kwi65728132 I always did like the styling of the 300C and it was on my short list for a new (to me) rear wheel drive, naturally aspirated V8 luxury sedan but I found a Hyundai Equus that was better optioned than any 300C I could find and for several grand less.