Our Reporter Reports on Consumer Reports

our reporter reports on consumer reports

For decades, Consumer Reports has been the American automobile buyer’s primary source for vehicle reliability information. Tens of millions of highly-educated, independent-minded people have made their car purchase based on a brace of red dots. While I don’t care for the dots– they’re a blunt instrument that can hide as much information as they convey– I’ve always assumed that Consumer Reports’ (CR) underlying data was solid. And then I took their survey…

Of the survey’s 19 questions, only one collects the data that's ultimately responsible for Consumer Reports' final, all-important reliability dots: question number 13.

“If you had any problems with your car in the last year (April 1, 2005 through March 31, 2006) that you considered SERIOUS because of cost, failure, safety or downtime, click the appropriate box(es) for each car. INCLUDE problems covered by warranty. DO NOT INCLUDE 1) problems resulting from accident damage; or 2) replacement of normal maintenance items (brake pads, batteries, mufflers) unless they were replaced much sooner or more often than expected.”

CR’s form then lists the car’s major systems, with a simple checkbox next to each. That means that multiple problems with a single system, such as ongoing hassles with a car’s electrics, count once. Equally troubling, respondents are supposed to remember a car problem that may have occurred over a year ago. They also need to remember whether incidents near the cutoff happened in March or April. Respondents that err on the safe side and report problems that might have happened within the timeframe, and do this year after year, are likely to report some problems twice.

There's an even more profound methodological iceberg dead ahead. CR’s dots signal “SERIOUS” problems [note the caps], yet never defines the term. I’ve always wondered how CR staffers decided whether a problem is “serious” enough to include in their survey. They don’t. CR’s question 13 requires that individual respondents make the call, based on “cost, failure, safety or downtime,” or other entirely subjective criteria.

This is a buck that should not be passed. Anyone with a significant other knows that two people hardly ever agree on what constitutes a “serious” problem. As CR does not provides clear guidelines as to which problems qualify as SERIOUS and which do not, the resulting data is not reliable. Would it be so hard for CR to provide a definition of that includes a dollar amount or the number of days out of service? Apparently so.

Without unambiguous guidelines, extraneous influences intrude. First, there’s the respondent’s general opinion of the car. Things gone right can ameliorate things gone wrong. Why else would some people keeping buying those pricey “black dot” jobs? Second, the reliability of cars past shapes consumers’ expectations. If the participant’s previous car lost a transmission, then a bad alternator may not seem so SERIOUS. Unless the current car is the same brand, and the participant is starting to feel twice fooled. Then a burned-out turn signal may seem SERIOUS. And third, if the dealer was smart enough to play nice, maybe kicking in a free loaner, then a SERIOUS problem will seem less severe.

Finally, we come to the part of the question which cautions that replacement of “normal maintenance items” shouldn’t be reported “unless they were replaced much sooner or more often than expected.” This instruction lumps maintenance and repair items together, with no way for CR’s analysts to separate the data later (should they be so ambitious). And, once again, the respondent must define terms, deciding what items count as “normal” and assess the gap between their expectations and reality (usually called irony).

If CR is going to include wear items, it should specify how long they should last. But how long should brake pads last? Expectations are going to vary. A lot. Brake pad life is heavily affected by driving style, driving conditions, a tire shop’s financial goals and other factors that have nothing to do with reliability. And batteries? How many times were the lights left on? How much crud has been allowed to build up around the terminals? Asking average car owners to gauge their vehicle’s parts wear against an entirely subjective ideal does not a scientific study make. If they really want to know about brake pads and batteries, they should at least ask about them separately, to keep the nasty things from contaminating the entire data set. And provide some guidelines.

I’m no triskaidekaphobic. But Consumer Reports’ question 13 does nothing to instill confidence in their reliability ratings, and much to cast doubt on their value. Respondents and readers need a more scientific and, ultimately, more useful guide to automotive reliability. Until CR’s survey undergoes a major overhaul, readers will be misled and manufacturers won’t have the valuable feedback they need to make genuine improvements.

[Michael Karesh operates www.truedelta.com , a vehicle reliability and price comparison site.]

Join the conversation
2 of 32 comments
  • EricGo EricGo on Jul 03, 2006

    M Karesh: I think your criticisms of the methods are valid if the goal is to predict repair frequences by system for any one model, but I at least do not use them in that fashion. I think their value is in comparing across models. It is analogous to the famous EPA fuel economy debate. While most people will not obtain the epa large print value, it is nonetheless true almost across the board that the relative ratios hold true. By that I mean that if my current car in my hands gets 80% of averaged city/highway EPA, I am very likely to get 80% of averaged city/highway EPA of my next car. Detroit spends a lot of money and effort in trying to discredit CR, arguing that import owners have a memory or value bias not present in domestic car owners. It is a logical question to ask, but I have *never* seen any data to back up the conjecture. The onus of proof is on you.

  • Michael Karesh Michael Karesh on Jul 04, 2006

    My original criticism of CR was that you really cannot compare cars using its ratings. You'll find it here: http://www.truedelta.com/pieces/shortcomings.php True, a "better than average" car is likely to be more reliable than an "average" car, but how much better? How and where does Detroit spend "a lot of money"? Can you provide a specific example? If they've been trying to discredit CR, they've done a poor, viritually invisible job of it.

  • Buickman how about LowIQ?
  • Gemcitytm Corey: As a native SW Ohioan, Powel Crosley, Jr. has always been an object of fascination for me. While you're correct that he wanted most of all to build cars, the story of the company he created with his brother Lewis, The Crosley Corporation, is totally fascinating. In the early 20's, Crosley was the nation's leading manufacturer of radio receivers. In the 1930's, working from an idea brought to him by one of his engineers, Crosley pioneered the first refrigerator with shelves in the door (called, of course, the "Shelvador"). He was the first to sell modular steel kitchen cabinets (made for him by Auburn in Connersville). He brought out the "IcyBall" which was a non-electric refrigerator. He also pioneered in radio broadcasting with WLW Radio in Cincinnati (wags said the calls stood for either "Whole Lotta Watts" or "World's Lowest Wages"). WLW was one of the first 50,000 watt AM stations and in 1934, began transmitting with 500,000 watts - the most powerful station in the world, which Mr. Crosley dubbed "The Nation's Station". Crosley was early into TV as well. The reason the Crosley operation died was because Mr. Crosley sold the company in 1945 to the AVCO Corporation, which had no idea how to market consumer goods. Crosley radios and TVs were always built "to a price" and the price was low. But AVCO made the products too cheaply and their styling was a bit off the wall in some cases. The major parts of the Crosley empire died in 1957 when AVCO pulled the plug. For the full story of Crosley, read "Crosley: Two Brothers and a Business Empire That Transformed the Nation" by Rutsy McClure (a grandson of Lewis Crosley), David Stern and Michael A. Banks, Cincinnati: Clerisy Press, ISBN-13: 978-1-57860-291-9.
  • AndyinMA Well, will they actually make any? Wranglers appear to be black only at this point, but I do admit to seeing a few Gladiators in other colors. A few.
  • Garrett The only way to send a message is to pull out of the transaction when the fee is disclosed unless the dealer pays for it...or just walk out regardless.If this happens enough, eventually someone will get the message.
  • Sgeffe I pay for the Remote and Security HondaLink stuff (remote functions from a phone app; accident notification, etc.), at roughly $200/yr. That’s value-added stuff. (A nice addition is that I can enable the crash-notification on ANY Honda vehicle to which I pair my phone if I wish, as long as the vehicle supports it.) I can cancel this stuff at any time, though! It looks like you CAN’T with Mary’s Folly!Typical GM! 🙄