Car Buying Tips: Consumer Reports, You Decide

Michael Karesh
by Michael Karesh
Consumer Reports has released the 2007 edition of its “Annual Auto Issue.” For the second year in a row, all CR’s “Top Picks” come from Japanese makes. For some industry observers, that’s a problem. They believe the magazine’s results indicate a hidden bias, especially against vehicles produced by domestic manufacturers. Which both is and isn’t true.

Consumer Reports’ road test engineers subjected every test vehicle to a thorough evaluation, using a pre-established set of criteria and weights. For example, emergency handling might get ten points, front seat comfort might receive eight and “feels like a Honda” might be worth 37 (just kidding— I hope). Whatever the formula, when the magazine totaled-up the points, they ended up with a list composed entirely of Japanese cars.

This process leads to an obvious question: what criteria and weights– what formula– does Consumer Reports use to rate any given vehicle? The press and Consumer Reports have a policy in this regard: don’t ask, won’t tell.

At the last Detroit auto show, I asked a Consumer Reports road test engineer why the magazine doesn’t publish its formulas. After all, nearly every enthusiast-oriented magazine does when conducting a comparison test. “It’s policy,” he replied. He went on to suggest that he didn’t make the policy, he didn’t necessarily support it, but as a Consumer Reports employee, he had no choice but to follow it.

It's time for Consumer Reports to declassify its formulas. Two days ago, someone made the same request on their forum. The moderator’s response was revealing (or not):

“Thank you for your comments. These forums are designed to help subscribers in selecting and buying a car. They are intended to be primarily peer to peer, with our Auto test experts helping out when available.“If you find errors we will be glad to look into them and make corrections, but we just don’t have the time or resources to engage in lengthy debates here.

“You can channel your inquiries through Customer Service. There is a link on the bottom of every forum page, and at the top of every CR on-line page. You are also welcome to visit our facilities when we hold an open house and speak directly with our test staff at that time.”

As an automotive data provider, I find Consumer Reports’ arrogance, intransigence and unaccountability completely unacceptable. Any company that depends on the public trust must strive for transparency. If you have nothing to hide, you hide nothing. That’s why I respond to any and all questions about TrueDelta’s methodology. Besides, engaging in open public debate can teach data providers better ways to do things.

The Consumer Reports moderator’s non-response indicates that the magazine doesn’t see how knowledge of its overall score formulas could further improve anyone’s ability to find the right car. These formulas are divulged on a need-to-know basis, and as far as they’re concerned car shoppers don’t need to know.

Sorry, but it just isn’t so. To keep things simple, let’s assume there are only two criteria, ride and handling. Let’s further assume that Consumer Reports’ editors have decided that ride quality is twice as important as handling when evaluating a minivan. Keeping the overall score formula secret implies that any reasonable minivan buyers should also weight ride quality twice as heavily as handling.

This is just plain wrong. There is no objective way to arrive at one best formula for everyone. For some minivan buyers handling is twice as important as ride quality, and there’s nothing inherently superior about either set of weights. These are necessarily subjective value judgments. The Toyota Sienna is Consumer Reports’ “top pick” among minivans. But a buyer who values handling above ride quality will be happier in a Honda Odyssey.

This is not to say that overall evaluations are necessarily useless. If the formula was provided, our minivanista could determine that giving extra weight to handling would tip the decision in favor of the Honda.

But the formula is not provided, so there’s no way for minivan shoppers to know how closely Consumer Reports’ criteria and weights match their own or how adjusting these might affect the decision. By withholding its formulas, Consumer Reports takes the stand that readers should let the Yonkers mob decide for them what matters most– and least– when choosing a vehicle.

It’s true that many, even most car buyers are intimidated by the process– to the point where they want an authority figure to tell them what to buy. But buyers truly interested in finding the best fit for their personal tastes are going to have to put forth more effort. Currently they’ll have to rely on sources other than Consumer Reports, since the magazine withholds information needed by people who want to think for themselves.

[Michael Karesh operates www.truedelta.com, a vehicle reliability and price comparison website.]

Michael Karesh
Michael Karesh

Michael Karesh lives in West Bloomfield, Michigan, with his wife and three children. In 2003 he received a Ph.D. from the University of Chicago. While in Chicago he worked at the National Opinion Research Center, a leader in the field of survey research. For his doctoral thesis, he spent a year-and-a-half inside an automaker studying how and how well it understood consumers when developing new products. While pursuing the degree he taught consumer behavior and product development at Oakland University. Since 1999, he has contributed auto reviews to Epinions, where he is currently one of two people in charge of the autos section. Since earning the degree he has continued to care for his children (school, gymnastics, tae-kwan-do...) and write reviews for Epinions and, more recently, The Truth About Cars while developing TrueDelta, a vehicle reliability and price comparison site.

More by Michael Karesh

Comments
Join the conversation
2 of 91 comments
  • Automaton Automaton on Mar 14, 2007

    I don't know if this has been mentioned before, but, CR doesn't publish the criteria for their tests on ANYTHING they test because they don't want manufacturers to build to that test criteria (in the same way car manufacturers build their cars to the CAFE standards and have, as a result, made that test inaccurate). It's a means to prompting the manufacurers to build the best possible product across all perameters.

  • Anonymous Anonymous on Oct 16, 2007

    [...] Re: Consmer Reports update Here, read this: Car Buying Tips: Consumer Reports, You Decide | The Truth About Cars I think CR is full of crap and is succumbing to pressure from domestic automakers in an attempt to preserve what is soon to be the end of an era - a US stronghold on the light truck market - a little while longer. When I think that as my head hits my pillow tonight, I will sleep just just fine. I hope they wake up and smell the coffee. My 2 cents. __________________ 2007 DC LTD, Silver Metallic. MODS: Line-X Bedliner; Toytec 3" Lift; HUGE Moto Metal 20" Wheels and 35*13.5*20 Procomp Xtreme A/T Tires; Sirius Radio; Nuvo GPS Navigation; Hornblasters; Black Westin Bull Bar with fog lights; Northern Tools Tool Box; TRD CAI; Color-matched Fender Flares; Boorla Catback Dual Exhaust. ***Pics in gallery. [...]

  • TheMrFreeze That new Ferrari looks nice but other than that, nothing.And VW having to put an air-cooled Beetle in its display to try and make the ID.Buzz look cool makes this classic VW owner sad 😢
  • Wolfwagen Is it me or have auto shows just turned to meh? To me, there isn't much excitement anymore. it's like we have hit a second malaise era. Every new vehicle is some cookie-cutter CUV. No cutting-edge designs. No talk of any great powertrains, or technological achievements. It's sort of expected with the push to EVs but there is no news on that front either. No new battery tech, no new charging tech. Nothing.
  • CanadaCraig You can just imagine how quickly the tires are going to wear out on a 5,800 lbs AWD 2024 Dodge Charger.
  • Luke42 I tried FSD for a month in December 2022 on my Model Y and wasn’t impressed.The building-blocks were amazing but sum of the all of those amazing parts was about as useful as Honda Sensing in terms of reducing the driver’s workload.I have a list of fixes I need to see in Autopilot before I blow another $200 renting FSD. But I will try it for free for a month.I would love it if FSD v12 lived up to the hype and my mind were changed. But I have no reason to believe I might be wrong at this point, based on the reviews I’ve read so far. [shrug]. I’m sure I’ll have more to say about it once I get to test it.
  • FormerFF We bought three new and one used car last year, so we won't be visiting any showrooms this year unless a meteor hits one of them. Sorry to hear that Mini has terminated the manual transmission, a Mini could be a fun car to drive with a stick.It appears that 2025 is going to see a significant decrease in the number of models that can be had with a stick. The used car we bought is a Mk 7 GTI with a six speed manual, and my younger daughter and I are enjoying it quite a lot. We'll be hanging on to it for many years.
Next