Car Buying Tips: Consumer Reports, You Decide

Michael Karesh
by Michael Karesh
Consumer Reports has released the 2007 edition of its “Annual Auto Issue.” For the second year in a row, all CR’s “Top Picks” come from Japanese makes. For some industry observers, that’s a problem. They believe the magazine’s results indicate a hidden bias, especially against vehicles produced by domestic manufacturers. Which both is and isn’t true.

Consumer Reports’ road test engineers subjected every test vehicle to a thorough evaluation, using a pre-established set of criteria and weights. For example, emergency handling might get ten points, front seat comfort might receive eight and “feels like a Honda” might be worth 37 (just kidding— I hope). Whatever the formula, when the magazine totaled-up the points, they ended up with a list composed entirely of Japanese cars.

This process leads to an obvious question: what criteria and weights– what formula– does Consumer Reports use to rate any given vehicle? The press and Consumer Reports have a policy in this regard: don’t ask, won’t tell.

At the last Detroit auto show, I asked a Consumer Reports road test engineer why the magazine doesn’t publish its formulas. After all, nearly every enthusiast-oriented magazine does when conducting a comparison test. “It’s policy,” he replied. He went on to suggest that he didn’t make the policy, he didn’t necessarily support it, but as a Consumer Reports employee, he had no choice but to follow it.

It's time for Consumer Reports to declassify its formulas. Two days ago, someone made the same request on their forum. The moderator’s response was revealing (or not):

“Thank you for your comments. These forums are designed to help subscribers in selecting and buying a car. They are intended to be primarily peer to peer, with our Auto test experts helping out when available.“If you find errors we will be glad to look into them and make corrections, but we just don’t have the time or resources to engage in lengthy debates here.

“You can channel your inquiries through Customer Service. There is a link on the bottom of every forum page, and at the top of every CR on-line page. You are also welcome to visit our facilities when we hold an open house and speak directly with our test staff at that time.”

As an automotive data provider, I find Consumer Reports’ arrogance, intransigence and unaccountability completely unacceptable. Any company that depends on the public trust must strive for transparency. If you have nothing to hide, you hide nothing. That’s why I respond to any and all questions about TrueDelta’s methodology. Besides, engaging in open public debate can teach data providers better ways to do things.

The Consumer Reports moderator’s non-response indicates that the magazine doesn’t see how knowledge of its overall score formulas could further improve anyone’s ability to find the right car. These formulas are divulged on a need-to-know basis, and as far as they’re concerned car shoppers don’t need to know.

Sorry, but it just isn’t so. To keep things simple, let’s assume there are only two criteria, ride and handling. Let’s further assume that Consumer Reports’ editors have decided that ride quality is twice as important as handling when evaluating a minivan. Keeping the overall score formula secret implies that any reasonable minivan buyers should also weight ride quality twice as heavily as handling.

This is just plain wrong. There is no objective way to arrive at one best formula for everyone. For some minivan buyers handling is twice as important as ride quality, and there’s nothing inherently superior about either set of weights. These are necessarily subjective value judgments. The Toyota Sienna is Consumer Reports’ “top pick” among minivans. But a buyer who values handling above ride quality will be happier in a Honda Odyssey.

This is not to say that overall evaluations are necessarily useless. If the formula was provided, our minivanista could determine that giving extra weight to handling would tip the decision in favor of the Honda.

But the formula is not provided, so there’s no way for minivan shoppers to know how closely Consumer Reports’ criteria and weights match their own or how adjusting these might affect the decision. By withholding its formulas, Consumer Reports takes the stand that readers should let the Yonkers mob decide for them what matters most– and least– when choosing a vehicle.

It’s true that many, even most car buyers are intimidated by the process– to the point where they want an authority figure to tell them what to buy. But buyers truly interested in finding the best fit for their personal tastes are going to have to put forth more effort. Currently they’ll have to rely on sources other than Consumer Reports, since the magazine withholds information needed by people who want to think for themselves.

[Michael Karesh operates www.truedelta.com, a vehicle reliability and price comparison website.]

Michael Karesh
Michael Karesh

Michael Karesh lives in West Bloomfield, Michigan, with his wife and three children. In 2003 he received a Ph.D. from the University of Chicago. While in Chicago he worked at the National Opinion Research Center, a leader in the field of survey research. For his doctoral thesis, he spent a year-and-a-half inside an automaker studying how and how well it understood consumers when developing new products. While pursuing the degree he taught consumer behavior and product development at Oakland University. Since 1999, he has contributed auto reviews to Epinions, where he is currently one of two people in charge of the autos section. Since earning the degree he has continued to care for his children (school, gymnastics, tae-kwan-do...) and write reviews for Epinions and, more recently, The Truth About Cars while developing TrueDelta, a vehicle reliability and price comparison site.

More by Michael Karesh

Comments
Join the conversation
2 of 91 comments
  • Automaton Automaton on Mar 14, 2007

    I don't know if this has been mentioned before, but, CR doesn't publish the criteria for their tests on ANYTHING they test because they don't want manufacturers to build to that test criteria (in the same way car manufacturers build their cars to the CAFE standards and have, as a result, made that test inaccurate). It's a means to prompting the manufacurers to build the best possible product across all perameters.

  • Anonymous Anonymous on Oct 16, 2007

    [...] Re: Consmer Reports update Here, read this: Car Buying Tips: Consumer Reports, You Decide | The Truth About Cars I think CR is full of crap and is succumbing to pressure from domestic automakers in an attempt to preserve what is soon to be the end of an era - a US stronghold on the light truck market - a little while longer. When I think that as my head hits my pillow tonight, I will sleep just just fine. I hope they wake up and smell the coffee. My 2 cents. __________________ 2007 DC LTD, Silver Metallic. MODS: Line-X Bedliner; Toytec 3" Lift; HUGE Moto Metal 20" Wheels and 35*13.5*20 Procomp Xtreme A/T Tires; Sirius Radio; Nuvo GPS Navigation; Hornblasters; Black Westin Bull Bar with fog lights; Northern Tools Tool Box; TRD CAI; Color-matched Fender Flares; Boorla Catback Dual Exhaust. ***Pics in gallery. [...]

  • MaintenanceCosts It's not a Benz or a Jag / it's a 5-0 with a rag /And I don't wanna brag / but I could never be stag
  • 3-On-The-Tree Son has a 2016 Mustang GT 5.0 and I have a 2009 C6 Corvette LS3 6spd. And on paper they are pretty close.
  • 3-On-The-Tree Same as the Land Cruiser, emissions. I have a 1985 FJ60 Land Cruiser and it’s a beast off-roading.
  • CanadaCraig I would like for this anniversary special to be a bare-bones Plain-Jane model offered in Dynasty Green and Vintage Burgundy.
  • ToolGuy Ford is good at drifting all right... 😉
Next