By on March 6, 2007

2007_ridgeline_rtl_6722.jpgConsumer Reports has released the 2007 edition of its “Annual Auto Issue.” For the second year in a row, all CR’s “Top Picks” come from Japanese makes. For some industry observers, that’s a problem. They believe the magazine’s results indicate a hidden bias, especially against vehicles produced by domestic manufacturers. Which both is and isn’t true.

Consumer Reports’ road test engineers subjected every test vehicle to a thorough evaluation, using a pre-established set of criteria and weights. For example, emergency handling might get ten points, front seat comfort might receive eight and “feels like a Honda” might be worth 37 (just kidding— I hope). Whatever the formula, when the magazine totaled-up the points, they ended up with a list composed entirely of Japanese cars.

This process leads to an obvious question: what criteria and weights– what formula– does Consumer Reports use to rate any given vehicle? The press and Consumer Reports have a policy in this regard: don’t ask, won’t tell. 

At the last Detroit auto show, I asked a Consumer Reports road test engineer why the magazine doesn’t publish its formulas. After all, nearly every enthusiast-oriented magazine does when conducting a comparison test. “It’s policy,” he replied. He went on to suggest that he didn’t make the policy, he didn’t necessarily support it, but as a Consumer Reports employee, he had no choice but to follow it. 

It's time for Consumer Reports to declassify its formulas. Two days ago, someone made the same request on their forum. The moderator’s response was revealing (or not):

“Thank you for your comments. These forums are designed to help subscribers in selecting and buying a car. They are intended to be primarily peer to peer, with our Auto test experts helping out when available.“If you find errors we will be glad to look into them and make corrections, but we just don’t have the time or resources to engage in lengthy debates here.

“You can channel your inquiries through Customer Service. There is a link on the bottom of every forum page, and at the top of every CR on-line page. You are also welcome to visit our facilities when we hold an open house and speak directly with our test staff at that time.”

As an automotive data provider, I find Consumer Reports’ arrogance, intransigence and unaccountability completely unacceptable. Any company that depends on the public trust must strive for transparency. If you have nothing to hide, you hide nothing. That’s why I respond to any and all questions about TrueDelta’s methodology. Besides, engaging in open public debate can teach data providers better ways to do things.

The Consumer Reports moderator’s non-response indicates that the magazine doesn’t see how knowledge of its overall score formulas could further improve anyone’s ability to find the right car. These formulas are divulged on a need-to-know basis, and as far as they’re concerned car shoppers don’t need to know.

Sorry, but it just isn’t so. To keep things simple, let’s assume there are only two criteria, ride and handling. Let’s further assume that Consumer Reports’ editors have decided that ride quality is twice as important as handling when evaluating a minivan. Keeping the overall score formula secret implies that any reasonable minivan buyers should also weight ride quality twice as heavily as handling.

This is just plain wrong. There is no objective way to arrive at one best formula for everyone. For some minivan buyers handling is twice as important as ride quality, and there’s nothing inherently superior about either set of weights. These are necessarily subjective value judgments. The Toyota Sienna is Consumer Reports’ “top pick” among minivans. But a buyer who values handling above ride quality will be happier in a Honda Odyssey.

This is not to say that overall evaluations are necessarily useless. If the formula was provided, our minivanista could determine that giving extra weight to handling would tip the decision in favor of the Honda.

But the formula is not provided, so there’s no way for minivan shoppers to know how closely Consumer Reports’ criteria and weights match their own or how adjusting these might affect the decision. By withholding its formulas, Consumer Reports takes the stand that readers should let the Yonkers mob decide for them what matters most– and least– when choosing a vehicle.

It’s true that many, even most car buyers are intimidated by the process– to the point where they want an authority figure to tell them what to buy. But buyers truly interested in finding the best fit for their personal tastes are going to have to put forth more effort. Currently they’ll have to rely on sources other than Consumer Reports, since the magazine withholds information needed by people who want to think for themselves.

[Michael Karesh operates, a vehicle reliability and price comparison website.]

Get the latest TTAC e-Newsletter!


91 Comments on “Car Buying Tips: Consumer Reports, You Decide...”

  • avatar
    gerhard trombley

    Perhaps it’s time for a comparo of consumer guide magazines. But not here.

  • avatar

    Refusing to reveal procedures is often (if not always) an indication that your procedures will not stand up to scrutiny. Maybe CR’s recommendations can be substantiated, maybe not. Until they show the contents of their “black boxes” we should disregard their recommendations, or at least take them with a healthy grain of salt.

  • avatar
    dean rune

    I wonder how the manufacturers feel about it? I guess it’s always a boost to be number one, but when you can’t explain why, kind of diminishes it.

  • avatar

    CR didn’t give favorable ratings for the Ford Escort when they wre new ( 91 – 96 models) for reliability but years later they were one of the few domestic small cars they recommend buying used. So when should a buyer follow their advice and when are they themselves wrong?

    No one has a crystal ball about long term reliability but we do see some qualities from day one. Some are subjective, some aren’t, but CR has a lot of influence and this influence even affects the depreciation a bit. Honda dealers are known to wag copies of their annual report around the showroom.

    CR rates other things under interesting criteria and once again we are supposed to trust their little
    “dot” scores, among other things, because they don’t have advertising and ask readers for their views.

  • avatar

    I remember when they blasted the Saturn Ion for having the center mounted gauge pod, but merely stated that the same design was an inconvenience in the Toyota Echo. I trust CR’s reliability ratings bacause they are based on statistics, but everything else is subjective.

  • avatar

    Simple scientific method: if you can’t defend the data, the test is worthless.

  • avatar

    One thing that could probably be clearer is what I meant by a “formula.” One would look like this:

    Criteria 1 * Weight 1 + C2*W2 + C3*W3 = Total

  • avatar

    CR seems to have a bias towards imports for the same (intangible) reasons that most car rags do: refinement. Their reviewers seem to try to like domestics (especially when the numbers closely match an import), but a coarse drivetrain, lousy fit/finish and safety features almost always drive the imports up a notch. The good thing about CU is their higher weighting of active/passive safety of a car, which they feel is as important as other aspects of performance/value. Ex: The Toyota Yaris, despite several positives (Reliability, fuel econ, price) was not recommended because of poor non-ABS braking performance (and the fact that ABS-equipped Yari are hard to find), and lousy at-the-limit handling in their “avoidance maneuver”. No, they’re not perfect, but they buy their vehicles ($2.8M worth last year), have their own test track (some level of consistency), and stress the attributes of a vehicle that are most important to the average consumer (their audience). They do have enthusiasts on staff, but they’re budget-limited, so few “high-end” cars are tested (Jeremy C. need not reply).

  • avatar
    richard whitman

    One of the biggest problems in CR evaluations is that they are done from a NE US suburban location and frame of mind. These have no bearing on California, Florida, Texas and Georgia driving environments.

  • avatar

    I agree with cheezeweggie. As a tool for predicting reliability CR is one of the best tools around. It’s not perfect, but its the best we got.

    As far as their “testing” is concerned. It’s really hard to figure out how they came up with a winner.

    At least with the buff books, you can ignore silly results in the “got to have it” category and come up with a true winner for yourself. CR just give us a “The Baxter Belchfire is the best… Trust us!”

  • avatar

    Please, bring the “Edit” feature back!
    “Coarse drivetrain, lousy fit/finish and lack of passive/active safety features on [em]domestics[/em] will drive the imports up a notch.”
    “Jeremy Clarkson need not [em]apply[/em]”.

  • avatar
    jerry weber

    There is more to consumer reports on weighting problems. First the difference between the best and worst cars has narrowed dramatically in the past few years. Thus if the good car has say 95 defects per 1000 and the bad car has say 120 you may statistically never notice. This compared to cars three and four times as bad as others years ago. Does consumer reports say we did our job and the statistical difference is negligible? No, they simply keep ranking the miniscule differences and choosing winners and losers accordingly. then there is the mercedes bmw thing. Are they much worse than lexus? I don't see them being towed in alongside the interstate, However, they both have a fussy complicated audio and climate control computer. This consumer reports would call a defect and blend it in with say engine or transmission malfunctions. Thus each of these cars have a defect that raises their scores through the roof and mercedes are worse than yugos in reliability. I prefer simple dials for the radio however that like weird styling and strange locations for gauges do not a defect make. Read the test reports (many of them) but always drive the car and if possible ask questions of owners before buying. I do and sometimes still make mistakes. But like the minivan blogger said if ride is important not handling, and you don't know why the honda didn't do as well as the toyota in the rankings you could make the wrong choice with a professionals help.

  • avatar

    I’m on the fence on this one. On the one hand, using numeric ratings to judge a car’s subjective values is a bit silly in general. It’s one thing to put a value on braking distance and emergency lane change time, but to add those values up with “steering feel” in some kind of formula only serves to cheapen the whole score. So if they’re going to do something that absurd, they might as well release their formulas so that buyers can un-screw the numbers back into a useful, subjective analysis.

    On the other hand, releasing the formulas to auto makers would lead to gaming of the system. This also shows how silly these formulas are. If it comes out that consumer reports gives equal weight to scores for “visibility” and “handling,” then car designers have a simple choice… spend millions improving suspension design, or spend thousands making the windows bigger. Designing cars to CR specs means everyone wins but the consumer.

    The real solution highlighted by all of this is to get rid of the formulas. Give me a subjective analysis, throw in some numbers if you have ’em, and let me rank the cars based on the available info. To think that even non-car-people can’t be bothered to do that for the second largest purchases of their lives really is arrogant.

  • avatar

    This is an excellent question.

    On the other hand, it is like watching a baseball game with an Umpire that has a clear bias to allow an inside pitch that other Umps wouldn’t have. As long as he consistently calls it the same on both pitchers, it all evens out.

    I agree that Consumer Reports ratings can be confusing. When I am shopping, I have to use their input as only one variable – no matter how “unbiased” they claim to be. But because they buy their own cars (no manufacturer gimmees) and seem to be accumuating “user” reviews en masse, I have to believe they have some credibilty. What motivation would they have to be down on North American cars if there wasn’t cause – I see no reason. If one looks at Manufacturer behaviours, like the GM and Ford intake manifold avoidance manoeuvere (deny, the avoid, then admit grudgingly), ya gotta wonder if there’s smoke, there has to be fire.

    As a prudent shopper, I have fond that if I line up their ratings and reviews, along with JD Power and Edmunds and a few others, I pretty much can determine a good from bad choice. Then it is up to me to decide.

    Or yell at the Ump.

  • avatar

    Of course bias,as is most of the media{TTAC accepted}
    The media bias is responsible for the quality perception gap that exists today.
    The Toyota Matrix and the Pontiac Vibe are identical cars.Made by the same people at the same time in the same plant.
    Guess what one gets the higher rating.

  • avatar

    One additional clarification: I’m talking about the overall road test scores here. To become a best pick, a car must:

    –have a road test score at or near the top
    –average or better reliability
    –adequate safety test scores

    From this the road test score is clearly the most important. Excellent reliability doesn’t improve a car’s standing compared to a car with average reliability. Oddly, this suggests that even CR feels that average reliability is good enough these days, and that once a car has attained that level other factors become more important.

  • avatar


    No, Vibe & Matrix get identical ratings, just got my copy of CR the other days. There was a couple points difference with ratings previously, only because one version had AWD which has been discontinued for 2007.

    I use CR as a starting point for looking for vehicles, as their biases are reasonably in-line with mine. There are vehicles they rate highly and recommend that I wouldn’t buy for myself nor recommend to friends/family, such as the Ford Focus.

  • avatar


    If the formula is valid, then you should want the manufacturers to “game the system,” as this would result in better cars. It’s only when a formula is seriously incomplete or inaccurate that gaming it would be a problem. For example, the EPA fuel economy test is not representative of the full range of real-world driving, so gaming it does not yield more efficient cars to nearly the same extent it yields higher numbers.

    To take your example, if visibility is truly important, then by all means give manufacturers a clear incentive to improve it.

    I say the same thing with my reliability work at TrueDelta. I’m eventually going to be reporting “days in the shop” as well as “times in the shop.” Someone once suggested that manufacturers might try to game the system by improving the supply of critical parts, so their cars would be in the shop for fewer days.

    That would be just awful, wouldn’t it?

  • avatar

    I ALWAYS consult Consumer Reports about everything I buy. They may be biased, but at least they are not bought and sold by advertisers. They are also not the only source of information for cars – but they are often the sole source for appliances and the like, and all sorts of other things I simply do not have the time or inclination to investigate. In a word, I trust them more than just about any other source of information.

    Cars are another matter. As an enthusiast, I know alot about cars. I read CR’s reports on reliability. They are assembled by real world tallys, so I beleive that they are accurate.

    For someone who does not feel they want to get a Ph.D. in Things Automotive, there are pity few relativley unbiased sources. None of the glossy car mags can be trusted. Few web sites either. We here like to think that TTAC is, but its really an enthusiasts site. For these people, CR is perfect. Read the review, see whats recommended, buy that, get on with life.

    Simple, Like me when i needed a new washer dryer. It took about 30 minutes to find the one i wanted using CR, another 30 minutes to price and buy it, using the internet. Done. Pop the cork on the merlot.

    I dont want to think about washer dryers. Lots of people dont want to think about cars. Until there is a better place to get info, there it is. Maybe more TTAC’s dn True Delta’s. I hope so. But until that time, CR it is!

  • avatar

    I wrote about the reliability survey side of CR here:

  • avatar
    Gardiner Westbound

    Consumer Reports is the automotive reporting gold standard. Pretenders have not closed the gap, so they nitpick and carp.

    The poor scores achieved by domestically branded automobiles are not the fruit of a dastardly plot as millions of owners will attest. They paid good money for abysmal quality products and suffered despicable customer care and large losses from poor resale values.

    The domestics have to reach and hold quality and reliability standards at least equal to their rivals for several years to attain the coveted CR “recommended” status.

  • avatar

    mikey, TTAC is absolutely biased. It’s a gearhead bias. You can’t deny that it’s there. TTAC implies it in every word they write. This is a good thing, it’s why we’re here.

    Disclosed bias _is_ _not_ _a_ _bad_ _thing_. Everybody has some preconceptions. It is something to take into account when evaluating what advice to trust and what advice to discard. What is troublesome is undisclosed/unindicated bias: hidden advertiser influence, claiming to be more objective than you actually are. Or say TTAC claimed to be a generic automobile consumer advocacy site. Fox News’s ‘Fair and Balanced’ slogan, for a real life example.

    I agree that CR should release the road test formula, but I don’t think it’s a death blow to the utility of the magazine. The meat of CR’s auto issue is statistics: reliability, owner satisfication, and depreciation. And those stats are devastating to American cars (and, to at least some extent, to all makes except Honda, Toyota, and Subaru).

    I’m not going to trust a CR reviewer or anyone else to tell me how the car drives or rides. I’m going to go to the dealer and take a test drive. The reviewers might be taller than I am, shorter than I am, fatter than I am, less padding in the ‘seat’, more padding in the ‘seat.’

  • avatar

    So CR’s standards are secret and questionable? Surprise, surprise. I quit listening to them regarding automobiles back in 1986 when they rated something new from GM (forget which model, maybe a Buick Regal) as excellent ride, excellent handling, excellent performance, DON’T BUY IT!!!!! Why? Because GM’s (in general) frequency of repair rate, and that, of course, negated everything else. Back then, it was Japanese, Japanese, Japanese, to the point of suggesting that, at the end of a test of a Mercedes 190E (which tested very well, no real complaints), it was recommended that you buy a Camry. In their eyes nothing was worth paying more than the price of a Camry.

  • avatar
    William C Montgomery

    I agree. However, I suppose that CR does not open the kimono and reveal their company secretes because it deems them to be proprietary. Making their formulae public might aid their competitors in imitating them. Google keeps a tight lid on exactly how they determine Page Rank in internet searches for the same reason.

    Ultimately, the proof is in the pudding, not the mixer. If the public disagrees with CR conclusions and recommendations, the company will loose credibility and its vital subscription base. Overall, it would appear that most people agree with them, so they are successful.

    But then again, most people are also sheep.

  • avatar

    Several years ago, they dinged a corvette because it did not have back seats.

  • avatar


    And poor Mike was dinged in those forums for being pro-import. He just can’t win.

    I did join his panel a few weeks ago, since he deserves as data point as he can get for all his hard work and I own a couple of vehicles TrueDelta is tracking.

  • avatar

    I stopped reading CR in the 80s. When it comes to reviews, every reviewer is biased. What I like about this site is that the reviewers wear their bias on their sleeves… there’s no hidden agenda. You say not just what you think is good or bad with a car, but also what your personal preferences are that give you that judgement. With CR, they have to be just as biased, but they claim to be unbiased, an impossibility. That, for me, makes them a bad source of information.

  • avatar

    A Corvette with back seats??? Sacrilege!

  • avatar

    Greg, that’s an interesting point about gaming the system. US News & World Report ranks colleges, and unlike CR, they are open about their formula. That has problems because 1) colleges totally game the system in response, 2) the measurements often have little to do with education quality, 3) many important measurements that should have been included are subjective and impossible to quantify, and 4) each prospective student’s ideal criteria is an individual matter, and will differ from their given criteria.

    Ranking cars shouldn’t be as complex as ranking colleges. But disclosing how you came to your conclusions is an absolute necessity.

  • avatar

    I think you are overlooking the overall formula that CR uses in any of it’s auto tests – How does the average Soccer Mom like the car. The reports on autos from the public is only from their subscribers so it is a case of preaching to the choir.

  • avatar

    Michael, is the main point of your article concerning the overall evaluation of the vehicle by CR rather than the reported reliability ratings?

    If you were talking about the overall evaluation of the car I would agree with you. If we are talking reliability ratings I will still stick with CR for one simple reason. Their ratings have consistently mirrored my car experiences. When I was younger I ignored them and I got precisely the same problems as reported in CR in my vehicles.

    Here is my take on this. The Detroit News reported this weekend that GM’s response to the CR news was that their warranty claims are now 40 percent lower than from 2002. My reaction to that is to question how good could a GM car from 1999 to 2002 be (I am using a typical 3 year warranty as a basis) in comparison to their competition? We are not talking about vegas and citations here we are talking about some models which are still produced by GM. Was GM admitting back in 2002 that their cars were not as good as Toyota and Honda in 2002 or were we all hearing the same arguments back then as now. I remeber reading the same arguments put forth by domestic automakers put out in the 80s, 90s and in 2000. You can only cry wolf so many times before people don’t believe you anymore.

    I believe the reliability of a most Toyota or Honda Model to its equivalent GM counterpart generally gets better reliability ratings because …..drumroll……they are actually are more reliable. Note I said most and not all.

    If GM’s warranty repair expenses were 40 percent higher in 2002 what does that say about GM cars? I am not talking about Vegas and Citations. Many of their models from 2002 are still in production. I have heard the same arguments put forth from the domestic car industry from the 80s, 90s and 2000. Our cars are just as good now blah blah, CR is biased. They said it in the 80s, they said it in the 90s and they said in 2000, Yet at the same time they always denied that there cars were less reliable. You can only cry wolf so many times before people stop believing you.

  • avatar

    In your response, I think you prefectly demonstrated one of Mr. Karesh’s points for him. You said “And those stats are devastating to American cars (and, to at least some extent, to all makes except Honda, Toyota, and Subaru).” To me it seems, that is the exact sort of impression that CR fosters due to their lack of published methods and numbers.

    CR’s method of mainly using relative reliability comparisons, rather than absolute comparisons, give the impression that Toyota and Honda are MUCH more reliable than other brands, when in fact the variation can be quite small.

    A question about reliability and CR recommendations. I read that in the last round of surveys, the Fusion/Milan V6 had better predicted long term reliability that either the Accord or Camry yet in terms of numbers, the Camry scored a 87, Accord a 89, and the Fusion a 77. Do those numbers reflect the surveys, or is that the mystery formula Michael is talking about.

  • avatar

    oophs didn’t proof read my past post with repeated paragraph

  • avatar


    Those numerical scores are the mystery formula Michael’s talking about. I sent a link to this article, asking them for a reply.

  • avatar

    I recently cancelled my CR online subscription after 3 years. The reason is quite simple: Internal inconsistency with consumer products. I figure if they can’t report consistent data from year to year (ie, how a certain product can go from best to worst in 12-18 months with no substantial changes in competition) leads me to question how well they can do this with cars.

  • avatar

    IMO they do what they intend to do, provide recommendations for people that have no idea about cars.
    Lets face it, if we have to go out and buy a car for our daughters (or other relatives) and just want them to have a car that does the job without being burdened by inconvenience, then Toyota and Honda are the only way to go based on playing the odds.
    When buying a car for ourselves we might go for a second rate car if we can buy it very heavily discounted, because we probably will be able to take care of the problem without going to the dealer.

    If one has to go to the dealer, generally speaking Honda and Toyota dealers are not as unpleasant and inconvenient. Not because they are dealers for Japanese makes, but because by being profitable they attract better people and a happier workforce that tends to solve problems without inconvenience.
    It follows that manufacturers and dealers that are unprofitable will squeeze customers at every turn.

    This is the segment of the market CR adresses.

  • avatar
    Steve Biro

    I tend to agree with jerseydevil. CR is a useful tool in terms of determining vehicle reliability, particularly because most enthusiast magazines don’t even approach the subject. But when it comes to selecting a vehicle based on utility and driving dynamics, I don’t even pay attention to them. That’s because CR’s automotive priorities are close to 180-degrees from mine. But that’s just me. I don’t pay attention to movie reviews for the same reason. Now… CR’s recommendations on a washer, dryer, vacuum cleaner, etc. are different. I pay attention to those because, when it come to household appliances, my feelings mirror most of the population: I simply want them to work and represent good value. And that’s the attitude most vehicle buyers bring with them. So CR’s evaluations might very well work for them. Just not for me. BTW, I have subscribed to CR three separate times over the last three decades. But I always end up allowing the subscription to expire. That should say something – about me, at least. :)

  • avatar

    I haven’t read Consumer Reports since I was a kid, which was over 30 years ago. My parents subscribed, and I loved poring over their tests because to me, it was applied science! I seem to recall that back then they actually published their methodologies, usually in sidebars with B&W photos of guys in lab coats and safety goggles. When did that stop?

    I agree that by NOT revealing their methods and measurement metrics, they have ZERO credibility. It reeks of bad science where answers come before questions, so that results fall in line with theory.


  • avatar

    If CR doesn’t publish its criteria & formulae, a well-informed auto buyer should dismiss it.

    Admittedly, I have an anti-domestic bias (due to reliability, fit, and refinement issues), but I remember reading a comparo between the Corvette Z06 and other Euro cars awhaile back in Car & Driver (or R&T?). The Z06 appears to win in every category, except for something the mag calls “Got to have it” factor. So instead of being at the top of the list, it was relegated to the back. Mind you, some of its competitors cost 3X as much as it. When one knows what the mag bases its “subjective” tests on, one can properly judge (or dismiss) the results. The same attitude should be taken to CR.

    Does a minivan buyer care about how fast it can go around a corner? Does a roadster buyer care much about whether the car has 4 or 8 cupholders? Does a Porsche 911 buyer care about the shitty, low-rent, subpar interior of the $80K+ sportster? NO.

    When I bought my M3, I ignored all the “cons” such as back-breaking suspension, loud cabin noise, low seating position, last-generation GPS, etc. Except for the lousy, outdated navigation system, most prospective buyers would consider the other traits positive.

    If CR refuses to disclose its formulae, then consumers should decline to part with their $. For what it’s worth, I only enjoy reading the back-page “Sellin’ it” section in CR.

  • avatar

    My guess is that CR does not want to reveal its testing secrets because they run the risk that others will copy them and they will lose their perch at the top of product analysis.

    Now perhaps the larger question is this – Is there any measurable evidence that CR’s (or indeed any car review organization) reports actually influence consumer behavior? Yes the Japanese are currently gaining market share and incrasing sales, but is this attributable in any way to the fact that their vehicles are highly endorsed by CR?

    Some of the worst reviewed vehicles have gone on to become sales successes, as you know.

    As MK pointed out, some factors that CR may value high or low may indeed not mesh with a consumer’s preferences. And how can you place objective, formula based, statistical value on some aspects of a vehicle including looks, ride, ergonomics etc. When it is the buyer that must ultimately determine the value of these aspects for himself.

    Is CR biased against the dom’s? Is there a determined conspiracy afoot? Hard to tell.

  • avatar

    To Sherman Lin

    The reason all the warranty statistics given by the domestics are irrelevant is because these days a significant number of people have to finance the ir cars for longer then the warranty period, ie 4 or 5 years.
    In my experience the domestic cars may be OK for a while, but people seem to incur enormous expenses very soon after going out of warranty. They do have planned obsolescence down to a fine art.

  • avatar

    One other point not brought up so far: Politically, CR definitely has always had a liberal bias, treating automobiles as absolutely unfortunate necessities that should be driven as little as possible – and do you really want to get car advice from a source that barely tolerates cars, considering them detrimental to the environment, etc.?

    Given that CR supposedly tests anything, when was the last time you read a motorcycle test? Answer: Never. When asked about it in a letters section back in the early 80’s, the answer (paraphrased) was, “We feel motorcycles are too dangerous and nobody should be riding them, so we’re not going to test them lest we give someone the idea that they should buy one.”

    How any pistonhead can read CR regarding car tests for anything other than a good laugh is completely beyond me. Can you imagine the results if they had ever tested a Lotus?

  • avatar

    Can you imagine the results if they had ever tested a Lotus?

    Yeah, they trashed it but not as badly as the Solstice.

  • avatar

    To Cowbell

    As you pointed out it was “predicted long term reliability”. Probably based on past reliability from that company. Thus Honda #1 Toyota #2 and the rest of the pack somewhere behind. As someone who keeps cars for a minimum of 10 years long term reliablility is very important to me. With the decline of German quality the only place left seems to be Asian brands with the exception of US brands if you can purchase a bulletproof 7/10 warrantee.


  • avatar
    Joe O

    I went to the CR website and I’d like to post some thoughts on their “top 10” list:

    1. Mazda Miata for “Fun to Drive” – I can understand this; it’s pretty well in the middle of this pack. However, at the end of their paragraph on why this car is their pick, they suggest the WRX (sedan or wagon) for fun to drive and practical. I don’t understand that pick anymore. As a previous owner of a WRX (I owned a Saab 9-2x Aero), it is no longer a class leader in many segments. It’s interior is spartan and not in a good way; it’s loud, unrefined in both the sense of noise, vibration, and engine and transmission mating.

    It is approx. 25k and for that money you can get some very nice options nowadays, such as the MazdaSpeed 3 (22-26k with a fair amount of options, Hatcback utility, alot more performance). Even the VW GTI, though past reliability does not indicate good things for the future.

    Again, there’s also a Honda Civic SI Sedan, etc. etc.

    Again, why was the WRX recommended (I’m not knocking on it, I just think there are definitely better suggestions for “fun to drive”)

    So I’ll move on in my thoughts on the top 10 list.

    Family Sedan was listed as Honda Accord, with seperate suggestions going to the VW Passat 3.6 and Camry. Again, the VW? Since this is a new model, they can only project reliability off of older data…which doesn’t bode well.

    Why not the Saturn Aura? I’m no fan of the domestics overall, but the Saturn Aura is a solid piece for a respectable “family sedan” price. The new 3.6 liter engine and 6-speed automatic available are quite nice.

    Anyway…just a few examples. I’d like to see why they don’t recommend these cars?


  • avatar

    I sometimes read CR, but not always.

    And when I do, I often don’t buy the item that came out of the review as “best.”

    So their test criteria being secret hasn’t affected me in my purchase decisions.

    I do think they should disclose their test criteria. But people should be careful what they ask for…if disclosed, the test criteria might actually turn out to be fair. Fancy that. And it would be yet another embarrassement for the domestics.

    And maybe for the car magazines, too.

  • avatar

    So CR will give a better score based on the brand rather than the individual merits of the car? Even if the Fusion is predicted to have better long-term reliability than the Camry or Accord it won’t be given a reliability recommendation because it’s a Ford and not a Toyota/Honda?

  • avatar

    Any attempt to wrangle something as esoteric as buyer preferences into a wooden formula is doomed to fail, if not at being popular then at actually providing any real information. There is no quantitative metric for qualitative variables, and I can’t help thinking that somebody who makes as large a purchase as a car based on somebody arbitrarily assigning it a 19 out of 20 is a fool. I think that rational free agents in the market will at least read entire reviews rather than skipping to star ratings at the end, and I would like to think that moniker applies to at least a great majority.

  • avatar
    Glenn A.

    March 6th, 2007 at 10:09 am
    Of course bias,as is most of the media{TTAC accepted}
    The media bias is responsible for the quality perception gap that exists today.
    The Toyota Matrix and the Pontiac Vibe are identical cars.Made by the same people at the same time in the same plant.
    Guess what one gets the higher rating.”

    Beg to differ, Mikey, but the Toyta Matrix is built in Ontario exclusively, and the Pontiac Vibe is built in California exclusively. They do “resemble” one another and are clearly based upon the same basic platform and even share window glass shapes and many components.

    But they are not “clones” coming down the same line at all. GM clearly had the opportunity to have input into the quality of product that comes out of the California plant badged as Pontiac, for better or worse.

    My best friend has a used Chevrolet Prizm which I advised would be a good used car purchase because it is “essentially” a California built car based upon the Corolla; so far it’s been almost flawless, and the resale value was significantly lower than a “real” Corolla. For my friend on a strict budget, it has been a win-win situation.

    I suggested the possibility of doing the same thing and obtaining a Pontiac Matrix next time, but I’m having second thoughts.

    Another possibility, if there is a “gap” between how the two cars stack up as they age, is that “GMers” buying cars may not take as much care of their vehicles as do Toyota purchasers? I dunno…..

  • avatar

    Who cares?

    Every car I’ve ever owned has received the Black Circles of Death in their frequency of repair charts.

  • avatar

    I agree with others that this seems to be the only source for reliability issues. They cover resell better as well.

    I apply much skepticism to the rest of what they say.

    Perhaps if the buff mags and internet players would take the bottom 10 percent manufacturers to task by ensuring that that they recommend AGAINST buying a vehicle if it’s recent brethren became bad ownership experiences, we could all dump CR.

  • avatar

    Joe O,

    The 2006 and newer WRX are quite different from the older ones – far nicer cabin, way better shifter and transmission, the bigger 2.5l turbo engine with way less turbo lag and more torque. Maybe that’s part of why they are still so high on the fun to drive rating. There’s also none of the torque steer of the Mazda zoom zoom FWD cars and very good reliability.

    I sure wish there were other options in the $25K range that were as practical and as fun as a WRX wagon, and not plagued by FWD.

  • avatar

    What? Seriously? Is this for real guys?

    CR’s system for ranking cars on a numeric scale and calling them “Top Picks” may be pretty pointless and stupid, but in general so is the idea of ranking cars at all.

    But this really isn’t a big deal AT ALL as they disclose what they thought of each individual aspect of the car. So as the article says, if a car buyer valued handling over ride quality they could simply see what CR thought of handling on that individual model, they wouldn’t have to follow CR’s top picks blindly.

    Give some credit to the consumer, they’re not nearly as stupid as that.

  • avatar

    “The Toyota Matrix and the Pontiac Vibe are identical cars.Made by the same people at the same time in the same plant. Guess what one gets the higher rating.”

    I’m looking at the ratings in the April 2007 Consumer Reports…Vibe gets a slightly higher reliability rating than the Matrix. Both are recommended.


  • avatar
    Mike P

    I see this Karesh post the same msgs everywhere… Not sure I see the beef. CR is biased toward safe, reliable cars that perform well in their battery of tests. The data is presented to be used by consumers and enthusiasts alike. That they don’t explain how their Excel sheets calculate every dot is a non issue. How many businesses in general go to that level of detail? Sure, it could be a competitve concern, but more likely, it is just info overkill. CR is a reference tool, not a source for a doctoral thesis. The beauty is they conduct more research and testing than ANY organization, then distill it down into easy-to-understand words and symbols. Is there any dispute with their findings? Does Koresh have more than a sample size of 1.3 million vehicles and decades experience analyzing the data? Nah, didn’t think so. They do a good job and let’s leave it at that. Would be far more interesting to discuss the latest findings. What’s next, asking how car mags conduct 60-0 brake tests as a requirement to believe anything they say?

  • avatar

    They do use one ‘test’ that I find invaluable. The would you by it again, yes or no, with the autos ranked in order of percent yes. Knowing the cars, you can guess as to what the owner was likely looking for. If your expectations are similar to that, you can get a feel for how satisfactory the car is.

  • avatar

    So you’re saying I should cancel my order for a Bosch dishwasher and order the Kenmore?

    While I’d like CR to reveal the weighting, and find it silly (or suspicious, even) that they won’t, I could care less. Like most of the others here, I read it, along with dozens of other sources (epinions, TTAC, Edmunds,, forums specific to the makes, every major car rag, etc.), research resale values and leasing factors, find people who own it, drive it myself, and then make my decision.

    Sometimes, with cars, we don’t always do what’s ultimately smart, because the first date (ummm, I mean first drive) is so damn compelling. But I won’t blame CR…or anyone but me.

    As a former police officer, I always listened to everyone’s “version” of the truth…

  • avatar

    Uh, this is the same Karesh that posts reviews at epinions based on dealer tests drives, and he’s being critical? In fairness, he does disclose that…

  • avatar

    “The Detroit News reported this weekend that GM’s response to the CR news was that their warranty claims are now 40 percent lower than from 2002.”

    Hmm..are their sales down 40 percent? Again, without knowing the facts behind the numbers, this is really meaningless.


  • avatar

    Claims being down could be for multiple reasons, including ability for customers to make claims. Doesn’t mean people aren’t having issues, especially on older models outside of warranty. Not surprising some manufacturers would be on the defensive. In the end, CR doesn’t answer to advertisers… just their subscribers.

  • avatar

    CR provides a good directional tool—as others have said—there are many other sources of information (like TTAC)for the consumer to use and draw conclusions.

    That said, I have 2 issues with CR:

    1) While they have 3M + in their survey—it is a self selected sample—members of the same country club if you will. Like a country club—there is very little turn in their subscriber base so it is not surprising that their results are very consistent year-on-year. Consistency is a good thing—but it does not make up for the fact that their survey is not scientific from a pure market research perspective.

    2) Toyota clearly has had quality issues over the past few 5 years—when will this show up in the CR data ? When will the 3.5 million people who had engine sludge problems own up to the fact that their car was not perfect when they fill out the CR survey ? In 2006 Toyota recalled more cars in the US than they sold—when will this show up in the CR data? Not saying that CR is biased—but their results are flawed if they do not show what is happening in the real world against ALL car brands !!

  • avatar

    The reality is the Japanese make the best vehicles out there right now. You can go and blame CR for giving their top picks to all Japanese models..actually hold that thought, no you can’t. In the small car category would you give the award to the Chevy Cobalt? Would you give the midsize sedan award to the Chevy Malibu? The midsize SUV award to the Ford Explorer? Are you kidding? The Japanese win the most spots because they make the highest quality cars, CR isn’t about the enthusiast, they are about the average car buyer. One choice I do not agree with is the Infiniti G35 as top Sports Sedan pick, it would definitely be the 335i, but of course, CR not being an enthusiasts publication, all they’ve tested is an ’06 325i, about as far from the ’07 335i as is possible.

  • avatar
    Larry Shearer

    Its obvious CR maintains a corporate culture that does not allow them to see beyond Japenese cars. Nobody on staff can dare suggest the consideration or recommendation of anything outside of Japenese makes for fear of being ostracized. Thing is, they’ve been that way for more than just 2 years, and their refusal to disclose criteria for comparison is just the nail in their credibility coffin.

  • avatar

    Without knowing any better, I am guessing that CR uses an assessment tool called Analytical Hierarchy. It is simply a process to assign weighting factors to an array of selection criteria. First, you rate each car for each criterion — 8 out of 10 points for visibility in the Chevy Cobalt, or whatever. Fair enough. Next, you must decide how important visibility, or braking, or reliability, is to you. This is where CR fails the transparency test. We have no idea what weighting factors the use for each assessment criterion.

    It seems they could run Analytical Hierarchy software on their web site, so their subscribers could enter their own weighting factors for the ratings criteria, and then compute rankings tailor-made to their own needs and requirements. But then, that would erode CR’s position as the ultimate arbiter of relative automotive value, wouldn’t it?

  • avatar

    I don’t buy the argument that CR has a pro-Japanese bias. I also take their ratings with a grain of salt, but find there full test reports more useful. For years the Passat was their top rated sedan and was recommended, but poor reliability scores did it in as well as the fact that Honda responded by redesigning the Accord with Passat-like handling as the target.

    Anyone who honestly things a Cobalt or Focus should outscore a Civic is the person who needs to defend their methodology.

    No, I’m not about to buy a car simply because it got high CR marks, but I will also say that the chance of picking a bad car from their list is pretty low!

  • avatar

    It’s been a busy day. I’ll do my best to belatedly respond to comments.

    On CR as a great source of reliability info:

    Even though this editorial is about CR’s road test ratings, unlike many who commented I think they do a much better job with the vehicle reviews than they do with the reliability research. I’m not saying the road tests are flawed, just that they’re withholding critical information. The reliability research is actually deeply flawed.

    For those of you who assume CR is the only game in town with the reliability research–that’s changing. I’m conducting my own reliability research, and my methods are far better than theirs. Sorry, Mr. Westbound, I’m way beyond just nitpicking and carping.

    On “bias”:

    I actually told RF I wouldn’t use that word in a CR piece. Yet it popped up in the final edit. But the way it’s used doesn’t charge CR with bias, so I’m fine with it. Like some who have commented, I don’t see CR’s problem as bias. Rather, it’s that they don’t make their tastes and formulas, the work behind the ratings, nearly as clear as they could.

    Putting it another way in response for ZoomZoom: I don’t expect their formulas to be any more unfair than any other formula. The “feels like a Honda” bit was just humor, nothing more. It’s possible they like how Honda’s feel, but everyone has preferences in this regard, and if theirs is toward the light, precise, delicate feel of a Honda, so be it.

    On the formulas as proprietary (Wm Montgomery):

    Which competitors would want to copy their formulas, and why? These things aren’t rocket science. And, unlike Google’s algorithm, they are essential to interpreting the results. Sure, it might be helpful to know Google’s algorithm if you thought they were ranking web pages in terms of quality, but they aren’t.

    Put another way, if CR divulged that ride quality does get twice as much weight as handling in calculating overall scores, why would anyone else then do the same thing? The formula doesn’t give them any particular advantage.

    On gaming the system (again):

    I already said a bit about this. But if CR believes that ride quality is twice as important as handling, and then automakers sought to game the system by prioritizing ride quality over handling, wouldn’t this be exactly what CR would want them to do?

    It’s very hard to learn from a test if you don’t know how the score is calculated.

    On domestic reliability:

    The gap has narrowed. But the fact remains that the domestics are behind, and in percentage terms the gap remains fairly large. In my own research fairly new Toyotas and Hondas often require about 0.3 successful repair trips a year, while GM and Chrysler products tend to congregate around 1.0. Ford has some cars in both camps. If my results are any indication, and the samples are small (unlike CR, I disclose them), the Five Hundred and Freestyle improved tremendously from 2005 to 2006.

    On Passat and Aura reliability (Joe O):

    The Passat has been out a year longer than the Aura, so they already have data on it. As do I. In both cases, the Passat is doing better than I’d expect a first-year VW to do.

    I’ll have results for the Aura in May. CR will have them in November. On average, my data will be over ten months “fresher” than theirs. Welcome to one way in which they are no longer “the gold standard” (as Mr. Westbound put it).


    One thing I’m learning from operating my site is that the great majority of people don’t care to dig into the details. They look at the “best picks” and that’s that. This is so much the case that I’m constantly under pressure to simplify my site. A big decision I’ve got to make is whether to design it for a relatively small population of people who love to dig through the details, or the masses that just get confused by them. CR consistently hides the details, perhaps largely for this reason. I’m not saying the consumer is stupid. It’s just as others have noted, people care to spend their time and energy in different ways, and some want to spend as little time as possible figuring out which car to buy. Which makes absolutely no sense to me–I test drive scads of cars just for funsies–but that’s just me.

    Mike P:

    The reason I have to keep posting my critiques is that they apparently haven’t gotten through to people like you just yet. Detroit had decades of experience building cars. That didn’t stop a small company like Honda for severely out-innovating them. I think you’ll find the same if you truly compare CR’s methods, which haven’t changed much in decades, with mine. Sometimes when an organization has been doing something for a long time, it’s not experienced, it’s just old.


    Those are my reviews, and as you say I disclose my methods. If you can find a concrete instance in which my evaluation of a car suffers from my limited exposure to the car, let me know. I must admit that one reason I don’t have much problem with CR’s reviews is that they tend to agree with my own. I take a test drive. They buy a car and subject it to dozens of controlled tests. We end up with similar conclusions, except my reviews provide more detail. Pretty funny when you think about it.

    For most cars I can learn 90 percent of what I need to learn in ten minutes or less. It wasn’t always this way, I’ll grant you, but after the first few hundred test drives you start to develop a finely tuned sense of how cars feel.


    CR doesn’t include recalls in its reliability results, and neither do I. I don’t know their rationale. Mine is that it’s generally a good thing when a manufacturer agrees to perform a recall, so I wouldn’t want to give them another reason to avoid taking this step.


    In the last day or so they have responded to the question raised in this editorial by stating that they will start offering a tool like you describe, though only as part of an expensive extra-cost package. Even then they won’t divulge how they add the subscores up. You’ll just be able to supply your own weights.

  • avatar

    I don’t pay attention to CR. I also don’t pay attention to Car and Driver, Motor Trend, and numerous other car magazines. My confidence in the economy doesn’t change everytime a new BusinessWeek comes in. I don’t understand why this is such a big deal to begin with… it’s still just opinion based on subjective facts.

    Their reliability testing…that is a different story (to a certain extent).

    Someone also brought up that they didn’t recommend a new Escort but endorsed a used one. That could make sense, especially if a big hit of depreciation up front leaves buyers with a reliable car. I would assume that is the reason, as such a car could still be a good value proposition.

  • avatar

    CR’s reliability ratings are probably a good guide to use, but from my perspective there is a slight bias towards Japanese cars. Back in the 80’s and 90’s I read the magazine every month, and would pour over the year-end car issues.

    Scores for a couple of the models they reported on were a bit confusing. The Ford Ranger always got more “average” and “below average” circles in the reliability graphs than did the Mazda B-Series truck. Same for the Toyota Corolla compared to the Chevrolet Prism. Now it wasn’t that big of a difference, but there was a difference. Odd…… these both came off the same assembly line. ( I know this has been mentioned before, but these are apples to apples comparisons ).

    This has to be due to perception, or possibly due to a smaller degree, the customer’s denial of the problems that really exist in the car they purchased. For example — the Japanese customer/CR reader may not consider an identical defect or problem in their car as serious as the American customer. And why shouldn’t they? CR has already told them that there car is more reliable, so it must be true.

    There also may be a bit of shame at play here. The Corolla customer is going to be a bit more reluctant to report a “serious” problem about their car, IF one of the deciding factors in their purchase was the fact that the car was supposed to be more reliable.

    It’s hard to be be truthful when reporting about any problems if that was the basis for buying the car in the first place. And this is CR’s demographic…………..

  • avatar

    Consumer Reports:

    GM: “I dont like how you rated our car”
    CR: “F-Off”


    GM: “I dont like how you rated our car”
    RT: “Ohh…um…Yes SIR…um…Ill fix that immediately…SIR”
    GM: “Now lets talk about that 4-page color glossy ad we are considering……”

  • avatar

    Plenty of people complain about Consumer Reports, but I have yet to see anyone put together anything close to as comprehensive of a testing program and customer data gathering effort. If you want to find methodology and disclosure to complain about, how about digging into the JD Powers Marketing BS Machine!

  • avatar

    And the top pick for pick up truck goes to…(drum roll)…The Honda Ridgeline!

    Is CR joking? There is all the proof anyone needs to see that there is bias in CR’s rating system.

  • avatar

    the sales trends are a better barometer than any magazine ex dealer

  • avatar

    They believe in man-made global warming, and it shows in there reviews.
    I used to subscribe.
    My wife has an XC-90. We got it cheaper oversees. (OSD program)
    Anyway. Last time I looked, every Volvo but the XC-90 was “recomended”. I dont know why a car that is heavier and safter would not be recomnded. But CR complained about the poor gas milage.

    (sorry for the rambling)

  • avatar

    Michael – Thanks for the response. To compare your reviews against CR, or other outlets, would take more homework than I care to commit to. On the surface, it is hard to believe that a short drive can provide the same depth as a long-term evaluation conducted by a team of experts, in CR’s case, automotive engineers. Your reports are longer than the reg road tests in the magazine, but no where near the depth of the expert report in their pricey buying kit.

    @ Luther – Very funny! And, true.

    @ Joe – Looking at the mag, the XC90 is probably not recommended due to reliability. It is the lowest-scoring Volvo in reliability.

  • avatar

    Michael Karesh:

    My point was simply that for the inquisitive CR reader, pretty much all the information you need is given to you which makes the overall-score/top picks pretty irrelevant.

    They do have excellent car reviews in my opinion however.

    Btw, you’ve got a great site, it’s a truly useful objective tool in measuring cars on a level plane.
    If there’s something I wish it had though, is more rebate information during the pricing comparison.

    Anytime I want to see if a car has a rebate, I use this site:

    But they don’t list if the rebates are available for MY region. Which makes the list sort of useless. I was hoping your website would have more information on that for a truer price comparison. Still, it’s great stuff though.

  • avatar

    Consumer Reports eats its own vomit.

    Mark, you touch on this, but I will expand that thought.
    1. They report how good certain vehicles are.
    2. Their subscribers (after reading the article)are biased to purchase and enjoy those vehicles.
    3. Their subscribers fill out the surveys about their new vehicles and rate them similarly to the way they were rated in the magazine.
    4. The magazine repeats to 1 again.

    You can see the obvious issue of using a self-selection process that guarantees that respondents highly value (indeed, they PAY for) information that is directly related to the item included on the survey.
    Could you pick a more biased sample (besides manufacturer employees) if you wanted to?

    CR could use a random sample and would deal with the high expense of doing so. Or perhaps they are concerned with how different the results could be when compared to the present methodology?

  • avatar
    Mike P

    The scale they have must limit car-owner bias. Already the program is very expensive. (Hope they buy a lot of the “forever” stamps to lock in prices.) I would be more concerned about the biases at True Delta where participants are actively recruited from enthusiast sites, like this one, where there is either an axe to grind or greater product loyalty than in the general public.

  • avatar

    Yes there are problems of self-selection bias with the CR methodology, yet NOBODY has put out a better program. The auto makers individually have all the data we would want, but none of them release it. Perhaps the gov’t could require publication of dollars spent on warranty claims per model in a way similar to the requirement that on time arrival rates are reported by flight number for airlines.

  • avatar
    Steve S

    CR is a tool, and a still useful one, but I generally do not use CR as the sole tool to determine my auto purchase. They have a bias, as do all car review sites and mags; in their case they are biased towards unexciting, practical cars in general. They are biased in much the same fashion that, say, Car & Driver is biased towards handling and accelleration uber alles in their tests.

    CR loves Japanese cars, but to say they have a bias against domestics is exaggeration. I bought an Olds Intrigue 5 years ago in part because both CR and Car & Driver raved about it. I found CR’s statement of “average” reliabilty for the Intrigue to be pretty accurate in my ownership of the car.

  • avatar

    I think there's a good analogy between truedelta/Consumer Reports and imports/domestics in the 70s. As many people have pointed out, CR is the only game in town right now, as the domestics were back in the day. And as is often pointed out often on this site, there were some serious flaws in the domestic cars back in the 70s, just as Michael is saying about CR. So truedelta is born because Mr. Karesh thinks there's a better way to do report the reliability of cars, just Honda and Toyota believed there was a better way to build cars.

  • avatar

    On the other hand, releasing the formulas to auto makers would lead to gaming of the system. This also shows how silly these formulas are. If it comes out that consumer reports gives equal weight to scores for “visibility” and “handling,” then car designers have a simple choice… spend millions improving suspension design, or spend thousands making the windows bigger. Designing cars to CR specs means everyone wins but the consumer.

    You really think Consumer Reports has THAT much juice? Aside from marketing meetings targeting the latte-sipping-and-NPR-listening-all-the-time demographic, no manufacture gives too much weight to Consumer Reports. Well, maybe Toyota…
    If a formula contains SUBJECTIVE factors like ride, handling, and aesthetics, it’d be nice to know how different vehicles stack up. (And how brutal inconsistency may reflect on the reviewers’ integrity).

    Years ago, “Best selling” book lists from Real Important Newspapers used ‘select’ bookstores to compile their data. Then along came Amazon and rude questions were asked about data…

    Good luck, TrueDelta.

  • avatar

    John: The auto makers individually have all the data we would want, but none of them release it.

    If you forced them to release this data (which I agree, is the best source for car quality info) then the Service Managers will be told to bury claims…Like they are doing now. Even Lexus buries repair work…NO doubt.

    I agree that customer feed-back loops are not 100% reliable/accurate for the simple truism that People Lie. A lot of people base their car purchases solely on pretty designs (still). The car, to them, is a “member of the family” (similar to the people who consider a dog or cat a “human” member of thier family) and not an inanimate tool for transportation. When they encounter problems with the car their ego gets in the way of objective reality. It is like a mother with a newborn child, that has arms protruding from its head and eyes in its chest, will always proclaim “isnt my baby the most perfect baby in the world”. Their are some that will say “that review was totally biased…How can they pick A over B as being a better car”. To those people the “better” car is the subjectively “prettier” one regardless of how reliable it is. Reliablity feed-back loops will continue to evolve toward 100% accuracy though simply because corporate deceit cannot survive for long in a world with worldwide, instantaneous communications (Internet). Decentralized communications will keep/make them honest…Or they will die. But until the feed-back loops are perfected, FOLLOW THE MONEY !!!

  • avatar

    For those looking for more insight into CR and its testing, Autoblog’s latest podcast goes inside the test facility. Interesting stuff…

  • avatar

    This is from an article on Allpar that is in line with your thinking:
    “the CR survey may over/understate the reliability of certain cars because the people that own them are not homogeneous. … many people will have a subconscious need to justify their purchase of a Japanese auto over of a domestic one, and they could do this by believing superior reliability is the reason they bought it. Because of cognitive dissonance, they would tend to overlook or downplay anything that would attack this mind-set. We do see many people who vehemently defend Japan’s cars’ reliability and smear that of others.”

    Mike P:
    March 7th, 2007 at 11:21 am
    The scale they have must limit car-owner bias.

    From the same article:
    A high response rate is the key to validity. Employee survey findings can be questioned when fewer than half of the employees respond. So how many people respond to a Consumer Reports survey? “Of over 4 million questionnaires sent this year, the magazine received responses regarding about 480,000 vehicles,” according to Detroit News. Now, if most people reported on two cars (because most families have two or more cars), that would put the response rate at a mere 6%. Even assuming one car per family – a highly dubious assumption – we have a taudry 12% response rate.

  • avatar

    Let’s not be cynical. CR is not perfect, just like Microsoft or the US government. But they are the best we have got so far. They certainly beat their competitions. In your own interest, you may trust them to a certain extent.

    Just because they refuse to release their method, CR’s credibility is not automatically zero. The Federal Reserve no more releases monetary supply data, does that imply the entire American economy has zero credibility?

  • avatar

    Windswords – I think you’re right about the scale. However, that allpar site information is years old. Looking at, there is a section on Reliability and a loooong FAQ.

    In all, the survey was sent to almost 6.7 million subscribers in 2006, and we received responses on about 1.3 million vehicles.

  • avatar

    Mike P, I agree the Allpar is old. but I thought the percentages would remain stable up to the present time. It does seem that the response rate has gotten better. However, the problem remains that it is still a “closed system”, open only to the subscribers.

  • avatar
    Mike P

    The “closed system” does carry certain benefits, though. The CR audience is presumably older and better educated than the general public, and certainly more than enthusiast forum particpants (no slight intended, just demographic reality). That actually may be a good thing for a measured, detailed response. Completely open, online data collection runs the risk of being diluted by the extremes. Those with the most passion, and problems, will be drawn in. While in balance they could average out, it is the enthusiasts that can better drive traffic. Nothing in life is perfect, though it still seems CR is doing a good job and providing information that would take many millions of dollars to outshine.

  • avatar

    Anyone who thinks you need a high response rate to get good data hasn’t studied sampling theory or statistics very well at all.

    Also, for a large part of the new car buying US population, CR’s reports are highly influential. They certainly have more influence than the enthusiast magazines. Speaking of which, if you correlate the actual reports from the traditional magazines with CR’s reports you will find that they are in agreement more often than not. The enthusiast magazines put ZERO weighting on crash test results or reliability history when they do their Comparison Tests, other than that, the data and write-ups generally have more in common than not. Both CR and Car and Driver found the Saturn Ion wanting. Bias? Not.

  • avatar

    I don’t know if this has been mentioned before, but, CR doesn’t publish the criteria for their tests on ANYTHING they test because they don’t want manufacturers to build to that test criteria (in the same way car manufacturers build their cars to the CAFE standards and have, as a result, made that test inaccurate). It’s a means to prompting the manufacurers to build the best possible product across all perameters.

Back to TopLeave a Reply

You must be logged in to post a comment.

Recent Comments

  • BunkerMan: It still looks like a sneaker from the side. I’d take the Mazda 3 over this.
  • Speed3: GM needs to give it up and acknowledge that it doesn’t not have the resources or talent to right this...
  • Noble713: Black metal and death metal. A 120GB iPod classic is my primary source of car music. It’s full with...
  • IHateCars: We’re on our second as my wife’s car, she initially had an ’07 FX35 which she loved...
  • RHD: Phase out.

New Car Research

Get a Free Dealer Quote


  • Contributors

  • Matthew Guy, Canada
  • Ronnie Schreiber, United States
  • Bozi Tatarevic, United States
  • Chris Tonn, United States
  • Corey Lewis, United States
  • Mark Baruth, United States
  • Moderators

  • Adam Tonge, United States
  • Corey Lewis, United States