Being in the survey business myself, there are few things I focus more on than how many surveys come in how soon. I can tell you that the current round is tracking almost exactly with the previous round. Maybe when you have 1.4m of them, though, you no longer care how many actually get returned, or when.

I don’t currently evaluate extended warranties. If I did, then having Warranty Direct ads would pose a conflict. Currently I don’t personally evaluate anything on the site itselft–the content is user-generated. My personal material is posted here and elsewhere.

Mr. Champion didn’t know what month most of the surveys were returned – big deal. Maybe the more appropriate question would have been “How old, on average, are the returned survey results?” Even then, we all know that this is pen/paper mail stuff here. Yes, is going to be a delay. I would guess that there are those who return the surveys the next day and those who get around to it several months later. Either way, they probably average out to some number. So he didn’t have the specific month. Did that mean he couldn’t get that info? Probably not. You say that their head of publicity was combative. Maybe we should also hear their side of the story.

Your second question about dot-to-problem rate correspondence is a good one. I would like to see their answer to that one. Perhaps your questions could be communicated via e-mail to them and their answers posted here.

I’m not going to be a blanket apologist for CR here. Politics and “stick to the goddamn toaster testing” comments aside, they do a commendable job in bringing information to me and millions of others who would otherwise be in the dark. Their work on safety testing in multiple areas (auto safety, lead, radon, et al) is enough in my book to have a permanent place on my reference shelf. They are a tireless advocate in these areas. They have a tremendous amount of respect. That same respect would not be there if they had advertising of any sort. And claiming on a website that “our opinions are in no way influenced by those ads floating next to that shit you’re trying to read” doesn’t cut it and everyone knows it.

Yes, you guys are competitors in a limited sense. I really like the TrueDelta concept and website. I’d like to see a lot more problem detail. And maybe some info as to what doesn’t match up to CR and others. I wish it could happen without ads though. For example, having a Warranty Direct ad on your page removes your ability to objectively discuss extended warranties – a very relevant subject in your realm.

I have always laughed when I read some of the forum crap from people dismissing CR auto testing. They do a pretty damn thorough job from what I’ve seen over the years. And they have their own track, buy their own vehicles, etc. Their summaries of most models are usually spot-on. They get hit, at least perceptually, for their board-stiff engineers assessment of things like sports cars. They usually hit the nail on the head though. I’ve always found it odd that the major rags have never given a shit about listing reliability for vehicles – aside from a handful of “long-term tests”. And these are cars that are usually “provided” to them, not purchased like CR’s policy. And safety results. How many rags have a damn bit of useful info on safety? Does C&D or Automobile give a mention to how safe this gleaming sled is? Usually not anywhere to be found. Guess they figure it’s a given that we’ll get it elsewhere.

Overall, I’d recommend treading lightly on the CR criticism until you provide us with their official response so that we can evaluate in black and white.

A very important point. Non-profits still want to be prosperous! As organizational theorists recognize, organizations seek to grow and prosper. How to be prosperous? Well, generate a revenue “surplus;” that makes new facilities, pay raises, etc., possible. And how can CR generate more revenue? One way is to persuade people that it is important for them to have CR’s guidance when they venture into the dark, scary world of the marketplace. The “colored dots” ranking system essentially dramatizes differences in reliability experience. Remember as a student how disappointed we were if we got 79% on a test? It’d be a mediocre “C,” not the respectable “B” grade from scoring one percentage point higher.

So absolute repair frequencies are important. As would be dollars expended. Moreover, I’d like to know CR’s weighting method. For example, a power window that won’t go down isn’t nearly as serious as a transmission that blows up, and not just in terms of cost.

The logical conclusion is that their refusal to accept advertising places them beyond reproach.

In fact, CR is chock full of ads–for all of the stuff CR sells. And, guess what? No one evaluates CR’s own products, some of which would not past muster in their own evaluations.

And this is the key to CR’s bias: at least as much as any other organization, CR is biased towards revenue growth. They present information in such a way that people will feel most in need of this information…

…which brings us to Dave Ruddell’s question. The only answer CR will give–shades of government bureaucracy here–is simply that they won’t release more than dots as a matter of “policy.” And no one in the media asks for actual numbers, much less presses for them.

Why not release the numbers? Partly out of habit. But also possibly because, if the numbers were released people might realize how small the differences actually are. And then they’d be less likely to subscribe.

It might also be because they lack faith that the absolute numbers are accurate, while trusting that all inaccuracies wash out in relative comparisons.

Three big reasons the numbers they won’t release might be highly inaccurate:

1. The question asks respondents to only report “problems that you considered serious.” This opens the door wide to under-reporting.

2. The data are old; in the summer, the numbers for “four-year-old cars” would actually be those of 2.5-year old cars.

3. The survey is conducted annually. I know from my own survey that people often forget repairs that occurred more than a few months earlier. I send the email for the survey monthly because I’m not sure even a quarterly survey wouldn’t stretch memories too far. Yearly? Forgedaboutit.

4. Combination of 1 & 3: problems that occurred a year ago are much less likely to seem serious than those that happened recently.

I was with you until you stated the above. Forgive me if I missed something, but I don’t see anyone making your stated argument. If I’m correct — and I invite you to prove me wrong — that suggests you’ve created a strawman.

Whether a media outlet relies upon advertising is hardly the sole factor in avoiding conflicts of interest (actual or perceived). However, it can be a major factor, particularly when some auto manufacturers have been known to bully publications that don’t follow script.

I’ve been reading auto buff magazines and CR for almost 40 years now and I’ve seen a fairly consistent pattern of the magazines sidestepping product quality issues that CR has dealt with head on. Many editors and reviewers have come and gone, yet the same general pattern tends to continue. I suspect that CR’s unusual funding structure has played an important role in protecting its journalistic independence.

I wish more media outlets would try that model. Does that mean CR is above criticism? Or that it is the only model worthy of emulation? Not at all.

I hope that you find great success in your important work. My guess is that there’s room in this world for both your approach and that of CR’s, particularly if the latter improves its transparency and responsiveness to feedback.

There’s no doubt that GM vehicles overall are far, far better than they used to be, but they still make major errors such as the intake manifold gasket leaks on all the V6 engines.

As time goes on, other problems still surface and the list is much longer than it should be. I’m waiting for the first turbo four cylinder engine from GM to go bang, but I think it will happen, since GM hasn’t a clue when it comes to small engine design.

I could get into Chryslers’ poor record, but since they won’t be around much longer, it’s no use beating a dead horse.

From what I’ve read, Ford is applying Mazda engineering practices and quality control techniques to all of its vehicle lines, not just the ones based on Mazda platforms.

Mr. Bailey,

Is is safe to say that Ford is the best of the domestics? And are GM and Chrysler products improving, too?

BUT it sounds like they just want to sell some of their equity and maintain their current ties.

That said, Mazda-based Fords aren’t the only ones doing well.

And refusing to accept it guarantees perfection, or at least means that all imperfections should be ignored?

Ridiculous.

And if one starts with a belief that CR is always right, one is likely to find that they’re always right.

I’ve got one thing I’d like someone to explain to me. Check out the most and least reliable Subarus here:

http://www.consumerreports.org/cro/cars/used-cars/reliability/best-worst-in-car-reliability-1005/how-makes-compare/0407_how-makes-compare.htm

The least reliable Subaru, with a repair rate about 15% worse than average: the Legacy Turbo.

The most reliable Subaru, with a repair rate about 45% better than average: the Outback Turbo.

That’s a large difference, almost equal to the spread from “worse than average” to “much better than average.”

Who ever knew that raising a car’s suspension and adding bodyside cladding could do so much to improve reliability?

See Legacies much more often than Outbacks in the shop? If you haven’t before, I bet you will now.

This is just one unexplained anomaly in their results. There are plenty of others.

But more than that, his methodology is transparent and he is open to criticism. He is always open to the possibility indeed the need for constant improvement and progress.

Consumer Reports has done– and continues to do– much good in the world. But their decision to shroud their data and “dumb down” the results brings them no honor.

Having said that, I have to agree with Mr. Bailey, as more than a few mechanics I’ve talked to have touted the accuracy of Consumer Reports survey results. So they must be on to something.

I’m also a little baffled at the barbs regarding the magazine’s alleged preference for wheeled appliances…the magazine also tests sports and sporty cars, and rates them on how much fun they are to drive. The days when it thought that a Dodge Dart sedan with a slant six and Torqueflite are all that anyone really needs are long gone.

Usually that charge is leveled at Consumer Reports by domestic fans miffed that their favorite brand’s poor showing on the test track and in the reliability surveys. As if a Pontiac G6 and Chevrolet Impala are more exciting than a Camry…

GS650G: CR dissed the Ford Escort when they were new,, when they turned out to be reliable cars they turned around and recommended them as used cars.

Are you referring to the Escort or the Focus? Because BOTH were initially unreliable, but Ford did work to correct the problems.

When we bought my wife’s 2005 Focus, I mentioned the improvement in reliability shown in Consumer Reports for the Focus after 2003. Even the Ford salesman said (in a low voice), “Yes, the first two years of that car had lots of problems; I’d avoid them, too.”

And note that the magazine has just said today that Ford continues to improve its reliability, and is gaining on the best of the Asians in this area.

There are at least a half dozen sites around the web that offer tens of thousands of reviews from folks who have actually owned the car for long periods of time. This is where you’ll find out whether certain components are truly aging well along with the vehicle overall. I actually encourage folks to visit those sites and related enthusiast sites if they want to learn more about the vehicles they purchase from me.

CR in particular has a rather nasty and stupid record of endorsing VW’s in the late 1990’s when the qualitative data in every site I know reflected the exact opposite of their findings. Guess which set of data won out in the end? They’re also a bit late when putting vehicles on the ‘avoid’ list and with generally understanding the actual corners cut by the automakers. Toyota’s decontenting over the last ten years in particular has resulted in a gradual loss of quality and durability vis-a-vis their past efforts. CR didn’t catch up to that fact until 2006.

CR isn’t trying to keep data away. They’re limited in the content they actually get and unfortunately, their narrow market demographic amongst their readers obscures their ability to look at and offer a more complete picture of the overall quality within the industry.

Detroit’s competitors started out small. Fans of Detroit initially wrote off the Japanese with “they’re only good with small cars.”

But what really gets me is when the people who claim to follow CR because “there’s no alternative” also do their best to prevent any alternative from developing. Many people–and I’m not saying you’re among them–seem to think CR should have no competition.

But a lack of competition has been at least as bad for CR as it ever was for Detroit.

They should stick to laundry detergent and can openers because they are simpler to test than a vehicle and are mostly US made.

Detroit faced a number of worthy competitors and LOST. CR still doesn’t really have any. Even with questions about their data and their lack of transparency, no one else provides data that is useful to average consumers on as large a number of cars and trucks as CR. Until someone else does, it’s CR.