Consumer Reports Announces Reliability Data; Not a TTAC Fan
After David Champion, head of Consumer Reports‘ auto testing, presented this year’s reliability results, I asked two simple questions. 1. What month were most surveys returned (i.e. how old are the data)? 2. What problem rates do the dots represent? Or, to keep it as simple as possible, what was the average problem rate for a 2008 car? Unfortunately, Mr. Champion did not know the answer to either question. He could only respond that the surveys went out in “the springtime,” and that the dots are relative. As if the actual problem rates they represent were of no consequence. In fact, both things matter. The truth about CR, as we’ve noted here: before: 1. The data are already about five months old, and will be 17 months old before they are updated again. 2. The differences between the dots for a 2008 model are about one problem for every thirty cars. But, since even the head of CR’s auto research doesn’t know these facts, it should come as no surprise that their millions of subscribers haven’t a clue. And then things got ugly…
Finally he brought up my comment on David Holzman’s piece about the future car conference CR hosted recently. My comment: “CR avoids corruption itself, but corrupts other media as much as possible.” Or something to that effect. My response: didn’t we just enjoy a nice meal, preceded by an open bar? Weren’t all of the questions other than mine easy? (Actually, mine should have been easy, to answer at least.) His only response: there were tough questions. But there weren’t. And I wouldn’t be surprised if we’re not invited back next year.
[CR paid for MK’s parking, lunch and the two ginger ales from the open bar. Fair disclosure: TTAC has a contract with MK’s TrueDelta for pricing and specification data]
Michael Karesh lives in West Bloomfield, Michigan, with his wife and three children. In 2003 he received a Ph.D. from the University of Chicago. While in Chicago he worked at the National Opinion Research Center, a leader in the field of survey research. For his doctoral thesis, he spent a year-and-a-half inside an automaker studying how and how well it understood consumers when developing new products. While pursuing the degree he taught consumer behavior and product development at Oakland University. Since 1999, he has contributed auto reviews to Epinions, where he is currently one of two people in charge of the autos section. Since earning the degree he has continued to care for his children (school, gymnastics, tae-kwan-do...) and write reviews for Epinions and, more recently, The Truth About Cars while developing TrueDelta, a vehicle reliability and price comparison site.
More by Michael Karesh
Comments
Join the conversation
I have been a CR subscriber for many years. I respect their independence from advertising. That doesn't make them perfect. No one claims that. I think they do a great job overall on a wide variety of product testing. They have also admitted when they screw up - car seat testing results from an outside contractor miscommunication comes to mind. They do seem to have integrity in their mission and in their testing methodologies. Mr. Champion didn't know what month most of the surveys were returned - big deal. Maybe the more appropriate question would have been "How old, on average, are the returned survey results?" Even then, we all know that this is pen/paper mail stuff here. Yes, is going to be a delay. I would guess that there are those who return the surveys the next day and those who get around to it several months later. Either way, they probably average out to some number. So he didn't have the specific month. Did that mean he couldn't get that info? Probably not. You say that their head of publicity was combative. Maybe we should also hear their side of the story. Your second question about dot-to-problem rate correspondence is a good one. I would like to see their answer to that one. Perhaps your questions could be communicated via e-mail to them and their answers posted here. I'm not going to be a blanket apologist for CR here. Politics and "stick to the goddamn toaster testing" comments aside, they do a commendable job in bringing information to me and millions of others who would otherwise be in the dark. Their work on safety testing in multiple areas (auto safety, lead, radon, et al) is enough in my book to have a permanent place on my reference shelf. They are a tireless advocate in these areas. They have a tremendous amount of respect. That same respect would not be there if they had advertising of any sort. And claiming on a website that "our opinions are in no way influenced by those ads floating next to that shit you're trying to read" doesn't cut it and everyone knows it. Yes, you guys are competitors in a limited sense. I really like the TrueDelta concept and website. I'd like to see a lot more problem detail. And maybe some info as to what doesn't match up to CR and others. I wish it could happen without ads though. For example, having a Warranty Direct ad on your page removes your ability to objectively discuss extended warranties - a very relevant subject in your realm. I have always laughed when I read some of the forum crap from people dismissing CR auto testing. They do a pretty damn thorough job from what I've seen over the years. And they have their own track, buy their own vehicles, etc. Their summaries of most models are usually spot-on. They get hit, at least perceptually, for their board-stiff engineers assessment of things like sports cars. They usually hit the nail on the head though. I've always found it odd that the major rags have never given a shit about listing reliability for vehicles - aside from a handful of "long-term tests". And these are cars that are usually "provided" to them, not purchased like CR's policy. And safety results. How many rags have a damn bit of useful info on safety? Does C&D or Automobile give a mention to how safe this gleaming sled is? Usually not anywhere to be found. Guess they figure it's a given that we'll get it elsewhere. Overall, I'd recommend treading lightly on the CR criticism until you provide us with their official response so that we can evaluate in black and white.
I actually thought that asking which month the surveys were returned--and 75% are online these days--was a less loaded question than "how old are the data," which would imply that the data are old. Maybe I was wrong in this, but I thought it was a very simple question. Being in the survey business myself, there are few things I focus more on than how many surveys come in how soon. I can tell you that the current round is tracking almost exactly with the previous round. Maybe when you have 1.4m of them, though, you no longer care how many actually get returned, or when. I don't currently evaluate extended warranties. If I did, then having Warranty Direct ads would pose a conflict. Currently I don't personally evaluate anything on the site itselft--the content is user-generated. My personal material is posted here and elsewhere.