El Lutzbo is At It Again

Karesh - No, I don't work for JD Power. In fact, a few months ago I was arguing with my coworkers against all the attention and money spent on looking through JD Power data. I didn't believe in their system was because I viewed it as flawed as you have described. But I changed my tune when I realized that there is no better tool or service out there that can be a positive benefit the way they've set up their "racket." Their methods are hardly perfect, but as it stands, they are the only consistent source of objective comparison data for the automakers. In my previous example with 4 alternatives, there is a whole bunch of waste and idiot decisions because of preconceived notions that JD Power is flawed. So instead of taking advantage of some extremely useful data, people write it off because they "know better" or simply hold the belief that a flawed system can have absolutely no benefit. Obviously my numbers above are made up for the sake of discussion; but let's continue that point. You would either have a group of people saying "JD Power data is stupid why should I worry about it?" You would also have a group of people looking to try to get incidents of tire wear down to zero. And a very few (but most often overruled) group would actually attempt to identify how to use their limited resources in order to pursue the items that they could improve the most. Let's say that that your 10 problems per hundred was well below the segment average for tire wear. But let's say the feedback for brakes was way above the segment average. Would you rather investment money and time to improve your brakes or investigate the cause of the tire wear? You have a whole bunch of these causal relationships mapped out in your discussion about their motivations. You're rifling out these slipperly slope ideas based on their potential actions, and it really sounds absurd. If an automaker could game the surveys by building cheater cars in the 3-month window (as you pointed out, every car in that timefrae would need to be a cheater car since they are gaming the entire sample), then that automaker's competency in execution would be through the roof. They'd be so good at their jobs that they wouldn't even have a "quality" problem to begin with. I have no doubt they want to make money, but if their test had no value because of idiotic behavior and "racketeering" then servies would appear that could provide better information and service than JD Power. A profit opportunity would exist for someone to give the automakers an objective source of information for them to evaluate short-term and long-term quality/reliability. A clear disconnect was occuring where customers had problems using something - and also customers having stuff failing to work. What would you do if you ran a survey and your customers (automakers) felt it necessary to differentiate the two types of incidents? Would you add a completely new survey or would you simply expand the scope of your existing one? And to Morea's point... good luck talking to any two people on this planet who have the same understanding of the word "quality."

So how does JDPower jump from surveying customers (most of whom know little about cars) about "quality" to being able to help automakers with problems like your (intriguing) example: So let’s pretend JD Power survey respondents also come up with 10 instances per hundred for the same car. And let’s say that is twice that of the segment average of all respondents. At this point, the automaker can assume that something is causing the car to exhibit uneven tire wear more frequently than others. Maybe it’s excessive camber by design. Maybe there is a high incident of failing ball joints that is causing uneven tire wear. Maybe the cars are getting damaged during shipping and their thrust angle is off the moment it hits the dealer lot. Will customer answers like "The tires wore out too fast" help an automaker answer your question? Shouldn't service managers be in a better position to provide a more cogent answer? I'm just asking, no rhetoric involved but plenty of skepticism since I am a believer that bad data leads to bad results (and that bad data is worse than no data). Asking customers what they think without careful consideration leads to "new Coke".

holydonut, For my Ph.D. I spent a year and a half inside General Motors, observing their product development activities. I've observed exactly what you're talking about: people spent far more time coming up with reasons they really didn't have to make any changes to the product than figuring out how to improve the product. Within that context there really is no current substitute for J.D. Power. Nor will you ever find me arguing that manufacturers shouldn't use the data. "The racket" has motivated such people, and provided them with information on which to act, when simple competitive pressure and their own warranty data has not. Now, a more proactive bunch would not have needed J.D. Power. But of course we've got to work with people as they are. I should note that it's not just Detroit. Mercedes only re-engineered its U.S.-market braking systems to produce less unsightly dust because they wanted to improve their J.D. Power score. Before that point, it was just "Silly Americans. Don't they realize that the best brakes make lots of black dust? Just wash it off." You ask whether I feel they should have two separate surveys to handle design quality and mechanical quality? No, one survey is fine. But they should keep the scores separate, and not combine them. In the end there are two problems with J.D. Power, both of which it's hard for any provider of reliability information to resist: 1. Must choose between serving industry and serving car buyers; they've chosen the industry, because that's where they started and that's where the money is, and this limits and distorts the assistance they can provide to car buyers. Unfortunately, for "the racket" to work they've still got to pretend they're providing the most useful information to car buyers. 2. CR of course chose car buyers. But both they and J.D. Power suffer from a second problem: making mountains out of molehills. Or at least mountains out of hills. When your product is reliability information, it's tempting to present this information in a way that maximizes the customer's demand for reliability information--i.e. make problems seem as large as possible. Car buyers (and perhaps the manufacturers as well) play right into #2, because they want information in the simplest form possible. Many people, when they must choose between having to think hard about something and being misled, would rather be misled. This justifies reporting results in the form of dots, among other things.

Great thread. I also agree that JD Powers has an inherent conflict of interest from the vehicle consumer's point of view. However, that is NOT their concern since OEM's are interested in looking good. So they try to make everyone happy by making hundreds of tiny niche segments so that everybody has a winner. One thing that hasn't been discussed is a very simple one: warranty costs as expressed as a percentage of sales. This value is very important to all automakers. Honda and Toyota keep theirs at about 1.5%. Ford has recently improved theirs to 2%. GM has kept mum what their % is. That tells me everything I need to know. And I love CR for calling out the same kind of bullshit that isn't tolerated on this site!