The Truth About J.D. Power's IQS
Another year, another J.D. Power survey. Since the non-profit Consumer Reports organization prohibits carmakers from using its ratings in their ads, “ranked highest in initial quality by J.D. Power and Associates” should start flooding the airwaves and Internet any minute now, with print sure to follow. But does all of this noise signify anything? Should those seeking trouble-free wheels be sure to buy one of J.D.’s winners? Hardly.
First, note the “initial” that qualifies “quality.” Power surveys car owners on “problems” encountered within their first 90 days of ownership. Most people understand that a car that’s reliable for 90 days isn’t necessarily reliable beyond that. But there’s a bigger issue. J.D. Power’s IQS has been redesigned (for the second time) to encompass a larger number of potential defects. And the more the IQS includes, the less it measures what most people want to know: vehicle reliability.
The previous redesign doubled the average number of reported problems per car by extending the IQS beyond defects (that can be fixed) to designed-in annoyances (that must be endured). For example, cupholder dissatisfaction famously slammed MINI’s score. The 2006 IQS report takes a step in the right direction by including subscores for "design quality" and "production quality." Combining two very different elements into a single score makes it unclear what the number represents. Yet this score receives 99 percent of the press coverage and 100 percent of the ad citations.
If you compare the rankings based on production quality alone, the brands’ relative positions change dramatically. BMW bounds 24 places to third; Buick jumps 14 to eighth; MINI ascends 13 to 16th; Mercedes-Benz climbs nine also to 16th; Subaru also gains nine to 19th. At the same time, Dodge drops eight to 27th; GMC plummets 13 rungs to 22nd; Nissan plunges ten, also to 22nd. Eight others change position by at least five slots. These include Chrysler, which shares many models with Dodge yet moves up five places, to fifth. Out of 37 brands, 16 rankings are heavily affected by the inclusion of design quality.
Beyond the cloudiness of the revised methodology, the way the results are reported and spun continues to put too much emphasis on relative rankings. In fact, absolute differences are often minuscule. Looking at defect rates alone, 22 out of 37 brands fall within one-tenth of a Problem per Car (PPC) of the 0.64 average. Thirty of 37 brands fall within two-tenths. Of the seven beyond this range, only one, Lexus, is on the top, and it only betters the average by 0.22 problems per car.
Stay with me here. The best brand, Lexus, has 0.42 problems per car, while the worst, Isuzu, has 1.10: a best-to-worst difference of 0.68 problems per car. Even this range results from a few especially low-scoring brands. The difference between number three (Toyota) and number 32 (Hummer) is a scant 0.27 problems per car. It’s ironic, since brands at the bottom of the chart receive the least attention in J.D. Power’s press releases. For years they didn’t even publicly release below-average scores.
Put another way, a Toyota compared to a Hummer has a one in four chance of having a single additional problem. Even comparing a car from Isuzu with one from Lexus, only two in three cars will have a single additional problem. What’s more, this additional problem is likely to be the only problem. Folks, we're talking about a single trip to the dealer for a single problem–which you still face nearly even odds of taking if you buy the best brand.
The reason why J.D. Power lumps design quality into the IQS is clear: without it, the differences between brands are rarely worth debating. And the smaller the differences, the less people care about IQS. And the less the pubic cares about IQS, the less automakers will pay to advertise IQS scores, and hire Power consultants to help improve them. This would truly be a problem– for J.D. Power.
J.D. needs to re-think their methodology and reporting. They should keep problems that require repair separate from other issues. Forget the brands, they don’t vary enough. Instead, emphasize model scores. Next, focus less on rankings and who is the best and more on the size of the differences and who is within spitting distance of the best. Finally, J.D. Power needs to shift their emphasis away from “initial quality” towards long-term durability. Manufacturers won’t like that a longer-term study keeps new models off J.D.-branded consumer radar, but anything less is, well, less.
Heck, J.D. Power might even rake in more cash this way. Anyone reasonably near the top—and not just those at the top—could advertise “ranked good enough in quality that you should focus on other criteria by J.D. Power and Associates.” No, it’s not punchy. Just the truth.
[Michael Karesh operates www.truedelta.com, a vehicle reliability and price comparison site.]
Michael Karesh lives in West Bloomfield, Michigan, with his wife and three children. In 2003 he received a Ph.D. from the University of Chicago. While in Chicago he worked at the National Opinion Research Center, a leader in the field of survey research. For his doctoral thesis, he spent a year-and-a-half inside an automaker studying how and how well it understood consumers when developing new products. While pursuing the degree he taught consumer behavior and product development at Oakland University. Since 1999, he has contributed auto reviews to Epinions, where he is currently one of two people in charge of the autos section. Since earning the degree he has continued to care for his children (school, gymnastics, tae-kwan-do...) and write reviews for Epinions and, more recently, The Truth About Cars while developing TrueDelta, a vehicle reliability and price comparison site.
More by Michael Karesh
Comments
Join the conversation
It looks like the Strategic Vision has succeeded where social scientists have repeatedly failed: scientific proof for Maslow's hierarchy of needs (which should be familiar to anyone who has had Marketing 101). http://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs This sort of theory of values is old stuff. My M.A. paper concerned this sort of research back in the 1950s and 1960s. Connecting such values to purchasing behavrior has always been iffy at best. Tends to fall apart like wet toilet paper when pressed. People just refuse to do what they're expected to do when it's time to buy a car. That said, Strategic Vision does try harder than JD Power to explore and capture the nuances of automotive satisfaction. It's hard to say how well they succeed at this without more details than they publicly release.
Yes, they are a bit reluctant to release their precise methodology. Commerical secret or too wild and wooly for public consumption? GM quotes their results an awful lot, and they're customers. But then all of these JD Power-type guys see no problem in accepting money from the people about whom they report. Who's to say the Chinese Walls don't have ears? OK, gotta pee.