By on August 28, 2006

x-type2222.jpg Every time an automotive research firm releases the results of a reliability survey, the focus is the same: who “won.” Firms like J.D. Power only publicly release model-level results for the top performers. Even where these firms release scores for all contenders at the make level, journalists focus on the winners. After all, John Q. wants someone to tell him which car to buy in as few words as possible. In the process, any car buyer truly interested in identifying the best car for their needs and wants gets left in the dark.

Yet few people realize this. It seems so natural to focus on the winners, whether the topic is an election, the NBA playoffs, or vehicle reliability. But let’s play “which one of these is not like the others” for a second. If someone wins an election by one vote, they get the office. If a team wins a deciding playoff game by one point, they get to move on to the next round. And if you buy the car that won an award by one point, you get…what?

In most cases, you do not get the car that best suits your needs. When evaluating a car, most people care about more than whether or not it’s the least likely to break. They’re also likely to be concerned about how it looks, how it drives, how well the seat fits their rear and more. Often the award winner isn’t nearly as attractive, as fun to drive or as comfortable as a competitor. Tradeoffs are a necessary part of the purchasing process.

Most reliability surveys provide dot ratings– with unspecified ranges of reliability– for the non-winners. But without the actual, precise scores for all the contenders, trading off quality against other factors is impossible. Buy the award winner and you’ll have a pretty good idea of how much style, performance, or comfort you’re giving up. But you won’t know how much “quality” you’re gaining in return. Say it’s one dot’s worth. Well, how much is that? Unfortunately, to make a wise choice, you need to know.

In a basketball game, a basket at the final buzzer can and should be the deciding factor. The closer the game, the more exciting it is to watch. But vehicle reliability isn’t about entertainment (though the news stories that cover the awards may be). The closer scores are, the less they matter. And the scores are often quite close.

In J.D. Power’s 2006 Initial Quality Study (IQS), 30 of 37 makes fell within two-tenths of a manufacturing defect per car of the average. The difference between number three (Toyota) and number 32 (Hummer) was 0.27 problems per car. In J.D.’s most recent Vehicle Dependability Study (VDS), 23 of 37 makes fell within half a problem per car of the 2.27 average. Only four makes— three of them domestic— bettered the average by more than half a problem per car.

Real-world problems occur in wholes. A car cannot have 1.79 problems. So Toyota’s VDS score of 179 implies that the typical Toyota has two problems in its third year. And Ford’s score of 224 implies… much the same thing. Buy a Toyota over an alleged “Fix or Repair Daily” car, and you gain no guarantees, just a middling chance of avoiding a single additional problem in the third year.

No one gets all hung up on vehicle reliability to avoid a single additional problem. Most consumers simply want to avoid buying a lemon that’s in the shop “all the time.” Well, reliability scores based on averages don’t help. Say we’ve got two basketball teams. On one, the average height is 6'5”. On the other, the average height is 6'7”. Which team has the most players over seven feet? Using averages alone, it’s impossible to say.

Time to buy a car. You know the award winners, and not much else. Play it safe and you’re likely sacrifice style, performance or comfort to maximize your odds of having one or two fewer mechanical problems. Ignore the surveys and there’s no telling how much car trouble you’ll have. Maybe none at all. Maybe a lot. The research firms know. But they’re not going to tell you. They only provide potentially helpful comprehensive data to corporate clients willing to pay the big bucks. 

Hang in there. My website, TrueDelta , is committed to clarifying how cars differ in reliability, from the "best" right down the “worst.” We’re working hard to collect unbiased real-world data on your behalf. And TTAC can always be counted on to go beyond the superficial story. (How many critiques of J.D. Power’s methodology have you seen in the mainstream automotive press?) Someday soon, you’ll be able to identify the car that best suits your needs and wants— without playing a guessing game that’s stacked against you.

Get the latest TTAC e-Newsletter!

59 Comments on “Quality Will Out...”


  • avatar
    jerry weber

    Someone else I read talked about the irrelevancy of numbers like JD Powers. The statistics have narrowed so greatly that the so called best are just marginally better than the rest. Yet, these statistics publication labs don’t every point this out. They just simply take the tighter numbers and publish them as if there is just as large difference in quality as there once was. It seems nobody wants to say, look I was the one to point this out and got the manufacturers to address it and now it is no longer relevant. No we keep reporting finishes by a nose as if the winner lapped the track.

  • avatar
    BuzzDog

    J.D. Power’s Initial Quality Study (IQS) is flawed on a few levels. Among the most significant is that unless you get a new car every two to four weeks, the number of defects at this point is not particularly relevant to the buyer, particularly when you consider that build quality is fairly high among all makes and models. Second is that the survey respondents are still in the “honeymoon” period; when you’ve just searched for and plunked down some big bucks for your new four-wheeled mate, you often tend to be more forgiving of minor quality issues. More telling are the reliability ratings compiled from many owners over three to five years – and perhaps, the answer to the age-old question: “Would you buy this car again if you have to do it all over?”

  • avatar
    philbailey

    On the other hand, Consumers Reports is dead accurate! Their “black spot, red spot” charts reflect real world reality. How do I know this? Because I get to fix ’em after they’re out of warranty and if I had to rely on Lexus business I’d be reduced (almost) to changing oil.

  • avatar
    JimP

    Many years ago when I started working at a specialty outdoor sports retailer, Consumer Reports came out with an article on life jackets (PFDs). The one we loved came in dead last; the one they loved, we loathed. I will grant that their winner had (marginally) more flotation and so in one (limited) sense was “better.” BUT it was so horribly uncomfortable that you’d never wear it, while the one we loved was almost unnoticeable to the wearer. It seemed obvious that there were other factors that they decided weren’t relevant, and yet they missed the biggest one of all: if you’re not wearing the PFD, it won’t do you much good when you need it. From that day on I decided to cancel my subscription to Consumer Reports. If they couldn’t get a simple thing like PFDs right, what were the odds they’d give me anything useful on cars?

  • avatar

    No doubt there’s some truth to what you say, Phil. But you’re focusing on out of warranty, i.e. after year three or four, while CR’s overall rating dots are based on years 0-2.5 (the oldest cars that influence these dots are 2.5 years old when the data is collected).

    There might be a fairly strong correlation between small differences in the early years (the absolute differences in these years are rarely large) and the larger differences that emerge in later years.

    But as far as I know CR has never published the results of an analysis with this conclusion. And why not provide the actual numbers so people could get a clearer idea of how large the differences are? Continuing, more and more people seem to be focusing on reliability in the later years, as you are. Why not at least provide a second prominently displayed dot, for years 4 to 7?

    Finally, as I say here, I think people are less concerned with the number of problems they’re most likely to have, the average, than their chances of getting an especially bad car. The dots are based on averages, and thus in no way indicate the chances of getting an outlier. If there is truth behind the common perception that American cars are more likely to be lemons, then it’s quite possible that stats based on the odds of getting a lemon would vary more than those based on averages.

  • avatar
    postman

    The results of these surveys have been so overused and abused by media marketing that they are virtually ignored by the general public. Most people are well aware that all statistics can be skewed to result in whatever outcome the user wishes to present

  • avatar

    I wouldn’t be so sure they are ignored. Many people are not so well informed as the readers of TTAC tend to be.

    It’s also possible to overstress how much statistical analysis can be manipulated. The main problem is not that the available stats have been manipulated, but that so few stats are actually available, often with an unclear basis, that people do not have enough data to work with to perform their own calculations. You end up either having to use very partial information, or ignore the stats altogether.

    The answer is not no stats, but more stats so that you can more easily know precisely what you want to know. Currently, it’s often necessary to infer what you’d like to know from stats based on something else. For example, people infer long-term reliability from stats based on short-term reliability.

  • avatar
    dhathewa

    First, Karesh is distorting the way the reliability statistics get used. Nobody looks at the reliability of a vehicle to the exclusion of all other factors. Nobody picks the car that won the award by one point if one is a New Beetle and the other is an F-350.

    Second, I think Karesh is underestimating the importance of reliability. What good is the car to me if the color, the style, the power, the road feel, etc, are all exactly what I want but it’s in the shop? Few of us can afford to buy a vehicle purely for pleasure. Mine gets used to get to work and whatnot. If it’s not ready for duty, it might as well be an anvil.

    Third, people like to take shots at CR and, no doubt about it, CR sometimes gets things wrong. However, nobody else has come along with something better. TrueDelta does look like a pretty good experiment and worth watching but its scope is autos only and what’s the value to date?

    JimP might have served himself and others better if he’d sent CR some feedback about their PFD testing methods. Could be he’s right and sharing the knowledge might improve CR. Without CR, what’s my best source of information for PFDs? The salesman in the shop JimP referred to? How do I know he’s working for me and not just recommending the PFD with the biggest markup?

  • avatar
    Martin Albright

    As the saying goes, there are lies, damn lies and statistics. “Problems” per car? What the hell does that mean? Glovebox doesn’t latch = 1 “problem”. Transmission fails while I’m crossing a railroad track with a train coming = 1 “problem.” In both cases the score is 1 “problem” but obviously one is more serious than the other – how do the surveys differentiate them?

    Also, how do driver’s expectations fit into this? Someone who pays $15k for a Hyundai or Chevy may be more willing to forgive niggling issues than someone who lays down $50k for a BMW or Caddy (and rightly so.)

    If these statistics are useful at all (other than to be misused as advertising tools, that is) then they might be useful to show very broad trends in manufacturers, but as far as individual models, I think they’re fairly useless to the average buyer, especially in the age of the internet. Now if you want to know what kinds of problems occur with your chosen car, just go to a message board that caters to that make/model and ask questions. For example, I never would have known of the dreaded Subaru 2.5l head gasket issue had I not gone to the internet. “Statistics” factories like CR and JD Powers won’t give you the kind of specificity that car buyers are looking for.

  • avatar
    Stephan Wilkinson

    A few years ago, I bought a new Porsche. Got the requisite J D Power survey form a couple of weeks later. The damn form was so complex and near-endless that I started to fill it out and soon gave up, since I loved the Porsche, hadn’t had the slightest problem with it, and got tired of checking off ‘5, 5, 5, 5…’ or whatever the top-grade number was.

    The lesson I took away from this is that people who love thir new car are far less likely to fill out the form than are people who hate it and can’t wait to tell the world why.

  • avatar
    carlisimo

    omg you’re the guy advertising TrueDelta in every single internet post you can

  • avatar
    radimus

    This editorial brought to you by…

  • avatar

    dhathewa,

    I make no judgements about the importance of reliability. It’s more important for some people than for others — otherwise some brands would sell far fewer cars.

    As for the one point, the problem is that you don’t know how large the margin of victory was, since only the winners are released. Might be one point, might be 100. So people might their decision just this way, but without realizing it.

  • avatar

    TTAC approves of Mr. Karesh’s website and its mission, and recommend his work therein.

    We do not receive any financial contribution from TrueDelta, nor is there any unofficial quid pro quo.

    I chose to publish this piece and allow the link because of the quality of Mr. Karesh’s arguments.

    As you are aware, all commentators are allowed to add links to their comments, provided the links apply to the article.

  • avatar
    radimus

    The site looks like it will be very useful if it takes off. I wish it all the best. I just couldn’t resist making the joke.

  • avatar
    nutbags

    As Martin Albright points out, there is no diffentiation as to what the problems are in the JD survey. At least CR does this although to a somewhat limited degree. At least they do not (or at least claim) accept any monies that may sway their stats unlike JD and the rest whose results are up for sale.
    Mr. Karesh, I am behind you and have been a member of your site for a few months now. Good Luck.

  • avatar
    KingElvis

    The ‘initial quality’ metric has been trumpeted by GM and seems mostly useful as a pseudoscience way of “proving” GM is closing the gap.

    Since the statistic is well, basically meaningless since the lowest score is just marginally different from the highest score, lets consider the REAL function of the statistic

    Lets rate the rating.

    Has it really helped GM’s marketing campaign all that much? (As “Kool Aid” pitcher) “OH YEAH!”

    How about GM sales? We can proudly declare it’s sold an extra .00000000002789 cars.

  • avatar

    Back around 1990 the Buick LeSabre won its class and was #2 overall in Initial Quality. This sold thousands of extra cars.

    But I agree that the current impact on sales seems to be much less significant, at least for GM. I think Hyundai has gotten a boost from the IQS.

  • avatar
    MRL

    Speakin’ ‘o quality: The babe with the Jag.

  • avatar
    PandaBear

    The only reliable source for reliability is internet forums dedicated for the model/make with sizable owners base. Places like tauruscarclub.com or honda-tech.com are the only place that a manufacture or marketing firm cannot skew the result by financial support.

    All the rests, like JD Power, or Consumer Reports, merely report stuff that are either skin deep or meaningless (i.e. glove box wouldn’t close has the same weight as transmission failure in rating).

    Those information wouldn’t show up until a model is a few years old, but then again, true reliability info wouldn’t show up until a few years later.

  • avatar
    Sajeev Mehta

    The only reliable source for reliability is internet forums dedicated for the model/make with sizable owners base. Places like tauruscarclub.com or honda-tech.com are the only place that a manufacture or marketing firm cannot skew the result by financial support.

    Amen to that. Well said, PandaBear.

  • avatar
    deVeritas

    Michael raises good points about the ‘quality’ of the Consumer Reports data. It should be pointed out that most respected scientific journals do publish their data so that interested readers can do the calculations themselves and come to their own conclusions.

    CR shouldn’t be considered a scientific publication, but as they do purport to base their conclusions on statistics, there is simply not enough data given to make those conclusions yourself. What’s the sample size, what’s the margin of error, and are the data points weighted? (Glove box or engine failure?) Yes, these are terrible terms for the math averse and would not be popular reading, but they are essential to fully understanding their conclusions. Maybe this data should be in an appendix with fine print?

    While CR may be immune to undue advertising influence, without more information, one can’t help but believe there is some subjectivity involved there, especially with their ‘recommended’ picks.

  • avatar

    For what it’s worth, this particular article was mostly directed at J.D. Power, Strategic Vision, and other research firms who only release model-level data on the winners. CR at least provides data on every car. Just no numbers. Apparently the American public cannot handle numbers. Like nursery school children, they can only comprehend red and black dots.

    I also believe it’s possible to separate CR’s reliability verdicts from their road test verdicts. The latter are necessarily more subjective. I’ve written about those here:

    http://www.truedelta.com/pieces/comparison_tests.php

  • avatar

    PandaBear, Sajeev,

    Forums can be one good source of info, but it’s not possible to gauge repair frequencies from them. Some of these forums have tens of thousands of members. If a few dozen report a particular problem, it could still be a very small percentage of the total.

    I’ve developed a few ways to measure problem severity. I don’t think the system is the right way to go about it, as even engine problems can vary a great deal. O2 sensor, or bent valves?

    Instead, I look at whether the car had to be towed, whether the repair had to be performed immediately, and how many days it took. If an engine problem did not prevent the car from being driven, and was repaired in a couple of hours, then how is it more severe than a problem with the glove compartment?

    In general, I try to measure what people seem to really care about as directly as possible. I want people to have to infer what they really want to know from what I eventually report as little as possible.

  • avatar
    1981.911.SC

    RF
    How many times are you going to let Mr. Karesh re-hash what car rating mean or don’t mean? AND then plug his web site? I hope most of your readers are astute enough to use ratings for part of their buying choice, not for 100%.
    Lets have some car reviews, how is the Toyota MR2, how about the Chrysler Mini Van? or (God forbid) how about the little Kia hatchback?
    Give me somthing to read, not a self-serving whine.

  • avatar
    maxxm

    Michael Karesh’s site, of course, is only as good (and complete) as the information upon which it is based is truly representative, so his dismissal of the value of owner-based internet sites is not necessarily valid. I belong to several such sites; there is no question that they generally disseminate much better and more thorough and reliable real-world information than the dealers seem to have, particularly for low-production or orphan models. None of us drives a statistic; we drive individual cars. As for those singular products, I believe that PandaBear and Sajeev Mehta are right on target with their observations.

  • avatar

    I do not mean to dismiss the value of online forums, not at all. If you want to get a sense of what kinds of problems a car might have, and thus which areas to check out when buying a used car or when maintaining your own car, forums are a good source. I’ve used them myself for this purpose.

    My only point was that you cannot judge repair frequencies from them. Many people go to a forum, see that many people are complaining about this or that problem, and wrongly conclude that the car is a troublesome model. With any model there will be some problems, and forums can make these problems seem much more common than they actually are.

    It’s kind of like the news. Many people feel that the world is an awful place because of all the bad stuff on the news. What they don’t factor in is that current technology makes it possible to learn about anything unusual that happens to one of the planet’s 6 billion people. I once read that the human mind is programmed to assume that anything it hears about is happening within a village of about 150 people, and thus must be at least somewhat common. Makes sense.

  • avatar

    Michael Karesh:
    August 28th, 2006 at 9:19 am
    >>No doubt there’s some truth to what you say, Phil. But you’re focusing on out of warranty, i.e. after year three or four, while CR’s overall rating dots are based on years 0-2.5 (the oldest cars that influence these dots are 2.5 years old when the data is collected).

    I don’t think that’s true. I’ve filled out the CR questionnaire on a six-year old car.

  • avatar

    PandaBear writes:
    >>The only reliable source for reliability is internet forums dedicated for the model/make with sizable owners base. Places like tauruscarclub.com or honda-tech.com are the only place that a manufacture or marketing firm cannot skew the result by financial support.

    >>All the rests, like JD Power, or Consumer Reports, merely report stuff that are either skin deep or meaningless (i.e. glove box wouldn’t close has the same weight as transmission failure in rating).

    Consumer Reports gets no financial support from companies of any sort. And while they may have some faults in the way they report stuff, they absolutely distinguish betw types of failure (engine vs transmission vs electrical, etc.)

    Michael Karesh writes:
    >>What they don’t factor in is that current technology makes it possible to learn about anything unusual that happens to one of the planet’s 6 billion people. I once read that the human mind is programmed to assume that anything it hears about is happening within a village of about 150 people, and thus must be at least somewhat common. Makes sense.

    People are unable to judge risks intuitively, and while I don’t know whether this is precisely true or not, the outcome is the same. This is why so many people are more afraid of flying than of being in an automobile, and why so many Americans outside of DC and NYC are afraid they might be killed by terrorists.

    I hope the website is a big success, because it sounds to me as if it could be very useful.

  • avatar

    They collect data on cars up to 7.5 years old. But the “predicted reliability” dots that receive the bulk of the attention are based on the first 2.5 years. Dots for individual years back to what they call “eight-year-old cars” (really 6.5 to 7.5 years old) are only provided with the system-level dots.

    They provide subscores for the different systems, but you cannot necessarily tell how severe the problem was from this information. As I said above, all engine problems are not equally severe.

    The above editorial was not about CR. This one was:

    https://www.thetruthaboutcars.com/?p=1719

  • avatar
    4runner

    Two comments on JD Power:

    1. Most automotive experts and the mainstream media quote JD Power VDS and IQS numbers to their hearts content but many do not realize that those publicly available numbers are a combination of (i) vehicle satisfaction and (ii) vehicle reliability. They are not a true gauge of a vehicle’s reliability because JD Power pollutes this data with highly subjective owner satisfaction scores.

    Case in point: BMW scores rather mediocre compared to its Japanese competition. However, when one dissects BMW’s JD Power scores, they will discover that BMW’s are reliable but leave the owner unsatisfied. (Some observers believe iDrive is the sole reason many to find BMW’s less satisfying.)

    2. JD Power is in business to make money. The real valuable data (mechanical defects by model) is made available only to paying OEM’s. Needless to say, this data is quite valuable and significatn safeguards are employed to keep this data a secret.

  • avatar
    rottenbob

    This was a great article and raies some important points. I am definitely going to check out the TrueDelta site.

    However, 1981.911.SC also raises an important issue: When will TTAC review the Kia Rio5 hatchback?

  • avatar

    Thanks.

    I’ve tried to drive a Rio5 a couple times–I like small cars and think it’s nicely styled–but whenever I say I’m interested in driving one the salesperson looks at me like I’m insane. They tell me right off I won’t like it. So I chicken out and drive something else.

  • avatar
    PandaBear

    Actually, in the hard drive industry that I used to work in, there is a site that does reliability survey very well and we can probably take a look at how it is done:

    http://www.storagereview.com

    It catalog each drive and its reliability by survying not only warranty, but also up time without incident, and how long does it last before the problem occur. Auto related problems are usually more complex, so it may take more efforts.

    I by no means suggested that true delta is bad, it is definitely a good start for us looking for another car. What is really hard to do, is to quantify the severity of problems and its cost. Is it possible to use “job quote hours” from Mitchel to quantify the severity?

  • avatar

    Thanks for the tip. I relied heavily on that site when buying drives in the past. I like how they word the intro.

    You’ve got consider people’s ability to answer a question. I don’t ask what specifically was repaired, at least not on the standard survey, because I don’t think most people would know without digging up the paperwork. And most people won’t dig up the paperwork.

    Your average computer owner would not be able to reliably fill out the StorageReview survey, though, because they won’t be able to figure out which model of hard drive they have. But perhaps this site mostly caters to hardcore computer enthusiasts that know this stuff.

    My research is designed so the average person can participate. So I ask questions anyone should be able to answer, especially since the survey is completed the month after the repair occurred:
    –was the car towed?
    –was a warning light on?
    –could they have safely and dependably driven the car another week before going into the shop?
    –how many days was the car gone?

    One more advantage of my approach: I only ask about problems that occur after someone signs up (with the exception of the first month). This way people don’t know what they’ll be reporting at the time they decide to participate. StorageReview attempts to deal with this possible source of bias by asking people to respond about every hard drive they’ve owned, not just the ones that failed. But this isn’t going to happen in many cases.

  • avatar

    Consumer Reports auto comparisons of decades ago had some strange recomendations. The one that stands in my mind was about small cars, including many imports. The auto that CR recomended (don’t remember the scoring) was….. AMC Pacer! I’ve never quite recoved.

    CR was fighting the war with Japan for a while and finally CR surrendered.

    Be fun to go back and review some of the werid reviews.

  • avatar
    Sajeev Mehta

    Michael: I see your point. I think your analysis and the information spread on forums caters to different people. Both are equally important.

    TrueDelta (and others) help those who look at new or late model cars, make informed decisions via facts and stats, and narrow it down a list of cars they will persue further.

    The forums help lurkers and members fix the trouble they’ve already gotten into, or give someone the information to justify the car they are dead-set on purchasing, stats be damned. Furthermore, a lot of the real valuable forum help comes after the warranty period expires, usually on the 2nd or 3rd owner by then.

    Or to make a stupid-insane sweeping generalization, I see more passionate right-brainers in one place and more informed left-brainers at the other.

  • avatar
    Terry Parkhurst

    Stephan Wilkinson makes a really important point (in earlier comments) about why someone who likes their car (or truck) but doesn’t want to be bothered with filling out a complicated survey (for nothing, or maybe a nominal $25 or so) won’t weigh in. That means the survey’s final tally is screwed – or to put it more eloquently “the equation is out of balance.”
    I think too many evaluations get lost in numbers, whether they are JD Powers survey or “comparos” in C/D or R&T. I happened to see a white Honda Fit today, walking to my neighborhood photo lab (which does digital and silver halide film work) and after admiring it, but beginning to walk away, spied the owner coming out to show it to a friend. I asked him what he thought of it and he told me.
    Now, admittedly he’d only had it for a short time. But we forget the importance of words and “seat of the pants” evaluations. What ever happened to talking to, or reading, someone’s feel for what is going on, underneath their derriere, as they drive the damn car (or truck, or ride a motorcycle).
    JD Powers is akin to a “think tank.” Thinking is good. But when you ask a guy, how is your marriage, he can usually tell you in the same manner, and with the same authority, the owner of that Fit – who loves it, by the way – did me, without hooking up any metering systems, or crunching any numbers. The old saying applies: play your hunches. And listen to owners of the car (or trucks or motorcycles) you want. JD Power’s information is not written on stone from above (or where ever the Higher Power which controls the universe may hang out).

  • avatar

    Sajeev,

    I have noticed that the passionate consumer is less likely to be interested in pricing and reliability. What had not dawned on me was that this was exactly the sort of person most likely to be avidly involved in a forum. But you’re exactly right.

    Minivans are most popular on my site, sports cars least popular. People don’t shift between alternative sports cars based on price and other quantifiable criteria; instead, they know which one they want.

    These should be complementary forms of information, but often people gravitate towards one or the other.

  • avatar
    Jan Andersson

    Again and again: no brand is better than its service. If I had a lemon with frequent problems, but loved it for other reasons, I wouldn’t mind so much if there was a competent dealer service shop in the neighborhood.

    A short call, and they should take in the car and lend me another similar without any hassle (paperwork) at all.

    If I couldn’t get my car moving, they should pick it up within 30 minutes, and leave the other car where I was.

  • avatar
    noley

    Having spent 13 years in a past life doing quantitative market research (some of it automotive), I’ve always questioned the methodologies used by CR, JDP and others. The questionnaires are usually poorly designed and can’t really gather much more than a fairly high-level view of any given make and model. They do provide decent directional info, but certainly not enough on which to make a buying decision. I think they are best at things like toasters and maybe home electronics but not really useful for automobiles.

    I find online forums are more practical, especially in terms of what to watch for in a given model. For example, I’m considering a used Porsche Boxster and learned at PelicanParts to look for rear seal leakage. CR or JDP would call that a “transmission problem,” but the specific info is a lot more valuable. For my Saab habit I go to Saabnet.com which has saved me a lot of coin in diagnosis and procedures, and enabled educated discussions with my indy tech on stuff I’m not inclined to do.

    In taking a quick look at the True Delta site it looks like it is on a path to rectify those shortcomings. I hope it does well.

  • avatar

    noley,

    Thanks for the vote of confidence. You’d probably enjoy my piece on CR’s survey, which I linked to earlier in the thread.

    Jan,

    You want good service? I’ve rarely found it. My most recent trip to a Chrysler dealer left me with lug nuts on so tight that I couldn’t change the tire when I got a flat. Destroyed the jack handle trying. AAA guy had to use a two-foot pipe as additional leverage on a four-way tire iron to get the things off.

    If you cannot properly tighten a lug nut, what can you do right?

  • avatar
    Jan Andersson

    If the dealers understood the importance of service quality for future car sales, they would start the reform work tomorrow. True story: in a mid size Swedish town, all taxis were Peugeot. In other cities, that was not true, only maybe one out of ten. Why? Quote from a driver: “If I call them half past three AM a Sunday morning, they will start the service work within 20 minutes and apologize for being late”.

  • avatar
    deVeritas

    Those complaining that this article was even posted should take note that you can click a link at the top called ‘Reviews’, and bookmark it. You won’t be bothered by non-review opinions that might matter to your current car anymore.

    As for forums, you must read everything with a grain of salt. How do you know that the person who made a comment isn’t a company representative? Or a cometitor representative? It’s known that with elections, finances, and other industries, people are often paid to generate or dismiss ‘hype’. As long as registration is free and unverified, you can’t be sure of anything really.

    The only reason you can lend any credence to any of the authors on this website, is because TTAC and RF have a reputation that they are trying to maintain. If RF allows a company representative to post on here, or if he was a company representative, he hopefully would disclose that. Because if it was brought to light otherwise, his and the TTAC repuation would be forever tarnished, his credibility would plummet along with the website hit count. However, if it was found that ‘1981.911.SC’ represented Porche, he/she can just change user names and start over. No big deal.

  • avatar
    PandaBear

    >>>>As for forums, you must read everything with a grain of salt. How do you know that the person who made a comment isn’t a company representative? Or a cometitor representative? It’s known that with elections, finances, and other industries, people are often paid to generate or dismiss ‘hype’. As long as registration is free and unverified, you can’t be sure of anything really.

    The nature of a good forum is a large member base that points out the obvious fault in people’s logic, like the way we point out how someone blindly praise Impala’s quality in one of our previous article comment got “owned” by various other members here, as well as the join date and post count of the member.

    There will always be bias, but as long as the bias is known and balanced by other members, or even other sites, it should give enough valuable info for the visitors who want to dig further. There will never be a “just buy this” opinion that every one agree to, and all buyers should do their own research at the end.

    Such is life, whether it is investment, finding the other half of your life, or which career path to choose, you can only find out so much info, and at the end you have to make the decision and take the risk yourself.

  • avatar
    RicardoHead

    A couple points.

    IMHO, people who buy import makes (particularly german makes) tend to think they have bought something “fine” and so when the POS breaks down or needs extra maintenance they consider it the “cost of owning something fine” and discount it in their minds as being a negative aspect of ownership. It’s an image that the imports marketers are more adept at creating, plus they have it easier because Americans generally have an inferiority complex about domestic makes. Japanese makes benefited from the “somthing fine” attitude in the past before they became more mainstream. Earlier adopters of Japanese makes were more like the purchasers of german makes (albeit with less cash to burn) and so they coddled their import baby.

    As Japanese makes have become more mainstream, their buying demographic has moved toward the GM and Ford average, which might mean an oil change every 30k miles whether it needs it or not, owners questioning “why da gaddammed tires need air ‘gain when ah jus’ pumped ’em full in ’04?” and the like. Hence the Japanese quality image starts taking a hit, and, true to the inferiority complex, I have heard Americans say “it’s because they started making them here.”

    That being said, in the last 25 years of my automotive awareness, my family has owned many vehicles, most domestic. With the exception of a famous POS british make, I have noticed no real difference in quality between Japanese and US cars when both are maintained decently and regularly. Paints, motor parts, all that seem very good. The flimsy body structure of my ’86 Accord (made in Japan) has caused me to shun Hondas, but I can’t complain too much about the motor, although it was at best average.

    I’m not counting the fun-to-drive factor here. But if people were buying cars because they are fun to drive, Camrys and Accords would NOT be selling.

  • avatar

    Good points. But I think the impact on maintenance can be overstressed. Many issues these days are electrical, and I’m personally not aware of periodic maintenance that should be performed on the electrical system.

    Or do some people keep their power window/seat/etc. motors well lubed?

  • avatar
    Sajeev Mehta

    Definitely lubricate window/seat runners on fully-depreciated hoopties.

    Other than that, Michael’s spot on with his point about electronics. Cut one too many corners and flaming cruise control systems are not far behind.

  • avatar
    RicardoHead

    My Accord had two power window motor failures. None of the Fords (or my brothers GMs) had window issues. The Accord also had electrical issues … overall I was generally disappointed with the Honda versus the hype. Fun motor to rev, though.

    Had to take my 02 Ford F150 in for the cruise control cable replacement. Took around 20 minutes including paperwork. Yes they recalled a buttload of vehicles because something like 50 or 80 vehicles out of like 5 million caught fire. I know any fire is bad, but the constant reminder for some clearly oddball situation is ridiculous.

    BTW – based on quality and my 7 years (recent) experience in Germany, I’ll never buy German or any other euro make. I ran a fleet of company cars over there and to diversify quality risk we changed cars from purely BMWs to all their makes. Across the board all were garbage, though Audis were the best of the crap. And German makes are the best over there. When I tried to mix in some Japanese makes the germans went nuts. I suppose if a person needs a Made In Germany label to appease their inferiority complex, then the purchasing decision makes sense, but I’ll stick with US and Japanese. They last, and since people here think 100mph is fast the US and Jpn makes have you well covered. I remember in Germany driving to a BMW dealership to pick up another stranded coworker again … I had brought over a Ford Mustang Convertible. The dealer gave me crap in the middle of the showroom about driving the stang, so I said loudly so his customers could here that “the reason I drive an American Ford is so I can reliably pick up BMW drivers from the garage.” He never gave me shit again.

    Oh – one other thing about the people saying Honda bends over for warranty issues. My 86 Accord was bought new at the time when third brakelights were not yet integrated into rear designs, and so the car had that plastic add-on protrusion was sticking out of the rear panel and pressed on the window. I’m cleaning dust back there after a month of ownership and notice the plastic is cracked, so I took it in to get it replaced under warranty. Honda gave me shit and said I broke it – which I hadn’t, it was simply forced in there with pressure causing a crack. After bickering, they replaced it. 3 months later the new one was cracked in the same spot … I took it in and took more shit from the dealer and after 2 hours got it replaced. It never broke again, but don’t tell me that Honda dealers are better at kissing my butt that Ford … on a couple minor issues with Fords I have gotten much better service and reception than at Honda, which also has reflected in my subsequent purchase decisions.

  • avatar
    Lumbergh21

    The validity of the reliability rankings also has a lot to do with the sample size. If the standard deviations in the average number of repairs for two cars is greater than the difference in their averages then little or nothing can be said about whether one car is more reliable than others. However, in tthe example given in the above article, the Ford has a 25% higher incidence of repairs, and this is undoubtedly statistically significant (I’m pretty sure the sample size for these two vehicles is relatively large unlike comparing a Ford GT to a Porsche 911). These average repair numbers are really providing you with an idea of how likely repairs are for a given car versus another car. In the example, they are 25% more likely. I will say that there is still more to rate of repair comparisons that is not typically discussed. The average cost of repairs. You may have a faulty door light switch that cost $30 to repair on one car (or $5, if you are at all mechanically inclined yourself) and a faulty clutch on another car that costs $500 to fix. Both of these “repairs” gets entered into the system as a repair but one is insignificant compared to the other. Or, even in the case of similar repairs, it may cost $200 for a new starter on a Ford versus $600 on a Toyota. That Ford starter would have to be woefully less reliable than the Toyota starter for the Toyota to be more cost effective. So, yes, the various reliability rankings do not tell the whole story and may provide a very inaccurate picture, but it is not correct to say that there is no real difference between an average of 2.24 repairs and 1.79 reapairs in 3 years of ownerchip just because both round off to 2 repairs.

  • avatar
    Lumbergh21

    “You want good service? I’ve rarely found it. My most recent trip to a Chrysler dealer left me with lug nuts on so tight that I couldn’t change the tire when I got a flat. Destroyed the jack handle trying. AAA guy had to use a two-foot pipe as additional leverage on a four-way tire iron to get the things off.

    If you cannot properly tighten a lug nut, what can you do right?”

    If this was the case, they could very well have damaged the car’s rotors, but you’re absolutely right. I can’t believe how tight the drain plug was on my car’s oil pan after having the oil changed at a dealer while some recall work was performed. I practically had to bench press the front end of the car to get it off when I went to change the oil myself a few months later. I guess I should be happy that they didn’t strip the threads. It seems that they only know hwo to use an impact wrench and have never heard of specific bolts, nuts, etc. having specific torque requirements. The sad thing is this stuff happens at a dealer’s service center where they are supposed to be the most knowledgable ones about that particualr make of car.

  • avatar
    RicardoHead

    After having my Honda’s oil overfilled twice by the dealer (well over the max) and an independent outfit strip my drain plug, I started doing most maintenance myself. If it’s not anything major, I get it done right, faster, and save money, and am more aware of anything going on with the vehicle. The only reason I had the dealer do it anyway is that Honda engineers located the filter is the most retarded location imaginable.

    Don’t even know how a dealer can overfill. That stuff comes out a hose that measures it, I believe. So I asked and the mechanic said that he didn’t see the oil on the stick so he added more. He apparently was not aware that new oil is not black …. then they explained he was new and not thoroughly trained. So you see, idiocy comes with all makes.

  • avatar
    bobinacar

    JD Powers surveys are worthless for all the reasons previously noted. Too bad they get so much attention. CR’s are not much better – their method of gathering data results in totally invalid statistics (refer to “Dewey Wins” for more information). In addition, they use overly broad categories to rate vehicle components so that some very minor problems get classed with some very major ones. Black dots don’t say very much about details.

    Add to it that they don’t quantify dealer response to problems i.e. “major engine problem but the factory contacted all the owners proactively to fix it” vs. “manufacturer argued endlessly that problem was not their fault and refused to pay for repairs”.

    Lastly, most of the people I know have few problems in the first couple years with a car and very few of them trade every three years. What really counts are the later years at high miles when a good car is trouble free- and a bad car is lots of trouble. No one seems to address that.

  • avatar
    TheOne

    I agree with RicardoHead on one thing at least I specifically purchased my “German… POS” BMW 3-series based on the fact that I liked how it drove. If I wanted something just to get me from point A to point B I would have purchased another Accord. I would consider myself a very knowledgeable car buyer and was well aware that the reliability was not rated as high as say a lexus.

    The point that is completely missed in standard reliability surveys is that there are various levels of service from different manuf./dealers.

    While my car has had a few repairs, RicardoHead is correct in that I dont consider it such a big deal. I get free bmw loaner cars to drive while my car was being fixed. My car sees the dlr about twice a year which usually includes a sched maint visit and one other type of repair. The lexus dealers here will drive to your home or work place to pick up your car and leave you a loaner while they work on your car.

    I think truedelta is on the right track if it can track things like number of dealership visits a year, did you receive a free loaner vehicle, was the problem fixed the first time, etc. This info makes a big difference when coupled with the std reliability stats..

  • avatar

    TheOne:

    Your post reminds me that at one point I wanted to ask if a free loaner was provided, as this can have a major impact on the inconvenience level. I cannot remember why I didn’t include this–probably because the questionnaire was getting too long. But now I’m having second thoughts.

    I might also have figured that this varies pretty much based on the brand’s policy. This is worth checking into.

    In a similar vein, I wonder how many dealerships will come get the car from you, leaving a loaner, and later return it to you, so that you never have to go to the dealer.

    What I probably should do is have a separate survey where people report their dealers’ policies in these areas. This way they would have to answer these questions only once per car brand, and I could ask a more comprehensive set of questions.

  • avatar

    Lumbergh21,

    I try to word things very carefully. I never say there is no difference between the Ford and Toyota numbers. What I do say is:

    “a middling chance of avoiding a single additional problem in the third year.”

    Given the stats provided, this is admittedly an approximation. It’s possible that one or both brands have very long tails, such that the median number of problems is actually one. Heck, the median could be higher for Toyota than for Ford, though this is unlikely. Ideally we’d have a frequency distribution–and more–for each. But the whole point of the editorial is that we don’t.

    I don’t believe your interpretation is as valid. A “25% higher incidence of repairs” falls into the problem that problems occur in wholes, and we’re dealing with small numbers. If the scores were 17.9 and 22.4 then, yes, your interpretation would make a lot of sense. But they’re 1.79 and 2.24.

    As I’m sure you’re aware, but that I’m equally sure many people are not aware, the phrase “statistically significant” can be quite misleading. It’s quite common in scientific journals to focus on the degree of statistical significance (i.e. probability that the difference is not zero).

    But the “significant” in “statistically significant” does not mean that the difference between two things is necessarily meaningful. While “significant” is a synonym for “meaningful” in the parlance of our times, “stastically significant” has absolutely nothing to do with meaning.

    Instead, all it means is that whatever difference is being measured in the study actually exists out in the real world. Nearly all of the time, all it means is that the difference is larger than zero, without suggesting how much larger than zero.

    The difference could be 0.01 problems per car and still be statistically significant if the sample is large enough and/or the standard deviation is small enough. But it wouldn’t be meangingful.

    Here’s a (hopefully) clear example. Someone decides to study the lengths of the Subaru Legacy and the Hyundai Sonata. They get a dozen of each and measure them. Every Sonata is 188.9 inches long. Every Legacy is 188.7 inches long. Though the sample sizes are not large, this difference is going to be statistically significant because it is larger than zero and there is no variance. But is it meaningful? Of course not.

  • avatar
    RicardoHead

    >>
    >> Every Sonata is 188.9 inches long. Every Legacy is 188.7 inches long.
    >>

    Did they measure the Sonatas on a hot day and the Legacies on a cold day? ;)

  • avatar
    Lumbergh21

    I guess what I was trying to get at was that the difference in the rate of repairs between the Toyota and the Ford is likely large enough that you could safely say the chances of having a “problem car” or “lemon” when you buy the Ford is higher than if you bought the Toyota. However, it is tru that this is not a given. It is more likely in my opinion that if the average number of repairs is higher then the number of cars requiring more than 2 repairs in the first three years is higher for the Ford than for the Toyota. I do understand that this is not necessarily true as the two cars could have significantly different distribution curves for their frequency of repair data. To reittirate, what I wold really like to see is good data on the average cost of repairs for the car over a ten year period as well as frequency of repairs and days out of service due to repair. I got rid of a Saturn Vue a couple of years ago because I was tired of the number of problems large and small that I had with the SUV. While none of these repairs cost me any money as they were performed under warranty, they did cost me time and aggravation. So, I understand that cost of repairs is not everything as well, though I may have made it seem that way in my first post.

  • avatar

    My point is that, with the numbers they provide, you cannot truly know much of anything.

    Get people into my panel, and I’ll give you exactly what you’re looking for. No inferences or opinions necessary.

    Topped 7,000 cars the other day. Which isn’t nearly the number I need, but it’s a good start.

Read all comments

Back to TopLeave a Reply

You must be logged in to post a comment.

Recent Comments

  • FreedMike: This youtube video proves everything you ever wanted to know about everything. Just watch it....
  • Imagefont: Well you made me look. Now I want the ‘69 white Alpha Spider with the long boat tail – beautiful...
  • jack4x: “I tend to troll for listings for my cars.” I also do this, so I keep an eye on 5th gen Vipers,...
  • Art Vandelay: B13 Sentra SE-R (and NX2000) Z32 300Z E39 540i (I was second on a low mileage one and the winner backed...
  • Old_WRX: I did find the video after all. https://www.youtube.com/watch? v=Du2wm5nhTXY Have you seen this? Comments?

New Car Research

Get a Free Dealer Quote

Who We Are

  • Adam Tonge
  • Bozi Tatarevic
  • Corey Lewis
  • Mark Baruth
  • Ronnie Schreiber