The S2000 is one of my top 3 favorite cars so I can’t argue against it, but I’d like to point out that it’s making about 150lb/ft at 6500rpm. The two engines are as “apples and oranges” as two normally aspirated engines could possibly be.

But it might as well not be there, as few people use it.

You are fighting an uphill battle. In all my discussions with people about cars, most bring up CR as the “Bible.” These people also know the least about cars. They turn up their nose on a car that has a half red circle, or just an average mark. They refuse to admit that the “red dot” supercar that they bought 8 years ago would likely only rate average today. Couple that thinking with biases that brand X has to suck, closed minded fanboys, etc. I really would like to know just what the real difference is just like you. Too bad this thread has been bumped off the home page, into the ether, never to be seen again…wish RF would consider a page two for bumped stuff.

Why are misperceptions and misinterpretations of CR so prevalent, and so deeply believed?

I honestly don’t understand this, and need to. It’s almost like discussing religion or politics. Why?

People tell me that I hate CR. Well, I don’t. I actually have positive feelings about CR as a whole. Where I get emotional (yes, it happens) is when people firmly believe that CR does things it does not do.

If someone says they want reliability information that is a year old, when they can have information that is much more recent, then I have no problem with that. Where I have a problem is when people think they’re getting one thing, but are really getting another.

In this I can sympathize with General Motors, though GM’s case is decidedly mixed. Some perceptions of GM continue to have a basis, others do not. All of them are difficult to change.

BODY INTEGRITY (Squeaks or rattles): Seals, and/or weather stripping, loose interior trim and moldings, air and water leaks, wind noise.

Water leaks can be serious. The rest, not so much. What I’ve never understood is why even list problem types that are never serious in a survey the is supposedly only asking for serious problems.

I don’t think their survey was as detailed in the past, and you might be recalling a past survey.

I’d love to never mention CR, and am generally taking this tack elsewhere. I discuss CR here because TTAC’s readership is more tolerant of facts that differ from “what everybody knows.”

Why mention them at all? Because misperceptions of what CR provides are so engrained, not only in the general public but in the media. As a result people think they’re getting something different than what they’re actually getting, and so don’t see the need to actually get what they think they’re getting. People are unaware of the problem, so they don’t recognize the need for a solution. Unless I help make people aware of the problem–no one else is doing it–then a better mousetrap by itself isn’t doing the trick.

TrueDelta’s results have two major advantages over CR’s:

1. Actual repair rates, so you can see the actual size of the differences between repair rates.

2. Updated promptly four times a year, for results that average over nine months ahead. Is there any other information where one source is over nine months ahead, yet you use the slower source?

The legend you remember reading, and so believe still exists, stated what range of repair rates each dot represented for the system-level dots: engine, transmission, etc. No such legend was ever provided for the overall reliability scores. And a few years ago they tossed the absolute scale for the system level scores as well.

This is covered in the FAQ here:

4.3. How has this approach differed from the way it was done in previous years?
CR has changed the way it presents reliability data, beginning with the 2005 survey.

In previous surveys, the symbol for each trouble spot represented a specific range of problem rates. This allowed readers to make direct comparisons in the rate at which people reported problems in different trouble spots and in different ages of cars. However, this previous approach made it difficult for readers to make sense of whether a particular trouble spot was better or worse than average, and in some cases, it limited our ability to identify unusually reliable or unreliable cars. Also, the Used Car Verdict and Predicted Reliability are relative values that compare a vehicle’s reliability record to the average model of the same age. The absolute scale of the trouble spots and the relative scale of the Verdict caused some confusion and frequent questions. So we changed our analysis in order to represent the data in a way that would be more clear and useful to readers.

To summarize: all of the ratings are now relative, and no actual problem rates are stated, in a legend or otherwise. This permits smaller and smaller differences to be reported as “meaningful.”

http://www.consumerreports.org/cro/cars/new-cars/auto-test/consumer-reports-car-reliability-faq-8-06/overview/0608_consumer-reports-carreliability-faq_ov.htm

In this FAQ they make a few ellusive attempts to respond to my critiques.

For example, the age of the data:

1.5. How current is the data?
All our reliability information is completely updated annually. We begin sending out each year’s survey in the spring. By late summer, we have collected and organized responses, and we complete our analysis and update the information online by mid October. The new information first appears in print in the Consumer Reports Best & Worst New Cars, on newsstands in November. Subsequent auto publications, such as the New Car Buying Guide, also use this new information. In the pages of Consumer Reports, we update Predicted Reliability and Recommendations in the vehicle Ratings beginning in the road tests in the November issue. Changes to new car recommendations are published in the December CR issue and used car results are published in the following April issue of CR. All reliability information we publish is based on subscribers’ experiences with cars in the 12-month period immediately preceding the survey.

What they don’t say in clear terms: the data are from a survey sent out in April 2008, which was nearly a year ago. There’s a reason the FDA requires clear expiration dates and not wordy paragraphs on milk cartons.

6.5. Since the average number of problems is small for most models, is Consumer Reports overemphasizing differences that may not be important?
Beyond statistical significance, we believe these differences are also meaningful to car buyers. We think that car buyers should expect a new car to be entirely problem-free in its first months or years of service. While the difference between a [red dot] and a [half-red dot] may be small, a pattern of several less-than-perfect trouble spots in a brand new car should be cause for concern and does not bode well for a model’s long-term reliability. We have not yet seen a single model in our survey that is entirely problem-free. More than that, one of the worst new models in the 2008 survey, the redesigned for 2008 Chrysler Sebring Convertible, has nearly four times as many problems as the average model, and 20 times as many problems as the Scion xD, which debuted in the same year. Those differences among models are important for car buyers to consider in choosing a car. We present these scores for trouble spots primarily to allow consumers to compare the relative incidence of problems among models. While there are no guarantees, you can improve your odds of buying a reliable car if you choose a model that has had a lower rate of problems in the past.

First, note that they distinguish between “statistically significant” and “meaningful,” recognizing that these are two different things. To prove that the differences are meaningful, they first imply that even a single problem in a sample of 100 cars is meaningful, then compare the two extreme models, and never provide any absolute problem rates. Generalizing from a comparison between two extreme cases to conclude that all differences are meaningful is simply poor logic. Overall, this is called “evading the question.” For other examples, see just about any Presidential debate or press conference.

I only paste the full paragraphs here because the number of words used to say very little is part of my point, and because these might be inaccessiable for non-members. If some can confirm that the above link works for non-members, they can be removed.

I don’t “believe” it, I’ve read it. It’s published in their materials. If I was a CR subscriber and had it handy, I’d type it out verbatim, but as I’m not, I’ll try to check into that later.

Actually, I think rattles and squeaks probably refer to, well, rattles and squeaks.

Yes, but if you are acquainted with CR’s survey, then you know that it uses categories such as “body hardware”, “suspension”, “engine” (or such similar terminology), etc.

Squeaks and rattles can come from a wide number of components, including those working in concert with one another to annoy the user. In addition, they often end up going undiagnosed, with the techs shrugging their shoulders and the problems never being isolated to their source.

Defining this as a category allows the user to complain about an annoyance that doesn’t fall neatly under one of the other labels. I just don’t see the problem here if the information is obtained more easily and reliably by posing the question.

You can see TrueDelta’s survey for the way I prefer to do it–with entirely objective terms. For example, “Was the car towed to the shop?” is both simple, and free of subjective interpretation.

It’s a fair question to ask, but it may have been a subjective choice on the part of the owner that led to the answer provided to that objective question. Towing the car for what turned out to be a dead battery would be quite different than having it towed because it left its transmission on the side of the road, for example. Some people drive cars that should have been towed in, while others tow unnecessarily, so the value of the question would depend upon what information that you’re trying to get. I’d say it’s a good question for determining whether the owner thought that it was worth towing, not necessarily for determining whether it needed to be towed.

All data has some noise in it, and the sample size should help to resolve some of it. CR has a million survey respondents, so the noise filter should be pretty decent.

Getting more to the heart of the matter, CR constructs a pretty accurate and trustworthy survey. The fact that it correlates well with other survey data suggests that it is not an oddball outlier in the world of surveys.

As a member of the public, I am more interested in hearing reasons why that I should consider your data to be at least on par with it than I am in getting a faulty critique of a rival. Your service is not improved one iota by whatever flaws CR has or doesn’t have, and as a data user, I am more interested in what is in it for me than I am in your attempts to demean the competition.

You’re not the first person to believe they provide such a legend. But they don’t. And never have for overall scores.

It’s a relative ranking. If I demand a well-above-average family sedan, then that limits my choices. If I broaden the field to include average or better, I get more choices. The user is provided with the meaning of the relative ranking, as well as a legend that converts terms such as “well above average” into a percentage, and is allowed to make his own decision.

The only percentage provided is the percentage difference from the average. What isn’t even provided as far as I can tell (I’ve looked), though it was sometimes in the past: the actual size of the average. So you might think you know the size of the difference between the cars you feel are acceptable and those you’ve decided are not acceptable–but you probably don’t.

In answer to your question, “For a 2008 model year car, how large do you think the mininum difference is between an “average” car and a “much better than average car,” in CR’s results?”, that answer is provided in their survey results.

Actually, it’s not provided, which is my point.

If all the cars in a class were exceptional, then yes, the curve would lose some meaning. That would be a bit like comparing the IQ’s of a room full of geniuses — everyone’s a winner, some just win a bit more than others, and the discrepancies may not be enough to matter. But just so long as the spread between the highly reliable and not-so-reliable remains statistically significant, as remains the case today, the ranking is still useful.

As explained in my response to Frustratedconsumer, “statistically significant” does not mean that a difference is meaningful.

I don’t know how much clearer that they can make it. They made it simple and fast to read, in order to encourage people to respond to and comprehend the survey. If they added a long list of caveats, most readers would gloss over them or misinterpret them, anyway, and the result would likely be no better. The large sample size should help to filter out that sort of noise.

It is hard to make a survey both clear and sufficiently simple. If there’s anything I’m painfully aware of, it’s this. You can see TrueDelta’s survey for the way I prefer to do it–with entirely objective terms. For example, “Was the car towed to the shop?” is both simple, and free of subjective interpretation. I get a bit more subjective with “could the car have been depedably driven for another week,” but this is short of asking people to report a problem they considered to be serious based on an unspecified amount of downtime, and unspecified cost, and so forth.

Those [rattles] are an area of annoyance for customers, and one that often doesn’t fall neatly into the other categories that they provide, given the numerous possible causes. This appears to be similar to JD Power’s effort to measure some of the less tangible issues in their ranking.

Actually, I think rattles and squeaks probably refer to, well, rattles and squeaks. People who count these as serious generally have a much looser definition of “serious” than those who consider anything that doesn’t prevent the car from running as not serious. JD Power measures things like the difficulty of using BMW’s iDrive, which is a design issue, not a mechanical issue that can be fixed. Two very different things.

Again, that data is provided in the survey. I’m not sure why that you claim that it isn’t. CR does explain its ranking system in its releases.

I claim it isn’t because it isn’t. If it is, just give me the number. I’ve never found such a number in CR.

I don’t why that should be the case, when CR provides a legend that associates the ranking with the percentage of owners who reported problems.

This affects thousands of purchase decisions.

It’s a relative ranking. If I demand a well-above-average family sedan, then that limits my choices. If I broaden the field to include average or better, I get more choices. The user is provided with the meaning of the relative ranking, as well as a legend that converts terms such as “well above average” into a percentage, and is allowed to make his own decision.

In answer to your question, “For a 2008 model year car, how large do you think the mininum difference is between an “average” car and a “much better than average car,” in CR’s results?”, that answer is provided in their survey results.

If all the cars in a class were exceptional, then yes, the curve would lose some meaning. That would be a bit like comparing the IQ’s of a room full of geniuses — everyone’s a winner, some just win a bit more than others, and the discrepancies may not be enough to matter. But just so long as the spread between the highly reliable and not-so-reliable remains statistically significant, as remains the case today, the ranking is still useful.

They’ve always given some guidance, and told people to exclude maintenance. But they don’t provide clear guidance.

I don’t know how much clearer that they can make it. They made it simple and fast to read, in order to encourage people to respond to and comprehend the survey. If they added a long list of caveats, most readers would gloss over them or misinterpret them, anyway, and the result would likely be no better. The large sample size should help to filter out that sort of noise.

Meanwhile, CR has an entire category for “rattles and squeaks.” How many of those would be reported at all with a reasonable definition of “serious?”

Those are an area of annoyance for customers, and one that often doesn’t fall neatly into the other categories that they provide, given the numerous possible causes. This appears to be similar to JD Power’s effort to measure some of the less tangible issues in their ranking.

To repeat the question above, in clearer terms: what do you think the minimum difference in problems per hundred cars per year is between an “average” 2008 model and a “much better than average” one?

Again, that data is provided in the survey. I’m not sure why that you claim that it isn’t. CR does explain its ranking system in its releases.

Your thought process is correct. What we have seen in CR’s results as a result is that fewer and fewer models, even Hondas and Toyotas, earn a “much better than average” rating. This despite the fact that the ratings differ by a percentage of the average, so that as the average declines the differences between ratings has also declined.

Earlier I asked about the minimum difference in problems per 100 cars between “average” and “much better than average” 2008s. No answers yet.

My attitude? I fear you’re projecting. Though I must admit that some of my own frustration is bleeding through–no matter what I write, some people act as if I’ve written whatever happens to be in their heads.

For example: where did I say there was no way to correct for the self-selection problem? What I actually provided was TWO ways TrueDelta does correct for self-selection, but that others do not.

What I did say was that I do not see how CR would correct for respondent bias given the data they have. I know what data they have because I know the questions they ask. And the common anomalies in their results–despite large sample sizes–suggest that they’ve not done nearly enough to adjust for extraneous variables.

Since you said it’s relatively easy to correct for respondent bias, I asked you how this might be done. Instead of providing an answer, you instead attack my “attitude,” which is generally done by someone who know the facts aren’t on their side or does’t really understand what they’re talking about.

Going back to methodology, you trivialize the questions asked and the timepoints. Well, anyone who knows anything about surveys knows that the way questions are worded is far from trivial. Ask the wrong question, and you’ll get the wrong answer, no matter how many people you ask.

As for the timepoints–I already gave some simple examples illustrating why these matter. Having people report what happens after they sign up removes a major potential source of distortion.

One pet peeve: “stastically significant” is horribly overused and misused. To say that a result is “statistically significant” only means that a basic statistical calculation has found that the difference in question is not zero, given a certain level of acceptable error. It says absolutely nothing about the quality of the data, the validity of the analysis, or the meaningfulness of the result.

A result can be “statistically significant,” and still incorrect or, more often, meaningless. Given a large enough sample, even the most minute difference can be “statistically significant.” It only has to be larger than zero. So, oddly enough, a result can be both significant in statistical terms and insignificant by any other measure. But because the lay public has been trained to look for this phrase, insignificant differences can be made to seem “significant.”

So, where’s your fix for respondent bias in CR’s data that doesn’t involve changing their questionnaire?

CR’s data, if I am correct, creates reliability data that is relative to the other samples, correct? So, if that is true, what happens in this hypothetical case:

Only two brands exist, say Toyota and Chevy. And say that Toyota is much better than average and Chevy is much worse. Now, over the course of one year, Chevy manages to make it problem rate match exactly what Toyota has. So, if all the ratings are relative to one another, what is the result? For all of Chevy’s progress, they would move up in CR to average, no more. Toyota, with no loss in reliability, would drop to average. Is this a correct assessment? Of course, there are way more makers than just two, but as the difference in reliability continues to narrow, wouldn’t this become more of an issue? According to most of the raw data that I have looked at, Chrysler’s reliability, compared to earlier Chryslers, has improved quite a bit. But because all automobiles as a group has improved more, Chryslers come across as getting worse which is not true. Comments?

I admit I’m confused.

You admit that TrueDelta has just as much of a self-selection problem AND you admit there is no real way to correct for that.

AND you admit you have no access to CR’s algorithms (or their data.) And thus can’t possibly offer an opinion on what they do. (Other than critique their survey questions)

Yet, you claim your process is superior.

I understand you have a PhD, which explains some of your attitude, but that doesn’t change the truth. TrueDelta – just because it asks different questions at different time points – doesn’t it make it more statistically significant. Especially when your 33K sample is what – a third of CRs?

“Respondent bias is much more correctable than the self-selection bias of the TrueDelta website. Anybody with a stats background would know that.”

Two points:

1. CR’s results are more subject to self-selection bias than TrueDelta’s. They don’t use a random sample, either, and unlike TrueDelta do not require continuous participation or limit reported problems to those that occur after joining (with the limited exception of the current month).

Some all-too-common situations: With CR, an unhappy owner can join and report everything that has happened in the past year. With TrueDelta, they cannot. Similarly, with CR people can respond only when they have a problem to report, and ignore the survey otherwise, and have their responses count every time. With TrueDelta, most such responses will be excluded from the analysis.

2. I have a stats background. Yet, without asking questions CR does not ask, it’s not clear to me how to correct for the respondent biases in their data. Since you suggest this is easily done, how would you suggest they do it?

Easy or not, they don’t appear to perform any such correction.

On the rest you seem to have some blindspots, willful or not.

My “pissing contest,” as you put it, is in their not reporting the actual problem rates. I believe that most people, when viewing just the dots, vastly overestimate the size of the differences between competing car models. This makes those differences seem more meaningful than they actually are in many, even most cases. This affects thousands of purchase decisions. At which point it’s not trivial.

For a 2008 model year car, how large do you think the mininum difference is between an “average” car and a “much better than average car,” in CR’s results? These are separated by an entire “better than average” rating, and are essentially the difference between an A and a C.

My critique of the “you considered serious” wording is not at all misleading. The way they qualify the question has not substantially changed (I’m not sure it has changed at all). They’ve always given some guidance, and told people to exclude maintenance. But they don’t provide clear guidance.

For example, some people will report an alternator failure, others will not by thinking “x brand is generally reliable. This must have been a fluke, and it was covered by the warranty.” I frequently follow up with people to close gaps in their responses. You would not believe some of the repairs some owners consider “not serious.”

Meanwhile, CR has an entire category for “rattles and squeaks.” How many of those would be reported at all with a reasonable definition of “serious?”

Bottom line: permitting each user to decide what counts as serious, with clear guidance concerning what should count ($x, y days, etc.) adds variables such as general satisfaction with the car, past experience with the brand, the general reputation of the brand, and dealer service quality to the analysis, without making it clear these affect the results and without any attempt to adjust for them.

To repeat the question above, in clearer terms: what do you think the minimum difference in problems per hundred cars per year is between an “average” 2008 model and a “much better than average” one?

It’s a relative ranking, equivalent to grading on a curve, with the least troublesome products within the pool getting a higher ranking. That’s an acceptable way of reporting results.

You count visits, CR reports trouble areas. The effort to argue that one method is inherently better than the other on that basis is fallacious. They’re different data points, both are useful and a smart user will consider both, rather than turn it into a pissing contest based upon the type of data array that is chosen.

The survey has people report problems “you considered serious.”

They add some clarification as to how to define it, so I believe that you are being a bit misleading. Their language makes it clear that they don’t want routine maintenance and accident repair to be included in the result.

Updated slowly once a year

The upside to that is that CR places a low burden on the respondent pool, which increases the likelihood of getting a large sample size, which they do. Where CR beats everyone handily is with the size of the sample, and that factor alone should improve accuracy.

Another benefit of the CR survey is that it is short, simple and easy to complete. Long surveys tend to encourage respondents to gloss over points, not necessarily due to dishonesty but just so that they can get it over with.

I have to wonder about some of the details that JD Power provides, for I can’t imagine that they have an easy time actually gathering some of their detailed data. I have no doubt that they ask the questions, but whether people diligently answer them is another issue.

Respondent bias is much more correctable than the self-selection bias of the TrueDelta website.

Anybody with a stats background would know that.

Well said. If you are knowledgeable passionate about some product, CR is NOT the place you look. Camera buff? Into skiing? Bikes, cars, whatever, if you are knowledgeable about it, you realize how lame CR reviews are. Conversely, if you view an automobile as a Maytag and grudgingly deal with car purchases, CR fills that void.

The automotive press offers a much better look into the realm of the automobile. And since you typically can find numerous reviews of the same car, you can take perceived biases into account.

Any halfway intelligent buyer would begin with CR, eliminating some vehicles, and then continue researching using vehicle-specific forums and blogs, speaking to other owners, test-driving all of their remaining choices multiple times and then making their own decision based on anecdotal and empirical evidence. I spent almost an entire year researching the car that I eventually purchased for my wife; the end result, a loaded ‘05 Ford Focus ZX5 SES (every available factory option except the automatic transmission) that she, and I, are very happy with. CR puts a lot of love on the Civic and the Corolla; I disliked the Corolla and found the Civic to be uninspired and boring, but I did check them out based on CR’s recommendation.

Of course, most buyers don’t buy cars in this manner. It’s usually an impulse buy, or one borne of neccessity, resulting in too high a price paid for what is more than likely a total POS.

Consumer Reports is nothing more than one of the many tools at my disposal, and is certainly valuable in that regard, but it’s not the only tool I use. Too many people, though, rely on CR as their only guide to a car purchase.

I don’t think CR is, as you put it, crap. But with any data source–including TrueDelta–you need to look at how the results were determined, including what specific questions were asked on the survey. To this end, I provide more information than others do, for those who really want to know what’s behind the ratings.

The key problems I have with CR, for those who are new to the discussion:

1. The survey has people report problems “you considered serious.” This opens the door wide to respondent bias. For example, people who generally like their car are less likely to report a repair, because they’re less likely to consider a repair as “serious.” CR doesn’t itself bias the reliability results, but respondents might.

2. Just dots, without any way to easily tell how many more repairs one model requires compared to another.

3. Updated slowly once a year, and then presented as if the information is “new” twice a year. TrueDelta’s results average over nine months ahead of CR’s.

None of this renders their results “crap,” just not nearly as good as they could be.

I’ve been arguing with my dad about CR ratings for 25 years. It goes back to when I was trying to convince him to get me an Audi 4000Q. Even then, CR’s definition of reliability was suspect to me. Fast forward 25 years. He calls me to tell me about something he read on the internet how it was crap. He didn’t remember the name, but I was able to tell him: true delta. He ended up getting a lexus after that. Sigh.

Like USNWR’s college ranking, or even BCS, CR’s car ranking mislead you badly. Here is a counter argument. CR has played a huge role in making news cars work until 100k. Honda and Toyota are the benchmark. Quality is vastly better than 20 or even 10 years ago. (with modern electronics, that statement may not be true)

But in today’s age of responsible drivers, we need to recognize that all cars need some tender loving care. Treat your car like an airplane. That way it will get to 200K, instead of 100K.

I tried this argument out when my dad wanted to get an extended warranty with the lexus. I lost again: he bought it. Sigh.

You are correct sir, for some reason I thought that both were built at NUMMI.

“Usually such differences are attributable to different demographics of the owners.”

There’s probably some truth to this. Car A is a garage queen, Car B is hammered and not maintained. One gets a good reliability rating, the other gets a poor one. However, this doesn’t help the person using the reviews because they are not aware of the caveats. In this case, the usual “some information is better than none at all” doesn’t hold true.

“Only if you completely ignore the 5.7liter hemi.”

I’d argue that a motor like that is the very definition of inefficient and unrefined. Chrysler/Dodge et al seem to never have any solution for high power other than massive, heavy, brute force via massive displacement. That they have only two valves per cylinder and twin plugs is 1960s technology. It was awesome in 2.0 liter Porsche flat 6 motors producing 220 hp in 1968. It’s unimpressive in a 5.7 liter V8 motor producing 390 hp in 200x. That’s specific power of ~68 hp per liter.

Give the nod to Honda: its first generation S2000 motor is 2.0 liters, four cylinders, and produces 240 hp, for a specific power output of 120 hp per liter. That’s more specific power than most Ferrari street motors.

Oh, and the 5.7 hemi also weighs an astronomical 485 lbs.