By on September 24, 2019

I can tell by the falling leaves and temperatures in the air surrounding My Old Kentucky Home that the season is here. No, not autumn. Not soccer season or even football season, which are already in full swing. No, it’s the season for the tried-and-true clickfest that is known as the Comparison Test. My friends at Road & Track are doing their Performance Car of the Year test out in California this week. Car & Driver should be doing Lightning Lap any day now at Virginia International Raceway. I’m sure Motor Trend does something similar, but I haven’t read a word of their writing in decades, even before that whole Apple Car nonsense.

The whole idea of the comparison test probably originated right around the time the second car was made — after all, it’s human nature to want to know what the best anything is. For as long as anyone can remember, the Camaro has been telling the Mustang to step outside, the 3 Series and the C Class have been toe to toe, and the 911 and the Corvette have been entering the ring. And if comparing 2 or 3 cars is good, then comparing 10 cars? Well, that’s obviously even better. Hence the mega-comparisons. Not only for performance cars, but for SUVs, too.

When I was younger, I eagerly awaited the results of such tests, treating them as virtual gospel. And over the seven plus years that I’ve been a contributor here, I’ve noticed that many of you do, as well. Any time that I’ve written a review of a car, some commenter will invariably say, “Yeah, well C&D ranked the Maibatsu Monstrosity best in their Midsized SUV Shootout, so your observations are obviously wrong, even though you just spent an entire week in one that wasn’t meticulously prepped by factory techs and accompanied by an on-site mechanic for the duration of the test!”

Well, I’m here to declare an end to all of it. The super comparison test needs to go away. Here’s why.

Before you call me a hypocrite, yes, I fully admit that I have been a tester in a super comparo or two in my day, and I had a great deal of fun doing it. If you haven’t been onsite when a dozen supercars show up to a track, all ready to go against the clock…well, you’ve missed out on something spectacular. I remember when my big brother was called upon to set the official times for R&T‘s Performance Car of the Year three years ago, and it was fantastic to watch.

But there is a sneaky truth about nearly every automotive publication out there today, and it’s a windmill I’ve been tilting against for years. Many of you are making the incorrect assumption that the people conducting the test are in any way qualified to do so. Nothing could be further from the truth. I once heard it said about police officers that there are two things most of them cannot do:

  1. Drive

2.  Shoot

The joke here is obviously that those are the two key functions of the job, and most of them can’t do either.

Nearly the same thing can be said about the majority of automotive writers, except that the two things they can’t do are Drive and Write. Again, if you’ve been a reader of mine for any length of time, this is not news to you. But the problem gets significantly exacerbated when you multiply the variables involved by, say, a dozen. I’ll explain.

Let’s say, for example, that you are asked to conduct a taste test of twelve wines, all of which have a price tag north of $200 a bottle. Let’s also assume that while you may like to have an occasional glass of wine with dinner, you are not truly capable of discerning the small differences between high-dollar wines (as most people are not).

The more wine you drink, the less able you become to discern the difference between them — and not because of the alcohol, but rather the way your brain processes the information. The sheer amount of stimuli starts to overwhelm your decision making ability. You don’t know a lot about wine to start with, and by the end of your sampling session, you feel like you know even less. In the end, you make your decision very differently than you would if you had the chance to evaluate each wine by itself.

Cars are no different. Even for professional drivers, it’s challenging enough to feel the difference between 1.2 and 1.3 Gs of lateral grip, or a 11.8 second quarter mile versus a 12.0, but then you multiply the amount of inputs and information by five or six, and your ability to split hairs disappears. Other thing start to creep into your decision making process, like the color of the car, or the logo on the steering wheel.

So then how are winners determined? You’ll notice that it’s rarely based on actual track times or performance numbers, which makes sense — after all, you wouldn’t expect a Hyundai Veloster N to compete with a McLaren Senna, would you? But then…why are we comparing them? When the Fiesta ST came out a few years ago, it was thrown in the deep end against things like the Audi R8 V10. Even if you do think that the aforementioned FiST or Veloster N are the best dollar-for-dollar performance value, are you really going to vote for them against a million-dollar supercar? How is a comparison between such cars even possible, much less relevant? Is anybody placing both of these cars in his or her consideration set? Peer pressure starts to become a real problem, especially when all of the writers involved start a group discussion. If your boss says he’s voting for a particular car, you’ve got to have some serious stones to vote the other way.

At the end, the awarding of the trophy becomes a nearly completely subjective conversation in which things like sticker price and statistics play a part, but are not hard and fast determining factors. In other words, you’re simply getting the opinion of people who may or may not be any particularly more qualified to make this decision than you are, and they’re doing it under far less than optimal conditions with cars that are factory-tech prepped and maintained throughout the competition — or, as some might call them, ringers.

It’s all excellent entertainment, which is why the super comparison tests will never stop, and we won’t stop clicking on them. So, maybe I will recant my call for them to end, as long as we all just try not to take them so seriously. Okay?

[Image: Sergey Akhrameev/Shutterstock, Ford, Mazda]

Get the latest TTAC e-Newsletter!

Recommended

32 Comments on “Bark’s Bites: Don’t Care, Don’t Compare...”


  • avatar
    FreedMike

    Pretty much agree – these are entertaining but useless. However, I would add:

    1) Some of these tests reveal some really embarrassing stuff, like the Tesla Model S that suffered an intermittent brake failure at 137 mph at a C/D Lightning Lap a few years back. Good to know irrelevant stuff like ***BRAKE FAILURE AT 137 MPH*** happens when the car’s stressed, you know?

    https://www.caranddriver.com/features/a15100055/tesla-model-s-p85d-at-lightning-lap-2016-feature/

    2) C/D groups the cars into classes, so the Accord 2.0T Sport is competing against similarly priced cars, not a Ford GT.

  • avatar
    quaquaqua

    I lived for the comparos when I was a kid. But that was the 80s and even the Consumer Reports reviews were SO entertaining to read. But that was the era where any comparison would inevitably include mostly crappy cars, and it was almost a race to the bottom to see which one was slowest, which one came with the worst defects, etc.

    C&D is still around and doing well, but the fact that all the cars they test are hand-delivered by the manufacturers that they take advertising money from really means you’re not gonna see much in the way of useful criticism. And anyone who thinks otherwise is kidding themselves.

  • avatar
    NeilM

    “Nearly the same thing can be said about the majority of automotive writers, except that the two things they can’t do are Drive and Write.”

    Well, your brother seems to do both pretty decently. So does R&T’s Sam Smith, and a whole bunch of others that could be listed. Then there are past journalist legends such as Paul Frère, or LJK Setright. The lesser online names at Jalopnik, or — dare we say it? — TTAC, well, who the hell knows.

    However the one thing they all have in common, something that most of us don’t, is the opportunity to sample a wide variety of vehicles, from the exotic to the mundane. That alone affords a useful perspective.

    So I call BS on your sweeping generalization. Like most such it’s true in part, untrue in other parts, and vaguely in between for the rest.

    • 0 avatar
      tonycd

      I’ve been a C/D reader since the Sixties. They had some pretty self-indulgent crap in about the ’80s or 90s, but they’ve been solid now for a number of years. I don’t buy that the only fine writers were in our hazy nostalgic past. I think C/D has several fine writers, including their recent addition Annie White. Phillips and Ezra Dyer are especially gifted with words. And they do critique the cars, provided you calibrate your expectation meter and read their 5.0 criticisms as an 8, which isn’t that hard to do.

      Personally, I find comparison tests very useful. Camaro vs. Mustang Part 96, who cares. Okay. But when the new Camry came out with its new platform and costlier suspension, everybody hailed it as an improvement. But the real question for me was, yeah, but has it caught up to Accord and Mazda6 over the road? The comparison tests answered that question: No. Sometimes there’s just no substitute for them.

  • avatar
    haroldhill

    The worst thing about comparison tests is that the parameters are overwhelmingly irrelevant to whether you or I are going to enjoy owning and driving the car. We’re obsessed with them partly because we’re trying to get a preview of whether or not a given car would be a good investment in our time and money. In the end the joy of driving a car boils down to enormously subjective qualities, such as the position of the gear shift, the response to small adjustments in pedal pressure, the weight of the steering, the seat (with 21 ways to make it miserably uncomfortable), and the ambiance – e.g. whether we feel special in it or whether it tickles our pragmatic, minimalist nature. No-one else’s writing can convey these answers; nor can a half-hour test drive. I’ve had great success chosing (and eliminating) future purchases via the rental experience. But never by reading comparison tests.

    In the meantime, can you post links to the comparison tests you referenced?

    • 0 avatar
      Arthur Dailey

      Well said.
      I do agree with Bark’s premise which is the reason why I first turned to TTAC. ‘Automotive journalism’ is largely an oxymoron. The testers rarely if ever address the issues/concerns of the actual new vehicle purchasing public. The act primarily as
      ‘shills’ for the manufacturers. Wittingly or unwittingly.

  • avatar
    MrIcky

    Those mega tests are amusing, and I used to think more of them than I do now. The problem is that unless you are actually planning on tracking a car, those tests are fairly meaningless. When the most humble car on the VIR megatest can pretty safely double the yellow ‘suggested speed’ for turns signs on real roads, it just is kind of bleh. I’m sure the SCCA crowd gets excited but then they just buy another miata- probably used.

  • avatar
    stingray65

    Who needs to drive or inspect when you can simply use the comparo test template:
    1. BMW has grown soft – no longer ultimate driver’s machine.
    2. Audi has wooden steering and/or ride, but fantastic interior.
    3. MB has bank vault doors and has grown into a hot rod, but some bits feel flimsy.
    4. Cadillac has great handling but no backseat and some lapses in quality.
    5. Japanese have great build quality, but no fun to drive, ugly styling.
    6. Koreans offer great value for money, but missing last 5% of refinement and brand lacks cachet.
    7. Other Americans (Ford, GM, FCA) mediocre build quality and suspect reliablity, but can offer great performance for the money.

  • avatar
    SPPPP

    I know it’s a joke, but if we really believed that the two key functions of a police officer’s job are to drive and shoot, that would be an indictment of our entire society. On TV, that might be true. But even on Starsky & Hutch, there was more thinking, running, shaking crooks down, and fighting than shooting.

    And in real life, those things are sometimes needed. But I am going to take a moment and reflect on how glad I am that guns are in fact not used in the vast majority of police interactions. Even in Texas.

    • 0 avatar
      Dan

      Never mind the vast majority of interactions, the vast majority of police work an entire career without shooting once. There are something like 850,000 cops in the country. There are about 3000 police shootings per year resulting in about 1000 deaths.

      • 0 avatar
        Arthur Dailey

        The majority of time spent by law enforcement officers, is on tasks/duties that could be completed by social workers.

        And the selection process used by a number of police ‘services’ reflects this.

        • 0 avatar
          slavuta

          In Boston, an officer must stand next to big pothole until it gets fixed.

        • 0 avatar
          Robotdawn

          I agree with the social worker aspect. I think of my father-in-law as a good cop because that’s pretty much what he is. A social worker with a badge. Problem is, most of the people he deals with wouldn’t listen to a regular social worker, and social workers can’t be bothered with real social issues (can’t half blame them either).

          I’d much rather he be doing what he is than having TV style shoot-outs.
          (He’s fired his weapon in anger once in 25 years.)

  • avatar
    thejohnnycanuck

    Does anyone still read Motor Trend? I know I haven’t been there since they got all butthurt and disabled their comments section.

    • 0 avatar
      SPPPP

      That irks me as well. What are they afraid of? Did I miss something big? I think they have shot themselves in the foot by disabling it – how much engagement do they expect for each story?

      • 0 avatar
        ToddAtlasF1

        I unsubscribed from their email, but now their magazine is showing up at my place. So does Car and Driver. I’m reasonably certain that they don’t have access to my credit cards, so they must be hard up for the right sort of subscribers. I believe Car and Driver started delivery just as I finally got Automobile to stop. The last MT I looked at had an article about what their staff would buy. It was bad enough trying care about their articles before I knew anything conclusive about their writers.

        • 0 avatar
          redapple

          Motor Trend is evil stinking trash. Like PBS’s Motor Week. The ‘reviews’ are milk toast. Bland crap.

          Road & Track is still the gold standard. More Peter Egan !!!

  • avatar
    slavuta

    ” Many of you are making the incorrect assumption that the people conducting the test are in any way qualified to do so”

    Well, I am not one of these. I disagreed with many “great reviewers”. Never trust! Try yourself! And I disagree that you can’t tell the difference after testing many cars. May be in the features – yes. But driving – no way. You can quickly tell, which cars you like to drive and which – not so much. I remember, when I tested bunch of cars and even wasn’t against some of them, but when got into Mazda3, I knew – I am buying this one. Same thing was with with Highlander this year. We test drove 5 other SUVs and it wasn’t hard to tell that HL was better suitable to drive than others.

  • avatar
    stuki

    Even sillier is that most tests focus on how “good” the cars are at doing sommersaults out of a plane, pulling a plow or getting around a scorching hot California Racetrack, despite no buyer ever buying them for such purposes, nor any engineer ever bothering all that much engineering for them.

    As for “being qualified,” the qualification which matters, is largely similarity of intent and preferences to the intended reader. Which is why constancy in employment matters. Over time, you learn to calibrate phenomena a writer describes in a certain way. Wrt motorcycles, whether I agree of disagree with him, I “know” what John Burns means when he writes something, as I “grew up” with the guy, and have ridden many of the same bikes over the years. Which is very different from how I treat a review by some random new dude.

    Another bias required in order to put much stock in magazine reviews, is that magazine reviewers are somehow able to discern “flaws” which the guys engineering the cars somehow do not. Which experience have taught me is rather rare, especially wrt bikes, which are pretty much all designed for “enthusiasts”. Wrt cars, being “an enthusiast” almost inevitably puts one away from the main targets most models are engineered for. Hence, an opinion form a fellow “enthusiast” may shed some basic light on exactly how short a shrift typical “enthusiast” concerns have gotten by the design team. But as long as I am shopping for an “enthusiast” car, I have found very little correlation between reviewer ratings, and which cars I personally prefer. The guys designing and test driving “enthusiast” focused models, know their stuff. Or they’d have another job. Perhaps at a magazine.

  • avatar
    eggsalad

    While I agree the writing is bad and the expectations of mega data processing by the writers is difficult, I would argue that comparison testing is *more* relevant today than in the olden times.

    The basis for this argument is that vehicles have become more and more similar over time. Every manufacturer is making essentially the same cars as its competitors. You want a 2 liter, 2 row CUV? There are seemingly dozens to choose from. I want to know which one has comfortable seats for road trips. I want to know which ones have goofy controls, or frustrating infotainment (gah, I hate that word!) systems.

    I can’t get all those answers from a 7-minute test drive at a dealership, and those are the answers I *need*.

    • 0 avatar
      hubcap

      The thing is, people have differing preferences. What may be a comfy seat for you, isn’t for someone else. Person A might be fine with a particular user interface while person B finds it clunky.

      There’s just no substitute for putting in quality time with the vehicle(s) you plan to spend your hard earned money on.

  • avatar
    Vanillasludge

    I lump car journo’s in with strippers, pro wrestling and politcal debates. There for your amusement, but best not taken seriously.

  • avatar
    redapple

    Good article.
    Gave me something to ponder that I never considered before.
    You know, that s what good writers do.

  • avatar
    MeJ

    While I agree with most of what was said, I have to add that sometimes the best thing to do is to compare the comparo’s and find any common thread. I would guess that if 9 out of 10 car journalists think a certain car is amazing and a good value (I’m thinking of the Honda Civic right now) then it might very well be worth a look.
    I would imagine that’s why the Miata and Porsche 911 (as well as a few others) always make it to the top ten lists.

  • avatar
    slavuta

    Oh, yea. Alex Sykes. This guy triggers me every time he suggests that the value of the car lies in how many **safety features** /nannies/ it has. For me, this is opposite.

  • avatar
    ToddAtlasF1

    A recent Motor Trend comparison test was between the RAV4 Advertisement and the Fiat Trailhawk. They tried finding something the Jeep could do off road that the Toyota could not. They failed again and again. First they would drive through an obstacle in the Jeep, only to follow it successfully in the RAV4. Finally, they got the Jeep stuck. They said they knew better than to try the hazard the Jeep couldn’t traverse in the Toyota and declared the category a win for the Cherokee.

    I’ve walked out of the woods twice because of mistakenly going there with people for whom it is a Jeep thing. I’ve never had to abandon a rented Toyota sedan, and I’ve taken them places where most transportation is accomplished on legs instead of wheels. To be fair I’ve also proven the ‘rental car as ultimate ATV theory’ with a Chevrolet Spectrum/Isuzu I-Mark. It’s possible that it is easier to get stuck in a RAV4 than a Tercel or Yaris, as the RAV4 is so much bigger and heavier. A RAV4 is still far less likely to be stopped by water or mechanical failure.

  • avatar
    millerluke

    I wish they’d include ‘predicted reliability and maintenance’ in the reviews. Sure, the Porsche is great driving, but when you have a $3000 oil change (I’m exaggerating) it makes it not so good. Or, when you have a new Camry and the tranny dies at 200km (as happened to me), it takes the glow off the good things about the car.

    Of course, if dealers weren’t so stingy with their test drives, we probably wouldn’t need the comparisons…

    • 0 avatar
      hubcap

      As I’m sure you realize your Camry experience was an outlier. It’s predicted reliability and maintenance would be among the best in the entire car universe. I doubt it would sway anyone from purchasing one.

      I said this but you’re right. The more information, generally the better.

  • avatar
    Prado

    Car reviews, comparison or not, are only good for a few things to me anymore. First, performance stats and technical details … they set my approximate expectation on what a car will be like to drive. Then likes and more importantly dislikes. This gives me the things to pay attention to, when I drive a car myself, to determine if those dislikes are issues to me. Everything else is pretty much just personal preference.

Read all comments

Back to TopLeave a Reply

You must be logged in to post a comment.

Recent Comments

  • Dale Houston: If I was buying for myself, I’d probably get the 3 over the CX-30. But our next car purchase is...
  • Pete Zaitcev: Every time I see the 2gen Ridge, I am reminded of the TFL episode where they just drove some trucks on...
  • jack4x: Being an engineer myself, I do have a lot of sympathy and respect for the folks who tried to come forward...
  • TR4: Remember when Quality was Job One? Oh, wait…
  • Dale Houston: Sales?

New Car Research

Get a Free Dealer Quote

Staff

  • Contributors

  • Timothy Cain, Canada
  • Matthew Guy, Canada
  • Ronnie Schreiber, United States
  • Bozi Tatarevic, United States
  • Chris Tonn, United States
  • Corey Lewis, United States
  • Mark Baruth, United States