By on October 13, 2016

Tesla Model S 85D, Image: © 2016 David Marek/The Truth About Cars

Tesla CEO Elon Musk vigorously defended his Autopilot system when accident reports rolled in earlier this year. Even when a fatal Florida crash was blamed on a temporarily blinded Autopilot, Musk assured citizens, Tesla owners, regulators and everyone else that the semi-autonomous driving system made his vehicles the safest things on the road.

Just do the math, Musk told the skeptics. Well, someone finally has.

Writing in Green Car Reports, David Noland tackles Musk’s claim that Autopilot-equipped Teslas handily beat America’s conventional vehicle fleet in terms of safety. By unpacking the claim, Noland shows why broad, hazy statistics trotted out as definitive proof often aren’t what they seem.

Critics pounced on Tesla and Musk after the fatal May crash, with many demanding that the system be pulled off the market until it could be proven safe. The National Highway Traffic Safety Administration and National Transportation Safety Bureau launched investigations. Musk shot back with numbers, claiming in a blog post that one fatality in 130 million Autopilot-driven miles beats the U.S. average of one death per 94 million miles driven.

On the outside, that seems pretty clear. Wrong, says Noland. Musk’s sample size — which officially stands at one death per 222 million miles — is just one of the problems with his claim.

“We’re talking here about the smallest possible sample size: one fatality,” Noland writes. And what about that videotaped fatal crash in China earlier this year? If that proves to be Autopilot’s fault, it doubles the system’s fatality rate.

Instantly, the “half a million lucky folks around the world allegedly saved by universal Autopilot would suddenly all be dead again,” Noland writes.

Counting the second fatality and using Musk’s 222 million mile claim, that makes for a fatality rate of one for every 111 million miles driven. That isn’t too far from the 94-million-mile average.

Many proponents of self-driving cars claim automation will reduce deaths, but it’s hard to back up that claim. On the subject of self-driving vehicles, the Rand Corporation claims, “Autonomous vehicles would have to be driven hundreds of millions of miles, and sometimes hundreds of billions of miles, to demonstrate their reliability in terms of of fatalities and injuries.”

That sample size issue again…

Noland then digs deeper into the stats on both sides. Musk apparently gets his 94 million figure from the NHTSA’s Fatality Rate per 100 Million VMT (Vehicle Miles Traveled). And guess what those fatalities also include? Pedestrians, cyclists, bus passengers and truck drivers.

Not only that, passenger car occupants made up only 36 percent of the country’s traffic fatalities last year. A boatload of Teslas could sail through the holes in Musk’s methodology. MIT Technology Review is one of the publications slamming his stats, claiming the comparison “has no meaning.”

There’s other factors keeping Tesla’s fatality rate low. Most Autopilot miles are traveled on major highways, in good weather, with older people behind the wheel (they can afford the vehicle, you see). So, Musk’s stats are drawn from the safest roads, driven in the safest weather, by the safest demographic of drivers.

Measured another way, Tesla’s vehicles are far more dangerous than an average car. The Insurance Institute for Highway Safety uses a different measure: driver deaths per million vehicle-years. The existing American fleet, it says, averages 28 deaths per million vehicle-years.

Using the average number of miles driven by Americans each year, Noland figures that works out to 28 fatalities per 12 billion miles driven, or one death per 428 million miles. Remember Tesla’s rate, assuming the Chinese death is Autopilot’s fault? One fatality per 111 million miles. Four times more dangerous than average.

Maybe Musk should cool it with the safety statistics for now.

[Image: © 2016 David Marek/The Truth About Cars]

Get the latest TTAC e-Newsletter!

42 Comments on “Musk’s Math: Tackling Tesla’s Dubious Autopilot Safety Stats...”


  • avatar
    orenwolf

    Two flaws in this argument:

    1 – it fails the “sniff test” – how many collisions are caused by distracted driving, a category not relevant to automation, let alone the category of collisions caused by human reaction time not being sufficient to respond? I personally refuse to believe removal of these two categories of collisions, if prevented by automation, would not be a net benefit
    2 – 222M miles is the largest sample size of automation, ever. Despite that, it’s only a tiny minority of vehicles. As the article points out, we need many more on the road to tell us the full story. And for that, I hope the other vehicle manufacturers will be as transparent as Tesla has with numbers, for the good of everyone. I doubt it, though. :/

    • 0 avatar
      Pch101

      Referring to misleading cheerleading as “transparency” is a joke.

      There simply aren’t enough miles traveled in order to judge Tesla, either way. Based upon the 2014 fatality rate of 1.08 fatalities per 100 million miles, one would expect 2.4 fatalities over the course of 222 million miles.

      In the real world, the difference between one, two and three fatalities is essentially a fluke.

      And as noted above, the miles driven in a car using Autopilot are the safest miles on a per-mile basis — limited-access highways have the lowest fatality rates and large sedans have below-average fatality rates — so the use of the overall 1.08 figure for a basis of comparison is itself optimistic.

      The way to assess the Autopilot system at this juncture is the extent to which it increases and decreases activities that contribute to crashes. If it has difficulty recognizing hazards and fails to respond accordingly while creating a false sense of security, then that’s a problem.

      • 0 avatar
        orenwolf

        “Referring to misleading cheerleading as “transparency” is a joke.”

        Tell me which of the other five car companies that support Lane Keeping as a feature on their cars has provided *any* data on the miles travelled in auto mode, or accidents therein, and then we’ll talk, sir.

        I’ll wait.

        • 0 avatar
          Pch101

          You’re actually excited that Musk is using inadequate data to mislead people as Tesla attempts to deflect any responsibility that it may have for a fatality. How did you get to be so gullible?

          • 0 avatar
            orenwolf

            Yes, I am excited that an auto company is releasing data on miles travelled in autonomous mode and on the number of fatalities in said mode, because I feel the data is important to the industry as a whole and that other manufacturers should, IMHO, follow suit. I can’t see how or why you would disagree with that stance, but I suppose anything is possible.

            If you feel the data is somehow meaningless because the data is being used in a way you feel is inappropriate, well I strongly suggest you avoid all datasets ever. :)

          • 0 avatar
            Pch101

            If you would bother to read the MIT Technology Review article that is referenced above, then you would understand why this is bogus and misleading.

            But I suppose that it must be difficult to click the link with those pom-poms in your hands.

          • 0 avatar
            orenwolf

            See, there’s the problem. You can’t separate your hatred of Tesla from the facts. The MIT article (indeed, the article in general) does not question whether Tesla’s statistics – deaths vs miles driven – are genuine, it questions the interpretation of those stats.

            Look, I get that this is probably the only website in the world that would slam car manufacturers for releasing data, but it seems to me you are conflating the apparent misuse of facts for an agenda – an aspect of this article *I haven’t even mentioned*, with the fact that we have data from a car manufacturer on miles travelled under automation, which IMHO is 1) unique and 2) significant.

            If applauding a manufacturer for releasing hard data, not just spin on the numbers is now cheerleading, then I genuinely wonder what you believe the “truth” in TTAC means exactly.

            Perhaps you believe car manufacturers should NOT be releasing hard data on automation, because the data can only be used to draw false conclusions or something?

            But please, go ahead and continue making fun of my position. I’m happy to stand up and defend being given raw data independent of spin, and will applaud ALL manufacturers for doing so regardless of what conclusions they draw from those numbers. Because those numbers give places like TTAC the ability to highlight independent analysis, and we are *all* the better for that, AFAIAC.

          • 0 avatar
            Pch101

            “I get that this is probably the only website in the world that would slam car manufacturers for releasing data”

            Er, you need to work on your reading skillz, because nobody did that.

            The problem is with Musk spinning the data, misinterpreting it, and using it to mislead people.

            I already explained above how it was misleading. MIT makes some points along similar lines.

            Meanwhile, Orenwolf the fan boy can’t figure out that misleading people is a bad thing. Elon Musk could walk up to a random guy and shoot him in the head, and you’d find some way to defend it because Tesla.

          • 0 avatar
            orenwolf

            See, the problem here appears to be that you believe events can only be about one thing. Like, if there’s some calamity in the world it’s bad form or something to talk about lesser events?

            The reason TTAC can even have a discussion about the numbers is because the data was released. Now, it’s clear that you believe they are being misused and therefore that is the only appropriate subject to comment on, but I believe it is relevant to point out the transparent release of the data itself and call out other manufacturers for not doing so, as without the data, this article could not exist.

            Yes, fanboy of the release of data, and quite happy to ignore the noise and praise the signal. I’m sorry if that is offensive, though, but I would equally praise *any* manufacturer that released raw data on autonomous miles driven. So.. I guess that makes me an autonomous vehicle data fanboy? Sure. I’ll take that title.

          • 0 avatar
            Pch101

            Musk released numbers because he knew that fanboy buffoons would misinterpret those numbers to Musk’s benefit.

            If you understood those numbers, then you would realize that they don’t say what Musk claims that they’re saying. But you revel in your ignorance and no one on the internet is going to fix you.

          • 0 avatar
            orenwolf

            Musk released numbers. No other auto manufacturer has done that.

            I understand that the article is about whether or not his conclusions are valid. I have given no opinion on them. If you wish to make your own assumptions on what I think of them, you are welcome to, but that wasn’t the point of my comment.

            My point was, that no one else has released this information, and that, even removed from the conclusions drawn by Musk, the numbers are valuable in their own right because they represent a transparent look at autonomous miles driven.

            Now, you may prefer that, like Audi, Volkswagen, Volvo, Mercedes and Infinity, Tesla/Musk not release their numbers, so that instead, one can only speculate on how many autonomous miles have been driven or how many deaths have occurred with those systems. Or you may wish to assume no one is using those systems, or that they have been perfect. I don’t presume to know your motives here. But *my* point is, there are five (well, 4.5 if you consider Audi/VW one org) also using this technology and NOT sharing details, which 1) makes analysis difficult and 2) sucks from a transparency standpoint.

            You are welcome to keep conflating the fact that I think release of data is good with the belief that I think the conclusions from Musk are valid. I’ve said no such thing, but I can’t stop you from believing what you want to believe. But I will continue to defend the release of data transparently and hope that other manufacturers will follow Tesla’s leadership in this way.

          • 0 avatar
            Pch101

            Musk didn’t just release numbers, he spun and misinterpreted them in an attempt to avoid liability for the Florida crash.

          • 0 avatar
            orenwolf

            “Musk didn’t just release numbers, he spun and misinterpreted them in an attempt to avoid liability for the Florida crash.”

            While that may be true, the release of the numbers, independent of the spin, is significant because no other auto manufacturer is doing that.

            The reason we can even *comment* on the effects autonomous driving is having are due to those numbers. If Tesla was doing what any other Manufacturer with this tech was doing, we’d all just be speculating without any hard data. To me, that’s far more important than any spin one person or one company places on the data.

            We should *all* be *requiring* that manufacturers with autonomous (Lane departure or better) technology regularly release their information on miles travelled. That’s the only way that we can build up a relevant dataset of how these technologies are working in the wild.

            My fear is that no one except Tesla has the telemetry required to even get this information. Maybe the Mercedes vehicles uplink statistics routinely, but I doubt anyone else is. That’s extremely unfortunate, because really the discussion can only be “How does Tesla’s autonomous technology fare”, not “How is autonomous driving technology performing in the wild?”, and I am MUCH more interested in the second question than the first, especially as more manufacturers adopt the tech!

          • 0 avatar
            Pch101

            And I have explained why the numbers don’t tell you anything.

            If you’ve successfully completed a university-level undergraduate statistics class, then this should not be difficult to grasp.

            This is akin to Michael Karesh’s various putdowns of Consumer Reports and JD Power. He likes to brag that’s he’s first and best because he releases his results early. What he neglects to mention is that his data is unreliable because his sample sizes are too small. Anyone who understands the basics knows that small samples produce large margins of error, which renders them unreliable.

            You are a willing dupe. There is obviously no fixing you. The math issue here should be obvious, yet you refuse to see it because you are blinded by your love for all things Tesla. If you applied your logic to a stats class assignment, then you would get a “F” and deserve it.

          • 0 avatar
            orenwolf

            “And I have explained why the numbers don’t tell you anything.”

            If you believe that miles travelled in autonomous mode and fatalities “don’t tell you anything”, then clearly we have no more to say to each other.

            However, I fully intend to continue to press for all manufacturers to release those numbers in the future, regardless of your position on the matter.

            Thank you for your time.

          • 0 avatar
            Pch101

            There haven’t been enough miles traveled to provide a reliable sample, because 200 million miles worth of travel in cars doesn’t generally produce a whole lot of deaths in any kind of car.

            How is this basic fact so difficult for you to grasp?

        • 0 avatar

          It may not be a transparency issue. They may not have all the data. Tesla track everything in minute detail.

          • 0 avatar
            orenwolf

            Indeed. There’s a very good chance that is the case. We’re going to have nothing but “estimates” from the rest of the manufactures for some time, at least until all vehicles are connected and can upload even basic telemetry.

            I’m actually surprised it’s taken so long – I figured OnStar and its’ ilk would have made connected vehicles standard by now, at least on all but the most basic models.

    • 0 avatar
      Kenmore

      “I hope the other vehicle manufacturers will be as transparent as Tesla has with numbers, for the good of everyone.”

      Orenwolf, please use a different avatar.

      You’re making chicks seem ditzy by association.

    • 0 avatar
      WRohrl

      Orenwolf didn’t just drink the kool-aid, he literally fell into the bucket and is currently drowning in it…

  • avatar

    There is another major factor that you’re missing that skew the Autopilot results. Autopilot only exists in Tesla’s – relatively new and expensive cars with the best safety systems. The figures that the Tesla Autopilot is being compared to represents all other cars on the road – two year old Volvo’s that are pretty safe, but also 10 year old Fords with poor tires and even 25 year old tiny little corollas with no brakes and Yugo’s that may have lasted the test of time.

    Autopilot has the distinct advantage of only being in new, expensive and safe cars, its deaths per million miles skewed by the fact its guaranteed to be surrounded, in virtually every car it operates, with state of the art mechanical, tire, brake, personal restraint, and safety systems.

    • 0 avatar
      rev0lver

      Also, the average Tesla driver is probably more likely to:

      1. Live in an urban area
      2. Live in a more temperate climate
      3. Older than the average driver

      Etc etc.

      Short answer. Tesla owners do not represent the average North American driver therefore any conclusions based on data collected from them is not generalizable.

  • avatar
    SCE to AUX

    First you say there isn’t enough data to make a claim… and then you make a claim.

    Then there is the pesky fact that a NHTSA Level 2 system *requires* an attentive driver.

    Tesla’s system may have its flaws, but the drivers own some of the blame.

    • 0 avatar
      ToddAtlasF1

      So you’re saying it is okay for Musk to make his claims because anyone challenging them would be relying on the same small pool of data that he is? I think the revelation here is that the mass-murdering passenger car only kills its occupants 36% as often as social engineers would have us believe. Maybe we don’t need to do anything about a freedom-granting technology that isn’t broken. People opposed to human-controlled cars are the ones completely full of it, and Musk can harvest his sheep as long as his creations don’t kill anyone worthy enough not to take the subsidies.

      • 0 avatar
        Erikstrawn

        Actually, he’s saying that whining about the numbers Musk is using to make his claim, then using those same numbers to make your own claim is hypocritical. I agree.

        I disagree with the assertion that Musk’s cars are killing people. The drivers were told this was beta testing and they should remain in control of the car. It’s also stupid for Tesla to expect the drivers to follow simple instructions.

        Self-driving cars are coming. All you luddites can whine about your loss of freedoms, but it won’t change things. People in my neighborhood still ride horses whenever they want, it just requires more money and interest to participate in the hobby. Driving in the future will be the same. It’ll require money and a strong desire to jump through all the hoops of owning and racing a car.

        • 0 avatar
          ToddAtlasF1

          Did you know that it wasn’t legislation that ended horse-provided mobility? It was a superior solution. Do any autonomous car grifters intend to wait for the market to demand that roads be reconfigured to suit their agendas? Civilizations are enriched when privileges become broadly available. They’re impoverished when rights become privileges. Maybe social unrest will shatter your delusions of grandeur. It might suit you.

          • 0 avatar
            orenwolf

            I’m not certain, but I’m fairly sure that horses were never the top cause of accidental death in a given country.

            I’m all for personal freedoms, but not if they put other lives at risk. Now, maybe we’ll find that autonomous vehicles aren’t safer. I personally doubt it, but that could be what we find. However, if instead we find that we can save tens of thousands of lives by restricting non-autonomous driving, then I’d support that, because I don’t believe my enjoyment of driving trumps others’ lives.

            That being said, I seriously doubt anything of the sort will happen for a very, very, very long time. We need a LOT more data before we can even come to this conclusion. Until then, I think we owe it to the drivers to the future to try and make roads safer for everyone, at least until we determine if autonomy is having an effect or not.

          • 0 avatar
            ToddAtlasF1

            Disease was common in cities because of horse droppings. Cars have saved more lives than they’ve cost. Just look at mortality rates before and after their spread. Transportation is always a high cause of accidental death. Try traveling by sailing ship, or steam locomotive, or…horse. If you want to die from fecal disease or packed into a hellish ghetto that’s on you. I’ll take my sedate suburban life and freedom of mobility. Some things are worth fighting for.

          • 0 avatar
            Kenmore

            My horse droppings… painted like EASTER EGGS!

            I’m so ANGRY!

          • 0 avatar
            Pch101

            In our urbanized environments, cars are an improvement over horses. But that isn’t particularly relevant to today’s situation. It’s not as if the alternative to a gas-guzzling pickup is a late-model Appaloosa.

          • 0 avatar
            nasch

            “Cars have saved more lives than they’ve cost. Just look at mortality rates before and after their spread.”

            Don’t confuse correlation with causation. Many other things happened during the spread of the automobile.

    • 0 avatar
      SSJeep

      Totally agreed SCE to AUX. Having driven both an Autopilot equipped Tesla and a handful of “semi-autonomous” vehicles with radar based collision avoidance and speed control, I can most assuredly state that the largest flaw in the system is the driver. Automatic driving systems take time to learn and more time to master.

      Say, for example, I am running my vehicle with radar based cruise and lane assist enabled. For the duration of time that those controls are on, I dont need to manipulate the steering wheel or brake. The vehicle does it for me. If traffic comes to a stop, those features are turned off by the computer. When I start going again, I may reflexively assume that those features are still enabled, which is a dangerous mistake.

      Semi-autonomous driving features make the act of driving itself much easier, especially in traffic. But they are no excuse to take one’s eyes off the road and hands off the wheel. That is exactly how accidents happen in these situations.

  • avatar
    bikegoesbaa

    Is the MIT Technology Review a professional wrestler?

    If not, why are they described as “slamming” anything?

    • 0 avatar
      Pch101

      The article’s use of terms such as “dubious”, “ludicrous”, “no meaning” and “apples and oranges” isn’t exactly high praise for Tesla. Given that the MIT Technology Review is not a publication that is inclined to resort to hyperbole for the sake of it, I would say that “slam” is appropriate.

  • avatar
    MeaMaximaCulpa

    And if we dive deeper in the figures (sh*tty sample size and all) shouldn’t the tesla with the autopilot be compared to cars of the same type, age and price? The American car fleet as a whole makes for a horrible comparison does it not?

    So let’s see, the oldest Tesla that’s fitted with the hardware required for the autopilot was manufactured exactly two years ago, the Tesla is a luxury sedan (or hatchback) with a base price around, what, $ 75 000.?So is the average car in the group “luxury sedans that were manufactured in the last two years and costing in excess of 75 k when new” safer then the average American car and/or the tesla?

    • 0 avatar
      stuki

      Of course, you could also take the tack that as long as Autopilot is not obviously less safe than the single least safe human driver that is not banned from the road, why ban Autopilot?

  • avatar
    RRocket

    Or you could use the IIHS methodology which is “per million registered vehicles”. When reviewing that data I was extremely shocked to see 9 different vehicles with ZERO deaths for 3 years. Pretty amazing considering some of them are large volume (Lexus RX for example).

    Anyways, if you use the “per million” calculation, these are all the vehicles that have fewer fatality rate than the Tesla

    Jeep Compass 2WD
    Jeep Patriot 4WD
    Honda Element 4WD
    Subaru Legacy 4WD
    Acura TSX
    Volkswagen CC
    Audi A4 4WD
    Acura TL 2WD
    BMW 328i sedan
    Mercedes-Benz C-Class
    Lexus ES 350
    Subaru Outback 4WD
    Volkswagen Jetta
    Kia Sorento 2WD
    Toyota Highlander hybrid 4WD
    Honda Pilot 4WD
    Chevrolet Equinox 2WD
    Chevrolet Equinox 4WD
    Ford Flex 2WD 5 (0-15)
    Mazda CX-9 4WD
    Jeep Grand Cherokee 4WD
    Toyota Highlander 2WD
    Honda Pilot 2WD
    Hyundai Santa Fe 4WD
    Mazda CX-9 2WD
    Dodge Nitro 4WD
    Toyota Venza 4WD
    Lexus RX 350 4WD
    Volvo XC90 4WD
    Mercedes-Benz M-Class 4WD
    Lexus RX 350 2WD
    Acura MDX 4WD
    Acura RDX 4WD
    Lincoln MKX 4WD
    Dodge Ram 1500 Quad 4WD
    Chevrolet Silverado 1500 Crew 2WD
    Honda Ridgeline 4WD
    Toyota Sequoia 4WD
    GMC Yukon 4WD
    Ford Expedition 4WD
    GMC Yukon 2WD
    Nissan Armada 2WD
    Buick Enclave 2WD
    Chevrolet Traverse
    GMC Acadia 4WD
    Buick Enclave 4WD
    Chevrolet Tahoe 4WD
    Chevrolet Traverse 4WD
    Mercedes-Benz GL-Class 4WD
    Land Rover Range Rover Sport 4WD
    Honda Odyssey
    GMC Yukon XL 1500 4WD

    • 0 avatar
      ToddAtlasF1

      There are some pretty surprising vehicles on the list of cars with superior safety records to the 2.5 ton Tesla. Good grief. The Tesla may be no better in terms of safety than cars of its mass produced decades ago. Or not, but jumping to plausible but improbable conclusions is the Tesla way.

      • 0 avatar

        I’m just waiting for Guccifer 2.0 to crack all of Musk’s email and give it to WikiLeaks. Some folks would still be buying his tonic even if we knew the whole truth.

        One thing we do know: We’re not getting the whole truth. We’re only getting Musk’s brand of truth. Kinda like watching Podesta on the news shows without reading his emails.

        • 0 avatar
          jkross22

          This.

          Never underestimate people’s emotional investment in a cause/company/politician/spiritual leader/cult. Delusion is a terrible thing, but unquestioning allegiance is a deal with the devil – it’s a 1:1 swap – critical thinking for stupidity.

Read all comments

Back to TopLeave a Reply

You must be logged in to post a comment.

Recent Comments

  • downunder: So China will still produce ICE/Hybrids when the rest of the world goes total ev. Hooray! Australians will...
  • FalconRTV: I would miss the V8 noises.
  • dal20402: Wheels color-coordinated to the butterscotch leather are pimp.
  • 28-Cars-Later: Four years is time enough for a recycle.
  • Corey Lewis: We done did TC in 2017! https://www.thetruthaboutcars. com/2017/11/rare-rides-a-19...

New Car Research

Get a Free Dealer Quote

Who We Are

  • Adam Tonge
  • Bozi Tatarevic
  • Corey Lewis
  • Mark Baruth
  • Ronnie Schreiber