When It Comes to Tesla's Accident-reducing Autosteer, Don't Believe the Numbers

There’s a study you should read, and it delivers black eyes to both Tesla and the National Highway Traffic Safety Administration.

You probably remember the fatal crash of a Tesla in Mountain View, California last March, a crash that occurred as the victim’s car cruised along in Autopilot mode. Unexpectedly, the vehicle steered itself out of a lane, impacting a highway divider at high speed. Once again, the effectiveness and safety of Tesla’s Autopilot system came under scrutiny as Tesla scrambled to defend itself. The automaker pointed to the findings of a 2017 NHTSA report released in the wake of a fatal crash from 2016. That study claimed the automaker’s Autosteer system, when introduced as part of the Autopilot suite of automated features, lowered Tesla crash rates by 40 percent.

Don’t believe everything you read, says R.A. Whitfield, director of Quality Control Systems. Whitfield filed a lawsuit and waited nearly two years to get to the bottom of that 40 percent figure.

As the NHTSA didn’t release the dataset behind the study, Whitfield requested it.

“Extraordinary claims ought to be backed by extraordinary evidence,” he told the Los Angeles Times.

Rebuffed, he filed a lawsuit under the Freedom of Information Act against the U.S. Department of Transportation. The data came into his possession in late November of last year.

After looking at the data, Whitfield discovered a serious problem with the methodology behind the controversial 40 percent figure. It has to do with the miles driven before and after Autosteer installation in data provided to the NHTSA by Tesla. The data covers a period from 2014 to 2016.

It’s a long document (you can read it here), but the report issued this month by Quality Control Systems breaks down the problems in the federal road safety agency’s calculations. Information about the number of pre-Autosteer miles traveled by certain vehicles in the data pool is missing, with other vehicles carrying vague info about when exactly Autosteer came online.

The NHTSA’s determination, the Maryland firm claims, was made “by examining the sums of the miles driven prior to Autosteer activation, miles driven after Autosteer activation, airbag deployment events prior to Autosteer activation and airbag deployment events after Autosteer activation for all of the subject vehicles.” That’s a quote from an NHTSA investigator.

In the report’s preamble, the firm states that, based on the incomplete data, it “recognized that NHTSA’s summarization of ‘miles driven prior to Autosteer activation [and] miles driven after Autosteer activation’ might not actually include all of the miles driven before or after Autosteer activation.”

After breaking the data down into different groups of vehicles (based on quality of reported mileage), the firm slammed the NHTSA. “The Agency’s treatment of missing or unreported mileage data in its calculations of exposure mileage as though the mileage were non-existent is not justifiable,” it says in its report.

The vast number of vehicles with missing mileage “results in the inflation of the overall ‘before Autosteer’ airbag deployment crash rate reported by NHTSA, but to a degree that can’t be known with certainty,” the report concludes.

In other words, the 40 percent crash reduction figure doesn’t have a leg to stand on. Exactly by how much Autosteer increased or reduced the crash rate can’t be known, given the incomplete data. The report concludes with a warning for lax agencies and automakers who hope to coax the public into supposedly safe self-driving vehicles.

“A very substantial fraction of the public simply doesn’t trust autonomous driving

[Image: Tesla]