It's Getting Hot In Here, So Turn Off All Your Motors — And Your ABS, Too

Toyota had an odd pair of recalls this week, highlighting both the increasing importance of software within the automobile and further reinforcing a pet theory held by your humble author.

• Toyota will update the motor/generator control ECU and hybrid control ECU software on certain Model Year 2010-2014 Prius vehicles. The software’s current settings could result in higher thermal stress in certain transistors, potentially causing them to become damaged. If this happens, various warning lights will illuminate and the vehicle can enter a failsafe mode. In rare circumstances, the hybrid system might shut down while the vehicle is being driven, resulting in the loss of power and the vehicle coming to a stop.
• Toyota will update the skid control ECU software on certain 2012 Toyota RAV4, 2012-2013 Toyota Tacoma, and 2012-2013 Lexus RX 350 models in order to address an electronic circuit condition that can cause the Vehicle Stability Control, Anti-lock Brake, and Traction Control functions to intermittently turn off. If these systems are off, standard braking operation remains fully functional.

Stuff like this is why engineering products for automobiles is about the worst career in the world. Not only are you faced with uncompromising cost controls, weight targets, and space constraints, you also have to consider the fact that your end user will be anywhere from Fairbanks to Death Valley with his foot flat to the floor, dead leaves in the radiator inlets, and dirty oil sloshing below the minimum mark in the sump. It’s nothing short of miraculous that cars work as well as they do, really.

In the case of the Prius, we have a situation where, presumably, too much current is being fed through something too close to a transistor, or perhaps the transistor is being too heavily loaded and overheating, the same way you can make the bottom of your laptop too hot to touch doing video editing in realtime, engaging in brute-force attacks on encrypted documents, or trying to load the various crazy Flash stuff embedded on this here website. Either way, it’s too hot to handle, so the system decides to lay off on the computing and/or the power transfer until the situation improves.

The other recall would appear to involve a potential “electronic circuit condition”. Allow me to take a wild-ass guess and say that it probably is a condition where a combination of inputs to the software creates a loop or a race condition. The latter term has nothing to do with civil rights or green flags; more a situation where a couple of variables are fighting it out for supremacy. When that happens, from the perspective of the user, the software simply goes out to lunch. Blue screen of death, endless spinning beachball, a Flappy Bird stuck in mid-air eternally because your phone rang right as you were also trying to cue up the next Fleet Foxes song. Of course, if the “user” is the ABS system in your Tacoma, then it, too, has to wait until the next reboot, which in the case of a car can range from a few minutes to the next time the battery is disconnected.

“Well what are you doing? Let’s get out of here!”

“Can’t. Computer’s jammed.”

“Jammed?”

“It says all its circuits are occupied. There’s no power anywhere in the ship.”

Ford moved away from the computer terminal, wiped a sleeve across his forehead and slumped back against the wall.

“Nothing we can do,” he said. He glared at nothing and bit his lip.

When Arthur had been a boy at school, long before the Earth had been demolished, he had used to play football. He had not been at all good at it, and his particular speciality had been scoring own goals in important matches. Whenever this happened he used to experience a peculiar tingling round the back of his neck that would slowly creep up across his cheeks and heat his brow. The image of mud and grass and lots of little jeering boys flinging it at him suddenly came vividly to his mind at this moment.

A peculiar tingling sensation at the back of his neck was creeping up across his cheeks and heating his brow.

He started to speak, and stopped.

He started to speak again and stopped again.

Finally he managed to speak.

“Er,” he said. He cleared his throat.

“Tell me,” he continued, and said it so nervously that the others all turned to stare at him. He glanced at the approaching yellow blob on the vision screen.

“Tell me,” he said again, “did the computer say what was occupying it? I just ask out of interest …”

Their eyes were riveted on him.

“And, er … well that’s it really, just asking.”

Zaphod put out a hand and held Arthur by the scruff of the neck.

“What have you done to it, Monkeyman?” he breathed.

“Well,” said Arthur, “nothing in fact. It’s just that I think a short while ago it was trying to work out how to …”

“Yes?”

“Make me some tea.”

“That’s right guys,” the computer sang out suddenly, “just coping with that problem right now, and wow, it’s a biggy. Be with you in a while.” It lapsed back into a silence that was only matched for sheer intensity by the silence of the three people staring at Arthur Dent.

As if to relieve the tension, the Vogons chose that moment to start firing. — Douglas Adams, The Hitchhiker’s Giude To The Galaxy

Your humble author is pretty good at getting cars to enter software failure modes. I experienced it recently in both the Nissan Juke and the Infiniti Q50S, in each case under conditions of speed, driver inputs, and available traction that I would cheerfully characterize as “abusive”. In fact, I’d say that it’s easier, in general, to “break” the dynamic systems of a car through hard driving than it is to break anything else. I’ve had far more ABS or stability-control failures than I’ve had, say, front wheel bearing seizures or dropped driveshafts.

There’s a reason for this, and now it’s time for my pet theory. Believe it or not, I’ve done a little bit of professional software development in my life. This will come as a great surprise to all of you who have considered my squeaky voice, prickly disposition, and tendency to quote Douglas Adams to be infallible evidence of a life spent as a Rhodesian mercenary. Do me a favor and keep quiet about this around the ladies, I always tell them that I paid for my Porsches by doing figure modeling. In any event, I’ve made some bucks writing software and I’ve spent some miserable hours dealing with other peoples’ work and I’ve participated in everything from solo development to the current XP/Agile/Kanban/Pivotal idiocy that’s sweeping the industry.

Once upon a time, software was written by people who knew what they were doing, like Mel and his descendants. They were generally solitary, socially awkward fellows with strong awareness of TSR gaming. They were hugely effective at doing things like getting an Atari 2600 to run Pac-Man or writing operating system kernels that never crashed, but they weren’t terribly manageable and they could be real pricks when you got in their way. I once worked with a fellow who had been at the company in question for twenty-three years and had personally written a nontrivial percentage of the nine million lines of code that, when compiled, became our primary product. He was un-fire-able and everybody knew it. There were things that only he knew.

This kind of situation might work out well for designing bridges or building guitars (not that Paul Reed Smith appears to miss Joe Knaggs all that much, to use an inside-baseball example) but it’s hell on your average dipshit thirty-five-year-old middle manager, who has effectively zero leverage on the wizard in the basement. Therefore, a movement started in the software business about fifteen years ago to ensure that no more wizards were ever created. It works like this: Instead of hiring five guys who really know their job at seventy bucks an hour each, you hire a team of fifty drooling morons at seven bucks an hour each. You make them program in pairs, with one typing and the other once watching him type (yes! This is a real thing! It’s called “extreme programming”!) or you use a piece of software to give them each a tiny bit of the big project.

This is what you get from a management perspective: fifty reports who are all pathetically grateful for the work instead of five arrogant wizards, the ability to fire anybody you like at any time withouiret consequence, the ability to demand outrageous work hours and/or conditions, (I was just told that a major American corporation is introducing “bench seating” for its programmers, to save space) and a product that nominally fulfills the spec. This is what you get from a user perspective: the kind of crapware that requires updates twice a week to fix bugs introduced with the previous updates. Remember the days when you could buy software that simply worked, on a floppy disk or cartridge, with no updates required? Those were the wizards at work. Today, you get diverse teams of interchangeable, agile, open-office, skill-compatible resources that produce steaming piles of garbage.

Enough of the rant. I can’t wait for the day when I never have to touch a computer again to make a living. Admittedly, it will be because I’m a sixty-three-year-old Wal-Mart greeter. But I’m looking forward to it. Where were we? Oh yes. An embarrassing amount of the software in the cars we drive is outsourced to programming farms where the wizards were long ago cut loose. Modern auto manufacturers sweat every detail of the unibody and the tire specs and the thickness of the rear door glass, and they create modern engineering wonders which they then proceed to load up with the cloacal expulsions of moronic bench-seated 120-IQ “programmers”. It’s no accident that software updates make up a large number of recalls nowadays. The software’s written by people who expect a chance at a do-over, not realizing that a Toyota Prius is a little harder to update than, say, a useless Android app.

Given the increasing evidence of this problem, what will the manufacturers do? Will they resurrect the wizards? Bring the programming in-house? Restore pride to the profession? Hell no. The future belongs to Internet-connected cars seamlessly upgrading their firmware twice a week. It sounds very advanced, and it is. But if you want something that reliably gets you to work or pumps its own brakes on an icy road, you might want to stick with the old stuff.