It's Getting Hot In Here, So Turn Off All Your Motors — And Your ABS, Too
Toyota had an odd pair of recalls this week, highlighting both the increasing importance of software within the automobile and further reinforcing a pet theory held by your humble author.
The substance of the recalls can be found on Toyota’s website, but here are the money shots:
- Toyota will update the motor/generator control ECU and hybrid control ECU software on certain Model Year 2010-2014 Prius vehicles. The software’s current settings could result in higher thermal stress in certain transistors, potentially causing them to become damaged. If this happens, various warning lights will illuminate and the vehicle can enter a failsafe mode. In rare circumstances, the hybrid system might shut down while the vehicle is being driven, resulting in the loss of power and the vehicle coming to a stop.
- Toyota will update the skid control ECU software on certain 2012 Toyota RAV4, 2012-2013 Toyota Tacoma, and 2012-2013 Lexus RX 350 models in order to address an electronic circuit condition that can cause the Vehicle Stability Control, Anti-lock Brake, and Traction Control functions to intermittently turn off. If these systems are off, standard braking operation remains fully functional.
Stuff like this is why engineering products for automobiles is about the worst career in the world. Not only are you faced with uncompromising cost controls, weight targets, and space constraints, you also have to consider the fact that your end user will be anywhere from Fairbanks to Death Valley with his foot flat to the floor, dead leaves in the radiator inlets, and dirty oil sloshing below the minimum mark in the sump. It’s nothing short of miraculous that cars work as well as they do, really.
In the case of the Prius, we have a situation where, presumably, too much current is being fed through something too close to a transistor, or perhaps the transistor is being too heavily loaded and overheating, the same way you can make the bottom of your laptop too hot to touch doing video editing in realtime, engaging in brute-force attacks on encrypted documents, or trying to load the various crazy Flash stuff embedded on this here website. Either way, it’s too hot to handle, so the system decides to lay off on the computing and/or the power transfer until the situation improves.
The other recall would appear to involve a potential “electronic circuit condition”. Allow me to take a wild-ass guess and say that it probably is a condition where a combination of inputs to the software creates a loop or a race condition. The latter term has nothing to do with civil rights or green flags; more a situation where a couple of variables are fighting it out for supremacy. When that happens, from the perspective of the user, the software simply goes out to lunch. Blue screen of death, endless spinning beachball, a Flappy Bird stuck in mid-air eternally because your phone rang right as you were also trying to cue up the next Fleet Foxes song. Of course, if the “user” is the ABS system in your Tacoma, then it, too, has to wait until the next reboot, which in the case of a car can range from a few minutes to the next time the battery is disconnected.
“Well what are you doing? Let’s get out of here!”
“Can’t. Computer’s jammed.”
“It says all its circuits are occupied. There’s no power anywhere in the ship.”
Ford moved away from the computer terminal, wiped a sleeve across his forehead and slumped back against the wall.
“Nothing we can do,” he said. He glared at nothing and bit his lip.
When Arthur had been a boy at school, long before the Earth had been demolished, he had used to play football. He had not been at all good at it, and his particular speciality had been scoring own goals in important matches. Whenever this happened he used to experience a peculiar tingling round the back of his neck that would slowly creep up across his cheeks and heat his brow. The image of mud and grass and lots of little jeering boys flinging it at him suddenly came vividly to his mind at this moment.
A peculiar tingling sensation at the back of his neck was creeping up across his cheeks and heating his brow.
He started to speak, and stopped.
He started to speak again and stopped again.
Finally he managed to speak.
“Er,” he said. He cleared his throat.
“Tell me,” he continued, and said it so nervously that the others all turned to stare at him. He glanced at the approaching yellow blob on the vision screen.
“Tell me,” he said again, “did the computer say what was occupying it? I just ask out of interest …”
Their eyes were riveted on him.
“And, er … well that’s it really, just asking.”
Zaphod put out a hand and held Arthur by the scruff of the neck.
“What have you done to it, Monkeyman?” he breathed.
“Well,” said Arthur, “nothing in fact. It’s just that I think a short while ago it was trying to work out how to …”
“Make me some tea.”
“That’s right guys,” the computer sang out suddenly, “just coping with that problem right now, and wow, it’s a biggy. Be with you in a while.” It lapsed back into a silence that was only matched for sheer intensity by the silence of the three people staring at Arthur Dent.
As if to relieve the tension, the Vogons chose that moment to start firing. — Douglas Adams, The Hitchhiker’s Giude To The Galaxy
Your humble author is pretty good at getting cars to enter software failure modes. I experienced it recently in both the Nissan Juke and the Infiniti Q50S, in each case under conditions of speed, driver inputs, and available traction that I would cheerfully characterize as “abusive”. In fact, I’d say that it’s easier, in general, to “break” the dynamic systems of a car through hard driving than it is to break anything else. I’ve had far more ABS or stability-control failures than I’ve had, say, front wheel bearing seizures or dropped driveshafts.
There’s a reason for this, and now it’s time for my pet theory. Believe it or not, I’ve done a little bit of professional software development in my life. This will come as a great surprise to all of you who have considered my squeaky voice, prickly disposition, and tendency to quote Douglas Adams to be infallible evidence of a life spent as a Rhodesian mercenary. Do me a favor and keep quiet about this around the ladies, I always tell them that I paid for my Porsches by doing figure modeling. In any event, I’ve made some bucks writing software and I’ve spent some miserable hours dealing with other peoples’ work and I’ve participated in everything from solo development to the current XP/Agile/Kanban/Pivotal idiocy that’s sweeping the industry.
Once upon a time, software was written by people who knew what they were doing, like Mel and his descendants. They were generally solitary, socially awkward fellows with strong awareness of TSR gaming. They were hugely effective at doing things like getting an Atari 2600 to run Pac-Man or writing operating system kernels that never crashed, but they weren’t terribly manageable and they could be real pricks when you got in their way. I once worked with a fellow who had been at the company in question for twenty-three years and had personally written a nontrivial percentage of the nine million lines of code that, when compiled, became our primary product. He was un-fire-able and everybody knew it. There were things that only he knew.
This kind of situation might work out well for designing bridges or building guitars (not that Paul Reed Smith appears to miss Joe Knaggs all that much, to use an inside-baseball example) but it’s hell on your average dipshit thirty-five-year-old middle manager, who has effectively zero leverage on the wizard in the basement. Therefore, a movement started in the software business about fifteen years ago to ensure that no more wizards were ever created. It works like this: Instead of hiring five guys who really know their job at seventy bucks an hour each, you hire a team of fifty drooling morons at seven bucks an hour each. You make them program in pairs, with one typing and the other once watching him type (yes! This is a real thing! It’s called “extreme programming”!) or you use a piece of software to give them each a tiny bit of the big project.
This is what you get from a management perspective: fifty reports who are all pathetically grateful for the work instead of five arrogant wizards, the ability to fire anybody you like at any time withouiret consequence, the ability to demand outrageous work hours and/or conditions, (I was just told that a major American corporation is introducing “bench seating” for its programmers, to save space) and a product that nominally fulfills the spec. This is what you get from a user perspective: the kind of crapware that requires updates twice a week to fix bugs introduced with the previous updates. Remember the days when you could buy software that simply worked, on a floppy disk or cartridge, with no updates required? Those were the wizards at work. Today, you get diverse teams of interchangeable, agile, open-office, skill-compatible resources that produce steaming piles of garbage.
Enough of the rant. I can’t wait for the day when I never have to touch a computer again to make a living. Admittedly, it will be because I’m a sixty-three-year-old Wal-Mart greeter. But I’m looking forward to it. Where were we? Oh yes. An embarrassing amount of the software in the cars we drive is outsourced to programming farms where the wizards were long ago cut loose. Modern auto manufacturers sweat every detail of the unibody and the tire specs and the thickness of the rear door glass, and they create modern engineering wonders which they then proceed to load up with the cloacal expulsions of moronic bench-seated 120-IQ “programmers”. It’s no accident that software updates make up a large number of recalls nowadays. The software’s written by people who expect a chance at a do-over, not realizing that a Toyota Prius is a little harder to update than, say, a useless Android app.
Given the increasing evidence of this problem, what will the manufacturers do? Will they resurrect the wizards? Bring the programming in-house? Restore pride to the profession? Hell no. The future belongs to Internet-connected cars seamlessly upgrading their firmware twice a week. It sounds very advanced, and it is. But if you want something that reliably gets you to work or pumps its own brakes on an icy road, you might want to stick with the old stuff.
U mad scientist on Feb 18, 2014
tedward> The dynamic of wizard vs manager is as old as time itself. Its merely run into a recent uptick in relevance due to the surge in managers being standardized by business schools and in company’s willingness to trust those schools to produce managers instead of doing that training and selection themselves. The obvious repercussion of this is that this new crop of managers lack much domain expertise, which is rather critical when making decisions about highly technical product development. -- In the larger picture, all of this is the product of an economic system which prioritizes short term financial success over long term consequences. Bean counting will make the next few quarterly results better for wall st (you know, the stuff that matters) than r&d. That means money guys at the top who naturally hire those like them as subordinates and it's downhill from there. For a while inertia will keep it going as marketing budgets and various gimmicks supersede product development, but eventually everyone is left wondering how the new upstart came to surpass them as the wheels fall off the wagon and the cycle starts anew. So to all those above wondering how the alphabet soup of business BS came to pass at their current job, consider it part of the natural circle of life for capitalist finance.
Robert on Feb 19, 2014
Jack: I didn't know you had programming chops, but I am not surprised. I've been in software for 16 years. Great Developers, I mean really really good developers, we're talking the best grade A talented ones, the the ones I buy the Macallan 18 and Edradour 1993 cask strength grey market scotch for because when the shit hits the fan they are the ones who can fix it, those devs all have one thing in common. They are all, to a man, also a musician. Talent levels notwithstanding, they all play instruments, are in bands, make pilgrimags to the Blues Hall of Fame in Memphis, and write some of the smartest code I've ever seen. So not surprised, but it confirms my own theory of the correlation between reading and playing music and coding.
Latest Car ReviewsRead more
Latest Product ReviewsRead more
- Bobbysirhan Some friends of mine were dazzled by a CUE demo that circulated on YouTube before this car reached the market. I was bewildered why anyone wanted a car as durable and dependable as their cellphones, but to each their own. One of them did actually show up with an XTS V-sport when the car first came out. He showed people CUE in my driveway, but I don't recall him offering demonstration rides to the assembled imported luxury car drivers. In the months that followed, I never saw or heard about the Cadillac again. He went back to driving his Yukon Denali until I moved away a year or two later.
- Scoutdude Yes you will have to wait between your 10 second bursts 200 electric ponies. The fact that it lists the continous output of 94 ponies means that is what the battery, wiring or motor can handle w/o overheating. Then there is the battery SOC. There will be some point at which it doesn't have enough charge to produce that 10 second burst and even if you started that 10 sec burst with enough power it may not be able to sustain that for a full 10 sec. So the question becomes which component is the weak link, how long will it take to cool down enough before you can repeat it. If it is the battery did that 10 sec blast no only heat up the battery but also drain it to the point where it needs to be recharged before it can sustain another 10 sec burst.
- Theflyersfan @Tim Healey: Like the idea and recommend keeping them interesting. We can get fluff piece reviews of the latest Corolla Cross "reviews" still in a Sunday paper! I'll say dig WAY back into the archives - I remember the review that brought me to the site - Farago's Lotus Elise review back in 2002 I think. There are the Lieberman reviews as well before he left and now we see him online and on TV. Now I'm trying to remember the names of the first group of reviewers here...
- SCE to AUX Few things are as boring as watching electric cars race.
- Bru65688995 I owned a 1965 Monza convertible. Had a blast until I could afford a 1967 SS396 Chevelle.