1. Introducing the smashing new Team FIAT T-Shirt !! To order yours click here : Team FIAT T-Shirt


Discussion in 'Hangout' started by varunrtr, May 30, 2011.

  1. Toyota's killer firmware: Bad design and its consequences

    Michael Dunn - October 28, 2013

    On Thursday October 24, 2013, an Oklahoma court ruled against Toyota in a case of unintended acceleration that lead to the death of one the occupants. Central to the trial was the Engine Control Module's (ECM) firmware.
    Embedded software used to be low-level code we'd bang together using C or assembler. These days, even a relatively straightforward, albeit critical, task like throttle control is likely to use a sophisticated RTOS and tens of thousands of lines of code.
    With all this sophistication, standards and practices for design, coding, and testing become paramount – especially when the function involved is safety-critical. Failure is not an option. It is something to be contained and benign.
    So what happens when an automaker decides to wing it and play by their own rules? To disregard the rigorous standards, best practices, and checks and balances required of such software (and hardware) design? People are killed, reputations ruined, and billions of dollars are paid out. That's what happens. Here's the story of some software that arguably never should have been.
    For the bulk of this research, EDN consulted Michael Barr, CTO and co-founder of Barr Group, an embedded systems consulting firm, last week. As a primary expert witness for the plaintiffs, the in-depth analysis conducted by Barr and his colleagues illuminates a shameful example of software design and development, and provides a cautionary tale to all involved in safety-critical development, whether that be for automotive, medical, aerospace, or anywhere else where failure is not tolerable. Barr is an experienced developer, consultant, former professor, editor, blogger, and author.
    Barr's ultimate conclusions were that:

    • Toyota’s electronic throttle control system (ETCS) source code is of unreasonable quality.
    • Toyota’s source code is defective and contains bugs, including bugs that can cause unintended acceleration (UA).
    • Code-quality metrics predict presence of additional bugs.
    • Toyota’s fail safes are defective and inadequate (referring to them as a “house of cards” safety architecture).

    • Misbehaviors of Toyota’s ETCS are a cause of UA.

    A damning summary to say the least. Let's look at what lead him to these conclusions:

    Although the investigation focused almost entirely on software, there is at least one HW factor: Toyota claimed the 2005 Camry's main CPU had error detecting and correcting (EDAC) RAM. It didn't. EDAC, or at least parity RAM, is relatively easy and low-cost insurance for safety-critical systems.
    Other cases of throttle malfunction have been linked to tin whiskers in the accelerator pedal sensor. This does not seem to have been the case here.

    The Camry ECM board. U2 is a NEC (now Renesas) V850 microcontroller.​
    The ECM software formed the core of the technical investigation. What follows is a list of the key findings.
    Mirroring (where key data is written to redundant variables) was not always done. This gains extra significance in light of …
    Stack overflow. Toyota claimed only 41% of the allocated stack space was being used. Barr's investigation showed that 94% was closer to the truth. On top of that, stack-killing, MISRA-C rule-violating recursion was found in the code, and the CPU doesn't incorporate memory protection to guard against stack overflow.
    Two key items were not mirrored: The RTOS' critical internal data structures; and—the most important bytes of all, the final result of all this firmware—the TargetThrottleAngle global variable.
    Although Toyota had performed a stack analysis, Barr concluded the automaker had completely botched it. Toyota missed some of the calls made via pointer, missed stack usage by library and assembly functions (about 350 in total), and missed RTOS use during task switching. They also failed to perform run-time stack monitoring.
    Toyota's ETCS used a version of OSEK, which is an automotive standard RTOS API. For some reason, though, the CPU vendor-supplied version was not certified compliant.
    Unintentional RTOS task shutdown was heavily investigated as a potential source of the UA. As single bits in memory control each task, corruption due to HW or SW faults will suspend needed tasks or start unwanted ones. Vehicle tests confirmed that one particular dead task would result in loss of throttle control, and that the driver might have to fully remove their foot from the brake during an unintended acceleration event before being able to end the unwanted acceleration.
    A litany of other faults were found in the code, including buffer overflow, unsafe casting, and race conditions between tasks.
    Last edited: Nov 18, 2013
    8 people like this.
  2. pdydeva

    pdydeva Regolare

    Thanks for posting Sat.

    That's a neatly done investigation by the US authorities. Fortunately/unfortunately, we have few Toyota Camry's running in India with the ETCS-i technology. Makes me wonder if India is getting too advanced in automotive sector when there is no such government bodies capable of handling such investigations. A clear example is the unclear investigation results of various burning cases (Vento, Nano, Volvo etc.)
    1 person likes this.
  3. rishike007

    rishike007 Esperto

    Pune- Mumbai
    Grande Punto 1.2
    One of my office friend captured this on Pune Satara road



    Sent from my iPhone using Tapatalk
    Last edited: Nov 22, 2013
  4. Dipesh_enr

    Dipesh_enr Amatore

  5. asimpleson

    asimpleson Esperto

    Linea 1.3
    That's very sad, what happened to the driver and rear passengers?
  6. Dipesh_enr

    Dipesh_enr Amatore

    It was only driver who succumbed to death while going to the hospital.:-(
  7. asimpleson

    asimpleson Esperto

    Linea 1.3
    Fortunately they also have very active class action suits against companies that sell faulty products. Nvidia used to sell some very badly designed chips that would heat and fry or desolder from the pc and laptop mainboards. They had to issue huge recalls in US atleast. Here in India, I know many people, friends, colleagues and their friends too who lost money on valueable laptops that died badly after a year or 2 never to be repaired again.
    2 people like this.
  8. Viny

    Viny Esperto

  9. Oh my god!! this is scary , I can't think of how those moments for the occupant of that ill fated car can be.
  10. DRIV3R

    DRIV3R Esperto

    The guy was reportedly driving back from work and this happened at around 5PM.

    Root cause is driver error, no doubt, but had the railings been as they should be, this guy should have been alive and kicking today. :(

    There are many safety lapses in our Indian roads.

    Another i20 drove straight into a river at night, where the area between the bridges were left without any barricades. Add to it, a dangerous curve just before that spot.

Share This Page