Tesla’s Autopilot Investigation - Would You Be Prepared?
Michael DiazLast week, NHTSA disclosed its request to Tesla for documentation and data related to their freshly-launched investigation into Tesla’s automated driving features. The list is lengthy (their definition of the word “Document” alone is at least 500 words) and the letter claims a whopping $114 Million maximum fine if they don’t deliver.
Today, I’m going to lay out what they asked for. While you go through the list, think about your organization: If NHTSA launched a probe into your ADAS or autonomous vehicles asking for these same things - would you be prepared? Would you be confident that it’s a few calls and clicks away?
Retrospect would like to help your team establish a culture and goal set built on safety such that the processes that would make complying with this request straightforward. We’d like to set you up so that you never become the subject of such an investigation - and if you somehow do, then the hardest part for you should be obtaining a copy of MS Access 2010.
Background
After the 12th time a Tesla vehicle with automated driving features crashed into an emergency vehicle, NHTSA announced the launch of an investigation. Just this week, NHTSA also published a letter to Tesla detailing what needs to be provided as part of the investigation. Below you’ll find a concise list. We’re sticking to the safety-relevant items here.
The List
I’ve compiled the highlights of the list that relate to safety. There are plenty more items and explanations given in the official NHTSA document, so give it a read. I’ve categorized the information requests into three groups for you to get a sense for the pain points that an organization receiving such a letter will face. I call them the straightforward, the complicated, and the controversial.
Straightforward
These are essentially informational requests. The data exists somewhere within the company databases. Things like “how many vehicles have which software version?”…You just need to do the proper queries. Note, even this easiest category of questions doesn’t necessarily make you look good or bad, such as “control limits on actuators” - they’ll just be straightforward answers.
Complicated
These are similar to straightforward items in that this information may already be in hand, but they may expose some difficult numbers (e.g. “number of complaints..”) or expose some questionable decisions (e.g. “Approach to driver attentiveness”). Tesla will have these answers - but they may not make you look good and may require additional context.
Controversial
These are your real pain points. With the first two categories, as begrudgingly as you may feel turning them over, you have the information or data. But these controversial ones are the ones Tesla may have not written down, allowed to be dictated implicitly or unconsciously (e.g. “Limitations to detecting signs, actors, spaces” and “law enforcement vehicles”) which will take time to think about, derive, and perhaps write down for the first time. These will likely pose a challenge to combine with the other questions about process control and history (“How long have these processes been in place”).
The responses to Controversial items are likely non-existent to date and will require much engineering work to answer - if they can be answered at all. Worse yet, the Controversial items may set Tesla up for painful follow-up questions like, “Why??” (E.g. “Metrics used to assess safety performance” and “Degree to which simulation data was relied upon in lieu of field testing” and “Effects of low light conditions on performance” and “Extent of field testing miles required prior to each release”). These are also the types of questions that, with enough “whys” in the follow-up, could lead an organization to admit to embarrassing practices, or lack of (like ‘we don’t monitor that’, ‘I don’t know’ and ‘it’s up to the AI’).
Here’s my rendition of the safety items in the list:
| # | NHTSA Requested Items (my words, emphasis added) | Difficulty |
|---|---|---|
| 1. | OE software/hardware/firmware versions | Straightforward |
| 2. | Most recent software / firmware / hardware updates on these vehicles | Straightforward |
| 3. | ite coordinates / involved objects / odometric data | Straightforward |
| 4. | timing of L2 system engagement/disengagement over the last 30 seconds leading to the crash | Straightforward |
| 5. | Crash warning / avoidance systems’ behavior prior to the incidents | Straightforward |
| 6. | Subject system logic intended to detects first responder vehicles on or off the roadway | Straightforward |
| 7. | ALL documents, telematics reports / data / logs related to the above | Straightforward |
| 8. | Control limits on actuators | Straightforward |
| 9. | Info/alerts/warnings/etc communicated to the driver when the system is running / requires intervention / detects impending crashes | Straightforward |
| 10. | Sensing mechanisms to detect / infer this | Straightforward |
| 11. | Minimum durations of engagement and between engagement periods to satisfy driver engagement -- and how this varies in different environments | Straightforward |
| 12. | Escalation / lockout strategies used to address inattentive drivers | Straightforward |
| 13. | What functions allowed to continue following a driver override | Straightforward |
| 14. | Which disabled functions resume on their own after an override and when | Straightforward |
| 15. | How the L2 systems interact with crash avoidance technologies | Straightforward |
| 16. | How long have these processes have been in place | Straightforward |
| 17. | Event data collection capabilities and limitations / countermeasures | Straightforward |
| 18. | Processes for investigating customer concerns or safety events | Straightforward |
| 19. | Cumulative mileage covered by each ‘subject vehicle’ (defined as all Tesla vehicles from 2014-2021 equipped with any form of Level 2 capability system at any time) with an L2 system engaged | Complicated |
| 20. | The number of complaints/reports/crashes/property damages/third party arbitrations/lawsuits related to your Level 2 system | Complicated |
| 21. | Summary of arbitrations/lawsuits and their facts and evidences | Complicated |
| 22. | Incident descriptions for the above including | Complicated |
| 23. | Copies of any reports produced by Tesla or outside parties in these lawsuits/arbitrations | Complicated |
| 24. | ODD specified to the customer intended for the system including: | Complicated |
| 25. | Types of roads, weather, the system is intended to bused on and those not to be used on | Complicated |
| 26. | Approach to enforcement of driver attentiveness | Complicated |
| 27. | Any modifications or changes made by or for Tesla that affect the L2 systems | Complicated |
| 28. | Strategies for detecting the presence of first responders including (flashing lights,flares,...) | Complicated |
| 29. | Scene detection strategy for first responders | Complicated |
| 31. | Processes in place to compare system performance in the field with the design intent | Complicated |
| 30. | Effects of low light conditions on these strategies | Controversial |
| 32. | If the system can be engaged / remain engaged outside the ODD specified describe how the system performance changes / warns the driver / adjusts | Controversial |
| 33. | List of conditions that may prompt system to require a take-over by driver -- including sequence of events and timing, intended behavior if no takeover occurs | Controversial |
| 34. | Detection capabilities within the ODD - what objects and events is it designed to detect (signs, actors, spaces, classifications | Controversial |
| 35. | LImitations to detecting each of the above | Controversial |
| 36. | Reasons why each modification/change was made | Controversial |
| 37. | Any processes governing extent of testing / validation req’d prior to release of the L2 systems or in-field updates in h/w or s/w including: | Controversial |
| 38. | Extent of field testing miles required prior to each release | Controversial |
| 39. | Extent of simulation or training data sets required prior to each release - and degree to which they are relied upon in lieu of field testing | Controversial |
| 40. | How test/val processes differ between first release and updates | Controversial |
| 41. | Metrics used to assess safety performance | Controversial |
| 42. | For every crash list the causal factors, failure mechanisms and modes, risk to safety | Controversial |
If you haven’t been overcome by the gravity of this request then here’s one more - they ask that this data be provided in a format compatible with Microsoft Access 2010.
In the end, this is an investigation into a particular company about the safety of its products. As part of that investigation, NHTSA wants to find out what an organization’s goals are. How does the organization set goals and execute to those goals?
Retrospect would like to help your team establish a culture and goal set built on safety such that the processes that would make complying with this request straightforward. We’d like to set you up so that you never become the subject of such an investigation - and if you somehow do, then the hardest part for you should be obtaining a copy of MS Access 2010.