How Safe is Safe Enough? - According to SOTIF
“How safe is safe enough?” is a question many in the autonomous vehicle industry are asking. One of the hopes is that the standardization effort, “Safety of the Intended Function,” or ISO/PAS 21448, will provide an answer to this question or at least provide the framework on how to answer it. In this post, we will provide key considerations for developers, regulators, and the SOTIF authors in their efforts to confidently claim, “This is why we know this AV is safe enough.”
We believe the path to safety-assured autonomous vehicles could be much closer than many of us realize. This is because safe autonomy is achieved more by a safety strategy than it is by additional technological improvements. The functional capabilities may not ultimately meet all of the expectations for autonomy, and yes, that will take time to improve, but safety assurance in autonomous technology is possible today and should be enabled for commercial opportunities in limited but gradually expanding applications.
In our earlier post on the SOTIF or ISO/PAS 21448 effort, we focused on the ways SOTIF may be misunderstood regarding whether it is really distinct from ISO 26262, or an application-specific subset within ISO 26262. In this post, we will focus on the draft SOTIF standard, itself, and how concept clarification can lead to safety assurance for autonomous vehicles. There is much alignment between this post and this safety paper, which we encourage you to check out, titled, “Towards an Operational Design Domain That Supports the Safety Argumentation of an Automated Driving System,” that was published in February of this year.
There are two main tenets to SOTIF, but in my opinion, this is how I would explain them in an ISO/PAS 21448 training course. The first tenet or belief is that all scenarios can be rightly divided along the lines of KNOWN-and-UNKNOWN and SAFE-and-UNSAFE. The second main tenet, and arguably the most critical one, is that the number of unsafe scenarios within a given set of use cases is dependent on improvements made during development, and not simply an invariant artifact stemming from the nature of the relevant use cases themselves.
While not a normative part of this standard, the following figure clearly illustrates these two main tenets and is essential to the idea of SOTIF. ISO/PAS 21448 states that the areas shown in this figure are used as a ”mental model to structure this document.”
The areas shown in this figure are used as a “mental model to structure” ISO/PAS 21448.
Looking at the first tenet of SOTIF (as I described it), we see that all the “relevant use cases” for the autonomous development are classified into four overlapping areas: 1. Safe-Known, 2. Unsafe-Known, 3. Unsafe-Unknown, and 4. Safe-Unknown. This fourth area, Safe-Unknown, is not considered in the remainder of the document. Only Areas 1, 2 ,and 3 are considered.
The figure shows the idea that the potential for an unsafe instance is relatively large at the beginning of development and becomes relatively small at the end, and that is straight forward. This is the second tenet. But the unfortunate part of this representation is that it loses any understanding of how the autonomous vehicle contributed to or avoided the occurrence of the unsafe scenario. Two suggestions are offered here to show this:
One method for reducing the unsafe areas, which ISO/PAS 21448 states, is to “restrict the use/performance of this function.” This would effectively be a modification of the original use case, and therefore a modification of the set of scenarios that could occur within the use case.
Another method for achieving safety is by “improving the function.” Presumably, this functional improvement would mean a change in functional requirements; if there is a change in functional requirements, then there would be fewer malfunctions leading to unsafe scenarios, thus arguably decreasing the relative size of the unsafe areas.
If we wanted, we could call this combination of Use Cases restrictions and Functional Requirements to be the “Operating Design Domain” or ODD; basically, it would state what the autonomous vehicle is designed to do and how it is going to do it. This can be a useful concept to invoke, and we will illustrate the ODD in the context of the Safe and Unsafe scenarios.
Rather than re-drawing the entire scenario map, as SOTIF shows in its “evolution” of scenario categories over the course of development, we will draw the functional improvements on top of the previous scenario map. But before we do this, I want to acknowledge that there are many sources of uncertainty in autonomous driving applications, and those sources of uncertainty, listed below, are still present even in our safe, well-defined ODD.
Sources of uncertainty:
- Uncertainty from underlying AI technology
- Uncertainty from potential systematic faults, e.g. incomplete requirements
- Uncertainty from external actors, e.g. driver suddenly swerves
- Uncertainty from environment
In these final two figures, we want to illustrate how the Use Case restrictions, as well as the Functional Requirements, can define our ODD and can define it safely. We also want to illustrate how uncertainties can still be present which can cause the autonomous vehicle to leave the ODD.
And the very last concept we will introduce here is the concept of using monitoring methods, such as the method we are developing at Retrospect, for systematically detecting faults in AV path planning. These monitoring methods could be part of an overall safety management system, encompassing the entire lifecycle of the autonomous driving product. Therefore, the remaining unsafe scenarios may be eliminated or otherwise argued as reasonably safe due to the early detection provided by the systematic fault monitor.
A suggestion for illustrating a safe, well-defined ODD plus Retrospect’s method of using a systematic fault monitor to detect potential ODD excursions, even if none actually occur
The purpose of including a risk-and-trajectory graph alongside the map of scenarios is to illustrate that the movement from one scenario to another is through a causal chain of events. This is similar to SOTIF’s definition of a scenario, a “description of the temporal development between several scenes in a sequence of scenes.” The work we have done so far, at Retrospect, in assuring safety of higher levels of autonomy has involved the determination of instantaneous risk profiles of a scene and a planned trajectory, to be used in a systematic monitoring tool for satisfying ISO 26262.
We haven’t talked about ISO 26262 so far, but one way in which SOTIF and ISO 26262 may closely interface is in definitions like the ODD, and a complimentary idea we might call the Non-ODD: the Non-Operating Design Domain, in which the AV should not operate and we can formally define and detect when the AV is entering. This Non-ODD can be, and should be, defined within the Known-Safe scenarios. This can account for the uncertainties that we know we will have – even if the true causes are unknown. This may also be required for legal compliance, for example, Germany has a draft law for AV approval which was described in this article to require that “[d]uring the automated phase of the journey, a ‘Technical Supervisor’ … must monitor the safety of the journey.”
The systematic monitoring method doesn’t have to only detect potentially unsafe faults. It can also be used to quantitatively measure how good of a job the AV does at tending to stay within a safe operating region. Again, this can be incorporated into an overall safety management system for the product lifecycle. Quantifiable metrics, like miles driven multiplied by average risk reduction rates, or robustness towards safety within a given scenario could be used instead of metrics used today, such as disengagements or miles driven without an accident. An illustration of this is shown, below.
A suggestion for illustrating robustness and non-linear stability characteristics towards safe operation
These modifications of the SOTIF images, which include Retrospect’s method for systematically monitoring risk of higher levels of autonomy, are simply meant to demonstrate how each unsafe scenario is necessarily realized through a causal sequence of events, a sequence along which we can potentially define where the autonomous behavior is no longer what we intend. In other words, we can define where the behavior becomes unintended. A systematic fault monitor which satisfies ISO 26262 may be very useful in detecting these early deviations, before they result in an actual hazardous event.
One final thought I hope to impart, has to do with the fact that I never really answered the original question: “How safe is safe enough?” I think there should be no answer to this question. I know this is often asked, but this is approaching safety from the wrong mindset. To get the answer we need to ask a different question. We need to ask, “How safe is too safe?” We need to be exploring to see how far we can go with safety assurance; not how little we could get away with. Once we’ve found the metrics, perhaps like the ones we’ve suggested here, that allow us to articulate and describe how “safe is too safe” we may be able gradually back down safety to a point where we’re not contributing any additional risk and it appears to be technically and commercially feasible. This is certainly our hope for a safe and beneficial beginning of commercial autonomy, and as we’ve outlined here, we don’t believe the use of this strategy is too far off.
To quickly summarize, we started with two core ideas, if not the two core ideas for SOTIF: scenarios are either Safe or Unsafe and Known or Unknown, and the likelihood of Unsafe scenarios actually occurring can be reduced through use case restriction and functional improvement. The clarity we wanted to add is how the use of requirements in functional improvement and use case definition can be used as an argument for completely safe operation, including the detection of when we are leaving a known operating state.
The use of systematic fault monitors, or risk monitors like ours, can be used to detect and mitigate the sources of uncertainty, thus reducing or eliminating the likelihood of the remaining unsafe scenarios from occurring. Any and all methods need to be combined until we can answer the question, “How safe is too safe?”
We hope you found this helpful, but encourage constructive feedback. If you are an author on the ISO/PAS 21448 standard and believe something is misrepresented or stated in error, please feel free to reach out and we will gladly correct it. For additional information on developing functional safety strategies and risk monitoring tools for your autonomous vehicle application of Level 3, 4, or 5, which is beyond the scope of ISO/PAS 21448, please check out our Autonomous Safety Project, or contact us. As always, be sure to sign up for our newsletter for more in-depth content and also follow us on LinkedIn for the next post.