An Important Whitepaper was Recently Released
How can we assure an AV is safe over its entire operating life? What if there are constant software updates? What proprietary details and sensitive data need to be publicly disclosed to show evidence of safety?
These are great questions that are beginning to get answered in a recent whitepaper that was put forth on IAMTS’s website titled, “Assuring Regulatory Compliance of Connected and Automated Vehicles during their Operational Lifetime.”
IAMTS stands for the International Alliance for Mobility Testing & Standardization which was formed under SAE in 2019 and now consists of representatives from SAE, IEEE, CATARC (China), TUEV SUED and other TUEV (TÜV) organizations (Germany, EU), and major automotive and technology developers.
The first thing about this whitepaper is that all of its contributors (authors) were from the testing or certifying services such as several TUEVs, CATARC, AVL, and FSD, and not from AV developers or suppliers who are also members of IAMTS. (Full disclosure, Retrospect is a supporter and member of IAMTS but was not a contributor to this publication effort.)
The next thing to note is the general structure of the white paper. Sections 1-3 and Appendix A cover the scope and definitions, and Sections 8-10 cover the contributor information and references. The meatier parts of the white paper are as follows:
Section 4 – covers existing regulations around the globe
Section 5 – discusses challenges to lifecycle compliance
Section 6 – reviews methods for compliance testing
Section 7 – makes recommendations for lifecycle compliance
In this blog we are only going to focus on some very important points within Section 6 - Consideration of applicable compliance testing methodologies.
There is a claim made up-front in this section. The claim is that a one-time, pre-launch validation activity, which is part of traditional automotive launches, is not feasible for autonomous vehicles. They write:
“A valid proof that automated vehicles fully meet all relevant requirements and prove to be reliable in real-world operation over the whole life-cycle in each possible situation cannot be provided before entering the market.”
They then introduce two types of validation assessment methods they call prospective and retrospective:
“Therefore, new assessment methods must be developed to ensure compliance of connected and automated vehicles during their operational lifetime. This includes a prospective safety assessment on the one hand and a validation in operation as a retrospective safety assessment on the other hand.”
These two methods are shown in a traditional V-model, with prospective safety assessment making up the right-hand side of the V-model, and retrospective validation constituting a new, feedback loop that compares operational field measurements to the original safety requirement specifications, bridging the top of the V-model.
They explain why both validation methods are necessary given many underlying reasons that are inherent in autonomous vehicle operation:
“Additionally, it is not possible to assess future adaptations to changing traffic conditions at the time of approval. Moreover, deterioration due to degradation, wear, tampering or damage as well as modifications due to regularly (over-the-air) software updates cannot be comprehensively determined at the time of approval, i.e., at the beginning of the product life cycle. Hence, a validation in operation as a retrospective safety assessment is vital to ensure road safety…”
They underscore how essential “Retrospective safety” is with the reference to the upcoming EU Regulation 2022/1426:
“Therefore, in-service monitoring and reporting (ISMR) performed by the manufacturers themselves is mandatory to apply for a type-approval of the automated driving system (ADS) of fully automated vehicles according to the Implementing Regulation (EU) 2022/1426 (cp. [2]).”
In further detailing the “Retrospective safety assessment (field monitoring)” approach, they highlight the traditional virtue of having truly independent assessment be performed to prevent a conflict of interest:
“In addition to ISMR performed by the manufacturers themselves, the validation of the performance of connected and automated vehicles should be performed by neutral, sovereign bodies as this supports a trusted third-party principle and complies with the market and field surveillance tasks of many countries…”
The most important quote is the following one, and it highlights the true advantage of “Retrospective safety” which is that a real-time validation algorithm can also act as a safety mechanism by reducing or halting the AV’s operation prior to an accident:
“If safety-relevant anomalies are detected, in worst case, the deactivation of the corresponding automated driving functions can be initiated…”
While the concept of real-time, “Retrospective safety” validation is straight-forward and desirable, the details of how such validation is implemented is extremely challenging, as the algorithms must be able to be sensitive enough to detect minor anomalies, yet robust from false positives. Furthermore, the algorithm’s evaluation of safety, itself, must be holistic and free from errors and this is hard to objectively define:
“[T]he performance of the driving task must be evaluated by appropriate algorithms. For this purpose, generally applicable, unambiguous and objective evaluation criteria and methods must be developed for validating the automated driving functions…“
Finally, the authors emphasize the applicability of such safety algorithms to real-word driving data in order to validate the validation algorithms.
“Therefore, a comprehensive analysis of real driving data is essential. Only if nominal driving behaviour can be described comprehensively, incidents in driving behaviour can be detected and evaluated.”
By getting a feel for how the real-time validation algorithms work in naturalistic or real-world data, and proving the algorithms are complete, correct, and comprehensive, both the AV developers and the public road users can have confidence in 3rd-party AV risk monitoring software.
If you are an AV developer, and your management has not bought into the need for 3rd-party AV risk monitoring software for in-vehicle and SIL-bench development, you should share this whitepaper from IAMTS.
If you have not had a comprehensive risk analysis of your existing data, please contact us for more information. If you are interested in trialing an off-the-shelf safety monitor, that is developed independently and with safety documentation, you may want to check out RiskEngineTM, the first and only objective AV risk analyzer.
Retrospect is the only company that offers an actual risk analysis and risk monitor using continuous quantification and first-principles based physics, and not a rules-based driving envelope that is only as good as the input it receives. Retrospect has been singularly focused on the problem of validating AV path-planning using in-vehicle safety monitors since our inception in 2018. It’s literally in our name and our identity.
Four years ago, in March of 2019, Retrospect was invited to present at the first-ever SOTIF conference in Munich on how to assure AV safety using real-time (Retrospective) safety validation during the Deployment phase of the AV lifecycle. The following slide was from that 2019 presentation:
We have always remained committed to equipping AV developers with tools and evidences they need to be convinced that public road users are free from risk from their AV technology, and we are convinced our approach is winning in the long-run over other approaches that have attempted to redefine risk thresholds and eliminate systematic safety requirements for AV developers.
We look forward to seeing more concepts such as IAMTS’s “Prospective Safety Validation” and “Retrospective Safety Validation” come to fruition very soon, as the industry transitions to responsible AV safety launches that holds public safety as their chief concern. Thank you for your support in helping make that a part of our industry’s culture. Please leave us a comment below or contact us for more discussion. We look forward to hearing from you.