Argo AI: Furthering Safety or Catching Up?
Last week marked the third time an AV company has announced a new, driverless AV operation on public roads in 2022. On Tuesday, Argo AI announced driverless testing had begun in Miami and Austin.
Waymo, also, announced last week an expansion of their Phoenix AV operations without a safety driver, and earlier, in March, Waymo began driverless AV testing in San Francisco.
In February, Cruise’s driverless ride-hailing service opened to the public in San Francisco. Prior to that, Cruise had been testing AVs without Safety Drivers since November 2021.
In none of these cases were the decisions to pull Safety Drivers out of the vehicles preceded with some sort of pronouncement, demonstration, or publication, such as the NHTSA Voluntary Safety Self-Assessment (VSSA), that indicated the management felt the standard of care had been met.
This is a problem.
It serves no purpose for AV companies to keep their safety argumentation secret when they’re potentially putting the public at risk. And yes, operating a vehicle, with or without a human driver, is inherently risky. This is a key concept we take for granted, unfortunately.
Keeping safety arguments secret is a problem because in the event of litigation, their past safety argumentation will end up being examined, anyway. And if anyone disagrees or cannot understand their safety argumentation, it would be better for them to have cleared that up sooner than later. Ideally, prior to bringing that risk onto public roads.
In Argo’s case, Sam Abuelsamid, contributor to Forbes, reported that an external company was brought in to review Argo’s safety operations. TÜV SÜD conducted a review of Argo’s Autonomous Driving System (ADS) and their procedures in Miami and Austin. While the details of this review are not public, Argo stated that TÜV SÜD said the concept was, “sufficiently effective and trustworthy for testing,” according to Keith Naughton at Bloomberg News.
The public needs to be made aware that they have every right to ask for a guarantee of their safety when a potentially risky activity is occurring in their proximity.
In the case of operating vehicles on public roads, these sorts of publicly available guarantees have been long enshrined in things like driver training, testing, licensing, and obtaining insurance, all of which can be provided as public evidence.
It’s not convincing when AV companies are not forthcoming with similar types of public evidences on ensuring the public’s safety. Again, I see no benefit to them in keeping the safety criteria for developing and operating an AV safely a secret.
Those who are ultimately going to own AV safety are going to be in one of two camps:
The AV developers, themselves, who strive to create and maintain safe technology
The public and regulators, who strive to prevent and keep unsafe technology off the streets
Back in January of this year, we wrote about the importance of ownership when it comes to AV safety. In reviewing a video of a Cruise AV being operated in a dust-storm in Arizona, we highlighted how essential it is for stakeholders in AV companies to be constantly looking for underlying risks in their technology. Do you think stakeholders are asking their teams to come up with more ways to identify underlying risks?
Looking back at our earlier blog, we want to remind these companies of the importance of Safety Culture in all their operating domains, including the public domain in which they, and other developers operate.
If there are methods for holistically detecting any and all safety-related bugs in AVs, it would benefit AV companies to share these methods openly. It doesn’t reveal competitive IP for how they ultimately solve those bugs, and it offers the public evidence they fulfill the necessary standard of care.
Thinking about how to systematically identify and own safety issues, such as perception errors or planning errors, let’s revisit a key concept from our earlier blog on Safety Culture and Management Ownership.
Safety culture is developed by those who are going to own safety in their company, in their industry, and in their community.
SAFETY CULTURE IS A CULTURE OF MANAGEMENT OWNERSHIP
Ultimately, management should want to own all those [safety] issues, and build their product and processes around finding those issues. Sometimes management doesn’t want to know.
In Cruise’s case, they 100% agreed with us that getting executive and management involvement is vital to managing safety, and they have a formal Safety Management System in place which includes management engagement early in the safety process.
There are two opposing philosophies when it comes to finding product issues. The first is “Ignorance is Bliss.” It’s better not to know. Sadly, this is not uncommon.
The second comes from the lean methods originating with Toyota and Deming, which I will summarize by Peter Drucker’s words, “No problem? No manager.”
Drucker tells the story in which a Toyota manager kept asking for weekly problem reports from two new American managers working under him. They kept assuring him everything was fine, as that was the custom they were used to. His response was if everything’s fine, then they don’t need a manager, “No problem? No manager,” and they quickly started finding problems to report.
It’s the second philosophy, the one in which management wants to know what the problem is that the employees are dealing with, that is the right approach and leads to success.
The other philosophy, the “ignorance is bliss” approach leads to overpromised planning, short-term gains, and eventually jumping roles as the ship takes on water. And let’s be blunt, it leads to far worse outcomes; potentially life-ending.
We find the correct philosophy in the first two of these 14 principles in the Lean Toyota Production System (TPS) as shown in this blog:
• Principle 1: Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals.
• Principle 2: Create continuous process flow to bring problems to the surface.
Bingo. Think long-term. Don’t worry about short-term financials. Create continuous process flow of issues to the surface for everyone to see.
When issues are found, like the issues shown with Cruise’s perception in the dust cloud, these are gold to management. These reveal where the structural weaknesses are most likely to be. They can plan accordingly. They can address the high-risk safety concerns first, launch sooner, and continuously improve functionality.
Systematically scrutinizing autonomous path planning for issues is not only great from a business perspective, it can actually become a part of the developer’s safety argument. This is precisely why we created RiskEngine™ and are continuing to work on quantitative AV monitoring solutions.
It allows developers to claim, “Yes, we likely have certain bugs and issues in our software we’re not aware of. But we can still operate safely by detecting those bugs early.”
They can claim this so long as the observability or detectability of those bugs, which can be tested through fault injection, is many orders of magnitude greater than the likelihood of the environmental circumstances occurring such that there is real harm caused by the bug.
In short, any bugs will be exposed in operation long before the bug actually leads to an accident. But, developers can only claim that if their management is using systematic risk monitoring of the path planning algorithm.
We should stop pretending that AVs will ever be “bug-free” and that by many miles of expensive testing we will somehow convince litigators that the AV was bug-free.
The sooner we all admit that software bugs are likely here to stay in autonomy, and we build an effective safety monitoring system around them, then the sooner AVs can be fully deployed and provide value to their customers and to their companies.
In closing, all AV company executives need to ask these questions:
Are the roles of the Functional Safety and Operational Safety teams to make safety problems go away, or to make safety problems known?
Are the safety team’s development milestones to show no more risks, or to start showing new risks?
Hopefully this blog has made the right answers to these questions fairly obvious.
MAKING THE CASE
Ultimately, in Argo’s case we can only say what we can see publicly: By getting an external audit from an automotive safety assessor, Argo appears to have gone further than any other major AV operator who has decided to remove their Safety Drivers.
One of our focuses at Retrospect is in making comprehensive safety reports that can be shared with 3rd parties, such as government regulators, litigators, and insurance companies. Part of that includes real-time safety monitoring, as well as offline data analytics.
The core metric for quantifying a road scene into a probabilistic injury metric is RiskEngine™. If you haven’t contacted us for a trial, or tried out our free online version, please schedule a call today.
Our goal is to enable those who want to own AV safety to make the safest and most productive AV technology possible.