I have sent quite a few people the summary of what I saw at the Autonomous Vehicles Test & Development Symposium in Stuttgart.
I was pleasantly surprised with the results – a lot of people sent thoughtful answers, or talked to me about it.
Here is a summary of what they said, anonymized (because this was mainly in the context of private conversations).
About whether AV people actually do CDV-style testing
Somebody suggested: “As a comment on the meeting as a whole, I think that there are probably players in the industry who were ‘conspicuous by their absence’, probably due to competitive fears. I wonder whether some of the ‘obvious’ names may already have developed some independent thinking and approaches to the questions of simulation, verification and validation that perhaps they did not want to share (yet) – I am keeping an eye on the standards space, because manufacturers have an incentive to ensure that their approaches are at least permitted, if not endorsed by relevant standards”.
However, somebody who should know (from the AV industry) answered: “I guess the car manufactures are working on this but I doubt that they have a solution ready”.
Another answer from the AV industry said “In a detailed picture on verification the automotive industry is a little bit better off than you perceive it, but this concerns verification of subsystems and components (like the semiconductors developed for automotive use semiconductor methodology for verification). For the complete automotive system the picture is indeed not perfect. An indicator is that there a rarely any presentations that address topics like test and verification strategy but rather focus on how to operate a particular tool.
The ISO26262 covers fairly well the issues of a simple system like some type of controller (speed controller, temperature control, etc.) where most of the risks are due to E/E failures. This subject has some tradition and is therefore more or less understood. Moving to ADAS or automated driving we are dealing more with some sort of a cognitive system, which also can fail due to E/E failure, but this is likely not the main risk. Systematic failure like not being able to cope with unforeseeable risky situations is much more of an issue.”.
Finally, I came up with the following article in the Guardian (don’t know how I never noticed it before), which suggests that Google probably does do something similar to the CDV I am talking about. More details are here. It seems like they did millions of miles of SW-only simulations, based on about 2000 miles of digitized roads.
Here is a key paragraph from the Guardian article: “Because much of Google’s virtual testing focused on rare and potentially hazardous scenarios, the company argues that the test equates to many lifetimes of human driving. “These are not straight and empty roads but four million interesting miles that actually teach us something and challenge the car,” said Jabbari.”
Also: “This allows Google engineers to assess new software for its robot cars in realistic virtual environments, testing how tweaks to their behavior might play out in reality. For example, a new emergency braking system was driven 10,000 miles in the simulator to check how often it kicked in and how it performed. And 50 virtual self-driving cars were dropped into a digital version of Google’s hometown of Mountain View to see how they interacted with each other, computer-generated pedestrians and simulated drivers.”
I’d love to find out more about what they do – let me know if you find anything.
About why (at least European) AV people don’t do CDV-style testing
Somebody from the chip verification industry said “The automotive industry is led by people with a mechanical background. Thus all management decisions are made based on mechanical terms. Electronics and software are considered “Neuland”, something new and mysterious that can’t be trusted.”.
Another person from chip verification said that automotive in general (not just AVs) does mainly HIL, not SIL. He heard from an automotive company that “80 (maybe 70) % of the failures they find usually with driver in the loop, could be found earlier with systematic verification.”.
But they don’t do it – he said. The general reason is “the European automotive industry likes to maintain all their DIL, HIL etc, because it’s part of a very well established infrastructure, that is controllable very well by the large companies. Any change into an EDA (Electronic Design Automation) verification directions will force
- additional education
- and break with existing roles and working/management mechanisms and dependencies
- reduced effort at the HIL/DIL side
1..3 are more or less strong blocking factors”.
Somebody who researches robotics verification (and also knows CDV) suggested that similar blocking issues exist in robotics, especially lack of CDV knowledge, and an assumption that “if we do SW-based verification, then it must be formal verification”. Perhaps there is simply no good “CDV fundamentals for non-chip engineers/managers” paper. Thinking a bit, I realized that indeed I don’t know of any.