AV coverage and performance metrics

Summary: Sooner or later, people dealing with AV (Autonomous Vehicle) verification encounter the difficult topic of how coverage, KPIs (Key Performance Indicators) and performance grades relate to each other. Also, there is often confusion between “Did this scenario happen” and “Did the AV perform well in this scenario” (and the fact that “KPIs” are sometimes … More AV coverage and performance metrics

Using program induction for verification

Summary: I discussed before (e.g. here) how connecting rule-based verification to the rule-less, amorphous Machine Learning world is really hard, and yet necessary. The current post talks about a somewhat-exotic technique called Program Induction (PI), and how it might (eventually) help bridge that gap. What’s program induction Background: I always liked the idea of synthesizing … More Using program induction for verification

Where Machine Learning meets rule-based verification

Summary: This post addresses some high-level questions like: Longer term, how much of the verification of Intelligent Autonomous Systems can be done with just Machine Learning (ML)? Should most requirements remain rule-based, and if so – how does that connect to the ML part? And how will the uneasy interface between ML and rules influence … More Where Machine Learning meets rule-based verification

About faults and bugs

Summary: This post tries to fit Fault Tree Analysis and CDV-based “normal” bug-finding into the same conceptual framework. FTA and CDV have a challenging, uneasy relationship with each other. In a nutshell (details below): FTA (and similar techniques) is a good, established way to reason about failures and reliability, but (1) it depends on humans … More About faults and bugs