Summary: This post talks about the DeepXplore paper, and uses it to revisit the topic of verification of ML-based systems The paper DeepXplore: Automated Whitebox Testing of Deep Learning Systems (by folks from Columbia U and Lehigh U) describes a new and (in my view) pretty important way to verify ML-based systems. And it somehow … More DeepXplore and new ideas for verifying ML systems
Summary: This post acts as a “table of contents” for the whole blog, linking (via tags) to the various posts. There are now quite a few posts in the Foretellix blog, and readers may get lost. So I added Tags to the various posts, and also created this “general orientation” post, which has links to … More Topics in this blog
Summary: This post will talk about “One-shot imitation learning” (a new and exciting direction in Machine Learning), and how that direction could help coverage maximization (which is important for verification). It will then speculate about the general role of ML in Intelligent Autonomous Systems verification. Note: You may have heard already about one-shot imitation learning … More One-shot imitation learning and verification
Summary: Here is a list of papers I read lately about verifying Autonomous Vehicles. For each, I say why I think it is interesting. These papers all start with something like “AVs are upon us, so they’d better be safe and well-verified”, and then go on to explain their (very diverse) ideas about how to … More Some papers about AV verification
Summary: This is another “What’s new in verification land” post. It describes a video and a paper from Mobileye, and takes that opportunity to revisit four topics: How Autonomous Vehicles should handle unstructured human interaction, how to balance Reinforcement Learning and safety, why simulation is the main way to validate safety in these unstructured environments, … More Misc stuff: Mobileye, simulations and test tracks
Summary: This is another one of those “What’s new in verification land” posts. It talks about the verification aspects of the recent Oscars fiasco. I was on vacation, and thus (mostly) missed this earth-shaking event. Luckily my spies are everywhere, and my friend Elliot Cohen sent me the following email with all the gory details: … More Misc stuff: The Oscars bug
Summary: This post describes some new attempts at formal verification of complex systems (mainly AVs and ML-based systems) In general, I think Formal Verification (FV) will only have a limited role in verifying Intelligent Autonomous Systems (IAS) and ML-based systems, because those systems are so complex and heterogenous. In FV has better PR than CDV … More Formal verification of really-complex systems
Summary: The interactions between Autonomous Vehicles and people can be complex, which complicates AV deployment. This post summarizes some recent related publications, and then tries to predict the verification implications of all that. For instance, it suggests that verification teams will try to track total-accidents, AV-specific-accidents and AV-specific-annoyances. There were several interesting publications lately about “bumps on … More Verifying interactions between AVs and people
Summary: This post discusses the annoying “Synthetic Sensor Input” (SSI) problem, i.e. the fact that it is very hard to synthesize realistic, synchronized streams of sensor inputs (e.g. Video+LiDAR+Radar). It explains why the SSI problem is a pain for Autonomous Vehicles verification (and for other things), and talks about the (imperfect) solutions. Let me start … More The “Synthetic Sensor Input” problem in AV verification
Summary: This post tries to explain (once and for all?) how the concept of coverage is used to optimize the verification process, what it means to auto-maximize coverage, and how people have tried to do it. I have been spending some time lately on coverage maximization via ML (which I described here). As is often … More Verification, coverage and maximization: The big picture
