Introduction and epistemic status: This is the first post in a planned series, “Alignment as a verification problem”. I co-originated coverage-driven verification (CDV), which became the standard methodology for chip verification and is heavily used in AV safety. Back in 2015 I wrote that verifying “Friendly AI” would be our biggest verification challenge (and that … More When is misalignment just a bug?
Summary: This post suggests that alignment training could benefit from coverage-driven verification. Anthropic recently reported that teaching Claude alignment rules (via pretraining-style next-token learning on alignment-related stories) is more effective than relying primarily on RL-style behavioral shaping. Some AV developers reached a related conclusion, but in addition tend to use a systematic, coverage-driven methodology for … More Coverage-driven alignment – What ‘Teaching Claude Why’ can borrow from AV verification
This blog post introduces the V&V method – please read the full 26-page paper here. Summary: AGI may arrive in the near future, and without first solving the alignment problem, this could be catastrophic. The V&V method is a concrete, practical framework for deploying AGI in a safer way. It is based on how complex, … More The V&V method – A step towards safer AGI
From now on this blog will focus on AI safety / AI alignment (and how they connect to V&V). Posts about autonomy V&V – my day job – will go directly to the Foretellix company blog (e.g. see this post). The company blog already has quite a few excellent autonomy / AV-related posts – please … More Changing the focus of this blog to AI safety
My new post is here on the company’s blog. It talks about: How AI-based autonomy influences verification How AI-based autonomy implementation is moving closer to verification The importance of abstractions in all of that Enjoy.
Summary: This short post does not talk about autonomous vehicles. It talks about a new paper describing new ways in which complex socio-technical systems could go bad. I then talk about the verification implications of that. Bruce Schneier is a well-known security researcher: I have been following his blog for years. I don’t mention him … More About “The coming AI hackers” paper
Summary: This post looks further into ways to compute the residual risk of ADS. In particular, it looks into some recent research directions, and concludes that (1) they show promise, and (2) they are not there yet. My previous post about ADS Residual Risk (RR) ended with the following tantalizing promise: A subsequent post (coming … More ADS residual risk – some related research
Summary: This post talks about estimating the residual risk of an ADAS / AV design. It looks at the ADS total-risk map, divides it into several areas, and discusses ways for estimating the residual risk of each. People who deal with ADS (Autonomous Driving Systems, which includes ADAS and Autonomous Vehicles) are naturally very interested … More Estimating the residual risk of ADAS / AV
Summary: This post talks about GPT-3, a new Machine Learning (ML) system currently making waves in the ML community. It explains why GPT-3 is a big deal, and then considers the verification implications of such systems. One way to look at GPT-3 (and the even-bigger GPT-4, GPT-5 etc. which are sure to follow) is as … More GPT-3 and verification
Summary: Autonomous Vehicles will enter various markets (robotaxis, delivery bots, autonomous mining and so on) at different rates (and the current pandemic will probably accelerate some while slowing others). This post is about the verification needs of these markets, and how a single, extensible, scenario-based verification language can be used to express both the common … More Autonomy markets and their potential bugs