The only way for this to cause tens of thousands of death by self-driving alone is for people to suddenly need to drive the cars themselves, and not being able to do it well. Unless I'm missing something.

The system could go haywire and crash in such unexpected circumstances. They might also not. It's hard to know how they will behave.