Industry Practices for Challenging Autonomous Driving Systems with Critical Scenarios

Testing autonomous driving systems for safety and reliability is essential, yet complex. A primary challenge is identifying relevant test scenarios, especially the critical ones that may expose hazards or harm to autonomous vehicles and other road users. Although numerous approaches and tools for critical scenario identification are proposed, the industry practices for selection, implementation, and evaluation of approaches, are not well understood. Therefore, we aim at exploring practical aspects of how autonomous driving systems are tested, particularly the identification and use of critical scenarios. We interviewed 13 practitioners from 7 companies in autonomous driving in Sweden. We used thematic modeling to analyse and synthesize the interview data. As a result, we present 9 themes of practices and 4 themes of challenges related to critical scenarios. Our analysis indicates there is little joint effort in the industry, despite every approach has its own limitations, and tools and platforms are lacking. To that end, we recommend the industry and academia combine different approaches, collaborate among different stakeholders, and continuously learn the field. The contributions of our study are exploration and synthesis of industry practices and related challenges for critical scenario identification and testing, and potential increase of industry relevance for future studies.