Designing an Interpretability Analysis Framework for Deep Reinforcement Learning (DRL) Agents in Highway Automated Driving Simulation

Explainability is a key requirement for users to effectively understand, trust, and manage artificial intelligence applications, especially those concerning safety. We present the design of a framework aimed at supporting a quantitative explanation of the behavioural planning performed in automated driving (AD) highway simulations by a high-level decision making agent trained through deep reinforcement learning (DRL). The framework relies on the computation of SHAP values and keeps into consideration a neural architecture featuring an attention layer. The framework is particularly devoted to study the relationship between attention and interpretability, and how to represent, analyze and compare attention and SHAP values in a 2D spatial highway environment. The framework features three main visualization areas, that are obtained by processing quantities such as attention, SHAP values, vehicular observations: Episode view, plotting quantities on an episode’s timeline; Frame view, reporting the measurement values step by step; Aggregated view, showing, also on 2D maps, statistical values from the aggregation of several simulation episodes. To the best of our knowledge, this is the first in-depth explainability analysis of a DRL-based decision-making AD system, also including a statistical analysis. We hope that the upcoming open source release of the designed framework will be useful to further advance research in the field.