Explaining a Deep Reinforcement Learning (DRL)-Based Automated Driving Agent in Highway Simulations

As deep learning models have become increasingly complex, it is critical to understand their decision-making, particularly in safety-relevant applications. In order to support a quantitative interpretation of an autonomous agent trained through Deep Reinforcement Learning (DRL) in the highway-env simulation environment, we propose a framework featuring three types of views for analyzing data: (i) episode timeline, (ii) frame by frame, and (iii) aggregated statistical analysis, also including heatmaps for a better spatial understanding. Our methodology allowed a novel, consistent description of the behavior of the agent. The main motivator for the taken action is typically the longitudinal distance from the second-closest and, to a lower extent, third-closest vehicle. In the overtakes, also the agent's position in lanes becomes relevant. The analysis identified interesting patterns and an issue in the last frames of an episode, when the agent is unable to overtake the last two vehicles, arguably because of the lack of reference vehicles ahead. We observed a clear differentiation between attention and SHAP values (estimating the importance of each feature for each decision), reflecting the architecture of the neural network, where the first layer implements the attention mechanism, while the deeper ones make the actual decision. Attention focuses on the proximity of the ego, while the decision is taken on a wider horizon, denoting a valuable anticipation capability. To support research, the proposed framework is released as open source.