Emergent self-awareness in multi-sensor physical agents

The cognitive approach to the development of autonomous vehicles takes inspiration from human reasoning, and, conversely to the computationalist approach, rejects formulating fixed mathematical models to describe each possible behavior of vehicles and objects around them. The computationalist approach indeed has a weakness: developers cannot examine and mathematically formulate all possible real-world situations that drivers may encounter. Cognitive approaches provide a solution to this problem, as they suggest that vehicles should continually learn through experience as humans do, which would allow them to progressively grasp rare and unexpected behaviors. This thesis mainly addresses the pivotal question of detecting when some unexpected behavior is occurring - referred to as anomaly detection - within a cognitive self-awareness framework. The adopted framework is characterized by several desirable characteristics, such as being Bayesian, hierarchical, multi-sensorial, data-driven, and interpretable. A set of modules for anomaly detection and localization of an agent are proposed. Dynamic Bayesian Networks are used, as they are interpretable probabilistic models allowing hierarchical representation of variables potentially coming from multiple sensors; moreover, the links between variables inside Dynamic Bayesian Networks can be learned from data. All methods in the thesis adopt a filter called the Markov Jump Particle Filter that can be described through a Dynamic Bayesian Network on a minimum of three levels. When elaborating image data, Variational Autoencoders are adopted to perform dimensionality reduction while maintaining a probabilistic representation; novel methods for joining Variational Autoencoders and Bayesian filters are proposed. This thesis focuses on the elaboration of vehicular video and odometry data. First, self-awareness anomaly detection approaches to separately handle video and odometry data are proposed; then, an approach fusing the two modalities is introduced; the capability to localize the vehicle is also added. The proposed methods are evaluated on real-world and simulated data from terrestrial and aerial vehicles.