Learning Self-Awareness Models for Physical Layer Security in Cognitive and AI-enabled Radios

Cognitive Radio (CR) is a paradigm shift in wireless communications to resolve the spectrum scarcity issue with the ability to self-organize, self-plan and self-regulate. On the other hand, wireless devices that can learn from their environment can also be taught things by malicious elements of their environment, and hence, malicious attacks are a great concern in the CR, especially for physical layer security. This thesis introduces a data-driven Self-Awareness (SA) module in CR that can support the system to establish secure networks against various attacks from malicious users. Such users can manipulate the radio spectrum to make the CR learn wrong behaviours and take mistaken actions. The SA module consists of several functionalities that allow the radio to learn a hierarchical representation of the environment and grow its long-term memory incrementally. Therefore, this novel SA module is a way forward towards realizing the original vision of CR (i.e. Mitola's Radio) and AI-enabled radios. This thesis starts with a basic SA module implemented in two applications, namely the CR-based IoT and CR-based mmWave. The two applications differ in the data dimensionality (high and low) and the PHY-layer level at which the SA module is implemented. Choosing an appropriate learning algorithm for each application is crucial to achieving good performance. To this purpose, several generative models such as Generative Adversarial Networks, Variational AutoEncoders and Dynamic Bayesian Networks, and unsupervised machine learning algorithms such as Self Organizing Maps Growing Neural Gas with different configurations are proposed, and their performances are analysed. In addition, we studied the integration of CR and UAVs from the physical layer security perspective. It is shown how the acquired knowledge from previous experience within the Bayesian Filtering facilitates the radio spectrum perception and allows the UAV to detect any jamming attacks immediately. Moreover, exploiting the generalized errors during abnormal situations permits characterizing and identifying the jammer at multiple levels and learning a dynamic model that embeds its dynamic behaviour. Besides, a proactive consequence can be drawn after estimating the jammer's signal to act efficiently by mitigating its effects on the received stimuli or by designing an efficient resource allocation for anti-jamming using Active Inference. Experimental results show that introducing the novel SA functionalities provides the high accuracy of characterizing, detecting, classifying and predicting the jammer's activities and outperforms conventional detection methods such as Energy detectors and advanced classification methods such as Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN) and Stacked Autoencoder (SAE). It also verifies that the proposed approach achieves a higher degree of explainability than deep learning techniques and verifies the capability to learn an efficient strategy to avoid future attacks with higher convergence speed compared to conventional Frequency Hopping and Q-learning.