Knowledge Transferability for Data-Efficient Deep Learning

Deep Learning has demonstrated remarkable progress in various Computer Vision tasks. However, its effectiveness often relies on the availability of large, well-annotated datasets. In many practical scenarios, there is often limited availability of such data. Furthermore, collecting or labeling more samples that align with the post-deployment environment can sometimes be challenging or impossible, resulting in reduced and sub-optimal performance. Transfer Learning (TL) emerges as a promising solution in these contexts, offering methods to leverage the knowledge acquired by a Deep Neural Network (DNN) on one task to enhance its performance on another, especially when data is scarce, or labels are absent. The main objective of this project is to study and analyze TL methodologies in depth, providing novel perspectives, suggesting new directions both for real-world applications and future research in data-efficient Deep Learning. To achieve this, we present a comprehensive analysis of prevalent TL pipelines, particularly focusing on the fine-tuning of DNNs weights from pre-trained models in real-world scenarios. We also conduct an extensive experimental study on Domain Adaptation techniques, assessing numerous datasets, a wide range of DNN architectures and pre-training strategies to determine which are the most important design choices for a successful transferability. Furthermore, we develop new algorithms aimed at enhancing data-efficiency and adaptability (like Object Detection in robotics). We also introduce a new Domain Adaptation technique for Image Classification, characterized by its straightforward design, which favorably competes with state-of-the-art approaches. Our exploration extends to validate Domain Adaptation methodologies in a Sim-to-Real application for robotics, focusing on surface recognition using vision-based tactile sensors. The result of this work is a comprehensive overview of Transfer Learning, emphasizing data and label efficiency, with new solutions to many challenges, addressing both specific real-world applications and more general algorithmic paradigms. From a broader perspective, the algorithms, analyses, and discoveries presented in this research study have substantial implications for the expansive field of data-efficient Deep Learning. In applications such as robotics, autonomous vehicles, industrial automation and healthcare, the refined TL techniques that we introduce can play a crucial role. Enhancing data efficiency allows for reductions in computational costs and training times, facilitating faster and more precise model deployments. Our methodologies emphasize adaptability, offering great value in situations where models face varied and dynamic environments. Furthermore, our findings serve as a robust foundation for subsequent research, allowing for exploration into deeper domain-specific adaptations or integration with emerging AI methodologies.