In recent years, there has been considerable interest in recording physiological signals from the human body to investigate various responses. Attention is one of the key aspects that physiologists, neuroscientists, and engineers have been exploring. Many theories have been established on auditory and visual selective attention. To date, the number of studies investigating the physiological responses of the human body to auditory attention on natural speech is, surprisingly, very limited, and there is a lack of public datasets. Investigating such physiological responses can open the door to new opportunities, as auditory attention plays a key role in many cognitive functionalities, thus impacting on learning and general task performance. In this thesis, we investigated auditory attention on the natural speech by processing physiological signals such as Electroencephalogram (EEG), Galvanic Skin Response (GSR), and Photoplethysmogram (PPG). An experiment was designed based on the well established dichotic listening task. In the experiment, we presented an audio stimulus under different auditory conditions: background noise level, length, and semanticity of the audio message. The experiment was conducted with 25 healthy, non-native speakers. The attention score was computed by counting the number of correctly identified words in the transcribed text response. All the physiological signals were labeled with their auditory condition and attention score. We formulated four predictive tasks exploiting the collected signals: Attention score, Noise level, Semanticity, and LWR (Listening, Writing, Resting, i.e., the state of the participant). In the first part, we analysed all the user text responses collected in the experiment. The statistical analysis reveals a strong dependency of the attention level on the auditory conditions. By applying hierarchical clustering, we could identify the experimental conditions that have similar effects on attention score. Significantly, the effect of semanticity appeared to vanish under high background noise. Then, analysing the signals, we found that the-state-of-the-art algorithms for artifact removal were inefficient for large datasets, as they require manual intervention. Thus, we introduced an EEG artifact removal algorithm with tuning parameters based on Wavelet Packet Decomposition (WPD). The proposed algorithm operates with two tuning parameters and three modes of wavelet filtering: Elimination, Linear Attenuation, and Soft-thresholding. Evaluating the algorithm performance, we observed that it outperforms state-of-the-art algorithms based on Independent Component Analysis (ICA). The evaluation was based on the spectrum, correlation, and distribution of the signals along with the performance in predictive tasks. We also demonstrate that a proper tuning of the algorithm parameters allows achieving further better results. After applying the artifact removal algorithm on EEG, we analysed the signals in terms of correlation of spectral bands of each electrode and attention score, semanticity, noise level, and state of the participant LWR). Next, we analyse the Event-Related Potential (ERP) on Listening, Writing and Resting segments of EEG signal, in addition to spectral analysis of GSR and PPG. With this thesis, we release the collected experimental dataset in the public domain, in order for the scientific community to further investigate the various auditory processing phenomena and their relation with EEG, GSR and PPG responses. The dataset can be used also to improve predictive tasks or design novel Brain-Computer-Interface (BCI) systems based on auditory attention. We also use the deeplearning approach to exploit the spatial relationship of EEG electrodes and inter-subject dependency of a model. As a domain application, we finally discuss the implications of auditory attention assessment for serious games and propose a 3-dimensional difficulty model to design game levels and dynamically adapt the difficulty to the player status.

Predictive analysis of auditory attention from physiological signals

BAJAJ, NIKESH
2019-05-03

Abstract

In recent years, there has been considerable interest in recording physiological signals from the human body to investigate various responses. Attention is one of the key aspects that physiologists, neuroscientists, and engineers have been exploring. Many theories have been established on auditory and visual selective attention. To date, the number of studies investigating the physiological responses of the human body to auditory attention on natural speech is, surprisingly, very limited, and there is a lack of public datasets. Investigating such physiological responses can open the door to new opportunities, as auditory attention plays a key role in many cognitive functionalities, thus impacting on learning and general task performance. In this thesis, we investigated auditory attention on the natural speech by processing physiological signals such as Electroencephalogram (EEG), Galvanic Skin Response (GSR), and Photoplethysmogram (PPG). An experiment was designed based on the well established dichotic listening task. In the experiment, we presented an audio stimulus under different auditory conditions: background noise level, length, and semanticity of the audio message. The experiment was conducted with 25 healthy, non-native speakers. The attention score was computed by counting the number of correctly identified words in the transcribed text response. All the physiological signals were labeled with their auditory condition and attention score. We formulated four predictive tasks exploiting the collected signals: Attention score, Noise level, Semanticity, and LWR (Listening, Writing, Resting, i.e., the state of the participant). In the first part, we analysed all the user text responses collected in the experiment. The statistical analysis reveals a strong dependency of the attention level on the auditory conditions. By applying hierarchical clustering, we could identify the experimental conditions that have similar effects on attention score. Significantly, the effect of semanticity appeared to vanish under high background noise. Then, analysing the signals, we found that the-state-of-the-art algorithms for artifact removal were inefficient for large datasets, as they require manual intervention. Thus, we introduced an EEG artifact removal algorithm with tuning parameters based on Wavelet Packet Decomposition (WPD). The proposed algorithm operates with two tuning parameters and three modes of wavelet filtering: Elimination, Linear Attenuation, and Soft-thresholding. Evaluating the algorithm performance, we observed that it outperforms state-of-the-art algorithms based on Independent Component Analysis (ICA). The evaluation was based on the spectrum, correlation, and distribution of the signals along with the performance in predictive tasks. We also demonstrate that a proper tuning of the algorithm parameters allows achieving further better results. After applying the artifact removal algorithm on EEG, we analysed the signals in terms of correlation of spectral bands of each electrode and attention score, semanticity, noise level, and state of the participant LWR). Next, we analyse the Event-Related Potential (ERP) on Listening, Writing and Resting segments of EEG signal, in addition to spectral analysis of GSR and PPG. With this thesis, we release the collected experimental dataset in the public domain, in order for the scientific community to further investigate the various auditory processing phenomena and their relation with EEG, GSR and PPG responses. The dataset can be used also to improve predictive tasks or design novel Brain-Computer-Interface (BCI) systems based on auditory attention. We also use the deeplearning approach to exploit the spatial relationship of EEG electrodes and inter-subject dependency of a model. As a domain application, we finally discuss the implications of auditory attention assessment for serious games and propose a 3-dimensional difficulty model to design game levels and dynamically adapt the difficulty to the player status.
3-mag-2019
File in questo prodotto:
File Dimensione Formato  
phdunige_4081078_1.pdf

Open Access dal 04/05/2020

Descrizione: pp. 1-87
Tipologia: Tesi di dottorato
Dimensione 11.64 MB
Formato Adobe PDF
11.64 MB Adobe PDF Visualizza/Apri
phdunige_4081078_2.pdf

Open Access dal 04/05/2020

Descrizione: pp. 88-96
Tipologia: Tesi di dottorato
Dimensione 11.92 MB
Formato Adobe PDF
11.92 MB Adobe PDF Visualizza/Apri
phdunige_4081078_3.pdf

Open Access dal 04/05/2020

Descrizione: pp. 97-128
Tipologia: Tesi di dottorato
Dimensione 10.41 MB
Formato Adobe PDF
10.41 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11567/943802
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact