Italian Multimodal Corpus: Verbal and Non-Verbal Communication in Political Domain

Abstract

- Speaker gestures are semantically co-expressive with speech and serve different pragmatic functions to accompany oral modality. Therefore, gestures are an inseparable part of the language system: they may add clarity to discourse, can be employed to facilitate lexical retrieval and retain a turn in conversations, assist in verbalizing semantic content and facilitate speakers in coming up with the words they intend to say. This aspect is particularly relevant in political discourse, where speakers try to apply communication strategies that are both clear and persuasive using verbal and non-verbal cues. This dissertation aims to analyze the co-speech gestures of several Italian politicians during face-to-face interviews using a multimodal linguistic approach. The work first introduces the corpus created: PoliModal corpus (Trotta et al., 2019, 2020), containing the transcripts of 56 TV face-to-face interviews of 14 hours, taken from the Italian political talk show “In mezz’ora in più” (for a total of 100,870 tokens) that has been manually annotated with information about metadata (i.e. tools used for the transcription, link to the interview etc.), pauses (used to mark a pause either between or within utterances), vocal expressions (marking non-lexical expressions such as coughs and semi-lexical expressions such as primary interjections), deletions (false starts, repetitions and truncated words), overlaps and facial displays, hand gestures and body posture. Then, the annotation scheme and the results of a series of statistical analyses aimed at understanding the relationship between annotated multimodal traits and language complexity are described in detail and testing the validity of existing studies on political orientation and language use. Finally, after the presentation of an additional semantic annotation layer related to the function assumed by hand movements, the relationship between them and other information layers such as a political party or non-lexical and semi-lexical tags is investigated. Concerning gesture speech relationship, the results obtained suggest that hand movements are mainly used with integrative and complementary functions. So, the information provided by such gestures adds precision and emphasis to spoken information. Its, also show that party affiliation does not significantly influence the gesture-speech relationship. Furthermore - testing the lexical retrieval hypothesis by calculating the association between the hand movements produced by each respondent and discourse disfluencies using weighted mutual information - it is shown that hand movements tend to co-occur with full pauses (i.e., repetition) and empty pauses (i.e., pause) and more frequently with interjections (semi-lexical tags), suggesting that gesticulation may represent an attempt at lexical retrieval. [edited by Author]