Syllabus
WS 22/23
Instructor: Mareike Hartmann (mareikeh@lst.uni-saarland.de)
Meetings: Thursdays 14:15 - 15:45, C7.1 Room U15
Office Hours: by appointment
Cross-lingual learning is a form of transfer learning, and refers to methods which learn from a task in one language and transfer this knowledge to a task in another language, which is particularly helpful to solve tasks in languages with few training resources. In this seminar, we will discuss different methods for cross-lingual learning, focusing on transfer via multilingual word embeddings and pre-trained multilingual language models, and cover recent applications for cross-lingual transfer.
Prerequisites: Background in natural language processing and deep learning is required.
Topics and papers to be discussed:\
| Topic | Readings |
| Multilingual word embeddings | Mikolov et al. (2013b): Exploiting Similarities among Languages for Machine Translation
Faruqui and Dyer (2014b): Improving Vector Space Word Representations Using Multilingual Correlation
Mrkšić et al. (2017): Semantic Specialization of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
Additional background material:
Ruder et al. (2019): A Survey of Cross-lingual Word Embedding Models|
| Bilingual dictionary induction: Inducing word-to-word translations from comparable data | Bergsma and van Durme (2011): Learning Bilingual Lexicons Using the Visual Similarity of Labeled Web Images
Kiela et al. (2015): Visual Bilingual Lexicon Induction with Transferred ConvNet Features
Rapp (1999): Automatic Identification of Word Translations from Unrelated English and German
Irvine and Callison-Birch (2017): A Comprehensive Analysis of Bilingual Lexicon Induction |
| Multilingual word embeddings: Unsupervised approache | Conneau et al. (2018): Word translation without parallel data
Artetxe et al. (2018): A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
Søgaard et al. (2019): On the Limitations of Unsupervised Bilingual Dictionary Induction|
| Evaluating multilingual word embeddings | Upadhyay et al. (2016): Cross-lingual Models of Word Embeddings: An Empirical Comparison
Glavaš et al. (2019): How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions](https://aclanthology.org/P19-1070.pdf)|
| Transformer-based multilingual language models | Conneau and Lample (2019): Cross-lingual Language Model Pretraining
Conneau et al. (2020): Unsupervised Cross-lingual Representation Learning at Scale |
| Cross-lingual effectiveness of multilingual language models | Wu and Dredze (2019): Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
Pires et al. (2019): How Multilingual is Multilingual BERT?|
| Essential elements for multilinguality| Karthikeyan et al. (2020): Cross-Lingual Ability of Multilingual BERT: An Empirical Study
Dufter and Schütze (2020): Identifying Elements Essential for BERT’s Multilinguality
Conneau et al. (2020): Emerging Cross-lingual Structure in Pretrained Language Models
Muller et al. (2021): First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
Lin et al. (2019): Choosing Transfer Languages for Cross-Lingual Learning|
| Seq2seq multilingual language models | Xue et al. (2021): mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Chi et al. (2021): mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs |
| Limitations of multilingual language models| Lauscher et al. (2020): From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Wu and Dredze (2020): Are All Languages Created Equal in Multilingual BERT?|
| Benchmarks for evaluating cross-lingual transfer| Hu et al. (2020): XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
Liang et al. (2020): XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
Ruder et al. (2021): XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation|
| Applications for cross-lingual transfer learning | de Vries et al (2022): Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages
Riabi et al. (2021): Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
Perez-Beltrachini and Lapata (2021): Models and Datasets for Cross-Lingual Summarisation
Kondratyuk and Straka (2019): 75 Languages, 1 Model: Parsing Universal Dependencies Universally
Müller-Eberstein et al. (2021): Genre as Weak Supervision for Cross-lingual Dependency Parsing
Zhang et al. (2021): On the Benefit of Syntactic Supervision for Cross-lingual Transfer in Semantic Role Labeling
Chen et al. (2021): Zero-Shot Cross-Lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders|