Workshop on Dutch language models for historical research
This workshop will be held on Dec 9th, 2022 at the Netherlands eScience Center.
If you're interested in attending the workshop, please contact Pim Huijnen (p.huijnen@uu.nl).
Description of workshop
The rise of transformer language models such as BERT has opened up possibilities to use contextualized word embeddings for downstream text processing tasks. This includes applications in humanities research, such as investigating the conceptual history of a certain topic in a collection of texts.
However, the current available language models for Dutch (at time of writing) fall short for historical research, because they are trained only on recent data. The performance of models can decline steadily when applied on data that lies outside of the distribution of the training on corpora. On the other hand, pretraining a new language model for each new research task is too computational intensive and not always desirable. Therefore, the need arises for broad general language models on historical texts. At the same time, the methods to properly use these models for historical research are still under development.
The aim of this workshop is to share knowledge on the state-of-the-art of language models for historical research, and to coordinate and lay out a strategy for training language models for historical Dutch. This strategy includes plans on training corpora, model sizes and architectures and model evaluation. The workshop is intended for researchers and practitioners developing or working with Dutch language models.
The morning session will consist of presentations on the following topics:
- Methodologies for using language models for historical research
- Evaluation of language models for historical research
The afternoon consists of discussion tables, providing input for a white paper on a strategy for training language models for historical Dutch. We will touch on the following topics:
- Suitable training corpora
- Suitable model size and architectures
- Pretraining vs. finetuning
- Evaluation of models
- Methodologies for using language models for historical research