kbatsuren

Motliere

A Python package to segment French words into morphological subwords.

zett

Code for Zero-Shot Tokenizer Transfer

lang-khk

Finite state and Constraint Grammar based analysers and proofing tools, and language resources for the Halh Mongolian language

pyfoma

Python Finite-State Toolkit

InfiniTransformer

Unofficial PyTorch/🤗Transformers(Gemma/Llama3) implementation of Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

WP-workflow-ComputEL2022

lexd

A lexicon compiler for non-suffixational morphologies

wtpsplit

Toolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.

ltg-bert

LTG-Bert

elc-bert

ml-tutorials

"Machine Learning" онлайн хуралд ашиглагдах материал юм

albert-mongolian

ALBERT trained on Mongolian text corpus

SWOWEN-2018

English Small World of Words SWOWEN-2018

genbench_cbt_2023

The official Genbench Collaborative Benchmarking Task repository 2023 (Archived)

length-generalization

Code for the paper "The Impact of Positional Encoding on Length Generalization in Transformers", NeurIPS 2023

baseline-pretraining

Code for pre-training BabyLM baseline models.

2023InflectionST

2023glossingST

A repo for the 2023 Sigmorphon glossing shared task

WAX

The respository describing a novel datasets for word association explanations

kuzdra

A nonce sentence translated into 100 languages

norare-cldf

CLDF dataset of norare-data

bilingual-abstracts-corpus

Bilingual corpus of scientific abstracts from ÚFAL Charles University publications.

2022-shared-tasks

Negative-Precedent-in-Legal-Outcome-Prediction

This is a repository for code used in the paper: On the Role of Negative Precedent in Legal Outcome Prediction

superbizarre

Code and data for "Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words"

text_characterization_toolkit

A library for computing diverse text characteristics and using them to analyze data sets and models with ease.

2022SegmentationST

SIGMORPHON 2022 Shared Task on Morpheme Segmentation

morphoeval

Evaluation for unsupervised morphological analysis and segmentation

wfdt

2022SegmentationST

SIGMORPHON 2022 Shared Task on Morpheme Segmentation