This repository contains a hand-curated of great machine (deep) learning resources for Natural Language Processing (NLP) with a focus on Bidirectional Encoder Representations from Transformers (BERT), attention mechanism, Transformer architectures/networks, and transfer learning in NLP.
Source)
Transformer (BERT) (Table of Contents
Expand Table of Contents
Papers
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context by Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le and Ruslan Salakhutdinov.
- Uses smart caching to improve the learning of long-term dependency in Transformer. Key results: state-of-art on 5 language modeling benchmarks, including ppl of 21.8 on One Billion Word (LM1B) and 0.99 on enwiki8. The authors claim that the method is more flexible, faster during evaluation (1874 times speedup), generalizes well on small datasets, and is effective at modeling short and long sequences.
- Conditional BERT Contextual Augmentation by Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han and Songlin Hu.
- SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering by Chenguang Zhu, Michael Zeng and Xuedong Huang.
- Language Models are Unsupervised Multitask Learners by Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever.
- The Evolved Transformer by David R. So, Chen Liang and Quoc V. Le.
- They used architecture search to improve Transformer architecture. Key is to use evolution and seed initial population with Transformer itself. The architecture is better and more efficient, especially for small size models.
- XLNet: Generalized Autoregressive Pretraining for Language Understanding by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.
-
A new pretraining method for NLP that significantly improves upon BERT on 20 tasks (e.g., SQuAD, GLUE, RACE).
-
"Transformer-XL is a shifted model (each hyper-column ends with next token) while XLNet is a direct model (each hyper-column ends with contextual representation of same token)." — Thomas Wolf.
-
A clever dual masking-and-caching algorithm.
- This is NOT "just throwing more compute" at the problem.
- The authors have devised a clever dual-masking-plus-caching mechanism to induce an attention-based model to learn to predict tokens from all possible permutations of the factorization order of all other tokens in the same input sequence.
- In expectation, the model learns to gather information from all positions on both sides of each token in order to predict the token.
- For example, if the input sequence has four tokens, ["The", "cat", "is", "furry"], in one training step the model will try to predict "is" after seeing "The", then "cat", then "furry".
- In another training step, the model might see "furry" first, then "The", then "cat".
- Note that the original sequence order is always retained, e.g., the model always knows that "furry" is the fourth token.
- The masking-and-caching algorithm that accomplishes this does not seem trivial to me.
- The improvements to SOTA performance in a range of tasks are significant -- see tables 2, 3, 4, 5, and 6 in the paper.
- CTRL: Conditional Transformer Language Model for Controllable Generation by Nitish Shirish Keskar, Richard Socher et al. [Code].
- PLMpapers - BERT (Transformer, transfer learning) has catalyzed research in pretrained language models (PLMs) and has sparked many extensions. This repo contains a list of papers on PLMs.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer by Google Brain.
- The group perform a systematic study of transfer learning for NLP using a unified Text-to-Text Transfer Transformer (T5) model and push the limits to achieve SoTA on SuperGLUE (approaching human baseline), SQuAD, and CNN/DM benchmark. [Code].
- Reformer: The Efficient Transformer by Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.
- "They present techniques to reduce the time and memory complexity of Transformer, allowing batches of very long sequences (64K) to fit on one GPU. Should pave way for Transformer to be really impactful beyond NLP domain." — @hardmaru
- Supervised Multimodal Bitransformers for Classifying Images and Text (MMBT) by Facebook AI.
- A Primer in BERTology: What we know about how BERT works by Anna Rogers et al.
- "Have you been drowning in BERT papers?". The group survey over 40 papers on BERT's linguistic knowledge, architecture tweaks, compression, multilinguality, and so on.
Articles
BERT and Transformer
- Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing from Google AI.
- The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning).
- Dissecting BERT by Miguel Romero and Francisco Ingham - Understand BERT in depth with an intuitive, straightforward explanation of the relevant concepts.
- A Light Introduction to Transformer-XL.
- Generalized Language Models by Lilian Weng, Research Scientist at OpenAI.
- What is XLNet and why it outperforms BERT
- Permutation Language Modeling objective is the core of XLNet.
- DistilBERT (from HuggingFace), released together with the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT.
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations paper from Google Research and Toyota Technological Institute. — Improvements for more efficient parameter usage: factorized embedding parameterization, cross-layer parameter sharing, and Sentence Order Prediction (SOP) loss to model inter-sentence coherence. [Blog post | Code]
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators by Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning - A BERT variant like ALBERT and cost less to train. They trained a model that outperforms GPT by using only one GPU; match the performance of RoBERTa by using 1/4 computation. It uses a new pre-training approach, called replaced token detection (RTD), that trains a bidirectional model while learning from all input positions. [Blog post | Code]
- Visual Paper Summary: ALBERT (A Lite BERT)
Attention Concept
- The Annotated Transformer by Harvard NLP Group - Further reading to understand the "Attention is all you need" paper.
- Attention? Attention! - Attention guide by Lilian Weng from OpenAI.
- Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) by Jay Alammar, an Instructor from Udacity ML Engineer Nanodegree.
- Making Transformer networks simpler and more efficient - FAIR released an all-attention layer to simplify the Transformer model and an adaptive attention span method to make it more efficient (reduce computation time and memory footprint).
- What Does BERT Look At? An Analysis of BERT’s Attention paper by Stanford NLP Group.
Transformer Architecture
- The Transformer blog post.
- The Illustrated Transformer by Jay Alammar, an Instructor from Udacity ML Engineer Nanodegree.
- Watch Łukasz Kaiser’s talk walking through the model and its details.
- Transformer-XL: Unleashing the Potential of Attention Models by Google Brain.
- Generative Modeling with Sparse Transformers by OpenAI - an algorithmic improvement of the attention mechanism to extract patterns from sequences 30x longer than possible previously.
- Stabilizing Transformers for Reinforcement Learning paper by DeepMind and CMU - they propose architectural modifications to the original Transformer and XL variant by moving layer-norm and adding gating creates Gated Transformer-XL (GTrXL). It substantially improve the stability and learning speed (integrating experience through time) in RL.
- The Transformer Family by Lilian Weng - since the paper "Attention Is All You Need", many new things have happened to improve the Transformer model. This post is about that.
OpenAI Generative Pre-Training Transformer (GPT) and GPT-2
- Better Language Models and Their Implications.
- Improving Language Understanding with Unsupervised Learning - this is an overview of the original GPT model.
- 🦄 How to build a State-of-the-Art Conversational AI with Transfer Learning by Hugging Face.
- The Illustrated GPT-2 (Visualizing Transformer Language Models) by Jay Alammar.
- MegatronLM: Training Billion+ Parameter Language Models Using GPU Model Parallelism by NVIDIA ADLR.
- OpenGPT-2: We Replicated GPT-2 Because You Can Too - the authors trained a 1.5 billion parameter GPT-2 model on a similar sized text dataset and they reported results that can be compared with the original model.
Additional Reading
- How to Build OpenAI's GPT-2: "The AI That's Too Dangerous to Release".
- OpenAI’s GPT2 - Food to Media hype or Wake Up Call?
- How the Transformers broke NLP leaderboards by Anna Rogers. 🔥🔥🔥
- A well put summary post on problems with large models that dominate NLP these days.
- Larger models + more data = progress in Machine Learning research ❓
- Transformers From Scratch tutorial by Peter Bloem.
- Real-time Natural Language Understanding with BERT using NVIDIA TensorRT on Google Cloud T4 GPUs achieves 2.2 ms latency for inference. Optimizations are open source on GitHub.
- NLP's Clever Hans Moment has Arrived by The Gradient.
- Language, trees, and geometry in neural networks - a series of expository notes accompanying the paper, "Visualizing and Measuring the Geometry of BERT" by Google's People + AI Research (PAIR) team.
- Benchmarking Transformers: PyTorch and TensorFlow by Hugging Face - a comparison of inference time (on CPU and GPU) and memory usage for a wide range of transformer architectures.
- Evolution of representations in the Transformer - An accessible article that presents the insights of their EMNLP 2019 paper. They look at how the representations of individual tokens in Transformers trained with different objectives change.
- The dark secrets of BERT - This post probes fine-tuned BERT models for linguistic knowledge. In particular, the authors analyse how many self-attention patterns with some linguistic interpretation are actually used to solve downstream tasks. TL;DR: They are unable to find evidence that linguistically interpretable self-attention maps are crucial for downstream performance.
- A Visual Guide to Using BERT for the First Time - Tutorial on using BERT in practice, such as for sentiment analysis on movie reviews by Jay Alammar.
- Turing-NLG: A 17-billion-parameter language model by Microsoft that outperforms the state of the art on many downstream NLP tasks. This work would not be possible without breakthroughs produced by the DeepSpeed library (compatible with PyTorch) and ZeRO optimizer, which can be explored more in this accompanying blog post.
Tutorials
- How to train a new language model from scratch using Transformers and Tokenizers tutorial by Hugging Face. 🔥
Videos
BERTology
- XLNet Explained by NLP Breakfasts.
- Clear explanation. Also covers the two-stream self-attention idea.
Official Implementations
- google-research/bert - TensorFlow code and pre-trained models for BERT.
Other Implementations
PyTorch and TensorFlow
- 🤗 Hugging Face Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between TensorFlow 2.0 and PyTorch. [Paper]
- spacy-transformers - a library that wrap Hugging Face's Transformers, in order to extract features to power NLP pipelines. It also calculates an alignment so the Transformer features can be related back to actual words instead of just wordpieces.
PyTorch
- codertimo/BERT-pytorch - Google AI 2018 BERT pytorch implementation.
- innodatalabs/tbert - PyTorch port of BERT ML model.
- kimiyoung/transformer-xl - Code repository associated with the Transformer-XL paper.
- dreamgonfly/BERT-pytorch - A PyTorch implementation of BERT in "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding".
- dhlee347/pytorchic-bert - A Pytorch implementation of Google BERT.
- pingpong-ai/xlnet-pytorch - A Pytorch implementation of Google Brain XLNet.
- facebook/fairseq - RoBERTa: A Robustly Optimized BERT Pretraining Approach by Facebook AI Research. SoTA results on GLUE, SQuAD and RACE.
- NVIDIA/Megatron-LM - Ongoing research training transformer language models at scale, including: BERT.
- deepset-ai/FARM - Simple & flexible transfer learning for the industry.
- NervanaSystems/nlp-architect - NLP Architect by Intel AI. Among other libraries, it provides a quantized version of Transformer models and efficient training method.
- kaushaltrivedi/fast-bert - Super easy library for BERT based NLP models. Built based on 🤗 Transformers and is inspired by fast.ai.
- NVIDIA/NeMo - Neural Modules is a toolkit for conversational AI by NVIDIA. They are trying to improve speech recognition with BERT post-processing.
- facebook/MMBT from Facebook AI - Multimodal transformers model that can accept a transformer model and a computer vision model for classifying image and text.
Keras
- Separius/BERT-keras - Keras implementation of BERT with pre-trained weights.
- CyberZHG/keras-bert - Implementation of BERT that could load official pre-trained models for feature extraction and prediction.
TensorFlow
- guotong1988/BERT-tensorflow - BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
- kimiyoung/transformer-xl - Code repository associated with the Transformer-XL paper.
- zihangdai/xlnet - Code repository associated with the XLNet paper.
Chainer
- soskek/bert-chainer - Chainer implementation of "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding".
Transfer Learning in NLP
As Jay Alammar put it:
The year 2018 has been an inflection point for machine learning models handling text (or more accurately, Natural Language Processing or NLP for short). Our conceptual understanding of how best to represent words and sentences in a way that best captures underlying meanings and relationships is rapidly evolving. Moreover, the NLP community has been putting forward incredibly powerful components that you can freely download and use in your own models and pipelines (It's been referred to as NLP's ImageNet moment, referencing how years ago similar developments accelerated the development of machine learning in Computer Vision tasks).
One of the latest milestones in this development is the release of BERT, an event described as marking the beginning of a new era in NLP. BERT is a model that broke several records for how well models can handle language-based tasks. Soon after the release of the paper describing the model, the team also open-sourced the code of the model, and made available for download versions of the model that were already pre-trained on massive datasets. This is a momentous development since it enables anyone building a machine learning model involving language processing to use this powerhouse as a readily-available component – saving the time, energy, knowledge, and resources that would have gone to training a language-processing model from scratch.
BERT builds on top of a number of clever ideas that have been bubbling up in the NLP community recently – including but not limited to Semi-supervised Sequence Learning (by Andrew Dai and Quoc Le), ELMo (by Matthew Peters and researchers from AI2 and UW CSE), ULMFiT (by fast.ai founder Jeremy Howard and Sebastian Ruder), the OpenAI transformer (by OpenAI researchers Radford, Narasimhan, Salimans, and Sutskever), and the Transformer (Vaswani et al).
ULMFiT: Nailing down Transfer Learning in NLP
ULMFiT introduced methods to effectively utilize a lot of what the model learns during pre-training – more than just embeddings, and more than contextualized embeddings. ULMFiT introduced a language model and a process to effectively fine-tune that language model for various tasks.
NLP finally had a way to do transfer learning probably as well as Computer Vision could.
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning by Sebastian Ruder et al. MultiFiT extends ULMFiT to make it more efficient and more suitable for language modelling beyond English. (EMNLP 2019 paper)
Other Resources
Expand Other Resources
- hanxiao/bert-as-service - Mapping a variable-length sentence to a fixed-length vector using pretrained BERT model.
- brightmart/bert_language_understanding - Pre-training of Deep Bidirectional Transformers for Language Understanding: pre-train TextCNN.
- algteam/bert-examples - BERT examples.
- JayYip/bert-multiple-gpu - A multiple GPU support version of BERT.
- HighCWu/keras-bert-tpu - Implementation of BERT that could load official pre-trained models for feature extraction and prediction on TPU.
- whqwill/seq2seq-keyphrase-bert - Add BERT to encoder part for https://github.com/memray/seq2seq-keyphrase-pytorch
- xu-song/bert_as_language_model - BERT as language model, a fork from Google official BERT implementation.
- Y1ran/NLP-BERT--Chinese version
- yuanxiaosc/Deep_dynamic_word_representation - TensorFlow code and pre-trained models for deep dynamic word representation (DDWR). It combines the BERT model and ELMo's deep context word representation.
- yangbisheng2009/cn-bert
- Willyoung2017/Bert_Attempt
- Pydataman/bert_examples - Some examples of BERT.
run_classifier.py
based on Google BERT for Kaggle Quora Insincere Questions Classification challenge.run_ner.py
is based on the first season of the Ruijin Hospital AI contest and a NER written by BERT. - guotong1988/BERT-chinese - Pre-training of deep bidirectional transformers for Chinese language understanding.
- zhongyunuestc/bert_multitask - Multi-task.
- Microsoft/AzureML-BERT - End-to-end walk through for fine-tuning BERT using Azure Machine Learning.
- bigboNed3/bert_serving - Export BERT model for serving.
- yoheikikuta/bert-japanese - BERT with SentencePiece for Japanese text.
- nickwalton/AIDungeon - AI Dungeon 2 is a completely AI generated text adventure built with OpenAI's largest 1.5B param GPT-2 model. It's a first of it's kind game that allows you to enter and will react to any action you can imagine.
Tools
- jessevig/bertviz - Tool for visualizing attention in the Transformer model.
- FastBert - A simple deep learning library that allows developers and data scientists to train and deploy BERT based models for NLP tasks beginning with text classification. The work on FastBert is inspired by fast.ai.
Tasks
Named-Entity Recognition (NER)
Expand NER
- kyzhouhzau/BERT-NER - Use google BERT to do CoNLL-2003 NER.
- zhpmatrix/bert-sequence-tagging - Chinese sequence labeling.
- JamesGu14/BERT-NER-CLI - Bert NER command line tester with step by step setup guide.
- sberbank-ai/ner-bert
- mhcao916/NER_Based_on_BERT - This project is based on Google BERT model, which is a Chinese NER.
- macanv/BERT-BiLSMT-CRF-NER - TensorFlow solution of NER task using Bi-LSTM-CRF model with Google BERT fine-tuning.
- ProHiryu/bert-chinese-ner - Use the pre-trained language model BERT to do Chinese NER.
- FuYanzhe2/Name-Entity-Recognition - Lstm-CRF, Lattice-CRF, recent NER related papers.
- king-menin/ner-bert - NER task solution (BERT-Bi-LSTM-CRF) with Google BERT https://github.com/google-research.
Classification
Expand Classification
- brightmart/sentiment_analysis_fine_grain - Multi-label classification with BERT; Fine Grained Sentiment Analysis from AI challenger.
- zhpmatrix/Kaggle-Quora-Insincere-Questions-Classification - Kaggle baseline—fine-tuning BERT and tensor2tensor based Transformer encoder solution.
- maksna/bert-fine-tuning-for-chinese-multiclass-classification - Use Google pre-training model BERT to fine-tune for the Chinese multiclass classification.
- NLPScott/bert-Chinese-classification-task - BERT Chinese classification practice.
- fooSynaptic/BERT_classifer_trial - BERT trial for Chinese corpus classfication.
- xiaopingzhong/bert-finetune-for-classfier - Fine-tuning the BERT model while building your own dataset for classification.
- Socialbird-AILab/BERT-Classification-Tutorial - Tutorial.
- malteos/pytorch-bert-document-classification - Enriching BERT with Knowledge Graph Embedding for Document Classification (PyTorch)
Text Generation
Expand Text Generation
- asyml/texar - Toolkit for Text Generation and Beyond. Texar is a general-purpose text generation toolkit, has also implemented BERT here for classification, and text generation applications by combining with Texar's other modules.
- Plug and Play Language Models: a Simple Approach to Controlled Text Generation (PPLM) paper by Uber AI.
Question Answering (QA)
Expand QA
- matthew-z/R-net - R-net in PyTorch, with BERT and ELMo.
- vliu15/BERT - TensorFlow implementation of BERT for QA.
- benywon/ChineseBert - This is a Chinese BERT model specific for question answering.
- xzp27/BERT-for-Chinese-Question-Answering
- facebookresearch/SpanBERT - Question Answering on SQuAD; improving pre-training by representing and predicting spans.
Knowledge Graph
Expand Knowledge Graph
- sakuranew/BERT-AttributeExtraction - Using BERT for attribute extraction in knowledge graph. Fine-tuning and feature extraction. The BERT-based fine-tuning and feature extraction methods are used to extract knowledge attributes of Baidu Encyclopedia characters.
- lvjianxin/Knowledge-extraction - Chinese knowledge-based extraction. Baseline: bi-LSTM+CRF upgrade: BERT pre-training.
License
Expand License
This repository contains a variety of content; some developed by Cedric Chee, and some from third-parties. The third-party content is distributed under the license provided by those parties.
I am providing code and resources in this repository to you under an open source license. Because this is my personal repository, the license you receive to my code and resources is from me and not my employer.
The content developed by Cedric Chee is distributed under the following license:
Code
The code in this repository, including all code samples in the notebooks listed above, is released under the MIT license. Read more at the Open Source Initiative.
Text
The text content of the book is released under the CC-BY-NC-ND license. Read more at Creative Commons.