gaoisbest / NLP-Projects

word2vec, sentence2vec, machine reading comprehension, dialog system, text classification, pretrained language model (i.e., XLNet, BERT, ELMo, GPT), sequence labeling, information retrieval, information extraction (i.e., entity, relation and event extraction), knowledge graph, text generation, network embedding

NLP-Projects

Natural Language Processing projects, which includes concepts and scripts about:

• 0_Word2vec

  • gensim, fastText and tensorflow implementations. See Chinese notes, 中文解读

• 1_Sentence2vec

  • doc2vec, word2vec averaging and Smooth Inverse Frequency implementations

• 2_Machine_reading_comprehension

• 3_Dialog_system

  • Categories and components of dialog system

• 4_Text_classification

  • tensorflow LSTM (See Chinese notes 1, 中文解读 1 and Chinese notes 2, 中文解读 2)
  • fastText implementation

• 5_Pretraining_LM

  • Principle of ELMo, ULMFit, GPT, BERT, XLNet

• 6_Sequence_labeling

  • Chinese_word_segmentation
    • HMM Viterbi implementations. See Chinese notes, 中文解读
  • Named_Entity_Recognition
    • Brands NER via bi-directional LSTM + CRF, tensorflow implementation. See Chinese notes, 中文解读

• 7_Information_retrieval

• 8_Information_extraction

• 9_Knowledge_graph

• 10_Text_generation

• 11_Network_embedding

Concepts

1. Attention

• Attention == weighted averages
• The attention review 1 and review 2 summarize attention mechanism into several types:
  • Additive vs Multiplicative attention
  • Self attention
  • Soft vs Hard attention
  • Global vs Local attention

2. CNNs, RNNs and Transformer

• Parallelization [1]

  • RNNs
    • Why not good ?
    • Last step's output is input of current step
  • Solutions
    • Simple Recurrent Units (SRU)
      • Perform parallelization on each hidden state neuron independently
    • Sliced RNNs
      • Separate sequences into windows, use RNNs in each window, use another RNNs above windows
      • Same as CNNs
  • CNNs
    • Why good ?
    • For different windows in one filter
    • For different filters

• Long-range dependency [1]

  • CNNs
    • Why not good ?
    • Single convolution can only caputure window-range dependency
  • Solutions
    • Dilated CNNs
    • Deep CNNs
      • N * [Convolution + skip-connection]
      • For example, window size=3 and sliding step=1, second convolution can cover 5 words (i.e., 1-2-3, 2-3-4, 3-4-5)
  • Transformer > RNNs > CNNs

• Position [1]

  • CNNs

    • Why not good ?
    • Convolution preserves relative-order information, but max-pooling discards them

  • Solutions

    • Discard max-pooling, use deep CNNs with skip-connections instead
    • Add position embedding, just like in ConvS2S

  • Transformer

    • Why not good ?
    • In self-attention, one word attends to other words and generate the summarization vector without relative position information

• Semantic features extraction [2]

  • Transformer > CNNs == RNNs

3. Pattern of DL in NLP models [3]

• Data

  • Preprocess
    • Sub-word segmentation to avoid OOV and reduce vocabulary size
      • sentencepiece
  • Pre-training (e.g., ELMO, BERT)
  • Multi-task learning
  • Transfer learning, ref_1, ref_2
    • Use source task/domain S to increase target task/domain T
  • If S has a zero/one/few instances, we call it zero-shot, one-shot, few-shot learning, respectively

• Model

  • Encoder
    • CNNs, RNNs, Transformer
  • Structure
    • Sequential, Tree, Graph

• Learning (change loss definition)

  • Adversarial learning
  • Reinforcement learning

References

• [1] Review
• [2] Why self-attention? A targeted evaluation of neural machine translation architectures
• [3] ACL 2019 oral

Awesome public apis

• Baidu AI Open Platform
• Tencent AI Open Platform
• Tencent NLP

Awesome packages

Chinese

• pyltp
• HanLP

English

• Spacy
• gensim
• Install tensorflow with one line: conda install tensorflow-gpu

Future directions

• Multi-task learning
• Self-training