Yngie-C / text-analytics

Unstructured Data Analysis (Graduate) @Korea University

text-analytics

Unstructured Data Analysis (Graduate) @Korea University

Notice

• Syllabus (download) [Video]

Recommended courses

• CS224d @Stanford: Deep Learning for Natural Language Processing
  • Course Homepage: http://cs224d.stanford.edu/
  • YouTube Video: https://www.youtube.com/playlist?list=PLlJy-eBtNFt4CSVWYqscHDdP58M3zFHIG
• CS224n @Stanford: Natural Language Processing Deep Learning
  • Course Homepage: http://web.stanford.edu/class/cs224n/
  • Youtube Video: https://www.youtube.com/playlist?list=PL3FW7Lu3i5Jsnh1rnUwq_TcylNr7EkRe6
• Deep Natural Lanugage Processing @Oxford
  • Course Homepage: https://github.com/oxford-cs-deepnlp-2017/lectures

Schedule

Topic 1: Introduction to Text Analytics

• Text Analytics: Backgrounds, Applications, & Challanges, and Process [Video]
• Text Analytics Process [Video]

Topic 2: Text Preprocessing

• Introduction to Natural Language Processing (NLP) [Video]
• Lexical analysis [Video]
• Syntax analysis & Other topics in NLP [Video]
• Reading materials
  • Cambria, E., & White, B. (2014). Jumping NLP curves: A review of natural language processing research. IEEE Computational intelligence magazine, 9(2), 48-57. (PDF)
  • Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug), 2493-2537. (PDF)
  • Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017). Recent trends in deep learning based natural language processing. arXiv preprint arXiv:1708.02709. (PDF)
  • NLP Year in Review - 2019 (Medium Post)

Topic 3: Neural Networks Basics (Optional, No Video Lectures)

• Perception, Multi-layered Perceptron
• Convolutional Neural Networks (CNN)
• Recurrent Neural Networks (RNN)
• Practical Techniques

Topic 4: Text Representation I: Classic Methods

• Bag of words, Word weighting, N-grams [Video]

Topic 5: Text Representation II: Distributed Representation

• Neural Network Language Model (NNLM) [Video]
• Word2Vec [Video]
• GloVe [Video]
• FastText, Doc2Vec, and Other Embeddings [Video]
• Reading materials
  • Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic language model. Journal of machine learning research, 3(Feb), 1137-1155. (PDF)
  • Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. (PDF)
  • Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111-3119). (PDF)
  • Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543). (PDF)
  • Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2016). Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. (PDF)

Topic 6: Dimensionality Reduction

• Dimensionality Reduction Overview, Supervised Feature Selection [Video]
• Unsupervised Feature Extraction [Video]
• Reading materials
  • Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American society for information science, 41(6), 391-407. (PDF)
  • Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse processes, 25(2-3), 259-284. (PDF)
  • Maaten, L. V. D., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of machine learning research, 9(Nov), 2579-2605. (PDF) (Homepage)

Topic 7: Topic Modeling as a Distributed Reprentation

• Topic modeling overview & Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis: pLSA [Video]
• LDA: Document Generation Process [Video]
• LDA Inference: Collapsed Gibbs Sampling, LDA Evaluation [Video]
• Reading Materials
  • Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American society for information science, 41(6), 391. (PDF)
  • Dumais, S. T. (2004). Latent semantic analysis. Annual review of information science and technology, 38(1), 188-230.
  • Hofmann, T. (1999, July). Probabilistic latent semantic analysis. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence (pp. 289-296). Morgan Kaufmann Publishers Inc. (PDF)
  • Hofmann, T. (2017, August). Probabilistic latent semantic indexing. In ACM SIGIR Forum (Vol. 51, No. 2, pp. 211-218). ACM.
  • Blei, D. M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77-84. (PDF)
  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning research, 3(Jan), 993-1022. (PDF)
• Recommended video lectures
  • LDA by D. Blei (Lecture Video)
  • Variational Inference for LDA by D. Blei (Lecture Video)

Topic 8: Language Modeling & Pre-trained Models

• Sequence-to-Sequence Learning [Slide], [Video]
• Transformer [Slide], [Video]
• ELMo: Embeddings from Language Models [Slide], [Video]
• GPT: Generative Pre-Training of a Language Model [Slide], [Video]
• BERT: Bidirectional Encoder Representations from Transformer [Slide], [Video]
• GPT-2: Language Models are Unsupervised Multitask Learners
• Reading Materials
  • Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in neural information processing systems (pp. 3104-3112). (PDF)
  • Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. (PDF)
  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008). (PDF)
  • Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv preprint arXiv:1802.05365. (PDF)
  • Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. (PDF)
  • Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. (PDF)
  • Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9. (PDF)

Topic 9: Document Classification

• Document classification overview, Vector Space Models (Naive Bayesian Classifier, k-Nearese Neighbor Classifier) [Slide], [Video]
• (Optional) Other VSM-based classsification (Lecture videos are taken from IMEN415 (Multivariate Data Analysis for Undergraudate Students @Korea University))
  • Logistic Regression: [Formulation], [Learning], [Interpretation]
  • Decision Tree: [Recursive Partitioning and Pruning]
  • Artificial Neural Network: [Perceptron], [Multi-layer Perceptron]
  • Ensemble Models" [Overview], [Bagging], [Random Forest], [AdaBoost], [Gradient Boosting Machine (GBM)]
• RNN-based document classification
• CNN-based document classification
• Reading materials
  • Kim, Y. (2014). Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. (PDF)
  • Zhang, X., Zhao, J., & LeCun, Y. (2015). Character-level convolutional networks for text classification. In Advances in neural information processing systems (pp. 649-657) (PDF)
  • Lee, G., Jeong, J., Seo, S., Kim, C, & Kang, P. (2018). Sentiment classification with word localization based on weakly supervised learning with a convolutional neural network. Knowledge-Based Systems, 152, 70-82. (PDF)
  • Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E. (2016). Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1480-1489). (PDF)
  • Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. (PDF)
  • Luong, M. T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. (PDF)

Topic 10: Sentiment Analysis

• Architecture of sentiment analysis
• Lexicon-based approach
• Machine learning-based approach
• Reading materials
  • Hamilton, W. L., Clark, K., Leskovec, J., & Jurafsky, D. (2016, November). Inducing domain-specific sentiment lexicons from unlabeled corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing (Vol. 2016, p. 595). NIH Public Access. (PDF)
  • Zhang, L., Wang, S., & Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4), e1253. (PDF)