stefan-mueller/ITAUR-WZB

Introduction to Text Analysis Using R

A One-Day Workshop at the Wissenschaftszentrum Berlin (WZB)

Kohei Watanabe, Department of Methodology, London School of Economics and Political Science

Stefan Müller, Department of Political Science, Trinity College Dublin

Kenneth Benoit, Department of Methodology, London School of Economics and Political Science

Date: January 31, 2018, Wissenschaftszentrum Berlin quanteda version: 0.9.9.65 (CRAN)

This repository contains the workshop materials for a one-day version of a workshop Introduction to Text Analysis Using R taught by Kohei Watanabe and Stefan Müller. The original version of this workshop was developed by Kenneth Benoit.

Structure

The course will taught interactively, as a series of "mini-modules" consisting of presentations of different aspects of quantitative text analysis using R, followed by practical exercises.

Module 1: Overview and demonstration of text analysis using R (9:00-9:30)
Module 2: Basic text data types and functions for text (9:30-10:00)
Module 3: Getting textual data into R (10:00-10:30)
Coffee break (10:30-10:50)
Module 4: Processing and preparing texts for analysis (10:50-12:30)
Lunch break (12:30-13:30)
Module 5: Descriptive analysis (13:30-15:00)
Break (15:00-15:15)
Module 6: Advanced analysis and working with other text packages (15:15-16:30)

Instructions for using this resource

You have three options for downloading the course material found on this page:

You can download the materials by clicking on each link.
You can "clone" repository, using the buttons found to the right side of your browser window as you view this repository. This is the button labelled "Clone in Desktop". If you do not have a git client installed on your system, you will need to get one here and also to make sure that git is installed. This is preferred, since you can refresh your clone as new content gets pushed to the course repository. (And new material will get actively pushed to the course repository at least once per day as this course takes place.)
Statically, you can choose the button on the right marked "Download zip" which will download the entire repository as a zip file.

You can also subscribe to the repository if you have a GitHub account, which will send you updates each time new changes are pushed to the repository.

Objectives

This workshop covers how to perform common text analysis and natural language processing tasks using R. Contrary to a belief popular among some data scientists, when used properly, R is a fast and powerful tool for managing even very large text analysis tasks.

The course comprises six mini "modules", each consisting of guided instruction in the form of presentation of methods followed by student implementation of those methods prior to advancing to the next module. Only after we have completed the practical exercises will we advance to the next stage.

We will cover how to format and input source texts, how to structure their metadata, and how to prepare them for analysis. This includes common tasks such as tokenisation, including constructing ngrams and "skip-grams", removing stopwords, stemming words, and other forms of feature selection. We show how to: get summary statistics from text, search for and analyse keywords and phrases, analyse text for lexical diversity and readability, detect collocations, apply dictionaries, and measure term and document associations using distance measures. Our analysis covers basic text-related data processing in the R base language, but most relies on the quanteda package (https://github.com/kbenoit/quanteda) for the quantitative analysis of textual data. We also cover how to pass the structured objects from quanteda into other text analytic packages for doing topic modelling, latent semantic analysis, regression models, and other forms of machine learning.

Prerequisites

While it is designed for those who have used R in some form previously, expertise in R is not required, and even those with no previous knowledge of R are welcome.

Reading materials

Designed to be done before the course or after, to augment what is presented during the course. These are just suggestions -- no reading before the course is required.

Browse the quanteda R package website, http://quanteda.io, especially the tutorials (also known as vignettes).
[Sanchez, G. (2013) Handling and Processing Strings in R Trowchez Editions. Berkeley, 2013.](http://www.gastonsanchez.com/Handling and Processing Strings in R.pdf)
stringi package page, which also includes a good discussion of the ICU library
Some guides to regular expressions: Zytrax.com's User Guide or the comprehensive resources from http://www.regular-expressions.info
See the quanteda tag on Stack Overflow, where you can pose questions and see some brilliant answers by our development team.

Schedule

Module 0: Installation and setup of R and relevant packages

CRAN for downloading and installing R
GitHub page for the quanteda package
Exercise:
- Make sure you have at least R 3.3.3 installed. Recommended is the latest version of R, as of 27 April 2017, is 3.4.0.
- Make sure your packages are up-to-date. From the command line, run
```
update.packages(ask = FALSE)
```
- Install quanteda from CRAN. From the "Packages" pane in RStudio, or from the command line:
```
install.packages("quanteda")
```
- Install readtext from GitHub, following these instructions.
- Try "knitting" this RMarkdown file: test_setup.Rmd. If it builds without error and looks like this, then you have successfully configured your system.

Module 1: Overview and demonstration of text analysis using R

Demonstration of text analysis using R: demo.R
Overview, motivation, and philosophy of the quanteda package (pdf slides)
Exercise: Step through execution of this example code workflow.Rmd.

Module 2: Basic text data types and functions for text

Basic text manipulation using R
Exercise: Step through execution of the .Rmd file.

Module 3: Getting textual data into R

Install the readtext package for this section.

Getting textual data into R
Example of importing and segmenting text: 2016 US Presidential Debates
Exercise: Try importing some texts of your own.
Sample data files: SOTU_metadata.csv, inaugTexts.csv, tweetSample.RData

Module 4: Processing and preparing texts for analysis

Text processing in R
Exercise: Step through execution of the .Rmd file.

Module 5: Descriptive analysis

Descriptive analysis of texts
Exercise: Step through execution of the .Rmd file.

Module 6: Advanced analysis and working with other text packages

Advanced Analysis

Advanced analysis
Analyzing social media
Analyzing a new corpus: Example from the Guardian
Exercise: Step through execution of the .Rmd file
Twitter analysis example, and the instructions for setting up your own Twitter app, in Twitter.Rmd.

Integration with other packages

TADA - The Text as Data App: A graphical user interface (GUI) for quantitative text analysis and visualisation based on various R packages.
stm: See the convert() function, but also I have an outstanding pull request with the stm package maintainer to work directly on a quanteda dfm-class object.
tokenizers: A package for tokenizing text in many different and flexible ways.
readtext: Import and handling for plain and formatted text files.
spacyr: An R wrapper to the spaCy "industrial strength natural language processing" Python library from http://spacy.io.
tidytext: Text mining using dplyr, ggplot2, and other tidy tools.

Module 7: Tell us about your problems

This session is intended for students to describe their own challenges and for the instructors to describe how to solve them. If you have some data you'd like us to work on live, as part of our interactive answers to your problems, you are encouraged to put them somewhere that can be accessed online, so that we will be able to access them in the class. Feel free to file issues at https://github.com/kbenoit/ITAUR/issues or even fork the repository, make a change, and issue a pull request to correct it!

stefan-mueller / ITAUR-WZB