dmzio / Sentiment-Analysis-in-Event-Driven-Stock-Price-Movement-Prediction

Use NLP to predict stock price movement associated with news

Sentiment Analysis for Event-Driven Stock Prediction

Use natural-language processing (NLP) to predict stock price movement based on Reuters News

1. Data Collection and Preprocessing

   1.1 get the whole ticker list to obtain the details of public companies

   1.2 crawl news from Reuters using BeautifulSoup

   1.3 crawl prices using urllib

2. Feature Engineering (Tokenization)

   2.1 Unify word format: unify tense, singular & plural, remove punctuations & stop words

   2.2 Implement one-hot encoding

   2.3 Pad word senquence (essentially a matrix) to keep the same dimension

3. Train a set of Bayesian Convolutional Neural Networks using Stochastic Gradient Langevin Dynamics to obtain more robustness

4. Use thinning models to predict future news

Requirement

• Python 3
• PyTorch > 0.4
• numpy
• NLTK
• Crawler tools
  • pip3 install lxml
  • pip3 install bs4
  • pip3 install urllib

Usage

Note: If you don't want to take time to crawl data and train the model, you can also directly go to step 4.

1. Data collection

1.1 Download the ticker list from NASDAQ

$ ./crawler/all_tickers.py 20  # keep the top e.g. 20% marketcap companies

1.2 Use BeautifulSoup to crawl news headlines from Reuters

Note: you may need over one month to fetch the news you want.

Suppose we find a piece of news about COO Lu Qi Resignation on May.18, 2018 at reuters.com

We can use the following script to crawl it and format it to our local file

$ ./crawler/reuters.py # we can relate the news with company and date, this is more precise than Bloomberg News

By brute-force iterating company tickers and dates, we can get the dataset with roughly 400,000 news in the end. Since a company may have multiple news in a single day, the current version will only use topStory news to train our models and ignore the others.

1.3 Use urllib to crawl historical stock prices

Improvement here, use normalized return [5] over S&P 500 instead of return.

$ ./crawler/yahoo_finance.py # generate raw data: stockPrices_raw.json, containing open, close, ..., adjClose
$ ./create_label.py # use raw price data to generate stockReturns.json

2. Feature engineering (Tokenization)

Unify the word format, project word to a word vector, so every sentence results in a matrix.

Detail about unifying word format are: lower case, remove punctuation, get rid of stop words, unify tense and singular & plural.

$ ./tokenize_news.py

3. Train a Bayesian ConvNet to predict the stock price movement.

Type the following to train a set of robust Bayesian models.

$ ./main.py -epochs 500 -static False

4. Prediction and analysis

Let's show one example how the thinning models react to Baidu Lu Qi's resignation

$ ./main.py -predict "Top executive behind Baidu's artificial intelligence drive steps aside"
>> Sell

The prediction makes sence, let's find another one.

Eli Lilly and Co (LLY.N)
FRI, JUN 1 2018
UPDATE 2-Lilly gets U.S. nod for arthritis drug, sets price well below rivals
* Drug priced at $25,000/year, 60 pct lower to AbbVie's Humira

$ ./main.py -predict "UPDATE 2-Lilly gets U.S. nod for arthritis drug  sets price well below rivals"
>> Sell

Lowering down drug prices looks helpful to gain market share in business, however, they didn't mention too much about the updates of technology, we are inclined to regard it as the virulent price competition, which does no help to the company earnings. Thus it is not a bad decision to sell Eli Lilly stocks.

Next, let's see what the buy options look like:

Alphabet Inc (GOOG.O)
WED, MAY 30 2018
Google launches second app in China, woos top smartphone market
* BEIJING Alphabet Inc's Google has launched a file managing tool in several Chinese app stores as it 
* looks for fresh inroads into the world's biggest smartphone market, where most of the internet 
* giant's top products remain banned.

$ ./main.py -predict "Google launches second app in China  woos top smartphone market"
>> Strong Buy

By now, you have basically understood how the models work, let's use backtesting to examine the performance on the news in the past two weeks.

$ ./main.py -eval True
>> Testing    - loss: 0.6761  acc: 58.07%(41.8/72.0) 83.50%(3.3/3.9) 100.00%(0.0/0.0) 0.00%(0.0/0.0) 

Note: the predictions are averaged, which explains why we have float numbers. From left to right, the predictions become more and more confident. 58% is actually much higher than my expectation, I believe when tested on a longer time horizon, the performance gets worse. However, as long as the predictions are better than random guesses (50%), you can't lose money betting on a favorable game (assume no trading cost and liquidity issue).

5. Future work

From the work by Tim Loughran and Bill McDonald, some words have strong indication of positive and negative effects in finance, we may need to dig into these words to find more information. A very simple but interest example can be found in Financial Sentiment Analysis part1, part2

As suggested by H Lee, we may consider to include features of earnings surprise due to its great value.

You are welcome to build a better stopword list and share it.

Issues

1. remove_punctuation() handles middle name (e.g., P.F -> pf)

References:

1. Yoon Kim, Convolutional Neural Networks for Sentence Classification, EMNLP, 2014
2. J Pennington, R Socher, CD Manning, GloVe: Global Vectors for Word Representation, EMNLP, 2014
3. Max Welling, Yee Whye Teh, Bayesian Learning via Stochastic Gradient Langevin Dynamics, ICML, 2011
4. Tim Loughran and Bill McDonald, 2011, “When is a Liability not a Liability? Textual Analysis, Dictionaries, and 10-Ks,” Journal of Finance, 66:1, 35-65.
5. H Lee, etc, On the Importance of Text Analysis for Stock Price Prediction, LREC, 2014
6. Xiao Ding, Deep Learning for Event-Driven Stock Prediction, IJCAI2015
7. IMPLEMENTING A CNN FOR TEXT CLASSIFICATION IN TENSORFLOW
8. Keras predict sentiment-movie-reviews using deep learning
9. Keras sequence-classification-lstm-recurrent-neural-networks
10. tf-idf + t-sne
11. Implementation of CNN in sequence classification
12. Getting Started with Word2Vec and GloVe in Python
13. PyTorch Implementation of Kim's Convolutional Neural Networks for Sentence Classification