gatesDect
A NLP classifier to determine whether an article was written by Bill Gates or someone else
You can find a full documentaion of what I did in this article.
Now a slight explanation of the files.
The xml files containing the URLs of the articles I want to scrape.
parseSitemapToFile.py: parses the XML file to a textfile containing only the URLs seperated by newline.
All programs that are starting with scrape are there to use the websitesURL files, generated by parseSitemapToFile.py, to scrape the contents of the articles find at the URLs and put the result either in raw_gates or raw_random_articles folder.
build_dataset.py : To take the files of the raw folders and split them into test , train and valid folder.
classifier_ml.py : A classifier which uses ML / NLP techniques.
###First optimization (26 October 2018) #####
-Added the folder "add_more_test_data" with some additional scraper programs in order to generate even more train/test data. -Edited the classifier.py class : - new word for stop_word list called "link" .
You can find a detailed change log here : https://medium.com/@julianhatzky/gatesdect-part-2-optimizing-nlp-d7f13e9e0e18