Stock prediction based on news articles and tweets.
Work is based on the work of Eliano Marques
The companies directory is composed of:
- The feed should be imported in Kylo : companies.feed.zip (Feed => + => Import File )
- A file to ingest with the list of the companies : companies.csv
- A sample to use when creating the feed : companies-sample.csv
Concerning the stock, you should import the template in Kylo and then create a feed. The directory contains the following files
- The template you should use : quoteTemplateForIngest.xml
- Input file getprices.goog.txt.
In case you would like to recreate a file you should use the following request :
https://www.google.com/finance/getprices?q=MMM&i=86400&p=40Y&f=d,c,v,k,o,h,l&df=cpct&auto=0&ei=Ef6XUYDfCqSTiAKEMg
Note that if you want the last quote from the day you should use the following request :
https://www.google.com/finance/getprices?q=GOOG&i=86400&p=2d&f=d,c,v,k,o,h,l&df=cpct&auto=0&ei=Ef6XUYDfCqSTiAKEMg
Some documentation on google financial api may be found on the unofficial documentation.
This directory all the stuff related to twitter ingestion :
WARNING
Concerning the GetTwitter processor, make sure the clock on the sandbox and the host are synchronized.
yum install -y ntp
service ntpd stop
//At home use
ntpdate pool.ntp.org
//At teradata office use
ntpdate -vd time01.teradata.com
You may also set these to be automatic:
vi /etc/ntp/step-tickers
//write in the file
time01.teradata.com
// setup the service to run automatically
service ntpd start
Nifi template for metadata ingestion for news articles is provided with NewsAPI.
NewsAPI provides over 70+ sources, for which we query one by one.
Directory composition:
- Template Template-NewsAPI-Ingest.xml to query for sources and articles. Input is generated automatically to query NewsAPI about current article sources, then it queries for articles on each source. Output is FlowFiles as an article metadata.
Service no longer available
Service is not free
Elastic is already installed in Kylo (port 9200 and port 9300).
If you want to have a description of your index:
curl -XGET 'localhost:9200/_cat/indices?v&pretty'
To setup Kibana you should follow this tutorial, main steps are :
- Download and install Kibana as a service.
- use Nginx as a reverse proxy.
You could use the port 8989 for kibana.
In the configuration file, instead of using example.com you may use sandbox.kylio.io.
In order to update this document you may use this markdown-cheatsheet.