BerkeleyPoulsen / Stock-price-scraper-for-SQL

A python script to scrape stock prices off NASDAQ API and feed it to MySQL

Stock-price-scraper-for-SQL

This is a simple python script to scrape stock prices off NASDAQ API and feed it to MySQL. I have spent the past 2 weeks pondering on what beginner project I should do to apply my SQL knowledge and where do I even obtain the data from? I spent some time researching and found a few option. I can import a csv file into SQLite table using SQLite3 or DB browser for SQLite or scrape the data off a webpage and feed it into MySQL. I decided to take up the challenge to extract the data myself since that is a great way for me to practise on my python skill. Do note that I am scrapping stock prices per minute and not per day.

Set up

Dependencies

• Python 3.8x
• python-dateutil 2.8.1 or newer
• ujson 1.35 or newer
• urllib3 1.25.8 or newer
• mysql-connector-python 8.0.18 or newer
• MySQL 8.0.19 or newer
• Internet connection

Installing dependencies

• Python: You can install Python using Anaconda. It is the easiest way to install python and it comes with a good amount of data science packages.
• MySQL: I used MySQL Community server. Please remember your password as it will need to use it everytime you log into the server.
• python-dateutil,ujson,mysql-connector-python,urllib3 can be installed using pip.

$ pip install mysql-connector-python

or conda

conda install -c anaconda mysql-connector-python

What to do before running code

• Log in using:

mysql -u root -p

• Create a database call stock

CREATE DATABASE stock;

• Select database

USE stock;

• Create a table on your database call nasdaq

CREATE TABLE nasdaq
  (
   datetime DATETIME,
   price FLOAT(2,2),
   symbol VARCHAR,
   created_at TIMESTAMP default now()
  );

• Edit the config file
  • You will need to change the db_user and db_password to your own. This is an essential step to prevent the disclosure of your server details.

How to check if your code works

• Using SELECT * from nasdaq

SELECT * from nasdaq

• You should able to see 4 columns of data (datetime, price, symbol and created_at)

Licensing

This scraper is licensed under the MIT license. You can check the information inside the LICENSE file. To make it short, I wanted it to be free and open, so that anyone can contribute to it.