WindBlaze1 / Google-Scraper

It scrapes google for multiple search queries by using user-provided proxies and stores the result in user's google spreadsheet

Google-Scraper

This Project Scrapes Google for a Search Query and Stores the results in a Google Spreadsheet

It has an Elegant and Clean UI developed in Bootstrap Studio and the backend is written in Django

Some Screenshots of its working:

Results of Scraping:

Logs:

How to Run:

First, extract the ZIP files in a folder, let's call it main.

You will need a Google service account json credentials file for authorization in pygshees:

1. Head to Google Developers Console and create a new project (or select the one you have.)

2. You will be redirected to the Project Dashboard, there click on “Enable Apis and services”, search for “Sheets API”.

3. In the API screen click on ‘ENABLE’ to enable this API.

4. Similarly enable the “Drive API”. We require drives api for getting list of spreadsheets, deleting them etc.
5. Go to Sheets API screen, then go to “Credentials” tab and choose “Create Credentials > Service Account Key”.
6. Next choose the service account as ‘App Engine default’ and Key type as JSON and click create:

7. You will now be prompted to download a .json file. This file contains the necessary private key for account authorization. Name the file as 'service_account_sheets.json'.

This is how this file may look like:

{ "type": "service_account", "project_id": "p....sdf", "private_key_id": "48.....", "private_key": "-----BEGIN PRIVATE KEY-----\nNrDyLw … jINQh/9\n-----END PRIVATE KEY-----\n", "client_email": "[e-mail]", "client_id": "10.....454", }

8. After getting the json file, paste it in the main folder.

Now, to install all the dependencies:

1. Open the main folder in the terminal.
2. Write this command:

pip install -r requirements.txt

3. After all dependencies are installed successfully, run the server:

python manage.py runserver

4. Now, you will get a link in the terminal, open that in a browser.