mervynteo / web-scraping-magic-with-scrapy-and-python

This repository contains my experiments with Scrapy for advanced web scraping in Python

Web Scraping in Python using Scrapy (with multiple examples)

This repository contains the code for the article "Web Scraping in Python using Scrapy (with multiple examples)" here - https://www.analyticsvidhya.com/blog/2017/07/web-scraping-in-python-using-scrapy/

Project Structure

.
+-- ourfirstscraper

|	+-- scrapy.cfg - scrapy configuration file

|	+-- ourfirstscraper
  
|	  +-- settings.py - contains all the settings associated with this project
   
|		  +-- spiders - contains the spiders from the tutorial
   
|			  +-- redditbot.py - Reddit Spider
 
|			  +-- shopclues.py - ShopClues Spider
 
|			  +-- techcrunch.py - Techcrunch Spider
 

Dependencies

• Python 2

• Scrapy1.4.0