lucky521 / pyspider

My Web Spider

PySpider

My Python Web Spider

On the Internet, Spider is one of the most interesting and useful technologies. It help us collect and analyse what we want. We need spider to be faster and smarter. My Python Web Spider now is Simple. But I will make it stronger little by little.

Specify Content

Regular Expression

match URL
match target data

Http Request Constructor

Header

Cookies

Sessions

Web URL Constructor

URL: baseURL + internal index

Logger

Status Report

Out to File

Format output

Acceleration

Muli Threading

Reconnection

Reference:

key board interrupt

HTTP header field