=========================================================================================================
The component and details of the project(__edition_0.5_)
---------------------------------------------------------------------------------------------------------
This is a Sina Weibo Spider archieved by server terminal and client terminal.
The Server terminal mainly deal with 3 tasks.
1. Get proxy from http proxy website. Unfortunately, not all proxy scratched from website
is valid. In fact, almost one quarter of them is valid. So, it is important to check if
those proxy is valid. One thread of server will manage a proxy pool.
Raw and unchecked proxy will scratched from website, and this thread will arrange these
raw proxy to several sub thread, each sub thread will check if the arranged proxy is
valid. If so, these proxy will be sent to valid proxy pool and ready for use.
Besides, the valid proxy may extend after some time. So it is necessary to check if the
proxy in proxy pool is valid.
2. The second part is communicating with clients. Clients will require proxy and task from
server. While client finish the task of getting info of users, they will return the user info
to server. When received this user info , server will store then in cache database.
3. The 3rd part of function is dealing with the data in cache databases. The attends of user
will be checked if already exist in user_info_table. If not, the user will be inserted into
user_info_table. And no matter whether it exists, the user will be deleted from ready_to_get
table. And ,as for attends, if they are not contained in user_info_table, they will be inserted
to ready_to_get table.
==========================================================================================================
UPDATE HISTORY:
_0.5_ : add the function of scratching the history of certain user
_0.4_ : if the average proxy size less than 30, refuse to assignment task to client
_0.3_ : drop the attends whoes fans num less than 1000
_0.2_ : add the partition to monitor the state of proxy pool
_0.1_ : the initial edition