aoeer39 / aio_webcrawler

Get images or infomation on website with aiohttp & aiofiles

异步卫星信息爬虫

Introduction

From the Gunter's Space Page The webcrawler use the asyncio to speed up the download process.

Requirements

• BeautifulSoup
• aiocsv
• aiofiles
• aiohttps

Usage

0. make the directories.

mkdir img
mkdir doc

1. get the satellites indexes

python3 ./aio_get_sat_urls.py

The program will save all satellites' url to all_satellites_urls.
And aio_get_sat_urls.py will crawl informations base on saved urls in all_satellites_urls.

2. get the satellites informations from the indexes

python3 ./aio_get_sat_info.py

Note To modify the webcrawler saved files' contexts, please read the comments in the code.