Zhihu Scraper is a simple Python package designed to scrape questions and their associated answers from the popular Chinese Q&A platform Zhihu.
Scrape question details including creation time, update time, title, comment count, vote count, follower count, and view count. Scrape answers for a specific question, capturing details such as author, content, creation and update time, comment count, vote count, and answer URL.
- Python 3.6 or newer
- Required Python packages:
requestsbeautifulsoup4pandasselectolax
- Clone this repository to your local machine.
git clone [URL of this repository]
- Navigate to the directory containing the zhihu_scraper package.
cd path_to_directory_containing_zhihu_scraper
- Install the required Python packages.
pip install -r requirements.txt
- Installing Locally:
Navigate to the directory containing setup.py in your terminal and run:
pip install .
Note: You might want to create a virtual environment to manage the dependencies.
- Prepare a CSV file named all_data.csv containing the list of Zhihu question IDs to be scraped.
- Navigate to the directory containing the zhihu_scraper package.
cd path_to_directory_containing_zhihu_scraper
- Execute the scraper:
python -m zhihu_scraper.main - After successful execution, the scraped data will be saved in a file named zhihu_data.csv.
The scraper is designed to fetch up to a maximum of 216 questions in one go, assuming an average of 4000 answers per question. Please split your list of question IDs accordingly to avoid potential scraping limits set by Zhihu.
This project is open-sourced and available for personal and research use. Please ensure to respect Zhihu's terms of service and use responsibly.