This repository contains a set of Python scripts utilizing Redis for data processing. The scripts perform various tasks, such as adding initial data to Redis sets, scraping URLs, and processing email tasks.
- Adds an initial email and URL to respective Redis sets.
- Don't forget to add the first email and urls in
- Usage:
python first_task.py- Continuously processes URLs and emails from Redis sets, adding them to corresponding queues.
- Usage:
python create_tasks.py- Scrapes content from URLs, extracts new URLs and email addresses, and adds them to Redis sets.
- Usage:
python execute_scraping_tasks.py- Periodically processes URLs from a Redis queue and updates a CSV file.
- Usage:
python execute_add_to_list_tasks.py- Processes email tasks from a Redis queue and updates a CSV file.
- Usage:
python process_email_tasks.pyModify the scripts with your actual Redis server information, queue names, file names, and intervals based on your requirements.
- Python (Version 3.x)
- Redis server running locally or with proper connection details
- Install requirements using requirements.txt
pip install -r requierment.txt