This is a simple Nodejs web scraping tool that runs a task scheduler(cron) which mines product information (particulary sneakers from an e-commerce store) to check for consistency in sneaker sizes. If there's any inconsistency, the server send an email to notify the user
- Server - NodeJS
- Headless Browser - Puppeteer
- Database - MongoDB
- ODM - Mongoose
- Message Queue - RabbitMQ
- Email Automation - Nodemailer + Sendgrid
- CLI tool - Inquirer
- Cloud PaaS - Heroku
- Scaffold application
- Install dependencies
- Initialize git
- Setup express server
- Setup config files
- Connect to database server
- Create scraper function
- Create database model and controller
- Implement CRUD functions
- Load database with mock data
- Manually test tracking process with mock data
- Setup email automation
- Setup task scheduler
- Load database with realtime data
- Test app in production env
- Abstract heavy tasks to workers - setup RabbitMq with amqp
- Create CLI tool for running database updates
- Write suitable tests -- in progress
- Deploy to production