kokseen1 / wall-e

A self-hosted Carousell scraper.

WALL-E

WALL-E is a self-hosted Carousell scraper that provides an intuitive Telegram interface via the Telegram Bot API.

WALL-E is easily deployed via Docker Compose either on your local machine or on Heroku for free. Configurations such as the polling frequency can be tweaked by modifying the Python source code.

Prerequisites

• Docker
• Docker Compose
• An available Telegram Bot

Deploying on Heroku

• Heroku CLI
• psql

Installation

Clone this repository

git clone https://github.com/kokseen1/wall-e.git
cd wall-e

Local Deployment

Create a .env file

touch .env

Set environment variables by adding the following lines in .env

BOT_TOKEN=<Telegram bot token>
POSTGRES_NAME=postgres
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
NO_SSL_VERIFY=True

Build and start the containers

sudo docker-compose up

Heroku Deployment

Login to Heroku

heroku login

Create a Heroku app and add the Git remote

heroku create -a <Heroku app name>
heroku git:remote -a <Heroku app name>

Set the stack of your app to container

heroku stack:set container

Add the Heroku postgresql addon

heroku addons:create heroku-postgresql:hobby-dev

Set environment variables

heroku config:set HEROKU_URL=https://<Heroku app name>.herokuapp.com/
heroku config:set BOT_TOKEN=<Telegram bot token>
heroku config:set POSTGRES_NAME=postgres
heroku config:set POSTGRES_USER=postgres
heroku config:set POSTGRES_PASSWORD=postgres

Connect to the database via psql

heroku pg:psql

Create tables

CREATE TABLE sent (
	item_id VARCHAR ( 255 ) NOT NULL,
	chat_id VARCHAR ( 255 ) NOT NULL
);

CREATE TABLE queries (
	query_text VARCHAR ( 255 ) NOT NULL,
	chat_id VARCHAR ( 255 ) NOT NULL
);

Exit from psql

exit

Push to Heroku

git push heroku main

Keep the application awake via a service like Kaffeine

Bot Usage

Add query

/add <query>

Remove query

/rm <query>

List queries

/ls

Fetch queries

/force