MemePlexPics / memeplex

A meme search engine that indexes underground Telergam

MemePlex

Описание

MemePlex - поисковая система для мемов русско- и украиноязычного нижнего телеграма.

Мы индексируем нишевые телеграм-каналы, которые обычно пролетают под радаром крупных индексаторов, которые к тому же, как правило, не позволяют искать кириллицей.

Наше главное отличие: мы не воруем контент, и всегда указываем ссылку на оригинал.

Если у вас есть телеграм-канал, вы можете рекомендовать его к добавлению здесь, либо же добавить его в featured channels, репостнув наш анонс и отписавшись в комментариях.

Description

MemePlex is a meme search engine and catalogue with a focus on Russian and Ukrainian underground Telegram content.

We index niche telegram channels that usually fly under the radar of big meme indexers, that usually don't even allow to search content in languages other than English.

Our distinguishing feature is that we don't steal the content (there's always a link to the original).

If you have a telegram channel, you can suggest it here, or add it to "featured channels" by reposting our announcement (see here) and let us know in the comments.

Links

• Website
• Telegram bot
• Telegram channel (news and announcements)

Requirements

• ocr.space API key
• TelegramApiServer
• TelegramRSS
• Docker
• RabbitMQ (will be installed automatically using the ./docker-compose.yml)
• ElasticSearch (will be installed automatically using the ./docker-compose.yml)

Setup

• Run npm run init to initialize the project
• Get free or paid ocr.space API key and put it into .env
• Set up TelegramApiServer (requires an app key, used to parse telegram channels)
• Set TG_API_ENDPOINT, TELEGRAM_API_ID and TELEGRAM_API_HASH in .env
• Set up TelegramRSS (needed to download image data)
• Set TG_RSS_ENDPOINT in .env
• Set up RabbitMQ - set AMQP_ENDPOINT in .env
• Set ELASTIC_ENDPOINT in .env
• Run docker compose up -d to start docker services
• Run bash ./scripts/elastic-certs.sh to create a self-signed TLS certificate for ElasticSearch
• Run npx tsx ./scripts/create-index.ts to create (or replace) an ElasticSearch index
• Run npm run migrations:run to create MySQL tables
• Run npm run build:front to build static frontend files
• Run npm run build:back to transpile files for backend

Startup

• Run docker compose up -d to start docker services
• Run npm run service:memesearch - Telegram parser
• Run npm run service:web - web server
• Run npm run service:pics - Telegram Pics bot server
• Run npm run service:publisher - Telegram Publisher bot server

Scripts

Sitemap generator

1. Run npx tsx ./scripts/generate-sitemap.ts once a day to generate the sitemap.xml (for all previous days when first run and for the previous day every time after that)
2. Rebuild the frontend using npm run build or just run cp -r ./frontend/src/public/sitemaps ./frontend/dist/. && cp ./frontend/src/public/sitemap.xml ./frontend/dist/sitemap.xml

• It might be a good idea to run it via Cron at the beginning of the day, like this:

0 0 * * * bash -c 'cd /path/to/project && npx tsx ./scripts/generate-sitemap.ts && cp -r ./frontend/src/public/sitemaps ./frontend/dist/. && cp ./frontend/src/public/sitemap.xml ./frontend/dist/sitemap.xml'

Backup

Files:

rsync --archive -progress --update user@host:remote_path local_path

ElasticSearch:

# Run to init a snapshot repository for ElasticSearch
bash ./scripts/backup/elastic-init.sh

# Run to start taking snapshot
bash ./scripts/backup/elastic-snapshot.sh

# Run to restore ElasticSearch from snapshots
bash ./scripts/backup/elastic-restore.sh

Mysql:

ssh user@host "cd ./path_to_project && bash ./scripts/backup/mysql.sh" | gzip > memeplex_$(date +%Y%m%d-%H%M%S).sql.gz