Deployment scripts for the podverse ecosystem
To test the Docker containers locally, use the docker-compose.local.yml file, and replace ### with "local".
For stage deployment, use the docker-compose.stage.yml file, and replace ### with "stage".
For prod deployment, use the docker-compose.prod.yml file, and replace ### with "prod".
Duplicate the config/podverse-api-###.example.env file, rename it to config/podverse-api-###.env, and update all of the environment variables to match what is needed for your environment. Repeat these steps for podverse-db-###.env and podverse-web-###.env.
Add your Google API key file to config/google/jwt.keys.json. Look in the config/google/jwt.keys.json.example for more info.
Replace the LETSENCRYPT_HOST and LETSENCRYPT_EMAIL environment variables in the docker-compose.###.yml file.
WARNING: If you use the letsencrypt for SSL on stage or prod, be careful not to run the letsencrypt container too many times. Every time the container starts it requests SSL certificates from letsencrypt, and if you reach that limit, you won't be able to request new SSL certificates for a week. For that reason you may want to remove podverse_letsencrypt_nginx from the following command while you're testing.
First you'll need to create the nginx-proxy network:
docker network create nginx-proxy
Then run the application containers:
docker-compose -f docker-compose.###.yml up -d podverse_nginx_proxy podverse_letsencrypt_nginx podverse_db podverse_api podverse_web
docker exec -d podverse_api_### npm --prefix /tmp run seeds:categories
To add podcasts to the database, you first need to add feed urls to the database, and then run the podcast parser with those feed urls.
You can pass multiple feed urls as a comma-delimited string parameter to the
npm run scripts:addFeedUrls command.
A list of sample podcast feed urls can be found in podverse-api/docs/sampleFeedUrls.txt.
docker exec podverse_api_### npm --prefix /tmp run scripts:addFeedUrls <feed urls>
Orphan feed urls do not have a podcast associated with them.
docker exec podverse_api_### npm --prefix /tmp run scripts:parseOrphanFeedUrls
To parse all non-orphan and public feed urls, you can run:
docker exec podverse_api_### npm --prefix /tmp run scripts:parsePublicFeedUrls
This project uses AWS SQS for its remote queue.
To add all orphan feeds to the queue:
docker exec podverse_api_### npm --prefix /tmp run scripts:addAllOrphanFeedUrlsToPriorityQueue
To add all non-orphan, public feeds to the queue:
docker exec podverse_api_### npm --prefix /tmp run scripts:addAllPublicFeedUrlsToQueue
To add all non-authority feedUrls (podcasts that do not have an authorityId):
docker exec podverse_api_### npm --prefix /tmp run scripts:addNonPodcastIndexFeedUrlsToPriorityQueue
To add all recently updated (according to Podcast Index), public feeds to the priority queue:
docker exec podverse_api_### npm --prefix /tmp run scripts:addRecentlyUpdatedFeedUrlsToPriorityQueue
After you have added feed urls to a queue, you can retrieve and then parse the feed urls by running:
docker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueue -- <retryTimeMS>
We also have a self-managed parsing queue, where we manually mark podcasts to be added to a separate queue for parsing at a regular cadence. The property is Podcast.parsingPriority and the parsingPriority is a value between 0-5. 0 is the default, and means the podcast should not be added to the self-managed queue. 1 is the most frequent, and 5 is the least frequent parsing.
At the time of writing this, 3 is the value we are using the most, which adds the feeds to the queue every 30 minutes.
The offset value is optional, and probably not needed.
docker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:addFeedsToQueueByPriority <parsingPriority> <offset>Then to parse from the self-managed queue call:
docker-compose -f docker-compose.###.yml run podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueueBelow is a sample cron command for adding feeds to queues.
0 */6 * * * docker-compose -f /home/mitch/podverse-ops/docker-compose.###.yml run --rm podverse_api_parser_worker npm run scripts:addAllPublicFeedUrlsToQueue
Feed parsing happens in worker containers that run continuously, and after the worker receives a "No messages found" response from the queue, the worker waits for a timeout period before making another message request to the queue.
docker-compose -f /home/mitch/podverse-ops/docker-compose.###.yml run -d --name podverse_api_parser_worker podverse_api_parser_worker npm run scripts:parseFeedUrlsFromQueue -- 1800000
Below is a sample cron config for requesting unique pageview data from Google Analytics, which is used throughout the site for sorting by popularity (not an ideal system for popularity sorting, but it's what we're using for now).
0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips hour
5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips day
10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips week
15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips month
25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips year
40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- clips allTime
0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes hour
5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes day
10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes week
15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes month
25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes year
40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- episodes allTime
0 * * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts hour
5 */2 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts day
10 */4 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts week
15 */6 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts month
25 */8 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts year
40 */12 * * * /usr/local/bin/docker-compose -f /home/mitch/podverse-ops/docker-compose.prod.yml run --rm podverse_api_stats npm run scripts:queryUniquePageviews -- podcasts allTime
You can run the scripts/db_backup.sh file with cron to automatically backup the database. Follow steps 1 and 2 in this tutorial by Rahul Kumar to install pg_dump, and follow this tutorial by Pranav Prakash to configure the script.
Sample cron command:
0 0 * * * /home/mitch/podverse-ops/scripts/db_backup.sh