Disboard Server Search Backend

This is the backend API for a Discord Server Search App with improved functiionality such as searching by multiple tags and by exlusion tags.

The python script found in /scripts/scrape.py is adapted from: https://github.com/DiscordFederation/DisboardScraper

Repo Tree

Files of interest can be found here:

client 
|---src // minimalist react client for demo of backend purposes only
scripts
|---scrape.py // scraper called in node.js
|---scrape_all_servers.py // script to populate mongoDB
server
|---app.js // load app
|---app.log // logging for server
|---connect.js // connect to mongoDB
|---helpers.js // parsing, formatting, and other helpers
|---server.js // server + controller

Usage

Install dependencies and run server(s)

cd scripts

Run a virtual shell:

pipenv shell

Install dependencies with makefile:

make

In the event of an inexplicable python error with parsing, please ensure that you are running a virtual env and have all dependencies loaded.

Run the express server:

cd ../server && npm start

If needed, run the react client:

cd ../client && npm start

Run scraper as standalone or debug

The script expects three args: <tag> <pages> <destination>

The tag is self-explanatory.

The max amount of pages allowed by Discord is 50.

The destination can be: server to return to the express server, json to print to terminal in a nice format, or mongodb to directly insert into the database.

For example:

python3 scrapy.py buffy 2 json

Also noteworthy is that if you try to run a scraper from AWS Lambda, your generated IP from AWS's known range will be blocked even. If you know of a workaround to this, please let me know here.

API Endpoints

GET to http://localhost:3333/search for an array of all tags.

POST to http://localhost:3333/tags and in your body, pleae include up to 5 inclusion and exclusion tags:

	body: JSON.stringify({
        tag1: tag1,
        tag2: tag2,
        tag3: tag3,
        tag4: tag4,
        tag5: tag5,
        exclude1: exclude1,
        exclude2: exclude2,
        exclude3: exclude3,
        exclude4: exclude4,
        exclude5: exclude5,
      }),

In the response, you will receive an array of JSON:

[
  {
    'Search Tag': 'buffy',
    'Server Name': 'Buffyverse',
    'Members Online': '208',
    'Creation Date': 'datetime.datetime(2019, 4, 18, 7, 33, 50, 504000',
    'Invite Link': 'https://disboard.org/server/568338386272780314',
    'Tag 1': 'angel',
    'Tag 2': 'buffy-the-vampire-slayer',
    'Tag 3': 'buffy',
    'Tag 4': 'btvs',
    'Tag 5': 'buffyverse'
  },
  {
    'Search Tag': 'buffy',
    'Server Name': 'Slayerverse',
    'Members Online': '82',
    'Creation Date': 'datetime.datetime(2022, 1, 16, 3, 1, 6, 13000',
    'Invite Link': 'https://disboard.org/server/932107165685137468',
    'Tag 1': 'supernatural',
    'Tag 2': 'angel',
    'Tag 3': 'buffy-the-vampire-slayer',
    'Tag 4': 'buffy',
    'Tag 5': 'buffyverse'
  }
]

Dev Notes

Inner single quotes (apostrophes) have been removed; you will instead see an empty space where there should be a single quote. For example, instead of the string "Devil's Land", you will receive "Devil s Land". Parse accordingly.
datetime has been left as is. Import and parse accordingly.
The current mongo collection in use is found at TheClusterName - discord.test_servers

Troubleshooting

If you get a weird python syntax error from the scraper script, check that you have:

run the virtual env in the script directory
loaded all dependencies for the script
run the server in the same shell as the virtual env

Note: Logs can be found in server/app.log

Report Bugs

Please let me know at: sujetdev@sudomail.com and include screenshots of the logs and other relevant details.

yffenim / disboard_search_backend