Giters

beb7 / gflare-tk

Open-Source Python Based SEO Web Crawler

Home Page:https://greenflare.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Custom robots.txt

beb7 opened this issue · comments

commented

Allow users to supply a custom robots.txt.