harryduong99 / links-crawler

crawl urls of websites

Command

go run main.go mongodb https://website.com

• Supporting databases:
  • Mysql mysql
  • MongoDb mongodb
  • PostgresSQL postgres
  • Also support storing in xlsx (excel) file xlsx

Config

CHUNK_STORING_LINKS = 3