nofreewill42 / ProjectIdeas

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ProjectIdeas

Ételekkel kapcsolatos

Maybe would be better to collect these from common crawl collect all hungarian sites, see if robots.txt exists if yes, extract the sitemap url if exists if doesn't exist, skip it

after all sitemap urls were collected, go through the skipeed websites without a known sitemap, and go through all the known sitemap formats, from most frequent to least...

About