Giters
jculvey
/
roboto
A web crawler/scraper/spider for nodejs
Geek Repo:
Geek Repo
Github PK Tool:
Github PK Tool
Stargazers:
67
Watchers:
9
Issues:
13
Forks:
25
jculvey/roboto Issues
Provide external logger
Updated
8 years ago
Question about roboto and memory usage
Updated
9 years ago
Unable to crawl a page with encoding other than utf8
Updated
9 years ago
get stats for each request
Updated
9 years ago
hashbang urls are discarded
Updated
9 years ago
infinite crawl
Updated
9 years ago
Comments count
4
Cant crawl Facebook.com
Updated
9 years ago
Custom queing logic?
Updated
9 years ago
Comments count
2
Crawl any page, parse only whitelisted and not blacklisted pages
Updated
10 years ago
Comments count
2
Remove folder npm_modules and have a .gitignore
Closed
10 years ago
Different Parsers for different domains
Updated
10 years ago
Add path for the item
Closed
10 years ago
Stop/resume
Updated
10 years ago
Comments count
2