fredwu / crawler

A high performance web crawler / scraper in Elixir.

fredwu/crawler Issues

Switch to Req
Closed 5 months ago
Request for a working example
Closed 10 months ago
High performance: Store gets flooded when too many pages are crawled
Closed 10 months ago6
Filter by dynamic base URL
Closed 10 months ago3
Crawler killed from consuming all memory or processes
Closed 10 months ago17
Retrieving all items in Crawler.Store
Closed 10 months ago2
could not compile dependency :crawler
Closed 4 years ago4
When finished
Closed 4 years ago1
Question: What did you use for the SVG graphics
Closed 4 years ago1
plug_cowboy dependency
Closed 5 years ago4
Cutting a new release?
Closed 5 years ago2
** (ArgumentError) invalid syntax, only "retry", "after" and "else" are permitted
Closed 6 years ago3
Is there a view to determine webfonts used?
Closed 6 years ago1
Links with space are not url encoded and return 404
Closed 6 years ago3
robots.txt
Closed 7 years ago1
Is the crawler able to get async javascript render contents?
Closed 7 years ago1
Why not use floki as a DSL for scraping page content ?
Closed 7 years ago3
Example Project for basic config
Closed 7 years ago8
customization strategy
Closed 7 years ago8