Remove URL fragment before storing in `Crawly.Middlewares.UniqueRequest`
tanguilp opened this issue · comments
Tangui commented
Usually a fragment leads to the same page.
oltarasenko commented
Yes, it will improve the situation! I will add it to the scope.
Oshosanya Michael commented
@Ziinc Is this issue still open? Looking for something to work on. If it is still open, please help with a description of the issue.
Matteo Redaelli commented
I think you should change the file /lib/crawly/middlewares/unique_request.ex
The fragment could be removed with something like
"http://example.com/faqs#one" |> URI.parse |> Map.put(:fragment, nil) |> URI.to_string
I could submit a pull request about this small change
Regards
Matteo