TurnerSoftware / InfinityCrawler

A simple but powerful web crawler library for .NET

TurnerSoftware/InfinityCrawler Issues

Allow controlling which links are visited
Updated 2 years ago8
Build versioning failed. If this keeps happening, please report it on GitHub
Updated 2 years ago5
Exception thrown on none http/https scheme
Closed 2 years ago2
Switching to AngleSharp
Updated 3 years ago
Javascript Support
Closed 3 years ago3
Add support for wildcard HostAliases
Updated 3 years ago4
Getting "Method not found" error
Closed 3 years ago3
Update NuGet package
Closed 3 years ago2
IAsyncEnumerable support
Closed 5 years ago1
Performance Settings are tied to "TaskHandlerOptions"
Closed 5 years ago1
Add MaxNumberOfPages setting
Closed 5 years ago1
Better support for X-Robots-Tag header
Closed 5 years ago1
Limited settings around the content parser
Closed 5 years ago