internetarchive / heritrix3

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

https://heritrix.readthedocs.io/

internetarchive/heritrix3 Issues

Question about memory usage
Closed 2 years ago3
Heritrix 3.4.0-SNAPSHOT-2022-03-08T19:15:59Z keeps pausing..
Updated 7 months ago1
Text versions of DNS should be recorded as WARC-Type resource instead of response
Updated 8 months ago
Redirect field in seeds-report.txt is only populated for status 301 and 302
Updated 9 months ago
Support for silent option when running a job
Updated 10 months ago
archive web crawler - crawl speed
Closed a year ago7
Error when more than 125 jobs are instantiated
Closed a year ago4
Provided seed files are updated (the more the job is repited, the more they are modified)
Updated a year ago
How to change auth type?
Updated a year ago
WARNING: politessDelay unset, returning default 5000
Updated a year ago
Maven build fails due to HTTP only upstream servers
Updated a year ago4
Compatibility problems with Sonatype release process
Closed a year ago
Error: Could not find or load main class org.archive.crawler.Heritrix Caused by: java.lang.ClassNotFoundException: org.archive.crawler.Heritrix
Closed a year ago2
Question about the size of the 'state' directory
Closed 2 years ago3
RateLimitGuard.authenticate() authentication failure
Closed 2 years ago2
SurtPrefixedDecideRule doesnt seem to work as expected
Closed 3 years ago3
Exclude PDF-Files
Closed 2 years ago8
Question: how to create a new log/report for a single class
Closed 2 years ago
Question re: cloudfront.net
Closed 2 years ago1
Question
Closed 2 years ago
How to scale Heritrix with Kubernetes?
Closed 2 years ago
How to cite?
Closed 2 years ago2
Impact of log4j CVE-2021-44228 on heritrix3?
Closed 2 years ago1
Authentication on servers using Oauth2
Closed 2 years ago4
cannot resolve these dependencies
Closed 2 years ago10
[Question] Where should i set the content obtained from http request ?
Closed 2 years ago1
Resume a crawl for later
Closed 2 years ago
Questions about TransclusionDecideRule
Closed 2 years ago6
Implicit max. value of URI cost and precedence (?) should raise warning if exceeded
Updated 2 years ago1
Time is not stopped when Disk Space Monitor is triggered and report files are removed
Closed 2 years ago5
Bean reference missing inherited properties
Updated 2 years ago
${launchId} is not being replaced (sometimes)
Updated 2 years ago1
Apple Silicon docker images
Updated 2 years ago1
Job build fails with NoClassDefFoundError: org/slf4j/LoggerFactory
Closed 2 years ago2
ExtractorHTML matches srcset attribute case-sensitively
Closed 2 years ago
Heritrix not ignoring robots.txt
Closed 2 years ago2
JDK18: ExtractorMultipleRegexTest fails due to Groovy asm incompatiblity
Closed 2 years ago
HTTP/2 protocol
Updated 2 years ago1
Overcrawling due to sitemap links acting like transclusions
Closed 2 years ago
"java.lang.NoClassDefFoundError: Could not initialize class org.archive.util.CLibrary" on Apple Silicon
Closed 2 years ago1
Setting of maxLogFileSize in the BDBModule is ineffective
Closed 2 years ago
Heritrix crasching on malformed Content-Length header
Closed 2 years ago
Commas in srcset-URLs are not handled correctly
Updated 2 years ago1
Crawl job stats and reports misleading when excluding PDF-Files (follow up to issue #453)
Updated 2 years ago3
Build failing via maven-assembly-plugin: group id is too big
Closed 3 years ago1
Allow plain HTTP console access (as a non-default option)
Updated 3 years ago1
Java version check throws StringIndexOutOfBoundsException on exact major versions
Closed 3 years ago
[Question] SEVERE Configuration problem: Unable to locate Spring NamespaceHandler for XML schema namespace
Closed 3 years ago4
force HTTP11 true - breaking warc records
Closed 3 years ago2
ExtractorChrome exception on images as data uris
Closed 3 years ago