tokenmill / crawling-framework

Easily crawl news portals or blog sites using Storm Crawler.

tokenmill/crawling-framework Issues

Deleting http source should invoke dialog menu
Closed a year ago
Relative date should be the date of parsing
Closed a year ago
Autocomplete on source field
Closed a year ago
Topology Worker Increase Not Doing Crawling.
Updated 5 years ago
Provide default values for ES index creation script
Closed 5 years ago
Start crawler with a simple `docker-compose up`
Closed 5 years ago
Setup a proper CI/CD
Closed 5 years ago
Separate integration tests from unit tests
Updated 5 years ago
Prepare a demo crawler with bbc configuration
Updated 5 years ago
ES index config should have `fielddata` enabled for text fields
Closed 6 years ago
tag cloud in kibana on text field
Closed 6 years ago1
Export some data by query
Updated 6 years ago
Extracted document should hava name from source for filtering
Updated 6 years ago
Article fingerprint for deduplication
Closed 6 years ago1
Use canonical url for duplicate detection
Closed 6 years ago
Search for tests
Closed 6 years ago
Inside source config indicate that the crawl is not happening
Updated 6 years ago1
Management UI should have paging for tests
Closed 6 years ago
Management UI related write actions should be commited Immediately
Closed 6 years ago
REST API to import/export crawler configuration
Updated 6 years ago
Evaluate http sources from CSV
Updated 6 years ago
URL date should not be used as publication date hint
Closed 6 years ago
Bulk update updates docs with wrong url
Closed 6 years ago
BulkProcessor and the RestHighLevelClient treats url encoded urls differently
Closed 6 years ago
Management UI null pointer exception
Closed 6 years ago
HttpArticle should include language attribute
Closed 6 years ago
Management UI paging of HTTP sources
Closed 6 years ago1
Management UI search as you type
Closed 6 years ago1
URL status after partial data analysis should have a unique status
Closed 6 years ago
Log actual LD+JSON on parsing error
Updated 7 years ago
Write installation instructions and documentation
Closed 7 years ago
Create example project which uses crawling-framework
Closed 7 years ago1
Auto-create ElasticSearch indexes if they don't exist
Updated 7 years ago