crwlr.software

crawler

Library for Rapid (Web) Crawler and Scraper Development

url

Swiss Army knife for urls.

query-string

A library for convenient handling of query strings used in HTTP requests.

schema-org

Extract schema.org objects from HTML documents

robots-txt

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

html-2-text

Convert HTML to formatted plain text.

laravel-crawler

Laravel adapter for the crwlr/crawler package.

utils

Utilities that are needed in multiple crwler packages.

package-template

Template repository for new crwlr packages

.github

graphics

Only images to show on github readme pages

crawler-ext-browser

Extension for the crwlr/crawler package containing steps utilizing a headless browser.

crwl-ext-browser

Extension configurations for integration of crwlr/crawler-ext-browser into the crwl.io app.

crwl-extension-utils

Utils for extension packages for the crwl.io app.