Crawls websites recursively. High Performance, with seed DB and store to index. Written in Rust.
Web Crawler is a proof-of-concept written in Rust. It reads a website from commandline or a seed database and starts to crawl all reachable sites, adds unknown external sites to a database and reads these. All found sites are added to a elastic search index.
The indexed sites can be accessed by web crawlers companion - the Search UI.
This is only valid for Mac - for those who likes windows: Good luck.
- Install Rust. The site has a good documentation.
- Install Postgres as an App - the most easy way to get Postgres up and running on a mac.
- Install and run Postgres Admin
- Install Elastic Search and Kibana (optional)
- Start all these (Run PostgresApp, Elastic Search)
Now you have the Rust language installed and all databases. You just need a Postgres Database instance and a seed table. Open Postgres Admin UI and create a Database named "webcrawler_dev". Copy contents of create_seed_table.sql to a SQL console in the admin and execute.
you can now run by
$ cargo run https://my_seed_site.comA good first-start url my be https://www.t-online.de/ or any site based what you are focussed at. or
$ cargo runIn the first case only your site will be crawled. In the second case the seed database is queried, all entries are scanned and will be crawled. If any external sites are linked, these will be added to this database. So database will grow.
If you have the Search UI installed from Github you now can open a search console on http://localhost:3000 and check your results
- Stabilize. After some thousands crawled sites it hangs
- Make some real logs. Don't let the log grow infinitely
- Improve read of seed-table. Update with last crawled date, add a 'max_deeph' factor
- Add an ignore table. Some external links should never be crawled like Add-Sites, Porn, Illegal stuff and so on. Ignore them early.
Free to use. Be polite and reference to its original creator
Written by Thorsten Claus, Dortmund, Germany