VRDate / spider-flow

A new generation of crawler platform that defines the crawler process in a graphical way, and completes the crawler without writing code.

introduction | feature | plugin | DEMO site | Documentation | Update Log | [Screenshot](#Project Part Screenshot) | [Other Open Source](#Other Open Source Project) | Disclaimer

introduce

The platform defines crawlers in the form of flow charts, and is a highly flexible and configurable crawler platform

Features

• Support Xpath/JsonPath/css selector/regex extraction/mashup extraction
• Support JSON/XML/Binary format
• Support multiple data sources, SQL select/selectInt/selectOne/insert/update/delete
• Support crawling JS dynamically rendered (or ajax) pages
• Support Proxy
• Support auto save to database/file
• Commonly used string, date, file, encryption and decryption functions
• Support for plugin extensions (custom executors, custom methods)
• Task monitoring, task log
• Support HTTP interface
• Support automatic cookie management
• Support for custom functions

Plugins

• Selenium plugin
• Redis plugin
• OSS plugin
• Mongodb plugin
• IP proxy pool plugin
• OCR recognition plugin
• Email plugin

Partial screenshot of the project

crawler list

crawler test

Debug

logs

Other open source projects

• spider-flow-vue, the front end of spider-flow
• magic-api, a framework that automatically maps to HTTP interfaces based on XML
• magic-api-spring-boot-starter

Disclaimer

Do not apply spider-flow to any work that may violate legal regulations and moral constraints, please use spider-flow friendly, abide by the spider agreement, and do not use spider-flow for any illegal purposes. If you choose to use spider-flow, it means that you abide by this agreement, and the author will not bear any legal risks and losses caused by your violation of this agreement, and all consequences will be borne by you.