GPT-Crawler-UI is a web-based interface for the GPT-Crawler, originally developed by BuilderIO. This project extends the functionality of the original GPT-Crawler by providing a user-friendly web interface, allowing users to easily interact with the crawler without delving into command-line operations.
The original GPT-Crawler can be found here: BuilderIO/gpt-crawler.
The GPT-Crawler-UI consists of two main components:
-
Frontend: A React-based user interface that allows users to input their crawling parameters and view the results in a structured format.
-
Backend: An Express server that handles requests from the frontend, executes the crawling process, and returns the results. The server is designed to restart after each request to ensure a fresh state for every operation.
- Built with Express and integrates the PlaywrightCrawler from Crawlee.
- Handles the crawling logic and data processing.
- The server is designed to restart after each request, ensuring that each crawl starts with a clean state.
Due to the nature of the crawling process and the need to maintain a clean state, the server is set up to restart after each request. This design choice may affect performance and should be considered when deploying the application in a production environment.
To get started with GPT-Crawler-UI:
-
Clone the repository:
git clone https://github.com/JaleelB/gpt-crawler-ui.git
-
Install dependencies:
cd gpt-crawler-ui pnpm i -
Start the server and frontend:
pnpm dev
If you have a suggestion that would make this better, send a PR!
- Special thanks to BuilderIO for the original GPT-Crawler which inspired this project.