How the crawler works?

[asac1995]

Hello, how are websites indexed? Is there a predefined site list? Isn't there an option for visitors to add their sites to be indexed?

[Ajink Gupta]

For visitors, there is an option to submit their website via the spider: http://khoj.wexr.site/spider/

However, in the current demo version, I have added a simple protection to prevent others from entering their website.

you can find it from sourcecode in spider folder !!
As of now this project is not meant for production !