/robots.txt is missing
onnimonni opened this issue · comments
Past Issues Searched
- I have searched open and closed issues to make sure that the bug has not yet been reported
Issue is a Bug Report
- This is a bug report and not a feature request, nor asking for support
Describe the bug
Currently when visiting https://lotusdocs.dev/robots.txt it returns 404 error page.
Expected behavior
In order to improve the SEO it would be helpful for the site to return /robots.txt with following content:
User-agent: *
Disallow:
Sitemap: https://lotusdocs.dev/sitemap.xml
This will allow crawling everything and point the crawlers to the sitemap.xml which is already available.
Source for sitemap not supporting relative urls: https://stackoverflow.com/a/14218476/1337062
Source for allowing everything: https://stackoverflow.com/a/44467157/1337062
Screenshots
No response
Environment
No response
If you add enableRobotsTXT=true to your hugo.toml, it should work.
You can also change the file by editing layouts/robots.txt.