simonw / til

Need a new plugin for this, based on https://simonwillison.net/2020/Jan/6/sitemap-xml/

https://til.simonwillison.net/sitemap.xml works now.

Oh interesting - https://til.simonwillison.net/robots.txt now returns:

User-agent: *
Disallow: /tils

That's because datasette-block-robots is also installed.

Interesting that the first time round it was 500 erroring from the datasette-sitemap plugin, but now it's doing the other thing. Reminder that plugin execution order is not guaranteed!

til/metadata.yaml

Lines 10 to 13 in 66dd394

    
             datasette-block-robots: 
        
               literal: |- 
        
                 User-agent: * 
        
                 Disallow: /tils

https://til.simonwillison.net/robots.txt now does:

Sitemap: https://til.simonwillison.net/sitemap.xml
User-agent: *
Disallow: /tils

	datasette-block-robots:
	literal: \|-
	User-agent: *
	Disallow: /tils

Add a sitemap.xml file