scrape hidden letters
boehs opened this issue · comments
not returned via api, often avaliable with direct navigation unless truely deleted
Found API endpoint for directly acessing letters, see https://github.com/boehs/ltc/blob/master/toolkit/endpoints.md
in progress as of 0a8532e
completed as of 9b41b8e
Statistics
- increase of 97756 letters
- 32693 letters were "hidden", either by a mod or the author
- 2314 letters were "feedback" for the moderators and not meant to be public
- 62749 letters were perplexingly missed from the batch archival, possibly from human (my) error
- I do know 700 of the very first letters aren't returned from the API, unclear why but
- all are there now, with no data loss
- 1529 letters were lost forever, perhaps deleted permanently from the source database