Determine last link seen timeout period for fetching links from reddit.
evancofer opened this issue · comments
Reddit's JSON api will only allow us to fetch 100 links at a time (https://www.reddit.com/dev/api). This presents a problem, as if all 100 of those links have been seen before, we need to use a link id for after
to find posts after the last of these 100 posts. After that, each iteration of 100 posts should do the same thing. However, this is rather time sensitive, as after
is not chronological, but instead correlated to ranking. We will likely need to store it in the cache of the user, and then timestamp their last access to reddigest in the database. With this, we can see if the users after
post is too old or not. On the other hand, if the after
post is still new, it's not too much of a problem, as the user can then just fetch and change after
until they find new posts.