skip some item
Brilliant opened this issue · comments
Problem the site refresh when crawling the site, the crawler will miss some item.
Fixup use a window to check the duplicate item, the current version roughly only one item. I think if there exists a continuous sequence, we should stop, and the length of the sequence is 5 or ...