How to prevent timeouts on my RPI4? Only one URL being scraped?
dxwoodward opened this issue · comments
If you see this screenshot here, I'm not very confident in the logs I'm seeing.
Is there something I can do to prevent this? I'm scraping every 15 seconds. I thought the Pi would be able to keep up at that rate? Right? https://imgur.com/a/ykUWdoz
I was browsing the source code here and I think this scraper uses Selenium primarily? I tried that at one point on my own scraper on a Pi2 and it was way to intensive, at least on the Best Buy site. BeautifulSoup is much more lightweight and as long as you pass the user-agent works for Best Buy too without having to get all the javascript etc.