Utilizing Selenium with a Scrapy project that has a pipeline into Azure containers?
haileyneorsd opened this issue · comments
Hi. I am adding a spider to the linked repo below, and so far my spider is working on my local machine.
It's a Scrapy project that has CI and Cron jobs to integrate the spiders into the repo. In Selenium when we use the webdriver to open the browser and extract data from the website, how does that work in a project with a pipeline ? This is a silly question, when the spider runs within the pipeline will there be a way to "open Chrome" and extract the data? Also, in the backend we are using Azure.
Thanks, I know my question sounds silly.