clemfromspace / scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The key not found error shows up

JDRanpariya opened this issue · comments

I started using this module first it shows yellow underline in vscode under module name and when i print response.meta dict it contains only following:
{'download_timeout': 180.0, 'download_slot': 'golden.com', 'download_latency': 0.34168577194213867}
it doesn't contain screenshot or driver etc.
what's the problem

+1, still same

Same issue here, @JDRanpariya , @boardsofcanada any clue ?

The issue was in my Selenium Contract tests. Testing by default yields normal Scrapy.Request objects. See source code here https://docs.scrapy.org/en/latest/_modules/scrapy/contracts.html

In order to workaround this and still use contracts to test ou spiders we need to create a new contract

from scrapy_selenium import SeleniumRequest
class WithSelenium(Contract):
    """ Contract to set the request class to be SeleniumRequest for the current call back method to test

    @with_selenium
    """
    name = 'with_selenium'
    request_cls = SeleniumRequest

Then use the contract @with_selenium in our testing.
Hope it helps

+1 Still the same, any updates!

for some reason when I copied the code for SeleniumRequest, SeleniumMiddelware to my project it all worked just fine!

Had the same issue before, for me, it's solved after install the driver (chromedriver or geckodriver, because the missing of driver made the selenium middleware disabled)