Ghost Selector
A scrapy selector that supports javascript querying.
How it works
Ghost Selector understands Selenium's PhantomJS webdriver and takes advantage of the API it offers to behave just like the built-in Scrapy selectors.
Under the hood, selenium uses GhostDriver to talk to PhantomJS via selenium's WebDriver Wire Protocol. All Ghost Selector does is wrap whatever selenium returns after running javascript.
Example
>>> from selenium.webdriver import phantomjs
>>> from ghostsel import WebdriverXPathSelector
>>> driver = phantomjs.webdriver.WebDriver()
>>> driver.get("http://en.wikipedia.org/wiki/Main_Page")
>>> selector = WebdriverXPathSelector(driver)
>>> list_items = selector.select_script("return $('#p-navigation li')")
>>> list_items[0].extract()
u'Main page'
>>>
See test for usage.
Credits
Thanks goes to my good friend Nicolas for sponsoring its development.