Scrapy12 Scrapy : cookpad.com/uk Infinite scroll pagination with a twist if response.xpath("//a/@rel='next\'").get() == "1": print("GET Next Page") next_page = response.xpath('//a[@rel="next"]/@href').get() yield response.follow(url=next_page,callback=self.parse)