matthewmueller / x-ray

The next web scraper. See through the <html> noise.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Want to capture the "div" which is created after scrolling the page

N-pharmarack opened this issue · comments

Subject of the issue

I want to capture the items from a site, but half of the items are loaded when I scroll the page(link of the page : https://www.91mobiles.com/mobile-price-list-in-india?page=1 ). A new div ( page_scroll_1) is created after I scroll the the page, and the items I want to capture are there in that div.

I have tried ".scrollTo" method from 'x-ray-phantom' but It didn't work

Your environment

  • version of node: 16.13.0
  • version of npm: 8.1.0

Steps to reproduce

const Xray      = require('x-ray');
const puppeteer = require('puppeteer');
var phantom     = require('x-ray-phantom');

var phantom_opts = {
    webSecurity: false,
    images: false,
    weak: false
};

var x = Xray().driver(phantom(phantom_opts, function (nightmare, done) {
    done
        .useragent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36")
        .goto(nightmare.request.req.url)
        .scrollTo(0, 500)
        .wait(3000)
        .scrollTo(0, 1000)
        .wait(3000)
        .scrollTo(0, 1500)
        .wait(3000)
        .scrollTo(0, 2500)
        .wait(3000)
        .scrollTo(0, 3000)
        .wait(3000)
        .scrollTo(0, 3500)
        .wait(3000)
        .scrollTo(0, 4000)
        .wait(3000)
}));


x('https://www.91mobiles.com/mobile-price-list-in-india?page=1', '.finder_snipet_wrap',[{
  mobile_model: '.name',
  Price: '.price',
  Url: '.name@href'
}])
.write('results.json')

Expected behaviour

It should capture all the item with '.finder_snipet_wrap'

Actual behaviour

Capture's only half items, and skips all the other '.finder_snipet_wrap' that is being created after page scrolling

Did you figure this out? I have a very similar issue.