Not getting any more posts, ending scrolling before limit

[dan-david]

I use Windows 10 Home and I have python 3.7 installed.

If I run on cmd py -3.7 crawl_profile.py danbilzerian I get the following:

DevTools listening on ws://127.0.0.1:51160/devtools/browser/5a390a57-f73b-4933-b2eb-614fd7b190c7
Extracting information from danbilzerian
INFO [2019-12-09 00:26:58] Extracting information from danbilzerian
INFO [2019-12-09 00:27:01] Alias name: Dan Bilzerian
INFO [2019-12-09 00:27:01] Bio: Content for 21+
Snapchat: dbilzerian
INFO [2019-12-09 00:27:01] Url: ignitecbd.co
INFO [2019-12-09 00:27:01] Posts: {'count': 1344}
INFO [2019-12-09 00:27:01] Follower: 29347018
INFO [2019-12-09 00:27:01] Following: {'count': 330}
INFO [2019-12-09 00:27:01] Is private: False
0
9
INFO [2019-12-09 00:27:11] breaking in 3...
If you believe this is only caused by slow internet, increase sleep time 'sleep_time_between_post_scroll' in settings.py
9
INFO [2019-12-09 00:27:16] breaking in 2...
If you believe this is only caused by slow internet, increase sleep time 'sleep_time_between_post_scroll' in settings.py
9
INFO [2019-12-09 00:27:21] breaking in 1...
If you believe this is only caused by slow internet, increase sleep time 'sleep_time_between_post_scroll' in settings.py
9
INFO [2019-12-09 00:27:26] breaking in 0...
If you believe this is only caused by slow internet, increase sleep time 'sleep_time_between_post_scroll' in settings.py
INFO [2019-12-09 00:27:26] Not getting any more posts, ending scrolling
INFO [2019-12-09 00:27:28] User danbilzerian has 0 comments.
Number of users who commented on their profile is  0

C:\Users\Daniel\instagram-profilecrawl\profiles\danbilzerian_commenters_2019-12-09 00-27-28.txt

Finished. The json file and nicknames of users who commented were saved in profiles direct

I am able to get the JSON file but only with the first posts from the profile. I tried that process with other profiles and I had the same problem. I was expecting information about much more posts since I set the settings.py file as below:

import os
from sys import platform as p_os


BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
OS_ENV = "windows" if p_os == "win32" else "osx" if p_os == "darwin" else "linux"

class Settings:
    profile_location = os.path.join(BASE_DIR, 'profiles')
    profile_commentors_location = os.path.join(BASE_DIR, 'profiles')
    profile_file_with_timestamp = True
    profile_commentors_file_with_timestamp = True
    limit_amount = 100
    scrape_posts_infos = True
    scrape_posts_likers = True
    scrape_follower = False
    output_comments = False
    sleep_time_between_post_scroll = 5
    sleep_time_between_comment_loading = 5
    mentions = True

    log_output_toconsole = True
    log_output_tofile = True
    log_file_per_run = False
    log_location = os.path.join(BASE_DIR, 'logs')

    #from Instpy
    # Set a logger cache outside object to avoid re-instantiation issues
    loggers = {}

    login_username = ''
    login_password = ''

    #chromedriver
    chromedriver_min_version = 2.36
    specific_chromedriver = f"chromedriver_{OS_ENV}"
    chromedriver_location = os.path.join(BASE_DIR, "assets", specific_chromedriver)

    if not os.path.exists(chromedriver_location):
        chromedriver_location = os.path.join(BASE_DIR, 'assets', 'chromedriver')

It seems like the scroll process on extractor.py is not working properly, but I am really not sure. How should I proceed?

Thank you for this awesome library!

[Shovan Paul]

Hi, I am encountering the same issue, but can't load any post. Did you manage to resolve it? If yes, how?

Edit: Figured it out -> Solution.