[Error] rate limit

[Ana Jimenez Santamaria]

I've used perceval.backends.core.github python library to fetch github data from a set of repos (26 repos). When executing the script, I got a rate limit error:

anajs@anajsana:~/Desktop/TFM$ python opendistro-git.py 
https://github.com/opendistro-for-elasticsearch/security.git
WARNING:perceval.backends.core.github:Rate limit not initialized: 401 Client Error: Unauthorized for url: https://api.github.com/rate_limit

It seems the rate limit starts from the very beginning, so I don't understand the reason. It might be because I'm not giving all the required token permissions? Permissions I gave were: admin:org, admin:public_key, read:packages, repo and user.

This is the script I'm using:

import requests
import time
import json
import logging
from datetime import datetime 

from perceval.backends.core.git import Git
from perceval.backends.core.github import GitHub

def extract_data(repo):

    data_repo = GitHub(owner="opendistro-for-elasticsearch", repository=repo, api_token="xxxxx")
    for item in data_repo.fetch(from_date=datetime.strptime('2020-01-01', '%Y-%m-%d')):
        print(item)
        return

def owner_repositories(name):
    query = "org:{}".format(name)
    page = 1
    repositories = []

    r = requests.get('https://api.github.com/search/repositories?q={}&page={}'.format(query, page))
    items = r.json()['items']

    while len(items) > 0:
        # github API has rate limit. This ask for a 3 sec inactivity
        time.sleep(3)
        for item in items:
            logging.info("Adding {} repository".format(item['clone_url']))
            repositories.append(item['clone_url'])
        page += 1
        r = requests.get('https://api.github.com/search/repositories?q={}&page={}'.format(query, page))
        items = r.json()['items']

    return repositories

def main():
    repos = owner_repositories('opendistro-for-elasticsearch')
    for repo in repos:
        #print(repo)
        extract_data(repo)

if __name__ == '__main__':
    main()

[Venu Vardhan Reddy Tekula]

Hi @anajsana
I think the access token should be parsed as a list of tokens. Ref: github.py#L82

So, the solution to your problem would be

def extract_data(repo):

-   data_repo = GitHub(owner="opendistro-for-elasticsearch", repository=repo, api_token="xxxxx")
+   data_repo = GitHub(owner="opendistro-for-elasticsearch", repository=repo, api_token=["token1","token2"])
    for item in data_repo.fetch(from_date=datetime.strptime('2020-01-01', '%Y-%m-%d')):
        print(item)
        return

I guess this should solve the issue.

EDIT: I just tried the solution on your script and it worked. 😄
Please let me know how it goes for you.

Also, WRT permissions,

It might be because I'm not giving all the required token permissions? Permissions I gave were: admin:org, admin:public_key, read:packages, repo and user.

I think repo is enough for perceval to retrieve the data.

[Miguel Ángel Fernández]

In addition to what @vchrombie said, I would recommend to use the sleep_for_rate option. This will pause the collection until the rate limit is reset.

data_repo = GitHub(owner="opendistro-for-elasticsearch", repository=repo, api_token=["<token>"], sleep_for_rate=True)

Note that if you activate this option, you may also need to set the sleep_time parameter (If I'm not wrong, GitHub API limit is reset every hour).

[Ana Jimenez Santamaria]

yes! It worked 👍 Thank you so much for your help and quick response @vchrombie and @mafesan