Skip changes limit if I want to start extraction from hits over 50k
marton-balazs-kovacs opened this issue · comments
When I use the iterative query to look up results starting over 50k which is the API limit, the iterative query function rewrites the hard-coded limit value. Thus, it is not possible to start the iteration over the API limit. I suspect the following code chunk is responsible for the behavior:
MAXLIMIT = maxlimit or 50000 flag_last_round = False if skip + limit >= MAXLIMIT: flag_last_round = True if skip + limit > MAXLIMIT: limit = MAXLIMIT - skip
The Analytics API has a built-in upper limit of 50k records for all queries.
So in dimcli
auto-iteration utility this is enforced, to avoid triggering an API error.
You can test it out via a simple query, by setting the skip
parameter at 50k:
![image](https://private-user-images.githubusercontent.com/3493663/256198795-a42e09de-ca8a-4018-b1fb-f13f14d7b04b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEzNTExNjcsIm5iZiI6MTcyMTM1MDg2NywicGF0aCI6Ii8zNDkzNjYzLzI1NjE5ODc5NS1hNDJlMDlkZS1jYThhLTQwMTgtYjFmYi1mMTNmMTRkN2IwNGIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcxOSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MTlUMDEwMTA3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MzFhNmI1MGQ0MWQ4ODc2MDY4MTc1NjA0NmI0MTlhZGQyNWQ5YzcwNTczZjczYThiMTRlNWIxNzVhNjZlODZmYyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.gfAjdlGdzVe3LNjsH2C9UN-FvynkgWzgNwhCAy6DjI0)
PS Let me know if this does not answer your question..