Azure Graph rate limit solution with powershell

[Nick]

@dirkjanm this a snip of what I use in powershell to get around graph rate limits on subscription and resources. A short one or two second pause should be worked into the loop so graph is not taxed.

`# Fetch the full array of subscription IDs
$subscriptions = Get-AzSubscription
$subscriptionIds = $subscriptions.Id

query

$query = "resources | order by subscriptionId asc";

Create a subscription counter, set the batch size, and prepare a variable for the results

$counter = [PSCustomObject] @{ Value = 0 }
$batchSize = 1000

Create array to hold results

$response = @()

Group the subscriptions into batches

$subscriptionsBatch = $subscriptionIds | Group -Property { [math]::Floor($counter.Value++ / $batchSize) }

Run the query for each batch

foreach ($batch in $subscriptionsBatch)
{

Create a resource counter, set the batch size, and prepare a variable for the results

$Skip = 0;
$First = 1000;

Get the data

$response += do {if ($Skip -eq 0) {$y = Search-AzGraph -Query $query -First $First -Subscription $batch.Group ; }
else {$y = Search-AzGraph -Query $query -Skip $Skip -First $First -Subscription $batch.Group } $cont = $y.Count -eq $First; $Skip = $Skip + $First; $y; } while ($cont) }

[Dirk-jan]

thx, latest commit on master uses a combination of token buckets and automatic throttling detection to work around this :)

[Nick]

That snip also gets past the hard limit on resources and subscriptions that can be retrieved via Graph.

[Dirk-jan]

do you have a reference on what those hard limits are? haven't run into them yet

[Nick]

So I work in one of the top 5 largest Azure tenants in the world with well over 5k subs and discovered this a couple years ago.

When a security principal has access to more than 5,000 subscriptions within the tenant or management group query scope, the response is limited to the first 5,000 subscriptions and the x-ms-tenant-subscription-limit-hit header is returned as true.

To get around the resource limit I use skip and first.

First currently has a maximum allowed value of 5000, which it achieves by paging results 1000 records at a time.
When First is configured to be greater than 1000 records, the query must project the id field in order for pagination to work. If it's missing from the query, the response won't get paged and the results are limited to 1000 records.

Here is a PowerShell Graph-Pagination-Sample I use to pull storage account endpoints.