How to know when the end of a cursor-iteration was reached?

[Roman Vottner]

We have a test workspace where we run manual and automatic end-2-end tests for our internal API stuff. In this process we create, edit, archive a lot of channels and groups. So over time quite a number of channels (5k+) were created (most of them though being archived). While we store the Slack specific channel and/or group identifier on creation, we have the case that some of the channels existed before and thus a typical channel creation would return name_taken.

As a fallback to that case we try to look up the the channel via name to learn the Slack ID for that channel and then perform an update for it. We therefore use the conversations.list API method which allows us to i.e. specify the size of the number of channels we want to receive per request (needs to be 999 or less) and a cursor to iterate through further results. I'm aware that filters are only applied AFTERWARDS and therefore I can receive less than the specified limit number of channels, which makes checking for receiving less than that limit to exit the loop impossible, at least when archived channels are not included in the result set. I therefore thought that for the last "page" the cursor will be set to null and could therefore be used to exit the loop and avoid having to restart from the beginning again, which reflects in the code below:

    // excludeArchived == true in this case which therefore returns less than the defined limit when 
    // channels are archived
    ConversationsListRequest.ConversationsListRequestBuilder rb =
            ConversationsListRequest.builder();
    rb.token(env.getBotToken())
            .limit(limit)
            .excludeArchived(excludeArchived);

    String cursor = null;
    do {
      if (cursor != null) {
        rb.cursor(cursor);
      }

      final ConversationsListResponse response =
              env.getApp().client().conversationsList(rb.build());
      if (response.isOk()) {
        LOG.debug("Retrieved {} channels", response.getChannels().size());
        for (Conversation conversation : response.getChannels()) {
          // do some stuff with the channels
        }
      } else {
        LOG.warn("Failure while retrieving channels. Reason: {}", response.getError());
      }
      LOG.debug("Updating cursor to retrieve next page of channels");
      cursor = response.getResponseMetadata().getNextCursor();

     // attempt to prevent rate-limiting errors as the API basically says 20+ requests per minute
      try {
        Thread.sleep(3000L);
      } catch (InterruptedException iEx) {
        LOG.debug("Sleep was interrupted while waiting for the next poll interval of available "
                + "channels");
        break;
      }
    } while (cursor != null);

With the given code I noticed that the code will "page" through all channels but then start from the beginning all over again and eventually end up in a rate limiting exception even though a sleep of 3 seconds should correspond to 20 requests per minute, which should be within the 20+ requests limit tier 2 operations have.

I thought that at the end of the iteration the cursor would return null to state that there are no further channels to retrieve. Unfortunately, I haven't found anything related to that in the documentation.

So, how do I prevent iterating though the same results over and over again? While a channel lookup per name or at least a filter that is applied BEFORE so that the returned size matches the specified limit where less than that would indicate the end of the list would be great, I'm not sure how much of a change this would be on the Slack API side.

[Eden Zimbelman]

Hey @RovoMe 👋 Really appreciate the code snippet and question! Having a conversations.lookupName method would be super useful here and I've shared this with our teams, but for now we can still tackle this with pagination.

Instead of checking for a null cursor for the end of results when paginating, a check for the empty string will be useful for knowing when no more results are present. Could you check if this change prevents the repeated looping?

-    } while (cursor != null);
+    } while (!cursor.isEmpty());

[Roman Vottner]

Will check that on Monday. Our DEV environment is shut down during off-hours since we should not work during these hours :)

In the meantime I did my own "circular loop breaker" logic by storing the first name that is returned and then checking after each request whether I see that name again and then just break the loop. A bit of a duct tape and glue solution but it at least gets the job done for now.

---

edit:

@zimeg I tested the change now with a locally running bot using the test-workspace and it did stop after the end was reached :)

Thank you for the input and suggestion :) I still root for the conversations.lookupName option as this would spare us from iterating through several pages of channels. Our production workspace has way fewer channels, but still somewhere around 100+ channels I guess. Our test-workspace for sure is extreme but also a good test case whether our bot will scale with an "expanding" production workspace in future :)

---

2nd edit:

Oh, by the way, we also have like 3500+ groups created in the meantime and here the usergroups.list method doesn't have a pagination and therefore returns all the groups in one-go. It would by handy if there is a matching usergroups.lookupName API method as well (maybe also a pagination approach similar to channels?)

[Eden Zimbelman]

@RovoMe great to hear! Just wanted to check that usergroups.list is returning all the responses with one call and not that it's missing pagination? I'll make a note of this too if so, but am not sure if it'll be tackled with any changes around the conversations.lookupName...

Thanks for checking the change and hope you enjoy the weekend!