Allow setting and reading continuation token for GetItemsFeed in delivery client

[Wim Vergouwe]

Motivation

We're exposing a data feed to a 3rd party that will generate sitemap.xml for us.
They require a feed of all pages. We don't want them to read to Kontent APIs directly, because the url building responsibility lies with us.
We can easily provide the functional part through the GetItemsFeed method on IDeliveryClient.
But pagination/continuation is not possible because we cannot set the continuation token.

Proposed solution

Add an optional parameter to the GetItemsFeed method:

public IDeliveryItemsFeed<T> GetItemsFeed<T>(IEnumerable<IQueryParameter> parameters = null, string continuationToken)

And expose the continuation token on IDeliveryItemsFeed

public interface IDeliveryItemsFeed<T>
{
    string ContinuationToken { get; }

Additional context

I'm open to creating a PR for this.

[Ondřej Chrastina]

Hello @WimVergouwe,

thanks for reaching out.

I am not sure if I understand the use-case completely, so correct me if I am wrong.

You need to fetch all items on your server, where you have the client initialized with Kontent API keys, and then you want to provide this data some service to build a sitemap for you right?

I guess you can use the approach described on the wiki: https://github.com/Kentico/kontent-delivery-sdk-net/wiki/Enumerating-all-items

Let me know if I am missing something.

[Wim Vergouwe]

Hi @Simply007,

Thx for the quick reply.
That's almost exactly what I want to do, BUT I want to provide a REST service that does this iteration and returns each batch to the 3rd party. For this to work efficiently and scalable, I need to round-trip the continuation token to my client.
So I need to be able to fetch only a single batch from a specific continuation token.

I hope this makes sense to you.

[Ondřej Chrastina]

Ahh OK, understood.

In that case, I will sync with the maintainers tomorrow if they see any problem in this change. I don't see any now, but I would rather be sure before welcoming the pull request.

I would vote to extend the IDeliveryItemsFeed<T> interface with the ContinuationToken property with a public getter and then in the implementation I would use public getter and private setter - I currently don't see any use cases for the setter to be enforced public.

Also, I would be super grateful if you share the short summary (ideally with code samples) of your use case so that we can place it in wiki/readme.

Me/somebody from team will loop back here once we walk through the potential effects of a change.

[Wim Vergouwe]

Ok, I'll wait for your feedback after talking with the maintainers.

[Wim Vergouwe]

Any update on this?

[Ondřej Chrastina]

Hello @WimVergouwe, I was waiting for the code sample for your use case.

Also, I would be super grateful if you share the short summary (ideally with code samples) of your use case so that we can place it in wiki/readme.

Because I think it is still possible to access the continuation token without any change.

I have checked GetStronglyTypedItemsFeed_MultipleBatches_FetchNextBatchAsync test and it gets the feed, then fetch the first batch and continuation token is propagated into IDeliveryItemsFeedResponse.ApiResponse.ContinuationToken properly.

Because calling GetItemsFeed and getting IDeliveryItemsFeed does not send the first request. So no continuation token is being propagated, it just prepare the feed and the first batch is being fetched when calling IDeliveryItemsFeed.FetchNextBatchAsync (returning IDeliveryItemsFeedResponse with Continuation token).

[Wim Vergouwe]

Hi @Simply007 ,

I have reviewed the test, but I still don't see a way of setting the continuation token.

I wasn't aware you were waiting for me to provide the summary and code sample. Here you go:

A 3rd party will handle sitemap xml generation for us. We need to provide them an api/feed that returns all our pages since a specific timestamp. Because this might be a large collection we need to be able to split the result in batches. The IDeliveryClient.GetItemsFeed seems ideal for this. But it doesn't allow us to get a single batch at a specific continuation token.

This is our current implementation:

        [HttpGet]
        public async Task<IActionResult> Get(
            DateTime? since = null,
            [FromHeader(Name = "X-Continuation")] string continuationToken = null)
        {
            var filters = new List<IQueryParameter>
            {
                new EqualsFilter("system.collection", _siteContext.GetBrand()) 
                // Todo: type filtering
            };

            if (since != null)
            {
                // For date filtering see https://kontent.ai/learn/reference/delivery-api#tag/Filtering-content/comparing-values
                filters.Add(
                    new RangeFilter(
                        "system.last_modified",
                        since.Value.ToString("O"),
                        DateTime.UtcNow.AddMonths(1).ToString("O")
                    )
                );
            }
            var feed = _deliveryClient.GetItemsFeed<object>(filters);

            if (continuationToken != null)
                SetContinuation(feed, continuationToken); // Hack: setting continuation token through reflection.

            while (feed.HasMoreResults)
            {
                var batch = await feed.FetchNextBatchAsync();
                if (batch == null) return NoContent();

                var pageDtos = new List<PageDto>();
                foreach (var pageModel in batch.Items.OfType<PageModel>().Where(x => x != null))
                {
                    pageDtos.Add(await ToPageDto(pageModel));
                }

                if (pageDtos.Count > 0)
                {
                    if (feed.HasMoreResults)
                    {
                        HttpContext.Response.Headers["X-Continuation"] = GetContinuation(feed);
                    }

                    return Ok(pageDtos);
                }
            }

            return NoContent();
        }

[Ondřej Chrastina]

Thanks, @WimVergouwe - now it is clear for me!

We have discussed this with the team and we like the idea of extending IDeliveryItemsFeed.FetchNextBatchAsync method with one optional string parameter, let's say continuationToken. Ideally, we shouldn't expose any information about the internal implementation of the batches pagination (that is why we are hesitant to expose continuation token as a property), but the optional parameter seems pretty small and fine by us. And ideally, we would like to keep the delivery client isolated from the implementation.

If we wanted to have it by the book, we should introduce a new interface next to the IDeliveryItemsFeed name i.e. IDeliveryItemsPagedBatch or something like that, and allow this one to work with continuation token with the defined interface, but it seems pretty complicated for such a small change.

So your code your would look like that with the optional parameter:

if (continuationToken != null)
{
    var feed = _deliveryClient.GetItemsFeed<object>(filters);

    var batch = await feed.FetchNextBatchAsync(continuationToken);
    
    var newContinuationToken = batch..ApiResponse.ContinuationToken;

    // place the continuation token up 
}

I have just one question about GetContinuation(feed) from the sample - why do you use feed to get a token since you can use the IDeliveryItemsFeedResponse.ApiResponse.ContinuationToken?

[Wim Vergouwe]

Alright, so you're ok that I make a pull request with an optional parameter called continuationToken, right?

About your question, why I don't reed continuation token from IDeliveryItemsFeedResponse.ApiResponse.ContinuationToken, that's just because I did not realize I could get it there ;-)

[Ondřej Chrastina]

Alright, so you're ok that I make a pull request with an optional parameter called continuationToken, right?

Yes, please! Ideally with the test validating the functionality (similar to GetStronglyTypedItemsFeed_MultipleBatches_FetchNextBatchAsync) but with providing the continuation token and verifying the HasMoreResults is set/cleared correctly and the data are returned correctly.

Feel free to ask if you are stuck.

Once the PR is merged I will update the wiki: https://github.com/Kentico/kontent-delivery-sdk-net/wiki/Enumerating-all-items to showcase the new functionality.

[Ondřej Chrastina]

Hello @WimVergouwe - we have released version 16 with continuation token set up feature - https://github.com/Kentico/kontent-delivery-sdk-net/releases/tag/16.0.0