Using LastEvaluatedKey to page through an index

[John Williams]

Description

Paging using LastEvaluatedKey and ExclusiveStartKey has stopped working properly in moto 5.0.4 and 5.0.5. It was working fine up till 5.0.3.

This seems similar to this closed issue: #7531

Code snippet

from collections.abc import Generator
from uuid import uuid4

import boto3
import pytest
from boto3.dynamodb.conditions import Key
from moto import mock_aws
from mypy_boto3_dynamodb.service_resource import Table


@pytest.fixture()
def temp_dynamodb_table() -> Generator[Table, None, None]:
    with mock_aws():
        dynamodb = boto3.resource("dynamodb", region_name="eu-west-2")

        table_name = uuid4().hex
        table = dynamodb.create_table(
            BillingMode="PAY_PER_REQUEST",
            TableName=table_name,
            KeySchema=[
                {"AttributeName": "pk", "KeyType": "HASH"},
            ],
            AttributeDefinitions=[
                {"AttributeName": "pk", "AttributeType": "S"},
                {"AttributeName": "index_pk", "AttributeType": "S"},
            ],
            GlobalSecondaryIndexes=[
                {
                    "IndexName": "MyGSI",
                    "KeySchema": [
                        {"AttributeName": "index_pk", "KeyType": "HASH"},
                    ],
                    "Projection": {"ProjectionType": "ALL"},
                },
            ],
        )
        table.meta.client.get_waiter("table_exists").wait(TableName=table_name)

        yield table

        table.delete()


def test_paging(temp_dynamodb_table: Table):
    three_items = [
        {
            "pk": "d71f4a29-7195-401e-a9cd-4bbdc966ae63",
            "index_pk": "a",
        },
        {
            "pk": "d71f4a29-7195-401e-a9cd-4bbdc966ae64",
            "index_pk": "a",
        },
        {
            "pk": "21a170d8-4003-485e-8215-8421fde3aae4",
            "index_pk": "a",
        },
    ]

    for item in three_items:
        temp_dynamodb_table.put_item(Item=item)

    key_condition_expression = Key("index_pk").eq("a")
    index_name = "MyGSI"

    first_response = temp_dynamodb_table.query(
        KeyConditionExpression=key_condition_expression, IndexName=index_name, Limit=2
    )

    assert len(first_response["Items"]) == 2
    last_evaluated_key = first_response["LastEvaluatedKey"]

    second_response = temp_dynamodb_table.query(
        KeyConditionExpression=key_condition_expression,
        IndexName=index_name,
        Limit=2,
        ExclusiveStartKey=last_evaluated_key,
    )

    assert len(second_response["Items"]) == 1  # this succeeds on moto 5.0.3. fails on 5.0.4 + 5.0.5 as len(items) == 0

Requirements.txt

pytest==8.1.1
mypy==1.9.0
boto3==1.34.11
boto3-stubs[athena,codebuild,codepipeline,dynamodb,events,firehose,kms,lambda,logs,s3,secretsmanager,sns,sqs,ssm,stepfunctions]==1.34.74
boto3-stubs[athena,dynamodb,events,firehose,kms,lambda,logs,s3,secretsmanager,sns,sqs,ssm,stepfunctions]==1.34.74
boto3-stubs[dynamodb,events,lambda,logs,s3,secretsmanager,sns,sqs,ssm]==1.34.74
botocore==1.34.11
botocore-stubs==1.34.69
moto[dynamodb]==5.0.5

Expected result

The assert in the last line of the test should succeed. i.e. the second page of results should have a single item in.

This was the result in 5.0.3

Actual result

The assert fails as the items returned is an empty list.

[Bert Blommers]

Hi @johnwilliams57-nhs, thanks for raising this and for providing the repro.

Moto 5.0.4/5.0.5 already fixed a few bugs around pagination, but this particularly use case is indeed still broken. I've opened a PR with a fix.