Using LastEvaluatedKey to page through an index
johnwilliams57-nhs opened this issue · comments
Description
Paging using LastEvaluatedKey and ExclusiveStartKey has stopped working properly in moto 5.0.4 and 5.0.5. It was working fine up till 5.0.3.
This seems similar to this closed issue: #7531
Code snippet
from collections.abc import Generator
from uuid import uuid4
import boto3
import pytest
from boto3.dynamodb.conditions import Key
from moto import mock_aws
from mypy_boto3_dynamodb.service_resource import Table
@pytest.fixture()
def temp_dynamodb_table() -> Generator[Table, None, None]:
with mock_aws():
dynamodb = boto3.resource("dynamodb", region_name="eu-west-2")
table_name = uuid4().hex
table = dynamodb.create_table(
BillingMode="PAY_PER_REQUEST",
TableName=table_name,
KeySchema=[
{"AttributeName": "pk", "KeyType": "HASH"},
],
AttributeDefinitions=[
{"AttributeName": "pk", "AttributeType": "S"},
{"AttributeName": "index_pk", "AttributeType": "S"},
],
GlobalSecondaryIndexes=[
{
"IndexName": "MyGSI",
"KeySchema": [
{"AttributeName": "index_pk", "KeyType": "HASH"},
],
"Projection": {"ProjectionType": "ALL"},
},
],
)
table.meta.client.get_waiter("table_exists").wait(TableName=table_name)
yield table
table.delete()
def test_paging(temp_dynamodb_table: Table):
three_items = [
{
"pk": "d71f4a29-7195-401e-a9cd-4bbdc966ae63",
"index_pk": "a",
},
{
"pk": "d71f4a29-7195-401e-a9cd-4bbdc966ae64",
"index_pk": "a",
},
{
"pk": "21a170d8-4003-485e-8215-8421fde3aae4",
"index_pk": "a",
},
]
for item in three_items:
temp_dynamodb_table.put_item(Item=item)
key_condition_expression = Key("index_pk").eq("a")
index_name = "MyGSI"
first_response = temp_dynamodb_table.query(
KeyConditionExpression=key_condition_expression, IndexName=index_name, Limit=2
)
assert len(first_response["Items"]) == 2
last_evaluated_key = first_response["LastEvaluatedKey"]
second_response = temp_dynamodb_table.query(
KeyConditionExpression=key_condition_expression,
IndexName=index_name,
Limit=2,
ExclusiveStartKey=last_evaluated_key,
)
assert len(second_response["Items"]) == 1 # this succeeds on moto 5.0.3. fails on 5.0.4 + 5.0.5 as len(items) == 0
Requirements.txt
pytest==8.1.1
mypy==1.9.0
boto3==1.34.11
boto3-stubs[athena,codebuild,codepipeline,dynamodb,events,firehose,kms,lambda,logs,s3,secretsmanager,sns,sqs,ssm,stepfunctions]==1.34.74
boto3-stubs[athena,dynamodb,events,firehose,kms,lambda,logs,s3,secretsmanager,sns,sqs,ssm,stepfunctions]==1.34.74
boto3-stubs[dynamodb,events,lambda,logs,s3,secretsmanager,sns,sqs,ssm]==1.34.74
botocore==1.34.11
botocore-stubs==1.34.69
moto[dynamodb]==5.0.5
Expected result
The assert in the last line of the test should succeed. i.e. the second page of results should have a single item in.
This was the result in 5.0.3
Actual result
The assert fails as the items returned is an empty list.
Hi @johnwilliams57-nhs, thanks for raising this and for providing the repro.
Moto 5.0.4/5.0.5 already fixed a few bugs around pagination, but this particularly use case is indeed still broken. I've opened a PR with a fix.