S3 bucket list: `max_keys: 0` is ignored

[Nick Bargnesi]

Summary

The max_keys parameter of s3_object_module is ignored if set to zero. This causes AWS API defaults to take effect, which for the max-keys URI request parameter is 1000.

AWS API documentation for S3 object listing: ListObjects, ListObjectsV2.

Note by ignoring max_keys: 0, Ansible will retrieve page after page of results, up to the last object in the bucket, possibly leading to considerable costs for the AWS account owner and consuming considerable CPU time and bandwidth.

Using one of the documented examples and setting max_keys: 0:

- name: List keys all options
  amazon.aws.s3_object:
    bucket: mybucket
    mode: list
    prefix: /my/desired/
    marker: /my/desired/0023.txt
    max_keys: 0

The AWS API request line will be:

GET https://mybucket.s3.amazonaws.com/?list-type=2&prefix=%2Fmy%2Fdesired%2F&start-after=%2Fmy%2Fdesired%2F0023.txt&encoding-type=url

Note the lack of the expected &max-keys=0 URI request parameter in the GET line.

This issue seems to be present in the following releases:

• 6.0.0
• 6.0.1
• 6.1.0
• 6.2.0
• 6.3.0
• 6.4.0
• 6.5.0
• 6.5.1
• 7.0.0
• 7.1.0
• 7.2.0

Issue Type

Bug Report

Component Name

modules.plugin.s3_object

Ansible Version

$ ansible --version
ansible [core 2.16.2]
  config file = None
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ~/.venvs/ansible-core/lib/python3.12/site-packages/ansible
  ansible collection location = ~/.ansible/collections:/usr/share/ansible/collections
  executable location = ~/.bin/ansible
  python version = 3.12.1 (main, Dec  7 2023, 20:45:44) [Clang 15.0.0 (clang-1500.0.40.1)]
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
Collection                   Version
---------------------------- -------
amazon.aws                   7.2.0

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.34.22
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.12/site-packages
Requires: botocore, s3transfer, jmespath
Required-by:
---
Name: botocore
Version: 1.34.22
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.12/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: s3transfer, boto3, awscli

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = None

OS / Environment

No response

Steps to Reproduce

- name: List keys all options
  amazon.aws.s3_object:
    bucket: mybucket
    mode: list
    prefix: /my/desired/
    marker: /my/desired/0023.txt
    max_keys: 0

Expected Results

I expected max-keys: 0 to be respected in the API request. It was ignored.

Actual Results

Ansible execution hung as the bucket I used never completed full pagination before the process was killed.

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[Nick Bargnesi]

Any other info needed for this issue? The bug has been around for over 6 months and drives up the cost of AWS bills.

[Nick Bargnesi]

Just in case it's not clear in the links above, there's already a PR for this #1954