s3_object put with metadata specifying Content-Type results in ContentType being None (and boto3 raising exception)
timdiggins opened this issue · comments
Summary
If I want to upload a file with a specific content-type, then I will set the metadata to have Content-Type:
- name: upload json
amazon.aws.s3_object:
bucket: "{{ discovery_bucket_name }}"
content: "{{ discovery_tempfile.stdout }}"
object: "discovery.json"
mode: put
metadata: 'ContentType=application/json'however this causes the following kind of error:
Traceback
Traceback (most recent call last):
File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 756, in upload_s3file
File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/retries.py", line 105, in deciding_wrapper
return retrying_wrapper(*args, **kwargs)
File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
return _retry_func(
File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
return func()
File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 636, in upload_fileobj
return future.result()
File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
return self._coordinator.result()
File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 266, in result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 139, in __call__
return self._execute_main(kwargs)
File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "/usr/local/lib/python3.9/site-packages/s3transfer/upload.py", line 764, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 936, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1010, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter ContentType, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
This seems to be because this following code:
https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/s3_object.py#L750
replaces extra["ContentType"] with the return value of get_content_type which, when present = truthy implicitly returns None:
https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/s3_object.py#L657-L665
This was introduced in #1193 -- I have added the following comment: https://github.com/ansible-collections/amazon.aws/pull/1193/files#r1406429232
Issue Type
Bug Report
Component Name
s3_object
Ansible Version
$ ansible --version
ansible [core 2.13.13]
config file = /canupload/provision/ansible.cfg
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.9.18 (main, Nov 21 2023, 19:14:17) [GCC 10.2.1 20210110]
jinja version = 3.1.2
libyaml = TrueCollection Versions
$ ansible-galaxy collection list
# /usr/local/lib/python3.9/site-packages/ansible_collections
Collection Version
----------------------------- -------
amazon.aws 3.5.0
ansible.netcommon 3.1.3
ansible.posix 1.4.0
ansible.utils 2.8.0
ansible.windows 1.12.0
arista.eos 5.0.1
awx.awx 21.10.0
azure.azcollection 1.14.0
check_point.mgmt 2.3.0
chocolatey.chocolatey 1.3.1
cisco.aci 2.3.0
cisco.asa 3.1.0
cisco.dnac 6.6.1
cisco.intersight 1.0.22
cisco.ios 3.3.2
cisco.iosxr 3.3.1
cisco.ise 2.5.9
cisco.meraki 2.13.0
cisco.mso 2.1.0
cisco.nso 1.0.3
cisco.nxos 3.2.0
cisco.ucs 1.8.0
cloud.common 2.1.2
cloudscale_ch.cloud 2.2.3
community.aws 3.6.0
community.azure 1.1.0
community.ciscosmb 1.0.5
community.crypto 2.9.0
community.digitalocean 1.22.0
community.dns 2.4.2
community.docker 2.7.3
community.fortios 1.0.0
community.general 5.8.3
community.google 1.0.0
community.grafana 1.5.3
community.hashi_vault 3.4.0
community.hrobot 1.6.0
community.libvirt 1.2.0
community.mongodb 1.4.2
community.mysql 3.5.1
community.network 4.0.2
community.okd 2.2.0
community.postgresql 2.3.1
community.proxysql 1.4.0
community.rabbitmq 1.2.3
community.routeros 2.5.0
community.sap 1.0.0
community.sap_libs 1.4.0
community.skydive 1.0.0
community.sops 1.5.0
community.vmware 2.10.2
community.windows 1.11.1
community.zabbix 1.9.0
containers.podman 1.10.1
cyberark.conjur 1.2.0
cyberark.pas 1.0.14
dellemc.enterprise_sonic 1.1.2
dellemc.openmanage 5.5.0
dellemc.os10 1.1.1
dellemc.os6 1.0.7
dellemc.os9 1.0.4
f5networks.f5_modules 1.21.0
fortinet.fortimanager 2.1.7
fortinet.fortios 2.2.1
frr.frr 2.0.0
gluster.gluster 1.0.2
google.cloud 1.0.2
hetzner.hcloud 1.9.0
hpe.nimble 1.1.4
ibm.qradar 2.1.0
ibm.spectrum_virtualize 1.10.0
infinidat.infinibox 1.3.12
infoblox.nios_modules 1.4.1
inspur.ispim 1.2.0
inspur.sm 2.3.0
junipernetworks.junos 3.1.0
kubernetes.core 2.3.2
lowlydba.sqlserver 1.2.0
mellanox.onyx 1.0.0
netapp.aws 21.7.0
netapp.azure 21.10.0
netapp.cloudmanager 21.21.0
netapp.elementsw 21.7.0
netapp.ontap 21.24.1
netapp.storagegrid 21.11.1
netapp.um_info 21.8.0
netapp_eseries.santricity 1.3.1
netbox.netbox 3.9.0
ngine_io.cloudstack 2.3.0
ngine_io.exoscale 1.0.0
ngine_io.vultr 1.1.2
openstack.cloud 1.10.0
openvswitch.openvswitch 2.1.0
ovirt.ovirt 2.4.1
purestorage.flasharray 1.15.0
purestorage.flashblade 1.10.0
purestorage.fusion 1.2.0
sensu.sensu_go 1.13.1
servicenow.servicenow 1.0.6
splunk.es 2.1.0
t_systems_mms.icinga_director 1.31.4
theforeman.foreman 3.7.0
vmware.vmware_rest 2.2.0
vultr.cloud 1.3.1
vyos.vyos 3.0.1
wti.remote 1.0.4
# /root/.ansible/collections/ansible_collections
Collection Version
---------------------- -------
amazon.aws 7.0.0
community.aws 7.0.0
community.digitalocean 1.24.0
vultr.cloud 1.10.0AWS SDK versions
$ pip3 show boto boto3 botocore
pip3 show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.29.6
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.32.6
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transferConfiguration
$ ansible-config dump --only-changed
OS / Environment
No response
Steps to Reproduce
- name: upload json
amazon.aws.s3_object:
bucket: "{{ discovery_bucket_name }}"
content: "{{ discovery_tempfile.stdout }}"
object: "discovery.json"
mode: put
metadata: 'ContentType=application/json'Expected Results
I expected the file to be uploaded with a content type of application/json
Actual Results
Error in Boto3 because ContentType is passed as None
Traceback (most recent call last):
File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 756, in upload_s3file
File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/retries.py", line 105, in deciding_wrapper
return retrying_wrapper(*args, **kwargs)
File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
return _retry_func(
File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
return func()
File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 636, in upload_fileobj
return future.result()
File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
return self._coordinator.result()
File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 266, in result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 139, in __call__
return self._execute_main(kwargs)
File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "/usr/local/lib/python3.9/site-packages/s3transfer/upload.py", line 764, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 936, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1010, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter ContentType, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
Code of Conduct
- I agree to follow the Ansible Code of Conduct