aiboto3 compatible version for using IRSA in eks

[sushama kothawale]

• Async AWS SDK for Python version: 9.2.0
• Python version: 3.7
• Operating System: EKS cluster nodes amd64 arch

Description

Our few services using aiboto3 9.2.0 version, recently we added support for IRSA (to use AWS IAM Roles for Service Accounts )with Amazon EKS . After adding this support services starts breaking with below errors:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 291, in _protected_refresh
    metadata = await self._refresh_using()
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 345, in fetch_credentials
    return await self._get_cached_credentials()
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 355, in _get_cached_credentials
    response = await self._get_credentials()
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/credentials.py", line 415, in _get_credentials
    return await client.assume_role_with_web_identity(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 155, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
ERROR: unexpected error - Not authorized to perform sts:AssumeRoleWithWebIdentity: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

sts:AssumeRoleWithWebIdentity these permissions are already attached to the role which is getting used in service, still getting above error.
It seems aiboto3 version is not compatible with IRSA.

Below is our requirements.txt file:

[aioboto3](https://pypi.org/project/aioboto3/)==9.2.0
[jsonschema](https://pypi.org/project/jsonschema/)==3.2.0
[kazoo](https://pypi.org/project/kazoo/)==2.6.0
[psycopg2](https://pypi.org/project/psycopg2/)==2.8.4
[SQLAlchemy](https://pypi.org/project/SQLAlchemy/)==1.3.12

Can someone please check this quickly? As our system is broken currently.

[Terri Cain]

There is nothing wrong with aioboto3 using IRSA

I just ran the following on my IRSA enabled kube cluster

import asyncio
import aioboto3


async def main():
    print(f"Version: {aioboto3.__version__}")

    session = aioboto3.Session()
    async with session.client("sts") as sts:
        resp = await sts.get_caller_identity()
        print(f"IAM: {resp['Arn'].split(':')[-1]}")


if __name__ == '__main__':
    asyncio.run(main())

and got:

root@test:/# python3 /tmp/a.py
Version: 12.0.0
IAM: assumed-role/homelab_pod_cert_manager/botocore-session-1699376436
root@test:/# 

I'd suggest you go check the assume role policy and that its correct for both the namespace and service account name.

[sushama kothawale]

Thanks @terrycain for quick response.

From the above output it looks like you are using 12.0.0 version. In our code we are using 9.2.0 aiboto3 package version wich is compatible with python 3.7.
so we need to know which aiboto3 version will be compatible with python 3.7 + IRSA?

[Terri Cain]

root@test:/# python3 /tmp/a.py
Version: 9.2.0
IAM: assumed-role/homelab_pod_cert_manager/botocore-session-1699377170

9.2.0 works fine. This is a problem on your end. And for reference you can look through the PyPI releases to see what aioboto3 versions work with 3.7, if i remember correction everything before version 12 does.