Add new version compatible with airflow 2.5.1

[Alborosso]

I'm trying to find a compatible version for my MWAA instance (Airflow v2.5.1). This is the list of base constraints for airflow v2.5.1: https://raw.githubusercontent.com/apache/airflow/constraints-2.5.1/constraints-3.10.txt.

If I test requirements for mwaa instance with with https://github.com/aws/aws-mwaa-local-runner/tree/v2.5.1 I incur in the following error:

The conflict is caused by:
    aiobotocore 2.8.0 depends on botocore<1.33.2 and >=1.32.4
    aiobotocore 2.7.0 depends on botocore<1.31.65 and >=1.31.16
    aiobotocore 2.6.0 depends on botocore<1.31.18 and >=1.31.17
    aiobotocore 2.5.4 depends on botocore<1.31.18 and >=1.31.17
    aiobotocore 2.5.3 depends on botocore<1.31.18 and >=1.31.17
    aiobotocore 2.5.2 depends on botocore<1.29.162 and >=1.29.161
    aiobotocore 2.5.1 depends on botocore<1.29.162 and >=1.29.161
    aiobotocore 2.5.0 depends on botocore<1.29.77 and >=1.29.76
    aiobotocore 2.4.2 depends on botocore<1.27.60 and >=1.27.59
    aiobotocore 2.4.1 depends on botocore<1.27.60 and >=1.27.59
    aiobotocore 2.4.0 depends on botocore<1.27.60 and >=1.27.59
    aiobotocore 2.3.4 depends on botocore<1.24.22 and >=1.24.21
    aiobotocore 2.3.3 depends on botocore<1.24.22 and >=1.24.21
    aiobotocore 2.3.2 depends on botocore<1.24.22 and >=1.24.21
    aiobotocore 2.3.1 depends on botocore<1.24.22 and >=1.24.21
    aiobotocore 2.3.0 depends on botocore<1.24.22 and >=1.24.21
    aiobotocore 2.2.0 depends on botocore<1.24.22 and >=1.24.21
    aiobotocore 2.1.2 depends on botocore<1.23.25 and >=1.23.24
    aiobotocore 2.1.1 depends on botocore<1.23.25 and >=1.23.24
    aiobotocore 2.1.0 depends on botocore<1.23.25 and >=1.23.24
    aiobotocore 2.0.1 depends on botocore<1.22.9 and >=1.22.8
    aiobotocore 2.0.0 depends on botocore<1.22.9 and >=1.22.8
    aiobotocore 1.4.2 depends on botocore<1.20.107 and >=1.20.106
    aiobotocore 1.4.1 depends on botocore<1.20.107 and >=1.20.106
    aiobotocore 1.4.0 depends on botocore<1.20.107 and >=1.20.106
    aiobotocore 1.3.3 depends on botocore<1.20.107 and >=1.20.106
    aiobotocore 1.3.2 depends on botocore<1.20.107 and >=1.20.106
    aiobotocore 1.3.1 depends on botocore<1.20.50 and >=1.20.49
    aiobotocore 1.3.0 depends on botocore<1.20.50 and >=1.20.49
    aiobotocore 1.2.2 depends on botocore<1.19.53 and >=1.19.52
    aiobotocore 1.2.1 depends on botocore<1.19.53 and >=1.19.52
    aiobotocore 1.2.0 depends on botocore<1.19.53 and >=1.19.52
    aiobotocore 1.1.2 depends on botocore<1.17.45 and >=1.17.44
    aiobotocore 1.1.1 depends on botocore<1.17.45 and >=1.17.44
    aiobotocore 1.1.0 depends on botocore<1.17.45 and >=1.17.44
    aiobotocore 1.0.7 depends on botocore<1.15.33 and >=1.15.32
    aiobotocore 1.0.6 depends on botocore<1.15.33 and >=1.15.32
    aiobotocore 1.0.5 depends on botocore<1.15.33 and >=1.15.32
    aiobotocore 1.0.4 depends on botocore<1.15.33 and >=1.15.32
    aiobotocore 1.0.3 depends on botocore<1.15.33 and >=1.15.32
    aiobotocore 1.0.2 depends on botocore<1.15.33 and >=1.15.32
    aiobotocore 1.0.1 depends on botocore<1.15.33 and >=1.15.32
    aiobotocore 1.0.0 depends on botocore<1.15.33 and >=1.15.32
    aiobotocore 0.12.0 depends on botocore<1.15.16 and >=1.15.3
    aiobotocore 0.11.1 depends on botocore<1.13.15 and >=1.13.14
    aiobotocore 0.11.0 depends on botocore<1.13.15 and >=1.13.14
    aiobotocore 0.10.4 depends on botocore<1.12.253 and >=1.12.252
    aiobotocore 0.10.3 depends on botocore<1.12.190 and >=1.12.189
    aiobotocore 0.10.2 depends on botocore<1.12.92 and >=1.12.91
    aiobotocore 0.10.1 depends on botocore<1.12.92 and >=1.12.91
    aiobotocore 0.10.0 depends on botocore<1.12.50 and >=1.12.49
    aiobotocore 0.9.4 depends on botocore<1.10.59 and >=1.10.58
    aiobotocore 0.9.3 depends on botocore<1.10.59 and >=1.10.58
    aiobotocore 0.9.2 depends on botocore<1.10.13 and >=1.10.12
    aiobotocore 0.9.1 depends on botocore<1.10.13 and >=1.10.12
    aiobotocore 0.9.0 depends on botocore<1.10.13 and >=1.10.12
    aiobotocore 0.8.0 depends on botocore<1.10.13 and >=1.10.12
    aiobotocore 0.7.0 depends on aiohttp<3.2.0 and >=3.1.0
    aiobotocore 0.6.0 depends on botocore<=1.8.21 and >=1.8.0
    aiobotocore 0.5.3 depends on aiohttp<3.0.0 and >=2.3.0
    aiobotocore 0.5.2 depends on aiohttp<3.0.0 and >=2.3.0
    aiobotocore 0.5.1 depends on aiohttp<3.0.0 and >=2.3.0
    aiobotocore 0.5.0 depends on aiohttp<3.0.0 and >=2.3.0
    aiobotocore 0.4.5 depends on aiohttp<=2.3.0 and >=2.0.4
    aiobotocore 0.4.4 depends on aiohttp<=2.3.0 and >=2.0.4
    aiobotocore 0.4.3 depends on aiohttp<=2.3.0 and >=2.0.4
    aiobotocore 0.4.2 depends on aiohttp<=2.3.0 and >=2.0.4
    aiobotocore 0.4.1 depends on aiohttp<=2.2.0 and >=2.0.4
    aiobotocore 0.4.0 depends on aiohttp<=2.2.0 and >=2.0.4
    aiobotocore 0.3.3 depends on botocore<=1.5.52 and >=1.5.34
    aiobotocore 0.3.2 depends on botocore<=1.5.52 and >=1.5.34
    aiobotocore 0.3.1 depends on botocore<=1.5.38 and >=1.5.0
    aiobotocore 0.3.0 depends on botocore<=1.5.33 and >=1.5.0
    aiobotocore 0.2.3 depends on aiohttp<2.0.0 and >=1.2.0
    aiobotocore 0.2.2 depends on botocore==1.5.0
    aiobotocore 0.2.1 depends on botocore==1.5.0
    aiobotocore 0.2.0 depends on botocore==1.5.0
    aiobotocore 0.1.1 depends on botocore==1.5.0
    aiobotocore 0.0.6 depends on botocore<=1.4.73 and >=1.4.29
    aiobotocore 0.0.5 depends on botocore<=1.4.28 and >=1.4.0
    The user requested (constraint) aiohttp==3.8.3
    The user requested (constraint) botocore==1.29.51

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Is it possible for you to add a compatible version for the following constraints?

• aiohttp==3.8.3
• botocore==1.29.51

[Jakob Keller]

I'm afraid that will not be possible due to the tight coupling of aiobotocore to a specific snapshot of the botocore codebase.

[Alexander Mohr]

we need to add a giant FAQ on the home page probably and warning on the issue template about this. looks like no one reads the docs ;)

[Alexander Mohr]

going to close for now, we can provide releases to support newer versions of botocore, but rarely, older versions. We just released a new version of aiobotocore, can you try using a newer version of airflow? If there is really no other way to resolve this we can look into a custom release.

[Alexander Mohr]

actually going to leave this open for a bit more

[Jarek Potiuk]

Just to add - once aiobotocore relaxed it's limits (aiobotocore 2.7.0) this became far less of a problem and indeed newer versions of Airflow use aiobotocore>2.7.0 and it allows for wider range of compatibility.

We are still held back on boto3 1.34 as aiobotocore only works with boto3 1.33.* but I guess it's coming soon.

[Jakob Keller]

Just to add - once aiobotocore relaxed it's limits (aiobotocore 2.7.0) this became far less of a problem and indeed newer versions of Airflow use aiobotocore>2.7.0 and it allows for wider range of compatibility.

We are still held back on boto3 1.34 as aiobotocore only works with boto3 1.33.* but I guess it's coming soon.

#1069 provides compatibility for botocore / boto3 <= 1.34.14

[Jarek Potiuk]

#1069 provides compatibility for botocore / boto3 <= 1.34.14

Coool :)