Add Python 3.10 support
mik-laj opened this issue · comments
Description
According to README.md - "Support for Python and Kubernetes versions":
We support a new version of Python/Kubernetes in main after they are officially released, as soon as we make them work in our CI pipeline (which might not be immediate due to dependencies catching up with new versions of Python mostly) we release new images/support in Airflow based on the working CI setup.
Python 3.10 has been released on October 18, 2021, so we should take steps to test Airflow with this release.
Use case/motivation
Support for new versions of Python is expected by users who wish to test the new features that this version added. After that, some users install this release by mistake and then have trouble using Airflow e.g. https://stackoverflow.com/questions/69615585/unable-to-start-airflow-webserver-with-python-3-10
Related issues
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
The ModuleNotFoundError: No module named 'pwd' error the user gets is weird though. Airflow should have no problem running on 3.10 (albeit not officially supported), and the pwd stdlib did not receive any significant change. That problem is likely separated from 3.10 support.
I'll submit a fix to deal with the distutils deprecation warning.
did python 3.10.1 release solved anything?
Python 3.10 support is missing for other reasons - unrelated to version support. it was missing because some dependencies did not supported it. But this might be a good thing to try again maybe we can get it workin. I am planning to do some further cleanups with images after Xms and possibly attemp to create even an ARM image (and there python 3.10 might actually be better than older versions).
Just try to build Python 3.10 - we are rather close - I think all our deps are almost ready - the last one is numpy limited by `apache-beam'. This is also blocker for ARM images. However it's been alrady merged in 'beam' repo:
apache/beam@d845a00#diff-1275c48808de339ef6f282d844c83ec441b5cfa0debc373fdcb7dba497da4fc8
And the next release should build both for Python 3.10 and allow us to build ARM images.
FYI. Apache Bem release that should be our "last" serious blocker is planned for first week of February. I could make some exclusions and make it, independently of Beam, but since it is only weeks away, I think it is better to wait.
It looks like new Apache Beam is released (assuming that is https://beam.apache.org/blog/beam-2.36.0/). Was that the last dependency holdout for this?
Yep. One more small things with #22017 and it shoud be possible.
All right. Now the only thing I need to do is to release all providers with version 3.10 support :).
It's already updated https://github.com/apache/airflow/blob/main/README.md . Only main version supports Python 3.10. What other update would you need? If it is missing somewhere in other readmes you see - feel free to open PR :)