Remove Python 3.9 support?

[Fokko]

Feature Request / Improvement

Python 3.9 will be EoL next month: https://devguide.python.org/versions/

When should we drop support?

It looks like <5% still uses 3.9:

[kevinjqliu]

i wonder how much of that 5% is our own CI haha

[jayceslesar]

In favor, is it worth it to eventually swap to all the newer type annotations within the code?

Could drop from __future__ import annotations as Im pretty sure in most cases 3.10 and above all type annotations are treated as strings

[gregunz]

It would also make this PR (#2536) simpler (or rather the ray package version choice simpler)

[Fokko]

I wonder how much of that 5% is our own CI haha

That's a good point xD

Let me send out an email on the dev list to collect more feedback.

[Fokko]

Sent out an email: https://lists.apache.org/thread/z3dhxx2wc0dhvgq9js5ktbqf4ggsn4z8

[Fokko]

Anyone interested in drafting up a PR?

[DavidEscott]

@Fokko I would urge you to consider having a longer support window for a project like pyiceberg. There are firms where the process of updating and deploying software is made particularly onerous because of internal frictions and restrictions. Those individuals are both more likely to be using tools like pyiceberg (because they can't easily spin up a full apache spark instance) and to be running old versions of python (because they can't easily deploy new images).

[Aniketsy]

@Fokko I’d love to work on this issue.

[Fokko]

@DavidEscott What would be acceptable in your opinion? We just released 1.10.0, and we could create patch releases for it. But it would be nice to support 3.13 as well (which is asked for frequently by the community).

[DavidEscott]

@Fokko I wish the python community would allow down a little bit. Sorry that I dont have a better answer for you than that. I just don't understand what benefits newer version of python could give you that could be worth the cost to developers who have to chase the python minor versions.

[kevinjqliu]

Thanks for chiming in @DavidEscott! I've worked at firms where upgrading the internally supported python versions was difficult. I do think the python version EOL is a great signal (and motivator) to move off of those versions. After the EOL, even security patches would not be accepted in those python version.

Specifically for pyiceberg's support for python 3.9, the community can still use python 3.9 up to the latest pyiceberg release, 0.10. Future pyiceberg releases will not include support for python 3.9.

For additional context, we last removed python 3.8 around its EOL timeline #1121

[DavidEscott]

The python EOL timeline is far too aggressive in my opinion. RHEL 9 release was may 2022 with 3.9, so adopters of it are getting python a couple years into the the 3.9 lifestyle. Add a year or two of internal testing and validation and you end up with end users seeing python rolled out to them during the final year of its support.

None of this is the fault of pyiceberg, but I suspect the userbase for pyiceberg may be more impacted than other projects and so would urge you to keep that in mind.

[kevinjqliu]

sounds good, lets see if we can find use cases for python 3.9 from the community.
Fokko has already started a thread on the devlist. I'll post the same on the Iceberg slack channel