Unpin pyarrow

[dhirschfeld]

The upper pin on pyarrow was removed in databricks-sql-python 3.7.0 which suggests it is also not required here:

• Remove upper caps on dependencies databricks-sql-python#452

Could the pin also be removed here as it makes it problematic to install alongside other libraries which also depend on pyarrow.

databricks-sqlalchemy/pyproject.toml

Line 14 in dd7b152

pyarrow = { version = ">=14.0.1,<17"}

[dhirschfeld]

...and it also creates security issues if the pin isn't being constantly updated to the latest:

• databricks-sqlalchemy depends on pyarrow versions with CVE-2024-52338 #13

[euanmacinnes]

it also makes it problematic to install in a docker image with alpine as the prebuilt arrow is only 18.0.0 onwards. Better to have arrow as an optional pip install databricks-sqlalchemy[arrow]

[akoshel]

pyarrow makes this incompatible with databricks-sql-connector for python 3.13
https://github.com/databricks/databricks-sql-python/blob/v4.0.1/pyproject.toml#L25