Make pyarrow a required dependency

[phofl]

Right now we run into a bunch of problems because pyarrow is potentially missing.

• It is pretty awkward to check for arrow inputs, e.g if we get an arrow array as input we have no good way of figuring this out (see https://github.com/pandas-dev/pandas/pull/52076/files)
• We can’t properly identify arrow scalars in all kinds of ops (e.g. indexing ops BUG: Setting a pyarrow scalar value with the same type that expands the series changes the dtype into object #52235, BUG: Pyarrow scalars inferred as object dtype with arrow backend enabled whwn assigning a new column #52056)
• Proper constructor support is not possible right now without causing significant slowdowns

This is the next step in supporting arrow fully in pandas. There are some things we can’t implement properly without requiring arrow as dependency.
So I think we should do it soonish, so that we can move forward.

cc @pandas-dev/pandas-core

[MarcoGorelli]

I'd be in favour

I've heard people raise objections because pyarrow is too heavy of a dependency - personally, I'd require it anyway, I don't think we should hold the project back because of this, just bringing it up for completeness

[jreback]

+1 from me - it's time

[datapythonista]

+1

I think main objections were not to do it yet as it was just fine to have it as a an optional dependency (see #50285). But if having it optional is an impediment.

Maybe PyArrow dependencies aren't ideal, but this should only be a temporary problem if PyArrow packages are expected to become more modular.

[lithomas1]

Are we planning on using Arrow Cython integration/C API stuff?

If not, requiring should be trivial on packaging side should be trivial (so no objections from me there).

Since from the few comments so far, most people seem to be in favor,
we should really start figuring out the specifics of this, e.g.

1. What will be the min. version of pyarrow that we will support?
   a) Would it be OK to have certain parts of pandas depend on a higher min. version of pyarrow?
2. How many version of pyarrow will we commit to supporting(e.g. for numpy we support from 1.20-1.24, which is ~5 versions)?

[jorisvandenbossche]

As long as it is not actually required (at the moment, it's strictly speaking an optional dependency, as you can perfectly fine use pandas without needing pyarrow, if you don't use any IO method relying on it and if you don't use the arrow-based dtypes), I don't see a good reason to make it an actual hard dependency.

The mentioned reasons that make it "awkward" to have it as an optional dependency don't sound that convincing to me, personally. I think it is quite easy to have helper methods that can check for pyarrow arrays and scalars (I commented on the mentioned PR: https://github.com/pandas-dev/pandas/pull/52076/files#r1160653924). That should already cover the first two points in the top post.

Proper constructor support is not possible right now without causing significant slowdowns

Is there some discussion about somewhere? I don't directly see why this would give a big slowdown.

---

When choosing to make it a hard requirement now, we have to be aware what the consequences are installation wise. For example, you can currently install pandas from source on most platforms as long as you have a compiler available (apart from that, we only have python dependencies / dependencies that can be installed by python in the same way as pandas). With pyarrow, that would no longer be true, as it cannot be installed from source. A typical example are docker images based on alpine linux (musl), for which there are no wheels at the moment.

[jbrockmendel]

+1

[simonjayhawkins]

Since from the few comments so far, most people seem to be in favor,

PDEP-1 first revision (scope) #51417 looks like it has moreless been accepted. The new text adds "New required dependency." as "Clear examples for potential PDEPs". However, IIUC this does not explicitly say a PDEP is required.

There is the following... "On the other hand, if an issue becomes controversial, i.e. it generated a significant
discussion, one could suggest opening a PDEP to formalize and document the discussion, making it
easier for the wider community to participate."

While this issue has not (here) "generated a significant discussion" a PDEP would be useful to "making it easier for the wider community to participate"

[MarcoGorelli]

agree - that, and the time bound on the voting period ensures we'll actually reach a decision rather than leaving the issue hanging around indefinitely. I think it'd be a good candidate

[jbrockmendel]

Making a PDEP shifts decision-making away from people who actually work on development and towards people willing to throw up walls of text.

We have five +1s, one -1, and one "no objections". Lets do this.

[simonjayhawkins]

-1. I don't agree that we should dismiss @jorisvandenbossche concerns.