ENH: Create Better IntervalDtype using PyArrow structs.

[randolf-scholz]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

Currently, pandas.IntervalArray suffer from 3 major limitations:

1. They are limited to data with the same closedness on both sides. no longer the case apparently
2. All datapoints are limited to the same closedness in the array. (i.e. the same array can only store closed intervals or only open intervals).
3. Intervals do not allow missing values
   • In particular one cannot represent unbounded intervals for data types that lack an actual infinity value like int32.
4. Some dtypes are not allowed like string

As a practical application for (1) that I am very interested in is storing information about the range of valid values for the columns of another DataFrame.

Feature Description

Given the better integration with pyarrow since 2.0, we can recreate IntervalDtype using pyarrow.struct:

import pyarrow as pa

def arrow_interval_dtype(subtype):
    fields = [
        ("lower_bound", subtype),
        ("upper_bound", subtype),
        ("lower_inclusive", pa.bool_()),
        ("upper_inclusive", pa.bool_()),
    ]
    return pa.struct(fields)

Contrary to the current IntervalDtype, this would solve all 3 major problems at once:

1. Each element of the resulting StructArray can have separate closedness
2. Pyarrow datatypes all support missing values
3. We can in principle use any ordered data type for the subtype.

Alternative Solutions

None.

Additional Context

Additionally, common request is adding extra operations for interval dtypes:

• ENH: Interval type should support intersection, union & overlaps & difference #21998
• ENH: Arithmetic operations on intervals #43629
• API: Implement interval-point joins #21901
• Features which Interval / IntervalIndex should probably have #19480

Additionally, one could imagine having a IntervalUnion type, that can represent finite unions of intervals, combining the interval type discussed here with pyarrow list-type. This type would naturally arise when performing unions of intervals, such as [0, 2]∪[3, 5]. The nice thing here is that the resulting space is mathematically closed under the standard set operations (union, intersection, complements, difference)

[jbrockmendel]

@jorisvandenbossche could the existing ArrowIntervalDtype be the solution here?

[Julian048]

@jbrockmendel is this something pandas wants to pursue?

[randolf-scholz]

@jbrockmendel This issue is the only match when googling "ArrowIntervalDtype".

[Julian048]

@randolf-scholz I was having trouble finding that aswell

arrow does have an interval type but only for time related which is not what this issue is on

[jbrockmendel]

i was referring to pandas.core.arrays.arrow.ArrowIntervalType

[randolf-scholz]

@jbrockmendel I took a look at the code, and one immediate limitation seems to be that again this restricts all intervals in the array to be of equal closedness, while the proposal here would allow storing intervals of different closedness in the same array.