refactor(series)!: 🕰️ drop TimestampSeries

[cmp0xff]

• Closes Series + Timedelta = no overload variant of ... #1355
• Closes str + Series -> Never #1359
• Addresses CLEAN: Investigate whether TimestampSeries, TimedeltaSeries, etc. can be removed #718
• Tests added: Please use assert_type() to assert the type of any return value

[Dr-Irv]

@cmp0xff you have a number of PRs submitted while I was out on vacation for 2 weeks. Can you let me know which ones I should prioritize for review?

[cmp0xff]

Hi @Dr-Irv, I hope you had a nice vacation. My pull requests are categorised below. Each category is independent, but those in a higher position have a slightly higher priority in my opinion.

Series: arithmetic operations

The following two PRs are independent. They migrate test_series.py to a subfolder series, and add quite a few test_*.py files there.

• add: feat(series): #1098 arithmetic addition #1275
• truediv: feat(series): #1098 arithmetic truediv #1280

DataFrame.to_dict

• fix(DataFrame): #799 to_dict #1283

Index.append

• feat(index): append #1282

Series: address #718

1. refactor(series)!: 🕰️ drop TimestampSeries #1274 - this is a prerequisite for the next one.
2. refactor: #718 also drop TimedeltaSeries #1273

[Dr-Irv]

Thanks for doing this. It's a lot of good work.

Main thing - if I'm going to merge this PR, it needs to be in a state where we don't need the followup PR.

Basic rule - we don't put ignore in the tests unless we are testing that the stubs should not accept something that is invalid. You have places where you have added ignore in the tests and I won't merge that in (unless we know it is a bug in the type checker)

[Dr-Irv]

Hi @Dr-Irv, I hope you had a nice vacation. My pull requests are categorised below. Each category is independent, but those in a higher position have a slightly higher priority in my opinion.

I've reviewed them all, except #1273 as noted there.

Thanks for all the great work.

[cmp0xff]

I've reviewed them all, except #1273 as noted there.

Thanks for all the great work.

Thank you very much for your quick and thorough reviews. I will be able to work on them next week.

[Dr-Irv]

Progressive approach

This is developed from my "consistent plan" in #1343 (comment). In contrast to the casting approach, we do

I'm fine with this approach. Let me know when I should review.

[cmp0xff]

Hi @Dr-Irv , I hope I have addressed all discussions around the stub files.

philosophy.md remains to be updated later in this PR.

[Dr-Irv]

Hi @Dr-Irv , I hope I have addressed all discussions around the stub files.

philosophy.md remains to be updated later in this PR.

Let me know when you have everything updated (including philosophy.md) and then I'll do a more complete review.

[cmp0xff]

Hi @Dr-Irv , I also proposed a new text for philosophy in ff37521. Now the PR is complete from my point of view.

[Dr-Irv]

Some small things and 2 larger things:

1. There are some tests that you removed that should work at runtime - inconsistencies in pandas interpretation that you can subtract a single datetime, but not a list of datetimes, so I'd like you to create an issue in pandas about any of those.
2. There are changes here that are using Series[Timedelta], but I'd like to have this PR only deal with Series[Timestamp] but still use TimedeltaSeries whenever we have Series[Timedelta]. Then the other PR can fix that.

[Dr-Irv]

I'm a little puzzled here. We can see that Series[Any] - list[datetime] is invalid, but why can't we see that Series[Any] - datetime is invalid?

Or maybe Series[Any] - list[datetime] should be allowed??

Same comment for the reverse operation and the .sub()

I think the issue here is that pandas is inconsistent, so can you report that in pandas?

[cmp0xff]

At run time,

• Series[Timestamp] - list[datetime] is not implemented. ENH: arithmetic between DatetimeArray and list pandas#62353
• Series[Timestamp] - datetime is valid.

  from datetime import datetime
  from typing import assert_type
  import pandas as pd
  
  arr = pd.to_datetime(["2020-01-01", "2020-01-02"]).array
  assert isinstance(arr, pd.arrays.DatetimeArray)
  
  print(arr - datetime(2020, 1, 1))

[Dr-Irv]

can you put back the tests that were there ? If we don't catch them in type checking, add a comment referring to the pandas issue.

[cmp0xff]

In the progressive plan, that is being implemented in this PR, Series[Any] - datetime gives TimedeltaSeries as the resulting type. When Series[Any] is actually Series[Timedelta] at run time, the subtraction will fail, of which type checkers are not aware.

Shall we add back left_td - s, which is Series[Any] - datetime for type checkers, but Series[Timedelta] - datetime at run time? Shall we do with pytrest.raise(...)?

[Dr-Irv]

Here is what I would prefer:
We have 3 possible cases:

1. An operation works at runtime, and we infer some type that makes sense in type checking
2. An operation fails at runtime, and we expect it to fail, and we are able to detect that with type checking
3. An operation fails at runtime, but we think it should work, and we do detect that with type checking. In this case, we should have the test, use if TYPE_CHECKING_INVALID_USAGE, but include a comment pointing to the pandas issue.

Examples of each:

1. Series[Timestamp] - datetime
2. Series[Timedelta] - datetime
3. Series[Timestamp] - list[datetime]

I think you had all of these kinds of tests currently in main, so when I look at this PR and see what has changed, each test should be there, but categorized appropriately.

[cmp0xff]

In the currently proposed philosophy, we have a fourth case:

• An operation fails at runtime, but static type checking cannot reveal the potential failure

I have tried to add old tests back in a0ae00d

[Dr-Irv]

I think the only thing that's now needed is some comments related to the things that are not working in pandas but we'd like to test. That means putting some tests back that you deleted with appropriate comments.

Ideally, if we believe it should work, but pandas says it doesn't, then we should have the type checker catch it until pandas fixes things (if ever). So I'd rather keep the tests that you deleted in tests/series/arithmetic/test_sub.py and add any appropriate comments that point to the pandas issues.

[Dr-Irv]

can you put back the tests that were there ? If we don't catch them in type checking, add a comment referring to the pandas issue.

[Dr-Irv]

The only remaining issues are in tests/arithmetic/test_sub.py. When looking at the comparison of the tests that are in main and the ones in this PR for that file, I should see that all the tests remain, and we just handle them appropriately as I suggest in another comment.

[Dr-Irv]

Here is what I would prefer:
We have 3 possible cases:

1. An operation works at runtime, and we infer some type that makes sense in type checking
2. An operation fails at runtime, and we expect it to fail, and we are able to detect that with type checking
3. An operation fails at runtime, but we think it should work, and we do detect that with type checking. In this case, we should have the test, use if TYPE_CHECKING_INVALID_USAGE, but include a comment pointing to the pandas issue.

Examples of each:

1. Series[Timestamp] - datetime
2. Series[Timedelta] - datetime
3. Series[Timestamp] - list[datetime]

I think you had all of these kinds of tests currently in main, so when I look at this PR and see what has changed, each test should be there, but categorized appropriately.

[Dr-Irv]

Thanks @cmp0xff ! It took us a while, but we got there. Great work.

[Dr-Irv]

@cmp0xff Do you think I should do a release with these changes, or wait for the removal of TimedeltaSeries ?

[cmp0xff]

@cmp0xff Do you think I should do a release with these changes, or wait for the removal of TimedeltaSeries ?

Hi @Dr-Irv , it depends on whether users make much use of TimedeltaSeries.

If users don't use it much (which I would guess so), releasing a new version is good (so that users know about our change in the philosophy earlier).

If users make much use of it, releasing now can be confusing.

[Dr-Irv]

@cmp0xff Do you think I should do a release with these changes, or wait for the removal of TimedeltaSeries ?

Hi @Dr-Irv , it depends on whether users make much use of TimedeltaSeries.

If users don't use it much (which I would guess so), releasing a new version is good (so that users know about our change in the philosophy earlier).

If users make much use of it, releasing now can be confusing.

I'm going to be offline from late Friday through Tuesday, 9/23, back online on 9/24, so I don't want to do a release and then have complaints where I think I should yank it.

So for now I will wait. and hope you make progress on the TimedeltaSeries one in the meantime.