BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill #27992

noritada · 2019-08-18T11:25:59Z

closes #19995

Timezone info is dropped in GroupBy.shift, bfill, and ffill
since index calculated by Cythonized functions are applied to
NumPy representation of values without timezone.

Could you please find and review this fix?

Many thanks,

-noritada

In [1]: import pandas as pd 
   ...: pd.__version__ 
Out[1]: '0.24.2'

In [2]: df = pd.DataFrame([ 
   ...:     ['a', pd.Timestamp('2019-01-01 00:00:00+09:00')], 
   ...:     ['b', pd.Timestamp('2019-01-01 00:05:00+09:00')], 
   ...:     ['a', None], 
   ...:     ['b', None], 
   ...:     ['a', pd.Timestamp('2019-01-01 00:20:00+09:00')], 
   ...:     ['b', pd.Timestamp('2019-01-01 00:25:00+09:00')], 
   ...: ], columns=['id', 'time']) 

In [3]: df['time'].shift(1) 
Out[3]: 
0                         NaT
1   2019-01-01 00:00:00+09:00
2   2019-01-01 00:05:00+09:00
3                         NaT
4                         NaT
5   2019-01-01 00:20:00+09:00
Name: time, dtype: datetime64[ns, pytz.FixedOffset(540)]

In [4]: df.groupby('id')['time'].shift(1) 
Out[4]: 
0                   NaT
1                   NaT
2   2018-12-31 15:00:00
3   2018-12-31 15:05:00
4                   NaT
5                   NaT
Name: time, dtype: datetime64[ns]

In [5]: df.groupby('id')['time'].bfill() 
Out[5]: 
0   2018-12-31 15:00:00
1   2018-12-31 15:05:00
2   2018-12-31 15:20:00
3   2018-12-31 15:25:00
4   2018-12-31 15:20:00
5   2018-12-31 15:25:00
Name: time, dtype: datetime64[ns]

In [6]: df.groupby('id')['time'].ffill() 
Out[6]: 
0   2018-12-31 15:00:00
1   2018-12-31 15:05:00
2   2018-12-31 15:00:00
3   2018-12-31 15:05:00
4   2018-12-31 15:20:00
5   2018-12-31 15:25:00
Name: time, dtype: datetime64[ns]

mroeschke · 2019-08-18T22:42:19Z

doc/source/whatsnew/v0.25.1.rst

@@ -120,6 +120,7 @@ Groupby/resample/rolling
 - Bug in :meth:`pandas.core.groupby.DataFrameGroupBy.transform` where applying a timezone conversion lambda function would drop timezone information (:issue:`27496`)
 - Bug in windowing over read-only arrays (:issue:`27766`)
 - Fixed segfault in `pandas.core.groupby.DataFrameGroupBy.quantile` when an invalid quantile was passed (:issue:`27470`)
+- Bug in :meth:`pandas.core.groupby.GroupBy.shift`, :meth:`pandas.core.groupby.GroupBy.bfill` and :meth:`pandas.core.groupby.GroupBy.ffill` where timezone information would be dropped (:issue:`xxxxx`)


If there is not an existing issue for this bug, use the number of this PR.

mroeschke · 2019-08-18T22:45:45Z

pandas/tests/groupby/test_timezone.py

+        ),
+    ],
+)
+def test_shift_bfill_ffill_tz(data, expected_shift, expected_bfill, expected_ffill):


Can you put this test in test_groupby.py

Also you can parameterize over shift, bfill, and ffill

…schke

…schke)

noritada · 2019-08-19T14:04:07Z

@mroeschke
Thank you very much for your reviews.
I've updated as per your suggestions.
Could you please check again?

Many thanks,

-noritada

mroeschke

Very nice. One small comment otherwise LGTM.

mroeschke · 2019-08-19T16:50:23Z

pandas/tests/groupby/test_groupby.py

@@ -1882,3 +1882,69 @@ def test_groupby_axis_1(group_name):
    results = df.groupby(group_name, axis=1).sum()
    expected = df.T.groupby(group_name).sum().T
    assert_frame_equal(results, expected)
+
+
+@pytest.mark.parametrize("tz", [None, "Asia/Tokyo"])


We have a tz_naive_fixture you can use in the function signature that will parameterize over more timezones.

def test_shift_bfill_ffill_tz(tz_naive_fixture, op, expected): tz = tz_naive_fixture ...

…mroeschke

…mroeschke

noritada · 2019-08-19T23:02:54Z

@mroeschke
Thank you so much again.
I should have found this function.
Test code updated again.

Many thanks,

-noritada

WillAyd · 2019-08-23T13:51:52Z

pandas/core/groupby/groupby.py

@@ -2220,26 +2220,28 @@ def _get_cythonized_result(
        base_func = getattr(libgroupby, how)

        for name, obj in self._iterate_slices():
+            values = obj._data._values


Hmm I'm not sure relying on underlying block values is the best way to go about this. Is it possible to just work with the obj here instead?

Sorry for my lack of understanding about underlying block values.
Should I leave the code using obj.values except for the last result = algorithms.take_nd(obj.values, result)?

WillAyd · 2019-08-23T13:52:09Z

Can you update whatsnew for 0.25.2 as well?

TomAugspurger · 2019-08-23T20:43:50Z

1.0.0, unless this is a regression in 0.25.x.

…

On Fri, Aug 23, 2019 at 8:52 AM William Ayd ***@***.***> wrote: ***@***.**** requested changes on this pull request. ------------------------------ In pandas/core/groupby/groupby.py <#27992 (comment)>: > @@ -2220,26 +2220,28 @@ def _get_cythonized_result( base_func = getattr(libgroupby, how) for name, obj in self._iterate_slices(): + values = obj._data._values Hmm I'm not sure relying on underlying block values is the best way to go about this. Is it possible to just work with the obj here instead? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#27992?email_source=notifications&email_token=AAKAOITRKMZJLG3H2KELYODQF7TQNA5CNFSM4IMSLLYKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCCQWFKQ#pullrequestreview-279012010>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIXQRK4RRN4LSTZ5RR3QF7TQNANCNFSM4IMSLLYA> .

jreback

looks good. can you do an issue search to see if we have anything about this (IIRC we do). if so, pls reference these issues.

jreback · 2019-09-08T16:20:57Z

doc/source/whatsnew/v0.25.1.rst

@@ -120,6 +120,7 @@ Groupby/resample/rolling
 - Bug in :meth:`pandas.core.groupby.DataFrameGroupBy.transform` where applying a timezone conversion lambda function would drop timezone information (:issue:`27496`)
 - Bug in windowing over read-only arrays (:issue:`27766`)
 - Fixed segfault in `pandas.core.groupby.DataFrameGroupBy.quantile` when an invalid quantile was passed (:issue:`27470`)
+- Bug in :meth:`pandas.core.groupby.GroupBy.shift`, :meth:`pandas.core.groupby.GroupBy.bfill` and :meth:`pandas.core.groupby.GroupBy.ffill` where timezone information would be dropped (:issue:`27992`)


can you move to 0.25.2

jreback · 2019-09-08T16:23:31Z

also pls merge master

…roupBy.shift

noritada · 2019-09-09T11:24:21Z

@WillAyd @TomAugspurger @jreback
Thanks for comments and sorry to be absent for 3 weeks.
I've merged master, move a whatsnew entry to v0.25.2 and document #19995 .

Could you please check this again?

Many thanks,

-noritada

jreback · 2019-09-09T12:06:08Z

thanks @noritada

lumberbot-app · 2019-09-09T12:06:33Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

$ git checkout 0.25.x
$ git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

$ git cherry-pick -m1 e0c63b4cfaa821dfe310f4a8a1f84929ced5f5bd

You will likely have some merge/cherry-pick conflict here, fix them and commit:

$ git commit -am 'Backport PR #27992: BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill'

Push to a named branch :

git push YOURFORK 0.25.x:auto-backport-of-pr-27992-on-0.25.x

Create a PR against branch 0.25.x, I would have named this PR:

"Backport PR #27992 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to suggest an improvement.

TomAugspurger · 2019-09-09T14:20:22Z

@jreback I think the plan was for 0.25.2 to just be Python 3.8 compat and 0.25 regressions: #28110. I don't believe this was a regression in 0.25.

jreback · 2019-09-09T15:24:05Z

ok sure np

alright then prob just need to move the actual release note

…andas-dev#27992)

BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill

8b38d7a

mroeschke reviewed Aug 18, 2019

View reviewed changes

mroeschke added Bug Groupby Timezones Timezone data dtype labels Aug 18, 2019

noritada added 3 commits August 19, 2019 10:42

Fill issue number placeholders with a PR number (pandas-dev#27992)

887c6e7

Relocate a test function as suggested by Matthew Roeschke (@mroeschke)

0812ccf

Parameterize methods as per a suggestion from Matthew Roeschke (@mroe…

43b42d5

…schke)

mroeschke requested changes Aug 19, 2019

View reviewed changes

Use a predefined fixture function for tz parameterization thanks to @…

b5a3735

…mroeschke

mroeschke approved these changes Aug 20, 2019

View reviewed changes

WillAyd requested changes Aug 23, 2019

View reviewed changes

jreback requested changes Sep 8, 2019

View reviewed changes

jreback added this to the 0.25.2 milestone Sep 8, 2019

noritada added 2 commits September 9, 2019 19:39

Merge with master and move whatsnew entry to v0.25.2

4b93034

Document additionally an existing issue number pandas-dev#19995 for G…

6eb0d28

…roupBy.shift

jreback approved these changes Sep 9, 2019

View reviewed changes

jreback merged commit e0c63b4 into pandas-dev:master Sep 9, 2019

lumberbot-app bot added the Still Needs Manual Backport label Sep 9, 2019

proost pushed a commit to proost/pandas that referenced this pull request Dec 19, 2019

BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill (p…

c657d9f

…andas-dev#27992)

proost pushed a commit to proost/pandas that referenced this pull request Dec 19, 2019

BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill (p…

0fc8e67

…andas-dev#27992)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill #27992

BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill #27992

noritada commented Aug 18, 2019 •

edited by jreback

Loading

mroeschke Aug 18, 2019

mroeschke Aug 18, 2019

noritada commented Aug 19, 2019

mroeschke left a comment

mroeschke Aug 19, 2019

noritada commented Aug 19, 2019

WillAyd Aug 23, 2019

noritada Sep 9, 2019

WillAyd commented Aug 23, 2019

TomAugspurger commented Aug 23, 2019 via email

jreback left a comment

jreback Sep 8, 2019

jreback commented Sep 8, 2019

noritada commented Sep 9, 2019

jreback commented Sep 9, 2019

lumberbot-app bot commented Sep 9, 2019

TomAugspurger commented Sep 9, 2019

jreback commented Sep 9, 2019

BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill #27992

BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill #27992

Conversation

noritada commented Aug 18, 2019 • edited by jreback Loading

mroeschke Aug 18, 2019

Choose a reason for hiding this comment

mroeschke Aug 18, 2019

Choose a reason for hiding this comment

noritada commented Aug 19, 2019

mroeschke left a comment

Choose a reason for hiding this comment

mroeschke Aug 19, 2019

Choose a reason for hiding this comment

noritada commented Aug 19, 2019

WillAyd Aug 23, 2019

Choose a reason for hiding this comment

noritada Sep 9, 2019

Choose a reason for hiding this comment

WillAyd commented Aug 23, 2019

TomAugspurger commented Aug 23, 2019 via email

jreback left a comment

Choose a reason for hiding this comment

jreback Sep 8, 2019

Choose a reason for hiding this comment

jreback commented Sep 8, 2019

noritada commented Sep 9, 2019

jreback commented Sep 9, 2019

lumberbot-app bot commented Sep 9, 2019

TomAugspurger commented Sep 9, 2019

jreback commented Sep 9, 2019

noritada commented Aug 18, 2019 •

edited by jreback

Loading