Possible Bug in Timezones in python Hourly data

[tristan-huber]

Hi y'all!

I've been loving using the python library for Meteostat. Recently, I've been looking at the tsun values from some locations in Germany and have become concerned that the Hourly python endpoint returns incorrectly shifted values when timezone is provided.

This example will take a second to talk through so please bear with me. First, let's consider the station 10505 near Bonn Germany. When I retrieve hourly data for Bonn for May 29 2021 and don't provide a timezone with the request I get back the following data (trimmed to just time, tsun, condition code in csv format):

time,tsun,coco
2021-05-29 04:00:00,0.0,1.0
2021-05-29 05:00:00,43.0,1.0
2021-05-29 06:00:00,60.0,1.0
2021-05-29 07:00:00,60.0,1.0
2021-05-29 08:00:00,60.0,1.0
2021-05-29 09:00:00,60.0,1.0
2021-05-29 10:00:00,60.0,1.0
2021-05-29 11:00:00,59.0,1.0
2021-05-29 12:00:00,55.0,1.0
2021-05-29 13:00:00,51.0,1.0
2021-05-29 14:00:00,54.0,2.0
2021-05-29 15:00:00,60.0,1.0
2021-05-29 16:00:00,43.0,1.0
2021-05-29 17:00:00,59.0,1.0
2021-05-29 18:00:00,60.0,1.0
2021-05-29 19:00:00,60.0,1.0
2021-05-29 20:00:00,10.0,1.0
2021-05-29 21:00:00,0.0,1.0
2021-05-29 22:00:00,0.0,1.0

Here we can see that the sky was mostly clear all day (condition code 1 or 2) and first light was detected at about 5:17 since there were 43 sunny minutes in the 05:00 hour. Similarly, it appears that sundown was around 8:10pm. If I check an external source, eg: https://www.timeanddate.com/sun/germany/bonn?month=5&year=2021, these values seem plausibly correct. Official sunrise occurs at 5:25 am (Europe/Berlin) slightly after first light is reported in this data extraction and official sunset is at 9:30 (Europe/Berlin), but it seems plausible that this station is shaded out before true sunset. Here is the first clue that something is up, these results seem consistent assuming their time stamps are in the local timezone.

If I request the same data from station 10505 but include timezone "Europe/Berlin" with the request, then I get back this data:

time,tsun,coco
2021-05-29 04:00:00+02:00,0.0,1.0
2021-05-29 05:00:00+02:00,0.0,1.0
2021-05-29 06:00:00+02:00,0.0,1.0
2021-05-29 07:00:00+02:00,43.0,1.0
2021-05-29 08:00:00+02:00,60.0,1.0
2021-05-29 09:00:00+02:00,60.0,1.0
2021-05-29 10:00:00+02:00,60.0,1.0
2021-05-29 11:00:00+02:00,60.0,1.0
2021-05-29 12:00:00+02:00,60.0,1.0
2021-05-29 13:00:00+02:00,59.0,1.0
2021-05-29 14:00:00+02:00,55.0,1.0
2021-05-29 15:00:00+02:00,51.0,1.0
2021-05-29 16:00:00+02:00,54.0,2.0
2021-05-29 17:00:00+02:00,60.0,1.0
2021-05-29 18:00:00+02:00,43.0,1.0
2021-05-29 19:00:00+02:00,59.0,1.0
2021-05-29 20:00:00+02:00,60.0,1.0
2021-05-29 21:00:00+02:00,60.0,1.0
2021-05-29 22:00:00+02:00,10.0,1.0
2021-05-29 23:00:00+02:00,0.0,1.0

The timestamps in the result now include the timezone offset. However, sunup now appears to occur at 7:17 local time, almost 2 hours after true sunrise. The last light of the day is now also shown as after sunset by ~40 minutes. It seems like adding the timezone parameter has caused all data to shift inaccurately by the timezone offset.

My current theory is that the raw date times are already in the local time zone (ie. the timezone returned by the Station api), and then are incorrectly shifted under the assumption that they're UTC time. This code in meteostat/utilities/mutations.py, called from here, seems consistent with that theory.

Additional questions I haven't explored but would love to know the answer to:

1. Does this affect other stations? In particular an example with a larger utc offset would be illustrative, however I haven't found any locations that report tsun outside of central europe, and it's hard to find ground-truth values for other weather properties which could be used to check for an offset.

2. What about DST? The raw time stamps don't have a DST, ie there's no skipped 2am on March 30th or whatnot. So.. are the times UTC+1 (no DST) or UTC+2 (with DST)? I'm not confident enough in the match up between tsun and calculated sunrise to make a call, especially since most non-DST dates are clouded out at this location. Or.. most scary to me, are the raw time stamps a mix of UTC+1 and UTC+2 and just have an interpolated row for the missing transitional hour.

Let me know what you think! If I can get an answer to question 2 above I'm happy to contribute a PR or a review to changing the code in mutations.py. Or, if there's something obvious I'm missing let me know that too :)

-Tristan

[clampr]

Hi Tristan,

Thanks for reaching out!

I just had a brief look, and it seems like something is off indeed. However, this doesn't affect data provided by DWD's open data interface (example: https://meteostat.net/en/station/10637?t=2021-05-29/2021-05-29).

I assume it's an issue with DWD POI data. I'll look into this and get back to you.

Thanks,
Chris

[clampr]

I did some further investigation and can confirm that the data you mentioned above is coming from DWD's POI dataset.

But I'm not sure whether there is a time zone issue. Here is the POI data for Frankfurt Airport:

And that's the data reported by DWD's quality-controlled hourly dataset:

Here, everything seems alright.

Now we need to figure out if something is wrong with the data for Bonn. DWD uses the UTC time zone for all datasets. It's important to remember that tsun always refers to the previous hour. So 2021-05-29 05:00:00,43.0,1.0 cannot be a CET timestamp if sunrise was at 5:17. I'd expect 43 minutes of sunshine to be reported at 6 CET if it was sunny throughout the hour.

Looking at data provided by Kachelmannwetter, it seems plausible that tsun == 0 at 4 UTC / 6 CET.

[tristan-huber]

Ah! ok, I think knowing that tsun refers to the previous/preceding hour explains most of my confusion here.

First and most basic, I did a double check of the data I'm getting back from the Hourly python endpoint for Frankfurt (station 10637) and it's consistent with what you've posted here. ie. if I don't provide a timezone with the request then I get back data showing...

time, tsun
2021-05-29 04:00:00,9.0
2021-05-29 05:00:00,45.0

and if I do provide a timezone then I see

time, tsun
2021-05-29 06:00:00+02:00,9.0

So this is consistent with the data you link in Kachelmannwetter, and is reasonably close to a computed sunrise of 5:22am CET.

The data for Bonn shows a bit more lag between computed sunrise (5:26 CET for May 29, 2021) and the first observed tsun value 2021-05-29 07:00:00+02:00,43.0. But as you point out this value from meteostat is consistent with Kachelmannwetter. So I'm willing to write off this lag as caused by some local factor, eg: shade from a hill or some other nearby obstruction whereas computed sunrise is based on the idealized horizon. So given that I believe I'm good to go here :)

Thanks again for looking into this and sharing what you found.

One final tangentially related question, you mentioned that tsun refers to the preceding hour, is this true for all data types returned from the Hourly api? eg: do prcp and wspd etc also describe the preceding hour?

[clampr]

Thank you for bringing it up @tristan-huber. Regarding Bonn I had my doubts, too.

Some of the parameters refer to the previous hour - mostly those which report sums (e.g. tsun or prcp). Others are singular measurements (e.g. temp or pres). And then there are some special cases. wpgt is the maximum wind gust within the previous hour and wspd - as far as I know - is the average wind speed in a 10-minute interval.

This should be in line with the WMO definitions. However, it might make sense adding some remarks to the documentation.