Why Do API and Website Timestamps Differ?

[jkreski]

I am seeing differences between the properties.timestamp and the last update time stamped on a reporting NWS station's page. For example, I see: 2024-11-16T18:19:00+00:00 on the API and I see: Last update | 16 Nov 1:56 pm EST on the website for: https://forecast.weather.gov/MapClick.php?CityName=Beulah&state=MI&site=APX&textField1=44.6296&textField2=-86.0955&e=0

How can they be different? Which one is accurate? Thanks!

[dwhitemv25]

Without knowing precisely when you made the API call vs loaded the MapClick page it's hard to say exactly what happened, but here is my analysis.

It looks like you just happened to hit a time where KMBL had issued a SPECI observation @ 13:19 EST, which is what you're seeing in the API call. You must have loaded the MapClick page much later, later enough that the usual synoptic METAR had been sent (which sends at 56 minutes past the hour for KMBL).

MADIS, the data source for API observations, has a long delay (20-45 mins) before publishing METAR/SPECI reports to allow for QC to run. I don't know offhand what data source MapClick is using, but it could be one that gets METARs faster.

[jkreski]

Also, who uses 45 minute old "current temp" data from the API and considers it quality data no matter how many times it's been checked? I am trying to build a web page where you can click a google map and get the "current" NWS closest station temp using the API. Seems this is not possible to get accurate data - or am I using the wrong data or something?

[scadergit]

Mapclick uses the non-qa SBN feed which only has METAR (airport) stations. The API uses the full MADIS feed, and is typically closer to 20m delayed (note some stations only report on the hour to begin with). The API will provide earlier, non-qa versions of an observation when provided by MADIS.

[dwhitemv25]

What are your requirements for "accuracy"?

[jkreski]

I would like the time stamp on the API to match the time stamp on the NWS website and have the "current" temps match as well or as close as possible. I'll attach two screen shots. They were both clicked and reloaded minutes ago. The times are hours apart.

[dwhitemv25]

As of this moment (15:33 UTC, 7:33am PST), looking at the last 10 observations in API, I can see the last 10 fine except for a missed observation at 09:56 UTC. So if there was a disruption in the data flow, it's since been resolved.

For the avoidance of any doubt, what is the exact URL you are using to fetch the observation?

[jkreski]

The NWS website is: https://forecast.weather.gov/MapClick.php?CityName=Beulah&state=MI&site=APX&textField1=44.6296&textField2=-86.0955&e=0 The reporting station for the observation URL is: https://api.weather.gov/stations/KTVC/observations/latest *Jon Kreski* AB9NN - FCC Ham Radio Call Sign (920) 931 - 5511 - Cell (231)3 83 - 4425 - Home My photos on my Instagram page <https://www.instagram.com/jkreski> My Facebook page <https://www.facebook.com/jon.kreski> *My Twitter page <https://twitter.com/Jon_Kreski>* My videos on my YouTube personal channel <https://www.youtube.com/channel/UC76atBn-lK98y6lSgvWCjYw?view_as=subscriber> My LinkedIn page <https://www.linkedin.com/in/jonkreski> Jon E. Kreski - is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to amazon.com, amazonsupply.com, or myhabit.com.

…

On Mon, Nov 18, 2024 at 10:38 AM dwhitemv25 ***@***.***> wrote: As of this moment (15:33 UTC, 7:33am PST), looking at the last 10 observations <https://api.weather.gov/stations/kmbl/observations?limit=10%22> in API, I can see the last 10 fine except for a missed observation at 09:56 UTC. So if there was a disruption in the data flow, it's since been resolved. For the avoidance of any doubt, what is the exact URL you are using to fetch the observation? — Reply to this email directly, view it on GitHub <#777 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF4E5WMISUUQ3BY44OWEIY32BICYVAVCNFSM6AAAAABR5EIDWKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRZGM3DGMI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dwhitemv25]

I think you meant KMBL for the station ID in your API call, since that is the nearest station returned on the MapClick page. But the form is correct.

If you're OK with just returning airports, the aviationweather.gov API gets METARs faster than the weather.gov API, but there is no QC.

[jkreski]

The API feed that corresponds to Manistee County - Blacker Airport (KMBL) seems to be concistantly hours old. Wonder if something needs to be re-synced or someone notified. Thanks for the hint for the aviation site. I will look into it. I was suprised that with my own home weather station I can see the near real-time temp and wind on the internet but the federal government with all their resources and technology can't seem to provide the same thing. Something seems amiss... Just this humble ex-Auditor's opinion!

[dwhitemv25]

I can check the MADIS QC data to see if that station is getting consistently flagged for problems. I had on and off problems with that with my CWOP station as there was a broken down station nearby that was throwing off its analysis.

[dwhitemv25]

I checked the MADIS QC data for KMBL and it doesn't seem to indicate any systemic problems. The temperature sensor, which had the highest bad observation count, was flagged 51 times in the last monthly review (10/7 through 11/3) for 6.3% of the 806 observations made. One per hour would be 776 for a 31 day month, and airports may send extra observations, so it doesn't seem any are missing. The latest weekly summary (11/11 through 11/17) doesn't show any massive number of missing observations either. So the data is getting to MADIS.

That would imply there is an issue with API not getting the latest observation when queried, or between API and you as there is a cache layer in place and it could be returning stale data.

[jkreski]

Thanks for the thorough investigation and time invested and patience with me! I guess that I can't solve the API not getting the latest info when queried but I think that's what it is. If it were a cache issue I would think that on subsequent queries it would return no or the same information. Rather, it returns different updated albeit old data. I'm attaching a screen shot that shows, on the right, the JavaScript code that I use to make the API call. Should I refer this to a JavaScript guru on the forum as a seperate post or is something that you have knowledge of? I am an old time long time Visual Basic programmer recently converted over to JavaScript for web development. I wondeer if there is some magic syntax to flush the cache either before or during a call?

[dwhitemv25]

It's not the web browser cache I'm referring to (most likely?), but the Akamai cache layer that serves api.weather.gov. The test is to look at the Cache-Control and Last-Modified headers from the API request (you can see these in the Network panel in the web developer tools) and see if they seem very out of date. Observations are cached for 5 minutes in the CDN.

For example, using the API test function, a query for the latest observation for a nearby station to me returns these relevant headers:

cache-control: public, max-age=296, s-maxage=300  
expires: Wed, 20 Nov 2024 00:00:05 GMT  
last-modified: Tue, 19 Nov 2024 21:54:00 GMT 

max-age will go down as the expiration time nears, once it hits zero and the page is queried again it will fetch it from the origin.

Some years back there was problems with the CDN and Last-Modified headers causing the cache to regurgitate pages from months prior. I believe all that functionality is disabled, so let's hope it's not that problem rearing its head again.

[sullynole]

This is not a cache issue. There are known issues with the ingest of MADIS data into the API backend, mentioned on the Updates tab here https://www.weather.gov/documentation/services-web-api

Can pull the data direct from MADIS here: https://madis-data.ncep.noaa.gov/public/sfcdumpguest.html

[jkreski]

A new rabbit hole! Where can I find the API test function? Thanks!

…

On Tue, Nov 19, 2024, 6:59 PM dwhitemv25 ***@***.***> wrote: It's not the web browser cache (most likely?), but the Akamai cache layer that serves api.weather.gov. The test is to look at the Cache-Control and Last-Modified headers from the API request (you can see these in the Network panel in the web developer tools) and see if they seem very out of date. Observations are cached for 5 minutes in the CDN. For example, using the API test function, a query for the latest observation for a nearby station to me returns these relevant headers: cache-control: public, max-age=296, s-maxage=300 expires: Wed, 20 Nov 2024 00:00:05 GMT last-modified: Tue, 19 Nov 2024 21:54:00 GMT max-age will go down as the expiration time nears, once it hits zero and the page is queried again it will fetch it from the origin. Some years back there was problems with the CDN and Last-Modified headers causing the cache to regurgitate pages from months prior. I believe all that functionality is disabled, so let's hope it's not that problem rearing its head again. — Reply to this email directly, view it on GitHub <#777 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF4E5WPK7YX4JADEO3KV3ST2BPGGNAVCNFSM6AAAAABR5EIDWKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZRGUZTCMA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dwhitemv25]

When I said "API test function," I was referring to the ability to submit API requests from the Specification page (https://www.weather.gov/documentation/services-web-api). It returns the response headers in a separate box.

You could do it just as easily from a web browser or curl.

[joepperkins]

I presume you are refining to a Swagger UI<https://petstore.swagger.io/> type self-documenting and testing Web API interface. ***@***.*** From: dwhitemv25 ***@***.***> Sent: Wednesday, November 20, 2024 12:47 PM To: weather-gov/api ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [weather-gov/api] Why Do API and Website Timestamps Differ? (Discussion #777) When I said "API test function," I was referring to the ability to submit API requests from the Specification page (https://www.weather.gov/documentation/services-web-api). It returns the response headers in a separate box. You could do it just as easily from a web browser or curl. — Reply to this email directly, view it on GitHub<#777 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADJHCGZVCQK3MEHBHLM2I332BTYK7AVCNFSM6AAAAABR5EIDWKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZSHA2TKNA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>

[jkreski]

Here is what I am seeing - the last update via the API is an hour behind the NWS website and cache-control: public, max-age=300, s-maxage=300 Print screens pasted in below: [image: Screenshot (224).png][image: Screenshot (226).png][image: Screenshot (225).png]

…

On Wed, Nov 20, 2024 at 3:46 PM dwhitemv25 ***@***.***> wrote: When I said "API test function," I was referring to the ability to submit API requests from the Specification page ( https://www.weather.gov/documentation/services-web-api). It returns the response headers in a separate box. You could do it just as easily from a web browser or curl. — Reply to this email directly, view it on GitHub <#777 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF4E5WP4YMEW2PI4F4F2Z732BTYK3AVCNFSM6AAAAABR5EIDWKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZSHA2TKNA> . You are receiving this because you authored the thread.Message ID: ***@***.***>