API returning no detailed data #720

kfaubel · 2024-05-03T18:29:48Z

kfaubel
May 3, 2024

I have been using this API for several years and it stopped working earlier this week. I do a GET of:
https://forecast.weather.gov/MapClick.php?lat=40.000&lon=-75.000&FcstType=digitalDWML
The data I now get is below.

The issue is that the element has the attributes I expect, but no actual data. There should be 169 elements under "humidity", "cloud-amount", "probability of precipitation", ...

Its not just one reporting location.

I looked at know issues and updates but I did not see anything. Any clues?

<dwml xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0"
    xsi:noNamespaceSchemaLocation="http://graphical.weather.gov/xml/DWMLgen/schema/DWML.xsd">
    <head>
        <product concise-name="tabular-digital" operational-mode="developmental" srsName="WGS 1984">
            <creation-date refresh-frequency="PT1H">2024-05-03T13:12:01-04:00</creation-date>
        </product>
        <source>
            <production-center>Philadelphia, PA</production-center>
            <credit>https://www.weather.gov/phi/</credit>
            <more-information>https://www.nws.noaa.gov/forecasts/xml/</more-information>
        </source>
    </head>
    <data>
        <location>
            <location-key>point1</location-key>
            <description>Cinnaminson NJ, NJ</description>
            <point latitude="40.01" longitude="-75" />
            <city state="NJ">Cinnaminson NJ</city>
            <height datum="mean sea level">95</height>
        </location>
        <moreWeatherInformation applicable-location="point1">
            //forecast.weather.gov/MapClick.php?lat=40.01&lon=-75&FcstType=digital</moreWeatherInformation>
        <time-layout time-coordinate="local" summarization="none">
            <layout-key>k-p1h-n1-0</layout-key>
            <start-valid-time>2024-05-03T14:00:00-04:00</start-valid-time>
            <end-valid-time>2024-05-03T15:00:00-04:00</end-valid-time>
            <start-valid-time>2024-05-03T15:00:00-04:00</start-valid-time>
            <end-valid-time>2024-05-03T16:00:00-04:00</end-valid-time>
            ... 300+ lines deleted ...
            <start-valid-time>2024-05-10T12:00:00-04:00</start-valid-time>
            <end-valid-time>2024-05-10T13:00:00-04:00</end-valid-time>
            <start-valid-time>2024-05-10T13:00:00-04:00</start-valid-time>
            <end-valid-time>2024-05-10T14:00:00-04:00</end-valid-time>
        </time-layout>
        <parameters applicable-location="point1">
            <humidity type="relative" units="percent" time-layout="k-p1h-n1-0" />
            <cloud-amount type="total" units="percent" time-layout="k-p1h-n1-0" />
            <probability-of-precipitation type="floating" units="percent" time-layout="k-p1h-n1-0" />
            <wind-speed type="sustained" time-layout="k-p1h-n1-0" />
            <temperature type="heat index" time-layout="k-p1h-n1-0" />
            <direction type="wind" units="degrees true" time-layout="k-p1h-n1-0" />
            <temperature type="dew point" time-layout="k-p1h-n1-0" />
            <wind-speed type="gust" time-layout="k-p1h-n1-0" />
            <hourly-qpf type="floating" units="inches" time-layout="k-p1h-n1-0" />
            <temperature type="hourly" time-layout="k-p1h-n1-0" />
        </parameters>
    </data>
</dwml>

akrherz · 2024-05-03T18:32:17Z

akrherz
May 3, 2024

See https://www.weather.gov/idp/forecastmigration

0 replies

kfaubel · 2024-05-03T20:20:03Z

kfaubel
May 3, 2024
Author

Thank you. I did not see that.

It does say:

These forecast types from MapClick.php are not functional: digitalDWML, digitalJSON, mgraphical, wxplan.

It's listed as a known issue but does not say that it has been discontinued. Any idea if its coming back? Any alternative for this kind of data?

Thank you.

1 reply

akrherz May 3, 2024

My suggestion would be to email [email protected] and ask to be added to the ticket that is tracking that particular issue with the forecast.weather.gov migration to IDP

paulnadler · 2024-05-03T21:03:06Z

paulnadler
May 3, 2024

I emailed them earlier today and was forwarded the ticket thread, but I don't see where we can continue to follow the status of the ticket. It was apparently fixed and closed at one point, then reopened again. And at the time was 139 in the queue. Is there somewhere we can follow the status of these (and other) tickets or are they internal only?

0 replies

jah1007 · 2024-05-03T21:49:12Z

jah1007
May 3, 2024

The forecastmigration page Daryl shared earlier will remain up-to-date.

0 replies

Kimball31 · 2024-05-07T21:33:24Z

Kimball31
May 7, 2024

I have been using this API for several years and it stopped working earlier this week. I do a GET of: https://forecast.weather.gov/MapClick.php?lat=40.000&lon=-75.000&FcstType=digitalDWML The data I now get is below.

I have the same issue with MapClick.php FcstType=digitalDWML
I agree. It's been out since May 1st at least.
It's out for any location I look at.

Emailing [email protected] is a blackhole. The support ticket thread shows lots of organizations have reported the service is broken. But the ticket contains nothing from NOAA in it about what's happening, what they are doing, and when to expect fix if at all.

0 replies

brewin · 2024-05-07T23:19:22Z

brewin
May 7, 2024

I reported this issue on April 24th when they did some kind of test of the migration and I first noticed the problem. Unfortunately, they decided to go through with the migration without fixing it.

0 replies

kollymars · 2024-05-08T13:31:28Z

kollymars
May 8, 2024
Maintainer

You can retrieve DWML from api.weather.gov on the /gridpoints/XXX/xx,xx/forecast/ and /forecast/hourly endpoints with the proper accept header. Although it's not a one for one, it will probably be a better solution long term. NWS is working on a fix for the forecast.weather.gov API, however there is no established timeline as of now for the fix.

1 reply

tkafka May 9, 2024

It definitely isn't one by one replacement - I have been using the MapClick endpoint to get a chartable weekly data, but for gridpoint hourly forecast, the precipitation data series end after 72 hours, and there seems to be no way of going beyond that :/

kfaubel · 2024-05-09T13:52:27Z

kfaubel
May 9, 2024
Author

This (https://api.weather.gov/gridpoints/BOX/55,102/forecast/hourly) appears to be close semantic replacement for the forecast.weather.gov APIs, but they are missing:

cloud-amount
hourly-qpf

I appreciate the service and I can migrate to the 2-step api.weather.com process. Moving to a JSON result will make my code cleaner for sure. Without cloud-amount and hourly-qpf I may stuck waiting for a fix to this issue, or some option I am not aware of right now.

I use the data to generate screens like this last success image on April 30th:

The light grey background is cloud cover, the dark blue is chance of precipitation, and the light blue is the expected precipitation amount.

The one thing I have not found anywhere, except the forecast.weather.gov website itself, is estimated snowfall/hour.

0 replies

netbymatt · 2024-05-09T14:08:48Z

netbymatt
May 9, 2024

Try looking at https://api.weather.gov/gridpoints/BOX/71,90 It includes quantitativePrecipitation and skyCover.

I could not get this information for the exact gridpoint you're looking at above (55,102). But switching to (71,90) which is directly on top of Boston works. I'm using that endpoint successfully to generate very similar charts. Grey is sky cover and those tiny yellow blips just above the x axis at 12am 5/11 and ~6pm 5/11 are quantitative precipitation.

For better visibility here's Chicago showing the yellow quantitative measurement a little better.

4 replies

kfaubel May 9, 2024
Author

This https://api.weather.gov/gridpoints/BOX/55,102 worked for me and it was exciting to see snowfallAmount.

It looks like some properties have 5 days of data (temperature, relativeHumidity, skyCover, ...) but others only have 2-3 days of data (dewPoint, windSpeed, snowfallAmount, probablilityOfPrecipitation, ...).

I compared the results for 71,90 with 55,102 and they are returning a different number of values for many of the properties.

I get that properties, like windSpeed, may have a low level of confidence 5 days out but its better than no data at all.

Cool graphs.

netbymatt May 9, 2024

Thanks!

You're absolutely correct not all the products are forecast out to the same number of days. You should also be aware that a lot of them change in granularity as you go further into the future. Probability of precipitation is a good example above. It's the blue fill, for about 48 hours it's up to 1-hour resolution, and then it jumps to 6-hour resolution with the plateaus between 6a~6p on 5/11.

Kimball31 May 9, 2024

Some values represent more than one hour. /PT1H - /PT23H is the numbers of hours. PT4H below means the value covers 4 hours.

"temperature": { "uom": "wmoUnit:degC", "values": [ { "validTime": "2024-05-09T07:00:00+00:00/PT4H", "value": 12.777777777777779 },

kfaubel May 9, 2024
Author

I missed the PT1H-PT23H. Thanks.

Looks like I have some work to do.

Kimball31 · 2024-05-09T18:05:12Z

Kimball31
May 9, 2024

for what it's worth, here is how I expand the time compressed elements:

`
function expandTimeElement($rawElement) {

$hours = array(
'PT1H' => 1,
'PT2H' => 2,
'PT3H' => 3,
'PT4H' => 4,
'PT5H' => 5,
'PT6H' => 6,
'PT7H' => 7,
'PT8H' => 8,
'PT9H' => 9,
'PT10H' => 10,
'PT11H' => 11,
'PT12H' => 12,
'PT13H' => 13,
'PT14H' => 14,
'PT15H' => 15,
'PT16H' => 16,
'PT17H' => 17,
'PT18H' => 18,
'PT19H' => 19,
'PT20H' => 20,
'PT21H' => 21,
'PT22H' => 22,
'PT23H' => 23,
'PT1D' => 24,
);

$expandedArray = [];
foreach ($rawElement as $compressedTime) {
    $timestamp = $compressedTime['validTime'];
    $timestampParts = explode('/', $timestamp);
    $compressed = $timestampParts[1];
    
    for ($i = 0; $i < $hours[$compressed]; $i++) {
        $expandedArray[] = $compressedTime['value'];
    }
}
return $expandedArray;

}
`

0 replies

paulnadler · 2024-05-21T22:53:55Z

paulnadler
May 21, 2024

There was a recent service chance notice about the hourly forecast data.
https://www.weather.gov/media/notification/pdf_2023_24/scn24-50_removal_of_digital_projects_from_forecast.pdf

Starting June 27th, hourly data from forecast.weather.gov will be migrated to api.weather.gov. However the API docs only list the /alerts endpoint as operational. Meaning the other endpoints can still change without notice.

Can we expect other endpoints to be added to the list of operational endpoints now that forecast.weather.gov APIs are being migrated?

6 replies

brewin May 22, 2024

digitalDWML and digitalJSON now redirect to f1.weather.gov:

https://forecast.weather.gov/MapClick.php?lat=35.4685&lon=-97.5213&unit=0&lg=english&FcstType=digitalJSON

Will that redirect disappear on June 27th? Or will all the old forecast.weather.gov APIs redirect to f1.weather.gov?

I would love to be able to use api.weather.gov, but I still see frequent errors. The notice even mentions that issue at the bottom and that it will be fixed "this fall". But that would mean there will be several months of random errors. We're supposed to retry until it works? I'm very confused.

jah1007 May 22, 2024

If there is any change the SCN will be updated accordingly. As it stands, yes the redirect to f1 will go dark on 6/27.

paulnadler May 22, 2024

@jah1007 will the replacement endpoints, /gridpoints and /forecast/hourly be subject to SCN or can they still change without notice?

jah1007 May 23, 2024

All endpoints on the API are subject to the Change Notification process. The banner on the documentation page that said contrary was old and has been removed.

dwhitemv25 Jun 6, 2024

Somewhat relevant for this discussion, SCN 24-50, which announced the decommissioning of the digitalJSON and digitalDWML formats, has been changed to occur at a future time. Those outputs are sticking around until they can port the missing elements to API and make the formats available there. Apparently there is still enough traffic in those formats to keep them alive.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API returning no detailed data #720

{{title}}

Replies: 11 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

API returning no detailed data #720

Replies: 11 comments · 12 replies

kfaubel May 3, 2024 Author

kollymars May 8, 2024 Maintainer

kfaubel May 9, 2024 Author

kfaubel May 9, 2024 Author

kfaubel May 9, 2024 Author

Replies: 11 comments 12 replies

kfaubel
May 3, 2024
Author

kollymars
May 8, 2024
Maintainer

kfaubel
May 9, 2024
Author

kfaubel May 9, 2024
Author

kfaubel May 9, 2024
Author