How to get older forecast data?

[ilarroche-altanova]

Dear reader,

I am currently trying to use NWS to have access to historical weather forecast data. However, when using the API, I only succeed to get forecast starting from the current date. I have tried different time queries in order to have forecasts for a given gridpoint on an older date, such as 4/5/2022, without success. Could you please tell me how I should proceed? If I do not have access to this data, is there another database from NCEI that provides it? If so, how can I have access to it ?

Thank you very much for your help!

Ines

[cmhettinger]

I have a similar question to add to this discussion. When using the /forecast/hourly API, the API starts with the current date/time (per original question). I would like to be able to get the full days worth of data, regardless of when the API call was made. For example, using curl at 9am I am missing the data from 12am-9am, and could really use the entire day. I understand this is a "forecast" endpoint which would imply going forward, but it would be very useful to be able to get the full days worth of datapoints, whether as a parameter to the /forecast/hourly endpoint, or a different endpoint for historical.

[scadergit]

Curious why you want expired forecast model data that was a prediction and possibly not what actually happened?

The solution that provides forecast data to the API only retains the current netcdf provided by the field office. These data files only have forward forecast periods. There is discussion to change this approach, and the new approach might lend itself to this feature, but there is no timeline to implement. New features are best communicated though your local office, as they represent teams that determine the feature priority.

[cmhettinger]

thanks for the follow-up...

"... is at best at the city or station level, expensive, or has an overly restrictive license" -- agreed

Thank you for the current forecast API, it meets 90% of my needs. Its unfortunate that there are not actuals vs forecast available, but I understand the scope of the current api and why that is problematic.

As a future/tomorrowland requirement, I'd like to see NWS be able to support APIs for actuals data. Seems a shame that we are all funding a National Weather Service, but for a "what actually happened" during the day, we need to go find a third party api provider. :)

[vigouredelaruse]

seconded feature request for timeseries of forecast / actual tuple

for justification note that consumers are building responsive time series models that do predictions and hindcasts that must be calibrated against actuals

thanks from a non-american

although: i just noticed this netcdf apparently upstream source and a brief scan seems to indicate archives https://madis-data.ncep.noaa.gov/

[sullynole]

If looking for current and past analysis in a gridded format, NWS produces a real time analysis dataset (RTMA/URMA) on the supercomputer that is available in grib2 format here:
https://www.nco.ncep.noaa.gov/pmb/products/rtma
or from NOMADS: https://nomads.ncep.noaa.gov
and also available in AWS open data: https://registry.opendata.aws/noaa-rtma/

Model data are not available via NWS API. The NOMADS application was designed to provide that type of service. NWS API does provide point observations from MADIS for realtime current/past data (up to 7 days) using the observations endpoints.

[cmhettinger]

Thanks for the references, I'll definitely check them out. That being said, a quick gander at the data shows some of those nomads files as 6gb monsters vs the KB files the API provides; :) It looks like we have some architectural decisions to work through on our side. Thanks again for the information, its been very useful.

[sullynole]

Yes, the size of model and analysis data with all the levels and variables available can be overwhelming. There are options that make retrieving the full grib2 files a little lighter to digest.

Within NOMADS there is the Grib filter option: https://nomads.ncep.noaa.gov/info.php?page=gribfilter

Also, UCAR provides a THREDDS Data Server:
https://www.unidata.ucar.edu/software/tds -> https://thredds.ucar.edu

Probably other options out there. I believe I have even seen some GIS layers of specific RTMA fields such as temperature and precipitation.

[vigouredelaruse]

useful discussion to the eyes of a non-domain software engineer currently building realtime and historical observation timeseries ingesters - thanks for the insider line

[dangelsaurus]

I know this isn't technically the correct place to ask, but looks like there are a lot of smart people here... Do any of the resources listed here have 7 day forecast as well? Unless I'm misunderstanding what I'm looking at (very good possibility) I can't find anything more than same day.

[dwhitemv25]

If by "resources listed here" you mean NOMADS/THREDDS, then yes, but in formats that would require processing before use.

If you mean the NWS API in general, then yes, see the API documentation for how to fetch it.

If that's not what you meant, can you be more specific?

[dangelsaurus]

yes sorry the NOMADS grib2 data, I got a bit lost in all the different resources available, but from what I could gather, it wasn't the 7 day forecast that we get from this API, rather just current day data?

When during more reading, it lead me to believe that since multi-day forecast data depends on world-wide forecast, that these large datasets weren't available from NWS in a grib2 format? (I should also reiterate that I got VERY lost in all the different resources so maybe I'm just missing something obvious.)

Essentially I'm wondering if there's any easier way to access the 2.5km grid 7 day forecast for EVERYWHERE in the US. (Not that I don't like using this service, but rather my use case might be a little different and I'm trying not to abuse the API with too many calls :-) )

[dwhitemv25]

This is a peek behind the scenes on where weather forecasts come from...

It may depend on what, exactly, a "forecast" is for your needs. Technically, the models generate forecasts for individual weather elements. The trick is that there are many models that handle some situations better than others and there is no automatic way to derive a "best" forecast between them.

This is where the forecasters at your local Weather Forecast Office come in -- they view the model data & observations and combine that with local knowledge to develop a forecast. The human-derived forecast is ultimately input into the National Digital Forecast Database (NDFD), which is available in gridded (grib2) formats.

The NDFD gridded elements, which is on a 5km grid, are available at https://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/. The description of the available elements is at https://vlab.noaa.gov/web/mdl/ndfd.

If you are looking for a text description, like what forecast.weather.gov outputs, but for any given point using the grids as a source, you will have to write a translator yourself.

Of course, you can choose another model, or set of models, and derive a forecast from that yourself. An example of how this is done can be found in the description of the Pirate Weather service, located at http://docs.pirateweather.net/en/latest/, starting under the "Data Sources" section.

For the record, the global model datasets ARE available on NOMADS, but yes, it's a LOT of data to sort through. Even NBM is a lot; I download the NBM GRIBs through hour 168 and that is still 10GB compressed.

Hope that helps. If you want to discuss further, let's start a new thread.