Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,20 @@ Use these roles for badges matching the Earth Data Hub webportal:
{bdg-geobrowser}`Geobrowser available`
```

## Required envars to properly build the documentation

- `READTHEDOCS_CANONICAL_URL`:\
Root of the documentation site
(can be also a root absolute path)
- `BASE_URL`:\
external full URL to the EDH portal
- `DATASTORE_INTERNAL_HOST`:\
Main EDH access domain (by default `data.earthdatahub.destine.eu`)
- `DATASTORE_HOST`:\
Alternative EDH access URL (required for restricted datasets, by default `api.earthdatahub.destine.eu`)
- `DOCUMENTATION_PORTAL`:\
Public URL to the documentation portal (by default `https://earthdatahub.destine.eu/docs`)

## Licenses

- Documentation and other content: [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/legalcode)
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ dependencies = [
"sphinx-llm>=1.1.0",
"sphinx-notfound-page>=1.1.0",
"sphinx-sitemap>=2.9.0",
"sphinx-substitution-extensions>=2026.8.13.1",
"sphinx-togglebutton>=0.4.5",
"sphinxcontrib-youtube>=1.5.0"
]
Expand Down
14 changes: 13 additions & 1 deletion source/404.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
hide_ai_links: true
---

# Page not found

We cannot find the page you are looking for.
Oh no! The page you're looking for can't be found.

**Error code: 404**

If the problem persists, please {edh_url}`contact us </contacts>`.

```{container} buttons
[Back to documentation](index)
```
105 changes: 102 additions & 3 deletions source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,13 @@
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

import os
import sys
import urllib
from pathlib import Path

from docutils import nodes

sys.path.append(str((Path(__file__).parent / "_ext").resolve()))

# -- Project information -----------------------------------------------------
Expand All @@ -30,6 +34,7 @@
"notfound.extension",
"sphinxcontrib.youtube",
"sphinx_iconify",
"sphinx_substitution_extensions",
"sphinx_llm.txt",
"edh_badges",
"edh_icons",
Expand All @@ -42,9 +47,14 @@
templates_path = ["_templates"]
exclude_patterns = ["_collections"]

myst_enable_extensions = [
"attrs_block",
]
myst_enable_extensions = ["attrs_block", "substitution"]

myst_substitutions = {
"datastore_host": os.getenv("VITE_DATASTORE_HOST", "api.earthdatahub.destine.eu"),
"datastore_internal_host": os.getenv(
"DATASTORE_INTERNAL_HOST", "data.earthdatahub.destine.eu"
),
}

collections = {
"edh-learning": {
Expand Down Expand Up @@ -182,3 +192,92 @@
# Meta nodes (notebook descriptions, see edh_llms_markdown) have no Markdown equivalent
llms_txt_suppress_unknown_node_warnings = ["meta"]
markdown_http_base = html_baseurl


def edh_url_with_camefrom_role(name, rawtext, text, lineno, inliner, options=None):
"""Generates configurable URL to EDH with a came_from parameter back to documentation portal.

Syntax:
{edh_url_with_camefrom}`Link Text </aaa/bb/ccc>`
Or just: {edh_url_with_camefrom}`/aaa/bb/ccc`
"""

options = options or {}

# Extract the current document name (e.g., 'folder/page')
env = inliner.document.settings.env
docname = env.docname
current_page = f"{docname}.html"

# Read the base URL from the environment variable.
base_url = os.getenv("BASE_URL", "https://earthdatahub.destine.eu").rstrip("/")
docs_root_url = os.getenv(
"DOCUMENTATION_PORTAL_ROOT", "https://earthdatahub.destine.eu/docs"
).rstrip("/")

# Parse the text (Link Text <path>)
if "<" in text and ">" in text:
link_text, path = text.split("<")
link_text = link_text.strip()
path = path.strip(">")
else:
# Fallback if the user just types {edh_url}`/aaa/bb/ccc`
link_text = text
path = text

# Ensure the path starts with a slash
path = path.strip()
if not path.startswith("/"):
path = "/" + path

# Safely URL-encode the current page name
encoded_page = urllib.parse.quote(f"{docs_root_url}/{current_page}")

# Construct the final URL
separator = "&" if "?" in base_url else "?"
full_url = f"{base_url}{path}{separator}came_from={encoded_page}"

# Create the docutils reference node
node = nodes.reference(rawtext, link_text, refuri=full_url, **options)
return [node], []


def edh_url_role(name, rawtext, text, lineno, inliner, options=None):
"""Generates configurable URL to EDH.

Syntax:
{edh_url}`Link Text </aaa/bb/ccc>`
Or just: {edh_url}`/aaa/bb/ccc`
"""

options = options or {}

# Read the base URL from the environment variable.
base_url = os.getenv("BASE_URL", "https://earthdatahub.destine.eu").rstrip("/")

# Parse the text (Link Text <path>)
if "<" in text and ">" in text:
link_text, path = text.split("<")
link_text = link_text.strip()
path = path.strip(">")
else:
# Fallback if the user just types {edh_url}`/aaa/bb/ccc`
link_text = text
path = text

# Ensure the path starts with a slash
path = path.strip()
if not path.startswith("/"):
path = "/" + path

# Construct the final URL
full_url = f"{base_url}{path}"

# Create the HTML anchor node
node = nodes.reference(rawtext, link_text, refuri=full_url, **options)
return [node], []


def setup(app):
app.add_role("edh_url", edh_url_role)
app.add_role("edh_url_with_camefrom", edh_url_with_camefrom_role)
2 changes: 1 addition & 1 deletion source/core-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ Best for analysing how variables change over time at specific locations.
No single chunking scheme fits every use case. We optimise each dataset for its most common access patterns and, when needed, publish multiple versions for different workflows.

```{hint}
The [Climate DT](https://earthdatahub.destine.eu/collections/climate-dt-2) high-resolution datasets are available with both map-optimised and time series-optimised chunking, so you can choose the version that best fits your workflow.
The {edh_url}`Climate DT </collections/climate-dt-2>` high-resolution datasets are available with both map-optimised and time series-optimised chunking, so you can choose the version that best fits your workflow.
```

## Webinar
Expand Down
37 changes: 24 additions & 13 deletions source/data-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,13 @@ Let's verify your environment. We have provided a **public test dataset** that y
Run the following in your Python console:

```{code-block} pycon
---
substitutions:
---
>>> import xarray as xr

>>> xr.open_dataset(
... "https://data.earthdatahub.destine.eu/public/test-dataset-v0.zarr",
... "https://|datastore_internal_host|/public/test-dataset-v0.zarr",
... chunks={}, # Tells Xarray to load data on-demand
... engine="zarr",
... )
Expand All @@ -80,8 +83,8 @@ And just like that, you are connected! The output above shows an Xarray `Dataset

To access our full suite of datasets, you will need a free **API Key**.

1. **Register an account** on the [DestinE Platform](https://earthdatahub.destine.eu/login?came_from=https%3A%2F%2Fearthdatahub.destine.eu%2Fdocs%2Fdata-access.html%23set-up-your-api-key).
1. **Visit your [Earth Data Hub account settings](https://earthdatahub.destine.eu/quota-api-keys#my-personal-access-tokens)**.
1. **Register an account** on the {edh_url_with_camefrom}`DestinE Platform </login>`.
1. Visit {edh_url}`API keys management section </quota-api-keys#my-personal-access-tokens>`.
1. **Copy your default API key** (or generate a new one).

```{admonition} Climate DT requires upgraded access.
Expand All @@ -99,12 +102,15 @@ Most datasets in the Earth Data Hub catalogue require your API key. Here are two

The quickest way to test protected data is to include your API key directly as the password in the dataset URL:

```python
```{code-block} python
---
substitutions:
---
import xarray as xr

# Replace <your API key> with your actual key
xr.open_dataset(
"https://edh:<your API key>@data.earthdatahub.destine.eu/private/test-dataset-v0.zarr",
"https://edh:<your API key>@|datastore_internal_host|/private/test-dataset-v0.zarr",
chunks={},
engine="zarr",
)
Expand All @@ -121,28 +127,32 @@ If you don't have it already, **create a file** in your home directory:

Then, **add the following lines** to the `.netrc` file:

```text
machine data.earthdatahub.destine.eu
```{code-block} text
---
substitutions:
---
machine |datastore_internal_host|
password <your API key>

machine api.earthdatahub.destine.eu
machine |datastore_host|
password <your API key>
```

```{note}
Earth Data Hub uses multiple endpoints, so your `.netrc` file needs an entry for each one. You can retrieve your `.netrc` file directly from your [EDH account settings](https://earthdatahub.destine.eu/quota-api-keys##my-personal-access-tokens) using the {octicon}`info` button.
Earth Data Hub uses multiple endpoints, so your `.netrc` file needs an entry for each one. You can retrieve your `.netrc` file directly from your {edh_url}`EDH account settings </quota-api-keys#my-personal-access-tokens>` using the {octicon}`info` button.
```

Now, you can load protected datasets cleanly by enabling environment trust (`trust_env=True`):

```{code-block} python
---
emphasize-lines: 5
substitutions:
---
import xarray as xr

xr.open_dataset(
"https://data.earthdatahub.destine.eu/private/test-dataset-v0.zarr",
"https://|datastore_internal_host|/private/test-dataset-v0.zarr",
storage_options={"client_kwargs": {"trust_env": True}}, # Auto-detect your .netrc
chunks={},
engine="zarr",
Expand All @@ -160,7 +170,7 @@ Direct authentication is fine for a quick test, but using a `.netrc` file is the

## Upgraded access

Some datasets, including the [Destination Earth Climate Adaptation Digital Twin (Climate DT)](https://earthdatahub.destine.eu/collections/climate-dt-2) collection, require upgraded access. You can easily recognise these datasets in the catalogue by their {bdg-restricted}`Restricted` badge.
Some datasets, including the \[Destination Earth Climate Adaptation Digital Twin ({edh_url}`Climate DT </collections/climate-dt-2>`) collection, require upgraded access. You can easily recognise these datasets in the catalogue by their {bdg-restricted}`Restricted` badge.

For example, if you try to access a Climate DT dataset without the required permissions, your request will fail with `HTTP 403 Forbidden`.

Expand All @@ -177,7 +187,7 @@ When you stream data on demand, each chunk is retrieved through an HTTP request
That may sound like a lot, and for most workflows, it is. But if you need to access large amounts of data, the key is to **stream only the data you need** and avoid downloading the same chunks repeatedly. To reduce unnecessary requests, see [](cache-data-locally).

```{note}
Your quota resets at midnight on the **first day of each month**. You can [check your quota usage](https://earthdatahub.destine.eu/quota-api-keys#quota) at any time.
Your quota resets at midnight on the **first day of each month**. You can {edh_url}`check your quota usage </quota-api-keys#quota>` at any time.
```

(cache-data-locally)=
Expand All @@ -191,11 +201,12 @@ Before running the example, replace **`URL`** with your dataset URL and **`CACHE
```{code-block} python
---
emphasize-lines: 10-15
substitutions:
---
import xarray as xr

# Replace with the Earth Data Hub dataset URL you'd like to access
URL = "https://data.earthdatahub.destine.eu/private/test-dataset-v0.zarr"
URL = "https://|datastore_internal_host|/private/test-dataset-v0.zarr"

# Replace with the local directory you'd like to use for caching
CACHE_STORAGE = "./edh_cache/"
Expand Down
2 changes: 1 addition & 1 deletion source/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ layout: landing
Welcome to the Earth Data Hub (EDH) documentation.

```{container} buttons
[Earth Data Hub](https://earthdatahub.destine.eu/)
{edh_url}`Earth Data Hub </>`
```

```{grid} 1 1 2 3
Expand Down
15 changes: 9 additions & 6 deletions source/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ Understand common errors and how to solve them.
If the dataset URL is correct but you get a `FileNotFoundError` or `NotImplementedError` when opening a dataset, your Zarr library may not support the **Zarr v3** format used by Earth Data Hub.

````{dropdown} FileNotFoundError
```python
FileNotFoundError: No such file or directory: 'https://data.earthdatahub.destine.eu/era5/era5-single-levels-atmosphere-daily-utc-v0.zarr'
```{code-block} python
:substitutions:
FileNotFoundError: No such file or directory: 'https://|datastore_internal_host|/era5/era5-single-levels-atmosphere-daily-utc-v0.zarr'
```
````

Expand Down Expand Up @@ -70,8 +71,9 @@ Some datasets, including those in the **Destination Earth Climate Adaptation Dig
To request access, follow the instructions in [](upgraded-access).

````{dropdown} ClientResponseError: 403 Forbidden
```python
ClientResponseError: 403, message='Forbidden', url='https://api.earthdatahub.destine.eu/climate-dt-2/IFS-NEMO-SSP3-7.0-sfc-hourly-standard-v0.zarr/zarr.json'
```{code-block} python
:substitutions:
ClientResponseError: 403, message='Forbidden', url='https://|datastore_host|/climate-dt-2/IFS-NEMO-SSP3-7.0-sfc-hourly-standard-v0.zarr/zarr.json'
```
````

Expand All @@ -82,7 +84,8 @@ If you receive a `ClientResponseError: 429 Too Many Requests`, Earth Data Hub ha
You will need to wait for your quota to reset before accessing more data. To learn more about quotas and how to make the most of them, see [](quotas).

````{dropdown} ClientResponseError: 429 Too Many Requests
```python
ClientResponseError: 429, message='Too Many Requests', url='https://data.earthdatahub.destine.eu/era5/era5-single-levels-atmosphere-daily-utc-v0.zarr/zarr.json'
```{code-block} python
:substitutions:
ClientResponseError: 429, message='Too Many Requests', url='https://|datastore_internal_host|/era5/era5-single-levels-atmosphere-daily-utc-v0.zarr/zarr.json'
```
````
26 changes: 26 additions & 0 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading