Skip to content

Commit e862aa0

Browse files
kthyngkthyng
authored
Merge pull request #56 from kthyng/update_intake
Update intake to v2
2 parents 75ff058 + 7ba8a34 commit e862aa0

29 files changed

+714
-974
lines changed

‎.github/workflows/test.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
fail-fast: false
1515
matrix:
1616
os: ["macos-latest", "ubuntu-latest", "windows-latest"]
17-
python-version: ["3.8", "3.9", "3.10"]
17+
python-version: ["3.9", "3.10", "3.11"]
1818
steps:
1919
- name: Checkout source
2020
uses: actions/checkout@v2

‎.pre-commit-config.yaml

Lines changed: 11 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -28,18 +28,12 @@ repos:
2828
exclude: docs/conf.py
2929
args: [--max-line-length=105 ]
3030

31-
- repo: https://github.com/pre-commit/mirrors-isort
32-
rev: v5.10.1
31+
- repo: https://github.com/pycqa/isort
32+
rev: 5.12.0
3333
hooks:
34-
- id: isort
35-
additional_dependencies: [toml]
36-
exclude: ^(docs|setup.py)
37-
args: [--project=gcm_filters, --multi-line=3, --lines-after-imports=2, --lines-between-types=1, --trailing-comma, --force-grid-wrap=0, --use-parentheses, --line-width=88]
38-
39-
- repo: https://github.com/asottile/seed-isort-config
40-
rev: v2.2.0
41-
hooks:
42-
- id: seed-isort-config
34+
- id: isort
35+
name: isort (python)
36+
args: ["--profile", "black", "--filter-files", "--lines-after-imports=2", "--project=gcm_filters", "--multi-line=3", "--lines-between-types=1", "--trailing-comma", "--force-grid-wrap=0", "--use-parentheses", "--line-width=88"]
4337

4438
- repo: https://github.com/psf/black
4539
rev: 22.10.0
@@ -56,9 +50,9 @@ repos:
5650
exclude: docs/source/conf.py
5751
args: [--ignore-missing-imports]
5852

59-
# - repo: https://github.com/codespell-project/codespell
60-
# rev: v1.16.0
61-
# hooks:
62-
# - id: codespell
63-
# args:
64-
# - --quiet-level=2
53+
- repo: https://github.com/codespell-project/codespell
54+
rev: v2.1.0
55+
hooks:
56+
- id: codespell
57+
args:
58+
- --quiet-level=2

‎.readthedocs.yaml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,10 @@ build:
1212
# uncomment to build from this exact version of package
1313
# the downside is the version listed in the docs will be a dev version
1414
# if uncommenting this, comment out installing pypi version of package in docs/env file
15-
# python:
16-
# install:
17-
# - method: pip
18-
# path: ./
15+
python:
16+
install:
17+
- method: pip
18+
path: ./
1919

2020
conda:
2121
environment: docs/environment.yml

‎MANIFEST.in

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎README.md

Lines changed: 44 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -24,15 +24,13 @@ For changes prior to 2022-10-19, all contributions are Copyright James Munroe, s
2424

2525

2626

27-
Intake is a lightweight set of tools for loading and sharing data in data
28-
science projects. Intake ERDDAP provides a set of integrations for ERDDAP.
27+
Intake is a lightweight set of tools for loading and sharing data in data science projects. Intake ERDDAP provides a set of integrations for ERDDAP.
2928

30-
- Quickly identify all datasets from an ERDDAP service in a geographic region,
31-
or containing certain variables.
29+
- Quickly identify all datasets from an ERDDAP service in a geographic region, or containing certain variables.
3230
- Produce a pandas DataFrame for a given dataset or query.
3331
- Get an xarray Dataset for the Gridded datasets.
3432

35-
The Key features are:
33+
The key features are:
3634

3735
- Pandas DataFrames for any TableDAP dataset.
3836
- xarray Datasets for any GridDAP datasets.
@@ -59,7 +57,7 @@ project is available on PyPI, so it can be installed using `pip`
5957
The following are prerequisites for a developer environment for this project:
6058

6159
- [conda](https://docs.conda.io/en/latest/miniconda.html)
62-
- (optional but highly recommended) [mamba](https://mamba.readthedocs.io/en/latest/) Hint: `conda install -c conda-forge mamba`
60+
- (optional but highly recommended) [mamba](https://mamba.readthedocs.io/en/latest/). Hint: `conda install -c conda-forge mamba`
6361

6462
Note: if `mamba` isn't installed, replace all instances of `mamba` in the following instructions with `conda`.
6563

@@ -83,126 +81,74 @@ Note: if `mamba` isn't installed, replace all instances of `mamba` in the follow
8381
pip install -e .
8482
```
8583

84+
Note that you need to install with `pip install .` once to get the `entry_points` correct too.
8685

8786
## Examples
8887

89-
To create an intake catalog for all of the ERDDAP's TableDAP offerings use:
88+
To create an `intake` catalog for all of the ERDDAP's TableDAP offerings use:
9089

9190
```python
92-
import intake
93-
catalog = intake.open_erddap_cat(
91+
import intake_erddap
92+
catalog = intake_erddap.ERDDAPCatalogReader(
9493
server="https://erddap.sensors.ioos.us/erddap"
95-
)
94+
).read()
9695
```
9796

9897

99-
The catalog objects behave like a dictionary with the keys representing the
100-
dataset's unique identifier within ERDDAP, and the values being the
101-
`TableDAPSource` objects. To access a source object:
98+
The catalog objects behave like a dictionary with the keys representing the dataset's unique identifier within ERDDAP, and the values being the `TableDAPReader` objects. To access a Reader object (for a single dataset, in this case for dataset_id "aoos_204"):
10299

103100
```python
104-
source = catalog["datasetid"]
101+
dataset = catalog["aoos_204"]
105102
```
106103

107-
From the source object, a pandas DataFrame can be retrieved:
104+
From the reader object, a pandas DataFrame can be retrieved:
108105

109106
```python
110-
df = source.read()
107+
df = dataset.read()
108+
```
109+
110+
Find other dataset_ids available with
111+
112+
```python
113+
list(catalog)
111114
```
112115

113116
Consider a case where you need to find all wind data near Florida:
114117

115118
```python
116-
import intake
119+
import intake_erddap
117120
from datetime import datetime
118121
bbox = (-87.84, 24.05, -77.11, 31.27)
119-
catalog = intake.open_erddap_cat(
122+
catalog = intake_erddap.ERDDAPCatalogReader(
120123
server="https://erddap.sensors.ioos.us/erddap",
121124
bbox=bbox,
125+
intersection="union",
122126
start_time=datetime(2022, 1, 1),
123127
end_time=datetime(2023, 1, 1),
124128
standard_names=["wind_speed", "wind_from_direction"],
125-
)
129+
variables=["wind_speed", "wind_from_direction"],
130+
).read()
126131

127-
df = next(catalog.values()).read()
132+
dataset_id = list(catalog)[0]
133+
print(dataset_id)
134+
df = catalog[dataset_id].read()
128135
```
129136

137+
Using the `standard_names` input with `intersection="union"` searches for datasets that have both "wind_speed" and "wind_from_direction". Using the `variables` input subsequently narrows the dataset to only those columns, plus "time", "latitude", "longitude", and "z".
130138

131-
<table class="align-default">
132-
<thead>
133-
<tr style="text-align: right;">
134-
<th></th>
135-
<th>time (UTC)</th>
136-
<th>wind_speed (m.s-1)</th>
137-
<th>wind_from_direction (degrees)</th>
138-
</tr>
139-
</thead>
140-
<tbody>
141-
<tr>
142-
<th>0</th>
143-
<td>2022-12-14T19:40:00Z</td>
144-
<td>7.0</td>
145-
<td>140.0</td>
146-
</tr>
147-
<tr>
148-
<th>1</th>
149-
<td>2022-12-14T19:20:00Z</td>
150-
<td>7.0</td>
151-
<td>120.0</td>
152-
</tr>
153-
<tr>
154-
<th>2</th>
155-
<td>2022-12-14T19:10:00Z</td>
156-
<td>NaN</td>
157-
<td>NaN</td>
158-
</tr>
159-
<tr>
160-
<th>3</th>
161-
<td>2022-12-14T19:00:00Z</td>
162-
<td>9.0</td>
163-
<td>130.0</td>
164-
</tr>
165-
<tr>
166-
<th>4</th>
167-
<td>2022-12-14T18:50:00Z</td>
168-
<td>9.0</td>
169-
<td>130.0</td>
170-
</tr>
171-
<tr>
172-
<th>...</th>
173-
<td>...</td>
174-
<td>...</td>
175-
<td>...</td>
176-
</tr>
177-
<tr>
178-
<th>48296</th>
179-
<td>2022-01-01T00:40:00Z</td>
180-
<td>4.0</td>
181-
<td>120.0</td>
182-
</tr>
183-
<tr>
184-
<th>48297</th>
185-
<td>2022-01-01T00:30:00Z</td>
186-
<td>3.0</td>
187-
<td>130.0</td>
188-
</tr>
189-
<tr>
190-
<th>48298</th>
191-
<td>2022-01-01T00:20:00Z</td>
192-
<td>4.0</td>
193-
<td>120.0</td>
194-
</tr>
195-
<tr>
196-
<th>48299</th>
197-
<td>2022-01-01T00:10:00Z</td>
198-
<td>4.0</td>
199-
<td>130.0</td>
200-
</tr>
201-
<tr>
202-
<th>48300</th>
203-
<td>2022-01-01T00:00:00Z</td>
204-
<td>4.0</td>
205-
<td>130.0</td>
206-
</tr>
207-
</tbody>
208-
</table>
139+
```python
140+
time (UTC) latitude (degrees_north) ... wind_speed (m.s-1) wind_from_direction (degrees)
141+
0 2022-01-01T00:00:00Z 28.508 ... 3.6 126.0
142+
1 2022-01-01T00:10:00Z 28.508 ... 3.8 126.0
143+
2 2022-01-01T00:20:00Z 28.508 ... 3.6 124.0
144+
3 2022-01-01T00:30:00Z 28.508 ... 3.4 125.0
145+
4 2022-01-01T00:40:00Z 28.508 ... 3.5 124.0
146+
... ... ... ... ... ...
147+
52524 2022-12-31T23:20:00Z 28.508 ... 5.9 176.0
148+
52525 2022-12-31T23:30:00Z 28.508 ... 6.8 177.0
149+
52526 2022-12-31T23:40:00Z 28.508 ... 7.2 175.0
150+
52527 2022-12-31T23:50:00Z 28.508 ... 7.4 169.0
151+
52528 2023-01-01T00:00:00Z 28.508 ... 8.1 171.0
152+
153+
[52529 rows x 6 columns]
154+
```

‎ci/environment-py3.10.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,14 @@ channels:
33
- conda-forge
44
dependencies:
55
- python=3.10
6+
- appdirs
7+
- fsspec
68
- numpy
79
- dask
810
- pandas
911
- erddapy
1012
- panel
1113
- intake
12-
- intake-xarray>=0.6.1
1314
- pytest
1415
- pytest-cov
1516
- isort
@@ -19,6 +20,8 @@ dependencies:
1920
- mypy
2021
- codecov
2122
- coverage[toml]
23+
- xarray
2224
- pip
2325
- pip:
26+
- git+https://github.com/intake/intake
2427
- cf-pandas

‎ci/environment-py3.8.yml renamed to ‎ci/environment-py3.11.yml

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,15 @@ name: test-env
22
channels:
33
- conda-forge
44
dependencies:
5-
- python=3.8
5+
- python=3.11
6+
- appdirs
7+
- fsspec
68
- numpy
79
- dask
810
- pandas
911
- erddapy
1012
- panel
11-
- intake
12-
- intake-xarray>=0.6.1
13+
# - intake
1314
- pytest
1415
- pytest-cov
1516
- isort
@@ -19,6 +20,8 @@ dependencies:
1920
- mypy
2021
- codecov
2122
- coverage[toml]
23+
- xarray
2224
- pip
2325
- pip:
26+
- git+https://github.com/intake/intake
2427
- cf-pandas

‎ci/environment-py3.9.yml

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,14 @@ channels:
33
- conda-forge
44
dependencies:
55
- python=3.9
6+
- appdirs
67
- numpy
78
- dask
89
- pandas
910
- erddapy
11+
- fsspec
1012
- panel
11-
- intake
12-
- intake-xarray>=0.6.1
13+
# - intake
1314
- pytest
1415
- pytest-cov
1516
- isort
@@ -19,6 +20,8 @@ dependencies:
1920
- mypy
2021
- codecov
2122
- coverage[toml]
23+
- xarray
2224
- pip
2325
- pip:
26+
- git+https://github.com/intake/intake
2427
- cf-pandas

‎docs/api.rst

Lines changed: 7 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -2,27 +2,12 @@
22
``intake-erddap`` Python API
33
=============================
44

5-
.. toctree::
6-
:maxdepth: 2
7-
:caption: Documentation
5+
.. currentmodule:: intake_erddap
86

7+
.. autosummary::
8+
:toctree: generated/
9+
:recursive:
910

10-
``intake-erddap`` catalog
11-
-------------------------
12-
13-
14-
.. autoclass:: intake_erddap.erddap_cat.ERDDAPCatalog
15-
:members: get_client, get_search_urls
16-
17-
``intake-erddap`` source
18-
------------------------
19-
20-
21-
.. autoclass:: intake_erddap.erddap.ERDDAPSource
22-
:members: get_client
23-
24-
.. autoclass:: intake_erddap.erddap.TableDAPSource
25-
:members: read, read_partition, read_chunked
26-
27-
.. autoclass:: intake_erddap.erddap.GridDAPSource
28-
:members: read_partition, read_chunked, to_dask, close
11+
ERDDAPCatalogReader
12+
TableDAPReader
13+
GridDAPReader

0 commit comments

Comments
 (0)