Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] add Rasters.jl benchmarks #18

Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
9f29bac
Add initial files for Rasters.jl
asinghvi17 Apr 12, 2024
97c4a36
Update gitignore
asinghvi17 Apr 12, 2024
d888a6a
get Rasters.jl benchmarks to working order
asinghvi17 Apr 12, 2024
258b1c7
get Rasters.jl kinda working
asinghvi17 Apr 12, 2024
c00040b
move from Rasters-jl to Rasters_jl
asinghvi17 Apr 12, 2024
c990831
Use Rasters.extract instead of indexing
asinghvi17 Sep 23, 2024
963b54e
Add Pixi and Julia projects to lock versions + make this reproducible
asinghvi17 Sep 24, 2024
e93077e
pixi still not working
asinghvi17 Sep 24, 2024
9c922b0
brief updates to files
asinghvi17 Sep 24, 2024
bbb3437
add Julia local project.toml
asinghvi17 Sep 24, 2024
cfde288
tune bencmarks
rafaqz Sep 24, 2024
ea7c536
Update pixi files to have separate envs per package
asinghvi17 Sep 24, 2024
5f5a2bc
Make benchmarks work, add plotting
asinghvi17 Sep 24, 2024
152a328
Plotting should never load unless not in benchmarking
asinghvi17 Sep 24, 2024
a8abb00
Make `load` eager
asinghvi17 Sep 24, 2024
f320a93
Set `crop` to the minimum value of the rest of the array
asinghvi17 Sep 24, 2024
8b590bf
Clean up, write tif not nc
asinghvi17 Sep 24, 2024
0df68ad
allow for potential rasterize test
asinghvi17 Sep 24, 2024
11b383f
add Rasters entry to readme
rafaqz Sep 24, 2024
a4c5231
clean up zonal
rafaqz Sep 24, 2024
5a95463
comments
rafaqz Sep 24, 2024
00806cd
bugfix nvdi
rafaqz Sep 24, 2024
4a30726
use `aggregate` (native Julia) instead of `resample` (GDAL)
asinghvi17 Feb 4, 2025
26816b1
use LZW compression in `write`
asinghvi17 Feb 4, 2025
a8bce32
Switch from `sum` to `mean` in downsample example
asinghvi17 Feb 4, 2025
ce9de4c
Go back to a bash script, remove pixi benchmarking
asinghvi17 Feb 12, 2025
8649823
Minor bugfixes
asinghvi17 Feb 12, 2025
2ae6cef
rename rasters_jl to rasters
asinghvi17 Feb 12, 2025
0825ba2
rename folder
asinghvi17 Feb 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# GitHub syntax highlighting
pixi.lock linguist-language=YAML linguist-generated=true
11 changes: 11 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,14 @@
# R files
.Rproj.user
.Rhistory
.Rdata
# pixi environments
.pixi
*.egg-info
# Julia environment
rasters_jl/Manifest.toml
# data
data/LC08_L1TP_190024_20200418_20200822_02_T1.zip
data/LC08_L1TP_190024_20200418_20200822_02_T1/*
data/Mammals_Terrestrial/*
data/stack.nc
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ You may also be interested in the [vector processing benchmarks](https://github.
- [raster](https://github.com/rspatial/raster)
- [exactextractr](https://github.com/isciences/exactextractr)

**Juliia**:
- [Rasters.jl](https://github.com/rafaqz/Rasters.jl)

## Reproduction
1. Download raster data (851 MB) from [Google Drive](https://drive.google.com/uc?id=1lzglfQJqlQh9OWT_-czc5L0hQ1AhoR8M&export=download) or [Earth Explorer](https://earthexplorer.usgs.gov/) (original source, registration required) and then unzip to `data/`.
2. Run all benchmarks using batch script (`run_benchmarks.sh`) or single benchmarks files.
Expand All @@ -40,6 +43,10 @@ Rscript stars/crop.R
python3 rasterio/crop.py
```

```
julia rasters_jl/crop.jl
```

## Dataset
Landsat 8 satellite scene (10 bands, 30 m resolution, 7771 x 7871 pixels) was used for tests.

Expand Down
8,372 changes: 8,372 additions & 0 deletions pixi.lock

Large diffs are not rendered by default.

74 changes: 74 additions & 0 deletions pixi.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
[project]
name = "raster-benchmark"
authors = ["Anshul Singhvi <[email protected]>"]
channels = ["conda-forge"]
description = "Benchmarking raster processing libraries"
platforms = ["osx-64", "linux-64"]
version = "0.1.0"

[tasks.benchmarks]

cmd = "echo 'Ran benchmarks'"
depends-on = ["r-raster.benchmark", "r-stars.benchmark", "r-terra.benchmark", "r-exactextractr.benchmark", "py-rasterio.benchmark", "py-rasterstats.benchmark", "py-rioxarray.benchmark", "julia-rastersjl.benchmark"]


[environments]
r-raster = ["r", "r-sf", "r-raster"]
r-stars = ["r", "r-sf", "r-stars"]
r-terra = ["r", "r-sf", "r-terra"]
r-exactextractr = ["r", "r-exactextractr", "r-terra"]

py-rasterio = ["python", "py-rasterio"]
py-rasterstats = ["python", "py-rasterstats"]
py-rioxarray = ["python", "py-rioxarray"]

julia-rasters = ["julia-rasters"]


[feature.r.dependencies]
r = ">=4.3,<5"
r-tibble = ">=3.2.1,<4"

[feature.r-sf.dependencies]
r = ">=4.3.3,<5"
r-sf = ">=1.0.15,<2"
r-s2 = ">=1.1.4,<2"

[feature.python.dependencies]
python = ">=3.11.6,<4"
xarray = ">=2024.9.0,<2026"
fiona = ">=1.9.4,<2"
pandas = ">=2.2.3,<3"
geopandas = ">=1,<2"

[feature.r-raster]
dependencies = {r-raster = ">=3.6_26,<4"}
# tasks = {benchmark = "cd raster; for path in \"${i}\"/*; do; echo \"$path\"; Rscript \"$path\"; done"}

[feature.r-stars]
dependencies = {r-stars = ">=0.6_6,<0.7", r-sf = ">=1.0_15,<2", r-s2 = ">=1.1.4,<2"}
# tasks = {benchmark = "cd stars; for path in \"${i}\"/*; do; echo \"$path\"; Rscript \"$path\"; done"}

[feature.r-terra]
dependencies = {r-terra = ">=1.8.5,<2"}
# tasks = {benchmark = "cd terra; for path in \"${i}\"/*; do; echo \"$path\"; Rscript \"$path\"; done"}

[feature.r-exactextractr]
dependencies = {r-exactextractr = ">=0.10,<0.11"}
# tasks = {benchmark = "cd exactextractr; for path in \"${i}\"/*; do; echo \"$path\"; Rscript \"$path\"; done"}

[feature.py-rasterio]
dependencies = {rasterio = ">=1.3.7,<2"}
# tasks = {benchmark = "cd rasterio; for path in \"${i}\"/*; do; echo \"$path\"; python3 \"$path\"; done"}

[feature.py-rasterstats]
dependencies = {rasterstats = ">=0.20.0,<0.21"}
# tasks = {benchmark = "cd rasterstats; for path in \"${i}\"/*; do; echo \"$path\"; python3 \"$path\"; done"}

[feature.py-rioxarray]
dependencies = {rioxarray = ">=0.18.0,<0.19"}
# tasks = {benchmark = "cd rioxarray; for path in \"${i}\"/*; do; echo \"$path\"; python3 \"$path\"; done"}

[feature.julia-rasters]
dependencies = {julia = ">=1.9.3,<2"}
# tasks = {benchmark = "cd rasters_jl; for path in \"${i}\"/*; do; echo \"$path\"; BENCHMARKING=true julia \"$path\"; done"}
Loading