Merge branch '0.8.0' into 'master'

From 0.8.0 into master

See merge request extracteo/eoreader!16

CHANGES.md

@@ -2,7 +2,47 @@
 
 ## X.Y.Z (YYYY-MM-DD)
 
-## 0.Y.Z (YYYY-MM-DD)
+## 0.8.0 (2021-10-25)
+
+- **BREAKING CHANGE: `crs`, `footprint`, `extent`, `wgs84_extent` are now properties !**
+- **BREAKING CHANGE: Removing raw `gdaldem` CLI from EOReader (the `HILLSHADE` and `SLOPE` bands are now slightly different !)** [#10](https://github.com/sertit/eoreader/issues/10)
+- **BREAKING CHANGE: `HILLSHADE` is given in `float32` instead of `uint8`**
+- **BREAKING CHANGE: `SLOPE` is given in degrees instead of percents**
+- **ENH: Adding the support of the PAZ SAR sensor**
+- **ENH: Adding the support of the Sentinel-2 processed with the [processing baseline 4.0](https://sentinels.copernicus.eu/web/sentinel/-/copernicus-sentinel-2-major-products-upgrade-upcoming)** [#11](https://github.com/sertit/eoreader/issues/11)
+- **ENH: Removing SNAP from Sentinel-3 pre-process -> Freeing optical data from SNAP dependency !** [#12](https://github.com/sertit/eoreader/issues/12)
+- **ENH: Enabling the use of other S3-SLSTR suffixes than `an` (stripe A at nadir position)**
+- **ENH: Thermal bands of Sentinel-3 SLSTR can now be used**
+- **ENH: All bands of Sentinel-3 SLSTR/OLCI can now be used (`S7`, `F1`, `F2` for SLSTR, `Oaxx` for OLCI)** [#14](https://github.com/sertit/eoreader/issues/14)
+- **ENH: `YELLOW` band is mapped to `Oa07` band of Sentinel-3 OLCI**
+- **ENH: Zipped Sentinel-3 products can now be processed**
+- **ENH: Allow the use of `kwargs` in `load`, mainly for `rasters.read` (and allowing ie. radiance adjustment in S3-SLSTR)**
+- OPTIM: `crs`, `footprint`, `extent`, `default_transform`, `wgs84_extent` are cached (using `@cached_property`) [#13](https://github.com/sertit/eoreader/issues/13)
+- OPTIM: `get_mean_sun_angles` and `default_transform` are now cached (using `@cache`) [#13](https://github.com/sertit/eoreader/issues/13)
+- OPTIM: `get_datetime`: Look for the date only if `datetime` attribute is None [#13](https://github.com/sertit/eoreader/issues/13)
+- OPTIM: Better management of `fspath` for cloud-stored products (download the files only once)
+- OPTIM: Stop downloading/extracting files if not necessary
+- FIX: Bands are correctly ordered in stacks
+- FIX: Only load a band once, even if asked several time in the bands
+- FIX: Use band size for cleaning optical pixel (instead of user resolution/size)
+- FIX: Always take the absolute value of the resolution when converting it to strings (for filenames)
+- FIX: Take the default resolution if nothing is given when converting it to strings (for filenames)
+- FIX: Always use `utils.read/write` instead of `rasters.read/write` (for Dask management)
+- FIX: Fixing a bug in `utils.write`
+- FIX: Add .xml files from `eoreader/data` in the MANIFEST.in
+- FIX: Add forgotten `@abstractmethod` where needed
+- FIX: Better management of `_tmp_process`
+- FIX: Fixing minor bug when trying to read metadata with a POSIX path
+- FIX: Fixing the `**kwargs` omission in `utils.read`
+- FIX: Better management of `_temp_process` directory
+- FIX: Landsats and TSX: Can use other filenames now
+- DEPR: `FAR_NIR` band is removed
+- REQ: Using `h5netcdf` instead of `netCDF4`
+- DOC: Add a Context paragraph in the README
+- DOC: Add a Conda x SNAP question in the FAQ
+- DOC: Creation of a Sentinel-3 notebook
+- DOC: Updates of notebooks
+- DOC: Numerous updates
 
 ## 0.7.1 (2021-09-29)
 

CI/SCRIPTS/test_satellites.py

@@ -15,10 +15,11 @@
     CI_EOREADER_BAND_FOLDER,
     DEM_PATH,
     S3_DB_URL_ROOT,
-    S3_DEF_RES,
     SAR_DEF_RES,
     TEST_USING_S3_DB,
 )
+from eoreader.keywords import SLSTR_RAD_ADJUST
+from eoreader.products.optical.s3_slstr_product import SlstrRadAdjust
 from eoreader.products.product import Product, SensorType
 from eoreader.reader import CheckMethod
 from eoreader.utils import EOREADER_NAME
@@ -86,36 +87,35 @@ def remove_dem_files(prod):
         files.remove(to_d)
 
 
-def test_invalid():
-    wrong_path = "dzfdzef"
-    assert READER.open(wrong_path) is None
-    assert not READER.valid_name(wrong_path, "S2")
-
-
-def _test_core_optical(pattern: str, dem_path=None, debug=False):
+def _test_core_optical(pattern: str, dem_path=None, debug=False, **kwargs):
     """
     Core function testing optical data
     Args:
         pattern (str): Pattern of the satellite
         debug (bool): Debug option
     """
     possible_bands = [RED, SWIR_2, HILLSHADE, CLOUDS]
-    _test_core(pattern, opt_path(), possible_bands, dem_path, debug)
+    _test_core(pattern, opt_path(), possible_bands, dem_path, debug, **kwargs)
 
 
-def _test_core_sar(pattern: str, dem_path=None, debug=False):
+def _test_core_sar(pattern: str, dem_path=None, debug=False, **kwargs):
     """
     Core function testing SAR data
     Args:
         pattern (str): Pattern of the satellite
         debug (bool): Debug option
     """
     possible_bands = [VV, VV_DSPK, HH, HH_DSPK, SLOPE, HILLSHADE]
-    _test_core(pattern, sar_path(), possible_bands, dem_path, debug)
+    _test_core(pattern, sar_path(), possible_bands, dem_path, debug, **kwargs)
 
 
 def _test_core(
-    pattern: str, prod_dir: str, possible_bands: list, dem_path=None, debug=False
+    pattern: str,
+    prod_dir: str,
+    possible_bands: list,
+    dem_path=None,
+    debug=False,
+    **kwargs,
 ):
     """
     Core function testing all data
@@ -156,8 +156,12 @@ def _test_core(
             )
 
             # Open product and set output
+            LOGGER.info("Checking opening solutions")
+            LOGGER.info("MTD")
             prod: Product = READER.open(path, method=CheckMethod.MTD, remove_tmp=False)
+            LOGGER.info("NAME")
             prod_name = READER.open(path, method=CheckMethod.NAME)
+            LOGGER.info("BOTH")
             prod_both = READER.open(path, method=CheckMethod.BOTH)
             assert prod is not None
             assert prod == prod_name
@@ -171,17 +175,6 @@ def _test_core(
                     # )
                     prod.output = tmp_dir
 
-                    # Env var
-                    # if (
-                    #     prod.platform == Platform.S3
-                    #     or prod.sensor_type == SensorType.SAR
-                    # ):
-                    #     os.environ[CI_EOREADER_BAND_FOLDER] = str(
-                    #         get_ci_data_dir().joinpath(prod.condensed_name)
-                    #     )
-                    # else:
-                    #     if CI_EOREADER_BAND_FOLDER in os.environ:
-                    #         os.environ.pop(CI_EOREADER_BAND_FOLDER)
                     os.environ[CI_EOREADER_BAND_FOLDER] = str(
                         get_ci_data_dir().joinpath(prod.condensed_name)
                     )
@@ -192,11 +185,10 @@ def _test_core(
                         os.environ[SAR_DEF_RES] = str(res)
                     else:
                         res = prod.resolution * 50
-                        os.environ[S3_DEF_RES] = str(res)
 
                     # Extent
                     LOGGER.info("Checking extent")
-                    extent = prod.extent()
+                    extent = prod.extent
                     assert isinstance(extent, gpd.GeoDataFrame)
                     extent_path = get_ci_data_dir().joinpath(
                         prod.condensed_name, f"{prod.condensed_name}_extent.geojson"
@@ -217,7 +209,7 @@ def _test_core(
 
                     # Footprint
                     LOGGER.info("Checking footprint")
-                    footprint = prod.footprint()
+                    footprint = prod.footprint
                     assert isinstance(footprint, gpd.GeoDataFrame)
                     footprint_path = get_ci_data_dir().joinpath(
                         prod.condensed_name, f"{prod.condensed_name}_footprint.geojson"
@@ -264,7 +256,7 @@ def _test_core(
                         tmp_dir, f"{prod.condensed_name}_stack.tif"
                     )
                     stack = prod.stack(
-                        stack_bands, resolution=res, stack_path=curr_path
+                        stack_bands, resolution=res, stack_path=curr_path, **kwargs
                     )
                     assert stack.dtype == np.float32
 
@@ -289,16 +281,17 @@ def _test_core(
                         ci_band = curr_path_band
 
                     band_arr = prod.load(
-                        first_band, size=(stack.rio.width, stack.rio.height)
+                        first_band, size=(stack.rio.width, stack.rio.height), **kwargs
                     )[first_band]
                     rasters.write(band_arr, curr_path_band)
                     assert_raster_almost_equal(curr_path_band, ci_band, decimal=4)
 
                 # CRS
                 LOGGER.info("Checking CRS")
-                assert prod.crs().is_projected
+                assert prod.crs.is_projected
 
                 # MTD
+                LOGGER.info("Checking Mtd")
                 mtd_xml, nmsp = prod.read_mtd()
                 assert isinstance(mtd_xml, etree._Element)
                 assert isinstance(nmsp, dict)
@@ -337,7 +330,7 @@ def test_s3_olci():
 def test_s3_slstr():
     """Function testing the correct functioning of the optical satellites"""
     # Init logger
-    _test_core_optical("*S3*_SL_1_*")
+    _test_core_optical("*S3*_SL_1_*", **{SLSTR_RAD_ADJUST: SlstrRadAdjust.SNAP})
 
 
 @s3_env
@@ -468,6 +461,7 @@ def test_csk():
     _test_core_sar("*csk_*")
 
 
+# Assume that tests TSX, TDX and PAZ sensors
 @s3_env
 def test_tsx():
     """Function testing the correct functioning of the optical satellites"""
@@ -491,3 +485,9 @@ def test_rcm():
 # TODO:
 # check non existing bands
 # check cloud results
+
+
+def test_invalid():
+    wrong_path = "dzfdzef"
+    assert READER.open(wrong_path) is None
+    assert not READER.valid_name(wrong_path, "S2")

CI/SCRIPTS_SNAP/test_snap.py

@@ -9,7 +9,7 @@
 
 from CI.SCRIPTS.scripts_utils import CI_EOREADER_S3, opt_path, s3_env, sar_path
 from eoreader.bands.alias import *
-from eoreader.env_vars import S3_DEF_RES, SAR_DEF_RES
+from eoreader.env_vars import SAR_DEF_RES
 from eoreader.products.product import Product, SensorType
 from eoreader.reader import Reader
 from eoreader.utils import EOREADER_NAME
@@ -99,7 +99,6 @@ def _test_core(pattern: str, prod_dir: str, possible_bands: list, debug=False):
                         os.environ[SAR_DEF_RES] = str(res)
                     else:
                         res = prod.resolution * 50
-                        os.environ[S3_DEF_RES] = str(res)
 
                     # Load data (just check if SNAP runs)
                     prod.load(stack_bands, resolution=res)

MANIFEST.in

@@ -1,3 +1,3 @@
-include README.md LICENSE NOTICE requirements.txt
+include README.md LICENSE NOTICE requirements.txt eoreader/data/*.xml
 recursive-include docs *
 prune CI

README.md

@@ -14,7 +14,7 @@ clouds, DEM and index in a sensor-agnostic way.
 
 |**Optical sensors** | **SAR sensors**|
 | --- | ---|
-|Sentinel-2 and Sentinel-2 Theia<br>Sentinel-3 OLCI and Sentinel-3 SLSTR<br>Landsat 1 to 8 (MSS, TM, ETM and OLCI)<br>PlanetScope<br>Pleiades<br>SPOT 6-7<br>WorldView-2 to 4, GeoEye-1 (and other Maxar sensors)| Sentinel-1<br>COSMO-Skymed<br>TerraSAR-X & TanDEM-X<br>RADARSAT-2<br>RADARSAT-Constellation|
+|Sentinel-2 and Sentinel-2 Theia<br>Sentinel-3 OLCI and Sentinel-3 SLSTR<br>Landsat 1 to 8 (MSS, TM, ETM and OLCI)<br>PlanetScope<br>Pleiades<br>SPOT 6-7<br>WorldView-2 to 4, GeoEye-1 (and other Maxar sensors)| Sentinel-1<br>COSMO-Skymed<br>TerraSAR-X, TanDEM-X & PAZ<br>RADARSAT-2<br>RADARSAT-Constellation|
 
 It also implements additional **sensor-agnostic** features:
 
@@ -88,7 +88,7 @@ For SAR data:
 >>> mtd, namespace = s1_prod.read_mtd()
 ```
 
-Sentinel-3 and SAR products need [`SNAP gpt`](https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503590/Creating+a+GPF+Graph) to be geocoded.
+SAR products need [`SNAP gpt`](https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503590/Creating+a+GPF+Graph) to be orthorectified and calibrated.
 Ensure that you have the folder containing your `gpt` executable in your `PATH`.
 
 ## Documentation
@@ -101,6 +101,7 @@ Available notebooks provided as examples:
 - [Basic tutorial](https://eoreader.readthedocs.io/en/latest/notebooks/base.html)
 - [SAR data](https://eoreader.readthedocs.io/en/latest/notebooks/SAR.html)
 - [VHR data](https://eoreader.readthedocs.io/en/latest/notebooks/VHR.html)
+- [Sentinel-3 data](https://eoreader.readthedocs.io/en/latest/notebooks/sentinel-3.html)
 - [Water detection](https://eoreader.readthedocs.io/en/latest/notebooks/water_detection.html)
 - [S3 Compatible Storage](https://eoreader.readthedocs.io/en/latest/notebooks/s3_compatible_storage.html)
 - [Dask](https://eoreader.readthedocs.io/en/latest/notebooks/dask.html)
@@ -129,6 +130,24 @@ You can install EOReader via conda:
 
 `conda install -c conda-forge eoreader`
 
+## Context
+
+SERTIT is part of the [Copernicus Emergency Management Service](https://emergency.copernicus.eu/)
+rapid mapping and risk and recovery teams.
+
+In these activations, we need to deliver information (such as flood or fire delineations, landslides mapping, etc.)
+based on various sensors (more than 10 optical and 5 SAR). As every minute counts in production,
+it seemed crucial to harmonize the ground on which are built our production tools, in order to make them
+as sensor-agnostic as possible.
+
+Thus, thanks to **EOReader**, these tools are made independent to the sensor:
+- the algorithm (and its developer) can focus on its core tasks (such as extraction)
+without taking into account the sensor characteristics
+(how to load a band, which band correspond to which band number, which band to use for this index...)
+- the addition of a new sensor is done effortlessly (if existing in **EOReader**) and without any modification of the algorithm
+- the maintenance is simplified and the code is way more readable (no more ifs regarding the sensor type!)
+- the testing is also simplified as the sensor-related parts are tested in this library
+
 ## License
 
 **EOReader** is licensed under Apache License v2.0. See LICENSE file for details.