Merge branch 'main' into get_template

Author: matthiasschaub

Diff for: .github/PULL_REQUEST_TEMPLATE.md

@@ -11,9 +11,9 @@ Closes #
 
 ### Checklist
 - [ ] I have updated my branch to `main` (e.g. through `git rebase main`)
-- [ ] My code follows the [style guide](https://github.com/GIScience/ohsome-quality-analyst/blob/main/CONTRIBUTING.md#style-guide) and was checked with [pre-commit](https://github.com/GIScience/ohsome-quality-analyst/blob/main/CONTRIBUTING.md#pre-commit) before committing
+- [ ] My code follows the [style guide](https://github.com/GIScience/ohsome-quality-api/blob/main/CONTRIBUTING.md#style-guide) and was checked with [pre-commit](https://github.com/GIScience/ohsome-quality-api/blob/main/CONTRIBUTING.md#pre-commit) before committing
 - [ ] I have commented my code
-- [ ] I have added sufficient unit and integration [tests](https://github.com/GIScience/ohsome-quality-analyst/blob/main/docs/development_setup.md#tests)
-- [ ] I have updated the [CHANGELOG.md](https://github.com/GIScience/ohsome-quality-analyst/blob/main/CHANGELOG.md)
+- [ ] I have added sufficient unit and integration [tests](https://github.com/GIScience/ohsome-quality-api/blob/main/docs/development_setup.md#tests)
+- [ ] I have updated the [CHANGELOG.md](https://github.com/GIScience/ohsome-quality-api/blob/main/CHANGELOG.md)
 
 Please check all finished tasks. If some tasks do not apply to your PR, please cross their text out (by using `~...~`) and remove their checkboxes.

Diff for: .gitignore

@@ -108,11 +108,14 @@ celerybeat.pid
 .env
 .venv
 env/
-venv/
+venv*/
 ENV/
 env.bak/
 venv.bak/
 
+# pyenv config
+.python-version
+
 # Spyder project settings
 .spyderproject
 .spyproject
@@ -143,8 +146,10 @@ cython_debug/
 # Ruff
 .ruff_cache/
 
-# OQT
+# oqapi
 /data/*
 /config/*
 !/config/sample.config.yaml
 !/config/logging.yaml
+/tests/*/fixtures/approved/*.received.txt
+/regression-tests/report/

Diff for: .pre-commit-config.yaml

@@ -1,14 +1,13 @@
 repos:
-  - repo: https://github.com/psf/black
-    rev: 23.3.0
-    hooks:
-      - id: black
-        args: ["--check", "--diff"]
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: v0.0.275
+    rev: v0.4.7
     hooks:
+      # Run the linter.
       - id: ruff
+      # Run the formatter.
+      - id: ruff-format
+        args: ["--check", "--diff"]
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.4.0
     hooks:

Diff for: CHANGELOG.md

@@ -2,18 +2,169 @@
 
 ## Current Main
 
+### Breaking Changes
+
+- report endpoint: has been deleted ([#818])
+
+
+### Other Changes
+
+- a new regression test suite has been added to support safer deployments of new versions ([#820])
+
+
+
+[#818]: https://github.com/GIScience/ohsome-quality-api/pull/818
+[#820]: https://github.com/GIScience/ohsome-quality-api/issues/820
+
+
+## Release 1.4.0
+
+
+### New Features
+
+- attribute-completeness: add workaround for attribute_key to use indicator in dashboard ([#800])
+- attribute-completeness: add validation for attribute-topic combination ([#810])
+- attribute-completeness: support for metadata and custom end-point ([#810])
+- attribute-completeness: improve generation of openAPI documentation ([#810])
+ 
+
+### Bug Fixes
+
+- road-comparison: bar for not covered roads is now shown in the plot ([#798])
+- attribute-completeness: fix bugs in result figure ([#800])
+
+
+### Other Changes
+
+- road-comparison: change color of bar to grey ([#798])
+- style: run ruff after version upgrade ([#806])
+- build: bump ruff version ([#806])
+- remove support for raster data from implementation and docs ([#815])
+
+
+[#798]: https://github.com/GIScience/ohsome-quality-api/pull/798
+[#800]: https://github.com/GIScience/ohsome-quality-api/pull/800
+[#815]: https://github.com/GIScience/ohsome-quality-api/issues/815
+[#810]: https://github.com/GIScience/ohsome-quality-api/pull/810
+
+## Release 1.3.0
+
+### Breaking Changes
+
+- major-roads-length: rename to `roads` and change filter to `highway=* and type:way` ([#786])
+
+### New Features
+
+- road-comparison: add new indicator which compares OSM roads with a reference dataset ([#778, #787, #788])
+- building-comparison: support comparison with multiple datasets ([#768])
+- building-comparison: improve figure and result description by adding more information ([#777, #789])
+
+### Other Changes
+
+- build: update dependencies (`rpy2` and `fastapi`) ([#775])
+- update dependencies ([#785])
+- test: use Approval Tests ([#789])
+- building-comparison: improve performance of request to database ([#790])
+
+
+[#768]: https://github.com/GIScience/ohsome-quality-api/pull/768
+[#775]: https://github.com/GIScience/ohsome-quality-api/pull/775
+[#777]: https://github.com/GIScience/ohsome-quality-api/issues/777
+[#778]: https://github.com/GIScience/ohsome-quality-api/issues/778
+[#785]: https://github.com/GIScience/ohsome-quality-api/pull/785
+[#786]: https://github.com/GIScience/ohsome-quality-api/issues/786
+[#787]: https://github.com/GIScience/ohsome-quality-api/issues/787
+[#788]: https://github.com/GIScience/ohsome-quality-api/issues/788
+[#789]: https://github.com/GIScience/ohsome-quality-api/pull/789
+[#790]: https://github.com/GIScience/ohsome-quality-api/pull/790
+[#806]: https://github.com/GIScience/ohsome-quality-api/pull/806
+
+## Release 1.2.0
+
+### New Features
+
+- api: add inverse parameter to coverage endpoint ([#762])
+- building-comparison: add caching of requests made to the database for EUBUCCO buildings ([#754])
+- building-comparison: add indicator to the list of core indicators and as such make it available on the ohsome Dashboard ([#765])
+
+[#754]: https://github.com/GIScience/ohsome-quality-api/pull/754
+[#762]: https://github.com/GIScience/ohsome-quality-api/issues/762
+[#765]: https://github.com/GIScience/ohsome-quality-api/issues/765
+
+## Release 1.1.1
+
+### Bug Fixes
+
+- fix(building-comparison): Use psycopg AsyncConnection. Database connection was synchronous. ([ba279b2])
+
+[ba279b2]: https://github.com/GIScience/ohsome-quality-api/commit/ba279b23b82404d9f6c377e2a34683b159803359
+
+## Release 1.1.0
+
+### Bug Fixes
+
+- building-comparison: is calculated for coverage area only ([#739])
+- building-comparison: result description now shows correct coverage percentage ([#739])
+- mapping-saturation: estimated total line is now always plotted, f it is not more than five times as large as the highest value ([#723])
+
+### New Features
+
+- building-comparison: AOI is now clipped to the coverage area ([#739])
+- mapping-saturation: add hover info to estimated total data line ([#723])
+- building-comparison: now has link to reference datasets ([#741])
+- building-comparison: added publication date of reference dataset and OSM Timestamp([#750])
+
+### Other Changes
+
+- building-comparison: no quality estimation for areas with strong difference to reference data ([#739])
+- test(db): add missing mock for getting coverage from database in tests ([#739])
+- building-comparison: request EUBUCCO database is now much faster ([#743])
+
+[#723]: https://github.com/GIScience/ohsome-quality-api/pull/723
+[#739]: https://github.com/GIScience/ohsome-quality-api/pull/739
+[#741]: https://github.com/GIScience/ohsome-quality-api/pull/741
+[#743]: https://github.com/GIScience/ohsome-quality-api/pull/743
+
+
+## Release 1.0.2
+
+### Bug Fixes
+
+- crs validation now covers all valid cases ([#721])
+
+### New Features
+
+- feat(figures): colors based on semantic UI CSS ([#591])
+- feat(building-comparison): add comparison indicator ([#718])
+- feat(api): add endpoint to retrieve coverage ([#718])
+- feat(db): query building area and coverage from EUBUCCO ([#718])
+
+[#591]: https://github.com/GIScience/ohsome-quality-api/issues/591
+[#718]: https://github.com/GIScience/ohsome-quality-api/pull/718
+[#721]: https://github.com/GIScience/ohsome-quality-analyst/pull/721
+
+## Release 1.0.1
+
+- refactor: rename "ohsome quality analyst" (OQT) to "ohsome quality API" (oqapi) ([#685])
+
+[#685]: https://github.com/GIScience/ohsome-quality-api/pull/685
+
+## Release 1.0.0
+
+This release includes a well specified [API](https://oqt.ohsome.org/api/docs) and two core indicators available through the [ohsome dashboard](https://dashboard.ohsome.org/#backend=oqapi).
+
 ### Breaking
 
 - api: remove `dataset`, `featureId` and `fidField` parameter ([#634])
-- api: computing mapping-saturation indicator for given data is now a separate endpoint ([#642])
+- api: computing mapping-saturation indicator for given data is now a separate (hidden) endpoint ([#642])
 - cli: remove CLI ([#634])
-- api: remove `include_svg`, `include_html` and `flatten` parameter ([#644])
+- api: remove `include_svg`, `include_html` `include_data` and `flatten` parameter ([#644])
 - api: only accept FeatureCollection as `bpolys` parameter ([#650])
 
 ### New Features
 
-- feat(api): add `includeFigure` parameter to `/indicators` endpoint ([#659])
-- feat(api): add indicator response models ([#670])
+- api: add `includeFigure` parameter to `/indicators` endpoint ([#659])
+- api: add indicator response models ([#670])
 
 ### Other Changes
 
@@ -38,6 +189,7 @@
     - now uses topic specific thresholds for bin creation ([#668])
     - plot now shows relative and absolute contributions ([#668])
     - further improvements to description and plot ([#668])
+    - using a geojson with wrong projection now throws specific error ([#667])
 - mapping-saturation: now shows the estimated total data in the plot ([#658])
 
 [#589]: https://github.com/GIScience/ohsome-quality-analyst/pull/589
@@ -57,6 +209,7 @@
 [#656]: https://github.com/GIScience/ohsome-quality-analyst/pull/656
 [#658]: https://github.com/GIScience/ohsome-quality-analyst/pull/658
 [#659]: https://github.com/GIScience/ohsome-quality-analyst/pull/659
+[#667]: https://github.com/GIScience/ohsome-quality-analyst/pull/667
 [#668]: https://github.com/GIScience/ohsome-quality-analyst/pull/668
 [#669]: https://github.com/GIScience/ohsome-quality-analyst/pull/669
 [#670]: https://github.com/GIScience/ohsome-quality-analyst/pull/670

Diff for: CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 ## How to Contribute
 
-Please contribute to this repository through creating [issues](https://github.com/GIScience/ohsome-quality-analyst/issues/new) and [pull requests](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests).
+Please contribute to this repository through creating [issues](https://github.com/GIScience/ohsome-quality-api/issues/new) and [pull requests](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests).
 
 
 ### Issues
@@ -47,13 +47,13 @@ All development work is based on the main branch (`main`). Merge requests are ex
 
 ### Tools
 
-This project uses [`black`](https://github.com/psf/black) and [`ruff`](https://github.com/astral-sh/ruff) to ensure consistent code style. See the `pyproject.toml` file for configuration.
+This project uses [`ruff`](https://github.com/astral-sh/ruff) to ensure consistent code style. See the `pyproject.toml` file for configuration.
 
 
 #### A Note on the Configuration
 
 ```bash
-poetry run black .
+poetry run ruff format .
 poetry run ruff .
 ```
 

Diff for: COPYRIGHTS.md

@@ -46,17 +46,11 @@ Please have a look at the API response field `attribution` to figure out which o
 - Attribution and use conditions: The development indicators available at the GDL Website can be freely downloaded and used by any interested person or party, provided that the Global Data Lab is mentioned as the source of the indicators and a link to the GDL Website is shown. 
 
 
-## OQT Regions
-
-Region geometry for which OQT provides pre-computed results.
-
-### GADM
-
-- Source: https://gadm.org/license.html
-- Attribution and use condition: The data are freely available for academic use and other non-commercial use. Redistribution or commercial use is not allowed without prior permission.
-
-
-### Nomenclature of Territorial Units for Statistics (NUTS)
-
-- Source: https://ec.europa.eu/eurostat/web/gisco/geodata/reference-data/administrative-units-statistical-units
-- Attribution and use condition: (1) The data will not be used for commercial purposes; (2) The source will be acknowledged.
+## EUBUCCO
+
+- Source: https://eubucco.com/data/
+- Attribution and use conditions: The EUBUCCO database is composed of files under different licenses:
+    - ODbL: Default (95+%)
+    - CC-BY-SA: Prague (v0_1-CZE_11-OTHER-LICENSE-CC-BY-SA.zip)
+    - CC-BY-NC: Abruzzo (v0_1-ITA_1-OTHER-LICENSE-CC-BY-NC.zip)
+- Citation: `Milojevic-Dupont, N., Wagner, F., Nachtigall, F. et al. EUBUCCO v0.1: European building stock characteristics in a common and open database for 200+ million individual buildings. Sci Data 10, 147 (2023). https://doi.org/10.1038/s41597-023-02040-2`

Diff for: Dockerfile

@@ -2,6 +2,10 @@
 # we have to use bullseye, because bookworm doesn't work with older Docker versions which are still in use
 FROM python:3.10-bullseye
 
+# Allow to set custom uid and gid values (i.e. for CI)
+ARG uid=1000
+ARG gid=1000
+
 # install R
 # to avoid caching issues combine apt-get update and install in one RUN statement.
 # to reduce image size, clean up the apt cache by removing /var/lib/apt/lists.
@@ -10,24 +14,25 @@ RUN apt-get update && \
     rm -rf /var/lib/apt/lists/*
 
 # within docker container: run without root privileges
-RUN useradd -md /home/oqt oqt
-WORKDIR /opt/oqt
-RUN chown oqt:oqt . -R
-USER oqt:oqt
+RUN groupadd -g "$gid" oqapi
+RUN useradd -l -md /home/oqapi -u "$uid" -g "$gid" oqapi
+WORKDIR /opt/oqapi
+RUN pip install --no-cache-dir poetry
+RUN chown oqapi:oqapi . -R
+USER oqapi:oqapi
 
 # make poetry binaries available to the docker container user
-ENV PATH=$PATH:/home/oqt/.local/bin
+ENV PATH=$PATH:/home/oqapi/.local/bin
 
 # install only the dependencies
-COPY --chown=oqt:oqt pyproject.toml pyproject.toml
-COPY --chown=oqt:oqt poetry.lock poetry.lock
-RUN pip install --no-cache-dir poetry
+COPY --chown=oqapi:oqapi pyproject.toml pyproject.toml
+COPY --chown=oqapi:oqapi poetry.lock poetry.lock
 RUN python -m poetry install --no-ansi --no-interaction --no-root
 
 # copy all the other files and install the project
-COPY --chown=oqt:oqt ohsome_quality_analyst ohsome_quality_analyst
-COPY --chown=oqt:oqt tests tests
-COPY --chown=oqt:oqt scripts/start_api.py scripts/start_api.py
-COPY --chown=oqt:oqt config/logging.yaml config/logging.yaml
+COPY --chown=oqapi:oqapi ohsome_quality_api ohsome_quality_api
+COPY --chown=oqapi:oqapi tests tests
+COPY --chown=oqapi:oqapi scripts/start_api.py scripts/start_api.py
+COPY --chown=oqapi:oqapi config/logging.yaml config/logging.yaml
 RUN python -m poetry install --no-ansi --no-interaction