Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ jobs:
cache-dependency-glob: "webknossos/uv.lock"

- name: Install proxay
run: npm install -g proxay
run: npm install -g proxay@1.9.0

- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}
Expand All @@ -176,14 +176,14 @@ jobs:
timeout-minutes: 30
env:
PYTHON_VERSION: ${{ matrix.python-version }}
run: ./test.sh --splits 3 --group ${{ matrix.group }} --splitting-algorithm least_duration --cov=webknossos --cov-report=
run: uv run test.py --splits 3 --group ${{ matrix.group }} --splitting-algorithm least_duration --cov=webknossos --cov-report=

- name: Python tests
if: ${{ matrix.python-version != '3.11' }}
timeout-minutes: 30
env:
PYTHON_VERSION: ${{ matrix.python-version }}
run: ./test.sh --splits 3 --group ${{ matrix.group }} --splitting-algorithm least_duration
run: uv run test.py --splits 3 --group ${{ matrix.group }} --splitting-algorithm least_duration

- name: Persist coverage reports
if: ${{ matrix.python-version == '3.11' }}
Expand Down Expand Up @@ -222,22 +222,21 @@ jobs:
enable-cache: true
cache-dependency-glob: "webknossos/uv.lock"

- name: Install proxay
run: npm install -g proxay

- name: Install minio
run: Invoke-WebRequest -Uri "https://dl.min.io/server/minio/release/windows-amd64/minio.exe" -OutFile "minio.exe"

- name: Install proxay
run: npm install -g [email protected]

- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Python tests
timeout-minutes: 30
shell: bash
env:
PYTHON_VERSION: ${{ matrix.python-version }}
MULTIPROCESSING_DEFAULT_START_METHOD: spawn
run: ./test.sh --splits 3 --group ${{ matrix.group }} --splitting-algorithm least_duration
run: uv run test.py --splits 3 --group ${{ matrix.group }} --splitting-algorithm least_duration

coverage_report:
needs: [webknossos_linux]
Expand Down
74 changes: 37 additions & 37 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: nightly

on:
schedule:
- cron: '00 06 * * *'
- cron: "00 06 * * *"
workflow_dispatch: ~

jobs:
Expand All @@ -19,46 +19,46 @@ jobs:
working-directory: webknossos

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v3

- name: Install uv
uses: astral-sh/setup-uv@v3
- name: Install uv
uses: astral-sh/setup-uv@v3

- name: Install proxay
run: npm install -g proxay
- name: Install proxay
run: npm install -g proxay

- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}
- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Check if git is dirty
run: |
git diff --no-ext-diff --quiet --exit-code
[[ -z $(git status -s) ]]
- name: Check if git is dirty
run: |
git diff --no-ext-diff --quiet --exit-code
[[ -z $(git status -s) ]]

- name: Python tests, refreshing the network snapshots
env:
WK_TOKEN: ${{ secrets.WK_TOKEN }}
run: ./test.sh --refresh-snapshots --splits 3 --group ${{ matrix.group }}
- name: Python tests, refreshing the network snapshots
env:
WK_TOKEN: ${{ secrets.WK_TOKEN }}
run: uv run test.py --refresh-snapshots --splits 3 --group ${{ matrix.group }}

- name: Python tests, using the new snapshots
env:
WK_TOKEN: ${{ secrets.WK_TOKEN }}
run: ./test.sh --refresh-snapshots --splits 3 --group ${{ matrix.group }}
- name: Python tests, using the new snapshots
env:
WK_TOKEN: ${{ secrets.WK_TOKEN }}
run: uv run test.py --refresh-snapshots --splits 3 --group ${{ matrix.group }}

- uses: slackapi/[email protected]
if: failure() || cancelled()
with:
webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
webhook-type: incoming-webhook
payload: |
attachments:
- color: "#f00"
blocks:
- type: "section"
text:
type: "mrkdwn"
text: ':fire: Job <${{github.server_url}}/${{github.repository}}/actions/runs/${{github.run_id}}|${{github.job}}> of workflow "${{github.workflow}}" failed'
- type: "context"
elements:
- type: mrkdwn
text: '<${{github.repositoryUrl}}|${{github.repository}}>'
- uses: slackapi/[email protected]
if: failure() || cancelled()
with:
webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
webhook-type: incoming-webhook
payload: |
attachments:
- color: "#f00"
blocks:
- type: "section"
text:
type: "mrkdwn"
text: ':fire: Job <${{github.server_url}}/${{github.repository}}/actions/runs/${{github.run_id}}|${{github.job}}> of workflow "${{github.workflow}}" failed'
- type: "context"
elements:
- type: mrkdwn
text: '<${{github.repositoryUrl}}|${{github.repository}}>'
14 changes: 7 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ See below for specifics of the different packages. Let's have a look at the comm
* `./format.sh`: ruff
* `./lint.sh`: ruff
* `./typecheck.sh`: mypy
* `./test.sh`: pytest and custom scripts
* `./test.py`: pytest and custom scripts

Those are also accessible via make commands from the top-level directory, running the respective scripts for each sub-project
, e.g. `make format`, `make lint`, …
Expand All @@ -133,26 +133,26 @@ Internal workflows for scalable minds:

The `webknossos` folder contains examples, which are not part of the package, but are tested via `tests/test_examples.py` and added to the documentation (see `docs/src/webknossos-py/examples`).

To run the `./test.sh` script it is necessary to install `proxay`. This is either done with [NPM](https://www.npmjs.com) or [yarn](https://yarnpkg.com/getting-started/install):
To run the `uv run test.py` script it is necessary to install `proxay`. This is either done with [NPM](https://www.npmjs.com) or [yarn](https://yarnpkg.com/getting-started/install):
```bash
npm install --global proxay

# or if you're using yarn
yarn global add proxay
```
When running the `./test.sh` script on MacOS, it is also necessary to install `minio` using the following command:
When running the `uv run test.py` script on MacOS, it is also necessary to install `minio` using the following command:

```bash
brew install minio
```

The tests also contain functionality for the WEBKNOSSOS client. There a two modes to run the tests:

1. `./test.sh --refresh-snapshots`, sending network requests to a WEBKNOSSOS instance:
This expects a local WEBKNOSSOS setup with specific test data, which is shipped with WEBKNOSSOS. If you're starting and running WEBKNOSSOS manually, please use port 9000 (the default) and run the `tools/postgres/dbtool.js prepare-test-db` script in the WEBKNOSSOS repository (⚠️ this overwrites your local WEBKNOSSOS database). Alternatively, a `docker compose` setup is started automatically for the tests, see `./test.sh` and `tests/docker-compose.yml` for details. The network requests & response are recorded as "cassettes" by [proxay](https://github.com/airtasker/proxay), see next point:
2. `./test.sh` replays responses from previous network snapshots using [proxay](https://github.com/airtasker/proxay).
1. `uv run test.py --refresh-snapshots`, sending network requests to a WEBKNOSSOS instance:
This expects a local WEBKNOSSOS setup with specific test data, which is shipped with WEBKNOSSOS. If you're starting and running WEBKNOSSOS manually, please use port 9000 (the default) and run the `tools/postgres/dbtool.js prepare-test-db` script in the WEBKNOSSOS repository (⚠️ this overwrites your local WEBKNOSSOS database). Alternatively, a `docker compose` setup is started automatically for the tests, see `uv run test.py` and `tests/docker-compose.yml` for details. The network requests & response are recorded as "cassettes" by [proxay](https://github.com/airtasker/proxay), see next point:
2. `uv run test.py` replays responses from previous network snapshots using [proxay](https://github.com/airtasker/proxay).

`./test.sh --store-durations` updates the durations for
`uv run test.py --store-durations` updates the durations for
[`pytest-split`](https://jerry-git.github.io/pytest-split),
which is used in the CI to split the tests for different runners.

Expand Down
14 changes: 7 additions & 7 deletions webknossos/Changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
### Breaking Changes
- The `endpoint_url` in `UPath` objects is now stored directly in the `storage_options` dict, instead of being stored in the `storage_options["client_kwargs"]` dict. [#1365](https://github.com/scalableminds/webknossos-libs/pull/1365)
- Removed Docker image `scalableminds/webknossos-cli` build. It will not be released further. [#1376](https://github.com/scalableminds/webknossos-libs/pull/1376)

### Added
- Dataset.add_layer_as_ref() now accepts RemoteLayer objects as well as Layer objects. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
- RemoteDataset.annotation_id is now exposed, when available. [#1380](https://github.com/scalableminds/webknossos-libs/pull/1380)
- RemoteDataset.open() now accepts an annotation url as well. [#1380](https://github.com/scalableminds/webknossos-libs/pull/1380)
### Changed
- Ported `test.sh` to `test.py`, run with `uv run test.py`. [#1379](https://github.com/scalableminds/webknossos-libs/pull/1379)
- `Dataset.add_layer_as_ref(remote_layer.path)` does not work anymore. Please use `Dataset.add_layer_as_ref(remote_layer)` instead. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
- Due to the refactoring, the imports of various classes have changed. Please update your code accordingly. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
- Deprecated a number of methods [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371]):
Expand All @@ -24,13 +31,6 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
- `Dataset.download()`, use `RemoteDataset.download()` instead.
- `mag_view.is_foreign`, use `layer.is_mag_view_foreign(mag_view)` instead.
- `Dataset.trigger_reload_in_datastore()`, use `RemoteDataset.trigger_reload_in_datastore()` instead.

### Added
- Dataset.add_layer_as_ref() now accepts RemoteLayer objects as well as Layer objects. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
- RemoteDataset.annotation_id is now exposed, when available. [#1380](https://github.com/scalableminds/webknossos-libs/pull/1380)
- RemoteDataset.open() now accepts an annotation url as well. [#1380](https://github.com/scalableminds/webknossos-libs/pull/1380)

### Changed
- Replaced `pathlib.Path` for `upath.UPath` internally. `pathlib.Path` is still supported for user-facing APIs. Adds support for `universal_pathlib` version 0.3.x. [#1374](https://github.com/scalableminds/webknossos-libs/pull/1374)
- Only use fs-based copy for mags on local file systems and S3. Other protocols, e.g. memory and HTTP, have caveats that break fs-based copying. [#1365](https://github.com/scalableminds/webknossos-libs/pull/1365)
- The `add_*` methods in `Attachments` now return the created attachment objects, similar to `add_layer` and `add_mag`. [#1365](https://github.com/scalableminds/webknossos-libs/pull/1365)
Expand Down
70 changes: 0 additions & 70 deletions webknossos/local_wk_setup.sh

This file was deleted.

Loading
Loading