Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EDITOR-527] E2E testing infrastructure #591

Merged
merged 13 commits into from
Feb 3, 2021
Merged

[EDITOR-527] E2E testing infrastructure #591

merged 13 commits into from
Feb 3, 2021

Conversation

zmoog
Copy link
Contributor

@zmoog zmoog commented Jan 15, 2021

Please check if the PR fulfills these requirements

  • The PR has no duplicates (please search among the Pull Requests
    before creating one)
  • Tests for the changes have been added (for bug fixes / features)
  • What kind of change does this PR introduce?

add e2e test infrastructure based on arduino-cli's one (using python and pytest)

  • What is the current behavior?
  • What is the new behavior?

test infrastructure is added, also to CI

  • Does this PR introduce a breaking change?

no

  • Other information:
  • add test infra
  • add in taskfile
  • develop some basic test (meaningfully tests will be added later along the way)
  • integrate in the CI

Sorry, something went wrong.

Maurizio Branca and others added 3 commits January 20, 2021 11:32

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
Cleanup

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
Refs: EDITOR-527

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
…n refused""

This reverts commit a3f7cbe.

The problem preventing CI to complete successfully is that the agent needs a GUI to run

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
…ers)

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
@umbynos
Copy link
Contributor

umbynos commented Jan 21, 2021

apparently triggering the \update on the cli version of the agent causes it to update to the GUI version 😢 thus causing the CI to fail (the runners do not have a GUI).
We need to fix this behavior

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
…cally mac mini with GUI support)"

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
@umbynos umbynos changed the title E2E testing infrastructure [EDITOR-527] E2E testing infrastructure Jan 26, 2021

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
@umbynos
Copy link
Contributor

umbynos commented Feb 2, 2021

============================= test session starts ==============================
platform darwin -- Python 3.9.1, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/runner/work/arduino-create-agent/arduino-create-agent, configfile: pyproject.toml
collected 6 items

test/test_certs.py ..                                                    [ 33%]
test/test_info.py .                                                      [ 50%]
test/test_update.py ss                                                   [ 83%]
test/test_v2.py .                                                        [100%]

========================= 4 passed, 2 skipped in 2.54s =========================

🥳
Although the test_update will be skipped for now: it has some weird behavior:

  • test_update_shutdown tries to test if the agent returns 200 along with a json(/update endpoint does not work properly on macos #608) shuts down correctly (we check this behavior waiting for the process to exit) -> there is a problem in this approach: the pytest fixture running the agent, by default, tries to kill it at the end of the test (but in this case is already killed)
  • test_latest_version tries to check if the version started after the update is the latest one (by checking the version number available on agent-version.json on s3). This test is correct, but the problem comes from the fact that another fixture is required: two instances of the agent will be running at the same time. One generated by the update (we do not have a way to kill it) and one generated by the fixture. Which will be the one responding to the /info? Hard to guess.

Another important thing to point out: the agent has been updated at this point: we need to rebuild it, otherwise the version under test will be the latest one, and not the one we are developing.

@umbynos umbynos marked this pull request as ready for review February 2, 2021 17:27

@pytest.fixture(scope="session")
def base_url():
return "http://127.0.0.1:8991"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be good to dynamically check the port from (8991 to 9000)

@umbynos umbynos self-assigned this Feb 2, 2021
@umbynos umbynos added the type: enhancement Proposed improvement label Feb 2, 2021
@umbynos umbynos self-requested a review February 3, 2021 11:52
Copy link
Contributor

@umbynos umbynos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zmoog approved 🏁

@umbynos umbynos merged commit 9123aae into devel Feb 3, 2021
@umbynos umbynos deleted the zmoog/e2e branch February 3, 2021 11:54
umbynos added a commit that referenced this pull request Feb 5, 2021

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
* Bootstrap E2E testing infrastructure

* add e2e tests in taskfile and in CI

* fix deprecation warning when running pytest

* Run e2e tests only with GUI version on macos (Github runners are basically mac mini with GUI support)"

* add first draft of tests for certs and v2 APIs

* add logs to .gitignore

* skip update, currently no way of testing this on gh runners

Co-authored-by: umbynos <[email protected]>
@umbynos umbynos added the test label Feb 25, 2021
@per1234 per1234 added topic: infrastructure Related to project infrastructure and removed topic: test labels Sep 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: infrastructure Related to project infrastructure type: enhancement Proposed improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants