chore: refactor test scripts (#4577)

* chore: refactor test scripts * chore: format * fix: UESIO_DEV needed for nx graph due to cypress.config.ts * fix: debug for nx failing to build graph * chore: add more diag info and explicitly ignore go-build-cache directory for nx * chore: finalize nx graph fix * chore: don't leak -e for env setup * chore: refactor build steps for docker tests * chore: ignore go-build-cache in .gitignore instead
ues-io · Feb 7, 2025 · ed612d8 · ed612d8
1 parent 8e577ca
commit ed612d8
Show file tree

Hide file tree

Showing 24 changed files with 310 additions and 193 deletions.
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -81,11 +81,6 @@ jobs:
         env:
           UESIO_DEV: "true" # required to ensure packui gets built
         run: |
-          # We lint/test/build affected but in order to build image, we need to ensure everything
-          # is built.  Build anything that hasn't been built yet (takes advantage of nx cache
-          # for anything already built from above)
-          npx nx run-many -t build
-
           # build values for docker image tags
           VERSION_SUFFIX=${{ github.run_number }}.${{ github.run_attempt }}
           FULL_SHA=${{ steps.setNxSHAs.outputs.head }}
@@ -155,9 +150,13 @@ jobs:
         run: |
           ./scripts/seed-etc-hosts.sh
 
+          cd apps/platform/ssl
+          bash ./create.sh
+          cd ../../../
+
           # Start up the Uesio app, and dependencies, in Docker
           # then run all Integration and E2E tests against the app
-          npm run tests-ci
+          npm run tests-docker
 
   check:
     name: Check format and lint

diff --git a/.gitignore b/.gitignore
@@ -78,3 +78,10 @@ __debug_bin
 queries/results
 .nx/cache
 .nx/workspace-data
+
+# CI workflows cache go which creates a go-build-cache directory that contains packages that contain package.json files
+# so whenever nx is run, it "thinks" they are "projects" in the nx workspace and then fails to parse them (e.g., they
+# do not have a "name" property).  The below could be included in a .nxignore file since the problem is nx specific,
+# however putting here instead because we would also never want to commit these files if we ever committed in
+# our workflows.
+/go-build-cache
diff --git a/README.md b/README.md
@@ -447,38 +447,54 @@ docker compose up -d
 bash apps/platform/migrations_test/test_migrations.sh
 ```
 
-## End-to-end Testing and Integration testing
+## Testing (Unit, Integration & E2E)
 
-To run E2E and Integration tests locally, there are a number of commands available:
+> ![IMPORTANT]
+> The default behavior for all tests is to run against `https://studio.uesio-dev.com:3000` so you must ensure that [SSL](#set-up-ssl) and [local DNS](#set-up-your-local-dns) have been configured.
 
-1. `npm run tests-all`
-   - Runs all Integration and E2E tests against your local Uesio app.
+To run the various test suites, there are a number of commands available:
+
+1. `npm run tests`
+   - Runs all Unit tests
+2. `npm run tests-all`
+   - Runs all Unit, Integration and E2E tests against your local Uesio app.
    - Use this when writing and debugging tests locally
-1. `npm run tests-ci`
-   - This is what we run in Github Actions [CI](./.github/workflows/ci.yaml) builds. It spins up all dependencies, and a Dockerized version of the Uesio app, runs integration and E2E tests against the app, and then spins down all Docker containers.
-1. `npm run tests-integration`
+3. `npm run tests-docker`
+   - This is what we run in Github Actions [CI](./.github/workflows/ci.yaml) builds for integration & E2E tests. It builds the latest code, spins up all dependencies, and a Dockerized version of the Uesio app, runs integration and E2E tests against the app, and then spins down all Docker containers.
+4. `npm run tests-integration`
    - Runs _just_ the Integration Tests (against your local app).
-1. `npm run tests-e2e`
+5. `npm run tests-e2e`
    - Runs _just_ the E2E Tests (against your local app).
+6. `npm run tests-init`
+   - Deletes the `uesio/tests` app if it exists and then creates the `uesio/tests` app with related workspaces and sites and loads seed data. All of the above scripts execute this script automatically so there is not typically a need to run it. However, if you want to run individual tests (via hurl, cypress, etc.) separate from one of the above scripts which runs an entire suite, you will need to run this script to prepare for test execution.
+7. `npm run tests-cleanup`
+   - Removes the `uesio/tests` app (if it exists). Similar to `tests-init`, the automated test suite scripts will execute this script prior to completion. However, if a test run terminates abnormally and/or if you ran `tests-init` manually, you can execute this script to remove the test related `uesio/tests` app.
+8. `npm run tests-cypress-open`
+   - Runs the cypress visual UI where you can run E2E tests from. See [E2E testing with cypress](#e2e-testing-with-cypress) for details.
+9. `npm run tests-cypress-run`
+   - Runs the cypress in headless mode, helpful when you want to run individual tests. See [E2E testing with cypress](#e2e-testing-with-cypress) for details.
 
 TO run just an individual E2E or Integration test, see the sections below.
 
+> [!NOTE]
+> You must manually run `npm run tests-init` in order to run individual tests. Depending on the test, you may need to re-run this script prior to every test execution. Additionally, ensure that `UESIO_DEV=true` environment variable is set prior to starting the server and for each test so that mock logins can be used.
+
 ### E2E testing with Cypress
 
-We use [Cypress](https://cypress.io) for writing end-to-end tests of the Uesio app. All E2E tests are defined in `cypress/e2e` directory.
+We use [Cypress](https://cypress.io) for writing end-to-end tests of the Uesio app. All E2E tests are defined in `apps/platform-e2e` directory.
 
 E2E tests are the most expensive and most brittle, and as such should be used sparingly.
 
-If you're running Uesio locally, you can use `npx cypress open` to launch Cypress' visual UI for running tests, or `npm run tests-e2e` to just run the tests in a headless runner.
+If you're running Uesio locally, you can use `npm run tests-cypress-open` to launch Cypress' visual UI for running tests, or `npm run tests-e2e` to just run all the tests in a headless runner. Note that when running using the visual UI or when running individual tests as per the below, you must have the `UESIO_DEV=true` environment variable set and have run `npm run tests-init`.
 
 #### Running a single E2E spec
 
 If you want to _visually_ run a single spec, use the Cypress visual UI and then select the individual spec.
 
-Or, use `npx cypress run --spec <path to spec>` to run a specific file in a headless Electron instance, e.g.
+Or, use `npm run tests-cypress-run -- --spec <path to spec>` to run a specific file in a headless Electron instance, e.g.
 
-```
-npx cypress run --spec cypress/e2e/builder.cy.ts
+```bash
+npx run tests-cypress-run -- --spec apps/platform-e2e/cypress/e2e/builder.cy.ts
 ```
 
 ### Integration / API testing with Hurl
@@ -489,13 +505,19 @@ To run API integration tests locally against your running Uesio container, use `
 
 #### Running a single Integration Test
 
-The easiest way to run a single Integration Test is to go into the `run-integration-tests.sh` file and comment out the lines where we run all tests, and uncomment the lines here:
+The easiest way to run a single Integration Test is to go into the `scripts/tests/start-integration-tests.sh` file and comment out the lines where we run all tests, and uncomment the lines here and then run `npm run tests-integration`:
 
 ```
-npx hurl --very-verbose -k --variable host=studio.uesio-dev.com --variable domain=uesio-dev.com --variable port=3000 hurl_specs/wire_collection_dependencies.hurl
+# npx hurl --very-verbose -k --variable host=$UESIO_TEST_HOST_NAME --variable domain=$UESIO_TEST_DOMAIN --variable port=$UESIO_TEST_PORT apps/platform-integration-tests/hurl_specs/wire_collection_dependencies.hurl
 ```
 
-You could run this from the CLI, but you would have to make sure that you are (a) in the right directory (b) have the right environment variables set up. (See the top of this `run-integration-tests.sh` for a better understanding).
+You could run the individual test from the CLI, but you would have to make sure that you have the test app created and the right environment variables set up. If you would like to run via the CLI:
+
+```bash
+npm run tests-init # initialize test app/workspace/site/data/etc.
+source scripts/tests/setup-env.sh # setup environment variables used in tests
+npx hurl --very-verbose -k --variable host=$UESIO_TEST_HOST_NAME --variable domain=$UESIO_TEST_DOMAIN --variable port=$UESIO_TEST_PORT apps/platform-integration-tests/hurl_specs/wire_collection_dependencies.hurl
+```
 
 # Continous integration (CI)
 

diff --git a/apps/platform-e2e/cypress.config.ts b/apps/platform-e2e/cypress.config.ts
@@ -7,6 +7,7 @@ const automationUsername = process.env.UESIO_AUTOMATION_USERNAME || "uesio"
 const automationPassword = process.env.UESIO_AUTOMATION_PASSWORD
 const useMockLogin = !automationPassword
 const inCi = process.env.CI === "true"
+
 export default defineConfig({
   e2e: {
     ...nxE2EPreset(__filename, {

diff --git a/apps/platform-e2e/cypress/e2e/wire_lookup_conditions.cy.ts b/apps/platform-e2e/cypress/e2e/wire_lookup_conditions.cy.ts
@@ -25,7 +25,7 @@ describe("Wire Lookup Conditions", () => {
       )
       cy.get("table[id$='contactsTable']>tbody>tr", {
         timeout: 4000,
-      }).should("have.length", 2)
+      }).should("have.length", 3)
 
       // Now initiate a manual requery by searching
       cy.getByIdFragment("input", "contactsSearch").type("Fred")

diff --git a/...n-tests/hurl_seeds/app_and_workspace.hurl → .../hurl_seeds/create_app_and_workspace.hurl b/...n-tests/hurl_seeds/app_and_workspace.hurl → .../hurl_seeds/create_app_and_workspace.hurl
@@ -13,59 +13,6 @@ user_id: jsonpath "$.user.id"
 POST https://{{host}}:{{port}}/site/perf/stats/reset
 HTTP 204
 
-# Delete the test site if it already exists
-DELETE https://{{host}}:{{port}}/site/api/v1/collection/uesio/studio/site?uesio/studio.name=eq.testsite
-x-uesio-app: uesio/tests
-HTTP 204
-
-# Delete the tests app if it already exists, which should cascade delete the workspace and all of its metadata
-DELETE https://{{host}}:{{port}}/site/api/v1/collection/uesio/studio/app?uesio/studio.fullname=eq.uesio%2Ftests
-HTTP 204
-
-# Verify that workspaces are cascade deleted
-POST https://{{host}}:{{port}}/site/wires/load
-{
-    "wires": [
-        {
-            "name": "workspaces",
-            "collection": "uesio/studio.workspace",
-            "conditions": [
-                {
-                    "field": "uesio/studio.app->uesio/studio.fullname",
-                    "operator": "EQ",
-                    "value": "uesio/tests"
-                }
-            ]
-        }
-    ]
-}
-HTTP 200
-[Asserts]
-jsonpath "$.wires[0].errors" not exists
-jsonpath "$.wires[0].data" count == 0
-
-# Verify that sites are cascade deleted
-POST https://{{host}}:{{port}}/site/wires/load
-{
-    "wires": [
-        {
-            "name": "sites",
-            "collection": "uesio/studio.site",
-            "conditions": [
-                {
-                    "field": "uesio/studio.app->uesio/studio.fullname",
-                    "operator": "EQ",
-                    "value": "uesio/tests"
-                }
-            ]
-        }
-    ]
-}
-HTTP 200
-[Asserts]
-jsonpath "$.wires[0].errors" not exists
-jsonpath "$.wires[0].data" count == 0
-
 # Create the tests app
 POST https://{{host}}:{{port}}/site/wires/save
 {

diff --git a/apps/platform-integration-tests/hurl_seeds/delete_app.hurl b/apps/platform-integration-tests/hurl_seeds/delete_app.hurl
@@ -0,0 +1,62 @@
+POST https://{{host}}:{{port}}/site/auth/uesio/core/mock/login
+{
+    "token": "uesio"
+}
+# Extract the sessionid from set-cookie header
+HTTP 200
+[Captures]
+session_id: cookie "sessid"
+user_id: jsonpath "$.user.id"
+
+# Delete the test site if it already exists
+DELETE https://{{host}}:{{port}}/site/api/v1/collection/uesio/studio/site?uesio/studio.name=eq.testsite
+x-uesio-app: uesio/tests
+HTTP 204
+
+# Delete the tests app if it already exists, which should cascade delete the workspace and all of its metadata
+DELETE https://{{host}}:{{port}}/site/api/v1/collection/uesio/studio/app?uesio/studio.fullname=eq.uesio%2Ftests
+HTTP 204
+
+# Verify that workspaces are cascade deleted
+POST https://{{host}}:{{port}}/site/wires/load
+{
+    "wires": [
+        {
+            "name": "workspaces",
+            "collection": "uesio/studio.workspace",
+            "conditions": [
+                {
+                    "field": "uesio/studio.app->uesio/studio.fullname",
+                    "operator": "EQ",
+                    "value": "uesio/tests"
+                }
+            ]
+        }
+    ]
+}
+HTTP 200
+[Asserts]
+jsonpath "$.wires[0].errors" not exists
+jsonpath "$.wires[0].data" count == 0
+
+# Verify that sites are cascade deleted
+POST https://{{host}}:{{port}}/site/wires/load
+{
+    "wires": [
+        {
+            "name": "sites",
+            "collection": "uesio/studio.site",
+            "conditions": [
+                {
+                    "field": "uesio/studio.app->uesio/studio.fullname",
+                    "operator": "EQ",
+                    "value": "uesio/tests"
+                }
+            ]
+        }
+    ]
+}
+HTTP 200
+[Asserts]
+jsonpath "$.wires[0].errors" not exists
+jsonpath "$.wires[0].data" count == 0
diff --git a/apps/platform/ssl/create.sh b/apps/platform/ssl/create.sh
@@ -1,9 +1,24 @@
 #!/bin/bash
 
-# Generate a new primary key
-openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -keyout private.key -days 3560 -out certificate.crt -config config.txt
+CERT_FILE="$PWD/certificate.crt"
 
-# OS X shortcut to automatically trust the certificate
-if [[ "$(uname)" == "Darwin" ]]; then 
-	sudo security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" "$PWD/certificate.crt"
+if [ -f "$CERT_FILE" ]; then
+    echo "SSL certificate already exists. Delete the existing certificate and re-run to generate a new SSL certificate."
+else
+  # Generate a new primary key
+  openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -keyout private.key -days 3560 -out "$CERT_FILE" -config config.txt
+  echo "SSL Certificate and private key created!"
+
+  # OS X shortcut to automatically trust the certificate
+  if [[ "$(uname)" == "Darwin" ]]; then
+    sudo security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" "$CERT_FILE"
+  else
+    # Update CA certificates so that CLI and Curl will not complain when connecting
+    # to our local Uesio instance with self-signed certificate
+    # TODO: update-ca-certificates should be fairly portable across distros but may need to handle differently in some cases
+    sudo cp "$CERT_FILE" /usr/local/share/ca-certificates/
+    sudo update-ca-certificates
+  fi
+  echo "System certificate trust store updated to trust SSL certificate!"
 fi
+
diff --git a/package.json b/package.json
@@ -16,12 +16,14 @@
     "get-module-version": "node -p \"require('./package').devDependencies['$npm_config_modulename']\" | sed 's/^\\^//g'",
     "graceful-shutdown": "lsof -t -i :3000 | xargs kill -15",
     "kill:webserver": "lsof -t -i :3000 | xargs kill -9",
-    "tests-cleanup": "bash scripts/tests-cleanup.sh",
+    "tests-cleanup": "bash scripts/cleanup-tests.sh",
     "tests-e2e": "bash scripts/run-e2e-tests.sh",
-    "tests-init": "bash scripts/tests-init.sh",
+    "tests-init": "bash scripts/init-tests.sh",
     "tests-integration": "bash scripts/run-integration-tests.sh",
     "tests-all": "bash scripts/run-all-tests.sh",
-    "tests-ci": "bash scripts/run-tests-ci.sh",
+    "tests-docker": "bash scripts/run-docker-tests.sh",
+    "tests-cypress-open": "cypress open --project apps/platform-e2e",
+    "tests-cypress-run": "cypress run --project apps/platform-e2e",
     "format": "nx format:write",
     "format:check": "nx format:check",
     "format:write": "nx format:write",

diff --git a/scripts/cleanup-tests.sh b/scripts/cleanup-tests.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+# Initial setup
+source ./scripts/tests/setup-common.sh
+
+# cleans up the tests app and workspaces,
+# and verifies that cleanup worked as expected
+bash "scripts/tests/cleanup-common.sh"
diff --git a/scripts/init-tests.sh b/scripts/init-tests.sh
@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+# Initial setup
+source ./scripts/tests/setup-common.sh
+
+# Initialize the sample app and seed test data
+bash "scripts/tests/init-common.sh"
diff --git a/scripts/run-all-tests.sh b/scripts/run-all-tests.sh
@@ -8,15 +8,19 @@
 
 set -e
 
+# Initial setup
+source ./scripts/tests/setup-common.sh
+
 # Runs unit tests
 npm run test
 
 # Runs Hurl integration tests against the app
-bash ./scripts/run-integration-tests.sh
+bash ./scripts/tests/start-integration-tests.sh
 
 # Runs Cypress End-to-End tests against the app
-bash ./scripts/run-e2e-tests.sh
+bash ./scripts/tests/start-e2e-tests.sh
 
-# Cleans up the tests app and workspaces
-bash ./scripts/tests-cleanup.sh
+# cleans up the tests app and workspaces,
+# and verifies that cleanup worked as expected
+bash ./scripts/tests/cleanup-common.sh
 
diff --git a/scripts/run-tests-ci.sh → scripts/run-docker-tests.sh b/scripts/run-tests-ci.sh → scripts/run-docker-tests.sh
@@ -11,19 +11,22 @@
 
 set -e
 
+# Initial setup
+source ./scripts/tests/setup-common.sh
+
 # Spins up dependencies and runs the app in Docker
-bash ./scripts/tests-setup.sh
+bash ./scripts/tests/setup-docker.sh
 
 # Runs Hurl integration tests against the app
-bash ./scripts/run-integration-tests.sh
+bash ./scripts/tests/start-integration-tests.sh
 
 # Runs Cypress End-to-End tests against the app
-bash ./scripts/run-e2e-tests.sh
+bash ./scripts/tests/start-e2e-tests.sh
 
 # cleans up the tests app and workspaces,
-# and verifies that these routines work as expected
-bash ./scripts/tests-cleanup.sh
+# and verifies that cleanup worked as expected
+bash ./scripts/tests/cleanup-common.sh
 
 # Spins down all docker containers
-bash ./scripts/tests-down.sh
+bash ./scripts/tests/cleanup-docker.sh