forked from DataBiosphere/toil
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Run CWL conformance tests on Kubernetes. (DataBiosphere#3323)
* Attempt to include and run CWL K8 tests. * Dollar dollar. * Minimize CWL build extras. * Correct batch system naming. * Only run CWL tests temporarily. * Run the kubernetes CWL tests. * Emit verbose CWL test output so we can see what's going on * Remember that we can't actually run anything on Kubernetes without an appliance * Uncomment all the other tests so we can merge properly * Delete all the tests we aren't currently interested in * Add a forgotten comma * Don't try and reconstruct exceptions * Accept the ConfigException that modern Kubernetes throws when there's no config * Make CWL Kubernetes tests invent their own job stores, and not run in parallel with a fixed job store name * Revert "Delete all the tests we aren't currently interested in" This reverts commit e02fd69. * Turn off other tests in a more mergeable way * Drop empty only * Improve spelling * Only run passing CWL1.0 tests on Kubernetes, and add 1.1 and 1.2 * Fix test syntax * Make Kubernetes CWL setup more like non-Kubernetes, and turn on more versions * Let all the CWL tests run for hours so we can see what actually fails * Remove extra _with from test name * Add back the _with and route caching tests through non-caching functions * Carve out for failing-on-Kubernetes 1.1 tests * Invent situation-appropriate job store names for CWL workflows * Run CWL tests in parallel and make the runner invent job stores * Rewrite Docker containers by wrapping Singularity Fixes DataBiosphere#3455. * Insist on wrapper and trace the env var * Validify YAML according to Gitlab's type constraints * Use correct path to Singularity wrapper * Upgrade Prometheus base to fix DataBiosphere#3457 * Add carveouts for Kubernetes CWL 1.2 tests * Carve out new failing tests * Try to carve out tests that failed on some other CWL versions * Drop tests 28,38,and 83 where they fail * Properly skip 136 and 137 for CWL 1.1 * Skip more tests observed to fail * Upgrade pip to try and fix DataBiosphere#3459 * Mark CWL tests on Kupernetes as expected to fail Around 20-40 of the CWL tests are intermittently failing on Kubernetes, and it's not worrth re-running the tests over and over to get all their numbers. Common failure modes include missing ordering constraints, and missing local file paths. This lets all the tests run (so we can see which are still broken), but expects the runs to fail. When we actually fix a test, we can add it to another test cakll that isn't xfail and runs it alone with selected_tests. Or, we can try and fix them all in a row and hope there's no backsliding, and then remove xfail. * Turn the non-CWL tests back on * Only pass the Singularity mirror if set * Actually insert the name in the missing env var error * Stop dumping environments * Go back to running CWL conformance tests under heavy CPU contention * Log some of what we are doing when fuzzing the filestore * Only set a default workDir with a non-default tmpdir_prefix * Drop some trailing whitespace * Pass Singularity mirror through to Cactus integration test * Fix Kubernetes indent causing credentials to never be refreshed * Run Singularity in wrapper's process * Note that we don't mirror all the Singularity calls we should * Also install wheel in the virtualenvs * Skip over things that aren't found when shutting down * Revert "Skip over things that aren't found when shutting down" This reverts commit 680c632. * Stop deleting all ASGs with tags * Address Lon's review comments * Make logging more verbose so we can fix the missing strings * Log more * Don't treat no provisioner as unrecognized * Skip CWL 1.0 and 1.1 tess because 1.2 covers them Co-authored-by: Adam Novak <[email protected]>
- Loading branch information
1 parent
1c7b6f2
commit 52cb9ca
Showing
14 changed files
with
409 additions
and
114 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Wrapper for Singularity that rewrites docker:// image specifiers that point to Docker Hub to use a registry mirror instead. | ||
| # Obeys SINGULARITY_DOCKER_HUB_MIRROR for the URL, including http:// or https:// protocol. | ||
|
|
||
| set -e | ||
|
|
||
| # Where is the real Singularity binary? | ||
| # The Dockerfile moves it from /usr/local/bin/singularity to here after installation | ||
| SINGULARITY_PATH=/usr/local/libexec/toil/singularity-real | ||
|
|
||
| # Read replacements from the environemnt | ||
| # TODO: really do that | ||
| MIRROR_HOST="" | ||
| MIRROR_HTTP=0 | ||
|
|
||
| if [[ ! -z "${SINGULARITY_DOCKER_HUB_MIRROR}" ]] ; then | ||
| MIRROR_HOST="${SINGULARITY_DOCKER_HUB_MIRROR##*://}" | ||
| MIRROR_PROTO="${SINGULARITY_DOCKER_HUB_MIRROR%%://*}" | ||
| if [[ "${MIRROR_PROTO}" == "http" ]] ; then | ||
| MIRROR_HTTP=1 | ||
| fi | ||
| fi | ||
|
|
||
| # Collect command line arguments | ||
| ARGC=$((${#} + 1)) | ||
| ARGV=($0 "${@}") | ||
|
|
||
| if [[ "${ARGC}" -ge "2" && "${ARGV[1]}" == "pull" && ! -z "${MIRROR_HOST}" ]] ; then | ||
| # We are doing a pull | ||
| # TODO: we will want to also catch `singularity build` | ||
| # TODO: we will want to be able to handle e.g. `singularity -q build` | ||
|
|
||
| # We will set this if we manage to replace a Docker name | ||
| REPLACED=0 | ||
|
|
||
| INDEX=2 | ||
| while [[ "${INDEX}" -lt "${ARGC}" ]] ; do | ||
| # For each argument other than the script name | ||
| if [[ "${ARGV[$INDEX]}" == docker://* ]] ; then | ||
| # If it doesn't have a / after the protocol, it needs "library/" inserted | ||
| NEW_SPEC="$(echo "${ARGV[$INDEX]}" | sed 's!^docker://\([^/][^/]*$\)!docker://library/\1!')" | ||
|
|
||
| # If it doesn't have a hostname with a dot before the first /, give it our hostname | ||
| NEW_SPEC="$(echo "${NEW_SPEC}" | sed 's!^docker://\([^.][^.]*/\)!docker://'${MIRROR_HOST}'/\1!')" | ||
|
|
||
| # Replace array item | ||
| ARGV[$INDEX]="${NEW_SPEC}" | ||
| REPLACED=1 | ||
| fi | ||
|
|
||
| let INDEX+=1 | ||
| done | ||
|
|
||
| # We will set this if we need to insert --nohttps for an insecure registry | ||
| HTTP_ARG="" | ||
|
|
||
| if [[ "${REPLACED}" == "1" && "${MIRROR_HTTP}" == "1" ]] ; then | ||
| # We need to use HTTP and not HTTPS for the mirror, so we need to isnert the argument | ||
| HTTP_ARG="--nohttps" | ||
| fi | ||
|
|
||
| # Run the pull with our extra args, and then all the args starting at 2 | ||
| # Run it as the current process so it gets signals intended for it | ||
| exec "${SINGULARITY_PATH}" pull ${HTTP_ARG} "${ARGV[@]:2}" | ||
| else | ||
| # Pass along all the args except the program name | ||
| # Run it as the current process so it gets signals intended for it | ||
| exec "${SINGULARITY_PATH}" "${ARGV[@]:1}" | ||
| fi | ||
|
|
||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.