Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge tesk-core project into source/tesk-core (#142)
* maintenance: restructure repo for hosting tesk-core * maintenance: make the package buildable again Fixed scm_setuptools configuration Tweaked tox configuration too * feature: add travis for CI * ntenance: add CI badges * maintenance: remove examples folder reason: should be centralized in the tesk repo * create ftp parent if not existing * actually make dir * upload directly to target iso creating new dir * remove pycurl experiment * turn on debug * remove debug msg * force debug through code * revert * always pull filer * correct syntax * create dir only * add some print to debug * make parent dir for file uploads too * create dir, not file * correct format * use debugging log for testing branch * remove local config loading on debug * new transput filer * rc1 of transput filer * Update README.md * use transput filer * add debug bits * remove pvc upon completion * suppport content field * add transput filer file * add taskmaster architecture * Create README.md * Update README.md * mv * Update README.md * maintenance: Use python's standard formatting autopep8 was used for applying the formatting, a few manual changes were made to make some changes more pallatable * fixes: closing files & conns; regex bugs Now files and connections are more consistently closed. Most regular expressions produced bugs and have been replaced. Transput class now contains all the virtual functions that are to be inherited. Logging messages are lazy-loaded now. Uploading HTTP folders now tries to upload all the files, not just the last one found. Renamed some variables to make their purpose more obvious Stop raising exceptions for the time being as the rest of the code does not handle them, return the error code instead * maintenance: less biolerplate to close connections Use transputs as context managers to be less verbose for closing connections * maintenance: use enums instead of strings This way the code won't fail in subtle ways * maintenance: add urllib as dependency * enhancement: minimize number of connections * fix: install backported enum support * fix: make filer usable on python < 3.6 * fix: convert from string to Enum Also added more debugging info * style: write more information on error messages * maintenance: update vulnerable dependencies * rm filer.py According to Ania, this is the old version that is not used anymore. * [#12] newTransput() * [#12] assertThrows() * [#12] assertThrows(): extracted to a separate class, in its own file. * [#12] FileTransput: dummy class + factory * [#12] FileTransput.download_file() TODO It's still using the host path. This has to be translated to a path inside the container. * [#12] containerPath() * [#12] validatePath() * [#12] ParserTest.test_defaults() * [#12] --localKubeConfig * [#12] ren transput_filer.py => filer.py * dockerBuild * dockerRun * [#12] ren transput_filer.py => filer.py (continued...) I forgot to rename in one of the tests. * [#12] Downgrading to Python 2 :'( * init * install * init: Activating virtualenv before ./install * src/: creating and moving tesk_core/ to it. * dockerBuild: - moved to scripts/ - it was always executing 'taskmaster' (hard-coded) * dockerRun: moving to scripts/ * dockerRun: forwarding cmd line args to the image. Before, they were simply discarded. * [#12] taskmaster: passing env vars HOST_BASE_PATH and CONTAINER_BASE_PATH to filer * scripts/run * [#12] install: no longer creating links Why? Because that way, './taskmaster' wouldn't run the current code: it would run _the last installed code_ ('pip install .'). I.e. one more thing for you to remember. If you forgot about this, it could easily lead to hours of head scratching and fruitless debugging. * [#12] Adding mount to CONTAINER_BASE_PATH ('/transfer') * test_mounts * test_run_task * [#12] ren set_volume_mounts() => add_volume_mount() * [#12] mount: the 'volumes' part * [#12] jobRunToCompletionMock(): printing the job name * [#12] /transfer mount: minor errors in the spec * 'hostPath' : { 'path' : '/transferAtNode' } * .append() => .extend() * run: deleting everything before + examples/ (everything = pods + jobs + pvc) * [#12] pprint * [#12] minor errors in the spec (21ca6e7): fixing broken test * [#12] 'Creating job...': adding the job name (production code) * [#12] transfer volume: hostPath => PVC * [#12] FileTransput.upload_dir() * [#12] upload/download methods: refactoring (transfer()) def download_file(self): self.transfer(shutil.copy , self.urlContainerPath , self.path) def upload_dir(self): self.transfer(shutil.copytree , self.path , self.urlContainerPath) * [#12] 'file:' protocol disabled if any of (HOST_BASE_PATH, CONTAINER_BASE_PATH) is undefined. * [#12] transfer mount: making it optional The relevant conf is added to filer spec only if both (HOST_BASE_PATH, CONTAINER_BASE_PATH) are defined. * [#12] FileTransput.upload_file() * [#12] FileTransput.download_dir() * run: parameterizing jsonFile * [#12] process_file(): removing '!= 0' checks from call sites. * [#12] copyDir() Limitation of shutil.copytree: > The destination directory, named by dst, must not already exist; it will be created as well as missing parent directories. * Reverting: if debug => imagePullPolicy = Always * [#12] TRANSFER_VOLUME_NAME (new env var) * [#12] Fix: TRANSFER_VOLUME_NAME => TRANSFER_PVC_NAME I parameterized the wrong constant. It should have been 'transfer-pvc' -- not 'transfer-volume'. * [#12] --pull-policy-always * [#12] --pull-policy-always: cmd line arg * [#12] --pull-policy-always: binding cmd line arg => what ends up in Filer() Plus: changed Filer() param from string => boolean. * Removing Python 2 support Dropping Python 2 support from the project: - setup.py no longer advertises Python 2 in supported interpreters - tox.ini does not use py27 environment - renamed some unit tests to be automatically picked up by pytest - pytest is meant to be used through tox (not from setup.py) for unit tests - removed unnecessary Python 2 dependencies (future, unittest2, enum34) - bumped up kubernetes to 6.1.0 because of syntax error when using 5.0.0 in Python 3.7 - using entry points instead of script names (renamed filer.py to filer) * Removing Python 2 dependencies from test_filer.py * Building Docker images with Python 3.7 - changed the base image to Alpine 3.10 with Python 3.7. - two staged build: (1) build wheel (2) install wheel and its dependencies - system-dependent Python directory layout knowlegde is no longer needed - docker will now ignore expensive .tox and other build directories while building images - image entrypoints edited to match "filer" and "taskmaster" installed binaries * Travis - set minimum Python version to 3.5 * Docker WORKDIR fixes, CI improvements - Removed WORKDIR from Dockerfiles - Python 3.7 added to Travis - Added extra Pylint step to fail on errors (E) only * Allow volumes (e.g. secrets) in the executor spec * Allow changing the default filer image name using -fn * Could not determine protocol, assuming 'file' * Limitations of shutil.copy: If does not interpret * as a glob, but as a character. * Fix indentation, add one space * Fix and add test Now a different function is used in the code. Also a couple of tests were added to increase coverage * fix(ftp): stopped ftp errors silencing FTP errors were not propagated, resulting in jobs completing while FTP errors were present (#20) * fixed spacing * feat(api): variable jobs backoff limits By defining new environment variables, TESK_API_FILER_ENVIRONMENT_BACKOFF_LIMIT and TESK_API_EXECUTOR_ENVIRONMENT_BACKOFF_LIMIT, the number of a retires before a job fails can be explicitly set * env vars renaming * bug fix: ImagePullErr * Update job.py * Add support to netrc in filer.py. If the environment variable 'NETRC_SECRET_NAME' is present, and the content of the secret is a valid netrc file, it will be used for authentication. * Catch the KeyError in case the HOME environment variable is not defined * Fix local filesystem regression of error checking * TESK API sends environment variables without TESK_API_TASKMASTER_ENVIRONMENT_ prefix * String/int conversion * added unit testing for taskmaster and job * Chore: Update kubernetes client to 9.0.0 and dep urllib * s3 credentials through secrets First try, untested code * fix for s3 credentials through secret Tested and working. Secret name is hardcoded. * add extract endpoint utility Since the 'endpoint_url' parameter inside the aws config file is not handled by boto, we need a simple utility module to extract it. * utility improvement Improvements for the extract_endpoint utility * added tests for utility and secret volume * fixed test_config dir * Try with the mount elsewhere and postStart * Fix UnboundLocalError * enrich filer tests Some simple unit tests test additions additional tests (pytest) Created more tests for ftp filer operations with pytest in a separate file additional tests for ftp, http removed print statement updated setup.py test dependencies enable Travis filer unit tests improvements Files are no longer created during tests, enhanced testing for ftp_download tests fixes removed file exists test Removed f-strings for python3.5 compliance http upload dir test add fs fixture correct fixture name added fs in test deps ignore calls order test for ftp upload dir add pytest config mock ftplib connect added ftp download dir test removed ftpserver.stop and redundant login, connect removed unnecessary dependencies filer tests enhancements tests adjustments for rebase file creation assertion & commented code removal assert mock_open called_with() fixed duplicate code issue * feat: dynamic storage provisioning The storage class used is now configurable through the use of the TESK_API_TASKMASTER_ENVIRONMENT_STORAGE_CLASS_NAME environment variable. If no value is provided, the default storage class will be used. (#111) * Move HOME for .netrc (#32) And we will move HOME for .netrc to fix #31 * Bump urllib3 from 1.22 to 1.24.2 Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.22 to 1.24.2. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/master/CHANGES.rst) - [Commits](urllib3/urllib3@1.22...1.24.2) Signed-off-by: dependabot[bot] <[email protected]> * added more testcases * Adding s3 support using boto3 * fixing bug in testcases * bug fix in tests * bug fix * moving s3 testcases to new file and reverting urllib version * downgrading boto3 * big fix * Allow unit tests to run under python3.8 * Remove Python3.5 support from unit tests * Add Python3.8 to travis testing and remove 3.5 * Updating taskmaster.py, job.py and pvc.py * adding testcases and condition to check particular exceptions * more logging * Bump urllib3 from 1.24.2 to 1.26.5 Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.24.2 to 1.26.5. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.24.2...1.26.5) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Install RUST compiler, upgrade version of boto3 and upgrade ubuntu dist * Add tesk-core-filer workflow * Add taskmaster workflow * Fix UID 1000 * Add user 100 to filer and taskmaster * Remove v3.6, add v3.7 * Upgrade versions of modules * Upgrade versions of modules * Add support for JSON_INPUT in gzipped configMap (#41) * Bump urllib3 from 1.26.5 to 1.26.18 Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.5 to 1.26.18. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.5...1.26.18) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Remove deprecated python versions and add the new ones * Add github action for tox tests * Fix quote * Fix config * Update how endpoint is handle * Fix some tests * Upgrade boto3 and tests * Add boto3 client mock * Make a bit better the scripts --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Pau Ruiz i Safont <[email protected]> Co-authored-by: erikvdbergh <[email protected]> Co-authored-by: Thiago Albuquerque <[email protected]> Co-authored-by: Zcowyzrg <[email protected]> Co-authored-by: Ania Niewielska <[email protected]> Co-authored-by: Kostas Liosis <[email protected]> Co-authored-by: cibin S B <[email protected]> Co-authored-by: Martin Rippin <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Clark-Casey <[email protected]> Co-authored-by: Cibin Sadasivan Baby <[email protected]> Co-authored-by: Kostis Zagganas <[email protected]> Co-authored-by: Milan <[email protected]> Co-authored-by: Alex Kanitz <[email protected]> Co-authored-by: xhejtman <[email protected]> Co-authored-by: Tristan <[email protected]> Co-authored-by: Tristan <[email protected]>
- Loading branch information
18 people
authored
Feb 20, 2024
1 parent
83b9c38
commit 2439f58