feat: test of adding Gitlab provider #27

vsoch · 2022-01-28T00:11:39Z

This will be probably a back and forth between myself and @nate-d-olson to test his particular issue #26. I think the first logical thing to try is to add a GitLab provider (the PR here) and then go from there!

@nate-d-olson if you'd care to test out this branch and report back, we can do exactly that :)

Signed-off-by: vsoch [email protected]

Signed-off-by: vsoch <[email protected]>

vsoch · 2022-01-28T00:13:51Z

I always struggle with the title 😆

nate-d-olson · 2022-01-28T18:05:24Z

Thanks @vsoch! the deploy workflow command worked but the resulting workflow/Snakefile uses github() instead of gitlab() for defining the code hosting provider, https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html#code-hosting-providers. I manually changed the command to gitlab("njd2/giab-asm-bench-whole-genome", path="Snakefile", tag="v0.003", host="gitlab.nist.gov/gitlab").

I am getting the following error when I try running the pipeline using snakemake --use-conda --dryrun -j 1

SyntaxError in line 1 of https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome/-/raw/v0.003/Snakefile:
invalid syntax
  File "/workspaces/test-deploy/workflow/Snakefile", line 18, in <module>

When I try downloading the Snakefile from https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome/-/raw/v0.003/Snakefile using wget I get an html file instead of the Snakefile. I don't know if this is due to the specific settings on our internal gitlab repository for more general to hosted gitlab repos. If this error is specific to our internal gitlab server configuration I am happy to use snakedeploy with a path to a local git repo.

vsoch · 2022-01-28T19:17:30Z

When I try downloading the Snakefile from https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome/-/raw/v0.003/Snakefile using wget I get an html file instead of the Snakefile. I don't know if this is due to the specific settings on our internal gitlab repository for more general to hosted gitlab repos. If this error is specific to our internal gitlab server configuration I am happy to use snakedeploy with a path to a local git repo.

I can't see that file, but I'd suspect it's the html view and not raw? Does GitLab have an equivalent raw view like GitHub? If you can find the path to the "raw file" could you share with me? We will need to make sure that is represented in the provider (right now it matches GitHub).

I am getting the following error when I try running the pipeline using snakemake --use-conda --dryrun -j 1

I can fix the directive to be gitlab but I think we might need to add it as acceptable syntax for a snakefile - I can look into that over this weekend.

vsoch · 2022-01-28T19:20:56Z

It does look like there is support for a GitlabFile (if I'm looking at the correct thing) but it might be new - https://github.com/snakemake/snakemake/blob/6e2ecd26e48eb64fa04c9c38dde591857e03c722/snakemake/sourcecache.py#L229

I suspect there is support for gitlab but we are just grabbing the wrong "raw url" so if you can send me that it would be great!

nate-d-olson · 2022-01-28T19:30:36Z

Here is the file I get when I use wget https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome/-/raw/v0.003/Snakefile. It looks like instead of downloading the file the url is redirected to the login page for our internal gitlab server. I generally avoid debugging this type of issue and am okay using local files. If this is an issue for other snakedeploy users and useful for you to debug I am happy to help with testing.

<!DOCTYPE html>
<html class="devise-layout-html">
<head prefix="og: http://ogp.me/ns#">
<meta charset="utf-8">
<link as="style" href="https://gitlab.nist.gov/gitlab/assets/application-969119f639d0837f445a10ced20d3a82d2ea69d682a4e74f39a48a4e7b443d5e.css" rel="preload">
<link as="style" href="https://gitlab.nist.gov/gitlab/assets/highlight/themes/white-0678dba52a34ddc3b009cf294c54cfbddb9bac5991d6377ab811afe156e5a0cb.css" rel="preload">

<meta content="IE=edge" http-equiv="X-UA-Compatible">

<meta content="object" property="og:type">
<meta content="GitLab" property="og:site_name">
<meta content="Sign in" property="og:title">
<meta content="Gitlab.nist.gov" property="og:description">
<meta content="https://gitlab.nist.gov/gitlab/assets/gitlab_logo-7ae504fe4f68fdebb3c2034e36621930cd36ea87924c11ff65dbcb8ed50dca58.png" property="og:image">
<meta content="64" property="og:image:width">
<meta content="64" property="og:image:height">
<meta content="https://gitlab.nist.gov/gitlab/users/sign_in" property="og:url">
<meta content="summary" property="twitter:card">
<meta content="Sign in" property="twitter:title">
<meta content="Gitlab.nist.gov" property="twitter:description">
<meta content="https://gitlab.nist.gov/gitlab/assets/gitlab_logo-7ae504fe4f68fdebb3c2034e36621930cd36ea87924c11ff65dbcb8ed50dca58.png" property="twitter:image">

<title>Sign in · GitLab</title>
<meta content="Gitlab.nist.gov" name="description">
<link rel="shortcut icon" type="image/png" href="/gitlab/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png" id="favicon" data-original-href="/gitlab/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png" />

<link rel="stylesheet" media="all" href="/gitlab/assets/application-969119f639d0837f445a10ced20d3a82d2ea69d682a4e74f39a48a4e7b443d5e.css" />
<link rel="stylesheet" media="all" href="/gitlab/assets/themes/theme_indigo-1e3c170ae7fd24d137960957cba8221abf63a78f8b108e77f131b0fed6a659c7.css" />


<link rel="stylesheet" media="all" href="/gitlab/assets/highlight/themes/white-0678dba52a34ddc3b009cf294c54cfbddb9bac5991d6377ab811afe156e5a0cb.css" />

<script>
//<![CDATA[
window.gon={};gon.features={"webauthn":false};
//]]>
</script>


<script src="/gitlab/assets/webpack/runtime.cf03c281.bundle.js" defer="defer"></script>
<script src="/gitlab/assets/webpack/main.51b8c03b.chunk.js" defer="defer"></script>
<script src="/gitlab/assets/webpack/commons-pages.admin.sessions-pages.ldap.omniauth_callbacks-pages.omniauth_callbacks-pages.profiles.t-f04c18ab.8bbc73af.chunk.js" defer="defer"></script>
<script src="/gitlab/assets/webpack/commons-pages.admin.sessions-pages.sessions-pages.sessions.new.1e0c0ee2.chunk.js" defer="defer"></script>
<script src="/gitlab/assets/webpack/pages.sessions.new.444bf31e.chunk.js" defer="defer"></script>



<meta name="csrf-param" content="authenticity_token" />
<meta name="csrf-token" content="S0kZbQRIf0uAWUNBllz/1o9V9nBlXbTvAMkpiN55iH4Ot2HZPNcEpnkKgQdQZlyo1sDHbP0DczAGUSE043ZUyg==" />

<meta name="action-cable-url" content="/gitlab/-/cable" />
<meta content="origin-when-cross-origin" name="referrer">
<meta content="width=device-width, initial-scale=1, maximum-scale=1" name="viewport">
<meta content="#474D57" name="theme-color">
<link rel="apple-touch-icon" type="image/x-icon" href="/gitlab/assets/touch-icon-iphone-5a9cee0e8a51212e70b90c87c12f382c428870c0ff67d1eb034d884b78d2dae7.png" />
<link rel="apple-touch-icon" type="image/x-icon" href="/gitlab/assets/touch-icon-ipad-a6eec6aeb9da138e507593b464fdac213047e49d3093fc30e90d9a995df83ba3.png" sizes="76x76" />
<link rel="apple-touch-icon" type="image/x-icon" href="/gitlab/assets/touch-icon-iphone-retina-72e2aadf86513a56e050e7f0f2355deaa19cc17ed97bbe5147847f2748e5a3e3.png" sizes="120x120" />
<link rel="apple-touch-icon" type="image/x-icon" href="/gitlab/assets/touch-icon-ipad-retina-8ebe416f5313483d9c1bc772b5bbe03ecad52a54eba443e5215a22caed2a16a2.png" sizes="152x152" />
<link color="rgb(226, 67, 41)" href="/gitlab/assets/logo-d36b5212042cebc89b96df4bf6ac24e43db316143e89926c0db839ff694d2de4.svg" rel="mask-icon">
<meta content="/gitlab/assets/msapplication-tile-1196ec67452f618d39cdd85e2e3a542f76574c071051ae7effbfde01710eb17d.png" name="msapplication-TileImage">
<meta content="#30353E" name="msapplication-TileColor">




</head>

<body class="application gl-browser-generic gl-platform-other login-page navless ui-indigo" data-page="sessions:new" data-qa-selector="login_page">

<script>
//<![CDATA[
gl = window.gl || {};
gl.client = {"isGeneric":true,"isOther":true};


//]]>
</script>
<div class="page-wrap">
<header class="navbar fixed-top navbar-empty">
<img class="brand-header-logo lazy" data-src="/gitlab/uploads/-/system/appearance/header_logo/1/simpLogo2.png" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" />
</header>

<div class="login-page-broadcast">


</div>
<div class="container navless-container">
<div class="content">
<div class="flash-container flash-container-page sticky">
<div class="flash-alert mb-2">
<svg class="s16 align-middle mr-1" data-testid="error-icon"><use xlink:href="/gitlab/assets/icons-795a2ef2fd636a0538bbef3b8d2787dd90927b42d7617fdda8620930016b333d.svg#error"></use></svg>
<span>You need to sign in or sign up before continuing.</span>
<div class="close-icon-wrapper js-close-icon">
<svg class="s16 close-icon gl-vertical-align-baseline!" data-testid="close-icon"><use xlink:href="/gitlab/assets/icons-795a2ef2fd636a0538bbef3b8d2787dd90927b42d7617fdda8620930016b333d.svg#close"></use></svg>
</div>
</div>
</div>

<div class="row mt-3">
<div class="col-sm-12">
<h1 class="mb-3 font-weight-normal">
Gitlab.nist.gov
</h1>
</div>
</div>
<div class="row mb-3">
<div class="col-sm-7 order-12 order-sm-1 brand-holder">
<img data-src="/gitlab/uploads/-/system/appearance/logo/1/logo-2x.png" class="lazy" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" />
<p data-sourcepos="1:1-1:136" dir="auto">If you want a public location to put your source code, check out <a href="https://github.com/usnistgov" rel="nofollow noreferrer noopener" target="_blank">https://github.com/usnistgov</a>  For help contact <a href="mailto:[email protected]">[email protected]</a></p>
<p data-sourcepos="1:1-1:218" dir="auto">Login with your normal NIST username and password to access, an account is automatically created for you. It is recommended to use Chrome to access the application as some versions of IE are not compatible with GitLab.</p>&#x000A;<p data-sourcepos="3:1-3:820" dir="auto">You are accessing a U.S. Government information system, which includes: 1) this computer, 2) this computer network, 3) all computers connected to this network, and 4) all devices and storage media attached to this network or to a computer on this network. You understand and consent to the following: you may access this information system for authorized use only; you have no reasonable expectation of privacy regarding any communication of data transiting or stored on this information system; at any time and for any lawful Government purpose, the Government may monitor, intercept, and search and seize any communication or data transiting or stored on this information system; and any communications or data transiting or stored on this information system may be disclosed or used for any lawful Government purpose.</p>&#x000A;<p data-sourcepos="5:1-5:174" dir="auto">By logging into and using this service, you are agreeing to adhere to the <a href="https://gitlab.nist.gov/gitlab/nist/help/raw/master/rob">NIST Gitlab Rules of Behavior</a>.</p>

</div>
<div class="col-sm-5 order-1 order-sm-12 new-session-forms-container">
<div id="signin-container">
<ul class="custom-provider-tabs nav nav-links nav-tabs new-session-tabs">

<li class="nav-item">
<a class="nav-link active" data-toggle="tab" data-qa-selector="ldap_tab" href="#ldapmain">NIST</a>
</li>

</ul>

<div class="tab-content">

<div class="active login-box tab-pane" id="ldapmain" role="tabpanel">
<div class="login-body">
<form id="new_ldap_user" class="gl-show-field-errors" action="/gitlab/users/auth/ldapmain/callback" accept-charset="UTF-8" method="post"><input name="utf8" type="hidden" value="&#x2713;" /><input type="hidden" name="authenticity_token" value="YjJcoY8zNurQPOY6RQ6jXfsed75Mr5azt7IvQyPO5VEnzCQVt6xNBylvJHyDNAAjootGotTxUWyxKif/HsE55Q==" /><div class="form-group">
<label for="username">NIST Username</label>
<input type="text" name="username" id="username" class="form-control top" title="This field is required." autofocus="autofocus" data-qa-selector="username_field" required="required" />
</div>
<div class="form-group">
<label for="password">Password</label>
<input type="password" name="password" id="password" class="form-control bottom" title="This field is required." data-qa-selector="password_field" required="required" />
</div>
<div class="remember-me">
<label for="remember_me">
<input type="checkbox" name="remember_me" id="remember_me" value="1" />
<span>Remember me</span>
</label>
</div>
<div class="submit-container move-submit-down">
<input type="submit" name="commit" value="Sign in" class="btn-success btn" data-qa-selector="sign_in_button" data-disable-with="Sign in" />
</div>
</form>
</div>
</div>


</div>
<div class="clearfix">
<div class="omniauth-container gl-mt-5">
<label class="label-bold d-block">
Sign in with
</label>
<div class="d-flex justify-content-between flex-wrap">
<form class="button_to" method="post" action="/gitlab/users/auth/saml"><button id="oauth-login-saml" class="btn d-flex align-items-center omniauth-btn text-left oauth-login qa-saml-login-button" type="submit"><span>
NIST SAML
</span>
</button><input type="hidden" name="authenticity_token" value="YpwVWDyANUldRsq9+yV7hgMVQ12A5QKWN9ZvsV/CMcQnYm3sBB9OpKQVCPs9H9j4WoByQRi7xUkxTmcNYs3tcA==" /></form></div>
<fieldset class="remember-me">
<label>
<input type="checkbox" name="remember_me" id="remember_me" class="remember-me-checkbox" />
<span>
Remember me
</span>
</label>
</fieldset>
</div>

</div>
</div>

</div>
</div>
</div>
</div>
<hr class="footer-fixed">
<div class="container footer-container">
<div class="footer-links">
<a href="/gitlab/explore">Explore</a>
<a href="/gitlab/help">Help</a>
<a href="https://about.gitlab.com/">About GitLab</a>
</div>
</div>

</div>
</body>
</html>

vsoch · 2022-01-28T19:34:49Z

Ah I see - and I just rendered this locally (without style which I didn't download files for) and indeed the request is not authenticated.

I don't think wget can support beyond a basic --username and --password so likely we would need to do a proper request with some parameters to authenticate, e.g., see here. Let's ping @johanneskoester on this one - have you seen cases like this / are there others trying to access private Snakefile or other assets, and if so, what is the suggested strategy? If not, let's discuss what is best to do.

we should have some discussion about whether non-traditional Snakefile paths are allowed (e.g., in the root as I have added) and if so, what additional changes are needed. Also, if th config directory does not exist should we really add it? It will generally break the workflow. And finally, how do we handle this only working for workflows that have all assets available remotely? E.g., what about local data directories? Signed-off-by: vsoch <[email protected]>

Signed-off-by: vsoch <[email protected]>

vsoch · 2022-02-26T20:54:58Z

okay I've done quite a bit of refactor with respect to the deployment logic: 862f311

And I think the main functionality should mostly work for GitLab and local, however I think we have some things to discuss!

The current snakedeploy does not really handle repos with Snakefile in the root. It will still generate a recipe to look for workflow/Snakefile. I don't see why this has to be the case, so I added a step to look for the correct path and use it. If this absolutely won't work, I can remove it, but instead we should fail the deploy and clearly state "you need to have the Snakefile in workflow/Snakefile"
The local deployment I'm not sure can easily handle a specific tag or branch because we already have the clone, and I'm not sure snakedeploy should take responsibility for trying to check out / fetch the correct tag. We can probably make that more clear.
snakedeploy will probably just work for repos that have all remote assets, is that correct? With a test workflow that requires data/ it didn't work. I guess I'm wondering what the benefits of snakedeploy are over say just git clone and then run with snakemake?

Signed-off-by: vsoch <[email protected]>

johanneskoester · 2022-05-18T10:57:28Z

Hi @vsoch and @nate-d-olson,
sorry I missed this. Regarding your questions:

workflow/Snakefile is the new standard, see the best practice docs. However, it is fine to have a little extra logic for the old style.
Yes, let us not make this too complicated. Local is a corner case anyway. Let us display a warning that tag etc. will be ignored for local.
Unsure what you mean here. Didn't work means there was an error? The repo itself should not contain any data. Snakedeploy is jsut supposed to create a new workflow that points to a remote repo via a module. The configuration of input and so on should be handled via whatever is deployed into the config folder. The advantage over cloning of workflows is that you can more easily upgrade and also push back changes because the deployment and the upstream workflow are no longer the same code base (see the modularization docs for a detailed outline of the benefits).

vsoch · 2022-05-18T17:12:32Z

For 3. I think I had tested a workflow that was missing data (and it wasn't retrieved) but I can't remember it was many months ago! @nate-d-olson do you want to give it another try, or is there a workflow or way to test this out so @johanneskoester can sign and approve? I don't have a good sense if it does the right thing because I haven't used snakedeploy a ton, so I would need your guidance about what to tweak in this PR.

We could also add and wait for others to try and submit feedback. I am good with whatever y'all want to do, I'll be here when you need / want me :)

johanneskoester · 2022-06-14T09:38:09Z

@vsoch, let's just add it.

snakedeploy/deploy.py

very simple test of adding Gitlab provider

6e83087

Signed-off-by: vsoch <[email protected]>

vsoch mentioned this pull request Jan 28, 2022

using snakedeploy with repos not on github #26

Open

vsoch changed the title ~~very simple test of adding Gitlab provider~~ bugfix: test of adding Gitlab provider Jan 28, 2022

vsoch changed the title ~~bugfix: test of adding Gitlab provider~~ feat: test of adding Gitlab provider Jan 28, 2022

vsoch added 2 commits February 26, 2022 13:48

linting

c200916

Signed-off-by: vsoch <[email protected]>

redundant definition of name

ce4e1e9

Signed-off-by: vsoch <[email protected]>

Merge branch 'main' into test/gitlab-provider

4e76114

johanneskoester requested changes Jun 14, 2022

View reviewed changes

snakedeploy/deploy.py Outdated Show resolved Hide resolved

rename to WorkflowDeployer

13a5d5d

johanneskoester approved these changes Jun 22, 2022

View reviewed changes

johanneskoester merged commit 47fc31b into main Jun 22, 2022

johanneskoester deleted the test/gitlab-provider branch June 22, 2022 15:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: test of adding Gitlab provider #27

feat: test of adding Gitlab provider #27

vsoch commented Jan 28, 2022

vsoch commented Jan 28, 2022

nate-d-olson commented Jan 28, 2022

vsoch commented Jan 28, 2022

vsoch commented Jan 28, 2022

nate-d-olson commented Jan 28, 2022

vsoch commented Jan 28, 2022

vsoch commented Feb 26, 2022

johanneskoester commented May 18, 2022

vsoch commented May 18, 2022

johanneskoester commented Jun 14, 2022

feat: test of adding Gitlab provider #27

feat: test of adding Gitlab provider #27

Conversation

vsoch commented Jan 28, 2022

vsoch commented Jan 28, 2022

nate-d-olson commented Jan 28, 2022

vsoch commented Jan 28, 2022

vsoch commented Jan 28, 2022

nate-d-olson commented Jan 28, 2022

vsoch commented Jan 28, 2022

vsoch commented Feb 26, 2022

johanneskoester commented May 18, 2022

vsoch commented May 18, 2022

johanneskoester commented Jun 14, 2022