Possible issue with CWL workflow with ".json" ending?

[mr-c]

Description

Maybe due to ".json" file extension?

Expected Behavior

Workflow rendered

Actual Behavior

Workflow or git commit could not be found

Possible Fix

Steps to Reproduce

Visualize
https://github.com/m-gall/registry-website/blob/master/database-build/qimr-somatic.json
→ https://view.commonwl.org/workflows/github.com/m-gall/registry-website/blob/master/database-build/qimr-somatic.json

[Anushka-shukla]

I'm trying to understand this issue I have used this link "https://github.com/m-gall/registry-website/blob/master/database-build/qimr-somatic.json" to see if the workflow is being generated or not right? All of these workflow links are present on "https://view.commonwl.org/workflows". When I'm searching for this workflow I'm not getting it.

[mr-c]

I'm trying to understand this issue I have used this link "https://github.com/m-gall/registry-website/blob/master/database-build/qimr-somatic.json" to see if the workflow is being generated or not right? All of these workflow links are present on "https://view.commonwl.org/workflows". When I'm searching for this workflow I'm not getting it.

If we enter the example URL https://github.com/m-gall/registry-website/blob/master/database-build/qimr-somatic.json in at https://view.commonwl.org/ under "URL to workflow" we get the error message "Workflow or git commit could not be found". It does not work at all. The point of this feature request is add the ability to accept a direct link to a CWL workflow in the "URL to the workflow" box.

[Anushka-shukla]

I tried understanding this and the only thing that I could understand was we need a ".cwl" file for the workflow to get rendered. I checked few more links form I could see there was the path of ".cwl" file was provided. While for this one https://github.com/m-gall/registry-website/blob/master/database-build/qimr-somatic.json it's missing.

Does this mean we have to create a cwl file for this one so that the workflow works? Am I thinking is correct way?

[mr-c]

I think the example URL I used was confusing, so I have copied the file to a new location.

Accessing https://people.debian.org/~crusoe/qimr-somatic.json gives you the raw CWL description.

Normally with view.commonwl.org we do not provide a raw URL, but a GitHub/GitLab repository HTTP(S) URL or a git URI. For example: https://github.com/common-workflow-language/workflows/tree/lobstr-v1/workflows/lobSTR/lobSTR-workflow.cwl

That link, if you visit it directly with wget or curl give you an HTML rendering of the GitHub interface and includes the contents of the lobSTR-workflow.cwl file.

As opposed to our earlier example where the URL was directly the CWL description, no HTML interface.

Does this make more sense?

[Anushka-shukla]

Sorry, I'm still not able to understand. I could understand we have to provide the GitHub/GitLab repository HTTP(S) URL in order to get the workflow generated. Is the .json file is still acceptable to generate the workflow? I mean this one https://github.com/m-gall/registry-website/blob/master/database-build/qimr-somatic.json

[tetron]

@mr-c I am also confused.

It seems like there are two things being discussed here.

The first one is giving it a github link to a JSON file. It apparently won't recognize the JSON as CWL even though it could be valid CWL. I assume cwlviewer has special pattern matching for github and gitlab URLs, and it clones the repository instead of expecting to fetch the CWL text over plain http URL.

The second is the ability to take a plain http URL and resolve the workflow from starting from the starting file.

Do we need to split this into two tickets?

[mr-c]

@tetron Sorry for the confusion, you are correct. I was confusing this issue with #210

[tetron]

Strictly speaking #210 isn't the same either, that is about accepting packed workflows, I'm talking about resolving a workflow over HTTP using relative URLs.

[mr-c]

about accepting packed workflows,

that are not hosted on git, on plain HTTP

resolving a workflow over HTTP using relative URLs

Sure, that would be the next step after accepting workflows via HTTP instead of Git

(resolving a workflow via Git using relative URLs already works)

[Anushka-shukla]

@mr-c I am also confused.

It seems like there are two things being discussed here.

The first one is giving it a github link to a JSON file. It apparently won't recognize the JSON as CWL even though it could be valid CWL. I assume cwlviewer has special pattern matching for github and gitlab URLs, and it clones the repository instead of expecting to fetch the CWL text over plain http URL.

The second is the ability to take a plain http URL and resolve the workflow from starting from the starting file.

Do we need to split this into two tickets?

How do I go about solving this issue? Can you please direct?

[Anushka-shukla]

As .json is not rendering the workflow there might be a reason that validation is only not happening. I could see from here:

cwlviewer/src/main/java/org/commonwl/view/workflow/WorkflowFormValidator.java

Line 41 in 4ce34de

// URL validation for cwl files on Github

there is no URL validation happening for URLs ending with ".json" can this be a reason that workflow is not generating?
I'm still understanding where exactly the fix is required.

[tetron]

@Anushka-shukla yes, I think that is right, it is enforcing that the file ends in .cwl. You probably either want to change it to (\\.cwl|\\.json) or remove \\.cwl from the regular expression entirely so it does not enforce a particular file extension.

[Anushka-shukla]

Okay, I'll start working on this one. Thanks for confirming.

[Anushka-shukla]

Hey @tetron, I tried both ways but nothing really changed. And tried changing with other URLs but no luck.