Suggestions to update the look of the workflow catalog

[m-jahn]

Hi fellow Snakemake and catalog users,

I would like to use this issue here to discuss about some changes that we can implement to make the Catalog more user-friendly, informative, and with an appealing design.

The problem

• as I have discussed with @johanneskoester recently, the catalog's design is not very interactive
• it also lacks deeper information about the individual workflows
• it's not clear for some users how and when the workflows get added or updated, and if not, why
• as the popularity of snakemake has grown, so has the catalog, and it needs to collect and hold information from more than 3000 github repos. This poses performance issues in the future.

Suggestions for an update

Right now the catalog's information is retrieved by crawling through eligible github repos. I would call this the back-end, and I don't think we need to change a lot there. In the future we could give users more options to include structured information from their workflows. One featrure could be a workflow outline/map in png or svg format, that is included in the individual workflow pages.

This brings me to the 'front-end' of the catalog, where we could actually change a lot. If we look at nf-core, they provide quite a detailed page for every pipeline. We could attempt something similar (of course with different prerequisites), but at least move away from the catalog being only a table. An updated catalog should have:

• an attractive front page
• a table or tile-like overview about the (top) workflows sorted by # stars etc
• single pages for each workflow
• easy integration into the current back-end
• simple web site deployment on Github Pages
• a free and open source page building framework

I have looked into the following options that can be used together with github pages:

• Jekyll -- often used for blogs and home pages, not really suitable to hold a lot of information
• Astro -- used for nf-core page. very flexible and can be customized to work as a catalog. I have tried the starlight for documentation, but navigation can not be customized
• Docusaurus -- an MIT licensed framework specifically for documentation websites. It turned out that it's very easy to set up, navigation can be customized. Workflows can be presented as 'Cards' (tiles) on front page. All workflows can get an individual markdown page, which just needs to be dumped in a folder. Menu structure is automatically created from the subdirs in docs.

I have created a test website on my github account with exemplary content from the workflow catalog back-end (10 entries). I have also added some standalone documentation pages and workflow overview pages with 'Cards'. These were all main features I wanted to test and they work well. Deployment is also easy with a premade Github Actions workflow.|

Here are some screenshots.

Front page:

Workflow overview

Single workflow page

Conclusion

• I have tested Docusaurus as a potential framework to build the catalog website and it brings all the features one can wish.
• it integrates well with the current back-end: all that needs to be done is to export one md page per workflow
• in order to not flood a future catalog with too many entries, I suggest to focus on standardized usage workflows
• these are anyway the workflows whose creators want them to be included in the catalog
• I tried a lot to make the test page look like the real website, just to make sure that Docusaurus is really suitable -- this doesn't have to be the final design!
• my apologies for this long post; please comment and make suggestions!

[johanneskoester]

Thanks a lot for trying this out!
First thoughts:

1. While I like having one dedicated page per workflow, I also like the tabular view, because it provides a better overview of all the workflows in a condensed way.
2. I definitely want to keep the scraping of all workflows, also the non-standardized. But indeed, as you say, the non-standardized ones don't need a page, just the table.
3. One issue with changing the framework is that this would also alter the URLs under which you can reach the workflow. Many of them have been already linked out somewhere via these URLs and changing that would break all those links.
4. I think we could achieve something similar to this docusaurus approach by simply modifying the HTML of the current catalog a bit. Basically having a full page per workflow just requires to render the current modal dialog differently. That one could then also be reformatted or extended with additional information. The advantage of this is that URLs used so far remain valid.

[m-jahn]

Hi Johannes,

Thanks for your feedback! Regarding some of the concerns you raised:

• about the links, yes that's indeed an issue. I haven't thought about that. One solution would be to include a redirection from the old format to the new format when building the site, that should be possible. I'm not sure if the links from the WF readme to the catalog are that important though, I've never used them
• table with all workflows instead of just featured ones: yes absolutely possible
• regarding the general frame work: I can see that you're hesitant switching to a different framework. But there's a lot to win and not so much to lose. I see on the plus side a much fresher look, many more features regarding cross linking and site layout, and most of all, modularization. Right now all functions, styles, and text are inside a single html, which is at least for me not possible to go through and make sensible changes. The modern react based frameworks all have a similar, more tractable structure, like separate files for CSS styling, front matter, and actual content, somewhat like this:

site
├── docs
│   ├── doc1.md
│   ├── doc2.md
├── src
│   ├── css
│   └── pages
│       ├── styles.module.css
│       └── index.js
├── static
│   └── img
│       └── image.png
├── docusaurus.config.js
├── README.md

This makes it IMO much easier to maintain, and also easier for other people to update.
Just think about it if this could be an option. The test page is on my github.

[johanneskoester]

Indeed, I see your point. We could move to a framework like that, but then I think we should ideally stick to the same that we use for the main docs and the plugin catalog (this way, we have completely the same styling, and can also simultaneously benefit from any improvements in that setup). That would be sphinx plus the sphinxawesome theme, in combination with myst for using markdown with sphinx.

What however definitely remains to be verified is whether such an approach integrates well with a datatable of the workflows and if we are really able to keep the URLs that point to the usage docs of individual workflows we had so far.

[johanneskoester]

The sources for the plugin catalog can be found here: https://github.com/snakemake/snakemake-plugin-catalog

[m-jahn]

Sphinx could be a viable option. I like the clean look. I fully agree that the different snakemake documentation pages should have a similar if not identical "corporate design". I can look into this as soon as I find time.

[johanneskoester]

Cool!

[mschilli87]

Could this change be the cause of snakemake-workflows/snakemake-workflow-template#11?