Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates to migrate to nf-schema #96

Draft
wants to merge 24 commits into
base: dev
Choose a base branch
from
Draft

Updates to migrate to nf-schema #96

wants to merge 24 commits into from

Conversation

DLBPointon
Copy link
Contributor

@DLBPointon DLBPointon commented Feb 21, 2025 •
edited
Loading

#95 Updating from NF-Validation to NF-Schema

This was causing crashes with external users.

Acceptance of PR needs the same done to Blobtoolkit.

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 24, 2025 •
edited
Loading

nf-core pipelines lint overall result: Failed ❌

Posted for pipeline commit 7698680

+| ✅ 170 tests passed       |+
#| ❔  32 tests were ignored |#
!| ❗  69 tests had warnings |!
-| ❌   1 tests failed       |-

❌ Test failures:

  • base_config - conf/base.config contains withName:GET_KMERS_COUNTS, but the corresponding process is not present in any of the Nextflow scripts.

❗ Test warnings:

  • files_exist - File not found: conf/igenomes_ignored.config
  • files_exist - File not found: ro-crate-metadata.json
  • nextflow_config - Config variable not found: validation.help.beforeText
  • nextflow_config - Config variable not found: validation.help.afterText
  • nextflow_config - Config variable not found: validation.summary.beforeText
  • nextflow_config - Config variable not found: validation.summary.afterText
  • nextflow_config - Config manifest.version should end in dev: 0.1.1
  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
  • pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
  • local_component_structure - get_lineage_for_top.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - filter_barcode.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - sed_sed.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - filter_comments.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - get_kmer_counts.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - create_btk_dataset.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - format_diamond_outfmt6.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - ascc_merge_tables.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - blast_get_top_hits.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - kmer_count_dim_reduction.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - sanger_tol_btk.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - kmer_count_dim_reduction_combine_csv.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - check_barcode.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - organelle_contamination_recommendations.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - convert_to_hits_file.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - samtools_depth_average_coverage.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - gc_content.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - chunk_assembly_for_vecscreen.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - validate_taxid.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - summarise_vecscreen_output.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - trailingns.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - filter_vecscreen_results.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - filter_fasta.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - reformat_full_outfmt6.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - blast_v5_database.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - autofiltering.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - generate_samplesheet.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - extract_contaminants.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - parse_fcsgx_result.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - blast_chunk_to_full.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - merge_btk_datasets.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - get_lineage_for_kraken.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - get_largest_scaff.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - run_read_coverage.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - trailingns_check.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - generate_genome.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - extract_tiara_hits.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - pe_mapping.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - organellar_blast.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - se_mapping.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - run_vecscreen.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - run_fcsgx.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - pacbio_barcode_check.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - get_kmers_profile.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - run_fcsadaptor.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - run_diamond.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - run_nt_kraken.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - essential_jobs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - extract_nt_blast.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • nfcore_yml - nf-core version in .nf-core.yml is not set to the latest version. Should be 3.2.0 but was 2.14.1

❔ Tests ignored:

  • files_exist - File is ignored: CODE_OF_CONDUCT.md
  • files_exist - File is ignored: assets/nf-core-ascc_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-ascc_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-ascc_logo_dark.png
  • files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: lib/nfcore_external_java_deps.jar
  • files_exist - File is ignored: lib/NfcoreSchema.groovy
  • files_exist - File is ignored: lib/NfcoreTemplate.groovy
  • files_exist - File is ignored: lib/Utils.groovy
  • files_exist - File is ignored: lib/WorkflowMain.groovy
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • nextflow_config - Config variable ignored: params.max_cpus
  • files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File ignored due to lint config: assets/nf-core-ascc_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-ascc_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-ascc_logo_dark.png
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/ascc/ascc/.github/workflows/awstest.yml
  • template_strings - template_strings

✅ Tests passed:

Run details

  • nf-core/tools version 3.2.0
  • Run at 2025-03-11 14:15:12

@DLBPointon
Copy link
Contributor Author

DLBPointon commented Feb 24, 2025 •
edited
Loading

@muffato and @tkchafin

Are there any plans to get blobtoolkit's schema updated to nf-schema now that nf-validation is depreciated?

On Slack it turned out that the reason a user couldn't use ASCC was because nf-validation is depreciated, updating the pipeline to nf-schema fixed it, but then it crashed on BTK because of the same reason. Thankfully, the parts of ASCC that ran were enough to give them the results they needed. However, it raises the issue that nf-validation is depreciated and end-user issues may become more common. Those able to run it may have the nf-validation plugin already cached in the plugin's directory.

The only issue with the upgrade seems to be shown in the above run logs, it doesn't like --profile_name which is what -profile is parsed into from what I can tell (No that was a me thing, a hangover variable from testing) and then a couple of linting bits look like they need a little bit of work too.

Another issue being that it isn't 100% compatible with NF-Tools and may break linting.
https://nextflow-io.github.io/nf-schema/latest/migration_guide/

@DLBPointon DLBPointon added enhancement New feature or request Release 1 labels Feb 24, 2025
@DLBPointon DLBPointon added the WIP Work In Progress label Feb 24, 2025
@DLBPointon DLBPointon marked this pull request as draft February 26, 2025 09:38
@muffato
Copy link
Member

muffato commented Feb 26, 2025

@DLBPointon . I've now created an issue to have a record: sanger-tol/blobtoolkit#138

Would you say it's something I could squeeze in like an hour ? We're going to release v0.7 soon and then pause development for a bit (we've been expressly asked to work on finalising and deploying the integration of the Genome After-Party), so now is really the time to get it done.

@DLBPointon
Copy link
Contributor Author

@DLBPointon . I've now created an issue to have a record: sanger-tol/blobtoolkit#138

Would you say it's something I could squeeze in like an hour ? We're going to release v0.7 soon and then pause development for a bit (we've been expressly asked to work on finalising and deploying the integration of the Genome After-Party), so now is really the time to get it done.

The nf-schema, yes I think you could get it done in an hour. Most of what you would need to do is in this PR.
NF-Test on the other hand maybe a day or so.

Happy to sort it for you if you don't have the time.

@muffato muffato mentioned this pull request Feb 27, 2025
Closed
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eeaunin eeaunin eeaunin approved these changes

@muffato muffato Awaiting requested review from muffato

Assignees
No one assigned
Labels
enhancement New feature or request Release 1 WIP Work In Progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@DLBPointon @muffato @eeaunin