Swap test file paths to use `testdata_base_path`

[SPPearce]

In the last few months we have swapped the file paths used in tests from:

file(params.test_data['sarscov2']['genome']['genome_fasta'], checkIfExists: true)

to direct paths like:

file(params.modules_testdata_base_path + 'genomics/sarscov2/genome/genome.fasta', checkIfExists: true)

This is in the guidelines now, but could do with a bulk update.

It would be great to have an automatic method to update all the filepaths at once, at least in the nf-test tests (but should be equivalent for pytest I think)
This could do with a linting check too, to ensure that new modules (or pytest -> nf-test conversions) still keep this.

[GallVp]

Hi @SPPearce

I have started this in batches. The first PR is here: #6048

I have created a script to detect and then automatically fix the paths which is available elsewhere:

1. https://github.com/GallVp/nxf-components/blob/main/.github/check_test_data_paths.sh
2. https://github.com/GallVp/nxf-components/blob/main/.github/get_test_data_path_update.nf

[SPPearce]

Oh thank you @GallVp !
Does it cope with paths specified with either single or double quotes?

[GallVp]

Yes, I think it does. It uses single quotes for the updated paths. I'll keep testing it and post updates here.

[GallVp]

Oh thank you @GallVp ! Does it cope with paths specified with either single or double quotes?

If you mean params.test_data["xyz"], then the current version does not cope with that: https://github.com/GallVp/nxf-components/blob/6eaf5c8313ddd3d0672c3388cfe981bafb872d70/.github/get_test_data_path_update.nf#L18

Needs an update!

[SPPearce]

Yeah, not sure if that is generally used but worth checking.
I have noticed some people put single or double quotes in the new file paths

[SPPearce]

I can't see any use of params.test_data["xyz"] with a quick search, so probably don't need to worry about that.

[GallVp]

Batch2: #6054

[GallVp]

Batch3: #6057

[GallVp]

Batch4: #6058

[GallVp]

Batch5: #6059

[GallVp]

List of modules with issues with their nf-tests

• fastqscreen/fastqscreen depends on bowtie/build; conda fail; nf-test (modules/nf-core/fastqscreen/fastqscreen, conda) is broken #6075
• fastqscreen/fastqscreen is also triggered by fastqscreen/buildfromindex
• centrifuge depends on untar; permissions failure on docker_self_hosted runner; Investigate issue extracting minigut_cf.tar.gz for centrifuge #5970
• hamronization/rgi triggers tar which fails
• merfin fails with a segmentation fault. There seems to be a new update for merfin: https://github.com/arangrhie/merfin which has to first go through Bioconda. For now, skipping merfin/hist. Its requirements are:
  • seqtk/mergepe
• mindagap/mindagap fails conda test and the issue may be linked with Latest version (4.3.0.38) fails on import - libGL.so.1: cannot open shared object file: No such file or directory opencv/opencv-python#370
• nf-test (modules/nf-core/decoupler/decoupler, conda) is broken #6134
• nf-test (modules/nf-core/pharokka/pharokka, conda) fails #6152
• flye tests fail due to insufficient memory #6165
• gatk4spark/markduplicates and gatk4spark/applybqsr fail on docker_self_hosted #6183
• nf-test (modules/nf-core/checkv/updatedatabase, docker_self_hosted) fails #6184
• nf-test (modules/nf-core/learnmsa/align, conda) fails #6185
• nf-test for vcf_impute_glimpse is broken because the test data has been moved #6186
• glimpse/chunk triggers vcf_impute_glimpse which is broken nf-test for vcf_impute_glimpse is broken because the test data has been moved #6186
• nf-test (modules/nf-core/hamronization/rgi, docker_self_hosted, singularity) use UNTAR which fails with tar (child): lbzip2: Cannot exec: No such file or directory
• nf-test (modules/nf-core/rgi/main, docker_self_hosted, singularity) use UNTAR which fails with tar (child): lbzip2: Cannot exec: No such file or directory