Merge pull request #83 from CenterForMedicalGeneticsGhent/dev

Release PR v1.1.0
nf-cmgg · Mar 14, 2023 · 20d4fda · 20d4fda
2 parents 38817d0 + 0c8e628
commit 20d4fda
Show file tree

Hide file tree

Showing 196 changed files with 8,783 additions and 3,843 deletions.
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -0,0 +1,27 @@
+{
+    "name": "nfcore",
+    "image": "nfcore/gitpod:latest",
+    "remoteUser": "gitpod",
+
+    // Configure tool-specific properties.
+    "customizations": {
+        // Configure properties specific to VS Code.
+        "vscode": {
+            // Set *default* container specific settings.json values on container create.
+            "settings": {
+                "python.defaultInterpreterPath": "/opt/conda/bin/python",
+                "python.linting.enabled": true,
+                "python.linting.pylintEnabled": true,
+                "python.formatting.autopep8Path": "/opt/conda/bin/autopep8",
+                "python.formatting.yapfPath": "/opt/conda/bin/yapf",
+                "python.linting.flake8Path": "/opt/conda/bin/flake8",
+                "python.linting.pycodestylePath": "/opt/conda/bin/pycodestyle",
+                "python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
+                "python.linting.pylintPath": "/opt/conda/bin/pylint"
+            },
+
+            // Add the IDs of extensions you want installed when the container is created.
+            "extensions": ["ms-python.python", "ms-python.vscode-pylance", "nf-core.nf-core-extensionpack"]
+        }
+    }
+}
diff --git a/.gitattributes b/.gitattributes
@@ -1,3 +1,4 @@
 *.config linguist-language=nextflow
+*.nf.test linguist-language=nextflow
 modules/nf-core/** linguist-generated
 subworkflows/nf-core/** linguist-generated
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -95,3 +95,19 @@ If you are using a new feature from core Nextflow, you may bump the minimum requ
 ### Images and figures
 
 For overview images and other documents we follow the nf-core [style guidelines and examples](https://nf-co.re/developers/design_guidelines).
+
+## GitHub Codespaces
+
+This repo includes a devcontainer configuration which will create a GitHub Codespaces for Nextflow development! This is an online developer environment that runs in your browser, complete with VSCode and a terminal.
+
+To get started:
+
+- Open the repo in [Codespaces](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-germline/codespaces)
+- Tools installed
+  - nf-core
+  - Nextflow
+
+Devcontainer specs:
+
+- [DevContainer config](.devcontainer/devcontainer.json)
+- [Dockerfile](.devcontainer/Dockerfile)
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -9,34 +9,46 @@ body:
       description: A clear and concise description of what the bug is.
     validations:
       required: true
-
   - type: textarea
     id: command_used
     attributes:
       label: Command used and terminal output
-      description: Steps to reproduce the behaviour. Please paste the command you used to launch the pipeline and the output from your terminal.
+      description: Steps to reproduce the behaviour. Please paste the command you used
+        to launch the pipeline and the output from your terminal.
       render: console
-      placeholder: |
-        $ nextflow run ...
+      placeholder: "$ nextflow run ...
+
 
         Some output where something broke
 
+        "
   - type: textarea
     id: files
     attributes:
       label: Relevant files
-      description: |
-        Please drag and drop the relevant files here. Create a `.zip` archive if the extension is not allowed.
-        Your verbose log file `.nextflow.log` is often useful _(this is a hidden file in the directory where you launched the pipeline)_ as well as custom Nextflow configuration files.
+      description: "Please drag and drop the relevant files here. Create a `.zip` archive
+        if the extension is not allowed.
+
+        Your verbose log file `.nextflow.log` is often useful _(this is a hidden file
+        in the directory where you launched the pipeline)_ as well as custom Nextflow
+        configuration files.
 
+        "
   - type: textarea
     id: system
     attributes:
       label: System information
-      description: |
-        * Nextflow version _(eg. 21.10.3)_
+      description: "* Nextflow version _(eg. 22.10.1)_
+
         * Hardware _(eg. HPC, Desktop, Cloud)_
+
         * Executor _(eg. slurm, local, awsbatch)_
+
         * Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter or Charliecloud)_
+
         * OS _(eg. CentOS Linux, macOS, Linux Mint)_
-        * Version of CenterForMedicalGeneticsGhent/nf-cmgg-germline _(eg. 1.1, 1.5, 1.8.2)_
+
+        * Version of CenterForMedicalGeneticsGhent/nf-cmgg-germline _(eg. 1.1, 1.5,
+        1.8.2)_
+
+        "
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -8,14 +8,14 @@ These are the most common things requested on pull requests (PRs).
 
 Remember that PRs should be made against the dev branch, unless you're preparing a pipeline release.
 
-Learn more about contributing: [CONTRIBUTING.md](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-germline/blob/master/.github/CONTRIBUTING.md)
+Learn more about contributing: [CONTRIBUTING.md](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-germline/tree/master/.github/CONTRIBUTING.md)
 -->
 
 ## PR checklist
 
 - [ ] This comment contains a description of changes (with reason).
 - [ ] If you've fixed a bug or added code that should be tested, add tests!
-  - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-germline/blob/master/.github/CONTRIBUTING.md)
+- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-germline/tree/master/.github/CONTRIBUTING.md)
 - [ ] Make sure your code lints (`nf-core lint`).
 - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
 - [ ] Usage Documentation in `docs/usage.md` is updated.

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -11,59 +11,52 @@ on:
 env:
   NXF_ANSI_LOG: false
 
+concurrency:
+  group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
+  cancel-in-progress: true
+
 jobs:
   test_all:
-    name: Run pipeline with test data (complete)
+    name: Run nf-test with ${{ matrix.test }}-${{ matrix.NXF_VER }}
     # Only run on push if this is the nf-core dev branch (merged PRs)
     if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'CenterForMedicalGeneticsGhent/nf-cmgg-germline') }}"
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        nxf_ver: ["21.10.3", ""]
+        NXF_VER:
+          - "22.10.5"
+          - "latest-everything"
         test:
           - "default"
           - "fails"
-          - "seqplorer_min"
-          - "seqplorer_vcfanno"
-          - "seqr_full"
-          - "seqr_no_genotyping"
+          - "annotate_test"
+          - "filter_test"
+          - "gemini_test"
+          - "full"
     steps:
       - name: Free some space
         run: |
           sudo rm -rf "/usr/local/share/boost"
           sudo rm -rf "$AGENT_TOOLSDIRECTORY"
 
       - name: Check out pipeline code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Install Nextflow
         uses: nf-core/setup-nextflow@v1
         with:
-          version: "${{ matrix.nxf_ver }}"
+          version: "${{ matrix.NXF_VER }}"
 
       - name: Install nf-test
         run: |
-          sudo bash; mkdir /opt/nf-test; cd /opt/nf-test; wget https://github.com/askimed/nf-test/releases/download/v0.6.0/nf-test-0.6.0.tar.gz; tar xvfz nf-test-0.6.0.tar.gz; chmod +x nf-test;
-          echo "/opt/nf-test" >> $GITHUB_PATH;
+          conda install -c bioconda nf-test
 
       - name: Run pipeline with test data
         run: |
-          nf-test test tests/${{ matrix.test }}.test
+          $CONDA/bin/nf-test test tests/${{ matrix.test }}.test --junitxml=default.xml
 
-      - name: Output log on failure
-        if: failure()
-        run: |
-          sudo apt install bat > /dev/null
-          batcat --decorations=always --color=always .nf-test/tests/*/meta/std.{out,err}
-      - name: Upload logs on failure
-        if: failure()
-        uses: actions/upload-artifact@v2
+      - name: Publish Test Report
+        uses: mikepenz/action-junit-report@v3
+        if: always() # always run even if the previous step fails
         with:
-          name: nf-test-logs
-          path: |
-            .nf-test/tests/*/meta/nextflow.log
-            .nf-test/tests/*/meta/std.out
-            .nf-test/tests/*/meta/std.err
-            .nf-test/tests/*/meta/trace.csv
-            .nf-test/tests/*/work
-            .nf-test/*/output
+          report_paths: "default.xml"
diff --git a/.github/workflows/fix-linting.yml b/.github/workflows/fix-linting.yml
@@ -24,7 +24,7 @@ jobs:
         env:
           GITHUB_TOKEN: ${{ secrets.nf_core_bot_auth_token }}
 
-      - uses: actions/setup-node@v2
+      - uses: actions/setup-node@v3
 
       - name: Install Prettier
         run: npm install -g prettier @prettier/plugin-php
@@ -34,9 +34,9 @@ jobs:
         id: prettier_status
         run: |
           if prettier --check ${GITHUB_WORKSPACE}; then
-            echo "::set-output name=result::pass"
+            echo "result=pass" >> $GITHUB_OUTPUT
           else
-            echo "::set-output name=result::fail"
+            echo "result=fail" >> $GITHUB_OUTPUT
           fi
 
       - name: Run 'prettier --write'

diff --git a/.github/workflows/linting.yml b/.github/workflows/linting.yml
@@ -4,6 +4,8 @@ name: nf-core linting
 # that the code meets the nf-core guidelines.
 on:
   push:
+    branches:
+      - dev
   pull_request:
   release:
     types: [published]
@@ -12,9 +14,9 @@ jobs:
   Prettier:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
 
-      - uses: actions/setup-node@v2
+      - uses: actions/setup-node@v3
 
       - name: Install Prettier
         run: npm install -g prettier
@@ -25,7 +27,7 @@ jobs:
   PythonBlack:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
 
       - name: Activate Black
         uses: psf/black@stable
@@ -56,12 +58,12 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Install Nextflow
         uses: nf-core/setup-nextflow@v1
 
-      - uses: actions/setup-python@v3
+      - uses: actions/setup-python@v4
         with:
           python-version: "3.7"
           architecture: "x64"
@@ -84,7 +86,7 @@ jobs:
 
       - name: Upload linting log file artifact
         if: ${{ always() }}
-        uses: actions/upload-artifact@v2
+        uses: actions/upload-artifact@v3
         with:
           name: linting-logs
           path: |

diff --git a/.github/workflows/linting_comment.yml b/.github/workflows/linting_comment.yml
@@ -18,7 +18,7 @@ jobs:
 
       - name: Get PR number
         id: pr_number
-        run: echo "::set-output name=pr_number::$(cat linting-logs/PR_number.txt)"
+        run: echo "pr_number=$(cat linting-logs/PR_number.txt)" >> $GITHUB_OUTPUT
 
       - name: Post PR comment
         uses: marocchino/sticky-pull-request-comment@v2

diff --git a/.nf-core.yml b/.nf-core.yml
@@ -1,14 +1,14 @@
 repository_type: pipeline
-
 lint:
   files_exist:
     - CODE_OF_CONDUCT.md
-    - assets/nf-core-nfcmggstructural_logo_light.png
-    - docs/images/nf-core-nfcmggstructural_logo_light.png
-    - docs/images/nf-core-nfcmggstructural_logo_dark.png
+    - assets/nf-core-nf-cmgg-germline_logo_light.png
+    - docs/images/nf-core-nf-cmgg-germline_logo_light.png
+    - docs/images/nf-core-nf-cmgg-germline_logo_dark.png
     - .github/ISSUE_TEMPLATE/config.yml
     - .github/workflows/awstest.yml
     - .github/workflows/awsfulltest.yml
+    - lib/WorkflowNf-cmgg-germline.groovy
   nextflow_config:
     - manifest.name
     - manifest.homePage
@@ -26,4 +26,6 @@ lint:
     - assets/sendmail_template.txt
     - lib/NfcoreTemplate.groovy
     - .prettierignore
+    - LICENSE
   actions_ci: false
+  pipeline_name_conventions: false
diff --git a/.prettierignore b/.prettierignore
@@ -1,5 +1,6 @@
 email_template.html
 adaptivecard.json
+slackreport.json
 .nextflow*
 .nf-test/
 work/
@@ -11,3 +12,5 @@ testing*
 *.pyc
 samplesheet*
 *.ped
+null
+bin/
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,34 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v1.1.0 - Glorious Ghent - [Mar 14 2023]
+
+### New Features
+
+1. BED file input is new optional (The regions are created from the FASTA index). Providing a BED file is still preffered for the most optimal runs.
+2. Added support for samples that aren't part of a family. Just leave the `ped` and `family_id` input fields in the samplesheet empty for a sample to be treated like this. This sample will go through exactly the same workflow but will be emitted as a single-sample VCF.
+3. Added `dump` functionality to lots of channels.
+4. Added the `dbsnp` option to `GATK HaplotypeCaller`. use `--dbsnp` and `--dbsnp_tbi` to supply these VCFs.
+5. Added the `vcf_extract_somalier` subworkflow to the pipeline. This also creates PED files inferred from the input multi-sample VCF.
+6. Added a validation subworkflow. All files that have a VCF in the `truth_vcf` column of the input samplesheet will be validated against this VCF. This can be turned off by supplying the `--validate false` flag to the pipeline run.
+
+### Improvements
+
+1. Improved the scatter/gather logic. This is now done with `goleft indexsplit` to define chunks of even coverage. The genotyping scattering now happens with `bedtools makewindows`. This creates chunks of even regions from the merged BED files for the family. By passing a padding of about 20 bps to the genotype tools, we make sure all variants on the edges of these regions are also genotyped. Duplicates are removed later when running `bcftools concat`
+2. Refactored a lot of the code to maintain the same style over the whole pipeline.
+3. Updated the minimum Nextlow version to `22.10.5` to make sure S3 staging works perfectly.
+4. The `post_processing` subworklow has been renamed to the better suiting `joint_genotyping` subworkflow. `reblockgvcf` has been moved to `germline_variant_calling` and the `filter` and `reheadering` has been moved to the main workflow.
+5. Merging VCFs of the same family now happens with `GATK GenomicsDBImport` instead of `GATK MergeGVCFs` or `bcftools merge`. This gives more reliable results.
+6. Improved the handling of `vcfanno`
+7. The PED headers can now be added to all the output VCFs that are part of a family instead of only those that were given a PED file as input. The PED file used is created using `somalier relate`. This feature can be turned on using the `--add_ped true` argument. This doesn't happen by default.
+
+### Bug fixes
+
+1. Fixed some issues when both the `ped` and `family_id` were given for a sample.
+2. Fixed the PED input for `rtgtools_pedfilter` (`-9` isn't recognized as unknown by the tool. Now these will be automatically converted to `0` before this tool)
+3. Fixed issues with the DBsnp index not being created correctly
+4. Fixed wrongly formed joins and added checks for mismatches and duplicates
+
 ## v1.0.1 - Happy Hollebeke - [Oct 7 2022]
 
 ### Changes

diff --git a/CITATIONS.md b/CITATIONS.md
@@ -8,10 +8,6 @@
 
 > Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311.
 
-## Pipeline tools
-
-TODO
-
 ## Software packaging/containerisation tools
 
 - [Anaconda](https://anaconda.com)