Skip to content

Commit a5f4510

Browse files
authored
Update README.md
1 parent 9c3d268 commit a5f4510

File tree

1 file changed

+9
-46
lines changed

1 file changed

+9
-46
lines changed

README.md

+9-46
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
# LEGEND L200 dataflow
22

3-
*Note: Still work in progress.*
4-
53
Implementation of an automatic data processing flow for L200
64
data, based on
75
[Snakemake](https://snakemake.readthedocs.io/).
@@ -10,23 +8,12 @@ data, based on
108
## Configuration
119

1210
Data processing resources are configured via a single site-dependent (and
13-
possibly user-dependent) configuration file, named "config.json" in the
11+
possibly user-dependent) configuration file, named `config.json` in the
1412
following. You may choose an arbitrary name, though.
1513

1614
Use the included [templates/config.json](templates/config.json) as a template
17-
and adjust the data base paths as necessary.
18-
19-
When running Snakemake, the path to the config file *must* be provided via
20-
`--configfile=path/to/configfile.json`. For example, run
21-
22-
```shell
23-
snakemake -j`nproc` --configfile=config.json file_to_generate
24-
```
25-
26-
## Snakefile
27-
28-
Snakemake is controlled using the Snakefile which specifies the rules to generate each file.
29-
The path to the Snakefile *must* be provided via `--snakefile path/to/Snakefile`.
15+
and adjust the data base paths as necessary. Note that, when running Snakemake,
16+
the default path to the config file is `./config.json`.
3017

3118

3219
## Key-Lists
@@ -47,11 +34,9 @@ which will generate the list of available file keys for all l200 files, resp.
4734
a specific period, or a specific period and run, etc.
4835

4936
For example:
50-
5137
```shell
52-
snakemake -j4 --configfile=config.json all-l200-myper.keylist
38+
$ snakemake all-l200-myper.keylist
5339
```
54-
5540
will generate a key-list with all files regarding period `myper`.
5641

5742

@@ -68,9 +53,8 @@ For file lists based on auto-generated key-lists like
6853
automatically, if it doesn't exist.
6954

7055
Example:
71-
7256
```shell
73-
snakemake -j4 --configfile=config.json all-mydet-mymeas-tier2.filelist
57+
$ snakemake all-mydet-mymeas-tier2.filelist
7458
```
7559

7660
File-lists may of course also be derived from custom keylists, generated
@@ -92,11 +76,9 @@ and produce all possible output for the given data tier, based on available
9276
tier0 files which match the target.
9377

9478
Example:
95-
9679
```shell
97-
snakemake -j`nproc` --configfile=config.json all-mydet-mymeas-tier2.gen
80+
$ snakemake all-mydet-mymeas-tier2.gen
9881
```
99-
10082
Targets like `my-dataset-raw.gen` (derived from a key-list
10183
`my-dataset.keylist`) are of course allowed as well.
10284

@@ -107,18 +89,14 @@ Snakemake supports monitoring by connecting to a
10789
[panoptes](https://github.com/panoptes-organization/panoptes) server.
10890

10991
Run (e.g.)
110-
11192
```shell
112-
panoptes --port 5000
113-
93+
$ panoptes --port 5000
11494
```
115-
11695
in the background to run a panoptes server instance, which comes with a
11796
GUI that can be accessed with a web-brower on the specified port.
11897

11998
Then use the Snakemake option `--wms-monitor` to instruct Snakemake to push
12099
progress information to the panoptes server:
121-
122100
```shell
123101
snakemake --wms-monitor http://127.0.0.1:5000 [...]
124102
```
@@ -131,24 +109,9 @@ instead supports Singularity containers via
131109
for greater control.
132110

133111
To use this, the path to `venv` and the name of the environment must be set
134-
in "config.json".
112+
in `config.json`.
135113

136114
This is only relevant then running Snakemake *outside* of the software
137115
container, e.g. then using a batch system (see below). If Snakemake
138116
and the whole workflow is run inside of a container instance, no
139-
container-related settings in "config.json" are required.
140-
141-
142-
## Running on a batch system
143-
144-
A template configuration to run the dataflow on an SGE batch system is
145-
included in [templates/snakemake-config](templates/snakemake-config).
146-
Copy the configuration into `"$HOME/.config/snakemake"` and adjust as
147-
necessary (especially batch-queue selection, number of jobs, etc.).
148-
149-
You should then be able to run data production on the batch system via
150-
(e.g.):
151-
152-
```shell
153-
snakemake --profile cluster-sge --jobs 20 --configfile=config.json all-l200-myper-dsp.gen
154-
```
117+
container-related settings in `config.json` are required.

0 commit comments

Comments
 (0)