Skip to content

Commit 219f86c

Browse files
authored
update doc and TOC (AliceO2Group#2175)
* update doc and TOC * Update DevelopersTips.md
1 parent 0defcf3 commit 219f86c

File tree

5 files changed

+96
-64
lines changed

5 files changed

+96
-64
lines changed

README.md

+61-52
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ For a general overview of our (O2) software, organization and processes, please
1717
* [Environment loading](doc/QuickStart.md#environment-loading)
1818
* [Execution](doc/QuickStart.md#execution)
1919
* [Basic workflow](doc/QuickStart.md#basic-workflow)
20-
* [Readout chain](doc/QuickStart.md#readout-chain)
20+
* [Readout chain (optional)](doc/QuickStart.md#readout-chain-optional)
2121
* [Post-processing example](doc/QuickStart.md#post-processing-example)
2222
* [Modules development](doc/ModulesDevelopment.md)
2323
* [Context](doc/ModulesDevelopment.md#context)
@@ -54,65 +54,74 @@ For a general overview of our (O2) software, organization and processes, please
5454
* [The post-processing framework](doc/PostProcessing.md#the-post-processing-framework)
5555
* [Post-processing interface](doc/PostProcessing.md#post-processing-interface)
5656
* [Configuration](doc/PostProcessing.md#configuration)
57+
* [Definition and access of user-specific configuration](doc/PostProcessing.md#definition-and-access-of-user-specific-configuration)
5758
* [Running it](doc/PostProcessing.md#running-it)
5859
* [Convenience classes](doc/PostProcessing.md#convenience-classes)
5960
* [The TrendingTask class](doc/PostProcessing.md#the-trendingtask-class)
6061
* [The SliceTrendingTask class](doc/PostProcessing.md#the-slicetrendingtask-class)
6162
* [The QualityTask class](doc/PostProcessing.md#the-qualitytask-class)
6263
* [The TRFCollectionTask class](doc/PostProcessing.md#the-trfcollectiontask-class)
64+
* [The BigScreen class](doc/PostProcessing.md#the-bigscreen-class)
6365
* [More examples](doc/PostProcessing.md#more-examples)
6466
* [Advanced topics](doc/Advanced.md)
65-
* [Framework](doc/Advanced.md#framework)
66-
* [Plugging the QC to an existing DPL workflow](doc/Advanced.md#plugging-the-qc-to-an-existing-dpl-workflow)
67-
* [Production of QC objects outside this framework](doc/Advanced.md#production-of-qc-objects-outside-this-framework)
68-
* [Multi-node setups](doc/Advanced.md#multi-node-setups)
69-
* [Batch processing](doc/Advanced.md#batch-processing)
70-
* [Moving window](doc/Advanced.md#moving-window)
71-
* [Writing a DPL data producer](doc/Advanced.md#writing-a-dpl-data-producer)
72-
* [Custom merging](doc/Advanced.md#custom-merging)
73-
* [QC with DPL Analysis](doc/Advanced.md#qc-with-dpl-analysis)
74-
* [Solving performance issues](doc/Advanced.md#solving-performance-issues)
75-
* [Understanding and reducing memory footprint](doc/Advanced.md#understanding-and-reducing-memory-footprint)
76-
* [CCDB / QCDB](doc/Advanced.md#ccdb--qcdb)
77-
* [Accessing objects in CCDB](doc/Advanced.md#accessing-objects-in-ccdb)
78-
* [Access GRP objects with GRP Geom Helper](doc/Advanced.md#access-grp-objects-with-grp-geom-helper)
79-
* [Custom metadata](doc/Advanced.md#custom-metadata)
80-
* [Details on the data storage format in the CCDB](doc/Advanced.md#details-on-the-data-storage-format-in-the-ccdb)
81-
* [Local CCDB setup](doc/Advanced.md#local-ccdb-setup)
82-
* [Asynchronous Data and Monte Carlo QC operations](doc/Advanced.md#asynchronous-data-and-monte-carlo-qc-operations)
83-
* [QCG](doc/Advanced.md#qcg)
84-
* [Display a non-standard ROOT object in QCG](doc/Advanced.md#display-a-non-standard-root-object-in-qcg)
85-
* [Canvas options](doc/Advanced.md#canvas-options)
86-
* [Local QCG (QC GUI) setup](doc/Advanced.md#local-qcg-qc-gui-setup)
87-
* [FLP Suite](doc/Advanced.md#flp-suite)
88-
* [Developing QC modules on a machine with FLP suite](doc/Advanced.md#developing-qc-modules-on-a-machine-with-flp-suite)
89-
* [Switch detector in the workflow <em>readout-dataflow</em>](doc/Advanced.md#switch-detector-in-the-workflow-readout-dataflow)
90-
* [Get all the task output to the infologger](doc/Advanced.md#get-all-the-task-output-to-the-infologger)
91-
* [Using a different config file with the general QC](doc/Advanced.md#using-a-different-config-file-with-the-general-qc)
92-
* [Enable the repo cleaner](doc/Advanced.md#enable-the-repo-cleaner)
93-
* [Configuration](doc/Advanced.md#configuration-1)
94-
* [Merging multiple configuration files into one](doc/Advanced.md#merging-multiple-configuration-files-into-one)
95-
* [Definition and access of task-specific configuration](doc/Advanced.md#definition-and-access-of-task-specific-configuration)
96-
* [Configuration files details](doc/Advanced.md#configuration-files-details)
97-
* [Miscellaneous](doc/Advanced.md#miscellaneous)
98-
* [Data Sampling monitoring](doc/Advanced.md#data-sampling-monitoring)
99-
* [Monitoring metrics](doc/Advanced.md#monitoring-metrics)
100-
* [Common check IncreasingEntries](doc/Advanced.md#common-check-increasingentries)
101-
* [Frequently asked questions](doc/FAQ.md)
102-
* [Git](doc/FAQ.md#git)
103-
* [Are there instructions how to use Git ?](doc/FAQ.md#are-there-instructions-how-to-use-git-)
104-
* [Coding Guidelines](doc/FAQ.md#coding-guidelines)
105-
* [Where are the Coding Guidelines ?](doc/FAQ.md#where-are-the-coding-guidelines-)
106-
* [How do I check my code or apply the guidelines ?](doc/FAQ.md#how-do-i-check-my-code-or-apply-the-guidelines-)
107-
* [Build](doc/FAQ.md#build)
108-
* [How do I add a dependency to my module ?](doc/FAQ.md#how-do-i-add-a-dependency-to-my-module-)
109-
* [How do I make my module depend on library XXX from AliceO2 ?](doc/FAQ.md#how-do-i-make-my-module-depend-on-library-xxx-from-aliceo2-)
110-
* [Run](doc/FAQ.md#run)
111-
* [Why are my QC processes using 100% CPU ?](doc/FAQ.md#why-are-my-qc-processes-using-100-cpu-)
112-
* [QCDB](doc/FAQ.md#qcdb)
113-
* [How to see which objects are stored in the CCDB ?](doc/FAQ.md#how-to-see-which-objects-are-stored-in-the-ccdb-)
114-
* [How to delete objects from the CCDB ?](doc/FAQ.md#how-to-delete-objects-from-the-ccdb-)
115-
* [My objects are not stored due to their size. What can I do ?](doc/FAQ.md#my-objects-are-not-stored-due-to-their-size-what-can-i-do-)
67+
* [Framework](doc/Advanced.md#framework)
68+
* [Plugging the QC to an existing DPL workflow](doc/Advanced.md#plugging-the-qc-to-an-existing-dpl-workflow)
69+
* [Production of QC objects outside this framework](doc/Advanced.md#production-of-qc-objects-outside-this-framework)
70+
* [Multi-node setups](doc/Advanced.md#multi-node-setups)
71+
* [Batch processing](doc/Advanced.md#batch-processing)
72+
* [Moving window](doc/Advanced.md#moving-window)
73+
* [Monitor cycles](doc/Advanced.md#monitor-cycles)
74+
* [Writing a DPL data producer](doc/Advanced.md#writing-a-dpl-data-producer)
75+
* [Custom merging](doc/Advanced.md#custom-merging)
76+
* [Critical, resilient and non-critical tasks](doc/Advanced.md#critical-resilient-and-non-critical-tasks)
77+
* [QC with DPL Analysis](doc/Advanced.md#qc-with-dpl-analysis)
78+
* [Uploading objects to QCDB](doc/Advanced.md#uploading-objects-to-qcdb)
79+
* [Getting AODs in QC Tasks](doc/Advanced.md#getting-aods-in-qc-tasks)
80+
* [Merging with other analysis workflows](doc/Advanced.md#merging-with-other-analysis-workflows)
81+
* [Enabling a workflow to run on Hyperloop](doc/Advanced.md#enabling-a-workflow-to-run-on-hyperloop)
82+
* [Solving performance issues](doc/Advanced.md#solving-performance-issues)
83+
* [Dispatcher](doc/Advanced.md#dispatcher)
84+
* [QC Tasks](doc/Advanced.md#qc-tasks-1)
85+
* [Mergers](doc/Advanced.md#mergers)
86+
* [Understanding and reducing memory footprint](doc/Advanced.md#understanding-and-reducing-memory-footprint)
87+
* [Analysing memory usage with valgrind](doc/Advanced.md#analysing-memory-usage-with-valgrind)
88+
* [CCDB / QCDB](doc/Advanced.md#ccdb--qcdb)
89+
* [Accessing objects in CCDB](doc/Advanced.md#accessing-objects-in-ccdb)
90+
* [Access GRP objects with GRP Geom Helper](doc/Advanced.md#access-grp-objects-with-grp-geom-helper)
91+
* [Global Tracking Data Request helper](doc/Advanced.md#global-tracking-data-request-helper)
92+
* [Custom metadata](doc/Advanced.md#custom-metadata)
93+
* [Details on the data storage format in the CCDB](doc/Advanced.md#details-on-the-data-storage-format-in-the-ccdb)
94+
* [Local CCDB setup](doc/Advanced.md#local-ccdb-setup)
95+
* [Instructions to move an object in the QCDB](doc/Advanced.md#instructions-to-move-an-object-in-the-qcdb)
96+
* [Asynchronous Data and Monte Carlo QC operations](doc/Advanced.md#asynchronous-data-and-monte-carlo-qc-operations)
97+
* [QCG](doc/Advanced.md#qcg)
98+
* [Display a non-standard ROOT object in QCG](doc/Advanced.md#display-a-non-standard-root-object-in-qcg)
99+
* [Canvas options](doc/Advanced.md#canvas-options)
100+
* [Local QCG (QC GUI) setup](doc/Advanced.md#local-qcg-qc-gui-setup)
101+
* [FLP Suite](doc/Advanced.md#flp-suite)
102+
* [Developing QC modules on a machine with FLP suite](doc/Advanced.md#developing-qc-modules-on-a-machine-with-flp-suite)
103+
* [Switch detector in the workflow <em>readout-dataflow</em>](doc/Advanced.md#switch-detector-in-the-workflow-readout-dataflow)
104+
* [Get all the task output to the infologger](doc/Advanced.md#get-all-the-task-output-to-the-infologger)
105+
* [Using a different config file with the general QC](doc/Advanced.md#using-a-different-config-file-with-the-general-qc)
106+
* [Enable the repo cleaner](doc/Advanced.md#enable-the-repo-cleaner)
107+
* [Configuration](doc/Advanced.md#configuration-1)
108+
* [Merging multiple configuration files into one](doc/Advanced.md#merging-multiple-configuration-files-into-one)
109+
* [Definition and access of simple user-defined task configuration ("taskParameters")](doc/Advanced.md#definition-and-access-of-simple-user-defined-task-configuration-taskparameters)
110+
* [Definition and access of user-defined configuration ("extendedTaskParameters")](doc/Advanced.md#definition-and-access-of-user-defined-configuration-extendedtaskparameters)
111+
* [Definition of new arguments](doc/Advanced.md#definition-of-new-arguments)
112+
* [Configuration files details](doc/Advanced.md#configuration-files-details)
113+
* [Global configuration structure](doc/Advanced.md#global-configuration-structure)
114+
* [Common configuration](doc/Advanced.md#common-configuration)
115+
* [QC Tasks configuration](doc/Advanced.md#qc-tasks-configuration)
116+
* [QC Checks configuration](doc/Advanced.md#qc-checks-configuration)
117+
* [QC Aggregators configuration](doc/Advanced.md#qc-aggregators-configuration)
118+
* [QC Post-processing configuration](doc/Advanced.md#qc-post-processing-configuration)
119+
* [External tasks configuration](doc/Advanced.md#external-tasks-configuration)
120+
* [Miscellaneous](doc/Advanced.md#miscellaneous)
121+
* [Data Sampling monitoring](doc/Advanced.md#data-sampling-monitoring)
122+
* [Monitoring metrics](doc/Advanced.md#monitoring-metrics)
123+
* [Common check IncreasingEntries](doc/Advanced.md#common-check-increasingentries)
124+
* [Update the shmem segment size of a detector](doc/Advanced.md#update-the-shmem-segment-size-of-a-detector)
116125

117126
### Where to get help
118127

doc/Advanced.md

+11-5
Original file line numberDiff line numberDiff line change
@@ -19,18 +19,18 @@ Advanced topics
1919
* [Monitor cycles](#monitor-cycles)
2020
* [Writing a DPL data producer](#writing-a-dpl-data-producer)
2121
* [Custom merging](#custom-merging)
22-
* [Critical and non-critical tasks](#critical-and-non-critical-tasks)
22+
* [Critical, resilient and non-critical tasks](#critical-resilient-and-non-critical-tasks)
2323
* [QC with DPL Analysis](#qc-with-dpl-analysis)
2424
* [Uploading objects to QCDB](#uploading-objects-to-qcdb)
2525
* [Getting AODs in QC Tasks](#getting-aods-in-qc-tasks)
2626
* [Merging with other analysis workflows](#merging-with-other-analysis-workflows)
2727
* [Enabling a workflow to run on Hyperloop](#enabling-a-workflow-to-run-on-hyperloop)
2828
* [Solving performance issues](#solving-performance-issues)
2929
* [Dispatcher](#dispatcher)
30-
* [QC Tasks](#qc-tasks)
30+
* [QC Tasks](#qc-tasks-1)
3131
* [Mergers](#mergers)
3232
* [Understanding and reducing memory footprint](#understanding-and-reducing-memory-footprint)
33-
* [Analysing memory usage with valgrind](#analysing-memory-usage-with-valgrind)
33+
* [Analysing memory usage with valgrind](#analysing-memory-usage-with-valgrind)
3434
* [CCDB / QCDB](#ccdb--qcdb)
3535
* [Accessing objects in CCDB](#accessing-objects-in-ccdb)
3636
* [Access GRP objects with GRP Geom Helper](#access-grp-objects-with-grp-geom-helper)
@@ -52,7 +52,8 @@ Advanced topics
5252
* [Enable the repo cleaner](#enable-the-repo-cleaner)
5353
* [Configuration](#configuration-1)
5454
* [Merging multiple configuration files into one](#merging-multiple-configuration-files-into-one)
55-
* [Definition and access of user-specific configuration](#definition-and-access-of-user-specific-configuration)
55+
* [Definition and access of simple user-defined task configuration ("taskParameters")](#definition-and-access-of-simple-user-defined-task-configuration-taskparameters)
56+
* [Definition and access of user-defined configuration ("extendedTaskParameters")](#definition-and-access-of-user-defined-configuration-extendedtaskparameters)
5657
* [Definition of new arguments](#definition-of-new-arguments)
5758
* [Configuration files details](#configuration-files-details)
5859
* [Global configuration structure](#global-configuration-structure)
@@ -769,7 +770,8 @@ ms_print massif.out.976329 > massif_abc_task.log
769770

770771
The MonitorObjects generated by Quality Control are stored in a dedicated repository (QCDB), which is based on CCDB.
771772
The run conditions, on the other hand, are located in another, separate database.
772-
The recommended way to access these is to use a `Lifetime::Condition` DPL input, which can be requested as in the query below:
773+
774+
The recommended way (excluding postprocessing) to access these conditions is to use a `Lifetime::Condition` DPL input, which can be requested as in the query below:
773775
```json
774776
"tasks": {
775777
"MyTask": {
@@ -787,6 +789,10 @@ Geometry and General Run Parameters (GRP) can be also accessed with the [GRP Geo
787789

788790
If your task accesses CCDB objects using `TaskInterface::retrieveCondition`, please migrate to using one of the methods mentioned above.
789791

792+
### Accessing from a Postprocessing task
793+
794+
PostProcessingTasks do not take DPL inputs, so in this case `TaskInterface::retrieveCondition` should be used.
795+
790796
## Access GRP objects with GRP Geom Helper
791797

792798
To get GRP objects via a central facility, add the following structure to the task definition and set its values

doc/DevelopersTips.md

+21-2
Original file line numberDiff line numberDiff line change
@@ -322,9 +322,9 @@ psql -h localhost ccdb ccdb_user -c "delete from ccdb_paths where pathid in (sel
322322
6. `cd /var/lib/pgsql/.globus `
323323
7. backup the old files
324324
8. copy hostcert and hostkey
325-
9. chmod 600 them
325+
9 chmod 600 them
326326
10. Make sure that `.globus` and the two files are owned by the user `postgres`
327-
327+
328328
### ControlWorkflows
329329

330330
#### Parameter `qcConfiguration` in tasks
@@ -528,3 +528,22 @@ then
528528
o2-qc --remote-batch QC_fullrun.root --config "json://QC_production.json" -b --override-values "qc.config.Activity.number=523897;qc.config.Activity.passName=apass2;qc.config.Activity.periodName=LHC22m"
529529
fi
530530
```
531+
532+
## Common QCDB cleanup tasks
533+
534+
In general:
535+
- DO NOT USE `truncate` ! `truncate is unstable and any mistake can have dire consequences.
536+
- All the scripts take the arguments `--dry-run`, `--print-list` and `--one-by-one`. Use them. Proceed with the actual deletion only, when you have thoroughly verified the output of a `print-list` and a `dry-run`
537+
- Deletion commands can only be run on the QCDB server itself.
538+
539+
#### Remove all objects in a path
540+
We use the script `o2-qc-repo-delete-time-interval` with an interval covering 0 to many years in the future.
541+
```shell
542+
o2-qc-repo-delete-time-interval --url http://localhost:8083 --log-level 10 --from 0 --to 1994150839053 --only-path-no-subdir --path qc/TRD/MO/Tracklets/layer1 --dry-run
543+
```
544+
545+
#### Move objects from one path to another
546+
It is important to put the full destination path, including the name of the folder we are moving.
547+
```shell
548+
o2-qc-repo-move-objects --url http://localhost:8083 --log-level 10 --path qc/TRD/MO/TrackletsTask/triggerspertimeframe --new-path qc/TRD/MO/Tracklets/triggerspertimeframe --dry-run
549+
```

doc/PostProcessing.md

+2-4
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,14 @@
77
* [The post-processing framework](#the-post-processing-framework)
88
* [Post-processing interface](#post-processing-interface)
99
* [Configuration](#configuration)
10-
* [Triggers configuration](#triggers-configuration)
10+
* [Definition and access of user-specific configuration](#definition-and-access-of-user-specific-configuration)
1111
* [Running it](#running-it)
1212
* [Convenience classes](#convenience-classes)
1313
* [The TrendingTask class](#the-trendingtask-class)
14-
* [Configuration](#configuration-1)
1514
* [The SliceTrendingTask class](#the-slicetrendingtask-class)
16-
* [Configuration](#configuration-2)
1715
* [The QualityTask class](#the-qualitytask-class)
18-
* [Configuration](#configuration-3)
1916
* [The TRFCollectionTask class](#the-trfcollectiontask-class)
17+
* [The BigScreen class](#the-bigscreen-class)
2018
* [More examples](#more-examples)
2119
<!--te-->
2220

doc/QuickStart.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
* [Environment loading](#environment-loading)
1111
* [Execution](#execution)
1212
* [Basic workflow](#basic-workflow)
13-
* [Readout chain (optional)](#readout-chain-\(optional\))
13+
* [Readout chain (optional)](#readout-chain-optional)
1414
* [Getting real data from readout](#getting-real-data-from-readout)
1515
* [Readout data format as received by the Task](#readout-data-format-as-received-by-the-task)
1616
* [Post-processing example](#post-processing-example)

0 commit comments

Comments
 (0)