MSPileup Interface

[haozturk]

Impact of the new feature
Pileup placement, campaign management

Is your feature request related to a problem? Please describe.
We need to implement an interface which will read the campaign configuration whenever there is a change and push the changes to MSPileup

Describe the solution you'd like
Currently, we run the following command whenever we change campaigns.json

python campaignsConfiguration.py --load campaigns.json

Now, I propose that this script should also trigger the pileup updates.

* A given pileup can be a member of multiple campaigns. A pileup should have the same settings in every campaign. Sometimes, operators might make a mistake and don't comply with this rule. We should scan the whole configuration and throw an error and exit if a pileup doesn't have the same settings in every campaign.

* If the previous step succeeds, whenever, we see a pileup in a campaign, we will get the corresponding pileup document from MSPileup
    * If it returns None, create a new pileup document with the following logic:

{"pileupName": <pileup name seen in the campaign config>,
  "pileupType": <to be discussed at [1]>,
  "expectedRSEs": <to be discussed at [2]>,
  "campaigns": <to be discussed at [3]>,
  "active": <to be discussed at [4]>
}

* Else (it means that pileup is already in place, we need to check if there is any update in its settings)
    * Check the expectedRSEs:
        * If it's not the same, set the expectedRSEs as the list of RSEs defined in the campaign
    * Check the campaigns:
        * If the current campaign name is included in `campaigns`, don't do anything. If not, append it
    * Check the active:
        * If the `active` coming from the microservice is different from what's defined in the campaign, update it according to the campaign config

[1] In Unified, if a pileup includes minbias, it's considered as classical mixing, else premix. Perhaps, we can use the same logic, however I see pileups such as /Hydjet_Quenched_Cymbal5Ev8_PbPbMinBias_5020GeV/HiFall15-75X_mcRun2_HeavyIon_v14_75X_mcRun2_HeavyIon_v14-v1/GEN-SIM in our campaign config and I don't what is its type. @z4027163 is this a pileup which requires heavy read as minbias?

[2] Set the expectedRSEs as the list of RSEs defined in the campaign

[3] A given pileup can be a member of multiple campaigns. We need to append the current campaign name to the already available list of campaigns coming from the microservice if it's not already there.

[4] In Unified campaigns, we have the go field which specifies whether a campaign is active or not. If it's true, workflows can pass through assignment, If not, they cannot. We need to decide whether we should set active as the same value go holds. I think this would work except the following case: Sometimes, we might need to pause a campaign by setting go: false, but we might not want to necessarily delete the pileup in this case. In that sense, it makes more sense to me to introduce a new field in our campaign configuration to specify explicitly whether we want to keep the pileup on disk or not. This new field can be named as active and it should be defined in the pileup level. @z4027163 @tyjyang what is your opinion on this?

Additional context
@amaltaro @z4027163 @tyjyang please let me know what you think of this proposal. I'm planning to work on this tomorrow.

[z4027163]

The plan looks good to me.

For [1], /Hydjet_Quenched_Cymbal5Ev8_PbPbMinBias_5020GeV/HiFall15-75X_mcRun2_HeavyIon_v14_75X_mcRun2_HeavyIon_v14-v1/GEN-SIM is heavy read.

For [4], is it normal that we only want to keep active on a certain pileup from a campaign? I think for most of the cases, if we pause a campaign, we want to keep all the pileups, so we can put "active" in campaign level to keep the cleaners of the json.
If we want to delete a pileup, is it feasible to remove the pileup from the campaign json?

[amaltaro]

@haozturk thanks for creating this description. I would like to add the following:
pileup updates are only required if you actually create/delete/update a secondary dataset (and/or their expected location)

• A given pileup can be a member of multiple campaigns. A pileup should have the same settings

perhaps on this, whenever you have a pileup, you could GET it from MSPileup and use its expectedRSEs in the campaign as well, ensuring the pileup configuration is the same (same locations).

If it returns None ...

note that an inexistent pileup in MSPileup will not return none, but it will return a "pileup name not found" error, or something similar.

In Unified, if a pileup includes minbias, it's considered as classical mixing, else premix...

As you can see in this PR:
https://github.com/dmwm/WMCore/pull/11521/files#diff-b1640fcaca1c880c4eefb6f8034d5d62983ec7eb6db54facedc634bb06952271R99
the logic we used is slightly different. Which basically says, if the datatier is PREMIX, then it's a premix pileup, all the rest is classic. Please let us know if this logic is not correct such that we can converge on the same convention. This OR also provides a dump of the current pileup configurations (from parsing the WMCore campaigns). Please let us know in case you spot any flaws and/or inconsistencies on that.

In Unified campaigns, we have the go field...

You captured this well! If you set a pileup configuration as active=false, the microservice will delete the rucio rules.

Other than these comments, this description seems pretty complete to me. Thanks!

[haozturk]

Thanks @amaltaro I noted those points. @z4027163 , Qier, can you please tell us which logic makes more sense to differentiate classical from premix? How can we identify cases such as /Hydjet_Quenched_Cymbal5Ev8_PbPbMinBias_5020GeV/HiFall15-75X_mcRun2_HeavyIon_v14_75X_mcRun2_HeavyIon_v14-v1/GEN-SIM?

[haozturk]

Important: In campaigns.json I'm seeing active pileups which don't have any replica on disk. Most probably these are not needed and should be cleaned up. Once we switch to MSPileup, we don't want it to create transfers for those. Example:

/MinBias_TuneCP5_14TeV-pythia8/Run3Winter20GS-110X_mcRun3_2021_realistic_v6-v1/GEN-SIM

@tyjyang we need to do a proper cleanup. For every pileup in campaigns.json, can you please check if they have rules protecting them on disk at the configured locations. If not, it means that it should revisited. We should get rid of them after talking to PDMV. The best way is to write a script to check. Let's create an issue for that, follow up. Please let me know if you have questions.