updates

Author: Tianyu Justin Yang

ACDC automation ideas.md

@@ -1,10 +1,3 @@
-# HLT and xrootd
-
--   For a given failed task, if there is a necessary site which is drain (marked as red and bold letters in the console):
-    -   Pre-select primary xrootd (don't touch the secondary xrootd at any step)
-    -   If either `T2_CH_CERN` or `T2_CH_CERN_HLT` is among the pre-selected sites, unselect them
-    -   It would be good to display a small message if console does this procedure in order to make it clear for new operators etc. It could be something like `There is a necessary site in drain, therefore enabling xrootd. Since xrootd is enabled, unselecting CERN and HLT since HLT cannot perform remote reads. We unselect CERN too, since jobs sent to CERN can be redirected to HLT internally`
-
 # filter workflows for group ACDC
 
 Sometimes the same type of error could affect multiple workflows, and the steps for manual ACDC are identical.  
@@ -24,6 +17,13 @@ for example, we could say:
 -   run ACDC without xrootd and splitting for all workflows in assistance-manual that failed at CERN sites with error 91009
 -   kill and clone for all workflows failed at TIFR that has < 90% statistics and < 1M events
 
+# HLT and xrootd
+
+-   For a given failed task, if there is a necessary site which is drain (marked as red and bold letters in the console):
+    -   Pre-select primary xrootd (don't touch the secondary xrootd at any step)
+    -   If either `T2_CH_CERN` or `T2_CH_CERN_HLT` is among the pre-selected sites, unselect them
+    -   It would be good to display a small message if console does this procedure in order to make it clear for new operators etc. It could be something like `There is a necessary site in drain, therefore enabling xrootd. Since xrootd is enabled, unselecting CERN and HLT since HLT cannot perform remote reads. We unselect CERN too, since jobs sent to CERN can be redirected to HLT internally`
+
 # Automatic site reassignment for draining sites
 When sites go into drain, we have to manually reassign workflows to other sites. The rules for which sites to reassign the workflows to could be automated:
 

manual operations.md

@@ -1,14 +1,15 @@
 # Manual operation procedures
 
-## Things to check before submitting ACDC
+## Permanent procedures
+### Things to check before submitting ACDC
 
 -   check if ReqMgr status is `completed`
 -   check if unified status is indeed `assistance-manual`, as sometimes there are delays in the cms-unified page
 -   look for existing jira tickets, if there is ongoing discussion, make sure the issues have been resolved before submitting ACDC
 -   check for solutions for specific errors in this [cheat sheet](https://docs.google.com/spreadsheets/d/12JBANxwzN0KWAV4o-yYxnA4Fe2juGXie5uiFXgF3bXo/edit#gid=0)
 -   check if there is an ongoing ACDC (Action: Pending on console)
 
-## Creating Jira tickets
+### Creating Jira tickets
 
 -   copy the full url of the workflow on Dima's site into description.
     for example, `https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_TOP-RunIISummer20UL18NanoAODv2-00233`
@@ -17,7 +18,7 @@
 -   tag interested parties in the description and component filed. (e.g. pdmv-conveners)
 -   **set labels** (e.g. T2_IN_TIFR, 8028-FallbackFileOpenError)
 
-## dealing with irrecoverable errors
+### dealing with irrecoverable errors
 
 -   if workflow not `ReReco`, aiming 90% stats,
     -   if there are recoverable errors
@@ -27,6 +28,34 @@
         -   else, **bypass**
 -   else, aiming 100% stats, same procedure as the above branch
 
+## Transient procedures
+### `50664` and `71304` workflows
+-   Put (`50664-PerformanceKill` OR `71304-JobKilled`) AND (label for campaign e.g. ) as jira labels
+-   wait for response
+-   if no response for some time, reject
+
+### `8028` for empty files
+When input files were produced okay, but later became empty, one will see this error message:
+```
+Fatal Exception (Exit code: 8028)
+An exception of category 'FallbackFileOpenError' occurred while
+[0] Constructing the EventProcessor
+[1] Constructing input source of type PoolSource
+[2] Calling RootFileSequenceBase::initTheFile()
+Additional Info:
+[a] XrdCl::File::Open(name='[root://cmsxrootd.hep.wisc.edu//store/mc/RunIISummer20UL18MiniAOD/HToSSTo2Mu2Hadrons_MH125_MS1p1_ctauS100_TuneCP2_13TeV-powheg-pythia8/MINIAODSIM/106X_upgrade2018_realistic_v11_L1v1-v1/2520000/A639C8D7-B843-F34E-98F7-8C077C24B6B2.root](root://cmsxrootd.hep.wisc.edu//store/mc/RunIISummer20UL18MiniAOD/HToSSTo2Mu2Hadrons_MH125_MS1p1_ctauS100_TuneCP2_13TeV-powheg-pythia8/MINIAODSIM/106X_upgrade2018_realistic_v11_L1v1-v1/2520000/A639C8D7-B843-F34E-98F7-8C077C24B6B2.root)', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] Too many attempts to gain dfs read access to the file
+' (errno=3011, code=400). No additional data servers were found.
+[b] Last URL tried: [root://cmsxrootd.hep.wisc.edu:1094//store/mc/RunIISummer20UL18MiniAOD/HToSSTo2Mu2Hadrons_MH125_MS1p1_ctauS100_TuneCP2_13TeV-powheg-pythia8/MINIAODSIM/106X_upgrade2018_realistic_v11_L1v1-v1/2520000/A639C8D7-B843-F34E-98F7-8C077C24B6B2.root?tried=](root://cmsxrootd.hep.wisc.edu:1094//store/mc/RunIISummer20UL18MiniAOD/HToSSTo2Mu2Hadrons_MH125_MS1p1_ctauS100_TuneCP2_13TeV-powheg-pythia8/MINIAODSIM/106X_upgrade2018_realistic_v11_L1v1-v1/2520000/A639C8D7-B843-F34E-98F7-8C077C24B6B2.root?tried=)
+[c] Problematic data server: cmsxrootd.hep.wisc.edu:1094
+[d] Disabled source: cmsxrootd.hep.wisc.edu:1094
+[e] Input file [root://cmsxrootd.hep.wisc.edu//store/mc/RunIISummer20UL18MiniAOD/HToSSTo2Mu2Hadrons_MH125_MS1p1_ctauS100_TuneCP2_13TeV-powheg-pythia8/MINIAODSIM/106X_upgrade2018_realistic_v11_L1v1-v1/2520000/A639C8D7-B843-F34E-98F7-8C077C24B6B2.root](root://cmsxrootd.hep.wisc.edu//store/mc/RunIISummer20UL18MiniAOD/HToSSTo2Mu2Hadrons_MH125_MS1p1_ctauS100_TuneCP2_13TeV-powheg-pythia8/MINIAODSIM/106X_upgrade2018_realistic_v11_L1v1-v1/2520000/A639C8D7-B843-F34E-98F7-8C077C24B6B2.root) could not be opened.
+Fallback Input file [root://cmsxrootd.fnal.gov//store/mc/RunIISummer20UL18MiniAOD/HToSSTo2Mu2Hadrons_MH125_MS1p1_ctauS100_TuneCP2_13TeV-powheg-pythia8/MINIAODSIM/106X_upgrade2018_realistic_v11_L1v1-v1/2520000/A639C8D7-B843-F34E-98F7-8C077C24B6B2.root?source=glow](root://cmsxrootd.fnal.gov//store/mc/RunIISummer20UL18MiniAOD/HToSSTo2Mu2Hadrons_MH125_MS1p1_ctauS100_TuneCP2_13TeV-powheg-pythia8/MINIAODSIM/106X_upgrade2018_realistic_v11_L1v1-v1/2520000/A639C8D7-B843-F34E-98F7-8C077C24B6B2.root?source=glow) also could not be opened.
+Original exception info is above; fallback exception info is below.
+[f] Fatal Root Error: @SUB=TStorageFactoryFile::ReadBuffer
+read from Storage::xread returned 0. Asked to read n bytes: 300 from offset: 0 with file size: 0
+```
+-   don't ACDC, instead apply `EmptyInputFile` as the jira label
+
 # Sites
 
 ## site peculiarities
@@ -78,12 +107,32 @@ Neutrino and MinBias are the major categories, and MinBias is particularly large
 
 To check if a workflow has secondary inputs, look for `minbias` in the error report, or check `MCPileup` on ReqMgr.
 
-# workflow composition
+WMCore-agent will only look for input blocks that are protected by their account.
+
+# workflow information
+
+## workflow composition
+
 workflows are made of **tasks**.
 Each task represents one step in the workflow production chain (e.g. GEN, SIM, DIGI...)
 
 Tasks are divided into **jobs** to be run on a batching system (e.g. condor).
 
+## checking workflow information
+
+### prepid and request ID (workflow ID)
+
+For MC workflows, prepid is contained in the request ID  
+FOr `ReReco` workflows, enter the request ID on `ReqMgr` to get prepid in the JSON file
+
+### worker node
+
+To check the specific node (specific machine within a site) at which the workflow was run:
+-   enter the workflow ID on [wmstats](https://cmsweb.cern.ch/wmstats/index.html),
+-   click on the box under the "L" column
+-   In the list of tasks under the workflow, click the "L" column box again for the task you want to check
+-   In the information shown at the bottom of the page, search for "worker node"
+
 # Computing Resources
 Each workflow can only take up a limited amount of computing resource before hitting PerformanceKill walls (exit code `50664` and `50660`).