Merge pull request ipfs-force-community#1084 from ipfs-force-community/feat/0x5459/support-sealing-daemons

feat(worker): support custom sealing daemon processor

Author: LinZexiao

CHANGELOG.zh.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.11.0
+- damocles-worker
+  - 支持以子进程方式启动自定义守护进程. 新增配置项目: `[[processors.sealing_daemons]]`[#1084](https://github.com/ipfs-force-community/damocles/pull/1084)
+
 ## 0.10.0-rc1
 
 - 支持 NV22 DDO [#1071](https://github.com/ipfs-force-community/damocles/pull/1071)

damocles-worker/src/config.rs

@@ -229,6 +229,9 @@ pub struct Processors {
 
     /// section for window_post processor
     pub window_post: Option<Vec<Ext>>,
+
+    /// section for sealing_daemons processor
+    pub sealing_daemons: Option<Vec<Ext>>,
 }
 
 impl Processors {

damocles-worker/src/run.rs

@@ -328,6 +328,7 @@ fn start_processors(
         transfer: construct_sub_processor!(transfer, cfg, limit),
         unseal: construct_sub_processor!(unseal, cfg, limit),
         window_post: construct_sub_processor!(window_post, cfg, limit),
+        sealing_daemons: construct_sub_processor!(sealing_daemons, cfg, limit),
     })
 }
 

damocles-worker/src/watchdog.rs

@@ -67,6 +67,8 @@ pub(crate) struct GlobalProcessors {
     pub transfer: Proc<TransferInput>,
     pub unseal: Proc<UnsealInput>,
     pub window_post: Proc<WindowPoStInput>,
+    #[allow(dead_code)]
+    pub sealing_daemons: Proc<()>,
 }
 
 impl Module for Box<dyn Module> {

damocles-worker/vc-processors/src/builtin/processors.rs

@@ -27,6 +27,16 @@ mod transfer;
 #[derive(Copy, Clone, Default, Debug)]
 pub struct BuiltinProcessor;
 
+impl Processor<()> for BuiltinProcessor {
+    fn name(&self) -> String {
+        String::new()
+    }
+
+    fn process(&self, _: ()) -> Result<<() as Task>::Output> {
+        Ok(())
+    }
+}
+
 impl Processor<AddPieces> for BuiltinProcessor {
     fn name(&self) -> String {
         "builtin AddPieces".to_string()

damocles-worker/vc-processors/src/core/mod.rs

@@ -51,3 +51,22 @@ impl<T: Task> Processor<T> for Box<dyn Processor<T>> {
         (**self).process(task)
     }
 }
+
+impl Task for () {
+    const STAGE: &'static str = "EMPTY";
+    type Output = ();
+}
+
+/// The DaemonProcessor does nothing
+#[derive(Debug, Default, Clone, Copy)]
+pub struct DaemonProcessor;
+
+impl Processor<()> for DaemonProcessor {
+    fn name(&self) -> String {
+        String::new()
+    }
+
+    fn process(&self, _: ()) -> Result<<() as Task>::Output> {
+        Ok(())
+    }
+}

docs/en/03.damocles-worker-config.md

@@ -589,7 +589,7 @@ Currently `{stage_name}` could be one of the following.
 - `c2`: for Commit2 phase
 - `unseal`: for unseal phase
 - `transfer`: used to customize the transfer method between local data and persistent data storage
-
+- `sealing_daemons`: used to start some custom daemons
 
 Each such configuration group translates to an external processor of the corresponding sealing phase to be started. If nothing is configured for one of the above `{stage_name}` and corresponding `[[processors.{stage_name}]]` line does not exist in the configuration file, 
 then `damocles-worker` will not start a child process for this `{stage_name}`. `damocles-worker` will use the built-in processor code to directly execute the corresponding `{stage_name}` task in `sealing_thread`, which means that the concurrent number of `{stage_name}` tasks depends on the corresponding number of `sealing_thread` and concurrent number of `{stage_name}` as configured in `[processors.limitation.concurrent]`. By not configuring external processor saves extra steps such as serializing task parameters and task output, but one loses capabilities such as more powerful concurrency control, cgroup control, and custom proof algorithms. You are feel to choose according to your own needs.

docs/zh/03.damocles-worker的配置解析.md

@@ -602,6 +602,7 @@ locks = ["gpu2"]
 - `c2`：用于 Commit2 阶段
 - `transfer`：用于自定义本地数据和持久化数据存储之间的传输方式
 - `unseal`: 用于 Unseal 阶段
+- `sealing_daemons` 用于启动一些自定义的守护进程
 
 
 每一个这样的配置组意味着将启动一个对应阶段的外部执行器。如果没有为上述的某个 `{stage_name}` 配置组配置任何内容，且配置中不存在对应的 `[[processors.{stage_name}]]` 这一行配置，