Add OpenTelemetry-native host and process monitoring

[mike-realuptime]

Summary

This PR adds OpenTelemetry Collector hostmetrics support for Linux and Windows infrastructure monitoring, including host-level metrics and logical process monitoring.

Key changes:

• Adds Linux and Windows hostmetrics mappings while preserving existing SkyWalking metric semantics.
• Adds process-hostmetrics-linux and process-hostmetrics-windows.
• Aggregates processes by normalized process name in the Collector before OTLP export.
• Normalizes Windows native hostmetrics fields, including state -> mode and process.handles -> process.open_handles.
• Enables the new process MAL rules by default.
• Preserves existing node-exporter contracts and fixes semantic differences for TCP state, CPU mode names, and cached memory.
• Adds deployable Linux and Windows Collector reference configurations.
• Extends MAL fixtures and E2E coverage, including raw Collector assertions and Windows OTLP fixture validation.

The final test suite is green across MAL runtime tests, backend build, Linux host/process E2E, Windows OTLP fixture coverage, raw Collector assertions, and the existing node-exporter regression E2E.

No Horizon/UI changes are included in this PR.

[Copilot]

🟡 Changes recommended

Several YAML files have an incomplete Apache 2.0 license header (missing the leading “Licensed to the Apache Software Foundation (ASF)…” line).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds OpenTelemetry-native host and process monitoring support (Linux + Windows) by extending existing MAL rules to accept either exporter-based metrics or OTel hostmetrics, and introduces a new E2E case that validates the OTel hostmetrics + pre-aggregated process pipeline end-to-end.

Changes:

• Extend Linux vm.yaml and Windows windows.yaml MAL rules to map both exporter and OTel hostmetrics sources into the same canonical SkyWalking meter targets (avoiding rule ownership collisions).
• Add new process MAL rules (process-hostmetrics-{linux,windows}.yaml) modeling normalized process groups as logical SkyWalking instances.
• Add a new E2E case (vm/otel-hostmetrics) including collector config, compose wiring, and workflow integration.

File summaries

File	Description
test/e2e-v2/cases/vm/prometheus-node-exporter/otel-rules/vm.yaml	Extends Linux VM MAL to accept both node-exporter and OTel hostmetrics under vm-monitoring.
test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/windows.yaml	Adds Windows VM MAL rules for the OTel-hostmetrics E2E case.
test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/vm.yaml	Adds Linux VM MAL rules for the OTel-hostmetrics E2E case.
test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-windows.yaml	Adds Windows process MAL rules mapping normalized process groups to instances.
test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/process-hostmetrics-linux.yaml	Adds Linux process MAL rules mapping normalized process groups to instances.
test/e2e-v2/cases/vm/otel-hostmetrics/otel-collector-config.yaml	Adds an otelcol-contrib config for host + process collection, normalization, grouping, and aggregation.
test/e2e-v2/cases/vm/otel-hostmetrics/expected/process-count-3.yml	Adds an assertion verifying grouped process count aggregation (=3).
test/e2e-v2/cases/vm/otel-hostmetrics/entrypoint.sh	Starts 3 sleep processes and runs the collector in the E2E container.
test/e2e-v2/cases/vm/otel-hostmetrics/e2e.yaml	Defines the new E2E verification steps for hostmetrics + process metrics.
test/e2e-v2/cases/vm/otel-hostmetrics/Dockerfile.otelcol	Builds a minimal image bundling otelcol-contrib plus the E2E entrypoint.
test/e2e-v2/cases/vm/otel-hostmetrics/docker-compose.yml	Wires OAP + BanyanDB + collector container; mounts MAL rules and collector config.
oap-server/server-starter/src/main/resources/otel-rules/windows.yaml	Updates production Windows infrastructure MAL to accept both windows-exporter and OTel hostmetrics sources.
oap-server/server-starter/src/main/resources/otel-rules/vm.yaml	Updates production Linux infrastructure MAL to accept both node-exporter and OTel hostmetrics sources.
oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-windows.yaml	Adds production Windows process MAL rules for normalized process groups.
oap-server/server-starter/src/main/resources/otel-rules/process-hostmetrics-linux.yaml	Adds production Linux process MAL rules for normalized process groups.
.github/workflows/skywalking.yaml	Adds the new “VM OpenTelemetry Hostmetrics” E2E job entry and pins otelcol-contrib version via env.

Review details

• Files reviewed: 16/16 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wu-sheng]

Thanks for the contribution. I reviewed the current head and reproduced the Collector path with otelcol-contrib:0.158.0. I think the following should be addressed before merge:

1. Fix the MAL unit-test data. All current unit-test jobs fail with 12 EMPTY results. This is not an existing exporter-output regression: the new VM/Windows hostmetrics-only expressions have no matching input in vm.data.yaml and windows.data.yaml. Please extend those fixtures, and add runtime fixtures for the two new process MAL files (which are currently compile-only).

2. The advertised Collector-side process aggregation does not occur. Every scrape still exports three process.count=1 datapoints for the three sleep processes, rather than one process.count=3 datapoint. The datapoints have slightly different timestamps, and aggregate_on_attributes includes the timestamp in its grouping key. MAL then sums the three values within the report, so the final OAP value is correctly 3 and repeated reports are averaged rather than accumulated. Please either remove the ineffective Collector aggregation and document that MAL performs the value aggregation, or implement actual pre-OTLP aggregation and assert the raw Collector output.

3. The native Windows path does not match the MAL inputs. windows.yaml filters/groups system_cpu_time by mode, but hostmetrics emits state; idle is therefore included and the per-mode result collapses. process-hostmetrics-windows.yaml expects process_open_handles, while native Windows hostmetrics emits process.handles (process_handles in MAL). No Windows normalization configuration or Windows test is included.

4. The process MAL rules are not enabled by the default OAP configuration. enabledOtelMetricsRules contains vm and windows, but neither process-hostmetrics-linux nor process-hostmetrics-windows. The E2E masks this by explicitly enabling the Linux process rule. Please either enable them by default or document the required opt-in configuration.

5. The existing node-exporter TopN-with-attributes assertion is regressed. The rewritten cpu_load1 expression removes the previous decorate({ me -> me.attr0 = me.layer.name() }), while the E2E still queries top_n(..., attr0='OS_LINUX'). Restore the decoration or intentionally update the contract and assertion.

6. Several mappings described as equivalent have different semantics:

   • tcp_curr_estab maps only state=established, whereas Tcp_CurrEstab includes ESTABLISHED and CLOSE-WAIT.
   • OTel CPU states interrupt and wait are copied unchanged, while the existing node-exporter label values are irq and iowait.
   • OTel/gopsutil cached memory includes SReclaimable, while node_memory_Cached_bytes does not, so memory_buff_cache changes meaning depending on its source.

7. process.count inherits the {threads} unit. copy_metric copies the descriptor from process.threads; the exact runtime output reports process.count Unit: {threads}. Please set the unit explicitly to {process}.

8. The production/reference configuration and Windows coverage are missing. The only Collector configuration added is explicitly Linux E2E configuration, despite the PR advertising Linux and Windows production support. Please add deployable reference documentation/configuration and Windows OTLP fixture coverage.

The process_memory_utilization * 100 expression is intentional fixed-point scaling for UI precision, so I do not consider it an output bug. Its comment should describe the basis-point scaling, and the corresponding UI expression should divide by 100.

[mike-realuptime]

Thank you very much for the detailed review.

I have addressed the points you raised and pushed an updated version of the PR.

This review has been a particularly valuable learning experience for me. Several of the issues you pointed out helped me better understand both SkyWalking's metric contracts and the OpenTelemetry Collector processing model. I sincerely appreciate the time you spent reproducing the setup and explaining the problems so precisely.

The changes are summarized below:

1. Extended the Linux VM and Windows MAL fixtures with the missing hostmetrics inputs and added runtime fixtures for both new process MAL rules.

2. Implemented real Collector-side logical process aggregation before OTLP export. The E2E now verifies the raw post-processor Collector output and confirms that three sleep processes become exactly one process.count=3 datapoint before OAP/MAL processing.

3. Added the required Windows normalization:

   • system.cpu.time: state -> mode
   • process.handles -> process.open_handles

4. Added process-hostmetrics-linux and process-hostmetrics-windows to the default enabledOtelMetricsRules.

5. Restored the historical cpu_load1 decoration so the TopN attr0=OS_LINUX contract is preserved.

6. Corrected the semantic mappings for:

   • Tcp_CurrEstab
   • CPU interrupt -> irq
   • CPU wait -> iowait
   • cached-memory semantics

7. process.count is now explicitly created with unit {process}.

8. Added deployable Linux and Windows Collector reference configurations and Windows OTLP host/process E2E coverage.

The final validation is green, including:

• MAL runtime tests
• backend build
• Linux hostmetrics E2E
• Linux process E2E
• raw Collector aggregation/normalization assertions
• Windows OTLP host/process E2E coverage
• existing node-exporter regression E2E

Thank you again for the thorough review and for the opportunity to improve both the implementation and my understanding of SkyWalking's architecture.

[wu-sheng]

Thanks for the updates. I re-reviewed 63a7ed3f52d05a6b4ce6bb768ff8bcde5b12a91b. The Linux CPU aggregation correction and documentation cleanup look good. I have two remaining requests, with the input data and verified outputs below.

1. Combine user and system before calling rate() for Windows normalized CPU.

The current expression calls rate('PT1M') separately for user and system, after each branch removes mode with .sum(['node_identifier_host_name']). Both calls consequently use the same counter-history key: metric owner, sample name, and remaining host label.

All CPU modes arrive together in one system_cpu_time sample family within each export. The collision occurs between the two rate() calls inside the same MAL evaluation.

I verified this with a complete Collector Contrib 0.158.0 export: 589 requests and 52,499 datapoints. The capture used the Linux reference configuration with 2-second collection intervals and a sleep process allowlist. I then applied the PR's Windows host normalization, including state to mode, preserving all values, timestamps, and request boundaries. This is real Linux collection replayed against the Windows MAL; I did not capture native Windows telemetry.

Each full request was decoded by the actual OpenTelemetryMetricRequestProcessor, then evaluated through compiled MAL and the production Expression.run wrapper. This replay stopped before storage/query processing.

Here is the CPU portion of two complete exports, exactly 60 seconds apart. CPU time values are cumulative seconds; all eight modes were included together in each export, along with the other metric families.

Input	First export	Export 60 seconds later
mode=user	4.99	5.53
mode=system	11.79	12.97
mode=idle	5520.20	6116.68
mode=interrupt	0	0
mode=nice	0	0
mode=softirq	1.47	1.80
mode=steal	0	0
mode=wait	3.14	3.14
system_cpu_logical_count	10	10

The expected normalized user + system percentage is:

((5.53 - 4.99) + (12.97 - 11.79)) / 60 / 10 * 100
= 0.2866666667%

MAL	Actual compiled output
Current PR: separate user/system rates	1.4200000000%
Proposed: combine user/system, then one rate	0.2866666667%

In the current expression, the system branch uses the user baseline 4.99, producing (12.97 - 4.99) / 60 * 100 = 13.3. Adding the user branch's 0.9 and dividing by 10 gives the incorrect 1.42%.

Please use:

- name: cpu_norm_percentage
  exp: >
    (system_cpu_time * 100)
      .tagMatch('mode','^(user|system)$')
      .sum(['node_identifier_host_name'])
      .rate('PT1M')
    /
    system_cpu_logical_count.sum(['node_identifier_host_name'])

The regex works in compiled MAL and selects exactly user and system. Please synchronize the production and E2E rule copies.

2. Move the intended CPU regression samples into input and update the assertions.

The new system_cpu_time block, containing user 20, system 12, interrupt 8, and idle 60, is under expected. It is therefore neither supplied as input nor checked as a rule output. The actual input still contains only user 20 and idle 80, with 4 logical CPUs.

This explains why the existing fixture passes without detecting the issue:

Test input	Expected normalized output	Current MAL	Proposed MAL
Existing fixture, using its normal counter priming	125	125 — PASS	125 — PASS
Two complete real exports shown above	0.2866666667	1.42 — FAIL	0.2866666667 — PASS

Both comparisons were executed with compiled MAL. Please move the misplaced samples into input, update the affected expectations, and include a regression with user and system present together. Include interrupt data to verify that it is excluded.

Validation: the backend build and all 1,437 local MAL module tests passed. The full exported-data replay evaluated each CPU expression 295 times; 294 outputs differed, with both expressions returning zero for the initial observation without counter history. Replaying the two complete exports above separately also reproduced the failure and verified the proposed result.

[wu-sheng]

@mike-realuptime Any update about this?

[mike-realuptime]

Thanks for the follow-up. I reproduced the issue and confirmed both remaining points.

I have updated the Windows normalized CPU expression to combine the user and system modes before calling rate(), using the suggested tagMatch('mode','^(user|system)$') expression. The production and E2E MAL copies are synchronized.

I also moved the user/system/interrupt/idle regression samples into the fixture input section and updated the expected values. The fixture now explicitly verifies that interrupt is present in the source data but excluded from normalized CPU calculation.

I reran the MAL tests and the relevant validation successfully.

Thanks again for the precise reproduction and explanation.

[wu-sheng]

Please submit your horizon side template updates.