Block use of internal and external snapshots on KVM #11039

JoaoJandre · 2025-06-16T18:37:01Z

Description

On KVM, there are two types of snapshots: internal and external. Most snapshot/backup solutions use external snapshots on ACS; save for disk-and-memory VM snapshots, which use internal snapshots (this is a limitation with KVM, as far as I know).

However, since internal snapshots are stored inside the VM's volume (hence the name), if an internal snapshot is taken after an external snapshot and the external snapshot is restored, the internal snapshot is lost.

Thus, this PR blocks the use of disk-and-memory VM snapshots alongside volume snapshots, NAS backups, and disk-only VM snapshots (at least the ones created using the default volume snapshot implementation).

I encourage maintainers of 3rd party storage providers to test if their implementation is compatible with disk-and-memory VM snapshots, if it is not it their simultaneous usage should be blocked.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

I created a VM and created a few disk-and-memory VM snapshots on it; then I tried to create NAS backups, volume snapshots and disk-only VM snapshots, all of them gave me an error, which is expected.

I validated that the opposite was also true for the aforementioned cases, e.g., create volume snapshot and try to create disk-and-memory VM snapshot.

I also validated that it was possible to create multiple NAS backups, disk-only VM snapshots and volume snapshots with no issues.

JoaoJandre · 2025-06-16T18:39:18Z

@slavkap @rp- I think it would be interesting to validate if the implementations done for Storpool and Linstor are compatible with disk-and-memory VM snapshots.

codecov · 2025-06-16T18:44:53Z

Codecov Report

❌ Patch coverage is 0% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.36%. Comparing base (574ed78) to head (9f9e5f2).
⚠️ Report is 166 commits behind head on main.

Files with missing lines	Patch %	Lines
.../storage/vmsnapshot/DefaultVMSnapshotStrategy.java	0.00%	15 Missing ⚠️
...rg/apache/cloudstack/backup/NASBackupProvider.java	0.00%	5 Missing and 1 partial ⚠️
...a/org/apache/cloudstack/backup/BackupProvider.java	0.00%	3 Missing ⚠️
...tack/storage/snapshot/DefaultSnapshotStrategy.java	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #11039    +/-   ##
==========================================
  Coverage     17.36%   17.36%            
- Complexity    15237    15245     +8     
==========================================
  Files          5888     5889     +1     
  Lines        525737   525855   +118     
  Branches      64164    64188    +24     
==========================================
+ Hits          91274    91304    +30     
- Misses       424163   424245    +82     
- Partials      10300    10306     +6

Flag	Coverage Δ
uitests	`3.63% <ø> (-0.01%)`	⬇️
unittests	`18.40% <0.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JoaoJandre · 2025-06-16T18:48:48Z

@blueorangutan package

blueorangutan · 2025-06-16T18:50:04Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-16T19:10:16Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13798

blueorangutan · 2025-06-17T07:45:56Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13809

DaanHoogland · 2025-06-17T08:43:04Z

@JoaoJandre :

08:35:48 [ERROR] /jenkins/workspace/acs-centos8-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.2.0-SNAPSHOT/engine/storage/snapshot/src/test/java/org/apache/cloudstack/storage/vmsnapshot/VMSnapshotStrategyKVMTest.java:32:8: Unused import - org.apache.cloudstack.backup.dao.BackupDao. [UnusedImports]

JoaoJandre · 2025-06-17T16:38:53Z

@blueorangutan package

blueorangutan · 2025-06-17T16:40:04Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-17T17:02:31Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13818

DaanHoogland · 2025-06-18T07:04:45Z

@blueorangutan package

blueorangutan · 2025-06-18T07:08:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-18T08:16:00Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13826

rp- · 2025-07-03T08:09:10Z

Linstor does currently not support memory snapshots (we check and throw an error if selected).
So I guess we are currently not affected by any of this?

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java

slavkap

code LGTM
I haven't tested it with NFS, but the StorPool smoke tests are executed successfully

DaanHoogland · 2025-07-08T15:00:32Z

@blueorangutan test

blueorangutan · 2025-07-08T15:02:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-07-09T06:43:28Z

[SF] Trillian test result (tid-13723)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53428 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11039-t13723-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache · 2025-08-28T13:23:15Z

@JoaoJandre
this seems to be included in PR #10632
do you still want it in 4.20.2 ?

JoaoJandre · 2025-08-28T13:31:46Z

@JoaoJandre this seems to be included in PR #10632 do you still want it in 4.20.2 ?

@weizhouapache PR #10632 blocks the usage of the feature introduced in #10632 and other incompatible features. This PR purposefully ignores #10632 and adds restrictions to avoid other interactions between internal and external snapshots; such as volume snapshot and disk-and-memory VM snapshot.

They are complementary. When merging this PR forward, care should be taken so that the validations of both PRs do not erase one another (I can make the merge forward if needed).

weizhouapache · 2025-08-28T13:36:51Z

@JoaoJandre this seems to be included in PR #10632 do you still want it in 4.20.2 ?

@weizhouapache PR #10632 blocks the usage of the feature introduced in #10632 and other incompatible features. This PR purposefully ignores #10632 and adds restrictions to avoid other interactions between internal and external snapshots; such as volume snapshot and disk-and-memory VM snapshot.

They are complementary. When merging this PR forward, care should be taken so that the validations of both PRs do not erase one another (I can make the merge forward if needed).

ok @JoaoJandre
I think the best option might be re-target this PR to 4.22 which includes #10632 , to avoid re-work.

JoaoJandre · 2025-09-01T18:07:47Z

@DaanHoogland @weizhouapache I rebased the changes so now I'm targeting main.

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java

weizhouapache · 2025-09-01T18:21:40Z

@JoaoJandre
thanks for the update, overall LGTM.
left a small comment

weizhouapache

code lgtm

not tested

DaanHoogland · 2025-09-08T10:59:07Z

@JoaoJandre this is ready to ship? (asking because I see scattered test reports and am not sure of completeness)

JoaoJandre · 2025-09-08T11:29:07Z

@DaanHoogland a new round of tests would be good since I rebased from 4.20 to main.

JoaoJandre · 2025-09-08T11:29:14Z

@blueorangutan package

blueorangutan · 2025-09-08T11:30:03Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-08T12:41:33Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 14898

JoaoJandre · 2025-09-08T14:46:45Z

@blueorangutan package

blueorangutan · 2025-09-08T14:48:04Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-08T15:11:20Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 14902

JoaoJandre · 2025-09-10T19:37:13Z

@blueorangutan package

JoaoJandre · 2025-09-10T19:37:23Z

Now it should work :)

DaanHoogland · 2025-09-15T09:25:41Z

@blueorangutan package

blueorangutan · 2025-09-15T09:28:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-15T11:16:56Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 15005

blueorangutan · 2025-09-15T13:39:23Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 15010

JoaoJandre · 2025-09-15T16:24:05Z

@DaanHoogland I'm pretty sure I built this PR and it worked (the github build also worked); could we try again later?

blueorangutan · 2025-09-16T08:38:19Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15030

github-actions · 2025-09-25T08:09:28Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

boring-cyborg bot added component:backup component:storage labels Jun 16, 2025

JoaoJandre added the Severity:Major label Jun 16, 2025

JoaoJandre added this to the 4.20.2 milestone Jun 16, 2025

DaanHoogland requested review from harikrishna-patnala, rp- and slavkap June 17, 2025 06:17

slavkap reviewed Jul 8, 2025

View reviewed changes

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java Outdated Show resolved Hide resolved

slavkap reviewed Jul 8, 2025

View reviewed changes

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java Outdated Show resolved Hide resolved

slavkap approved these changes Jul 8, 2025

View reviewed changes

boring-cyborg bot added the component:integration label Sep 1, 2025

JoaoJandre changed the base branch from 4.20 to main September 1, 2025 18:05

weizhouapache reviewed Sep 1, 2025

View reviewed changes

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java Outdated Show resolved Hide resolved

address review

8938966

weizhouapache approved these changes Sep 3, 2025

View reviewed changes

weizhouapache modified the milestones: 4.20.2, 4.22.0 Sep 3, 2025

DaanHoogland closed this Sep 8, 2025

DaanHoogland reopened this Sep 8, 2025

fix tests

9f9e5f2

github-actions bot added the status:has-conflicts label Sep 25, 2025

Block use of internal and external snapshots on KVM #11039

Are you sure you want to change the base?

Block use of internal and external snapshots on KVM #11039

Conversation

JoaoJandre commented Jun 16, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

JoaoJandre commented Jun 16, 2025

Uh oh!

codecov bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JoaoJandre commented Jun 16, 2025

Uh oh!

blueorangutan commented Jun 16, 2025

Uh oh!

blueorangutan commented Jun 16, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

DaanHoogland commented Jun 17, 2025

Uh oh!

JoaoJandre commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

DaanHoogland commented Jun 18, 2025

Uh oh!

blueorangutan commented Jun 18, 2025

Uh oh!

blueorangutan commented Jun 18, 2025

Uh oh!

rp- commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

slavkap left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jul 8, 2025

Uh oh!

blueorangutan commented Jul 8, 2025

Uh oh!

blueorangutan commented Jul 9, 2025

Uh oh!

weizhouapache commented Aug 28, 2025

Uh oh!

JoaoJandre commented Aug 28, 2025

Uh oh!

weizhouapache commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoaoJandre commented Sep 1, 2025

Uh oh!

Uh oh!

weizhouapache commented Sep 1, 2025

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Sep 8, 2025

Uh oh!

JoaoJandre commented Sep 8, 2025

Uh oh!

JoaoJandre commented Sep 8, 2025

Uh oh!

blueorangutan commented Sep 8, 2025

Uh oh!

blueorangutan commented Sep 8, 2025

Uh oh!

JoaoJandre commented Sep 8, 2025

codecov bot commented Jun 16, 2025 •

edited

Loading

weizhouapache commented Aug 28, 2025 •

edited

Loading