Skip to content

Commit 7097bcd

Browse files
carlbraganzacarlbraganza
committed
Updated instructions in docs/example-snapshot-metadata.md for the
alpha release of the external snapshot metadata sidecar.
1 parent a7a88c4 commit 7097bcd

File tree

2 files changed

+121
-105
lines changed

2 files changed

+121
-105
lines changed

‎deploy/kubernetes-1.30/hostpath/csi-snapshot-metadata-sidecar.patch

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
- name: csi-snapshot-metadata
22
# TODO: Replace main tag with actual released tag once external-snapshot-metadata is released
3-
image: gcr.io/k8s-staging-sig-storage/csi-snapshot-metadata:canary
3+
image: registry.k8s.io/sig-storage/csi-snapshot-metadata:v0.1.0
44
imagePullPolicy: "IfNotPresent"
55
command:
66
args:

‎docs/example-snapshot-metadata.md

+120-104
Original file line numberDiff line numberDiff line change
@@ -4,81 +4,87 @@ The CSI HostPath driver now includes support for the CSI [SnapshotMetadata](http
44

55
This document outlines the steps to test this feature on a Kubernetes cluster.
66

7-
### Deploying CSI Hostpath driver with SnapshotMetadata service
7+
### Deploy the CSI Hostpath driver with the SnapshotMetadata service
88

9-
Setting up CSI Hostpath driver with SnapshotMetadata service requires provisioning TLS certificates, creating TLS secrets, SnapshotMetadata custom resource, patching up csi-hostpathplugin deployments, etc. These steps are automated in `deploy.sh` script used to deploy CSI Hostpath driver.
9+
Setting up CSI Hostpath driver with SnapshotMetadata service requires provisioning TLS certificates, creating TLS secrets, SnapshotMetadata custom resource, patching up csi-hostpathplugin deployments, etc.
10+
These steps are automated in the `deploy.sh` script used to deploy CSI Hostpath driver when invoked with the
11+
appropriate environment variable.
1012

11-
Follow the steps below to deploy CSI Hostpath driver with SnapshotMetadata service:
12-
13-
a. Create `SnapshotMetadata` CRD
13+
Follow the steps below to deploy CSI Hostpath driver with a Kubernetes SnapshotMetadata service:
1414

15+
a. Create the `SnapshotMetadata` CRD using the definition in the
16+
[external-snapshot-metadata](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/examples/snapshot-metadata-lister) repository.
1517
```
16-
$ kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/main/client/config/crd/cbt.storage.k8s.io_snapshotmetadataservices.yaml
18+
kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/main/client/config/crd/cbt.storage.k8s.io_snapshotmetadataservices.yaml
1719
```
1820

19-
b. Execute deploy script to setup hostpath plugin driver with external-snapshot-metadata change
20-
21+
b. Execute the deploy script to setup the hostpath plugin driver with the external-snapshot-metadata sidecar:
2122
```
22-
$ SNAPSHOT_METADATA_TESTS=true HOSTPATHPLUGIN_REGISTRY=gcr.io/k8s-staging-sig-storage HOSTPATHPLUGIN_TAG=canary ./deploy/kubernetes-1.27/deploy.sh
23+
SNAPSHOT_METADATA_TESTS=true HOSTPATHPLUGIN_REGISTRY=registry.k8s.io/sig-storage HOSTPATHPLUGIN_TAG=v1.16.1 ./deploy/kubernetes-latest/deploy.sh
2324
```
25+
Specifying the `SNAPSHOT_METADATA_TESTS=true` environment variable causes it to configure the `external-snapshot-metadata` sidecar.
26+
If you wish to specify a specific hostpath image registry or tag then also set the `HOSTPATHPLUGIN_REGISTRY` and
27+
`HOSTPATHPLUGIN_TAG` environment variables.
2428

25-
### Setup SnapshotMetadata client
29+
### Setup a Kubernetes SnapshotMetadata client
2630

2731
The `SnapshotMetadata` service implements gRPC APIs. A gRPC client can query these APIs to retrieve metadata about the allocated blocks of a CSI VolumeSnapshot or the changed blocks between any two CSI VolumeSnapshot objects.
2832

29-
For our testing, we will be using a sample client implementation in Go provided as a example in [external-snapshot-metadata](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/examples/snapshot-metadata-lister) repo.
33+
For our testing, we will be using a sample client implementation in Go provided as a example in the
34+
[external-snapshot-metadata](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/examples/snapshot-metadata-lister) repository.
35+
This client performs the actions that would be taken by a real backup application to fetch snapshot metadata
36+
on allocated or changed blocks.
3037

31-
Follow the following steps to setup client with all the required permissions:
38+
The following sets up the client with all the required permissions:
3239

33-
1. Setup RBAC
40+
1. Setup RBAC for the client
3441

35-
a. Create `ClusterRole` containing all the required permissions for the client
42+
a. Create a ClusterRole containing all the required permissions for the client
3643

3744
```bash
38-
$ kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/main/deploy/snapshot-metadata-client-cluster-role.yaml
45+
kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/main/deploy/snapshot-metadata-client-cluster-role.yaml
3946
```
4047

41-
b. Create a namespace to deploy client
48+
b. Create a Namespace to deploy the client
4249

4350
```
44-
$ kubectl create namespace csi-client
51+
kubectl create namespace csi-client
4552
```
4653

47-
c. Create service account
54+
c. Create a ServiceAccount in the namespace
4855

4956
```
50-
$ kubectl create serviceaccount csi-client-sa -n csi-client
57+
kubectl create serviceaccount csi-client-sa -n csi-client
5158
```
5259

53-
d. Bind the clusterrole to the service account
60+
d. Bind the ClusterRole to this ServiceAccount
5461

5562
```
56-
$ kubectl create clusterrolebinding csi-client-cluster-role-binding --clusterrole=external-snapshot-metadata-client-runner --serviceaccount=csi-client:csi-client-sa
63+
kubectl create clusterrolebinding csi-client-cluster-role-binding --clusterrole=external-snapshot-metadata-client-runner --serviceaccount=csi-client:csi-client-sa
5764
```
5865

59-
2. Deploy sample client pod which contains [snapshot-metadata-lister](https://github.com/kubernetes-csi/external-snapshot-metadata) tool which can be used as a client to call SnapshotMetadata APIs
60-
66+
2. Deploy a Pod with the
67+
[snapshot-metadata-lister](https://github.com/kubernetes-csi/external-snapshot-metadata) tool,
68+
using the sample YAML from the
69+
[external-snapshot-metadata](https://github.com/kubernetes-csi/external-snapshot-metadata/tree/main/examples/snapshot-metadata-lister) repository:
6170
```
62-
$ kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/main/examples/snapshot-metadata-lister/deploy/snapshot-metadata-lister-pod.yaml -n csi-client
71+
kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/main/examples/snapshot-metadata-lister/deploy/snapshot-metadata-lister-pod.yaml -n csi-client
72+
```
73+
This takes some time to complete because an init container is used to compile the tool.
74+
Wait for the Pod to become ready before continuing:
75+
```
76+
kubectl wait -n csi-client --for=jsonpath='{.status.phase}'=Running --timeout=5m pod/csi-client
6377
```
6478
65-
This client performs following actions:
66-
1. Find Driver name for the snapshot.
67-
2. Discover `SnapshotMetadataService` resource for the driver which contains endpoint, audience and CA cert.
68-
3. Create SA Token with expected audience and permissions.
69-
4. Make gRPC call `GetMetadataAllocated` and `GetMetadataDelta` with appropriate params from `SnapshotMetadataService` resource, generated SA token.
70-
5. Stream response and print on console.
71-
72-
73-
### Test GetMetadataAllocated
79+
### Test the GetMetadataAllocated RPC
7480
75-
1. Create CSI Hostpath storageclass
81+
1. Create the CSI hostpath driver's StorageClass:
7682
7783
```
78-
$ kubectl create -f examples/csi-storageclass.yaml
84+
kubectl create -f examples/csi-storageclass.yaml
7985
```
8086
81-
2. Create a volume with Block mode access
87+
2. Create a dynamically provisioned PVC with Block mode access for this StorageClass:
8288
8389
```
8490
kubectl apply -f - <<EOF
@@ -97,7 +103,7 @@ This client performs following actions:
97103
EOF
98104
```
99105
100-
3. Mount the PVC to a pod
106+
3. Mount the PVC to a Pod:
101107
102108
```
103109
kubectl apply -f - <<EOF
@@ -123,8 +129,24 @@ This client performs following actions:
123129
claimName: pvc-raw
124130
EOF
125131
```
132+
Here the volume is mounted at the `/dev/loop3` device path.
133+
Wait for the Pod to become ready before continuing:
134+
```
135+
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/pod-raw
136+
```
137+
138+
4. Add some data blocks to the mounted volume:
139+
140+
```
141+
kubectl exec -it pod-raw -- dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=1 conv=notrunc
142+
kubectl exec -it pod-raw -- dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=3 conv=notrunc
143+
kubectl exec -it pod-raw -- dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=5 conv=notrunc
144+
kubectl exec -it pod-raw -- dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=7 conv=notrunc
145+
kubectl exec -it pod-raw -- dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=9 conv=notrunc
146+
```
147+
The calls above add 4Ki sized blocks of random data at block addresses 1, 3, 5, 7 and 9.
126148
127-
4. Snapshot PVC `pvc-raw`
149+
5. Snapshot the PVC creating VolumeSnapshot `raw-pvc-snap-1`
128150
129151
```
130152
kubectl apply -f - <<EOF
@@ -139,71 +161,46 @@ This client performs following actions:
139161
EOF
140162
```
141163
142-
Wait for snapshot to be ready
143-
164+
Wait for the snapshot to be ready:
165+
```
166+
kubectl get vs raw-pvc-snap-1
167+
```
168+
The output should look something like this when the snapshot is available:
144169
```
145-
$ kubectl get vs raw-pvc-snap-1
146170
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
147171
raw-pvc-snap-1 true pvc-raw 10Mi csi-hostpath-snapclass snapcontent-e17ba543-b8be-4a8e-9b0f-d708d664a0ee 99s 100s
148172
```
149173
150-
5. Now, inside `csi-client` pod which is created in previous steps, use `snapshot-metadata-lister` tool query allocated blocks metadata
174+
6. Now, inside the `csi-client` Pod that was created in previous steps, use the `snapshot-metadata-lister` tool to query
175+
for metadata on the allocated blocks:
151176
152177
```
153-
$ kubectl exec -n csi-client csi-client -c run-client -- /tools/snapshot-metadata-lister -n default -s raw-pvc-snap-1
154-
155-
Record# VolCapBytes BlockMetadataType ByteOffset SizeBytes
178+
kubectl exec -n csi-client csi-client -c run-client -- /tools/snapshot-metadata-lister -n app -s raw-pvc-snap-1
179+
```
180+
The command above requests metadata on the allocated blocks of the current snapshot (`-s` flag).
181+
The output should look something like this:
182+
```
183+
Record# VolCapBytes BlockMetadataType ByteOffset SizeBytes
156184
------- -------------- ----------------- -------------- --------------
157-
1 10485760 FIXED_LENGTH 0 4096
158-
1 10485760 FIXED_LENGTH 4096 4096
159-
1 10485760 FIXED_LENGTH 8192 4096
160-
1 10485760 FIXED_LENGTH 12288 4096
161-
1 10485760 FIXED_LENGTH 16384 4096
162-
1 10485760 FIXED_LENGTH 20480 4096
163-
1 10485760 FIXED_LENGTH 24576 4096
164-
1 10485760 FIXED_LENGTH 28672 4096
165-
1 10485760 FIXED_LENGTH 32768 4096
166-
1 10485760 FIXED_LENGTH 36864 4096
167-
1 10485760 FIXED_LENGTH 40960 4096
168-
.
169-
.
170-
.
171-
.
172-
10 10485760 FIXED_LENGTH 10452992 4096
173-
10 10485760 FIXED_LENGTH 10457088 4096
174-
10 10485760 FIXED_LENGTH 10461184 4096
175-
10 10485760 FIXED_LENGTH 10465280 4096
176-
10 10485760 FIXED_LENGTH 10469376 4096
177-
10 10485760 FIXED_LENGTH 10473472 4096
178-
10 10485760 FIXED_LENGTH 10477568 4096
179-
10 10485760 FIXED_LENGTH 10481664 4096
185+
1 10485760 FIXED_LENGTH 4096 4096
186+
1 10485760 FIXED_LENGTH 12288 4096
187+
1 10485760 FIXED_LENGTH 20480 4096
188+
1 10485760 FIXED_LENGTH 28672 4096
189+
1 10485760 FIXED_LENGTH 36864 4096
180190
```
191+
There should be 5 allocated blocks.
181192
193+
### Test the GetMetadataDelta RPC
182194
183-
### Test GetMetadataDelta
184-
185-
1. Change couple of blocks in the mounted device file in `pod-raw` Pod
186-
195+
1. Now make some changes to the mounted volume:
187196
```
188-
$ kubectl exec -ti pod-raw -- sh
189-
190-
### change blocks 12, 13, 15 and 20
191-
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=12 conv=notrunc
192-
1+0 records in
193-
1+0 records out
194-
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=13 conv=notrunc
195-
1+0 records in
196-
1+0 records out
197-
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=15 conv=notrunc
198-
1+0 records in
199-
1+0 records out
200-
/ # dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=20 conv=notrunc
201-
1+0 records in
202-
1+0 records out
203-
197+
kubectl exec -it pod-raw -- dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=2 conv=notrunc
198+
kubectl exec -it pod-raw -- dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=4 conv=notrunc
199+
kubectl exec -it pod-raw -- dd if=/dev/urandom of=/dev/loop3 bs=4K count=1 seek=9 conv=notrunc
204200
```
201+
The calls above add new blocks at block addresses 2 and 4, and modifies block at block address 9.
205202
206-
2. Snapshot `pvc-raw` again
203+
2. Snapshot the PVC again, creating VolumeSnapshot `raw-pvc-snap-2`:
207204
208205
```
209206
kubectl apply -f - <<EOF
@@ -218,25 +215,44 @@ This client performs following actions:
218215
EOF
219216
```
220217
221-
Wait for snapshot to be ready
222-
218+
Wait for the snapshot to be ready:
219+
```
220+
kubectl get vs raw-pvc-snap-2
221+
```
222+
The output should look something like this when the snapshot is available:
223223
```
224-
$ kubectl get vs
225224
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
226-
raw-pvc-snap-1 true pvc-raw 10Mi csi-hostpath-snapclass snapcontent-e17ba543-b8be-4a8e-9b0f-d708d664a0ee 7m40s 7m40s
227-
raw-pvc-snap-2 true pvc-raw 10Mi csi-hostpath-snapclass snapcontent-188562cb-03b3-4b70-b12d-28900527bca8 23s 23s
225+
raw-pvc-snap-2 true pvc-raw 10Mi csi-hostpath-snapclass snapcontent-630fc6d8-6b42-48e2-8ec0-c215a1f65882 7s 7s
228226
```
229227
230-
3. Using `external-snapshot-metadata-client` which uses `GetMetadataDelta` gRPC to allocated blocks metadata
231-
228+
3. Once again use the `snapshot-metadata-lister` tool, but this time to view the changes between the two snapshots:
229+
```
230+
kubectl exec -n csi-client csi-client -c run-client -- /tools/snapshot-metadata-lister -n default -s raw-pvc-snap-2 -p raw-pvc-snap-1
231+
```
232+
The command requests metadata on the changed blocks between the current snapshot (`-s` flag) and the previous snapshot (`-p` flag).
233+
The ouptut should look something like this:
232234
```
233-
$ kubectl exec -n csi-client csi-client -c run-client -- /tools/snapshot-metadata-lister -n default -s raw-pvc-snap-1 -p raw-pvc-snap-2
234-
235-
236235
Record# VolCapBytes BlockMetadataType ByteOffset SizeBytes
237236
------- -------------- ----------------- -------------- --------------
238-
1 10485760 FIXED_LENGTH 49152 4096
239-
1 10485760 FIXED_LENGTH 53248 4096
240-
1 10485760 FIXED_LENGTH 61440 4096
241-
1 10485760 FIXED_LENGTH 81920 4096
242-
```
237+
1 10485760 FIXED_LENGTH 8192 4096
238+
1 10485760 FIXED_LENGTH 16384 4096
239+
1 10485760 FIXED_LENGTH 36864 4096
240+
```
241+
242+
4. View the allocated blocks again:
243+
```
244+
kubectl exec -n csi-client csi-client -c run-client -- /tools/snapshot-metadata-lister -n default -s raw-pvc-snap-2
245+
```
246+
The output should look something like this:
247+
```
248+
Record# VolCapBytes BlockMetadataType ByteOffset SizeBytes
249+
------- -------------- ----------------- -------------- --------------
250+
1 10485760 FIXED_LENGTH 4096 4096
251+
1 10485760 FIXED_LENGTH 8192 4096
252+
1 10485760 FIXED_LENGTH 12288 4096
253+
1 10485760 FIXED_LENGTH 16384 4096
254+
1 10485760 FIXED_LENGTH 20480 4096
255+
1 10485760 FIXED_LENGTH 28672 4096
256+
1 10485760 FIXED_LENGTH 36864 4096
257+
```
258+
There should be 7 allocated blocks.

0 commit comments

Comments
 (0)