Skip to content

Commit d7bdba4

Browse files
matthew-richersonmatthew-richerson
authored
Merge release v0.1.7
Release v0.1.7
2 parents 707e4d5 + 82bd630 commit d7bdba4

File tree

4 files changed

+170
-1
lines changed

4 files changed

+170
-1
lines changed

‎docs/guides/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
* [Global Lustre](global-lustre/readme.md)
1818
* [Directive Breakdown](directive-breakdown/readme.md)
1919
* [User Interactions](user-interactions/readme.md)
20+
* [System Storage](system-storage/readme.md)
2021

2122
## NNF User Containers
2223

‎docs/guides/system-storage/readme.md

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
---
2+
authors: Matt Richerson <[email protected]>
3+
categories: provisioning
4+
---
5+
6+
# System Storage
7+
8+
## Background
9+
10+
System storage allows an admin to configure Rabbit storage without a DWS workflow. This is useful for making storage that is outside the scope of any job. One use case for system storage is to create a pair of LVM VGs on the Rabbit nodes that can be used to work around an `lvmlockd` bug. The lockspace for the VGs can be started on the compute nodes, holding the `lvm_global` lock open while other Rabbit VG lockspaces are started and stopped.
11+
12+
## NnfSystemStorage Resource
13+
14+
System storage is created through the `NnfSystemStorage` resource. By default, system storage creates an allocation on all Rabbits in the system and exposes the storage to all computes. This behavior can be modified through different fields in the `NnfSystemStorage` resource. A `NnfSystemStorage` storage resource has the following fields in its `Spec` section:
15+
16+
| Field | Required | Default | Value | Notes |
17+
|-------|----------|---------|-------|-------|
18+
| `SystemConfiguration` | No | Empty | `ObjectReference` to the `SystemConfiguration` to use | By default, the `default`/`default` `SystemConfiguration` is used |
19+
| `IncludeRabbits` | No |Empty | A list of Rabbit node names | Rather than use all the Rabbits in the `SystemConfiguration`, only use the Rabbits contained in this list |
20+
| `ExcludeRabbits` | No |Empty | A list of Rabbit node names | Use all the Rabbits in the `SystemConfiguration` except those contained in this list. |
21+
| `IncludeComputes` | No | Empty | A list of compute node names | Rather than use the `SystemConfiguration` to determine which computes are attached to the Rabbit nodes being used, only use the compute nodes contained in this list |
22+
| `ExcludeComputes` | No | Empty | A list of compute node names | Use the `SystemConfiguration` to determine which computes are attached to the Rabbits being used, but omit the computes contained in this list |
23+
| `ComputesTarget` | Yes | `all` | `all`,`even`,`odd`,`pattern` | Only use certain compute nodes based on their index as determined from the `SystemConfiguration`. `all` uses all computes. `even` uses computes with an even index. `odd` uses computes with an odd index. `pattern` uses computes with the indexes specified in `Spec.ComputesPattern` |
24+
| `ComputesPattern` | No | Empty | A list of integers [0-15] | If `ComputesTarget` is `pattern`, then the storage is made available on compute nodes with the indexes specified in this list. |
25+
| `Capacity` | Yes | `1073741824` | Integer | Number of bytes to allocate per Rabbit |
26+
| `Type` | Yes | `raw` | `raw`, `xfs`, `gfs2` | Type of file system to create on the Rabbit storage |
27+
| `StorageProfile` | Yes | None | `ObjectReference` to an `NnfStorageProfile`. This storage profile must be marked as `pinned` |
28+
| `MakeClientMounts` | Yes | `false` | Create `ClientMount` resources to mount the storage on the compute nodes. If this is `false`, then the devices are made available to the compute nodes without mounting the file system |
29+
| `ClientMountPath` | No | None | Path to mount the file system on the compute nodes |
30+
31+
`NnfSystemResources` can be created in any namespace.
32+
33+
### Example
34+
35+
```yaml
36+
apiVersion: nnf.cray.hpe.com/v1alpha1
37+
kind: NnfSystemStorage
38+
metadata:
39+
name: gfs2-systemstorage
40+
namespace: systemstorage
41+
spec:
42+
excludeRabbits:
43+
- "rabbit-1"
44+
- "rabbit-9"
45+
- "rabbit-14"
46+
excludeComputes:
47+
- "compute-32"
48+
- "compute-49"
49+
type: "gfs2"
50+
capacity: 10000000000
51+
computesTarget: "pattern"
52+
computesPattern:
53+
- 0
54+
- 1
55+
- 2
56+
- 3
57+
- 4
58+
- 5
59+
- 6
60+
- 7
61+
makeClientMounts: true
62+
clientMountPath: "/mnt/nnf/gfs2"
63+
storageProfile:
64+
name: gfs2-systemstorage
65+
namespace: systemstorage
66+
kind: NnfStorageProfile
67+
```
68+
69+
## lvmlockd Workaround
70+
71+
System storage can be used to workaround an `lvmlockd` bug that occurs when trying to start the `lvm_global` lockspace. The `lvm_global` lockspace is started only when there is a volume group lockspace that is started. After the last volume group lockspace is stopped, then the `lvm_global` lockspace is stopped as well. To prevent the `lvm_global` lockspace from being started and stopped so often, a volume group is created on the Rabbits and shared with the computes. The compute nodes can start the volume group lockspace and leave it open.
72+
73+
The system storage can also be used to check whether the PCIe cables are attached correctly between the Rabbit and compute nodes. If the cables are incorrect, then the PCIe switch will make NVMe namespaces available to the wrong compute node. An incorrect cable can only result in compute nodes that have PCIe connections switched with the other compute node in its pair. By creating two system storages, one for compute nodes with an even index, and one for compute nodes with an odd index, the PCIe connection can be verified by checking that the correct system storage is visible on a compute node.
74+
75+
### Example
76+
77+
The following example resources show how to create two system storages to use for the `lvmlockd` workaround. Each system storage creates a `raw` allocation with a volume group but no logical volume. This is the minimum LVM set up needed to start a lockspace on the compute nodes. A `NnfStorageProfile` is created for each of the system storages. The `NnfStorageProfile` specifies a tag during the `vgcreate` that is used to differentiate between the two VGs. These resources are created in the `systemstorage` namespace, but they could be created in any namespace.
78+
79+
```yaml
80+
apiVersion: nnf.cray.hpe.com/v1alpha1
81+
kind: NnfStorageProfile
82+
metadata:
83+
name: lvmlockd_even
84+
namespace: systemstorage
85+
data:
86+
xfsStorage:
87+
capacityScalingFactor: "1.0"
88+
lustreStorage:
89+
capacityScalingFactor: "1.0"
90+
gfs2Storage:
91+
capacityScalingFactor: "1.0"
92+
default: false
93+
pinned: true
94+
rawStorage:
95+
capacityScalingFactor: "1.0"
96+
commandlines:
97+
pvCreate: $DEVICE
98+
pvRemove: $DEVICE
99+
sharedVg: true
100+
vgChange:
101+
lockStart: --lock-start $VG_NAME
102+
lockStop: --lock-stop $VG_NAME
103+
vgCreate: --shared --addtag lvmlockd_even $VG_NAME $DEVICE_LIST
104+
vgRemove: $VG_NAME
105+
---
106+
apiVersion: nnf.cray.hpe.com/v1alpha1
107+
kind: NnfStorageProfile
108+
metadata:
109+
name: lvmlockd_odd
110+
namespace: systemstorage
111+
data:
112+
xfsStorage:
113+
capacityScalingFactor: "1.0"
114+
lustreStorage:
115+
capacityScalingFactor: "1.0"
116+
gfs2Storage:
117+
capacityScalingFactor: "1.0"
118+
default: false
119+
pinned: true
120+
rawStorage:
121+
capacityScalingFactor: "1.0"
122+
commandlines:
123+
pvCreate: $DEVICE
124+
pvRemove: $DEVICE
125+
sharedVg: true
126+
vgChange:
127+
lockStart: --lock-start $VG_NAME
128+
lockStop: --lock-stop $VG_NAME
129+
vgCreate: --shared --addtag lvmlockd_odd $VG_NAME $DEVICE_LIST
130+
vgRemove: $VG_NAME
131+
```
132+
133+
Note that the `NnfStorageProfile` resources are marked as `default: false` and `pinned: true`. This is required for `NnfStorageProfiles` that are used for system storage. The `commandLine` fields for LV commands are left empty so that no LV is created.
134+
135+
```yaml
136+
apiVersion: nnf.cray.hpe.com/v1alpha1
137+
kind: NnfSystemStorage
138+
metadata:
139+
name: lvmlockd_even
140+
namespace: systemstorage
141+
spec:
142+
type: "raw"
143+
computesTarget: "even"
144+
makeClientMounts: false
145+
storageProfile:
146+
name: lvmlockd_even
147+
namespace: systemstorage
148+
kind: NnfStorageProfile
149+
---
150+
apiVersion: nnf.cray.hpe.com/v1alpha1
151+
kind: NnfSystemStorage
152+
metadata:
153+
name: lvmlockd_odd
154+
namespace: systemstorage
155+
spec:
156+
type: "raw"
157+
computesTarget: "odd"
158+
makeClientMounts: false
159+
storageProfile:
160+
name: lvmlockd_odd
161+
namespace: systemstorage
162+
kind: NnfStorageProfile
163+
```
164+
165+
The two `NnfSystemStorage` resources each target all of the Rabbits but a different set of compute nodes. This will result in each Rabbit having two VGs and each compute node having one VG.
166+
167+
After the `NnfSystemStorage` resources are created, the Rabbit software will create the storage on the Rabbit nodes and make the LVM VG available to the correct compute nodes. At this point, the `status.ready` field will be `true`. If an error occurs, the `.status.error` field will describe the error.

‎external/nnf-dm

Submodule nnf-dm updated 33 files

‎mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ nav:
2323
- 'Disable or Drain a Node': 'guides/node-management/drain.md'
2424
- 'Debugging NVMe Namespaces': 'guides/node-management/nvme-namespaces.md'
2525
- 'Directive Breakdown': 'guides/directive-breakdown/readme.md'
26+
- 'System Storage': 'guides/system-storage/readme.md'
2627
- 'RFCs':
2728
- rfcs/index.md
2829
- 'Rabbit Request For Comment Process': 'rfcs/0001/readme.md'

0 commit comments

Comments
 (0)