sriov, Assign only one PF for the worker (kubevirt#510)

* sriov, Fix sriov pfs scan error detection

In case there are no sriov pfs available,
current sriov pfs scan logic would not detect it.
Bash would keep the array non empty,
and the only element would not be a valid pf
but the path itself.

Fix logic by using find, which returns an empty
output in case no valid matches found.

Since we use local keyword, the command would not
fail the script, even that we use -e.
Therefore add a check if the array is empty,
and fail fast in this case.
This will detect the problems as soon as possible
without trailing errors.

Signed-off-by: Or Shoval <[email protected]>

* sriov, cosmetics of move_sriov_pfs_netns_to_node function

Rename ifs_name to pf_name as its actually a pf.

Add missing double quotes when using bash variables
in order to avoid word splitting, and in order to
comply with the other instances in the function.

Signed-off-by: Or Shoval <[email protected]>

* sriov, Assign only one PF for the worker

Currently sriov operator manifest actually uses just one of the PFs,
but allocates all available whitelisted PFs to the worker node.
Therefore this commit update logic
to allocate just one PF for the node, in default.

Modularity improvement:
Currently the PFs allocation function gets
the node it should allocate PFs to.
In order to increase modularity,
add a parameter which determines how many PFs it should allocate to the node.
This way we would be able to use it in a modular
way in case we need to control the number of PFs
we allocate to that node instead allocating
all the PFs to that node.

Signed-off-by: Or Shoval <[email protected]>

Author: oshoval

cluster-up/cluster/kind-k8s-sriov-1.17.0/config_sriov.sh

@@ -9,10 +9,13 @@ KUBECONFIG_PATH="${KUBEVIRTCI_CONFIG_PATH}/$KUBEVIRT_PROVIDER/.kubeconfig"
 
 MASTER_NODE="${CLUSTER_NAME}-control-plane"
 WORKER_NODE_ROOT="${CLUSTER_NAME}-worker"
+PF_COUNT_PER_NODE=${PF_COUNT_PER_NODE:-1}
 
 OPERATOR_GIT_HASH=8d3c30de8ec5a9a0c9eeb84ea0aa16ba2395cd68  # release-4.4
 SRIOV_OPERATOR_NAMESPACE="sriov-network-operator"
 
+[ $PF_COUNT_PER_NODE -le 0 ] && echo "FATAL: PF_COUNT_PER_NODE must be a positive integer" >&2 && exit 1
+
 # This function gets a command and invoke it repeatedly
 # until the command return code is zero
 function retry {
@@ -219,26 +222,34 @@ function apply_sriov_node_policy {
 
 function move_sriov_pfs_netns_to_node {
   local -r node=$1
+  local -r pf_count_per_node=$2
   local -r pid="$(docker inspect -f '{{.State.Pid}}' $node)"
   local pf_array=()
 
   mkdir -p /var/run/netns/
   ln -sf /proc/$pid/ns/net "/var/run/netns/$node"
 
-  local -r sriov_pfs=( /sys/class/net/*/device/sriov_numvfs )
-  for ifs in "${sriov_pfs[@]}"; do
-    local ifs_name="${ifs%%/device/*}"
-    ifs_name="${ifs_name##*/}"
+  local -r sriov_pfs=( $(find /sys/class/net/*/device/sriov_numvfs) )
+  [ "${#sriov_pfs[@]}" -eq 0 ] && echo "FATAL: Could not find available sriov PFs" >&2 && return 1
+
+  for pf in "${sriov_pfs[@]}"; do
+    local pf_name="${pf%%/device/*}"
+    pf_name="${pf_name##*/}"
 
-    if [ $(echo "${PF_BLACKLIST[@]}" | grep "${ifs_name}") ]; then
+    if [ $(echo "${PF_BLACKLIST[@]}" | grep "${pf_name}") ]; then
       continue
     fi
 
-    ip link set "$ifs_name" netns "$node"
-    pf_array+=($ifs_name)
+    ip link set "$pf_name" netns "$node"
+    pf_array+=("$pf_name")
+    [ "${#pf_array[@]}" -eq "$pf_count_per_node" ] && break
   done
 
-  echo ${pf_array[@]}
+  [ "${#pf_array[@]}" -lt "$pf_count_per_node" ] && \
+    echo "FATAL: Not enough PFs allocated, PF_BLACKLIST (${PF_BLACKLIST[@]}), PF_COUNT_PER_NODE ${PF_COUNT_PER_NODE}" >&2 && \
+    return 1
+
+  echo "${pf_array[@]}"
 }
 
 # The first worker needs to be handled specially as it has no ending number, and sort will not work
@@ -255,7 +266,7 @@ if [[ "$SRIOV_NODE" == "${WORKER_NODE_ROOT}0" ]]; then
   SRIOV_NODE=${WORKER_NODE_ROOT}
 fi
 
-NODE_PFS=($(move_sriov_pfs_netns_to_node $SRIOV_NODE))
+NODE_PFS=($(move_sriov_pfs_netns_to_node "$SRIOV_NODE" "$PF_COUNT_PER_NODE"))
 
 SRIOV_NODE_CMD="docker exec -it -d ${SRIOV_NODE}"
 ${SRIOV_NODE_CMD} mount -o remount,rw /sys     # kind remounts it as readonly when it starts, we need it to be writeable