system-dpdk: Test with mlx5 devices.

The DPDK unit test only runs if vfio or igb_uio kernel modules are loaded:
on systems with only mlx5, this test is always skipped.

Besides, the test tries to grab the first device listed by dpdk-devbind.py,
regardless of the PCI device status regarding kmod binding.

Remove dependency on this DPDK script and use a minimal script that
reads PCI sysfs.

This script is not perfect, as one can imagine PCI devices bound to
vfio-pci for virtual machines.
Plus, this script only tries to take over vfio-pci devices. mlx5 devices
can't be taken over blindly as it could mean losing connectivity to the
machine if the netdev was in use for this system.

For those two reasons, add a new environment variable DPDK_PCI_ADDR for
testers to select the PCI device of their liking.
For consistency and grep, the temporary file PCI_ADDR is renamed
to DPDK_PCI_ADDR.

Reviewed-by: Maxime Coquelin <[email protected]>
Acked-by: Eelco Chaudron <[email protected]>
Signed-off-by: David Marchand <[email protected]>
Acked-by: Eelco Chaudron <[email protected]>

Documentation/topics/testing.rst

@@ -343,15 +343,20 @@ To see a list of all the available tests, run::
 
 These tests support a `DPDK supported NIC`_. The tests operate on a wider set of
 environments, for instance, when a virtual port is used.
-They do require proper DPDK variables (``DPDK_DIR`` and ``DPDK_BUILD``).
 Moreover you need to have root privileges to load the required modules and to bind
-the NIC to the DPDK-compatible driver.
+a PCI device to the DPDK-compatible driver.
 
 .. _DPDK supported NIC: https://core.dpdk.org/supported/#nics
 
+The phy test will skip if no suitable PCI device is found.
+It is possible to select which PCI device is used for this test by setting the
+DPDK_PCI_ADDR environment variable, which is especially useful when testing
+with a mlx5 device::
+
+    # DPDK_PCI_ADDR=0000:82:00.0 make check-dpdk
+
 All tests are skipped if no hugepages are configured. User must look into the DPDK
 manual to figure out how to `Configure hugepages`_.
-The phy test will skip if no compatible physical device is available.
 
 .. _Configure hugepages: https://doc.dpdk.org/guides-22.11/linux_gsg/sys_reqs.html
 

tests/automake.mk

@@ -188,6 +188,7 @@ SYSTEM_OFFLOADS_TESTSUITE_AT = \
 
 SYSTEM_DPDK_TESTSUITE_AT = \
 	tests/system-common-macros.at \
+	tests/system-dpdk-find-device.py \
 	tests/system-dpdk-macros.at \
 	tests/system-dpdk-testsuite.at \
 	tests/system-dpdk.at

tests/system-dpdk-find-device.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from pathlib import Path
+import os
+import sys
+
+# The tester might want to select a PCI device, if so, trust it.
+if 'DPDK_PCI_ADDR' in os.environ:
+    print(os.environ['DPDK_PCI_ADDR'])
+    sys.exit(0)
+
+for device in sorted(Path('/sys/bus/pci/devices').iterdir()):
+    class_path = device / 'class'
+    # Only consider Network class devices
+    if class_path.read_text().strip() != '0x020000':
+        continue
+    kmod_path = device / 'driver' / 'module'
+    kmod_name = kmod_path.resolve().name
+    # Only care about devices bound to vfio_pci or igb_uio.
+    if kmod_name not in ['vfio_pci', 'igb_uio']:
+        continue
+    print(device.resolve().name)
+    sys.exit(0)
+
+sys.exit(1)

tests/system-dpdk-macros.at

@@ -19,14 +19,8 @@ m4_define([OVS_DPDK_PRE_PHY_SKIP],
   [dnl Perform the precheck
    OVS_DPDK_PRE_CHECK()
 
-   dnl Check if VFIO or UIO driver is loaded
-   AT_SKIP_IF([ ! (lsmod | grep -E "igb_uio|vfio") ], [], [stdout])
-
-   dnl Find PCI address candidate, skip if there is no DPDK-compatible NIC
-   AT_CHECK([$DPDK_DIR/usertools/dpdk-devbind.py -s | head -n +4 | tail -1], [], [stdout])
-   AT_CHECK([cat stdout | cut -d" " -s -f1 > PCI_ADDR])
-   AT_SKIP_IF([ ! test -s PCI_ADDR ])
-
+   dnl Check if a device is available for DPDK
+   AT_SKIP_IF([ ! $abs_top_srcdir/tests/system-dpdk-find-device.py > DPDK_PCI_ADDR ])
 ])
 
 

tests/system-dpdk.at

@@ -63,7 +63,7 @@ OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 sleep 2
 
@@ -240,7 +240,7 @@ OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS and add policer
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl set interface phy0 ingress_policing_rate=10000 ingress_policing_burst=1000])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 sleep 2
@@ -380,7 +380,7 @@ OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS and add egress policer
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 OVS_WAIT_UNTIL([ovs-vsctl set port phy0 qos=@newqos -- --id=@newqos create qos type=egress-policer other-config:cir=1250000 other-config:cbs=2048])
 AT_CHECK([ovs-appctl -t ovs-vswitchd qos/show phy0], [], [stdout])
 sleep 2
@@ -509,7 +509,7 @@ dnl First set MTU to its default value and confirm that value, then increase the
 
 dnl Add userspace bridge and attach it to OVS with default MTU value
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 
 dnl Check default MTU value in the datapath
@@ -546,7 +546,7 @@ dnl First set an increased MTU value and confirm that value, then decrease the M
 
 dnl Add userspace bridge and attach it to OVS and modify MTU value
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl set Interface phy0 mtu_request=9000])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 
@@ -665,7 +665,7 @@ OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS and set MTU value to max upper bound
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl set Interface phy0 mtu_request=9702])
 AT_CHECK([ovs-vsctl show], [], [stdout])
 
@@ -703,7 +703,7 @@ OVS_DPDK_START()
 
 dnl Add userspace bridge and attach it to OVS and set MTU value to min lower bound
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
-AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout], [stderr])
+AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0 type=dpdk options:dpdk-devargs=$(cat DPDK_PCI_ADDR)], [], [stdout], [stderr])
 AT_CHECK([ovs-vsctl set Interface phy0 mtu_request=68])
 AT_CHECK([ovs-vsctl show], [], [stdout])