RPMsg in Userspace outside of Petalinux #63

sunoc · 2024-12-04T07:19:51Z

sunoc
Dec 4, 2024

Hi,

Not sure if this is the best place to ask, but here is my situation:

I'm current running RPMsg on a KRIA KR260 board. The Cortex-A runs Ubuntu and communicates with FreeRTOS on the Cortex-R. The current setup is the "default" one, with a modified device tree that allows me to load the FreeRTOS firmware from the Linux.

However, I'm now trying to run the same setup but in userspace, as described in the documentation page, mostly in order to compare it with my current system.

Unfortunately I cannot really figure out where to start, what needs to be modified, what doesn't.

According to another page on the Atlassian website, a mysterious rpmsg-echo-ping-shared program should in theory answer all my questions, but so far I was unable to find the source anywhere.

Is there a resource or documentation out there that I missed and that I should have had follow? Any pointer or advice is very welcome!

Many thanks in advance for your help!

Answered by tnmysh

Dec 4, 2024

@sunoc

Above reference documentation is little dated, but most of it is still correct.

rpmsg-echo-ping-shared binary is built here: https://github.com/Xilinx/open-amp/blob/f9039c27a00caa7f1548ffd53d863776edc6f223/apps/examples/echo/CMakeLists.txt#L20

I located source file from above reference here: https://github.com/Xilinx/open-amp/blob/master/apps/examples/echo/rpmsg-ping.c

I believe it's using shared libraries (so) in linux hence "-shared" name was appended to app name.

I am not sure if the application can load firmware though as described in the documentation.
You might have to use debugger to load firmware. Or have only remoteproc nodes in kernel device-tree.
Do not include mbox, mbo…

View full answer

tnmysh · 2024-12-04T15:14:03Z

tnmysh
Dec 4, 2024
Maintainer

@sunoc

Above reference documentation is little dated, but most of it is still correct.

rpmsg-echo-ping-shared binary is built here: https://github.com/Xilinx/open-amp/blob/f9039c27a00caa7f1548ffd53d863776edc6f223/apps/examples/echo/CMakeLists.txt#L20

I located source file from above reference here: https://github.com/Xilinx/open-amp/blob/master/apps/examples/echo/rpmsg-ping.c

I believe it's using shared libraries (so) in linux hence "-shared" name was appended to app name.

I am not sure if the application can load firmware though as described in the documentation.
You might have to use debugger to load firmware. Or have only remoteproc nodes in kernel device-tree.
Do not include mbox, mbox-names, vrings and vdevbuffer information in remoteproc nodes.

That way only remoteproc part is available in kernel, and You can load firmware via remoteproc.

Thanks.

1 reply

sunoc Dec 5, 2024
Author

This is great, thank you very much, I'll check these! I guess one key part I was missing is the fact that I must find an alternative way to load the firmware for the R5 core.

tnmysh · 2024-12-04T15:25:04Z

tnmysh
Dec 4, 2024
Maintainer

Any reason for not to use RPMsg in kernelspace ? RPMSG in userspace has many limitations. Also it's not actively maintained as RPMsg in kernelspace.

1 reply

sunoc Dec 5, 2024
Author

It was more for being able to compare both ways, but indeed if the userspace system is less convenient to use, I might give up on it.

tnmysh · 2024-12-04T15:26:32Z

tnmysh
Dec 4, 2024
Maintainer

I put reference link from Xilinx open-amp repo, as petalinux is building it from there as of now. Soon all those apps will be moved out of open-amp repo to openamp-system-reference repo.

0 replies

sunoc · 2024-12-17T03:24:02Z

sunoc
Dec 17, 2024
Author

Okay, so the discussion here actually lead me to figure out how to use RPMsg in userspace on the KRIA board while still being able to load the firmware from a running Linux. I'll write it down here for the record:

Custom device tree

The device tree was modified in order to have the vring, shared memory and IPI memory areas to be "compatible with UIO", also with matching addresses with the tutorials:

--- system.dts.orig	2024-12-17 02:53:50.728339591 +0000
+++ system_uio.dts	2024-12-12 05:34:30.816562575 +0000
@@ -203,7 +203,7 @@
 
 	firmware {
 
-		zynqmp-firmware {
+		zynqmp_firmware: zynqmp-firmware {
 			compatible = "xlnx,zynqmp-firmware";
 			#power-domain-cells = <0x01>;
 			method = "smc";
@@ -715,7 +715,7 @@
 			phandle = <0x41>;
 		};
 
-		interrupt-controller@f9010000 {
+		gic: interrupt-controller@f9010000 {
 			compatible = "arm,gic-400";
 			#interrupt-cells = <0x03>;
 			reg = <0x00 0xf9010000 0x00 0x10000 0x00 0xf9020000 0x00 0x20000 0x00 0xf9040000 0x00 0x20000 0x00 0xf9060000 0x00 0x20000>;
@@ -1575,7 +1575,7 @@
 			pinctrl-names = "default";
 			u-boot,dm-pre-reloc;
 			compatible = "xlnx,zynqmp-uart\0cdns,uart-r1p12";
-			status = "okay";
+			status = "disabled";
 			interrupt-parent = <0x05>;
 			interrupts = <0x00 0x16 0x04>;
 			reg = <0x00 0xff010000 0x00 0x1000>;
@@ -1921,18 +1921,6 @@
 		reg = <0x00 0x00 0x00 0x80000000 0x08 0x00 0x00 0x80000000>;
 	};
 
-	reserved-memory {
-		#address-cells = <0x02>;
-		#size-cells = <0x02>;
-		ranges;
-
-		pmu@7ff00000 {
-			reg = <0x00 0x7ff00000 0x00 0x100000>;
-			no-map;
-			phandle = <0x7a>;
-		};
-	};
-
 	gpio-keys {
 		compatible = "gpio-keys";
 		autorepeat;
@@ -1973,6 +1961,72 @@
 		pwms = <0x1f 0x02 0x9c40 0x00>;
 	};
 
+       reserved-memory {
+           #address-cells = <2>;
+           #size-cells = <2>;
+           ranges;
+           rproc_0_reserved: rproc_0_reserved@3ec00000 {
+               no-map;
+               reg = <0x0 0x3ec00000 0x0 0x1000000>;
+	   };
+
+       };
+       tcm_0a: tcm_0a@ffe00000 {
+           no-map;
+           reg = <0x0 0xffe00000 0x0 0x10000>;
+           status = "okay";
+           compatible = "mmio-sram";
+           power-domain = <&zynqmp_firmware 15>;
+       };
+       tcm_0b: tcm_0b@ffe20000 {
+           no-map;
+           reg = <0x0 0xffe20000 0x0 0x10000>;
+           status = "okay";
+           compatible = "mmio-sram";
+           power-domain = <&zynqmp_firmware 16>;
+       };
+
+
+       rf5ss@ff9a0000 {
+           compatible = "xlnx,zynqmp-r5-remoteproc";
+           xlnx,cluster-mode = <1>;
+           ranges;
+           reg = <0x0 0xFF9A0000 0x0 0x15000>;
+           #address-cells = <0x2>;
+           #size-cells = <0x2>;
+           r5f_0 {
+               compatible = "xilinx,r5f";
+               #address-cells = <2>;
+               #size-cells = <2>;
+               ranges;
+               sram = <&tcm_0a &tcm_0b>;
+	       memory-region = <&rproc_0_reserved>;
+	       power-domain = <&zynqmp_firmware 7>;
+		};
+	};
+
+	amba_pl: amba_pl@0 {
+                #address-cells = <2>;
+                #size-cells = <2>;
+                compatible = "simple-bus";
+                ranges ;
+
+		vring: vring@0 {
+			compatible = "vring_uio";
+			reg = <0x0 0x3ed40000 0x0 0x40000>;
+		};
+		shm0: shm@0 {
+			compatible = "shm_uio";
+			reg = <0x0 0x3ed20000 0x0 0x0100000>;
+		};
+		ipi0: ipi@0 {
+			compatible = "ipi_uio";
+			reg = <0x0 0xff340000 0x0 0x1000>;
+			interrupt-parent = <&gic>;
+			interrupts = <0 29 4>;
+		};
+	};
+
 	__symbols__ {
 		pinctrl_usb1_default = "/firmware/zynqmp-firmware/pinctrl/usb1-default";
 		pinctrl_usb0_default = "/firmware/zynqmp-firmware/pinctrl/usb0-default";

OpenAMP

The library and example apps were build from source, including the libmetal. From there, test were made running the rpmsg-echo-ping-shared example.

Cortex-R5F firmware

The remotecore firmware was simply adapted by changing the addresses from what was used for kernel space mode in order to match the new device tree.

This firmware is cross-compiled on another machine, moved to the KRIA board Ubuntu; then loaded to the Cortex-R core using the usual:

echo kria-rpmsg-base-echo.elf > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state

Many thanks again for the help!

0 replies

tnmysh · 2024-12-17T18:24:38Z

tnmysh
Dec 17, 2024
Maintainer

I think this can be fixed:

+		vring: vring@0 {
+			compatible = "vring_uio";
+			reg = <0x0 0x3ed40000 0x0 0x40000>;
+		};
+		shm0: shm@0 {
+			compatible = "shm_uio";
+			reg = <0x0 0x3ed20000 0x0 0x0100000>;
+		};

To:

+		shm0: shm@0 {
+			compatible = "shm_uio";
+			reg = <0x0 0x3ed40000 0x0 0x0100000>;
+		};

I am not sure if you really need vring node. Only need shm0 node with vring address space.
Again, I am not sure, please give it a try.

2 replies

tnmysh Dec 18, 2024
Maintainer

I think node name should be: shm@3ed40000.

sunoc Dec 20, 2024
Author

Alright, this setup seems to work:

shm0: shm@3ed20000 {
		compatible = "shm_uio";
		reg = <0x0 0x3ed20000 0x0 0x0140000>;
};
ipi0: ipi@ff340000 {
		compatible = "ipi_uio";
		reg = <0x0 0xff340000 0x0 0x1000>;
		interrupt-parent = <&gic>;
		interrupts = <0 29 4>;
};

I adjusted the area for shm so it covers the vring part that was set in the programs already.

Thanks again for the input! I appreciate it!

sunoc · 2024-12-20T04:28:23Z

sunoc
Dec 20, 2024
Author

I have a doubt, however, on the actual "userspace-ness" of my system. Maybe you can help me with that?
The Linux program that sends the ping-pong messages is from the open-amp repository so it should work.

However I noticed that the firmware I'm loading to the Cortex-R spins a few suspicious kernel modules after being launched. Here are the extra modules returned by lsmod after the fw is loaded:

Module                  Size  Used by
uio_pci_generic        16384  0
vfio_pci               16384  0
vfio_pci_core          65536  1 vfio_pci
vfio_virqfd            20480  1 vfio_pci_core
vfio_iommu_type1       49152  0
vfio                   45056  2 vfio_pci_core,vfio_iommu_type1
uio_dmem_genirq        20480  0
virtio_rpmsg_bus       24576  0
rpmsg_ns               20480  1 virtio_rpmsg_bus
rpmsg_core             20480  2 virtio_rpmsg_bus,rpmsg_ns

I'll look it up and see if I can disable stuff in the code of my firmware...

3 replies

tnmysh Dec 20, 2024
Maintainer

Well, virtio_rpmsg* are expected. I am not sure about vfio though.

sunoc Dec 20, 2024
Author

Oh right, I was wondering about the rpmsg_ns and rpmsg_core, but since they are used by the virtio_rpmsg, I guess it's fine.

I'm not familiar with the VFIO, but the kernel doc says that it allows "non-privileged, userspace drivers", so I'm guessing it's the UIO equivalent to access the IOMMU.

edmooring Dec 21, 2024
Maintainer

The Xililnx OpenAMP library tree does use VFIO, if I remember correctly. I don't think that support has been upstreamed yet.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RPMsg in Userspace outside of Petalinux #63

{{title}}

Replies: 6 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

RPMsg in Userspace outside of Petalinux #63

sunoc Dec 4, 2024

Replies: 6 comments · 7 replies

tnmysh Dec 4, 2024 Maintainer

sunoc Dec 5, 2024 Author

tnmysh Dec 4, 2024 Maintainer

sunoc Dec 5, 2024 Author

tnmysh Dec 4, 2024 Maintainer

sunoc Dec 17, 2024 Author

Custom device tree

OpenAMP

Cortex-R5F firmware

tnmysh Dec 17, 2024 Maintainer

tnmysh Dec 18, 2024 Maintainer

sunoc Dec 20, 2024 Author

sunoc Dec 20, 2024 Author

tnmysh Dec 20, 2024 Maintainer

sunoc Dec 20, 2024 Author

edmooring Dec 21, 2024 Maintainer

sunoc
Dec 4, 2024

Replies: 6 comments 7 replies

tnmysh
Dec 4, 2024
Maintainer

sunoc Dec 5, 2024
Author

tnmysh
Dec 4, 2024
Maintainer

sunoc Dec 5, 2024
Author

tnmysh
Dec 4, 2024
Maintainer

sunoc
Dec 17, 2024
Author

tnmysh
Dec 17, 2024
Maintainer

tnmysh Dec 18, 2024
Maintainer

sunoc Dec 20, 2024
Author

sunoc
Dec 20, 2024
Author

tnmysh Dec 20, 2024
Maintainer

sunoc Dec 20, 2024
Author

edmooring Dec 21, 2024
Maintainer