Below are information to complement and clarify UG1186 "Getting Started Guide" for Zynq UltraScale+ MPSoC.
Quick try!
Here are the basic steps to boot Linux and run an openamp application using pre-built images.
e.g for ZCU102:
The echo-test application sends packets from Linux running on quad-core Cortex-A53 to a single cortex-R5 running FreeRTOS which send them back.
Extract files BOOT.BIN, image.ub and openamp.dtb files from a pre-built Petalinux BSP tarball to sdcard
host shell$ tar xvf xilinx-zcu102-v2017.2-final.bsp --strip-components=4 --wildcards */BOOT.BIN */image.ub */openamp.dtb
host shell$ cp BOOT.BIN image.ub openamp.dtb <your sd card>
Note: Alternatively, if you already created a Petalinux project with a provided BSP for your board, pre-built images can also be found under the <your project>/pre-built/linux/images/ directory.
Go to u-boot prompt and boot Linux from sdcard
...
Hit any key to stop autoboot: 0
ZynqMP> mmcinfo && fatload mmc 0 ${netstart} ${kernel_img} && fatload mmc 0 0x14000000 openamp.dtb
Device: sdhci@ff170000
...
reading image.ub
31514140 bytes read in 2063 ms (14.6 MiB/s)
reading openamp.dtb
38320 bytes read in 18 ms (2 MiB/s)
ZynqMP> bootm $netstart $netstart 0x14000000
...
Note: As an alternative to all steps above to sd-boot, you can jtag-boot the board. For this you need to have connected a jtag cable, installed jtag drivers and created a Petalinux project using a provided BSP. You would then go into the <your project>/pre-built/linux/images directory and replace file system.dtb by openamp.dtb, then enter: "petalinux-boot --jtag --prebuilt 3"
At Linux login prompt enter 'root' for user and 'root' for password and run echo-test demo
plnx_aarch64 login: root
Password:
root@plnx_aarch64:~# echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware
root@plnx_aarch64:~# echo start > /sys/class/remoteproc/remoteproc0/state
[ 177.375451] remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
[ 177.384705] remoteproc remoteproc0: Booting fw image image_echo_test, size 644144
[ 177.396832] remoteproc remoteproc0: registered virtio0 (type 7)
[ 177.399108] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 177.412370] zynqmp_r5_remoteproc ff9a0100.zynqmp_r5_rproc: RPU boot from TCM.
[ 17Starting application...
Try to init remoteproc resource
Init remoteproc resource succeeded
Waiting for events...
7.422089] remoteproc remoteproc0: remote processor ff9a0100.zynqmp_r5_rproc is now up
[ 177.442121] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1
root@plnx_aarch64:~# modprobe rpmsg_user_dev_driver
[ 188.089835] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
[ 188.101250] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!
root@plnx_aarch64:~# echo_test
Echo test start
Open rpmsg dev!
[ 190.364739] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: Sent init_msg to target 0x1.
****************************************
Please enter command and press enter key
****************************************
1 - Send data to remote core, retrieve the echo and validate its integrity ..
2 - Quit this application ..
CMD>
Docs and source code:
Documents:
The following document describes libmetal APIs:
{libmetal-doc-20170418.pdf}
URLs to source code:
Xilinx Openamp related code:
The following location provide access to the code:
https://github.com/Xilinx/open-amp/tree/xilinx-v2017.2
https://github.com/Xilinx/libmetal/tree/xilinx-v2017.2
https://github.com/Xilinx/meta-openamp/tree/rel-v2017.2
https://github.com/Xilinx/linux-xlnx/tree/xilinx-v2017.2
Xilinx SDK template (which uses https://github.com/Xilinx/embeddedsw/tree/xilinx-v2017.2)
e.g for main components:
Demo Applications:
https://github.com/Xilinx/meta-openamp/tree/rel-v2017.2/recipes-openamp/rpmsg-examples
https://github.com/Xilinx/open-amp/tree/xilinx-v2017.2/apps
https://github.com/Xilinx/libmetal/tree/xilinx-v2017.2/examples/system
Xilinx SDK template for RPU firmware (echo test, matrix multiplication and rpc demo)
Libraries:
https://github.com/Xilinx/open-amp/tree/xilinx-v2017.2/lib
https://github.com/Xilinx/libmetal/tree/xilinx-v2017.2/lib
LKMs
https://github.com/Xilinx/meta-openamp/tree/rel-v2017.2/recipes-kernel/rpmsg-user-module/files
https://github.com/Xilinx/meta-openamp/tree/rel-v2017.2/recipes-kernel/rpmsg-proxy-module/files
https://github.com/Xilinx/linux-xlnx/tree/xilinx-v2017.2/drivers/rpmsg
https://github.com/Xilinx/linux-xlnx/tree/xilinx-v2017.2/drivers/remoteproc
OpenAMP framework OSS:
https://github.com/OpenAMP
FAQ:
Is there a way to reduce Petalinux build time with OpenAMP?
To reduce extra (re)-compilation time for the remote processor firmware built with Petalinux and to preserve the source code in the temporary build directory:
Edit your <petalinux-project>/project-spec/meta-user/conf/petalinuxbsp.conf file and add:
RM_WORK_EXCLUDE += "openamp-fw-echo-testd openamp-fw-mat-muld openamp-fw-rpc-demo"
Remote firmware failed to boot and now I see an error saying "failed to declare rproc mem as DMA mem", why?
This happens after an invalid image is provided to remoteproc and this one exited before freeing all allocated memory, preventing further allocation on next run.
In this situation, in order to load a new openamp image you need to reboot Linux.
The patch below will take care of fixing remoteproc so that you are not forced to reboot Linux.
{0001-remoteproc-resource_cleanup-releases-DMA-declared-me.patch}
Note this however doesn't fix the root cause of the issue, which is probably the footprint of the elf image provided to remotproc doesn't match the allocated memory in the DTS.
Additional examples:
ZynqMP Linux Master running on APU with RPMsg in kernel space and 2 RPU slaves.
Enabling Linux Drivers and other packages
Proceed as indicated in UG1186 to enable Linux remoteproc driver support and other openamp packages.
Device tree:
Add content of file openamp-overlay-split.dtsi to file:
<petalinux project>/project-spec/meta-user/recipes-bsp/device-tree/file/system-user.dtsi
rebuild the device tree
petalinux-build -c device-tree
Building remote processor demo applications to run on RPU 0 (cortex_r5_0) with Xilinx SDK
Proceed as documented in UG1186 to generate remote processor openamp applications with Xilinx SDK.
RPU 0 is also used by default for the pre-built applications provided with Petalinux BSPs.
Building remote processor demo applications to run on RPU 1 (cortex_r5_1) with Xilinx SDK
Remote processor applications (echo_test, matrix multiply, rpc demo) code is by default set to run with RPU 0 and need to be slightly modified for RPU-1.
When RPU-1 is selected in Xilinx SDK, the code generated need to be modified as follow:
Edit platform_info.h and replace IPI_IRQ_VECT_ID value 65 by 66
Edit platform_info.c and replace IPI_BASE_ADDR value 0xFF310000 by 0xFF320000
Check that the application linker script (lscript.ld) addresses match and fit the DTS zynqmp_r5_rproc memory sections.
Check that the appplication rsc_table.c address for RSC_RPROC_MEM carveout is not overlapping the linker script addresses.
Example: Running two echo_test application concurrently on Linux, each communicating to one RPU
Use Petalinux to build/boot your target and then login to Linux console serial port.
If you haven't added the remote processor firmware applications to your Linux root filesystem (see UG1186 ch. 3) you can tftp them in the target directory /lib/firmware
Check remoteproc driver is already loaded (normally it is if your device tree is properly configured):
root@plnx_aarch64:/lib/firmware# lsmod
Tainted: G
virtio_rpmsg_bus 20480 0 - Live 0xffffff800098e000
rpmsg_core 16384 1 virtio_rpmsg_bus, Live 0xffffff800097c000
zynqmp_r5_remoteproc 16384 0 - Live 0xffffff800096a000
remoteproc 40960 1 zynqmp_r5_remoteproc, Live 0xffffff8000959000
virtio 16384 2 virtio_rpmsg_bus,remoteproc, Live 0xffffff8000951000
virtio_ring 20480 2 virtio_rpmsg_bus,remoteproc, Live 0xffffff8000948000
uio_pdrv_genirq 16384 0 - Live 0xffffff8000940000
Load rpmsg_user_dev_driver LKM:
root@plnx_aarch64:/lib/firmware# modprobe rpmsg_user_dev_driver
Start RPU-0:
root@plnx_aarch64:/lib/firmware# echo image_echo_test_r5_0 > /sys/class/remoteproc/remoteproc0/firmware
root@plnx_aarch64:/lib/firmware#
root@plnx_aarch64:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc0/state
root@plnx_aarch64:/lib/firmware#
[70982.961635] remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
[70982.971366] remoteproc remoteproc0: Booting fw image image_echo_test_r5_0, size 638724
[70982.985672] virtio_rpmsg_bus virtio0: rpmsg host is online
[70982.993691] remoteproc remoteproc0: registered virtio0 (type 7)
[70983.002197] zynqmp_r5_remoteproc ff9a0100.zynqmp_r5_rproc: RPU boot from TCM.
[7Starting application...
Try to init remoteproc resource
Init remoteproc resource succeeded
Waiting for events...
0983.012367] remoteproc remoteproc0: remote processor ff9a0100.zynqmp_r5_rproc is now up
[70983.032821] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1
[70983.043731] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
root@plnx_aarch64:/lib/firmware# [70983.055535] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!
Start RPU-1:
root@plnx_aarch64:/lib/firmware# echo image_echo_test_r5_1 > /sys/class/remoteproc/remoteproc1/firmware
root@plnx_aarch64:/lib/firmware#
root@plnx_aarch64:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc1/
[71185.157615] remoteproc remoteproc1: powering up ff9a0200.zynqmp_r5_rproc
[71185.167453] remoteproc remoteproc1: Booting fw image image_echo_test_r5_1, size 639140
[71185.182180] virtio_rpmsg_bus virtio1: rpmsg host is online
[71185.190226] remoteproc remoteproc1: registered virtio1 (type 7)
[71185.198724] zynqmp_r5_remoteproc ff9a0200.zynqmp_r5_rproc: RPU boot from TCM.
[7Starting application...
Try to init remoteproc resource
Init remoteproc resource succeeded
Waiting for events...
1185.208915] remoteproc remoteproc1: remote processor ff9a0200.zynqmp_r5_rproc is now up
[71185.229420] virtio_rpmsg_bus virtio1: creating channel rpmsg-openamp-demo-channel addr 0x1
[71185.240367] rpmsg_user_dev_driver virtio1:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
root@plnx_aarch64:/lib/firmware# [71185.252200] rpmsg_user_dev_driver virtio1:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!
Run echo_test Linux application with RPU-0 using either the serial port or other telnet or ssh connection:
root@plnx_aarch64:/lib/firmware# echo_test
Echo test start
Open rpmsg dev!
[71507.962881] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: Sent init_msg to target 0x1.
****************************************
Please enter command and press enter key
****************************************
1 - Send data to remote core, retrieve the echo and validate its integrity ..
2 - Quit this application ..
CMD>
Run a concurrent echo_test Linux application with RPU-1 using another connection (telnet, ssh...):
root@plnx_aarch64:/lib/firmware# echo_test -d /dev/rpmsg1
Echo test start
Open rpmsg dev!
****************************************
Please enter command and press enter key
****************************************
1 - Send data to remote core, retrieve the echo and validate its integrity ..
2 - Quit this application ..
CMD>
__Note__: The order in which you start the RPU determines which /dev/rpmsgX device is being used with that RPU.
In the above case /dev/rpmsg0 is used for RPU-0.
If however RPU-1 was started first, it would have been associated with /dev/rpmsg0 and RPU-0 would have been using /dev/rpmsg1.
ZynqMP Linux Master running on APU with RPMsg in kernel space and only one RPU slave or RPU in lockstep.
When running with RPU in split mode and only one RPU is an OpenAMP slave, the second RPU can still run another non-openamp application.
RPU-0 slave:
Petalinux BSPs provide a default template to generate a DTB with support for OpenAMP running on RPU-0, see:
<petalinux project>/project-spec/meta-usr/recipes-bsp/device-tree/files/openamp-overlay.dtsi
Add its content to file <petalinux project>/project-spec/meta-user/recipes-bsp/device-tree/file/system-user.dtsi
RPU-1 slave:
Proceed as for the two RPU configuration above and edit your device tree to remove the unused 'zynmp_r5_rproc' entry and associated nodes (tcm, pd,...) that may not be needed any more.
RPU in lockstep:
When running with RPU in lockstep mode, the setup is almost as if running on RPU-0, however the device tree is slightly different, please see openamp-overlay-lockstep.dtsi
Note: Depending on your BSP, you may need to update the Vivado design to add RPU to the isolation configuration, mark it non-secure, and assign 4 TCMs.
ZynqMP Linux Master running on APU with RPMsg in kernel space, APU boots RPU and then later APU restart while RPU keeps running
In this use case, APU boots first into Linux, then boots the remote processor firmware on RPU with remoteproc.
The APU subsystem is then restarted while the RPU firmware keeps running.
After APU is restarted, you try to load the RPU firmware with remoteproc, this one then check:
whether RPU is running
whether firmware checksum resource type "RSC_FW_CHKSUM" is present in the resource table.
If present, it checks if the firmware checksum matches the value calculated from the loadable sections of the firmware.
If the checksum matches, remoteproc does not restart the firmware, and simply notify the RPU to reset virtio
Then both RPU and APU will initialize the RPMsg/virtio to reinitialize the communication
If the firmware is not running or checksum resource is not present or checksum does not match, the RPU will be loaded with the new firmware.
Configure and Build PetaLinux
Please follow same steps as for use case with Linux APU running remoteproc and RPMsg in kernel space to configure PetaLinux.
Additionally, check and if necessary enable ZynqMP debug FS support from kernel menuconfig:
petalinux-config -c kernel
Kernel hacking --->
Compile-time checks and compiler options --->
[*] Debug Filesystem
Device Drivers --->
SOC (System On Chip) specific Drivers ---->
[*] Xilinx Zynq MPSoC driver support ---->
[*] Enable Xilinx Zynq MPSoC Power Management API debugfs functionality
[*] Enable Zynq MPSoC generic PM domains
Building remote processor demo applications with Xilinx SDK
Proceed as documented in UG1186 to generate remote processor openamp applications with Xilinx SDK.
For this example we use the "echo test" demo for RPU-0, and we want it to ignore APU requests to shutdown generated on reboot by rpmsg_user_dev_driver LKM.
Modify echo_test.c and comment out the 'break' that is being called in the main while loop when 'evt_chnl_deleted' is received.
/* we got a shutdown request, exit */
if (evt_chnl_deleted) {
//break;
}
Running the demo step-by-step:
Log into Linux, then start RPU firmware, e.g:
echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
modprobe rpmsg_user_dev_driver
Restart APU subsystem only:
echo system_shutdown 2 0 > /sys/kernel/debug/zynqmp_pm/power
reboot
When Linux boots again, log into Linux, then:
echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
You shall be able to see on the console a message from remoteproc LKM saying:
"remoteproc remoteproc0: remote is already running. Do not restart"
Re-create the connection to RPU and start the Linux application:
modprobe rpmsg_user_dev_driver
echo_test
ZynqMP Linux Master running on APU Linux loads arbitrary RPU Firmware
Overview
The information below is intended to provide guidance to users who wish to set up a Linux + Bare-metal,RTOS, etc.This configuration relies on the FSBL to start the software running on the APU, and then APU Linux using remoteproc will load the RPU.
To Boot RPU Firmware via APU with Linux
These instructions assume the user has already generated firmware for the RPU and that the user is using Petalinux to create their embedded Linux solution.
1. As directed in User Guide 1186 Chapter 3, create an application inside of the Petalinux project to install the firmware into the Linux host's file system in /lib/firmware.
If creating a new application with the SDK, you may need to update the linker script DDR address to match the DTS address below (0x3ed00000).
2. Modify the device tree at project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi. For example:
/ {
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
rproc_0_reserved: rproc@3ed000000 {
no-map;
/* DDR memory reserved for RPU firmware.
* If you want to use predefined shared memory,
* you should also reserved them here.
*/
reg = <0x0 0x3ed00000 0x0 0x1000000>;
};
};
power-domains {
/* For TCM memories, you will need specify the power domain
* IDs. As APU will need to use the power domain ID to request
* access through PMU FW.
*/
pd_r5_0: pd_r5_0 {
#power-domain-cells = <0x0>;
pd-id = <0x7>;
};
pd_tcm_0_a: pd_tcm_0_a {
#power-domain-cells = <0x0>;
pd-id = <0xf>;
};
pd_tcm_0_b: pd_tcm_0_b {
#power-domain-cells = <0x0>;
pd-id = <0x10>;
};
};
amba {
/* You will need to specify the firmware memory as "mmio-sram". */
r5_0_tcm_a: tcm@ffe00000 {
compatible = "mmio-sram";
reg = <0 0xFFE00000 0x0 0x10000>;
pd-handle = <&pd_tcm_0_a>;
};
r5_0_tcm_b: tcm@ffe20000 {
compatible = "mmio-sram";
reg = <0 0xFFE20000 0x0 0x10000>;
pd-handle = <&pd_tcm_0_b>;
};
elf_ddr_0: ddr@3ed00000 {
compatible = "mmio-sram";
reg = <0 0x3ed00000 0x0 0x40000>;
};
test_r50: zynqmp_r5_rproc@0 {
compatible = "xlnx,zynqmp-r5-remoteproc-1.0";
reg = <0x0 0xff9a0100 0 0x100>, <0x0 0xff340000 0 0x100>, <0x0 0xff9a0000 0 0x100>;
reg-names = "rpu_base", "ipi", "rpu_glbl_base";
dma-ranges;
core_conf = "split0";
/* Specify the firmware memories here */
sram_0 = <&r5_0_tcm_a>;
sram_1 = <&r5_0_tcm_b>;
sram_2 = <&elf_ddr_0>;
pd-handle = <&pd_r5_0>;
interrupt-parent = <&gic>;
interrupts = <0 29 4>;
} ;
};
};
3. Run the following to build your petalinux project.
petalinux-build
4. After booting the Petalinux project, run the following to boot the RPU firmware onto RPU.
echo <name of firmware> > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state