NXP S32G BSP


1. About this document
======================
This document describes the common and non-hardware specific information.
Please refer to README.hardware for hardware specific information.

Dependencies
------------
This layer depends on the oe-core version supplied with linux-yocto kernel.


Maintenance
-----------
This layer is maintained by Wind River Systems, Inc.
Contact <[email protected]> or your support representative for more
information on submitting changes.

Building the nxp-s32g layer
---------------------------
This layer should be added to bblayers.conf. This is done automatically
when using the Wind River setup.sh wrapper.

License
-------
Copyright (C) 2018-2020 Wind River Systems, Inc.

Source code included in the tree for individual recipes is under the LICENSE
stated in the associated recipe (.bb file) unless otherwise stated.

The metadata is under the following license unless otherwise stated.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

2. BSP Kernel and Distros
=========================

The following table summarizes the valid Wind River Linux distros for this BSP.
'Y' in each content cell stands for supported; 'N' stands for not supported:

  +--------------+-------------+-------------+-------------+
  | valid/distro |   wrlinux   | wrlinux-cgl | wrlinux-ovp |
  +--------------+-------------+-------------+-------------+
  |    valid     |      Y      |      N      |      N      |
  +--------------+-------------+-------------+-------------+

For the supported kernel type for this BSP, please check the TARGET_SUPPORTED_KTYPES
by running 'bitbake -e virtual/kernel | grep "^TARGET_SUPPORTED_KTYPES="'.

Note: The preempt-rt ktype is available for this BSP/Machine at this time.


3. Board Specific Patches
=========================

To get a list of patches applied to the kernel specific to this BSP along with
patch descriptions use git-whatchanged on the default kernel. For example:

	%> cd tmp-glibc/work-shared/<bsp_name>/kernel-source
	%> git whatchanged <kernel_version>/standard/base..<kernel_version>/standard/<bsp_name>


4. Boot Instructions
====================

The typical u-boot settings apply to these boards. You will need to use
setenv, printenv and saveenv, to configure, display and store respectively
your network configuration details and kernel command line. In order to
TFTP a kernel, you need at a minimum to set the following:

	- ipaddr
	- gatewayip
	- netmask
	- serverip
	- loadaddr
	- dtbfile
	- dtbaddr
	- ethaddr
	- eth1addr

Your board may ship with the values for the MAC address of
the Ethernet interfaces set to the defaults compiled into the
u-boot image.  You should set the ethaddr, eth1addr and so on
as per the manufacturer assigned values, usually indicated with
a sticker on the board.

Target specifics are usually set in the "bootargs" variable, and the
kernel image is set in the "bootfile" variable. Don't forget that if you
want these values to be available after a power cycle you will need to run
the saveenv command.

Typically convenience macros exist that will create a complete bootargs
command line for the kernel based on variables that contain individual
settings.  Since these macros exist as env variables themselves, they
may or may not be present depending on who installed and configured
u-boot for the board.

The console device for the board is ttyLF0 at 115200 baud. For the DTB
address, 0x8FE00000  was used with success by Wind River. The load address
must be high enough to not interfere with kernel decompression. A value
of 0x8A080000 was used with success for all kernels tested by Wind River.

4.1 NFS Root File System
------------------------

Example settings for the monitor (u-boot) are shown below:

nfsboot=setenv bootargs root=/dev/nfs rw nfsroot=$serverip:$rootpath \
ip=$ipaddr:$serverip:$gatewayip:$netmask:$hostname:$netdev:off \
console=$consoledev,$baudrate $othbootargs;tftp $loadaddr $bootfile; \
tftp $fdtaddr $fdtfile;booti $loadaddr - $fdtaddr
ethaddr=00:04:9F:02:00:FD
eth1addr=00:04:9F:02:01:FD
eth2addr=00:04:9F:02:02:FD
loadaddr=0x8A080000
fdtaddr=0x8FE00000
gatewayip=192.168.1.1
netmask=255.255.255.0
ipaddr=192.168.1.100
serverip=192.168.1.2
netdev=eth0
bootfile=Image
fdtfile=fsl-s32g274a-evb.dtb or fsl-s32g274a-rdb2.dtb
rootpath=/tftpboot/rootfs
consoledev=ttyLF0
baudrate=115200

Note:
By default, in u-boot eth0 is occupied by PFE mode. When in non-PFE mode
in order to switch eth0 to the GMAC by explicitly disabling and enabling
the GMAC driver. Run below commands in u-boot:

# s32ccgmac disable
# s32ccgmac enable
# setenv ethact eth_eqos

5. Creating Partitioned Images(WIC)
===================================

User can use the OpenEmbedded Image Creator, wic, to create the properly
partitioned image on a SD card. The wic command
generates partitioned images from existing OpenEmbedded build artifacts.
User can refer to the below URL to get more WIC details:

https://docs.yoctoproject.org/singleindex.html#creating-partitioned-images-using-wic

This BSP supports disk images for SD card.
After build the project, user will get a WIC image under the directory
tmp-glibc/deploy/images/<bsp name>/ ,such as:

tmp-glibc/deploy/images/nxp-s32g/wrlinux-image-<rootfs_type>-nxp-s32g-<board>.wic

Then user can write the output image to a SD card:

5.1 Burn images to SD card
--------------------------

To burn WIC image to SD card, user only need to execute the below command:

For S32G274A-EVB:
# dd if=wrlinux-image-small-nxp-s32g-evb.wic of=/dev/your_sd_dev

For S32G274A-RDB2:
# dd if=wrlinux-image-small-nxp-s32g-rdb2.wic of=/dev/your_sd_dev

5.2 Set uboot env
-----------------

Board can boot automatically by set the below uboot environment variables:

=> setenv bootfile Image; setenv fdtfile <fsl-s32g274a-evb.dtb or fsl-s32g274a-rdb2.dtb>; setenv loadaddr 0x8A080000; setenv fdtaddr 0x8FE00000;

=> setenv bootargs console=ttyLF0,115200n8 root=/dev/mmcblk0p2 rw

=> setenv bootcmd 'fatload mmc 0:1 $loadaddr $bootfile; fatload mmc 0:1 $fdtaddr $fdtfile; booti $loadaddr - $fdtaddr'

=> saveenv; run bootcmd;

Note:
Because PFE and Ethernet GMAC ports are mutually exclusive, in order to enable
PFE or Ethernet GMAC ports on WIC image need to run u-boot commands first
before "run bootcmd":

To use Ethernet GMAC ports:
=> s32ccgmac disable;s32ccgmac enable;setenv ethact eth_eqos;

To use PFE ports:
=> pfeng stop

6. Features
===========

6.1 FLEXCAN
-----------

Detail verification commands as below:

	# ip link set can0 up type can bitrate 800000 dbitrate 8000000 fd on
	# ip link set can1 up type can bitrate 800000 dbitrate 8000000 fd on
	# candump can0 &
	# cansend can1 5A1#11.22.33.44.55.66.77.88
	# ip -d -s link show can0
	# ip -d -s link show can1

If can0 dumps data from can1, FLEXCAN feature works fine.

Note:
For S32G274A-EVB, can0 is the J79 on the S32G-PROCEVB-S board, can1 & can2 are the J19 & J20
on the S32GRV-PLATEVB board.
For S32G274A-RDB2, can0 & can1 are on the J6 on the S32G-VNP-RDB2 board and validated,
can 2 & can3 are no external interfaces and no validation.

6.2 HSE
-------

The NXP HSE (Hardware Security Engine) is a security subsystem aimed at running
relevant security functions for applications with stringent confidentiality and
authenticity requirements. This chapter describes the supported HSE features for
the S32G274 board. Detailed information can refer to "HSE Reference Manual".

HSE is a component of S32G274A SOC. It must be installed with the firmware to use.
HSE firmware is not shipped with SDK. It is released through NXP's Flexera, customers
have to contact NXP to get the access to Flexera.

6.2.1 HSE firmware
------------------

Before to use the HSE, HSE firmware must be installed and the key catalogs must
be formatted.

If you get permission to use HSE, you can get packages "HSE_FW_S32G274_0_0_8_5.exe"
and "HSE_FW_DEMO_APP_S32G274_0_0_8_5.exe" from NXP's official website. you can find
the HSE firmware pink image from "HSE_FW_S32G274_0_0_8_5.exe"

6.2.2 HSE Demo
--------------

The demo application is provided separately and contains details on how to
provision HSE FW on virgin devices and demonstrates common use cases of its
security features.

The HSE Demo is from the package "HSE_FW_DEMO_APP_S32G274_0_0_8_5.exe" of
NXP's official website.

Frist step in installing the HSE FW on a device is to generate the blob image,
the blob image is a binary image which defines the layout of external flash memory.
Contains all images used by BootRom/HSE FW to boot and configure the system at start-up.

Below steps is an example about how to verify the HSE function on M7 cores,
detailed information can refer to the document of "HSE_FW_DEMO_APP_S32G274_0_0_8_5_ReadMe.pdf"

1) Generate the blob image using scripts\blob.py script:

%> set PLATFORM=s32g2xx
%> cd demo_security_installer\scripts
%> python3 blob.py -o new -j json/s32g2xx_new_blob.json -b images/blob_s32g2xx_sd.bin -i sd

2) Flashing the blob image:

%> dd if=blob_s32g2xx_sd.bin of=/dev/your_sd_dev

3) Boot target from SD:

There will be a display below on the serial to confirm HSE is up and running.

	HSE FW Version: 0.1.0_0.8.5
	HSE FW up and running! Status:
	      HSE_STATUS_RNG_INIT_OK
	      HSE_STATUS_INIT_OK
	      HSE_STATUS_CUST_SUPER_USER

NOTICE:
	the script uses rev2 version HSE firmware to create the new blob
	image by dedault, if user's board is rev1 version, please modify the
		json/s32g2xx_new_blob.json
	file to be compatible with the HSE version.

6.2.3 HSE linux
---------------

For linux, there have a demo to verify HSE and it provides PKCS#11 functionality.
The steps are as following:

 1. Build project with HSE built-in into WIC image

    If users want to use HSE feature in linux, please specify the location
    of HSE firmware, like the following:

	HSE_LOCAL_FIRMWARE_EVB_BIN = "/path/to/hse/firmware/of/evb/board"

    in the project <build-dir>/conf/local.conf.
    Then build the project as normal:

	bitbake wrlinux-image-std

    after build finish, HSE firmware has been put into the WIC image.

 2. Initialize HSE firmware and format key stores

    After step1, using the SD card to boot the board, and in U-Boot stage,
    using the following command to format HSE key stores:

	hse_keystore_format

    If succeed, it should shows:

	MMC read: dev # 0, block # 0, count 1 ...1 blocks read: OK
	Formatting NVM and RAM key stores...
	Generating SYS_IMG...
	Writing SYS_IMG to SDcard...
	MMC write: dev # 0, block # 905, count 128...128 blocks written: OK
	Updating SYS_IMG pointer...
	MMC write: dev # 0, block # 0, count 1...1 blocks written: OK
	MMC write: dev # 0, block # 8, count 1...1 blocks written: OK

    then boot the board as normal.

 3. Run the HSE & PKCS#11 demo

    When the linux starts up, run the following command:

	pkcs-keyop /usr/lib64/libpkcs11-hse.so.1.0

    If succeed, the messages should like:

	root@nxp-s32g:~# pkcs-keyop /usr/lib64/libpkcs11-hse.so.1.0
	hse: UIO device open

	1 slots available

	Using token:
	Manufacturer......: NXP Semiconductors
	Description.......: NXP HSE Slot
	Token label.......: NXP HSE Token

	Key pair stored

	Keys available: 1
	Enumerated key label: HSE Key Pair

	Key removed
	hse: UIO device closed
	root@nxp-s32g:~#

NOTICE:

    The linux HSE & PKCS#11 demo supports S32G274AEVB board only for now,
    because the RDB2 board does not include the HSE related configurations
    and dts node in U-Boot. And this will be fixed in future. Moreover,
    there will be "hse-rng: no data available" message shown with HSE 0.8.5
    version firmware, and using 1.0.0 version HSE firmware can fix it.

6.2.4 Known issues
------------------

For HSE feature, due to the board boot flow changed from uboot to ATF by
default, and  HSE enablement flow changed accordingly, such that HSE is not
available for now, and its support status will be updated when NXP SDK bsp32
officially released.


6.3 PFE
-------

The NXP PFE (Packet Forwarding Engine) offloads the processing of Ethernet packets
from the host cores, yielding higher performance and lower power than pure software
processing can achieve.

PFE is a component of S32G274A SOC. It must be installed with the firmware to use.
PFE firmware is not shipped with SDK, customers have to contact NXP to get permission
and download the PFE firmware.

6.3.1 PFE firmware
------------------

PFE is a firmware-based module. Load the firmware before using PFE.
The firmwares name are "s32g_pfe_class.fw" and "s32g_pfe_util.fw" with
version 1.1.1, which have to be downloaded from NXP Software.

6.3.2 Steps to verify
---------------------

1. Add the "pfe" package to the system through including the
following line in your local.conf:

	IMAGE_INSTALL_append_nxp-s32g = " pfe"
        PFE_LOCAL_FIRMWARE_DIR = "<The firmwares fully directory path, eg: lib/firmware/>"

    The variable PFE_LOCAL_FIRMWARE_DIR indicates the location where can
    get the firmwares s32g_pfe_class.fw and s32g_pfe_util.fw.
    And after setting the above variables, the firmwares can be installed
    into WIC image's first boot partition automatically when build WIC image.

2. Built and install them:

	# bitbake wrlinux-image-<small/std>

Normally the PFE feature should use the WIC booting system to verify instead of
using NFS, the u-boot will default use the eth_pfeng Ethernet port mode and boot
the wrlinux. You can set the ip address for the pfe port to do a simple test
in the wrlinux. More complex usage and testing can refer to the document
"S32G Linux PFE Driver User’s Manual".

About the detailed pfe port map information can refer to the chapter "Ethernet Hardware Support on S32G2"
of the document "S32G2_LinuxBSP_31.0_User_Manual.pdf".

Note:

1. Same as NXP SDK, only the "pfe1" interface for S32G274A-EVB board and
"pfe2" interface for S32G274A-RDB2 board are verified fine. Others pfe ports
connected with "SJA1105Q" or "SJA1110A" don't work.

2. In order to use the Ethernet port with GMAC active instead of pfe in u-boot, you can
remove the s32g_pfe_class.fw and s32g_pfe_util.fw from sdcard first boot partition,

3. For S32G274A-EVB board, since the "pfe1" and GMAC Ethernet port share the same physical
port, and u-boot will force the GMAC to be disabled if detecting the pfe
firmware. So the GMAC will not be usable at this time, you can use the WIC method to
test the pfe feature or adding a PCIe Ethernet port as a replacement.

4. For S32G274A-RDB2 board, if using NFS to verify the PFE feature. You should
first switch eth to GMAC in u-boot, please refer to above chapter "4.1 NFS Root File System"
to switch eth to GMAC.

5. For S32G274A-EVB board, the connection over the SJA1105 automotive switch requires
firmware to be preloaded to the switch on every reboot:
	=> sja probe 5:0
	=> sja speed -,-,-,1G,1G 5:0
If the s32g_pfe_class.fw is not loaded, u-boot will print the following failure logs:
	Failed to change the clock source of mux 6 to 55 (CGM = 0000000044018000)
	Failed to enable clock: 20005
	Failed to enable 93 clock
	Failed to enable mac2_rx_rgmii clock

6.4 LLCE
--------

NXP LLCE (Low Latency Communication Engine) is an autonomous accelerator
designed to offload communication tasks on the CAN, LIN and Flexray
Protocols. It is a self-contained module, separate from the application
cores that would run applications (host cores). For now, LLCE CAN has
been supported, LIN and Flexray are not, same as SDK.

LLCE is a component of S32G274A SOC. It must be installed with the firmwares
to use. LLCE firmwares are not shipped with SDK. It is released through NXP's
Flexera, customers have to contact NXP to get the access to Flexera.

6.4.1 LLCE firmwares
--------------------

LLCE is a firmware-based module. Load the firmwares before using it.
There are 4 LLCE firmwares:

	dte.bin
	frpe.bin
	ppe_tx.bin
	ppe_rx.bin

with version 1.0.3 to be downloaded from NXP Software.

6.4.2 LLCE CAN verify steps
---------------------------

If users want to verify LLCE CAN feature, please add the following lines in
<build-dir>/conf/local.conf of wrlinux project:

	PNWHITELIST_wr-nxp-s32g += "llce"
	IMAGE_INSTALL_append_nxp-s32g = " llce"
	LLCE_LOCAL_FIRMWARE_DIR = "<path-to-llce-firmwares-for-s32g2-boards>"
	LLCE_LOCAL_FIRMWARE_DIR_S32G3 = "<path-to-llce-firmwares-for-s32g3-boards>"

	the variable LLCE_LOCAL_FIRMWARE_DIR indecates the firmwares(dte.bin,
	frpe.bin, ppe_tx.bin and ppe_rx.bin) location.

then build the project as usual. Boot up the board with the new built WIC
image, make sure that the test llcecan ports are connected. As an example,
using llcecan0 and llcecan14 to verify. steps as following:

  1. configure llcecan0 and llcecan14 interfaces

	ip link set up llcecan0 type can bitrate 500000 dbitrate 5000000 fd on
	ip link set up llcecan14 type can bitrate 500000 dbitrate 5000000 fd on

  2. capture can messages

	candump any,0:0,#FFFFFFFF > llce_can.log &

  3. send a packet using llcecan14

	cansend llcecan14 15575555##300112233445566778899aabbcceedd112233

  4. verify the capture output

	cat llce_can.log

  If succeed, it should output like:
	root@nxp-s32g:~# cat llce_can.log
	  llcecan0  15575555  [20]  00 11 22 33 44 55 66 77 88 99 AA BB CC EE DD 11 22 33 00 00
	  llcecan14  15575555  [20]  00 11 22 33 44 55 66 77 88 99 AA BB CC EE DD 11 22 33 00 00

6.5 ATF
-------

The ARM Trusted Firmware (ATF, or TF-A) is a software component of
the linux BSP with a double role as a bootloader and Secure Monitor.
As a bootloader, it runs before U-Boot on the A53 boot core and takes
over some of the core and SoC initializations such as clocks or MC_ME
partitions (mostly for those peripherals and subsystems that the ATF
needs). As a Secure Monitor, the TF-A provides power management services
to other software components, via a standard API/ABI called PSCI. For
details of ATF, please refer to the sections

	"21 ARM Trusted Firmware"
and
	"19 Power Management"

in "S32G2_LinuxBSP_31.0_User_Manual.pdf".

NOTICE:
    For hardware reasons, the power management features are only supported
    on S32G274A RDB REV B2 or newer boards.


6.5.1 Prerequirement
--------------------

Currentlly, the ATF is an optional template feature, if users want to enable
it, please adding the following line in <build-dir>/conf/local.conf of the
project:

	WRTEMPLATE = "feature/atf-s32g"

This will enable the ATF feature and compile and install the related files
into WIC image.

6.5.2 STR verify steps
----------------------

STR(Sleep To Ram) is a complex power management feature and is a high-level
concept, implemented by the linux kernel and the ATF, and underlaid by the
S32G SoC’s Standby state. With ATF enabled, users can verify STR using the
following command after the board boot up:

	rtcwake -d rtc0 -m mem -s 10

this command will excute STR and try to wakeup the system with RTC after
10 seconds. If succeed, the output messages should like the following:

	root@nxp-s32g:~# rtcwake -d rtc0 -m mem -s 5
	rtcwake: assuming RTC uses UTC ...
	rtcwake: wakeup from "mem" using rtc0 at Thu Jan  1 00:27:29 1970
	PM: suspend entry (deep)
	Filesystems sync: 0.000 seconds
	Freezing user space processes ... (elapsed 0.001 seconds) done.
	OOM killer disabled.
	Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
	printk: Suspending console(s) (use no_console_suspend to debug)
	NOTICE:  S32G TF-A: s32g_pwr_domain_off
	NOTICE:  S32G TF-A: s32g_pwr_domain_off_down_wfi: cpu = 1
	NOTICE:  S32G TF-A: s32g_pwr_domain_pwr_down_wfi: cpu = 2
	TICE:  S32G TF-A: s32g_pwr_domain_off
	NOTICE:  S32G TF-A: s32g_pwr_domain_pwr_down_wfi: OTICE:  S32G TF-A: s32g_get_sys_suspend_power_state
	NOTICE:  S32G TF-A: s32g_pwr_domain_suspend_pwrdown_early
	NOTICE:  S32G TF-A: s32g_pwr_domain_suspend
	NOTICE:  S32G TF-A: s32g_pwr_domain_pwr_down_wfi: cpu = 0
	NOTICE:  S32G TF-A: s32g_pwr_domain_suspend_finish
	NOTICE:  S32G TF-A: s32g_pwr_domain_on: booting up core 1
	NOTICE:  S32G TF-A: s32g_pwr_domain_on_finish: cpu 1 running
	NOTICE:  S32G TF-A: s32g_pwr_domain_on: booting up core 2
	NOTICE:  S32G TF-A: s32g_pwr_domain_on_finish: cpu 2 running
	NOTICE:  S32G TF-A: s32g_pwr_domain_on: booting up core 3
	NOTICE:  S32G TF-A: s32g_pwr_domain_on_finish: cpu 3 running
	s32cc-dwmac 4033c000.ethernet eth0: Link is Down
	s32cc-dwmac 4033c000.ethernet: Set TX clock to 125M
	s32cc-dwmac 4033c000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
	s32cc-dwmac 4033c000.ethernet eth0: Link is Down
	fsl-edma 40144000.dma-controller: WARN: There is non-idle channel.
	Disabling non-boot CPUs ...
	IRQ78: set affinity failed(-22).
	CPU1: shutdown
	psci: CPU1 killed (polled 4 ms)
	IRQ78: set affinity failed(-22).
	CPU2: shutdown
	psci: CPU2 killed (polled 4 ms)
	IRQ78: set affinity failed(-22).
	CPU3: shutdown
	psci: CPU3 killed (polled 4 ms)
	Enabling non-boot CPUs ...
	Detected VIPT I-cache on CPU1
	GICv3: CPU1: found redistributor 1 region 0:0x00000000508a0000
	CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
	CPU1 is up
	Detected VIPT I-cache on CPU2
	GICv3: CPU2: found redistributor 100 region 0:0x00000000508c0000
	CPU2: Booted secondary processor 0x0000000100 [0x410fd034]
	CPU2 is up
	Detected VIPT I-cache on CPU3
	GICv3: CPU3: found redistributor 101 region 0:0x00000000508e0000
	CPU3: Booted secondary processor 0x0000000101 [0x410fd034]
	CPU3 is up
	s32cc-dwmac 4033c000.ethernet: phy mode set to RGMII
	s32cc-dwmac 4033c000.ethernet: Failed to reset the dma
	s32cc-dwmac 4033c000.ethernet eth0: stmmac_hw_setup: DMA engine initialization failed
	s32cc-dwmac 4033c000.ethernet eth0: configuring for phy/rgmii link mode
	OOM killer enabled.
	Restarting tasks ... done.
	PM: suspend exit
	root@nxp-s32g:~#

6.5.2 CPU Hotplug verify steps
------------------------------

With ATF enabled, CPU Hotplug is supported, below is the steps
to verify this feature:

  1. check current online cpu numbers

	root@nxp-s32g:~#  nproc
	4

  2. offline cpu1

	root@nxp-s32g:~# echo 0 > /sys/devices/system/cpu/cpu1/online
	NOTICE:  SIRQ78: set affinity failed(-22).
	process 201 (rngd) no longer affine to cpu1
	process 199 (rngd) no longer affine to cpu1
	CPU1: shutdown
	psci: CPU1 killed (polled 0 ms)
	root@nxp-s32g:~# nproc
	3

  3. online cpu1

	root@nxp-s32g:~# echo 1 > /sys/devices/system/cpu/cpu1/online
	NOTICE:  S32G TF-A: s32g_pwr_domain_on: booting up core 1
	NOTICE:  S32G TF-A: s32g_pwr_domain_on_finish: cpu 1 running
	Detected VIPT I-cache on CPU1
	GICv3: CPU1: found redistributor 1 region 0:0x00000000508a0000
	CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
	root@nxp-s32g:~#
	root@nxp-s32g:~# nproc
	4
	root@nxp-s32g:~#

6.5.3 Known issues
------------------

For ATF feature, there are some known issues.

  1. USB devices can not be identified after STR

     Same as SDK, the usb devices can not be identified after STR, because
     the USB driver does not support STR currentlly, there is a STR supported
     driver list in section:
	"19.3 Suspend to RAM" in "S32G2_LinuxBSP_31.0_User_Manual.pdf"

     And this list will be updated in future.


6.5.4 Build ATF
---------------

By default, the atf image is compiled and installed. If you want atf to be manually
compiled separately, use the following command：

        # bitbake atf-s32g

6.6 IPCF
--------

IPCF is short for "Inter-Platform Communication Framework" on NXP S32G
platform, it is designed to work on various hardware platforms and operating
systems. Prior to being used it needs to be integrated in the operating
systems running on each platform. For S32G, to use this feature, users need
to manually prepare application for M7 cores(refer to the following section
6.6.1 "IPCF application on M7 side"), and the A53 linux side components
have already integrated into wrlinux(please refer to the following section
6.6.2 "Enable IPCF feature in wrlinux").

6.6.1 IPCF application on M7 side
---------------------------------

Please note that, application on M7 side is not a pre-built binary file, it
should be built manually, as preparation, the following(all of them can be
found on Flexera) shoul be setup in the following order on user's local host
machine:

	1. S32 Design Studio 3.4
	2. S32 Design Studio 3.4 Development Packages for Offline Use,
	   Support for S32G Family
	3. Real Time Drivers (RTD) 4.4 version 3.0.0 D2112 (use file
	   with _updatesite.zip in the name and install through
	   Extensions and Updates in S32 Design Studio)
	4. IPCF version 4.5.0 D2201 (use file with _updatesite.zip in
	   the name and install through Extensions and Updates in
	   S32 Design Studio)

For detail informations about the above packages setup, installation and
build steps, please refer to the documents:

	"S32DS_Installation_Guide_3.4.0.pdf"
	"SW32_RTD_4.4_3.0.0_D2112_ReleaseNotes.pdf"
	"SW32G_IPCF_4.5.0_D2201_ReleaseNotes.pdf"

and etc., which can be found in related packages downloaded from flexra.

And if the IPCF application on M7 side build successfully following the
related documents, then users can find the binary files:

	"IPCF_Example_S32G274.bin" and "IPCF_Example_S32G399_M7_0.bin"

from S32DS IPCF example project.

6.6.2 Enable IPCF feature in wrlinux
------------------------------------

If users want to enable IPCF feature on A53 side linux system, please add
the following lines in the <project-build-dir>/conf/local.conf:

	IPCF_M7_APP_BIN_DIR = "/path/to/your-ipcf-m7-app/"	
	IMAGE_INSTALL_append = ' ipc-shm'

then build the project as normal:

	bitbake wrlinux-image-std

the IPCF related drivers and M7 side app binaries(IPCF_Example_S32G274.bin and
IPCF_Example_S32G399_M7_0.bin) will be built into WIC image.

NOTICE:
    The M7 side binary default names are IPCF_Example_S32G274.bin and
IPCF_Example_S32G399_M7_0.bin, and they are defined by two variables
IPCF_M7_APP_BIN_NAME and IPCF_M7_APP_BIN_NAME_S32G3 respectively, users can
use different binary names by modifying these variables, or set related
variable(s) to empty if it is unneeded for users.


6.6.3 IPCF verify steps
-----------------------

If both the IPCF application on M7 side and IPCF related drivers in linux
are all build successfully, then users can follow the below steps to verify
this feature:

	1. dd WIC image(build following 6.6.2 section) into one SD-card

	2. using the SD-card to bootup the board

	3. stop in u-boot stage, and excute following commands

		a> disable Data Cache from uboot:
			=>dcache off
		b> zero-set SRAM shared memory used by both sample apps:
			=>mw.q 0x34000000 0x0 0x100000
		c> load IPCF binary in SRAM:
		   for S32G2:
			=>fatload mmc 0:1 0x80000000 IPCF_Example_S32G274_M7_0.bin
		   for S32G3:
			=>fatload mmc 0:1 0x80000000 IPCF_Example_S32G399_M7_0.bin
		   then:
			=>cp.q 0x80000000 0x34300000 0x30000
			=>dcache on;
		d> start M7 core (the argument is the address of the Interrupt Vector):
		   for S32G2:
			=>startm7 0x34401000;
		   for S32G3:
			=>startm7 0x34381000;
		e> boot Linux:
			=>boot

	4. after linux system startup, enable ipcf driver support on linux

		modprobe ipc-shm-sample


	5. run ipcf test command

		echo 10 > /sys/kernel/ipc-shm-sample/ping

If all the above steps succeed, the result messages should be like:

	root@nxp-s32g:~# echo 10 > /sys/kernel/ipc-shm-sample/ping
	ipc-shm-sample: starting demo...
	ipc-shm-sample: ch 0 >> 20 bytes: SENDING MESSAGES: 10
	ipc-shm-sample: ch 1 >> 32 bytes: #0 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 1 << 32 bytes: #0 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 2 >> 32 bytes: #1 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 2 << 32 bytes: #1 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 1 >> 32 bytes: #2 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 1 << 32 bytes: #2 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 2 >> 32 bytes: #3 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 2 << 32 bytes: #3 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 1 >> 32 bytes: #4 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 1 << 32 bytes: #4 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 2 >> 32 bytes: #5 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 2 << 32 bytes: #5 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 1 >> 32 bytes: #6 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 1 << 32 bytes: #6 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 2 >> 32 bytes: #7 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 2 << 32 bytes: #7 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 1 >> 32 bytes: #8 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 1 << 32 bytes: #8 HELLO WORLD! from CORE 4
	ipc-shm-sample: ch 2 >> 32 bytes: #9 HELLO WORLD! from KERNEL
	ipc-shm-sample: ch 0 << 20 bytes: REPLIED MESSAGES: 10
	ipc-shm-sample: ch 2 << 32 bytes: #9 HELLO WORLD! from CORE 4
	ipc-shm-sample: exit demo
	root@nxp-s32g:~#


6.7 Build U-Boot
----------------

By default, the u-boot image is compiled and installed. If you want u-boot to be manually
compiled separately, use the following command：

        # bitbake u-boot-s32