hdmi2csi

The HDMI2CSI module

Quicklinks

• Drivers
• Compiling (or use our prebuilt image)
• Capturing Video
• Future Development
• Tests

---

Introduction

The High-Performance Multimedia Group has developed a High Definition Multimedia Interface (HDMI®) to MIPI®Camera Serial Interface Type 2 (CSI-2) converter module (HDMI2CSI) as a plug in to the NVIDIA Jetson TX1 development kit.

The HDMI2CSI module supports 4K video resolution for next-generation embedded Ultra High Definition video applications. The HDMI2CSI module offers two 4K/2K HDMI video and audio streams to be simultaneously converted in MIPI CSI-2 video and TDMI audio format that can be processed by the Jetson TX1 processor.

More information about the hardware as well as contact information for ordering is available at the HPMM-blog.

Fig. 1: The HDMI2CSI Board attached to the Nvidia TX1 Evaluation Board

Drivers

The drivers to use the HDMI2CSI board are available in the https://github.com/InES-HPMM/linux-l4t/ repository. There are different branches for different versions of the underlying operating system. The OS is Linux4Tegra from Nvidia, which is based on the Linux kernel 3.10. The hdmi2csi/l4t-r23-1 branch is a fork of the tegra-l4t-r23.1 tag by Nvidia with custom drivers for the TC358840 HDMI-to-CSI bridge chip on top. There is also a branch for the newest Release R24.1 and a development branch for new driver features.

Features

The supported features vary based on the branch:

Branch	L4T Version	Dynamic Format Resolve	HDMI-In Ports	Max. Resolution	Status	EDID	Audio
hdmi2csi/l4t-r24-2.1	R24.2.1	Yes	A and B (requires U-Boot modification, see #1	2160p30	Under active development	Fixed. Native: 2160p30, 1080p50. Extended: 2160p30, 1080p60, 720p29.97/30, 1080p30, 1080p50, 1080p29.97/30.	Stereo Capture on HDMI In Port B with separate cable
hdmi2csi/l4t-r24-1-dev-4K	R24.1	Yes	A (B can be used parallel from commit a2b2fc9	2160p30	Development stopped	Fixed (checksum error)	-

Deprecated: hdmi2cs/l4t-r23-1, hdmi2cs/l4t-r23-1-dev-4K, hdmi2cs/l4t-r24-1

#####Drivers for L4T R24.2.1 Work In Progress!

hdmi2csi/l4t-r24-2.1

A first port of our drivers for L4T R24.2.1 is available. We have successfully tested HDMI capture of 3840x2160p30 on HDMI-A, but it is still very much in development. We expect to solve most of the issues in the coming days.

Before building Image/modules/dtbs, make sure to load the correct kernel configuration for the hdmi2csi board: make tegra21_hdmi2csi_defconfig

If you are interested in contributing to driver development, please contact us via email ([email protected]) or via this repository.

Issues:

• HDMI-Capturing on HDMI-B is temporarily disabled due to the init failing --> Workaround Pinmux
• Note that we are building the drivers on R24.2**.1** (even newer than 24.2). Please use a compatible Root-FS (e.g. the Ubuntu16.04 coming with the new Jetpack 2.3**.1**)
• Nvidia has introduced the driver for the TC358840 in 24.2, but we are not (yet) compatible with their camera implementation. Nvidia seems to currently only support max. 4 CSI lanes and capturing for 1080p60. Therefore we will wait until they support full 8 CSI lanes and capturing 2160p30 before possibly using their interfaces for the TC358840.
  • It should still be possible to use the new Multimedia API with our new drivers. We will publish more information as soon as we have tested this.
  • If you are interested in the TC358840 implementation of NVidia, take a look at http://nv-tegra.nvidia.com/gitweb/?p=linux-3.10.git&a=search&h=4206e3f9c9b3a1757c36bb798ca751c09e2db8c5&st=commit&s=tc358840
  • Note that the dt-bindings we use currently differ from those described by Nvidia.

Structure

The drivers are written in the Video4Linux2 (V4L2) framework and have full device tree support.

Three main components are necessary for a V4L2 driver:

• Host driver: tegra_vi2 - This driver has originally been developed by Avionic Design for the Meerkat Nvidia TK1 platform and was ported to the TX1 by the HPMM research group
• Sensor (subdevice) driver: tc358840 - Based on the predecessor TC358743, this driver was developed by the HPMM research group and is getting into Linux Media
• Video buffer driver: tegra_vi2_vidq - The videobuf2 API is used with contiguous DMA buffers

Fig.2: Hardware and software components for using HDMI2CSI with a Nvidia Jetson TX1 host processor

Configuration

The hardware-specific device configuration is handled in the Device Tree source file. It consists of two parts:

• The two TC358840 subdevices as I2C nodes

i2c7: i2c@546c0000 {
		/* HDMI IN A (4K) */
		tc358840xbg@0f {

			...

		};

		/* HDMI IN B (Full HD) */
		tc358840xbg@1f {
			
			...

		};
};

• and the video interface of the TX1 (in short: VI2) with two ports

host1x {
		vi {
			...

			/* HDMI IN A */
			port@0 {
				...
			};

			/* HDMI IN B */
			port@2 {
				...
				};
			};
		};
};

Compiling the kernel and drivers

More information about compiling the kernel from the sources are available at Custom Kernel Compilation.

Capturing HDMI video with the HDMI2CSI module

EDID / HDMI Source

The hdmi2csi board acts as HDMI sink (with limited feature set). Every HDMI sink defines the formats (resolutions, timings,..) it supports in an EDID file. In our case the EDID (v1.3) is fixed and included in the tc358840 driver. Currently these formats are enabled:

• 16:9
  • 3840x2160p30
  • 1920x1080p24/25/30/50/60
  • 1280x720p30
• 64:27 ("21:9")
  • 1920x1080p29.97/30/60
  • 1280x720p29.97/30

Tested Sources:

• Nvidia Shield (1920x1080p60, 3840x2160p30)
• WD Live TV (1920x1080p60)
• GoPro Hero 4 (1920x1080p60)
• QuantumData 780B (all enabled formats)

Load Driver

Make sure the subdevice driver is loaded:

sudo modprobe tc358840

The GStreamer framework is supported for video capturing.

Capture 1080p60 with any branch

The following command can be used to capture a HDMI input at 1080p60 plugged into the HDMI-A input and display it on a display attached to the HDMI output:

gst-launch-1.0 v4l2src ! 'video/x-raw, width=1920, height=1080, framerate=60/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=60/1, format=I420' ! nvoverlaysink sync=false

Capture 2160p30 with hdmi2csi/l4t-r24-1-dev-4K or hdmi2csi/l4t-r24-2.1

Capturing 2160p30 is currently only supported by the hdmi2csi/l4t-r24-1-dev-4K and hdmi2csi/l4t-r24-2.1](https://github.com/InES-HPMM/linux-l4t/tree/hdmi2csi/l4t-r24-2.1) dev branches. It is also necessary to set the TX1 to maximum performance by running the performance script: jetson_clocks_max.sh from the Nvidia Documentation. There is a script in the Tegra Linux Driver Package R24.1 Release Notes PDF in chapter 2.3 "Maximizing TX1 performance" which is similar.

• jetson_clocks_max.sh
• max_perf.sh

sudo su
./jetson_clocks_max.sh
./max_perf.sh

Use one of the following two methods to capture 2160p30 video on the HDMI-input and display it on the HDMI output.

MMAP mode and hardware-accelerated path

This method is based on the (slow) method of MMAP for passing buffers through the pipeline. The advantage of this method is that we can use the default GStreamer version 1.2.4. But this is not the best performing method in terms of CPU usage.

This uses the MMAP io-mode (per default) and the hardware-accelerated path with the Nvidia video convert/scaler nvvidconv in the video memory memory:NVMM to write directly on top of the video buffer (using nvoverlaysink).

# Set display resolution to 2160p30 (if not automatically done)
export DISPLAY=:0
xrandr --output HDMI-0 --mode 3840x2160 --rate 30.0

# Launch capture pipeline
gst-launch-1.0 v4l2src device=/dev/video0 do-timestamp=true ! 'video/x-raw, width=3840, height=2160, framerate=30/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=3840, height=2160, framerate=30/1, format=I420' ! nvoverlaysink sync=false

## io-mode=2 is the default for v4l2src (MMAP method for buffer passing)

Userptr/Dmabuf mode

Alternatively we can use different modes of passing buffers through the pipeline (e.g. Userptr or Dmabuf mode). This method requires a current version of GStreamer (at least 1.6.0) and some patches (see build GStreamer manually for details). This method can also be used to write to a openGL display sink, and these methods can achieve better CPU performance than MMAP. More information about the io-modes is available here.

Some example pipelines are:

#Dmabuf
gst-launch-1.0 v4l2src io-mode=4 device=/dev/video0 do-timestamp=true ! 'video/x-raw, width=3840, height=2160, framerate=30/1, format=UYVY' ! xvimagesink sync=false
#Userptr
gst-launch-1.0 v4l2src io-mode=3 device=/dev/video0 do-timestamp=true ! 'video/x-raw, width=3840, height=2160, framerate=30/1, format=UYVY' ! xvimagesink sync=false

Future Development

The performance for achieving 2160p30 video capturing varies between the different methods of buffer passing. Also there are limitations for which elements can be used with certain modes (especially with the hardware accelerated video conversion nvvidconv). Furthermore Userptr and Dmabuf modes require specific versions of GStreamer and additional workarounds to function.

The next development step is to improve usability of the different modes and hopefully allow the use of nvvidconv with the better performing modes.

Also the drivers in the dev-branches will be rewritten to improve readability and maintainability.

The current state of support for io-modes is described in Support for io-modes.

If you are interested in collaborating, we are happy to test out changes you made to the code or talk about possible improvements. You can contact us via email to [email protected].

Tests

Coming soon..