MS Kinect V2 on NVIDIA Jetson TX1

With a USB firmware patch and an updated libfreenect2, the Microsoft Kinect V2 now runs on the Jetson TX1 Development Kit. Looky here:

Background

For a stretch there it was not possible to run the open source Kinect V2 driver libfreenect2 on the Jetson TX1 because of an issue with the USB firmware. Fortunately NVIDIA has issued a firmware patch
(see the Jetson TX1 Forum, USB 3 Transfer Failures ) which fixes the issue. As you might recall, Microsoft now offers the Kinect V2 as a two part kit a Xbox One Kinect Sensor Bar along with a Kinect Adapter for Windows. You will need both a Kinect Xbox One sensor and the adapter for use with the Jetson, or the discontinued Kinect for Windows. The Kinect Adapter for Windows converts the output from the Kinect to USB 3.0. The advantage of this setup is that you can use the Kinect sensor from your Xbox One, or at least have an excuse to get a Xbox One + Kinect for “research” purposes.

Installation

The installLibfreenect2 repository on the JetsonHacks Github account contains convenience scripts for installing libfreenect2 and the USB firmware patch. First, get the repository:

$ git clone https://github.com/jetsonhacks/installLibfreenect2

Second, install libfreenect2 and compile the library and examples:

$ cd installLibfreenect2
$ ./installLibfreenect2

Third, you will need to patch the USB firmware:

$ ./firmwarePatch.sh

After installing the USB firmware patch, it is necessary to reboot the machine in order for the firmware changes to take effect.

When the machine reboots, you can run the example:

$ cd ~/libfreenect2/build/bin
$ ./Protonect

Some Notes

The installation of libfreenect2 in the video is on L4T 24.1, flashed by JetPack 2.2. CUDA is required. Both 32 bit and 64 bit versions of L4T are shown in the video, installation of libfreenect2 and the firmware patch is the same in both cases.

The Mesa libs installed by libfreenect2 overwrite libGL.so, which causes issues. The fix as of this writing (July, 2016) is to link libGL.so to the Tegra version.

The repository contains a script jetson_max_l4t.sh which sets the CPU and GPU to maximum clock values. This will increase the Jetson TX1 performance at the cost of power consumption.

L4T 23.X Notes

The JPEG decompressor under L4T 23.X produces RGBA format, where as the Protonect viewer consumes BGRA format. The repository contains a patch which adds a simplistic algorithm to rearrange the bytes appropriately. If you intend to use this script with L4T 23.X, you will need to uncomment the line:

# patch -p 1 -i $PATCHDIR/bgra.patch

Also, if you plan to use this library in production L4T 23.X code, your application should consider writing specialized code to do the RGBA→BGRA conversion more efficiently.

For L4T 24.1, there is no patch applied as the JPEG decompressor produces BGRA format natively.

The post MS Kinect V2 on NVIDIA Jetson TX1 appeared first on JetsonHacks.