Slow memcpy when using TegraJPEG.

[ksze]

I'm trying to find and fix bottlenecks in libfreenect2.

One of the issues I found is that memcpy'ing the decoded image from TegraFrame.data (which is effectively jpeg_decompress_struct dinfo.jpegTegraMgr->buff[0]) is very slow. We're talking about around 55 ms at full optimization (-O3) using clang++-3.8.

In contrast, if I try to allocate plain char arrays of the same size on the heap, fill them with random data, and then do memcpy, I get around 9.5 ms, which is not great, but still much better.

So something is very wonky about using plain memcpy on that chunk of memory allocated by the hardware accelerated library. There must be a way to get fast access to the memory, otherwise TegraJPEG seems pretty pointless (the slow read access would defeat the purpose of the fast decode). I hope somebody familiar with the TegraJPEG/gstreamer stuff can pick up and work on this issue.

[xlz]

Probably has something to do with cache modes? I don't think it is related to code optimization level.

Can you try this thing: compute the byte sum of all bytes in tegraframe.data? This tests the performance of read access.

[ksze]

@xlz You mean just summing all the byte values of tegraframe.data into a size_t variable and timing how long that takes?

[xlz]

Something like that yes. The summing is to prevent optimization from deleting the apparently useless code. And you need to print that sum.

[ksze]

@xlz I just tried that.
Summing tegraframe.data takes about 70 to 90 ms, whereas summing a plain unsigned char array of the same size takes about 9.6 ms (I made sure to populate the unsigned char array with random values).

[xlz]

So it's just different memory having read performance. Sorry this is beyond my understanding. And I don't have a TX1 to test with now.

I would ask the question why this memory is slower than that memory and look into /proc/pid/smaps and the like and find their difference.

[ksze]

@xlz I just looked into /proc/pid/smaps. It looks like the TegraFrame.data is actually mmap'ed from DMA buffer:

a1407000-a1c07000 rw-s 00000000 00:09 4207       anon_inode:dmabuf
Size:               8192 kB
Rss:                8100 kB
Pss:                8100 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:      3828 kB
Private_Dirty:      4272 kB
Referenced:         8100 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr sh mr mw me ms dc de dd

The 8100 kB corresponds neatly with 1920 * 1080 * 4.

This is some quick info I found about DMA buffers: http://elinux.org/images/a/a8/DMA_Buffer_Sharing-_An_Introduction.pdf

After reading that short intro, I think it means that there is an extra address translation step when accessing DMA buffers; there's something about "scatterlist".

[xlz]

Oh it's DMA memory. The rule of thumb of DMA is that you always copy in large chunks because DMA usually has high latency. I guess memcpy does it in small chunks.

Well, I don't know how to properly copy this part of memory to the main memory. Maybe I only happened to find jpegTegraMgr->buff[0] which is a basic but non-optimal way to access decoded pixels.

Here is another attempt at accessing the decoded rgb pixels from GPU #744 (comment)

I looked really hard at the context of jpegTegraMgr->buff[0] when I first reverse engineered it and there was so many layers of abstraction and internal interfaces that I didn't find anything better than just this pointer. Anyway, what we need here is either

• a method to issue a DMA copy of the whole region to the main memory
• or a way to register it as CUDA device memory so no copying happens.

[ksze]

As an alternative, do you know how to use jpeg_read_raw_data() instead of jpeg_read_scanlines()?

I looked at gstjpeg_src/gst-jpeg/gst-jpeg-1.0/ext/jpeg/gstjpegdec.c and I get the feeling that NVIDIA's custom gstreamer can still get hardware acceleration while using jpeg_read_raw_data() instead of jpeg_read_scanlines(), although the output would be downsampled YUV422P, and not RGBX.

I think jpeg_read_raw_data() would write to any regular pointer you give it. If that's the case, we could maybe make it write to a cudaHostAlloc()-ed buffer, and use CUDA to do the yuv422p to RGB conversion.

I dumped a raw JPEG packet from the Kinect sensor and have successfully decoded it using jpeg_read_raw_data(), but using the non-hardware-accelerated libjpeg on the Jetson TK1. I have verified that the decoding is correct, by feeding the decoded raw YUV data to ffmpeg and encoding that as a PNG.

However, if I compile and link the same code with the hardware-accelerated libnvjpeg, I get segfault when I run the compiled executable.

Here is the source code:

/**
 * Compile this with:
 *     g++-4.8 -std=c++11 -o tegrajpeg_raw_accelerated tegrajpeg_raw.cpp -I/home/ubuntu/libfreenect2/build/gstjpeg_src/nv_headers -L/usr/lib/arm-linux-gnueabihf/tegra/ -lnvjpeg -O3
 * or
 *     g++-4.8 -std=c++11 -o tegrajpeg_raw tegrajpeg_raw.cpp -ljpeg -O3
 */

/* Standard library */
#include <chrono>
#include <cstring>
#include <fstream>
#include <iostream>
#include <iterator>
#include <memory>
#include <string>
#include <vector>

/* libjpeg */
#include "jpeglib.h"

using namespace std;
using namespace ::std::chrono;

vector<char> read_jpg_file() {
    string filepath{"/home/ubuntu/kinect_raw.jpg"};
    vector<char> jpg_buffer;

    ifstream jpg_file(filepath, ios::binary | ios::ate);

    if (jpg_file.is_open()) {
        jpg_buffer.reserve(jpg_file.tellg());

        jpg_file.seekg(0, ios::beg);

        jpg_buffer.assign(istreambuf_iterator<char>(jpg_file), istreambuf_iterator<char>());
        jpg_file.close();
    }
    else {
        cerr << "Error reading kinect_raw.jpg!" << endl;
    }

    return jpg_buffer;
}

int main(int argc, char** argv) {
    // read a jpeg file
    vector<char> jpg_buffer = read_jpg_file();  //<char> because ifstream works with char, not unsigned char. :(
    unsigned char* output_buffer = new unsigned char[1920 * 1080 * 2];

    int i, j;

    unsigned char**lines[3];
    unsigned char*y[4 * DCTSIZE] = {NULL, };
    unsigned char*u[4 * DCTSIZE] = {NULL, };
    unsigned char*v[4 * DCTSIZE] = {NULL, };
    int v_samp[3];
    unsigned char *base[3], *last[3];

    // How many bytes per row of Y, U, and V data.
    const int stride[3] = {1920, 960, 960}; 

    const unsigned int height{1080};

    lines[0] = y;
    lines[1] = u;
    lines[2] = v;

    struct jpeg_decompress_struct cinfo;
    struct jpeg_error_mgr jerr;

    cinfo.err = jpeg_std_error(&jerr);
    jpeg_create_decompress(&cinfo);

    jpeg_mem_src(&cinfo, reinterpret_cast<unsigned char*>(jpg_buffer.data()), jpg_buffer.size());

    jpeg_read_header(&cinfo, true);
    
    cinfo.raw_data_out = 1;

    jpeg_start_decompress(&cinfo);

    v_samp[0] = cinfo.comp_info[0].v_samp_factor;
    v_samp[1] = cinfo.comp_info[1].v_samp_factor;
    v_samp[2] = cinfo.comp_info[2].v_samp_factor;

    // Starting positions of the Y, U, and V components in the output_buffer
    base[0] = output_buffer;            // Y
    base[1] = base[0] + 1920 * 1080;    // U
    base[2] = base[1] + 960 * 1080;     // V
    
    for (i = 0; i < height; i += v_samp[0] * DCTSIZE) { 
        for (j = 0; j < (v_samp[0] * DCTSIZE); j++) {
            lines[0][j] = base[0] + (i + j) * stride[0];
            lines[1][j] = base[1] + (i + j) * stride[1];
            lines[2][j] = base[2] + (i + j) * stride[2];
        }

        jpeg_read_raw_data(&cinfo, lines, v_samp[0] * DCTSIZE);
    }

    jpeg_finish_decompress(&cinfo);

    ofstream out_yuv{"/home/ubuntu/out.yuv", ::std::ios::out | ::std::ios::binary};
    out_yuv.write(reinterpret_cast<const char*>(output_buffer), 1920 * 1080 * 2);
    out_yuv.close();

    jpeg_destroy_decompress(&cinfo);

    delete[] output_buffer;
}

[xlz]

gdb?

…

On May 22, 2017 4:18 AM, "Kal Sze" ***@***.***> wrote: As an alternative, do you know how to use jpeg_read_raw_data() instead of jpeg_read_scanlines()? I looked at gstjpeg_src/gst-jpeg/gst-jpeg-1.0/ext/jpeg/gstjpegdec.c and I get the feeling that NVIDIA's custom gstreamer can still get hardware acceleration while using jpeg_read_raw_data() instead of jpeg_read_scanlines(), although the output would be downsampled YUV422P, and not RGBX. I *think* jpeg_read_raw_data() would write to any regular pointer you give it. If that's the case, we could maybe make it write to a cudaHostAlloc()-ed buffer, and use CUDA to do the yuv422p to RGB conversion. I dumped a raw JPEG packet from the Kinect sensor and have successfully decoded it using jpeg_read_raw_data(), but using the non-hardware-accelerated libjpeg on the Jetson TK1. I have verified that the decoding is correct, by feeding the decoded raw YUV data to ffmpeg and encoding that as a PNG. However, if I compile and link the same code with the hardware-accelerated libnvjpeg, I get segfault when I run it compiled executable. Here is the source code: /** * Compile this with: * g++-4.8 -std=c++11 -o tegrajpeg_raw_accelerated tegrajpeg_raw.cpp -I/home/ubuntu/libfreenect2/build/gstjpeg_src/nv_headers -L/usr/lib/arm-linux-gnueabihf/tegra/ -lnvjpeg -O3 * or * g++-4.8 -std=c++11 -o tegrajpeg_raw tegrajpeg_raw.cpp -ljpeg -O3 */ /* Standard library */ #include <chrono> #include <cstring> #include <fstream> #include <iostream> #include <iterator> #include <memory> #include <string> #include <vector> /* libjpeg */ #include "jpeglib.h" using namespace std; using namespace ::std::chrono; vector<char> read_jpg_file() { string filepath{"/home/ubuntu/kinect_raw.jpg"}; vector<char> jpg_buffer; ifstream jpg_file(filepath, ios::binary | ios::ate); if (jpg_file.is_open()) { jpg_buffer.reserve(jpg_file.tellg()); jpg_file.seekg(0, ios::beg); jpg_buffer.assign(istreambuf_iterator<char>(jpg_file), istreambuf_iterator<char>()); jpg_file.close(); } else { cerr << "Error reading kinect_raw.jpg!" << endl; } return jpg_buffer; } int main(int argc, char** argv) { // read a jpeg file vector<char> jpg_buffer = read_jpg_file(); //<char> because ifstream works with char, not unsigned char. :( unsigned char* output_buffer = new unsigned char[1920 * 1080 * 2]; int i, j; unsigned char**lines[3]; unsigned char*y[4 * DCTSIZE] = {NULL, }; unsigned char*u[4 * DCTSIZE] = {NULL, }; unsigned char*v[4 * DCTSIZE] = {NULL, }; int v_samp[3]; unsigned char *base[3], *last[3]; // How many bytes per row of Y, U, and V data. const int stride[3] = {1920, 960, 960}; const unsigned int height{1080}; lines[0] = y; lines[1] = u; lines[2] = v; struct jpeg_decompress_struct cinfo; struct jpeg_error_mgr jerr; cinfo.err = jpeg_std_error(&jerr); jpeg_create_decompress(&cinfo); jpeg_mem_src(&cinfo, reinterpret_cast<unsigned char*>(jpg_buffer.data()), jpg_buffer.size()); jpeg_read_header(&cinfo, true); cinfo.raw_data_out = 1; jpeg_start_decompress(&cinfo); v_samp[0] = cinfo.comp_info[0].v_samp_factor; v_samp[1] = cinfo.comp_info[1].v_samp_factor; v_samp[2] = cinfo.comp_info[2].v_samp_factor; // Starting positions of the Y, U, and V components in the output_buffer base[0] = output_buffer; // Y base[1] = base[0] + 1920 * 1080; // U base[2] = base[1] + 960 * 1080; // V for (i = 0; i < height; i += v_samp[0] * DCTSIZE) { for (j = 0; j < (v_samp[0] * DCTSIZE); j++) { lines[0][j] = base[0] + (i + j) * stride[0]; lines[1][j] = base[1] + (i + j) * stride[1]; lines[2][j] = base[2] + (i + j) * stride[2]; } jpeg_read_raw_data(&cinfo, lines, v_samp[0] * DCTSIZE); } jpeg_finish_decompress(&cinfo); ofstream out_yuv{"/home/ubuntu/out.yuv", ::std::ios::out | ::std::ios::binary}; out_yuv.write(reinterpret_cast<const char*>(output_buffer), 1920 * 1080 * 2); out_yuv.close(); jpeg_destroy_decompress(&cinfo); delete[] output_buffer; } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#852 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUkRvyKJw3XLg9tYiUBi_G-zOUJoDUPks5r8W8egaJpZM4Nddbf> .