Multiple kinect

[chengzijiazu]

Overview Description:
multiple kinect: I am working with the multiple kinect v2 on windows base on libfreenect2 0.2.
When I debuy the programmer,the kinects can start and programmer shut down after a while.I don't while.Could some one give me a hand? When connect with two kienct ,it can work well.

Computer DELL Precision M6800 notebook
Cpu Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz
RAM 32.0GB
Graphic card 1、NVIDIA Quadro K3100M
2、Intel(R) HD Graphics 4600
Disk 1、PLEXTOR PX-128M6S+ (128GB)
2、HGST HTS545050A7E380 (500GB)
Mainboard 0XD1M5 (A00)
Network card 1、Intel(R) Ethernet Connection I217-LM
2、Dell Wireless 1550 802.11ac

display LGD:da02 :1920x1080
system Windows 8.1 64

the code:
/*

• This file is part of the OpenKinect Project. http://www.openkinect.org
  *
• Copyright (c) 2011 individual OpenKinect contributors. See the CONTRIB file
• for details.
  *
• This code is licensed to you under the terms of the Apache License, version
• 2.0, or, at your option, the terms of the GNU General Public License,
• version 2.0. See the APACHE20 and GPL2 files for the text of the licenses,
• or the following URLs:
• http://www.apache.org/licenses/LICENSE-2.0
• http://www.gnu.org/licenses/gpl-2.0.txt
  *
• If you redistribute this file in source form, modified or unmodified, you
• may:
• 1. Leave this header intact and distribute it under the same terms,

•  accompanying it with the APACHE20 and GPL20 files, or
  

• 2. Delete the Apache 2.0 clause and accompany it with the GPL2 file, or
• 3. Delete the GPL v2 clause and accompany it with the APACHE20 file
• In all cases you must keep the copyright notice intact and include a copy
• of the CONTRIB file.
  *
• Binary distributions must follow the binary distribution requirements of
• either License.
  */

/** @file Protonect.cpp Main application file. */

include

include

include <signal.h>

/// [headers]

include <libfreenect2/libfreenect2.hpp>

include <libfreenect2/frame_listener_impl.h>

include <libfreenect2/registration.h>

include <libfreenect2/packet_pipeline.h>

include <libfreenect2/logger.h>

/// [headers]

ifdef EXAMPLES_WITH_OPENGL_SUPPORT

include "viewer.h"

endif

bool protonect_shutdown = false; ///< Whether the running application should shut down.

void sigint_handler(int s)
{
protonect_shutdown = true;
}

bool protonect_paused = false; ///< Whether the running application should pause.
libfreenect2::Freenect2Device *devtopause;
libfreenect2::Freenect2Device *devtopause1;
libfreenect2::Freenect2Device *devtopause2;

//Doing non-trivial things in signal handler is bad. If you want to pause,
//do it in another thread.
//Though libusb operations are generally thread safe, I cannot guarantee
//everything above is thread safe when calling start()/stop() while
//waitForNewFrame().
void sigusr1_handler(int s)
{
if (devtopause == 0)
return;
/// [pause]
if (protonect_paused)
devtopause->start();
else
devtopause->stop();

if (devtopause1 == 0)
    return;
/// [pause]
if (protonect_paused)
    devtopause1->start();
else
    devtopause1->stop();

if (devtopause2 == 0)
    return;
/// [pause]
if (protonect_paused)
    devtopause2->start();
else
    devtopause2->stop();

protonect_paused = !protonect_paused;
/// [pause]

}

//The following demostrates how to create a custom logger
/// [logger]

include

include

class MyFileLogger : public libfreenect2::Logger
{
private:
std::ofstream logfile_;
public:
MyFileLogger(const char *filename)
{
if (filename)
logfile_.open(filename);
level_ = Debug;
}
bool good()
{
return logfile_.is_open() && logfile_.good();
}
virtual void log(Level level, const std::string &message)
{
logfile_ << "[" << libfreenect2::Logger::level2str(level) << "] " << message << std::endl;
}
};
/// [logger]

/// [main]
/**

• Main application entry point.
  *

• Accepted argumemnts:

• - cpu Perform depth processing with the CPU.

• - gl Perform depth processing with OpenGL.

• - cl Perform depth processing with OpenCL.

• - Serial number of the device to open.

• • -noviewer Disable viewer window.
    */
    int main(int argc, char *argv[])
    /// [main]
    {
    std::string program_path(argv[0]);
    std::cerr << "Version: " << LIBFREENECT2_VERSION << std::endl;
    std::cerr << "Environment variables: LOGFILE=<protonect.log>" << std::endl;
    std::cerr << "Usage: " << program_path << " [-gpu=] [gl | cl | cuda | cpu] []" << std::endl;
    std::cerr << " [-noviewer] [-norgb | -nodepth] [-help] [-version]" << std::endl;
    std::cerr << " [-frames ]" << std::endl;
    std::cerr << "To pause and unpause: pkill -USR1 Protonect" << std::endl;
    size_t executable_name_idx = program_path.rfind("Protonect");

  std::string binpath = "/";

  if (executable_name_idx != std::string::npos)
  {
  binpath = program_path.substr(0, executable_name_idx);
  }

if defined(WIN32) || defined(_WIN32) || defined(WIN32)

// avoid flooing the very slow Windows console with debug messages
libfreenect2::setGlobalLogger(libfreenect2::createConsoleLogger(libfreenect2::Logger::Info));

else

// create a console logger with debug level (default is console logger with info level)
/// [logging]
libfreenect2::setGlobalLogger(libfreenect2::createConsoleLogger(libfreenect2::Logger::Debug));
/// [logging]

endif

/// [file logging]
MyFileLogger *filelogger = new MyFileLogger(getenv("LOGFILE"));
if (filelogger->good())
    libfreenect2::setGlobalLogger(filelogger);
else
    delete filelogger;
/// [file logging]

/// [context]
libfreenect2::Freenect2 freenect2;

libfreenect2::Freenect2Device *dev = 0;
libfreenect2::Freenect2Device *dev1 = 0;
libfreenect2::Freenect2Device *dev2 = 0;

libfreenect2::PacketPipeline *pipeline = 0;
libfreenect2::PacketPipeline *pipeline1 = 0;
libfreenect2::PacketPipeline *pipeline2 = 0;

/// [context]

std::string serial = "";
std::string serial1 = "";
std::string serial2 = "";

bool viewer_enabled = true;
bool enable_rgb = true;
bool enable_depth = true;
int deviceId = -1;
size_t framemax = -1;

for (int argI = 1; argI < argc; ++argI)
{
    const std::string arg(argv[argI]);

    if (arg == "-help" || arg == "--help" || arg == "-h" || arg == "-v" || arg == "--version" || arg == "-version")
    {
        // Just let the initial lines display at the beginning of main
        return 0;
    }
    else if (arg.find("-gpu=") == 0)
    {
        if (pipeline)
        {
            std::cerr << "-gpu must be specified before pipeline argument" << std::endl;
            return -1;
        }
        deviceId = atoi(argv[argI] + 5);
    }
    else if (arg == "cpu")
    {
        if (!pipeline)
            /// [pipeline]
            pipeline = new libfreenect2::CpuPacketPipeline();
        if (!pipeline1)
            /// [pipeline]
            pipeline1 = new libfreenect2::CpuPacketPipeline();      
        if (!pipeline2)
            /// [pipeline]
            pipeline2 = new libfreenect2::CpuPacketPipeline();      
    }
    else if (arg == "gl")
    {

ifdef LIBFREENECT2_WITH_OPENGL_SUPPORT

        if (!pipeline)
            pipeline = new libfreenect2::OpenGLPacketPipeline();
        if (!pipeline1)
            pipeline1 = new libfreenect2::OpenGLPacketPipeline();
        if (!pipeline2)
            pipeline2 = new libfreenect2::OpenGLPacketPipeline();

else

        std::cout << "OpenGL pipeline is not supported!" << std::endl;

endif

    }
    else if (arg == "cl")
    {

ifdef LIBFREENECT2_WITH_OPENCL_SUPPORT

        if (!pipeline)
            pipeline = new libfreenect2::OpenCLPacketPipeline(deviceId);
        if (!pipeline1)
            pipeline1 = new libfreenect2::OpenCLPacketPipeline(deviceId);
        if (!pipeline2)
            pipeline2 = new libfreenect2::OpenCLPacketPipeline(deviceId);

else

        std::cout << "OpenCL pipeline is not supported!" << std::endl;

endif

    }
    else if (arg == "cuda")
    {

ifdef LIBFREENECT2_WITH_CUDA_SUPPORT

        if (!pipeline)
            pipeline = new libfreenect2::CudaPacketPipeline(deviceId);

else

        std::cout << "CUDA pipeline is not supported!" << std::endl;

endif

    }
    else if (arg.find_first_not_of("0123456789") == std::string::npos) //check if parameter could be a serial number
    {
        serial = arg;
    }
    else if (arg == "-noviewer" || arg == "--noviewer")
    {
        viewer_enabled = false;
    }
    else if (arg == "-norgb" || arg == "--norgb")
    {
        enable_rgb = false;
    }
    else if (arg == "-nodepth" || arg == "--nodepth")
    {
        enable_depth = false;
    }
    else if (arg == "-frames")
    {
        ++argI;
        framemax = strtol(argv[argI], NULL, 0);
        if (framemax == 0) {
            std::cerr << "invalid frame count '" << argv[argI] << "'" << std::endl;
            return -1;
        }
    }
    else
    {
        std::cout << "Unknown argument: " << arg << std::endl;
    }
}

if (!enable_rgb && !enable_depth)
{
    std::cerr << "Disabling both streams is not allowed!" << std::endl;
    return -1;
}

/// [discovery]
if (freenect2.enumerateDevices() == 0)
{
    std::cout << "no device connected!" << std::endl;
    return -1;
}
else
{
    std::cout << "Found device connected!" << freenect2.enumerateDevices() << std::endl;
}

if (serial == "")
{
    serial = freenect2.getDeviceSerialNumber(0);
    std::cout << "Device 0 is :" << serial << std::endl;
    //serial = freenect2.getDefaultDeviceSerialNumber();
}
if (serial1 == "")
{
    serial1 = freenect2.getDeviceSerialNumber(1);
    std::cout << "Device 1 is :" << serial1 << std::endl;
    //serial = freenect2.getDefaultDeviceSerialNumber();
}
if (serial2 == "")
{
    serial2 = freenect2.getDeviceSerialNumber(2);
    std::cout << "Device 2 is :" << serial2 << std::endl;
    //serial = freenect2.getDefaultDeviceSerialNumber();
}
/// [discovery]

if (pipeline)
{
    /// [open]
    std::cout << "Opening device 0 with pipeline!" << std::endl;
    dev = freenect2.openDevice(serial, pipeline);
    /// [open]
}
else
{
    std::cout << "Opening device 0!" << std::endl;
    dev = freenect2.openDevice(serial);
}
if (pipeline1)
{
    /// [open]
    std::cout << "Opening device 1 with pipeline!" << std::endl;
    dev1 = freenect2.openDevice(serial1, pipeline1);
    /// [open]
}
else
{
    std::cout << "Opening device 1!" << std::endl;
    dev1 = freenect2.openDevice(serial1);
}
if (pipeline2)
{
    /// [open]
    std::cout << "Opening device 2 with pipeline!" << std::endl;
    dev2 = freenect2.openDevice(serial2, pipeline2);
    /// [open]
}
else
{
    std::cout << "Opening device 2!" << std::endl;
    dev2 = freenect2.openDevice(serial2);
}

if (dev == 0)
{
    std::cout << "failure opening device 0!" << std::endl;
    return -1;
}

if (dev1 == 0)
{
    std::cout << "failure opening device 1!" << std::endl;
    return -1;
}

if (dev2 == 0)
{
    std::cout << "failure opening device 2!" << std::endl;
    return -1;
}

devtopause = dev;
devtopause1 = dev1;
devtopause2 = dev2;

signal(SIGINT, sigint_handler);

ifdef SIGUSR1

signal(SIGUSR1, sigusr1_handler);

endif

protonect_shutdown = false;

/// [listeners]
int types = 0;
if (enable_rgb)
    types |= libfreenect2::Frame::Color;
if (enable_depth)
    types |= libfreenect2::Frame::Ir | libfreenect2::Frame::Depth;
libfreenect2::SyncMultiFrameListener listener(types);
libfreenect2::SyncMultiFrameListener listener1(types);
libfreenect2::SyncMultiFrameListener listener2(types);
libfreenect2::FrameMap frames;
libfreenect2::FrameMap frames1;
libfreenect2::FrameMap frames2;

dev->setColorFrameListener(&listener);
dev->setIrAndDepthFrameListener(&listener);
dev1->setColorFrameListener(&listener1);
dev1->setIrAndDepthFrameListener(&listener1);
dev2->setColorFrameListener(&listener2);
dev2->setIrAndDepthFrameListener(&listener2);
/// [listeners]

/// [start]
if (enable_rgb && enable_depth)
{
    if (!dev->start())
        return -1;
    if (!dev1->start())
        return -1;
    if (!dev2->start())
        return -1;
}
else
{
    if (!dev->startStreams(enable_rgb, enable_depth))
        return -1;
    if (!dev1->startStreams(enable_rgb, enable_depth))
        return -1;
    if (!dev2->startStreams(enable_rgb, enable_depth))
        return -1;
}

//std::cout << "device serial: " << dev->getSerialNumber() << std::endl;
//std::cout << "device firmware: " << dev->getFirmwareVersion() << std::endl;
/// [start]

/// [registration setup]
libfreenect2::Registration* registration = new libfreenect2::Registration(dev->getIrCameraParams(), dev->getColorCameraParams());
libfreenect2::Registration* registration1 = new libfreenect2::Registration(dev1->getIrCameraParams(), dev1->getColorCameraParams());
libfreenect2::Registration* registration2 = new libfreenect2::Registration(dev2->getIrCameraParams(), dev2->getColorCameraParams());
libfreenect2::Frame undistorted(512, 424, 4), registered(512, 424, 4);
libfreenect2::Frame undistorted1(512, 424, 4), registered1(512, 424, 4);
libfreenect2::Frame undistorted2(512, 424, 4), registered2(512, 424, 4);
/// [registration setup]

size_t framecount = 0;
size_t framecount1 = 0;
size_t framecount2 = 0;

ifdef EXAMPLES_WITH_OPENGL_SUPPORT

Viewer viewer;
if (viewer_enabled)
    viewer.initialize();
Viewer viewer1;
if (viewer_enabled)
    viewer1.initialize();
Viewer viewer2;
if (viewer_enabled)
    viewer2.initialize();

else

viewer_enabled = false;

endif

/// [loop start]
while (!protonect_shutdown && (framemax == (size_t)-1 || framecount < framemax))
{
    listener.waitForNewFrame(frames);
    libfreenect2::Frame *rgb = frames[libfreenect2::Frame::Color];
    libfreenect2::Frame *ir = frames[libfreenect2::Frame::Ir];
    libfreenect2::Frame *depth = frames[libfreenect2::Frame::Depth];
    listener.waitForNewFrame(frames1);
    libfreenect2::Frame *rgb1 = frames1[libfreenect2::Frame::Color];
    libfreenect2::Frame *ir1 = frames1[libfreenect2::Frame::Ir];
    libfreenect2::Frame *depth1 = frames1[libfreenect2::Frame::Depth];
    listener.waitForNewFrame(frames2);
    libfreenect2::Frame *rgb2 = frames2[libfreenect2::Frame::Color];
    libfreenect2::Frame *ir2 = frames2[libfreenect2::Frame::Ir];
    libfreenect2::Frame *depth2 = frames2[libfreenect2::Frame::Depth];
    /// [loop start]

    if (enable_rgb && enable_depth)
    {
        /// [registration]
        registration->apply(rgb, depth, &undistorted, &registered);
        registration1->apply(rgb1, depth1, &undistorted1, &registered1);
        registration2->apply(rgb2, depth2, &undistorted2, &registered2);
        /// [registration]
    }

    framecount++;
    framecount1++;
    framecount2++;
    if (!viewer_enabled)
    {
        if (framecount % 100 == 0)
            std::cout << "The viewer is turned off. Received " << framecount << " frames. Ctrl-C to stop." << std::endl;
        listener.release(frames);
        if (framecount1 % 100 == 0)
            std::cout << "The viewer is turned off. Received " << framecount1 << " frames. Ctrl-C to stop." << std::endl;
        listener.release(frames1);
        if (framecount2 % 100 == 0)
            std::cout << "The viewer is turned off. Received " << framecount2 << " frames. Ctrl-C to stop." << std::endl;
        listener.release(frames2);
        continue;
    }

ifdef EXAMPLES_WITH_OPENGL_SUPPORT

    if (enable_rgb)
    {
        viewer.addFrame("RGB", rgb);
        viewer1.addFrame("RGB", rgb1);
        viewer2.addFrame("RGB", rgb2);
    }
    if (enable_depth)
    {
        viewer.addFrame("ir", ir);
        viewer.addFrame("depth", depth);
        viewer1.addFrame("ir", ir1);
        viewer1.addFrame("depth", depth1);
        viewer2.addFrame("ir", ir2);
        viewer2.addFrame("depth", depth2);
    }
    if (enable_rgb && enable_depth)
    {
        viewer.addFrame("registered", &registered);
        viewer1.addFrame("registered", &registered1);
        viewer2.addFrame("registered", &registered2);
    }

    protonect_shutdown = protonect_shutdown || viewer.render();

endif

    /// [loop end]
    listener.release(frames);
    listener1.release(frames1);
    listener2.release(frames2);
    /** libfreenect2::this_thread::sleep_for(libfreenect2::chrono::milliseconds(100)); */
}
/// [loop end]

// TODO: restarting ir stream doesn't work!
// TODO: bad things will happen, if frame listeners are freed before dev->stop() :(
/// [stop]
dev->stop();
dev->close();
dev1->stop();
dev1->close();
dev2->stop();
dev2->close();
/// [stop]

delete registration;
delete registration1;
delete registration2;

return 0;

}

Actual Results:

Expected Results:

Reproducibility:

Additional Information:

[djivani]

I believe you cannot run more than 2 Kinects on a single USB 3.0 controller card, which is probably what exists in your DELL Precision M6800 notebook. The solution is to use a desktop with spare PCIe slots where you can plug-in extra USB 3.0 controllers, ideally 1 for each Kinect.

[chengzijiazu]

Thank you! I will try to modified the code !

[kalluwa]

Hi ,
i have the same problem with U, libfreenect2 works fine with two kinects but failed with 3 kinects.
i am working at DELL precision 7510 and not sure for problems of PCIe.
have u made it workable or any suggestions?

[chengzijiazu]

Not yet,but I am still working to solve this problem.

[kalluwa]

after the reading the issue :
320

Yes, I can confirm that each Kinect needs its own USB-controller. Current main boards typically have one or two USB 3.0 controllers, so one needs to plug in USB 3.0 expansion card(s). In my current setup I use 4 x Transcend TS-PDU3 USB3.0 Expansion Cards and one on board controller. Running Ubuntu 14.04 LTS.

i believe that THE HARDWARE can just support 2 kinects just as djivani said, and i have to choose anothing way(change the PC) to use multiple kinects, 😞

waiting for your good news.

[djivani]

I am working on a setup of 6 Kinects using 5 Transcend USB 3.0 Dual Expansion Card PCI Express Interface Model TS-PDU3 cards and the onboard USB 3.0 controller.

[chengzijiazu]

Could you give me your code about the multiple kinects.（I think there must be something wrong with my code）I am starting again and working for skeleton tracking project of my postgraduate tast. @djivani

[sunwtGitHub]

Hi, I am working with three number of kinect v2 on windows 10.
I installed two Transcend USB 3.0 Expansion cards (one on a PCI-E 1x and one on a PCI-E 16x).
I am sure every devices has thier own usb3 controller.

In my program, I used three number of threads to process the data from each device.
When I open the devices and begin to receive the data. At first, all of the threads were working well.
After two minutes(or shorter), one or two threads were blocked(the last one still worked well), and the function waitForNewFrame was never return.

I'm trying to find a way to solve this problem.
So I refer to the freenect2 project code.
The problem is that the code "processor_->process(packet)" in function "void DepthPacketStreamParser::onDataReceived(...)" will never be executed.
Because variables "footer->sequence" and "footer->subsequence" are all values greater than 50 thousand...

Is there anyone who can help me?