Merging Multiple view ARCore point clouds using COLMAP camera poses

[himmetozcan]

Hi there. In one of my project, I am trying to merge ARCore point cloud output, and COLMAP outputs. Particularly, I am trying to use the camera poses from COLMAP, to merge the multiple view ARCore point clouds.

I see that you are converting the COLMAP point cloud to ARCore world, maybe vice versa. Can you point me to the right direction.? For example, can u at least provide a few data from ARCore so that I can understand how the code is working.

[alexs7]

@himmetozcan , thanks for showing interest in my project. Are you trying to align the point clouds from ARCore and COLMAP ?

[himmetozcan]

Thanks for replying. The end goal is merging multiple point clouds of an object using ArCore. After some research, I think I do not know enough information about ArCore's capabilities. Currently, I save the depth image, RGB image, and camera parameters and generate a point cloud from them. The depth image and RGB image are already pixel aligned. I take pictures of the same item from different views, let's say it is an object such as a chair, and try to merge the individual point clouds so that I can have a full 3D view of the object. First I thought using COLMAP would give me the camera poses and locations and so I can use this information to merge the point clouds from ArCore. Now I see that maybe I can already get multiple point cloud frames from ArCore, that are merged already? Like using this function from ArCore. What do u think is the right way to do it?

[alexs7]

ARCore exposes the pointcloud (https://developers.google.com/ar/reference/java/com/google/ar/core/PointCloud), and you can save that.
Check my code here: https://github.com/alexs7/arcore-android-sdk/blob/dc6df300ea17f6862d2a984f263f1cd93b2d89fa/samples/hello_ar_java/app/src/main/java/com/google/ar/core/examples/java/helloar/ClientWrapper.java#L46

Have you looked into ICP methods ? https://en.wikipedia.org/wiki/Iterative_closest_point

If your images are overlapping then COLMAP might be the way to go if you are looking to offload the point cloud creation.
I am curious too see what you can do with ARCore only

[himmetozcan]

Very nice code, I see that you are sending the data to a local server. Any chance if you can share the code for listening to the send data (maybe a small flask project?).

I have tried ICP on ArCore point cloud, without using any information about camera poses. It is not working that good, and it is very slow (taking 10-15 seconds). I am using the ICP algorithm in open3d.

[alexs7]

Hello, the server is a Node.js server that run also a visual AR debugger. The code is here:
https://github.com/alexs7/Mobile-Pose-Estimation-Pipeline-Prototype/tree/demo_version/threejs_viewer
If you cd in that dir and run npm start it should start a GUI with a server listening to localhost:3000/

Let me know if you need any help

[himmetozcan]

Hi again. I have managed to run the android app, but couldn't run the threejs_server. npm start isn't working since there is no package.json file. I ve created one with npm init but since there is no start script I ve just run node main.js and getting this

error:
app.on('ready', createWindow);
   ^

TypeError: Cannot read property 'on' of undefined
   at Object.<anonymous> (J:\ArCore\Mobile-Pose-Estimation-Pipeline-Prototype-demo_version\threejs_viewer\main.js:36:5)

I am not familiar with node/npm so I am stuck here. Any help is appreciated.

[alexs7]

Apologies for the late reply, here is the package.json file, try this and let me know if it works.
"Start" will start electron.

{
  "name": "arcore_debugger",
  "version": "1.0.0",
  "description": "",
  "main": "main.js",
  "scripts": {
    "start": "electron ."
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "electron": "^8.0.1",
    "electron-packager": "^14.2.1"
  },
  "dependencies": {
    "body-parser": "^1.19.0",
    "bootstrap": "^4.4.1",
    "express": "^4.17.1",
    "jquery": "^3.4.1",
    "pngjs": "^3.3.2",
    "popper": "^1.0.1",
    "quaternion": "^1.1.0",
    "sleep": "^5.1.1",
    "three": "^0.91.0"
  }
}