Task: Processing 3D data

There are sensors, that produce 3D point data directly. An example data from Intel RealSense camera are available here. Every data file contain a single scene, there are several point-clouds in a file from different view-points. The 3D points of each view-point are in coordinate system centered/oriented at the actual sensor position. Additionally, each view-point contains (imprecise) rotation and translation with respect to global coordinate system.

The task is to load the data from chosen scene, align particular point clouds together and store/show them as PLY 3D model file.

  1. use the stored rotation and translation for alignment. Note, that they transforms from world frame to $i$-th view sensor frame: $X_i = R X_w + t$

  2. implement and use ICP (Iterative Closest Point) method for refining the rotations and translations, and produce improved PLY file.

Note: there is a simple package ''ge'' for writing a PLY file. The RealSense data are stored in .pckl file, an example of reading 3D data and storing to a PLY file follows. The recommended viewer for the 3D PLY files is MeshLab.

In [3]:
import numpy as np
import pickle
import ge
import matplotlib.cm

file = "data/realsense/2019-12-09_11-06-02_data.pckl"

# load .pckl data from the RealSense sensor
with open( file, "rb") as fh:
    data = pickle.load( fh )

# a single view is stored in data[i]
Nview = len( data ) # number of views

# colormap for storing each view with different color
cmap = matplotlib.cm.get_cmap( 'jet')

# open PLY output object
g = ge.GePly( file[:-5] + '.ply' ) # change extension .pckl to .ply

# iterate views
for i in range( 0,  Nview ):

    # 3D points from a single view - 3 x n numpy array
    X = data[i]["points"].T
    
    # rotation matrix, 3x3 numpy array
    R = data[i]['extrinsics'][:3, :3]
    
    # translation vector, 3x1 numpy array
    t = data[i]['extrinsics'][:3, [3]]

    # transform the points X here using R and t
    # R = .....
    # t = .....

    # color for this view
    C = cmap( i / ( Nview - 1 ) )[:3] # skip alpha

    # ply output
    g.points( X, color=C )

g.close()