Mastering 3D Computer Vision & Point Cloud Processing-Mod 9-Point Cloud to Mesh and Point Cloud to Voxel Grid Conversion with Code

Introduction:

Welcome to the ✍️“3D Computer Vision & Point Cloud Processing Blog Series”. This series of blogs is your 🚀 hands-on guide to mastering 3D point cloud processing with Python.

In this post, we’ll delve into how the point clouds data can be converted to Mesh and voxel grid.

Topics to Be Discussed in this Blog

1. Introduction to 3D Data Types
2. Why convert between different 3D Data Types
3. Point Cloud to Mesh Conversion
4. Point Cloud to Voxel Grid Conversion
5. Summary

Introduction to 3D Data Types

There are several 3D data types and most common ones are point clouds, meshes, and voxel grid;

1. Point Cloud: A collection of points in 3D space, often representing surfaces or features of objects. Each point typically has coordinates (x, y, z) and may include additional information such as color or intensity.
2. Mesh: A collection of vertices, edges, and faces that define the shape of a 3D object or surface. Meshes are used to represent solid objects with surfaces.
3. Voxel Grid: A three-dimensional grid where each cell (voxel) represents a value or attribute, such as density or material composition. Voxel grids are often used to represent volumetric data.

Other data types include CAD Models, Animation Data. More details about “Differences among Different 3D Data Types”, “Selection of Data Types Based on Applications”, “Advantages and Disadvantages of Different 3D Data Types”, please read my previous blog.

Why convert between different 3D Data Types

The need for conversion between 3D data formats is primarily due to two reasons:

1. Different data types have different strengths and weaknesses in terms of simplicity, storage, compression, transmission, processing, and so on. For example, in terms of storage efficiency, point cloud is the best choice among others.
2. Different applications demand different data types. For example, Mesh is the best choice for 3D printing applications.

Point Cloud to Mesh Conversion

1. Converting point clouds to meshes is crucial for several reasons,

1. Surface Reconstruction: Point clouds are collections of discrete points in 3D space, which can be irregularly sampled. Converting them to a mesh creates a continuous surface representation, making it easier to visualize and analyze.
2. Visualization: Meshes offer smoother and visually appealing representations, ideal for 3D graphics and simulation applications.
3. Computational Geometry: Meshes simplify geometric algorithms like collision detection and ray tracing, making them more efficient than point clouds.

2. Applications of Meshes

Meshes are widely used in various fields,

1. Computer Graphics: For rendering 3D scenes, character modeling, and animation.
2. Medical Imaging: To represent anatomical structures.
3. Robotics: For environment modeling and path planning.

3. Advantages of Converting to Mesh

1. Smooth Surfaces: Meshes offer smoother surface representations than point clouds.
2. Efficiency: Meshes are memory & computationally efficient for many algorithms and simulations.

4. Disadvantages of Converting to Mesh

1. Increased Complexity: Meshes have more complex data structures, making some operations more challenging.
2. Mesh Artifacts: Meshes can have artifacts like gaps or non-manifold geometry, depending on the reconstruction method.
3. Information Loss: Converting to meshes can lose fine-grained details from the original point cloud.

5. Let’s code

Follow the below steps to run the script.

1. Create a file named “pcd-to-mesh.py” and paste the following code into it.
2. Download “bunny-pcd.ply” ply data from https://github.com/MatPixel/dataset-for-3d-pointcloud-processing-3d-deep-learning/blob/main/bunny-pcd.ply and place it in the same directory as the Python script.
3. Open the file in viewer such as MeshLab (https://www.meshlab.net/) for understanding the data.
4. Run the script using a Python interpreter such as visual studio code https://code.visualstudio.com/.
5. The script will load the point cloud, estimate normals if needed, convert it to a mesh, and save the resulting mesh as “bunny-pcd-to-mesh.ply” in the same directory.

CODE:

# Import libraries
import numpy as np
import open3d as o3d

# Load a point cloud
ply_file_path_read = "bunny-pcd.ply"
point_cloud = o3d.io.read_point_cloud(ply_file_path_read)

# Compute normals if PCD does not have
if point_cloud.has_normals() == False:
    print("Estimating normals...")
    point_cloud.estimate_normals()
else:
    print("PCD already has normals. So, skip estimating normals")

# Orient the computed normals w.r.t to the tangent plane. 
# This step will solve the normal direction issue. If this step is skipped, there might be holes in the mesh surfaces.
o3d.geometry.PointCloud.orient_normals_consistent_tangent_plane(point_cloud, 10)

# Estimate radius for rolling ball
distances = point_cloud.compute_nearest_neighbor_distance()
avg_dist = np.mean(distances)
radius = 1.5 * avg_dist   
print("Minimum neighbour distance = {:.6f}".format(np.min(distances)))
print("Maximum neighbour distance = {:.6f}".format(np.max(distances))) 
print("Average neighbour distance = {:.6f}".format(np.mean(distances)))

mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_ball_pivoting(
                                    point_cloud, 
                                    o3d.utility.DoubleVector([radius, radius * 2]))

print("PCD Detail: {}".format(point_cloud))
print("Mesh Details: {}".format(mesh))

# Visualize and save the mesh generated from point cloud
#mesh.paint_uniform_color(np.array([0.5, 1.0, 0.5])) # to uniformly color the surface
o3d.visualization.draw_geometries([mesh], window_name='mesh with estimated normals', width=1200, height=800)
o3d.io.write_triangle_mesh("bunny-pcd-to-mesh.ply", mesh, write_ascii=True)

OUTPUT:

Estimating normals...
Minimum neighbour distance = 0.000006
Maximum neighbour distance = 0.002240
Average neighbour distance = 0.00100347
PCD Detail: PointCloud with 35947 points.
Mesh Details: TriangleMesh with 35947 points and 71600 triangles.

The plots below show the results of the point cloud to mesh conversion. The top plot is the original PCD, and the bottom plot is the converted mesh.

Bunny Point Cloud Data

Converted Bunny Mesh data

In the above code, we have used “Estimate normals” and “Estimate radius for rolling ball” blocks. These are not explained here, and we will discuss them in detail in upcoming blogs, keeping the learning flow in mind. For now, we are focusing on “how to convert” rather than “how it works.” Thank you for your understanding.

Point Cloud to Voxel Grid Conversion

1. Converting point clouds to voxel grid is needed for many reasons,

1. Regularization: Point clouds are often irregularly sampled. Converting them to a voxel grid imposes a regular structure, providing a more organized and standardized representation of the 3D space.
2. Quantization: Voxel grids quantize the 3D space into small cubes (voxels). This simplifies spatial indexing, allowing for efficient and uniform data processing.
3. Resampling: Voxel grids enable uniform resampling of the 3D space, allowing you to change the voxel size to control data resolution.

2. Applications of Voxel Grid

1. Robotics: Voxel grids are commonly used in robotics navigation for tasks like occupancy mapping, path planning, and collision detection.
2. Medical Imaging: Voxel grids are used in medical imaging for representing volumetric data, such as CT scans and MRI images.

3. Benefits of Converting to Voxel Grid

1. Structured Representation: Voxel grids provide a structured, grid-based representation that simplifies spatial operations.
2. Memory Efficiency: Depending on the voxel size and sparsity, voxel grids can be memory-efficient for large-scale data.
3. Uniform Resolution: Voxel grids allow for uniform resampling of the data, enabling control over data resolution.

4. Cons of Converting to Voxel Grid

1. Complexity: Voxel grids introduce additional data structures and indexing mechanisms, increasing computational complexity for certain operations.
2. Information Loss: Fine-grained details present in the original point cloud can be lost depending on the voxel size.

5. Let’s code

Follow the below steps to run the script.

1. Create a file named “pcd-to-voxel.py” and paste the following code into it.
2. Download “bunny-pcd.ply” ply data from https://github.com/MatPixel/dataset-for-3d-pointcloud-processing-3d-deep-learning/blob/main/bunny-pcd.ply and place it in the same directory as the Python script.
3. Open the file in viewer such as MeshLab (https://www.meshlab.net/) for understanding the data.
4. Run the script using a Python interpreter such as visual studio code https://code.visualstudio.com/.
5. The script will load the point cloud, convert it to a voxel, and save the results as “bunny-pcd-to-voxel-001-unitcube.ply, “bunny-pcd-to-voxel-002-unitcube.ply”, “bunny-pcd-to-voxel-003-unitcube.ply” for different voxel size” in the same directory.

CODE:

# Import libraries
import numpy as np
import open3d as o3d

# Load a point cloud
ply_file_path_read = "bunny-pcd.ply"
point_cloud = o3d.io.read_point_cloud(ply_file_path_read)
#point_cloud = point_cloud.uniform_down_sample(every_k_points=10) # if there are huge points
point_cloud.scale(1 / np.max(point_cloud.get_max_bound() - point_cloud.get_min_bound()),center=point_cloud.get_center()) # Fit to unit cube
o3d.io.write_point_cloud("bunny-pcd-unitcube.ply", point_cloud, write_ascii=True)

# Estimate radius for rolling ball
distances = point_cloud.compute_nearest_neighbor_distance()
avg_dist = np.mean(distances)
print(f"Average neighbour distance = {np.mean(distances):.6}")

# Guess different voxel sizes based on points' distances
voxelsize = np.round(avg_dist, 5)
voxelsize_half = voxelsize/2
voxelsize_double = voxelsize*2

print(f"Different voxel size considered based on points' distances: {voxelsize}, {voxelsize_double}, {voxelsize_half}")

voxel_grid = o3d.geometry.VoxelGrid.create_from_point_cloud(point_cloud, voxel_size=voxelsize)
print(f"Voxel center in X,Y,Z = {voxel_grid.get_center()}")
print(f"Sample Voxels: \n {voxel_grid.get_voxels()[:3]}")
o3d.visualization.draw_geometries([voxel_grid], width=1200, height=800)
o3d.io.write_voxel_grid("bunny-pcd-to-voxel-001-unitcube.ply", voxel_grid, write_ascii=True)

voxel_grid = o3d.geometry.VoxelGrid.create_from_point_cloud(point_cloud, voxel_size=voxelsize_half)
o3d.visualization.draw_geometries([voxel_grid], width=1200, height=800)
o3d.io.write_voxel_grid("bunny-pcd-to-voxel-002-unitcube.ply", voxel_grid, write_ascii=True)
print(f"Voxel center in X,Y,Z = {voxel_grid.get_center()}")
print(f"Sample Voxels: \n {voxel_grid.get_voxels()[:3]}")

voxel_grid = o3d.geometry.VoxelGrid.create_from_point_cloud(point_cloud, voxel_size=voxelsize_double)
o3d.visualization.draw_geometries([voxel_grid], width=1200, height=800)
o3d.io.write_voxel_grid("bunny-pcd-to-voxel-003-unitcube.ply", voxel_grid, write_ascii=True)
print(f"Voxel center in X,Y,Z = {voxel_grid.get_center()}")
print(f"Sample Voxels: \n {voxel_grid.get_voxels()[:3]}")

OUTPUT:

Average neighbour distance = 0.00644491
Different voxel size considered based on points distances: 0.00644, 0.01288, 0.00322

Voxel center in X,Y,Z = [-0.02642576  0.09283475  0.00802933]
Sample Voxels: 
 [Voxel with grid_index: (67, 83, 47), color: (0, 0, 0), 
Voxel with grid_index: (50, 68, 104), color: (0, 0, 0), 
Voxel with grid_index: (87, 6, 96), color: (0, 0, 0)]

Voxel center in X,Y,Z = [-0.02655916  0.09503492  0.00905491]
Sample Voxels: 
 [Voxel with grid_index: (184, 159, 85), color: (0, 0, 0), 
Voxel with grid_index: (29, 86, 116), color: (0, 0, 0), 
Voxel with grid_index: (160, 190, 132), color: (0, 0, 0)]

Voxel center in X,Y,Z = [-0.02872222  0.09150737  0.00555566]
Sample Voxels:
 [Voxel with grid_index: (69, 31, 38), color: (0, 0, 0), 
Voxel with grid_index: (46, 48, 42), color: (0, 0, 0), 
Voxel with grid_index: (32, 28, 17), color: (0, 0, 0)]

The below plots show the results of point cloud to voxel conversion. Plots from left to right and top to bottom: voxel grid with voxel size = 0.00644, 0.01288, 0.00322 and the combined one respectively.

Left: Voxel grid with voxel size = 0.00644 , Right: Voxel grid with voxel size = 0.01288

Left: voxel size = 0.00322, Right: combined one

In the above code, we have used the “Estimate radius for rolling ball”, “Guess different voxel sizes based on points distances” and, “VoxelGrid.create_from_point_cloud()” blocks. These are not explained here, and we will discuss them in detail in upcoming blogs, keeping the learning flow in mind. For now, we are focusing on “how to convert” rather than “how it works.” Thank you for your understanding.

Summary:

In this blog, we explored the conversion of point clouds to meshes and voxel grids. We discussed the reasons for converting between different 3D data types, the process of conversion, and their applications in various fields. We also provided code examples for both conversions and visualized the results.

✨Happy exploring! Happy learning!✨

📝Next Blog Preview:

In the upcoming post, 🚀“Mastering 3D Computer Vision & Point Cloud Processing- Mod 10 — Mesh to Point Cloud and Mesh to Voxel Grid Conversion with Code”.

Topics to Be Discussed in the Next Blog

1. Mesh to Point Cloud Conversion
2. Mesh to Voxel Grid Conversion

Topics to Be Discussed in the Upcoming Blogs — Immediate Focus

• Discussion on Commonly used File formats
• Most used Dataset
• Most Point Cloud Preprocessing Techniques