3D 数据的持续学习正在成为机器学习和理解我们周围世界的越来越重要的部分。 随着新的 3D 数据提取硬件（如深度相机和 LiDAR）在闭路电视、相机和智能手机中变得司空见惯，越来越多的人正在使用它提供的额外维度。 此外，摄影测量和 Structure from Motion 正在成为 3D 重建和建模流程的正常组成部分，并且提取和处理大型 3D 数据集正在成为必需品。 3D 深度学习的非结构化数据可以有不同的表示：
点云 [3,4,11]
体素和体素网格 [1, 8, 9]
深度图 [2]
图 [5, 10]
这些远非所有可能的 3D 数据演示，还有其他的，如参数化 CAD 模型、多视图图像、体积等。要真正了解其中的一些内容，你可以阅读 Florent Paux 撰写的信息丰富的文章“如何表示 3D 数据？”。
在本文中，我们将专注于将 3D 数据表示为体素。 但首先什么是体素？ 最简单的比较是体素是 3D 像素。 体素被排列成体素网格，这可以看作是图像有序结构的 3D 等价物。 当点云或网格变成体素表示时，它与体素网格相交。 点云或网格中的点然后落在某些体素中。 这些体素被留下，而所有其他不与任何点相交的体素要么被丢弃，要么被归零，我们剩下的是对象的雕刻表示。 体素化可以只是表面级别或整个网格/点云体积。
# Read the bunny statue point cloud using numpy's loadtxt point_cloud_path = os.path.join('point_cloud','bunnyStatue.txt') point_cloud = np.loadtxt(point_cloud_path, delimiter=' ') # Separate the into points, colors and normals array points = point_cloud[:,:3] colors = point_cloud[:,3:6] normals = point_cloud[:,6:]
# Initialize a point cloud object pcd = o3d.geometry.PointCloud() # Add the points, colors and normals as Vectors pcd.points = o3d.utility.Vector3dVector(points) pcd.colors = o3d.utility.Vector3dVector(colors) pcd.normals = o3d.utility.Vector3dVector(normals)
# Create a voxel grid from the point cloud with a voxel_size of 0.01 voxel_grid=o3d.geometry.VoxelGrid.create_from_point_cloud(pcd,voxel_size=0.01)
# Initialize a visualizer object vis = o3d.visualization.Visualizer() # Create a window, name it and scale it vis.create_window(window_name='Bunny Visualize', width=800, height=600)
# Add the voxel grid to the visualizer vis.add_geometry(voxel_grid)
# We run the visualizater vis.run() # Once the visualizer is closed destroy the window and clean up vis.destroy_window()
# Parameters class holding variables that change in the callback function classparams(): # counter for the rotation counter = 0 # counter for selecting a new voxel size sizes_counter = 0 # array of voxel sizes between 0.01 and 0.1 voxel_sizes = np.arange(0.01,0.1,0.005) # empty TriangleMesh object that will contain the cubes vox_mesh = o3d.geometry.TriangleMesh() # boolean value used for initial initialization of the voxel mesh initialize = True # Callback function used to construct and rotate the voxel meshes defrotate_and_change(vis):
# When the counter is 0 generate the voxel grid and construct the voxel mesh if params.counter == 0: # generate the voxel grid using the voxel sizes setup in the params class voxel_grid=o3d.geometry.VoxelGrid.create_from_point_cloud(pcd,voxel_size=params.voxel_sizes[params.sizes_counter]) # get all voxels in the voxel grid voxels_all= voxel_grid.get_voxels() # geth the calculated size of a voxel voxel_size = voxel_grid.voxel_size # loop through all the voxels for voxel in voxels_all: # create a cube mesh with a size 1x1x1 cube=o3d.geometry.TriangleMesh.create_box(width=1, height=1, depth=1) # paint it with the color of the current voxel cube.paint_uniform_color(voxel.color) # scale the box using the size of the voxel cube.scale(voxel_size, center=cube.get_center()) # get the center of the current voxel voxel_center = voxel_grid.get_voxel_center_coordinate(voxel.grid_index) # translate the box to the center of the voxel cube.translate(voxel_center, relative=False) # add the box to the TriangleMesh object params.vox_mesh+=cube # on the first run of the callback loop initialize the Triangle mesh by adding it to the Visualizer. In subsequent iterations just update the geometry if params.initialize: vis.add_geometry(params.vox_mesh) params.initialize = False else: vis.update_geometry(params.vox_mesh)
# We create a 3D rotation matrix from x,y,z rotations, the rotations need to be given in radians R = params.vox_mesh.get_rotation_matrix_from_xyz((0, np.deg2rad(2), 0)) # The rotation matrix is applied to the specified object - in our case the voxel mesh. We can also specify the rotation pivot center params.vox_mesh.rotate(R, center=(0, 0, 0)) # tick the counter up params.counter+=1 # For the changes to be seen we need to update both the geometry that has been changed and to update the whole renderer connected to the visualizer vis.update_geometry(params.vox_mesh) vis.update_renderer()
# When the counter gets to 180 we zero it out. This is done because we rotate the mesh by 2 degrees on an iteration if params.counter >= 180: params.counter=0 # we tick the voxel size counter params.sizes_counter +=1 # if the voxel size counter becomes equal to the size of the voxel sizes array, reset it if params.sizes_counter >= len(params.voxel_sizes): params.sizes_counter=0 # each time we clear the mesh. This is important, because without it we will just add more and more box primitives to the mesh object params.vox_mesh.clear()
# Read the bunny statue point cloud using numpy's loadtxt point_cloud_path = os.path.join('point_cloud','bunnyStatue.txt') point_cloud = np.loadtxt(point_cloud_path, delimiter=' ') # Separate the into points, colors and normals array points = point_cloud[:,:3] colors = point_cloud[:,3:6] normals = point_cloud[:,6:]
# Initialize a point cloud object pcd = o3d.geometry.PointCloud() # Add the points, colors and normals as Vectors pcd.points = o3d.utility.Vector3dVector(points) pcd.colors = o3d.utility.Vector3dVector(colors) pcd.normals = o3d.utility.Vector3dVector(normals)
# Create a voxel grid from the point cloud with a voxel_size of 0.01 # voxel_grid=o3d.geometry.VoxelGrid.create_from_point_cloud(pcd,voxel_size=0.01)
# Initialize a visualizer object vis = o3d.visualization.Visualizer() # Create a window, name it and scale it vis.create_window(window_name='Bunny Visualize', width=800, height=600)
# Add the voxel grid to the visualizer # vis.add_geometry(voxel_grid)
# Register the callback function vis.register_animation_callback(rotate_and_change) # We run the visualizater vis.run() # Once the visualizer is closed destroy the window and clean up vis.destroy_window()
# Helper parameters class containing variables that will change in the callback function classparams(): # voxels counter that will stop the voxel mesh generation when there are no more voxels in the voxel grid counter = 0 vox_mesh=o3d.geometry.TriangleMesh()
# Voxel builder callback function defbuild_voxels(vis): # check if the counter is more than the amount of voxels if params.counter < len(voxels_all): # get the size of the voxels voxel_size = voxel_grid.voxel_size # create a box primitive of size 1x1x1 cube=o3d.geometry.TriangleMesh.create_box(width=1, height=1, depth=1) # paint the box uniformly with the color of the voxel cube.paint_uniform_color(voxels_all[params.counter].color) # scale the box to the size of the voxel cube.scale(voxel_size, center=cube.get_center()) # get the center position of the current voxel voxel_center = voxel_grid.get_voxel_center_coordinate(voxels_all[params.counter].grid_index) # translate the box to the voxel center cube.translate(voxel_center, relative=False) # add the box primitive to the voxel mesh params.vox_mesh+=cube # on the first loop create the geometry and on subsequent iterations update the geometry if params.counter==0: vis.add_geometry(params.vox_mesh) else: vis.update_geometry(params.vox_mesh)
# update the renderer vis.update_renderer() # tick up the counter params.counter+=1
# Read the bunny statue point cloud using numpy's loadtxt point_cloud_path = os.path.join('point_cloud','bunnyStatue.txt') point_cloud = np.loadtxt(point_cloud_path, delimiter=' ') # Separate the into points, colors and normals array points = point_cloud[:,:3] colors = point_cloud[:,3:6] normals = point_cloud[:,6:]
# Initialize a point cloud object pcd = o3d.geometry.PointCloud() # Add the points, colors and normals as Vectors pcd.points = o3d.utility.Vector3dVector(points) pcd.colors = o3d.utility.Vector3dVector(colors) pcd.normals = o3d.utility.Vector3dVector(normals) # Create a voxel grid from the point cloud with a voxel_size of 0.01 voxel_grid=o3d.geometry.VoxelGrid.create_from_point_cloud(pcd,voxel_size=0.01) # Get all the voxels in the voxel grid voxels_all= voxel_grid.get_voxels()
# get all the centers and colors from the voxels in the voxel grid all_centers=[] all_colors=[] for voxel in voxels_all: voxel_center = voxel_grid.get_voxel_center_coordinate(voxel.grid_index) all_centers.append(voxel_center) all_colors.append(voxel.color)
# Initialize a visualizer object vis = o3d.visualization.Visualizer() # Create a window, name it and scale it vis.create_window(window_name='Bunny Visualize', width=800, height=600) # Create a point cloud that will contain the voxel centers as points pcd_centers = o3d.geometry.PointCloud() # Tranform the numpy array into points for the point cloud pcd_centers.points = o3d.utility.Vector3dVector(all_centers) # pcd_centers.colors = o3d.utility.Vector3dVector(all_colors)' pcd_centers.paint_uniform_color([1, 0, 0]) # Add the voxel centers point cloud to the visualizer vis.add_geometry(pcd_centers)
# Invoke the callback function vis.register_animation_callback(build_voxels) # We run the visualizater vis.run() # Once the visualizer is closed destroy the window and clean up vis.destroy_window()
# Set up the path to the point cloud point_cloud_path = os.path.join('point_cloud','bunnyStatue.txt') # Load the point cloud. As internally from_file is calling Pandas, we set Pandas input parameters like separator, header and column names cloud = PyntCloud.from_file(point_cloud_path, sep=" ", header=0, names=["x","y","z","r","g","b","nx","ny","nz"])
# We use the imported point cloud to create a voxel grid of size 64x64x64. voxelgrid_id = cloud.add_structure("voxelgrid", n_x=64, n_y=64, n_z=64) # We use the calculated occupied voxel grid ids to create the voxel representation of the point cloud voxelgrid = cloud.structures[voxelgrid_id] # We extract the density feature for each occupied voxel that we will use for coloring the voxels density_feature_vector = voxelgrid.get_feature_vector(mode="density") # Calculate the maximum density to normalize the colors max_density = density_feature_vector.max() # We extract the shape of a voxel, as well as the position of each point in X, Y, Z in the voxel grid voxel_size = voxelgrid.shape x_cube_pos = voxelgrid.voxel_x y_cube_pos = voxelgrid.voxel_y z_cube_pos = voxelgrid.voxel_z
from pyntcloud import PyntCloud import os import open3d as o3d
# Set up the path to the point cloud point_cloud_path = os.path.join('point_cloud','bunnyStatue.txt') # Load the point cloud. As internally from_file is calling Pandas, we set Pandas input parameters like separator, header and column names cloud = PyntCloud.from_file(point_cloud_path, sep=" ", header=0, names=["x","y","z","r","g","b","nx","ny","nz"])
# We use the imported point cloud to create a voxel grid of size 64x64x64. voxelgrid_id = cloud.add_structure("voxelgrid", n_x=64, n_y=64, n_z=64) # We use the calculated occupied voxel grid ids to create the voxel representation of the point cloud voxelgrid = cloud.structures[voxelgrid_id] # We extract the density feature for each occupied voxel that we will use for coloring the voxels density_feature_vector = voxelgrid.get_feature_vector(mode="density") # Calculate the maximum density to normalize the colors max_density = density_feature_vector.max() # We extract the shape of a voxel, as well as the position of each point in X, Y, Z in the voxel grid voxel_size = voxelgrid.shape x_cube_pos = voxelgrid.voxel_x y_cube_pos = voxelgrid.voxel_y z_cube_pos = voxelgrid.voxel_z
# Initialize a open3d triangle mesh object vox_mesh = o3d.geometry.TriangleMesh()
# go through all voxelgrid voxels for idx inrange(0, len(voxelgrid.voxel_n)): # get the id of the current voxel in the voxel grid curr_number = voxelgrid.voxel_n[idx] # get the center of the voxel grid voxel voxel_center = voxelgrid.voxel_centers[curr_number] # get the density value of the current voxel. Because the density matrix is in the shape X,Y,Z, where they are the coordinates in the voxel grid # we use the voxel grid positions we already curr_density = density_feature_vector[x_cube_pos[idx],y_cube_pos[idx],z_cube_pos[idx]] # we normalize the value using the maximum density curr_density_normalized = curr_density / max_density # create a box primitive in open3d primitive=o3d.geometry.TriangleMesh.create_box(width=1, height=1, depth=1) # paint the box uniformly using the normalized density primitive.paint_uniform_color((curr_density_normalized,curr_density_normalized,curr_density_normalized)) # scale the cube using the saved voxel size primitive.scale(voxel_size[0], center=primitive.get_center()) # we translate the box to the center position of the voxel primitive.translate(voxel_center, relative=True) # add to the voxel mesh vox_mesh+=primitive
# Initialize a visualizer object vis = o3d.visualization.Visualizer() # Create a window, name it and scale it vis.create_window(window_name='Bunny Visualize', width=800, height=600) # add the voxel mesh to the visualizer vis.add_geometry(vox_mesh) vis.run() # Once the visualizer is closed destroy the window and clean up vis.destroy_window()
# Selection function which takes a name of the primitive and returns a mesh object # For the sphere and torus we lower the default resolution values so they can be visualized easier defchoose_primitive(name): return { 'box': o3d.geometry.TriangleMesh.create_box(), 'cone': o3d.geometry.TriangleMesh.create_cone(), 'sphere': o3d.geometry.TriangleMesh.create_sphere(resolution=5), 'cylinder': o3d.geometry.TriangleMesh.create_cylinder(), 'icosahedron': o3d.geometry.TriangleMesh.create_icosahedron(), 'octahedron': o3d.geometry.TriangleMesh.create_octahedron(), 'torus': o3d.geometry.TriangleMesh.create_torus( radial_resolution=10, tubular_resolution=5) }[name]
# Load the rooster mesh. Trimesh directly detects that the mesh is textured and contains a material mesh_path = 'mesh/rooster.obj' mesh = trimesh.load(mesh_path)
# Voxelize the loaded mesh with a voxel size of 0.01. We also call hollow() to remove the inside voxels, which will help with color calculation angel_voxel = mesh.voxelized(0.01).hollow()
# Transform the texture information to color information, mapping it to each vertex. Transform it to a numpy array only_colors = mesh.visual.to_color().vertex_colors only_colors = np.asarray(only_colors) # If we want to add the color information to the mesh uncomment this part # mesh.visual = mesh.visual.to_color()
# Extract the mesh vertices mesh_verts = mesh.vertices
# We use the ProximityQuery built-in function to get the closest voxel point centers to each vertex of the mesh _,vert_idx = trimesh.proximity.ProximityQuery(mesh).vertex(angel_voxel.points)
# We initialize a array of zeros of size X,Y,Z,4 to contain the colors for each voxel of the voxelized mesh in the grid cube_color=np.zeros([angel_voxel.shape[0],angel_voxel.shape[1],angel_voxel.shape[2],4])
# We loop through all the calculated closest voxel points for idx, vert inenumerate(vert_idx): # Get the voxel grid index of each closets voxel center point vox_verts = angel_voxel.points_to_indices(mesh_verts[vert]) # Get the color vertex color curr_color = only_colors[vert] # Set the alpha channel of the color curr_color[3] = 255 # add the color to the specific voxel grid index cube_color[vox_verts[0],vox_verts[1], vox_verts[2],:] = curr_color
# generate a voxelized mesh from the voxel grid representation, using the calculated colors voxelized_mesh = angel_voxel.as_boxes(colors=cube_color)
# Initialize a scene s = trimesh.Scene() # Add the voxelized mesh to the scene. If want to also show the intial mesh uncomment the second line and change the alpha channel of in the loop to something <100 s.add_geometry(voxelized_mesh) # s.add_geometry(mesh) s.show()
# Load mesh and texture into PyVista mesh_path = os.path.join('mesh','rooster.obj') mesh = pv.read(mesh_path) tex_path = os.path.join('mesh','rooster01.jpg') tex = pv.read_texture(tex_path)
# Initialize the plotter object with four sub plots pl = pv.Plotter(shape=(2, 2)) # First subplot show the mesh with the texture pl.subplot(0, 0) pl.add_mesh(mesh,name='rooster',texture = tex)
# Second subplot show the voxelized repsentation of the mesh with voxel size of 0.01. We remove the surface check as the mesh has small imperfections pl.subplot(0, 1) voxels = pv.voxelize(mesh, density=0.01, check_surface=False) # We add the voxels as a new mesh, add color and show their edges pl.add_mesh(voxels, color=True, show_edges=True)
# Third subplot shows the voxel representation using cones pl.subplot(1,0) glyphs = voxels.glyph(factor=1e-3, geom=pv.Cone()) pl.add_mesh(glyphs)
# Forth subplot shows the voxels together with a contour showing the per voxel distance to the mesh pl.subplot(1,1) # Calculate the distance between the voxels and the mesh. Add the results as a new scalar to the voxels voxels.compute_implicit_distance(mesh, inplace=True) # Create a contour representing the calculated distance contours = voxels.contour(6, scalars="implicit_distance") # Add the voxels and the contour with different opacity to show both pl.add_mesh(voxels, opacity=0.25, scalars="implicit_distance") pl.add_mesh(contours, opacity=0.5, scalars="implicit_distance")
# Link all four views so all cameras are moved at the same time pl.link_views() # Set camera start position pl.camera_position = 'xy' # Show everything pl.show()
# Function to generate density for each voxel and add it as a field defcalculate_neighbours(mesh, voxel_size = 0.01): # voxelize the given mesh with a specified size voxels voxels = pv.voxelize(mesh, density=voxel_size, check_surface=False) # Get the voxel center points voxel_centers = voxels.cell_centers().points # Get the mesh vertices mesh_vertices = mesh.points # Calculate the KDTree of the mesh vertices from Scipy kd_tree_vertices = KDTree(mesh_vertices) # Call the sphere radius function and calculate the new radius radius = calculate_sphere_radius(voxel_size) # Use the calculated KDTree and radius to get the neighbors for each voxel center neighbours = kd_tree_vertices.query_ball_point(voxel_centers,radius) # Count the number of points for each voxel center neighbour_count = [len(curr_neighbours) for curr_neighbours in neighbours] # Cast to array and normalize between 0 and 1 neighbour_count = np.array(neighbour_count, dtype=np.float32) neighbour_density = neighbour_count/neighbour_count.max() # Add the density as a field to the voxels voxels['density'] = neighbour_density
import pyvista as pv import os import numpy as np from scipy.spatial import KDTree
# Function to calculate the radius of a sphere with an equal volume as a cube with a given side defcalculate_sphere_radius(voxel_size = 0.01): voxel_volume = voxel_size ** 3 radius = ((3*voxel_volume)/(4*np.pi))**(1/3) return radius
# Function to generate density for each voxel and add it as a field defcalculate_neighbours(mesh, voxel_size = 0.01): # voxelize the given mesh with a specified size voxels voxels = pv.voxelize(mesh, density=voxel_size, check_surface=False) # Get the voxel center points voxel_centers = voxels.cell_centers().points # Get the mesh vertices mesh_vertices = mesh.points # Calculate the KDTree of the mesh vertices from Scipy kd_tree_vertices = KDTree(mesh_vertices) # Call the sphere radius function and calculate the new radius radius = calculate_sphere_radius(voxel_size) # Use the calculated KDTree and radius to get the neighbors for each voxel center neighbours = kd_tree_vertices.query_ball_point(voxel_centers,radius) # Count the number of points for each voxel center neighbour_count = [len(curr_neighbours) for curr_neighbours in neighbours] # Cast to array and normalize between 0 and 1 neighbour_count = np.array(neighbour_count, dtype=np.float32) neighbour_density = neighbour_count/neighbour_count.max() # Add the density as a field to the voxels voxels['density'] = neighbour_density
return voxels
# Function to visualize and threshold the voxel representation based on the calculated density defvisualize_thresh(voxels): p = pv.Plotter() p.add_mesh_threshold(voxels,show_edges=True) p.show()
# Class for interactive voxelization of a mesh with different sizes voxel based on a GUI slider classvisualize_voxel_levels(): # class initialization def__init__(self,mesh, plotter): self.plot = plotter self.mesh = mesh # Function for voxelization, which is called as a callback from the slider widget defvoxelize_mesh(self, value): # The function gets the slider value as a input and calculates the voxel representation voxels = pv.voxelize(self.mesh, density=value, check_surface=False) # It then adds the voxels as a new mesh to the Plotter. Here it's important to not forget the name in the add_mesh, # as this tells PyVista that this is the same mesh so it does not create a new one each time self.plot.add_mesh(voxels, name='voxel_mesh', show_edges=True) return # Function for adding the widget to the Plotter and visualizing everything defshow_different_voxel_levels(self): self.plot.camera_position = 'xy' # Create a widget give it the callback function, a minimum and maximum range, initial value, title and event type - # in our case it calls the callback function every time the widget slider is interacted with self.plot.add_slider_widget(self.voxelize_mesh, [0.01, 0.2], 0.01, title='Voxel Size', event_type = "always") self.plot.show() if __name__ == '__main__': # Load the rooster statue mesh mesh_path = os.path.join('mesh','rooster.obj') mesh = pv.read(mesh_path)
# Initialize the class and call the visualization function vis_vox = visualize_voxel_levels(mesh, pv.Plotter()) vis_vox.show_different_voxel_levels()
