Assignment 5: Point Cloud Processing

16-825 Learning for 3D Vision | Fall 2026 | Jinyao Zhou (jinyaozh)

Q1. Classification Model (30 points)

Q: Report the test accuracy.

Test accuracy: 0.977

Q: Visualize a few random test point clouds and mention the predicted classes for each.


#393: GT chair, pred chair

#523: GT chair, pred chair

#644: GT vase, pred vase

#648: GT vase, pred vase

#722: GT lamp, pred lamp

#736: GT lamp, pred lamp

Q: Also visualize at least 1 failure prediction for each class (chair, vase and lamp), and provide an interpretation of the failure in a few sentences.

Chair: no failure. The model classifies all 617 test chairs correctly (chair accuracy 1.000).

Vase accuracy: 0.902 (92 / 102).

Lamp accuracy: 0.949 (222 / 234).


#673: GT vase, pred lamp

#685: GT vase, pred lamp

#859: GT lamp, pred vase

#870: GT lamp, pred vase

Interpretation: Almost all errors are between vase and lamp. Both classes are tall objects that are approximately symmetric about the up axis, and some vases have a narrow neck and a wide top that look like a lamp shade, while some lamps have a wide base and a short body that look like a vase. A global max-pooled feature describes the overall shape, so it cannot easily separate these similar outlines. Chairs have no errors because their shape (seat, back, legs) is very different from the other two classes, and 65% of the training objects are chairs.

Q2. Segmentation Model (30 points)

Q: Report the test accuracy.

Test accuracy: 0.903

Q: Visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth

Q: Report the prediction accuracy for each object, and provide an interpretation of the result in a few sentences.

Left: ground truth. Right: prediction. Colors: ■ 0 other, ■ 1 headrest, ■ 2 back, ■ 3 arm, ■ 4 base, ■ 5 seat. The handout does not name the labels; I inferred the names from the part positions.


#315: accuracy 0.916

#392: accuracy 0.833

#523: accuracy 0.831

#541: accuracy 0.903

#351 (bad): accuracy 0.475

#235 (bad): accuracy 0.463

#163 (bad): accuracy 0.687

Interpretation: The large parts (back, seat, base) are accurate. Most errors are at the boundaries between parts that touch with no clear edge, for example back and headrest, or arm and seat. Headrest is the worst part (test accuracy 0.127): it is rare in the training data (61 / 4489 training chairs have a headrest; 58,518 / 44,890,000 training points, 0.13%), and about 90% of its points are predicted as back, because it continues the back surface (#163). When the headrest is a separate cushion with a clear gap above the back, the model segments it correctly (#541).

The other two bad objects have unusual shapes. #351 is a barrel chair where the back and the arms are one continuous surface; its ground truth has no base, and the whole outer shell down to the floor is labeled arm, but the model predicts the lower shell as base, which is the usual layout of a chair. #235 is a sofa-like chair, and its ground truth is ambiguous: 60% of the points are labeled arm, and in the seat and the front panel the arm and seat labels are mixed point by point. The back, arm and seat regions in the prediction are cleaner than in the ground truth. But, as in #351, the model predicts the lower part of the body as base, while the ground-truth base is only the four small feet.

PointNet labels each point from only its own coordinates and one global feature of the whole object. It does not know the local structure around the point, for example the part-level shape that the point is on, so it labels points mostly by their position (for example, by height). This fails when two parts meet with no clear edge, or when a part is not at its usual position.

Q3. Robustness Analysis (20 points)

Experiment 1: number of points per object

Q: Briefly describe your procedure

For each test object, I keep N random points (the same point indices for all objects) and remove the other points. I evaluate the Q1 and Q2 models on these points with no retraining. For segmentation, the accuracy uses the labels of the kept points.

Q: For each task (classification and segmentation), report test accuracy and visualization on a few samples, in comparison with your results from Q1 & Q2.

Points per object10000 (Q1, Q2)500010005001005010
Classification0.9770.9760.9720.9690.9180.7740.282
Segmentation0.9030.9030.8980.8870.8190.7900.663
10000 points (Q1)1000 points100 points50 points
#393 (chair)
pred chair

pred chair

pred chair

pred chair
#644 (vase)
pred vase

pred vase

pred lamp

pred lamp
#736 (lamp)
pred lamp

pred lamp

pred lamp

pred lamp
10000 points (Q2)1000 points100 points50 points
#315
accuracy 0.916

accuracy 0.918

accuracy 0.910

accuracy 0.920
#392
accuracy 0.833

accuracy 0.828

accuracy 0.710

accuracy 0.740

Q: Provide some interpretation of the results in a few sentences.

Interpretation: Accuracy stays almost the same down to 500–1000 points, and falls clearly only below 100 points. The global max pooling takes its value from only the few points that give the maximum value in at least one of the 1024 feature channels (the critical points). As long as the random subset of the points still covers the overall shape, the global feature changes little. With very few points, the outline is lost: classification falls to 0.282 at 10 points, and vases start to be predicted as lamps already at 100 points (#644).

Classification is more sensitive to the number of points than segmentation, because its prediction depends only on the global feature. Segmentation is on one category (chairs), where the part of a point depends mostly on its own coordinates (base low, seat in the middle, back high), and each kept point still has its correct coordinates. Thus the global feature is less important for segmentation, and its accuracy falls more slowly.

Experiment 2: rotation of the input

Q: Briefly describe your procedure

I rotate each test object about the x, y (up) or z axis through the origin, by an angle from 0° to 180°. I evaluate the Q1 and Q2 models on the rotated objects with no retraining. The visualizations show rotations about the x axis.

Q: For each task (classification and segmentation), report test accuracy and visualization on a few samples, in comparison with your results from Q1 & Q2.

Angle0° (Q1, Q2)30°60°90°180°
Classification, about x0.9770.7470.4180.3130.707
Classification, about y0.9770.8950.5900.6550.339
Classification, about z0.9770.7000.3320.2700.608
Segmentation, about x0.9030.7390.4550.2630.346
Segmentation, about y0.9030.7890.6460.5460.602
Segmentation, about z0.9030.6780.5330.4160.367
Classification, #393 (chair)Rotation axis0° (Q1)30°60°90°180°
about x
axis x (black line)

pred chair

pred chair

pred lamp

pred lamp

pred chair
about y
axis y (black line)

pred chair

pred chair

pred chair

pred chair

pred lamp
about z
axis z (black line)

pred chair

pred chair

pred chair

pred vase

pred lamp
Segmentation, #315Rotation axis0° (Q2)30°60°90°180°
about x
axis x (black line)

accuracy 0.916

accuracy 0.821

accuracy 0.708

accuracy 0.313

accuracy 0.274
about y
axis y (black line)

accuracy 0.916

accuracy 0.850

accuracy 0.656

accuracy 0.510

accuracy 0.485
about z
axis z (black line)

accuracy 0.916

accuracy 0.742

accuracy 0.672

accuracy 0.502

accuracy 0.379

Q: Provide some interpretation of the results in a few sentences.

Interpretation: Accuracy falls quickly for all three axes, already at 30°. The model has no T-Net and the training data has no rotation augmentation, so the network learns features of absolute coordinates in one fixed orientation. A tilted chair becomes a tall object and is predicted as a lamp (#393). In the segmentation renders at 60°–90°, the predicted parts look like horizontal layers (low points become base, high points become back), so the model uses height more than shape. Rotation about the up axis (y) does the least damage to segmentation, because the height of each point does not change. Some classification curves go up again near 180°, because an upside-down vase or lamp is still tall and narrow. Rotation augmentation, a T-Net, or rotation-invariant features can make the model more robust.

Q4. Bonus Question - Locality (20 points)

Q: Incorporate certain kind of locality as covered in the lecture (e.g. implement PointNet++, DGCNN, Transformers (https://arxiv.org/pdf/2012.09164v2.pdf), etc).

Q: Specify and briefly describe the model you have implemented

I implemented PointNet++ with single-scale grouping: two set abstraction levels (512 centroids, radius 0.2, 32 neighbors; 128 centroids, radius 0.4, 64 neighbors), a global level, and for segmentation three feature propagation levels. Each set abstraction level selects centroids with farthest point sampling, groups the points in a ball around each centroid, and runs a small PointNet on each group, with coordinates relative to the centroid. Classification uses the same fully connected head as Q1 on the global feature. Segmentation interpolates the features back to all points (inverse distance, 3 nearest points) and concatenates them with the features of the same level. Training settings are the same as Q1 and Q2.

Q: For each task (classification and segmentation), report the test accuracy of your best model, in comparison with your results from Q1 & Q2.

Test accuracyPointNet (Q1, Q2)PointNet++
Classification0.9770.976
Segmentation0.9030.913

Q: Visualize some results in comparison to ones obtained in the earlier questions

Classification samples

11 test objects are wrong with PointNet and correct with PointNet++, and 12 objects are the opposite. Confusion matrices (number of test objects; rows are ground truth, columns are predictions):

Ground truthPointNetPointNet++
chairvaselampclass accuracychairvaselampclass accuracy
chair (617)617001.000615020.997
vase (102)19290.902188130.863
lamp (234)0122220.949072270.970

Examples where PointNet++ is better than PointNet:


#870: GT lamp, PointNet vase, PointNet++ lamp

#667: GT vase, PointNet lamp, PointNet++ vase

Opposite examples, where PointNet++ is worse than PointNet:


#518: GT chair, PointNet chair, PointNet++ lamp

#832: GT lamp, PointNet lamp, PointNet++ vase

Segmentation samples

Test accuracy of each part (all test points with that ground-truth label):

Part0 other1 headrest2 back3 arm4 base5 seat
PointNet0.9290.1270.9270.7380.9030.924
PointNet++0.9000.1660.9320.8020.9230.921

The object accuracy of PointNet++ is higher than PointNet on 408 test objects and lower on 206. Left to right in each gif: ground truth, PointNet (Q2), PointNet++.

Improved

#212 (largest gain): PointNet 0.766, PointNet++ 0.950

#489 (second largest gain): PointNet 0.686, PointNet++ 0.860

#315 (Q2 random sample): PointNet 0.916, PointNet++ 0.952

#392 (Q2 random sample): PointNet 0.833, PointNet++ 0.864
Not improved

#351 (Q2 bad sample): PointNet 0.475, PointNet++ 0.473

#135 (largest loss): PointNet 0.864, PointNet++ 0.746

Interpretation: Each set abstraction level of PointNet++ runs a small PointNet on the points in a ball, with coordinates relative to the ball center. Thus each point gets features of the local shape around it, not only its position and one global feature.

Classification. Classification does not improve (0.977 and 0.976). The class of an object depends mostly on its global shape, which the global feature of PointNet already describes well. The main addition of PointNet++ is local receptive fields, which help to label each point but give little help to classify the whole object.

Segmentation. Segmentation improves by 1%, and most of the gain is on arm (0.738 to 0.802), a part that touches the seat and the back with no clear boundary and needs more fine-grained perception of geometry. The improved samples show that PointNet++ labels points more by the local shape and less by their coordinates. In #212, the seat is inside a bucket-shaped back; PointNet labels 1355 of the 3541 seat points and 568 of the 1249 arm points as back, because they are in the region of the back, and PointNet++ labels almost all of them correctly. These points are hidden behind the back in the render, so the difference is hard to see. #489 has no arms, but PointNet labels 1907 seat points at the sides as arm, because they are at the usual position of arms; PointNet++ sees that they are on the same surface as the seat and labels only 149 as arm.

PointNet++ still does not generalize to unusual shapes. #351 does not improve (0.475 to 0.473): its ground truth has no base and the whole shell is labeled arm, which is rare in the training data. #135 becomes worse (0.864 to 0.746): its seat is a curved surface above a round base, and PointNet++ labels most of the seat as base, because the local shape of the curved seat is similar to the base.

Robustness: PointNet vs PointNet++

Same procedure as Q3, with the PointNet++ models. Mean test accuracy over the rotation angles:

Rotation axisClassification, 15°–30°Classification, 15°–180°Segmentation, 15°–30°Segmentation, 15°–180°
PointNetPointNet++PointNetPointNet++PointNetPointNet++PointNetPointNet++
x0.8310.8820.5830.4550.7910.8470.4560.437
y (up)0.9290.9320.6090.6360.8270.8840.6500.802
z0.8220.9270.4490.5270.7500.8440.5050.537



Interpretation: PointNet++ is more robust to small pose changes, but less robust to fewer points.

Rotation. For both tasks, PointNet++ is better for small rotations (15°–30°) about all axes, because its local features use coordinates relative to the ball center. For segmentation, it is also better for all rotations about the up axis (y): the mean accuracy over 15°–180° is 0.802, PointNet 0.650. For classification, it is worse for large tilts about x.

Number of points. PointNet++ is much worse with few points: at 100 points, classification is 0.257 (PointNet 0.918) and segmentation is 0.608 (PointNet 0.819). The ball radius is fixed and the model saw only dense 10000-point objects in training, so with sparse input most balls contain only a few points. The PointNet++ paper makes the model adapt to point density with multi-scale grouping (MSG) or multi-resolution grouping (MRG), and with random input dropout in training (each training object loses a random 0–95% of its points). My implementation uses single-scale grouping and no input dropout, so it is sensitive to the number of points. These density-adaptive methods should decrease this problem.