16-825 Learning for 3D Vision | Fall 2026 | Jinyao Zhou (jinyaozh)
Q: Report the test accuracy.
Test accuracy: 0.977
Q: Visualize a few random test point clouds and mention the predicted classes for each.
![]() #393: GT chair, pred chair |
![]() #523: GT chair, pred chair |
![]() #644: GT vase, pred vase |
![]() #648: GT vase, pred vase |
![]() #722: GT lamp, pred lamp |
![]() #736: GT lamp, pred lamp |
Q: Also visualize at least 1 failure prediction for each class (chair, vase and lamp), and provide an interpretation of the failure in a few sentences.
Chair: no failure. The model classifies all 617 test chairs correctly (chair accuracy 1.000).
Vase accuracy: 0.902 (92 / 102).
Lamp accuracy: 0.949 (222 / 234).
![]() #673: GT vase, pred lamp |
![]() #685: GT vase, pred lamp |
![]() #859: GT lamp, pred vase |
![]() #870: GT lamp, pred vase |
Interpretation: Almost all errors are between vase and lamp. Both classes are tall objects that are approximately symmetric about the up axis, and some vases have a narrow neck and a wide top that look like a lamp shade, while some lamps have a wide base and a short body that look like a vase. A global max-pooled feature describes the overall shape, so it cannot easily separate these similar outlines. Chairs have no errors because their shape (seat, back, legs) is very different from the other two classes, and 65% of the training objects are chairs.
Q: Report the test accuracy.
Test accuracy: 0.903
Q: Visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth
Q: Report the prediction accuracy for each object, and provide an interpretation of the result in a few sentences.
Left: ground truth. Right: prediction. Colors: ■ 0 other, ■ 1 headrest, ■ 2 back, ■ 3 arm, ■ 4 base, ■ 5 seat. The handout does not name the labels; I inferred the names from the part positions.
![]() #315: accuracy 0.916 |
![]() #392: accuracy 0.833 |
![]() #523: accuracy 0.831 |
![]() #541: accuracy 0.903 |
![]() #351 (bad): accuracy 0.475 |
![]() #235 (bad): accuracy 0.463 |
![]() #163 (bad): accuracy 0.687 |
Interpretation: The large parts (back, seat, base) are accurate. Most errors are at the boundaries between parts that touch with no clear edge, for example back and headrest, or arm and seat. Headrest is the worst part (test accuracy 0.127): it is rare in the training data (61 / 4489 training chairs have a headrest; 58,518 / 44,890,000 training points, 0.13%), and about 90% of its points are predicted as back, because it continues the back surface (#163). When the headrest is a separate cushion with a clear gap above the back, the model segments it correctly (#541).
The other two bad objects have unusual shapes. #351 is a barrel chair where the back and the arms are one continuous surface; its ground truth has no base, and the whole outer shell down to the floor is labeled arm, but the model predicts the lower shell as base, which is the usual layout of a chair. #235 is a sofa-like chair, and its ground truth is ambiguous: 60% of the points are labeled arm, and in the seat and the front panel the arm and seat labels are mixed point by point. The back, arm and seat regions in the prediction are cleaner than in the ground truth. But, as in #351, the model predicts the lower part of the body as base, while the ground-truth base is only the four small feet.
PointNet labels each point from only its own coordinates and one global feature of the whole object. It does not know the local structure around the point, for example the part-level shape that the point is on, so it labels points mostly by their position (for example, by height). This fails when two parts meet with no clear edge, or when a part is not at its usual position.
Q: Briefly describe your procedure
For each test object, I keep N random points (the same point indices for all objects) and remove the other points. I evaluate the Q1 and Q2 models on these points with no retraining. For segmentation, the accuracy uses the labels of the kept points.
Q: For each task (classification and segmentation), report test accuracy and visualization on a few samples, in comparison with your results from Q1 & Q2.
| Points per object | 10000 (Q1, Q2) | 5000 | 1000 | 500 | 100 | 50 | 10 |
|---|---|---|---|---|---|---|---|
| Classification | 0.977 | 0.976 | 0.972 | 0.969 | 0.918 | 0.774 | 0.282 |
| Segmentation | 0.903 | 0.903 | 0.898 | 0.887 | 0.819 | 0.790 | 0.663 |
| 10000 points (Q1) | 1000 points | 100 points | 50 points | |
|---|---|---|---|---|
| #393 (chair) | ![]() pred chair |
![]() pred chair |
![]() pred chair |
![]() pred chair |
| #644 (vase) | ![]() pred vase |
![]() pred vase |
![]() pred lamp |
![]() pred lamp |
| #736 (lamp) | ![]() pred lamp |
![]() pred lamp |
![]() pred lamp |
![]() pred lamp |
| 10000 points (Q2) | 1000 points | 100 points | 50 points | |
|---|---|---|---|---|
| #315 | ![]() accuracy 0.916 |
![]() accuracy 0.918 |
![]() accuracy 0.910 |
![]() accuracy 0.920 |
| #392 | ![]() accuracy 0.833 |
![]() accuracy 0.828 |
![]() accuracy 0.710 |
![]() accuracy 0.740 |
Q: Provide some interpretation of the results in a few sentences.
Interpretation: Accuracy stays almost the same down to 500–1000 points, and falls clearly only below 100 points. The global max pooling takes its value from only the few points that give the maximum value in at least one of the 1024 feature channels (the critical points). As long as the random subset of the points still covers the overall shape, the global feature changes little. With very few points, the outline is lost: classification falls to 0.282 at 10 points, and vases start to be predicted as lamps already at 100 points (#644).
Classification is more sensitive to the number of points than segmentation, because its prediction depends only on the global feature. Segmentation is on one category (chairs), where the part of a point depends mostly on its own coordinates (base low, seat in the middle, back high), and each kept point still has its correct coordinates. Thus the global feature is less important for segmentation, and its accuracy falls more slowly.
Q: Briefly describe your procedure
I rotate each test object about the x, y (up) or z axis through the origin, by an angle from 0° to 180°. I evaluate the Q1 and Q2 models on the rotated objects with no retraining. The visualizations show rotations about the x axis.
Q: For each task (classification and segmentation), report test accuracy and visualization on a few samples, in comparison with your results from Q1 & Q2.
| Angle | 0° (Q1, Q2) | 30° | 60° | 90° | 180° |
|---|---|---|---|---|---|
| Classification, about x | 0.977 | 0.747 | 0.418 | 0.313 | 0.707 |
| Classification, about y | 0.977 | 0.895 | 0.590 | 0.655 | 0.339 |
| Classification, about z | 0.977 | 0.700 | 0.332 | 0.270 | 0.608 |
| Segmentation, about x | 0.903 | 0.739 | 0.455 | 0.263 | 0.346 |
| Segmentation, about y | 0.903 | 0.789 | 0.646 | 0.546 | 0.602 |
| Segmentation, about z | 0.903 | 0.678 | 0.533 | 0.416 | 0.367 |
| Classification, #393 (chair) | Rotation axis | 0° (Q1) | 30° | 60° | 90° | 180° |
|---|---|---|---|---|---|---|
| about x | ![]() axis x (black line) |
![]() pred chair |
![]() pred chair |
![]() pred lamp |
![]() pred lamp |
![]() pred chair |
| about y | ![]() axis y (black line) |
![]() pred chair |
![]() pred chair |
![]() pred chair |
![]() pred chair |
![]() pred lamp |
| about z | ![]() axis z (black line) |
![]() pred chair |
![]() pred chair |
![]() pred chair |
![]() pred vase |
![]() pred lamp |
| Segmentation, #315 | Rotation axis | 0° (Q2) | 30° | 60° | 90° | 180° |
|---|---|---|---|---|---|---|
| about x | ![]() axis x (black line) |
![]() accuracy 0.916 |
![]() accuracy 0.821 |
![]() accuracy 0.708 |
![]() accuracy 0.313 |
![]() accuracy 0.274 |
| about y | ![]() axis y (black line) |
![]() accuracy 0.916 |
![]() accuracy 0.850 |
![]() accuracy 0.656 |
![]() accuracy 0.510 |
![]() accuracy 0.485 |
| about z | ![]() axis z (black line) |
![]() accuracy 0.916 |
![]() accuracy 0.742 |
![]() accuracy 0.672 |
![]() accuracy 0.502 |
![]() accuracy 0.379 |
Q: Provide some interpretation of the results in a few sentences.
Interpretation: Accuracy falls quickly for all three axes, already at 30°. The model has no T-Net and the training data has no rotation augmentation, so the network learns features of absolute coordinates in one fixed orientation. A tilted chair becomes a tall object and is predicted as a lamp (#393). In the segmentation renders at 60°–90°, the predicted parts look like horizontal layers (low points become base, high points become back), so the model uses height more than shape. Rotation about the up axis (y) does the least damage to segmentation, because the height of each point does not change. Some classification curves go up again near 180°, because an upside-down vase or lamp is still tall and narrow. Rotation augmentation, a T-Net, or rotation-invariant features can make the model more robust.
Q: Incorporate certain kind of locality as covered in the lecture (e.g. implement PointNet++, DGCNN, Transformers (https://arxiv.org/pdf/2012.09164v2.pdf), etc).
Q: Specify and briefly describe the model you have implemented
I implemented PointNet++ with single-scale grouping: two set abstraction levels (512 centroids, radius 0.2, 32 neighbors; 128 centroids, radius 0.4, 64 neighbors), a global level, and for segmentation three feature propagation levels. Each set abstraction level selects centroids with farthest point sampling, groups the points in a ball around each centroid, and runs a small PointNet on each group, with coordinates relative to the centroid. Classification uses the same fully connected head as Q1 on the global feature. Segmentation interpolates the features back to all points (inverse distance, 3 nearest points) and concatenates them with the features of the same level. Training settings are the same as Q1 and Q2.
Q: For each task (classification and segmentation), report the test accuracy of your best model, in comparison with your results from Q1 & Q2.
| Test accuracy | PointNet (Q1, Q2) | PointNet++ |
|---|---|---|
| Classification | 0.977 | 0.976 |
| Segmentation | 0.903 | 0.913 |
Q: Visualize some results in comparison to ones obtained in the earlier questions
11 test objects are wrong with PointNet and correct with PointNet++, and 12 objects are the opposite. Confusion matrices (number of test objects; rows are ground truth, columns are predictions):
| Ground truth | PointNet | PointNet++ | ||||||
|---|---|---|---|---|---|---|---|---|
| chair | vase | lamp | class accuracy | chair | vase | lamp | class accuracy | |
| chair (617) | 617 | 0 | 0 | 1.000 | 615 | 0 | 2 | 0.997 |
| vase (102) | 1 | 92 | 9 | 0.902 | 1 | 88 | 13 | 0.863 |
| lamp (234) | 0 | 12 | 222 | 0.949 | 0 | 7 | 227 | 0.970 |
Examples where PointNet++ is better than PointNet:
![]() #870: GT lamp, PointNet vase, PointNet++ lamp |
![]() #667: GT vase, PointNet lamp, PointNet++ vase |
Opposite examples, where PointNet++ is worse than PointNet:
![]() #518: GT chair, PointNet chair, PointNet++ lamp |
![]() #832: GT lamp, PointNet lamp, PointNet++ vase |
Test accuracy of each part (all test points with that ground-truth label):
| Part | 0 other | 1 headrest | 2 back | 3 arm | 4 base | 5 seat |
|---|---|---|---|---|---|---|
| PointNet | 0.929 | 0.127 | 0.927 | 0.738 | 0.903 | 0.924 |
| PointNet++ | 0.900 | 0.166 | 0.932 | 0.802 | 0.923 | 0.921 |
The object accuracy of PointNet++ is higher than PointNet on 408 test objects and lower on 206. Left to right in each gif: ground truth, PointNet (Q2), PointNet++.
| Improved |
|---|
![]() #212 (largest gain): PointNet 0.766, PointNet++ 0.950 |
![]() #489 (second largest gain): PointNet 0.686, PointNet++ 0.860 |
![]() #315 (Q2 random sample): PointNet 0.916, PointNet++ 0.952 |
![]() #392 (Q2 random sample): PointNet 0.833, PointNet++ 0.864 |
| Not improved |
![]() #351 (Q2 bad sample): PointNet 0.475, PointNet++ 0.473 |
![]() #135 (largest loss): PointNet 0.864, PointNet++ 0.746 |
Interpretation: Each set abstraction level of PointNet++ runs a small PointNet on the points in a ball, with coordinates relative to the ball center. Thus each point gets features of the local shape around it, not only its position and one global feature.
Classification. Classification does not improve (0.977 and 0.976). The class of an object depends mostly on its global shape, which the global feature of PointNet already describes well. The main addition of PointNet++ is local receptive fields, which help to label each point but give little help to classify the whole object.
Segmentation. Segmentation improves by 1%, and most of the gain is on arm (0.738 to 0.802), a part that touches the seat and the back with no clear boundary and needs more fine-grained perception of geometry. The improved samples show that PointNet++ labels points more by the local shape and less by their coordinates. In #212, the seat is inside a bucket-shaped back; PointNet labels 1355 of the 3541 seat points and 568 of the 1249 arm points as back, because they are in the region of the back, and PointNet++ labels almost all of them correctly. These points are hidden behind the back in the render, so the difference is hard to see. #489 has no arms, but PointNet labels 1907 seat points at the sides as arm, because they are at the usual position of arms; PointNet++ sees that they are on the same surface as the seat and labels only 149 as arm.
PointNet++ still does not generalize to unusual shapes. #351 does not improve (0.475 to 0.473): its ground truth has no base and the whole shell is labeled arm, which is rare in the training data. #135 becomes worse (0.864 to 0.746): its seat is a curved surface above a round base, and PointNet++ labels most of the seat as base, because the local shape of the curved seat is similar to the base.
Same procedure as Q3, with the PointNet++ models. Mean test accuracy over the rotation angles:
| Rotation axis | Classification, 15°–30° | Classification, 15°–180° | Segmentation, 15°–30° | Segmentation, 15°–180° | ||||
|---|---|---|---|---|---|---|---|---|
| PointNet | PointNet++ | PointNet | PointNet++ | PointNet | PointNet++ | PointNet | PointNet++ | |
| x | 0.831 | 0.882 | 0.583 | 0.455 | 0.791 | 0.847 | 0.456 | 0.437 |
| y (up) | 0.929 | 0.932 | 0.609 | 0.636 | 0.827 | 0.884 | 0.650 | 0.802 |
| z | 0.822 | 0.927 | 0.449 | 0.527 | 0.750 | 0.844 | 0.505 | 0.537 |



Interpretation: PointNet++ is more robust to small pose changes, but less robust to fewer points.
Rotation. For both tasks, PointNet++ is better for small rotations (15°–30°) about all axes, because its local features use coordinates relative to the ball center. For segmentation, it is also better for all rotations about the up axis (y): the mean accuracy over 15°–180° is 0.802, PointNet 0.650. For classification, it is worse for large tilts about x.
Number of points. PointNet++ is much worse with few points: at 100 points, classification is 0.257 (PointNet 0.918) and segmentation is 0.608 (PointNet 0.819). The ball radius is fixed and the model saw only dense 10000-point objects in training, so with sparse input most balls contain only a few points. The PointNet++ paper makes the model adapt to point density with multi-scale grouping (MSG) or multi-resolution grouping (MRG), and with random input dropout in training (each training object loses a random 0–95% of its points). My implementation uses single-scale grouping and no input dropout, so it is sensitive to the number of points. These density-adaptive methods should decrease this problem.