Test accuracy: 97.90% (PointNet without transform blocks, all 10000 points, 953 test objects).
![]() #39 pred: chair true: chair | ![]() #256 pred: chair true: chair | ![]() #293 pred: chair true: chair | ![]() #485 pred: chair true: chair | ![]() #604 pred: chair true: chair | ![]() #806 pred: vase (wrong) true: lamp |
| chair 0 of 617 chairs wrong shown: other objects wrongly called chair | ![]() #869 pred: chair (wrong) true: lamp | |
| vase 9 of 102 vases wrong | ![]() #618 pred: lamp (wrong) true: vase | ![]() #651 pred: lamp (wrong) true: vase |
| lamp 11 of 234 lamps wrong | ![]() #721 pred: vase (wrong) true: lamp | ![]() #787 pred: vase (wrong) true: lamp |
Training: Adam, learning rate 0.001, batch size 32, 100 epochs, all 10000 points. train.py keeps the best epoch (epoch 93).
Class ids: 0 = chair, 1 = vase, 2 = lamp. Only 20 of 953 test objects are wrong. All 617 chairs are right. 9 vases are called lamps, 10 lamps are called vases, and 1 lamp is called a chair.
Why it fails: vases and lamps are both round, tall and often symmetric, so their global shapes are close. The failures are unusual shapes. Vase #618 is a closed cylinder with a lid, like a lantern. Vase #651 is a flower on a long curved stem, like a desk lamp arm with a head. Lamp #721 is a cylinder on thin rods, and lamp #787 is a round ball, so both look like vases. No chair is wrong, so the chair row shows lamp #869 instead: it has a flat rectangular frame, like a chair back. PointNet only uses one global max-pooled feature, so it does not look at small local parts that could fix these cases. There are also fewer vases (741) and lamps (1554) than chairs (4489) in training.
Test accuracy: 89.79% (fraction of correct points over all 617 test chairs, 10000 points each).
| object | ground truth | prediction |
|---|---|---|
| #294 good accuracy 94.72% | ![]() ground truth | ![]() prediction |
| #329 good accuracy 98.16% | ![]() ground truth | ![]() prediction |
| #537 good accuracy 95.71% | ![]() ground truth | ![]() prediction |
| #351 bad accuracy 45.00% | ![]() ground truth | ![]() prediction |
| #235 bad accuracy 49.56% | ![]() ground truth | ![]() prediction |
Training: Adam, learning rate 0.001, batch size 32, 100 epochs, all 10000 points. train.py keeps the best epoch (epoch 97).
Colors: white = class 0 and magenta = class 1 (small, rare parts; white is hard to see), cyan = back, yellow = arms, blue = legs or base, red = seat.
Most chairs are segmented well (95% to 98% on the good examples). The errors are mostly at part borders, for example where the seat meets the back or the legs.
The two bad objects are unusual chairs. #351 is a closed barrel chair: the ground truth calls its outer wall back and arms, but the model calls the lower half legs/base, because on normal chairs the low points are legs. #235 is a sofa where the ground truth mixes arm and seat labels on the same surface, and the model gives a cleaner but different split. The model relies a lot on the height of a point, so it fails when a part is not at its usual height or when the labels are not clear.
Procedure: rotate every test point cloud about the z axis (this tilts the object sideways) by 15, 30, 45, 60 and 90 degrees. Use the Q1 and Q2 models with no retraining, all 10000 points (eval_cls.py / eval_seg.py with --rot_deg).
| rotation (deg, about z) | classification acc | segmentation acc |
|---|---|---|
| 0 | 97.90% | 89.79% |
| 15 | 92.65% | 82.88% |
| 30 | 71.88% | 69.98% |
| 45 | 32.00% | 60.92% |
| 60 | 27.70% | 53.95% |
| 90 | 25.50% | 43.92% |
Classification (left column = Q1 result):
![]() 0 deg #39 pred: chair true: chair | ![]() 45 deg #39 pred: vase (wrong) true: chair | ![]() 90 deg #39 pred: vase (wrong) true: chair |
![]() 0 deg #256 pred: chair true: chair | ![]() 45 deg #256 pred: vase (wrong) true: chair | ![]() 90 deg #256 pred: vase (wrong) true: chair |
![]() 0 deg #293 pred: chair true: chair | ![]() 45 deg #293 pred: vase (wrong) true: chair | ![]() 90 deg #293 pred: vase (wrong) true: chair |
Segmentation (0 deg = Q2 result):
![]() ground truth | ![]() 0 deg pred accuracy 94.72% | ![]() 45 deg pred accuracy 58.28% | ![]() 90 deg pred accuracy 34.42% |
![]() ground truth | ![]() 0 deg pred accuracy 98.16% | ![]() 45 deg pred accuracy 64.26% | ![]() 90 deg pred accuracy 51.58% |
Both models are not rotation invariant. Accuracy already drops at 15 degrees and falls fast after that. At 45 degrees, classification falls from 97.9% to 32.0%, and segmentation from 89.8% to 60.9%.
The models were trained only on upright objects, with no rotation augmentation and no transform block, so they learn features in a fixed frame. A tilted chair looks like a vase to the classifier (at 90 degrees, 724 of 953 objects are called vase). The segmentation model still labels points by their old height, so after tilting, the labels follow the wrong axis.
Procedure: keep only N random points per object (the same random subset for all objects, seed 0), N = 5000, 2000, 1000, 500, 100, 50, 10. Use the Q1 and Q2 models with no retraining (eval_cls.py / eval_seg.py with --num_points).
| points per object | classification acc | segmentation acc |
|---|---|---|
| 10000 | 97.90% | 89.79% |
| 5000 | 97.69% | 89.86% |
| 2000 | 97.69% | 89.85% |
| 1000 | 97.38% | 89.63% |
| 500 | 96.75% | 88.83% |
| 100 | 89.93% | 81.78% |
| 50 | 67.47% | 77.69% |
| 10 | 27.07% | 66.66% |
Classification (left column = Q1 result):
![]() 10000 points #39 pred: chair true: chair | ![]() 100 points #39 pred: chair true: chair | ![]() 50 points #39 pred: chair true: chair |
![]() 10000 points #256 pred: chair true: chair | ![]() 100 points #256 pred: chair true: chair | ![]() 50 points #256 pred: chair true: chair |
![]() 10000 points #293 pred: chair true: chair | ![]() 100 points #293 pred: chair true: chair | ![]() 50 points #293 pred: lamp (wrong) true: chair |
Segmentation (10000 points = Q2 result):
![]() 10000 points ground truth | ![]() 10000 points pred accuracy 94.72% | ![]() 100 points ground truth | ![]() 100 points pred accuracy 90.00% | ![]() 50 points ground truth | ![]() 50 points pred accuracy 80.00% |
![]() 10000 points ground truth | ![]() 10000 points pred accuracy 98.16% | ![]() 100 points ground truth | ![]() 100 points pred accuracy 87.00% | ![]() 50 points ground truth | ![]() 50 points pred accuracy 88.00% |
The models are very robust to fewer points. With 500 points (5% of the input), classification is 96.7% and segmentation is 88.8%, almost the same as with 10000 points. Accuracy drops clearly only below 100 points.
PointNet uses a max pool, so the global feature only depends on a small set of key points that cover the shape. A random subset still covers the shape well. With 50 or 10 points the shape is lost and the classifier mostly says lamp. Segmentation stays at 67% even with 10 points, because the label of a point depends a lot on its height.
Model: PointNet++ (single-scale grouping), in models.py (cls_pointnet2_model, seg_pointnet2_model). It adds locality by learning features of small neighborhoods:
Set abstraction 1: farthest point sampling picks 512 centers, a ball query (radius 0.2) takes 32 neighbors per center, and a shared MLP (64, 64, 128) with max pool encodes each neighborhood in local coordinates. Set abstraction 2: 128 centers, radius 0.4, 64 neighbors, MLP (128, 128, 256). Set abstraction 3: one global MLP (256, 512, 1024) and max pool.
Classification: the global feature goes through FC layers 512, 256, 3 with dropout. Segmentation: three feature propagation layers bring the features back to all 10000 points (inverse-distance interpolation from the 3 nearest centers, skip links, MLPs), then a per-point classifier.
Training is the same as Q1 and Q2 (Adam, 0.001, batch 32, 100 epochs, all 10000 points). Best epochs: 42 (classification) and 93 (segmentation). Sampling and grouping use pytorch3d ops.
| model | classification acc | segmentation acc |
|---|---|---|
| PointNet (Q1, Q2) | 97.90% | 89.79% |
| PointNet++ (Q4) | 97.48% | 90.45% |
Classification: the Q1 failure cases again.
| object | PointNet | PointNet++ |
|---|---|---|
| #869 true: lamp | ![]() PointNet: chair (wrong) | PointNet++: vase (wrong) |
| #618 true: vase | ![]() PointNet: lamp (wrong) | PointNet++: lamp (wrong) |
| #651 true: vase | ![]() PointNet: lamp (wrong) | PointNet++: vase (right) |
| #721 true: lamp | ![]() PointNet: vase (wrong) | PointNet++: lamp (right) |
| #787 true: lamp | ![]() PointNet: vase (wrong) | PointNet++: vase (wrong) |
Segmentation: the Q2 objects again.
| object | ground truth | PointNet | PointNet++ |
|---|---|---|---|
| #294 | ![]() ground truth | ![]() PointNet accuracy 94.72% | ![]() PointNet++ accuracy 97.01% |
| #329 | ![]() ground truth | ![]() PointNet accuracy 98.16% | ![]() PointNet++ accuracy 98.04% |
| #537 | ![]() ground truth | ![]() PointNet accuracy 95.71% | ![]() PointNet++ accuracy 98.24% |
| #351 | ![]() ground truth | ![]() PointNet accuracy 45.00% | ![]() PointNet++ accuracy 47.11% |
| #235 | ![]() ground truth | ![]() PointNet accuracy 49.56% | ![]() PointNet++ accuracy 49.54% |
Segmentation gets better: 90.45% vs 89.79%. Part borders are sharper and thin parts are cleaner (for example #537 goes from 95.7% to 98.2%), because each point now sees its local neighborhood. The two bad chairs stay bad, because their problem is the unusual shape and unclear labels, not missing detail.
Classification stays about the same: 97.48% vs 97.90% (4 more wrong objects out of 953). PointNet++ fixes 11 of the 20 PointNet failures, but makes new mistakes, mostly vases called lamps. Local features do not help much for this easy 3-class task.
Note: all numbers above use the same eval scripts, which shuffle the point order. The ball query takes the first 32 points inside the ball, so PointNet++ depends a little on the point order. With the original point order it gets 98.53% (classification) and 91.76% (segmentation). PointNet does not depend on the order.