Everything was trained on my RTX 5090.
Test accuracy = 97.69%
My model is PointNet without the transformation blocks. Every point goes through the same MLP (3 to 64 to 64 to 64 to 128 to 1024, written as 1x1 convolutions with batch norm and ReLU), a max pool over all the points gives one 1024-d feature for the whole object, and 3 fully connected layers (1024 to 512 to 256 to 3) give the class scores. I trained with the default settings (250 epochs, batch size 32, learning rate 0.001), and train.py kept the checkpoint from epoch 191.
| True class | Predicted chair | Predicted vase | Predicted lamp | Accuracy |
|---|---|---|---|---|
| Chair | 617 | 0 | 0 | 100% |
| Vase | 1 | 91 | 10 | 89.2% |
| Lamp | 1 | 10 | 223 | 95.3% |
2 random test objects from each class (most of the test set is chairs, so I picked them per class).
Chairs

Left: predicted chair. Right: a long lounge chair, also predicted chair.
Vases

Left: predicted vase. Right: predicted lamp, which is wrong. It's a shallow dish with a stick coming out of it, so it looks like the base and arm of a desk lamp.
Lamps

Both predicted lamp.
Chair, predicted lamp (from the epoch 190 checkpoint)

My best model got all 617 test chairs right, so I took this failure from the checkpoint saved at epoch 190 (96.75% test accuracy), which called this chair a lamp (79% lamp). It's a tall, closed box with flat sides and no separate legs, seat or back, so it looks more like a tall column lamp than a chair. The best model got it right but was still unsure (59% chair, 41% lamp), and it's the least confident chair in the whole test set.
Vase, predicted lamp

The left one is a wide short cylinder that looks just like a drum lamp shade, and the right one is a flat dish with a branch sticking up, which looks like a lamp base with an arm. The model was 100% sure both were lamps.
Lamp, predicted vase

Both of these have a round, bulging base under the shade, and that base looks like the body of a vase (both were about 67% vase and 33% lamp, so the model wasn't very sure).
Almost all of the mistakes are between vases and lamps, since both are mostly round and symmetric, and the model only sees the overall shape through one max-pooled feature. Chairs never get confused because their legs and backs look very different. Vases are also the smallest class (741 training objects vs 4489 chairs and 1554 lamps), which is probably why they have the lowest accuracy.
Test accuracy = 90.07% (fraction of all test points labeled correctly)
The segmentation model uses the same shared MLP, but for every point it concatenates the point's own 64-d feature from the first layers with the 1024-d max-pooled feature of the whole chair, and then a per point MLP (1088 to 512 to 256 to 128 to 6) gives the part scores. I trained with the default settings (250 epochs, batch size 32, learning rate 0.001), and train.py kept the checkpoint from epoch 164. The colors are cyan for the back, red for the seat, blue for the legs and base, yellow for the arms and magenta for the small part on top of some backs.
Object 459 (random): accuracy 97.8%

Object 85 (random): accuracy 86.4%

Object 323 (random): accuracy 73.9%

Object 4 (random): accuracy 69.3%

Object 235 (bad prediction, the worst object): accuracy 44.7%

Object 26 (bad prediction, the second worst object): accuracy 46.7%

Ground truth is on the left and the prediction is on the right. The 4 random objects were picked randomly and the 2 bad ones are the lowest accuracy objects in the test set.
Simple chairs like 459 come out almost perfect, since the back, seat and legs are clearly separate pieces. The problems are on sofas and armchairs, where the seat, arms and back are one soft connected shape and the boundaries between them are not obvious. In 323 and 4 the model puts the boundary between the seat and the back or arms in the wrong place. The two worst objects are sofas whose labels are pretty unusual (most of 235 is labeled as arms), and the model predicts a more typical layout instead, with a seat in the middle and legs at the bottom. It also misses the small magenta part on top of 26, which is only 0.1% of the test points. The edges between parts are also noisy because PointNet labels each point from its own position and one global feature, without looking at its neighbors. Over the test set the median object accuracy is 93.5%, so most chairs are segmented well and the average is pulled down by the harder sofas.
For both experiments I used the same trained models from Q1 and Q2 and only changed the test inputs (I added --rotate options to eval_cls.py and eval_seg.py, and robustness.py runs all the settings). In the data y is up and the objects are centered at the origin.
I rotated every test point cloud around the x axis (tilting the object forward) and around the y axis (spinning it around the up direction) by 0 to 180 degrees.
| Rotation (degrees) | 0 (Q1/Q2) | 15 | 30 | 45 | 60 | 90 | 135 | 180 |
|---|---|---|---|---|---|---|---|---|
| Classification, tilt | 97.7% | 94.5% | 89.0% | 63.5% | 38.0% | 28.0% | 64.5% | 76.2% |
| Classification, spin | 97.7% | 96.9% | 90.6% | 76.6% | 61.9% | 69.9% | 31.3% | 35.6% |
| Segmentation, tilt | 90.1% | 83.7% | 73.7% | 63.8% | 42.0% | 25.7% | 24.8% | 34.6% |
| Segmentation, spin | 90.1% | 85.7% | 76.1% | 71.1% | 67.8% | 56.3% | 51.6% | 57.0% |

Classification (chair 459, vase 637, lamp 871 from Q1, which were all predicted correctly there)
Original (Q1): chair, vase, lamp

Tilted 45 degrees: predicted lamp, vase, lamp

Spun 90 degrees: predicted vase, vase, lamp

Segmentation (objects 459 and 85 from Q2, ground truth then prediction)
Tilted 45 degrees: object 459 accuracy 65.1% (97.8% in Q2), object 85 accuracy 58.4% (86.4% in Q2)

Spun 90 degrees: object 459 accuracy 84.6%, object 85 accuracy 42.3%

Both models get a lot worse once the object is rotated more than about 30 degrees, since they were only trained on upright objects facing the same way and PointNet works directly on the xyz coordinates (there's no transformation block or rotation augmentation). Spinning around the up axis barely matters for vases and lamps because they're round (85 to 93% at 90 and 180 degrees), but chairs drop to 59% at 90 degrees and 6% at 180 degrees, so the model learned that chairs face one direction. Segmentation falls apart the most when tilted (26% at 90 degrees) because the parts are mostly told apart by height, legs at the bottom and back at the top, and tilting moves them to different heights. Training with random rotations would probably fix most of this.
I evaluated with --num_points from 10,000 (the Q1/Q2 setting) down to 10, which randomly keeps that many of the points of every object.
| Points per object | 10000 (Q1/Q2) | 5000 | 2000 | 1000 | 500 | 200 | 100 | 50 | 20 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Classification | 97.7% | 97.4% | 97.5% | 97.2% | 96.6% | 95.1% | 92.0% | 78.5% | 48.3% | 30.4% |
| Segmentation | 90.1% | 90.1% | 90.0% | 89.5% | 88.4% | 85.6% | 83.1% | 79.9% | 73.2% | 66.5% |

Classification (same chair, vase and lamp as above)
100 points: predicted vase, vase, lamp

20 points: predicted lamp, lamp, lamp

Segmentation (objects 459 and 85, ground truth then prediction)
100 points: object 459 accuracy 93.0% (97.8% in Q2), object 85 accuracy 63.0% (86.4% in Q2)

20 points: object 459 accuracy 80.0%, object 85 accuracy 65.0%

Both models barely change down to about 500 points, because the max pool only needs a few points on each part of the object to get almost the same global feature. Below 100 points classification drops quickly, and with 20 points it calls almost everything a lamp (every lamp is right but none of the vases are), since a handful of scattered points looks like the thin parts of a lamp. Segmentation goes down much more slowly (73% with 20 points), probably because each point's label mostly depends on its own height in the chair, which doesn't change when the other points are removed.
I implemented PointNet++ with single scale grouping. Instead of one max pool over the whole object, it runs a small PointNet on local neighborhoods and builds them up in layers. Each set abstraction layer picks centers with farthest point sampling, takes up to K points within a radius of each center (ball query), runs a shared MLP on their coordinates relative to the center and max pools inside each group. I used 3 layers: 512 centers with radius 0.2 and K = 32, 128 centers with radius 0.4 and K = 64, and then one group with everything, which gives a 1024-d feature for the object. For classification, fully connected layers (1024 to 512 to 256 to 3) give the class scores. For segmentation, 3 feature propagation layers bring the features back to all 10,000 points (each point takes the inverse distance weighted average of its 3 nearest centers and concatenates the features from the matching earlier layer), and a per point layer gives the 6 part scores. I used the pytorch3d ops for the sampling, ball query and nearest neighbors, and trained both with the same default settings as Q1 and Q2. train.py kept the checkpoints from epoch 170 (classification) and 180 (segmentation).
Classification test accuracy = 98.85% (PointNet in Q1: 97.69%)
Segmentation test accuracy = 91.80% (PointNet in Q2: 90.07%)
| PointNet (Q1/Q2) | PointNet++ | |
|---|---|---|
| Classification | 97.69% | 98.85% |
| Chairs | 100% (617/617) | 99.8% (616/617) |
| Vases | 89.2% (91/102) | 94.1% (96/102) |
| Lamps | 95.3% (223/234) | 98.3% (230/234) |
| Segmentation | 90.07% | 91.80% |
| Segmentation, median object | 93.5% | 95.2% |
One thing I noticed is that PointNet++ depends a little on the order of the points. The ball query keeps the first K points it finds in each ball, and the points in the data files are stored nicely spread out, so in file order those points cover the ball evenly. eval_cls.py and eval_seg.py shuffle the points, and with shuffled points PointNet++ gets 98.11% and 90.64%. The numbers above use the file order (I added --keep_order to the eval scripts), which is also what train.py used to pick the best model. PointNet gets the same accuracy either way.
Q1 failures (vases 618 and 651, lamps 726 and 812)

PointNet predicted lamp, lamp, vase, vase. PointNet++ gets all four right (vase, vase, lamp, lamp), and also vase 655 and chair 77 from Q1. Their overall shapes look like the other class, so looking at local details probably helps.
PointNet++ failures (chair 406 predicted lamp, vase 670 predicted lamp, lamp 787 predicted vase)

PointNet got 406 and 787 right and 670 wrong. 406 is a chair that is almost completely flat, 670 is a thin vase full of sticks that look like the arms of a lamp, and 787 is a lamp with a big round ball for a base, which looks like the body of a vase.
| Object | 459 | 85 | 323 | 4 | 235 | 26 |
|---|---|---|---|---|---|---|
| PointNet (Q2) | 97.8% | 86.4% | 73.9% | 69.3% | 44.7% | 46.7% |
| PointNet++ | 98.8% | 75.6% | 87.4% | 72.1% | 49.8% | 64.7% |
Ground truth, then PointNet (Q2), then PointNet++.
Object 323: PointNet 73.9%, PointNet++ 87.4%

Object 26: PointNet 46.7%, PointNet++ 64.7%

Object 85: PointNet 86.4%, PointNet++ 75.6%

On 323 PointNet labels the lower front of the seat as legs, and on 26 it mixes up the seat, arms and legs. PointNet++ gets these mostly right (the arms of 26 go from 34% to 78% correct), probably because each point's label now also depends on the shape around it and not just its height. It's not always better though: on 85 it gets the arms and legs right but puts patches of legs and arms inside the big seat, so the accuracy goes down, and it still misses the small magenta part on top of 26. Overall it improves 5 of these 6 objects and the median object accuracy goes from 93.5% to 95.2%.