Assignment 5: Point Cloud Classification and Segmentation
16-825: Learning for 3D Vision · Name: Zixin Guo · Andrew ID: zixinguo
Q1 and Q2 train a PointNet (without the T-Nets) to classify point clouds as chair / vase / lamp and to segment chairs into six parts. Q3 checks how these two models react to rotated inputs and to inputs with fewer points. Q4 swaps the PointNet encoder for a DGCNN encoder, which looks at local neighborhoods.
Both tasks use one design (models.py): an encoder returns a feature for every point and one global feature for the cloud, and a small head turns them into class scores. All models were trained on one A40 with the handout defaults: Adam, lr 1e-3, batch 32, no augmentation. As in the provided train.py, the checkpoint with the best test accuracy is kept.
Q1. Classification Model
Model. Every point goes through the same MLP (3 → 64 → 64 → 64 → 128 → 1024, BatchNorm and ReLU after each layer). A max over all points gives a 1024-d global feature, and a second MLP (1024 → 512 → 256 → 3, dropout 0.3) outputs the class scores. Trained for 250 epochs on all 10000 points per object.
Test accuracy: 97.80 % (932 of 953 objects, python eval_cls.py).
| class | test objects | correct | accuracy |
|---|---|---|---|
| chair | 617 | 617 | 1.000 |
| vase | 102 | 94 | 0.922 |
| lamp | 234 | 221 | 0.944 |

Six random test clouds. The text in each cell is the object index, the ground truth and the prediction. All six are correct.

Failure cases, one for every kind of confusion that occurs. Red text marks a wrong prediction.
- Chair → lamp (#406). The best checkpoint gets all 617 test chairs right, so this example comes from the last saved checkpoint (epoch 240, 96.4 %). It is a narrow armchair with a tall, round-topped back and thick posts for legs. As an outline that is a tall body with a dome on top, closer to a lantern than to the wide, low chairs that fill the training set. It is a borderline case: 10 of the last 14 saved checkpoints get it wrong.
- Vase → lamp (#651). A bowl with one flower on a long bent stem. A thin arm with a blob at the end, standing on a base, is what a desk lamp looks like.
- Vase → chair (#671). A low, wide planter with small plants sticking up. A wide flat body with a few things rising from it reads as a seat with a back.
- Lamp → vase (#750). A lantern: a closed round body on a short stand, with no pole and no shade. Geometrically it is a container.
- Lamp → chair (#869). A wall lamp with a flat rectangular mounting plate and a long body below it. A flat plate attached to a box is the back-plus-seat layout of a chair.

All 21 failures of the best checkpoint.
The global feature is a max over points, so it describes the overall outline of the object and not its details. Chairs have an outline no other class has (legs, seat, back) and are two thirds of the training set, so they are easy. Vases and lamps are both roughly symmetric around the vertical axis and differ more in purpose than in shape. Most of the 13 misclassified lamps have no pole (ceiling domes, lanterns, plain cylinders), and most of the 8 misclassified vases are tall thin ones or have plants in them.
Q2. Segmentation Model
Model. The same encoder as in Q1. The 64-d feature of every point (after the first two layers) is concatenated with the 1024-d global feature, and a shared MLP (1088 → 512 → 256 → 128 → 6) classifies each point. Trained for 250 epochs on all 10000 points per chair.
Test accuracy: 90.10 % over all points of the 617 test chairs (python eval_seg.py).
Colors: cyan = back, red = seat, blue = leg, yellow = armrest, magenta = headrest, gray = footrest (white in the handout code; I changed it because white does not show on a white background).

Top: ground truth. Bottom: prediction with the accuracy of that object. From left to right: the objects at the 90 %, 50 % and 10 % quantile of per-object accuracy, then the two worst objects of the test set.
| object | accuracy | what happens |
|---|---|---|
| #490 | 0.981 | Plain chair. The only errors are a thin band where the seat meets the back and the legs. |
| #187 | 0.932 | The median chair. The bottom of the back and the feet of the armrests are labeled seat. |
| #572 | 0.778 | The arm supports run down to the floor. The ground truth calls the lower half leg; the model calls the whole side panel armrest. |
| #235 | 0.468 | A box-shaped club chair. The ground truth labels almost everything armrest. The model predicts the usual layout: legs at the bottom, seat in the middle, back on top, arms on the sides. |
| #61 | 0.461 | A wide, sofa-like chair with low arms. The model labels the whole body seat, misses the armrests, and calls the top of the back a headrest. |
Two thirds of the test chairs are above 90 % and only 4 % are below 70 %. The model works well on ordinary chairs and fails on unusual ones. The reason is that each point is labeled from just two things: its own coordinates and the global shape feature. Nothing tells a point what its neighbors look like. So the model learns where each part usually sits inside the chair (“low points are legs, the top is the back”), which is right for a standard chair and wrong for a sofa or a box. In #235 and #572 the ground truth is also debatable, since where the armrest ends and the leg begins is a labeling choice.
Per part, the share of points that is labeled correctly is 0.93 for seat, 0.92 for back, 0.92 for footrest, 0.88 for leg, 0.79 for armrest and 0.18 for headrest. Armrests are thin and touch three other parts. Headrests appear in only 8 test chairs (0.1 % of all points) and are just the top of the back in terms of shape, so the model almost never predicts them.
Q3. Robustness Analysis
Both experiments use the Q1 and Q2 checkpoints as they are. Only the test input changes. python analysis.py q3 runs both; a single setting can be run with the eval scripts, e.g. python eval_cls.py --rotate 30 or python eval_seg.py --num_points 1000.
3.1 Rotation
Procedure. Every test cloud is rotated around the origin by the same angle, from 0° to 180° in steps of 15°, about one coordinate axis. Rotating about z or x tips the object over; y is the vertical axis, so rotating about y turns the object on the spot.
| rotation about z | 0° (Q1, Q2) | 15° | 30° | 45° | 60° | 90° | 180° |
|---|---|---|---|---|---|---|---|
| classification | 0.978 | 0.940 | 0.726 | 0.444 | 0.326 | 0.302 | 0.645 |
| segmentation | 0.901 | 0.827 | 0.698 | 0.597 | 0.532 | 0.412 | 0.313 |

Test accuracy against the rotation angle for the three axes.

Classification of a chair, a vase and a lamp rotated about z. The text is the prediction; red means wrong.

Predicted segmentation of three chairs rotated about z, with the accuracy of each.
Interpretation. The model was trained without rotation augmentation and has no T-Net, and every training object stands upright and faces the same way. The network sees raw xyz, so what it learned is tied to that pose. A 15° tilt costs little (4 points for classification, 7 for segmentation), 30° costs a lot, and from 45° on both models are broken.
- Classification. Chairs break first. About z, the share of chairs still called chair goes 1.00 → 0.76 (30°) → 0.33 (45°) → 0.11 (90°) → 0.00 (105°–135°), while vases and lamps stay near 0.6–0.7. A chair is recognized by “legs below, back above”; a tilted chair no longer has that and is called a vase or a lamp. Since 65 % of the test set is chairs, the accuracy drops far below what always answering “chair” would give.
- Turning about the vertical axis does not bother vases and lamps (about 0.9 at every angle, because they are nearly symmetric around that axis), but chairs facing backwards are almost never recognized (0.06 at 180°). The model only knows chairs that face one way.
- The curves are not monotonic. Upside-down chairs (180° about z) are recognized again 62 % of the time, and chairs rotated 120°–165° about x 80–90 % of the time. The model never saw such inputs, so its output there is an accident of the learned features, not understanding.
- Segmentation. The labels follow the position in space, not the part. At 180° the back hangs down and is painted as leg, and the legs point up and are painted as seat and armrest: only 3 % of leg points and 9 % of back points are still correct. About y, seat and leg mostly survive (0.82 and 0.86 at 180°) because they are defined by height, which this rotation keeps, while the back drops to 0.2.
3.2 Number of points
Procedure. Only a random subset of n points per object is given to the model, for n from 10000 down to 50. The smaller subsets are contained in the larger ones. The models were trained with 10000 points.
| points per object | 10000 (Q1, Q2) | 5000 | 2000 | 1000 | 500 | 200 | 100 | 50 |
|---|---|---|---|---|---|---|---|---|
| classification | 0.978 | 0.976 | 0.974 | 0.971 | 0.972 | 0.962 | 0.923 | 0.842 |
| segmentation | 0.901 | 0.901 | 0.901 | 0.901 | 0.892 | 0.856 | 0.821 | 0.772 |

Test accuracy against the number of points per object (log scale).

Classification of the same three objects with fewer points. With 50 points the vase is called a chair.

Predicted segmentation of the same three chairs with fewer points. These three are easy chairs and stay near 0.9 even with 50 points; the test average at 50 points is 0.77.
Interpretation. Both models barely notice. With 1000 points, a tenth of the input, classification loses 0.7 points and segmentation nothing. The global feature is a max over points, so each of its 1024 values is set by a single point. A random subset almost always contains a point close to that one, and the max hardly changes. The per-point branch never depended on the other points in the first place.
Below 200 points the outline gets patchy and the errors are uneven. In classification nearly all of the loss is vases: with 50 points only 17 % of vases are recognized, against 91 % of chairs and 96 % of lamps, because a few scattered points of a vase no longer show a closed body. In segmentation the small parts go first (armrest 0.79 → 0.50, footrest 0.92 → 0.00 at 50 points), while leg and back stay above 0.8.
Q4. Bonus Question - Locality
Model: DGCNN. PointNet processes every point alone and only mixes information in one global max pool. DGCNN replaces the per-point MLP with EdgeConv layers. An EdgeConv layer finds the k = 20 nearest neighbors of every point, builds one feature per neighbor from the point and the offset to the neighbor, [x_j − x_i, x_i], runs a shared MLP on it, and takes the max over the neighbors. I use four layers (64, 64, 128, 256 channels). The neighbors of the first layer are found in xyz; later layers search in the feature space of the previous layer, so the graph changes from layer to layer. The four outputs are concatenated into a 512-d point feature, which is lifted to 1024-d and max-pooled into the global feature. The heads are the ones from Q1 and Q2: only the encoder is swapped (--arch dgcnn).
Training. The neighbor search needs all pairwise distances, so I train with 2000 random points per object (a new subset for every batch) instead of 10000, for 100 epochs. Everything else is the same as before. The models are tested with 2000 points per object, and PointNet is tested on exactly the same points for a fair comparison.
| PointNet, 10000 points (Q1, Q2) | PointNet, 2000 points | DGCNN, 2000 points | |
|---|---|---|---|
| classification | 0.978 | 0.974 | 0.986 |
| segmentation | 0.901 | 0.901 | 0.916 |

Objects on which the two classifiers disagree. Black: DGCNN is right (4 of 18 such objects). Red: DGCNN is wrong (2 of 8).

Top: ground truth. Middle: PointNet. Bottom: DGCNN. The first three chairs are the ones DGCNN improves most, the last two the ones it hurts most.
Interpretation.
- Classification. On the same clouds the errors go from 25 to 13 out of 953. DGCNN fixes 13 lamps that PointNet called vase or chair and 4 vases with plants or sticks in them that PointNet called lamp. These are the objects whose outline is ambiguous (Q1). Local neighborhoods add what the outline lacks, for example whether a thin part is a straight pole or a branching stem.
- Segmentation. 71 % of the chairs get better and the median object accuracy goes from 0.93 to 0.95. The gain is largest for armrests (0.77 → 0.81 of the points correct) and backs (0.92 → 0.94). With neighborhood features a point knows what kind of surface it lies on: DGCNN finds the armrests of the box-shaped chair #180, which PointNet labels seat, and mostly stops calling the base of #426 a footrest. Headrests are still not learned (0.11).
- Where it is worse. On #77, a chair that is little more than a frame, DGCNN labels the upper part of the legs armrest. On the rocking lounge chair #292 it invents a headrest and a footrest. Local shape alone can be misleading when the object is unusual.
- Point density. DGCNN is tied to the density it was trained with. Given all 10000 points, its accuracy falls to 0.943 and 0.875, below PointNet, because the 20 nearest neighbors then cover a much smaller patch than in training. PointNet has no such problem (Q3.2).