Assignment 5: Point Cloud Classification and Segmentation

16-825: Learning for 3D Vision · Name: Zixin Guo · Andrew ID: zixinguo

Q1 and Q2 train a PointNet (without the T-Nets) to classify point clouds as chair / vase / lamp and to segment chairs into six parts. Q3 checks how these two models react to rotated inputs and to inputs with fewer points. Q4 swaps the PointNet encoder for a DGCNN encoder, which looks at local neighborhoods.

Both tasks use one design (models.py): an encoder returns a feature for every point and one global feature for the cloud, and a small head turns them into class scores. All models were trained on one A40 with the handout defaults: Adam, lr 1e-3, batch 32, no augmentation. As in the provided train.py, the checkpoint with the best test accuracy is kept.


Q1. Classification Model

Model. Every point goes through the same MLP (3 → 64 → 64 → 64 → 128 → 1024, BatchNorm and ReLU after each layer). A max over all points gives a 1024-d global feature, and a second MLP (1024 → 512 → 256 → 3, dropout 0.3) outputs the class scores. Trained for 250 epochs on all 10000 points per object.

Test accuracy: 97.80 % (932 of 953 objects, python eval_cls.py).

class test objects correct accuracy
chair 617 617 1.000
vase 102 94 0.922
lamp 234 221 0.944

random test clouds

Six random test clouds. The text in each cell is the object index, the ground truth and the prediction. All six are correct.

failure cases

Failure cases, one for every kind of confusion that occurs. Red text marks a wrong prediction.

all failures

All 21 failures of the best checkpoint.

The global feature is a max over points, so it describes the overall outline of the object and not its details. Chairs have an outline no other class has (legs, seat, back) and are two thirds of the training set, so they are easy. Vases and lamps are both roughly symmetric around the vertical axis and differ more in purpose than in shape. Most of the 13 misclassified lamps have no pole (ceiling domes, lanterns, plain cylinders), and most of the 8 misclassified vases are tall thin ones or have plants in them.

Q2. Segmentation Model

Model. The same encoder as in Q1. The 64-d feature of every point (after the first two layers) is concatenated with the 1024-d global feature, and a shared MLP (1088 → 512 → 256 → 128 → 6) classifies each point. Trained for 250 epochs on all 10000 points per chair.

Test accuracy: 90.10 % over all points of the 617 test chairs (python eval_seg.py).

Colors: cyan = back, red = seat, blue = leg, yellow = armrest, magenta = headrest, gray = footrest (white in the handout code; I changed it because white does not show on a white background).

segmentation results

Top: ground truth. Bottom: prediction with the accuracy of that object. From left to right: the objects at the 90 %, 50 % and 10 % quantile of per-object accuracy, then the two worst objects of the test set.

object accuracy what happens
#490 0.981 Plain chair. The only errors are a thin band where the seat meets the back and the legs.
#187 0.932 The median chair. The bottom of the back and the feet of the armrests are labeled seat.
#572 0.778 The arm supports run down to the floor. The ground truth calls the lower half leg; the model calls the whole side panel armrest.
#235 0.468 A box-shaped club chair. The ground truth labels almost everything armrest. The model predicts the usual layout: legs at the bottom, seat in the middle, back on top, arms on the sides.
#61 0.461 A wide, sofa-like chair with low arms. The model labels the whole body seat, misses the armrests, and calls the top of the back a headrest.

Two thirds of the test chairs are above 90 % and only 4 % are below 70 %. The model works well on ordinary chairs and fails on unusual ones. The reason is that each point is labeled from just two things: its own coordinates and the global shape feature. Nothing tells a point what its neighbors look like. So the model learns where each part usually sits inside the chair (“low points are legs, the top is the back”), which is right for a standard chair and wrong for a sofa or a box. In #235 and #572 the ground truth is also debatable, since where the armrest ends and the leg begins is a labeling choice.

Per part, the share of points that is labeled correctly is 0.93 for seat, 0.92 for back, 0.92 for footrest, 0.88 for leg, 0.79 for armrest and 0.18 for headrest. Armrests are thin and touch three other parts. Headrests appear in only 8 test chairs (0.1 % of all points) and are just the top of the back in terms of shape, so the model almost never predicts them.

Q3. Robustness Analysis

Both experiments use the Q1 and Q2 checkpoints as they are. Only the test input changes. python analysis.py q3 runs both; a single setting can be run with the eval scripts, e.g. python eval_cls.py --rotate 30 or python eval_seg.py --num_points 1000.

3.1 Rotation

Procedure. Every test cloud is rotated around the origin by the same angle, from 0° to 180° in steps of 15°, about one coordinate axis. Rotating about z or x tips the object over; y is the vertical axis, so rotating about y turns the object on the spot.

rotation about z 0° (Q1, Q2) 15° 30° 45° 60° 90° 180°
classification 0.978 0.940 0.726 0.444 0.326 0.302 0.645
segmentation 0.901 0.827 0.698 0.597 0.532 0.412 0.313

accuracy vs rotation

Test accuracy against the rotation angle for the three axes.

classification under rotation

Classification of a chair, a vase and a lamp rotated about z. The text is the prediction; red means wrong.

segmentation under rotation

Predicted segmentation of three chairs rotated about z, with the accuracy of each.

Interpretation. The model was trained without rotation augmentation and has no T-Net, and every training object stands upright and faces the same way. The network sees raw xyz, so what it learned is tied to that pose. A 15° tilt costs little (4 points for classification, 7 for segmentation), 30° costs a lot, and from 45° on both models are broken.

3.2 Number of points

Procedure. Only a random subset of n points per object is given to the model, for n from 10000 down to 50. The smaller subsets are contained in the larger ones. The models were trained with 10000 points.

points per object 10000 (Q1, Q2) 5000 2000 1000 500 200 100 50
classification 0.978 0.976 0.974 0.971 0.972 0.962 0.923 0.842
segmentation 0.901 0.901 0.901 0.901 0.892 0.856 0.821 0.772

accuracy vs number of points

Test accuracy against the number of points per object (log scale).

classification with fewer points

Classification of the same three objects with fewer points. With 50 points the vase is called a chair.

segmentation with fewer points

Predicted segmentation of the same three chairs with fewer points. These three are easy chairs and stay near 0.9 even with 50 points; the test average at 50 points is 0.77.

Interpretation. Both models barely notice. With 1000 points, a tenth of the input, classification loses 0.7 points and segmentation nothing. The global feature is a max over points, so each of its 1024 values is set by a single point. A random subset almost always contains a point close to that one, and the max hardly changes. The per-point branch never depended on the other points in the first place.

Below 200 points the outline gets patchy and the errors are uneven. In classification nearly all of the loss is vases: with 50 points only 17 % of vases are recognized, against 91 % of chairs and 96 % of lamps, because a few scattered points of a vase no longer show a closed body. In segmentation the small parts go first (armrest 0.79 → 0.50, footrest 0.92 → 0.00 at 50 points), while leg and back stay above 0.8.

Q4. Bonus Question - Locality

Model: DGCNN. PointNet processes every point alone and only mixes information in one global max pool. DGCNN replaces the per-point MLP with EdgeConv layers. An EdgeConv layer finds the k = 20 nearest neighbors of every point, builds one feature per neighbor from the point and the offset to the neighbor, [x_j − x_i, x_i], runs a shared MLP on it, and takes the max over the neighbors. I use four layers (64, 64, 128, 256 channels). The neighbors of the first layer are found in xyz; later layers search in the feature space of the previous layer, so the graph changes from layer to layer. The four outputs are concatenated into a 512-d point feature, which is lifted to 1024-d and max-pooled into the global feature. The heads are the ones from Q1 and Q2: only the encoder is swapped (--arch dgcnn).

Training. The neighbor search needs all pairwise distances, so I train with 2000 random points per object (a new subset for every batch) instead of 10000, for 100 epochs. Everything else is the same as before. The models are tested with 2000 points per object, and PointNet is tested on exactly the same points for a fair comparison.

PointNet, 10000 points (Q1, Q2) PointNet, 2000 points DGCNN, 2000 points
classification 0.978 0.974 0.986
segmentation 0.901 0.901 0.916

classification, PointNet vs DGCNN

Objects on which the two classifiers disagree. Black: DGCNN is right (4 of 18 such objects). Red: DGCNN is wrong (2 of 8).

segmentation, PointNet vs DGCNN

Top: ground truth. Middle: PointNet. Bottom: DGCNN. The first three chairs are the ones DGCNN improves most, the last two the ones it hurts most.

Interpretation.