Assignment 5: Point Cloud Classification and Segmentation

16-825: Learning for 3D Vision · Kashu Yamazaki

Q1 Classification · Q2 Segmentation · Q3 Robustness · Q4 DGCNN

Q1. Classification Model

Model and training

The classifier is PointNet without the T-Net blocks. A shared per-point MLP with widths 3–64–64–64–128–1024 (each layer a 1×1 convolution, BatchNorm and ReLU) is followed by a max pool over the points and a 1024–512–256–3 fully connected head, with BatchNorm and ReLU after the first two layers and dropout 0.3 before the last. I trained it for 100 epochs with Adam (lr 1e-3, batch 32, no schedule) on all 10000 points of each object. The run took 22.7 minutes on an RTX 3080, and the best epoch was 54.

Checkpoint selection. The provided train.py keeps the checkpoint with the highest test accuracy, so the accuracies in Q1, Q2 and Q4 come from an epoch selected on the test set and are slightly optimistic compared with selection on a held-out split. Classification test accuracy first passed 0.95 at epoch 6 (epochs count from 0), and 65 of the 80 epochs from epoch 20 on scored 0.96 or higher. Segmentation test accuracy varied between 0.804 and 0.902 from epoch 100 on.

Test accuracy

Test accuracy is 0.9759 (930 of 953 objects), with all 10000 points per object. Vases have the highest error rate and are also the smallest class in training (741 of 6784 objects).

ClassTest objectsCorrectAccuracyMisclassified as
Chair6176160.9981 lamp
Vase102870.85313 lamp, 2 chair
Lamp2342270.9705 vase, 2 chair

Random test objects

Two objects per class, drawn with np.random.default_rng(0). Each GIF is a 30-frame turntable, and each point is coloured by its normalised xyz position so that depth and separate parts are visible. The number in brackets is the predicted class probability.

Test object 524, a chair, rotating.
Object 524: chair, predicted chair (1.000)
Test object 393, a chair, rotating.
Object 393: chair, predicted chair (1.000)
Test object 669, a vase, rotating.
Object 669: vase, predicted vase (0.998)
Test object 644, a vase holding a leafy plant, rotating.
Object 644: vase, predicted lamp (0.821). Misclassified; a leafy plant in a pot.
Test object 791, a lamp, rotating.
Object 791: lamp, predicted lamp (0.970)
Test object 728, a lamp, rotating.
Object 728: lamp, predicted lamp (1.000)

Failure cases

Object 543 is the only misclassified chair. The vase and lamp failures shown are two of the 15 misclassified vases and two of the 7 misclassified lamps.

Test object 543, a ladder-back chair, rotating.
Object 543: chair, predicted lamp (0.651)
Test object 651, an orchid in a bowl, rotating.
Object 651: vase, predicted lamp (0.990)
Test object 671, a low planter tray, rotating.
Object 671: vase, predicted chair (0.873)
Test object 827, a lamp with a drum shade, rotating.
Object 827: lamp, predicted vase (0.982)
Test object 922, a candelabra lamp, rotating.
Object 922: lamp, predicted vase (0.996)

Chair 543 is a ladder-back chair whose backrest is several times taller than the seat is wide, so most of its points lie on a tall, thin vertical strip above a small base. That outline is closer to a floor lamp than to a typical chair, and the model gave lamp 0.65.

Vase 651 is an orchid in a shallow bowl. The flower stalk curves up and over the bowl and ends in a wide flower head, which has the layout of a desk lamp: round base, bent arm, shade at the end. Vase 644 in the random set, a leafy plant in a pot, is the same kind of error. Vases that hold plants have spiky geometry at the top that resembles lamp arms or shades.

Vase 671 is a wide, low planter tray with small plants along the top. Most vases in the data are tall and closed, while this one is a flat rectangular slab about as high as a seat, and the model labelled it chair.

Lamp 827 is a drum shade on a short base with no visible stem. Its points form a closed shell that is close to rotationally symmetric, which is the usual vase shape. Lamp 922 is a candelabra with several curved arms that end in small shades on a small base, and it looks like stems in a narrow vase.

Of the 23 test errors, 18 are confusions between lamps and vases (13 vases called lamps, 5 lamps called vases). Both classes contain rotationally symmetric bodies and thin upright parts. PointNet reduces the object to one max-pooled feature, which encodes which local shapes are present but not how they connect, so a stem in a vase and a lamp arm give similar features.

Q2. Segmentation Model

Model and training

The segmentation network is PointNet without T-Nets. A per-point MLP 3–64–64 gives a 64-d local feature, and a second MLP 64–64–128–1024 followed by a max pool gives a 1024-d global feature. The global feature is copied to every point and concatenated with its local feature (1088-d), and a per-point MLP 1088–512–256–128 and a 128–6 convolution give the per-point logits. I trained it for 150 epochs with Adam (lr 1e-3, no schedule) at batch 16, on all 10000 points of each object; batch 32 peaked at 8.96 GB, too close to the 10 GB of the RTX 3080. The run took 64.1 minutes, and the best epoch was 136.

Test accuracy

Test accuracy is 0.9021, counted over all 6,170,000 test points (617 chairs of 10000 points). Per-object accuracy ranges from 0.470 (object 235) to 0.997 (object 471).

In the GIFs, cyan is the back, red the seat, blue the legs or base, and yellow the arms. The provided viz_seg colours label 0 white, the same as the background, so label 0 points do not show. Label 0 is 0.33% of test points, and none of the five objects below has a label 0 or label 1 point in either the ground truth or the Q2 prediction. Label 0 predictions appear later on the page, on chair 392 under yaw in Q3 and on chair 426 in Q4, and those captions give the counts.

ObjectChosen asPer-object accuracy
392random0.8060
315random0.9426
523random0.8629
235worst in the test set0.4695
351second worst0.4719

The random objects were drawn with np.random.default_rng(0).choice(617, 3, replace=False).

Chair 392, ground-truth part labels, rotating.
Object 392, ground truth
Chair 392, predicted part labels, rotating.
Object 392, prediction (0.806)
Chair 315, ground-truth part labels, rotating.
Object 315, ground truth
Chair 315, predicted part labels, rotating.
Object 315, prediction (0.943)
Chair 523, ground-truth part labels, rotating.
Object 523, ground truth
Chair 523, predicted part labels, rotating.
Object 523, prediction (0.863)
Chair 235, ground-truth part labels, rotating.
Object 235, ground truth
Chair 235, predicted part labels, rotating.
Object 235, prediction (0.470). Bad prediction.
Chair 351, ground-truth part labels, rotating.
Object 351, ground truth
Chair 351, predicted part labels, rotating.
Object 351, prediction (0.472). Bad prediction.

Interpretation

Object 315 is an office chair with a five-star base. The prediction matches the ground truth except along the seat–back join and where the arms meet the seat. Object 392 is a rocking chair; the model extends the seat label up into the lower part of the backrest and onto the arm supports, and gets the rockers and the back right. Object 523 is a chair without arms whose rear legs continue past the seat, and its errors are on the leg tops next to the seat and on the seat edge. In all three, the errors sit at part boundaries. A point's local feature in this network comes from its own coordinates only, so points just either side of a boundary get nearly the same feature.

Object 235 is a sofa-like armchair with closed sides and a skirt down to the floor. The ground truth labels the side panels and much of the box as arms (6165 of 10000 points) and the lower front as seat. The model labels the bottom of the box as legs and the cushion area as seat, which is where those parts are on a typical chair. Object 351 is a cylindrical barrel chair with a cut-out in the front. Its ground truth labels the shell as arms and back down to the floor and has no leg points, while the model splits the shell by height, with legs on the bottom half and back on the top half. Neither chair has visible legs, and on both the model assigns labels by height and position as it would on an ordinary four-legged chair.

Q3. Robustness Analysis

Procedure

robustness.py evaluates the Q1 and Q2 checkpoints under three perturbations of the test set, without retraining. Every setting starts from the same subsample as eval_cls.py and eval_seg.py (np.random.seed(0), then np.random.choice(10000, N), applied to every object and to the segmentation labels), so the unperturbed rows reproduce Q1 (0.9759) and Q2 (0.9021). Classification accuracy is over the 953 test objects and segmentation accuracy over all points of the 617 test chairs.

The data are y-up: chair legs sit at mean y = −0.47 and backs at y = +0.34, and chairs face +z. The grids below render each perturbed cloud from one fixed camera (elevation 20°, azimuth 30°). Classification grids show the first test object of each class (0 chair, 617 vase, 719 lamp), coloured by the unperturbed xyz so that colours follow the same points across columns; a red title is a wrong prediction. Segmentation grids show chair 392 from Q2, ground truth above the prediction, with that object's accuracy in the title.

Experiment 1: rotation

All points are rotated about the origin by 0 to 180 degrees about one of two axes, with all 10000 points kept. Rotation about x tips the object forward or backward, so the up axis no longer points up. Rotation about y (yaw) spins the object in place on the floor, which is the orientation change that a real scan of an upright object would show.

Angle (deg)Cls, x-axisCls, y-axisSeg, x-axisSeg, y-axis
0 (Q1/Q2)0.97590.97590.90210.9021
150.92340.97060.82350.8681
300.86040.92440.70600.8087
450.81010.84160.61180.7289
600.66950.74500.46620.6463
900.32420.69250.22110.5979
1350.66000.32210.21530.5136
1800.65900.41760.25220.6518
Test accuracy against rotation angle for classification and segmentation, about the x and y axes.
Accuracy against rotation angle. Solid lines rotate about x, dashed lines about y.

Per-class classification accuracy at selected angles, from the confusion matrix over the full test set (617 chairs, 102 vases, 234 lamps). Answering “chair” for every object would score 0.647.

RotationChairVaseLampMisclassified chairs
x, 90°0.1650.3730.722381 vase, 134 lamp
x, 180°0.6220.2550.932142 vase, 91 lamp
y, 90°0.5770.8240.940201 vase, 60 lamp
y, 135°0.0030.8630.927364 vase, 251 lamp
y, 180°0.1510.8240.94425 vase, 499 lamp
Classification of a chair, vase and lamp rotated about x by 0, 45, 90 and 180 degrees.
Classification, rotation about x by 0, 45, 90, 180°. Chair 0 becomes vase at 90° and lamp at 180°; vase 617 becomes chair at 45, 90 and 180°; lamp 719 stays correct.
Classification of a chair, vase and lamp rotated about y by 0, 45, 90 and 135 degrees.
Classification, rotation about y by 0, 45, 90, 135°. Chair 0 is called vase at 90 and 135°; the vase and lamp stay correct.
Segmentation of chair 392 rotated about x by 0, 45, 90 and 180 degrees.
Segmentation of chair 392, rotation about x by 0, 45, 90, 180° (accuracy 0.81, 0.59, 0.21, 0.19).
Segmentation of chair 392 rotated about y by 0, 45, 90 and 135 degrees.
Segmentation of chair 392, rotation about y by 0, 45, 90, 135° (accuracy 0.81, 0.63, 0.49, 0.47). The white gaps in the 45 and 135° predictions are points predicted as label 0 (812 and 979 of 10000).

Turntable GIFs of chair 392 under rotation, next to the unrotated Q2 result. The GIF camera orbits the rotated cloud, so a tilted chair stays tilted throughout.

Chair 392, ground truth, unrotated.
0° (Q2), ground truth
Chair 392, prediction, unrotated.
0° (Q2), prediction (0.806)
Chair 392 rotated 45 degrees about x, ground truth.
x 45°, ground truth
Chair 392 rotated 45 degrees about x, prediction.
x 45°, prediction (0.59)
Chair 392 rotated 90 degrees about x, ground truth.
x 90°, ground truth
Chair 392 rotated 90 degrees about x, prediction.
x 90°, prediction (0.21); labels follow height
Chair 392 rotated 90 degrees about y, ground truth.
y 90°, ground truth
Chair 392 rotated 90 degrees about y, prediction.
y 90°, prediction (0.49); back and arm labels swap

Neither model is rotation invariant. Without a T-Net, PointNet takes raw xyz coordinates, and the training objects are all in the same canonical orientation, so the features encode where parts are in the world frame. Tilting about x lowers accuracy fastest on both tasks. At 90° of tilt, classification falls to 0.32, below the 0.647 of a constant “chair” answer, and most chairs are called vases.

Under yaw, vases and lamps keep 0.82 to 0.94 accuracy at 90, 135 and 180°, as expected for shapes that are close to rotationally symmetric about y. Chairs drop to 2 of 617 correct at 135°, so the classifier recognises chairs partly by which way they face.

Segmentation degrades sooner than classification: at 15° of tilt it loses 0.08 against 0.05 for classification. The model labels points mostly by height. In the x-rotation grid, at 90 and 180° the lowest points are predicted as legs (blue) and the highest as back or arms whatever part they belong to, and at 180° the upside-down back is labelled as legs. Under yaw, segmentation accuracy recovers to 0.65 at 180° because the height order of the parts is unchanged and only front and back swap. At 90° of yaw the back is labelled as arms (yellow) and the arms as back (cyan), so the model also uses the z coordinate to find the back.

Experiment 2: number of points

Each object is subsampled to N = 10000, 5000, 1000, 500, 100, 50 or 10 points with the same random indices for every object, with no other change. The models were trained on 10000 points.

PointsClsSeg
10000 (Q1/Q2)0.97590.9021
50000.97480.9023
10000.97270.8964
5000.96540.8813
1000.91920.7907
500.80480.7174
100.27070.5801
Test accuracy against number of points per object, log scale.
Accuracy against points per object (log scale).
Classification of a chair, vase and lamp with 10000, 1000, 100, 50 and 10 points.
Classification with N = 10000, 1000, 100, 50, 10. Point radius grows as N shrinks so that sparse clouds stay visible. Vase 617 fails at 50 (chair) and 10 (lamp); chair 0 fails at 10 (lamp); lamp 719 stays correct.
Segmentation of chair 392 with 10000, 1000, 100, 50 and 10 points.
Segmentation of chair 392 with N = 10000, 1000, 100, 50, 10 (accuracy 0.81, 0.83, 0.78, 0.70, 0.60).
Chair 392 with 100 points, ground truth.
100 points, ground truth
Chair 392 with 100 points, prediction.
100 points, prediction (0.78). Compare the 10000-point Q2 result above (0.806).

Accuracy is almost flat from 10000 down to 500 points: classification goes from 0.9759 to 0.9654 and segmentation from 0.9021 to 0.8813. The global feature is a max over points, so each channel depends on a few extreme points, and a random 5% subsample still lands near most of them. The loss becomes visible at 100 points and large at 10. At N = 10 the classifier predicts lamp for 929 of the 953 objects, including 593 of 617 chairs and all 102 vases; at N = 50 chairs are still 0.843 correct but vases drop to 0.176 (77 of 102 called lamp).

Segmentation loses more than classification at 100 and 50 points, but at N = 10 it still scores 0.58, because part labels correlate with height (legs low, back high) and a single point carries its own height. A max over fewer points is never larger in any channel, so at small N the pooled features fall below the values that the BatchNorm layers after the pool saw in training at N = 10000, which may account for part of the drop. I did not retrain at small N to check.

Experiment 3: Gaussian noise

Independent Gaussian noise with standard deviation 0, 0.01, 0.02, 0.05 or 0.1 is added to each coordinate of all 10000 points (torch generator seeded with 0). Objects are scaled so that the farthest point is at distance 1 from the origin; over the classification test set the median extents are 0.94 (x), 1.59 (y) and 1.01 (z).

Noise stdClsSeg
0 (Q1/Q2)0.97590.9021
0.010.97270.9002
0.020.96640.8922
0.050.93810.8438
0.10.88770.7436
Test accuracy against Gaussian noise standard deviation.
Accuracy against noise standard deviation.
Classification of a chair, vase and lamp with noise std 0, 0.02, 0.05 and 0.1.
Classification with noise std 0, 0.02, 0.05, 0.1. Only vase 617 fails, at 0.1 (lamp).
Segmentation of chair 392 with noise std 0, 0.02, 0.05 and 0.1.
Segmentation of chair 392 with noise std 0, 0.02, 0.05, 0.1 (accuracy 0.81, 0.78, 0.71, 0.56).

The noise in these two grids is drawn separately from the noise in the accuracy run, so a prediction in a grid can differ from that object's prediction in the run that produced the table.

Both models lose at most 0.01 up to std 0.02. At std 0.1, about 6% of a typical object's height and 10% of its width or depth, the shapes in the classification grid are visibly blurred and classification still holds 0.89, while segmentation falls to 0.74. Classification depends on the overall shape, which survives the blur, whereas a point's part label flips when the noise moves it across a part boundary.

Q4. Locality: DGCNN

Model

I implemented DGCNN (Wang et al., “Dynamic Graph CNN for Learning on Point Clouds”). Its building block is EdgeConv: for every point it finds the k = 20 nearest neighbours in the current feature space and builds an edge feature [xj − xi, xi] for each neighbour j. A shared 1×1 Conv2d, BatchNorm and LeakyReLU(0.2) MLP runs over those edges, and a max over the 20 neighbours gives the new point feature. Each layer rebuilds the graph from its input features, so later layers group points that are similar in feature space as well as close in 3D. The neighbour search is torch.cdist and topk in PyTorch, with no compiled extensions.

Training and evaluation protocol

The neighbour search builds an N×N distance matrix per object in every EdgeConv layer. At batch 32 and 10000 points that matrix is 32 × 10000² × 4 bytes = 12.8 GB in float32, more than the 10 GB of the RTX 3080; at 1024 points it is 134 MB and at 2048 points 537 MB. In training, each batch draws a fresh random subset of 1024 points per object for classification and 2048 for segmentation, shared by all objects in the batch. The per-epoch test accuracy that train.py uses to pick the checkpoint also comes from random subsets of the same size. Training used the train.py defaults (Adam, lr 1e-3, batch 32, 250 epochs, no schedule, no augmentation beyond the resampling) on the babel cluster, with classification and segmentation on two NVIDIA L40S GPUs at the same time, about 19 s per epoch each. The best epochs were 205 for classification and 198 for segmentation.

eval_local.py evaluates PointNet and DGCNN on the same points: 5 random subsets of point indices (seeds 0 to 4), each shared by every object and both models, reported as mean ± std over the 5 draws. Per-object numbers and GIFs use draw 0. The Q1/Q2 PointNet models were trained on 10000 points and are evaluated here on the same 1024 or 2048 points as DGCNN.

Test accuracy

TaskPointsPointNet (Q1/Q2)DGCNN
Classification10240.9717 ± 0.00230.9832 ± 0.0007
Classification100000.9759not run
Segmentation20480.9007 ± 0.00050.9205 ± 0.0003
Segmentation100000.9021not run

On classification draw 0, PointNet misclassifies 29 of 953 test objects: 2 chairs called lamps, 17 vases (15 called lamps, 2 called chairs) and 10 lamps (8 called vases, 2 called chairs). DGCNN misclassifies 17: 1 chair called a lamp, 6 vases called lamps and 10 lamps called vases.

Classification results

Each GIF has three panels, left to right: ground truth, PointNet and DGCNN, all on the same 1024 points. Points are coloured by the class assigned to the whole object: blue chair, red vase, magenta lamp. The first three objects are Q1 failures.

Chair 543: ground truth, PointNet and DGCNN class colours.
Chair 543 (Q1 failure, a tall ladder-back chair): PointNet lamp, DGCNN chair.
Vase 671: ground truth, PointNet and DGCNN class colours.
Vase 671 (Q1 failure, a low planter tray): PointNet chair, DGCNN vase.
Lamp 827: ground truth, PointNet and DGCNN class colours.
Lamp 827 (Q1 failure, a drum shade): both models predict vase.
Lamp 750: ground truth, PointNet and DGCNN class colours.
Lamp 750: PointNet lamp, DGCNN vase. Among the four objects shown, this is the only one where DGCNN is wrong and PointNet is right.

Segmentation results

Ground truth, PointNet and DGCNN, left to right, on the same 2048 points, with the Q2 colours. The numbers are per-object accuracy for PointNet and DGCNN. Label 0 is white here too; of these four chairs, only 426 has label 0 predictions (165 from PointNet, 15 from DGCNN).

Chair 426: ground truth, PointNet and DGCNN part labels.
Chair 426: PointNet 0.616, DGCNN 0.967. PointNet labels the lower half of the seat block as legs (648 of 2048 points, against 57 in the ground truth) or as label 0 (165 points, white and hard to see); DGCNN labels 1578 of the 1618 seat points correctly.
Chair 212: ground truth, PointNet and DGCNN part labels.
Chair 212, a cantilever chair whose seat curves up into the backrest: PointNet 0.653, DGCNN 0.974. PointNet labels 608 of the 732 seat points as back; DGCNN labels 695 of them correctly. Both models get the arms right.
Chair 235: ground truth, PointNet and DGCNN part labels.
Chair 235 (Q2 worst case, a sofa-shaped chair): PointNet 0.447, DGCNN 0.536. PointNet labels the bottom as legs where the ground truth has seat and arms; DGCNN predicts few legs (55 points) and recovers some of the seat, but labels 630 of the 1252 arm points as seat.
Chair 351: ground truth, PointNet and DGCNN part labels.
Chair 351 (Q2 second worst, a barrel chair): PointNet 0.450, DGCNN 0.454. The ground truth labels the lower cylinder as back and arms, and both models predict a base there.

Two of PointNet's 15 worst chairs, 426 and 212, go above 0.96 with DGCNN. Others barely move or drop slightly, for example chair 26 (0.534 to 0.531) and chair 331 (0.636 to 0.551).

Interpretation

DGCNN is more accurate on both tasks at the same number of points: 0.9717 to 0.9832 on classification and 0.9007 to 0.9205 on segmentation, with a smaller spread across the five point subsets. Moving PointNet from 10000 points to these subsets lowers it from 0.9759 to 0.9717 on classification and from 0.9021 to 0.9007 on segmentation, so most of the gap comes from the model and little from PointNet seeing fewer points than it was trained on.

In classification, nearly all of the gain is on vases: PointNet calls 15 vases lamps and DGCNN calls 6. Lamp errors stay at 10 for both models. PointNet sees each point only through its absolute coordinates before one global max pool. In DGCNN, each EdgeConv layer sees a point together with the offsets to its 20 neighbours, which describe the local surface shape (a thin stem, a curved wall, a flat seat). My reading is that this lets it keep most vases out of the lamp class. Lamps with large rounded shades (827, 750) still look like vases to it: all 10 of DGCNN's lamp errors are vase predictions, while most of PointNet's errors go from vase to lamp.

In segmentation, PointNet's errors on 426 and 212 follow height and position, as in Q2 and Q3: it labels the lower part of a deep seat as legs, and a seat that curves up into the backrest as back. DGCNN's labels follow the part boundaries on both chairs. Points on the same surface have similar neighbourhoods and so get similar edge features, which fits this result. Chairs with unusual geometry or annotation stay hard for both models. On 235, a sofa with a skirt down to the floor, PointNet predicts legs in the lower region and DGCNN labels half of the arm points as seat. On 351 both predict a base where the ground truth has back and arms.

DGCNN costs more to run. Its neighbour search is quadratic in the number of points (12.8 GB per layer at 10000 points and batch 32, as computed above), so I trained and evaluated it on 1024 or 2048 points rather than 10000.