Assignment 5: Point Cloud Learning

Guo Yu · guoy

16-825 Learning for 3D Vision · Classification, segmentation, robustness, and locality

Q1. Classification Model

I implemented a PointNet classifier without transformation blocks. Shared pointwise layers map coordinates to 64, 128, and 1024 channels. Global max pooling produces an object descriptor, followed by fully connected layers of 512 and 256 units and three class logits. The model distinguishes chairs, vases, and lamps.

Test accuracy: 98.11% (935 / 953 objects), using 10000 points per object.

Random test examples

Six randomly selected test objects. Each panel shows the object index, ground-truth class, and predicted class. Colors indicate height, not semantic labels.
Six randomly selected test objects. Each panel shows the object index, ground-truth class, and predicted class. Colors indicate height, not semantic labels.

Failure examples

One misclassified example for each true class: chair #406, vase #716, and lamp #883.
One misclassified example for each true class: chair #406, vase #716, and lamp #883.

These explanations are hypotheses based on the visible geometry, rather than verified causes of the network’s decisions.

Q2. Segmentation Model

The PointNet segmenter combines a 64-channel feature at each point with a broadcast 1024-channel global max-pooled descriptor. Shared prediction layers produce six part logits per point. The task segments chair point clouds into six semantic part classes.

Test point accuracy: 90.35%, evaluated on 617 objects with 10000 points each.

The following five objects include two difficult predictions (#351 and #61). Accuracy is the fraction of correctly labeled points in each object.

Test objectPoint accuracyExample
#095.06%Test example
#198.72%Test example
#289.60%Test example
#35144.55%Difficult prediction
#6146.28%Difficult prediction

Part 0Part 1Part 2Part 3Part 4Part 5

Object #0 — 95.06%

Ground-truth segmentation of chair 0, rotating view
Ground truth
PointNet segmentation of chair 0, rotating view
PointNet prediction

The model largely separates the back, seat, and legs correctly. Remaining errors occur around connections between parts, where adjacent regions have similar coordinates and weak boundaries.

Object #1 — 98.72%

Ground-truth segmentation of chair 1, rotating view
Ground truth
PointNet segmentation of chair 1, rotating view
PointNet prediction

The thin frame, seat, back, and armrests are mostly recovered. Their clear spatial separation makes this an easier case, although small errors remain around narrow junctions.

Object #2 — 89.60%

Ground-truth segmentation of chair 2, rotating view
Ground truth
PointNet segmentation of chair 2, rotating view
PointNet prediction

The main seat, back, and pedestal are recognizable, but the rounded seat shell blends into the back and side regions. These smooth transitions make precise part boundaries harder than in a simple frame chair.

Object #351 — 44.55%

Ground-truth segmentation of chair 351, rotating view
Ground truth
PointNet segmentation of chair 351, rotating view
PointNet prediction

This chair has a continuous, enclosing shell with broad side regions. The prediction labels much of the lower shell as leg/support points instead of recovering the ground-truth side structure, showing that global shape cues are insufficient for this unusual geometry.

Object #61 — 46.28%

Ground-truth segmentation of chair 61, rotating view
Ground truth
PointNet segmentation of chair 61, rotating view
PointNet prediction

The squat, box-like shape has a back and sides that are less distinct than those of a conventional chair. A large region is predicted as seat, with additional part confusion near the upper structure, producing a low accuracy despite a plausible overall outline.

Q3. Robustness Analysis

I evaluated the fixed Q1 and Q2 PointNet checkpoints under two input perturbations, with no retraining. Classification accuracy is measured per object; segmentation accuracy is measured per point. The full, unrotated 10000-point input is the reference.

Experiment 1: Rotation

Procedure. Rotate each test point cloud about the vertical y axis through the origin by 45° and 90°. All 10000 points are retained, and point labels remain unchanged. The model is run in evaluation mode.

ConditionClassificationDrop (pp)SegmentationDrop (pp)
Original (0°, 10000 points)98.11%+0.0090.35%+0.00
45° rotation49.74%+48.3771.27%+19.09
90° rotation36.73%+61.3957.30%+33.06

Drops are measured in percentage points (pp). A negative drop indicates an improvement.

Rotation accuracy for both tasks. The 0° results reproduce the Q1/Q2 reference to the displayed precision.
Rotation accuracy for both tasks. The 0° results reproduce the Q1/Q2 reference to the displayed precision.

Interpretation. Rotation sharply reduces accuracy: at 90°, classification drops by 61.39 percentage points and segmentation by 33.06. PointNet’s max pooling is invariant to point ordering, but the coordinate-dependent features are not inherently rotation invariant. The results suggest that the models rely strongly on the canonical orientations seen during training.

Comparison with Q1 and Q2: 90° rotation

Classification: the same three objects before and after rotation, with ground-truth and predicted classes. Chair #84 becomes incorrect, while vase #621 and lamp #734 remain correct.
Classification: the same three objects before and after rotation, with ground-truth and predicted classes. Chair #84 becomes incorrect, while vase #621 and lamp #734 remain correct.
Segmentation: objects #0, #1, and #351. Each row shows baseline ground truth, baseline prediction, rotated ground truth, and rotated prediction; accuracies are printed above the panels.
Segmentation: objects #0, #1, and #351. Each row shows baseline ground truth, baseline prediction, rotated ground truth, and rotated prediction; accuracies are printed above the panels.

The simple frame chairs retain some recognizable parts after rotation, but their labels become less consistent. Object #351 is already difficult before rotation and becomes worse, falling from 44.56% to 23.14% in this evaluation.

Additional comparisons at 45°

Classification comparison at 45° for the same three objects.
Classification comparison at 45° for the same three objects.
Segmentation comparison at 45°. Ground truth rotates with the input geometry.
Segmentation comparison at 45°. Ground truth rotates with the input geometry.

Experiment 2: Number of Input Points

Procedure. Evaluate each model using 5000, 1000, and 256 points instead of 10000, without rotating the cloud. A fixed random seed (42) defines a common point permutation; smaller samples are nested subsets. Segmentation labels are selected with exactly the same indices. Segmentation accuracy is evaluated only on the retained points.

ConditionClassificationDrop (pp)SegmentationDrop (pp)
Original (0°, 10000 points)98.11%+0.0090.35%+0.00
5000 points98.22%-0.1090.41%-0.05
1000 points97.90%+0.2190.36%-0.01
256 points96.85%+1.2687.36%+2.99
Accuracy as the number of input points changes. The full 10000-point result is the reference.
Accuracy as the number of input points changes. The full 10000-point result is the reference.

Interpretation. Both models are much more robust to reduced point density than to rotation. At 1000 points, classification remains at 97.90% and segmentation at 90.36%; at 256 points, they fall to 96.85% and 87.36%. Sparse inputs can omit fine geometry or points that contribute strongly to global max-pooled features. Small improvements at some densities reflect these particular subsets; one seed does not establish that fewer points are generally better.

Comparison with Q1 and Q2: 256 points

Classification with full inputs and 256-point inputs. All three displayed objects remain correct, even though accuracy over the full test set decreases.
Classification with full inputs and 256-point inputs. All three displayed objects remain correct, even though accuracy over the full test set decreases.
Segmentation with full inputs and 256-point inputs, including the corresponding sampled ground truth. Major parts remain visible, but fine boundary evidence is sparse.
Segmentation with full inputs and 256-point inputs, including the corresponding sampled ground truth. Major parts remain visible, but fine boundary evidence is sparse.

Individual examples need not follow the aggregate trend. In particular, the apparent improvement for object #351 is measured on a different, smaller set of points and does not imply that the model now segments the discarded points correctly.

Additional comparisons at 5000 and 1000 points

Classification: full inputs versus 5000-point inputs.
Classification: full inputs versus 5000-point inputs.
Segmentation: full inputs versus 5000-point inputs, with matched ground truth.
Segmentation: full inputs versus 5000-point inputs, with matched ground truth.
Classification: full inputs versus 1000-point inputs.
Classification: full inputs versus 1000-point inputs.
Segmentation: full inputs versus 1000-point inputs, with matched ground truth.
Segmentation: full inputs versus 1000-point inputs, with matched ground truth.

Q4. Locality: Dynamic Graph CNN

Implemented model

I implemented a compact DGCNN with three EdgeConv blocks of 64, 64, and 128 channels and a 256-channel embedding. At each block, a new k-nearest-neighbor graph (k = 20) is constructed in the current feature space. A shared network processes the edge feature [xᵢ, xⱼ − xᵢ], followed by max pooling over neighbors. This explicitly uses local relationships, while the first PointNet processes individual points before global pooling.

The classifier combines global max and mean pooled features. The segmenter concatenates the local EdgeConv features with a broadcast global descriptor and predicts six labels per point. This is a smaller adaptation of DGCNN, without a transformation network or category conditioning.

Test accuracy compared with Q1 and Q2

ModelTraining points per objectClassification accuracySegmentation accuracy
PointNet (Q1/Q2)1000098.11%90.35%
Compact DGCNN500098.01%91.95%

DGCNN correctly classifies 934 / 953 test objects. Relative to PointNet, classification changes by -0.10 percentage points, and segmentation improves by 1.60 percentage points.

The training settings differ in point count: PointNet uses 10000 points, while DGCNN uses 5000. The saved DGCNN segmentation evaluation also uses 5000 points per object. These results compare the obtained systems and do not isolate architecture alone. For context, Q3’s PointNet evaluation at 5000 points gives 98.22% classification and 90.41% segmentation; it was still trained on 10000 points.

Classification comparison

The same diagnostic test objects with PointNet and DGCNN predictions. Each displayed cloud contains 2500 sampled points for readability; these are visualization samples, not a new model evaluation.
The same diagnostic test objects with PointNet and DGCNN predictions. Each displayed cloud contains 2500 sampled points for readability; these are visualization samples, not a new model evaluation.

DGCNN corrects the chair #406 and lamp #883 errors shown in Q1, but both models still confuse vase #716 with a lamp. Conversely, lamp #869 is correct for PointNet and incorrect for DGCNN. Local features help some shapes but do not eliminate the geometric ambiguity between lamps and vases, and the overall classification score does not improve.

Segmentation comparison

The same five Q2 objects are shown below. Ground truth and PointNet use the original 10000-point clouds; DGCNN predictions use the saved 5000-point samples. The label colors are the same across models. Per-object accuracies use each model’s evaluated points.

Test objectPointNet (10000 points)DGCNN (5000 points)Change (pp)
#095.06%95.24%+0.18
#198.72%99.30%+0.58
#289.60%87.86%-1.74
#35144.55%48.46%+3.91
#6146.28%56.04%+9.76

Object #0

Ground-truth segmentation of comparison object 0
Ground truth
PointNet prediction for comparison object 0
PointNet · 95.06%
DGCNN prediction for comparison object 0
DGCNN · 95.24%

Object #1

Ground-truth segmentation of comparison object 1
Ground truth
PointNet prediction for comparison object 1
PointNet · 98.72%
DGCNN prediction for comparison object 1
DGCNN · 99.30%

Object #2

Ground-truth segmentation of comparison object 2
Ground truth
PointNet prediction for comparison object 2
PointNet · 89.60%
DGCNN prediction for comparison object 2
DGCNN · 87.86%

Object #351

Ground-truth segmentation of comparison object 351
Ground truth
PointNet prediction for comparison object 351
PointNet · 44.55%
DGCNN prediction for comparison object 351
DGCNN · 48.46%

Object #61

Ground-truth segmentation of comparison object 61
Ground truth
PointNet prediction for comparison object 61
PointNet · 46.28%
DGCNN prediction for comparison object 61
DGCNN · 56.04%

Interpretation. Local relationships help DGCNN improve the simple frame examples (#0 and #1) and the difficult enclosing shapes (#351 and #61). Object #61 improves most among these examples, but its prediction still confuses a large region with the seat. The rounded chair #2 becomes slightly worse, showing that neighborhood features do not resolve every smooth part transition. The two difficult examples remain well below the overall accuracy, so locality helps without fully solving unusual part layouts.

Method reference: Wang et al., Dynamic Graph CNN for Learning on Point Clouds.