Assignment 5: Point Cloud Processing

Andrew ID: yunxianm

I compared PointNet with a model that groups nearby points. The classification dataset has 6,784 training objects and 953 test objects from three classes: chair, vase, and lamp. The segmentation dataset has 4,489 training chairs and 617 test chairs. Each input contains 10,000 points. The scatterplots display up to 3,000 points for readability.

All four models were trained for 250 epochs with batch size 32, Adam, learning rate 0.001, and seed 42. I used the starter's rule of saving the checkpoint with the highest test accuracy. These scores are test-selected, so they are not independent held-out estimates.

Model Training epochs Epoch saved (1-250)
PointNet classification 250 112
PointNet segmentation 250 214
Local classification 250 71
Local segmentation 250 143

Q1. Classification

The classifier uses shared layers with widths 3, 64, 128, and 1024, followed by max pooling over points. Its fully connected head has widths 1024, 512, 256, and 3, with batch normalization, ReLU, and dropout 0.3. It has 801,539 parameters. I used the PointNet architecture without a transformation network.

The model correctly classifies 934/953 test objects (98.01%).

Class Correct / total Recall
Chair 616/617 99.84%
Vase 96/102 94.12%
Lamp 222/234 94.87%

Six random classification examples

I sampled six test objects using seed 42. Objects 621, 734, 412, 818, 417, and 84 are all classified correctly.

One classification error from each class

Chair #406, predicted as vase. Its rounded back and side surfaces enclose a broad volume, while the legs are thin. The outer shape resembles a vessel, although the legs are still visible.

Vase #651, predicted as lamp. A low bowl-like body supports a thin curved structure with a small upper cluster. This arrangement resembles a lamp with a base, stem, and shade.

Lamp #750, predicted as vase. A broad faceted body dominates the shape. Its short stem is less distinctive than the long curved stem of the correctly classified lamp #818.

These examples suggest that the global shape can dominate smaller details that distinguish the classes.

Q2. Segmentation

The segmenter combines a 64-dimensional feature at each point with the 1024-dimensional global feature. Shared layers of widths 1088, 512, 256, 128, and 6 produce the point labels. The model has 867,462 parameters.

It correctly labels 5,559,589/6,170,000 test points (90.11%).

Part (ID) Correct / total points Recall
Footrest-like part (0) 18,280/20,188 90.55%
Headrest (1) 681/7,865 8.66%
Backrest (2) 1,800,994/1,972,401 91.31%
Armrests (3) 419,668/563,263 74.51%
Base and legs (4) 1,102,769/1,252,860 88.02%
Seat (5) 2,217,197/2,353,423 94.21%

I inferred the part names above from the labeled shapes and kept the numeric IDs in the figures. The headrest (ID 1) makes up only about 0.13% of test points. Its 8.66% recall shows a weakness that is easy to miss in the overall accuracy.

Ground truth and predicted segmentation for five chairs

The first two chairs have the lowest test accuracy. The other three are random examples sampled with seed 42. Accuracies use all 10,000 points.

Test object Accuracy Observation
235 44.23% Much of an ID 3 region is assigned ID 5, changing a broad central and lower area. Some lower points also become ID 4.
351 48.98% The prediction adds a large ID 4 region in the lower structure, although the ground truth contains no ID 4 points.
404 98.45% The backrest, seat, and lower support regions closely follow the ground truth. Only 155 points are incorrect.
54 71.75% ID 5 extends across a broad ID 3 region. Much of the upper backrest region remains correct.
477 91.88% The main regions look similar, but ID 5 extends into neighboring regions, leaving 812 errors.

Q3. Robustness

I tested two changes to the inputs while keeping the Q1/Q2 checkpoints fixed: rotation and reduced point count. Both experiments use the full test sets.

Rotation. I rotated the point clouds around the y axis by 0, 15, 30, 45, 90, and 180 degrees. The transformation was x_new = cos(a)*x + sin(a)*z, y_new = y, and z_new = -sin(a)*x + cos(a)*z. Each segmentation label stayed with its point.

Rotation (degrees) Classification accuracy Segmentation accuracy
0 98.01% 90.11%
15 96.75% 84.71%
30 84.89% 74.60%
45 69.67% 68.42%
90 69.67% 53.39%
180 34.52% 59.61%

Classification before and after a 45-degree rotation

Segmentation before and after a 45-degree rotation

The filters still operate on XYZ coordinates, so rotating an object changes the point features and the pooled feature. I trained on the original orientations without rotation augmentation. The models can therefore learn position and orientation patterns that do not transfer to the rotated inputs.

At 180 degrees, chair recall drops to 16/617 (2.59%); 521 chairs are predicted as lamps. Segmentation reaches its lowest accuracy at 90 degrees and recovers somewhat at 180 degrees.

The classification scores at 45 and 90 degrees are equal: both have 664 correct predictions, but 256 object predictions differ. Between those angles, 119 objects change from wrong to correct and 119 change from correct to wrong. The remaining 18 change to a different wrong label.

In the 45-degree examples, chairs #0 and #1 become a vase and a lamp, while chair #86 stays correct. Segmentation accuracy for #389 falls from 94.61% to 36.80%, for #35 from 93.35% to 39.24%, and for #55 from 90.98% to 74.03%.

Point count. I sampled 5,000, 1,000, and 100 points without replacement, using seeds 42, 43, and 44. I kept coordinates and segmentation labels together and used the same sampled point positions across objects in each run. The table gives the mean and standard deviation across the three subsets. The 10,000-point row uses the original inputs.

Input points Classification %, mean (SD) Segmentation %, mean (SD)
10,000 98.01 (0.00) 90.11 (0.00)
5,000 97.73 (0.05) 90.16 (0.02)
1,000 97.06 (0.09) 90.06 (0.04)
100 93.81 (0.31) 82.66 (0.05)

Classification with 10,000 and 1,000 points

Segmentation with 10,000 and 1,000 points

Accuracy under rotation and reduced point count

Removing points is less damaging than rotation. At 1,000 points, classification loses about one percentage point and segmentation stays near 90%. Redundant points on large surfaces can be removed while leaving the overall shape recognizable. At 100 points, small structures and boundaries are more easily missed, and segmentation drops to 82.66%.

Individual objects can change substantially even when the overall score barely moves. For the examples below, I compared both predictions on the same 1,000 retained points. The original-input figure titles show accuracy on all 10,000 points, so they differ slightly from this table.

Segmentation object Original prediction on retained points Prediction using only 1,000 points Change (pp)
339 92.50% 75.30% -17.20
272 80.10% 68.40% -11.70
54 71.50% 81.90% +10.40

Objects #339 and #272 show large changes between the seat and lower support regions. Random example #54 improves because more of its ID 3 region is preserved. The first two examples were selected for large losses. The slight overall increase at 5,000 points also depends on which labels remain in the sampled subsets.

Q4. Locality

I added a local encoder using 128 farthest-point anchors and 32 nearest neighbors per anchor. The anchors spread the groups across the object while keeping the distance calculation smaller than grouping around all 10,000 points. A 32-point group provides nearby shape information. I kept both settings fixed for classification and segmentation.

Each neighborhood uses both offsets from its anchor and absolute XYZ coordinates. The offsets describe the local shape; absolute coordinates retain where that shape lies on the object. Shared layers of widths 6, 64, 128, and 256 are followed by max pooling over neighbors. A separate point network with widths 3, 64, and 128 processes all input points.

For classification, I combined the global point and anchor features. For segmentation, I interpolated features from the three nearest anchors to each point, using inverse-distance weights, then combined them with pointwise and global features. This preserves a prediction for every input point.

The local classifier has 184,259 parameters and the local segmenter has 250,182. Their parameter counts differ from PointNet, so this comparison does not isolate locality alone.

Task PointNet accuracy Local model accuracy Change (pp)
Classification 98.01% 98.32% +0.31
Segmentation 90.11% 89.99% -0.12

PointNet and local-model accuracy

Classification predictions from both models

Ground truth and predictions from both segmentation models

The local classifier gets 937/953 objects correct, three more than PointNet. Lamp recall rises from 94.87% to 98.72%, while vase recall falls from 94.12% to 88.24%. Chair recall stays at 99.84%.

Object and true class PointNet prediction Local prediction
670 (vase) lamp vase
714 (vase) lamp vase
84 (chair) chair chair
620 (vase) vase lamp

The local model corrects vases #670 and #714, which have tall or irregular protrusions. Both models correctly classify chair #84, while rounded vase #620 becomes a lamp under the local model.

The local segmenter gets 5,552,357 points correct, 7,232 fewer than PointNet. Recalls for IDs 2, 3, and 4 improve, but ID 5 recall falls from 94.21% to 91.24%. ID 1 recall falls from 8.66% to 0.09% (7/7,865 points).

Segmentation object PointNet accuracy Local accuracy Change (pp)
105 59.21% 82.51% +23.30
178 76.90% 93.16% +16.26
54 71.75% 86.60% +14.85
447 98.46% 57.66% -40.80

For #105, the local prediction restores more of the ID 5 region where PointNet assigns ID 3. For random example #54, it preserves more of the ID 3 region. On #178, the seat region follows the ground truth more closely. On #447, much of the seat is assigned ID 4, producing the largest regression. The figures show two large improvements, one random example, and the largest regression.

AI assistance was used for implementation, running the experiments, and drafting this report.