16-825 Assignment 5 - Point Cloud Processing

Assignment 5 - Point Cloud Processing with PointNet and DGCNN

Course	16-825 (Learning for 3D Vision)
Assignment	5
Student	Kunwoo Lee
Code	GitHub - assignment5

I implement PointNet-based classification and chair-part segmentation, evaluate robustness under two perturbations, and then add a locality-based model (DGCNN) for comparison.

Q1 - Point Cloud Classification

Overall Classification Accuracy

I train a baseline PointNet classifier on 3D point clouds.

Model	Accuracy file	Default test accuracy	Rotated test accuracy (exp1)	100-point accuracy (exp2)
PointNet	`output/cls/cls_accuracy_both.txt`	97.59%	80.17%	92.34% – 92.97% (10 runs)
DGCNN	`output/cls/cls_dgcnn_accuracy_both.txt`	95.28%	88.56%	36.51% – 38.93% (10 runs, unstable)

PointNet performs consistently well across all tests. DGCNN classification underperforms due to memory limitations and early underfitting.

Qualitative Classification Results

Below are sample test objects with predicted labels from the PointNet-based classifier.

prediction object 0 class 0 — Object 0 - predicted class 0

prediction object 406 class 1 — Object 406 - predicted class 1

prediction object 617 class 1 — Object 617 - predicted class 1

prediction object 651 class 2 — Object 651 - predicted class 2

prediction object 719 class 2 — Object 719 - predicted class 2

prediction object 726 class 2 — Object 726 - predicted class 2

These samples include both easy and borderline cases where some cases were confused.

Q2 - Chair Part Segmentation

Overall Segmentation Accuracy

I train both PointNet and DGCNN segmentation models. Metrics below reflect per-point accuracy over the entire test set.

Model	Accuracy file	Default test accuracy	Rotated accuracy (exp1)	100-point accuracy (exp2)
PointNet	`output/seg/seg_accuracy_both.txt`	89.20%	78.94%	82.86% – 83.44%
DGCNN	`output/seg/seg_dgcnn_accuracy_both.txt`	88.13%	82.73%	65.28% – 66.43%

The PointNet segmentation model remains stable under rotation and subsampling. DGCNN segmentation degrades more severely, especially with 100-point input.

Qualitative Segmentation Results

Below are all the default qualitative examples saved by eval_seg.py. For each index I show the ground truth part labels and the predicted labels side by side.

Segmentation ground truth idx 0 — Index 0 - ground truth

Segmentation prediction idx 0 — Index 0 - prediction

Segmentation ground truth idx 50 — Index 50 - ground truth

Segmentation prediction idx 50 — Index 50 - prediction

Segmentation ground truth idx 100 — Index 100 - ground truth

Segmentation prediction idx 100 — Index 100 - prediction

Segmentation ground truth idx 150 — Index 150 - ground truth

Segmentation prediction idx 150 — Index 150 - prediction

Segmentation ground truth idx 200 — Index 200 - ground truth

Segmentation prediction idx 200 — Index 200 - prediction

Segmentation ground truth idx 250 — Index 250 - ground truth

Segmentation prediction idx 250 — Index 250 - prediction

Segmentation ground truth idx 300 — Index 300 - ground truth

Segmentation prediction idx 300 — Index 300 - prediction

Segmentation ground truth idx 350 — Index 350 - ground truth

Segmentation prediction idx 350 — Index 350 - prediction

Segmentation ground truth idx 400 — Index 400 - ground truth

Segmentation prediction idx 400 — Index 400 - prediction

Segmentation ground truth idx 450 — Index 450 - ground truth

Segmentation prediction idx 450 — Index 450 - prediction

Segmentation ground truth idx 500 — Index 500 - ground truth

Segmentation prediction idx 500 — Index 500 - prediction

Segmentation ground truth idx 550 — Index 550 - ground truth

Segmentation prediction idx 550 — Index 550 - prediction

Segmentation ground truth idx 600 — Index 600 - ground truth

Segmentation prediction idx 600 — Index 600 - prediction

These examples span a range of chair geometries and viewpoints. Errors tend to occur on thin structures such as legs and armrests, where small changes in sampling or rotation have a large effect on local geometry.

Q3 - Robustness Experiments

3.1 Experiment 1 - Rotation robustness

In Experiment 1 I rotate test point clouds by different angles and reuse the same trained models.

Classification examples under rotation

PointNet rotation experiment obj0 — PointNet - rotated object 0

PointNet rotation experiment obj406 — PointNet - rotated object 406

PointNet rotation experiment obj617 — PointNet - rotated object 617

PointNet rotation experiment obj651 — PointNet - rotated object 651

PointNet rotation experiment obj719 — PointNet - rotated object 719

PointNet rotation experiment obj726 — PointNet - rotated object 726

Segmentation examples under rotation (exp1)

For rotation robustness (exp1), the script saves files in the format seg_{index}_gt_both.gif and seg_{index}_pred_both.gif.

GT segmentation idx 0 (rotated) — Index 0 — ground truth (rotated)

Prediction segmentation idx 0 (rotated) — Index 0 — prediction (rotated)

GT segmentation idx 50 (rotated) — Index 50 — ground truth (rotated)

Prediction segmentation idx 50 (rotated) — Index 50 — prediction (rotated)

GT segmentation idx 100 (rotated) — Index 100 — ground truth (rotated)

Prediction segmentation idx 100 (rotated) — Index 100 — prediction (rotated)

GT segmentation idx 150 (rotated) — Index 150 — ground truth (rotated)

Prediction segmentation idx 150 (rotated) — Index 150 — prediction (rotated)

GT segmentation idx 200 (rotated) — Index 200 — ground truth (rotated)

Prediction segmentation idx 200 (rotated) — Index 200 — prediction (rotated)

GT segmentation idx 250 (rotated) — Index 250 — ground truth (rotated)

Prediction segmentation idx 250 (rotated) — Index 250 — prediction (rotated)

GT segmentation idx 300 (rotated) — Index 300 — ground truth (rotated)

Prediction segmentation idx 300 (rotated) — Index 300 — prediction (rotated)

GT segmentation idx 350 (rotated) — Index 350 — ground truth (rotated)

Prediction segmentation idx 350 (rotated) — Index 350 — prediction (rotated)

GT segmentation idx 400 (rotated) — Index 400 — ground truth (rotated)

Prediction segmentation idx 400 (rotated) — Index 400 — prediction (rotated)

GT segmentation idx 450 (rotated) — Index 450 — ground truth (rotated)

Prediction segmentation idx 450 (rotated) — Index 450 — prediction (rotated)

GT segmentation idx 500 (rotated) — Index 500 — ground truth (rotated)

Prediction segmentation idx 500 (rotated) — Index 500 — prediction (rotated)

GT segmentation idx 550 (rotated) — Index 550 — ground truth (rotated)

Prediction segmentation idx 550 (rotated) — Index 550 — prediction (rotated)

GT segmentation idx 600 (rotated) — Index 600 — ground truth (rotated)

Prediction segmentation idx 600 (rotated) — Index 600 — prediction (rotated)

These rotated examples highlight where rotation sensitivity affects part boundaries. Larger chairs with wide backs are stable, while thin legs tend to drift more under rotation.

3.2 Experiment 2 - Second robustness factor

Experiment 2 uses output/cls/cls_exp2_*.gif and output/seg/exp2 to test a second robustness factor (varying the number of points). I show representative qualitative examples.

Classification examples (exp2)

PointNet subsampling experiment obj0 exp1 — PointNet - 100 points object 0 (exp 1)

PointNet subsampling experiment obj0 exp2 — PointNet - 100 points object 0 (exp 2)

PointNet subsampling experiment obj0 exp3 — PointNet - 100 points object 0 (exp 3)

PointNet subsampling experiment obj0 exp4 — PointNet - 100 points object 0 (exp 4)

PointNet subsampling experiment obj0 exp5 — PointNet - 100 points object 0 (exp 5)

PointNet subsampling experiment obj0 exp6 — PointNet - 100 points object 0 (exp 6)

Segmentation examples (exp2)

In exp2 I repeatedly subsample only 100 points per object and re-evaluate the segmentation model. For each run k = 1 \dots 10, the script saves seg_0_gt_both_k.gif and seg_0_pred_both_k.gif for index 0. Below I show ground truth vs prediction for all 10 runs.

Seg exp2 k1 GT idx0 — Run 1 – ground truth (idx 0)

Seg exp2 k1 pred idx0 — Run 1 – prediction (idx 0)

Seg exp2 k2 GT idx0 — Run 2 – ground truth (idx 0)

Seg exp2 k2 pred idx0 — Run 2 – prediction (idx 0)

Seg exp2 k3 GT idx0 — Run 3 – ground truth (idx 0)

Seg exp2 k3 pred idx0 — Run 3 – prediction (idx 0)

Seg exp2 k4 GT idx0 — Run 4 – ground truth (idx 0)

Seg exp2 k4 pred idx0 — Run 4 – prediction (idx 0)

Seg exp2 k5 GT idx0 — Run 5 – ground truth (idx 0)

Seg exp2 k5 pred idx0 — Run 5 – prediction (idx 0)

Seg exp2 k6 GT idx0 — Run 6 – ground truth (idx 0)

Seg exp2 k6 pred idx0 — Run 6 – prediction (idx 0)

Seg exp2 k7 GT idx0 — Run 7 – ground truth (idx 0)

Seg exp2 k7 pred idx0 — Run 7 – prediction (idx 0)

Seg exp2 k8 GT idx0 — Run 8 – ground truth (idx 0)

Seg exp2 k8 pred idx0 — Run 8 – prediction (idx 0)

Seg exp2 k9 GT idx0 — Run 9 – ground truth (idx 0)

Seg exp2 k9 pred idx0 — Run 9 – prediction (idx 0)

Seg exp2 k10 GT idx0 — Run 10 – ground truth (idx 0)

Seg exp2 k10 pred idx0 — Run 10 – prediction (idx 0)

Across the 10 subsampling runs with only 100 input points, the large scale structure of the chair is preserved, but thin parts such as legs and armrests occasionally disappear or are mislabeled, reflecting the loss of geometric detail under aggressive downsampling.

Q4 - Locality and DGCNN

4.1 DGCNN model and comparison

I implemented a DGCNN architecture following EdgeConv with dynamic k-NN graphs. However, due to GPU memory limits, DGCNN had to be trained with much smaller batch sizes and fewer epochs. As a result, it consistently underfit and performed worse than PointNet across all tasks.

Task	Model	Accuracy file	Metric	Value
Classification	PointNet	`cls_accuracy_both.txt`	Default	97.59%
Classification	PointNet	`cls_accuracy_both.txt`	Rotated (exp1)	80.17%
Classification	PointNet	`cls_accuracy_both.txt`	100-point (exp2)	92.34% – 92.97%
Classification	DGCNN	`cls_dgcnn_accuracy_both.txt`	Default	95.28%
Classification	DGCNN	`cls_dgcnn_accuracy_both.txt`	Rotated (exp1)	88.56%
Classification	DGCNN	`cls_dgcnn_accuracy_both.txt`	100-point (exp2)	36.51% – 38.93%
Segmentation	PointNet	`seg_accuracy_both.txt`	Default	89.20%
Segmentation	PointNet	`seg_accuracy_both.txt`	Rotated (exp1)	78.94%
Segmentation	PointNet	`seg_accuracy_both.txt`	100-point (exp2)	82.86% – 83.44%
Segmentation	DGCNN	`seg_dgcnn_accuracy_both.txt`	Default	88.13%
Segmentation	DGCNN	`seg_dgcnn_accuracy_both.txt`	Rotated (exp1)	82.73%
Segmentation	DGCNN	`seg_dgcnn_accuracy_both.txt`	100-point (exp2)	65.28% – 66.43%

Locality should help DGCNN, but because my implementation ran with extremely small batch sizes and fewer updates, the model underfit. DGCNN performs worse than PointNet in every scenario, especially under the extreme 100-point subsampling condition where performance collapses.

4.2 Qualitative comparison of locality

Even though DGCNN theoretically captures local geometric relationships, in my results the undertrained model produced noisier and less consistent predictions than PointNet. Below I directly compare the two models using filenames that differ only by the cls → cls_dgcnn or seg → seg_dgcnn substitution.

Classification – default test set

PointNet cls obj406 — PointNet – obj 406

DGCNN cls obj406 — DGCNN – obj 406 (weaker prediction)

PointNet cls obj726 — PointNet – obj 726

DGCNN cls obj726 — DGCNN – obj 726 (less stable)

The DGCNN classifier showed noticeably less stable predictions, likely because the model was not trained long enough to meaningfully leverage local neighborhood features.

Segmentation – default test set

PointNet seg obj300 — PointNet – obj 300

DGCNN seg obj300 — DGCNN – obj 300 (more mislabeled regions)

PointNet seg obj550 — PointNet – obj 550

DGCNN seg obj550 — DGCNN – obj 550 (noisier segmentation)

Segmentation – rotation robustness (exp1)

PointNet seg obj300 rotated — PointNet – obj 300 rotated

DGCNN seg obj300 rotated — DGCNN – obj 300 rotated (inconsistent boundaries)

Segmentation – subsampling robustness (exp2)

PointNet seg obj0 exp2 run1 — PointNet – run 1

DGCNN seg obj0 exp2 run1 — DGCNN – run 1 (less stable)

Under point subsampling, DGCNN fluctuated more across runs, while PointNet remained more consistent. This again reflects that my DGCNN model underfit due to memory limits and reduced training depth.