Q1. Classification Model (30 points)¶

Report the test accuracy:¶

  • 97.80% on the 953 test objects
Class Accuracy Correct / Total
chair (0) 99.84% 616 / 617
vase (1) 90.20% 92 / 102
lamp (2) 95.73% 224 / 234

Visualize a few random test point clouds and mention the predicted classes for each.¶

q1

Also visualize at least 1 failure prediction for each class (chair, vase and lamp), and provide an interpretation of the failure in a few sentences.¶

q1_failure

Most of the chairs are recognized. More errors occur in type vase and lamp. The reasonable reason may be they are both symmetric objects, so their global shapes overlap.

Q2. Segmentation Model (30 points)¶

Report the test accuracy.¶

  • 90.10% point-level accuracy on the 617 test objects

Visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth¶

[Object 0]

Prediction:

pred_q2_i0

Ground Truth:

gt_q2_i0

[Object 1]

Prediction:

pred_q2_i1

Ground Truth:

gt_q2_i1

[Object 471]

Prediction:

pred_q2_i471

Ground Truth:

gt_q2_i471

worst accuracy:

[Object 26]

Prediction:

pred_q2_i26

Ground Truth:

gt_q2_i26

[Object 235]

Prediction:

pred_q2_i235

Ground Truth:

gt_q2_i235

Report the prediction accuracy for each object, and provide an interpretation of the result in a few sentences.¶

  • Object 0: 94.24%

  • Object 1: 98.15%

  • Object 471: 99.75% (best one)

  • Object 26: 47.27% (worse one)

  • Object 235: 47.68%

The model learned the part of chair and most objects are segmented well. The worse segmentation object have a pattern: shaped like armchairs or sofa chairs, and the good segmentation are chairs with thin legs. A plausible explanation is that the wide seat make the boundary ambigous, and thin-legged chairs have clear part structure.

Q3. Robustness Analysis (20 points)¶

1. You can rotate the input point clouds by certain degrees and report how much the accuracy falls¶

Briefly describe your procedure¶

Rotate the input point clouds by 30, 50 , 90 degree with Y axis. Point subsampling uses seed 0, so the only changing factor is the angle.

For each task (classification and segmentation), report test accuracy and visualization on a few samples, in comparison with your results from Q1 & Q2.¶

Angle Classification accuracy Segmentation accuracy
0° (Q1/Q2) 97.80% 90.10%
30° 86.88% 77.25%
60° 48.27% 66.01%
90° 43.13% 47.07%

Classification:

q3

Segmentation:

[Ground Truth]

gt_q3a_rot30

[30°]

Prediction:

gt_q3a_rot30

[60°]

Prediction:

gt_q3a_rot60

[90°]

Prediction:

gt_q3a_rot90

Provide some interpretation of the results in a few sentences.¶

Both classification and segmentation are sensitive to rotation. It is as expected because PointNet doesn't have rotation invariance. The type of vase and lamp are less affected because they are symmetric, but the chair isn't symmetric. The performance of classification and segmentation is worser when the rotation is larger.

2. You can input a different number of points points per object (modify --num_points when evaluating models in eval_cls.py and eval_seg.py)¶

Briefly describe your procedure¶

I test different --num_points with the value of 5000 and 1000, and the baseline is 10000.

For each task (classification and segmentation), report test accuracy and visualization on a few samples, in comparison with your results from Q1 & Q2.¶

Points per object Classification accuracy Segmentation accuracy
10000 (Q1/Q2) 97.80% 90.10%
5000 97.69% 90.09%
1000 97.27% 89.66%

Classification:

cls_random_q3b_n1000

Segmentation:

[--num_points 5000]

Prediction:

pred_q3b_n5000

Ground Truth:

gt_q3b_n5000

[--num_points 1000]

Prediction:

pred_q3b_n1000

Ground Truth:

gt_q3b_n1000

Provide some interpretation of the results in a few sentences.¶

The classification and segmentation models are still robust when the input point number decrease. We reduce the input point number to 0.1, and the performance doesn't decrease a lot, it still reach 97% for classification and 89% for segmentation. The global feature remains stable when the point set is sparser.