Q1. Classification Model (30 points)¶
Report the test accuracy:¶
- 97.80% on the 953 test objects
| Class | Accuracy | Correct / Total |
|---|---|---|
| chair (0) | 99.84% | 616 / 617 |
| vase (1) | 90.20% | 92 / 102 |
| lamp (2) | 95.73% | 224 / 234 |
Visualize a few random test point clouds and mention the predicted classes for each.¶

Also visualize at least 1 failure prediction for each class (chair, vase and lamp), and provide an interpretation of the failure in a few sentences.¶

Most of the chairs are recognized. More errors occur in type vase and lamp. The reasonable reason may be they are both symmetric objects, so their global shapes overlap.
Q2. Segmentation Model (30 points)¶
Report the test accuracy.¶
- 90.10% point-level accuracy on the 617 test objects
Visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth¶
[Object 0]
Prediction:

Ground Truth:

[Object 1]
Prediction:

Ground Truth:

[Object 471]
Prediction:

Ground Truth:

worst accuracy:
[Object 26]
Prediction:

Ground Truth:

[Object 235]
Prediction:

Ground Truth:

Report the prediction accuracy for each object, and provide an interpretation of the result in a few sentences.¶
Object 0: 94.24%
Object 1: 98.15%
Object 471: 99.75% (best one)
Object 26: 47.27% (worse one)
Object 235: 47.68%
The model learned the part of chair and most objects are segmented well. The worse segmentation object have a pattern: shaped like armchairs or sofa chairs, and the good segmentation are chairs with thin legs. A plausible explanation is that the wide seat make the boundary ambigous, and thin-legged chairs have clear part structure.
Q3. Robustness Analysis (20 points)¶
1. You can rotate the input point clouds by certain degrees and report how much the accuracy falls¶
Briefly describe your procedure¶
Rotate the input point clouds by 30, 50 , 90 degree with Y axis. Point subsampling uses seed 0, so the only changing factor is the angle.
For each task (classification and segmentation), report test accuracy and visualization on a few samples, in comparison with your results from Q1 & Q2.¶
| Angle | Classification accuracy | Segmentation accuracy |
|---|---|---|
| 0° (Q1/Q2) | 97.80% | 90.10% |
| 30° | 86.88% | 77.25% |
| 60° | 48.27% | 66.01% |
| 90° | 43.13% | 47.07% |
Classification:

Segmentation:
[Ground Truth]

[30°]
Prediction:

[60°]
Prediction:

[90°]
Prediction:

Provide some interpretation of the results in a few sentences.¶
Both classification and segmentation are sensitive to rotation. It is as expected because PointNet doesn't have rotation invariance. The type of vase and lamp are less affected because they are symmetric, but the chair isn't symmetric. The performance of classification and segmentation is worser when the rotation is larger.
2. You can input a different number of points points per object (modify --num_points when evaluating models in eval_cls.py and eval_seg.py)¶
Briefly describe your procedure¶
I test different --num_points with the value of 5000 and 1000, and the baseline is 10000.
For each task (classification and segmentation), report test accuracy and visualization on a few samples, in comparison with your results from Q1 & Q2.¶
| Points per object | Classification accuracy | Segmentation accuracy |
|---|---|---|
| 10000 (Q1/Q2) | 97.80% | 90.10% |
| 5000 | 97.69% | 90.09% |
| 1000 | 97.27% | 89.66% |
Classification:

Segmentation:
[--num_points 5000]
Prediction:

Ground Truth:

[--num_points 1000]
Prediction:

Ground Truth:

Provide some interpretation of the results in a few sentences.¶
The classification and segmentation models are still robust when the input point number decrease. We reduce the input point number to 0.1, and the performance doesn't decrease a lot, it still reach 97% for classification and 89% for segmentation. The global feature remains stable when the point set is sparser.