Assignment 5
PointNet
Q1. Classification Model (40 points)
test accuracy:0.9832
Success samples
gt: chair, pred: chair -- gt: vase, pred: vase -- gt: lamp, pred: lamp
Failure samples
gt: chair, pred: lamp -- gt: lamp, pred: vase -- gt: vase, pred: lamp
The first chair is folded, which is different from most of the open ones in the training set. The second lamp is very flat and round, and even to a human it’s hard to tell whether it’s a lamp or a vase. There is a plant inside the third vase, which makes it look very much like a lamp.
Q2. Segmentation Model (40 points)
test accuracy:0.8959
Success samples (left: GT, right: Pred)
ID: 0 Accuracy: 0.9261
ID: 444 Accuracy: 0.9704
ID: 456 Accuracy: 0.9152
bad predictions:
ID: 9 Accuracy: 0.6743
ID: 587 Accuracy: 0.7635
interpretation: for normal shape chairs, the prediction results are generally good. But for chair in uncommon shape, they are in the long tail distribution of training set, and are not well learned by models.
Q3. Robustness Analysis (20 points)
CLS task:procedure: rotate input pc by 30 degrees
accuracy: 0.9832(no rotation) --> 0.6747
procedure: rotate input pc by 60 degrees
accuracy: 0.9832(no rotation) --> 0.2896
SEG task:
procedure: rotate input pc by 30 degrees
accuracy: 0.8960(no rotation) --> 0.7038
procedure: rotate input pc by 60 degrees
accuracy: 0.8960(no rotation) --> 0.5101
gt -- pred -- pred rotated 30 -- pred rotated 60
gt -- pred -- pred rotated 30 -- pred rotated 60
gt -- pred -- pred rotated 30 -- pred rotated 60
For both cls and seg tasks, when the rotation is getting larger, the accuracy drops larger. This is probably because the prior in training set indicates the pc object should be placed at the flat ground without angles. And the part should some how have a high possibility to be horizontal to the ground plane.
CLS task:
procedure: downsample the number of points to 5000
accuracy: 0.9832(no downsample) --> 0.9832
procedure: downsample the number of points to 500
accuracy: 0.9832(no downsample) --> 0.9759
SEG task:
procedure: downsample the number of points to 5000
accuracy: 0.8960(no downsample) --> 0.8962
gt -- pred
procedure: downsample the number of points to 500
accuracy: 0.8960(no downsample) --> 0.8823
For both CLS and SEG tasks, it seems only a very large portion of downsampling will decrease the accuracy.