16825 - HW5

1 Classification Model

I implemented the pointNet model with residual blocks, largely following the pointnet paper. I trained the model for 250epochs and tested it. I obtained a final test accuracy of 0.932.

Point Cloud Visualization: Reference vs. Prediction

Label: Chair	Classified as: Chair
Label: Chair	Classified as: Chair
Label: Chair	Classified as: Chair

Point Cloud Visualization: Failure Cases

Label: Chair	Classified as: Vase	The chair is designed from cutting down a polygon, which kept most of the mass and it does not have the usual distinct legs and backrest of a typical chair.
Label: Chair	Classified as: Lamp	The chair is focused to its hollow legs, which made it look similar to hanging lamps.
Label: Lamp	Classified as: Vase	The lamp has a very close base and a short stand, making it look similar to a vase.

2 Segmentation Model

I implemented the segmentation model with backbone from Q1 and evaluated it. I obtained a final test accuracy of 0.860.

Segmentation Visualization

	Accuracy: 79%	Very standard sofa shape, making it easy for the network.
	Accuracy: 71%	Stand of chair is evident.
	Accuracy: 97%	Clear legs and backrest.
	Accuracy: 41%	This one is hard because the chair is using a unconventional curve design for the legs, and it is overall futuristic. The network may not have seem such chair during training.
	Accuracy: 40%	I am quite confused for this one because the shape of the chair is still pretty evident. I would expect higher accuracy. The only explination I have is that its think legs made the network think it is a sofa.

3.1 Robustness Analysis - Scale

To test does scaling the point cloud affacts the network's performance, I scaled the points by 0.5x, 1.0x, and 2.0x and evaluated the model. The results are as follows:

Scale Applied	Classification Accuracy	Segmentation Accuracy
0.5x	0.311	0.770
1.0x	0.932	0.860
2.0x	0.842	0.702

As we can see, scaling the object to be smaller have a detrimental effect on the network's generalization. This is because 1. the shapes are not augmented for scale during training, therefore the network have overfitted to the scale in the training set. 2. The network may have learnt certain threshold to identify and differentiate parts, which failes to generalize when the object is scaled down too much.

Index	Segmentation GT	Segmentation Result 0.5x	Classification 0.5x	Segmentation Result 1.0x	Classification 1.0x	Segmentation Result 2.0x	Classification 2.0x
1			Chair -> Vase		Chair -> Lamp		Chair -> Vase
256			Chair -> Chair		Chair -> Chair		Chair -> Chair
512			Chair -> Chair		Chair -> Chair		Chair -> Chair

From the above table, we can see as the scale changes, the segmentation boundaries have also overflow to incorrect regions. This partly supports the hypothesis that the network have learnt certain thresholds to differentiate parts, which fails when the object is scaled. Classification is impacted the most when the object is scaled down, as the overall shape and structure is harder to identify.

3.2 Robustness Analysis - Point Downsampling

To test does downsampling the point cloud affacts the network's performance, I randomly downsampled the points to 10000(baseline), 5000, and 2000 points and evaluated the model. The results are as follows:

Points Used	Classification Accuracy	Segmentation Accuracy
10000	0.932	0.860
5000	0.927	0.862
2000	0.938	0.866

As we can see, downsampling the points does not have significant effect on the network's generalization. This is likely because the network is designed to have a invariant architecture to a set of points, and the point clouds are still dense enough to represent the overall shape.

Index	Segmentation GT	Segmentation Result 10000	Classification 10000	Segmentation Result 5000	Classification 5000	Segmentation Result 2000	Classification 2000
1			Chair -> Chair		Chair -> Chair		Chair -> Chair
256			Chair -> Chair		Chair -> Chair		Chair -> Chair
512			Chair -> Chair		Chair -> Chair		Chair -> Chair

From the above visualization, we can see that the number of points does not significantly affect the segmentation boundaries or classification results, supporting the previous conclusion.