Q1. Classification Model¶

Report the test accuracy.

Test accuracy: 0.9769150052465897¶

Visualize a few random test point clouds and mention the predicted classes for each.

A few point clouds and their predicted classes¶

I selected a few random objects by their indices.

Object 0: predicted chair, is a chair

Object 300: predicted chair, is a chair

Object 700: predicted vase, is a vase

Object 800: predicted lamp, is a lamp

failure predictions¶

Chair has no failure predictions

Vase:

Predicted lamp

It makes some sense because purely by looking at the geometry it was also hard for me to distinguish between this and potentially a bell shaped lamp.

Lamp:

Predicted vase

The shape of this lamp, with a narrow bottom and a fat top does look a little similar to the overall shape/ geometry of a vase, which may mean that this overall distribution relationship between points (this shape) could be easier misled to a vase.

Q2. Segmentation Model¶

Test accuracy: 0.9040074554294976¶

Visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth

Report the prediction accuracy for each object, and provide an interpretation of the result in a few sentences.

Segmentation results¶

Object 0:

Accuracy: 0.9562000036239624

Object 100:

Accuracy: 0.9567000269889832

Object 200:

Accuracy: 0.754800021648407

Object 500:

Accuracy: 0.7782999873161316

Object 600:

Accuracy: 0.9923999905586243

Overall I think the object is well segmented when it shows clear distinctions between their parts. For those chairs with clear structures, they generally perform

Comparing with the other 3 objects, I would say object 200 and object 500 are the two failed ones with accuracies of around 0.75 and 0.78. With the three better performing chairs, they are of cleaner shapes which would be easier to segment - with their legs, back part, seat and arm cleanly distinguishable. While the two objects with lower accuracy had overlapping regions (obj 200) or has unconventional shaped base (obj 500) that may make this segmentation task more difficult as the point. These cases when there is no clear boundary between parts the model shapes falls back to typical chair parts which performs less well for unconventional chair shapes.

Q3. Robustness Analysis¶

I experimented with different rotation and number of points as suggested in the readme.

Rotate the input points:¶

I tried rotating in each of the three axies, 30°, 60° and 90° separately. Q1/Q2 result are kept in 0° box in the table below.

Test accuracy¶

task 0° x 30° x 60° x 90° y 30° y 60° y 90° z 30° z 60° z 90°
cls 0.9769 0.7891 0.3610 0.2718 0.9297 0.7891 0.6915 0.8006 0.3841 0.3001
seg 0.9040 0.7406 0.4205 0.3034 0.8037 0.6693 0.6043 0.7241 0.5529 0.4718

Classification (same objects as Q1 chosen)¶

Object 0 (chair):

axis 0° 30° 60° 90°
x
y
z
predicted 0° 30° 60° 90°
x chair chair lamp vase
y chair chair lamp lamp
z chair chair vase vase

Object 300 (chair):

axis 0° 30° 60° 90°
x
y
z
predicted 0° 30° 60° 90°
x chair chair vase vase
y chair chair chair chair
z chair chair chair vase

Object 700 (vase):

axis 0° 30° 60° 90°
x
y
z
predicted 0° 30° 60° 90°
x vase vase vase lamp
y vase vase vase vase
z vase vase vase lamp

Object 800 (lamp):

axis 0° 30° 60° 90°
x
y
z
predicted 0° 30° 60° 90°
x lamp lamp chair lamp
y lamp lamp lamp lamp
z lamp lamp lamp chair

Segmentation (same objects as Q2, ground truth in Q2)¶

Object 0:

axis 0° 30° 60° 90°
x
y
z
accuracy 0° 30° 60° 90°
x 0.9562 0.8204 0.4763 0.3427
y 0.9562 0.9289 0.7856 0.7372
z 0.9562 0.8483 0.6625 0.4730

Object 100:

axis 0° 30° 60° 90°
x
y
z
accuracy 0° 30° 60° 90°
x 0.9567 0.7510 0.4274 0.2433
y 0.9567 0.8415 0.7009 0.6799
z 0.9567 0.7435 0.5047 0.3937

Object 200:

axis 0° 30° 60° 90°
x
y
z
accuracy 0° 30° 60° 90°
x 0.7548 0.6693 0.4789 0.2144
y 0.7548 0.6534 0.5547 0.3630
z 0.7548 0.6161 0.3862 0.3056

Object 500:

axis 0° 30° 60° 90°
x
y
z
accuracy 0° 30° 60° 90°
x 0.7783 0.7320 0.4411 0.3194
y 0.7783 0.7479 0.6772 0.6279
z 0.7783 0.7371 0.6977 0.5401

Object 600:

axis 0° 30° 60° 90°
x
y
z
accuracy 0° 30° 60° 90°
x 0.9924 0.7859 0.3627 0.3773
y 0.9924 0.9047 0.7552 0.6600
z 0.9924 0.7414 0.5244 0.6661

For classification in general, the higher the rotation angle, in all three axis, the lower the test accuracy. The robustness lasted upto around 30 degrees and 60 degrees onwards had a much greater drop comparing to 0 to 30 degrees rotation. This goes with prediction since the model is only trained on upright (0 degree) objects and has no transformation block to align the input, so a rotated object is just a different point distribution to the model. Small rotations still look close enough to the training data while larger ones do not.

An interesting observation is that the model is much more robust to rotations in the y axis comparing to the x and z axis (0.69 vs 0.27/ 0.30 at 90 degrees). y is the up axis here, so rotating about y only spins the object around and keeps how tall/ how fat the shape is, which is likely where most of the differentiating factors come from since what's trained are the 0 degree data. Rotating about x or z tips the object over and changes exactly that, e.g. chairs 0 and 300 are predicted as vase or lamp at 60-90 degrees about x.

Segmentation follows the same trend, with y again being the most robust. The part labels are closely tied to height (legs at the bottom, back at the top), so once the chair is tipped over the parts likely get labelled by where they are rather than what they are. This drop happens to all objects, for instance object 600 goes from 0.99 to 0.36 at 60 degrees about x.

Number of points per object:¶

I evaluated with 10000, 5000, 1000 and 100 points per object. Q1/Q2 result listed as 10000.

Test accuracy¶

task 10000 5000 1000 100
cls 0.9769 0.9769 0.9727 0.8615
seg 0.9040 0.9039 0.8978 0.8187

Classification (same objects as Q1)¶

object 10000 5000 1000 100
0
300
700
800
predicted (true) 10000 5000 1000 100
0 (chair) chair chair chair chair
300 (chair) chair chair chair chair
700 (vase) vase vase vase vase
800 (lamp) lamp lamp lamp lamp

Segmentation (same objects as Q2, ground truth in Q2)¶

object 10000 5000 1000 100
0
100
200
500
600
accuracy 10000 5000 1000 100
0 0.9562 0.9536 0.9520 0.9100
100 0.9567 0.9544 0.9450 0.9200
200 0.7548 0.7552 0.7190 0.4800
500 0.7783 0.7756 0.7960 0.7200
600 0.9924 0.9932 0.9960 0.9100

The model is quite robust to the number of points, for classification, the drop was barely noticeable up until only having 100 points. This makes sense since max pooling only keeps the max of each feature over all points, so as long as there are enough points to cover the overall shape, the global feature stays about the same.

Segmentations followed a similar trend, much less affected than how rotation changes things, likely for a similar reason as compared to classfication. For the per object accuracy, it generally stayed quite close to the accuracy at full scale. Only at 100 points when the shape gets too sparse, for some objects like object 200 it had a large drop, likely due to the fact that less points made it even harder to distinguish the borders between parts.

In [ ]: