Q1. Classification Model¶
Report the test accuracy.
Test accuracy: 0.9769150052465897¶
Visualize a few random test point clouds and mention the predicted classes for each.
A few point clouds and their predicted classes¶
I selected a few random objects by their indices.
Object 0: predicted chair, is a chair

Object 300: predicted chair, is a chair

Object 700: predicted vase, is a vase

Object 800: predicted lamp, is a lamp

failure predictions¶
Chair has no failure predictions
Vase:

Predicted lamp
It makes some sense because purely by looking at the geometry it was also hard for me to distinguish between this and potentially a bell shaped lamp.
Lamp:

Predicted vase
The shape of this lamp, with a narrow bottom and a fat top does look a little similar to the overall shape/ geometry of a vase, which may mean that this overall distribution relationship between points (this shape) could be easier misled to a vase.
Q2. Segmentation Model¶
Test accuracy: 0.9040074554294976¶
Visualize segmentation results of at least 5 objects (including 2 bad predictions) with corresponding ground truth
Report the prediction accuracy for each object, and provide an interpretation of the result in a few sentences.
Segmentation results¶
Object 0:

Accuracy: 0.9562000036239624
Object 100:

Accuracy: 0.9567000269889832
Object 200:

Accuracy: 0.754800021648407
Object 500:

Accuracy: 0.7782999873161316
Object 600:

Accuracy: 0.9923999905586243
Overall I think the object is well segmented when it shows clear distinctions between their parts. For those chairs with clear structures, they generally perform
Comparing with the other 3 objects, I would say object 200 and object 500 are the two failed ones with accuracies of around 0.75 and 0.78. With the three better performing chairs, they are of cleaner shapes which would be easier to segment - with their legs, back part, seat and arm cleanly distinguishable. While the two objects with lower accuracy had overlapping regions (obj 200) or has unconventional shaped base (obj 500) that may make this segmentation task more difficult as the point. These cases when there is no clear boundary between parts the model shapes falls back to typical chair parts which performs less well for unconventional chair shapes.
Q3. Robustness Analysis¶
I experimented with different rotation and number of points as suggested in the readme.
Rotate the input points:¶
I tried rotating in each of the three axies, 30°, 60° and 90° separately. Q1/Q2 result are kept in 0° box in the table below.
Test accuracy¶
| task | 0° | x 30° | x 60° | x 90° | y 30° | y 60° | y 90° | z 30° | z 60° | z 90° |
|---|---|---|---|---|---|---|---|---|---|---|
| cls | 0.9769 | 0.7891 | 0.3610 | 0.2718 | 0.9297 | 0.7891 | 0.6915 | 0.8006 | 0.3841 | 0.3001 |
| seg | 0.9040 | 0.7406 | 0.4205 | 0.3034 | 0.8037 | 0.6693 | 0.6043 | 0.7241 | 0.5529 | 0.4718 |
Classification (same objects as Q1 chosen)¶
Object 0 (chair):
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| predicted | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | chair | chair | lamp | vase |
| y | chair | chair | lamp | lamp |
| z | chair | chair | vase | vase |
Object 300 (chair):
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| predicted | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | chair | chair | vase | vase |
| y | chair | chair | chair | chair |
| z | chair | chair | chair | vase |
Object 700 (vase):
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| predicted | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | vase | vase | vase | lamp |
| y | vase | vase | vase | vase |
| z | vase | vase | vase | lamp |
Object 800 (lamp):
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| predicted | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | lamp | lamp | chair | lamp |
| y | lamp | lamp | lamp | lamp |
| z | lamp | lamp | lamp | chair |
Segmentation (same objects as Q2, ground truth in Q2)¶
Object 0:
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| accuracy | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | 0.9562 | 0.8204 | 0.4763 | 0.3427 |
| y | 0.9562 | 0.9289 | 0.7856 | 0.7372 |
| z | 0.9562 | 0.8483 | 0.6625 | 0.4730 |
Object 100:
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| accuracy | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | 0.9567 | 0.7510 | 0.4274 | 0.2433 |
| y | 0.9567 | 0.8415 | 0.7009 | 0.6799 |
| z | 0.9567 | 0.7435 | 0.5047 | 0.3937 |
Object 200:
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| accuracy | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | 0.7548 | 0.6693 | 0.4789 | 0.2144 |
| y | 0.7548 | 0.6534 | 0.5547 | 0.3630 |
| z | 0.7548 | 0.6161 | 0.3862 | 0.3056 |
Object 500:
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| accuracy | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | 0.7783 | 0.7320 | 0.4411 | 0.3194 |
| y | 0.7783 | 0.7479 | 0.6772 | 0.6279 |
| z | 0.7783 | 0.7371 | 0.6977 | 0.5401 |
Object 600:
| axis | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | ![]() |
![]() |
![]() |
![]() |
| y | ![]() |
![]() |
![]() |
![]() |
| z | ![]() |
![]() |
![]() |
![]() |
| accuracy | 0° | 30° | 60° | 90° |
|---|---|---|---|---|
| x | 0.9924 | 0.7859 | 0.3627 | 0.3773 |
| y | 0.9924 | 0.9047 | 0.7552 | 0.6600 |
| z | 0.9924 | 0.7414 | 0.5244 | 0.6661 |
For classification in general, the higher the rotation angle, in all three axis, the lower the test accuracy. The robustness lasted upto around 30 degrees and 60 degrees onwards had a much greater drop comparing to 0 to 30 degrees rotation. This goes with prediction since the model is only trained on upright (0 degree) objects and has no transformation block to align the input, so a rotated object is just a different point distribution to the model. Small rotations still look close enough to the training data while larger ones do not.
An interesting observation is that the model is much more robust to rotations in the y axis comparing to the x and z axis (0.69 vs 0.27/ 0.30 at 90 degrees). y is the up axis here, so rotating about y only spins the object around and keeps how tall/ how fat the shape is, which is likely where most of the differentiating factors come from since what's trained are the 0 degree data. Rotating about x or z tips the object over and changes exactly that, e.g. chairs 0 and 300 are predicted as vase or lamp at 60-90 degrees about x.
Segmentation follows the same trend, with y again being the most robust. The part labels are closely tied to height (legs at the bottom, back at the top), so once the chair is tipped over the parts likely get labelled by where they are rather than what they are. This drop happens to all objects, for instance object 600 goes from 0.99 to 0.36 at 60 degrees about x.
Number of points per object:¶
I evaluated with 10000, 5000, 1000 and 100 points per object. Q1/Q2 result listed as 10000.
Test accuracy¶
| task | 10000 | 5000 | 1000 | 100 |
|---|---|---|---|---|
| cls | 0.9769 | 0.9769 | 0.9727 | 0.8615 |
| seg | 0.9040 | 0.9039 | 0.8978 | 0.8187 |
Classification (same objects as Q1)¶
| object | 10000 | 5000 | 1000 | 100 |
|---|---|---|---|---|
| 0 | ![]() |
![]() |
![]() |
![]() |
| 300 | ![]() |
![]() |
![]() |
![]() |
| 700 | ![]() |
![]() |
![]() |
![]() |
| 800 | ![]() |
![]() |
![]() |
![]() |
| predicted (true) | 10000 | 5000 | 1000 | 100 |
|---|---|---|---|---|
| 0 (chair) | chair | chair | chair | chair |
| 300 (chair) | chair | chair | chair | chair |
| 700 (vase) | vase | vase | vase | vase |
| 800 (lamp) | lamp | lamp | lamp | lamp |
Segmentation (same objects as Q2, ground truth in Q2)¶
| object | 10000 | 5000 | 1000 | 100 |
|---|---|---|---|---|
| 0 | ![]() |
![]() |
![]() |
![]() |
| 100 | ![]() |
![]() |
![]() |
![]() |
| 200 | ![]() |
![]() |
![]() |
![]() |
| 500 | ![]() |
![]() |
![]() |
![]() |
| 600 | ![]() |
![]() |
![]() |
![]() |
| accuracy | 10000 | 5000 | 1000 | 100 |
|---|---|---|---|---|
| 0 | 0.9562 | 0.9536 | 0.9520 | 0.9100 |
| 100 | 0.9567 | 0.9544 | 0.9450 | 0.9200 |
| 200 | 0.7548 | 0.7552 | 0.7190 | 0.4800 |
| 500 | 0.7783 | 0.7756 | 0.7960 | 0.7200 |
| 600 | 0.9924 | 0.9932 | 0.9960 | 0.9100 |
The model is quite robust to the number of points, for classification, the drop was barely noticeable up until only having 100 points. This makes sense since max pooling only keeps the max of each feature over all points, so as long as there are enough points to cover the overall shape, the global feature stays about the same.
Segmentations followed a similar trend, much less affected than how rotation changes things, likely for a similar reason as compared to classfication. For the per object accuracy, it generally stayed quite close to the accuracy at full scale. Only at 100 points when the shape gets too sparse, for some objects like object 200 it had a large drop, likely due to the fact that less points made it even harder to distinguish the borders between parts.




















































































































