Hyojae Park
Test accuracy: 0.9769150052465897
Successful examples:
Chair:
Vase:
Lamp:
Failure examples:
Chair incorrectly predicted as lamp:
Vase incorrectly predicted as lamp:
Lamp incorrectly predicted as vase:
The failure cases show extreme examples of each object type where its geometry can confuse the model because of exotic features that are not usually found in that object. For instance, the chair example shows a chair that is wavy and long, unlike most chairs. In addition, the vase contains a flower that is resting on one of its ends, but this long stem resembles a desk lamp, confusing the model. Lastly, the lamp example shows that tall lamp that contains multiple bulbs, which the model may confuse as a flower.
On the other hand, successful examples exhibit geometry that can be easily attributed to each object (ex: chair has legs), helping the model be accurate.
Test accuracy: 0.9031752025931928
In the following figures, the left is the ground truth, and right is the prediction.
Good prediction #1 (accuracy: 0.992):
Good prediction #2 (accuracy: 0.984):
Good prediction #3 (accuracy: 0.9875):
Bad prediction #1 (accuracy: 0.5023):
Bad prediction #2 (accuracy: 0.5418):
As evident by these examples, the model performs extremely well when the object is composed of clearly separated components (legs, seat, back, arm rest). However, for more complicated examples such as couches, which contain up to 6 segments, the model does not do well. For instance, in the second bad prediction, it is not clear that the dark blue segment (the bottom of the chair) and the red segment (the seat) should be separate components. Therefore, we see that our model treats it as one larger segment, which is incorrect. This vagueness in couches make their prediction less accurate to other chairs which have components that are clearly distinct.
I first evaluated the trained models by giving it only a 100 points per object (as opposed to 10,000), hence only 1% of the information.
I collected the following results:
Classification accuracy with 100 points per object: 0.9328436516264428
Example positive result (left is 100 points, right is 10,000):
Example negative result (lamp classified as chair):
Overall, reducing the number of points did not significantly destroy the performance of the classification model, even when the number of points decreased dramatically. For instance, the chair example, although large portions of its legs is missing in the 100 points example, was still categorized correctly.
In the negative result, the model failed when given only a 100 points, and incorrectly categorized a lamp as a chair. This shows that as the number of points decreases, it also diminishes key features that distinguish the object category from others. In this example, it may be that the lack of points at the bottom of the lamp makes the base look like legs of a chair, leading to an incorrect result.
The segmentation model also showed similar results:
Segmentation accuracy with 100 points per object: 0.8318962722852512
Example successful segmentation result (>0.99 accuracy for both 100 and 10,000 points examples):
Example failed segmentation result (0.37 accuracy):
Similar to the classification model, the segmentation model still performs relatively well even when given a small number of points. This is especially true for objects that the model did extremely well in (in the 10,000 case), indicating potential signs of overfitting. This is seen with the example successful result above, which succesfully segments the chair with four different components. However, for difficult tasks such as couches, it fails significantly, with an accuracy of 0.37.
Next, I rotated each object by 180 degrees (flipped upside down) before evaluating the model. The following are the results:
Classification accuracy after rotating each object: 0.621196222455404
Succesful result:
Failed result:
The classification accuracy significantly dipped, even when the number of points remained 10,000. This indicates that the learned model is not rotation invariant, highlighting the need for transformation blocks.
The figures above show examples that, when flipped, do not have the features of the original object in the same location. For instance, flipping vase with a flower now has the complex features of the flower at the bottom of the object, resulting in incorrect results.
Segmentation accuracy after rotating each object: 0.34476353322528364
The segmentation accuracy dropped even more than the classification accuracy. As evident by the example above, which now has an accuracy of 0.358 (from the previous 0.984), we see that our model still attempts to maintain the height ordering of each segment (light blue at top, red in middle, and blue at bottom), leading to significantly incorrect results.