CMU 16-825 Learning for 3D Vision

Assignment 5:
PointNet: Classification and Segmentation

Sichen Guo

Task 1 /

Classification Model

1.1   Test Accuracy

Test accuracy: 98.11% (935 / 953 objects), using 10,000 points per object and the best classification checkpoint.

1.2   Random Test Point Clouds

Six randomly selected test objects are shown below.

Test #621: ground truth vase; prediction vase.
Test #621: ground truth vase; prediction vase.
Test #734: ground truth lamp; prediction lamp.
Test #734: ground truth lamp; prediction lamp.
Test #412: ground truth chair; prediction chair.
Test #412: ground truth chair; prediction chair.
Test #818: ground truth lamp; prediction lamp.
Test #818: ground truth lamp; prediction lamp.
Test #417: ground truth chair; prediction chair.
Test #417: ground truth chair; prediction chair.
Test #84: ground truth chair; prediction chair.
Test #84: ground truth chair; prediction chair.

1.3   Failure Predictions

Test #716: vase → lamp (81.9% confidence).
Test #716: vase → lamp (81.9% confidence).
Test #883: lamp → vase (99.2% confidence).
Test #883: lamp → vase (99.2% confidence).

Vase → lamp, test #716: This vase has a large tapered body above a narrow stem and a stepped base, giving it a silhouette similar to a table lamp. The visible geometry provides no clear cue about whether the upper body is a container or a lampshade. This shape ambiguity may explain the incorrect lamp prediction.

Lamp → vase, test #883: This lamp has a broad, rounded body on a short base, without an obvious long stem or separate shade. Its compact, vessel-like shape resembles a vase, which may explain the confusion. The model assigns 99.2% confidence to the wrong class, showing that high confidence does not guarantee a correct prediction on an ambiguous shape.

Chair: All 617 chair test objects are classified correctly, so the unmodified test set contains no chair failure to visualize.

Task 2 /

Segmentation Model

2.1   Test Accuracy

Test point accuracy: 89.85%, evaluated on all 617 test chairs with 10,000 points per object.

2.2   Three Best Predictions

The three highest-accuracy objects are shown below. Ground truth is on the left and prediction on the right, using the same camera sequence and class-color mapping.

Test #471 — Accuracy: 99.71%

Test #471: ground-truth segmentation.
Test #471: ground-truth segmentation.
Test #471: predicted segmentation; point accuracy 99.71%.
Test #471: predicted segmentation; point accuracy 99.71%.

Test #562 — Accuracy: 99.40%

Test #562: ground-truth segmentation.
Test #562: ground-truth segmentation.
Test #562: predicted segmentation; point accuracy 99.40%.
Test #562: predicted segmentation; point accuracy 99.40%.

Test #569 — Accuracy: 99.37%

Test #569: ground-truth segmentation.
Test #569: ground-truth segmentation.
Test #569: predicted segmentation; point accuracy 99.37%.
Test #569: predicted segmentation; point accuracy 99.37%.

2.3   Two Worst Predictions

The two lowest-accuracy objects illustrate substantial segmentation failures. Their results are compared with the corresponding ground truth using the same views.

Test #255 — Accuracy: 39.36%

Test #255: ground-truth segmentation.
Test #255: ground-truth segmentation.
Test #255: predicted segmentation; point accuracy 39.36%.
Test #255: predicted segmentation; point accuracy 39.36%.

Test #351 — Accuracy: 44.55%

Test #351: ground-truth segmentation.
Test #351: ground-truth segmentation.
Test #351: predicted segmentation; point accuracy 44.55%.
Test #351: predicted segmentation; point accuracy 44.55%.

Interpretation: The successful cases (#471, #562, and #569) have clearly separated backs, seats, and legs, and their predicted labels closely match the ground truth, with accuracies above 99%. In contrast, the compact, box-like chair #255 and rounded chair #351 have continuous surfaces and less distinct part boundaries, leading to substantial confusion between semantic regions. For example, much of the lower blue region in #255 is predicted as red, while #351 is divided into broad upper and lower regions that do not match the ground truth. These results suggest that the model handles conventional chair structures well but struggles with unusual shapes whose parts blend together.

Task 3 /

Robustness Analysis

3.1   Experiment 1: Input Rotation

Classification

Procedure: I rotate each test point cloud by 30°, 60°, or 90° about the Y axis through its own centroid, retaining all 10,000 points. I evaluate the same trained classification model on all 953 test objects without retraining. The unrotated input is the Q1 baseline. Each static image places the original input on the left and the actual rotated input on the right, with the same fixed front camera and scale. The right point cloud retains its 30°, 60°, or 90° input rotation.

0° — Q1 baseline:
Test accuracy: 98.11%

30° rotation:
Test accuracy: 74.19% (23.92 percentage points drop)

60° rotation:
Test accuracy: 61.91% (36.20 percentage points drop)

90° rotation:
Test accuracy: 63.27% (34.84 percentage points drop)

30° rotation

Test #0 (ground truth chair): left, baseline → chair; right, 30° rotation → vase. Baseline correct → perturbed wrong.
Test #0 (ground truth chair): left, baseline → chair; right, 30° rotation → vase. Baseline correct → perturbed wrong.
Test #707 (ground truth vase): left, baseline → vase; right, 30° rotation → chair. Baseline correct → perturbed wrong.
Test #707 (ground truth vase): left, baseline → vase; right, 30° rotation → chair. Baseline correct → perturbed wrong.
Test #734 (ground truth lamp): left, baseline → lamp; right, 30° rotation → vase. Baseline correct → perturbed wrong.
Test #734 (ground truth lamp): left, baseline → lamp; right, 30° rotation → vase. Baseline correct → perturbed wrong.

60° rotation

Test #0 (ground truth chair): left, baseline → chair; right, 60° rotation → vase. Baseline correct → perturbed wrong.
Test #0 (ground truth chair): left, baseline → chair; right, 60° rotation → vase. Baseline correct → perturbed wrong.
Test #663 (ground truth vase): left, baseline → vase; right, 60° rotation → lamp. Baseline correct → perturbed wrong.
Test #663 (ground truth vase): left, baseline → vase; right, 60° rotation → lamp. Baseline correct → perturbed wrong.
Test #734 (ground truth lamp): left, baseline → lamp; right, 60° rotation → vase. Baseline correct → perturbed wrong.
Test #734 (ground truth lamp): left, baseline → lamp; right, 60° rotation → vase. Baseline correct → perturbed wrong.

90° rotation

Test #0 (ground truth chair): left, baseline → chair; right, 90° rotation → vase. Baseline correct → perturbed wrong.
Test #0 (ground truth chair): left, baseline → chair; right, 90° rotation → vase. Baseline correct → perturbed wrong.
Test #619 (ground truth vase): left, baseline → vase; right, 90° rotation → lamp. Baseline correct → perturbed wrong.
Test #619 (ground truth vase): left, baseline → vase; right, 90° rotation → lamp. Baseline correct → perturbed wrong.
Test #724 (ground truth lamp): left, baseline → lamp; right, 90° rotation → vase. Baseline correct → perturbed wrong.
Test #724 (ground truth lamp): left, baseline → lamp; right, 90° rotation → vase. Baseline correct → perturbed wrong.

Interpretation: Accuracy falls from 98.11% to 74.19%, 61.91%, and 63.27% at 30°, 60°, and 90°, respectively. PointNet's shared pointwise features and max pooling provide invariance to point ordering, but do not inherently provide invariance to 3D rotation. These results show strong sensitivity to object orientation; the decrease is not strictly monotonic, since 90° performs slightly better than 60°.

Sample comparison: All nine displayed rotation comparisons are baseline-correct objects that become incorrect. Chair #0 changes from chair to vase at all three angles; the object retains its chair structure, while only its orientation changes. At the dataset level, chair accuracy falls from 100% to 46.35% at 60°, while vase and lamp accuracies remain 90.20% and 90.60%, respectively, so chairs account for much of the overall degradation. One possible reason is that a chair’s appearance changes substantially across viewing angles. In particular, side or rear views may obscure its characteristic features, leading to misclassification.

Segmentation

Procedure: I evaluate the Q2 segmentation checkpoint on all 617 test chairs after rotating each point cloud by 30°, 60°, or 90° about the Y axis through its centroid. For each angle, I choose three objects with baseline accuracy at least 90% that exhibit large accuracy drops.

0° — Q2 baseline:
Test point accuracy: 89.85%

30° rotation:
Test point accuracy: 76.31% (13.54 percentage points drop)

60° rotation:
Test point accuracy: 67.81% (22.05 percentage points drop)

90° rotation:
Test point accuracy: 62.47% (27.38 percentage points drop)

30° rotation — segmentation

Test #224: baseline 95.16%; perturbed 51.49%; same-point drop 43.67 percentage points.
Test #224: baseline 95.16%; perturbed 51.49%; same-point drop 43.67 percentage points.
Test #474: baseline 94.84%; perturbed 52.42%; same-point drop 42.42 percentage points.
Test #474: baseline 94.84%; perturbed 52.42%; same-point drop 42.42 percentage points.
Test #447: baseline 96.34%; perturbed 54.94%; same-point drop 41.40 percentage points.
Test #447: baseline 96.34%; perturbed 54.94%; same-point drop 41.40 percentage points.

60° rotation — segmentation

Test #541: baseline 90.52%; perturbed 26.26%; same-point drop 64.26 percentage points.
Test #541: baseline 90.52%; perturbed 26.26%; same-point drop 64.26 percentage points.
Test #389: baseline 94.58%; perturbed 36.36%; same-point drop 58.22 percentage points.
Test #389: baseline 94.58%; perturbed 36.36%; same-point drop 58.22 percentage points.
Test #216: baseline 95.28%; perturbed 37.07%; same-point drop 58.21 percentage points.
Test #216: baseline 95.28%; perturbed 37.07%; same-point drop 58.21 percentage points.

90° rotation — segmentation

Test #169: baseline 96.41%; perturbed 24.13%; same-point drop 72.28 percentage points.
Test #169: baseline 96.41%; perturbed 24.13%; same-point drop 72.28 percentage points.
Test #389: baseline 94.58%; perturbed 24.03%; same-point drop 70.55 percentage points.
Test #389: baseline 94.58%; perturbed 24.03%; same-point drop 70.55 percentage points.
Test #462: baseline 98.54%; perturbed 28.39%; same-point drop 70.15 percentage points.
Test #462: baseline 98.54%; perturbed 28.39%; same-point drop 70.15 percentage points.

Interpretation: Test point accuracy decreases from 89.85% to 76.31%, 67.81%, and 62.47% as the rotation angle increases. For these settings, both classification and segmentation are sensitive to orientation. Shared pointwise features and max pooling do not make PointNet inherently invariant to spatial rotation, so changing coordinates can alter the predicted part labels even though the object geometry is preserved.

Sample comparison: Test #224 drops from 95.16% to 51.49% at 30°, with large regions no longer matching the ground-truth colors. At 90°, Test #169 drops from 96.41% to 24.13%; much of its cyan back region is predicted as blue, despite the unchanged semantic structure. The errors extend beyond narrow part boundaries, indicating substantial part-label confusion.

3.2   Experiment 2: Number of Input Points

Classification

Procedure: I randomly sample 5,000, 1,000, or 500 points per test object without replacement, leaving the original orientation unchanged.

10,000 points — Q1 baseline:
Test accuracy: 98.11%

5,000 points:
Test accuracy: 98.01% (0.10 percentage points drop)

1,000 points:
Test accuracy: 97.80% (0.31 percentage points drop)

500 points:
Test accuracy: 97.27% (0.84 percentage points drop)

5,000 points

Test #703 (ground truth vase): left, baseline → vase; right, 5,000 points → lamp. Baseline correct → perturbed wrong.
Test #703 (ground truth vase): left, baseline → vase; right, 5,000 points → lamp. Baseline correct → perturbed wrong.
Test #55 (ground truth chair): left, baseline → chair; right, 5,000 points → chair.
Test #55 (ground truth chair): left, baseline → chair; right, 5,000 points → chair. .
Test #900 (ground truth lamp): left, baseline → lamp; right, 5,000 points → lamp.
Test #900 (ground truth lamp): left, baseline → lamp; right, 5,000 points → lamp. .

1,000 points

Test #406 (ground truth chair): left, baseline → chair; right, 1,000 points → lamp. Baseline correct → perturbed wrong.
Test #406 (ground truth chair): left, baseline → chair; right, 1,000 points → lamp. Baseline correct → perturbed wrong.
Test #687 (ground truth vase): left, baseline → vase; right, 1,000 points → lamp. Baseline correct → perturbed wrong.
Test #687 (ground truth vase): left, baseline → vase; right, 1,000 points → lamp. Baseline correct → perturbed wrong.
Test #839 (ground truth lamp): left, baseline → lamp; right, 1,000 points → vase. Baseline correct → perturbed wrong.
Test #839 (ground truth lamp): left, baseline → lamp; right, 1,000 points → vase. Baseline correct → perturbed wrong.

500 points

Test #335 (ground truth chair): left, baseline → chair; right, 500 points → vase. Baseline correct → perturbed wrong.
Test #335 (ground truth chair): left, baseline → chair; right, 500 points → vase. Baseline correct → perturbed wrong.
Test #645 (ground truth vase): left, baseline → vase; right, 500 points → lamp. Baseline correct → perturbed wrong.
Test #645 (ground truth vase): left, baseline → vase; right, 500 points → lamp. Baseline correct → perturbed wrong.
Test #796 (ground truth lamp): left, baseline → lamp; right, 500 points → vase. Baseline correct → perturbed wrong.
Test #796 (ground truth lamp): left, baseline → lamp; right, 500 points → vase. Baseline correct → perturbed wrong.

Interpretation: Accuracy is 98.01% with 5,000 points, 97.80% with 1,000 points, and 97.27% with 500 points. Even after removing 95% of the points, accuracy decreases by only 0.84 percentage points. For these tested settings, the classifier is much more robust to random point removal than to rotation. The remaining samples appear sufficient to preserve useful global shape evidence, although sparse sampling can miss small or distinctive structures.

Sample comparison: At 5,000 points, only one originally correct object becomes incorrect: vase #703 is predicted as a lamp. As the number of input points decreases, prediction accuracy declines slightly but remains high overall. This may be due to the model’s ability to extract global features: even with fewer points, it can still capture the object’s overall shape and maintain high classification accuracy. However, in examples such as Test #335 and Test #645, fine structures such as chair legs and flower stems become discontinuous or lose detail when the point cloud is sparse, which may impair classification performance.

Segmentation

Procedure: I evaluate the same Q2 model with 5,000, 1,000, or 500 randomly sampled points per chair.

Baseline on retained points: This measures the full-input model’s accuracy only on the points retained after subsampling.

10,000 points — Q2 baseline:
Test point accuracy: 89.85%

5,000 points:
Test point accuracy: 89.89% (0.04 percentage points improvement)

1,000 points:
Test point accuracy: 89.75% (0.10 percentage points drop)

500 points:
Test point accuracy: 89.02% (0.84 percentage points drop)

5,000 points — segmentation

Test #112: baseline 91.71%; baseline on retained points 91.80%; perturbed 89.90%; same-point drop 1.90 percentage points.
Test #112: baseline 91.71%; baseline on retained points 91.80%; perturbed 89.90%; same-point drop 1.90 percentage points.
Test #473: baseline 90.78%; baseline on retained points 90.56%; perturbed 89.44%; same-point drop 1.12 percentage points.
Test #473: baseline 90.78%; baseline on retained points 90.56%; perturbed 89.44%; same-point drop 1.12 percentage points.
Test #67: baseline 93.15%; baseline on retained points 92.98%; perturbed 91.94%; same-point drop 1.04 percentage points.
Test #67: baseline 93.15%; baseline on retained points 92.98%; perturbed 91.94%; same-point drop 1.04 percentage points.

1,000 points — segmentation

Test #576: baseline 94.49%; baseline on retained points 94.30%; perturbed 81.50%; same-point drop 12.80 percentage points.
Test #576: baseline 94.49%; baseline on retained points 94.30%; perturbed 81.50%; same-point drop 12.80 percentage points.
Test #480: baseline 92.24%; baseline on retained points 91.20%; perturbed 85.40%; same-point drop 5.80 percentage points.
Test #480: baseline 92.24%; baseline on retained points 91.20%; perturbed 85.40%; same-point drop 5.80 percentage points.
Test #112: baseline 91.71%; baseline on retained points 91.30%; perturbed 85.90%; same-point drop 5.40 percentage points.
Test #112: baseline 91.71%; baseline on retained points 91.30%; perturbed 85.90%; same-point drop 5.40 percentage points.

500 points — segmentation

Test #576: baseline 94.49%; baseline on retained points 94.80%; perturbed 62.20%; same-point drop 32.60 percentage points.
Test #576: baseline 94.49%; baseline on retained points 94.80%; perturbed 62.20%; same-point drop 32.60 percentage points.
Test #541: baseline 90.52%; baseline on retained points 91.40%; perturbed 71.40%; same-point drop 20.00 percentage points.
Test #541: baseline 90.52%; baseline on retained points 91.40%; perturbed 71.40%; same-point drop 20.00 percentage points.
Test #339: baseline 95.31%; baseline on retained points 94.00%; perturbed 78.00%; same-point drop 16.00 percentage points.
Test #339: baseline 95.31%; baseline on retained points 94.00%; perturbed 78.00%; same-point drop 16.00 percentage points.

Interpretation: Accuracy is 89.89%, 89.75%, and 89.02% at 5,000, 1,000, and 500 points, respectively. The small improvement at 5,000 points is only 0.04 percentage points, while removing 95% of the input points reduces overall accuracy by 0.84 percentage points. As the same in classification, the model is much more robust to random point removal than to the tested rotations.

Sample comparison: Individual objects can deteriorate much more than the dataset average suggests. For Test #576, baseline accuracy on the retained points is 94.30% at 1,000 points and 94.80% at 500 points, but sparse-input prediction reaches only 81.50% and 62.20%, respectively. Missing samples may weaken global context and evidence around small structures or part boundaries in certain cases.