/ Machine Learning - Gaussian Mixture Model
Machine Learning - Gaussian Mixture Model¶
Gaussian Mixture Models (GMM) are probabilistic generative models that assume data is drawn from a mixture of K multivariate Gaussian distributions. Each distribution, called a component, has its own mean, covariance, and mixing weight. GMM uses the Expectation-Maximization (EM) algorithm to learn these parameters from data.
This notebook covers:
- Definition and background (NIST/SEMATECH e-Handbook)
- How GMM works — key equations and the EM algorithm
- Strong modelling assumptions and when they break
- Real-world example: classification on the Breast Cancer Wisconsin dataset
- Evaluation and interpretation of results
Definition¶
According to the NIST/SEMATECH e-Handbook of Statistical Methods:
"A finite mixture model assumes that the population consists of a finite number of sub-populations (components), and that an observation from the population is an observation from one of the sub-populations. When the component distributions are all multivariate normal, the model is called a Gaussian mixture model."
— NIST/SEMATECH e-Handbook of Statistical Methods, Section on Cluster Analysis
In a GMM with K components, the probability density of an observation x is:
$$p(\mathbf{x}) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)$$
where:
- $\pi_k$ is the mixing weight of component $k$ (probability that an observation belongs to $k$), with $\sum_k \pi_k = 1$
- $\mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)$ is the multivariate Gaussian with mean $\boldsymbol{\mu}_k$ and covariance $\boldsymbol{\Sigma}_k$
How GMM Works¶
Soft Assignment (vs. K-Means Hard Assignment)¶
K-Means assigns each point to exactly one cluster. GMM instead computes a responsibility $r_{nk}$ — the probability that observation $\mathbf{x}_n$ was generated by component $k$:
$$r_{nk} = \frac{\pi_k \, \mathcal{N}(\mathbf{x}_n \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)}{\sum_{j=1}^{K} \pi_j \, \mathcal{N}(\mathbf{x}_n \mid \boldsymbol{\mu}_j, \boldsymbol{\Sigma}_j)}$$
A point on the boundary between two clusters gets roughly equal responsibility for both, rather than being forced into one.
The EM Algorithm¶
GMM parameters $(\pi_k, \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)$ are found by Expectation-Maximization:
- E-step (Expectation): Using current parameter estimates, compute responsibilities $r_{nk}$ for every observation and component.
- M-step (Maximization): Re-estimate parameters to maximize the expected log-likelihood:
- $\hat{\pi}_k = \frac{1}{N}\sum_n r_{nk}$ (effective fraction of points in component $k$)
- $\hat{\boldsymbol{\mu}}_k = \frac{\sum_n r_{nk} \mathbf{x}_n}{\sum_n r_{nk}}$ (responsibility-weighted mean)
- $\hat{\boldsymbol{\Sigma}}_k$ = responsibility-weighted covariance around $\hat{\boldsymbol{\mu}}_k$
Repeat until the log-likelihood stops increasing. EM is guaranteed to increase the log-likelihood
at every step but may converge to a local optimum — always use multiple random restarts (n_init > 1).
Covariance Types in scikit-learn¶
GaussianMixture(covariance_type=...) controls the shape of each component's covariance matrix:
| Type | Shape | Meaning |
|---|---|---|
'full' |
Arbitrary ellipsoid | Each component has its own unconstrained covariance |
'tied' |
Shared ellipsoid | All components share one covariance matrix |
'diag' |
Axis-aligned ellipsoid | Each component has its own diagonal covariance |
'spherical' |
Circle / sphere | Each component has a single scalar variance |
For most real datasets, 'full' is the safest default. Use 'diag' or 'spherical' when the
number of samples per component is small relative to the number of features.
Strong Assumptions¶
GMM is a powerful model, but it rests on several assumptions that are frequently violated in practice. Understanding them prevents misuse.
1. Data is generated by exactly K Gaussian components. The number of components K must be specified in advance. In practice K is unknown, so it is selected by minimising AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion) over a range of values. Choosing K too small merges distinct groups; too large splits coherent ones.
2. Each component follows a multivariate Gaussian distribution. Real clusters can be skewed, heavy-tailed, ring-shaped, or lie on a curved manifold — none of which a Gaussian can represent faithfully. When shapes deviate strongly from ellipsoidal, GMM will produce misleading fits and responsibilities.
3. The covariance structure matches the chosen type.
Using covariance_type='spherical' on elongated clusters forces circular component shapes,
distorting cluster boundaries. Choosing a covariance type that is more constrained than the
true data geometry introduces model misspecification bias.
4. Samples are independent and identically distributed (i.i.d.). GMM assumes each observation is drawn independently from the same mixture distribution. Time-series data (temporal autocorrelation), spatial data (neighbourhood correlation), or repeated measures on the same subject all violate this assumption and can inflate apparent cluster certainty.
5. EM converges to the global optimum.
EM is a local optimisation algorithm. It is sensitive to initialisation and can converge to
a poor local maximum of the log-likelihood, especially when components overlap or data are
high-dimensional. Mitigate by setting n_init to run multiple random restarts and keep the
best result.
6. The mixture is identifiable.
If two components are nearly identical (overlapping means and covariances), the model is
unidentifiable — the same likelihood is achieved by swapping or merging them. Regularise
with reg_covar to prevent degenerate (near-zero-determinant) covariance matrices.
Example: Breast Cancer Wisconsin Dataset¶
The Breast Cancer Wisconsin dataset contains 569 samples with 30 numeric features computed from digitised images of fine-needle aspirate (FNA) of breast masses. Each sample is labelled malignant (212 samples) or benign (357 samples).
We apply GMM with two components (matching the two known classes) and evaluate how well the unsupervised clustering recovers the true labels. Because the dataset is 30-dimensional, we project to 2D via PCA for visualisation only — the GMM is fitted on all 30 standardised features.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_breast_cancer
from sklearn.mixture import GaussianMixture
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (
adjusted_rand_score,
normalized_mutual_info_score,
classification_report,
)
sns.set_theme(style="whitegrid")
Step 1 — Load and Inspect the Data¶
data = load_breast_cancer(as_frame=True)
df = data.frame
df["target"] = data.target
df["label"] = df["target"].map({0: "malignant", 1: "benign"})
print("Shape:", df.shape)
print("\nClass distribution:")
print(df["label"].value_counts())
print("\nFirst 5 rows (selected features):")
df[["mean radius", "mean texture", "mean perimeter", "mean area", "label"]].head()
Shape: (569, 32) Class distribution: label benign 357 malignant 212 Name: count, dtype: int64 First 5 rows (selected features):
| mean radius | mean texture | mean perimeter | mean area | label | |
|---|---|---|---|---|---|
| 0 | 17.99 | 10.38 | 122.80 | 1001.0 | malignant |
| 1 | 20.57 | 17.77 | 132.90 | 1326.0 | malignant |
| 2 | 19.69 | 21.25 | 130.00 | 1203.0 | malignant |
| 3 | 11.42 | 20.38 | 77.58 | 386.1 | malignant |
| 4 | 20.29 | 14.34 | 135.10 | 1297.0 | malignant |
Step 2 — Standardise Features and Project to 2D¶
GMM is sensitive to feature scale. We standardise all 30 features to zero mean and unit variance before fitting. The PCA projection to 2D is used only for plotting — the GMM is fitted on the full 30-dimensional standardised data.
feature_cols = data.feature_names.tolist()
X = df[feature_cols].values
y_true = df["target"].values
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
pca2 = PCA(n_components=2, random_state=42)
X_pca = pca2.fit_transform(X_scaled)
explained = pca2.explained_variance_ratio_
print(f"PC1 explains {explained[0]:.1%} of variance")
print(f"PC2 explains {explained[1]:.1%} of variance")
print(f"Total (2 PCs): {explained.sum():.1%}")
PC1 explains 44.3% of variance PC2 explains 19.0% of variance Total (2 PCs): 63.2%
Step 3 — Fit the Gaussian Mixture Model¶
We fit a GMM with two components (n_components=2) using full covariance matrices.
n_init=10 runs EM from 10 different random initialisations and keeps the best result,
reducing sensitivity to local optima.
gmm = GaussianMixture(
n_components=2,
covariance_type="full",
n_init=10,
random_state=42,
)
gmm.fit(X_scaled)
y_pred = gmm.predict(X_scaled)
y_prob = gmm.predict_proba(X_scaled)
print(f"Converged: {gmm.converged_}")
print(f"Iterations: {gmm.n_iter_}")
print(f"Log-likelihood: {gmm.lower_bound_:.4f}")
sizes = {int(k): int(v) for k, v in zip(*np.unique(y_pred, return_counts=True))}
print(f"\nPredicted component sizes: {sizes}")
print(f"Mixing weights (π): {gmm.weights_.round(3)}")
Converged: True
Iterations: 8
Log-likelihood: 0.1693
Predicted component sizes: {0: 214, 1: 355}
Mixing weights (π): [0.376 0.624]
Step 4 — Evaluate Against True Labels¶
GMM is unsupervised — it assigns component labels 0 and 1 without knowing which corresponds to malignant or benign. We align predicted labels to true labels via majority vote: whichever true class is most common in each predicted component becomes that component's assigned label.
Metrics:
- Adjusted Rand Index (ARI): agreement between predicted and true labels, corrected for chance. Range [−1, 1]; 1 = perfect.
- Normalized Mutual Information (NMI): shared information between predictions and true labels. Range [0, 1]; 1 = perfect.
- Classification report: precision, recall, F1 per class after label alignment.
# Majority-vote label alignment
def align_labels(y_true, y_pred):
"""Map each predicted component to the majority true label within it."""
mapping = {}
for component in np.unique(y_pred):
mask = y_pred == component
majority = np.bincount(y_true[mask]).argmax()
mapping[component] = majority
return np.array([mapping[p] for p in y_pred])
y_aligned = align_labels(y_true, y_pred)
ari = adjusted_rand_score(y_true, y_pred)
nmi = normalized_mutual_info_score(y_true, y_pred)
print(f"Adjusted Rand Index (ARI): {ari:.4f}")
print(f"Normalized Mutual Information (NMI): {nmi:.4f}")
print()
print(classification_report(
y_true, y_aligned,
target_names=data.target_names,
))
Adjusted Rand Index (ARI): 0.7740
Normalized Mutual Information (NMI): 0.6611
precision recall f1-score support
malignant 0.92 0.92 0.92 212
benign 0.95 0.95 0.95 357
accuracy 0.94 569
macro avg 0.94 0.94 0.94 569
weighted avg 0.94 0.94 0.94 569
Step 5 — Visualise GMM Components in 2D¶
We project the 30-dimensional data to 2D via PCA and colour points by their GMM-predicted component. The true class boundary is shown as marker shape (circle = benign, cross = malignant) to reveal where the model agrees with or differs from the ground truth.
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Left panel: GMM predicted components
palette = {0: "#E07B54", 1: "#5B8DB8"}
component_names = {0: "Component 0", 1: "Component 1"}
for comp in [0, 1]:
mask = y_pred == comp
axes[0].scatter(
X_pca[mask, 0], X_pca[mask, 1],
c=palette[comp], label=component_names[comp],
alpha=0.6, edgecolors="white", linewidths=0.3, s=40,
)
axes[0].set_title("GMM Predicted Components")
axes[0].set_xlabel(f"PC1 ({explained[0]:.1%} variance)")
axes[0].set_ylabel(f"PC2 ({explained[1]:.1%} variance)")
axes[0].legend()
# Right panel: True labels
true_palette = {0: "#C0392B", 1: "#27AE60"}
true_names = {0: "malignant", 1: "benign"}
for cls in [0, 1]:
mask = y_true == cls
axes[1].scatter(
X_pca[mask, 0], X_pca[mask, 1],
c=true_palette[cls], label=true_names[cls],
alpha=0.6, edgecolors="white", linewidths=0.3, s=40,
)
axes[1].set_title("True Labels")
axes[1].set_xlabel(f"PC1 ({explained[0]:.1%} variance)")
axes[1].set_ylabel(f"PC2 ({explained[1]:.1%} variance)")
axes[1].legend()
fig.suptitle(
"Breast Cancer Wisconsin — GMM Components vs. True Labels (PCA 2D projection)",
fontsize=13, y=1.01,
)
plt.tight_layout()
plt.show()
Summary¶
| Step | Action | Result |
|---|---|---|
| 1 | Loaded Breast Cancer Wisconsin dataset | 569 samples × 30 features; 357 benign, 212 malignant |
| 2 | Standardised features with StandardScaler |
All features on equal scale before GMM fitting |
| 3 | Projected to 2D with PCA (visualisation only) | PC1+PC2 explain ~63% of variance |
| 4 | Fitted GaussianMixture(n_components=2, n_init=10) |
EM converged; two components with distinct mixing weights |
| 5 | Evaluated with ARI and NMI | Both metrics > 0.5 — clusters partially recover true classes |
| 6 | Visualised components in PCA space | Side-by-side plot shows alignment between GMM and true labels |
GMM worked reasonably here because:
- The two classes (malignant / benign) have different distributional profiles across the 30 features.
- Full covariance matrices capture the different within-class feature correlations.
Limitations observed:
- Some malignant / benign samples are interleaved in PCA space — GMM cannot perfectly separate them without supervision.
- ARI and NMI less than 1.0 reflect the assumption violations: the two classes are not perfectly Gaussian, and K=2 is imposed rather than selected by AIC/BIC.
© 2026 Ivan Cao-Berg Pittsburgh Supercomputing Center, Carnegie Mellon University
Licensed under the GNU General Public License v2.0 (GPL-2).
You may redistribute and/or modify this work under the terms of GPL-2.
This work is distributed WITHOUT ANY WARRANTY.
Happy computing.