A Unifying Review of Linear Gaussian Models
| Author | Sam Roweis and Zoubin Ghahramani |
|---|---|
| Published | 1999 |
| Format | Research paper |
| Original | Read the original |
TL;DR — Roweis and Ghahramani’s 1999 review showing that factor analysis, PCA, Gaussian mixtures, Kalman filters and hidden Markov models are all versions of one generative model.
The paper starts from one basic model: a hidden state that evolves by linear first-order Markov dynamics, with each observation a linear function of the state, and both processes corrupted by Gaussian noise. By varying its assumptions (static or dynamic data, continuous or discrete state, with or without a limit of zero noise), the authors derive factor analysis, principal component analysis, mixtures of Gaussian clusters, vector quantization, Kalman filter models (linear dynamical systems) and hidden Markov models. A simple nonlinearity, a “winner-take-all” step, links the discrete-state models to the continuous ones. Other nonlinearities bring in independent component analysis.
Along the way they gather derivations scattered across earlier papers, introduce “sensible PCA” (SPCA), a version of PCA with a proper probabilistic interpretation, and propose spatially adaptive observation noise. They also show that factor analysis and Gaussian mixtures can be built as autoencoder neural networks trained with squared error plus the same regularization term. The closing discussion argues that inference in every variant is the same probability propagation, that learning in every variant is the EM algorithm, and that questions such as choosing the number of clusters, factors or hidden states are really one question. An appendix gives pseudocode for inference and learning in each model.
The authors were at the Gatsby Computational Neuroscience Unit, University College London. This copy is the October 1998 preprint marked “In Press” for Neural Computation, Vol. 11, No. 2 (1999), and runs to 34 pages.
What it is: A peer-reviewed journal paper and a standard reference for seeing classical unsupervised learning methods as one family. It is a review and synthesis, not a report of new experiments. The archived PDF has no usable text layer, so it reads as images.
Where this came from
- Will Manidis’s “most underrated PDF” thread, July 2023 — routing record: The Most Underrated PDF on the Internet (July 2023)
34 pages. A copy is archived locally against link rot; the header links the original source.