Mixture model

From Wikipedia, the free encyclopedia

In statistics, a mixture model is a probabilistic model for density estimation using a mixture distribution. A mixture model can be regarded as a type of unsupervised learning or clustering.

[edit] Examples

The normal distribution is plotted using different means and variances

Suppose researchers are trying to find the optimal mixture of ingredients for a fruit punch consisting of grape juice, mango juice, and pineapple juice. A mixture model ^[1] is suitable here because the results of the taste tests will not depend on the amount of ingredients used to make the batch but rather on the fraction of each ingredient present in the punch. The components always sum to a whole, which a mixture model takes into account.

As another example, financial returns often behave differently in normal situations and during crisis times. A mixture model for return data seems reasonable. Some model this as a jump-diffusion model, or as a mixture of two normal distributions.

[edit] Direct and indirect applications of mixture models

The financial example above is one direct application of the mixture model, a situation in which we assume an underlying mechanism so that each observation belongs to one of some number of different sources or categories. This underlying mechanism may or may not, however, be observable. In this form of mixture, each of the sources is described by a component probability density function, and its mixture weight is the probability that an observation comes from this component.

In an indirect application of the mixture model we do not assume such a mechanism. The mixture model is simply used for its mathematical flexibilities. For example, a mixture of two normal distributions with different means may result in a density with two modes, which is not modeled by standard parametric distributions. Another example is given by the possibility of mixture distributions to model fatter tails than the basic Gaussian ones, so as to be a candidate for modeling more extreme events. When combined with dynamical consistency, this approach has been applied to financial derivatives valuation in presence of the volatility smile in the context of local volatility models.

[edit] Types of mixture model

Main article: Mixture density

[edit] Probability mixture model

In statistics, a probability mixture model is a probability distribution that is a convex combination of other probability distributions.

Suppose that the discrete random variable $X$ is a mixture of $n$ component discrete random variables $Y i$ . Then, the probability mass function of $X$ , $f X (x)$ , is a weighted sum of its component distributions:

$f_{X}(x) = \sum_{i=1}^{n} a_i f_{Y_i}(x)$

for some mixture proportions $0 \leq a_{i} \leq 1$ where $a_{1} + \cdots + a_{n} = 1$ .

The definition is the same for continuous random variables, except that the functions $f$ are probability density functions.

[edit] Parametric mixture model

In the parametric mixture model, the component distributions are from a parametric family, with unknown parameters $θ i$ :

$f_{X}(x) = \sum_{i=1}^{n} a_i f_Y(x ; \theta_i).$

[edit] Continuous mixture

A continuous mixture is defined similarly:

$f_{X}(x) = \int_\Theta h(\theta) f_Y(x ; \theta) \, d\theta$

where

$0 \le h(\theta) \quad \forall \theta \in \Theta$

and

$\int_\Theta h(\theta) \, d\theta = 1.$

[edit] Identifiability

Identifiability refers to the existence of a unique characterization for any one of the models in the class being considered. Estimation procedure may not be well-defined and asymptotic theory may not hold if a model is not identifiable.

[edit] Example

Let $J$ be the class of all binomial distributions with $n = 2$ . Then a mixture of two members of $J$ would have

p 0 = π(1 - θ 1) 2 + (1 - π)(1 - θ 2) 2

p 1 = 2πθ 1 (1 - θ 1) + 2(1 - π)θ 2 (1 - θ 2)

and $p 2 = 1 - p 0 - p 1$ . Clearly, given $p 0$ and $p 1$ , it is not possible to determine the above mixture model uniquely, as there are three parameters ( $π,θ 1,θ 2$ ) to be determined.

[edit] Definition

Consider a mixture of parametric distributions of the same class. Let

$J=\{f(\cdot ; \theta):\theta\in\Omega\}$

be the class of all component distributions. Then the convex hull $K$ of $J$ defines the class of all finite mixture of distributions in $J$ :

$K=\{p(\cdot):p(\cdot)=\sum_{i=1}^n a_i f_i(\cdot ; \theta_i), a_i>0, \sum_{i=1}^n a_i=1, f_i(\cdot ; \theta_i)\in J\ \forall i,n\}$

$K$ is said to be identifiable if all its members are unique, that is, given two members $p$ and $p'$ in $K$ , being mixtures of $k$ distributions and $k'$ distributions respectively in $J$ , we have $p = p'$ if and only if, first of all, $k = k'$ and secondly we can reorder the summations such that $a i = a i'$ and $f i = f i'$ for all $i$ .

[edit] Common approaches for estimation in mixture models

Parametric mixture models are often used when we know the distribution $Y$ and we can sample from $X$ , but we would like to determine the $a i$ and $θ i$ values. Such situations can arise in studies in which we sample from a population that is composed of several distinct subpopulations.

It is common to think of probability mixture modeling as a missing data problem. One way to understand this is to assume that the data points under consideration have "membership" in one of the distributions we are using to model the data. When we start, this membership is unknown, or missing. The job of estimation is to devise appropriate parameters for the model functions we choose, with the connection to the data points being represented as their membership in the individual model distributions.

[edit] Expectation maximization (EM)

The Expectation-maximization algorithm can be used to compute the parameters of a parametric mixture model distribution (the $a i$ 's and $θ i$ 's). It is an iterative algorithm with two steps: an expectation step and a maximization step. Practical examples of EM and Mixture Modeling are included in the SOCR demonstrations.

[edit] The expectation step

With initial guesses for the parameters of our mixture model, "partial membership" of each data point in each constituent distribution is computed by calculating expectation values for the membership variables of each data point. That is, for each data point $x j$ and distribution $Y i$ , the membership value $y i, j$ is:

$y_{i,j} = \frac{a_i f_Y(x_j;\theta_i)}{f_{X}(x_j)}.$

[edit] The maximization step

With expectation values in hand for group membership, plug-in estimates are recomputed for the distribution parameters.

The mixing coefficients $a i$ are the means of the membership values over the $N$ data points.

$a_i = \frac{1}{N}\sum_{j=1}^N y_{i,j}$

The component model parameters $θ i$ are also calculated by expectation maximization using data points $x j$ that have been weighted using the membership values. For example, if $θ$ is a mean $μ$

$\mu_{i} = \frac{\sum_{j} y_{i,j}x_{j}}{\sum_{j} y_{i,j}}.$

With new estimates for $a i$ and the $θ i$ 's, the expectation step is repeated to recompute new membership values. The entire procedure is repeated until model parameters converge.

[edit] Markov-chain Monte Carlo

As an alternative to the EM algorithm, the mixture model parameters can be deduced using posterior sampling as indicated by Bayes' theorem. This is still regarded as an incomplete data problem whereby membership of data points is the missing data. A two-step iterative procedure known as Gibbs sampling can be used.

The previous example of a mixture of two Gaussian distributions can demonstrate how the method works. As before, initial guesses of the parameters for the mixture model are made. Instead of computing partial memberships for each elemental distribution, a membership value for each data point is drawn from a Bernoulli distribution (that is, it will be assigned to either the first or the second Gaussian). The Bernoulli parameter $θ$ is determined for each data point on the basis of one of the constituent distributions.^[vague] Draws from the distribution generate membership associations for each data point. Plug-in estimators can then be used as in the M step of EM to generate a new set of mixture model parameters, and the binomial draw step repeated.

[edit] Spectral method

Some problems in mixture model estimation can be solved using spectral methods. In particular it becomes useful if data points $x i$ are points in high-dimensional Euclidean space, and the hidden distributions are known to be log-concave (such as Gaussian distribution or Exponential distribution).

Spectral methods of learning mixture models are based on the use of Singular Value Decomposition of a matrix which contains data points. The idea is to consider the top $k$ singular vectors, where $k$ is the number of distributions to be learned. The projection of each data point to a linear subspace spanned by those vectors groups points originating from the same distribution very close together, while points from different distributions stay far apart.

One distinctive feature of the spectral method is that it allows us to prove that if distributions satisfy certain separation condition (e.g. not too close), then the estimated mixture will be very close to the true one with high probability.

[edit] Other methods

Some of them can even probably learn mixtures of heavy-tailed distributions including those with infinite variance (see links to papers below). In this setting, EM based methods would not work, since the Expectation step would diverge due to presence of outliers.

[edit] A simulation

To simulate a sample of size N that is from a mixture of distributions F_i, i=1 to n, with probabilities p_i (sum p_i=1):

Generate N random numbers from a Categorical distribution of size n and probabilities p_i for i=1 to n. These tell you which of the F_i each of the N values will come from. Denote by m_i the quantity of random numbers assigned to the i^th category.
For each i, generate m_i random numbers from the F_i distribution.

[edit] See also

[edit] Mixture

[edit] Hierarchical models

[edit] Outlier detection

RANSAC

[edit] References

^ Dinov, ID. "Expectation Maximization and Mixture Modeling Tutorial". California Digital Library, Statistics Online Computational Resource, Paper EM_MM, http://repositories.cdlib.org/socr/EM_MM, December 9, 2008

[edit] Further reading

[edit] Books on mixture models

Titterington, D., A. Smith, and U. Makov "Statistical Analysis of Finite Mixture Distributions," John Wiley & Sons (1985).
McLachlan, G.J. and Peel, D. Finite Mixture Models, , Wiley (2000)
Marin, J.M., Mengersen, K. and Robert, C.P. "Bayesian modelling and inference on mixtures of distributions". Handbook of Statistics 25, D. Dey and C.R. Rao (eds). Elsevier-Sciences (to appear). available as PDF
Lindsay B.G., Mixture Models: Theory, Geometry, and Applications. NSF-CBMS Regional Conference Series in Probability and Statistics Vol. 5, Institute of Mathematical Statistics, Hayward (1995).
McLachlan, G.J. and Basford, K.E. "Mixture Models: Inference and Applications to Clustering", Marcel Dekker (1988)
Everitt, B.S. and Hand D.J. "Finite mixture distributions", Chapman & Hall (1981)

[edit] Application of Gaussian mixture models

D.A. Reynolds and R.C. Rose (1995). "Robust text-independent speaker identification using Gaussianmixture speaker models". IEEE Transactions on Speech and Audio Processing.
H.H. Permuter, J. Francos and I.H. Jarmyn (2003). "Gaussian mixture models of texture and colour for image database retrieval". IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings (ICASSP '03).
Wolfgang Lemke (2005). Term Structure Modeling and Estimation in a State Space Framework. Springer Verlag. ISBN 978-3540283423.
Damiano Brigo and Fabio Mercurio (2001). "Displaced and Mixture Diffusions for Analytically-Tractable Smile Models". Mathematical Finance - Bachelier Congress 2000. Proceedings, Springer Verlag.
Damiano Brigo and Fabio Mercurio (2002). "Lognormal-mixture dynamics and calibration to market volatility smiles". International Journal of Theoretical and Applied Finance 5 (4).
Carol Alexander (2004). "Normal mixture diffusion with uncertain volatility: Modelling short- and long-term smile effects". Journal of Banking & Finance 28 (12).

[edit] External links

The SOCR demonstrations of EM and Mixture Modeling
Interactive Mixture of Normal-distributions Java applet
Mixture modelling page (and the Snob program for Minimum Message Length (MML) applied to finite mixture models), maintained by D.L. Dowe.
PyMix - Python Mixture Package, algorithms and data structures for a broad variety of mixture model based data mining applications in Python
em - A Python package for learning Gaussian Mixture Models with Expectation Maximization, currently packaged with SciPy
GMM.m Matlab code for GMM Implementation

[0] Dinov, ID. "Expectation Maximization and Mixture Modeling Tutorial". California Digital Library, Statistics Online Computational Resource, Paper EM_MM, http://repositories.cdlib.org/socr/EM_MM, December 9, 2008

[1]