A random vector \(X\) is a vector, each of whose element is a random variable: \[X = \begin{bmatrix} X_1 \\ X_2 \\ \vdots \\ X_n \end{bmatrix}\]
The mean vector of the random vector \(X\) is \[E[X]= \begin{bmatrix} E[X_1] \\ E[X_2] \\ \vdots \\ E[X_n] \end{bmatrix}\]
A random matrix \(M\) is a matrix, each of whose element is a random variable: \[X = \begin{bmatrix} X_{11} & X_{12} & \ldots & X_{1n}\\ X_{21} & X_{22} & \ldots & X_{2n} \\ \vdots & \cdots & \vdots & \vdots \\ X_{m1} & X_{m2} & \ldots & X_{mn} \end{bmatrix} = [X_{ij}]_{m\times n}\]
The mean matrix of the random matrix \(M\) is \[E[M]= \begin{bmatrix} E[X_{11}] & E[X_{12}] & \ldots & E[X_{1n}]\\ E[X_{21}] & E[X_{22}] & \ldots & E[X_{2n}] \\ \vdots & \cdots & \vdots & \vdots \\ E[X_{m1}] & E[X_{m2}] & \ldots & E[X_{mn}] \end{bmatrix} = [E[X_{ij}]]_{m\times n}\]
The covariance matrix of a random vector \(X\) with mean vector \(\mu\) is defined as \[\mathrm{Cov}(X) = E[(X-\mu)(X-\mu)^T]\] which can be simplified to \[\mathrm{Cov}(X) = \begin{bmatrix} \mathrm{Var}(X_1) & \mathrm{Cov}(X_1, X_2) & \ldots & \mathrm{Cov}(X_1, X_n) \\ \mathrm{Cov}(X_2, X_1) & \mathrm{Var}(X_2) & \ldots & \mathrm{Cov}(X_2, X_n) \\ \vdots & \vdots & \ldots & \vdots \\ \mathrm{Cov}(X_n, X_1) & \mathrm{Cov}(X_n, X_2) & \ldots & \mathrm{Var}(X_n) \end{bmatrix}\]
Let \(X\) be an \(n\)-dimensional random vector and the random vector \(Y\) be defined as \[Y=AX+b,\] where \(A\) is a constant \(m \times n\) matrix and \(b\) is a fixed \(m\)-dimensional vector. Then: \[E[Y]=AE[X]+b\] and \[\mathrm{Cov}(Y) = A \mathrm{Cov}(X)A^T\]
Two random variable \(X\) and \(Y\) are said to be bivariate normal or jointly normal if \(aX+bY\) has a normal distribution for all \(a,b\in\mathbb{R}\).
If \(X\) and \(Y\) are jointly normal, then \(X\) and \(Y\) must be individually normal.
If \(X\sim \mathcal{N}(\mu_X, \sigma_X^2)\) and \(Y\sim \mathcal{N}(\mu_Y, \sigma_Y^2)\) are independent, then they are jointly normal.
If \(X\sim \mathcal{N}(\mu_X, \sigma_X^2)\) and \(Y\sim \mathcal{N}(\mu_Y, \sigma_Y^2)\) are jointly normal then \[X+Y \sim \mathcal{N}\left( \mu_X+\mu_Y, \sigma_X^2 + \sigma_Y^2 + 2 \rho \sigma_X \sigma_Y \right)\] where \(\rho\) is the correlation coefficient of \(X\), and \(Y\).
If \(X\) and \(Y\) are bivariate normal and uncorrelated, then they are independent.
If for \(i=1, 2, \ldots n\), the random variables \(Z_i\)’s are i.i.d. and standard normal: \[Z_i \sim \mathcal{N}(0,1), \quad i=1, 2, \ldots, n\] then the vector \[Z = \begin{bmatrix} Z_1 \\ Z_2 \\ \vdots \\ Z_n \end{bmatrix}\] is called the standard normal vector.
Taking \(z = [z_1\ z_2\ \ldots \, z_n]^T\), the pdf of the standard normal vector \(Z\) is \[f_Z(z) = \dfrac{1}{(2\pi)^{n/2}} e^{-\frac{1}{2}z^Tz} = \dfrac{1}{(2\pi)^{n/2}} e^{-\frac{1}{2}\Vert z \Vert^2}\]
\(E[Z] = 0\) and \(\mathrm{Cov}(Z) = I\), the unit matrix.
Let \(A\) be a (non-singular) \(n\times n\) matrix and \(\mu\) be an \(n\)-dimensional vector and let \[X = AZ + \mu\] Introduce the notation \(\Sigma = AA^T\). Then \[E[X] = A E[Z] + \mu = \mu\] and \[\mathrm{Cov}(X) = AA^T = \Sigma\]
Transformation: \[X = AZ + \mu\] gives \[Z=A^{-1}(X-\mu)\] Then Jacobian is \(\det(A^{-1})= \dfrac{1}{\sqrt{\det(\Sigma)}}\)
The pdf of \(X\) is \[f_X(x) = \dfrac{1}{(2\pi)^{n/2} \sqrt{\Sigma}} e^{-\frac{1}{2} (x-\mu)^T \Sigma^{-1}(x-\mu)}\] Hence, \(X\sim \mathcal{N}(\mu, \Sigma)\)
Let \(X\sim \mathcal{N}(\mu_X, \Sigma_X)\) be an \(n\)-dimensional normal vector. Further, suppose \(B\) be a full rank \(m\times n\) constant matrix and \(b\) is an \(m\)-dimensional constant vector. Then the random vector \[Y = BX + b\] is also distributed normally with mean vector \(\mu_Y\) and covariance matrix \(\Sigma_Y\), given by \[\begin{matrix} & \mu_Y& = & B \mu_X +b \\[3ex] \text{and} & \Sigma_Y &=& B \Sigma_X B^T \end{matrix}\] That is \[BX+b \sim \mathcal{N}\left(B\mu_X + b , B\Sigma_X B^T\right)\]
If \(X\sim \mathcal{N}(\mu, \Sigma)\), then its components \(X_i\) and \(X_j\) are independent if and only if \(\Sigma_{ij} = 0\).
Let \(X_1\), \(X_2\), \(X_3\), \(\ldots \), \(X_n\) be a random i.i.d. sample drawn from a distribution with a parameter \(\theta\), which can be a vector \(\theta = [\theta_1\, \theta_2\, \ldots, \theta_k]^T\). Suppose that \(x_1\), \(x_2\), \(x_3\),\(\ldots \), \(x_n\) are the observed values of \(X_1\), \(X_2\), \(X_3\),\(\ldots \), \(X_n\). The likelihood function is then defined as \[\begin{aligned}[t] L(x_1, x_2, \ldots, x_n; \theta) & = f_{X_1, X_2, \ldots, X_n}(x_1, x_2, \ldots, x_n; \theta) \\[2ex] & = \prod_{i=1}^n f_{X_i}(x_i;\theta) \end{aligned}\]
A maximum likelihood estimate (MLE) of \(\theta\), represented as \(\hat{\theta}_{ML}\) is a value of \(\theta\) that maximizes the likelihood function.
Often, it is easier to maximize the log likelihood function \[\ln{L(x_1, x_2, \ldots, x_n; \theta)} = \sum_{i=1}^n \ln{f_{X_i}(x_i;\theta)}\]
MLE of some common distributions on the basis of the observed sample values \(X_1 = x_1\), \(X_2=x_2\), …, \(X_n = x_n\), with \[\bar{x} = \dfrac{x_1+x_2+\ldots +x_n}{n}\] and \[\mathrm{S.D.} = \sqrt{\dfrac{1}{n} \sum_{i=1}^n (x_i-\bar{x})^2}\]
| Distribution | pmf/pdf | Parameter (\(\theta\)) | MLE (\(\hat{\theta}_{ML})\) |
|---|---|---|---|
| Bernoulli | \(\begin{matrix} p^x(1-p)^{1-x}, \\ x\in \{0, 1\} \end{matrix}\) | \(p\) | \(\hat{p} = \bar{x}\) |
| Binomial | \(\begin{matrix} {}^nC_x \ p^x(1-p)^{n-x}, \\ x=0,1, \ldots, n \end{matrix}\) | \(p\) | \(\hat{p} = \dfrac{\text{no. of success}}{n}\) |
| Poisson | \(\begin{matrix} \dfrac{\lambda^x e^{-\lambda}}{x!}, \\[2ex] x = 0, 1, 2, \ldots \end{matrix}\) | \(\lambda\) | \(\hat{\lambda} = \bar{x}\) |
| Uniform | \(\begin{matrix} \dfrac{1}{b-a}, x\in[a,b] \\ 0, \ \mathrm{otherwise} \end{matrix}\) | \(a,b\) | \(\begin{matrix} \hat{a} = \min \{x_i\} \\ \hat{b} = \max \{x_i\} \end{matrix}\) |
| Exponential | \(\begin{matrix} \lambda e^{-\lambda x}, \\ x \ge 0 \end{matrix}\) | \(\lambda\) | \(\hat{\lambda} = \dfrac{1}{\bar{x}}\) |
| Normal | \(\begin{matrix} \dfrac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2\sigma^2}(x-\mu)^2}, \\ x \in \mathbb{R} \end{matrix}\) | \(\mu, \sigma\) | \(\begin{matrix} \hat{\mu} = \bar{x} \\ \hat{\sigma} = \mathrm{S.D.} \end{matrix}\) |
Given the data set \[\mathcal{D} = \{ (x_i, y_i) \mid i=1,2,\ldots, n\}\] where \(x_i\in \mathbb{R}^d\) and \(y_i \in \mathbb{R}\), a linear regression model tries to find the “best” linear function that fits the data set.
Defining the vector \(X=[x_1\, x_2\, \ldots\, x_n]^T\) and \(Y=[y_1\, y_2\, \ldots\, y_n]^T\), a model is \[Y=w^TX + \epsilon\] for a weight vector \(w \in \mathbb{R}^d\) and a zero mean Normal vector whose each component are i.i.d. and \(\epsilon_i\sim \mathcal{N}(0, \sigma^2)\). Then \[Y\mid X \sim \mathcal{N}(w^TX, \sigma^2)\] Note that \(X\) and \(w\) are constant vectors but \(Y\) is a random vector.
Considering the unknown weight \(w\) as a parameter, maximum likelihood estimation can be used to obtain the optimum weight. This turns out to be the same problem as minimize \[\dfrac{1}{2} \sum_{i=1}^n (w^Tx_i - y_i)^2\]
An important unsupervised task in Machine Learning is to group an unlabeled data \[\mathcal{D} = \{x_1, x_2, \ldots, x_n\}, \quad x_i \in \mathbb{R}^d\] into some kind of homogeneous clusters.
An approach is to assume that the data was produced by a generative model and then adjust the model parameters to maximize the probability that the model would produce exactly the data that is observed.
Let the clusters be assigned labels as \(1\), \(2\), \(\ldots\), \(K\), and let \(Z\) be a random variable that takes on values from these cluster labels with certain probabilities: \[P(Z=k) = \pi_k , \quad k = 1, 2, \ldots, K\]
As a result, \(\pi_k \ge 0\) for \(k = 1, 2, \ldots, K\) and \[\sum_{k=1}^K \pi_k = 1\]
Assume that the observed data vector \(x=[x_1\, x_2\, \ldots \, x_n]^T\) is the value attained by the random vector \[X = \begin{bmatrix} X_1 \\ X_2 \\ \vdots \\ X_n \end{bmatrix}\]
A mixture model assumes a probability density of the form \[P(X=x) = \sum_{k=1}^K P(Z=k) P(X=x|Z=k)\]
It is assumed that given \(Z=k\), the random vector \(X\) is distributed normally with mean vector \(\mu_k\) and variance matrix \(\Sigma_k\): \[X|Z=k \sim \mathcal{N}\left( \mu_k, \Sigma_k \right)\]
The Gaussian mixture model (GMM) represents the distribution \[P(X=x) = \sum_{k=1}^K \pi_k \,\mathcal{N}\left( x \mid \mu_k, \Sigma_k\right)\] with \(\pi_k \ge 0\) for \(k=1, 2, \ldots, K\) and \({\displaystyle \sum_{k=1}^K \pi_k = 1}\).
GMM is a density estimator.
The complete GMM is parametrized by the mean vectors, covariance matrices and the mixture weights from all component densities, collectively represented as \[\lambda = \{ \pi_k, \mu_k, \Sigma_k \}, \quad k=1, 2, \ldots, K\]
The maximum likelihood estimate of the parameters \(\lambda\) is obtained iteratively using a variant of expectation-maximization (EM) algorithm.
In EM algorithm, start from an initial parameter set \(\lambda_0\) and iterate through the data points one by one and perform the following two steps till either the algorithm converges or an upper limit of iteration is reached.
For the data point \(x_\ell\):
E step: Evaluate the responsibilities using the current parameter: \[\gamma(z_{\ell k}) = \dfrac{\pi_k \, \mathcal{N}(x_\ell | \mu_k, \Sigma_k)} {\displaystyle \sum_{j=1}^K \pi_j \, \mathcal{N}(x_\ell | \mu_j, \Sigma_j)}\] The responsibility for a data point is the probability that it belongs to a cluster: \[\gamma(z_{\ell k}) = P(Z=k|X=x_\ell)\]
M step: Re-estimate the parameters using the current responsibilities: \[\mu_k^\mathrm{new} = \dfrac{1}{N_k} \sum_{\ell=1}^n \gamma(z_{\ell k}) x_\ell\] \[\Sigma_k^\mathrm{new} = \dfrac{1}{N_k} \sum_{\ell=1}^n \gamma(z_{\ell k}) (x_\ell -\mu_k^\mathrm{new})(x_\ell -\mu_k^\mathrm{new})^T\] \[\pi_{k}^\mathrm{new} = \dfrac{N_k}{n}\] where \[N_k = \sum_{\ell = 1}^n \gamma(z_{\ell k})\]
Markov’s Inequality: Let \(X\) be a non-negative random variable with a finite mean \(\mu\). Then \[P(X\ge c) \le \dfrac{\mu}{c}\]
Chebyshev’s Inequality: Let \(X\) be a random variable with \[E[X]=\mu \quad \text{and} \quad \mathrm{Var}(X) = \sigma^2\] Then \[P(|X-\mu|\ge k\sigma ) \le \dfrac{1}{k^2}\]
Hoeffding Inequality: Suppose \(X_1\), \(X_2\), …, \(X_n\) be i.i.d.’s so that \(E[X_i] = \mu\) and \(a\le X_i \le b\). Define \[\overline{X}= \dfrac{1}{n} \sum_{i=1}^n X_i\] Then \[P(|\bar{X}-\mu| \ge \delta ) \le 2 \exp{\left(-\dfrac{2n\delta^2}{(b-a)^2}\right)}\]
Weak law of large numbers: Suppose \(X_1\), \(X_2\), …, \(X_n\) \(\sim\) i.i.d. \(X\) so that \(E[X] = \mu\) and \(\mathrm{Var}(X)=\sigma^2\). Define the sample mean \[\overline{X} = \dfrac{1}{n} \sum_{i=1}^n X_i\] Then \[P(|\bar{X}-\mu| \ge \delta ) \le \dfrac{\sigma^2}{n\delta^2}\]
Central Limit Theorem (CLT): Suppose \(X_1\), \(X_2\), …, \(X_n\) \(\sim\) i.i.d. \(X\) so that \(E[X] = \mu\) and \(\mathrm{Var}(X)=\sigma^2\). Define \[Y = X_1 + X_2 + \ldots + X_n\] Then \[\dfrac{Y-n\mu}{\sigma \sqrt{n}} \approx \mathcal{N}(0, 1)\]