A vector-valued random variable $X=[\begin{array}{lll}{X}_1& \cdots & X_n\end{array}{]}^T$$X=[\begin{array}{l}{X}_1\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\cdots \hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}{X}_n\end{array}{]}^T$X=[{:[X_(1),cdots,X_(n)]:}]^(T)X=[\begin{array}{lll}X_{1} & \cdots & X_{n}\end{array}]^{T} is said to have a multivariate normal (or Gaussian) distribution with mean $\mu \in {\mathbf{R}}^n$$\mu \in {\mathbf{R}}^n$mu inR^(n)\mu \in \mathbf{R}^{n} and covariance matrix $\mathrm{\Sigma }\in {\mathbf{S}}_{++}^n$$\mathrm{\Sigma }\in {\mathbf{S}}_{++}^n$Sigma inS_(++)^(n)\Sigma \in \mathbf{S}_{++}^{n}^[1] if its probability density function^[2] is given by

We write this as $X\sim \mathcal{N}(\mu ,\mathrm{\Sigma })$$X\sim \mathcal{N}(\mu ,\mathrm{\Sigma })$X∼N(mu,Sigma)X \sim \mathcal{N}(\mu, \Sigma). In these notes, we describe multivariate Gaussians and some of their basic properties.

Recall that the density function of a univariate normal (or Gaussian) distribution is given by

Here, the argument of the exponential function, $-\frac{1}{2{\sigma }^2}(x-\mu {)}^2$$-\frac{1}{2{\sigma }^2}(x-\mu {)}^2$-(1)/(2sigma^(2))(x-mu)^(2)-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}, is a quadratic function of the variable $x$$x$xx. Furthermore, the parabola points downwards, as the coefficient of the quadratic term is negative. The coefficient in front, $\frac{1}{\sqrt{2\pi }\sigma }$$\frac{1}{\sqrt{2\pi }\sigma }$(1)/(sqrt(2pi)sigma)\frac{1}{\sqrt{2 \pi} \sigma}, is a constant that does not depend on $x$$x$xx; hence, we can think of it as simply a "normalization factor" used to ensure that

The figure on the left shows a univariate Gaussian density for a single variable $X$$X$XX. The figure on the right shows a multivariate Gaussian density over two variables $X_1$$X_1$X_(1)X_{1} and $X_2$$X_2$X_(2)X_{2}.

In the case of the multivariate Gaussian density, the argument of the exponential function, $-\frac{1}{2}(x-\mu {)}^T{\mathrm{\Sigma }}^{-1}(x-\mu )$$-\frac{1}{2}(x-\mu {)}^T{\mathrm{\Sigma }}^{-1}(x-\mu )$-(1)/(2)(x-mu)^(T)Sigma^(-1)(x-mu)-\frac{1}{2}(x-\mu)^{T} \Sigma^{-1}(x-\mu), is a quadratic form in the vector variable $x$$x$xx. Since $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma is positive definite, and since the inverse of any positive definite matrix is also positive definite, then for any non-zero vector $z,z^T{\mathrm{\Sigma }}^{-1}z>0$$z,z^T{\mathrm{\Sigma }}^{-1}z>0$z,z^(T)Sigma^(-1)z > 0z, z^{T} \Sigma^{-1} z>0. This implies that for any vector $x\ne \mu$$x\ne \mu$x!=mux \neq \mu,

Like in the univariate case, you can think of the argument of the exponential function as being a downward opening quadratic bowl. The coefficient in front (i.e., $\frac{1}{(2\pi {)}^{n/2}|\mathrm{\Sigma }{|}^{1/2}}$$\frac{1}{(2\pi {)}^{n/2}|\mathrm{\Sigma }{|}^{1/2}}$(1)/((2pi)^(n//2)|Sigma|^(1//2))\frac{1}{(2 \pi)^{n / 2}|\Sigma|^{1 / 2}}) has an even more complicated form than in the univariate case. However, it still does not depend on $x$$x$xx, and hence it is again simply a normalization factor used to ensure that

The concept of the covariance matrix is vital to understanding multivariate Gaussian distributions. Recall that for a pair of random variables $X$$X$XX and $Y$$Y$YY, their covariance is defined as

When working with multiple variables, the covariance matrix provides a succinct way to summarize the covariances of all pairs of variables. In particular, the covariance matrix, which we usually denote as $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma, is the $n\times n$$n\times n$n xx nn \times n matrix whose $(i,j)$$(i,j)$(i,j)(i, j) th entry is $\mathrm{Cov}[X_i,X_j]$$\mathrm{Cov}\left[X_i,X_j\right]$Cov[X_(i),X_(j)]\operatorname{Cov}\left[X_{i}, X_{j}\right].

The following proposition (whose proof is provided in the Appendix A.1) gives an alternative way to characterize the covariance matrix of a random vector $X$$X$XX :

Proposition 1. For any random vector $X$$X$XX with mean $\mu$$\mu$mu\mu and covariance matrix $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma,

In the definition of multivariate Gaussians, we required that the covariance matrix $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma be symmetric positive definite (i.e., $\mathrm{\Sigma }\in {\mathbf{S}}_{++}^n$$\mathrm{\Sigma }\in {\mathbf{S}}_{++}^n$Sigma inS_(++)^(n)\Sigma \in \mathbf{S}_{++}^{n}). Why does this restriction exist? As seen in the following proposition, the covariance matrix of any random vector must always be symmetric positive semidefinite:

Proposition 2. Suppose that $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma is the covariance matrix corresponding to some random vector $X$$X$XX. Then $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma is symmetric positive semidefinite.

Proof. The symmetry of $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma follows immediately from its definition. Next, for any vector $z\in {\mathbf{R}}^n$$z\in {\mathbf{R}}^n$z inR^(n)z \in \mathbf{R}^{n}, observe that

Here, (2) follows from the formula for expanding a quadratic form (see section notes on linear algebra), and (3) follows by linearity of expectations (see probability notes).

To complete the proof, observe that the quantity inside the brackets is of the form $\sum _i\sum _j{x}_i{x}_j{z}_i{z}_j=(x^{T}z{)}^2\ge 0$$\sum _i \sum _j x_i{x}_j{z}_i{z}_j=(x^{T}z{)}^2\ge 0$sum_(i)sum_(j)x_(i)x_(j)z_(i)z_(j)=(x^(T)z)^(2) >= 0\sum_{i} \sum_{j} x_{i} x_{j} z_{i} z_{j}=(x^{T} z)^{2} \geq 0 (see problem set #1). Therefore, the quantity inside the expectation is always nonnegative, and hence the expectation itself must be nonnegative. We conclude that $z^T\mathrm{\Sigma }z\ge 0$$z^T\mathrm{\Sigma }z\ge 0$z^(T)Sigma z >= 0z^{T} \Sigma z \geq 0.

From the above proposition it follows that $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma must be symmetric positive semidefinite in order for it to be a valid covariance matrix. However, in order for ${\mathrm{\Sigma }}^{-1}$${\mathrm{\Sigma }}^{-1}$Sigma^(-1)\Sigma^{-1} to exist (as required in the definition of the multivariate Gaussian density), then $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma must be invertible and hence full rank. Since any full rank symmetric positive semidefinite matrix is necessarily symmetric positive definite, it follows that $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma must be symmetric positive definite.

To get an intuition for what a multivariate Gaussian is, consider the simple case where $n=2$$n=2$n=2n=2, and where the covariance matrix $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma is diagonal, i.e.,

In this case, the multivariate Gaussian density has the form,

where we have relied on the explicit formula for the determinant of a $2\times 2$$2\times 2$2xx22 \times 2 matrix^[3], and the fact that the inverse of a diagonal matrix is simply found by taking the reciprocal of each diagonal entry. Continuing,

The last equation we recognize to simply be the product of two independent Gaussian densities, one with mean ${\mu }_1$${\mu }_1$mu_(1)\mu_{1} and variance ${\sigma }_1^2$${\sigma }_1^2$sigma_(1)^(2)\sigma_{1}^{2}, and the other with mean ${\mu }_2$${\mu }_2$mu_(2)\mu_{2} and variance ${\sigma }_2^2$${\sigma }_2^2$sigma_(2)^(2)\sigma_{2}^{2}.

More generally, one can show that an $n$$n$nn-dimensional Gaussian with mean $\mu \in {\mathbf{R}}^n$$\mu \in {\mathbf{R}}^n$mu inR^(n)\mu \in \mathbf{R}^{n} and diagonal covariance matrix $\mathrm{\Sigma }=\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_n^2)$$\mathrm{\Sigma }=\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_n^2)$Sigma=diag(sigma_(1)^(2),sigma_(2)^(2),dots,sigma_(n)^(2))\Sigma=\operatorname{diag}(\sigma_{1}^{2}, \sigma_{2}^{2}, \ldots, \sigma_{n}^{2}) is the same as a collection of $n$$n$nn independent Gaussian random variables with mean ${\mu }_i$${\mu }_i$mu_(i)\mu_{i} and variance ${\sigma }_i^2$${\sigma }_i^2$sigma_(i)^(2)\sigma_{i}^{2}, respectively.

Another way to understand a multivariate Gaussian conceptually is to understand the shape of its isocontours. For a function $f:{\mathbf{R}}^2\to \mathbf{R}$$f:{\mathbf{R}}^2\to \mathbf{R}$f:R^(2)rarrRf: \mathbf{R}^{2} \rightarrow \mathbf{R}, an isocontour is a set of the form

for some $c\in \mathbf{R}$$c\in \mathbf{R}$c inRc \in \mathbf{R}.^[4]

What do the isocontours of a multivariate Gaussian look like? As before, let's consider the case where $n=2$$n=2$n=2n=2, and $\mathrm{\Sigma }$$\mathrm{\Sigma }$Sigma\Sigma is diagonal, i.e.,

As we showed in the last section,

Now, let's consider the level set consisting of all points where $p(x;\mu ,\mathrm{\Sigma })=c$$p(x;\mu ,\mathrm{\Sigma })=c$p(x;mu,Sigma)=cp(x ; \mu, \Sigma)=c for some constant $c\in \mathbf{R}$$c\in \mathbf{R}$c inRc \in \mathbf{R}. In particular, consider the set of all $x_1,x_2\in \mathbf{R}$$x_1,x_2\in \mathbf{R}$x_(1),x_(2)inRx_{1}, x_{2} \in \mathbf{R} such that