# Properties of the Covariance Matrix¶

The covariance matrix of a random vector $$\mathbf{X} \in \mathbf{R}^{n}$$ with mean vector $$\mathbf{m}_{x}$$ is defined via:

$\mathbf{C}_{x}=E\left[(\mathbf{X}-\mathbf{m})(\mathbf{X}-\mathbf{m})^{T}\right] .$

The $$(i, j)^{\text {th }}$$ element of this covariance matrix $$\mathbf{C}_{x}$$ is given by

$C_{i j}=E\left[\left(X_{i}-m_{i}\right)\left(X_{j}-m_{j}\right)\right]=\sigma_{i j} .$

The diagonal entries of this covariance matrix $$\mathbf{C}_{x}$$ are the variances of the components of the random vector $$\mathbf{X}$$, i.e.,

$C_{i i}=E\left[\left(X_{i}-m_{i}\right)^{2}\right]=\sigma_{i}^{2} .$

Since the diagonal entries are all positive the trace of this covariance matrix is positive, i.e.,

$\operatorname{Trace}\left(\mathbf{C}_{x}\right)=\sum_{i=1}^{n} C_{i i}>0 .$

This covariance matrix $$\mathbf{C}_{x}$$ is symmetric, i.e., $$\mathbf{C}_{x}=\mathbf{C}_{x}^{T}$$ because :

$C_{i j}=\sigma_{i j}=\sigma_{j i}=C_{j i} .$

The covariance matrix $$\mathbf{C}_{x}$$ is positive semidefinite, i.e., for $$\mathbf{a} \in \mathbf{R}^{n}$$ :

\begin{aligned} E\left\{\left[(\mathbf{X}-\mathbf{m})^{T} \mathbf{a}\right]^{2}\right\} & =E\left\{\left[(\mathbf{X}-\mathbf{m})^{T} \mathbf{a}\right]^{T}\left[(\mathbf{X}-\mathbf{m})^{T} \mathbf{a}\right]\right\} \geq 0 \\ E\left[\mathbf{a}^{T}(\mathbf{X}-\mathbf{m})(\mathbf{X}-\mathbf{m})^{T} \mathbf{a}\right] & \geq 0, \quad \mathbf{a} \in \mathbf{R}^{n} \\ \mathbf{a}^{T} \mathbf{C}_{x} \mathbf{a} & \geq 0, \quad \mathbf{a} \in \mathbf{R}^{n} . \end{aligned}

Since the covariance matrix $$\mathbf{C}_{x}$$ is symmetric, i.e., self-adjoint with the usual inner product its eigenvalues are all real and positive and the eigenvectors that belong to distinct eigenvalues are orthogonal, i.e.,

$\mathbf{C}_{x}=\mathbf{V} \boldsymbol{\Lambda} \mathbf{V}^{T}=\sum_{i=1}^{n} \lambda_{i} \vec{v}_{i} \vec{v}_{i}^{T} .$

As a consequence, the determinant of the covariance matrix is positive, i.e.,

$\operatorname{Det}\left(\mathbf{C}_{X}\right)=\prod_{i=1}^{n} \lambda_{i} \geq 0 .$

The eigenvectors of the covariance matrix transform the random vector into statistically uncorrelated random variables, i.e., into a random vector with a diagonal covariance matrix. The Rayleigh coefficient of the covariance matrix is bounded above and below by the maximum and minimum eigenvalue :

$\lambda_{\min } \leq \frac{\mathbf{a}^{T} \mathbf{C}_{x} \mathbf{a}}{\mathbf{a}^{T} \mathbf{a}}, \quad \mathbf{a} \in \mathbf{R} \leq \lambda_{\max } .$