Understanding the Relationship Between Leverage and Mahalanobis Distance

In Multiple Linear Regression, it is useful to detect the presence of outliers in the sample. An indicator of the presence of outliers is the leverage (McCullagh & Nelder, 1989, p. 405). The leverage is a measure of the distance between an observation in the sample and the sample mean vector (p. 405). Given that the leverage is a measure of distance, it is perhaps unsurprising that it is related to the another measure of distance known as the Mahalanobis distance (Mahalanobis, 1936). In this post, I prove the following mathematical relationship between the leverage and the Mahalanobis distance:

$$d_i^2=(n-1)\left(\begin{array}{c}{h}_{ii}–\frac{1}{n}\end{array}\right)$$ where $d_i^2$ is the square of the Mahalanobis distance between the $i$-th observation and the sample mean vector, and $h_{ii}$ is the leverage of the $i$-th observation, for $i=1,2,\dots ,n$.

Notations

Before we proceed to prove the mathematical relationship between the leverage and the Mahalanobis distance, it is useful to introduce the notation that will be used in the proof.

Sample. Let $X=(x_1,x_2,\dots ,x_n{)}^T$ be an $n\times p$ matrix which represents a sample of $n$ observations across $p$ covariates. In $X$, the $i$-th row represents the $i$-th observation and the $j$-th column represents the $j$-th covariate, for $i=1,2,\dots ,n$ and $j=1,2,\dots ,p$.

Sample Means. Let $\mu =({\mu }_1,{\mu }_2,\dots ,{\mu }_p{)}^T=\frac{1}{n}{X}^T{J}_{n,1}$ be a column vector with $p$ elements which represents the sample mean vector, where $J_{n,1}$ is an column vector with $n$ $1$s.

Sample Covariances. Let $\mathrm{\Sigma }=\frac{1}{n-1}(X-J_{n,1}{\mu }^T{)}^T(X-J_{n,1}{\mu }^T)$ be a $p\times p$ matrix which represents the sample covariance matrix.

Model Matrix. Let $(J_{n,1}X)=((1x_1^T{)}^T,(1x_2^T{)}^T,\dots ,(1x_n^T{)}^T{)}^T$ be an $n\times (p+1)$ matrix which represents the model matrix which includes an intercept in addition to the $p$ covariates.

Leverage. The leverage is a measure of the distance between an observation in the sample and the sample mean vector (McCullagh & Nelder, 1989, p. 405). Let $h_{ii}$ be the leverage of the $i$-th observation in the sample, for $i=1,2,\dots ,n$. By definition,

$$\begin{array}{rl}{h}_{ii}& =(1x_i^T)((J_{n,1}X{)}^T(J_{n,1}X){)}^{-1}(1x_i^T{)}^T\end{array}$$

Mahalanobis Distance. The Mahalanobis distance between two vectors is a measure of the distance between the vectors. Let $d_i^2$ be the square of the Mahalanobis distance between the $i$-th observation in the sample and the sample mean vector, for $i=1,2,\dots ,n$ in the sample. By definition,

$$\begin{array}{rl}{d}_i^2& =(x_i-\mu {)}^T{\mathrm{\Sigma }}^{-1}(x_i-\mu )\end{array}$$

Result

Now that we have introduced the notation that will be used in the proof, let us proceed to prove the mathematical relationship between the leverage and the Mahalanobis distance.

Theorem 1. The Mahalanobis distances are related to the leverages as follows: $$d_i^2=(n-1)\left(\begin{array}{c}{h}_{ii}–\frac{1}{n}\end{array}\right)$$ Proof:

$$\begin{array}{rl}{h}_{ii}& =(1x_i^T)((J_{n,1}X{)}^T(J_{n,1}X){)}^{-1}(1x_i^T{)}^T\\ & =(1x_i^T){\left(\begin{array}{cc}n& n{\mu }^T\\ n\mu & X^{T}X\end{array}\right)}^{-1}(1x_i^T{)}^T\\ & =(1x_i^T)\left(\begin{array}{cc}\frac{1}{n}+\frac{1}{n}n{\mu }^T\frac{1}{n-1}{\mathrm{\Sigma }}^{-1}n\mu \frac{1}{n}& -\frac{1}{n}n{\mu }^T\frac{1}{n-1}{\mathrm{\Sigma }}^{-1}\\ -\frac{1}{n-1}{\mathrm{\Sigma }}^{-1}n\mu \frac{1}{n}& \frac{1}{n-1}{\mathrm{\Sigma }}^{-1}\end{array}\right)(1x_i^T{)}^T\\ & =(1x_i^T)\left(\begin{array}{cc}\frac{1}{n}+\frac{1}{n-1}{\mu }^T{\mathrm{\Sigma }}^{-1}\mu & -\frac{1}{n-1}{\mu }^T{\mathrm{\Sigma }}^{-1}\\ -\frac{1}{n-1}{\mathrm{\Sigma }}^{-1}\mu & \frac{1}{n-1}{\mathrm{\Sigma }}^{-1}\end{array}\right)(1x_i^T{)}^T\\ & =\left(\begin{array}{cc}\frac{1}{n}+\frac{1}{n-1}{\mu }^T{\mathrm{\Sigma }}^{-1}\mu -\frac{1}{n-1}{x}_i^T{\mathrm{\Sigma }}^{-1}\mu & -\frac{1}{n-1}{\mu }^T{\mathrm{\Sigma }}^{-1}+\frac{1}{n-1}{x}_i^T{\mathrm{\Sigma }}^{-1}\end{array}\right)(1x_i^T{)}^T\\ & =\frac{1}{n}+\frac{1}{n-1}\left(\begin{array}{c}{\mu }^T{\mathrm{\Sigma }}^{-1}\mu –x_i^T{\mathrm{\Sigma }}^{-1}\mu -{\mu }^T{\mathrm{\Sigma }}^{-1}{x}_i+x_i^T{\mathrm{\Sigma }}^{-1}{x}_i\end{array}\right)\\ & =\frac{1}{n}+\frac{1}{n-1}\left[\begin{array}{c}-(x_i^T-{\mu }^T){\mathrm{\Sigma }}^{-1}\mu +(x_i^T-{\mu }^T){\mathrm{\Sigma }}^{-1}{x}_i\end{array}\right]\\ & =\frac{1}{n}+\frac{1}{n-1}\left[\begin{array}{c}(x_i^T-{\mu }^T){\mathrm{\Sigma }}^{-1}(x_i-\mu )\end{array}\right]\\ & =\frac{1}{n}+\frac{1}{n-1}{d}_i^2\end{array}$$

Therefore,

$$d_i^2=(n-1)\left(\begin{array}{c}{h}_{ii}–\frac{1}{n}\end{array}\right)$$ which was to be demonstrated.

In the proof,

$$\left(\begin{array}{cc}n& n{\mu }^T\\ n\mu & X^{T}X\end{array}\right)$$ is inverted blockwise using the analytic inversion formula:

$${\left(\begin{array}{cc}\mathbf{\text{A}}& \mathbf{\text{B}}\\ \mathbf{\text{C}}& \mathbf{\text{D}}\end{array}\right)}^{-1}=\left(\begin{array}{cc}{\mathbf{\text{A}}}^{-1}+{\mathbf{\text{A}}}^{-1}\mathbf{\text{B}}(\mathbf{\text{D}}-{\mathbf{\text{CA}}}^{-1}\mathbf{\text{B}}{)}^{-1}{\mathbf{\text{CA}}}^{-1}& -{\mathbf{\text{A}}}^{-1}\mathbf{\text{B}}(\mathbf{\text{D}}-{\mathbf{\text{CA}}}^{-1}\mathbf{\text{B}}{)}^{-1}\\ -(\mathbf{\text{D}}-{\mathbf{\text{CA}}}^{-1}\mathbf{\text{B}}{)}^{-1}{\mathbf{\text{CA}}}^{-1}& (\mathbf{\text{D}}-{\mathbf{\text{CA}}}^{-1}\mathbf{\text{B}}{)}^{-1}\end{array}\right)$$

In the analytic inversion formula,

$$\begin{array}{rl}\mathbf{\text{D}}-{\mathbf{\text{CA}}}^{-1}\mathbf{\text{B}}& =X^{T}X-n\mu \frac{1}{n}n{\mu }^T\\ & =X^{T}X-n\mu {\mu }^T\\ & =(n-1)\mathrm{\Sigma }\end{array}$$

This is because

$$\begin{array}{rl}\mathrm{\Sigma }& =\frac{1}{n-1}(X-J_{n,1}{\mu }^T{)}^T(X-J_{n,1}{\mu }^T)\\ & =\frac{1}{n-1}(X^{T}X-X^T{J}_{n,1}{\mu }^T-\mu J_{1,n}X+\mu J_{1,n}{J}_{n,1}{\mu }^T)\\ & =\frac{1}{n-1}(X^{T}X-n\mu {\mu }^T-n\mu {\mu }^T+n\mu {\mu }^T)\\ & =\frac{1}{n-1}(X^{T}X-n\mu {\mu }^T)\end{array}$$

References

Mahalanobis, C. P. (1936). On the generalized distance in statistics. In Proceedings of the National Institute of Sciences of India (Vol. 2, No. 1, pp. 49-55).

McCullagh, P., & Nelder, J. A. (1989). Generalized linear models. Springer.

Appendix

In the Appendix, I present some results on the equivalence between statistics obtained from samples without and with mean-centering.

Notations

Sample. Let $X^c=X-J_{n,1}{\mu }^T=(x_1^c,x_2^c,\dots ,x_n^c{)}^T$ be an $n\times p$ matrix which represents the mean-centred sample.

Sample Means. Let ${\mu }^c=({\mu }_1^c,{\mu }_2^c,\dots ,{\mu }_p^c{)}^T=0_{p,1}$ be a column vector with $p$ elements which represents the mean-centred sample mean vector.

Sample Covariances. Let $^{c}=(X{c}-J_{n,1}(^{c}){T})^{T}(X{c}-J_{n,1}(^{c}){T})=(X^{c}){T}(X^{c})$ be a $p\times p$ matrix which represents the mean-centred sample covariance matrix.

Model Matrix. Let $(J_{n,1}{X}^c)$ be an $n\times (p+1)$ matrix which represents the model matrix using the mean-centred sample.

Leverage. Let $h_{ii}^c$ be the leverage of the $i$-th observation in the mean-centred sample, for $i=1,2,\dots ,n$. By definition,

$$\begin{array}{rl}{h}_{ii}^c& =(1(x_i^c{)}^T)((J_{n,1}{X}^c{)}^T(J_{n,1}{X}^c){)}^{-1}(1(x_i^c{)}^T{)}^T\\ & =(1x_i^T-{\mu }^T)((J_{n,1}{X}^c{)}^T(J_{n,1}{X}^c){)}^{-1}(1x_i^T-{\mu }^T{)}^T\end{array}$$

Mahalanobis Distance. Let $(d_i^c{)}^2$ be the square of the Mahalnobis distance between the $i$-th observation and ${\mu }^c$, for $i=1,2,\dots ,n$ in the mean-centred sample. By definition,

$$\begin{array}{rl}(d_i^c{)}^2& =(x_i^c-{\mu }^c{)}^T({\mathrm{\Sigma }}^c{)}^{-1}(x_i^c-{\mu }^c)\\ & =(x_i-\mu {)}^T({\mathrm{\Sigma }}^c{)}^{-1}(x_i-\mu )\end{array}$$

Results

Proposition 1. The sample covariance matrix is the same regardless of whether the sample has been mean-centred. In other words, $\mathrm{\Sigma }={\mathrm{\Sigma }}^c$.

Proof:

$$\begin{array}{rl}\mathrm{\Sigma }& =\frac{1}{n-1}(X-J_{n,1}{\mu }^T{)}^T(X-J_{n,1}{\mu }^T)\\ & =\frac{1}{n-1}(X^c{)}^T(X^c)\\ & ={\mathrm{\Sigma }}^c\end{array}$$

which was to be demonstrated.

Proposition 2. The leverages are the same regardless of whether the sample has been mean-centred. In other words, $h_{ii}=h_{ii}^c$.

Proof:

$$\begin{array}{rl}{h}_{ii}^c& =(1x_i^T-{\mu }^T)((J_{n,1}{X}^c{)}^T(J_{n,1}{X}^c){)}^{-1}(1x_i^T-{\mu }^T{)}^T\\ & =(1x_i^T-{\mu }^T){\left(\begin{array}{cc}n& 0_{1,p}\\ 0_{p,1}& (X^c{)}^T{X}^c\end{array}\right)}^{-1}(1x_i^T-{\mu }^T{)}^T\\ & =(1x_i^T-{\mu }^T){\left(\begin{array}{cc}n& 0_{1,p}\\ 0_{p,1}& (n-1)\mathrm{\Sigma }\end{array}\right)}^{-1}(1x_i^T-{\mu }^T{)}^T\\ & =(1x_i^T-{\mu }^T)\left(\begin{array}{cc}\frac{1}{n}& 0_{1,p}\\ 0_{p,1}& \frac{1}{n-1}{\mathrm{\Sigma }}^{-1}\end{array}\right)(1x_i^T-{\mu }^T{)}^T\\ & =\left(\begin{array}{cc}\frac{1}{n}& \frac{1}{n-1}(x_i^T-{\mu }^T){\mathrm{\Sigma }}^{-1}\end{array}\right)(1x_i^T-{\mu }^T{)}^T\\ & =\frac{1}{n}+\frac{1}{n-1}\left[\begin{array}{c}(x_i^T-{\mu }^T){\mathrm{\Sigma }}^{-1}(x_i-\mu )\end{array}\right]\\ & =\frac{1}{n}+\frac{1}{n-1}{d}_i^2\\ & =h_{ii}\end{array}$$

which was to be demonstrated.

In the proof, $\left(\begin{array}{cc}n& 0_{1,p}\\ 0_{p,1}& (n-1)\mathrm{\Sigma }\end{array}\right)$ is a diagonal block matrix and therefore inverted blockwise.

Proposition 3. The Mahalanobis distances are the same regardless of whether the sample has been mean-centred. In other words, $d_i=d_i^c$.

Proof:

$$\begin{array}{rl}{d}_i^2& =(x_i-\mu {)}^T{\mathrm{\Sigma }}^{-1}(x_i-\mu )\\ & =(x_i-\mu {)}^T({\mathrm{\Sigma }}^c{)}^{-1}(x_i-\mu )\\ & =(d_i^c{)}^2\end{array}$$

Therefore,

$$d_i=d_i^c$$

which was to be demonstrated.