Assignment 1 · Problem 1 · submitted 11 Sep 2023

Covariance & eigen playground

Before any principal component analysis, the assignment checked the basics: which matrices can be covariance matrices at all, and how a symmetric matrix factors into rotations and stretches. Drag the entries below and the spectral decomposition $\Sigma = \Gamma\Lambda\Gamma^{\top}$ and its ellipse follow along.

Problem 1(a)

Which a and b make this a covariance matrix?

The question gave $\Sigma=\begin{pmatrix}1&a\\3&b\end{pmatrix}$ . A covariance matrix must be symmetric (because $\operatorname{Cov}(X_1,X_2)=\operatorname{Cov}(X_2,X_1)$ ) and positive semi-definite. Symmetry forces $a=3$ ; then the determinant $b-9\ge 0$ gives $b\ge 9$ . The valid set is a single ray, not a region.

What the 2023 answer said

The submission skipped the symmetry condition and only asked for a non-negative determinant of the matrix as written,

1\cdot b-3a\ge 0

, concluding

b\ge 3a

. That half-plane accepts matrices such as

\begin{pmatrix}1&1\\3&4\end{pmatrix}

, which are not symmetric and so cannot be covariance matrices. Switch between the two rules to compare.

1.0	1.0
3.0	4.0

a (top-right entry)1.0

b (bottom-right variance)4.0

Symmetric: a = 3 (Cov(X₁,X₂) = Cov(X₂,X₁))
Variances on the diagonal are non-negative (1 and b ≥ 0)
Determinant 1·b − 3a = 1.00 ≥ 0

Correct rule (a = 3, b ≥ 9)

not a covariance matrix

Submitted rule (b ≥ 3a)

covariance matrix

The (a, b) plane

valid covariance matrices: a = 3, b ≥ 9
accepted by the 2023 rule b ≥ 3a

Problems 1(b) and 1(d)

Eigen-decomposition you can drag

Every symmetric positive semi-definite matrix factors as $\Sigma=\Gamma\Lambda\Gamma^{\top}$ : an orthogonal matrix $\Gamma$ of eigenvectors (a rotation) and a diagonal $\Lambda$ of eigenvalues (the variances along those directions). The density contours of a normal distribution with covariance $\Sigma$ are ellipses whose axes are exactly these eigenvectors.

σ₁₁ = Var(X₁)4.00

σ₂₂ = Var(X₂)2.00

σ₁₂-1.73

σ₂₁-1.73

Keep σ₁₂ = σ₂₁ (symmetric)

4.00	-1.73
-1.73	2.00

= Γ Λ Γᵀ, with

0.866	0.500
-0.500	0.866

Γ (eigenvectors)

5.000	0
0	1.000

Λ (eigenvalues)

0.866	-0.500
0.500	0.866

Γᵀ

check: ΓΛΓᵀ = [[4.000, -1.732], [-1.732, 2.000]]

Valid covariance matrix (positive definite)

λ₁ = 5.000, λ₂ = 1.000; total variance tr Σ = 6.000; implied correlation ρ = -0.612.

Ellipse of constant density

The axes are the eigenvectors γ₁, γ₂; their half-lengths are 1 and 2 standard deviations √λ.

1σ and 2σ ellipses
γ₁ (largest variance)
γ₂
260 simulated draws

Problem 1(b), worked by hand

The 2 × 2 example without software

Find the eigenvalues and orthonormal eigenvectors of $\Sigma=\begin{pmatrix}4&-\sqrt3\\-\sqrt3&2\end{pmatrix}$ using only the characteristic polynomial, then write it as $\Gamma\Lambda\Gamma^{\top}$ .

1 · eigenvalues

\begin{aligned}\det(\Sigma-\lambda I)&=(4-\lambda)(2-\lambda)-3\\&=\lambda^2-6\lambda+5\\&=(\lambda-5)(\lambda-1)\end{aligned}

so $\lambda_1=5$ and $\lambda_2=1$ . Their sum is the trace (6) and their product the determinant (5), a quick check.

2 · eigenvectors

\begin{aligned}(\Sigma-5I)x=0&:\; -x_1-\sqrt3\,x_2=0\\&\Rightarrow\; \gamma_1=\tfrac12\begin{pmatrix}-\sqrt3\\1\end{pmatrix}\end{aligned}

\begin{aligned}(\Sigma-I)x=0&:\; 3x_1-\sqrt3\,x_2=0\\&\Rightarrow\; \gamma_2=\tfrac12\begin{pmatrix}1\\\sqrt3\end{pmatrix}\end{aligned}

Both are normalised to length 1 and are orthogonal ( $\gamma_1^{\top}\gamma_2=0$ ), as they must be for a symmetric matrix with distinct eigenvalues. Signs are arbitrary.

3 · spectral decomposition

\begin{gathered}\Sigma=\Gamma\Lambda\Gamma^{\top},\\\Gamma=\begin{pmatrix}-\tfrac{\sqrt3}{2}&\tfrac12\\\tfrac12&\tfrac{\sqrt3}{2}\end{pmatrix},\quad\Lambda=\begin{pmatrix}5&0\\0&1\end{pmatrix}\end{gathered}

\Sigma=\underbrace{\begin{pmatrix}-\tfrac{\sqrt3}{2}&\tfrac12\\\tfrac12&\tfrac{\sqrt3}{2}\end{pmatrix}}_{\Gamma}\underbrace{\begin{pmatrix}5&0\\0&1\end{pmatrix}}_{\Lambda}\underbrace{\begin{pmatrix}-\tfrac{\sqrt3}{2}&\tfrac12\\\tfrac12&\tfrac{\sqrt3}{2}\end{pmatrix}}_{\Gamma^{\top}}

This part of the submission was right

The 2023 answer derived

\lambda=5

and

\lambda=1

with the eigenvectors above and confirmed them with R's eigen(). The explicit

\Gamma

and

\Lambda

are written out here for completeness. Load the “Problem 1(b)” preset in the playground to see them numerically.

Problem 1(d) does the same for the 7 × 7 sample covariance of the wheat measurements; it is shown with the PCA it feeds into.

S = ΓΛΓᵀ for the wheat data