Skip to content

Spectrum and eigendecomposition

· 3 min read

In the vault of things that I get amused every time I remember is the connection between the time series’ spectrum and matrices’ spectrum.

Having been trained as an engineer during my undergraduate years, I instinctively associate a signal’s spectrum with its oscillations. It’s the spectrum that unveils the predominant oscillatory components within the signal under analysis. But as a graduate student learning the dark arts of magic statistics, I’ve often encountered colleagues whose notion of spectrum was completely different from mine: for them, spectrum represents the sum of eigenvalues of the signal’s covariance matrix (as a consequence of the Wiener–Khinchin–Einstein theorem.)

The discussion I show here might seem too detailed for a book yet too basic for a academic paper. Nonetheless, I find it to be a stimulating mental exercise in linking seemingly distant concepts.

Let us start with spectral definitions in linear algebra: the spectrum of a square matrix is the set of its eigenvalues. (I think a different post is needed to discuss the proper natural interpretation of eigenvectors and eigenvalues.)

In the classical definition of a time series spectrum (or power spectrum density, PSD), the spectrum is the Fourier transform of the covariance function:

\[X\left(\omega\right) =\left|\mathcal{F} \left\{ \sigma_{xx}\left(\tau\right) \right\} \left(\omega\right) \right|^{2}\]

However, consider a scenario where we only have \(T\) points sampled from a weakly stationary time series. Then, the spectrum could be defined as

\[X\left(\omega\right)=\sum_{t=0}^{T-1}\sigma_{xx}\left(\tau\right)e^{-j\omega t}\]

Then, we can construct a covariance matrix \(\Sigma_{xx}\) as

\[\Sigma_{xx}=\left(\begin{array}{cccc} \sigma_{xx}\left(0\right) & \sigma_{xx}\left(1\right) & \cdots & \sigma_{xx}\left(T-1\right)\\ \sigma_{xx}\left(-1\right) & \sigma_{xx}\left(0\right) & \cdots & \sigma_{xx}\left(T-2\right)\\ \vdots & \vdots & \ddots & \vdots\\ \sigma_{xx}\left(-T+1\right) & \sigma_{xx}\left(-T+2\right) & \cdots & \sigma_{xx}\left(0\right) \end{array}\right)\]

The nice trick here is to remember that due to weak stationarity, the correlation at \(-\tau\), \(\sigma_{xx}\left(-\tau\right)\), will be the same at \(\tau\), \(\sigma_{xx}\left(\tau\right)\). This symmetric property makes \(\Sigma_{xx}\) to be a circulant matrix.

Then, let’s proceed with an eigendecomposition of \(\Sigma_{xx}\), where \(Y\) will be the eigenvectors and \(\gamma\) the eigenvalues:

\[\Sigma_{xx}Y=\gamma Y\]

While this expression may seem complex, focusing on the \(k\)-th row of \(\Sigma_{xx}\) simplifies matters:

\[\gamma_{k}Y_{k}=\sum_{\tau=0}^{T-1}\sigma_{xx}\left(\tau-k\right)Y_{\tau}=\sum_{\tau=-k}^{T-1-\tau}\sigma_{xx}\left(\tau\right)Y_{\tau+k}\]

Recall that \(k\) represents the lag index and it can go from \(0\) to \(T-1\): \(\left(0\le k\le T-1\right)\).

Now, a last trick: assume that \(Y_{k}\) has some periodicity. That means that the values \(Y_{k}\) would start repeating after some period. Even if we do not think that condition is true, the period can be made as large as possible so that, in practice, no periodic values would be observed.

Considering the periodicity assumption for \(Y_{k}\), let’s say \(Y_{k}=e^{-j2\pi\frac{k}{T}\tau}\) for some \(\tau\). This allows us to define the \(k\)-th eigenvalue as

\[\gamma_{k}e^{-j2\pi\frac{k^{2}}{T}}=\sum_{\tau=0}^{T-1}\sigma_{xx}\left(\tau\right)e^{-j2\pi\frac{k}{T}\left(\tau+k\right)}\]

Simplifying further,

\[\gamma_{k}=\sum_{\tau=0}^{T-1}\sigma_{xx}\left(\tau\right)e^{-j2\pi\frac{k}{T}\tau}\]

Now, replacing \(\omega=2\pi\frac{k}{T}\), we notice striking similarities with the PSD equation:

\[X\left(\omega\right)=\sum_{t=0}^{T-1}\sigma_{xx}\left(\tau\right)e^{-j\omega t}\]

This expression establishes a deeper connection- the PSD is also the set of eigenvalues of the Fourier transform of the covariance matrix.

In essence, the matrix’s spectrum and a time series’ spectrum share a special link that was unclear to me for a long time. And I would try to explain why it is still amusing to me:

  • In my mind, I picture a matrix as a fixed object, as something that holds information in rows and columns. This object has nice descriptive elements: the eigenvalues/eigenvectors. We can calculate trace and determinants with them. And we can check if systems will be stable using them too. Yet, a matrix, per se, remains unchanged unless it is decided to be time-varying.

  • On the other hand, we have a time series, a perpetually fluctuating entity, ever-changing from one moment to the next. And a crucial information of this time-living object is the spectrum.

    • In this time series, a matrix emerges when you calculate the average correlation between each point in the signal with any other past value and organize this information into rows and columns. Again, this matrix is a fixed, non-time-varying entity. But, the spectrum of this newly-constructed matrix mirrors the spectrum of the original time series.