{"id":1302,"date":"2022-10-11T15:30:07","date_gmt":"2022-10-11T15:30:07","guid":{"rendered":"https:\/\/www.ethanepperly.com\/?p=1302"},"modified":"2022-10-11T15:30:10","modified_gmt":"2022-10-11T15:30:10","slug":"low-rank-approximation-toolbox-nystrom-approximation","status":"publish","type":"post","link":"https:\/\/www.ethanepperly.com\/index.php\/2022\/10\/11\/low-rank-approximation-toolbox-nystrom-approximation\/","title":{"rendered":"Low-Rank Approximation Toolbox: Nystr\u00f6m Approximation"},"content":{"rendered":"\n

<\/p>\n\n\n\n

Welcome to a new series for this blog, Low-Rank Approximation Toolbox<\/em>. As I discussed in a previous post<\/a>, many matrices we encounter in applications are well-approximated by a matrix with a small rank<\/a>. Efficiently computing low-rank approximations has been a major area of research, with applications in everything from classical problems in computational physics<\/a> and signal processing<\/a> to trendy topics like data science<\/a>. In this series, I want to explore some broadly useful algorithms and theoretical techniques in the field of low-rank approximation.<\/p>\n\n\n\n

I want to begin this series by talking about one of the fundamental types of low-rank approximation, the Nystr\u00f6m approximation of a \"N\times N\" (real symmetric or complex Hermitian<\/a>) positive semidefinite<\/strong> (psd) matrix<\/a> \"A\". Given an arbitrary \"N\times k\" “test matrix” \"\Omega\", the Nystr\u00f6m approximation is defined to be<\/p>\n\n\n\n

(1) <\/span>   <\/span>\"\[A\langle \Omega\rangle := A\Omega \, (\Omega^*A\Omega)^{-1} \, \Omega^*A. \]\"<\/p><\/p>\n\n\n\n

This formula is sensible whenever \"\Omega^*A\Omega\" is invertible; if \"\Omega^*A\Omega\" is not invertible, then the inverse \"{}^{-1}\" should be replaced by the Moore\u2013Penrose pseudoinverse<\/a> \"{}^\dagger\". For simplicity, I will assume that \"\Omega^* A \Omega\" is invertible in this post, though everything we discuss will continue to work if this assumption is dropped. I use \"{}^*\" to denote the conjugate transpose<\/a> of a matrix, which agrees with the ordinary transpose \"{}^\top\" for real matrices. I will use the word self-adjoint to refer to a matrix which satisfies \"A=A^*\".<\/p>\n\n\n\n

The Nystr\u00f6m approximation (1) answers the question<\/p>\n\n\n\n

What is the “best” rank-\"k\" approximation to the psd matrx \"A\" provided only with the matrix\u2013matrix product \"A\Omega\", where \"\Omega\" is a known \"N\times k\" matrix (\"k\ll N\")?<\/p><\/blockquote>\n\n\n\n

Indeed, if we let \"Y = A\Omega\", we observe that the Nystr\u00f6m approximation can be written entirely using \"Y\" and \"\Omega\":<\/p>\n\n\n\n

  <\/span>   <\/span>\"\[A\langle \Omega\rangle = Y \, (\Omega^* Y)^{-1}\, Y^*.\]\"<\/p><\/p>\n\n\n\n

This is central advantage of the Nystr\u00f6m approximation: to compute it, the only<\/em> access to the matrix \"A\" I need is the ability to multiply the matrices \"A\" and \"\Omega\". In particular, I only need a single pass over the entries of \"A\" to compute the Nystr\u00f6m approximation. This allows the Nystr\u00f6m approximation to be used in settings when other low-rank approximations wouldn’t work, such as when \"A\" is streamed<\/em> to me as a sum of matrices that must be processed as they arrive and then discarded<\/a>.<\/p>\n\n\n\n

Choosing the Test Matrix<\/h2>\n\n\n\n

Every choice of \"N\times k\" test matrix \"\Omega\" defines a rank-\"k\" Nystr\u00f6m approximation1<\/a><\/sup>Actually, \"A\langle \Omega\rangle\" is only rank at most<\/em> \"k\". For this post, we will use rank-\"k\" to mean “rank at most \"k\"“.<\/span> \"A\langle \Omega\rangle\" by (1). Unfortunately, the Nystr\u00f6m approximation won’t be a good<\/em> low-rank approximation for every choice of \"\Omega\". For an example of what can go wrong, if we pick \"\Omega\" to have columns selected from the eigenvectors of \"A\" with small eigenvalues, the approximation \"A\langle \Omega\rangle\" will be quite poor.<\/p>\n\n\n\n

The very best choice of \"\Omega\" would be the \"k\" eigenvectors associated with the largest eigenvalues. Unfortunately, computing the eigenvectors to high accuracy is computationally costly. Fortunately, we can get decent low-rank approximations out of much simpler \"\Omega\"‘s:<\/p>\n\n\n\n

  1. Random:<\/strong> Perhaps surprisingly, we get a fairly low-rank approximation out of just choosing \"\Omega\" to be a random matrix, say, populated with statistically independent<\/a> standard normal<\/a> random entries. Intuitively, a random matrix is likely to have columns with meaningful overlap with the large-eigenvalue eigenvectors of \"A\" (and indeed with any \"k\" fixed orthonormal vectors). One can also pick more<\/a> exotic<\/a> kinds<\/a> of random matrices, which can have computational benefits.<\/li>
  2. Random then improve:<\/strong> The more similar the test matrix \"\Omega\" is to the large-eigenvalue eigenvectors of \"A\", the better the low-rank approximation will be. Therefore, it makes sense to use the power method<\/a> (usually called subspace iteration<\/a> in this context) to improve a random initial test matrix \"\Omega_{\rm init}\" to be closer to the eigenvectors of \"A\".2<\/a><\/sup>Even better than subspace iteration is block Krylov iteration<\/a>.<\/span> See section 11.6 of the following survey<\/a> for details.<\/li>
  3. Column selection:<\/strong> If \"\Omega\" consists of columns \"i_1,i_2,\ldots,i_k\" of the identity matrix, then \"A\Omega\" just consists of columns \"i_1,\ldots,i_k\" of \"A\": In MATLAB notation<\/a>,

      <\/span>   <\/span>\"\[A(:,\{i_1,\ldots,i_k\}) = A\Omega \quad \text{for}\quad \Omega = I(:,\{i_1,i_2,\ldots,i_k\}).\]\"<\/p> This is highly appealing as it allows us to approximate the matrix \"A\" by only reading a small fraction of its entries<\/em> (provided \"k\ll N\")! Producing a good low-rank approximation requires selecting the right column indices \"i_1,\ldots,i_k\" (usually under the constraint of reading a small number of entries from \"A\"). In my research with Yifan Chen<\/a>, Joel A. Tropp<\/a>, and Robert J. Webber<\/a>, I’ve argued that the most well-rounded algorithm for this task is a randomly pivoted partial Cholesky decomposition<\/a>.<\/li><\/ol>\n\n\n\n

    The Projection Formula<\/h2>\n\n\n\n

    Now that we’ve discussed the choice of test matrix, we shall explore the quality of the Nystr\u00f6m approximation as measured by the size of the residual<\/a> \"A - A\langle \Omega\rangle\". As a first step, we shall show that the residual is psd. This means that \"A\langle \Omega\rangle\" is an underapproximation<\/em> to \"A\".<\/p>\n\n\n\n

    The positive semidefiniteness of the residual follows from the following projection formula<\/em> for the Nystr\u00f6m approximation:<\/p>\n\n\n\n

      <\/span>   <\/span>\"\[A\langle \Omega \rangle = A^{1/2} P_{A^{1/2}\Omega} A^{1/2}.\]\"<\/p><\/p>\n\n\n\n

    Here, \"P_{A^{1/2}\Omega}\" denotes the the orthogonal projection<\/a> onto the column space<\/a> of the matrix \"A^{1/2}\Omega\". To deduce the projection formula, we break down \"A\" as \"A = A^{1/2}\cdot A^{1/2}\" in (1):<\/p><\/p>\n\n\n\n

      <\/span>   <\/span>\"\[A\langle \Omega\rangle = A^{1/2} \left( A^{1/2}\Omega \left[ (A^{1/2}\Omega)^* A^{1/2}\Omega \right]^{-1} (A^{1/2}\Omega)^* \right) A^{1/2}.\]\"<\/p><\/p>\n\n\n\n

    The fact that the paranthesized quantity is \"P_{A^{1/2}\Omega}\" can be verified in a variety of ways, such as by QR factorization<\/a>.3<\/a><\/sup>Let \"A^{1/2} \Omega = QR\", where \"Q\" has orthonormal columns and \"R\" is square and upper triangular. The orthogonal projection is \"P_{A^{1/2}\Omega} = QQ^*\". The parenthesized expression is \"(QR)(R^*Q^*QR)^{-1}R^*Q^* = QRR^{-1}R^{-*}R^*Q^* = QQ^* = P_{A^{1/2}\Omega}\".<\/span>\n\n\n\n

    With the projection formula in hand, we easily obtain the following expression for the residual:<\/p>\n\n\n\n

      <\/span>   <\/span>\"\[A - A\langle \Omega\rangle = A^{1/2} (I - P_{A^{1/2}\Omega}) A^{1/2}.\]\"<\/p><\/p>\n\n\n\n

    To show that this residual is psd, we make use of the conjugation rule<\/a>.<\/p>\n\n\n\n

    Conjugation rule:<\/strong> For a matrix \"B\" and a self-adjoint matrix \"H\", if \"H\" is psd then \"B^*HB\" is psd. If \"B\" is invertible, then the converse holds: if \"B^*HB\" is psd, then \"H\" is psd.<\/p><\/blockquote>\n\n\n\n

    The matrix \"I - P_{A^{1/2}\Omega}\" is an orthogonal projection and therefore psd. Thus, by the conjugation rule, the residual of the is Nystr\u00f6m approximation is psd:<\/p>\n\n\n\n

      <\/span>   <\/span>\"\[A - A\langle \Omega\rangle = \left(A^{1/2}\right)^* (I-P_{A^{1/2}\Omega})A^{1/2} \quad \text{is psd}.\]\"<\/p><\/p>\n\n\n\n

    Optimality of the Nystr\u00f6m Approximation<\/h2>\n\n\n\n

    There’s a question we’ve been putting off that can’t be deferred any longer:<\/p>\n\n\n\n

    Is the Nystr\u00f6m approximation actually a good<\/em> low-rank approximation?<\/p>\n\n\n\n

    As we discussed earlier, the answer to this question depends on the test matrix \"\Omega\". Different choices for \"\Omega\" give different approximation errors. See the<\/a> following<\/a> papers<\/a> for Nystr\u00f6m approximation error bounds with different choices of \"\Omega\". While the Nystr\u00f6m approximation can be better or worse depending on the choice of \"\Omega\", what is true about Nystr\u00f6m approximation for every<\/em> choice of \"\Omega\" is that:<\/p>\n\n\n\n

    The Nystr\u00f6m approximation \"A\langle \Omega\rangle\" is the best possible<\/em> rank-\"k\" approximation to \"A\" given the information \"A\Omega\".<\/p>\n\n\n\n

    In precise terms, I mean the following:<\/p>\n\n\n\n

    Theorem:<\/strong> Out of all self-adjoint matrices \"\hat{A}\" spanned by the columns of \"A\Omega\" with a psd residual \"A - \hat{A}\", the Nystr\u00f6m approximation has the smallest error as measured by either the spectral<\/a> or Frobenius<\/a> norm<\/a> (or indeed any unitarily invariant norm<\/a>, see below).<\/p><\/blockquote>\n\n\n\n

    Let’s break this statement down a bit. This result states that the Nystr\u00f6m approximation is the best approximation \"\hat{A}\" to \"A\" under three conditions:<\/p>\n\n\n\n

    1. \"\hat{A}\" is self-adjoint.<\/li>
    2. \"\hat{A}\" is spanned by the columns of \"A\Omega\".<\/li><\/ol>\n\n\n\n

      I find these first two requirements to be natural. Since \"A\" is self-adjoint, it makes sense to require our approximation \"\hat{A}\" to be as well. The stipulation that \"\hat{A}\" is spanned by the columns \"A\Omega\" seems like a very natural requirement given we want to think of approximations which only use the information \"A\Omega\". Additionally, requirement 2 ensures that \"\hat{A}\" has rank at most \"k\", so we are really only considering low-rank approximations to \"A\".<\/p>\n\n\n\n

      The last requirement is less natural:<\/p>\n\n\n\n

      1. The residual \"A - \hat{A}\" is psd.<\/li><\/ol>\n\n\n\n

        This is not an obvious requirement to impose on our approximation. Indeed, it was a nontrivial calculation using the projection formula to show that the Nystr\u00f6m approximation itself satisfies this requirement! Nevertheless, this third stipulation is required to make the theorem true. The Nystr\u00f6m approximation (1) is the best “underapproximation” to the matrix \"A\" to in the span of \"A\Omega\".<\/p>\n\n\n\n

        Intermezzo: Unitarily Invariant Norms and the Psd Order<\/h2>\n\n\n\n

        To prove our theorem about the optimality of the Nystr\u00f6m approximation, we shall need two ideas from matrix theory: unitarily invariant norms and the psd order. We shall briefly describe each in turn.<\/p>\n\n\n\n

        A norm \"\left\|\cdot\right\|_{\rm UI}\" defined on the set of \"N\times N\" matrices is said to be unitarily invariant<\/strong> if the norm of a matrix does not change upon left- or right-multiplication by a unitary matrix<\/a>:<\/p>\n\n\n\n

          <\/span>   <\/span>\"\[\left\|UBV\right\|_{\rm UI} = \left\|B\right\|_{\rm UI} \quad \text{for all unitary matrices $U$ and $V$.}\]\"<\/p><\/p>\n\n\n\n

        Recall that a unitary matrix \"U\" (called a real orthogonal matrix<\/a> if \"U\" is real-valued) is one that obeys \"U^*U = UU^* = I\". Unitary matrices preserve the Euclidean lengths<\/a> of vectors, which makes the class of unitarily invariant norms highly natural. Important examples include the spectral<\/a>, Frobenius<\/a>, and nuclear<\/a> matrix norms:<\/p>\n\n\n\n

        The unitarily invariant norm of a matrix \"B\" depends entirely on its singular values<\/a> \"\sigma(B)\". For instance, the spectral, Frobenius, and nuclear norms take the forms<\/p>\n\n\n

          <\/span>   <\/span>\"\begin{align*}\left\|B\right\|_{\rm op} &= \sigma_1(B),& &\text{(spectral)} \\\left\|B\right\|_{\rm F} &= \sqrt{\sum_{j=1}^N (\sigma_j(B))^2},& &\text{(Frobenius)} \\\left\|B\right\|_{*} &=\sum_{j=1}^N \sigma_j(B)).& &\text{(nuclear)}\end{align*}\"<\/p><\/p>\n\n\n

        In addition to being entirely determined by the singular values, unitarily invariant norms are non-decreasing<\/em> functions of the singular values: If the \"j\"th singular value of \"B\" is larger than the \"j\"th singular value of \"C\" for \"1\le j\le N\", then \"\left\|B\right\|_{\rm UI}\ge \left\|C\right\|_{\rm UI}\" for every unitarily invariant norm \"\left\|\cdot\right\|_{\rm UI}\". For more on unitarily invariant norms, see this short and information-packed blog post<\/a> from Nick Higham<\/a>.<\/p>\n\n\n\n

        Our second ingredient is the psd order<\/strong> (also known as the Loewner order<\/strong>). A self-adjoint matrix \"A\" is larger than a self-adjoint matrix \"H\" according to the psd order, written \"A\succeq H\", if the difference \"A-H\" is psd. As a consequence, \"A\succeq 0\" if and only if \"A\" is psd, where \"0\" here denotes the zero matrix of the same size as \"A\". Using the psd order, the positive semidefiniteness of the Nystr\u00f6m residual can be written as \"A - A\langle \Omega\rangle \succeq 0\".<\/p>\n\n\n\n

        If \"A\" and \"H\" are both psd matrices and \"A\" is larger than \"H\" in the psd order, \"A\succeq H\succeq 0\", it seems natural to expect that \"A\" is larger than \"H\" in norm. Indeed, this intuitive statement is true, at least when one restricts oneself to unitarily invariant norms.<\/p>\n\n\n\n

        Psd order and norms.<\/strong> If \"A\succeq H\succeq 0\", then \"\left\|A\right\|_{\rm UI} \ge \left\|H\right\|_{\rm UI}\" for every unitarily invariant norm \"\left\|\cdot\right\|_{\rm UI}\".<\/p><\/blockquote>\n\n\n\n

        This fact is a consequence of the following observations:<\/p>\n\n\n\n