{"id":2154,"date":"2025-08-04T23:57:01","date_gmt":"2025-08-04T23:57:01","guid":{"rendered":"https:\/\/www.ethanepperly.com\/?p=2154"},"modified":"2026-02-13T16:44:07","modified_gmt":"2026-02-13T16:44:07","slug":"gaussian-integration-by-parts","status":"publish","type":"post","link":"https:\/\/www.ethanepperly.com\/index.php\/2025\/08\/04\/gaussian-integration-by-parts\/","title":{"rendered":"Gaussian Integration by Parts"},"content":{"rendered":"\n

Gaussian random variables<\/a> are wonderful, and there are lots of clever tricks for doing computations with them. One particularly nice tool is the Gaussian integration by parts formula<\/a>, which I learned from my PhD advisor Joel Tropp<\/a>. Here it is:<\/p>\n\n\n\n

\n

Gaussian integration by parts.<\/strong> Let \"z\" be a standard<\/a> Gaussian random variable. Then \"\expect[zf(z)] = \expect[f'(z)]\".<\/p>\n<\/blockquote>\n\n\n\n

This formula makes many basic computations effortless. For instance, to compute the second moment<\/a> \"\expect[z^2]\" of a standard Gaussian random variable \"z\", we apply the formula with \"f(x) = x\" to obtain

  <\/span>   <\/span>\"\[\expect[z^2] = \expect[z f(z)] = \expect[f'(z)] = \expect[1] = 1.\]\"<\/p>Therefore, the second moment is one. Since \"z\" has mean zero, this also means that the variance<\/a> of a standard Gaussian random variable is one.<\/p>\n\n\n\n

The fourth moment is no harder to compute. Using \"f(x) = x^3\", we compute

  <\/span>   <\/span>\"\[\expect[z^4] = \expect[zf(z)] = \expect[f'(z)] = 3\expect[z^2] = 3.\]\"<\/p>Easy peesy. We’ve shown the fourth moment of a standard Gaussian variable is three\u2014no fancy integral tricks required. <\/p>\n\n\n\n

Iterating this trick, we can compute all the even moments of a standard Gaussian random variable<\/a>. Indeed,

  <\/span>   <\/span>\"\[\expect[z^{2p}] = \expect[z\cdot z^{2p-1}] = (2p-1) \expect[z^{2p-2}] = (2p-1)(2p-3) \expect[z^{2p-4}] = \cdots = (2p-1)(2p-3)\cdots 1.\]\"<\/p>We conclude that the \"(2p)\"th moment of a standard Gaussian random variable is \"(2p-1)!!\", where \"!!\" indicates the (in)famous double factorial<\/a> \"(2p-1)!! = (2p-1)\times(2p-3)\times\cdots \times 3 \times 1\".<\/p>\n\n\n\n

As a spicier application, let us now compute \"\expect[|z|]\". To do so, we choose \"f(x) = \operatorname{sign}(x)\" to be the sign function<\/a>:

  <\/span>   <\/span>\"\[\operatorname{sign}(x) = \begin{cases} 1, & x > 0, \\ 0 & x = 0, \\ -1 & x < 0.\end{cases}\]\"<\/p>This function is not differentiable in a “Calculus 1” sense because it is discontinuous at zero. However, it is differentiable in a “distributional sense”<\/a> and its derivative is \"f'(x) = 2\delta(x)\", where \"\delta\" denotes the famous Dirac delta “function”<\/a>. We then compute

  <\/span>   <\/span>\"\[\expect[|z|] = \expect[zf(z)] = \expect[f'(z)] = 2\expect[\delta(z)].\]\"<\/p>To compute \"\expect[\delta(z)]\", write the integral out using the probability density function<\/a> \"\phi\" of the standard Gaussian distribution:

  <\/span>   <\/span>\"\[\expect[|z|] = 2\expect[\delta(z)] = 2\int_{-\infty}^\infty \phi(x) \delta(x) \, \mathrm{d} x = 2\phi(0).\]\"<\/p>The standard Gaussian distribution has density<\/a>

  <\/span>   <\/span>\"\[\phi(x) = \frac{1}{\sqrt{2\pi}} \exp \left( - \frac{x^2}{2} \right),\]\"<\/p> so we conclude

  <\/span>   <\/span>\"\[\expect[|z|] = 2 \cdot \frac{1}{\sqrt{2 \pi}} = \sqrt{\frac{2}{\pi}}}.\]\"<\/p>A computation involving integrating against the Gaussian density has again been made trivial by using the Gaussian integration by parts formula.<\/p>\n\n\n\n

Application: Power Method from a Random Start<\/h2>\n\n\n\n

As an application of the Gaussian integration by parts formula, we can analyze the famous power method<\/a> for eigenvalue computations with a (Gaussian) random initialization. This discussion is adapted from the tutorial of Kireeva and Tropp (2024)<\/a>.<\/p>\n\n\n\n

Setup<\/h3>\n\n\n\n

Before we can get to the cool application of the Gaussian integration by parts formula, we need to setup the problem and do a bit of algebra. Let \"A\" be a matrix, which we’ll assume for simplicity to be symmetric<\/a> and positive semidefinite<\/a>. Let \"\lambda_1 > \lambda_2 \ge \lambda_3 \ge \cdots \ge \lambda_n \ge 0\" denote the eigenvalues of \"A\". We assume the largest eigenvalue \"\lambda_1\" is strictly<\/mark><\/em> larger than the next eigenvalue \"\lambda_2\". <\/p>\n\n\n\n

The power method computes the largest eigenvalue of \"A\" by repeating the iteration \"x \gets Ax / \norm{Ax}\". After many iterations, \"x\" approaches an eigenvector of \"A\" and \"x^\top A x\" approaches an eigenvalue. Letting \"x^{(0)}\" denote the initial vector, the \"t\"th power iterate is \"x^{(t)} = A^t x^{(0)} / \norm{A^t x^{(0)}}\" and the \"t\"th eigenvalue estimate is

  <\/span>   <\/span>\"\[\mu^{(t)} = \left(x^{(t)}\right)^\top Ax^{(t)}= \frac{\left(x^{(0)}\right)^\top A^{2t+1} x^{(0)}}{\left(x^{(0)}\right)^\top A^{2t} x^{(0)}}.\]\"<\/p><\/p>\n\n\n\n

It is common to initialize the power method with a vector \"x^{(0)}\" with (independent) standard Gaussian random coordinates. In this case, the components \"z_1,\ldots,z_n\" of \"x^{(0)}\" in an eigenvector basis of \"A\" are also independent standard Gaussians, owing to the rotational invariance of the (standard multivariate) Gaussian distribution<\/a>. Then the \"t\"th eigenvalue estimate is

  <\/span>   <\/span>\"\[\mu^{(t)} = \frac{\sum_{i=1}^n \lambda_i^{2t+1} z_i^2}{\sum_{i=1}^n \lambda_i^{2t} z_i^2},\]\"<\/p>and the error of \"\mu^{(t)}\" as an approximation to the dominant eigenvalue \"\lambda_1\" is

  <\/span>   <\/span>\"\[\frac{\lambda_1 - \mu^{(t)}}{\lambda_1} = \frac{\sum_{i=1}^n (\lambda_1 - \lambda_i)/\lambda_1 \cdot \lambda_i^{2t} z_i^2}{\sum_{i=1}^n \lambda_i^{2t} z_i^2}.\]\"<\/p><\/p>\n\n\n\n

Analysis<\/h3>\n\n\n\n

Having set everything up, we can now use the Gaussian integration by parts formula to make quick work of the analysis. To begin, observe that the quantity

  <\/span>   <\/span>\"\[\frac{\lambda_1 - \lambda_i}{\lambda_1}\]\"<\/p>is zero for \"i = 1\" and is at most one for \"i > 1\". Therefore, the error is bounded as

  <\/span>   <\/span>\"\[\frac{\lambda_1 - \mu^{(t)}}{\lambda_1} \le \frac{\sum_{i=2}^n \lambda_i^{2t} z_i^2}{\lambda_1^{2t} z_1^2 + \sum_{i=2}^n \lambda_i^{2t} z_i^2} = \frac{c^2}{\lambda_1^{2t} z_1^2 + c^2} \quad \text{for } c^2 = \sum_{i=2}^n \lambda_i^{2t} z_i^2.\]\"<\/p>We have introduced a parameter \"c\" to consolidate the terms in this expression not depending on \"z_1\". Since \"z_1,\ldots,z_n\" are independent, \"z_1\" and \"c\" are independent as well.<\/p>\n\n\n\n

Now, let us bound the expected value of the error. First, we take an expectation \"\expect_{z_1}\" with respect to only the randomness in the first Gaussian variable \"z_1\". Here, we use Gaussian integration by parts in the reverse<\/em> direction. Introduce the function

  <\/span>   <\/span>\"\[f'(x) = \frac{c^2}{\lambda_1^{2t}x^2 + c^2}.\]\"<\/p>This function is the derivative of

  <\/span>   <\/span>\"\[f(x) = \frac{c}{\lambda_1^t} \arctan \left( \frac{\lambda_1^t}{c} \cdot x \right).\]\"<\/p>Thus, by the Gaussian integration by parts formula, we have

  <\/span>   <\/span>\"\[\expect_{z_1}\left[\frac{\lambda_1 - \mu^{(t)}}{\lambda_1}\right] \le \expect_{z_1} \left[z_1 \cdot \frac{c}{\lambda_1} \arctan \left( \frac{\lambda_1^t}{c} \cdot z_1 \right) \right] = \expect_{z_1} \left[|z_1| \cdot \frac{c}{\lambda_1^t} \left|\arctan \left( \frac{\lambda_1^t}{c} \cdot z_1 \right)\right| \right].\]\"<\/p>In the last line, we observed that the bracketed quantity is nonnegative, so we are free to introduce absolute value signs. The arctangent function is always at most \"\pi/2\", so we can bound

  <\/span>   <\/span>\"\[\expect_{z_1}\left[\frac{\lambda_1 - \mu^{(t)}}{\lambda_1}\right] \le = \frac{\pi}{2} \cdot \frac{c}{\lambda_1^t} \cdot \expect_{z_1} \left[|z_1|\right] \le \sqrt{\frac{\pi}{2}} \cdot \frac{c}{\lambda_1^t}.\]\"<\/p>Here, we used our result from above that \"\expect[|z|] = \sqrt{2/\pi}\" for a standard Gaussian variable \"z\".<\/p>\n\n\n\n

We’re in the home stretch! We can bound \"c\" as

  <\/span>   <\/span>\"\[c = \sqrt{\sum_{i=2} \lambda_i^{2t} z_i^2} \le \lambda_2^{t} \sqrt{\sum_{i=2}^n z_i^2} = \lambda_2^t \norm{(z_2,\ldots,z_n)}.\]\"<\/p>We see that \"c\" is at most \"\lambda_2^t\" times the length of a vector of \"n-1\" standard Gaussian entries. As we’ve seen before on this blog<\/a>, the expected length of a Gaussian vector with \"n-1\" entries is at most \"\sqrt{n-1}\". Thus, \"\expect[c] \le \lambda_2^t \sqrt{n-1}\". We conclude that the expected error for power iteration is

  <\/span>   <\/span>\"\[\expect\left[\frac{\lambda_1 - \mu^{(t)}}{\lambda_1}\right] = \expect_c\left[\expect_{z_1} \left[ \frac{\lambda_1 - \mu^{(t)}}{\lambda_1} \right]\right] \le \sqrt{\frac{\pi}{2}} \cdot \expect\left[ \frac{c}{\lambda_1^t} \right] =\sqrt{\frac{\pi}{2}} \cdot \left(\frac{\lambda_2}{\lambda_1}\right)^t \cdot \sqrt{n-1} .\]\"<\/p>We see that the power iteration converges geometrically at a rate of at least \"(\lambda_2/\lambda_1)^t\".<\/p>\n\n\n\n

The first analyses of power iteration from a random start were done by Kuczy\u0144ski and Wo\u017aniakowski (1992)<\/a> and require pages of detailed computations involving integrals. This simplified analysis, due to Tropp (2020)<\/a>, makes the analysis effortless by comparison.<\/p>\n","protected":false},"excerpt":{"rendered":"

Gaussian random variables are wonderful, and there are lots of clever tricks for doing computations with them. One particularly nice tool is the Gaussian integration by parts formula, which I learned from my PhD advisor Joel Tropp. Here it is: Gaussian integration by parts. Let be a standard Gaussian random variable. Then . This formulaRead more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-2154","post","type-post","status-publish","format-standard","hentry","category-expository"],"_links":{"self":[{"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/posts\/2154","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/comments?post=2154"}],"version-history":[{"count":16,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/posts\/2154\/revisions"}],"predecessor-version":[{"id":2301,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/posts\/2154\/revisions\/2301"}],"wp:attachment":[{"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/media?parent=2154"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/categories?post=2154"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/tags?post=2154"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}