This post is part of a new series for this blog, Note to Self, where I collect together some notes about an idea related to my research. This content may be much more technical than most of the content of this blog and of much less wide interest. My hope in sharing this is that someone will find this interesting and useful for their own work.
This post is about a fundamental tool of high-dimensional probability, the Hanson–Wright inequality. The Hanson–Wright inequality is a concentration inequality for quadratic forms of random vectors—that is, expressions of the form where is a random vector. Many statements of this inequality in the literature have an unspecified constant ; our goal in this post will be to derive a fairly general version of the inequality with only explicit constants.
The name subgaussian is appropriate as the tail probabilities of Gaussian random variables exhibit the same square-exponential decrease .
A (non-obvious) fact is that if is subgaussian in the sense (1) and centered (), then ‘s cumulant generating function (cgf)
is subquadratic: There is a constant (independent of and ), for which
Since properties (1) and (2) are equivalent (up to a change in the parameter ), we are free to fix a version of property (2) as our definition for a (centered) subgaussian random variable.
Definition (subgaussian random variable): A centered random variable is said to be -subgaussian or subgaussian with variance proxy if its cgf is subquadratic:
For instance, a mean-zero Gaussian random variable with variance has cgf
and is thus subgaussian with variance proxy equal to its variance.
Theorem (Hanson–Wright): Let be a random vector with independent centered -subgaussian entries and let be a square matrix. Then
where is a constant (not depending on , , , or ).2Here, and denote the Frobenius and spectral norms.
This type of concentration is exactly the same type as provided by Bernstein’s inequality (which I discussed in my post on concentration inequalities). In particular, for small deviations , the tail probabilities decay are subgaussian with variance proxy :
For large deviations , this switches to subexponential tail probabilities with decay rate :
Now we come to a first version of the Hanson–Wright inequality with explicit constants, first for a matrix which is diagonal-free—that is, having all zeros on the diagonal. I obtained this version of the inequality myself, though I am very sure that this version of the inequality or an improvement thereof appears somewhere in the literature.
Theorem (Hanson–Wright, explicit constants, diagonal-free): Let random vector with independent centered -subguassian entries and let be a diagonal-free square matrix. Then we have the cgf bound
As a consequence, we have the concentration bound
Let us begin proving this result. Our proof will follow the same steps as Vershynin’s proof in High-Dimensional Probability (which in turn is adapted from an article by Rudelson and Vershynin), but taking care to get explicit constants. Unfortunately, proving all of the relevant tools from first principles would easily triple the length of this post, so I make frequent use of results from the literature.
We seek to compare the bilinear form to the Gaussian bilinear form where and are independent standard Gaussian vectors. We begin with the following cgf bound for the Gaussian quadratic form :
This equation is the result of Example 2.12 in Boucheron, Lugosi, and Massart’s Concentration Inequalities. By applying this result to the Hermitian dilation of in ‘s place, one obtains a similar result for the decoupled bilinear form :
We now seek to compare to . To do this, we first evaluate the cgf of only over the randomness in . Since we’re only taking an expectation over the random variable , we can apply the subquadratic tail condition (3) to obtain
Now we perform a similar computation for the quantity in which has been replaced by the Gaussian vector :
We stress that this is an equality since the cgf of a Gaussian random variable is given by (4). Thus we can substitute the left-hand side of the above display into the right-hand side of (7), yielding
We now perform this same trick again using the randomness in :
Packaging up (8) and (9) gives
Combining all these results (5), (6), and (10), we obtain
Fact (Bernstein concentration from Bernstein cgf bound): Suppose that a random variable satisfies the cgf bound for . Then
Now, here’s a more general result (with worse constants) which permits the matrix to possess a diagonal.
Theorem (Hanson–Wright, explicit constants): Let random vector with independent centered -subguassian entries and let be an arbitrary square matrix. Then we have the cgf bound
As a consequence, we have the concentration bound
Decompose the matrix into its diagonal and off-diagonal portions. For any two random variables and (possibly highly dependent), we can bound the cgf of their sum using the following “union bound”:
The two equality statements are the definition of the cumulant generating function and the inequality is Cauchy–Schwarz.
Using the “union bound”, it is sufficient to obtain bounds for the cgfs of the diagonal and off-diagonal parts and . We begin with the diagonal part. We compute
Substituting this result into (12) gives
For the second inequality, we used the facts that and .
We now look at the off-diagonal part . We use a version of the decoupling bound (5) where we compare to , where we’ve both replaced one copy of with an independent copy and reinstated the diagonal of (see Remark 6.1.3 in Vershynin’s High-Dimensional Probability):
We can now just repeat the rest of the argument for the diagonal-free Hanson–Wright inequality, yielding the same conclusion
Combining (11), (13), and (14), we obtain
As with above, this cgf bound implies the desired probability bound.