This is the second of a sequence of two posts on sketching, which I’m doing on the occasion of my new paper on the numerical stability of the iterative sketching method. For more on what sketching is and how it can be used to solve computational problems, I encourage you to check out the first post.
The goals of this post are more narrow. I seek to answer the question:
Which sketching matrix should I use?
To cut to the chase, my answer to this question is:
Sparse sign embeddings are a sensible default for sketching.
There are certainly cases when sparse sign embeddings are not the best type of sketch to use, but I hope to convince you that they’re a good sketching matrix to use for most purposes.
Let’s start things off with some numerical experiments.1Code for all numerical experiments can be found on the blogpost branch of the Github for my recent paper. We’ll compare three types of sketching matrices: Gaussian embeddings, a subsampled randomized trigonometric transform (SRTT), and sparse sign embeddings. See the last post for descriptions of these sketching matrices. I’ll discuss a few additional types of sketching matrices that require more discussion at the end of this post.
Recall that a sketching matrix seeks to compress a high-dimensional matrix or vector to a lower-dimensional sketched matrix or vector . The quality of a sketching matrix for a matrix is measured by its distortion , defined to be the smallest number such that
Here, denotes the column space of the matrix .
We begin with timing test. We will measure three different times for each embedding:
- Construction. The time required to generate the sketching matrix .
- Vector apply. The time to apply the sketch to a single vector.
- Matrix apply. The time to apply the sketch to an matrix.
We will test with input dimension (one million) and output dimension . For the SRTT, we use the discrete cosine transform as our trigonometric transform. For the sparse sign embedding, we use a sparsity parameter .
Here are the results (timings averaged over 20 trials):
Our conclusions are as follows:
- Sparse sign embeddings are definitively the fastest to apply, being 3–20× faster than the SRTT and 74–100× faster than Gaussian embeddings.
- Sparse sign embeddings are modestly slower to construct than the SRTT, but much faster to construct than Gaussian embeddings.
Overall, the conclusion is that sparse sign embeddings are the fastest sketching matrices by a wide margin: For an “end-to-end” workflow involving generating the sketching matrix and applying it to a matrix , sparse sign embeddings are 14× faster than SRTTs and 73× faster than Gaussian embeddings.2More timings are reported in Table 1 of this paper, which I credit for inspiring my enthusiasm for the sparse sign embedding l.
Runtime is only one measure of the quality of a sketching matrix; we also must care about the distortion . Fortunately, for practical purposes, Gaussian embeddings, SRTTs, and sparse sign embeddings all tend to have similar distortions. Therefore, we are free to use the sparse sign embeddings, as they as typically are the fastest.
Consider the following test. We generate a sparse random test matrix of size for and using the MATLAB sprand function; we set the sparsity level to 1%. We then compare the distortion of Gaussian embeddings, SRTTs, and sparse sign embeddings across a range of sketching dimensions between 100 and 10,000. We report the distortion averaged over 100 trials. The theoretically predicted value (equivalently, ) is shown as a dashed line.
To me, I find these results remarkable. All three embeddings exhibit essentially the same distortion parameter predicted by the Gaussian theory.
It would be premature to declare success having only tested on one type of test matrix . Consider the following four test matrices:
- Sparse: The test matrix from above.
- Dense: is taken to be a matrix with independent standard Gaussian random values.
- Khatri–Rao: is taken to be the Khatri–Rao product of three Haar random orthogonal matrices.
- Identity: is taken to be the identity matrix stacked onto a matrix of zeros.
The performance of sparse sign embeddings (again with sparsity parameter ) is shown below:
We see that for the first three test matrices, the performance closely follows the expected value . However, for the last test matrix “Identity”, we see the distortion begins to slightly exceed this predicted distortion for .
To improve sparse sign embeddings for higher values of , we can increase the value of the sparsity parameter . We recommend
With this higher sparsity level, the distortion closely tracks for all four test matrices:
Implemented appropriately (see below), sparse sign embeddings can be faster than other sketching matrices by a wide margin. The parameter choice is enough to ensure the distortion closely tracks for most test matrices. For the toughest test matrices, a slightly larger sparsity parameter can be necessary to achieve the optimal distortion.
While these tests are far from comprehensive, they are consistent with the uniformly positive results for sparse sign embeddings reported in the literature. We believe that this evidence supports the argument that sparse sign embeddings are a sensible default sketching matrix for most purposes.
Sparse Sign Embeddings: Theory and Practice
Given the highly appealing performance characteristics of sparse sign embeddings, it is worth saying a few more words about these embeddings and how they perform in both theory and practice.
Recall that a sparse sign embedding is a random matrix of the form
Each column is an independent and randomly generated to contain exactly nonzero entries in uniformly random positions. The value of each nonzero entry of is chosen to be either or with 50/50 odds.
The goal of sketching is to reduce vectors of length to a smaller dimension . For linear algebra applications, we typically want to preserve all vectors in the column space of a matrix up to distortion :
To use sparse sign embeddings, we must choose the parameters appropriately:
Given a dimension and a target distortion , how do we pick and ?
Based on the experiments above (and other testing reported in the literature), we recommend the following parameter choices in practice:
The parameter choice is advocated by Tropp, Yurtever, Udell, and Cevher; they mention experimenting with parameter values as small as . The value has demonstrated deficiencies and should almost always be avoided (see below). The scaling is derived from the analysis of Gaussian embeddings. As Martinsson and Tropp argue, the analysis of Gaussian embeddings tends to be reasonably descriptive of other well-designed random embeddings.
The best-known theoretical analysis, due to Cohen, suggests more cautious parameter setting for sparse sign embeddings:
The main difference between Cohen’s analysis and the parameter recommendations above is the presence of the factor and the lack of explicit constants in the O-notation.
If a sparse matrix library is unavailable, then either pursue a dedicated implementation or use a different type of embedding; sparse sign embeddings are just as slow as Gaussian embeddings if they are stored in an ordinary non-sparse matrix format!
Even with a sparse matrix format, it can require care to generate and populate the random entries of the matrix . Here, for instance, is a simple function for generating a sparse sign matrix in MATLAB:
function S = sparsesign_slow(d,n,zeta)
cols = kron((1:n)',ones(zeta,1)); % zeta nonzeros per column
vals = 2*randi(2,n*zeta,1) - 3; % uniform random +/-1 values
rows = zeros(n*zeta,1);
for i = 1:n
rows((i-1)*zeta+1:i*zeta) = randsample(d,zeta);
S = sparse(rows, cols, vals / sqrt(zeta), d, n);
Here, we specify the rows, columns, and values of the nonzero entries before assembling them into a sparse matrix using the MATLAB sparse command. Since there are exactly nonzeros per column, the column indices are easy to generate. The values are uniformly and can also be generated using a single line. The real challenge to generating sparse sign embeddings in MATLAB is the row indices, since each batch of row indices much be chosen uniformly at random between and without replacement. This is accomplished in the above code by a for loop, generating row indices at a time using the slow randsample function.
As its name suggests, the sparsesign_slow is very slow. To generate a sparse sign embedding with sparsity requires 53 seconds!
Fortunately, we can do (much) better. By rewriting the code in C and directly generating the sparse matrix in the CSC format MATLAB uses, generating this same 200 by 10 million sparse sign embedding takes just 0.4 seconds, a speedup of 130× over the slow MATLAB code. A C implementation of the sparse sign embedding that can be used in MATLAB using the MEX interface can be found in this file in the Github repo for my recent paper.
Other Sketching Matrices
Let’s leave off the discussion by mentioning other types of sketching matrices not considered in the empirical comparison above.
Another family of sketching matrices that we haven’t talked about are coordinate sampling sketches. A coordinate sampling sketch consists of indices and weights . To apply , we sample the indices and reweight them using the weights:
Coordinate sampling is very appealing: To apply to a matrix or vector requires no matrix multiplication of trigonometric transforms, just picking out some entries or rows and rescaling them.
In order for coordinate sampling to be effective, we need to pick the right indices. Below, we compare two coordinate sampling sketching approaches, uniform sampling and leverage score sampling (both with replacement), to the sparse sign embedding with the suggested parameter setting for the hard “Identity” test matrix used above.
We see right away that the uniform sampling fails dramatically on this problem. That’s to be expected. All but 50 of 100,000 rows of are zero, so picking rows uniformly at random will give nonsense with very high probability. Uniform sampling can work well for matrices which are “incoherent”, with all rows of being of “similar importance”.
Conclusion (Uniform sampling). Uniform sampling is a risky method; it works excellently for some problems, but fails spectacularly for others. Use only with caution!
The ridge leverage score sampling method is more interesting. Unlike all the other sketches we’ve discussed in this post, ridge leverage score sampling is data-dependent. First, it computes a leverage score for each row of and then samples rows with probabilities proportional to these scores. For high enough values of , ridge leverage score sampling performs slightly (but only slightly) worse than the characteristic scaling we expect for an oblivious subspace embedding.
Ultimately, leverage score sampling has two disadvantages when compared with oblivious sketching matrices:
- Higher distortion, higher variance. The distortion of a leverage score sketch is higher on average, and more variable, than an oblivious sketch, which achieve very consistent performance.
- Computing the leverage scores. In order to implement this sketch, the leverage scores have to first be computed or estimated. This is a nontrivial algorithmic problem; the most direct way of computing the leverage scores requires a QR decomposition at cost, much higher than other types of sketches.
There are settings when coordinate sampling methods, such as leverage scores, are well-justified:
- Structured matrices. For some matrices , the leverage scores might be very cheap to compute or approximate. In such cases, coordinate sampling can be faster than oblivious sketching.
- “Active learning”. For some problems, each entry of the vector or row of the matrix may be expensive to generate. In this case, coordinate sampling has the distinct advantage that computing or only requires generating the entries of or rows of for the randomly selected indices .
Ultimately, oblivious sketching and coordinate sampling both have their place as tools in the computational toolkit. For the reasons described above, I believe that oblivious sketching should usually be preferred to coordinate sampling in the absence of a special reason to prefer the latter.
Tensor Random Embeddings
There are a number of sketching matrices with tensor structure; see here for a survey. These types of sketching matrices are very well-suited to tensor computations. If tensor structure is present in your application, I would put these types of sketches at the top of my list for consideration.
The CountSketch sketching matrix is the case of the sparse sign embedding. CountSketch has serious deficiencies, and should only be used in practice with extreme care.
Consider the “Identity” test matrix from above but with parameter , and compare the distortion of CountSketch to the sparse sign embedding with parameters :
We see that the distortion of the CountSketch remains persistently high at 100% until the sketching dimension is taken , 20× higher than .
CountSketch is bad because it requires to be proportional to in order to achieve distortion . For all of the other sketching matrices we’ve considered, we’ve only required to be proportional to (or perhaps ). This difference between for CountSketch and for other sketching matrices is a at the root of CountSketch’s woefully bad performance on some inputs.3Here, the symbol is an informal symbol meaning “proportional to”.
The fact that CountSketch requires is simple to show. It’s basically a variant on the famous birthday problem. We include a discussion at the end of this post.4In fact, any oblivious sketching matrix with only 1 nonzero entry per column must have . This is Theorem 16 in the following paper.
There are ways of fixing the CountSketch. For instance, we can use a composite sketch , where is a CountSketch of size and is a Gaussian sketching matrix of size .5This construction is from this paper. For most applications, however, salvaging CountSketch doesn’t seem worth it; sparse sign embeddings with even nonzeros per column are already way more effective and reliable than a plain CountSketch.
By now, sketching is quite a big field, with dozens of different proposed constructions for sketching matrices. So which should you use? For most use cases, sparse sign embeddings are a good choice; they are fast to construct and apply and offer uniformly good distortion across a range of matrices.